Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash in otel_metrics.go #907

Closed
kentquirk opened this issue Nov 29, 2023 · 1 comment
Closed

Crash in otel_metrics.go #907

kentquirk opened this issue Nov 29, 2023 · 1 comment
Labels
type: bug Something isn't working

Comments

@kentquirk
Copy link
Contributor

Using a current refinery, a user has reported a crash with fatal error: concurrent map read and map write. The stack trace points to github.com/honeycombio/[email protected]/transmission/transmission.go:228 which is a call to the metrics object. It appears that the conflicting call is related to an error returned from a transmission. This is the stack trace there:

net/http.(*persistConn).roundTrip(0xc4fcc64fc0, 0xc28887ea40)
	net/http/transport.go:2620 +0x974
net/http.(*Transport).roundTrip(0xc00053e140, 0xc2d8a92000)
	net/http/transport.go:595 +0x7ba
net/http.(*Transport).RoundTrip(0xc2d8a92000?, 0xecd920?)
	net/http/roundtrip.go:17 +0x19
net/http.send(0xc18ffbff00, {0xecd920, 0xc00053e140}, {0xd5d720?, 0x48b501?, 0x1439100?})
	net/http/client.go:251 +0x5f7
net/http.(*Client).send(0xc1482413e0, 0xc18ffbff00, {0x0?, 0xc09dcf5910?, 0x1439100?})
	net/http/client.go:175 +0x9b
net/http.(*Client).do(0xc1482413e0, 0xc18ffbff00)
	net/http/client.go:715 +0x8fc
net/http.(*Client).Do(...)
	net/http/client.go:581
github.com/honeycombio/libhoney-go/transmission.(*batchAgg).fireBatch(0xc288418800, {0xc521716000?, 0xed, 0x100})
	github.com/honeycombio/[email protected]/transmission/transmission.go:452 +0x916
github.com/honeycombio/libhoney-go/transmission.(*batchAgg).Fire(0xc288418800, {0x7fd13639e1b0?, 0xc000010120?})
	github.com/honeycombio/[email protected]/transmission/transmission.go:325 +0x109
created by github.com/facebookgo/muster.(*Client).worker.func1
	github.com/facebookgo/[email protected]/muster.go:133 +0xfb

I think this is probably relating to the map of Gauge metrics being both written by an error reporting function and read by the periodic metrics posting logic. But to date I have not been able to reproduce the error.

This is to document the error in case we see it again.

@kentquirk kentquirk added the type: bug Something isn't working label Nov 29, 2023
@fchikwekwe fchikwekwe self-assigned this Jan 2, 2024
@fchikwekwe fchikwekwe removed their assignment Jan 11, 2024
@kentquirk
Copy link
Contributor Author

We haven't seen this lately, so we think it's been fixed. Please reopen if it recurs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants