Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

heplify-server stuck in restarting cycle #355

Closed
nakchak opened this issue Apr 21, 2020 · 1 comment
Closed

heplify-server stuck in restarting cycle #355

nakchak opened this issue Apr 21, 2020 · 1 comment

Comments

@nakchak
Copy link

nakchak commented Apr 21, 2020

While testing a fix for sipcapture/homer7-docker#73
I seem to have stumbled upon more issues, I have sucessfully updatedt he loki config locally to stay up using the new 0s syntax for zero length durations, but now im getting a panic logged every time heplify-server attempts to come up

Could not find toml config file, use flag defaults. <nil>
2020/04/21 14:13:26.414607 server.go:83: INFO start heplify-server 1.29 with config.HeplifyServer{HEPAddr:"0.0.0.0:9060", HEPTCPAddr:"", HEPTLSAddr:"0.0.0.0:9060", ESAddr:"", ESDiscovery:true, ESUser:"", ESPass:"", LokiURL:"http://loki:3100/api/prom/push", LokiBulk:400, LokiTimer:2, LokiBuffer:100000, LokiHEPFilter:[]int{1, 5, 100}, ForceHEPPayload:[]int(nil), PromAddr:"0.0.0.0:9096", PromTargetIP:"", PromTargetName:"", DBShema:"homer7", DBDriver:"postgres", DBAddr:"db:5432", DBUser:"root", DBPass:"homerSeven", DBDataTable:"homer_data", DBConfTable:"homer_config", DBBulk:400, DBTimer:4, DBBuffer:400000, DBWorker:8, DBRotate:true, DBPartLog:"2h", DBPartIsup:"6h", DBPartSip:"2h", DBPartQos:"6h", DBDropDays:5, DBDropDaysCall:0, DBDropDaysRegister:0, DBDropDaysDefault:0, DBDropOnStart:false, Dedup:false, DiscardMethod:[]string(nil), AlegIDs:[]string(nil), CustomHeader:[]string(nil), SIPHeader:[]string{"ruri_user", "ruri_domain", "from_user", "from_tag", "to_user", "callid", "cseq", "method", "user_agent"}, LogDbg:"rotator", LogLvl:"debug", LogStd:true, LogSys:false, Config:"./heplify-server.toml", ConfigHTTPAddr:"", ConfigHTTPPW:"", Version:false}
2020/04/21 14:13:26.414923 prometheus.go:38: INFO expose metrics without or unbalanced targets
panic: not implemented
        panic: not implemented

goroutine 76 [running]:
github.com/golang/protobuf/proto.(*InternalMessageInfo).Size(...)
        /go/src/github.com/golang/protobuf/proto/deprecated.go:91
github.com/sipcapture/heplify-server/remotelog/logproto.(*PushRequest).XXX_Size(0xc000054500, 0xed3740)
        /go/src/github.com/sipcapture/heplify-server/remotelog/logproto/logproto.pb.go:49 +0x3a
github.com/gogo/protobuf/proto.Marshal(0xed3740, 0xc000054500, 0xc0005db8f8, 0xc00001ec20, 0xcb2580, 0xbc6683, 0xbc7aaa)
        /go/src/github.com/gogo/protobuf/proto/table_marshal.go:2938 +0x278
github.com/sipcapture/heplify-server/remotelog.encodeBatch(0xc0005dbe98, 0x7f76cf61a000, 0x0, 0x4cab51, 0xc000090040, 0xc0005db9f0)
        /go/src/github.com/sipcapture/heplify-server/remotelog/loki.go:194 +0x1dc
github.com/sipcapture/heplify-server/remotelog.(*Loki).sendBatch(0xc0001a8e70, 0xc0005dbe98, 0x0, 0x0)
        /go/src/github.com/sipcapture/heplify-server/remotelog/loki.go:173 +0x3d
github.com/sipcapture/heplify-server/remotelog.(*Loki).start.func1(0xc0001a8e70, 0xc0005dbe28)
        /go/src/github.com/sipcapture/heplify-server/remotelog/loki.go:83 +0x3c
panic(0xc53b20, 0xeb6060)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/golang/protobuf/proto.(*InternalMessageInfo).Size(...)
        /go/src/github.com/golang/protobuf/proto/deprecated.go:91
github.com/sipcapture/heplify-server/remotelog/logproto.(*PushRequest).XXX_Size(0xc0000542c0, 0xed3740)
        /go/src/github.com/sipcapture/heplify-server/remotelog/logproto/logproto.pb.go:49 +0x3a
github.com/gogo/protobuf/proto.Marshal(0xed3740, 0xc0000542c0, 0xc0005dbc30, 0xc00001e010, 0xc0005dbd10, 0x43f283, 0xc0005dbec8)
        /go/src/github.com/gogo/protobuf/proto/table_marshal.go:2938 +0x278
github.com/sipcapture/heplify-server/remotelog.encodeBatch(0xc0005dbe98, 0xc000059320, 0xc0005dbef0, 0xc000059380, 0xc000138b08, 0xc000138bb8)
        /go/src/github.com/sipcapture/heplify-server/remotelog/loki.go:194 +0x1dc
github.com/sipcapture/heplify-server/remotelog.(*Loki).sendBatch(0xc0001a8e70, 0xc0005dbe98, 0x0, 0x0)
        /go/src/github.com/sipcapture/heplify-server/remotelog/loki.go:173 +0x3d
github.com/sipcapture/heplify-server/remotelog.(*Loki).start(0xc0001a8e70, 0xc00016c900)
        /go/src/github.com/sipcapture/heplify-server/remotelog/loki.go:161 +0x1373
github.com/sipcapture/heplify-server/remotelog.(*Remotelog).Run.func1(0xc00000c0e0)
        /go/src/github.com/sipcapture/heplify-server/remotelog/remotelog.go:36 +0x3c
created by github.com/sipcapture/heplify-server/remotelog.(*Remotelog).Run
        /go/src/github.com/sipcapture/heplify-server/remotelog/remotelog.go:35 +0x78

Looking at the file in question (loki.go) line 83 seems to be sending data to loki, i suspect that the payload is possibly using the old format for a duration and loki is rejecting it is that plausible? There is nothing logged in loki's logs other many "creating table" entries

@negbie
Copy link
Member

negbie commented Apr 21, 2020

Duplicate of #354

@negbie negbie marked this as a duplicate of #354 Apr 21, 2020
@negbie negbie closed this as completed Apr 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants