boomcatch sends only `rt` to telegraf (statsd) #111

rschoessler · 2019-03-28T14:57:47Z

#99 seems to have a similar problem, but that solution is not working for me.

Here's my setup:

boomerang -> boomcatch -> telegraf -> influxdb

My webpage is pretty simple and includes the following in the <head>:

 <script src="assets/js/boomerang/boomerang.js"></script>
  <script src="assets/js/boomerang/plugins/rt.js"></script>
  <script src="assets/js/boomerang/plugins/painttiming.js"></script>
  <script>
    BOOMR.init({
      beacon_url: "http://localhost:8080/beacon",
      PaintTiming: {
        enabled: true,
        clearOnBeacon: true
      }
    });
  </script>

It doesn't matter if I use painttiming.js or navtiming.js or usertiming.js, the only data that is ever sent BY boomcatch is rt.firstbyte, rt.lastbyte and rt.load:

2019-03-28 10:44:15 INFO boomcatch: sending rt.firstbyte:315|ms
rt.lastbyte:3159|ms
rt.load:3159|ms

2019-03-28 10:44:15 INFO boomcatch: sent 56 bytes

If I debug the browser/sources in developer tools and break on boomerang:3372:

BOOMR.sendBeaconData(varsSent);

I can see that the timings that I am requesting are being sent:

The rt data gets into my database, but nothing else.

Ideally I want to get usertiming as I have created custom user timing marks that I am most interested in

What am I missing??? Thanks in advance.

cc: @josebolos

The text was updated successfully, but these errors were encountered:

rschoessler · 2019-04-01T12:21:01Z

@josebolos would you be able to assist me with this? Asking directly as I see that you are the most (only?) contributor to this project of late. Thanks if you are able to help.

josebolos · 2019-04-04T15:51:13Z

Hi @rschoessler,

You haven't attached boomcatch's config so I'm just guessing here. What mapper are you using?

If you're using the default mapper, which is statsd, I think that this mapper only translates certain data, as you can see in the statsd mapper docs.

Each boomcatch user has different needs in terms of what needs to be mapped and sent, so the recommended course of action is to write your own mapper. It's quite easy to do!.

Before starting to write your own, I would suggest to check the unmapped mapper just to confirm that this is indeed the problem.

Hope that it helps!

rschoessler · 2019-04-05T14:11:44Z

@josebolos I'm running boomcatch from command line. Essentially, I was using the default mapper which I think is statsd. That is ONLY sending the rt data from my original post over to telegraf (which forwards to influxdb). I've tried the unmapped mapper and that gets an error on the telegraf side:

Error: splitting '|', Unable to parse metric: {"data":

...which I assume is because the format is not statsd which telegraf is expecting.

Here's my command (for statsd...have switched to unmapped, doesn't work):

H:\>boomcatch --host localhost --port 8080 --maxSize 2048 --mapper statsd

...which sends:

2019-04-05 10:06:47 INFO boomcatch: sending rt.firstbyte:311|ms
rt.lastbyte:2144|ms
rt.load:2144|ms

2019-04-05 10:06:47 INFO boomcatch: sent 56 bytes

I've tried to use statsd but seems like it has issues sending to my version of influxdb. I'm still testing things out, so I'm open to other suggestions on what boomcatch should send to that will ultimately get it into influxdb...would you have any suggestions?

Thanks!

rschoessler · 2019-04-05T15:18:38Z

Also based on the statsd mappers documentation it would seem that the navtiming data should be supported by my current setup/approach (i.e. rt is making it, but navtiming is not).

rschoessler · 2019-04-11T20:45:46Z

@josebolos ,

I've changed my configuration to use boomerang -> boomcatch -> statsd/graphite docker.

I can see that the navtiming data is going from the browser and into boomcatch by adding in a little logging:

2019-04-11 16:31:55 INFO boomcatch: state.body: rt.start=navigation&rt.bmr=401%2C92%2C67%2C44&rt.tstart=1555014709659&rt.bstart=1555014710324&rt.end=1555014715403&t_resp=312&t_page=5432&t_done=5744&t_other=t_domloaded%7C4165%2Cboomerang%7C5095%2Cboomr_fb%7C665&rt.tt=297186&rt.obo=4&nt_nav_
st=1555014709659&nt_fet_st=1555014709663&nt_dns_st=1555014709667&nt_dns_end=1555014709667&nt_con_st=1555014709667&nt_con_end=1555014709969&nt_req_st=1555014709969&nt_res_st=1555014709971&nt_res_end=1555014709972&nt_domloading=1555014710005&nt_domint=1555
014713822&nt_domcontloaded_st=1555014713822&nt_domcontloaded_end=1555014713823&nt_domcomp=1555014715393&nt_load_st=1555014715393&nt_load_end=1555014715403&nt_unload_st=1555014709988&nt_unload_end=1555014709988&nt_enc_size=1706&nt_dec_size=1706&nt_trn_siz
e=210&nt_red_cnt=0&nt_nav_type=1&u=http%3A%2F%2Flocalhost%3A4200%2F&r=http%3A%2F%2Flocalhost%3A4200%2F&v=%25boomerang_version%25&rt.si=u0w74xfvwk-ppt6n7&rt.ss=1555006435083&rt.sl=42&vis.st=visible&ua.plt=Win32&ua.vnd=Google%20Inc.&pid=dglw42zw&n=1&sb=1

-- log that I added for mappedData
2019-04-11 16:31:55 INFO boomcatch: mappedData: rt.firstbyte:312|ms
rt.lastbyte:5744|ms
rt.load:5744|ms

-- what is actually sent
2019-04-11 16:31:55 INFO boomcatch: sending rt.firstbyte:312|ms
rt.lastbyte:5744|ms
rt.load:5744|ms

So, this confirms for me that boomcatch seems to be either filtering out or ignoring the navigation timing data that is sent in.

Again, the command that I run from my terminal is:
boomcatch --host localhost --port 4201 --mapper statsd --path /beacon

What am I missing?

vedranf · 2019-04-19T14:04:25Z

Hello,

I think i have the same/similar issue. With unmapped, I see that data is being received:

2019-04-19 13:48:13 INFO boomcatch: sending {"data":{"restiming":"{"https://":{"www." ....
2019-04-19 13:48:13 INFO boomcatch: sent 6543 bytes

but upon switching to statsd, this is all I get:

2019-04-19 13:50:21 INFO boomcatch: referer=https://www.../ user-agent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36 address=127.0.0.1[...] method=POST url=/
2019-04-19 13:50:21 INFO boomcatch: sent 0 bytes
...

I run it like this:

$ ./src/cli.js --host 127.0.0.1 --port 2222 --workers 1 --mapper statsd --path / --forwarder console

Also, with waterfall I get the same error as in #86. Is it possible that boomcatch fails to parse the data? Perhaps a version mismatch with boomerang? Or it's sending invalid json? This "restiming" key appears to be json embedded as a string with quotes escaped. I'm not sure if that is expected.

Thanks,
Vedran

vedranf · 2019-04-19T16:47:27Z

Hello,

I just found examples under client/ directory. That worked, however, format being use there doesn't match what's described in:
https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.ResourceTiming.html

That boomerang uses json and trie structure, while boomcatch seems to uses application/x-www-form-urlencoded key-values pairs.

Regards,
Vedran

aftabnaveed · 2019-10-23T06:45:32Z

Before forwarding the data you probably need to decompress it via

var ResourceTimingDecompression = require('resourcetiming-compression').ResourceTimingDecompression
ResourceTimingDecompression.decompressResources(JSON.parse(rt)));

That kind of works but I am having an issue with it which is specified here
nicjansma/resourcetiming-compression.js#28

genesis-teddy · 2019-12-10T06:59:08Z

Before forwarding the data you probably need to decompress it via
var ResourceTimingDecompression = require('resourcetiming-compression').ResourceTimingDecompression
ResourceTimingDecompression.decompressResources(JSON.parse(rt)));
That kind of works but I am having an issue with it which is specified here
nicjansma/resourcetiming-compression.js#28

Which files should have this? Does anyone have a patch?

nevesing · 2020-06-16T03:06:52Z

I am also having the same problem. Any help would be appreciated.

genesis-teddy · 2020-06-17T13:22:41Z

I am also having the same problem. Any help would be appreciated.

https://github.com/genesis-teddy/boomcatch/wiki/Notes

nevesing · 2020-06-17T14:40:16Z

@genesis-teddy - Thanks for the info. So I reinstalled Boomcatch using your repo and I am able to see metrics from RT and ResTiming BUT not from NT yet:

npm install -g https://github.com/genesis-teddy/boomcatch

Startup cmd: boomcatch --prefix Jun17 --port 7000 --host 0.0.0.0 --path / --mapper statsd --forwarder udp --forwardHost mystatsd_host

BOOMR.init():

e.detail.BOOMR.init({ beacon_url: "https://myboomcatch_host", beacon_type: "GET", ResourceTiming: { enabled: true, clearOnBeacon: true }, RT: { enabled: true, cookie: "Cookie-Sample" }, NavigationTiming: { enabled: true } });

Console output: It shows metrics related to RT and ResTiming only. Tried disabling ResTiming or RT and I dont see metrics for NT. It is not the beacon issue as the 'unmapped' mapper shows all the metrics.

genesis-teddy · 2020-06-18T01:45:59Z

@genesis-teddy - Thanks for the info. So I reinstalled Boomcatch using your repo and I am able to see metrics from RT and ResTiming BUT not from NT yet:

npm install -g https://github.com/genesis-teddy/boomcatch

Startup cmd: boomcatch --prefix Jun17 --port 7000 --host 0.0.0.0 --path / --mapper statsd --forwarder udp --forwardHost mystatsd_host

BOOMR.init():

e.detail.BOOMR.init({ beacon_url: "https://myboomcatch_host", beacon_type: "GET", ResourceTiming: { enabled: true, clearOnBeacon: true }, RT: { enabled: true, cookie: "Cookie-Sample" }, NavigationTiming: { enabled: true } });

Console output: It shows metrics related to RT and ResTiming only. Tried disabling ResTiming or RT and I dont see metrics for NT. It is not the beacon issue as the 'unmapped' mapper shows all the metrics.

Glad it helped. I have not tried NT on my POC, so I have nothing to add, other than the documentation indicates it is available in the statsd mapper. Kindly share what you find out 🙂

nevesing · 2020-06-18T02:03:29Z

I dont have much experience in development or javascript so I am not sure.

genesis-teddy · 2020-06-19T01:51:28Z

I dont have much experience in development or javascript so I am not sure.

I’m curious to test, but too busy to help right now.
Could you share a screenshot of grafana tree?

nevesing · 2020-06-19T02:56:09Z

Thanks for showing Interest. I am not sure what do you mean by grafana tree screenshot. NT shows NavigationTiming API metrics.

genesis-teddy · 2020-06-20T01:44:23Z

Thanks for showing Interest. I am not sure what do you mean by grafana tree screenshot. NT shows NavigationTiming API metrics.

nevesing · 2020-06-20T03:17:30Z

Oh ok. This screenshot is from Graphite for RT and ResTiming. Similarly I should see for NT as well once it is fixed. Also if I capture metrics from 10 different applications, how does Graphite differentiate metrics by each app I dont know.

genesis-teddy · 2020-06-20T11:36:47Z

Oh ok. This screenshot is from Graphite for RT and ResTiming. Similarly I should see for NT as well once it is fixed. Also if I capture metrics from 10 different applications, how does Graphite differentiate metrics by each app I dont know.

Yes, Graphite. I got the two mixed up 😅
Somewhere in boomcatch, there’s a reference to a prefix.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

boomcatch sends only `rt` to telegraf (statsd) #111

boomcatch sends only `rt` to telegraf (statsd) #111

rschoessler commented Mar 28, 2019

rschoessler commented Apr 1, 2019

josebolos commented Apr 4, 2019

rschoessler commented Apr 5, 2019

rschoessler commented Apr 5, 2019

rschoessler commented Apr 11, 2019

vedranf commented Apr 19, 2019 •

edited

Loading

vedranf commented Apr 19, 2019

aftabnaveed commented Oct 23, 2019 •

edited

Loading

genesis-teddy commented Dec 10, 2019

nevesing commented Jun 16, 2020

genesis-teddy commented Jun 17, 2020

nevesing commented Jun 17, 2020

genesis-teddy commented Jun 18, 2020

nevesing commented Jun 18, 2020

genesis-teddy commented Jun 19, 2020

nevesing commented Jun 19, 2020

genesis-teddy commented Jun 20, 2020

nevesing commented Jun 20, 2020

genesis-teddy commented Jun 20, 2020

boomcatch sends only rt to telegraf (statsd) #111

boomcatch sends only rt to telegraf (statsd) #111

Comments

rschoessler commented Mar 28, 2019

rschoessler commented Apr 1, 2019

josebolos commented Apr 4, 2019

rschoessler commented Apr 5, 2019

rschoessler commented Apr 5, 2019

rschoessler commented Apr 11, 2019

vedranf commented Apr 19, 2019 • edited Loading

vedranf commented Apr 19, 2019

aftabnaveed commented Oct 23, 2019 • edited Loading

genesis-teddy commented Dec 10, 2019

nevesing commented Jun 16, 2020

genesis-teddy commented Jun 17, 2020

nevesing commented Jun 17, 2020

genesis-teddy commented Jun 18, 2020

nevesing commented Jun 18, 2020

genesis-teddy commented Jun 19, 2020

nevesing commented Jun 19, 2020

genesis-teddy commented Jun 20, 2020

nevesing commented Jun 20, 2020

genesis-teddy commented Jun 20, 2020

boomcatch sends only `rt` to telegraf (statsd) #111

boomcatch sends only `rt` to telegraf (statsd) #111

vedranf commented Apr 19, 2019 •

edited

Loading

aftabnaveed commented Oct 23, 2019 •

edited

Loading