@jimis Would you be able to help us collect a few profiles? Our documentation for profiling needs a bit of work first though, we will get back to you with an update that has more precise instructions.

Yes I can do some profiling. I already tried to grab some gdb backtraces to see where the threads are kept busy, but all I got was question marks instead of function names. Are you binaries shipping with debug info?

FYI, on this server it is InfluxDB that has thousands of connections in ESTABLISHED state. There is an issue there, because I see a slow but constant upwards trend in the number of connections (as logged by telegraf), and the number gets reset only when I restart influxd. I hope I'll fix this issue soon, which will significantly reduce the number of connections. Until then I can definitely profile telegraf.

I don't think profiling is necessary, I have verified that the slowness is in the inputs.netstat plugin.

What I did to verify this:

• disabled netstat plugin and restarted telegraf
• CPU utilisation went very low
• re-enable netstat plugin and restart telegraf
• CPU utilisation goes to 70% on average

I believe it should be easy for you to replicate the scenario, by providing a fake /proc/net/tcp6 file with 120K lines. I can provide you with one such file if needed.

EDIT: I might be wrong about /proc/net/tcp6 file. I had a look at gopsutil net_linux.go and I believe it goes through and parses all of /proc/*/fd/*, which would explain why it is horrendously slow. :-)

Opened shirou/gopsutil#784 for that, feel free to comment.

In my opinion, telegraf shouldn't use gopsutil for something so simple, but better open directly the relevant file.

@glinton I see you added the upstream label, but I have to disagree, after reading a bunch of comments in issues in gopsutil. gopsutil is having the job to parse and return information from every single process in the system. While telegraf can do much better by opening only one generic file.

@danielnelson I see you removed the ready label, what more information would you need from me? I don't think profiling will be useful, the culprit was easy to identify.

I'll answer over on the gopsutil issue too, it may make sense for us to have a custom linux implementation because gopsutil is reporting the pid, and we don't need it in this plugin. I don't think the pid can be obtained from /proc/net/tcp and it appears from strace that when given the -p flag netstat follows a similar strategy to gopsutil.

If you can work on an implementation that uses the /proc/net/tcp* files, we can either add it to Telegraf or into gopsutil depending on what is decided.

I can get tcp netstat from /proc/net/tcp*, but I don't know what the metrics udp_socket means, and where is it from?

I can get tcp netstat from /proc/net/tcp*, but I don't know what the metrics udp_socket means, and where is it from?

@JoveYu I would /guess/ it is the number of udp sockets the system keeps track of. Probably the number of lines of some of the files /proc/net/udp* ?

i hava rewrite netstat plugin in linux, it can help fix this issue

@danielnelson thanks for pointing me to this open ticket.

What version of telegraf client has the fix @JoveYu and @danielnelson

Still working on the fix in #6772.