How to collect statistics/ logs from multiple connected distributed application instances?

  architecture, c++, distributed, networking, statistics

This question is more about application software architecture design, but links, libraries, samples of code would really help.


I have my network console application (The Application) written in C++ running on 400+ PC that are distributed between several networks. Each instance of The Application is connected to bunch of instances of The Application and every instance is sending/receiving some data to/from other instances of The Application. Every instance is equal, there is no server by design.

The Application is sending some data that have each piece labelled with unique ID. So f.e. if we have 3 instances (i#1, i#2, i#3) of The Application and 2 networks (nw#1, nw#2) and i#3 instance is using both nw#1 and nw#2 the path for data piece labelled as "UniquePiece#1" could be:

DataSent{UniquePiece#1} from i#1 PC#1

[i#1(nw#1)] -> [i#3(nw#1), i#3(nw#2)] -> [i#2(nw#2)]

DataReceived{UniquePiece#1} in i#3 PC#3

What I need to implement:

Some GUI to get the real-time statistics like:

  1. Graph of the instances (like network graph). Visualize networks that The Application is part of and their connections. Pings, latency.
  2. Data logs. Visualize data flows between The Application instances, get statistics about sent/received packets, its contents etc.
  3. Error logs. Visualize errors that are happening while The Application is running.

My idea is to implement/use some protocol for The Application to send between instances statistic info. This task raises several questions like:

  • How to shrink network usage to minimum (traffic)?
  • How to get ping and latency info?
  • How to find all nodes of The Application in network (each instance)?
  • What protocol to use or do I need to implement some?
  • What tools to use (C++ libraries)?


I understand that the problem is very "broad" but do now know where to start or what to search. Any hints how such task could be implemented in C++ or some mature implementations (libraries, articles) would be really appreciated.

Source: Windows Questions C++