Add Basic Metrics

[ansermino]

Story

As an operator/developer
I want get insights into the performance of the system
So that I can debug issues, adjust my setup, tune performance

Background

We currently have very little insight into how the system is functioning, beyond the UI. It would be nice to have some basic metrics included so we can understand the performance better.

Details

Some things we might want to measure:

• Peers discovered
• Libp2p peer connections
• IP resoloution requests
• DB reads/writes/updates
• Total number of records
• Fork digests

Scenarios

N/A

Implementation details

We should implement prometheus within the application and then use Datadog or Grafana to visualize them. We should not use Datadog library to add metrics as this is an OS project.

Testing details

Acceptance criteria

• Basic metrics are agreed upon and rationalized (ie. why are we measuring this?)
• Metric collection is implemented
• Dashboard to display metrics is implemented