Working with Memory Metrics from Node Exporter

Run stress -m 1 on your server before starting this lesson.

When it comes to looking at our memory metrics, there are a few core metrics we want to consider. Memory metrics for Prometheus and other monitoring systems are retreived through the /proc/meminfo file; in Prometheus in particular, these metrics are prefixed with node_memory in the expression editor, and quite a number of them exist. However, of the vast array of memory information we have access to, there are only a few core ones we will have to concern ourselves with much of the time:

• node_memory_MemTotal_bytes
• node_memory_MemFree_bytes
• node_memory_MemAvailable_bytes
• node_memory_Buffers_bytes
• node_memory_Cached_bytes

Those who do a bit of systems administration, incident response, and the like have probably used free before to check the memory of a system. The metric expressions listed above provide us with what is essentially the same data as free but in a time series where we can witness trends over time or compare memory between multiple system builds.

node_memory_MemTotal_bytes provides us with the amount of memory on the server as a whole — in other words, if we have 64 GB of memory, then this would always be 64 GB of memory, until we allocate more. While on its own this is not the most helpful number, it helps us calculate the amount of in-use memory:

node_memory_MemTotal_bytes - node_memory_MemFree_bytes

Here, node_memory_MemFree_bytes denotes the amount of free memory left on the system, not including caches and buffers that can be cleared. To see the amount of available memory, including caches and buffers that can be opened up, we would use node_memory_MemAvailable_bytes. And if we wanted to see the cache and buffer data itself, we would use node_memory_Cached_bytes and node_memory_Buffers_bytes, respectively.