Working with Disk Metrics from Node Exporter

Run stress -i 40 on your server before starting this lesson.

Disk metrics are specifically related to the performance of reads and writes to our disks, and are most commonly pulled from /proc/diskstats. Prefixed with node_disk, these metrics track both the amount of data being processed during I/O operations and the amount of time these operations take, among some other features.

The Node Exporter filters out any loopback devices automatically, so when we view our metric data in the expression editor, we get only the information we need without a lot of noise. For example, if we run iostat -x on our terminal, we’ll receive detailed information about our xvda device on top of five loop devices.

Now, we can collect information similar to iostat -x itself across a time series via our expression editor. This includes using irate to view the disk usage of this I/O operation across our host:

irate(node_disk_io_time_seconds_total[30s])

Additionally, we can use the node_disk_io_time_seconds_total metric alongside our node_disk_read_time_seconds_total and node_disk_write_time_seconds_total metrics to calculate the percentage of time spent on each kind of I/O operation:

irate(node_disk_read_time_seconds_total[30s]) / irate(node_disk_io_time_seconds_total[30s])

irate(node_disk_write_time_seconds_total[30s]) / irate(node_disk_io_time_seconds_total[30s])

Additionally, we’re also provided with a gauge-based metric that lets us see how many I/O operations are occurring at a point in time:

node_disk_io_now

Other metrics include:

• node_disk_read_bytes_total and node_disk_written_bytes_total, which track the amount of bytes read or written, respectively
• node_disk_reads_completed_total and node_disk_writes_completed_total, which track the amount of reads and writes
• node_disk_reads_merged_total and node_disk_writes_merged_total, which track read and write merges