PGCon2013 - includes video

PGCon 2013
The PostgreSQL Conference

Jeff Davis
Day Talks - Day 1 - 2013-05-23
Room MNT 202
Start time 11:30
Duration 01:00
ID 586
Event type Lecture
Track DBA
Language used for presentation English

Corruption Detection and Containment

Strategies and Roadmap

This will not be the most exciting talk, nor is there (currently) a simple answer to make hardware corruption problems go away. But it's important -- without being careful, it's easy for corruption to spread to replicas and backups, leaving data hopelessly lost. Or, a strange crash due to corruption could take many engineering resources to analyze.

This talk is about kinds of hardware corruption that can and do happen, and the ways to detect and contain the corruption as quickly as possible. Additionally, we'll discuss a roadmap of improvements to postgresql to make this an easier process; as well as alternatives (such as detecting corruption in the filesystem).

Note: Some storage systems do provide strong protections against data corruption. This talk is primarily (though not exclusively) targeted at users of the local filesystem, particularly on Linux.

These are the topics that will be addressed:

  1. Why not deal with this in the filesystem?
  2. The different kinds of corruption.
  3. When to detect the corruption, and how to contain it.
  4. Data page checksums
  5. Backups and corruption
  6. Replication and corruption
  7. Background and offline detection
  8. More work to be done