0 Preface
Parts 1 and 2 are analysis of the logic of XLOG generation and cleaning. The processing of XLOG surge is directly viewed in Part 3.
1 WAL archive
# Maximum number of log file segments between automatic WAL checkpointscheckpoint_segments = # Maximum time between automatic WAL checkpointscheckpoint_timeout = # Relieve io stresscheckpoint_completion_target = # Minimum number of log file segments to keep more segments for the librarywal_keep_segments = # Completed WAL segments are sent to archive storage via archive_commandarchive_mode = # Force timeout to switch to a new wal segment filearchive_timeout = max_wal_size = min_wal_size =
1.1 When archive is not enabled
The number of files is controlled by the following parameters, usually not exceeding
(2 + checkpoint_completion_target) * checkpoint_segments + 1
or
checkpoint_segments + wal_keep_segments + 1
a file.
If an old segment file is no longer needed, it will be renamed and then continued to be overwritten. If the short-term log output peak results in more than
3 * checkpoint_segments + 1
Delete files directly.
1.2 When archiving is enabled
Number of files: Delete the archived segment file successfully
In abstractly, a running PG generates an infinitely long WAL log sequence. Each segment is 16M, and the names of these segment files are numerical names, reflected in the location in the WAL sequence. When you do not use WAL archives, the system usually just creates a few segment files and then recycles them by renaming the segment files that are no longer used to a higher segment number.
The archive command returns zero if and only if the archive command succeeds. After obtaining a zero-value result, PostgreSQL will assume that the WAL segment file has been archived successfully and will delete the segment file later. A non-zero value tells PostgreSQL that the file is not archived and will be retry periodically until successful.
2 PG source code analysis
2.1 Delete logic
Trigger the delete action
RemoveOldXlogFiles > CreateCheckPoint > CreateRestartPoint
wal_keep_segments judgment (call this function to modify _logSegNo, and then pass it in RemoveOldXlogFiles)
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo) { XLogSegNo segno; XLogRecPtr keep; XLByteToSeg(recptr, segno); keep = XLogGetReplicationSlotMinimumLSN(); /* compute limit for wal_keep_segments first */ if (wal_keep_segments > 0) { /* avoid underflow, don't go below 1 */ if (segno <= wal_keep_segments) segno = 1; else segno = segno - wal_keep_segments; } /* then check whether slots limit removal further */ if (max_replication_slots > 0 && keep != InvalidXLogRecPtr) { XLogSegNo slotSegNo; XLByteToSeg(keep, slotSegNo); if (slotSegNo <= 0) segno = 1; else if (slotSegNo < segno) segno = slotSegNo; } /* don't delete WAL segments newer than the calculated segment */ if (segno < *logSegNo) *logSegNo = segno; }
Delete logic
static void RemoveOldXlogFiles(XLogSegNo segno, XLogRecPtr endptr) { ... ... while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL) { /* Ignore files that are not XLOG segments */ if (strlen(xlde->d_name) != 24 || strspn(xlde->d_name, "0123456789ABCDEF") != 24) continue; /* * We ignore the timeline part of the XLOG segment identifiers in * deciding whether a segment is still needed. This ensures that we * won't prematurely remove a segment from a parent timeline. We could * probably be a little more proactive about removing segments of * non-parent timelines, but that would be a whole lot more * complicated. * * We use the alphanumeric sorting property of the filenames to decide * which ones are earlier than the lastoff segment. */ if (strcmp(xlde->d_name + 8, lastoff + 8) <= 0) { if (XLogArchiveCheckDone(xlde->d_name)) # Archive closes and returns true # The done file returns true # Existence.ready returns false # Recheck has done file returns true # Rebuild the .ready file and return false { /* Update the last removed location in shared memory first */ UpdateLastRemovedPtr(xlde->d_name); # Recycle or delete directly, clean up .done and .ready files RemoveXlogFile(xlde->d_name, endptr); } } } ... ... }
2.2 Archive Logic
static void pgarch_ArchiverCopyLoop(void) { char xlog[MAX_XFN_CHARS + 1]; # Get the oldest xlog file name that has not been archived while (pgarch_readyXlog(xlog)) { int failures = 0; for (;;) { /* * Do not initiate any more archive commands after receiving * SIGTERM, nor after the postmaster has died unexpectedly. The * first condition is to try to keep from having init SIGKILL the * command, and the second is to avoid conflicts with another * archiver spawned by a newer postmaster. */ if (got_SIGTERM || !PostmasterIsAlive()) return; /* * Check for config update. This is so that we'll adopt a new * setting for archive_command as soon as possible, even if there * is a backlog of files to be archived. */ if (got_SIGHUP) { got_SIGHUP = false; ProcessConfigFile(PGC_SIGHUP); } # archive_command is not set and will no longer be executed # Our command is not set, this branch is going if (!XLogArchiveCommandSet()) { /* * Change WARNING to DEBUG1, since we will left archive_command empty to * let external tools to manage archive */ ereport(DEBUG1, (errmsg("archive_mode enabled, yet archive_command is not set"))); return; } # Execute the archive command! if (pgarch_archiveXlog(xlog)) { # Successfully, change .ready to .done pgarch_archiveDone(xlog); /* * Tell the collector about the WAL file that we successfully * archived */ pgstat_send_archiver(xlog, false); break; /* out of inner retry loop */ } else { /* * Tell the collector about the WAL file that we failed to * archive */ pgstat_send_archiver(xlog, true); if (++failures >= NUM_ARCHIVE_RETRIES) { ereport(WARNING, (errmsg("archiving transaction log file \"%s\" failed too many times, will try again later", xlog))); return; /* give up archiving for now */ } pg_usleep(1000000L); /* wait a bit before retrying */ } } } }
2.3 ready generation logic
static void XLogWrite(XLogwrtRqst WriteRqst, bool flexible) { ... if (finishing_seg) { issue_xlog_fsync(openLogFile, openLogSegNo); /* signal that we need to wakeup walsenders later */ WalSndWakeupRequest(); = ; /* end of page */ # Archive open && wal_level >= archive if (XLogArchivingActive()) # Generate ready file XLogArchiveNotifySeg(openLogSegNo); XLogCtl->lastSegSwitchTime = (pg_time_t) time(NULL); ...
2.4 Summary
As long as the ready file satisfies archive_mode=on and wal_lever>=archive, it will always be generated (generated by XLogWrite function call)
Because archive_command is set empty, the consumption of ready file is completely controlled by external programs
The processing of done file is done by PG, and the two places will trigger the processing of done file, checkpoint and restart point.
How many done files are processed is controlled by wal_keep_segments and replication_slot (KeepLogSeg function)
3 Reasons for WAL segment accumulation (Look for total?)
Note: Be careful not to manually delete the xlog file anyway
Note: The log back generated by checkpoint does not generate ready files immediately, but is generated in the next xlog piece.
3.1 ReplicationSlot
Opened the streaming copy slot
-- Stream copy slot -- ifrestart_lsnand currentXLOGThe number of bytes with very large differences, Need to checkslotCan subscribers receive it normallyXLOG, -- Or is the subscriber normal. Not to beslotData removal, pg_xlogThe directory may burst select pg_xlog_location_diff(pg_current_xlog_location(),restart_lsn), * from pg_replication_slots;
delete
select pg_drop_replication_slot('xxx');
After deletion, PG will clean up xlog in the next checkpoint
3.2 Larger wal_keep_segments
Check the parameter configuration, note that turning on this parameter will delay xlog and ready.
3.3 There is a problem with recycling
If the PG automatic recycling mechanism is not used, the database relies on external programs to modify the .ready file, and the recycling process needs to be detected
(archive_mode=on archive_command='')
3.4 Checkpoint interval is too long
Check parameter configuration
The above is personal experience. I hope you can give you a reference and I hope you can support me more. If there are any mistakes or no complete considerations, I would like to give you advice.