Clarify and implement core astropy feature freeze policy #96

Proposed by: @perrygreenfield @mhvk @eteq @taldcroft
Expanded by: @pllim (feel free to edit if I got things wrong here)

@perry -- A PR must be submitted before the freeze date to receive consideration for inclusion into the release, but that does not guarantee inclusion. Given the overwhelming rush of PRs in past freezes, it is simply not practical to handle them all at the last minute. Part of the process at freeze is an initial triage of the PRs, those that certainly will be part of release, those that might be, and those that will not be. The middle category will be included if there is sufficient time and resources. The last category generally will cover those that require a lot of reviewing and testing resources (e.g., large changes or commits) that cannot be accommodated or are not considered significant enough to be included at the last minute... Those making these decisions should be heavily weighted by those that have to do the work of the release.

@mhvk -- ... ensure that we don't implicitly force the release team in roles of gatekeepers: how about implementing that after some pre-freeze point, any of the subpackage maintainers can (effectively) block inclusion of PRs with a "want to review but don't have time" request-changes review - I guess this falls a bit in a more general category of timely review... as in many open source
projects, review is becoming the bottleneck

@eteq -- we should move the release to whatever time scale allows the testing to happen... the intent was that that time line also got shifted by the additional week necessary to allow this.

@taldcroft -- I don't see any reason not to amend APE 2... it is worth recognizing that late PR review is also a problem alongside late submissions

Problems to solve:

• Late PRs.
• Late PR reviews.
• Late What's New entries (see DOC: Require PR author to add What's New entry astropy#9676)

Action items based on the proposal above:

• Telecon(s) to refine the freeze and triaging process that is proposed above, allowing for ample time after the freeze for release manager to do the release without undue stress of PRs keep sneaking in after freeze.
• Amend APE2 with an addendum PR, documenting the new policy from decisions made in telecon(s). For clarify, it is important to separate the existing APE 2 content and the amendment.
• Create a checklist for core maintainers to set clear expectations for feature freeze time.
• Perhaps a bot can enforce some of the rules; e.g., label PRs opened after a freeze date and milestoned to the associated release as "late". With the bot acting as neutral party would take any hostility/stress off core maintainers and release manager.

See Also

#95