Repository navigation
Formal governance document #5676
Description
Activity
I have one suggestion that might sound too radical. Why not simply put data.table under the umbrella of rOpenSci?
https://github.com/ropensci
rOpenSci already has a well defined governance structure, so data.table contributors only need to focus on the coding part. Moreover, as I mentioned in another issue, rOpenSci has already been funded by NumFOCUS.
#5675Reacted by stewartli and R.Andrés CastañedaI have one suggestion that might sound too radical. Why not simply put data.table under the umbrella of rOpenSci? https://github.com/ropensci rOpenSci already has a well defined governance structure, so data.table contributors only need to focus on the coding part. Moreover, as I mentioned in another issue, rOpenSci has already been funded by NumFOCUS. #5675
hi @waynelapierre thanks for your comment. Can you clarify what is the "well defined governance structure" of rOpenSci? I found https://softdev4research.github.io/4OSS-lesson/04-contributions/ which lists some general recommendations about what an open-source project governance document should contain, but I was not able to find a document describing governance of the rOpenSci project.
Hi Toby - thank you for this!
I won't rush to comment save for one initial thought regarding triaging of both issues and PRs. Triage is a somewhat hidden layer between contributors and reviewers. I'll often see Jan diligently handling this and think it would be good if this process was captured in the document.
Reacted by Frank and Michael ChiricoWould the reviewers have commit rights to the main branch? The chart implies that they don't and combined with:
Matt has written me several emails, and has told me that going forward he will have very little time to devote to data.table development.
That keeps the current problems around...
I would recommend a read of the substrait governance which is based on the ASF system of a group of committers and a PMC with some changes to make room for automatic releases and such (which currently are not possible for ASF projects) .
Reacted by Grant McDermott and David BudzyńskiPerhaps a survey would help in accumulating feedback faster and from more people?
In addition to the questions listed above, it could cover some extra ground such as:- respondent's background & if/how they'd be willing to contribute to data.table
- what to prioritize in future development
- feedback on more significant potential/upcoming changes (e.g. DT(), env)
Reacted by David BudzyńskiIt would indeed be nice to hear from @mattdowle if he has any strict requirements for the new governance -- i.e., proposals/guidelines to which he would not give final approval.
Regarding timelines, it would be good to build in an acceleration mechanism, whereby we can move to the next phase in approval once sufficient consensus is achieved (e.g. certainly among @tdhock @jangorecki and myself, with perhaps a few more). It does seem to me surprising that we expect it will take a year to establish the new governance & then move to releasing new data.table code.
Reacted by JoshOBrien, minemR, Toby Dylan Hocking, Hongyuan Jia, benoit, Ben Gorman, Dirk Eddelbuettel, Jan Gorecki, Benjamin Schwendinger, Kyle Haynes and 4 morePerhaps a survey would help in accumulating feedback faster and from more people? In addition to the questions listed above, it could cover some extra ground such as:
* respondent's background & if/how they'd be willing to contribute to data.table * what to prioritize in future development * feedback on more significant potential/upcoming changes (e.g. DT(), env)Hi @sluga thanks for sharing. The survey sounds like a good idea, that would help data.table developers and contributors get an idea about what kinds of features the users would like to prioritize. Would you be willing to set that up? (maybe this could be an additional role mentioned in the governance document, survey manager)
I would like to invite the following people to comment on this proposal, because they are listed among the top contributors https://github.com/Rdatatable/data.table/graphs/contributors
@st-pasha @lianos @tshort @eantonya @shrektan @HughParsonage @MarkusBonsch @ColeMiller1 @sritchie73 @tlapak @dracodoc @rsaporta @philippechataignon @eddelbuettel @OfekShilon @oseiskar @2005m @KyleHaynes @JoshOBrien @DavidArenburg @heavywatal @mcol @JenspederM @ajdamico @franknarf1 @eliocamp @UweBlock
If you have time and interest, I would appreciate your input/feedback on this issue; otherwise, if you don't have time to participate, or don't have any opinion to share, that is fine too.Reacted by Rick Saporta9 remaining items
Thanks @tdhock for spearheading this! Do you have an estimated timeline? It feels like this process is dragging. Perhaps it needs less formal discussion and more unilateral action? ..or perhaps some concrete dates so that we don't suffer paralysis by analysis.
Anyways I don't mean to complain. Really appreciate your work. I'll happily make a small donation to the team if it helps and I'd love to promote data.table if/when new releases start happening.
Reacted by jd4dsJust came across this document written by cURL creator/maintainer Daniel Sternberg. Sharing as relevant:
Reacted by ismirsehregal, Jan Gorecki, Steve Lianoglou, Ofek and Lluís RevillaReacted by Jan Gorecki, Tyson Barrett and JoshOBrienWhat a great read! Thanks for sharing Michael. Here are some relevant passages
Contributors will not stick around -> My experience says that you will have better success in getting more maintainers if you (as an existing maintainer) ask those you consider being contenders, rather than waiting and hoping for them to ask.
https://un.curl.dev/code/quality#how-do-you-achieve-good-code-quality
Roles: https://un.curl.dev/maintain BDFL? security? release manager (use a checklist), web, reviewing, support, blog, debug, merging, feature dev, doc writers, event planning, stickers, presenters, world monitoring (surveys).
Reacted by Tyson Barrett- unpinned this issue
on Oct 11, 2023 Hi, just a friendly reminder to check on the progression of the new governance. We are about to be one month for the deadline proposed by @tdhock. Do we have already a core-team for the future data.table? Are the funding issues closed? Will we claim back the throne of speed from other packages such as collapse?
End of November was the first milestone date mentioned by Toby, so we still have time.
As for benchmark with collapse, I invite you to submit new issue, for each task you are troubled by DT being slower than collapse.
It happened multiple times that scaling data up, or cardinality up, resulted in DT being faster. And not only vs collapse but in general.
There are also other places where we could provide faster functions but we rather hold to make them built-in, to reduce number of functions that user has to learn/discover, like fsum, fmean, etc.As benchmarking is off-topic to this issue I kindly ask to not continue that topic here but create new issue if needed.
Reacted by Toby Dylan Hocking- pinned this issue
on Oct 17, 2023 In my original post, I asked the questions, What roles and/or permissions should we define? What process must a contributor follow in order to obtain special roles and/or permissions? Here are some more detailed answers to these questions. Please comment constructively, discuss strengths/weaknesses of this structure, and propose concrete alternatives.
- Contributor: a user who has written/commented at least one issue, worked to label/triage issues, written a blog post, given a talk, etc. This is not a formal role (no need to keep a list of contributors), but contributors should be encouraged to submit their first PR to become a project member.
- Project member: some one who has submitted at least one PR that has been merged into master. Any user can become a member by submitting a PR, then having it reviewed and merged into master. Members are credited via
role="ctb"in DESCRIPTION (so they appear in Author list on CRAN), and they are added to https://github.com/orgs/Rdatatable/teams/project-members so they can create new branches in the Rdatatable/data.table GitHub repo. Note: I like the term "member" here instead of "contributor" because I believe there are many ways to contribute to the project without having to submit a PR. Currently there are 50 members. - Reviewer: a member who has volunteered to do code reviews for some features/files. After one or more significant PRs to a given file, a member should be invited to add their name as a reviewer of that file in CODEOWNERS, and after that is merged into master, then they are considered a reviewer. Same credit in DESCRIPTION as a regular member,
role="ctb"(so they appear in Author list on CRAN). Note: having your name in CODEOWNERS does not give any special permission, but it does mean that you will be notified whenever there is a new PR with changes to that file. Current reviewers listed in add CODEOWNERS file #5629 are @tdhock, @ben-schwen, @jangorecki, @MichaelChirico but it would be great to expand this to include more people who have submitted PRs in the last few years. - GitHub maintainer: permission to merge PRs into master branch. After a reviewer has a consistent history of careful reviews of others' PRs, then a current GitHub maintainer should ask all other current GitHub maintainers if they approve promoting the reviewer to GitHub maintainer, and it should be done if there is general consensus/agreement. Credited via
role="aut"in DESCRIPTION (so they appear in Author list on CRAN), and added to https://github.com/orgs/Rdatatable/teams/maintainers which gives permission to merge PRs into master branch. Note: to avoid confusion with CRAN maintainer, should we use a different name for this role? "GitHub masters" or "committers" (thanks @assignUser) because they can commit to master branch? Currently there are two GitHub maintainers: Matt and Arun, but I would suggest expanding this to include at least @tdhock, @ben-schwen, @jangorecki, @MichaelChirico - CRAN maintainer: in charge of communication with CRAN. Responsible for submitting releases to CRAN on a regular basis, and in response to requests from CRAN. Credited via
role="cre"in DESCRIPTION (so they appear as Maintainer on CRAN). Note: in the past this role has been filled by one of the GitHub maintainers, but CRAN maintainer does not actually need permission to merge PRs into master branch. Currently this is Matt, but for the future I would suggest @jangorecki or @MichaelChirico
Reacted by Angel Esteban Feliz-
Project member - a rule for listing into DESCRIPTION file was once nicely defined by Matt, possibly in a video, but is also mentioned here (that non-code contributions may not qualify): fixes #504 fread now handles all kind of NAs without coercion to char #1236 (comment)
-
GitHub maintainer - we need to define how many GH maintainers have to give green light for PR to be merged. Proposed list (Matt, Arun, Michael, Ben, Toby, Jan) has 6 maintainers, normally I would then say 3 "lgtm" should be fine to merge a PR but considering only 4 of those 6 are active it feels like 3 out of 4 is quite a lot. Could as well be 1 GH maintainer plus 1 reviewer/GH maintainer.
Reacted by Dirk Eddelbuettel and Toby Dylan Hocking-
should we use a different name for this role?
'Committer' as used by the ASF is quite descriptive imo
how many GH maintainers have to give green light for PR to be merged
To keep development velocity I would suggest >= 1 👍 and no 👎 for smaller day to day changes (bugfixes, enhancements etc) and some higher requirement for more important/influential changes (e.g. changes to core api/implementation). This could also be a discussion + vote outside of the PR (e.g. for arrow we have to vote on the ML for things like format changes/additions).
Reacted by Toby Dylan Hocking, Jan Gorecki and Dereck MezquitaFor CRAN maintainer I think it would be most suitable to choose a person whose time will be already funded by NSF (or any other company/foundation willing to sponsor that). Preparing release and CRAN communication have often short deadlines and it can be quite time consuming, therefore relying on a volunteer to handle that does not seem to be fair.
Reacted by Jacob Wujciak-Jens, Toby Dylan Hocking and Angel Esteban Feliz- unpinned this issue
on Dec 14, 2023
data.table has no formal governance document; Matt Dowle, the original author and only current maintainer, has commit permissions on github, and he submits the package to CRAN. The only other author, Arun, has been inactive for several years.
Matt has done a fantastic job at creating a highly efficient and widely used R package, and he continues to submit "patch" releases to CRAN (containing minimal fixes so that the package continues to pass CRAN checks). data.table has many contributors, who have submitted PRs, which have been reviewed and merged by Matt. This form of project leadership/governance is similar to the former python model, Benevolent Dictator For Life (BDFL). This form of governance can handle as many contributions/PRs as the BDFL (Matt) has time to review and merge. The purpose of this issue is to discuss alternative forms of governance which may be able to handle more contributions, from a larger and more diverse group of contributors.
I therefore propose that we use this issue to write constructive, thoughtful, respectful, and inclusive comments containing concrete propositions for the future governance of the data.table project. At the end of three months (end of November 2023), I will synthesize the comments into a draft governance document, which I will publish as a PR that creates a new GOVERNANCE.md file. After that, there will be a
threeone month period of open comments, where people can discuss the strengths/weaknesses of the draft, and we can discuss possible edits to make. At the end of that second period (end of Dec 2023), I will consider the consensus of the comments, make corresponding edits, and publish a second draft (by editing the PR). If that draft is sufficient, I will ask contributors to sign that document by adding their names to the GOVERNANCE.md file in that PR. If there are still significant concerns, we can use anotherthreeone month period of comments to resolve them, after which I will publish another (hopefully final) draft in end of Jan 2024.Some significant questions that I suggest we try to answer in the governance document: (others are welcome too)
data.tableprinciples #5693We are not the first open-source project to have a governance document, here is a reading list about open-source governance, which can inform our discussion:
About the roles to define, I suggest replacing the current flat leadership model (one maintainer role at the top, many contributors on the bottom), with a hierarchical model containing intermediate "reviewers," similar to the successful model of subsystem maintainers from the linux kernel project. See figure below, but note that the names are totally arbitrary (for example, I do not expect Kelly to be release manager, but it would be nice to have someone take the role of release manager).
I would suggest that each intermediate “reviewer” volunteer to be in charge of reviewing and merging PRs for specific features/files, as defined in the CODEOWNERS file, #5629 So far only @ben-schwen @MichaelChirico and @jangorecki have volunteered to be reviewers.
In addition to the reviewer role in the figure above, there could be at least five other roles (with responsibilities):
I would volunteer to be reverse dependency manager, as I have set up the revdep-check system
I would nominate @MichaelChirico for translation manager and @jangorecki for binary manager.
Who would volunteer for release manager and performance testing manager?