Repository navigation
Custom changelog seems to ignore update mode #1132
Copy link
Copy link
Closed
Labels
Description
Question
How do I make sure the changelog entries operate in 'update' mode, even with a custom changelog? I noticed that the update mode was working perfectly until I switched to a custom template.
My end goal is simply to include the body of the commit in the changelog entry, hence why I have a custom template.
Configuration
Semantic Release Configuration
[tool.semantic_release]
version_variables = ["__init__.py:__version__"]
upload_to_pypi = false
tag_format = "@[app]/[service]-v{version}"
commit_message = "chore(release): @[app]/[service]-v{version} [skip ci]"
[tool.semantic_release.changelog]
template_dir = "../../../documentation/templates"
mode = "update"
[tool.semantic_release.changelog.default_templates]
changelog_file = "../../../documentation/docs/[service]/changelog.md"
[tool.semantic_release.branches.test-semantic-release]
match = "test-semantic-release"
prerelease = falseAdditional context
my template (if there is an easier way to do this, please let me know):
{# <root>/documentation/templates/documentation/docs/[service]/changelog.md.j2 #}
{% for version, release in ctx.history.released.items() %}
{{ "## %s (%s)" | format(version.as_semver_tag(), release.tagged_date.strftime("%Y-%m-%d")) }}
{% for type_, commits in (release.elements | dictsort) if type_ != "unknown" %}
{% set filtered_commits = [] %}
{% for commit in commits %}
{% if "[skip ci]" not in commit.message %}
{% set _ = filtered_commits.append(commit) %}
{% endif %}
{% endfor %}
{% if filtered_commits %}
{{ "### %s" | format(type_ | title) }}
{% for commit in commits %}
{{ "* %s ([`%s`](%s))" | format(
commit.descriptions[0] | capitalize,
commit.hexsha[:7],
commit.hexsha | commit_hash_url
) }}
{% set lines = commit.message.split('\n') %}
{% set bullet_lines = [] %}
{% for line in lines[1:] %}
{% if line and (line.startswith('-') or line.startswith('*')) %}
{% set _ = bullet_lines.append(line) %}
{% endif %}
{% endfor %}
{% if bullet_lines %}
{% for bline in bullet_lines %}
{{ bline }}
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% endfor %}# cicd.yml
# ...
- name: Create [service] github release
if: needs.changes.outputs.[service] == 'true'
uses: python-semantic-release/[email protected]
with:
directory: poweralpha/services/[service]
changelog: true
commit: true
push: true
tag: true
vcs_release: true
github_token: ${{ secrets.GITHUB_TOKEN }}
root_options: "-vv"Reactions are currently unavailable
Activity
codejedi365 commentedon Jan 5, 2025
Hello @marc-at-brightnight,
Did you happen to look at the Incrementally Updating Changelog Template section of the documentation? Was this helpful or not understandable?
Secondly, if I understand your desire, you would like the commit body to be a bulleted list rather than how the default template displays the full commit body? I think I first read it as you wanted only the subject line which is what most people ask me about, but by the look of the template it is more of a template formatting change. Either way I have provided instructions on what components to modify for handling the commit body within issue #1122. There is a comment for how to only display the commit subject line but in your case it points out the section you should modify for formatting details. The issue also includes more detail than the above documentation of what to change from the default to a custom template which would preserve other features (ex updating) that PSR has provided.
Also, note that we already split the commit messages up by newlines and include them in the
commit.descriptionslist so I don't think you would need to do that part yourself.Lastly, I would recommend using the
changelog.exclude_commit_patternssetting to enable PSR to remove those undesirable commits rather than you having to do that inside of the template.codejedi365 commentedon Jan 5, 2025
Separately, may I ask what information are you trying to preserve and include from the commit body?
Over the past few months and for a few more in the future, I have been significantly changing the default changelog contents and on that roadmap is to reduce down to only displaying the commit subject line. The precursor to this is the support for parsing squash commits so that they can be each presented per version. Both are very close to publish. I have also added the features to parse out PR/MR numbers and parse out issue resolution numbers which are available to consumers in each
ParsedCommitobject. I have been systematically changing the default template to display these components in pretty formats. My reason for asking the question above is to understand the users of PSR and what other info they may desire that I have not already considered. Thanks!marc-niclas commentedon Jan 5, 2025
I overlooked it. I think maybe changing the title might help, perhaps "Using a changelog template in update mode"?
If I use the default template, this is what I see in the changelog:
This was my commit:
Instead, I would like my changelog entry to read:
Which is why I turned to a custom template. It seems like the default template only includes the subject header. For our use case, if the user wants to provide more details in the form of bullets in the commit message, those bullets should be reflected in the changelog.
Got it 👍🏼
Very good 👍🏼
Thank you very much @codejedi365 for the extremely good work on this repo and diligence on provided A+ user support. It is refreshing to encounter such professionalism on an open-source project and it does not go unnoticed 🫡
marc-niclas commentedon Jan 5, 2025
See my previous response. If that doesn't answer your question fully, happy to expound.
codejedi365 commentedon Jan 5, 2025
Thanks, I'll consider it. That specific section is for updating a custom changelog previously made to now be an updating changelog, not exactly to explain update mode. Update mode is explained at the top of the page under the default templates. I'll try to add some clarity there.
I believe you are hitting an unexpected edge case of the parser due to your commit format to give you a result that only includes the commit subject. From the conventional commit standard (i.e. the newer variant of the angular guidelines), It is expected for there to be a double newline (a blank line) between the commit subject and the commit body. Since your commit does not have a double newline, somehow the bullets are being stripped during parsing.
If you amend your commit to be the following then I suspect you will get a different result (and maybe the one you desire):
I should probably clarify now that I have discussed commit formatting above. The
commit.descriptionslist is a product of the following logic:That means a lot, as user support takes a lot more time than expected. Thank you!
marc-niclas commentedon Jan 5, 2025
Interesting. I tried that and got the following:
with commit:
Not exactly what I want so will stick with custom template for now.
Ah, that explains why
.descriptionsdoesn't work for me. I need it split up by single newlines, not double. No matter, easy fix.I think I have what I need (mostly just needed the pointer to the right section of the documentation but the extra context and help has been super helpful), except I have one more question, if you don't mind:
I have multiple services and would like to avoid duplicating the changelog template in different folders that mirrors our documentation structure. is there a way I can have multiple services use the same changelog?
My latest template, in case it is helpful for others encountering this thread: (sorry for bad formatting, my first jinja2 template)
codejedi365 commentedon Jan 5, 2025
Unfortunately, I don't think the provided example helps illustrate a real life use of what the bulleted information would be. Currently PSR's parsers support extracting additional paragraphs in the form of the prefix
BREAKING CHANGE:which then in the default template will be consolidated and placed in a bulleted list at the bottom in a### BREAKING CHANGESsection.Another feature that is not yet released (but will be soon) is the additional support of an additional paragraph prefix
NOTICE:(reference issue: #223) which will similarly create a separate section### ADDITIONAL RELEASE INFORMATIONwith a bulleted list of additional instructions/details for product consumers. The intent was to provide deprecation notices which are not breaking changes within themselves.I recognize that the standard although detailed does not comment as much on what the content should be within commit sections. I personally use
commitizenwhen I started out which helped frame what information should be described in a commit message. With those tools as a base and many personal experiences after, I have come to believe that the subject line is written from a user perspective (how does this change affect me [the end user]) and the commit body paragraph is written for the programmer that comes after you with a hint of how the specific change maps to the summary line. Then if you have additional info to give the users, use one of our two prefixes (or manually write/update the CHANGELOG). And lastly include any git footer prefixes that close issues, add references, or enrich the commit details (signed-off-by, etc.).With my approach above, I don't find any additional information that needs to be provided to users that is not captured in these ways. What additional information are you trying to provide to users that wouldn't fit in this approach?
Lastly, something I forgot to mention earlier, If you are intending for the JIRA issue reference to be parsed as a linked issue then in its current format it will not be detected by PSR. I implemented PSR to mirror the 4 VCS's PSR supports with the caveat that it must be specified in the colon-defined git footer syntax (referenced by conventional commit standard). See Common Issue Identifier Detection
codejedi365 commentedon Jan 5, 2025
Partially. The first problem was from the double newline from the subject. I recently added support to detect single newline bulleted lists and it will not collapse those single newlines. In fact, it actually normalizes to insert an additional newline to meet the markdown style guidelines which then allow step 3 to properly separate into separate entries in
commit.descriptionscodejedi365 commentedon Jan 5, 2025
Understandable, I didn't put much effort in to this as it was a temporary fix and will be changed in the future. There was some weirdness with additional indents that I tried to fix with the word wrap filter and the indent parameter but it was hard to solve generically.
Unfortunately not. Primarily because Jinja has a poor path resolution algorithm that attempts to prevent path traversal attacks but errors out if it ever detects a
..parent directory path even if it is valid within its sandboxed directory. One day I plan to open a PR for it but I haven't gotten around to it. I believe #845 matches your situation and I haven't gotten around to that fix yet either.PSR does not yet support monorepos fully and this is one of those situations I believe.
codejedi365 commentedon Jan 5, 2025
@marc-at-brightnight, I looked over your template. I think line 6 will cause you issues on the next release. You are looping through the entire list of releases again upon each new release rather than just the latest release. All the previous releases will be in the changelog contents as part of the update.
marc-niclas commentedon Jan 5, 2025
Yeah I think I agree with that. For us, the purpose is exclusively for the programmer, not so much the user. Sometimes it is helpful to include additional detail beyond the commit header. The idea is to make sure this added detail is included in the changelog entry.
11 remaining items
codejedi365 commentedon Jan 6, 2025
Well since we copied the default templates the only way you would see a difference is if you specifically ran it against a version that had a commit body that has a bullet list in it (I forgot about this when creating my example project). The reason you have to move the release notes template is so that PSR detects your template and doesn't automatically fall back to the internal one. Your custom release notes templates import your custom changes template that displays commit bodies with hanging indented bullets
Got the invite, I can try it out tonight.
Will have to look into what didn't happen later. I was running everything locally.
marc-niclas commentedon Jan 7, 2025
Ok SO I gave it another shot and you were definitely right, it was using the default changelog template on the first PR.
The second PR seems to work wonderfully. What wasn't working earlier is resolved now, looks like the tags on the repo were behind what was already in the changelog so the changelog writer was intelligently omitting those entries so I just didn't give it enough credit.
The bad news is that I can't seem to replicate this success in my bigger repo. I have to deep dive tomorrow on why the sample monorepo is working and why its not working in my prod monorepo and what the differences might be. It's totally weird: it's writing the changelog in init mode (previous changelog is gettting wiped out so that's one problem), the changelog and version changes are getting committed to the repo BUT the github job fails with the following traceback. So I'm not sure why that's happening when the changelog entry is coming through, it is bizarre.
I'll follow up with what I find, just wanted to give a quick update...
ci/cd traceback in prod poweralpha repo
codejedi365 commentedon Jan 7, 2025
This is a totally random thought but I see you chose kebab-case for the base templates directory. Since Jinja uses Python compiler type things under the hood, maybe switch to snake_case because Python fails to import kebab-case file paths (probably one of the dumbest pythonic rules ever).
codejedi365 commentedon Jan 7, 2025
I didn't realize you were using a path filter so now that makes a bit more sense to how you were handling which service to release rather than solely relying on PSR to figure it out based on commits. The latter being how the open issues related to monorepos expect PSR to function.
Its probably because I'm super picky about the content of changelogs but since you had different changelogs (one for each service) I figured you would only want relevant commits to service 1 included in the service 1 changelog and no service 2 commits included in the service 1 changelog. Without a custom parser to differentiate when a commit is only relevant to a specific service, all commits that cause a change will be included into either changelog. You can see this in the example repo I made:
The above changelog excerpt came from the following unreleased commits.
Note that I tried to solve this problem with
exclude_commit_patternsusing a negative lookahead regular expression and it works for any commit that does not cause a version bump (i.e. notfeat,fix, orperf). The regular expression will attempt to match any conventional commit type with a scope prefix that is not the service itself (i.e.srvc1-for service 1).marc-niclas commentedon Jan 7, 2025
It ended up being really silly. Turns out I had a reference to ".base-changelog-template" and the folder was named ".base-changelog-templates". RE your comment regarding kebab-case, I have replaced this folder named with underscores.
Thanks again for all the superb support, definitely would have been quite lost without all the help. Cheers!
marc-niclas commentedon Jan 7, 2025
just saw this comment.
Ah fair enough, will give this a shot.
marc-niclas commentedon Jan 7, 2025
I just observed another behavior that's a bit strange. I tested a double-release, where 2 services are marked for release at the same time. In this scenario, I would expect both services to receive the same changelog entry, but with their respective versions. However, what seemed to happen is that srvc1 (the first service to release based on the ordering in cicd.yml) had its changelog overwritten by srvc2's changelog. You can see this behavior in this commit.
For now, I will take measures to prevent a double-release, since it's not a common scenario and generally we would want unique changelog entries. However, if you have any idea why this is happening, I would be curious but if not, we can put it as a parking lot item, as you continue to further support for monorepos.
codejedi365 commentedon Jan 7, 2025
@marc-at-brightnight, I'm not sure how the double-release differs from my offline test that calls both shell scripts sequentially. I feel this is part of your setup because for one it reinserted an "initial release" statement which means it didn't detect any previous tags?
I can't explain what happened in the other changelog. I do think the recursive copy of the base templates directory must only make one copy at a time otherwise it will write two changelogs when it releases a single service.
marc-niclas commentedon Jan 7, 2025
This was the right instinct. I added
rm -rf documentation/templates/documentationto each step before release and this solved the problem.This repo is now setup very similar to how prod poweralpha is setup, if its helpful for future development: https://github.com/marc-at-brightnight/psr-monorepo-poweralpha
codejedi365 commentedon Jan 7, 2025
I did work on a custom parser for monorepos yesterday (in line with path filtering in #614, and scoped variations discussed above), will try to do some testing now that I have an artificial repository. If this is of use or desire for you, then please let me know and it would be helpful if you are able to test it out further for your environment.
marc-niclas commentedon Jan 7, 2025
I think it could certainly be of use to monorepos with strict separation between packages. However, we have a libraries folder where services use common functions within. Changing the name or arguments, for example, of one of these library functions would result in changes in many services. This could cause releases in multiple services if we relied on path filtering only. Therefore, we opted for a more manual solution, where the developer has to specify which services to release, as I described earlier.
codejedi365 commentedon Jan 7, 2025
I think we are talking past each other. Yes, #614 wants path filtering, and yes with strict separation of packages they would not need your elegant solution to solve service release determination. I was not necessarily expecting to replace your release determination because of the unknown complexities of your system. If I could simplify it, I would, but ultimately I was only trying to solve the changelog issue described above which I had indicated would require a custom parser. I also added the scope relevancy as a filter and/or alternative to the path filtering variant, neither which would necessarily work for a shared library (thats new to me).
What is your opinion of the following?
only relevant commits to service 1 should be included in the service 1 changelog and no service 2 commits included in the service 1 changelog
marc-niclas commentedon Jan 14, 2025
understood 👍🏼
💯