Bug: Relative template_dir path causes TemplateNotFound error in monorepo configuration

The problem

TemplateNotFound error thrown by Jinja2 when generating changelog in a subfolder. The directory structure is as follows:

- main_program
+-- templates
    +-- main_program
        +-- CHANGELOG.md.j2
+-- CHANGELOG.md
+-- pyproject.toml
+-- ...
- lib1
- lib2

To be clear, the templates are discovered by semantic-release, but when they're passed to Jinja, a relative path is used, and Jinja is not able to find them.

Expected behavior

I expect a generated CHANGELOG.md in the same level as pyproject.toml by using the templates provided in the configuration.

Environment

Python Version: v3.8.18
Pip Version: v23.0.1
Semantic Release Version: v9.1.0

pip freeze

annotated-types==0.6.0
antlr4-python3-runtime==4.13.1
antlr4-tools==0.2
argcomplete==3.2.2
argparse-addons==0.12.0
asn1tools==0.166.0
bcrypt==4.0.1
Beaker==1.12.1
beautifulsoup4==4.12.2
bitstruct==8.17.0
build==1.0.3
cantools==39.0.0
capstone==5.0.0rc2
certifi==2024.2.2
cffi==1.15.1
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
colored-traceback==0.3.0
crccheck==1.3.0
cryptography==40.0.2
cssselect==1.2.0
decli==0.6.1
decorator==5.1.1
diskcache==5.6.1
docutils==0.20.1
dotty-dict==1.3.1
exceptiongroup==1.1.3
filebytes==0.10.2
FormEncode==2.1.0
future==0.18.3
gitdb==4.0.11
GitPython==3.1.42
idna==3.6
importlib-metadata==7.0.1
importlib-resources==6.1.1
iniconfig==2.0.0
install-jdk==1.1.0
intervaltree==3.1.0
jaraco.classes==3.3.1
jeepney==0.8.0
Jinja2==3.1.3
keyring==24.3.0
lark==1.1.8
-e git+<redacted>
lxml==4.9.3
Mako==1.2.4
markdown-it-py==3.0.0
MarkupSafe==2.1.2
mdurl==0.1.2
mikai==4.0.1
more-itertools==10.2.0
msgpack==1.0.5
-e git+<redacted>
nh3==0.2.15
nose==1.3.7
numpy==1.24.4
packaging==23.1
paramiko==3.1.0
Paste==3.7.1
PasteDeploy==3.1.0
PasteScript==3.3.0
pkginfo==1.9.6
pluggy==1.3.0
plumbum==1.8.1
prompt-toolkit==3.0.36
psutil==5.9.5
pwntools==4.9.0
pycparser==2.21
pydantic==2.6.1
pydantic_core==2.16.2
pyelftools==0.29
Pygments==2.16.1
Pylons==1.0.3
PyNaCl==1.5.0
pyparsing==3.1.0
pyproject_hooks==1.0.0
pyserial==3.5
PySide6==6.6.1
PySide6-Addons==6.6.1
PySide6-Essentials==6.6.1
PySocks==1.7.1
pytest==7.4.2
python-can==4.2.2
python-dateutil==2.8.2
python-gitlab==4.4.0
python-semantic-release==9.1.0
pytz==2023.3
PyYAML==6.0.1
questionary==2.0.1
readme-renderer==42.0
repoze.lru==0.7
requests==2.31.0
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.7.0
ROPGadget==7.3
ropper==1.13.8
Routes==2.5.1
rpyc==5.3.1
SecretStorage==3.3.3
shellingham==1.5.4
shiboken6==6.6.1
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
sortedcontainers==2.4.0
soupsieve==2.5
Tempita==0.5.2
termcolor==2.4.0
textparser==0.24.0
tomli==2.0.1
tomlkit==0.12.3
twine==5.0.0
typing_extensions==4.7.1
tzdata==2023.3
unicorn==2.0.1.post1
urllib3==2.2.1
waitress==2.1.2
wcwidth==0.2.9
WebError==0.13.1
WebHelpers==1.3
WebOb==1.8.7
WebTest==3.0.0
wrapt==1.15.0
zipp==3.17.0

Using build==1.0.3 and setuptools=56.0.0.

Configuration

[tool.semantic_release]
assets = []
commit_message = "chore: bumped version to {version}\n\nAutomatically generated by python-semantic-release"
commit_parser = "angular"
logging_use_named_masks = false
major_on_zero = true
tag_format = "main_program-v{version}"
version_variables = [
  "main_program/__init__.py:__version__"
]


[tool.semantic_release.changelog]
template_dir = "main_program/templates"
changelog_file = "CHANGELOG.md"
exclude_commit_patterns = []

Logs

GitHub Attachment to Logs

Additional context

There's something wrong here:

python-semantic-release/semantic_release/changelog/template.py

Lines 77 to 123 in 82bfcd3

    
           def recursive_render( 
        
               template_dir: Path, 
        
               environment: Environment, 
        
               _root_dir: str | os.PathLike[str] = ".", 
        
           ) -> list[str]: 
        
               rendered_paths: list[str] = [] 
        
               for root, file in ( 
        
                   (Path(root), file) 
        
                   for root, _, files in os.walk(template_dir) 
        
                   for file in files 
        
                   # we slice relpath.parts[1:] to allow the top-level 
        
                   # template folder to have a dot prefix. 
        
                   # e.g. to permit ".github/psr-templates" to contain the templates, 
        
                   # rather than enforcing a top-level, non-hidden directory 
        
                   if not any( 
        
                       elem.startswith(".") 
        
                       for elem in Path(root).relative_to(template_dir).parts[1:] 
        
                   ) 
        
                   and not file.startswith(".") 
        
               ): 
        
                   output_path = (_root_dir / root.relative_to(template_dir)).resolve() 
        
                   log.info("Rendering templates from %s to %s", root, output_path) 
        
                   output_path.mkdir(parents=True, exist_ok=True) 
        
                   if file.endswith(".j2"): 
        
                       # We know the file ends with .j2 by the filter in the for-loop 
        
                       output_filename = file[:-3] 
        
                       # Strip off the template directory from the front of the root path - 
        
                       # that's the output location relative to the repo root 
        
                       src_file_path = str((root / file).relative_to(template_dir)) 
        
                       output_file_path = str((output_path / output_filename).resolve()) 
        
                       log.debug("rendering %s to %s", src_file_path, output_file_path) 
        
                       stream = environment.get_template(src_file_path).stream() 
        
                       with open(output_file_path, "wb+") as output_file: 
        
                           stream.dump(output_file, encoding="utf-8") 
        
                       rendered_paths.append(output_file_path) 
        
                   else: 
        
                       src_file = str((root / file).resolve()) 
        
                       target_file = str((output_path / file).resolve()) 
        
                       log.debug( 
        
                           "source file %s is not a template, copying to %s", src_file, target_file 
        
                       ) 
        
                       shutil.copyfile(src_file, target_file) 
        
                       rendered_paths.append(target_file) 
        
               return rendered_paths

specifically, line 109

python-semantic-release/semantic_release/changelog/template.py

Lines 107 to 110 in 82bfcd3

    
           log.debug("rendering %s to %s", src_file_path, output_file_path) 
        
           stream = environment.get_template(src_file_path).stream()

@deme3

@deme3 Thanks for the bug report and the initial troubleshooting you have accomplished. I have been working an extensive rewrite of the changelog testing and this will definitely be one of the new tests to add along with a fix.

@codejedi365

@codejedi365 thanks for letting me know. :)

Until we get the new release with the changelog rewrite, I can suggest the following patch for whoever is using this tool in a monorepo like me (note that I have no idea if this is good, I haven't looked much into the codebase, I just needed a quick fix for now):

unchanged:
--- a/cli/commands/changelog.py
+++ b/cli/commands/changelog.py
@@ -3,6 +3,7 @@ from __future__ import annotations
 import logging
 import os
 from typing import TYPE_CHECKING
+from pathlib import Path
 
 import click
 
@@ -76,7 +77,8 @@ def changelog(ctx: click.Context, release_tag: str | None = None) -> None:
                 f"{template_dir!r} relative to {repo.working_dir!r}"
             )
         else:
-            recursive_render(template_dir, environment=env, _root_dir=repo.working_dir)
+            project_root = Path(runtime.global_cli_options.config_file).absolute().parent
+            recursive_render(template_dir, environment=env, _root_dir=project_root)
 
     if release_tag:
         if runtime.global_cli_options.noop:
@@ -116,4 +118,4 @@ def changelog(ctx: click.Context, release_tag: str | None = None) -> None:
                 )
             except Exception as e:
                 log.exception(e)
-                ctx.fail(str(e))
\ No newline at end of file
+                ctx.fail(str(e))
unchanged:
--- a/cli/config.py
+++ b/cli/config.py
@@ -395,9 +395,10 @@ class RuntimeContext:
         )
 
         # changelog_file
+        project_root = Path(global_cli_options.config_file).absolute().parent
         changelog_file = Path(raw.changelog.changelog_file).resolve()
 
-        template_dir = Path(repo.working_tree_dir or ".") / raw.changelog.template_dir
+        template_dir = Path(project_root or ".") / raw.changelog.template_dir
 
         template_environment = environment(
             template_dir=raw.changelog.template_dir,
@@ -434,4 +435,4 @@ class RuntimeContext:
         # credential masker
         self.apply_log_masking(self.masker)
 
-        return self
\ No newline at end of file
+        return self
only in patch2:
unchanged:
--- a/cli/commands/version.py
+++ b/cli/commands/version.py
@@ -5,6 +5,7 @@ import os
 import subprocess
 from contextlib import nullcontext
 from datetime import datetime
+from pathlib import Path
 from typing import TYPE_CHECKING, ContextManager, Iterable
 
 import click
@@ -402,6 +403,7 @@ def version(  # noqa: C901
         hvcs_client=hvcs_client, release_history=rh
     )
     changelog_context.bind_to_environment(env)
+    project_root = Path(runtime.global_cli_options.config_file).absolute().parent
 
     updated_paths: list[str] = []
     if update_changelog:
@@ -409,13 +411,13 @@ def version(  # noqa: C901
             if opts.noop:
                 noop_report(
                     f"would have recursively rendered the template directory "
-                    f"{template_dir!r} relative to {repo.working_dir!r}. "
+                    f"{template_dir!r} relative to {project_root!r}. "
                     "Paths which would be modified by this operation cannot be "
                     "determined in no-op mode."
                 )
             else:
                 updated_paths = recursive_render(
-                    template_dir, environment=env, _root_dir=repo.working_dir
+                    template_dir, environment=env, _root_dir=project_root
                 )
 
         else:
@@ -425,13 +427,13 @@ def version(  # noqa: C901
             if opts.noop:
                 noop_report(
                     "would have written your changelog to "
-                    + str(changelog_file.relative_to(repo.working_dir))
+                    + str(changelog_file.relative_to(project_root))
                 )
             else:
                 changelog_text = render_default_changelog_file(env)
                 changelog_file.write_text(changelog_text, encoding="utf-8")
 
-            updated_paths = [str(changelog_file.relative_to(repo.working_dir))]
+            updated_paths = [str(changelog_file.relative_to(project_root))]
 
         if commit_changes and opts.noop:
             noop_report(

which instead of using the repository root directory as reference, uses the folder where the config is located.

@deme3

@deme3 I hope to have this in next week. My PR has been pending since the weekend.

I was a bit concerned that this could break other changelog designs but we didn't have any testing in place to validate either scenario

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

Need to circle back around to this issue. It slipped my mind, unfortunately the patch above will not work anymore given the refactor I did in v9.8.4 but if you port it to the changelog_writer.py file it should be easy enough to get back to your fix. I don't have a good pythonic monorepo example to go off of which is been part of the delay in support of monorepos. If you have an example open source project I can mirror for testing that would be great.

@codejedi365

@codejedi365 https://github.com/langchain-ai/langchain/tree/master/libs is a pretty good open source python monorepo

@JonZeolla

@JonZeolla, thank you!

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

Still in backlog

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

Still in backlog

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

Still in backlog

It has been 90 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

Still in the backlog

It has been 90 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

Navigation Menu

Repository navigation

Bug: Relative template_dir path causes TemplateNotFound error in monorepo configuration #845

The problem

Expected behavior

Environment

Configuration

Logs

Additional context

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def recursive_render(
	template_dir: Path,
	environment: Environment,
	_root_dir: str \| os.PathLike[str] = ".",
	) -> list[str]:
	rendered_paths: list[str] = []
	for root, file in (
	(Path(root), file)
	for root, _, files in os.walk(template_dir)
	for file in files
	# we slice relpath.parts[1:] to allow the top-level
	# template folder to have a dot prefix.
	# e.g. to permit ".github/psr-templates" to contain the templates,
	# rather than enforcing a top-level, non-hidden directory
	if not any(
	elem.startswith(".")
	for elem in Path(root).relative_to(template_dir).parts[1:]
	)
	and not file.startswith(".")
	):
	output_path = (_root_dir / root.relative_to(template_dir)).resolve()
	log.info("Rendering templates from %s to %s", root, output_path)
	output_path.mkdir(parents=True, exist_ok=True)
	if file.endswith(".j2"):
	# We know the file ends with .j2 by the filter in the for-loop
	output_filename = file[:-3]
	# Strip off the template directory from the front of the root path -
	# that's the output location relative to the repo root
	src_file_path = str((root / file).relative_to(template_dir))
	output_file_path = str((output_path / output_filename).resolve())

	log.debug("rendering %s to %s", src_file_path, output_file_path)
	stream = environment.get_template(src_file_path).stream()

	with open(output_file_path, "wb+") as output_file:
	stream.dump(output_file, encoding="utf-8")

	rendered_paths.append(output_file_path)
	else:
	src_file = str((root / file).resolve())
	target_file = str((output_path / file).resolve())
	log.debug(
	"source file %s is not a template, copying to %s", src_file, target_file
	)
	shutil.copyfile(src_file, target_file)
	rendered_paths.append(target_file)
	return rendered_paths

Bug: Relative template_dir path causes TemplateNotFound error in monorepo configuration #845

Description

The problem

Expected behavior

Environment

Configuration

Logs

Additional context

Activity

codejedi365 commented on Feb 23, 2024

deme3 commented on Feb 23, 2024

codejedi365 commented on Mar 6, 2024

github-actions commented on Jul 5, 2024

codejedi365 commented on Jul 6, 2024

JonZeolla commented on Aug 12, 2024

codejedi365 commented on Aug 13, 2024

github-actions commented on Dec 10, 2024

codejedi365 commented on Dec 10, 2024

5 remaining items

github-actions commented on Apr 12, 2025

codejedi365 commented on Apr 18, 2025

github-actions commented on Jun 18, 2025

codejedi365 commented on Jun 18, 2025

github-actions commented on Sep 18, 2025

codejedi365 commented on Sep 24, 2025

github-actions commented on Dec 23, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions