Repository navigation
mtree bugs with filenames containing wildcards #101
Description
Activity
Isn't mtree part of the base system? Could you send this as a bug report upstream please? (https://www.netbsd.org/support/send-pr.html)
- Sure - pkg/56733 submitted. I'm not quite sure where to go with all these things; my apologies.
I also submitted an upstream issue (about another problem) on 2022-01-10.
Received zero response so far.So I am not sure what is the workflow here.
@bsiegert So I just realized something:
There is mtree in two places:
https://github.com/NetBSD/pkgsrc/tree/trunk/pkgtools/mtree/files (which is what this issue pertains to)
and
https://github.com/NetBSD/src/tree/trunk/usr.sbin/mtree
(and of course their CVS origins)
I note that these two trees aren't identical, and in fact are rather divergent. The pkgsrc version -- this one -- is portable, and in fact I maintain mtree-netbsd for Debian based on this version.
The NetBSD/src version is not portable, but appears newer (for instance, having added -F).
I did have engagement on 56733 but did not at the time realize that we were talking about two different mtrees!
So I am very much talking about the pkgsrc one, which is https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/pkgtools/mtree/index.html . Presumably it is portable because pkgsrc itself is portable, to *BSD, Linux, and so forth.
I don't know how the NetBSD/src mtree flows into the pkgsrc mtree -- perhaps, at the moment, the answer is "it doesn't" - fair enough. All I know is that this patch fixes the issue for me:
--- verify.c.orig 2018-08-22 15:48:38.000000000 -0500 +++ verify.c 2022-03-01 09:45:04.854574448 -0600 @@ -142,9 +142,7 @@ if (specdepth != p->fts_level) goto extra; for (ep = level; ep; ep = ep->next) - if ((ep->flags & F_MAGIC && - !fnmatch(ep->name, p->fts_name, FNM_PATHNAME)) || - !strcmp(ep->name, p->fts_name)) { + if (!strcmp(ep->name, p->fts_name)) { ep->flags |= F_VISIT; if (compare(ep, p)) rval = MISMATCHEXIT;Noting this from an email conversation:
There are some other mtree implementations and forks from NetBSD; for instance:
- https://github.com/archiecobbs/mtree-port (fork from pkgsrc; very old, doesn't even have -C)
- https://github.com/archiecobbs/nmtree (more current fork from pkgsrc)
Like the pkrsrc mtree, these two seem to be outdated relative to the
usr.sbin mtree.Other implementations:
- libarchive, of course. (does not do verification)
- FreeBSD has an semi-compatible mtree. I found it has issues with
filenames containing tildes, so I avoided it. - https://github.com/mratajsky/libmtree (appears to originate from
FreeBSD; apparently abandoned) - https://github.com/eric-switzer/mtree-parse (Python)
- https://github.com/vbatts/go-mtree (re-implementation in Go)
- https://github.com/vbatts/go-mtree (Rust parser)
The NetBSD pkgsrc tree is directly used as part of the Arch Linux
packaging format; https://aur.archlinux.org/packages/nmtree for details.
They also have a "git" mtree derived from archiecobbs/nmtree as above.So effectively NetBSD pkgsrc tree is the ultimate upstream for a number
of other projects. It is probably reasonable to point many of those to
use archiecobbs/nmtree but I thought I ought to write down all the
options here for others that may be looking.- added 2 commits that reference this issue
on May 2, 2022 - added 2 commits that reference this issue
on Aug 11, 2022 - added a commit that references this issue
on Sep 21, 2022 - added a commit that references this issue
on Oct 18, 2022 - added a commit that references this issue
on Nov 14, 2022 - added a commit that references this issue
on Dec 7, 2022 - added a commit that references this issue
on Jan 22, 2023 139 remaining items
- added 7 commits that reference this issue
on May 15, 2026 - added a commit that references this issue
on May 21, 2026 - added 2 commits that reference this issue
on May 31, 2026 - added a commit that references this issue
on Jun 24, 2026 - added a commit that references this issue
on Jul 14, 2026 - added 2 commits that reference this issue
on Jul 26, 2026 - added a commit that references this issue
on Sep 24, 2026
Let's say I have a directory containing these files, one of which contains an asterisk:
file*
file2.txt
file.txt
file3.txt
Further assume they are different sizes and have different content.
If, in the output from mtree -c, file* is emitted before the others, the later mtree verify will apparently match all four files using the rule file* and emit errors on the other three. I observed this in the wild.
A workaround is to manually edit the emitted mtree data and move file* to be after the other non-wildcard files; then it will properly verify.
Additionally, the support for wildcards in the mtree format is not documented that I can see, and probably should be. Or perhaps it should be removed, since AFAICT, no tools are generating mtree files intentionally using this feature.
I observe a call to fnmatch in verify.c which MAY be the cause of this issue. (I am not certain).