Repository navigation
awk detection is broken on Musl-based Linux distros #132
Description
Activity
@jperkin responded on IRC on Sunday, March 17th, 2024:
I'd start from first-principles
firstly, does it not exist, or is /usr/pkg/bin/nawk there but somehow built with an invalid interpreter or something?
nawk is only bootstrapped if a native awk isn't found or is unsuitable, so I'd also check to see why your native awk isn't sufficient and if it can be provided/fixed instead
then if it is required and is missing, I'd be looking at the bootstrap log to figure out why it failed and how that wasn't reported as a failure to the bootstrap script
as this is Linux you'll be looking at mk/tools/tools.Linux.mk for how the native awk is found
To be continued.
Just to document my latest finding. It looks like the path for AWK is decided in
mk/tools/replace.mk:226 .if !defined(TOOLS_IGNORE.awk) && !empty(_USE_TOOLS:Mawk) 227 . if ${PKGPATH} == lang/nawk 228 MAKEFLAGS+= TOOLS_IGNORE.awk= 229 . elif !empty(_TOOLS_USE_PKGSRC.awk:M[yY][eE][sS]) 230 TOOLS_DEPENDS.awk?= nawk>=20040207:../../lang/nawk 231 TOOLS_CREATE+= awk 232 TOOLS_PATH.awk= ${TOOLBASE}/bin/nawk 233 . endif 234 .endif... with
TOOLBASEbeing set inmk/bsd.prefs.mk:607 LOCALBASE?= /usr/pkg 608 TOOLBASE:= ${LOCALBASE}- added a commit that references this issue
on Mar 25, 2024 It seems mk/tools/tools.Linux.mk tries to find "gawk" specifically, and otherwise will not assign anything. That whole file basically assumes GNU tooling. (You kind of already partly answered your own question there in #130 (comment) .)
A dependency may indeed be specified in mk/tools/replace.mk where you quote, in certain contexts only.
(If you're looking for a single place where everything is revealed, well, a lot of the pkgsrc infrastructure isn't like that. It's more complicated.)
Reacted by Saulius Krasuckas- added a commit that references this issue
on Mar 31, 2024 - changed the title
[-]awk detection is broken on a Busybox- and Musl-based distro[/-][+]awk detection is broken on a Busybox- and Musl-based distros[/+]on Apr 20, 2024 Confirmed on the following distros, attaching the logs inline:
Quoting @jperkin's IRC response and inviting for assistance:
firstly, does it not exist, or is /usr/pkg/bin/nawk there but somehow built with an invalid interpreter or something?
Surely the file does not exist yet:
# ls /usr/pkg/ ls: /usr/pkg/: No such file or directorynawk is only bootstrapped if a native awk isn't found or is unsuitable, so I'd also check to see why your native awk isn't sufficient and if it can be provided/fixed instead
My impression is that nawk was not bootstrapped. That's the root problem I see.
then if it is required and is missing, I'd be looking at the bootstrap log to figure out why it failed and how that wasn't reported as a failure to the bootstrap script
Jonathan, I still doesn't understand that part. Sounds too abstract, maybe even ambiguous to me.
A guidance is needed :)as this is Linux you'll be looking at mk/tools/tools.Linux.mk for how the native awk is found
The only awk references I found in
mk/tools/tools.Linux.mkare:105 . if exists(${_path}/gawk) 106 TOOLS_PLATFORM.gawk?= ${_path}/gawk 107 . endif ... 280 281 # assuming grep/sed/tar/awk are usually the GNU versions, is this safe? 282 283 .if !empty(TOOLS_PLATFORM.bison) 284 TOOLS_PLATFORM.bison-yacc?= ${TOOLS_PLATFORM.bison} -y 285 .endif 286 287 TOOLS_PLATFORM.awk?= ${TOOLS_PLATFORM.gawk} 288Which @dhgutteridge explained above:
It seems mk/tools/tools.Linux.mk tries to find "gawk" specifically, and otherwise will not assign anything. That whole file basically assumes GNU tooling.
Debian 12 also provides non-GNU implementation by default, the
mawk. Interestingly,pkgsrcdecides to bootstrap its' ownnawkhere just fine resulting in/usr/pkg/bin/nawk.From 05-bootstrap-builds-nawk-on-Debian-12.log:
--- /root/debug/src/github.com/NetBSD/pkgsrc.git/debug/logs/03-bootstrap-misses-nawk-on-Alpine-Linux.log +++ /root/debug/src/github.com/NetBSD/pkgsrc.git/debug/logs/05-bootstrap-builds-nawk-on-Debian-12.log @@ -1,16 +1,16 @@ ===> bootstrap command: bootstrap/bootstrap -===> bootstrap started: Fri Apr 19 20:26:12 EEST 2024 +===> bootstrap started: Sat Apr 20 08:29:37 AM EEST 2024 Working directory is: /root/debug/src/github.com/NetBSD/pkgsrc.git/work ... -===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh make-bootstrap.sh) ... -===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake +===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /usr/bin/sh make-bootstrap.sh) ... +===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake ===> Building libnbcompat -===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat -===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /bin/sh ./configure -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake -j1) +===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat +===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /usr/bin/sh ./configure -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake -j1) configure: creating cache config.cache -checking build system type... x86_64-pc-linux-musl -checking host system type... x86_64-pc-linux-musl +checking build system type... x86_64-pc-linux-gnu +checking host system type... x86_64-pc-linux-gnu checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out @@ -265,13 +267,11 @@ checking whether ln -s works... yes checking for ranlib... ranlib checking for gawk... no -checking for mawk... no -checking for nawk... no -checking for awk... awk +checking for mawk... mawk checking for ar... ar checking for fparseln in -lutil... no -checking for grep that handles long lines and -e... /bin/grep -checking for egrep... /bin/grep -E +checking for grep that handles long lines and -e... /usr/bin/grep +checking for egrep... /usr/bin/grep -E checking for _Bool... yes checking for stdbool.h that conforms to C99... yes checking for alloca.h... yes ... ranlib libnbcompat.a +===> Bootstrapping awk +===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk +===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake -j1 -f Makefile CC="cc" CFLAGS="") +cc -c awkgram.tab.c +cc -c b.c +cc -c main.c +cc -c parse.c +cc -c proctab.c +cc -c tran.c +cc -c lib.c +cc -c run.c +cc -c lex.c +cc awkgram.tab.o b.o main.o parse.o proctab.o tran.o lib.o run.o lex.o -lm -o a.out +===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk/a.out /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/nawk ===> Bootstrapping pkgtools
I wonder what makes the bootstrap fail to reach (or just skip) this step on the topic distros.
OK, found the answer in
bootstrap/bootstrap:737 # Debian/Ubuntu's awk is mawk, and mawk does not understand 738 # some regexps used in pkgsrc/mk. 739 if [ -f /etc/debian_version ]; then 740 need_awk=yes
So it's tied to the hardcoded OS name. Waiting for the suggestions on how to proceed in a clean way :)
- changed the title
[-]awk detection is broken on a Busybox- and Musl-based distros[/-][+]awk detection is broken on a Busybox- and Musl-based Linux distros[/+]on Apr 24, 2024 So it's tied to the hardcoded OS name. Waiting for the suggestions on how to proceed in a clean way :)
The main thing we would avoid changing (active users and all...) is Enterprise Linux. Perhaps you can tell us if GNU awk is specifically identifiable on distributions like CentOS @sskras (perhaps installed as "gawk" in some directory)? Otherwise we can set need_awk=yes on all Linux...
72 remaining items
- added 7 commits that reference this issue
on May 15, 2026 - added a commit that references this issue
on Jun 29, 2026 - added a commit that references this issue
on Jul 7, 2026 - added a commit that references this issue
on Jul 14, 2026 - added a commit that references this issue
on Sep 12, 2026
Forking the issue #130 (comment).
Currently I test only the OpenMPTCProuter distro, but OpenWRT probably suffers in the same way.
The distro provides BusyBox version of AWK which seems to have a mix of features from
nawk,gawkand probably some other implementations too: https://wiki.alpinelinux.org/wiki/AwkMy idea was that the
boostrap/bootstrapscript should either pick up the OS-native version (/usr/bin/awk -> ../../bin/busybox) or bootstrap its' own version (/usr/pkg/bin/nawk).The bootstrapping seems to to ignore the former and require the latter. The problem is that the latter isn't built yet at the moment it gets required:
(I trimmed down details from all earlier steps except for the last three ones)