Updates to bootstrap and build on macOS 14-15 / Xcode 16 - #267

How does your iconv pointer fix compare to Apple's upstream ?
https://github.com/apple-oss-distributions/libiconv/tree/libiconv-64

How does your iconv pointer fix compare to Apple's upstream ?

They are using size_t the same, but are passing 0 as altslash
https://github.com/apple-oss-distributions/libiconv/blob/a167071feb7a83a01b27ec8d238590c14eb6faff/libiconv/lib/utf8mac.h#L1575
which makes sense in a way since it is a u_int16_t, although the doc suggests it should be used as char – since it is never used at all, does not really matter, so I'll go with Apple's solution.

Tested bootstrap now on 15.1, too, and 14.6, which indeed has 5.34 as /usr/bin/perl as well. Still cannot tell if this has always been the default version on Sonoma.

I believe (but not certain) 14.x had perl5.30 earlier and now it has 5.34 Both at least are in /usr/bin now.

dpkg doesn't need to have its internal gettext bumped?

Building into an installed tree for comparison, libintl.8.dylib has bumped its compatibility version to 13.0.0 with %v=0.22.5

Error: Shlibs field says compatibility version for /opt/sw/lib/libintl.8.dylib is 10.0.0, but it is actually 13.0.0.

Good catch, forgot the bootstrap builds are all run without --validate.

dpkg doesn't need to have its internal gettext bumped?

The one in fink-distributions? Would probably make sense for consistency, but I have never installed that over the bootstrapped one – as it is still at 1.10.21 vs. 1.19.7, it would almost definitely not build on any arm64 system.

Oops about the dpkg note. I had forgotten that new dpkg no longer has an internal gettext.

I've created a PR for the same updates to fink-distributions.

Added a patch for Sonoma 14.7 and tested there too, but still only on the Sequoia 15.1 beta, since I can't seem to be able to downgrade that installation to the released 15.0.

I just created #269 which has a small subset of these changes to get bootstrap to "work" on 10.7; however I don't have the libcxx and other changes, so things end up running into issues with libiconv etc. I'll patch your version in and see if that fixes things

This is my first time contributing to Fink so I'm not sure what the community norms are; but if we have multiple folks working on this, wouldn't it be better to set up a development branch for this? (so we can more easily incorporate all contributions, and then merge into the main branch once it's ready to go)

I just tried this PR on macos14.7 and all the basic setup[1] seems to work without error; in particular libiconv appears to compile fine. There are still a bunch of warnings, but most of them are future-proofing things that should be handled upstream instead (sprintf deprecated, missing prototypes, etc). The only two warnings that seem particularly relevant to us are:

• configure: WARNING: unrecognized options: --with-finkvirtuals
• ld: warning: -bind_at_load is deprecated on macOS

However, after the basic setup, when I try installing things I get the error Can't resolve dependency "perl5303-core" for package "locale-gettext-pm5303-1.07-1" (same as with #269)

[1]: That is, ./bootstrap && fink selfupdate-git && fink self-update && fink index -f && fink update-all

Yes, I encountered the same issues testing the bootstrap again on 14.7, though I don't recall seeing it in my first attempt on Sonoma. I was hesitating to set perlcmd to perl5.34 as well, since as noted by #267 (comment), 5.34 seems to have only been introduced some time after the first 14.0 release. But because $system_perl_version is checking /usr/bin/perl, which now is 5.34.1, the all the relevant packages in fink-distributions will now need their system-perl5341 versions as well.
The only alternative I see is to somehow hack Services.pm to force $system_perl_version to the older version on 14.x only, but since the pm5341 versions will be needed for Sequoia anyway, I don't think that's a useful option.

where is this I'm in a position to test this today as I no longer have fink now :\

I'm just adding a fix to Bootstrapping for inject to force new perl, and I'll commit the core parts of this PR, I'm not testing a full bootstrapping so I can't comment on the package changes yet.

part of this PR was added, I haven't bootstrapped I only did an update via inject so I can't approve the info file changes yet.

Made a fresh bootstrap from main on 15.1; that works. But the

elsif ((&version_cmp($perlversion, '=', "5.34.1")) and Fink::Services::get_kernel_vers() == '24') {

will still make it fail on current 14.x when called with /usr/bin/perl[5.34], so I needed to change that part to get_kernel_vers() >= '23'.

MAIN also fails bootstrap on Sonoma now (having Xcode 16.1 as well) without the gettext + libiconv patches.

./utf8mac.h:1576:61: error: incompatible pointer to integer conversion passing 'void *' to parameter of type 'u_int16_t' (aka 'unsigned short') [-Wint-conversion]
 1576 |     ret = utf8_decodestr(s, n, ucsp, &ucslen, sizeof(ucsp), NULL, flags, &consumed);
...
In file included from ./iconv.c:112:
./encodings.def:75:15: error: incompatible function pointer types initializing 'int (*)(conv_t, ucs4_t *, const unsigned char *, size_t)' (aka 'int (*)(struct conv_struct *, unsigned int *, const unsigned char *, unsigned long)') with an expression of type 'int (conv_t, ucs4_t *, const unsigned char *, int)' (aka 'int (struct conv_struct *, unsigned int *, const unsigned char *, int)') [-Wincompatible-function-pointer-types]
   75 |             { utf8mac_mbtowc, NULL },     { utf8mac_wctomb, NULL })
      |               ^~~~~~~~~~~~~~
./iconv.c:111:5: note: expanded from macro 'DEFENCODING'
  111 |   { xxx_ifuncs1,xxx_ifuncs2, xxx_ofuncs1,xxx_ofuncs2, ei_##xxx##_oflags },
      |     ^~~~~~~~~~~
In file included from ./iconv.c:112:
./encodings.def:75:45: error: incompatible function pointer types initializing 'int (*)(conv_t, unsigned char *, ucs4_t, size_t)' (aka 'int (*)(struct conv_struct *, unsigned char *, unsigned int, unsigned long)') with an expression of type 'int (conv_t, unsigned char *, ucs4_t, int)' (aka 'int (struct conv_struct *, unsigned char *, unsigned int, int)') [-Wincompatible-function-pointer-types]
   75 |             { utf8mac_mbtowc, NULL },     { utf8mac_wctomb, NULL })
      |                                             ^~~~~~~~~~~~~~
./iconv.c:111:30: note: expanded from macro 'DEFENCODING'
  111 |   { xxx_ifuncs1,xxx_ifuncs2, xxx_ofuncs1,xxx_ofuncs2, ei_##xxx##_oflags },
      |                              ^~~~~~~~~~~

NB it does not properly fail though; despite

### execution of /tmp/fink.FaP_G failed, exit code 2
phase compiling: libiconv-1.16-3 failed

bootstrap proceeds to tell me (despite a missing dpkg-deb)

SKIPPING: Can't read control for 'dists/stable/main/binary-darwin-arm64/base/file-sharedir-pm_1.118-1.1_darwin-arm64.deb': No such file or directory

You should now have a working Fink installation in '/opt/sw4'. You still need package descriptions if you want to compile packages yourself. You can get them
by running either of the commands: 'fink selfupdate-rsync', to update via rsync (generally preferred); or 'fink selfupdate-svn', to update via svn (more likely
to work through a firewall). You should also run 'sudo xcodebuild -license' and accept the Xcode license, since some packages will require this.

Run '. /opt/sw4/bin/init.sh' to set up this terminal session environment to use Fink. To make the software installed by Fink available in all of your future
terminal shells, add '. /opt/sw4/bin/init.sh' to the init scripts '.zprofile', '.profile', or '.bash_profile' in your home directory. The program
/opt/sw4/bin/pathsetup.sh can help with this. Enjoy.

In fact at this point all there is in %p/bin is

apt-get-lockwait*	bzgrep*			fink*			gzexe*			xmlwf*			zgrep*
bunzip2*		bzip2*			fink-instscripts*	gzip*			zcat*			zless*
bzcat*			bzip2recover*		fink-scanpackages*	init.csh		zcmp*			zmore*
bzcmp@			bzless@			fink-virtual-pkgs*	init.sh			zdiff*			znew*
bzdiff*			bzmore*			gperf*			pager*			zegrep*
bzegrep@		dpkg-lockwait*		gunzip*			pathsetup.sh*		zfgrep*
bzfgrep@		editor*			gzcat@			uncompress*		zforce*

With just the updates in 10.9-libcxx from this PR libiconv and dpkg build, completing bootstrap, but then the next installation attempt fails at

Reading dependency for intltool40-0.51.0-1204...
WARNING: While resolving dependency "system-perl5303" for package "intltool40-0.51.0-1204", package "system-perl5303" was not found.
Can't resolve dependency "system-perl5303" for package "intltool40-0.51.0-1204" (no matching packages/versions found)
Exiting with failure.

So I think this confirms the above change to allow $perlversion="5.34.1" is needed on 23.6 (unless system-perl can somehow be forced to 5303 rather than 5341 on current Sonoma releases).

>= 23 will affect 26 etc etc so that needs to be tighter I honestly forgot about it I'll try to fix it right now

EDIT: nvm reading the code it'll work fine since it matches perl version first. Updating it now.

Yep, and can't know for sure what system perl 25 or 26 will have anyway – at least there is a hypothetical chance they may still have 5341.

totally, I thought this was a different section of code is why. This is perfectly fine and is now committed to main.

The package changes still haven't been added yet as I haven't had a chance to attempt to bootstrap fresh. Which I need to do for fink/fink-distributions#1031 at some point anyhow.

The package changes still haven't been added yet as I haven't had a chance to attempt to bootstrap fresh. Which I need to do for fink/fink-distributions#1031 at some point anyhow.

I have rebased and pulled in the updates from fink/fink-distributions#1158 for those (bootstrap last tested under 15.1.1).

Rebased including those two commits and bootstrapped this successfully on 15.3.1.

Just tried the latest version on {aarch64, OSX 14.7, Darwin 23.6.0, Perl 5.34.1}, and the bootstrap goes through okay (yay!), however when trying to install something I get Can't resolve dependency "perl5303-core" for package "locale-gettext-pm5303-1.07-1" (no matching packages/versions found).

I'm guessing that's because fink isn't choosing the right perl version? I'd offer to patch it, but I'm not quite sure where that code is (is it Fink::PkgVersion->get_perl_dir_arch?)

You do have perl5341-core and system-perl-5.34.1-1 then? I think everything should be set on this side, though you might still need to pull in #277 to get the 14.4 Distribution properly set up.
The perlmods part would rather be a fink-distributions issue, as most pm5341 updates are still in review limbo in fink/fink-distributions#1181. For your system there may possibly also a number of pm-14.4.info packages needed now.
Can you determine what is pulling in locale-gettext-pm5303 as a dependency (should be locale-gettext-pm5341)?

I do have perl5341-core (as virtual package), though it's not showing system-perl-5.34.1-1 as installed. (Fwiw my system does have /usr/bin/perl5.30 in addition to /usr/bin/perl5.34 ~= /usr/bin/perl; not that that helps.) I didn't have to pull #277 in; I tried, but all the changes are already in this PR or already landed on master

The dependency chain looks something like gnupg2 > ghostscript-esp > ??? > help2man-perl5303 > locale-gettext-pm5303. But I can't figure out what's pulling in help2man-perl5303; doing fink -vvv install ghostscript-esp looks the same as with -vv, and running fink show-deps on all the things before help2man-perl5303 doesn't show any culprits that I could notice. I'm not sure if there's a better approach for trying to get a list of all the recursive dependencies

fink list -f dotty[-build] -r | awk '$3~/help2man-perl/' should get it;
although I don't see it as direct dependency for ghostscript-esp must be help2man then – currently that resolves to

  (%type_pkg[systemperl] = 5303) 14.0,
  (%type_pkg[systemperl] = 5341) 15.0

But if your system correctly had Distribution: 14.4 that should actually result in "no package help2man available", since it would still need the addition

  (%type_pkg[systemperl] = 5341) 14.4,

I do have perl5341-core (as virtual package), though it's not showing system-perl-5.34.1-1 as installed. (Fwiw my system does have /usr/bin/perl5.30 in addition to /usr/bin/perl5.34 ~= /usr/bin/perl; not that that helps.) I didn't

That's strange; what does /usr/bin/perl -v say? Default should be 5.34.1 on recent Sonoma releases, thus the extra distribution for 14.4+. Should then have

 i   system-perl                      5.34.1-1                 [virtual package representing perl]
 p   system-perl5341                                           [virtual package]

There could be intermediate levels of dependencies in the graph. Possible culprits include flex and libtool2: they have dependencies on help2man and are themselves dependencies of many other packages possibly including those that ghostscript-esp lists as dependencies. I just pushed a commit that removes the flex->help2man dependency (wasn't actually used). I'm working on a similar fix for libtool2.

Yes, probably libtool2 via e.g. libidn.

fink/fink-distributions#1224 will remove libtool2:builddepends:help2man

The dependency chain looks something like gnupg2 > ghostscript-esp > ??? > help2man-perl5303 > locale-gettext-pm5303. But I can't figure out what's pulling in help2man-perl5303

Fwiw, looking through some older buildlogs (for earlier versions of this PR), I found a message saying that gsasl19-shlibs-2.2.1-2 was one wanting to pull in help2man. But yeah, as dmacks says, libtool2 is a more obvious transitive dependency and that does directly depend on help2man.

That's strange; what does /usr/bin/perl -v say? Default should be 5.34.1 on recent Sonoma releases, thus the extra distribution for 14.4+. Should then have

> ls -1 $(which perl)5*
/usr/bin/perl
/usr/bin/perl5.30
/usr/bin/perl5.34

> perl -v
This is perl 5, version 34, subversion 1 (v5.34.1) built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)
[...]

> fink list system-perl
Information about 15261 packages read in 1 seconds.
 i   system-perl          5.34.1-1        [virtual package representing perl]
 p   system-perl5341
# Okay, that's weird. Last time I checked it wasn't showing it...

> uname -r
23.6.0

> sw_vers  | grep ProductVersion
ProductVersion:		14.7

> grep Distribution /opt/sw/etc/fink.conf
Distribution: 14.0

oof. The perl bump from 5.30 to 5.34 in macOS 14.3 -> 14.4 is #276 and dealt with in #277. I'm very swamped at the moment. @wrengr there was the a question of $ver warning in #277 . Was that pinpointed to a specific .info or a general thing with any package?

Ugh, I just noticed that the branch I was working on hadn't pulled in the #277 stuff like it should've. Sorry about that. I just started over again with a clean branch: starting at d2a2290, then applying #277, then applying #267. So now the bootstrap correctly gets Distribution: 14.4. Still getting issues about libtool2 depending on help2man, but that should be handled by fink/fink-distributions#1224 once I pull that in.

I never did track down the root of the $ver warnings in #277. Just now on the new branch I ran fink selfupdate-git && fink update-all which updates:

The following package will be installed or updated: 
 objtools 
The following 10 additional packages will be installed: 
 cmake fink-package-precedence libidn2.0-dev libidn2.0-shlibs libunistring5 libunistring5-shlibs 
 libzstd1-dev libzstd1-shlibs lz4-dev lz4-shlibs 

And the uninitialized $ver error messages show up around root-fink-package-precedence-0.36-1, but that's the only one.

Could you verify if fink/fink-distributions/pull/1224 allows you to successfully build libtool2? If so, that will be one whole level of mess cleared up (and generally make things better even for older platforms).

Could you verify if fink/fink-distributions/pull/1224 allows you to successfully build libtool2? If so, that will be one whole level of mess cleared up (and generally make things better even for older platforms).

It seems to build okay with that PR. It has a bunch of -Wcompound-token-split-by-macro warnings, but afaict those are innocuous. Toward the end it did also give a warning:
dpkg-gensymbols: warning: unknown executable format in file '/opt/sw/src/fink.build/root-libtool2-shlibs-2.5.4-1/opt/sw/lib/libltdl.7.dylib' but I'm not sure whether that's serious or not.

However, despite it appearing to build fine I'm not entirely sure if it has or not. When trying to install ghostscript-esp (which depends on libtool) it ends up failing on one of the dependencies xft2-dev-2.2.0-5 (which does not directly depend on libtool). The last line before the error is an invocation of libtool, which gives linker errors about /opt/X11/lib/libXrender.1.dylib and /opt/X11/lib/libX11.6.dylib not having symbols for arm64.

Can you post the actual command and error message about symbols and libX11 ? The Xquartz installer 2.8.5 (latest stable) should be arm64 compatible:

nieder $ lipo -info /opt/X11/lib/libX11.6.dylib 
Architectures in the fat file: /opt/X11/lib/libX11.6.dylib are: i386 x86_64 arm64 

And what's the output of otool -hv /opt/sw/lib/libltdl.7.dylib ?

$ otool -hv /sw/lib/libltdl.7.dylib
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64  X86_64        ALL  0x00       DYLIB    13       1536   NOUNDEFS DYLDLINK TWOLEVEL NO_REEXPORTED_DYLIBS

The fragment of the error log:

/bin/sh ../libtool  --tag=CC   --mode=link gcc -Wall -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fno-strict-aliasing -Wbad-function-cast -Wformat=2  -g -O2 -version-number 2:2:0 -no-undefined -L/opt/sw/lib/aarch64-darwin -L/opt/sw/lib -Wl,-headerpad_max_install_names -o libXft.la -rpath /opt/sw/lib/xft2/lib xftcolor.lo xftcore.lo xftdbg.lo xftdpy.lo xftdraw.lo xftextent.lo xftfont.lo xftfreetype.lo xftglyphs.lo xftinit.lo xftlist.lo xftname.lo xftrender.lo xftstr.lo xftswap.lo xftxlfd.lo -L/opt/sw/lib -lfontconfig -lfreetype -L/opt/sw/lib -lfreetype -L/opt/X11/lib -lXrender -lX11  
libtool: link: gcc -dynamiclib  -o .libs/libXft.2.dylib  .libs/xftcolor.o .libs/xftcore.o .libs/xftdbg.o .libs/xftdpy.o .libs/xftdraw.o .libs/xftextent.o .libs/xftfont.o .libs/xftfreetype.o .libs/xftglyphs.o .libs/xftinit.o .libs/xftlist.o .libs/xftname.o .libs/xftrender.o .libs/xftstr.o .libs/xftswap.o .libs/xftxlfd.o   -L/opt/sw/lib/aarch64-darwin -L/opt/sw/lib /opt/sw/lib/fontconfig2/lib/libfontconfig.dylib /opt/sw/lib/freetype219/lib/libfreetype.dylib -L/opt/X11/lib -lXrender -lX11  -Wl,-headerpad_max_install_names   -pthread -install_name  /opt/sw/lib/xft2/lib/libXft.2.dylib -compatibility_version 5 -current_version 5.0  
ld: warning: ignoring file '/opt/X11/lib/libXrender.1.dylib': fat file missing arch 'arm64', file has 'i386,x86_64' 
ld: warning: ignoring file '/opt/X11/lib/libX11.6.dylib': fat file missing arch 'arm64', file has 'i386,x86_64' 
Undefined symbols for architecture arm64: 
  [...]
ld: symbol(s) not found for architecture arm64 
clang: error: linker command failed with exit code 1 (use -v to see invocation) 
make[2]: *** [libXft.la] Error 1 
make[1]: *** [all-recursive] Error 1 
make: *** [all] Error 2 

The requested command output:

$ lipo -info /opt/X11/lib/libX11.6.dylib
Architectures in the fat file: /opt/X11/lib/libX11.6.dylib are: i386 x86_64
$ otool -hv /opt/sw/lib/libltdl.7.dylib
/opt/sw/lib/libltdl.7.dylib:
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64    ARM64        ALL  0x00       DYLIB    16       1376   NOUNDEFS DYLDLINK TWOLEVEL NO_REEXPORTED_DYLIBS

For X11, it looks like you have an older version of XQuartz installed that doesn't have arm64. Release 2.8.0 from 2021 was the first one to have Apple Silicon support and current release is 2.8.5.

For libtool2, that looks fine. Not sure what happened during the build and I don't know now dpkg-gensymbols works

Why hasn't this been merged yet? It's been a year with no activity. What's the hold-up?

Ping

Why hasn't this been merged yet? It's been a year with no activity. What's the hold-up?

Unless support has been added already in a different commit/PR, in which case this PR should be closed

Sidenote: My original comment from January was a bit too aggressive, and I apologize for that

I have not followed the libX11 issues above, and don't have a Sequoia system to test on at this time, but it seems all relevant changes (except perhaps removing the broken fink-mirrors) have made it into main, which is bootstrapping successfully on Tahoe 26.4. So I am inclined to to close this.

I have not followed the libX11 issues above, and don't have a Sequoia system to test on at this time, but it seems all relevant changes (except perhaps removing the broken fink-mirrors) have made it into main, which is bootstrapping successfully on Tahoe 26.4. So I am inclined to to close this.

Perhaps you could install in a VM (assuming you have a Mac), VMware Fusion is free nowadays and supports macOS guests on both Intel and Apple Silicon. And I do think both macOS 13 and 14 hosts are still supported by the latest version of Fusion, which does support macOS 15 guests (if you're on 11 or 12 you might also be able to use a slightly earlier version of Fusion with support for macOS 15 guests)

Edit: I was wrong, although MacOS guests are supported in Fusion for Intel hosts, they are NOT supported for Apple Silicon. However, you could instead use Parallels (paid) or UTM (free, based on QEMU), both of these should work on either Intel or Apple Silicon hosts

I've rebased to remove the spurious timestamp-only changes, showing that the only programmatic change remaining is the altslash -> (int) 0 replacement. This actually bootstraps both with and without the change on 26.4 and Big Sur 11.7 with Xcode 13.2, so I am not sure if the pointer mismatch was an Xcode 16-only failure, or if it has been resolved there as well. I suggest to squash if merging this.
Unless I hear otherwise from @wrengr, I assume the libX11 issues were an XQuartz version problem indeed. I never saw anything of that kind even when testing this on Sonoma and Sequoia.