Repository navigation
Hetzner Benchmarking Machine Replacements #3657
Description
Activity
These machines have been provisioned and added to Jenkins with the same labels/configs as the former nearform machines.
Next steps: Can somebody with permssions can kick off a benchmarking jobs and some v8 builds to verify that all is working as intended?
I've marked the two Nearform machines offline in Jenkins and started a V8 build which is running on test-hetzner-ubuntu2204-x64-1:
https://ci.nodejs.org/job/node-test-commit-v8-linux/5876/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/I've marked the two Nearform machines offline in Jenkins and started a V8 build which is running on test-hetzner-ubuntu2204-x64-1: https://ci.nodejs.org/job/node-test-commit-v8-linux/5876/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/
This has failed:
02:47:26 + DEPOT_TOOLS_DIR=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools 02:47:26 + PATH=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools:/home/iojs/build/workspace/node-test-commit-v8-linux/depot_tools:/home/iojs/venv/bin:/home/iojs/nghttp2/src:/home/iojs/wrk:/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin tools/dev/v8gen.py x64.release --no-goma 02:47:26 02:47:26 Hint: You can raise verbosity (-vv) to see the output of failed commands. 02:47:26 02:47:26 Traceback (most recent call last): 02:47:26 File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 309, in <module> 02:47:26 sys.exit(gen.main()) 02:47:26 File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 303, in main 02:47:26 return self._options.func() 02:47:26 File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 162, in cmd_gen 02:47:26 self._call_cmd([ 02:47:26 File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 211, in _call_cmd 02:47:26 output = subprocess.check_output( 02:47:26 File "/usr/lib/python3.10/subprocess.py", line 421, in check_output 02:47:26 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, 02:47:26 File "/usr/lib/python3.10/subprocess.py", line 526, in run 02:47:26 raise CalledProcessError(retcode, process.args, 02:47:26 subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'tools/mb/mb.py', 'gen', '-f', 'infra/mb/mb_config.pyl', '-m', 'developer_default', '-b', 'x64.release', 'out.gn/x64.release']' returned non-zero exit status 1. 02:47:26 make: *** [Makefile:303: v8] Error 1
Logging into the machine and running the failing command with
-vv(as suggested):iojs@test-hetzner-ubuntu2204-x64-1:~/build/workspace/node-test-commit-v8-linux/deps/v8$ PATH=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools:/home/iojs/build/workspace/node-test-commit-v8-linux/depot_tools:/home/iojs/venv/bin:/home/iojs/nghttp2/src:/home/iojs/wrk:/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin tools/dev/v8gen.py x64.release --no-goma -vv ################################################################################ /usr/bin/python3 -u tools/mb/mb.py gen -f infra/mb/mb_config.pyl -m developer_default -b x64.release out.gn/x64.release Writing """\ dcheck_always_on = false is_debug = false target_cpu = "x64" """ to /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/x64.release/args.gn. /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/buildtools/linux64/gn gen out.gn/x64.release --check -> returned 1 ERROR at //build/config/linux/pkg_config.gni:104:17: Script returned non-zero exit code. pkgresult = exec_script(pkg_config_script, args, "json") ^---------- Current dir: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/x64.release/ Command: python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py -s /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/linux/debian_bullseye_amd64-sysroot -a x64 glib-2.0 gmodule-2.0 gobject-2.0 gthread-2.0 Returned 1. stderr: Traceback (most recent call last): File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py", line 247, in <module> sys.exit(main()) File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py", line 142, in main prefix = GetPkgConfigPrefixToStrip(options, args) File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py", line 80, in GetPkgConfigPrefixToStrip prefix = subprocess.check_output([options.pkg_config, File "/usr/lib/python3.10/subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.10/subprocess.py", line 503, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.10/subprocess.py", line 971, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'pkg-config' See //build/config/linux/BUILD.gn:58:3: whence it was called. pkg_config("glib") { ^------------------- See //build/config/compiler/BUILD.gn:300:18: which caused the file to be included. configs += [ "//build/config/linux:compiler" ] ^------------------------------ GN gen failed: 1 Traceback (most recent call last): File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 309, in <module> sys.exit(gen.main()) File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 303, in main return self._options.func() File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 162, in cmd_gen self._call_cmd([ File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 211, in _call_cmd output = subprocess.check_output( File "/usr/lib/python3.10/subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'tools/mb/mb.py', 'gen', '-f', 'infra/mb/mb_config.pyl', '-m', 'developer_default', '-b', 'x64.release', 'out.gn/x64.release']' returned non-zero exit status 1. iojs@test-hetzner-ubuntu2204-x64-1:~/build/workspace/node-test-commit-v8-linux/deps/v8$
Looking at the other nearform benchmarking machines I see a considerable amount of manual package installations that are outside of ansible's setup.
There's at least 220 packages that have been manually installed.
Theres also the likelyhood that I was supposed to configure ansible differently to set up these machines beyond what I understood.
I attempted to add the 'is_benchmark' = true to add the benchmark role as I see that pkg-config is installed that way, but some of the other packages in that role have been dropped from ubuntu and that role hasnt been updated for six years, so its probably got some stale packages in there.
List of missing packages
apport-symptoms aptitude apt-transport-https bind9-host bison bsdmainutils build-essential busybox-initramfs bzip2 ca-certificates clang cloud-guest-utils cloud-initramfs-copymods cloud-initramfs-dyn-netconf console-setup-linux coreutils cryptsetup-bin debianutils dh-python distro-info-data dmeventd dmsetup dnsmasq-base dns-root-data dnsutils dpkg eject file flex ftp gawk geoip-database gettext-base git-man gnutls-bin gpgv groff-base grub-common grub-legacy-ec2 grub-pc ifenslave ifupdown initramfs-tools initramfs-tools-bin initramfs-tools-core init-system-helpers install-info isc-dhcp-common keyboard-configuration klibc-utils krb5-locales language-selector-common libacl1 libapparmor1 libatm1 libattr1 libaudit1 libaudit-common libblkid1 libbsd0 libbz2-1.0 libc6 libcap2 libcap2-bin libcap-dev libc-ares-dev libc-bin libcunit1-dev libcurl4 libdb5.3 libdevmapper1.02.1 libdevmapper-event1.02.1 libdumbnet1 liberror-perl libestr0 libev-dev libevent-dev libexpat1 libfdisk1 libfribidi0 libgcrypt20 libglib2.0-data libgmp10 libgnutls30 libgnutls-openssl27 libgpg-error0 libgpm2 libjansson-dev libkeyutils1 libklibc libkmod2 liblocale-gettext-perl liblxc1 liblz4-1 liblzma5 liblzo2-2 libmagic1 libmnl0 libmount1 libmspack0 libncurses5 libncursesw5 libnetfilter-conntrack3 libnewt0.52 libp11-kit0 libpam0g libpam-modules libpam-modules-bin libpam-runtime libpcre3 libpolkit-agent-1-0 libpopt0 libpython3-stdlib libreadline6 libsasl2-modules libseccomp2 libselinux1 libsemanage-common libsigsegv2 libslang2 libsmartcols1 libsqlite3-0 libss2 libssl-dev libstdc++6 libsystemd0 libtasn1-6 libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl libtinfo5 libudev1 libusb-0.1-4 libustr-1.0-1 libutempter0 libuuid1 libwrap0 libx11-data libxml2-dev linux-base linux-generic linux-headers-generic lsb-base ltrace lxcfs makedev mime-support mlocate ncurses-base ncurses-term ntfs-3g openjdk-8-jre-headless openssh-sftp-server openssl pastebinit perl perl-base pkg-config policykit-1 popularity-contest powermgmt-base python2 python3.7-distutils python3-apport python3-apt python3-chardet python3-commandnotfound python3-dbus python3-debian python3-distupgrade python3-gdbm python3-gi python3-minimal python3-newt python3-pkg-resources python3-problem-report python3-pycurl python3-requests python3-setuptools python3-six python3-software-properties python3-systemd python3-update-manager python3-urllib3 python-apt-common r-base readline-common rename resolvconf run-one sed sgml-base shared-mime-info snap-confine snapd squashfs-tools tar tasksel tcpd telnet traceroute ubuntu-cloudimage-keyring ubuntu-core-launcher ubuntu-minimal ubuntu-standard ucf uidmap unzip util-linux vim-common vim-runtime vlan xauth xdg-user-dirs xkb-data xml-core zerofree zlib1g zlib1g-devTake a look at the above list and we should decide if we need to update ansible to include some of this setup or not.
I suspect it is the
.is_benchmarkingvariable that installed those packages, and that is has bit rotted. For example,pkg-config(the missing thing in #3657 (comment)) is listedWe should probably add
as it appears needed to build V8.pkg-configtoI'm less familiar with what is needed to run the benchmarks.
I attempted to add the 'is_benchmark' = true to add the benchmark role as I see that pkg-config is installed that way, but some of the other packages in that role have been dropped from ubuntu and that role hasnt been updated for six years, so its probably got some stale packages in there.
Adding pkg-config this way has allowed the V8 CI to build and run tests.
- Node.js
mainbranch failed due to Brokentest/v8-updates/test-linux-perf-loggernode#51308: https://ci.nodejs.org/job/node-test-commit-v8-linux/5878/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/consoleFull - Node.js 18 branch failed due to missing
perf: https://ci.nodejs.org/job/node-test-commit-v8-linux/5879/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/console
Neither encountered the networking issues we had with the Nearform hosted benchmark machines (🎉).
For the missing
perf, maybe we need to add the hetzner machines to:build/ansible/playbooks/jenkins/worker/create.yml
Lines 74 to 77 in 51ad778
- Node.js
For the missing perf, maybe we need to add the Hetzner machines to:
That intel tag was what was being used to target the Nearform intel donated machines. We should definitely change that to target the Hetzner ones now.
Adding pkg-config this way
How shall we approach getting these machines into a stable usable state going forward? I can continue to adjust which packages are installed as part of the ansible setup, but I don't want to inadvertently step on or undo any work that anybody else is doing. (Though I also lack any background in what the jobs do/accomplish)
Ran the benchmark job which fails - https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1498/
Looks like there may be a directory missing. That may have been created manually as these machines were set up a long time ago.
@ryanaslett have you added the linux-perf role? That might be all that is needed to get the v8 jobs running as well as they were before on the machines. @richardlau is that your expectation?
manually creating the directory /w owned by iojs and with group iojs has let the benchmark run get further
@ryanaslett if you are updating the ansible scripts, is there a section which is specific to the benchmark machines that we can add the creation of the /w directory owned by iojs with group iojs?
Job to see if perf job runs ok after adding the /w directory - https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1499/
@ryanaslett if you are updating the ansible scripts, is there a section which is specific to the benchmark machines that we can add the creation of the /w directory owned by iojs with group iojs?
Thats sort of what I was asking in #3657 (comment) - mostly who should be doing this.
28 remaining items
I left a comment in nodejs/performance#157 -- I think the script is currently limiting the benchmarks to a single CPU, which is probably not the intent 🙂. See nodejs/node#52233 (comment). Perhaps the script could parameterize the
--setparameter and then we could set it in the Jenkins job.Done: nodejs/performance#158
@rluvaton btw we can use
nproccommand to max cores instead of deafult0-11in the scriptnprocdoesn't know the difference between performance and efficiency cores.Reacted by Ali Hassan@rluvaton based on #3657 (comment), this does not seem to be working as quite a lot of results are NaN.
This was before my use of 0-11 cpu, should it change anything?
Not sure what's the problem but there was no analysis of the result.
If this is still happening after the change of 0-11 CPU I will revert it...
Currently the CI is locked
as it is still not working, reverted
@thisalihassan can you please take a look why all the NaN?
@rluvaton sure also where can I see the logs?
You have this example: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1514/consoleFull
but you can run locally
FWIW there's a difference in output, e.g.
https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/1510/console21:42:04 "new","test_runner/suite-tests.js","concurrency='no' testType='async' testsPerSuite=1000 numberOfSuites=100",34812.26643390852,2.872550691vs
https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/1513/console13:39:14 test_runner/suite-tests.js concurrency="no" testType="async" testsPerSuite=1000 numberOfSuites=100: 29,112.48474928658i.e. as if the
"new",(or"old",) field and opening"of the second field is missing.@rluvaton nodejs/node#52456 "ipc" I untintentially removed this in the spawn stdio, that's why child.on message wasn't listening events
in Fork IPC is established by default but in spawn it is not established by default hence listener wasn't working
The procurement process has completed, and I have created two EX44's (https://www.hetzner.com/dedicated-rootserver/ex44/) at Hetzner.
If all goes well we should be able to have these online and running benchmark tests.
I believe the next steps are