generate version data file in build phase by techatrix · Pull Request #1409 · zigtools/zls

techatrix · 2023-08-16T14:47:18Z

Instead of having to run zig build gen whenever builtins get changed so that master.zig is updated, I thought that this could be converted into a build step.

I am not entirely satisfied with how the solution ended up in build.zig because caching prevents the version data from being updated whenever the website changes but never caching forces you to always have a internet connection when you build ZLS.

What I did is to invalidate the cache once every day based on the current timestamp which feels more like a hacky trick rather than a proper solution.

zls/build.zig

Lines 147 to 153 in b55525e

    
           const version_data_file_name = blk: { 
        
               // invalidate version data periodically from cache because the website content may change 
        
               // setting `has_side_effects` would also be possible but that would always force a re-run 
        
               const timestamp = @divFloor(std.time.timestamp(), std.time.s_per_day); 
        
               break :blk b.fmt("version_data_{s}_{d}.zig", .{ data_version, timestamp }); 
        
           }; 
        
           const version_data_path = gen_version_data_cmd.addOutputFileArg(version_data_file_name);

Some feedback on this would be appreciated. Or just tell me to scrap this idea which is also fine.

leecannon · 2023-08-16T16:08:12Z

This looks good.

I've also needed to use a few similar hacks to do cache invalidation for steps depending on external stuff.

Would be cool if zig's cache supported time to live for its entries.

A commit is still needed.

BratishkaErik · 2023-08-18T12:23:47Z

Hello everyone, sorry for being late, I just did not noticed this update when building ZLS (I thought it was just bad internet). IIUC building now requires access to the Internet during compilation? Gentoo's network sandbox rightfully blocks this access and compiling fails with this error:

>>> Compiling source in /tmp/portage/dev-zig/zls-9999/work/zls-9999 ...
 * zig-9999 build -Doptimize=ReleaseSafe -Ddata_version=master --verbose
/usr/lib64/zig/9999/bin/zig build-exe /tmp/portage/dev-zig/zls-9999/work/zls-9999/src/config_gen/config_gen.zig --cache-dir /tmp/portage/dev-zig/zls-9999/work/zls-9999/zig-cache --global-cache-dir /tmp/portage/dev-zig/zls-9999/homedir/.cache/zig --name zls_gen --listen=- 
steps [2/6] zig build-exe zls_gen Debug native... LLVM Emit Object... /tmp/portage/dev-zig/zls-9999/work/zls-9999/zig-cache/o/b2db62685cbf5fd7e07b8c05d638942b/zls_gen --generate-version-data master --generate-version-data-path /tmp/portage/dev-zig/zls-9999/work/zls-9999/zig-cache/o/1352759e7fc620c17346be54b7670f40/version_data_master_19587.zig 
run zls_gen (version_data_master_19587.zig): error: error: failed to download https://raw.githubusercontent.com/ziglang/zig/master/doc/langref.html.in: error.TemporaryNameServerFailure
error: DownloadFailed
/usr/lib64/zig/9999/lib/std/net.zig:45:9: 0x493224 in parseIp (zls_gen)
        return error.InvalidIPAddressFormat;
        ^
/usr/lib64/zig/9999/lib/std/net.zig:75:29: 0x493567 in parseExpectingFamily (zls_gen)
            os.AF.UNSPEC => return parseIp(name, port),
                            ^
/usr/lib64/zig/9999/lib/std/net.zig:1414:48: 0x49cc73 in linuxLookupNameFromDns (zls_gen)
    if (ap[0].len < 4 or (ap[0][3] & 15) == 2) return error.TemporaryNameServerFailure;
                                               ^
/usr/lib64/zig/9999/lib/std/net.zig:1358:5: 0x49e1a2 in linuxLookupNameFromDnsSearch (zls_gen)
    return linuxLookupNameFromDns(addrs, canon, name, family, rc, port);
    ^
/usr/lib64/zig/9999/lib/std/net.zig:996:17: 0x49ef15 in linuxLookupName (zls_gen)
                try linuxLookupNameFromDnsSearch(addrs, canon, name, family, port);
                ^
/usr/lib64/zig/9999/lib/std/net.zig:933:9: 0x4419b1 in getAddressList (zls_gen)
        try linuxLookupName(&lookup_addrs, &canon, name, family, flags, port);
        ^
/usr/lib64/zig/9999/lib/std/net.zig:709:18: 0x360a8f in tcpConnectToHost (zls_gen)
    const list = try getAddressList(allocator, name, port);
                 ^
/usr/lib64/zig/9999/lib/std/http/Client.zig:888:45: 0x32c104 in connectUnproxied (zls_gen)
        error.TemporaryNameServerFailure => return error.TemporaryNameServerFailure,
                                            ^
/usr/lib64/zig/9999/lib/std/http/Client.zig:947:9: 0x2e9d16 in connect (zls_gen)
        return client.connectUnproxied(host, port, protocol);
        ^
/usr/lib64/zig/9999/lib/std/http/Client.zig:1012:44: 0x2cfdba in request (zls_gen)
    const conn = options.connection orelse try client.connect(host, port, protocol);
                                           ^
/tmp/portage/dev-zig/zls-9999/work/zls-9999/src/config_gen/config_gen.zig:949:19: 0x2cf2d5 in httpGET (zls_gen)
    var request = try client.request(.GET, uri, .{ .allocator = allocator }, .{});
                  ^
/tmp/portage/dev-zig/zls-9999/work/zls-9999/src/config_gen/config_gen.zig:832:9: 0x2d4d52 in generateVersionDataFile (zls_gen)
        return error.DownloadFailed;
        ^
/tmp/portage/dev-zig/zls-9999/work/zls-9999/src/config_gen/config_gen.zig:1085:9: 0x2de079 in main (zls_gen)
        try generateVersionDataFile(gpa, data_version, output_path);
        ^

Previously, I used zig build --help in src_unpack phase, so that ZLS downloads all required dependencies before compilation, and compilation itself works fine without Internet. It worked before, but now it doesn't, and I don't see any way to replicate this behaviour again (renaming command above to zig build gen --help doesn't help, it still wants Internet during compilation). Here's relevant code:
https://github.com/gentoo/guru/blob/61fa0e6d172f8855d5a757f82cd0cd8d37db7828/dev-zig/zls/zls-9999.ebuild#L106-L115

Again, sorry for not bringing it earlier, I simply didn't know about this change.

hwchen · 2023-08-18T15:54:36Z

I'm also getting this error using nix flakes.

#1409 was mistake

zigtools#1409 was mistake

generate version data file in build phase

b55525e

leecannon previously approved these changes Aug 16, 2023

View reviewed changes

leecannon approved these changes Aug 16, 2023

View reviewed changes

leecannon merged commit adcc686 into master Aug 16, 2023

techatrix deleted the techatrix/gen-version-data branch August 17, 2023 00:11

techatrix mentioned this pull request Aug 17, 2023

nix: add missing dependency on zig language reference file #1410

Merged

askazakov added a commit to askazakov/zls that referenced this pull request Aug 29, 2023

drop irrelevant toc item after zigtools#1409

ae46527

askazakov mentioned this pull request Aug 29, 2023

README: drop irrelevant toc item #1429

Merged

leecannon pushed a commit that referenced this pull request Aug 29, 2023

drop irrelevant toc item after #1409 (#1429)

719565d

KoltPenny pushed a commit to KoltPenny/zls that referenced this pull request Oct 18, 2023

generate version data file in build phase (zigtools#1409)

3c36cb9

KoltPenny pushed a commit to KoltPenny/zls that referenced this pull request Oct 18, 2023

drop irrelevant toc item after zigtools#1409 (zigtools#1429)

bd7596c

techatrix added a commit that referenced this pull request Aug 16, 2024

commit langref.html.in to source control instead of downloading it

dc6669e

#1409 was mistake

techatrix mentioned this pull request Aug 16, 2024

commit langref.html.in to source control instead of downloading it #2000

Merged

techatrix added a commit that referenced this pull request Aug 16, 2024

commit langref.html.in to source control instead of downloading it

6f75315

#1409 was mistake

techatrix added a commit that referenced this pull request Aug 22, 2024

commit langref.html.in to source control instead of downloading it

414fb35

#1409 was mistake

techatrix added a commit that referenced this pull request Aug 22, 2024

commit langref.html.in to source control instead of downloading it

e917fdd

#1409 was mistake

llogick pushed a commit to llogick/zls that referenced this pull request Aug 28, 2024

commit langref.html.in to source control instead of downloading it

66af995

zigtools#1409 was mistake

Navigation Menu

Repository navigation

generate version data file in build phase - #1409

generate version data file in build phase#1409
leecannon merged 1 commit into
masterfrom
techatrix/gen-version-data

techatrix commented Aug 16, 2023

Uh oh!

leecannon commented Aug 16, 2023

Uh oh!

BratishkaErik commented Aug 18, 2023 •

edited

Loading

Uh oh!

hwchen commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	const version_data_file_name = blk: {
	// invalidate version data periodically from cache because the website content may change
	// setting `has_side_effects` would also be possible but that would always force a re-run
	const timestamp = @divFloor(std.time.timestamp(), std.time.s_per_day);
	break :blk b.fmt("version_data_{s}_{d}.zig", .{ data_version, timestamp });
	};
	const version_data_path = gen_version_data_cmd.addOutputFileArg(version_data_file_name);

Conversation

techatrix commented Aug 16, 2023

Uh oh!

leecannon commented Aug 16, 2023

Uh oh!

BratishkaErik commented Aug 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hwchen commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

BratishkaErik commented Aug 18, 2023 •

edited

Loading