packageTimestamp setting - #6237

Wait what? Does any other build tool or packaging tool nuke timestamps like that?

Since when does does reproducible build mean "same input file contents -> same file timestamp"? Does something break if timestamps are preserved?

Web resource caching is one major reason to preserve timestamps but I'm sure there are others, nuking them seems unwise without a really strong reason, no?

@nafg The point of reproducible builds is to get bit-for-bit equivalent binary artifact regardless of who builds things to make sure the JAR is not tampered by the developer etc. iiuc wiping out the timestamp is fairly common thing to do. See for example:

• https://salsa.debian.org/reproducible-builds/strip-nondeterminism.
• https://zlika.github.io/reproducible-build-maven-plugin/

replaces file timestamps in ZIP entries with a fixed value,

This feature should be opt-in rather than opt-out.

I certainly see the value in reproducible builds, but there are obviously use cases where the developer may be using the timestamps for something meaningful. In those cases their application will break when they upgrade sbt, which is a very surprising thing to happen when upgrading to a new minor version.

The biggest issue to me is that you're setting the timestamp to an intentionally incorrect value by default. It's a reasonable hack to solve the problem of reproducible builds, but it's still a hack.

I agree with @nafg's ideas on the API.

I do think it's great for sbt to provide an easy way to reset timestamps to some fixed value, and I'm sure a lot of people will prefer that behavior. I just think it's not a good default.

I should cc @raboof here.

For incremental compilation, relying only on timestamp is not always reliable due to the time resolution in Docker filesystems etc, so I've switched to using non-cryptographic hash to perform invalidation in sbt 1.4.x.

For the purpose of reproducible builds, the original intent was to wipe out the timestamps of *.class files since they would change depending on when the CI machine or local laptop ran the compile task. Maybe a compromise middle ground may be to somehow distinguish resources vs *.class files?

The biggest issue to me is that you're setting the timestamp to an intentionally incorrect value by default.

What is the correct value? Note that git intentionally does not record modified time.

I'm sorry to see this change has caused some surprising behavior.

Since when does does reproducible build mean "same input file contents -> same file timestamp"? Does something break if timestamps are preserved?

Indeed it doesn't - I agree the best thing to do is to preserve timestamps from the original file modification times. However, since

git intentionally does not record modified time

... in the majority of cases the modified-time of resources is somewhat-arbitrary, and the modified-time of classes is "whenever the compilation was done", which is also somewhat-arbitrary. I don't think we should allow those to end up in the artifact by default.

The biggest issue to me is that you're setting the timestamp to an intentionally incorrect value by default. It's a reasonable hack to solve the problem of reproducible builds, but it's still a hack

I think using 'epoch' to signify "unknown" (in a file format that does not allow leaving out the timestamp) is pretty reasonable. I think setting the timestamp to "unknown" rather than a somewhat-arbitrary value is an improvement, even though I agree it is a bit of a sledgehammer.

For a somewhat less 'sledgehammer' (but still rough) workaround, you could set SOURCE_DATE_EPOCH to the timestamp of the last commit at build time. This will set the timestamp to a stable, somewhat-reasonable value, though it will of course over-estimate the timestamp of files that were not changed on the last commit.

The 'ideal' solution would be to keep the source modification timestamps for those files where we know those timestamps can be assumed to be 'correct'. Unfortunately since in most typical scenario's (git) no timestamps can assumed to be 'correct', I think this should be opt-in.

For the purpose of reproducible builds, the original intent was to wipe out the timestamps of *.class files since they would change depending on when the CI machine or local laptop ran the compile task. Maybe a compromise middle ground may be to somehow distinguish resources vs *.class files?

I think doing it only for class files makes a bit more sense, as it's sbt's job to compile your code to class files, but it still seems risky to do by default. What if users have other tools that rely on those modify times?

What is the correct value? Note that git intentionally does not record modified time.

The "correct" time is the time that the file was last modified according to the filesystem. Of course that value may not be universally useful or reliable, but in some cases users do care what it is.

The 'ideal' solution would be to keep the source modification timestamps for those files where we know those timestamps can be assumed to be 'correct'. Unfortunately since in most typical scenario's (git) no timestamps can assumed to be 'correct', I think this should be opt-in.

sbt doesn't make any particular guarantees about the file timestamps being "correct", does it? That's typically the responsibility of the user or the user's filesystem to determine. Do other build tools reset the timestamps by default?

There are plenty of situations where you'll get predictable, meaningful values for the modified timestamps. If I clone a git repository fresh, I can generally assume the modified times will be close to the time the repository was cloned. It would not surprise me if people are relying on that assumption, even if it might not be a good idea.

The guarantee is that if you put println("hello world") you get a bytecode that does hello world. Everything else is somewhat arbitrary and implementation detail. Some build tools use timestamp as invalidation mechanism to implement incremental compilation, whereas sbt 1.4.x doesn't. As much as I do respect playing nice with existing norms, I think there's a real benefit in being able to git clone a repository in 2020 or 2030 and getting bit-for-bit identical JAR file that can be validated using SHA-256 (or whatever hash algorithm we'd use in 10 years).

Do other build tools reset the timestamps by default?

It looks like Bazel does.

$ unzip -v bazel-bin/greeting/greeting-bin.jar
Archive:  bazel-bin/greeting/greeting-bin.jar
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
       0  Stored        0   0% 01-01-2010 00:00 00000000  META-INF/
      97  Stored       97   0% 01-01-2010 00:00 5a07476d  META-INF/MANIFEST.MF
       0  Stored        0   0% 01-01-2010 00:00 00000000  com/
       0  Stored        0   0% 01-01-2010 00:00 00000000  com/twitter/
       0  Stored        0   0% 01-01-2010 00:00 00000000  com/twitter/greeting/
    2481  Defl:N     1014  59% 01-01-2010 00:00 c6eb159c  com/twitter/greeting/Main$.class
     860  Defl:N      491  43% 01-01-2010 00:00 5429a12f  com/twitter/greeting/Main$delayedInit$body.class
    1021  Defl:N      653  36% 01-01-2010 00:00 55e294cb  com/twitter/greeting/Main.class
--------          -------  ---                            -------
    4459             2255  49%                            8 files

It's not surprising that Bazel sets the timestamps to a fixed value, given that it's very opinionated about reproducible builds.

The main problem is that we're introducing a breaking change, since sbt is choosing to take responsibility for something it previously was not responsible for. If sbt wants to be opinionated about this—which in itself is certainly not a bad thing—then the change should wait for 2.0. Most users expect to be able to upgrade to a new minor version without breaking changes. I don't even see any migration notes for 1.4.x mentioning this.

Like I've said, I have no problem with the motivation behind the change or the value of reproducible builds, but also I believe maintainers have a responsibility to their users to not break things unless they have a really good reason to. sbt is now a very widely used tool, and I don't think being opinionated about reproducibility is a good enough reason to change the default behavior.

The main problem is that we're introducing a breaking change, since sbt is choosing to take responsibility for something it previously was not responsible for. If sbt wants to be opinionated about this—which in itself is certainly not a bad thing—then the change should wait for 2.0. Most users expect to be able to upgrade to a new minor version without breaking changes. I don't even see any migration notes for 1.4.x mentioning this.

Regardless of the outcome of this particular feature, I want to clarify this notion about minor version upgrade. I adopted Semantic Versioning for sbt so we can introduce new features and sometimes breaking changes without burning the forest down with binary incompatibility. Since sbt 1.0 in 2017, we shipped feature releases around year-apart with fairly aggressive changes like introducing slash syntax, switching to Coursier, and implementing remote caching. Following the logic that any breaking change would require major version bump, we'd need to do that annually, requiring unnecessary republishing of plugins.

FWIW sbt 1.4.0 release notes do mention this as breaking change.

It's of course up to sbt's maintainers how the versioning scheme should work, and I see the benefit of introducing changes without having to re-publish plugins. But even from a loose semver perspective the new behavior is very surprising, especially because people are much more likely to notice the issue once the application is deployed, rather than at build time.

FWIW sbt 1.4.0 release notes do mention this as breaking change.

You're right, it does mention this:

Makes JAR file creation repeatable by sorting entry by name and dropping timestamps

But there are no migration notes about that at all, I guess because it wasn't obvious what the impact would be.

For a somewhat less 'sledgehammer' (but still rough) workaround, you could set SOURCE_DATE_EPOCH to the timestamp of the last commit at build time. This will set the timestamp to a stable, somewhat-reasonable value, though it will of course over-estimate the timestamp of files that were not changed on the last commit.

I find this to be a very elegant solution - the last commit is always known, and changing the commit necessarily voids the expectation of reproducibility.

Using the timestamp of the last commit would be a more reasonable default. That should give you reproducibility and should also be suitable for the web resource caching situations. Then the timestamp is actually somewhat meaningful.

I also like the idea of Option[Instant] setting like @nafg suggested that lets you override with a specific timestamp if you prefer.

+1 on timestamp of the last commit as the default.
If we wanted, the timestamps could be more granual -- theoretically, a timestamp could be determined for each individual file based on the commit that modified the file last. But I don't know if it's worth it.

Anyway, reproducibility is a hugely important thing, so thanks for that.

How can it be the default, if there's no guarantee SBT is running in a git repo?

It's nothing complicated: you can fallback to 0 (or the actual timestamp from the file system) if there is no git repository to be found (or any other VCS sbt will know to work with).

The fallback I was thinking of would be to not set a timestamp at all. Maybe we could also display a warning to let the user know how to manually define the timestamp strategy. We could attempt to check for other version control systems, though I think git is pervasive enough that it would be fine to only support git by default.

Actually, a more user-friendly way to be opinionated here would be to display a warning when packaging a project unless you explicitly set something.

For example, when packaging the project, you could display something like:

[warn] This build is not reproducible! Timestamps may differ between builds.
[warn]
[warn] To remove this warning, choose a timestamp strategy. For example:
[warn]
[warn]   Compile / packageBin / timestampStrategy := Package.gitCommitTimestamp
[warn] 
[warn] For more details see: <link>

Other options could be keepTimestamp, fixedTimestamp(instant), etc. Obviously the exact naming, etc. could change, but the idea is you force the user to be aware of their decision.

Displaying warning by default is a tactic taken sometimes by different tooling or language, but I don't think it's usually a great option. Either the user doesn't have enough context to make the right choice, or it's not relevant to the problem they want to solve. A recent example that comes to mind is where we deprecated floating-point sorting by displaying a deprecation message.

I could create a global setting with type Option[Long] and default it to 2010-01-01, and maybe provide a function that calls out to git if people want to rewire the setting.

2010 feels way too arbitrary. 1970 at least communicates to the user that it's the zero-timestamp.

Time of the git commit > 0 > 2010-01-01, IMHO.

Displaying warning by default is a tactic taken sometimes by different tooling or language, but I don't think it's usually a great option. Either the user doesn't have enough context to make the right choice, or it's not relevant to the problem they want to solve.

IMO a warning would be preferable to a breaking change. If the user doesn't have enough context and makes the wrong choice, then the fact that they made an explicit change will make it easier to debug any issue that arises. If you make a suggestion that works for most people—like the git timestamp—then most users will be fine.

A recent example that comes to mind is where we deprecated floating-point sorting by displaying a deprecation message.

I actually thought that decision made sense. There are tradeoffs between the IEEE 754 ordering and the total ordering, and a warning is a good way to make sure the user has the opportunity to make an informed choice.

I could create a global setting with type Option[Long] and default it to 2010-01-01, and maybe provide a function that calls out to git if people want to rewire the setting.

I think a setting like that would work. We just don't have consensus on what the default should be.

@sideeffffect 1970-01-01 doesn't work unfortunately because the WinZip format apparently does not support dates earlier than 1980-01-01, and since it doesn't carry timezone information 1980-01-01 itself probably should be avoided. I'm not sure why Bazel chose 2010-01-01, but bazelbuild/bazel@475f40a shows that it used to be 1980, but it was changed to 2010 in 2018.

In terms of the scope of reproducibility, I don't think it matters whether something is a library or open sourced. In fact, the attackers might avoid public JARs if they wanted to stay stealth. Anyone who deploys anything with sensitive data probably should care about it.

A default of not preserving the actual modification time when archiving makes no sense to me

The modification time is an arbitrary number of when someone git cloned, or ran the build. If someone uses remote caching feature, the binary timestamp could even be earlier than the source timestamp on a local machine. It would be an irrational thing to do if I started doing this on my own, but as previously mentioned this is a common technique, and I think we're in a good company to pursue reproducibility - https://reproducible-builds.org/projects/ lists Google, Arch Linux, Debian, and Tor among others.

The HTTP Last-Modified is an interesting use case of JAR timestamps (it's an interesting use case because getting a ballpark number is should provide useful caching), but I feel like it's more of an exception not the rule.

We shouldn't even need to depend on a system installed git binary, since there's JGit, no?

JGit doesn't support git worktrees (https://bugs.eclipse.org/bugs/show_bug.cgi?id=477475) so I default to system git in sbt-dynver (and then I never needed to rebuild the indirection that sbt-git had over system git/JGit.)

I think we can assume most developers have git on their system, as long as we fail in an obvious way to give them an opportunity to fix the issue or otherwise override the setting.

I guess one aspect worth considering is that the cloned git repository could be copied and built on some environment that didn't have git, in which case the build wouldn't reproduce.

I've added packageTimestamp setting that can be used to opt out of the current behavior. Not sure if we can all come to a full consensus on #6235, but I think having this setting is strictly better than the status quo, so I'm merging this.