Repository navigation
go test fails on offline system #8928
Description
Activity
@pdostal thanks for the detailed report. This is definitely not a desirable state. Aside from what you've mentioned it's also introducing a source of flakiness.
@malancas can you take a look at this please?
What is the testing seam here? Is it acceptable to stub the
SigStoreVerifieraltogether and assume it is tested correctly elsewhere? Should we be stubbing the http responses? Let me know your thoughts.@williammartin Taking a look now. I'm going to read through the code and figure out the best place to either stub the code out.
@williammartin I think stubbing out the
SigstoreVerifierin theverifypackage unit tests makes the most sense here. I'll work on a PR for that.All looks fine now. Thank you!
Great! Sorry for the friction and thank you for your help. Please let us know if we mess this up again in the future.
Reacted by Pavel Dostál@steiza @malancas I'd like us to get a handle on tests that really have to reach out to external resources or not. Can you have a think about how the attestation commands tests could be restructured, lints we could write, or any other approach to prevent this happening. It seems like a very easy thing to do, without much visibility with the nature of the tuf client.
Happy to get together and pair on this with you.
Ah, sorry, I didn't realize that the tests were intended to work offline!
#9340 should be a quick fix. If we wanted to re-enable happy-path testing we could mock out a client, but I think we'd need some small support changes in sigstore-go.
In terms of how to ensure tests work offline in the future, ideally CI would run tests in a restricted network, maybe by using Azure private networking? Now that I know we want tests to run offline, I can be aware of that in future changes.
A linter detection for this specific issue would be pretty difficult.
tuf.Newdoes use the network in the happy path, but it doesn't in thefails because the root cannot be foundtest case that remains. And of course, this wouldn't detect other functions potentially included in future tests that would make network calls.Hello,
I package gh for openSUSE, where the entire pipeline operates completely offline. This design choice ensures build reproducibility and guarantees that the package is not externally influenced.
When a test fails due to the absence of a network connection, I can disable it, but I appreciate having the package thoroughly tested.
While you can flag each test that requires a network connection, I believe it's beneficial to maintain the test suite as self-sufficient as possible.
Thank you very much for your work and support for the offline scenario.
Describe the bug
when running the tests offline there are errors recently introduced by #8698.
Steps to reproduce the behavior
hgfrom this repositorymake testExpected vs actual behavior
All the other tests are successfully passing; however, there are a few specific tests that are failing. A potential workaround involves using the command
-skip="TestRunInspect|TestJSONOutput|TestNewSigstoreVerifier|TestRunVerify|TestTUFRootVerify", though this approach lacks elegance.To address this issue more gracefully, one of two strategies could be employed: either emulate the remote call with a local one or categorize these tests as online tests so they can be selectively skipped. This could streamline the testing process while maintaining the integrity of the test suite.