Flaky test fails intermittently in GitHub Action

Test case passes on some runs and fails on others: org.json.junit.JSONObjectTest.issue654StackOverflowInputWellFormed

Test Run: https://github.com/stleary/JSON-java/actions/runs/6644866305/job/18054932906#step:5:79

Code Link:

JSON-java/src/test/java/org/json/junit/JSONObjectTest.java

Lines 3668 to 3676 in b5b9f63

    
           @Test(expected = JSONException.class) 
        
           public void issue654StackOverflowInputWellFormed() { 
        
               //String input = new String(java.util.Base64.getDecoder().decode(base64Bytes)); 
        
               final InputStream resourceAsStream = JSONObjectTest.class.getClassLoader().getResourceAsStream("Issue654WellFormedObject.json"); 
        
               JSONTokener tokener = new JSONTokener(resourceAsStream); 
        
               JSONObject json_input = new JSONObject(tokener); 
        
               assertNotNull(json_input); 
        
               fail("Excepected Exception."); 
        
           }

Appears to be due to differences in resources available on each run. The exception expected is a "JSONException", but that's because it is wrapping a "StackOverflowError". StackOverflowError is going to be dependent on the environment and may not hit consistently.

I think this test case should be removed, as it never guarantees the behavior. To identify the max depth supported, we can test through configuration. And environment resource related runtime error expectation need not be tested.

Might be better to see if the test can be fixed, although if you see a chance to add a unit test, that is fine too.

@stleary

@stleary , I'm leaning towards just marking the test as a "skip/ignore" or removing it altogether. It seems like this test would need regular maintenance to ensure that it was consistently hitting stack overflows.

@Ignore("This test relies on system constraints and may not always pass.")
@Test(expected = JSONException.class) 
 public void issue654StackOverflowInputWellFormed() { 
     //String input = new String(java.util.Base64.getDecoder().decode(base64Bytes)); 
     final InputStream resourceAsStream = JSONObjectTest.class.getClassLoader().getResourceAsStream("Issue654WellFormedObject.json"); 
     JSONTokener tokener = new JSONTokener(resourceAsStream); 
     JSONObject json_input = new JSONObject(tokener); 
     assertNotNull(json_input); 
     fail("Excepected Exception."); 
 }

@johnjaylward

@johnjaylward Fair enough, better to silence the test, than have it fail intermittently.

@johnjaylward

@johnjaylward - are you annotating the test with Ignore?

Fixes stleary#821

Navigation Menu

Repository navigation

Flaky test fails intermittently in GitHub Action #821

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	@Test(expected = JSONException.class)
	public void issue654StackOverflowInputWellFormed() {
	//String input = new String(java.util.Base64.getDecoder().decode(base64Bytes));
	final InputStream resourceAsStream = JSONObjectTest.class.getClassLoader().getResourceAsStream("Issue654WellFormedObject.json");
	JSONTokener tokener = new JSONTokener(resourceAsStream);
	JSONObject json_input = new JSONObject(tokener);
	assertNotNull(json_input);
	fail("Excepected Exception.");
	}

Flaky test fails intermittently in GitHub Action #821

Description

Activity

rudrajyotib commented on Oct 28, 2023

stleary commented on Oct 29, 2023

johnjaylward commented on Oct 30, 2023

stleary commented on Oct 31, 2023

rudrajyotib commented on Nov 4, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions