ERROR: cannot find the current segment/subsegment when segment is open and uploading file to s3.  #377

Activity

1. carolabadeer commented on Jan 30, 2023

   

   carolabadeer

   on Jan 30, 2023

   Contributor

   More actions

   Hello, I have a few questions that will hopefully help clarify where this issue is coming from:

   1. Is there any currently active context that should be passed in when configuring the xray_recorder? Are you running in Lambda?
   2. If you add another xray_recorder.current_segment().trace_id before the first call to upload_file, does that trace ID match that of "test segment"? Can you also try xray_recorder.get_trace_entity() to see what the currently active entity (segment or subsegment) is?
   3. Is this code snippet all in one place/function? Is there any additional code that is being executed between the block that throws an error and the one that doesn't?
2. RandyVuu commented on Jan 31, 2023

   

   RandyVuu

   on Jan 31, 2023

   Last edited by RandyVuu

   Author

   More actions

   Hello, to answer your questions

   1. There is no active context to pass to the xray_recorder. I discovered this issue when using xray sdk on glue pyshell and then recreated it locally.
   2. There is no additional code inbetween the lines. The code ran exactly as is.

   Heres a snippet including more trace_id logs and the corresponding output

   if __name__ == "__main__":
       patch_all()
       xray_recorder.configure(context_missing='LOG_ERROR', daemon_address='localhost:2000')
       segment = xray_recorder.begin_segment('test segment')
       logger.info(f'created segment trace_id:{segment.trace_id} ')
       s3_resource = boto3.resource("s3")
       logger.info(f'current trace entity trace_id: {xray_recorder.get_trace_entity().trace_id}')
       # this give the error
       logger.info(f'trace_id before error: {xray_recorder.current_segment().trace_id}')
       s3_resource.Bucket('resultfiles-test' ).upload_file('hello_world.txt', 'hello_world.txt')
       logger.info(f'trace_id after error: {xray_recorder.current_segment().trace_id}')
       # this works
       logger.info(f'trace_id before working: {xray_recorder.current_segment().trace_id}')
       s3_resource.Bucket('resultfiles-test' ).put_object(Key='hello_world.txt', Body=b'hello world')
       logger.info(f'trace_id after working: {xray_recorder.current_segment().trace_id}')
       xray_recorder.end_segment()

   output

   INFO:aws_xray_sdk.core.patcher:successfully patched module botocore
   INFO:aws_xray_sdk.core.patcher:successfully patched module sqlite3
   INFO:aws_xray_sdk.core.sampling.sampler:No effective centralized sampling rule match. Fallback to local rules.
   INFO:root:created segment trace_id:1-63d85b07-0a470e1a68d647efad69bce2 
   INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
   INFO:root:current trace entity trace_id: 1-63d85b07-0a470e1a68d647efad69bce2
   INFO:root:trace_id before error: 1-63d85b07-0a470e1a68d647efad69bce2
   ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment, please make sure you have a segment open
   WARNING:aws_xray_sdk.core.recorder:No segment found, cannot begin subsegment s3.
   ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment, please make sure you have a segment open
   INFO:root:trace_id after error: 1-63d85b07-0a470e1a68d647efad69bce2
   INFO:root:trace_id before working: 1-63d85b07-0a470e1a68d647efad69bce2
   INFO:root:trace_id after working: 1-63d85b07-0a470e1a68d647efad69bce2
   

   When I set the logging level to debug to see the segment sent, it includes the second working call to s3.

   {
     "id": "a69a1a6eb4e12c86",
     "name": "test segment",
     "start_time": 1675123698.393169,
     "in_progress": false,
     "aws": {
       "xray": {
         "sdk": "X-Ray for Python",
         "sdk_version": "2.11.0"
       }
     },
     "subsegments": [
       {
         "id": "1130954ac4c9383e",
         "name": "s3",
         "start_time": 1675123699.1555898,
         "parent_id": "a69a1a6eb4e12c86",
         "in_progress": false,
         "http": {
           "response": {
             "status": 200
           }
         },
         "aws": {
           "operation": "PutObject",
           "region": "us-east-1",
           "request_id": "CTQFC52JXK7BED7F",
           "id_2": "4lazYkItN+LCNrSfBTnved/tEByY50Od+KArL3cX5uWrx8BPHDXNMCnxOi/hhT4Tv9wd+aGbGdw=",
           "key": "hello_world.txt",
           "bucket_name": "resultfiles-test"
         },
         "trace_id": "1-63d85bf2-67e791f683029bb451728c4a",
         "type": "subsegment",
         "namespace": "aws",
         "end_time": 1675123699.3337479
       }
     ],
     "trace_id": "1-63d85bf2-67e791f683029bb451728c4a",
     "service": {
       "runtime": "CPython",
       "runtime_version": "3.9.8"
     },
     "end_time": 1675123699.335922
   }
   
3. RandyVuu commented on Mar 2, 2023

   

   RandyVuu

   on Mar 2, 2023

   Author

   More actions

   I discovered the issue can be resolved if I disable multi-threading on upload_file.