[Python] SerializationException for XRay exporter

I have a python 3.8 lambda instrumented with the aws layer, after following the documentation here: https://aws-otel.github.io/docs/getting-started/lambda/lambda-python

Layer version: aws-otel-python38-ver-1-3-0:1 (contains OpenTelemetry for Python v1.3.0 with Contrib v0.22b0 and ADOT Collector for Lambda v0.11.0)
The lambda is a scheduled lambda (using AWS eventbridge)
I have not made any changes to the default collector config provided by the layer:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
  awsxray:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      exporters: [logging]

After running the lambda, the layer initialises successfully, but there is an error with the xray exporter after the function execution completes:
2021-08-02T11:12:09.614Z	error	exporterhelper/queued_retry.go:245	Exporting failed. Try enabling retry_on_failure config option.	{
    "kind": "exporter",
    "name": "awsxray",
    "error": "Permanent error: SerializationException: \n\tstatus code: 400
[...]

2021-08-03T10:53:10.222Z	error	exporterhelper/queued_retry.go:175	Exporting failed. Dropping data. Try enabling sending_queue to survive temporary failures.
{
    "kind": "exporter",
    "name": "awsxray",
    "dropped_items": 2
}

I have not tried the suggestions to survive temporary failures because this error happens consistently.
Is there something missing in my setup or is it a bug somewhere?

Hey @bellmatt !
Thanks for filing this issue and describing your setup. Just to confirm, when you say you didn't modify the collector-config you mean you expect the config is the same config we have on this repo correct?
I just tried using ARN arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python38-ver-1-3-0:1, followed the steps in the docs you linked, did not modify the collector config either, and followed this quick tutorial on how to trigger a Lambda function with AWS Eventbridge. Doing all that my Lambda runs successfully and exports to X-Ray where I can see the trace. When I check the CloudWatch logs for this Lambda function I don't see the error you mentioned.
Is it possible for you to try manually triggering your Lamba and seeing if traces appear in X-Ray that way? If that's not possible, would you be able to describe your Lambda function in more detail? I'm not as suspicious of EventBridge since it only calls the Lambda function without doing much more to modify any OTel setup.
Something else that could be helpful is to post more CloudWatch logs to see if they tell us anything else that is potentially weird in the OTel setup on Lambda?
Thanks!

Hi @NathanielRN! sorry for the very late reply and thanks for taking a look and trying to reproduce!
I've had some more time to retry this after updating to the latest layer (python38-1-5-0), but I'm still seeing the error.
Hopefully these answer your questions:

Triggering the lambda manually results in the same error
I do see a trace in Xray, but it's the same one I would get just by enabling active tracing on the lambda
It's a fairly small python 3.8 lambda that makes several external HTTP calls and puts a file in S3, otherwise very little is different from the sample app in this repo
I am effectively using the default config from this repo yes, I can verify this in the logs too

And here are all the otel related logs I get in cloudwatch
|   timestamp   |                                                                                message
|<PHONE_NUMBER>822 | 2021/11/05 11:04:27 [collector] Launching OpenTelemetry Lambda extension, version:  v0.1.0                                                                                |
|<PHONE_NUMBER>822 | 2021-11-05T11:04:27.822Z info service/collector.go:303 Starting otelcol... {"Version": "v0.1.0", "NumCPU": 2}                                                                                |
|<PHONE_NUMBER>822 | 2021-11-05T11:04:27.822Z info service/collector.go:242 Loading configuration...                                                                                |
|<PHONE_NUMBER>825 | 2021-11-05T11:04:27.825Z info service/collector.go:258 Applying configuration...                                                                                |
|<PHONE_NUMBER>829 | 2021-11-05T11:04:27.829Z info builder/exporters_builder.go:264 Exporter was built. {"kind": "exporter", "name": "logging"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/exporters_builder.go:264 Exporter was built. {"kind": "exporter", "name": "awsxray"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/pipelines_builder.go:214 Pipeline was built. {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/pipelines_builder.go:214 Pipeline was built. {"pipeline_name": "traces", "pipeline_datatype": "traces"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/receivers_builder.go:227 Receiver was built. {"kind": "receiver", "name": "otlp", "datatype": "traces"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/receivers_builder.go:227 Receiver was built. {"kind": "receiver", "name": "otlp", "datatype": "metrics"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info service/service.go:143 Starting extensions...                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info service/service.go:188 Starting exporters...                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/exporters_builder.go:93 Exporter is starting... {"kind": "exporter", "name": "awsxray"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/exporters_builder.go:98 Exporter started. {"kind": "exporter", "name": "awsxray"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/exporters_builder.go:93 Exporter is starting... {"kind": "exporter", "name": "logging"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/exporters_builder.go:98 Exporter started. {"kind": "exporter", "name": "logging"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info service/service.go:193 Starting processors...                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/pipelines_builder.go:52 Pipeline is starting... {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/pipelines_builder.go:63 Pipeline is started. {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/pipelines_builder.go:52 Pipeline is starting... {"pipeline_name": "traces", "pipeline_datatype": "traces"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/pipelines_builder.go:63 Pipeline is started. {"pipeline_name": "traces", "pipeline_datatype": "traces"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info service/service.go:198 Starting receivers...                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info builder/receivers_builder.go:71 Receiver is starting... {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.831Z info otlpreceiver/otlp.go:75 Starting GRPC server on endpoint <IP_ADDRESS>:4317 {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info otlpreceiver/otlp.go:137 Setting up a second GRPC listener on legacy endpoint <IP_ADDRESS>:55680 {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info otlpreceiver/otlp.go:75 Starting GRPC server on endpoint <IP_ADDRESS>:55680 {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info otlpreceiver/otlp.go:93 Starting HTTP server on endpoint <IP_ADDRESS>:4318 {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info otlpreceiver/otlp.go:159 Setting up a second HTTP listener on legacy endpoint <IP_ADDRESS>:55681 {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info otlpreceiver/otlp.go:93 Starting HTTP server on endpoint <IP_ADDRESS>:55681 {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info builder/receivers_builder.go:76 Receiver started. {"kind": "receiver", "name": "otlp"}                                                                                |
|<PHONE_NUMBER>833 | 2021-11-05T11:04:27.832Z info service/collector.go:206 Setting up own telemetry...                                                                                |
|<PHONE_NUMBER>835 | 2021-11-05T11:04:27.835Z info service/telemetry.go:99 Serving Prometheus metrics {"address": ":8888", "level": 0, "service.instance.id": "7cbb0866-e130-4cc3-a119-7370cdfe3456"}                                                                                |
|<PHONE_NUMBER>835 | 2021-11-05T11:04:27.835Z info service/collector.go:218 Everything is ready. Begin running and processing data.                                                                                |
|<PHONE_NUMBER>839 | 2021/11/05 11:04:27 Registered extension ID: "5d0c57a4-c58d-4a3f-9d84-f9c0a1d74ee6"                                                                                |
|<PHONE_NUMBER>839 | 2021/11/05 11:04:27 [collector] Register response: {                                                                                |
|<PHONE_NUMBER>839 |  "functionName": "fcn-xxx”,                                                                                |
|<PHONE_NUMBER>839 |  "functionVersion": "$LATEST",                                                                                |
|<PHONE_NUMBER>839 |  "handler": "app.lambda_handler"                                                                                |
|<PHONE_NUMBER>839 | }                                                                                |
|<PHONE_NUMBER>839 | 2021/11/05 11:04:27 [collector] Waiting for event...                                                                                |
|<PHONE_NUMBER>201 | START RequestId: 9bedd347-e140-4730-96f6-3e5a9a342cde Version: $LATEST                                                                                |
|<PHONE_NUMBER>201 | EXTENSION Name: collector State: Ready Events: [INVOKE,SHUTDOWN]                                                                                |
|<PHONE_NUMBER>203 | 2021/11/05 11:04:29 [collector] Received event: {                                                                                |
|<PHONE_NUMBER>203 |  "eventType": "INVOKE",                                                                                |
|<PHONE_NUMBER>203 |  "deadlineMs":<PHONE_NUMBER>202,                                                                                |
|<PHONE_NUMBER>203 |  "requestId": "9bedd347-e140-4730-96f6-3e5a9a342cde",                                                                                |
|<PHONE_NUMBER>203 |  "invokedFunctionArn": "arn:aws:lambda:eu-west-1:584614786727:function:fcn-sprouts-function-pagerdutydailys3backup",                                                                                |
|<PHONE_NUMBER>203 |  "tracing": {                                                                                |
|<PHONE_NUMBER>203 |   "type": "X-Amzn-Trace-Id",                                                                                |
|<PHONE_NUMBER>203 |   "value": "Root=1-61850fbb-5a15b15d1a7576de62df2228;Parent=11818bad54f5b1cb;Sampled=1"                                                                                |
|<PHONE_NUMBER>203 |  }                                                                                |
|<PHONE_NUMBER>203 | }                                                                                |
|<PHONE_NUMBER>203 | 2021/11/05 11:04:29 [collector] Waiting for event...                                                                                |

[ edit: I have removed the function logging ]

|<PHONE_NUMBER>823 | 2021-11-05T11:04:43.822Z error exporterhelper/queued_retry.go:254 Exporting failed. Try enabling retry_on_failure config option. {"kind": "exporter", "name": "awsxray", "error": "Permanent error: SerializationException: \n\tstatus code: 400, request id: ac117941-92be-4543-840d-3f06e6b12211"}                                                                        |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/exporter/exporterhelper/queued_retry.go:254                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/exporter/exporterhelper/traces.go:116                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/exporter/exporterhelper/queued_retry.go:174                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/exporter/exporterhelper/traces.go:95                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/consumer/consumerhelper.ConsumeTracesFunc.ConsumeTraces                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/consumer/consumerhelper/traces.go:29                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/receiver/otlpreceiver/internal/trace/otlp.go:60                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector/model@v0.33.0/otlpgrpc/traces.go:84                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector/model@v0.33.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.22.0/interceptor.go:327                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector/model@v0.33.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219                                                                                |
|<PHONE_NUMBER>823 | google.golang.org/grpc.(*Server).processUnaryRPC                                                                                |
|<PHONE_NUMBER>823 |  google.golang.org/grpc@v1.40.0/server.go:1297                                                                                |
|<PHONE_NUMBER>823 | google.golang.org/grpc.(*Server).handleStream                                                                                |
|<PHONE_NUMBER>823 |  google.golang.org/grpc@v1.40.0/server.go:1626                                                                                |
|<PHONE_NUMBER>823 | google.golang.org/grpc.(*Server).serveStreams.func1.2                                                                                |
|<PHONE_NUMBER>823 |  google.golang.org/grpc@v1.40.0/server.go:941                                                                                |
|<PHONE_NUMBER>823 | 2021-11-05T11:04:43.823Z error exporterhelper/queued_retry.go:176 Exporting failed. Dropping data. Try enabling sending_queue to survive temporary failures. {"kind": "exporter", "name": "awsxray", "dropped_items": 2}                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/exporter/exporterhelper/queued_retry.go:176                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/exporter/exporterhelper/traces.go:95                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/consumer/consumerhelper.ConsumeTracesFunc.ConsumeTraces                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/consumer/consumerhelper/traces.go:29                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector@v0.33.0/receiver/otlpreceiver/internal/trace/otlp.go:60                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector/model@v0.33.0/otlpgrpc/traces.go:84                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector/model@v0.33.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.22.0/interceptor.go:327                                                                                |
|<PHONE_NUMBER>823 | go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler                                                                                |
|<PHONE_NUMBER>823 |  go.opentelemetry.io/collector/model@v0.33.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219                                                                                |
|<PHONE_NUMBER>823 | google.golang.org/grpc.(*Server).processUnaryRPC                                                                                |
|<PHONE_NUMBER>823 |  google.golang.org/grpc@v1.40.0/server.go:1297                                                                                |
|<PHONE_NUMBER>823 | google.golang.org/grpc.(*Server).handleStream                                                                                |
|<PHONE_NUMBER>823 |  google.golang.org/grpc@v1.40.0/server.go:1626                                                                                |
|<PHONE_NUMBER>823 | google.golang.org/grpc.(*Server).serveStreams.func1.2                                                                                |
|<PHONE_NUMBER>823 |  google.golang.org/grpc@v1.40.0/server.go:941                                                                                |
|<PHONE_NUMBER>845 | END RequestId: 9bedd347-e140-4730-96f6-3e5a9a342cde                                                                                |
|<PHONE_NUMBER>845 | REPORT RequestId: 9bedd347-e140-4730-96f6-3e5a9a342cde Duration: 14638.64 ms Billed Duration: 14639 ms Memory Size: 256 MB Max Memory Used: 255 MB Init Duration: 1599.57 ms  XRAY TraceId: 1-61850fbb-5a15b15d1a7576de62df2228 SegmentId: 11818bad54f5b1cb Sampled: true                                                                                |

Is there a way to enable debug logging for aws-otel-lambda that may add more context for the SerializationException?

Thanks for the added context @bellmatt ! I especially appreciate you manually triggering the Lambda so we can eliminate Eventbridge and the scheduling from the equation for now.
It makes sense that you see the traces for just Lambda in X-Ray, as those are sent by the Lambda service independent of OpenTelemetry and so regardless of OTel's success/failure those traces should make it to X-Ray every time.
The logs are also helpful at least to confirm that the problem occurs at the time of the ADOT collector (packaged with the python38-ver-1-5-0:3 layer) exporting the traces, and not with the OTel Python SDK itself.
However, I have never seen this Serialization error from the collector... You say that:

It's a fairly small python 3.8 lambda that makes several external HTTP calls and puts a file in S3

I'm wondering if somehow the upstream OTel Python instrumentation (bundled in the layer) of your HTTP calls is trying to record a response/request parameter that would throw an exception at the point of being exported by the awsxray: exporter in the ADOT collector? I'm less suspicious of S3 because you're just "put"-ing and not "get"-ing so less chance for wonky data in the serialization step I would think...
Unfortunately it's not easy to add logging to the Lambda Layers because they're already published with their set configurations... But it would be helpful to see what kind of traces OTel Python is sending to the collector and seeing if anything is suspiciously un-serializable. Since you say you app is not too different from the upstream sample app for OTel Python Lambda Function maybe you can comment out your code and try that sample app code first to confirm the difference is the Python function you are using? If it works with the Sample App you may have discovered a bug in OTel Python Instrumentation or the collector itself that we should fix up there where a user's code cannot be instrumented correctly. At that point we can use your code with a modified layer that has debug logging enable. Otherwise, If you still have the same error then we can double check exactly what we package in the python layer since it wouldn't depend on user code at that point.
If you're able to share what libraries your Lambda function is using and how it is using them that would be helpful! I understand if you can't, and we'll try to work around it in that case. Either way taking the steps above first is worthwhile to determine if we are going down the right path 🙂
I'll ping @jefchien and @sethAmazon in here who work on the https://github.com/aws-observability/aws-otel-collector just in case they've seen this error on the collector before.

 Message content: {'kind': 'custom', 'myfunction': 'myparams'}

[SNIP]

2021-11-17T06:53:59.550Z	error	exporterhelper/queued_retry.go:165	Exporting failed. Try enabling retry_on_failure config option.	{     "kind": "exporter",     "name": "awsxray",     "error": "Permanent error: SerializationException: \n\tstatus code: 400, request id: d0d75484-69c0-4aca-8ed3-16ccae02c729" }
--
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/queued_retry.go:165
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/traces.go:134
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/queued_retry.go:99
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/traces.go:113
go.opentelemetry.io/collector/consumer/consumerhelper.ConsumeTracesFunc.ConsumeTraces
go.opentelemetry.io/collector@v0.36.0/consumer/consumerhelper/traces.go:29
go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export
go.opentelemetry.io/collector@v0.36.0/receiver/otlpreceiver/internal/trace/otlp.go:60
go.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export
go.opentelemetry.io/collector/model@v0.36.0/otlpgrpc/traces.go:102
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1
go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.24.0/interceptor.go:325
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler
go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219
google.golang.org/grpc.(*Server).processUnaryRPC
google.golang.org/grpc@v1.40.0/server.go:1297
google.golang.org/grpc.(*Server).handleStream
google.golang.org/grpc@v1.40.0/server.go:1626
google.golang.org/grpc.(*Server).serveStreams.func1.2
google.golang.org/grpc@v1.40.0/server.go:941
2021-11-17T06:53:59.551Z	error	exporterhelper/queued_retry.go:101	Exporting failed. Dropping data. Try enabling sending_queue to survive temporary failures.	{     "kind": "exporter",     "name": "awsxray",     "dropped_items": 7 }
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/queued_retry.go:101
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/traces.go:113
go.opentelemetry.io/collector/consumer/consumerhelper.ConsumeTracesFunc.ConsumeTraces
go.opentelemetry.io/collector@v0.36.0/consumer/consumerhelper/traces.go:29
go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export
go.opentelemetry.io/collector@v0.36.0/receiver/otlpreceiver/internal/trace/otlp.go:60
go.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export
go.opentelemetry.io/collector/model@v0.36.0/otlpgrpc/traces.go:102
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1
go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.24.0/interceptor.go:325
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler
go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219
google.golang.org/grpc.(*Server).processUnaryRPC
google.golang.org/grpc@v1.40.0/server.go:1297
google.golang.org/grpc.(*Server).handleStream
google.golang.org/grpc@v1.40.0/server.go:1626
google.golang.org/grpc.(*Server).serveStreams.func1.2
google.golang.org/grpc@v1.40.0/server.go:941
END RequestId: 42505b52-6e26-5de6-8ca5-015cc71472f4
REPORT RequestId: 42505b52-6e26-5de6-8ca5-015cc71472f4	Duration: 2441.25 ms	Billed Duration: 2442 ms	Memory Size: 256 MB	Max Memory Used: 159 MB	Init Duration: 2773.14 ms	XRAY TraceId: 1-6194a702-5d2440a8f3a49b97335cd62c	SegmentId: 3bb118a61ed3cef6	Sampled: true

However, API Gateway and manually triggered Lambda do not have the issue.

Hey @pdecat thanks for providing your logs 🙂 Unfortunately it looks like the logs only include grpc code logs and don't have logs from the awsxray exporter itself which would have the logs we need to see for the Serialization error 😕 Especially hoping that the logs reveal what the collector is trying to serialize.
As I mentioned in the comment above, I think this is more likely some miscommunication between the OTel Python SDK & the ADOT Collector which uses the OTel Collector Contrib - AWS X-Ray Exporter. Either the SDK is not serializing correctly or the Collector is not. The fact that it is SQS where users can send whatever they want makes it easier for there to be Serialization error so that the SDK or Collector are provided with values they did not account for.
It also makes sense that you see the error in SQS but not Eventbridge because the JSON in the Lambda Event is very different depending on if the Lambda was triggered by SQS or if the Lambda was triggered by EventBridge. If we knew the Lambda event that causes the Serialization error with this layer that would be a great help to get us to reproduce the error.
However, as it is right now, this still isn't enough information to figure out why you are getting a Serialization error. Today I created a Lambda triggered from SQS with the aws-otel-python38-ver-1-5-0:3 python layer and it completed successfully.
That's why the best way to get to the bottom of this is to get logs for what the SDK is sending to the collector. We want to see what traces are being sent by the OTel Python SDK before it gets to the collector. Like I said before this isn't necessarily easy with the already published layers, but the Lambda event is a great start. If that's not the issue then we should look into your Lambda code, then configuration, and so on.
If it looks fine then we want to investigate the ADOT Collector exporter code and want to figure out why it can't serialize the span it receives.
Please let me know if that makes sense 🙂 Also please let me know if you're able to provide me with your trouble-causing Lambda event, Lambda Function Code, your SQS setup, how you configure your Lambda function (it looks to be set to Active tracing), etc so we can try to reproduce the issue.

@NathanielRN Thanks for the really detailed replies and additional context!
I have got this working with another of my python lambdas that has exactly the same config/layer setup. So I think your theory that this issue relates to the code that is being instrumented may be correct.
I'll look into how I can use a layer with debugging enabled to see what's going on with the function where I get the error

@bellmatt That’s so awesome I’m really glad to hear it!! 😄
Yes please let us know if you can share the code that can reproduce this issue 🙂 Even if it is some unique user code, the exporter shouldn’t be throwing an exception or at least it should be a more helpful one… thanks again!

Hi @NathanielRN, I have enabled the logging exporter in debug logging level:
receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  awsxray:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, awsxray]
    metrics:
      receivers: [otlp]
      exporters: [logging]

Here are traces with this configuration (slightly redacted but number of characters preserved):
2021/11/19 16:53:52 [collector] Waiting for event...
2021-11-19 16:53:52,551 New SQS received. Message content: {'kind': 'channel', 'channel_name': 'fr-team-test', 'message': ':team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n'}
2021-11-19 16:53:52,797 Loading s3:s3
2021-11-19 16:53:52,816 Loading s3:Object
2021-11-19 16:53:52,817 Calling s3:get_object with {'Bucket': 's3-preprod-team-slack-bot-global-eu-west-3-012345678901', 'Key': 'cache/slack_channels.json'}
2021-11-19 16:53:53,012 Response: {'ResponseMetadata': {'RequestId': 'K848WGG97HHNX3H3', 'HostId': 'bAsPLmzu76mEWQt1N7wzpQ5vQa/f+llhlo0mW0R6H9Gx02uXWWxIKjyCROfWq5pnFmWk3/lp/94=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'bAsPLmzu76mEWQt1N7wzpQ5vQa/f+llhlo0mW0R6H9Gx02uXWWxIKjyCROfWq5pnFmWk3/lp/94=', 'x-amz-request-id': 'K848WGG97HHNX3H3', 'date': 'Fri, 19 Nov 2021 16:53:53 GMT', 'last-modified': 'Wed, 15 Sep 2021 08:53:26 GMT', 'etag': '"ca4e9884f9fb4f382fa0d9d54630c522"', 'x-amz-version-id': 'rhMyJVvp35YcSFcn6DinSOnpBeyoM8U5', 'accept-ranges': 'bytes', 'content-type': 'binary/octet-stream', 'server': 'AmazonS3', 'content-length': '3338817'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2021, 9, 15, 8, 53, 26, tzinfo=tzutc()), 'ContentLength': 3338817, 'ETag': '"ca4e9884f9fb4f382fa0d9d54630c522"', 'VersionId': 'rhMyJVvp35YcSFcn6DinSOnpBeyoM8U5', 'ContentType': 'binary/octet-stream', 'Metadata': {}, 'Body': <botocore.response.StreamingBody object at 0x7fd18c122bb0>}
2021-11-19 16:53:53,497 Executing post_simple_message with parameter: {'channel_id': '***********', 'message': ':team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n', 'kind': 'channel', 'channel_name': 'fr-team-test'}
/opt/python/slack_sdk/web/internal_utils.py:271: UserWarning: The `text` argument is missing in the request payload for a chat.postMessage call - It's a best practice to always provide a `text` argument when posting a message. The `text` argument is used in places where content cannot be rendered such as: system push notifications, assistive technology such as screen readers, etc.
warnings.warn(message, UserWarning)
2021-11-19 16:53:53,515 Sending a request - url: https://www.slack.com/api/chat.postMessage, query_params:
{}
, body_params:
{}
, files:
{}
, json_body: {'channel': '***********', 'blocks': [{'type': 'section', 'text': {'type': 'mrkdwn', 'text': ':team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n'}}]}, headers: {'Content-Type': 'application/json;charset=utf-8', 'Authorization': '(redacted)', 'User-Agent': 'Bolt/1.6.1 Python/3.8.11 slackclient/3.11.2 Linux/4.14.246-198.474.amzn2.x86_64'}
2021-11-19 16:53:53,963 Received the following response - status: 200, headers: {'date': 'Fri, 19 Nov 2021 16:53:53 GMT', 'server': 'Apache', 'access-control-allow-origin': '*', 'referrer-policy': 'no-referrer', 'x-slack-backend': 'r', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'access-control-allow-headers': 'slack-route, x-slack-version-ts, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags', 'access-control-expose-headers': 'x-slack-req-id, retry-after', 'x-oauth-scopes': 'chat:write,commands,incoming-webhook,app_mentions:read,channels:history,channels:read,groups:history,users:read,groups:read', 'x-accepted-oauth-scopes': 'chat:write', 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'cache-control': 'private, no-cache, no-store, must-revalidate', 'pragma': 'no-cache', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'x-slack-req-id': '7f632e42ebfab3c613c1309e26567ecb', 'vary': 'Accept-Encoding', 'content-type': 'application/json; charset=utf-8', 'x-envoy-upstream-service-time': '349', 'x-backend': 'main_normal main_bedrock_normal_with_overflow main_canary_with_overflow main_bedrock_canary_with_overflow main_control_with_overflow main_bedrock_control_with_overflow', 'x-server': 'slack-www-hhvm-main-iad-8mc7', 'x-slack-shared-secret-outcome': 'no-match', 'via': 'envoy-www-iad-u992, envoy-edge-lhr-selo', 'x-edge-backend': 'envoy-www', 'x-slack-edge-shared-secret-outcome': 'no-match', 'connection': 'close', 'transfer-encoding': 'chunked'}, body:
{
    "ok": true,
    "channel": "***********",
    "ts": "1637340833.026600",
    "message": {
        "bot_id": "***********",
        "type": "message",
        "text": "This content can't be displayed.",
        "user": "***********",
        "ts": "1637340833.026600",
        "team": "***********",
        "bot_profile": {
            "id": "***********",
            "app_id": "A024MS7V8RY",
            "name": "*****-Test",
            "icons": {
                "image_36": "https://avatars.slack-edge.com/2021-06-21/*************_8d9b3d67513acf315dde_36.png",
                "image_48": "https://avatars.slack-edge.com/2021-06-21/*************_8d9b3d67513acf315dde_48.png",
                "image_72": "https://avatars.slack-edge.com/2021-06-21/*************_8d9b3d67513acf315dde_72.png"
            },
            "deleted": false,
            "updated":<PHONE_NUMBER>,
            "team_id": "***********"
        },
        "blocks": [
            {
                "type": "section",
                "block_id": "lgSW",
                "text": {
                    "type": "mrkdwn",
                    "text": ":team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n",
                    "verbatim": false
                }
            }
        ]
    }
}

2021-11-19T16:53:54.015Z	INFO	loggingexporter/logging_exporter.go:41	TracesExporter
{
    "#spans": 5
}

2021-11-19T16:53:54.015Z	DEBUG	loggingexporter/logging_exporter.go:51	ResourceSpans #0
Resource labels:
     -> telemetry.sdk.language: STRING(python)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.5.0)
     -> cloud.region: STRING(eu-west-3)
     -> cloud.provider: STRING(aws)
     -> faas.name: STRING(team-slack-bot)
     -> faas.version: STRING($LATEST)
     -> service.name: STRING(team-slack-bot)
InstrumentationLibrarySpans #0
InstrumentationLibrary opentelemetry.instrumentation.botocore 0.24b0
Span #0
    Trace ID       : 7afe3987293befaea019f397f1c61de1
    Parent ID      :
    ID             : 82495e7a7211ec94
    Name           : secretsmanager
    Kind           : SPAN_KIND_CLIENT
    Start time     : 2021-11-19 16:53:52.004638344 +0000 UTC
    End time       : 2021-11-19 16:53:52.077158974 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> aws.operation: STRING(GetSecretValue)
     -> aws.region: STRING(eu-west-3)
     -> aws.service: STRING(secretsmanager)
     -> aws.request_id: STRING(e411e04e-da75-472f-ba63-e8a26a35313b)
     -> retry_attempts: INT(0)
     -> http.status_code: INT(200)
Span #1
    Trace ID       : 7ea2d8472bb17969d5d8724bed11ba60
    Parent ID      :
    ID             : 2528edcde60e561f
    Name           : secretsmanager
    Kind           : SPAN_KIND_CLIENT
    Start time     : 2021-11-19 16:53:52.315715612 +0000 UTC
    End time       : 2021-11-19 16:53:52.39317674 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> aws.operation: STRING(GetSecretValue)
     -> aws.region: STRING(eu-west-3)
     -> aws.service: STRING(secretsmanager)
     -> aws.request_id: STRING(5a947ff6-d90b-4ec1-a4a8-aa84cfd06714)
     -> retry_attempts: INT(0)
     -> http.status_code: INT(200)
Span #2
    Trace ID       : bca5ac01b559ce88ffe08d2b04753c20
    Parent ID      :
    ID             : 033f72c42403a4b2
    Name           : sqs
    Kind           : SPAN_KIND_CLIENT
    Start time     : 2021-11-19 16:53:52.412566147 +0000 UTC
    End time       : 2021-11-19 16:53:52.447696825 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> aws.operation: STRING(GetQueueUrl)
     -> aws.region: STRING(eu-west-3)
     -> aws.service: STRING(sqs)
     -> aws.request_id: STRING(cdef0b08-70c7-59b7-a8d1-5119a8273cfa)
     -> retry_attempts: INT(0)
     -> http.status_code: INT(200)
Span #3
    Trace ID       : 6197d69d69846f2d792172ae1d92a3a3
    Parent ID      : 64988892b117aa41
    ID             : 8ed5326f67f8fd4b
    Name           : s3
    Kind           : SPAN_KIND_CLIENT
    Start time     : 2021-11-19 16:53:52.835015137 +0000 UTC
    End time       : 2021-11-19 16:53:53.012767137 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> aws.operation: STRING(GetObject)
     -> aws.region: STRING(eu-west-3)
     -> aws.service: STRING(s3)
     -> aws.request_id: STRING(K848WGG97HHNX3H3)
     -> retry_attempts: INT(0)
     -> http.status_code: INT(200)
InstrumentationLibrarySpans #1
InstrumentationLibrary opentelemetry.instrumentation.aws_lambda 0.16.dev0
Span #0
    Trace ID       : 6197d69d69846f2d792172ae1d92a3a3
    Parent ID      : 937e3f012e0b69cb
    ID             : 64988892b117aa41
    Name           : *****.main.lambda_handler
    Kind           : SPAN_KIND_SERVER
    Start time     : 2021-11-19 16:53:52.55144591 +0000 UTC
    End time       : 2021-11-19 16:53:53.964310971 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> faas.execution: STRING(bae4419a-1677-5e6f-858a-1b8bcea4b43f)
     -> faas.id: STRING(arn:aws:lambda:eu-west-3:012345678901:function:team-slack-bot)
     -> faas.name: STRING(team-slack-bot)
     -> faas.version: STRING($LATEST)

2021-11-19T16:53:54.228Z	error	exporterhelper/queued_retry.go:165	Exporting failed. Try enabling retry_on_failure config option.
{
    "kind": "exporter",
    "name": "awsxray",
    "error": "Permanent error: SerializationException: \n\tstatus code: 400, request id: 6181b57b-c12a-4520-8318-1520f2d712cb"
}

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
	go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/queued_retry.go:165
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
	go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/traces.go:134
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
	go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/queued_retry.go:99
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
	go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/traces.go:113
go.opentelemetry.io/collector/consumer/consumerhelper.ConsumeTracesFunc.ConsumeTraces
	go.opentelemetry.io/collector@v0.36.0/consumer/consumerhelper/traces.go:29
go.opentelemetry.io/collector/service/internal/fanoutconsumer.traceConsumer.ConsumeTraces
	go.opentelemetry.io/collector@v0.36.0/service/internal/fanoutconsumer/consumer.go:79
go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export
	go.opentelemetry.io/collector@v0.36.0/receiver/otlpreceiver/internal/trace/otlp.go:60
go.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export
	go.opentelemetry.io/collector/model@v0.36.0/otlpgrpc/traces.go:102
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1
	go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.24.0/interceptor.go:325
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler
	go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219
google.golang.org/grpc.(*Server).processUnaryRPC
	google.golang.org/grpc@v1.40.0/server.go:1297
google.golang.org/grpc.(*Server).handleStream
	google.golang.org/grpc@v1.40.0/server.go:1626
google.golang.org/grpc.(*Server).serveStreams.func1.2
	google.golang.org/grpc@v1.40.0/server.go:941
2021-11-19T16:53:54.228Z	error	exporterhelper/queued_retry.go:101	Exporting failed. Dropping data. Try enabling sending_queue to survive temporary failures.
{
    "kind": "exporter",
    "name": "awsxray",
    "dropped_items": 5
}

go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
	go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/queued_retry.go:101
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
	go.opentelemetry.io/collector@v0.36.0/exporter/exporterhelper/traces.go:113
go.opentelemetry.io/collector/consumer/consumerhelper.ConsumeTracesFunc.ConsumeTraces
	go.opentelemetry.io/collector@v0.36.0/consumer/consumerhelper/traces.go:29
go.opentelemetry.io/collector/service/internal/fanoutconsumer.traceConsumer.ConsumeTraces
	go.opentelemetry.io/collector@v0.36.0/service/internal/fanoutconsumer/consumer.go:79
go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export
	go.opentelemetry.io/collector@v0.36.0/receiver/otlpreceiver/internal/trace/otlp.go:60
go.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export
	go.opentelemetry.io/collector/model@v0.36.0/otlpgrpc/traces.go:102
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1
	go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.24.0/interceptor.go:325
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler
	go.opentelemetry.io/collector/model@v0.36.0/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219
google.golang.org/grpc.(*Server).processUnaryRPC
	google.golang.org/grpc@v1.40.0/server.go:1297
google.golang.org/grpc.(*Server).handleStream
	google.golang.org/grpc@v1.40.0/server.go:1626
google.golang.org/grpc.(*Server).serveStreams.func1.2
	google.golang.org/grpc@v1.40.0/server.go:941
END RequestId: bae4419a-1677-5e6f-858a-1b8bcea4b43f
REPORT RequestId: bae4419a-1677-5e6f-858a-1b8bcea4b43f	Duration: 1680.31 ms	Billed Duration: 1681 ms	Memory Size: 256 MB	Max Memory Used: 174 MB	Init Duration: 2652.43 ms
XRAY TraceId: 1-6197d69d-69846f2d792172ae1d92a3a3	SegmentId: 79ecfe4c2fc4e126	Sampled: true

Also, I just noticed that the error does not happen with the very same Lambda function and some other SQS messages:
2021/11/19 16:54:03 [collector] Waiting for event...
2021-11-19 16:54:03,041 New SQS received. Message content: {'kind': 'channel', 'channel_name': 'fr-team-test', 'message': ':team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n'}
2021-11-19 16:54:03,043 Executing post_simple_message with parameter: {'channel_id': '***********', 'message': ':team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n', 'kind': 'channel', 'channel_name': 'fr-team-test'}
2021-11-19 16:54:03,043 Sending a request - url: https://www.slack.com/api/chat.postMessage, query_params:
{}
, body_params:
{}
, files:
{}
, json_body: {'channel': '***********', 'blocks': [{'type': 'section', 'text': {'type': 'mrkdwn', 'text': ':team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n'}}]}, headers: {'Content-Type': 'application/json;charset=utf-8', 'Authorization': '(redacted)', 'User-Agent': 'Bolt/1.6.1 Python/3.8.11 slackclient/3.11.2 Linux/4.14.246-198.474.amzn2.x86_64'}
2021-11-19 16:54:03,408 Received the following response - status: 200, headers: {'date': 'Fri, 19 Nov 2021 16:54:03 GMT', 'server': 'Apache', 'access-control-allow-origin': '*', 'referrer-policy': 'no-referrer', 'x-slack-backend': 'r', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'access-control-allow-headers': 'slack-route, x-slack-version-ts, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags', 'access-control-expose-headers': 'x-slack-req-id, retry-after', 'x-oauth-scopes': 'chat:write,commands,incoming-webhook,app_mentions:read,channels:history,channels:read,groups:history,users:read,groups:read', 'x-accepted-oauth-scopes': 'chat:write', 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'cache-control': 'private, no-cache, no-store, must-revalidate', 'pragma': 'no-cache', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'x-slack-req-id': 'ca6443fe89944cf739993add6d0fbbe2', 'vary': 'Accept-Encoding', 'content-type': 'application/json; charset=utf-8', 'x-envoy-upstream-service-time': '279', 'x-backend': 'main_normal main_bedrock_normal_with_overflow main_canary_with_overflow main_bedrock_canary_with_overflow main_control_with_overflow main_bedrock_control_with_overflow', 'x-server': 'slack-www-hhvm-main-iad-8gnd', 'x-slack-shared-secret-outcome': 'no-match', 'via': 'envoy-www-iad-v704, envoy-edge-lhr-fg4e', 'x-edge-backend': 'envoy-www', 'x-slack-edge-shared-secret-outcome': 'no-match', 'connection': 'close', 'transfer-encoding': 'chunked'}, body:
{
    "ok": true,
    "channel": "***********",
    "ts": "1637340843.026700",
    "message": {
        "bot_id": "***********",
        "type": "message",
        "text": "This content can't be displayed.",
        "user": "***********",
        "ts": "1637340843.026700",
        "team": "***********",
        "bot_profile": {
            "id": "***********",
            "app_id": "***********",
            "name": "*****-Test",
            "icons": {
                "image_36": "https://avatars.slack-edge.com/2021-06-21/*************_8d9b3d67513acf315dde_36.png",
                "image_48": "https://avatars.slack-edge.com/2021-06-21/*************_8d9b3d67513acf315dde_48.png",
                "image_72": "https://avatars.slack-edge.com/2021-06-21/*************_8d9b3d67513acf315dde_72.png"
            },
            "deleted": false,
            "updated":<PHONE_NUMBER>,
            "team_id": "***********"
        },
        "blocks": [
            {
                "type": "section",
                "block_id": "iLVm",
                "text": {
                    "type": "mrkdwn",
                    "text": ":team: Cloud Custodian report for account `012345678901`:\nmark-unattached-old-ebs-volumes-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-attached-ebs-volumes-marked-for-deletion affected aws.ebs count:\n- eu-west-3: 0\nunmark-used-ebs-snapshots-marked-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nmark-unused-old-ebs-snapshots-for-deletion affected aws.ebs-snapshot count:\n- eu-west-3: 0\nterminate-marked-ec2-instances affected aws.ec2 count:\n- eu-west-3: 0\ndelete-marked-ebs-volumes affected aws.ebs count:\n- eu-west-3: 0\nmark-old-ec2-instances-named-packer-builder-for-termination affected aws.ec2 count:\n- eu-west-3: 0\nunmark-used-amis-marked-for-deletion affected aws.ami count:\n- eu-west-3: 0\nmark-unused-old-amis-for-deregistration affected aws.ami count:\n- eu-west-3: 0\nderegister-marked-amis affected aws.ami count:\n- eu-west-3: 0\ndelete-marked-ebs-snapshots affected aws.ebs-snapshot count:\n- eu-west-3: 0\n",
                    "verbatim": false
                }
            }
        ]
    }
}

2021-11-19T16:54:03.415Z	INFO	loggingexporter/logging_exporter.go:41	TracesExporter
{
    "#spans": 1
}

2021-11-19T16:54:03.415Z	DEBUG	loggingexporter/logging_exporter.go:51	ResourceSpans #0
Resource labels:
     -> telemetry.sdk.language: STRING(python)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.5.0)
     -> cloud.region: STRING(eu-west-3)
     -> cloud.provider: STRING(aws)
     -> faas.name: STRING(team-slack-bot)
     -> faas.version: STRING($LATEST)
     -> service.name: STRING(team-slack-bot)
InstrumentationLibrarySpans #0
InstrumentationLibrary opentelemetry.instrumentation.aws_lambda 0.16.dev0
Span #0
    Trace ID       : 6197d6ab06a9913d0910e417499762de
    Parent ID      :<PHONE_NUMBER>61cffe
    ID             : 5591b7433c79b36d
    Name           : *****.main.lambda_handler
    Kind           : SPAN_KIND_SERVER
    Start time     : 2021-11-19 16:54:03.040898845 +0000 UTC
    End time       : 2021-11-19 16:54:03.40887126 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> faas.execution: STRING(8d6a3ba0-824f-556a-923b-61b15aacbf3a)
     -> faas.id: STRING(arn:aws:lambda:eu-west-3:012345678901:function:team-slack-bot)
     -> faas.name: STRING(team-slack-bot)
     -> faas.version: STRING($LATEST)

END RequestId: 8d6a3ba0-824f-556a-923b-61b15aacbf3a
REPORT RequestId: 8d6a3ba0-824f-556a-923b-61b15aacbf3a	Duration: 395.10 ms	Billed Duration: 396 ms	Memory Size: 256 MB	Max Memory Used: 175 MB
XRAY TraceId: 1-6197d6ab-06a9913d0910e417499762de	SegmentId: 243432f020d35f04	Sampled: true

@pdecat Thanks for providing that!
This is super interesting. However, I'm unable to reproduce the Serialization Error still on my side 😕 I used the same configuration and was able to call the S3, SQS, and secretsmanager services and have them be exported to AWS X-Ray:

It is difficult to say exactly what is causing the Serialization Error but we can try some things.

It definitely feels like user code is getting a response that is unserializable. Then, when the OTel Python SDK tries to put that in a trace, it makes it to the Collector and the error occurs. When I did a diff of your two logs I noticed that the 2 Lambdas had the same SQS message content according to the Message content: log. However, the "bad" call has a call to s3 which has a section that has <botocore.response.StreamingBody object at 0x7fd18c122bb0> in its response. Maybe that's not allowed? It would be interesting to remove this call to S3 and see if that is the difference in causing the Serialization Error. Similarly, you can extend this trail-and-error method of remove the other calls in the Lambda and find out which one is causing the problem? I know you said it works fine when it is triggered manually vs triggered from SQS but it also looks like you're able to trigger from SQS and get it working successfully? That's why I believe it is possible it is something in your actually Lambda function code?
Note that in my example I call S3 ListBuckets not GetObject, in case that is the problem causing the Serialization Error?
I noticed that you make an external call to what seems to be a Slack bot? I'm curious why this does not show up as a trace on the loggingexporter. This python38-ver-1-5-0:3 layer has instrumentation for the requests library so unless you are using urllib3 we should see a trace for it in the logging exporter?
In the "bad" example you have 5 traces: 2 from secretesmanager, 1 from s3, 1 from sqs and 1 from aws_lambda. Well, that's what we assume based on the loggingexporter output. I wonder if a trace is being created in addition to these traces but the loggingExporter is also experiencing a SerializationError and just silently failing as opposed to throwing an error like the awsXRayExporter... That's why while adding debug logging to the Collector is better, it would be best to see exactly what the OTel Python SDK has to send to the Collector to break the Collector. It's hard to do that without the Python Code that reproduces the error... this is why you can take your Lambda code and run it locally using manual instrumentation as explained in our dogs and then turn on debug logging on the OTel Python App locally. Then you'll see what gets sent to the Collector before it makes it there!
Otherwise if there are no hidden traces then it's very surprising that the awsXRayExporter fails because nothing in the loggingExporter's output looks particularly troublesome to serialize... In fact it looks like the calls we have regularly CI tests for in (calls using boto3 and the aws_lambda instrumentation itself)...

Please let me know if you try these ideas out! The most helpful would definitely be configuration that allows us to reproduce the error 🙂

I can reproduce the error using function code that does nothing except call S3 GetObject (or PutObject). Whereas if I then change the code to call ListObjects instead on the same bucket, it works and I can see trace data in xray and S3 appears on the service map.

@bellmatt I am able to GetObject from an S3 bucket and see the Service Trace map succeed:

Is what you have in the S3 content non-sensitive data that I could test trying to get myself? Otherwise could you try doing GetObject for something simpler (like a simple text file) and tell me if that works?

Hey @bellmatt @pdecat ! I was working on some stuff on AWS Lambda and I finally came across your error by chance 🙂
However, I am thinking it is an issue in the upstream ADOT Collector instead of anything in control of the managed Lambda layer.
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/6808
Can you both please look at it when you have a chance and let me know if this looks like something you are trying to do with your Lambdas? Create a Trace with a Parent Context that has a non X-Ray compliant format?
If you are, then electing to not use the auto-instrumentation /opt/otel-instrument script in your AWS_LAMBDA_EXEC_WRAPPER environment variable and instead manually instrumenting should prevent this issue, at the cost of not getting auto instrumentation or the trace generated by the opentelemetry-instrumentation-aws-lambda package which instruments a call of you Lambda function.
Will notify our team about the issue above so we can investigate as soon as possible!

The code has been merged and should be available in release 16 of otel. If you would like to test this code I can create a unofficial docker image for you.

Sorry it took me so long to get back to you, but this error doesn't occur with the latest aws-otel-lambda layer, so I think this can be closed


Script deletes itself and text files, but not audio file

Self explanatory.
There are two options on how to deal with this.

Don't delete any files

Can completely remove the escape key trap and its corresponding function
As such, can change shebang back to /bin/sh
May require a significant rework, but should be cleaner

Delete audio.opus as well

Easier to implement
Still requires Bash

While I personally think option 1 is better in the long run, it would require a significant rework that I'm not entirely sure how to approach.
Option 2 is suitable as a temporary fix. It would just require adding rm audio.opus to the ctrl_c function in script.sh.

When executing rat it deletes the entire folder, so it's not a problem there. The easiest way to do things as far as I can see would be just to remove the rm commands (and in turn the ctrl_c trap) from the script all together, as it's not necessary outside of the rat archive anyway. I'll try that now.

Thanks!

Also, for future reference, it's possible to use an exit trap instead of an interrupt trap.
trap <function> exit

I've reworked it all a bit - check 6fe51ea7c9390c94acbc05e3b7b3df91fd382b78. I removed the interrupt trap from script.sh and moved it all to the rat executable instead, so script.sh now has the original shebang and runs in ash etc. (still haven't fixed the weird glitch where it shows the audio progress bar but I'll get around to it).
rat automatically cleans up the whole temp folder anyway, so this issue seems resolved.

All seems resolved on this front!


Add checkbox support for multi select

Added checkbox support for multi select
Fixes #1311

Please add example using checkbox to index.html in order to test.

@interactivellama the first example will work. All that is needed is multi-select enabled.

Do we want any kind of set spacing on these checkboxes?

@BenjaminNeilDavis I think this is older code I'm referring to, but is there a reason that the action.html key is a function and not a string? Line 244: var html = action.html(); is a little confusing because it's so close to http://api.jquery.com/html/.

Please merge in glyphicons into actions dropdown. #1410

Are the double border line to the left side of actions column intentional? This is with column sizing on.

@interactivellama Did you do anything different to get that double line? What page are you looking at? What browser are you using? I am looking at index page in chrome and do not see that

Reviewed: Assigned to @kevinparkerson for additional review.

"Auto flip" of the dropdown to cause the actions dropdown to flip up has been added in this PR.
It does not use data-target due to wrapping div not having a unique ID, etc. and therefore transverses the all parent DOM notes for overflow:hidden.

@BenjaminNeilDavis It's Chrome. I think it has something to do with column sizing being on.

I see nothing wrong with the code after a look at it. Would like to test it locally, but the clone download percentage is still only at 34% after 20 minutes. Slow today, for some reason. Might check it out later after it finishes.

@interactivellama and @kevinparkerson approved offline


Use extract_plain_text to render plain text in /serials

connect to https://github.com/bm-sms/daimon-news-misc/issues/147

Looks like it's working for me. Any thoughts?

It works 👍
I think this PR is merge ready when https://github.com/bm-sms/daimon-news/pull/558#discussion_r68879511 is fixed.

🚀


Organization of some lib/scripts

Due diligence

[x] I am familiar with the "Reporting issues" section of CONTRIBUTING.md
[x] I have searched the existing issues and haven't found one that matches what I'm reporting or requesting now.
[x] If this is a bug report, I have searched the Bash changelog for information that may explain or provide clues to the behavior I'm observing and reference it in the body of the report.

Description
I am not sure I understand correctly the organization of the library files, but I thought I would ask anyway.
Under lib/ there is the bats-main script. This is the entry point to ./go test.
Then, almost each test file loads the tests/environment.bash file.
This, then, loads the "main" environment file lib/testing/environment plus the assertions library lib/bats/assertions.
If I understand the naming correctly, lib/bats/ is for libraries adding functionality that Bats is missing (more or less). So, we have the assertions library and background-process there. Then, lib/testing/ is for files that the test files will load on-demand if they need some special functionality to drive the application, eg the lib/testing/stack-trace file.
If the above are correct, here are my questions:

Shouldn't lib/bats/assertion-test-helpers be under lib/testing as it contains special functions to test a part of the application?
Shouldn't lib/bats/helpers, lib/bats/helper-function, lib/testing/environment be under a separate 3rd folder named perhaps lib/test-helpers or lib/testing-common or lib/testing/common? My thinking is that what these 3 files have in common (and what distinguishes them among the rest) is that they provide functions to write richer test functions.

This is not a functionally important issue but re-organizing might help a new contributor understand more easily what is going on and what each file is supposed to do. At least so it seems to me after spending ~1 hour trying to find where functions came from and the hierarchy of sourced files.
What do you think?

Fair questions. I do have a small explanation in the README to explain my intent with the current structure: https://github.com/mbland/go-script-bash#bats-test-assertions-and-helpers

Bats test assertions and helpers
The assertions and helpers from the test suite have been extracted into the lib/bats libraries. While these are not modules you can import with _GO_USE_MODULES, they are completely independent of the rest of the core framework and you may source them in your own Bats tests. (Whether or not these will ever become a separate library remains an open question.)
Variables, helper functions, and assertions for testing features based on the core framework are available in the lib/testing directory. The lib/bats-main library makes it easy to write a ./go test command script with the same interface and features as the core framework's ./go test command.

The key thing is that everything in lib/bats depends on nothing from the core framework at all (even though it's all tested with ./go test bats-*). Ideally I'd like to hoist that into a separate library (perhaps as a bats-core project).
Regardless, even within this repository, it might be worthwhile to reorganize the directories, just so long as everything in lib/bats remains completely independent.


issue #3544: Correct py/px to dy/dx in d3.t.ts GraphNode interface"

@nestalk could you review this PR?

Hello and thank you for your contribution!
Due to an excessively long queue of pull requests that have become stale, we are "declaring bankruptcy" and closing all PRs opened before May 1, 2016. If you'd still like to merge this code in, please open a new PR that has been merged and rebased with the master branch.
Going forward, we are committing to review or merge all PRs on a regular basis so this bankruptcy will not occur again. We apologize for the incovenience and hope you will continue to contribute to DefinitelyTyped in the future.
Thanks
The TypeScript and DefinitelyTyped teams


[admin] (Hyperlinks) Fixing link anchors for new OPS build

OPS Build status updates of commit 2fda8e9:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/design/using-office-ui-fabric-react.md
:white_check_mark:Succeeded

docs/excel/excel-add-ins-ranges.md
:white_check_mark:Succeeded

docs/excel/excel-add-ins-worksheets.md
:white_check_mark:Succeeded

docs/reference/objectmodel/preview-requirement-set/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/preview-requirement-set/office.context.mailbox.md
:white_check_mark:Succeeded

docs/reference/objectmodel/preview-requirement-set/office.context.md
:white_check_mark:Succeeded

docs/reference/objectmodel/preview-requirement-set/outlook-requirement-set-preview.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.1/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.1/outlook-requirement-set-1.1.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.2/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.3/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.3/outlook-requirement-set-1.3.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.4/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.5/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.5/office.context.mailbox.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.5/office.context.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.6/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.6/office.context.mailbox.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.6/office.context.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.6/outlook-requirement-set-1.6.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.7/office.context.mailbox.item.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.7/office.context.mailbox.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.7/office.context.md
:white_check_mark:Succeeded

docs/reference/objectmodel/requirement-set-1.7/outlook-requirement-set-1.7.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Binary download : assembly not strong named

Hi, in the binary download of the TaskScheduler assembly, the assembly is not signed.  Probably it is a good idea distribute it as a strong named assembly.
Also, the source code contains a (the ?) signing key file.
So, for now I made a compilation of my own and signed it with this key, but I believe you better make available a signed binary.
Joris Spriet
Originally posted: 2011-10-04T08:07:51

Thank you for your suggestion. The reason the key is there is specifically so it can be compiled and signed by those that need it. When I released it a couple of years ago signed, I had some big backlash from the community.
Originally posted: 2011-10-04T12:53:11

If you can elaborate I would like to know what requirements you have for strongly named assemblies so I can make a more informed decision. Also, what problems have you run into with strongly named assemblies in the past and how have you worked around them?
Originally posted: 2011-10-04T12:57:31

At my company, we sign all our assemblies.  It serves as a sign to our customers the assembly can be trusted - and it was produced on our build server.
As a consequence, we only can use third-party assemblies that are signed as well.  If a third-party component is not available signed,
we basically don't use it.  For TaskScheduler, as the source contained a snk, I compiled and signed it myself.
The library was just what I needed, and, okay, so I cheated here, but that is why I posted this remark...
Joris.
Originally posted: 2011-10-04T17:48:52


Revocation Flag Returns false when valid

When checking Revocation for a Verifiable Credentials the library returns valid = false when the credential is not revoked. When the credential is revoked is returns valid = true. This behavior is confusing and not consistent with how the other valid flags work.
Here is an example when a Verifiable Credentials passed both the revocation and json schema validation.

I revised the interface for status checks here: https://github.com/transmute-industries/verifiable-credentials/pull/13
Please let me know what you think


Editor doesn't let go of cursor when resizing width

Describe the bug
When resizing the code container, it's impossible to set a specific width.
To Reproduce

Go to https://carbon.now.sh
Click on right edge of code container.
Move cursor to one side
Although code container resizes, there's no way to set it at a particular width.

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots

Info (please complete the following information):

OS [e.g. macOS, Linux, Windows, iOS]: Linux
Browser [e.g. chrome, safari]: Chrome 83 (also tested in Firefox 77)

@b3u I have a feeling this is just because your pointer went offscreen. If you drag the window normally, staying within the window, does that work for you?

I believe 6e1b115 will have fixed any issues you noticed.


Don't cleanup comments with a regular expression, use ast

Hi @jpsca
Here is a proposal to fix the issue #85
I think that the code talk by himself.
Feel free to ask for comment.

Thank you


Error after deleting a project

After visiting /projects/foo/delete, I was presented with a yes/no prompt page, clicked yes, and then got an error page:
Could not find template match for "projectDeleted.tpl", checked paths:
html-templates/projects/not_in_philly/delete/projectDeleted.tpl
html-templates/projects/not_in_philly/projectDeleted.tpl
html-templates/projects/projectDeleted.tpl
html-templates/projectDeleted.tpl
The project does appear to be properly deleted.

This is a cosmetic issue just requiring a simple template to be setup for printing the post-delete status page. One could just copy the *Deleted.tpl for another record type and replace the nouns


[BUG] Compile failed on Databricks 13.3  ---  MockTaskContext.scala:69: overriding method getKillReason in class TaskContext of type ()Option[String]

Describe the bug
Build FAILED on Databricks-13.3 : MockTaskContext.scala:69: overriding method getKillReason in class TaskContext of type ()Option[String]
 [WARNING] Pruning sources from previous analysis, due to incompatible CompileSetup.
 [INFO] Compiling 136 Scala sources and 1 Java source to /home/ubuntu/spark-rapids/tests/target/spark341db/test-classes...
 [ERROR] /home/ubuntu/spark-rapids/tests/src/test/scala/org/apache/spark/sql/rapids/metrics/source/MockTaskContext.scala:69: overriding method getKillReason in class TaskContext of type ()Option[String];
  method getKillReason has weaker access privileges; it should be public
 [ERROR]   override private[spark] def getKillReason() = None
 [ERROR]                               ^
 [ERROR] one error found

Not sure if the failure has something to do with the latest Databricks maintenance?
https://docs.databricks.com/en/release-notes/runtime/maintenance-updates.html

dup: https://github.com/NVIDIA/spark-rapids/issues/10063


The /audit/storage page is slow to load on dev

My guess is that this is due to the number of product-location queries.  They're done for every row of the table, so it's an NxM query situation. On my local machine, with near-zero latency to local database per-query, it's quite fast to load.
(this is all a guess, of course we should measure)
Those calculations could be cached heavily as they're the same for everybody, don't change often, and are only estimates anyway (based on a tiny sample).

Alternative option: pre-calculate them with summary generation. It could then be an exact answer, rather than an estimate.

Between #268 and increasing the cache duration, this is probably no longer an issue. Closing.


Is there a way to display search results from my custom Graph connector under the "All" Vertical?

Hi, Is there a way to display results from my custom Graph connector under the "All" Vertical? I have indexed data that shows correctly under my custom Vertical, but does not appear under the "All" vertical.  Not having my custom connector data displayed under the "All" vertical makes any custom connector almost useless.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e785be0b-a0aa-13f2-21b9-2dd7defd39ea
Version Independent ID: cba8efaa-5cdb-0c2d-db36-df4b740694c5
Content: Manage Microsoft Graph Connectors for Microsoft Search
Content Source: MicrosoftSearch/manage-connector.md
Service: mssearch
GitHub Login: @monaray97
Microsoft Alias: mecampos

Hi @VelinGeorgiev, you can enable a result cluster for your connector which will show a set of connector results under "All." See this page here for more information: https://docs.microsoft.com/en-us/microsoftsearch/result-cluster. Let me know if you have any other questions.

Thanks for your response @monaray97!
However, the article you pasted did not help.

Here is what I did (according to the article)

Opened my Microsoft Search Admin panel
Navigated to the Verticals
Selected "All" and set it to "Show Connector Results"
Waited 24 hours.
Tested with my custom connector data, but results did not appear under the "All" Vertical while testing/searching.

Looks like custom Graph Connectors do not appear under the "All" Vertical making the Graph Connectors something that is not mature and usable at the moment. It is just not worth the configuration and development effort to create a connector that does not display data under the "All" Vertical.

@jthake-msft not sure if you can help, but I am desperately seeking for at least yes, no answer here.

Hi @VelinGeorgiev , can you validate if the "title" and the "content" are set up correctly for your custom connector. Discovery of the content is dependent upon this. Excerpt from the article "Relevant results are chosen based on the "title" property and "content" property of each item. The content property is marked as isContent=true on the schema."
Your custom connector schema should have "title" label added to the correct properties for discoverability. Some reference articles:
https://docs.microsoft.com/en-us/graph/api/externalconnectors-schema-create?view=graph-rest-1.0&tabs=http
https://docs.microsoft.com/en-us/microsoftsearch/configure-connector#step-5-assign-property-labels
https://docs.microsoft.com/en-us/microsoftsearch/configure-connector#content-property
Hope this helps

Hi @VelinGeorgiev
To add to @jypal6 comment, Result cluster will display results from custom verticals on All page with a high relevance threshold. While there might be matching results in custom verticals, if they fall below the relevance cutoff they will not be displayed on Result Cluster in All page.
Mapping title and other semantic labels to the connector properties can lead to better relevance.
If you are unable to get desired results, please do share your connection schema to look at.
Thanks.

Hi @VelinGeorgiev
Were the above pointers helpful. Do let us know if the results are now showing under the All vertical.


Not accepting servicetoken

As many times I try it says - Provided session token was invalid or no longer active and couldn't access Mi Cloud.
I have even tried userId, guserid, mUserId

Thank you for support.
I am doing exactly as mentioned in readme and as you guided here.
Check screenshots, same error
Latest chrome

Choosing proper values by copying complete value

Error screen

@ArijitU
Just released new version with different approach about determining whether user has signed in or not.
Please check if this resolved issue, personally I cannot reproduce this behaviour.

Tried new release, sorry didn't work.
Have you hardcoded the URL in the software anywhere?
Asking because I noticed in developer tools, under cookies yours is https://us.i.mi.com/ and ours is https://i.mi.com/ so if your software is trying to go to us.i.mi.com that won't work here.

I checked with Fiddler, indeed your app is trying to negotiate with us.i.mi.com, so that could be the issue
CONNECT us.i.mi.com:443 HTTP/1.1
Host: us.i.mi.com:443
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.125 Safari/537.36
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

@ArijitU
Thank you for the reply.
Yes, I'm using us.i.mi.com domain as this is where Mi Cloud redirects me after signing in.
After thinking I've come up with two root causes:

There may be region issue, based on region there are multiple domains for the same app (I'm in Europe, and based on your language code I reckon you are in India)
Domain may be based on used authentication method. I need info if you're using third party auth (eg. Facebook or Google)

Both will take me some time to consider and check if they are really responsible for that behaviour.

Yes I am in India, a suggestion could be to pass URL as field value alongside session and userid
No I am not using third party authentication, plain simple MI email+password authentication

Passing URL along with rest of the data could be a solution, but it will complicate whole process which is not suitable for end user.
When I decided to share this on GitHub I had simplicity on my mind, but as we see, it didn't work out.
At the same moment I will resolve issue whenever user is signing in with third party auth (check whether cookies are different or which are necessary to keep session alive).

Hi @ArijitU, as I had some spare time I managed to create separate release for you to test.
It provides support for alternative support for multiple domains. You can download it from here.
I reckon cookies wouldn't be different between domains.
Let me know if this resolves this issue, and thank you for participate in tests!

Hi, it seems that I'm having the same issue, though not related to India, my domain is us.i.mi.com. I've tried using Firefox and Chrome to no avail.

Hi, @HansLau, are you using the newest release, and the issue still persist?

Yes I've used 1.07 and the 1.08 pre release.

So, even if you're using pre-release (with potential fix) you're still getting "servicetoken is invalid" error?

Yea. I don't think it's related to domain name, but I'm not sure what the issue is

Have the same issue.
Chrome: Version 118.0.5993.118 (Official Build) (64-bit)
App: 1.1.2
.NET: 6.0.24
Default domain of us.i.mi.com

I have the same issue also , Any help please ?

@ArijitU @Crolug @HansLau @tabark92
There is a new release that resolves this issue.
Mi Cloud have changed their auth system, so the issue became more apparent.
Because it was regional issue back then I couldn't do much. I couldn't reproduce the behaviour.
Now it stopped working completely and I had to slightly modify the code.
Please let me know if this issue has been resolved with this update. Thanks.

Closing as it seems to be resolved.


feat: Add Adaptation Project's OData Service and SAPUI5 Model prompting

Feat for #2082.

Adds OData Service and SAPUI5 Model prompts to be used for prompting independent of environment.
Add CLI command to prompt and generate a change.

changeset ok
code adheres to coding standards
good test coverage
local test failed :(

Problem: I added the url '/test/abc' and the change and everything was created correctly however the preview fails because a url must end with a /. I'd suggest to either enhance the validators to catch this or (my preference) just to add the / if it is missing when writing the change.

Fixed with: e04528eaaaff66cadb23e6d0aab8a64683fb8685


Merge pull request #57 from Bonface-Osoro/main

Merge pull request #56 from Bonface-Osoro/refine_plots

Orbits added


Create arrows by clicking on link buttons in cards

Fixes #165

This is a bit sad, but I suppose it's due to my "small" screen.
And this will be fixed later, when styling is done.
Furthermore, it's looking good.

Extracted issue for that in #184


[SPARK-33960][SQL] Push down Limit through Sort

What changes were proposed in this pull request?
This pr add support push down Limit through Sort.
Why are the changes needed?
Improve query performance.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test and benchmark test:
 spark.range(500000000).write.saveAsTable("t1")
 spark.sql("select * from t1 order by  id limit 2").show

Before this pr
After this pr

Test build #133594 has started for PR 30992 at commit e82ff4c.

Test build #133594 has started for PR 30992 at commit e82ff4c.

Test build #133594 has finished for PR 30992 at commit e82ff4c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133594 has finished for PR 30992 at commit e82ff4c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38183/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38183/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133594/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133594/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38183/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38183/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38183/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38183/

Doesn't this change the output?
In the Range example the values are in order, but what if they are not?
For values 1, 3, 2 query Limit 2 -> Sort would sort it before and return 1, 2. With this change it would limit before sorting and return 1, 3.

Doesn't this change the output?
In the Range example the values are in order, but what if they are not?
For values 1, 3, 2 query Limit 2 -> Sort would sort it before and return 1, 2. With this change it would limit before sorting and return 1, 3.

Sorry, close it.

Sorry, close it.


Appstudio update mvp-test-component

Pipelines as Code configuration proposal

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
30 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
13 seconds
inspect-image

✅ Succeeded
11 seconds
deprecated-base-image-check

✅ Succeeded
18 seconds
clair-scan

✅ Succeeded
1 minute
clamav-scan

✅ Succeeded
14 seconds
sbom-json-check

✅ Succeeded
20 seconds
label-check

✅ Succeeded
18 seconds
optional-label-check

✅ Succeeded
7 seconds
show-summary


Increase max-items for minute specificity

Problem
When using --date-spec=min the visitor's graph does only show the past 6 hours. This is probably fine for real-time reporting, but not suitable to generate daily graphs. Turns out, this behavior is related to --max-items defaulting to 366.
Proposal
When --date-spec=min is set, the default for --max-items could be increased to 1440 to display the data for the last 24 hours.
Alternative: If the described use case (daily graphs) is not strong enough, a note in the man pages could be added, indicating, that number of displayed data point might be increased via --max-items to render a longer time range.
Side note
While searching for an answer to my problem, I found two ticket that are probably fixed and could be closed:

Need ability to break down VISITORS data by minute (#2277)
Add option to use actual minutes, instead of every 'tenth minute' (#1251)

I agree. I've made the change, it should now default to 1440 when --date-spec=min. Please feel free to build from development and let me know how it goes. Otherwise, it will be pushed out in the upcoming release.
Thanks for the feedback!


Preliminary Xtensa support

Based on the work in #804.
I hope non-upstream target support is acceptable :).

I've now reverted to using the critical section to avoid the race condition, for reference the CS impl I used looked like this:
struct CriticalSection;
critical_section::custom_impl!(CriticalSection);

static mut VPS: u32 = 0;
unsafe impl critical_section::Impl for CriticalSection {
    unsafe fn acquire() -> u8 {
        core::arch::asm!("rsil {0}, 15", out(reg) VPS);
        0
    }

    unsafe fn release(_token: u8) {
        core::arch::asm!("wsr.ps {0}", in(reg) VPS)
    }
}

With the next release of critical_section, I believe I can get rid of the static mut and pass it through the variable-sized token instead to restore processor state.

Hmm, if waiti 0 sets INTLEVEL=0, it's effectively exiting the critical section. I wonder if this can be used to create unsoundness, for example calling run() within a CS will cause it to end early, but I can't immediately see how to exploit that because run() is noreturn...

I'm not sure there is anything we can do here, waiti doesn't restore the previous INTLEVEL so you're right it does effectively end the CS there. However, this is only possible because we are using unsafe and we make sure that we don't include anything that needs to be inside the CS after the waiti call. This is sort of similar to cortex-m in that you can call interrupt::enable inside the critical section to end the critical section there but of course, this method is marked as unsafe in cortex-m.

FYI this file is what I am basing my implementation on, which came straight from Tensilica (now Cadence). Might be worth a quick skim read, just in case I'm missing something obvious :sweat_smile:.

To the best of my knowledge, I think this is the best we can do for now. I've added a comment to warn that the CS ends early when using waiti. It kinda sucks it works this way, it means we can't implement any CPU usage tracking etc. Hopefully, I can find a better way to do this in the future.
I hope this is acceptable as is, I'd be happy to manage any issues relating to Xtensa (or RISCV) if this gets merged :).


[DUPLICATE] Redesign actions in notification detailed view

DUPLICATE OF https://github.com/eclipse-tractusx/traceability-foss/issues/618

As a user ,
I want i want to have consistent design on notification actions ,
so that I always know intuitively where to find which action .
Hints / Details

Miro UI Concept Board : LINK

Add only icons with tooltip (Description of the action)
As Side nav container
Link necessary (new) Action icons to action

needs #620
Acceptance Criteria

[ ] implemented action icons for notification detailed view (as in concept)
[ ] Cypress e2e test user journey covers this improved detail view as part of the complete user journey
[ ] Added tooltip to new close action "Close Notification"

Out of Scope

...

I copied the ticket as the original tickets were deleted due to a spam account.
Orginal Ticket: https://github.com/eclipse-tractusx/traceability-foss/issues/887