ZNC 1.6.1 release archive doesn't contain submodules

make is not happening on release archive (https://github.com/znc/znc/releases/tag/znc-1.6.1):
root@services163:~/znc-znc-1.6.1/znc# make
Packing man page znc.1.gz...
Packing man page znc-buildmod.1.gz...
It looks like git submodules are not initialized. Run: git submodule update --init --recursive
make: *** [third_party/Csocket/Csocket.h] Error 1

The issue is: release archive does not contain '.git' folder, so there is no way to do git submodule update --init --recursive.
Solved by using not release archive, but git cloning particular release commit (in this case: aa6e7e7).
Could be reference to #806

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/27025526-znc-1-6-1-release-archive-doesn-t-contain-submodules?utm_campaign=plugin&utm_content=tracker%2F1759&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1759&utm_medium=issues&utm_source=github).

This issue is invalid and caused by ZNC using submodules which GitHub doesn't support with those zip/tarballs, but they cannot be disabled.
Please download ZNC from http://znc.in/releases/ instead.

I see. Thanks =]
Probably there is no reason to keep releases on GitHub then.

Probably there is no reason to keep releases on GitHub then.

I was unclear at first sorry, the tar/zip files are automatically generated by GitHub and they cannot be disabled. I have requested disabling them at least once from GitHub support.
This issue doesn't affect only ZNC, but also Atheme, https://github.com/atheme/atheme/issues/382.

@Mikaela
Thanks for explanation. Sorry for taking your time then :sweat_smile:

You're welcome, no problem. Could you close this next to the comment box?


Implement a chain combinator

Two parallel iterators of the same type act as one big iterator.  All
common properties are maintained: bounded, exact, and indexed.
VecDeque gets to be the first user, replacing its temporary vectors
instead with a chain over its pair of slices.

Very cool!

My only question is the comment, which seems a bit misleading, or maybe I'm confused.


Query Builder: Add ability to group by columns

Related: #15563
It is currently only possible to generate queries of the form
from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(<filter>)
  |> aggregateWindow(every: v.windowPeriod, fn: <agg>)

Since flux does a default group by *, it is not possible to apply a different grouping of data before aggregating.

Aggregate all series over time into a single series

More of less the query we'll want to be able to express is
from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "cpu")
  |> filter(fn: (r) => r._field == "usage_user")
  |> group()
  |> aggregateWindow(every: v.windowPeriod, fn: mean)
  |> yield(name: "mean")

POC I whipped up:

Chatted with Alex Pax about this on Slack. We both think there needs to be more affordance for switching between Filters and Groups so I'm going to make the Filter / Group a dropdown and allow selecting it.

Chatted with @mavarius about the design and UX of this. For the first initial pass (what I'm calling Group By 0.5) we're going to add a dropdown to select between filter and group. Adding a new BuilderCard will select filter by default, initially.

After the initial 0.5 functionality lands, we'll evaluate the best way to change the function with fewer clicks. Some initial thoughts are to have a list of functions to add in place of the single + icon, or a dropdown above Aggregate Functions with a list of functions to add.

Related: https://github.com/influxdata/influxdb/issues/16076


SWEndpoint LOG LEVEL error

#ifdef DEBUG
__unused static const int ddLogLevel = LOG_LEVEL_VERBOSE;
#else
__unused static const int ddLogLevel = LOG_LEVEL_OFF;
#endif
REDEFINITION OF LOG LEVEL

This is because the pod had a prefix header and I remove that in this version since prefix headers sometimes cause builds to not work properly. When you do pod install with version 0.2.0 then it should work properly.

4569ee13a3ca17c65327a25dc6582be0e66ca222 has been pushed. Should fix this issue


Feature Request: enable rechanging tf.device of a tensor

It seems that once a tensor's GPU was defined using tf.device, the GPU cannot be changed anymore.
When loading saved graphs, the graph will use the same GPU that was chosen years ago and it cannot be set again.
thanks.

Can you provide an example that demonstrates this?

Here I demonstrate the missing feature on the mnist classic example
# using tensorflow 1.3

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

# create a model and save the graph and the tensors for later training
def createModelAndSave(my_gpu):
    learning_rate = 0.01
    with tf.device(my_gpu): # all model's tensors are on this gpu, for example '/gpu:0'
        x = tf.placeholder("float", [None, 784])
        y = tf.placeholder("float", [None, 10])

        # Set model weights
        W = tf.Variable(tf.zeros([784, 10]))
        b = tf.Variable(tf.zeros([10]))

        model = tf.nn.softmax(tf.matmul(x, W) + b)

        # cost function
        cost_function = -tf.reduce_sum(y*tf.log(model))

        # Gradient descent
        optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost_function)

    # save all tensors in the collection
    tf.add_to_collection('cost_function',cost_function)
    tf.add_to_collection('optimizer',optimizer)
    tf.add_to_collection('x', x)
    tf.add_to_collection('y', y)

    saver = tf.train.Saver()
    with tf.Session() as sess:
        init = tf.global_variables_initializer()
        sess.run(init)
        saver.save(sess,'myModel.ckpt')

# load the saved graph and the tensors for training
def loadModelAndTrain():
    mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
    training_iteration = 1000
    batch_size = 100

    # load graph
    tf.train.import_meta_graph('myModel.ckpt.meta')
    with tf.Session() as sess:
        # load tensors
        cost_function = sess.graph.get_collection("cost_function")[0]
        optimizer = sess.graph.get_collection("optimizer")[0]
        x = sess.graph.get_collection("x")[0]
        y = sess.graph.get_collection("y")[0]

        # ~~~~~~~
        # At this point the tensors and the graph are loaded but all of them are on GPU that was chosen when the model was saved.
        # for example, the training will run on '/gpu:0' as chosen years ago.
        # assuming we don't have the function to create the model again, we cannot relocate the tensors to anther GPU.
        # ~~~~~~~
        init = tf.global_variables_initializer()
        sess.run(init)

        # Training cycle - all on the chosen GPU
        for iteration in range(training_iteration):
            avg_cost = 0.
            total_batch = int(mnist.train.num_examples/batch_size)
            for i in range(total_batch):
                batch_xs, batch_ys = mnist.train.next_batch(batch_size)
                sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})
                avg_cost += sess.run(cost_function, feed_dict={x: batch_xs, y: batch_ys})/total_batch
            # Display logs per iteration step
            print ("Iteration:", '%04d' % (iteration + 1), "cost=", "{:.9f}".format(avg_cost))

if __name__ == "__main__":

    # ~~~~~ first run createModelAndSave(), later comment this function and run loadModelAndTrain()

    createModelAndSave('/gpu:0') # save a model to a specific GPU
    # loadModelAndTrain() # load the saved model and train it, cannot rechoose GPU

https://www.tensorflow.org/api_docs/python/tf/train/import_meta_graph
import_meta_graph(
meta_graph_or_file,
clear_devices=False,
import_scope=None,
**kwargs
)

Thank you for your answer.
It's indeed clear the devices and rest them to their default GPU:0, but it is still not possible to choose another one. (GPU:2 for example).
we are a team that work  on several GPUs. The feature to LOAD each model to a different GPU is very important and still missing.


Improve task request details page UI and UX

Issue Description
Redesign the task requests details page. With the following requirements:

Task request status need to be shown and update when user approves/rejects.
Task request type need to be shown
All requestors name profile photo and request details need to be shown along with button to approve.
A button to reject task request need to be shown. This will reject all users request
If request type is CREATION GitHub issue details need to be shown
If request type is ASSIGNMENT task details need to be shown.

Checklist

[ ] I have read and followed the project's code of conduct.
[ ] I have searched for similar issues before creating this one.
[ ] I have provided all the necessary information to understand and reproduce the issue.
[ ] I am willing to contribute to the resolution of this issue.

Thank you for contributing to our project! We appreciate your help in improving it.

Current bugs: https://github.com/Real-Dev-Squad/website-dashboard/issues/603


fails to configure/install

Seems to install ok, but...
configuring gateway with wizzard does not work, nothing happens when pressing the spyglass icon.
manual configurations seems to work
when depolying node-red: Flows stopped due to missing node types.
telldus-gateway
telldus-device
log from install:
Aug  5 08:56:40 raspberrypi Node-RED[1654]: 5 Aug 08:56:40 - [info] Installing module: node-red-contrib-telldus-zwave-v2-local, version: 1.3.4
Aug  5 08:56:40 raspberrypi Node-RED[1654]: 5 Aug 08:56:40 - [trace]<EMAIL_ADDRESS>Aug  5 08:57:10 raspberrypi Node-RED[1654]: 5 Aug 08:57:10 - [info] Installed module: node-red-contrib-telldus-zwave-v2-local
Aug  5 08:57:11 raspberrypi Node-RED[1654]: 5 Aug 08:57:11 - [info] Added node types:
Aug  5 08:57:11 raspberrypi Node-RED[1654]: 5 Aug 08:57:11 - [info]  - node-red-contrib-telldus-zwave-v2-local:telldus-gateway : SyntaxError: missing ) after argument list
Aug  5 08:57:11 raspberrypi Node-RED[1654]: 5 Aug 08:57:11 - [info]  - node-red-contrib-telldus-zwave-v2-local:telldus-device : SyntaxError: missing ) after argument list
Aug  5 08:57:11 raspberrypi Node-RED[1654]: 5 Aug 08:57:11 - [info]  - node-red-contrib-telldus-zwave-v2-local:telldus-metricsensor : SyntaxError: Unexpected token function
Aug  5 08:57:11 raspberrypi Node-RED[1654]: 5 Aug 08:57:11 - [info]  - node-red-contrib-telldus-zwave-v2-local:telldus-switchsensor : SyntaxError: Unexpected token function
Manage pallet shows the node-red-contrib-telldus-zwave-v2-local, last object with a red exklamation mark!
nodes depreciated after restart:
5 Aug 10:21:15 - [info] Node-RED version: v0.18.7
5 Aug 10:21:15 - [info] Node.js  version: v6.14.2
5 Aug 10:21:15 - [info] Linux 4.14.34-v7+ arm LE
5 Aug 10:21:16 - [info] Loading palette nodes
5 Aug 10:21:26 - [info] Dashboard up and running
5 Aug 10:21:30 - [info] Dashboard version 2.6.2 started at /ui
5 Aug 10:21:32 - [warn] ------------------------------------------------------
5 Aug 10:21:32 - [warn] [node-red-contrib-telldus-zwave-v2-local/telldus-gateway] SyntaxError: missing ) after argument list
5 Aug 10:21:32 - [warn] [node-red-contrib-telldus-zwave-v2-local/telldus-device] SyntaxError: missing ) after argument list
5 Aug 10:21:32 - [warn] [node-red-contrib-telldus-zwave-v2-local/telldus-metricsensor] SyntaxError: Unexpected token function
5 Aug 10:21:32 - [warn] [node-red-contrib-telldus-zwave-v2-local/telldus-switchsensor] SyntaxError: Unexpected token function
5 Aug 10:21:32 - [warn] ------------------------------------------------------
I also have "node telldus live" and "node-red-contrib-tellstick" could there be a conflict?

Been seeing weird errors from people using old node versions. Take the appropriate steps to upgrade your node version and try again.

Works fine with Node.js  version: v8.11.3, Thanks!


Feature: Implementing linear release logic in the economic model

Implementing linear release logic in the economic model.
When the mainnet was launched, genesis released 15% of the tokens, and the remaining 85% needed to be released linearly. With reference to the economic model, a linear release logic is implemented.
Total Allocation
The Initial Dex Offering(IDO) 1%, and the genesis is fully released.
LP mining rewards, 60%, genesis release 5%, and the rest will be released linearly for two years.
Stake rewards 10%, genesis release 5%, and the rest will be released linearly for one year.
Community incentives 5%, for community incentives, airdrops, etc. Genesis releases 2%, and the rest will be released linearly for two years.
Developer Funds 10%, released linearly for two years.
Protocol Treasury 14%, used for initial liquidity, community governance, ecological construction, etc. Genesis release 2%, the rest is locked in DAO Treasury, and it will be operated by voting after DAO is launched.
Refer ： https://docs.starswap.xyz/v/en/tokenomics/tokenomics

this has implement in https://github.com/Elements-Studio/starswap-core/pull/18


Update Brooklyn codebase to build the newly donated UI

See discussion about the donated UI<EMAIL_ADDRESS>Note that it also requires:

https://github.com/apache/brooklyn-server/pull/974

looks good, will merge once apache/brooklyn-server#974 is in


Add rust-toolchain and set MSRV to 1.79

Having a set MSRV is a good idea, as is enforcing it with a rust-toolchain file.
We can probably use an earlier version if you prefer; i just went with 1.79 for no particular reason. Let me know!

Looks like we couldn't go much further back than 1.73, the least common denominator of all the current dependencies. I'll merge it as-is, and if anyone stumbles on this requirement, please open an issue!
Thank you @emilk! 🙌


Account suspension: No information on claim information

In [1] it states that there are relevant claim values and they can be viewed by following another document.
By default, the claim values of the identity claims used in this feature are stored in the JDBC datasource configured in the identity.xml file. See Configuring Claims for more information on how to store the claim values in the user store.

If any claims are used for this feature then they needs to be mentioned here. Please mention any such here itself
[1] https://docs.wso2.com/display/IS540/User+Account+Suspension

Thank you for your contribution!
We are closing this issue since it has not been prioritized for a long time. Chances are that it has already been solved in more recent versions. If not, we will be re-evaluating this when it becomes a priority.


Finalize PSA config support

Description
Add missing PSA_WANT_CCM/GCM/CMAC. This completes the set of PSA_WANT config options given the current support of PSA crypto in Mbed TLS. Then update the test code/data dependencies using those PSA config option and those introduced in #4009.
Status
READY
Requires Backporting
NO, PSA only.
Steps to test or reproduce
CI testing is enough.

Some of these changes have already been done in #4240...

Yes the coordination has not been really good on AEAD. I mentioned a couple of times that I would do it though.

"This block of code was accidentally left in place in ab5631.
Please remove it."
Indeed, thanks for catching this. This should be fixed now.

@bensze01 Is that ok with you now that I've addressed your latest comment?


Add armeria-scala module

Motivation:
When using Armeria in a Scala project, a user will often find the
following issues:

Conversion between Java futures and Scala futures
Conversion between Java collections and Scala collections
Conversion between EventLoop/Executor and ExecutionContext
Inability to use lambda expressions due to incomplete type inference

Modifications:

Added a new module armeria-scala.

See site/src/pages/docs/advanced-scala.mdx for the complete
feature list.

Cleaned up Scala builds

Extracted scala-library and gradle-scalafmt version to
dependencies.yml.

Updated scala-library from 2.13.4 to 2.13.5 and 2.12.12 to 2.12.13

Removed the grpc-scalapb_2.13 flag from :scalapb_2.12 because we
reuse the source code generated by :scalapb_2.13.
Added the no_aggregation flag to :scalapb_2.12 because it conflicts
with :scalapb_2.13 during aggregation. We currently don't do any
aggregation for Scala code, but we will get an error once we start to
aggregate.
Made :scalapb_2.12 depend on :scalapb_2.13:generateTestProto
instead of :scalapb_2.13:compileTestScala.

Result:

Better Scala interoperability
Cleaner Scala build

Preview:

@minwoox Gentle ping :bow:

Thanks for reviewing, everyone. :bow:


KML - Double click point and left click functionality breaks

https://cesiumjs.org/Cesium/Apps/Sandcastle/index.html?src=KML.html&label=Showcases
The KML showcase in the Cesium Sandcastle shows 3 sample KML files. Allow it to load the default selected KML, KML - Global Science Facilities, and double click one of the points displayed on the map.
After you are zoomed into that point (is this an intended feature?) you can no longer left click and drag to pan the map. Left click behaves like middle click in that it rotates and tilts the camera. Middle click no longer works.

Yes, that is the intended behavior.  Cesium has first class support for time dynamic and moving objects, so when you are double click on something, the default behavior is to go into tracking mode.  You can disable the default double click handling and add your own.  Here's an example that zooms to the entity rather than tracking it.
viewer.screenSpaceEventHandler.setInputAction(function(e) {
    var picked = viewer.scene.pick(e.position);
    if (Cesium.defined(picked)) {
        var id = Cesium.defaultValue(picked.id, picked.primitive.id);
        if (id instanceof Cesium.Entity) {
            var entity = id;
            if (Cesium.defined(entity)) {
                viewer.zoomTo(entity);
            }
        }
    }
}, Cesium.ScreenSpaceEventType.LEFT_DOUBLE_CLICK);

In the future, please start a forum discussion for any questions you might have rather than opening a GitHub issue.  You are likely to get faster responses on the forum (it was a fluke that I answered this so quickly) and more people will see it.  We try to reserve GitHub issues for confirmed bugs.
If you're new to Cesium, I also recommend you check out the Visualizing Spatial Data tutorial.  Which will hopefully answer a lot of questions up front.


Add integration tests for new services

This adds basic smoke tests for the services added in the last
6 months or so.
cc @kyleknap

Two questions:

Are any of these services in preview (i.e. do you need to subscribe to them)? Just asking because sometimes for some of the newer services, you need to subscribe to them as they are in preview, or they will throw errors.
What is your thoughts on adding integration tests to the error smoke tests for these services?

Nope, AFAIK these are all generally available.  Verified these pass for me on multiple AWS accounts.

Not opposed.  We started that, but it looks like it's currently incomplete.  Would not be opposed to another PR that fleshes this out more.

Awesome. :ship: It would be nice as principle for every service to have one success case and one error case for integration test. We can add a PR for that later down the road.

Coverage decreased (-1.11%) to 93.34% when pulling a474f67318e48426b388810ef22db680f447da32 on jamesls:more-smoke-tests into 875dfe5c495689015c0fda3eadf2a7d8ccf820d0 on boto:develop.


Add rails and rjsf groups to dependabot.yml

Motivation and Context

As I was reviewing dependabot PRs, I noticed two more dependabot groups that we can specify to simplify the PRs.
Your Changes

Description: Added rails and rjsf groups to dependabot.yml.
Type of change (select all that apply):

[x] Other (please specify): dependabot configuration

Checklist

[x] I have performed a self-review of my own code.
[ ] I have verified that the pre-commit.ci checks have passed.
[ ] I have verified that the CI tests have passed.
[ ] I have reviewed the test coverage changes reported on Coveralls.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 91.814%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
38229

Relevant Lines:
41073

💛  - Coveralls


refactor(connector): tidy tiny code by removing redundancy code.

Signed-off-by: tabVersion<EMAIL_ADDRESS>Co-authored-by: xiekeyi98<EMAIL_ADDRESS>Signed-off-by: xiekeyi98<EMAIL_ADDRESS>What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!
Please explain IN DETAIL what the changes are in this PR and why they are needed:

Summarize your change (mandatory)

Tidy code by removing redundancy code(UPSTREAM_SOURCE_KEY).

How does this PR work? Need a brief introduction for the changed logic (optional)
Describe clearly one logical change and avoid lazy messages (optional)
Describe any limitations of the current code (optional)

Checklist

[x] I have written necessary docs and comments
[x] I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)
related #1963 ( cherry-pick from https://github.com/singularity-data/risingwave/pull/1963/commits/ccba8562bc424ff2d630e71c79a2c30f76fda26c)

PTAL @tabVersion  thanks.


v3: misc fixes

@alainjobart

Plumb initialization context from main function.
Verify GetShardNames in vschema test.

This change is

LGTM with one minor comment on a comment. ;)


Fix ARN of STS

Issue #, if available:
Description of changes:
The STS ARN is not the good one. it came from an older workshop
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I believe this has been addressed with other edits.


Rleayers update

Update existing relayers to 11.1 v
Checkout drives, setup monitors.
Setup relayer for Gravity-Bostrom

Relayers have been updated.


Local sync?

Is it possible to use SyncedStore without one of the Y.js providers ?

https://docs.yjs.dev/ecosystem/connection-provider
https://docs.yjs.dev/ecosystem/connection-provider

I wonder if SyncedStore could be used like any other state library (redux, mobx...) and only upgrade/configure it once you really need collaboration?

Hi Eric!
Can you specify more what functionality you’re looking for exactly (what’s
the functionality you’d like to “keep”?).
It should be possible to use syncedstore, and just not connect any sync
provider.
Alternatively, perhaps the Reactive library could provide what you’re
looking for?
https://github.com/YousefED/reactive
On Sun, 23 Jan 2022 at 20:20, Eric Charles @.***> wrote:

Is it possible to use SyncedStore without one of the Y.js providers ?

https://docs.yjs.dev/ecosystem/connection-provider
https://docs.yjs.dev/ecosystem/connection-provider

I wonder if SyncedStore could be used like any other state library
(redux, mobx...) and only upgrade/configure it once you really need
collaboration?
—
Reply to this email directly, view it on GitHub
https://github.com/YousefED/SyncedStore/issues/46, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC2BWNOQLI6BDKYFRDWR3TUXRIHLANCNFSM5MTWWOIQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Can you specify more what functionality you’re looking for exactly (what’s
the functionality you’d like to “keep”?).

Just a local sync without depending on a y.js provider (so no webrtc, no websocket, no indexdb...).
I guess it should be possible to go throught the Y.Doc in memory and just update the store via that local Y.Doc.

It should be possible to use syncedstore, and just not connect any sync provider.

Yes, but for non-syncing appliation, using a sync provider could not be mandatory.
I have looked for a "in-memory" sync provider, but did not find any.
Even if such a in-memory would exist, I guess it would create a data structure that would not be needed in case of local (non-syncing) state.

Alternatively, perhaps the Reactive library could provide what you’relooking for? https://github.com/YousefED/reactive

Yes, I have looked at that and have opened a question on https://github.com/YousefED/reactive/issues/5 because I don't see how to use it from different React components.

Yes, but for non-syncing appliation, using a sync provider could not be mandatory.

A sync provider is not mandatory for SyncedStore, just like it's not mandatory for Yjs. If you just create a Y.Doc, everything will work in memory. Only if you want to persist data or enable collaboration, you'll need a sync provider.
Hope this makes sense! I'll close this issue for now, but feel free to reply with additional questions

Makes sense! Thx @YousefED


Review_readme_15_LoginForm_#23

Review finished. I think that it is very useful to put the tree folder while you are reading the Readme.md

Good stuff, proceeding to merge please updaate issue and kanban


ROS2 CI dev

I think you want to remove the "ROS2 Drivetrain Description" commit or squash the last two commits together

Woah @younesr1 r ya taking over the pr?

Woah @younesr1 r ya taking over the pr?

nah lol just hadnt realised i was on ci-dev


Error in virtual-audio.js

Hey first of all I wanna say this library is awesome, really impressive work! I've been wanting to use WebAudio in a project, and I love your virtual audio graph approach. I used the library to put together a small sequencer-type demo, and it worked great! One small thing I had to change though was in virtual-audio.js. When scheduling future updates to audio parameters, Elm is sending out ScheduledUpdate messages, but virtual-audio.js is expecting a ScheduledAudioParam message. Changing ScheduledAudioParam to ScheduledUpdate in the Javascript fixed the problem.
Not sure if you're looking for collaborators, but I'm really interested in this project :)

Oh excellent, the virtual-audio implemention is currently a handy convenience rather than an official part of the package (right now).
I think the virtual-audio.js is slightly out of date compared to a more current implementation I have for a separate project. Thanks for pointing it out; I’ll make sure everything is working a bit later tonight.
Neat sequencer by the way!

Awesome, looking forward to seeing what your working on!


jaeger-agent reproducible memory leak in 1.21.0

Describe the bug
I am observing very high and rapidly increasing memory usage of jaeger-agent which may be a memory leak.
Eventually the agent (container) may run out of memory and crash.
I am able to reproduce the behavior reliably.
It is happening at very low span rates of <= 30 or 50 Spans/sec according to jaeger_collector_spans_received_total
I am using a dev setup running Demo ASP.NET Core Webservices, using opentelemetry-dotnet for instrumentation. Since these are dummy projects in a home lab environment, I am able to provide the full source code of the .NET solution if necessary.
Possible Cause & Fix
https://github.com/open-telemetry/opentelemetry-dotnet/issues/1372
It looks like this problem can be fixed by by using MaxPayloadSizeInBytes = 65000;
which was the default until mid september.
Is this memory consumption by jaeger-agent expected behavior if a client library misbehaves?
Or is this something the jaeger team would like to investigate?

I am observing this behavior running jaeger-all-in-one native on windows. or in a linux container on DockerDesktop WSL2, or in a Linux Hyper-V VM. At first I was using and blaming badger local storage. I then switched to elasticsearch storage.
I have no split up to running agent, collector and query containers on WSL 2 so I can pinpoint the memory usage to agent.
Agent is currently not on localhost where instrumented client application is running, but I tried this also and the issue happend too. Will try this again now that I am no longer using all in one image.
The issue does not seem to occur under very light load. I am curl'ing my services to generate spans. At first memory is stable and low. Then I started curl'ing  in 10 parallel loops increasing span creation rate.
After some minutes agents memory jumps from < 50 MB to > 2GB and then > 5 GB.
The container has currently a hard memory limit of mem_limit: 8000m.

At the current moment it sites "stable" at 4.6 GB but I have seen it go beyond 8 GB as well.
A symptom or maybe the cause for this are log errors starting to appear in agent logs. While running up to about 3 curl loops there are no log messages. A litte more requests and these start trickling in:
jaeger-agent        | {"level":"error","ts":1605909716.7062023,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: error reading list begin: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
jaeger-agent        | {"level":"error","ts":1605909751.7633333,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 2 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
jaeger-agent        | {"level":"error","ts":1605909761.783476,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: Required field TraceIdLow is not set","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
jaeger-agent        | {"level":"error","ts":1605909771.80936,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading
struct: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
jaeger-agent        | {"level":"error","ts":1605909791.8287015,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 8 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}

These 5 log errors correlate with metric
jaeger_agent_thrift_udp_t_processor_handler_errors_total{protocol="compact",model="jaeger"}

I am not sure if the instrumentation library is to blame and this is a concurrency issue there. If requests are reduced, no more log errors are happening, memory is stable (16 MB).
When increasing request load error rate increases again, and at some point memory jumps a few gigabytes:

I took 300 log lines and de-duplicated them a bit to these 50:
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 6 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: Required field TraceIdLow is not set","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Log error reading struct: Required field Timestamp is not set","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: Invalid data length","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: error reading list begin: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 8 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 9 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 8 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 11 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: Unknown data type 0","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 8 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 2 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 2 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 6 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 23 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 12 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 15 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: EOF","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 22 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 6 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.SpanRef error reading struct: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.SpanRef error reading struct: *jaeger.SpanRef field 6 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field -24 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 25 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 12 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 20 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 33 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 14 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 5 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: error reading list begin: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.SpanRef error reading struct: *jaeger.SpanRef field 5 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 10 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 13 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 12 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 13 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.SpanRef error reading struct: Required field RefType is not set","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 4 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 10 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 25 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 16 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 13 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 5 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 18 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 11 read error: don't know what type: 15","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Log error reading struct: *jaeger.Log field 9 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 19 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 25 read error: don't know what type: 13","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 22 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}

To Reproduce
Steps to reproduce the behavior:

Run ASP.NET Core 3.1 Demo Solution using opentelemetry instrumentation
Create load to generate spans
jaeger-agent memory increase by some gigabyte and agent may crash.

If the logs / other steps are not enough I should be able to provide you with a ready to run docker-compose setup, but it will take me some time.
Expected behavior
jaeger-agent should not consume this much memory with such little load.
Screenshots
See above.
Version (please complete the following information):

OS: Windows 10 2020H2 + Docker Desktop <IP_ADDRESS> on WSL 2` or natively on windows.
Jaeger version: 1.20.0 and 1.20.1
Deployment: agent on bare metal windows, or WSL2 linux container, or linux vm. Agent is currently not on localhost where instrumented client application is running, but I tried this also and the issue happend too.
opentelemetry-dotnet version 1.0.0-RC1 or 0.8.0-beta

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.
I did enable debug logging for agent and collector, files attached. For this run, agent memory did only increase up until 1.9 GB.
jaeger-agent-debug.log
jaeger-collector-debug.log
Not sure which other steps would apply. I am no go dev, so using debug images would be of no use.
Additional context

Thanks for the detailed report, @Mario-Hofstaetter! I'll try to reproduce this, but I'm not sure I'll be able to build/run a dotnet example on Linux. Would you be able to capture the payload the opentelemetry-dotnet library is sending? This would make it easier to reproduce/fix this.

This might be a duplicate of #2459.

This might be a duplicate of #2459.

@jpkrohling I guess in my case, this here was the cause and not badger. The original author of #2459 (lichenglin) did not state which instrumentation library he is using.
I have yet to retry badger local storage with the opentelemetry-dotnet fix mentioned above in first post.

I'll try to reproduce this, but I'm not sure I'll be able to build/run a dotnet example on Linux. Would you be able to capture the payload the opentelemetry-dotnet library is sending? This would make it easier to reproduce/fix this.

I'm pretty sure I can provide a small solution that will run on linux (it's .net core after all), at least a docker-compose setup.
What do you mean by capture the payload?
Wireshark / pcap?
Thanks.

I'm pretty sure I can provide a small solution that will run on linux (it's .net core after all), at least a docker-compose setup.

That would be awesome!

What do you mean by capture the payload?
Wireshark / pcap?

Yes, something like that. If you are using plain HTTP for the communication, instead of gRPC, it would be even easier as I could just use the HTTP payload with cURL.

I think we can make use of /debug/pprof/heap url at jaeger-agent's admin host-port endpoint to get the live heap and see what is taking the heap so much.

I think we can make use of /debug/pprof/heap url at jaeger-agent's admin host-port endpoint to get the live heap and see what is taking the heap so much.

Is this url available by default or does it need to be enabled?
I've not yet had time to prepare the demo solution ...

Hi @Mario-Hofstaetter
It will be very useful to have that to debug more. I am new to golang but I can give it a try. Someone will definitely be able to help.

@Mario-Hofstaetter, you can open http://localhost:14269/debug/pprof/  in your browser, and navigate over the raw data. You might also want to try this:
$ go tool pprof -web http://localhost:14269/debug/pprof/heap

This should open your web browser with the current heap as an image. You can try the same for the other profiles (see the first URL for the available profiles).

@Mario-Hofstaetter do you have any news on this? Are you able to provide with a dotnet application that runs on Linux, so that I can reproduce it?

@Mario-Hofstaetter do you have any news on this? Are you able to provide with a dotnet application that runs on Linux, so that I can reproduce it?

@jpkrohling Apologies for the long delay, will schedule this for the weekend to do.

This fairly reliably reproduces it for me:
ConsoleApp1.csproj
<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net5.0</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
      <PackageReference Include="OpenTelemetry" Version="1.0.0-rc1.1" />
      <PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="1.0.0-rc1.1" />
    </ItemGroup>

</Project>

Program.cs
using System;
using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

namespace ConsoleApp1
{
    class Program
    {
        public static ActivitySource ActivitySource = new("foo");
        static void Main()
        {
            Console.WriteLine("Hello World!");
            using var tracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddSource(ActivitySource.Name)
                .AddJaegerExporter()
                .Build();

            for (var i = 0; i < 1000; i++)
            {
                using var _ = ActivitySource.StartActivity(i.ToString());
            }

        }
    }
}

How can I run this on Linux?

How can I run this on Linux?

Just copy these two files somwhere and do dotnet run, after installing .NET 5: https://dotnet.microsoft.com/download

How can I run this on Linux?

@jpkrohling If you don't feel like installing .NET, I have pushed the sample app to https://github.com/Mario-Hofstaetter/jaegertracing-jaeger-issue-2638 including a Dockerfile and was able to reproduce on https://labs.play-with-docker.com . Thanks @mberginbh for the tiny example.
You can point the app to any agent adress/port and change the loopCount to change the load.
Steps to reproduce
$ git clone https://github.com/Mario-Hofstaetter/jaegertracing-jaeger-issue-2638.git && cd jaegertracing-jaeger-issue-2638
$ docker-compose up -d
# takes some time for elasticsearch to spin up, so collector and query-ui containers will be restarting
# c# demo app image is build with tag 'jaegerdemoclient'

Run demo app:
# Repeat a few times
$ docker-compose up jaegerdemoclient
Starting jaegerdemoclient ... done
Attaching to jaegerdemoclient
jaegerdemoclient    | Hello World!
jaegerdemoclient    | Using agenthost 'jaeger-agent:6831'
jaegerdemoclient    | Running  '500' loops
jaegerdemoclient    | Starting loop
jaegerdemoclient    | Loop done, wait 5 seconds because it seems not everything gets flushed
jaegerdemoclient exited with code 0

Memory usage increases:
$ docker stats --no-stream
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
c178db9e972d        jaeger-agent        0.00%               2.095GiB / 31.42GiB   6.67%               445kB / 103kB       0B / 0B             11

Change agent adress or loopcount
using compose:
$ docker-compose run -e "DEMOCLIENT_LOOPCOUNT=5000" -e "JAEGER_AGENTHOST=my-dev-machine" jaegerdemoclient
Hello World!
Using agenthost 'my-dev-machine:6831'
Running  '1000' loops

without compose:
docker run --rm -it -e "DEMOCLIENT_LOOPCOUNT=5000" -e "JAEGER_AGENTHOST=my-dev-machine" jaegerdemoclient

A loop of 5000 is crashing the agent for me on labs.play-with-docker.com .
To run it natively just run dotnet run in the repo (with dotnet 5 installed),
the app will default to localhost:6831 for the agent.

Thank you to both of you! I'll give it a try soon.

Quick status update: I was able to reproduce this issue, and I'm convinced it's the same as #2452. The code where this is mostly observable is here:
https://github.com/apache/thrift/blob/b75e88a33d67ae05ef9b5fa001d2a63a2effe377/lib/go/thrift/compact_protocol.go#L557-L576
I'm sure the root cause is elsewhere, and I'm still investigating, but wanted to share some findings in case someone has an idea on what's going on:

most of the time, this function runs fine, but given enough retries of the reproducing code, memory will just explode
when the memory explodes, the usage at the beginning of this function is typically quite low. One example execution: the agent was using 3MiB of memory before the buffer allocation (lines 570-574 in the code above), and 1987MiB after that.
the length (L558) was<PHONE_NUMBER> for that case
the total UDP package payload length was 4083 (https://github.com/jaegertracing/jaeger/blob/1b209472dc6a35f7de90c7a9048cb4e960cc2335/cmd/agent/app/processors/thrift_processor.go#L118). Here's where I'm confused: as far as I understand, this payload is what will then be read at the protocol code, so, the length (previous point) would have to be lower than the payload size
I might need to read more about Go's memory management, but the heap part of pprof shows a value close to the buffer size for the line 576 in compact_protocol. Does it mean that the buffer is still in memory, or will Go reclaim this space?

cc @yurishkuro, as you might be able to confirm that my expectation on item 4 is correct.

the length (L558) was<PHONE_NUMBER> for that case

Isn't this the root cause? It sounds like a malformed payload.

The payload itself was within bounds (4083). While the payload was indeed malformed, we shouldn't just explode in memory consumption when receiving bad data.

I agree, but this sounds like a bug(ish) in the Thrift library, if it just blindly allocates that much memory based on a malformed value from the payload.

Alright, I created https://issues.apache.org/jira/browse/THRIFT-5322 to track this there, but I still need to make sure it's indeed in the Thrift library.

Status update: I got a reproducer that touches far less code, so, might be easier to isolate and fix the issue:
package main

import (
	"context"
	"encoding/base64"
	"fmt"
	"runtime"

	"github.com/apache/thrift/lib/go/thrift"
	"github.com/jaegertracing/jaeger/cmd/agent/app/customtransport"
	"github.com/jaegertracing/jaeger/thrift-gen/agent"
	"github.com/jaegertracing/jaeger/thrift-gen/jaeger"
	"github.com/jaegertracing/jaeger/thrift-gen/zipkincore"
)

func main() {
	payloadHex := "[encoded data, 5452 characters]"
	payloadBin, err := base64.StdEncoding.DecodeString(payloadHex)
	if err != nil {
		fmt.Printf("error while decoding: %v\n", err)
		return
	}

	transport := &customtransport.TBufferedReadTransport{}
	protocol := thrift.NewTCompactProtocol(transport)
	_, err = transport.Write(payloadBin)
	if err != nil {
		fmt.Printf("error while processing: %v\n", err)
		return
	}

	rep := &Reporter{}

	processor := agent.NewAgentProcessor(rep)

	var before runtime.MemStats
	runtime.ReadMemStats(&before)

	_, err = processor.Process(context.Background(), protocol, protocol)
	if err != nil {
		fmt.Printf("error while processing: %v\n", err)
	}

	var after runtime.MemStats
	runtime.ReadMemStats(&after)

	fmt.Printf("memory usage before: %d MiB\n", before.Alloc/1024/1024)
	fmt.Printf("memory usage after: %d MiB\n", after.Alloc/1024/1024)

	fmt.Println("finished")
}

type Reporter struct {
}

func (r *Reporter) EmitBatch(ctx context.Context, batch *jaeger.Batch) (err error) {
	fmt.Println("sending batch to collector")
	return nil
}
func (r *Reporter) EmitZipkinBatch(ctx context.Context, spans []*zipkincore.Span) (err error) {
	return nil
}

The payload is based on what the reproducer from @mberginbh generated.

Related PR: https://github.com/apache/thrift/pull/2292

Hi there,
I see that https://github.com/apache/thrift/pull/2292 is merged. Is it necessary to wait for new apache/thrift release to catch the fix in Jaeger?

Yes, we need a numbered thrift release.

I faced with the same issue that was described in https://github.com/open-telemetry/opentelemetry-dotnet/issues/1372 on jaeger 1.22.0.
And yes, MaxPayloadSizeInBytes = 65000 in OpenTelemetry.Exporter.Jaeger settings fixes the issue.
cc: @yurishkuro / @jpkrohling

I'll try @Mario-Hofstaetter's reproducer against the released binary and see if I can reproduce it.

I got the same issue , have tried 1.21 and 1.22 versions , both  agent memory > 10G occasionally.

I currently have no spare time for detailled tests 😢
I just did a quick run on play with docker (as described in https://github.com/jaegertracing/jaeger/issues/2638#issuecomment-740962869 ),
with docker images latest and 1.22.0,
I did not see memory exploding to gigabytes, but I got it to a few hundert megabytes, and just then noticed that the agent container has already crashed a few times, maybe I overdid it with -e "DEMOCLIENT_LOOPCOUNT=50000" ...

Looks like I could get to this with the original reproducer in our master:
$ go tool pprof -top http://localhost:14271/debug/pprof/heap
Fetching profile over HTTP from http://localhost:14271/debug/pprof/heap
Saved profile in /home/jpkroehling/pprof/pprof.__debug_bin.alloc_objects.alloc_space.inuse_objects.inuse_space.057.pb.gz
File: __debug_bin
Type: inuse_space
Time: Mar 11, 2021 at 3:51pm (CET)
Showing nodes accounting for 67.52MB, 100% of 67.52MB total
      flat  flat%   sum%        cum   cum%
   67.02MB 99.26% 99.26%    67.02MB 99.26%  github.com/jaegertracing/jaeger/cmd/agent/app/servers.NewTBufferedServer.func1

$ go tool pprof -top http://localhost:14271/debug/pprof/heap
Fetching profile over HTTP from http://localhost:14271/debug/pprof/heap
Saved profile in /home/jpkroehling/pprof/pprof.__debug_bin.alloc_objects.alloc_space.inuse_objects.inuse_space.058.pb.gz
File: __debug_bin
Type: inuse_space
Time: Mar 11, 2021 at 3:51pm (CET)
Showing nodes accounting for 11333.37MB, 99.40% of 11401.76MB total
Dropped 73 nodes (cum <= 57.01MB)
      flat  flat%   sum%        cum   cum%
11265.35MB 98.80% 98.80% 11265.85MB 98.81%  github.com/jaegertracing/jaeger/thrift-gen/jaeger.(*Span).ReadField10
   67.02MB  0.59% 99.39%    67.02MB  0.59%  github.com/jaegertracing/jaeger/cmd/agent/app/servers.NewTBufferedServer.func1
    0.50MB 0.0044% 99.40% 11322.85MB 99.31%  github.com/jaegertracing/jaeger/thrift-gen/jaeger.(*Batch).ReadField2
    0.50MB 0.0044% 99.40% 11323.86MB 99.32%  github.com/jaegertracing/jaeger/thrift-gen/agent.(*AgentEmitBatchArgs).ReadField1

@fishy, does this ring a bell? I'll continue investigating but looks like THRIFT-5322 might not be fixed after all...

Note that the thrift fix does not "prevent all memory allocations". It only prevents the blindly allocation according to a malformed/malicious string header.
If you have malicious/malformed message with large string header claiming that this string has N bytes, in the old code it allocates N bytes upfront. In the fix it uses bytes.Buffer, read at most N bytes and grow the buffer as it reads if needed. You can read bytes.Buffer.Grow's doc/code to learn more about how it works, but it does involve allocating a new, larger buffer, and copy from the old smaller buffer to the new buffer, when needed.
So say you have a string header claiming to have a string with size of 100MB, and you do have 90MB left in this message, it will still read all the remaining of your message (90MB) until it fails (because it doesn't have enough content to read all 100MB), and allocate 90MB (plus the smaller ones before it hits 90MB) buffers along the way.
Another thrift change after the fix implemented TConfiguration, which allows you to change default max message size (which is 100MB if you don't override it through TConfiguration), and we do check max message size when reading string header and reject it directly (without any future read) if the string header is larger than max message size. So if your code have to deal with malicious/malformed messages a lot, you do want to use TConfiguration and set a smaller max message size.

@fishy, I agree with everything you said and I was about to argue something very similar in my previous message, but then I tried to add more pressure in the reproducer and came up with the scenario I've shown earlier: in under 10s, the memory consumption went from near zero to more than 11GB for the github.com/jaegertracing/jaeger/thrift-gen/jaeger.(*Span).ReadField10, in a path that accepts data only via UDP, meaning that the packet size is at most about 65KB. The memory consumption then rapidly went back to very low levels, but this spike in memory would be enough to get the process killed depending on the process manager (systemd, Kubernetes, ...)
Are you able to try the reproducer above, or should I attempt to come up with an example payload that triggers the problem?

Since it's udp only, can you try to use &TConfiguration{MaxMessageSize: 100*1024} (I used something larger than 65KB because it could be zipped payload), and see how's the memory usage?

Also since you mentioned ReadField10, which thrift struct is it and what's the type of the field with numeric id 10?

It's a list: https://github.com/jaegertracing/jaeger/blob/78019b6849133bdedebe35d2a28fe00cce875958/thrift-gen/jaeger/jaeger.go#L1147-L1156
I'll try your suggestion with the custom TConfiguration tomorrow. Thanks for your support!

Containers (list/set/map) are not in that fix, and TConfiguration won't help them either (but I do think you should still apply the TConfiguration fix anyways). So this is actually a different issue (but kinda related to THRIFT-5322)
In some other thrift language packages (java I believe) we do have a sanity check on container size before doing the allocations. we do not have the in go library yet. you are welcomed to create a ticket and PR to do so in go library (also via TConfiguration, ideally).
That being said, for container size check, the only thing you can do is set a max container size and reject requests with a container header larger than that. As long as the container header is smaller than the limit we still do the allocation for the whole container with the given size upfront. This can be changed in thrift compiler code, but doing that has performance implications.

@fishy not sure I understand that. Should this check be done in our business code, or should I change the generated code? I believe that it shouldn't be possible at all to allocate 11GB for a payload of under 65KB, no matter what data type is used in the payload, no?

The sanity check on container header should be done in thrift go library (github.com/apache/thrift/lib/go/thrift)
If you want to avoid allocation of the containers upfront after it passed the header sanity check, then that needs to be don in thrift compiler (in generated go code)

Thanks for the clarification! I just opened https://issues.apache.org/jira/browse/THRIFT-5369 to address this.

I'm not sure if this is the same issue, but I hope so. I see very large allocations as well but on the TraceIdHigh here:

Which is an int64:
https://github.com/jaegertracing/jaeger/blob/78019b6849133bdedebe35d2a28fe00cce875958/thrift-gen/jaeger/jaeger.go#L1060-L1070

@jpkrohling no I can't reproduce it myself. I run it as a service and somebody is probably sending something bad. But it is consistently getting oom killed. Any tips on how to track down the spans at a collector level? I can only think of building a fork with some more logging.
Or if you can point me to branch with the https://github.com/apache/thrift/pull/2359 I can build and verify that it fixes it.

@jpkrohling no I can't reproduce it myself. I run it as a service and somebody is probably sending something bad. But it is consistently getting oom killed. Any tips on how to track down the spans at a collector level? I can only think of building a fork with some more logging.
Or if you can point me to branch with the apache/thrift#2359 I can build and verify that it fixes it.

@forestsword You can run go mod edit --replace=github.com/apache/thrift=github.com/fishy/thrift@1bc6892 && go mod tidy to use the code from the PR. You do need to pass in a TConfiguration with a reasonable MaxMessageSize set to actually limit the memory usage.
The PR actually also contains compiler changes and go mod edit would only replace the library code, but the library code (with reasonable MaxMessageSize configuration) should be enough to improve the situation.

But it is consistently getting oom killed.

If we provide you with a proposed fix, would you be able to determine whether the fix works?
I should be working soon on this one and I can prepare a container image with @fishy's and my own changes.

Thanks @fishy that helped. @jpkrohling I was able to create a build with the thrift fix and had it run overnight to see if it fixed it and it most certainly did. The 1.22 version collectors oom multiple times an hour, the build hasn't been killed and runs with a stable memory usage, whereas the others would have 8-15gb spikes and be killed off. I'm happy to verify again with whatever you give me.

Wonderful news! Are you able to open a PR with the changes you had to make? You might want to leave the dependency changes aside, as we'll bump it properly once @fishy's PR is merged and available in a release.

I didn't need to change any code I only included @fishy's PR. I suppose that means it used the default of 100mb as MaxMessageSize.

Looks like it then. How's the memory usage, are you able to share some numbers?

So I made it hard for myself to get those numbers. It was previously running to get a 4th of the traffic. I've cut-over now to only use the build for all of the traffic here, which isn't much here (~300 spans per second). I had multiple containers previously to try and offset the oom kills. I'm back to 1 collector with this build with 4Gb limit.
Maybe not the most helpful shot but it gives you the ranges. Red being restarts of the pods. The rest is working set bytes. The cut-over around 13:30.

Looks nice, thanks for sharing!

Good to hear that it helped!

Wonderful news! Are you able to open a PR with the changes you had to make? You might want to leave the dependency changes aside, as we'll bump it properly once @fishy's PR is merged and available in a release.

A release would be half year away (we plan to do 0.15.0 release around September and there's no plan to do 0.14.2 release at this time, and this bug alone is unlikely to change that). You probably want to use the master branch version once that PR is merged (go mod actually makes that easy, but I also totally understand it if you only want to use tagged releases).
Also I'd still strongly suggest y'all to set a lower MaxMessageSize.

A release would be half year away (we plan to do 0.15.0 release around September and there's no plan to do 0.14.2 release at this time, and this bug alone is unlikely to change that).

We can probably use what we did before 0.14.0, in that we forked the Thrift repository and tagged v1.13.0-patch1 with your first fix. References:
https://github.com/jaegertracing/thrift
https://github.com/jaegertracing/jaeger/pull/2780

Also I'd still strongly suggest y'all to set a lower MaxMessageSize.

Will do, thanks! In theory, this could be the same as the max size of a UDP packet, right? I would set it to a couple of MiBs just to be on the safe side.

Also, since your tests shown that the library change alone is enough to fix your problems, I'm going to revert the compiler change from my PR, because that would have negative performance impacts for everyone. I can add that back later if it's proven necessary.

Sounds good!

I was also seeing these memory issues frequently with the 1.18 build of the agent. As per the discussion, updating to the 1.22 version went a long way to addressing the issue although there were still occasional spikes.
I have made the modification recommended by @fishy and deployed, and we have not seen a spike for over 48 hours.
Thanks for the info. The detail in this discussion was really informative.

I believe @rubenvp8510 is working on backporting @fishy's fix to the version of Thrift that Jaeger uses. After that, we'll propose releasing 1.23, which would then fix this issue here once and for all (hopefully) :-)


Templates not expanded for shell_command via automation

Home Assistant release (hass --version):
0.64.1
Python release (python3 --version):
Python 3.5.3
Component/platform:
shell_command called from automation
Description of problem:
Template variables are not expanded when a shell_command is called via automation. When the shell_command is called via http://raspi:8123/dev-service everything works fine.
Expected:
shell_command with template variables should also work via automation.
Problem-relevant configuration.yaml entries and steps to reproduce:

Add this into configuraion.yaml:

shell_command:
  433mhz_send: '/home/homeassistant/.homeassistant/shell_commands/433mhz_send.py -r {{ remote }} -c {{ character }} -a {{ action }}'

Add this into automations.yaml:

- alias: turn on speaker when chromecast starts playing
  trigger:
    platform: state
    entity_id: media_player.livingroom
    to: playing
  action:
    service: shell_command.433mhz_send
    data:
      remote: 3
      character: a
      action: on

Trigger the automation by setting the media_player to "playing"

Log Details (ERROR)
Error running command: `/home/homeassistant/.homeassistant/shell_commands/433mhz_send.py -r {{ remote }} -c {{ character }} -a {{ action }}`, return code: 1
NoneType: None

Additional info:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.3 (stretch)
Release:        9.3
Codename:       stretch

Have you tried it with:
    data:
      remote: '3'
      character: 'a'
      action: 'on'

Well, that works. Sorry for stealing your time. :-)
I made my script write all input parameters to a logfile (should have done this before raising this issue) and the problem was:
action: on

Was interpreted as boolean value by the yaml parser which lead to the variable being expanded to True. But my script expected a literal on.


Typo in correcting roughtail stingray?

https://github.com/mpinsky/OceanAdapt/blob/master/complete_r_script.R#L247
@JWMorley
When doing this for my version of the trawl data set if I do X[SPECIESCODE==8713050104] I get output (340 lines).
When I do X[haulid==20010106] # my haulid is the original COLLECTIONNUMBER I get 26 lines of output.
But if I do X[haulid==20010106 & SPECIESCODE==8713050104], I don't get anything.
Is this a typo of sorts?

Note that this relates to rBatt/trawlData#6

'haulid' is made from EVENTNAME, not COLLECTIONNUMBER, which I think you
know. The first and last line of code makes sense, but I'm not sure why
`X[haulid==20010106] would produce anything, because there is no EVENTNAME
with that number.....

https://github.com/mpinsky/OceanAdapt/blob/master/complete_r_script.R#L247
@JWMorley
When doing this for my version of the trawl data set if I do
X[SPECIESCODE==8713050104] I get output (340 lines).
When I do X[haulid==20010106] # my haulid is the original COLLECTIONNUMBER I get 26 lines of output.
But if I do X[haulid==20010106 & SPECIESCODE==8713050104], I don't get
anything.
Is this a typo of sorts?

Reply to this email directly or view it on GitHub:
https://github.com/mpinsky/OceanAdapt/issues/45

Actually, I didn't know. I must have messed that up. Crap.
I probably never noticed because I was only using the inner strata (are the 2 collections per event reflecting 1 inner and 1 outer per event??).
I'll be in today and ask in person. Thanks for helping me sort this out.

Given that I may have used the wrong column for haulid, this still doesn't explain why I don't get any output for that last line.

The two COLLECTIONNUMBERS are basically replicate tows within an
EVENTNAME. The catches of each COLLECTIONNUMBER are conducted
independently, but all other variables (depth, strata, temperature, gps,
etc.) are identical within an EVENTNAME.
You most likely will want to drop the outer strata of seus entirely. They
were only sampled for 10 years in spring and fall, and were of limited
latitude coverage.

Actually, I didn't know. I must have messed that up. Crap.
I probably never noticed because I was only using the inner strata (are
the 2 collections per event reflecting 1 inner and 1 outer per event??).
I'll be in today and ask in person. Thanks for helping me sort this out.

Reply to this email directly or view it on GitHub:
https://github.com/mpinsky/OceanAdapt/issues/45#issuecomment-154114002

OK, gotcha. I did drop the Outer strata for those reasons.
Is this a situation where there's a net hanging off of each side of the boat, or are the towing through the area two separate times?

They're towed at the same time, and I think the two nets are connected to
eachother (ie share a center trawl-door or something like that).

OK, gotcha. I did drop the Outer strata for those reasons.
Is this a situation where there's a net hanging off of each side of the
boat, or are the towing through the area two separate times?

Reply to this email directly or view it on GitHub:
https://github.com/mpinsky/OceanAdapt/issues/45#issuecomment-154125430


this one has been working for me and now grace

Indeed, works for me too, workflow1 now generates good images.


enum problem

I just started playing with this code a bit and I am running into an issue. I cannot refer to suits as 'c' or 's' etc.
If I try to make a combo it does not understand '2s3d' for example. It says "ValueError: s is not a valid Suit"
So it seems like the enum coding that provides the 'c' or 'clubs' equivalent references is not working for some reason.
Any idea why this is happening?

A bit more info...
The error occurs in the code within the __common.py file on line 29 with the return statement:
'return super(_PokerEnumMeta, cls).call(value)'
which causes the init.py of enum to call the following function on line 330:
cls.new(cls, value)
and finally having the error in that same file on line 642 with the following try/except code:
try:
    if value in cls._value2member_map_:
        return cls._value2member_map_[value]
except TypeError:
    # not there, now do long search -- O(n) behavior
    for member in cls._member_map_.values():
        if member.value == value:
            return member
raise ValueError("%s is not a valid %s" % (value, cls.__name__))

which raises the ValueError and outputs:
ValueError: s is not a valid Suit
...
ok, last edit...
It looks like capital letters work for suits but not lowercase. So I can successfully call Card('AS') but not Card('As')
Thanks

This is definetely a bug, see #20. I will fix it, but until then use unicode_literals future or try Card(u'As') instead of Card('As')

The codebase is now Python 3 only.


Crash bug on null number in Duckdb

The query below generate NULLs as the result of sums.   I'm getting hard exceptions in DuckDb when running this.
terminate called after throwing an instance of 'duckdb::InternalException'
  what():  INTERNAL Error: Calling GetValueInternal on a value that is NULL
Aborted

        source: state_facts is table('malloytest.state_facts') {
          measure:
            c is airport_count.sum()
            ava is c {where: popular_name = 'Ava'}

          query: basic is {
            group_by: popular_name
            aggregate: c,ava
          }

          query: nested is {
            aggregate: c
            nest: basic is {
              group_by: popular_name
              aggregate: c,ava
            }
          }
        }
        query: state_facts->nested

Here is a test case that will fail for nomodel.spec.ts
  it(`number as null- ${databaseName}`, async () => {
    // a cross join produces a Many to Many result.
    // symmetric aggregate are needed on both sides of the join
    // Check the row count and that sums on each side work properly.
    const result = await runtime
      .loadQuery(
        `
        source: s is table('malloytest.state_facts') + {
        }
        query: s-> {
          group_by: state
          nest: ugly is {
            group_by: popular_name
            aggregate: foo is NULLIF(sum(airport_count)*0,0)+1
          }
        }
      `
      )
      .run();
    expect(result.data.path(0, "ugly", 0, "foo").value).toBe(null);
  });

erros with
terminate called after throwing an instance of 'duckdb::InternalException'
  what():  INTERNAL Error: Calling GetValueInternal on a value that is NULL

The fix has landed in duckdb, it should be available later this evening so I will probably bump duckdb in the morning.

Fixed by #667


Keep the scrooll when refreshing (Preview to the side)

I would like to know if it is possible to keep the scroll when modifying the html file? I mean when I edit a html file, each time a refresh is done the scrollbar goes up.
This is a little bit annoying. I need to scroll down again to see the my last position.

Ideally it would only update the preview on save


landing page: change style

Proposed changes
Update the landing page of the documentation according to the design
Unreleased project version (optional)
https://5eb16ad8f08678ca4b0cfed3--wip-docs.netlify.app/

Thanks for taking a look @thaJeztah.


mutes all audio clips it can find. this could be a problem

since any and all kinds of audio clips are forbidden by the forum rules, this isn't an issue


Recorded video requires too much storage space

Describe the bug

I plan to record the mkv video first, store the video locally, and then input the video locally to track the human skeleton, but if I record the rgb, ir, and depth data streams, the saved mkv file is very large, which requires 4G in one minute. RAM. So I want to ask if there is any solution? I thought of a way to record only the rgb and depth data streams. Can human bone tracking only input the rgb and depth data streams, and not the ir data stream? I only input the rgb and depth data streams, the program will have errors, but because the k4abt_tracker_enqueue_capture () function is a dll format file, I ca n’t. I plan to record the mkv video first, store the video locally, and then save the video locally Input, track human bones, but if you record rgb, ir and depth data streams, the saved mkv file is very large and requires 4G of memory in one minute. So I want to ask if there is any solution? I thought of a way to record only the rgb and depth data streams. Can human bone tracking only input the rgb and depth data streams, and not the ir data stream? I only input the rgb and depth data streams, the program will have errors, but since the
k4abt_tracker_enqueue_capture () function is from a dll format file, I cannot view the function.
Expected behavior

Logs

Screenshots

Desktop (please complete the following information):

OS with Version:
SDK Version:
Firmware version:

Additional context

Body tracking only requires the depth recording. You do not need to record the rgb camera (unless you have other reasons to record the rgb camera). We are investigating options to compress depth both at rest and in transit.

I only input a video that contains the depth and rgb data streams, but does not include the ir data stream, but an error occurs. If I only enter the depth data stream without the ir data stream, it seems that it is not possible to track human bones. An error occurs, why is it wrong?

You need to input the depth and IR streams. You do not need the RGB stream.


GPU accelerated FFT

For even faster convolution, can we augment fftw with pyvkfft?
https://github.com/vincefn/pyvkfft
https://pyvkfft.readthedocs.io/en/latest/

Yes let's look into this some more.


sst.aws.Router - Request invalidation feature as an option

Invalidation right now are limited to the various JS frameworks like Next.js, Astro, Remix, etc...
It would be great if we could also have an CloudFront invalidation option for the sst.aws.Router as well.

@frank maybe we expose this?

@thdxr @fwang it has been awhile. Just checking in to see if this is a good suggestion and will be implemented down the road?

@ruchernchong In the next release, u can pass in invalidation: true or an object to customize the behavior.


[Bloc] Improve rendering performance by introducing composition layers

The compositor works now, we will open more specific issues.


[backport] Remove unnecessary tests

Backport of #4966

Jenkins CI test (for commit 4c5878908c9b1dfaf80e113ba00d3d85f73bb107) succeeded without errors!


[FEA] Further improvements to histogram-based median

A histogram-based median for 2D 8 and 16-bit integer images was implemented in #317. There are some areas where this could be further improved in the future.
improved boundary handling
The histogram-based median proposed in #317 currently requires some edge padding to handle all of the boundary conditions available. This padding prevents being able to directly write into a user-provided output array from the kernel. Ideally we would fully implement the boundary handling in the kernels to avoid the overhead of pre-padding before the kernel and cropping afterward.
extension to 3D data
It should be possible to extend this concept to 3D, but more care to stay within GPU memory limits will be required. I think one could store histograms on a per-plane rather than per-line basis in that case to reduce the number of histogram copies that need to be kept in-memory.
Use CUB for Histogram operations
Look into using CUB's BlockScan or similar for the histogram median lookups
Extend testing with 16-bit ranges beyond 10-bit depth
Locally 12-bit range (e.g. [0, 4096] as common in DICOM) works, but testing with a range of values larger than 1024 fails on remote tests. We need to investigate why this is.

Also, I don't think we currently use them anywhere in cuCIM, but this same implementation from #317 can also be used to acceleration CuPy's percentile_filter or rank_filter just by changing the median_pos variable passed to a kernel to a position that corresponds to the desired rank.


[Documentation] Tutorial?

Not sure if you ever have the time, but in the distant future perhaps you could either add
to the main README, or to another file, and explain the code as well on a semi-high
level? Just as an entry point and starter, does NOT have to be complete, just something
that gives an overview, and adds a bit more to that overview. Thank you for reading.

Hi @rubyFeedback. Maybe this helps? https://viewsourcecode.org/snaptoken/kilo/