feat(ecs): add codepipeline deploy-action to ecs cluster

This will implement #1386
No Unit-Test, one Integration Test ist present. The similar compontent s3 deploy-action has no unit-test (didnt found it), so I oriented there. Please give me feedback when required.
I have tested the integration test in my private AWS Account, and it worked.
Notes:

I did not implement toCodePipelineInvokeAction (on service?) as service and repository is required, as I didnt figured out an easy api for this. Need feedback if this is required.
The default IAM rules are maybe too open (ecs:* for *), I oriented on the default AWS CodePipeline Role: https://docs.aws.amazon.com/codepipeline/latest/userguide/how-to-custom-role.html
Not included in the component, but in the integration-test: sts:AssumeRole and iam:PassRole for * - same reason as point before. The following would be ideal, but I didnt figured out how to construct this with cdk:            - Resource: "*"
              Effect: Allow
              Action:
                - iam:PassRole
              Condition:
                StringEqualsIfExists:
                  iam:PassedToService:
                    - ec2.amazonaws.com
                    - ecs-tasks.amazonaws.com

Pull Request Checklist

[ ] Testing

Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
CLI change?: coordinate update of integration tests with team
cdk-init template change?: coordinated update of integration tests with team

[x] Docs

jsdocs: All public APIs documented
README: README and/or documentation topic updated

[x] Title and Description

Change type: title prefixed with fix, feat will appear in changelog
Title: use lower-case and doesn't end with a period
Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"

[x] Sensitive Modules (requires 2 PR approvers)

IAM Policy Document (in @aws-cdk/aws-iam)
EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

@skinny85, @SoManyHs... opinions?

The build failure seems like something transient. Can you rebase, and try again? (there are some conflicts with current master, so that would have to be done sometime anyway). Thanks!

Hej @skinny85 - Thank you very much for your code-review! :) I will fix the issues in the next days :+1:

Hey @aweiher ,
there's been a pretty big revolution in the way the Actions are represented in code in PR #2098 . This might make rebasing this PR particularly challenging. Given how close to the final shape this PR is, I'd be willing to take it over from you - I was the one doing the changes in #2098, so I'm quite familiar with what needs to be changed in this new way of doing things.
Let me know!

Hi @skinny85 - No Problem, take this PR over. I really wanted to implement this by myself, but an other task is eating all my time unfortunately. I added one commit/comment regarding the permission, hope it helps.

Thanks, will do!

I've rebased the PR on top of master. @RomainMuller and @SoManyHs , I would appreciate if you took a look.

Missed a new feature in ECS when rebasing that changes the expected file for the integ tests.


You need to equip your Personal Ventilation Unit

In a perfect world the script would make an outfit checkpoint, equip the unit while not displacing any Guzzlr gear,, complete the delivery and then restore the original outfit.
`Visit to Conspiracy Island: The Secret Government Laboratory in progress...
You need to equip your Personal Ventilation Unit.
Failed to adventure in [The Secret Government Laboratory]
Stack trace:
  at guzzlr_deliverLoop (guzzlr.ash:684)
  at guzzlr_deliver (guzzlr.ash:702)
  at guzzlr_aftercore (guzzlr.ash:719)
  at main (guzzlr.ash:770)
printStackOnAbort => false
breakableHandling => 1
choiceAdventureScript => scripts\CounterChecker.ash
promptAboutCrafting => 1`

hypothetically fixed with 8206784ff7f913f8b6c1e5fbcd53dc6c62a00839
Please let me know if the problem still occurs

I'm running a Platinum delivery every day so it should come up eventually.


Release

What is left:
### Tasks
- [ ] https://github.com/get10101/nostr-dice/issues/14
- [ ] https://github.com/get10101/nostr-dice/issues/15
- [ ] https://github.com/get10101/nostr-dice/issues/17
- [ ] https://github.com/get10101/nostr-dice/issues/13
- [ ] https://github.com/get10101/nostr-dice/issues/21
- [ ] https://github.com/get10101/nostr-dice/issues/22
- [ ] https://github.com/get10101/nostr-dice/issues/23
- [ ] https://github.com/get10101/nostr-dice/issues/16
- [x] Don't allow users to zap old notes
- [x] optional: expire multiplier notes (so that they will get deleted after the round)
- [x] optional: Check if pubkey (of zap request) has a lightning address, reject otherwise.

All done besides: https://github.com/nostrdice/nostrdice/issues/17


IncludeOptimized does not work with private set

1. Description
If an Entity class have a property with private set then IncludeOptimized does not work.
2. Exception
System.ArgumentException: Property set method not found.
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
   at Z.EntityFramework.Plus.PropertyOrFieldAccessor.SetValue(Object obj, Object value)
   at Z.EntityFramework.Plus.QueryIncludeOptimizedNullCollection.CheckNullRecursive(Object currentItem, List`1 paths, Int32 index)
   at Z.EntityFramework.Plus.QueryIncludeOptimizedNullCollection.NullCollectionToEmpty(Object item, List`1 childs)
   at Z.EntityFramework.Plus.QueryIncludeOptimizedParentQueryable`1.CreateEnumerable()
   at Z.EntityFramework.Plus.QueryIncludeOptimizedParentQueryable`1.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

3. Fiddle or Project
public class Entity
{
    public Collection<Child> Children { get; private set; }
}

public class Child{}

4. Any further technical details
I think it might be that you get the properties from the Entity Framework wrapper class type instead of the type from the actual class, since if you get it from the actual class then SetValue from PropertyInfo should work just fine for private setters.
Does not work:
using System;
using System.Reflection;
using System.Reflection.Emit;

public class BaseClass
{
    public virtual String Prop
    {
        get;
		private set;
    }
}

public class ImpClass : BaseClass
{
}

public class TypeMain
{
    public static void Main()
    {
        Type myType =(typeof(ImpClass));
	var obj = new ImpClass();
        PropertyInfo[] myPropertyInfo = myType.GetProperties(BindingFlags.Public|BindingFlags.Instance);
	myPropertyInfo[0].SetValue(obj, "test", null);
	obj.Dump();
    }
}

But this does:
using System;
using System.Reflection;
using System.Reflection.Emit;

public class BaseClass
{
    public virtual String Prop
    {
        get;
		private set;
    }
}

public class ImpClass : BaseClass
{
}

public class TypeMain
{
    public static void Main()
    {
        Type myType =(typeof(BaseClass));
        var obj = new ImpClass();
        PropertyInfo[] myPropertyInfo = myType.GetProperties(BindingFlags.Public|BindingFlags.Instance);
	myPropertyInfo[0].SetValue(obj, "test", null);
	obj.Dump();
    }
}

Hello @SpaceOgre ,
Do you think you could create a Fiddle with the issue?
My developer tried to reproduce it, but he tells me everything is working: https://dotnetfiddle.net/2ab36j
Best Regards,
Jon

Sponsorship
Help us improve this library
Performance Libraries
context.BulkInsert(list, options => options.BatchSize = 1000);
Entity Framework Extensions • Dapper Plus
Runtime Evaluation
Eval.Execute("x + y", new {x = 1, y = 2}); // return 3
C# Eval Function

@JonathanMagnan will see if I can manage that.
But now i see that I totally forgot to mention that I saw the problem on Net Framework 4.8, sorry about that.

Hello @SpaceOgre ,
Look like my developer cannot reproduce it either with EF6: https://dotnetfiddle.net/dsnsPF
Could you check what he did wrong on the Fiddle and provide one with the issue?
Best Regards,
Jon

@JonathanMagnan I finally tracked down the problem...
My first report was not correct, the problem arises when include a sub-child in the child include with at select:
context.Set<Parent>().IncludedOptimized(x => x.Children.Select(y => y.Toy))

See this fiddle: https://dotnetfiddle.net/5q2VuO

Hello @SpaceOgre ,
Unfortunately, we will have to abandon this one.
While we can successfully reproduce this issue, we have not found any easy way to fix it.
The proxy type created by Entity Framework makes handling this scenario harder.
So at this moment, we will treat this scenario as unsupported.
Best Regards,
Jon

Are you finding this library useful? If so, please consider supporting its continued development by becoming a sponsor.
Additionally, if you think your enterprise would benefit from this library, we encourage you to suggest they become a sponsor as well. Your support will help ensure this library remains alive and well-supported.

@JonathanMagnan no problem, totalt get that it might be hard to fix. And the use of private setter in these cases is probably pretty rare. An easy fix is to use protected instead of private and you get more or less the same protection as private.


[Question] Alter table drop partition on cluster errors

Hello.
We have a cluster of 6 nodes (3 shards). Replication configs include login/pass authorization.
......
        <shard>
                <replica>
                    <host>serever1</host>
                    <port>9000</port>
                    <user>user1</user>
                    <password>pass1</password>
                </replica>
                <replica>
                    <host>serever11</host>
                    <port>9001</port>
                    <user>user1</user>
                    <password>pass1</password>
                </replica>
            </shard>
........

For insert, select, create queries - all works fine
But when we run a query like:
ALTER TABLE table1 on cluster cluster1 DROP PARTITION '2018-10-01';
sometimes we got such exception:
┌─host───────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ server21 │ 9002 │      0 │       │                   5 │                2 │
│ server3  │ 9000 │      0 │       │                   4 │                2 │
│ server31 │ 9003 │      0 │       │                   3 │                2 │
│ server2  │ 9000 │      0 │       │                   2 │                2 │
└────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘
┌─host───────┬─port─┬─status─┬─error──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─num_hosts_remaining─┬─num_hosts_active─┐
│ server11 │ 9001 │    192 │ Code: 192, e.displayText() = DB::Exception: Received from server1:9000, <IP_ADDRESS>. DB::Exception: Unknown user default., e.what() = DB::Exception │                   1 │                1 │
└────────────┴──────┴────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────┴──────────────────┘
┌─host──────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ server1 │ 9000 │      0 │       │                   0 │                0 │
└───────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

The main word is sometimes.
We can run such a query several times and get the same exception, but at some moment a query may works as it should.
Maybe someone has already encountered this?
I would appreciate any tips.

It looks like you trying to make operation via user 'default.' not 'default'. There's a strange dot at the end of name. Look at your users.xml files on all the nodes. Is there any <default.> instead of ?
Other place where you may look at is connection messages, such as:
<Debug> TCPHandler: Connected ClickHouse client version 18.14.0, revision: 54409, user: default.

Is there only one dot in all such messages?

Hi.
There was no user "default." or "default" in configs. At the same time, everything was fine in the logs too.
As a fix I just created user "default" on all nodes in the cluster with strict network limitation (only for cluster nodes). And It seems helps us.

I run a 20 node replicated cluster (10 shards * 2 replicas). Cluster is configured with internal_replication and using user "distributed" for inter-cluster communication.
I have encountered this same issue both using distributed queries (ON CLUSTER) and local (single Replica Pair).
Either ALTER TABLE "foo"."bar_local" ON CLUSTER "my-cluster" DROP PARTITION '201901' or ALTER TABLE "foo"."bar_local" DROP PARTITION '201901' occationally throw the following error:
Code: 195. DB::Exception: Received from localhost:9000, <IP_ADDRESS>. DB::Exception: Received from <replica_host_name>:9000, <replica_host_ip>. DB::Exception: User default is not allowed to connect from address ::ffff:<local_host_external_ip>.
The user "default" is indeed restricted not work for this IP address, but the default user should not be used anywhere here. I run the command from clickhouse client as the user "local" and cluster is configured to use the user "distributed" internally. This configuration is validated on both hosts.
Running the same command from the replica host there is no exception indicating that this is not a bi-directional relationship.

It's looks like a duplicate of https://github.com/yandex/ClickHouse/issues/1861
It was fixed in 19.1


ALE not picking up .eslintrc

My ale isn't picking up my eslintrc https://github.com/instructure/canvas-lms if you were to run yarn and open up https://github.com/instructure/canvas-lms/blob/master/app/jsx/permissions/actions.js you can see that it doesn't pick up the local eslintrc.
v8.0.459
Operating System: 10.13.5
:ALEInfo
Current Filetype:
Available Linters: []
Enabled Linters: []
Linter Variables:
Global Variables:
let g:ale_cache_executable_check_failures = v:null
let g:ale_change_sign_column_color = v:null
let g:ale_command_wrapper = v:null
let g:ale_completion_delay = v:null
let g:ale_completion_enabled = 0
let g:ale_completion_max_suggestions = v:null
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '%code: %%s'
let g:ale_echo_msg_info_str = 'Info'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 0
let g:ale_fixers = {'javascript': ['eslint', 'remove_trailing_lines']}
let g:ale_history_enabled = 1
let g:ale_history_log_output = 1
let g:ale_keep_list_window_open = v:null
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_filetype_changed = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 1
let g:ale_lint_on_insert_leave = 0
let g:ale_linter_aliases = {}
let g:ale_linters = {}
let g:ale_linters_explicit = 0
let g:ale_list_window_size = v:null
let g:ale_list_vertical = v:null
let g:ale_loclist_msg_format = v:null
let g:ale_max_buffer_history_size = v:null
let g:ale_max_signs = v:null
let g:ale_maximum_file_size = v:null
let g:ale_open_list = v:null
let g:ale_pattern_options = {}
let g:ale_pattern_options_enabled = 0
let g:ale_set_balloons = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = v:null
let g:ale_sign_error = v:null
let g:ale_sign_info = v:null
let g:ale_sign_offset = v:null
let g:ale_sign_style_error = v:null
let g:ale_sign_style_warning = v:null
let g:ale_sign_warning = v:null
let g:ale_statusline_format = ['%d error(s)', '%d warning(s)', 'OK']
let g:ale_type_map = v:null
let g:ale_use_global_executables = v:null
let g:ale_warn_about_trailing_blank_lines = 1
let g:ale_warn_about_trailing_whitespace = 1

So I have an update on this issue.  It looks like jshint is conflicting with jslint or both are running at the same time.  I have worked around this by setting
:let b:ale_fixers = {'javascript': ['eslint']}
:let b:ale_linters = {'javascript': ['eslint']}

in my vimrc but there are some js files I still have to do that.

Was your problem that jshint was reporting some problems, so you had to select only ESLint? ALE will run everything it can find. I recommend setting the following in your ftplugin file if you only want to run ESLint.
" In ~/.vim/ftplugin/javascript.vim

let b:ale_linters = ['eslint']

The problem was that jshint was taking precedence over eslint.  I don't have jshint configured in my repro at all which is why it was weird it was trying to use jshint over eslint. I ended up just removing the jshint file from ale itself because the above lets stopped working in my vimrc :S
will your vim also be true for fixing? Like will this also apply to fixing? I also use vim-javascript so would I add that to my path vim-javascript/after/ftplugin/javascript.vim

You can add let b:ale_fixers = ['eslint'] too if you want to fix files with only ESLint. No fixers are enabled by default. It sounds like you just need to configure ALE so it doesn't run JSHint, so I'll close this.


How To Add/Customize Titles and Labels

Filters and search bar don't have label tags, which is a requirement for accessibility. Pagination buttons have titles numbered "1" to "x" but aren't customizable, which results in title text being a duplicate of the element text - also not accessible. Is there a way in the current package to customize any of these elements by adding labels to form inputs or modifying title tags? If so, how could this be accomplished?

Filters and search bar don't have label tags,

Can you explain it more and give it an example, I can implement it, thanks

@AllenFang In order for a form control (in this case a text input or select dropdown) to be accessible, it must have an associated label tag which describes its purpose to users of assistive technology. You can hide the label visually using the Bootstrap "sr-only" (screen reader only) tag, which will make it available for those who need it without altering the UI.
Here's what I'm using for my custom search bar to make it 508-compliant:
<label htmlFor={`table-search-bar-${tableId}`} className="w-100">
        Search This Data Table
        <input
          id={`table-search-bar-${tableId}`}
          className="form-control form-control-sm"
          ref={n => input = n}
          type="text"
          placeholder="Search"
          onChange={handleChange}
        />

Notes on the above: I'm passing a 'tableId' prop because I have multiple tables per page, and each 'htmlFor' - 'id' pair must be unique. The text inside the span tag is the actual label describing the form control's purpose, hidden from view to normal users with the 'sr-only' bootstrap class.
A custom select component (column filter dropdown), if I could build it, would look something like this:
<label htmlFor={`filter-column-${column.title}`}>
          {column.title}
          <select id={`filter-column-${column.title}`} onChange={handleChange}>
            {column.options.map(val => (
              <option value={val.title}>{val.title}</option>
            ))}
          </select>

...with column.title, column.options, and val.title being pseudo-code for their respective properties.
Another area of react-bootstrap-table-2 that could use a simple fix would be in the pagination buttons, which render with a default title attribute equal to the page, ie., for the button with text "1" it renders title="1". Any title tags used should differ from the text content of the element. So for example if these were changed to title="page-1", title="page-2" etc. this would meet the requirements for accessibility (of course, the title attribute could be removed entirely as well, as the main requirement is that IF title tags are used they must not be identical or redundant to the element text or ALT text).
Let me know if I can clarify anything further. Thanks again for your attention to this!

@Jayboy75 thanks your explanation, that is clear for me and I enhance this when I have time, sorry I'm busy recently, so PR is also welcome 👍

@AllenFang sure thing, pull request submitted - https://github.com/react-bootstrap-table/react-bootstrap-table2/pull/890

RELEASE NOTE
Please update related packages listed in above notes, thanks

@AllenFang @Jayboy75 I recently upgrade my react-bootstrap-table2-paginator module to 2.0.6, but I am still getting this accessibility warning - IDs used in ARIA and labels must be unique. Do you know if your PR addresses this? Perhaps I upgraded incorrectly.

I am getting this warning on the pagination size dropdown button, and I am just using the standard paginationFactory

@jmmadsen My PR didn't alter the pagination buttons specifically - I just built out my own custom pagination buttons, which you can do using examples in the docs.

That makes sense, thank you.


Attempting to validate an Inline oneOf union with a discriminator raises key not found: "$ref" error

Given a specification like
openapi = {
  'openapi' => '3.0.1',
  'components' => {
    'schemas' => {
      'Pets' => {
        'oneOf' => [
          {
            'type' => 'object',
            'properties' => {
              'id' => {
                'type' => 'integer',
              },
              'petType' => {
                'type' => 'string',
                'enum' => ['cat'],
              },
              'meow' => {
                'type' => 'string',
              },
            },
          },
          {
            'type' => 'object',
            'properties' => {
              'id' => {
                'type' => 'integer',
              },
              'petType' => {
                'type' => 'string',
                'enum' => ['dog'],
              },
              'bark' => {
                'type' => 'string',
              },
            },
          },
        ],
        'discriminator' => {
          'propertyName' => 'petType',
        },
      },
    },
  },
}

and then attempting to validate an input (which can either be valid or invalid)
schemer = JSONSchemer.openapi(openapi).schema('Pet')
schemer.valid_schema? # returns true
schemer.valid?({ "id": 1, "petType": "cat", "meow": "Prrr" }) # Error

An error is thrown: key not found: "$ref".
If I omit the discriminator property from Pet, everything works as expected. So inline unions work in the absence of discriminator, but do not work when provided.
Poked around source a bit, and from what I can gather, the presence of the discriminator property tells json_schemer to validate unions differently, where it will only look for the $refs when finding the subschemas.
I don't know if this is the expected behaviour or not; not sure if Openapi specifies whether or not $refs must be used when defining tagged unions. Looking at the spec docs, I see:

oneOf - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.

Which seems to suggest that they can either be inline or references, as long as they are a valid schema object.

The OAS says : "When using the discriminator, inline schemas will not be considered."
This means that the example (without the $refs), should not work.
I have also asked in the OAS Slack channel about this and it was confirmed that it should not work with inline-schemas:

oneOf entries need to be $refs to entries in the Components Object, because discriminator locates the schemas based on component names.  If you use mapping with all explicit URIs, then you can technically use inline schemas, although that's not very clear from the text and tools may or may not handle it correctly.

@moberegger discriminator is meant to pick a schema based on its name (in #/components) or by full URI. For simplicity, I decided not to consider inline schema URIs (spec: "When using the discriminator, inline schemas will not be considered."), but in your example it wouldn't make a difference. From what I can tell, your example doesn't need discriminator at all, because only one of your oneOf schemas will match based on petType. Is there another reason you're trying to use discriminator here?

This means that the example (without the $refs), should not work. I have also asked in the OAS Slack channel about this and it was confirmed that it should not work with inline-schemas:

oneOf entries need to be $refs to entries in the Components Object, because discriminator locates the schemas based on component names.  If you use mapping with all explicit URIs, then you can technically use inline schemas, although that's not very clear from the text and tools may or may not handle it correctly.

Thanks for looking into this, @ahx! I did consider looking through inline schemas—unfortunately I can't remember why I avoided it.


Deleted the Pass file, replacing it with [[nodiscard]]

C++ changes look OK but also all project files will need to have the deleted header removed.


Bug fix when the price is a whole number

When the price is a whole number, the number_format() function errors out with a notice that the 1st argument is a string, but a float is expected.

this was fixed in a recent commit: https://github.com/VividWeb/vivid_store/commit/c7d19a3549ebd412e6aa6e2e32cb8e96c4fa2867


[Vision] add test cases for flip, normalize, to_tensor

https://github.com/apache/incubator-mxnet/issues/8556

I'll add tests for brightness/contrast... once this pr is approved

Hello @javelinjs, please rebase your PR.

@marcoabreu You mean rebase from master? Is there anything related to these operators?


Fixed GPUParticles2D preprocessing bug

When awaiting on timer before switching to a scene containing the particles, GPUParticles2D is first preprocessed with an initial position, and then updated to the set position on the next frame. Details of the fix in my comment on the bug page.
Fixes #75723

Out of curiosity, I checked whether this would solve the seemingly related 3D equivalent of this bug #71935, but it doesn't seem to.


Fixing TraceContext NPE issue

Description
When query timed out it's possible that one thread tries to register the trace after the request is already unregistered. So need to check null before putting the trace.
Sample stacktrace:
2022/02/04 00:01:06.604 ERROR [ServerQueryExecutorV1Impl] [pqr-7] Exception processing requestId 1193
java.lang.RuntimeException: Caught exception while running CombinePlanNode.
        at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:164)
        at org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:41)
        at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)
        at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:290)
        at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:192)
        at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)
        at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154)
        at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
        at shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
        at shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:?]
5:10
        at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:154)
        ... 15 more
Caused by: java.lang.NullPointerException
        at org.apache.pinot.core.util.trace.TraceContext.registerThreadToRequest(TraceContext.java:140)
        at org.apache.pinot.core.util.trace.TraceCallable.call(TraceCallable.java:41)
        ... 8 more

Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

Codecov Report

Merging #8126 (9c7ab22) into master (3c98a44) will decrease coverage by 57.12%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #8126       +/-   ##
=============================================
- Coverage     71.35%   14.23%   -57.13%
+ Complexity     4303       81     -4222
=============================================
  Files          1617     1579       -38
  Lines         83874    82261     -1613
  Branches      12537    12393      -144
=============================================
- Hits          59851    11706    -48145
- Misses        19943    69676    +49733
+ Partials       4080      879     -3201

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
?

unittests2
14.23% <0.00%> (+0.08%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c98a44...9c7ab22. Read the comment docs.


Probe whether the kernel has ODP vxlan support

It's possible to build that kernel with openvswitch, and vxlan, but
without the vport_vxlan module that glues them together.  And at
least one distro has actually done this.
Addresses #1599.

This got merged into master, but the issue it is meant to be resolving has 1.2.1 milestone.

Hmm. But the issue indicates that something did get merged into 1.2.1.
@awh please explain what happened here.

But the issue indicates that something did get merged into 1.2.

Well, the commit message says that, but it's actually a merge into master. Except it also show up on the 1.2 branch.
What happened?

What happened?

I merged it into 1.2 on the command line, then 1.2->master.


fix: capture argv[0] earlier

Because CommandLine.arguments is a mutable var it's better to capture it at startup, just in case it gets changed before the signal handler runs.

Closing in favour of #10


⚠ PrivateHD has degraded performance

In 02ce356, PrivateHD ($PHD) experienced degraded performance:

HTTP code: 200
Response time: 2201 ms

Resolved: PrivateHD performance has improved in bfda993.


Copy-SqlServerAgent fails to copy over SSIS jobs - Proxy Accounts

Client environment tried to copy over SQL Agent and gives warning that jobs are being skipped because the proxy account does not exist....when it actually copied it over correctly. All jobs tied to that proxy though are being skipped.

Just ran Copy-SqlJob and it copies them without issues or warnings...None of the jobs were skipped.

Adding one more note, once client tried to run the SSIS jobs I had to reapply the password on the credential tied to the proxy account. Will have to see if I can recreate this issue in my domain lab and will update.

@wsmelton : any update on this ?

Closing this one out until I can reproduce it.


Add detailed deployment instructions

Make it clear to users how to deploy the application to AWS with detailed setup instructions assuming the user is starting from scratch.

tracked internally, closing ticket


Slow on Editable table

I find that with 1000 users in the editable table, it has good response times.
However, when I add my own data, the response after each key stroke takes about 1 second, often missing keys.
I have tried reducing the number of members of the User model, but still get slow responses.  I am sure it is something with my code, but the I am not seeing it.  What have you seen that kills response times on the editable table?
thank you.

I honestly don't know. I don't think there is any relationship between the editable feature and how many items there are in the database. I have a database with 10,000 users and I can edit any of them without any noticeable delay. Besides, no requests are sent to the server while editing a field, so the size of the database doesn't really matter at all.

Thank you.


bugfix: 非直连 Windows 机器安装 Agent, gsecmdline 报错  (closed #664)

Codecov Report

:exclamation: No coverage uploaded for pull request base (V2.2.15-rc@fd40d5d). Click here to learn what that means.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             V2.2.15-rc     #773   +/-   ##
=============================================
  Coverage              ?   71.03%
=============================================
  Files                 ?      294
  Lines                 ?    21553
  Branches              ?        0
=============================================
  Hits                  ?    15310
  Misses                ?     6243
  Partials              ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd40d5d...f20f7dc. Read the comment docs.

代码没有问题，辛苦验证存量和新增 Proxy 是否正常下发或同步 ntrights.exe

之前已经验证


Check tutorial about Kafka, Flink and CrateDB with the vanilla PostgreSQL JDBC Driver

Hi there,
because @proddata just asked about the state of the CrateDB JDBC Driver, I would like to put down that note here.
When refreshing the resources [1,2] the other day, building upon [3] by @kovrus, @carlotas19 converged the resources into [4] (cheers!). While supporting that, I also created some other accompanying resources at [5], which add some infrastructure and documentation to run the example in a reproducible manner out of the box.
My memories about the details are a bit faded, but I remember that the example did not work with the vanilla PostgreSQL JDBC Driver. Apparently, I already took a little note about it at the place where you would be able to switch the driver ^driver-switch-gradle, alongside ^driver-switch-code:

Currently, org.postgresql:postgresql croaks with
org.postgresql.util.PSQLException: No hstore extension installed.

We should get back to this and use latest software versions of the corresponding components when testing again, this time specifically focused on shedding some more light onto the problem discovered here.
With kind regards,
Andreas.
/cc @hammerhead
[1] https://www.ververica.com/blog/smart-systems-iot-use-case-open-source-kafka-flink-cratedb
[2] https://crate.io/resources/white-papers/lp-wp-flink-kafka-cratedb
[3] https://github.com/crate/cratedb-flink-jobs
[4] https://dev.to/crate/build-a-data-ingestion-pipeline-using-kafka-flink-and-cratedb-1h5o
[5] https://github.com/crate/cratedb-examples/tree/main/spikes/kafka-flink

Temporarily fixed by: https://github.com/crate/cratedb-examples/pull/26


Involve the community in bug-hunting

Since most of our users will be tech-savy people we try to get them involved in bug-hunting.
I sugest that we send as many details from the back-end about the error as possible and dispaly those details in  the browser console along with an encouragement to take a look at our code, file an issue and try to fix the error if it was a bug.
This can be a nice way to interact with the community. Thoughts?

Sounds good. The fact that Coduno is open source also lifts restrictions on "leaking" stack traces.
I just pushed this changeset which pulls util.Trace to api/controllers as tracable(err) err and puts more effort in dealing with errors.


Fix PR 7131 bugs and vulnerabilities

Description
This PR fixes the bugs and vulnerabilities found by SonarCloud in PR https://github.com/apache/cloudstack/pull/7131.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[X] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5426

@blueorangutan test

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-6018)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42155 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6018-kvm-centos7.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

@rohityadavcloud , I am satisfied with just smoketests for this, would you agree?
@JoaoJandre , will you apply/answer my suggestions?

@JoaoJandre, could you look at the code smells as well?

SonarCloud Quality Gate failed.
  0 Bugs   0 Vulnerabilities   0 Security Hotspots   19 Code Smells
 0.0% Coverage  0.0% Duplication

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5453

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-6040)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43196 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6040-kvm-centos7.zip
Smoke tests completed. 102 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_01_add_primary_storage_disabled_host
Error
0.68
test_primary_storage.py

test_01_primary_storage_nfs
Error
0.11
test_primary_storage.py

ContextSuite context=TestStorageTags>:setup
Error
0.21
test_primary_storage.py

test_01_non_strict_host_anti_affinity
Failure
112.86
test_nonstrict_affinity_group.py

test_02_non_strict_host_affinity
Error
82.22
test_nonstrict_affinity_group.py

test_03_deploy_and_scale_kubernetes_cluster
Failure
34.19
test_kubernetes_clusters.py

test_07_deploy_kubernetes_ha_cluster
Failure
69.75
test_kubernetes_clusters.py

test_08_upgrade_kubernetes_ha_cluster
Failure
44.08
test_kubernetes_clusters.py

test_09_delete_kubernetes_ha_cluster
Failure
42.11
test_kubernetes_clusters.py

ContextSuite context=TestKubernetesCluster>:teardown
Error
131.14
test_kubernetes_clusters.py

test_01_secure_vm_migration
Error
158.20
test_vm_life_cycle.py

test_02_unsecure_vm_migration
Error
262.84
test_vm_life_cycle.py

test_03_secured_to_nonsecured_vm_migration
Error
140.72
test_vm_life_cycle.py

test_08_migrate_vm
Error
43.75
test_vm_life_cycle.py

test_hostha_enable_ha_when_host_in_maintenance
Error
303.86
test_hostha_kvm.py

@blueorangutan test

Trillian Build Failed (tid-6080)

@JoaoJandre, could you look at the code smells as well?

SonarCloud Quality Gate failed.
  0 Bugs   0 Vulnerabilities   0 Security Hotspots   19 Code Smells
 0.0% Coverage  0.0% Duplication

@GutoVeronezi done :)

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5511

@blueorangutan test matrix

@DaanHoogland a Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

Trillian test result (tid-6114)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42475 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6114-kvm-centos7.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

Trillian test result (tid-6112)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43063 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6112-xenserver-71.zip
Smoke tests completed. 106 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_02_cancel_host_maintenace_with_migration_jobs
Error
1762.58
test_host_maintenance.py

test_02_cancel_host_maintenace_with_migration_jobs
Error
1762.62
test_host_maintenance.py

Trillian test result (tid-6113)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 44338 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6113-vmware-67u3.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

I am hesistent to merge this before 4.18 as it introduces no functionality or fixes any reported use case from failing.
@rohityadavcloud @weizhouapache @GutoVeronezi , can you give your opinions on how serious the fixes in here are?

I am hesistent to merge this before 4.18 as it introduces no functionality or fixes any reported use case from failing. @rohityadavcloud @weizhouapache @GutoVeronezi , can you give your opinions on how serious the fixes in here are?

@DaanHoogland, as it is just a refactoring (to fix the bugs and vulnerabilities reported by the SonarCloud), I see no problem in adding this one to 4.18. We just have to guarantee that nothing broke with the refactoring; by the results of the tests and the code analysis, the changes seems OK; manual tests are welcome, though.

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5561

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-6165)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40903 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6165-kvm-centos7.zip
Smoke tests completed. 106 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_02_deploy_vm_with_extraconfig_kvm
Error
0.22
test_deploy_vm_extra_config_data.py

test_03_update_vm_with_extraconfig_kvm
Error
0.18
test_deploy_vm_extra_config_data.py

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5743

@blueorangutan test matrix

Trillian test result (tid-6293)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 41933 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6293-vmware-67u3.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

Trillian test result (tid-6292)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50058 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6292-xenserver-71.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_02_upgrade_kubernetes_cluster
Error
3978.13
test_kubernetes_clusters.py

test_03_deploy_and_scale_kubernetes_cluster
Failure
285.50
test_kubernetes_clusters.py

test_07_deploy_kubernetes_ha_cluster
Failure
62.40
test_kubernetes_clusters.py

test_08_upgrade_kubernetes_ha_cluster
Failure
0.05
test_kubernetes_clusters.py

test_09_delete_kubernetes_ha_cluster
Failure
0.04
test_kubernetes_clusters.py

ContextSuite context=TestKubernetesCluster>:teardown
Error
225.42
test_kubernetes_clusters.py

Trillian test result (tid-6294)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50144 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7140-t6294-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_02_upgrade_kubernetes_cluster
Error
3969.44
test_kubernetes_clusters.py

test_08_upgrade_kubernetes_ha_cluster
Error
4054.17
test_kubernetes_clusters.py

test_09_delete_kubernetes_ha_cluster
Failure
0.04
test_kubernetes_clusters.py

ContextSuite context=TestKubernetesCluster>:teardown
Error
114.51
test_kubernetes_clusters.py

This looks good to me, the kubernetes errors seem to be bugs in the tests assuming a state is returned when it is not in the assertions. If not that these are environmental.
cc @shwstppr @davidjumani

@DaanHoogland this was merged into 4.18 instead of main, is this correct?


Fix bug with #111

Looks good. Why does this fix the issue?

Looks good. Why does this fix the issue?

Before, only the timestamp of the current day was retrieved. So the difference was always 0, now I use the correct timestamp function.


update the spotify green

sampled from the press logos provided at https://press.spotify.com/us/images.  sadly no svg update available yet

Thanks for the update!


Rewrite testAcquireLockShutout to use Countdown Latch

Should hopefully fix the failing unit test

was waiting for jenkins to finish testing that....


GPU memory leak while running sweeps

wandb --version && python --version && uname
Weights and Biases version: 0.9.7
Python version: 3.7.9
Operating System: Ubuntu 18.04LTS
Description
I'm running sweeps, and I notice that every so often one of the GPUs doesn't reclaim all its memory after a training job goes away.  It ends up in this horrible CUDA-bug state where nvidia-smi reports that the memory is used in the top half, but in the bottom half doesn't report any processes that owns that memory.  I can only reclaim the memory by rebooting the machine.  (I've read that sometimes nvidia-smi -r will fix this, but it's never let me reset the GPU that way I think because X-windows is running on it.)
What I Did
This is not a great bug report, because I don't know how to repro it.  I'm not even sure it's anything to do with wandb, or just some bug between CUDA & pytorch or something.  But I've seen it three or four times now, and only when running wandb sweeps.  I've mostly been using hyperband early termination with my sweeps.  And I sometimes will kill jobs manually from the wandb web UI.  So I suspect it's maybe got something to do with the way the agent kills the python process that's using the GPU - maybe it's not cleaning up properly.

Thanks for reporting! We're looking into this.

hello, i meet the same error while using sweep module. The program can not free the GPU memory by itself, i have to clean up the GPU memory after that.  If I press ctrl+c twice, then the program is killed with leaked GPU memory. Any solutions on this issue? Thanks a lot.

hey @zoeyuchao , is this happening to you in an older version wandb, or the latest one?

hey @zoeyuchao , is this happening to you in an older version wandb, or the latest one?

I tried the older versions and the latest one, still get the same issue.

I see, thanks for following up. We're looking into fixing this issue.

I am also having a similar problem. I tried it with older versions and now with 0.10.10 and still the same problem that the GPU runs out of memory on the 3rd or 4th run of the sweep

I'm also having this issue. Roughly 10% of memory (1 gig) isn't being released between runs. I'm using version 0.10.12 on ubuntu 20.04 with cuda 11.0. you can clearly see the leak in successive runs in the attached screenshot.

Hey @MLBurnham thanks for the update
Could you share a simple working script with us so that we can reproduce this issue?
Thanks is advance :smile:

I have same problem. It happens after training and validation before new epoch. I don't log gradients and parameters, only metrics. Memory doesn't leak on all gpu. See screenshot(1 epoch =~24.5hours). Second time it was on 4th epoch and I have out of memory.

Using version 0.10.10.

Hey @reymondzzzz I'm assuming this is within the context of a sweep?  If you're not logging gradients I can't think of any wandb feature that would interact with GPU memory.  In the context of a sweep it could be possible that multiple processes are trying to use the same GPU if not configured properly.  We would need more information about the environment and the exact behavior to understand what the root cause could be.

Hey @MLBurnham thanks for the update
Could you share a simple working script with us so that we can reproduce this issue?
Thanks is advance

Sorry for the slow reply. Here's the sweep script I ran. The memory leak only occurs when I run the sweep. When running the sweep I can run ~4 iterations of the model before I run out of memory. When doing a manual grid search with the out of the box simple transformers library, I can run dozens of iterations of the model back to back.
config
sweep_config = {
"method": "bayes",  # grid, random
"metric": {"name": "accuracy", "goal": "maximize"},
"parameters": {
"num_train_epochs": {"min": 1, "max": 5},
"learning_rate": {"min": 0, "max": 0.00010000
},
},
"early_terminate": {"type": "hyperband", "min_iter":6,},
}
sweep_id = wandb.sweep(sweep_config, project="Political Tweet Classifier")
logging
logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)
data import
df = pd.read_csv("Tweets/oct_sample.csv")
df = df[['text', 'political']].dropna()
train test split
train = df.sample(frac = 0.8, random_state = 907)
test = df.drop(train.index).reset_index(drop=True)
train = train.reset_index(drop = True)
train_df = train[['text', 'political']]
eval_df = test[['text', 'political']]
convert strings to lower
train_df['text'] = train_df['text'].str.lower()
eval_df['text'] = eval_df['text'].str.lower()
args
model_args = ClassificationArgs()
model_args.evaluate_during_training = True
model_args.evaluate_during_training_silent = False
model_args.evaluate_during_training_steps = 100
model_args.learning_rate = 4e-4
model_args.manual_seed = 907
model_args.max_seq_length = 256
model_args.no_cache = True
model_args.no_save = True
model_args.num_train_epochs = 10
model_args.overwrite_output_dir = True
model_args.reprocess_input_data = True
model_args.train_batch_size = 16
model_args.eval_batch_size = 16
model_args.train_custom_parameters_only = False
model_args.wandb_project = "Political Tweet Classifier"
training function
def train():
# Initialize a new wandb run
wandb.init(resume = True)
# Create a TransformerModel
model = ClassificationModel(
    "electra",
    "google/electra-base-discriminator",
    use_cuda=True,
    args=model_args,
    sweep_config=wandb.config,
)

# Train the model
model.train_model(
    train_df,
    eval_df=eval_df,
    accuracy=lambda truth, predictions: accuracy_score(
        truth, [round(p) for p in predictions]
    ),
)

# Sync wandb
wandb.join()

train
wandb.agent('ab5cvrjn', train)

Ahh, my guess is something in the TransformerModel is holding onto gpu memory between iterations.  If you can refactor the code to use process based executions that should address the issue.  Assuming you can write the above code to a file named train.py, something like the following should work:
# Add the following to your sweep config
# program: "train.py"

# Add the following to the end of your script
if __name__ == "__main__":
    train()

Then instead of calling wandb.agent(...) inside of python, call it from the command line: wandb agent SWEEP_ID.  This will launch each trial in a separate process which should guarantee you don't hold onto any memory.

Hi @vanpelt is there another solution besides running script from command line? Using google colab for GPU access which uses notebooks, which as I understand can cause some issues with wandb sweeps.

Hi @vanpelt is there another solution besides running script from command line? Using google colab for GPU access which uses notebooks, which as I understand can cause some issues with wandb sweeps.

Hey @nhsmith85 we'll look into the root cause for in process sweeps, but you should be able to run a script from colab pretty easily.  You can use the the %%writefile train.py.  Yo write a file, and then call !wandb agent SWEEP_ID from a different cell.

Hey @nhsmith85 we'll look into the root cause for in process sweeps, but you should be able to run a script from colab pretty easily.  You can use the the %%writefile train.py.  Yo write a file, and then call !wandb agent SWEEP_ID from a different cell.

I'm having the same issue https://github.com/huggingface/transformers/issues/10885

@jwa018 With huggingface, you could try: os.environ['WANDB_WATCH'] = 'false'
Otherwise if you're doing a parameter search, you should check out the sweeps documentation as you can structure your code to run as independent processes (not sure if it's what you do already).

I'm running sweeps, and I notice that every so often one of the GPUs doesn't reclaim all its memory after a training job goes away. It ends up in this horrible CUDA-bug state where nvidia-smi reports that the memory is used in the top half, but in the bottom half doesn't report any processes that owns that memory.

This is also happening to me when I'm running sweeps. After the agent finishes the first job, the memory in the GPU is not freed and the next job doesn't run due to a CUDA out of memory error.
Did anyone find a fix?

@goncalomcorreia running sweeps from within a single process using wandb.agent can cause these errors if you aren't manually freeing the objects that connect to your GPU in the train function you passed to wandb.agent.  The simplest solution is to use our command line agent.  This way you run wandb agent from the command line and it will execute a program you specify for each trial thereby guaranteeing all object release GPU memory.

Hi! thanks for the help @vanpelt! I think this is what I'm doing. I'm running wandb agent goncalomcorreia/*project_name*/pctdctfe and this happens after the first trial.
In my sweep configuration I have:
command:
- python
- -m
- nmt.train
- --config
- nmt_config.yaml
- ${args}
early_terminate:
  eta: 3
  max_iter: 27
  s: 2
  type: hyperband
method: grid
metric:
  goal: maximize
  name: val_BLEU
parameters:
  model.init_args.prior_latent_size:
    values:
    - 128
    - 256
    - 512

This either means your previous trial didn't exit cleanly which is really strange.  Or it means there are other processes on the box consuming GPU memory.  When executing agents this way we execute the python command, wait for it to exit, then execute a new python command.  When python exits it should release all GPU memory, so something else must be around holding onto it.  You could try running nvidia-smi as this is happening to see the PID of the processes using GPU memory, then do a ps aux | grep $PID to see what's holding onto the ram.

Before the process ends:
(3.9.7) goncalo@server:~$ ps aux | grep 2559215
goncalo  2559215 94.0 24.7 58570208 32577444 pts/8 Dl+ 00:32 499:48 /home/goncalo/.pyenv/versions/nmt/bin/python -m nmt.train --config nmt_config.yaml  --model.init_args.prior_latent_size 2048
goncalo  2905447  0.0  0.0   6432  1968 pts/19   S+   09:23   0:00 grep --color=auto 2559215

After the process finishes, nothing appears in nvidia-smi but the memory is being used as if the process was still running. ps aux | grep $PID now gives:
(3.9.7) goncalo@server:~$ ps aux | grep 2559215
goncalo  2926176  0.0  0.0   6432   676 pts/19   S+   12:46   0:00 grep --color=auto 2559215

exactly as @leopd described above:

nvidia-smi reports that the memory is used in the top half, but in the bottom half doesn't report any processes that owns that memory.

@goncalomcorreia what does it say about used memory?  Can you just copy the output of nvidia-smi and share it here?

@goncalomcorreia bumping Chris' suggestion

here it is:
goncalo@server:~$ nvidia-smi
Wed Nov 10 08:02:02 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:09:00.0 Off |                  N/A |
| 37%   63C    P2   15W / 250W  |   1MiB / 11019MiB    |     0%      Default  |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:0B:00.0 Off |                  N/A |
| 35%   57C    P2    8W / 250W  |   1MiB / 11019MiB    |     0%      Default  |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:41:00.0 Off |                  N/A |
| 48%   84C    P2   15W / 250W  |   9012MiB / 11016MiB |     0%      Default  |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|                                                                             |
+-----------------------------------------------------------------------------+

Notice how in GPU 2 the memory was not released.

@goncalomcorreia Are you still experiencing these types of issues with our most recent version of the client library 0.12.7?

I'm actually still having this problem. And it's very similar to what happened before. nvidia-smi show nothing in process but memory is used. But in nvtop (https://github.com/Syllo/nvtop), it shows a ghost process there and it's using the memory. But it's not killable (i.e. killing it doesn't do anything).
So in my experience, if that happens, the python process is actually still there. If I kill all process kill ( ps aux | grep <my python script> | awk '{print $2}'), the memory usage will be gone. or another way would be I can also do fuser /dev/nvidia* -k to kill all process using the GPU (but it also kills all other process using the GPU, not very nice). So given such experience, would that make sense to say that wandb didn't actually cleanup the child process correctly perhaps? But the thing is it's not happening everytime. (like same scripts, training 1 epochs didn't have problem but 250 epochs had) But it happens to multiple different scripts of mine which is quite annoying. (also not using wandb sweep was totally fine.)
I know this thread is kinda old but I don't really know how to search this issue and this is the only one I found that was related to my case.
My current wandb version is 0.12.17.

PS: I saw some comments on watching the model parameters (which I did use), so I'm trying to do it without the watch.

Albeit closed, I thought I would offer my 2 cents on this since I was wrestling with this issue for the past few days. Looks like freeing up the resources at the end of the training functions does the trick.
# main training loop
def train(config=None):
  with wandb.init(config=config):
    config = wandb.config

    ...

  # cleanup
  del model
  torch.cuda.empty_cache()

# running sweep
wandb.agent(sweep_id, train)
wandb.finish()

That way you can still use the python approach without having to define the .yaml file.

Albeit closed, I thought I would offer my 2 cents on this since I was wrestling with this issue for the past few days. Looks like freeing up the resources at the end of the training functions does the trick.
# main training loop
def train(config=None):
  with wandb.init(config=config):
    config = wandb.config

    ...

  # cleanup
  del model
  torch.cuda.empty_cache()

# running sweep
wandb.agent(sweep_id, train)
wandb.finish()

That way you can still use the python approach without having to define the .yaml file.

This doesn't help the "memory leak" that occurs when I Ctrl+C to terminate the programs though.


WIP: Fix tf lstm

Overview
Closes #3344, #3343
Description of the changes proposed in this pull request:
The problem is that the tensorflow ops were not captured in eager mode (so self._init_between_tensors was not executed which resulted in an error cause self.in_between_ops was not set). This is fixed by introducing _init_between_tensors_eager method, that executes the graph once for the data input and captures all ops that are called. From there on the flow continues as before. Since tensorflow2 seems to introduce a couple more ops, these also need to be added to the op_handlers.
Checklist
I would suggest the following steps in the given order:

[x] The failing test test_tf_keras_lstm2 might be an indication that one of the recently added ops shouldn't be passed through. Understand what is going on here and how we can do this better.
[ ] Check in paper if additivity is conserved with this approach (I guess it should)
[x] Get more test cases to make sure that the issue is correctly fixed (at least 5 different tests, search in previous issues to find some and include feedback from the community). Inspired by this comment, we should at least test RNN, GRU and LSTM layers.
[x] Clean up this PR and make it ready for review. Decide on the tests that we really want to include in the test suite on a permanent basis.
[ ] check out if this fixes #2765

Later:

[x] All pre-commit checks pass.
[x] Unit tests added (if fixing a bug or adding a new feature)

Note: Feel free to review and also post some previously failing examples in here

Thanks for taking on this issue, it would be great to get those issues addressed.
In terms of how to arrange the PRs, I don't really have anything specific to say about this PR but only general advice. It's helpful to break PRs down into small incremental changes wherever possible. It's up to your judgement on how best to split any large PRs.
Your emphasis on adding test cases looks very prudent - this seems like it will be critical, given that there is a huge spaec of possible models that DeepExplainer has to be able to explain.

The errors seem mainly to happen in 2 tests and whether the shap values sum up as expected is heavily dependent on the input. Not sure if I should mark this tests as xfail for now and create a seperate issue. The thing here is that we have a lot of issues where people are complaining about the Deepexplainer outputs not summing up as expected (not passing assert_additivity) and this PR is still a clear improvement over the status quo even without fixing the underlying assert_additivity issue (which I suspect might be a mixture of rounding issues + some wrong assignments in the ophandler). Any feedback on this is very welcome

Hi @CloseChoice @connortann I am very closely following this issue and I am happy to test the releases or fixes done Kindly let me know if you need any support from my side happy to contribute to fix this issue

Hi @CloseChoice @connortann I am very closely following this issue and I am happy to test the releases or fixes done Kindly let me know if you need any support from my side happy to contribute to fix this issue

Thanks for the reply. Your feedback is most welcome. If you could test this branch on data and tell us whether you realise if the results make sense or if there are strong inconsistencies.

Hi @CloseChoice
Sure, Happy to do so.
Just double checking I am good If I run this setup:
this experiment  -->  https://www.kaggle.com/code/billlucas/explaining-cnn-lstm-using-shap/notebook
With this branch code --> Fix-tf-lstm (Branch)
with python version 3.11 and tensorflow latest version
Is my understanding correct ? If there any specific versions I need to use Please point me to the url where can I refer these details
Thank you

Hi @CloseChoice
Sure, Happy to do so.
Just double checking I am good If I run this setup:
this experiment --> https://www.kaggle.com/code/billlucas/explaining-cnn-lstm-using-shap/notebook With this branch code --> Fix-tf-lstm (Branch) with python version 3.11 and tensorflow latest version
Is my understanding correct ? If there any specific versions I need to use Please point me to the url where can I refer these details Thank you

That looks good. Hope with this fix, we are capable to run any tensorflow version above 2.4 aswell, so feel free to use the latest version (our test suite will test against the latest either way). I did not test all layers that I can see in that examples, so am pretty curious about the results.

hi @CloseChoice
i am using it for some LSTM code, as i couldn't compute SHAP values. Now code is working, but I am getting the gradients as zero. Looking into it, I found that in function phi_symbolic (line 365) and then in function grad_graph (line 352), x_grad is zero. Now i see you are still calling self._init_between_tensors(out.op, shap_rAnD) instead of self._init_between_tensors_eager(out.op, shap_rAnD, data..?)? Maybe that is making gradient go to zero?

hi @CloseChoice
i am using it for some LSTM code, as i couldn't compute SHAP values. Now code is working, but I am getting the gradients as zero. Looking into it, I found that in function phi_symbolic (line 365) and then in function grad_graph (line 352), x_grad is zero. Now i see you are still calling self._init_between_tensors(out.op, shap_rAnD) instead of self._init_between_tensors_eager(out.op, shap_rAnD, data..?)? Maybe that is making gradient go to zero?

i also did
x_grad = tape.gradient(
                        out,
                        shap_rAnD,
                        unconnected_gradients=tf.UnconnectedGradients.NONE,
                    )

to see if there is issue in network, but gradient is still zero, which means that gradients are going to zero

hi @CloseChoice
i am using it for some LSTM code, as i couldn't compute SHAP values. Now code is working, but I am getting the gradients as zero. Looking into it, I found that in function phi_symbolic (line 365) and then in function grad_graph (line 352), x_grad is zero. Now i see you are still calling self._init_between_tensors(out.op, shap_rAnD) instead of self._init_between_tensors_eager(out.op, shap_rAnD, data..?)? Maybe that is making gradient go to zero?

i also did
x_grad = tape.gradient(
                        out,
                        shap_rAnD,
                        unconnected_gradients=tf.UnconnectedGradients.NONE,
                    )

to see if there is issue in network, but gradient is still zero, which means that gradients are going to zero

Thanks for looking into this. I also found that all shap values are zero (due to the gradients being zero as you found out). The problem here is that tensorflow is hiding the exact gradients from us and just exposed a PartitionedCall to us but we need to get the exact ops that are called. I'll try to find a workaround for this.

What about call to self._init_between_tensors(out.op, shap_rAnD) in grad_graph function? I changed it to self._init_between_tensors_eager(out.op, shap_rAnD, data..?) but then got other errors. maybe if that is fixed, gradients would be correct

What about call to self._init_between_tensors(out.op, shap_rAnD) in grad_graph function? I changed it to self._init_between_tensors_eager(out.op, shap_rAnD, data..?) but then got other errors. maybe if that is fixed, gradients would be correct

Nope, that won't help since the _init_between_tensors_eager is also just catching the PartitionedCell but we need to get the ops underlying this cell.

@CloseChoice Do we have a status update on this? Would really appreciate it because my model is using some tf2 features and require shap lstm explainer.

@CloseChoice Do we have a status update on this? Would really appreciate it because my model is using some tf2 features and require shap lstm explainer.

Hey, thanks for having your eyes on this. I do not have worked on this for quite a while. The problems lay somewhere deep in tensorflow hiding their API from us. Can't give any timeframe on this.

Note: take a deep look how this repo does it: https://github.com/alewarne/Layerwise-Relevance-Propagation-for-LSTMs


Add Bulk Recount/Reject Functionality for Not Counted Cycle Count Items

Description
Currently, there is no functionality to bulk recount or reject items that have not been counted in the cycle count process. This limitation forces users to manually recount or reject each item individually, which can be time-consuming and inefficient, especially when dealing with a large number of items.

I WANT TO CONTRIBUTE


std.conv.to for POD structs

bearophile_hugs reported this on 2014-11-13T10:42:15Z
Transfered from https://issues.dlang.org/show_bug.cgi?id=13728
CC List

john.loughran.colvin

Description
You can convert 3 strings into a fixed-size array of length 3 using std.conv.to:

void main() {
    import std.conv: to;
    string[] data = ["1", "2", "3"];
    auto foo = data.to!(int[3]);
}

So to simplify some code I'd like std.conv.to to support conversions to POD structs too (without constructors):

void main() {
    import std.conv: to;
    string[] data = ["1.5", "2", "3"];
    static struct Foo { double a; uint b, c; }
    auto foo = data.to!Foo;
}

That is similar to:

void main() {
    import std.conv: to;
    string[] data = ["1.5", "2", "3"];
    static struct Foo { double a; uint b, c; }
    auto foo = Foo(data[0].to!(typeof(Foo.tupleof[0])),
                   data[1].to!(typeof(Foo.tupleof[1])),
                   data[2].to!(typeof(Foo.tupleof[2])));
}

Similar code for std.typecons.tuples could be supported:

void main() {
    import std.conv: to;
    import std.typecons: Tuple;
    string[] data = ["1.5", "2", "3"];
    alias Foo = Tuple!(double,"a", uint,"b", uint,"c");
    auto foo = data.to!Foo;
}

john.loughran.colvin commented on 2014-12-17T21:56:42Z
Why not non-POD structs and classes too?

Also, what do you think about situations where only a few of the members are set:

struct A { int a, b, c; }
auto a = A(3, 4);

is completely valid, but would you want to allow

auto a = [3, 4].to!A;

?

bearophile_hugs commented on 2014-12-17T22:15:54Z
(In reply to John Colvin from comment #2)
> Why not non-POD structs and classes too?

To keep this Enhancement simple and well defined. Further extensions and enhancements are possible later.

> Also, what do you think about situations where only a few of the members are
> set:
>
> struct A { int a, b, c; }
> auto a = A(3, 4);
>
> is completely valid, but would you want to allow
>
> auto a = [3, 4].to!A;
>
> ?

Seems possible, but also adds complexity to the proposal, so better to keep things simple at first.

bearophile_hugs commented on 2014-11-13T10:59:02Z
"data.to!Foo" is also more DRY, this helps avoid some bugs if later you change the definition of Foo.


Login through PagerDuty incident link or ID

What type of PR is this?
feature
What this PR does / Why we need it?
--
Which Jira/Github issue(s) does this PR fix?
Resolves OSD-19682
Special notes for your reviewer
I'm currently out of ideas how to test the scenario described in the test case "should fail to extract cluster ID when the 'details' field under Common Event Format Details is missing or invalid" due to the sensitivity of the token information involved. Considering this limitation, I'm leaning towards removing this test case from the suite. Unless someone has an idea which would be secure.
Pre-checks (if applicable)

[x] Ran unit tests locally
[ ] Validated the changes in a cluster
[x] Included documentation changes with PR

Fyi, code review has been completed in original PR https://github.com/openshift/backplane-cli/pull/337

Codecov Report
Attention: Patch coverage is 35.71429% with 63 lines in your changes are missing coverage. Please review.

Project coverage is 47.98%. Comparing base (6bc8b44) to head (5e28c42).


Remove Support of Py3.7 and Support Py3.11

Issue #, if available:
Description of changes:

Remove support for Python Version 3.7, as its marked End-of-Life for more than 1 year [ref];
Add support for Python Version 3.11

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Thanks for the version bump. README.md also needs an update though.

Thanks for the version bump. README.md also needs an update though.

Thanks for the catch. Just pushed another revision which also updates the README.md.


Scene Not loaded properly in Raspberry Pi 3b

Following readme instructions, including setting up gcc 4.9 we found this will build.
However, it does not load the default scene correctly. Instead of showing Manhattan, it shows a zoomed out world map.
You can zoom and pan, however sometimes the q key does not respond and there is no way to exit the running application. It seems to not see fonts, although they are in the project.
We have a scene that runs fine on tangram-es compiled using the os x readme instructions, that scene causes a segmentation fault when trying to load on the rpi 3.

Hi, sorry for not responding sooner. Unfortunately our team doesn't have an RPi 3 to test this issue. Can you try the build again and send the log output? Thanks.

It's okay we found a workaround. I don't have access to this device currently but if I put my hands on it again will test and reply.

Great, thanks for the reply. I'll close this until we're able to repro the problem.


feat: side by side

This is a feature branch merge. All commits have had their own PR with respective code reviews.

So excited for this feature branch! For visibility, I added a comment for two bugs PTAL thank you 🙏
https://github.com/chanzuckerberg/single-cell-explorer/pull/964#issuecomment-2166131817
CC: @seve @kaloster


Removed console.log()

Removed console.log()

Oh man, good catch! Can you run npm run lib and commit that as well?

Done :-)

Thanks for this PR and for merging it. I'm still seeing this though on v2.2.0 which is the latest on npm.
Any chance to push the updated code to a new npm version?

Well yes it seems on this merge the log came back
https://github.com/casesandberg/react-color/commit/f29c918290e9c63851176fcd3288bbb5e767219f#diff-05beea2169d9a4fc0b781db8e7ceceb9R20391


[benchmark] Small Driver Tweaks

Equalize reported memory usage with & without --verbose

I guess this one can be closed: https://github.com/apple/swift/pull/20808

@nordicio Umm… yes. Sorry, haven't seen your's and I still had another tweak hanging around, so I bundled them together. But thank you for looking into that!

@eeckstein Please review 🙏

@swift-ci benchmark

@swift-ci smoke test

@swift-ci smoke test

@swift-ci benchmark

Build comment file:
No performance and code size changes


About Uploading in Heroku

I want to upload this project to Heroku, but I don't know how I can do.
Please tell me is there any way to upload it, please tell me how to if there is.
If there isn't a way, that's ok! Just be free about testing.
Thank you, All of Qingdao University.

i think is not easy to do so, and i'm not familiar with Heroku.

Hmm...

Then how can I upload this OJ to the WEB?

You can see: https://github.com/QingdaoU/OnlineJudgeDeploy/blob/2.0/README.en.md

I think I should use AWS instead of Heroku

@virusdefender @renbaoshuo than what services did you use to publish qduoj.com to web?

@virusdefender @renbaoshuo than what services did you use to publish qduoj.com to the web?

You can use any server to deploy it.
See: https://github.com/QingdaoU/OnlineJudgeDeploy/blob/2.0/README.en.md

you should use a normal Linux based server


IRC/circe fixes: TLS connection workaround and default server settings

This PR addresses the problem mentioned in issue #1862 where the IRC module circe is hanging on TLS connections when using OpenSSL versions > 1.1.0. In this case, tls.el does not correctly determine the end of the info block and hangs.
This workaround was first proposed here: https://github.com/jorgenschaefer/circe/issues/340#issuecomment-454521683
Since it looks like it will not be fixed upstream, this might be a good place to incorporate the workaround.
Additionally, this PR fixes `circe-network-defaults' being overwritten on init: by default, this list is filled with predefined server details for e.g. Freenode.net which is rather useful.
Please note that this code works fine on my setup and with my OpenSSL setup (1.1.1a on Ubuntu Mate 19.10) connecting to Freenode.net but I have not yet been able to test it on another system/server connection.

Thanks for the feedback -- fixed! First elisp-based PR, so I really appreciate someone looking over even these rather trivial changes :)

Looks good to me! Thanks for the help and the quick response!


Feature: Download canister .did and .most

Please add a command to download canister's .did and .most.
Currently I do my code hacky:
%.install: %.wasm FORCE
	dfx canister create $(*F)
	dfx canister install --network=$(NETWORK) -m install --wasm=$< $(*F)

%.upgrade: %.wasm %.most FORCE
	mkdir -p $(DFXDIR)/.dfx/local/canisters/$(*F)
	cp -f $*.most $(DFXDIR)/.dfx/local/canisters/$(*F)/ # hack!
	cp -f $*.did $(DFXDIR)/.dfx/local/canisters/$(*F)/constructor.did # hack!
	dfx canister install --network=$(NETWORK) -m upgrade --wasm=$< $(*F)

I want to use moc --stable-compatible explicitly to avoid hacks, what is possible only when I get direct access to such canister data.

To get it from deployed canisters: dfx canister metadata <canister_id> motoko:stable-types.
To get it from wasm: ic-wasm <my.wasm> metadata motoko:stable-types.
If you want to make it everything explicit and scriptable, I encourage you to try ic-repl.

OK, dfx canister metadata <canister_id> motoko:stable-types gives .did. But how to get .most? (And I don't really understand what is the distinction between .did and .most.)
Also, the following demonstrates that stresser does have main shared method, but it is missing in the actor:
$ dfx canister metadata stresser motoko:stable-types && dfx canister call stresser main '()'
// Version: 1.0.0
actor {

};
Error: Failed update call.
Caused by: Failed update call.
  The replica returned a replica error: reject code CanisterReject, reject message IC0501: Canister bkyz2-fmaaa-aaaaa-qaaaq-cai is out of cycles, error code None

$ dfx --version
dfx 0.18.0

motoko:stable-types gives you .most. candid:service gives you .did

This doesn't seem to be an issue with Rust CDK but rather with general development using dfx.