Sidekiq::Status tries to connect to Redis in RSpec

Ruby version: 3.0.1
Rails version: 7.0.7
Sidekiq / Pro / Enterprise version(s): 7.1.3
sidekiq initializer:
require 'sidekiq'
require 'sidekiq-status'

# SIDEKIQ_STATUS_EXPIRATION = 48.hours.to_i
Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
  Sidekiq.strict_args!(false)
  # Sidekiq::Status.configure_client_middleware(config,
                                              # expiration: SIDEKIQ_STATUS_EXPIRATION)
end

After updating from Sidekiq 6 to Sidekiq 7, running sidekiq worker tests with RSpec that involve using Sidekiq::Status or Sidekiq::Batch modules are failing because they are trying to connect to a redis server (which should not happen in RSpec, and also confirmed that in Sidekiq 6 it works as expected).
Even trying to put the Sidekiq::Status.configure_client_middleware line in the sidekiq initializer (commented for now) and trying to run a RSpec test with sidekiq will result into the following error:
RedisClient::CannotConnectError:
       Connection refused - connect(2) for <IP_ADDRESS>:6379
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:117:in `connect'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:49:in `initialize'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client.rb:688:in `new'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client.rb:688:in `block in connect'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:12:in `connect'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client.rb:687:in `connect'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client.rb:674:in `raw_connection'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client.rb:641:in `ensure_connected'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client.rb:415:in `multi'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client/decorator.rb:55:in `multi'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-status-3.0.3/lib/sidekiq-status/storage.rb:17:in `block in store_for_id'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-status-3.0.3/lib/sidekiq-status/storage.rb:112:in `block in redis_connection'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-status-3.0.3/lib/sidekiq-status/storage.rb:111:in `redis_connection'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-status-3.0.3/lib/sidekiq-status/storage.rb:16:in `store_for_id'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-status-3.0.3/lib/sidekiq-status/client_middleware.rb:42:in `call'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/middleware/chain.rb:182:in `traverse'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-batch-0.1.9/lib/sidekiq/batch/middleware.rb:11:in `call'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/middleware/chain.rb:182:in `traverse'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/middleware/chain.rb:173:in `invoke'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/client.rb:88:in `push'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/job.rb:361:in `client_push'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/job.rb:198:in `perform_async'
     # /Users/andrew/.rvm/gems/ruby-3.0.1/gems/sidekiq-7.1.3/lib/sidekiq/job.rb:286:in `perform_async'
     # ./spec/workers/associations_dac_billing_report_spec.rb:154:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Errno::ECONNREFUSED:
     #   Connection refused - connect(2) for <IP_ADDRESS>:6379
     #   /Users/andrew/.rvm/gems/ruby-3.0.1/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:117:in `connect'

If I comment that line then the initializer runs successfully, but then it will fail where I use Sidekiq::Status or Sidekiq::Batch in the workers that are being tested. For Sidekiq::Batch I found a workaround with rspec-sidekiq, but I can't find one to prevent Sidekiq::Status to connect to redis server.
I'm using the latest sidekiq-status, 3.0.3, and we really want to upgrade the Sidekiq gem from 6 to 7, so any help would be appreciated, thank you!

I have the same problem, if I remove the include in my Job of Sidekiq Status it works

any updates on this ? @andrewt77 Did you try something ?

This solutions works, but I think is a temporary solution:
spec_helper.rb
  config.before(:suite) { require 'sidekiq-status/testing/inline' }

This wasn't working for me, instead I've completely removed the middleware from my test environment. Here's what my Sidekiq initializer looks like:
require "sidekiq"
require "sidekiq-status"

Sidekiq.configure_client do |config|
  break if Rails.env.test?

  Sidekiq::Status.configure_client_middleware config, expiration: 60.minutes.to_i
end

Sidekiq.configure_server do |config|
  break if Rails.env.test?

  Sidekiq::Status.configure_server_middleware config, expiration: 60.minutes.to_i
  Sidekiq::Status.configure_client_middleware config, expiration: 60.minutes.to_i
end

@angusm73 this is actually a good solution. we still need to figure out why is this happening though

Suggested solutions here didn't resolve an issue. This is what helped:
require "sidekiq"
require "sidekiq-status"
require "sidekiq-status/testing/inline" if Rails.env.test?

Sidekiq.configure_client do |config|
  Sidekiq::Status.configure_client_middleware config, expiration: 60.minutes.to_i
end

Sidekiq.configure_server do |config|
  Sidekiq::Status.configure_server_middleware config, expiration: 60.minutes.to_i
  Sidekiq::Status.configure_client_middleware config, expiration: 60.minutes.to_i
end


Minor change

I would have contributed more to this project ;)

This whole page needs rewriting really.. Will try to have a run through of the site content again now that the site structure is in place :+1:


Supporting 3D buildings in Mapbox GL JS

Recently Mapbox GL JS added support for mapping 3D buildings features found in OSM data. I have implemented some features to support basic building shapes which will allow shapes to be rendered, with sensible default values if height or levels data is not available.
As Mapbox GL JS's current implementation supports only flat roofs, using height, min_height, building:levels, building:min_level tags would suffice.

Great. Thanks!
Part of the work on 3d has been done already in OSM2VectorTiles V3 / OpenMapTiles:
https://openmaptiles.github.io/klokantech-3d-gl-style/#14.47/40.7086/-74.0049/16/56
Please see: https://github.com/openmaptiles/openmaptiles/tree/master/layers/building

@klokan Thanks for reviewing my PR!
OpenMapTiles looks cool and it is new to me, is this going to be the successor to OSM2VectorTiles?
Nevertheless, with the development of OSM2VectorTiles V3 in mind, this might still be a useful for users who have adopted V2 in their existing work.


no response from alpaca-electron

Tried the macOS x86 version. It starts. But when loading the Alpaca model and entering a message, it never responds. In other words: can't make it work on MacOS.
Using MacOS 13.2 on an MacBook Pro M1 (2020).

Yeah, doing the same here. was working fine last night. loaded it this morning and it just won't respond. its like it is not loading the model.

Tried the macOS x86 version. It starts. But when loading the Alpaca model and entering a message, it never responds. In other words: can't make it work on MacOS.
Using MacOS 13.2 on an MacBook Pro M1 (2020).

Why are you using the x64 version? It runs really slow on ARM64 Macs. Use the ARM64 version instead. Or does the ARM64 build not work?

Tried the macOS x86 version. It starts. But when loading the Alpaca model and entering a message, it never responds. In other words: can't make it work on MacOS.
Using MacOS 13.2 on an MacBook Pro M1 (2020).

Why are you using the x64 version? It runs really slow on ARM64 Macs. Use the ARM64 version instead. Or does the ARM64 build not work?

The ARM64 version is broken and cannot be loaded.  I am having all the same issues as above.

I don't have an arm64 Mac. But I guess it's either an arm64 thing or just the M1 chipset. Its pretty quick on my Intel Mac virtual machine.


Fixed Regular Expression Denial of Service (ReDoS)

📊 Metadata *
Bounty URL: https://www.huntr.dev/bounties/1-npm-urlregex/
⚙ Description *
Implemented a different regex engine that is not vulnerable to the same ReDoS attacks caused by catastrophic backtracking.
💻 Technical Description *
From the readme in Google's repo:

RE2 is a fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python. It is a C++ library.

Unlike the native NodeJS regex engine, which is vulnerable to denial of service attacks when a malicious user supplies a very long URL, RE2 lacks the backreference and lookahead capabilities required for this attack, making it safer to use on user supplied input. Given that this package does not accept user supplied regular expressions, and the existing regex was not making use of these operations, I imported the Node bindings for RE2, which use almost identical syntax to the native RegExp.
More information about the Node bindings can be found here.
I understand that the purpose of this package may have been to remain free of dependencies, but I believe that using a regex engine without these capabilities is the right solution, especially given that the dangerous features were not being used.
🐛 Proof of Concept (PoC) *
const urlRegex = require("urlregex");
const isValid = urlRegex().test(
  "http://huntr.devtestvulnerability2312321.testvulnerability2312321.testvulnerability2312321.testvulnerability2312321.testvulnerability2312321.testvulnerability2312321.testvulnerability2312321"
);

console.log(isValid);

🔥 Proof of Fix (PoF) / User Acceptance Testing (UAT)
Unit tests

Congratulations bbeale - your fix has been selected! 🎉
Thanks for being part of the community & helping secure the world's open source code.
If you have any questions, please respond in the comments section. Your bounty is on its way - keep hunting!


Use TLS/SSL for node state query

(Requested from other issue)
Currently the code is using the non-encrypted port (9020) to do the node look up - if it is possible look in to ways of getting the node ips using the MGT api instead. If it is necessary do hit the 'data' api, use the SSL (9021) by default, or have a flag to turn it on
Thanks!

The 1.2 release has a fix for this and uses SSL by default.  Closing out this issue.


Add envoy helm charts

Summary

Extract helm charts for envoy from scalardl(https://github.com/scalar-labs/helm-charts/tree/main/charts/scalardl/templates/envoy) ( thanks @superbrothers )
Added support for charts using dashboards to dynamically change possible values.
After releasing Envoy charts, I will apply them to scalardb, scalardl, and scalardl-audit.

Details
Results of applying it to scalardl and scalardl-audit
$ kubectl get deployments -n default
scalardl-audit-auditor       3/3     3            3           4h16m
scalardl-audit-envoy-envoy   3/3     3            3           4h12m
scalardl-envoy-envoy         3/3     3            3           4h25m
scalardl-ledger              3/3     3            3           4h25m

$ kubectl get pods -n default
scalardl-audit-auditor-6886c85d56-k7xp5       1/1     Running     0          3h36m
scalardl-audit-auditor-6886c85d56-ksqs4       1/1     Running     0          3h36m
scalardl-audit-auditor-6886c85d56-s7nxr       1/1     Running     0          3h36m
scalardl-audit-envoy-envoy-86dd94ddf4-fpv2q   1/1     Running     0          3h32m
scalardl-audit-envoy-envoy-86dd94ddf4-jjktg   1/1     Running     0          3h32m
scalardl-audit-envoy-envoy-86dd94ddf4-zdzlh   1/1     Running     0          3h32m
scalardl-envoy-envoy-79c5c645bd-8mwzf         1/1     Running     0          3h45m
scalardl-envoy-envoy-79c5c645bd-twc8p         1/1     Running     0          3h45m
scalardl-envoy-envoy-79c5c645bd-v6ctr         1/1     Running     0          3h45m
scalardl-ledger-7f889464dd-2prgb              1/1     Running     0          3h45m
scalardl-ledger-7f889464dd-6wqft              1/1     Running     0          3h45m
scalardl-ledger-7f889464dd-fwctt              1/1     Running     0          3h45m

$ kubectl get svc -n default
scalardl-audit-auditor-headless      ClusterIP      None             50051/TCP,50053/TCP,50052/TCP     3h36m
scalardl-audit-envoy-envoy           LoadBalancer   <IP_ADDRESS>    40051:30272/TCP,40052:31673/TCP   3h32m
scalardl-audit-envoy-envoy-metrics   ClusterIP      <IP_ADDRESS>    9001/TCP                          3h32m
scalardl-audit-metrics               ClusterIP      <IP_ADDRESS>    8080/TCP                          3h36m
scalardl-envoy-envoy                 LoadBalancer   <IP_ADDRESS>    50051:32748/TCP,50052:30591/TCP   3h45m
scalardl-envoy-envoy-metrics         ClusterIP      <IP_ADDRESS>   9001/TCP                          3h45m
scalardl-ledger-headless             ClusterIP      None             50051/TCP,50053/TCP,50052/TCP     3h45m
scalardl-metrics                     ClusterIP      <IP_ADDRESS>      8080/TCP                          3h45m

$ kubectl get configmap -n monitoring
scalardl-audit-auditor-grafana-dashboard                       2      3h38m
scalardl-audit-envoy-envoy-grafana-dashboard                   1      3h34m
scalardl-envoy-envoy-grafana-dashboard                         1      3h47m
scalardl-ledger-grafana-dashboard                              2      3h47m

$ kubectl get servicemonitor -n monitoring
scalardl-audit-envoy-envoy-metrics              3h35m
scalardl-audit-metrics                          3h38m
scalardl-envoy-envoy-metrics                    12m
scalardl-metrics                                3h47m

Is it possible to switch between scalardl, scalardb and scalardl-audit in one dashboard by using Variables / Query of Grafana dashboard? I feel that we should avoid templating a JSON file.

Is it possible to switch between scalardl, scalardb and scalardl-audit in one dashboard by using Variables / Query of Grafana dashboard? I feel that we should avoid templating a JSON file.

@superbrothers
I was thinking of creating separate envoy proxy dashboards for scalardl, scalardl-audit, and scalardb, but Can I correct in thinking that it would be better to use grafana's variables to select scalardl, scalardl-audit, and scalardb on a single dashboard?

@superbrothers
Is this what you think?

Is this what you think?

It looks good! In addition, it would be nice to be able to filter the pods by the selected type.

Thank you for your confirmation :bow:
I'll try to create this content.

@superbrothers
The dashboard has been changed to be integrated into a single dashboard.
When Service Type is changed, Pods will also be extracted.
Please check it out :bow:
https://github.com/scalar-labs/helm-charts/pull/26/commits/0bc3c84784477261900ba1d394feb6ce327bbdaf

@feeblefakie
I corrected the points you pointed out.
PTAL :bow:

@miseyu Also, CIs fail. Please check when you get a chance.

@miseyu Also, CIs fail. Please check when you get a chance.
I'm very sorry :bow:

The dashboard has been changed to be integrated into a single dashboard.
When Service Type is changed, Pods will also be extracted.
Please check it out 🙇

Great:+1:  It's fine, but I prefer Values to be in the order ServiceType -> Pods.

I noticed that when each dashboard is enabled, three similar dashboards are created. One dashboard may be more convenient for users, but it makes the installation a little complicated. Umm...

I've installed multiple envoy Charts with Grafana dashboard enabled, but it seems that only one dashboard is enabled. There are no errors.
$ k get -n monitoring cm | grep envoy
envoy-1631542964-envoy-grafana-dashboard                       1      4m30s
envoy-1631542966-envoy-grafana-dashboard                       1      4m28s

This may be fine, but it is a little confusing.

This may be fine, but it is a little confusing.

Yes, this situation was known, but I am very aware that it is difficult to understand.
I'll write a comment or something to the effect that it is difficult to understand.

Great👍 It's fine, but I prefer Values to be in the order ServiceType -> Pods.

I'll fix the order.

@feeblefakie
@superbrothers
I am very sorry about yesterday.
I have been in a flurry of activity, and I deeply apologize for the mix of unintentional commits.
I'll be careful not to do this in the future.
I took care of the issues you pointed out, and would appreciate it if you could check again.
PTAL
https://github.com/scalar-labs/helm-charts/pull/26/commits/b1ce6479e1a042e4f922801427fc24a7e082382f

@feeblefakie
If this is OK, I'll update the index.yaml file after the merge and send out the PR for envoy-charts release. :bow:

@superbrothers PTAL again when you get a chance. :bow: (We can correct other minor issues in a later PR.)


Add netmask/subnet size

Is your feature request related to a problem? Please describe.
We currently have the option to get a lot of network information about the client. However, we are not able to get the current network mask or subnet size (e.g. <IP_ADDRESS> or /24)
Describe the solution you'd like
IMO the best solution would be to add a "netmask": key and value for each of the interfaces in the networkInterfaces() feature.

@muffeeee ... yes very clear. Will add it!

@muffeeee just added subnet mask for ip4 and ip6. Version 4.24.0 just released.


Declare pkg-config dependencies

It fails Travis CI (for some reason Cabal 2.5 can't parse output of pkg-config --modversion) and not all systems have pkg-config so it's necessary to take it into account somehow.
Do you really need pkg-config? HsOpenSSL has a lot of options for various platforms and even tries autodetect OpenSSL installation path on macOS.

Thanks for the quick response.
I have no idea if I absolutely need this, but this was needed to make this package buildable on NixOS. Maybe @bgamari can provide more info on this as he suggested me adding this field.

Just confirmed with @phadej that the pkg-config --modversions parsing problem is a Cabal bug and it now fixed in https://github.com/haskell/cabal/issues/163 .
It'll take some time for that change to be shipped though.

Great. It seems that following scheme is used in tests related to fix:
Flag use-pkg-config
    default: False
    manual:  True
...
Library
    ...
    if flag(use-pkg-config)
        pkgconfig-depends: ...
    else
        ...

Perhaps it worth to adopt it so HsOpenSSL could be used on all systems with or without pkg-config

Closing this PR, since #51 was merged.


noImplicitOverride reports in JS files with checkJs enabled, can only ts-ignore the line

Bug Report
🔎 Search Terms
override checkjs jsdoc
🕗 Version & Regression Information

This changed between versions 4.2 and 4.3

⏯ Playground Link
Playground Link
💻 Code
class Foo {
  method() {
    return 1234
  }
}

class Bar extends Foo {
  method() {
    return super.method() + 42;
  }
}

🙁 Actual behavior
In my projects, I set checkJs to enforce that our non-TS files also typecheck. We also use the noImplicitOverride rule.
But, if you try to create a class with method overrides in JS files, you get the override errors, even though there doesn't appear to be a way to silence them except via // @ts-ignore.
🙂 Expected behavior
noImplicitOverride is not reported in JS files (or, there's some JSDoc way to add it, like types can be).

The issue is labeled "Domain: Error Messages". What should be the expected error message?
/cc @DanielRosenwasser

What should be the expected error message?

@sandersn might have opinions but
This member must have a leading JSDoc comment with an `@override` tag because it overrides a member in the base class 'Foo'.

TypeScript does not have good enough tools to handle jsdoc/comments yet. It's hard to add a jsdoc tag to something if there's already exists leading comments.

This is a good point. We would accept a PR that doesn't include a quick fix, but maybe Nathan also has some ideas there.

Here's a PR to try to add quickfix support in js file. https://github.com/Kingwl/TypeScript/pull/8/files.

It's hard to add a jsdoc tag to something if there's already exists leading comments.

I believe inferFromUsage has code for this.

@Kingwl see if there's anything there you might be able to leverage for your current work, and let us know if you run into any issues.

I guess it's might helpful to @stkevintan.


Fully guide for using RSBuilding

Hi there,
After searching a week in order to use RSBuilding. Now i know how to install and using RSBuilding with WHU dataset.
Please follow very carefully these steps, guys.
STEP 1:  INSTALL
Similarly with author's guide:

Install RSBuilding
git clone git@github.com:Meize0729/RSBuilding.git
cd RSBuilding
Environment Installation
Step 0: Install Miniconda.
Step 1: Create a virtual environment named rsbuilding and activate it.
conda create --name rsbuilding python=3.9
conda activate rsbuilding
Step 2: Install [PyTorch]
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
Step 3: Install [MMCV]
pip install mmcv==2.0.1 -f https://download.openmmlab.com/mmcv/dist/cu117/torch2.0/index.html -i https://pypi.tuna.tsinghua.edu.cn/simple
Step 4: Install other dependencies.
pip install -r requirements.txt

STEP 2:  PREPARING DATASET
Download WHU dataset from this site: http://gpcv.whu.edu.cn/data/building_dataset.html

Extract and store like this:

Move your cursor in to cd tools/data_process/WHU_Building_Dataset
Check your path by pwd. For example, mine is: /BUILDING_DETECTION/RSBuilding/tools/data_process/WHU_Building_Dataset
Export your data_dir path by this command: export data_dir='Your specific path' For example, mine is: export data_dir='/BUILDING_DETECTION/RSBuilding/data_dir'
Recheck by this command: echo $data_dir/. If return same with what you input, that means you successfully assign your data_dir path.
Open file generate_data_list.py in RSBuilding/tools/data_process/WHU_Building_Dataset and modify line 15:
from label_filename = filename.split('.')[0] + '.png' to label_filename = filename.split('.')[0] + '.tif'
The reason is: our data in *.tif format
Run the commands which authors provide:
for train
python generate_data_list.py
--input-img-dir="$data_dir/whubuilding/train/image"
--input-mask-dir="$data_dir/whubuilding/train/label"
--output-txt-dir="$data_dir/whubuilding/train.txt"
for val
python generate_data_list.py
--input-img-dir="$data_dir/whubuilding/val/image"
--input-mask-dir="$data_dir/whubuilding/val/label"
--output-txt-dir="$data_dir/whubuilding/val.txt"
for test
python generate_data_list.py
--input-img-dir="$data_dir/whubuilding/test/image"
--input-mask-dir="$data_dir/whubuilding/test/label"
--output-txt-dir="$data_dir/whubuilding/test.txt"

test.txt will look like this:

Go to this directory: RSBuilding/data_list/whu and modify 2 file: test.txt and train.txt

You have to remove # sign in this line
fill with absolute path to your test.txt, train.txt in data_dir path. Ex, mine is: /BUILDING_DETECTION/RSBuilding/data_dir/whubuilding/test.txt

like image below:

STEP 3:  COLLECTING PRETRAINED MODEL
Download from Baidu as authors' guide

STEP 4: CONFIGURATION
I will use Swin_T and i will upload fully config file below. You can copy then make a new config file, store in your folder and use it directly. NOTE: Change some important parts in this config file. Check the file below.
Swin_T_config.zip
STEP 5: RUN TEST COMMAND
Run test command. For ex, mine is:

Maybe you will got this error:

This error is due to we did not register all the modules of Opencd in to mmengine. In order to register, just add some code lines in to file RSBuilding/tools/test.py.

Do the same maner with file RSBuilding/tools/train.py.
Re-run the test.py then results will look like:

#3

Great tutorial thanks ! I was personally stuck with the opencd import error

I have a visualization question. I set draw = True in default_hooks, but it can not generate results during testing.
visualization=dict(
draw=True,
img_shape=(
512,
512,
3,
),
interval=50,
type='CDVisualizationHook')

I have a visualization question. I set draw = True in default_hooks, but it can not generate results during testing.
visualization=dict( draw=True, img_shape=( 512, 512, 3, ), interval=50, type='CDVisualizationHook')

I haven't tried using the default hook for visualization. We haven't officially released the visualization code, but I might have commented out some of the visualization code during the evaluation process. You can try to look for it. The link is at https://github.com/Meize0729/RSBuilding/blob/main/opencd/evaluation/metrics/iou_base_metric.py.

Hi there,
After searching a week in order to use RSBuilding. Now i know how to install and using RSBuilding with WHU dataset.
Please follow very carefully these steps, guys.
STEP 1: INSTALL Similarly with author's guide:

Install RSBuilding
git clone git@github.com:Meize0729/RSBuilding.git
cd RSBuilding
Environment Installation
Step 0: Install Miniconda.
Step 1: Create a virtual environment named rsbuilding and activate it.
conda create --name rsbuilding python=3.9
conda activate rsbuilding
Step 2: Install [PyTorch]
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
Step 3: Install [MMCV]
pip install mmcv==2.0.1 -f https://download.openmmlab.com/mmcv/dist/cu117/torch2.0/index.html -i https://pypi.tuna.tsinghua.edu.cn/simple
Step 4: Install other dependencies.
pip install -r requirements.txt

STEP 2: PREPARING DATASET Download WHU dataset from this site: http://gpcv.whu.edu.cn/data/building_dataset.html
Extract and store like this:

Move your cursor in to cd tools/data_process/WHU_Building_Dataset
Check your path by pwd. For example, mine is: /BUILDING_DETECTION/RSBuilding/tools/data_process/WHU_Building_Dataset
Export your data_dir path by this command: export data_dir='Your specific path' For example, mine is: export data_dir='/BUILDING_DETECTION/RSBuilding/data_dir'
Recheck by this command: echo $data_dir/. If return same with what you input, that means you successfully assign your data_dir path.
Open file generate_data_list.py in RSBuilding/tools/data_process/WHU_Building_Dataset and modify line 15: from label_filename = filename.split('.')[0] + '.png' to label_filename = filename.split('.')[0] + '.tif' The reason is: our data in *.tif format
Run the commands which authors provide:
for train
python generate_data_list.py --input-img-dir="$data_dir/whubuilding/train/image" --input-mask-dir="$data_dir/whubuilding/train/label" --output-txt-dir="$data_dir/whubuilding/train.txt"
for val
python generate_data_list.py --input-img-dir="$data_dir/whubuilding/val/image" --input-mask-dir="$data_dir/whubuilding/val/label" --output-txt-dir="$data_dir/whubuilding/val.txt"
for test
python generate_data_list.py --input-img-dir="$data_dir/whubuilding/test/image" --input-mask-dir="$data_dir/whubuilding/test/label" --output-txt-dir="$data_dir/whubuilding/test.txt"

test.txt will look like this:
Go to this directory: RSBuilding/data_list/whu and modify 2 file: test.txt and train.txt

You have to remove # sign in this line
fill with absolute path to your test.txt, train.txt in data_dir path. Ex, mine is: /BUILDING_DETECTION/RSBuilding/data_dir/whubuilding/test.txt

like image below:
STEP 3: COLLECTING PRETRAINED MODEL Download from Baidu as authors' guide
STEP 4: CONFIGURATION I will use Swin_T and i will upload fully config file below. You can copy then make a new config file, store in your folder and use it directly. NOTE: Change some important parts in this config file. Check the file below. Swin_T_config.zip
STEP 5: RUN TEST COMMAND Run test command. For ex, mine is:
Maybe you will got this error:
This error is due to we did not register all the modules of Opencd in to mmengine. In order to register, just add some code lines in to file RSBuilding/tools/test.py.  Do the same maner with file RSBuilding/tools/train.py.
Re-run the test.py then results will look like:
#3

Cool!
Thank you for providing guidance to everyone. This is much simpler and more readable than my Readme. Thank you~
However, the installation process for OpenCD can be done via
python setup.py develop

I have a visualization question. I set draw = True in default_hooks, but it can not generate results during testing.
visualization=dict( draw=True, img_shape=( 512, 512, 3, ), interval=50, type='CDVisualizationHook')

I modified this file to export binary mask.
backup your visualization_hook.py and replace by mine in this path: RSbBuilding/opencd/engine/hooks/visualization_hook.py
visualization_hook.zip


There isn't any file or folder under the /mnt/c(WSL could not read the windows files and windows also coult not read the wsl files)

Version
Windows 10 17763.253
WSL Version

[ ] WSL 2
[ ] WSL 1

Kernel Version
No response
Distro Version
Ubuntu 20.04
Other Software
No response
Repro Steps
cd /mnt/c
ls
Expected Behavior
show all of the files and folders under the windows c
Actual Behavior
there isn't any files and folders， last week, the function is correctly.
Diagnostic Logs
No response

Clear the /etc/wsl.conf and restart to solved the problem


Update CODEOWNERS

add owners for /src/azure-cli-core/azure/cli/core/aaz/
add @necusjz as owner of /src/azure-cli/azure/cli/command_modules/monitor

Related command

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

monitor

/azp run


Add components

[ ] install react and fix linters
[ ] add components

thank you for your approval
On Fri, Dec 9, 2022 at 11:01 AM Amkam @.***> wrote:

@.**** approved this pull request.
Status: APPROVED 🟢🟢🟢
Hi @atatm https://github.com/atatm,
Your project is complete!
Highlights:
✅ You used Gitflow correctly 👍
✅ Linter checks passed 👍
✅ Used class-based components for Calculator and App components 👍
✅ Calculator app is nicely styled 👍
✅ Well organized code 👍
There is nothing else to say other than... it's time to merge it: ship it:
Congratulations! 🎉
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something
is not 100% clear.
Please, remember to tag me in your question so I can receive the
notification.
As described in the Code reviews limits policy
https://microverse.zendesk.com/hc/en-us/articles/1500004088561 you have a
limited number of reviews per project (check the exact number in your
Dashboard). If you think that the code review was not fair, you can request
a second opinion using this form https://airtable.com/shrQAqnBwek5a0O0s.
—
Reply to this email directly, view it on GitHub
https://github.com/atatm/math-magicians/pull/2#pullrequestreview-1211261135,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZKNXDEOONQHVRK4TSTWJXDWMLRMZANCNFSM6AAAAAASZBGJUM
.
You are receiving this because you were mentioned.Message ID:
@.***>


Merge Conflict Course - Octocat edition

Overview
TL;DR
Changing the narrative of the Merge conflict course to use our Octocats. ❤  :octocat:
Questions

Are we good with the selected Octocats? Other options are all visible here

Next Steps

[ ] Re-record the instructions gifs
[ ] Ensure all non-Octocat mentions are removed
[ ] 🚢

Review
No @ mentions at this time.

@beardofedu Looked over this and I think it looks great 👍 I'm happy to be associated with the 49ers. 😄
One nit-picky thing that has NOTHING to do with the course, and really belongs in its own PR - Instead of having 'continue' at the bottom of the IDE intro page, it would be cool to have buttons that point to each IDE. But, ignore that comment in this PR. I just wanted to be sure to write it down somewhere.
Before:

After:

@brianamarie

One nit-picky thing that has NOTHING to do with the course, and really belongs in its own PR - Instead of having 'continue' at the bottom of the IDE intro page, it would be cool to have buttons that point to each IDE. But, ignore that comment in this PR. I just wanted to be sure to write it down somewhere.

UGH why did you have to have such a good idea ❤  I think this is a great idea, are we ok with having multiple buttons at the bottom as we increase the number of GUI interfaces that we cover?
GitKraken, Eclipse, GH Desktop, etc

are we ok with having multiple buttons at the bottom as we increase the number of GUI interfaces that we cover?

This seems like a good question for @hectorsector or @JasonEtco - but for my eyes, I think...yes?

@beardofedu 👋 How can I help with this PR?

@brianamarie I just updated the course content and updated the course repository to include all octocat information and remove any unwanted references. The biggest thing you could do to help push this forward is to run through the class and make sure that there aren't any non-Octocat references in the activities, that the activities work, and I think we are good to go.
I still need to update the gifs for the course, but that might be done by the time you try it out.

@beardofedu Could I help with the gifs? Do all of the gifs need to be re-done?

@brianamarie it is all done. let me know what you think. ❤

@beardofedu I made a few changes

The link users are taking and importing is under githubtraining, but on the first page it asked users to ask questions in the class repository on githubschool. I updated the githubschool URL to match the import URL on `githubtraining.
When I pressed the "continue" button, it would go in a different order than the nav. I changed this so it now matches the nav.
Added an oxford comma because I can't not
I removed the "continue" button for IDEs, because it only went to Atom and then back to the main IDE page. Now, there's a note saying "click in the navigation bar for specific walkthroughs".

Please feel free to undo any of this work as always @beardofedu. This looks ready to :ship: to me!


I used the -l option to install, why is it not installed to the specified location on the hard disk?

winget install DuongDieuPhap.ImageGlass -l "D:\ImageGlass"
or
winget install ImageGlass -l "D:\ImageGlass"
Neither installed to location "D:\ImageGlass"，Instead it installed to the default location C:\Program Files\ImageGlass

Not all installers support the -l switch. In some cases the installers can be marked as not having a location switch at all, and in other cases the location switch is simply unknown.
https://github.com/microsoft/winget-cli/blob/fe1eef3400b6dc03df66a93211b85fec0d7a44e3/src/AppInstallerCLIPackage/Shared/Strings/en-us/winget.resw#L340-L342

Are there any configurations that I can download and install to the path "D:\winget" by default?
I install a software, such as winget install abcd, and then the software abcd is installed in the "d:\winget" folder.

As of right now there is only the defaultInstallRoot , but that is only activated if the installer requires a location


pass the output of getCallerIdentity as the name of the federated user

What changed?
Why?
How did you test it?
Potential risks
Is patch release candidate?
Link to relevant docs PRs

I would have preferred that federation token name be composed of splitting caller.Arn instead of using caller.UserId. Something like a strings.SplitAfter( caller.Arn, ":user/" )
The ARN will have the canonical username. It's much clearer than seeing the abstraction of AIDA... as the username in CloudTrail.
But at least we now have a determinate name, so that my console preferences will persist. Thanks!!

@Eddie023 @JoshuaWilkes This is great, but it is almost unusable for Cloudtrail audits where we want to query by username. Now we instead have the User ID in the logs.


chore(flake): update flake.lock

Description of change
I just updated the flake.lock because for some reason nix is trying to update flake.lock while building the package.
for difference:
$ nix run github:orhun/binsider

and to try with updated flake.lock
$ nix run github:osbm/binsider

How has this been tested? (if applicable)
I ran the above code

I mean current flake works but with this flag --no-write-lock-file:
❯ nix run github:orhun/binsider
error:
       … while updating the lock file of flake 'github:orhun/binsider/85485672f055af0c42b9de0c93239317ba6f735c?narHash=sha256-Tvt%2Birgm8o%2B%2B39rCr78f72S0eb9QFVfocdpuHUEhRQo%3D'

       error: cannot write modified lock file of flake 'github:orhun/binsider' (use '--no-write-lock-file' to ignore)
~
❯ nix run github:orhun/binsider --no-write-lock-file
warning: not writing modified lock file of flake 'github:orhun/binsider':
• Updated input 'naersk/nixpkgs':
    'path:/nix/store/lryfc8mhk1czqsa421di2y5nzz5c3b8m-source?lastModified=0&narHash=sha256-HP89HZOT0ReIbI7IJZJQoJgxvB2Tn28V6XS3MNKnfLs%3D' (1970-01-01)
  → follows 'nixpkgs'
~ took 18m3s
❯


When I delete a work from My Work Listing I get forwarded to the catalog

Copy of https://github.com/projecthydra/sufia/issues/2447 for workflow

Sprint 3 PSU


update gengo usage to grpc-ecosystem

github.com/gengo/grpc-gateway now redirects to github.com/grpc-ecosystem/grpc-gateway, which it appears is disliked by golang and results in the following when running go get -u github.com/philips/grpc-gateway-example:
package github.com/philips/grpc-gateway-example
	imports github.com/philips/grpc-gateway-example/cmd
	imports github.com/gengo/grpc-gateway/runtime
	imports github.com/grpc-ecosystem/grpc-gateway/runtime/internal: use of internal package not allowed

Usage of gengo is updated throughout the repo, and I generated new protobufs with the edited makefile and protoc 3.0.0-beta-4.

Hey @sgarciapdx, were you able to get the server up and running ?


Deploy new ksonnet site

ksonnet has a new site based on Hugo. It needs to be deployed to production

Site has been deployed


[$250] Profile - Display Name Not Live Updated When Adding Legal Name to New Account

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v9.0.65-1
Reproducible in staging?: Y
Reproducible in production?: Y
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause Internal Team
Action Performed:

Create a new account and navigate to Settings.
Go to Profile, add a legal name, and observe that the Display Name is not updated.
Refresh the page; the Display Name remains unchanged.

Expected Result:
When a legal name is added to a newly created account (or an account without a previously set display name), the Display Name should automatically inherit the legal name and update in real time.
Actual Result:
The Display Name does not update live after adding a legal name. It only updates after navigating back.
Workaround:
Unknown
Platforms:

[ ] Android: Standalone
[ ] Android: HybridApp
[x] Android: mWeb Chrome
[ ] iOS: Standalone
[ ] iOS: HybridApp
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/ee3cee57-2ad9-4a2d-86c2-cd780b3f6896

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~021861461957491636641
Upwork Job ID:<PHONE_NUMBER>491636641
Last Price Increase: 2024-11-26

Issue OwnerCurrent Issue Owner: @alitoshmatov

@alitoshmatov we already have two proposals, please can you review? Thanks!

Proposal
Please re-state the problem that we are trying to solve in this issue.
The Display Name does not update live after adding a legal name. It only updates after navigating back.
What is the root cause of that problem?
Now, if the user doesn't set the display name, the default display name is the user's login. When we update the legal name and the display name is still the default, the backend only returns the updated data for firstName and lastName in the current user's personal details.
This bug will not happen in the past because displayName is empty by default and in this case will display ${firstName} ${lastName} or login.
Now displayName is the login by default so the UI still displays it if we only update firstName, lastName.
What changes do you think we should make in order to solve the problem?

We decided the optimistic data was unimportant when we updated the legal name here. And it can be wrong in the case of device 1 we update the legal name offline and online after device 2 updates the display name online.

For the bug the display name is not live updated, it has the same RCA with a bug here that is closed because we don't prioritize. We can use my solution in this issue to fix the case here. Update the displayName in Onyx whenever it's outdated.

callback: (val) => {
    Object.keys(val ?? {}).map((accountID) => {
        const details = val?.[accountID];
        const createDisplayName = PersonalDetailsUtils.createDisplayName(details?.login ?? '', details);
        if (createDisplayName !== details?.displayName) {
            Onyx.merge(ONYXKEYS.PERSONAL_DETAILS_LIST, {
                [accountID]: {
                    displayName: createDisplayName
                }
            })
        }
    })
    allPersonalDetails = val
},

https://github.com/Expensify/App/blob/c4ff235cbfb6bc6582a3bb045b813e4d43151135/src/libs/actions/PersonalDetails.ts#L47
What alternative solutions did you explore? (Optional)

@Themoonalsofall Thank you for your proposal, your RCA is correct and solution solve the issue

@huult Thank you for your proposal, your RCA is also correct. your solution is very similar to previous proposal

@nkdengineer Thank you for your proposal, your solution is still not working in offline and I don't think overloading callback function with another onyx update is a good idea.

I think we can go with @Themoonalsofall 's proposal which applies optimistically updates first name, lastname, and display name when display name is default value
C+ reviewed 🎀 👀 🎀

@alitoshmatov

@huult Thank you for your proposal, your RCA is also correct. your solution is very similar to previous proposal

My proposal is significantly different from the previous proposal.

My proposal provides more cases.
Explains how to implement them.
Outlines what needs to be done in the code.

When we call api updateLegalName we are not updating displayName in personalDetails Onyx

This is the RCA of the previous proposal

like BE is returning when we call OpenReport API

I understand the previous proposal added ONYXKEYS.PERSONAL_DETAILS_LIST to optimisticData into https://github.com/Expensify/App/blob/main/src/libs/actions/PersonalDetails.ts#L125. However, it’s not working.

My proposal provides more cases.

True, but we are only interested in conditions related to our issue, hypothetically we could come up with a lot of irrelevant cases

Explains how to implement them.

Yes, not doubting that

Outlines what needs to be done in the code.

Same as your second point

This is the RCA of the previous proposal

like BE is returning when we call OpenReport API

This is not RCA(Root cause Analysis)

I understand the previous proposal added ONYXKEYS.PERSONAL_DETAILS_LIST to optimisticData into https://github.com/Expensify/App/blob/main/src/libs/actions/PersonalDetails.ts#L125. However, it’s not working because UPDATE_LEGAL_NAME does not respond to ONYXKEYS.PERSONAL_DETAILS_LIST

Not sure what you meant here, you can simply add new onyx update to any optimistica data. Moreover your solution calls PersonalDetails.updateDisplayName which would have caused unnecessary API call.

@alitoshmatov Thank you for the more detailed explanation <3


Add '--directory,-C' flag for changing current dir before build

This implements the suggestion in #10098 to make cargo change cwd before
completing config processing and starting the build. It is also an
alternative to --manifest-path that resolves the issue described
in #2930.
The behavior of this new flag makes cargo build function exactly the same when run at the root of the project as if run elsewhere outside of the project.  This is in contrast to --manifest-path, which, for example, results in a different series of directories being searched for .cargo/config.toml between the two cases.
Fixes #10098
Reduces impact of #2930 for many, possibly all impacted, by switching to this new cli argument.
How should we test and review this PR?
The easiest way to reproduce the issue described in #2930 is to create an invalid .cargo/config.toml file in the root of a cargo project, for example ! as the contents of the file.  Running cargo with the current working directory underneath the root of that project will quickly fail with an error, showing that the config file was processed.  This is correct and expected behavior.
Running the the same build with the current working directory set outside of the project, e.g. /tmp, and passing the --manifest-path /path/to/project/Cargo.toml.  The build will proceed without erroring as a result of reading the project's .cargo/config.toml, because config file searching is done from cwd (e.g. in /tmp and /) without including the project root, which is a surprising result in the context of the cargo config documentation, which suggests that a .cargo/config.toml file checked into the root of a project's revision control will be processed during the build of that project.
Finally to demonstrate that this PR results in the expected behavior, run cargo similar to the previous run, from /tmp or similar, but instead of --manifest-path /path/to/project/Cargo.toml, pass --directory /path/to/project to cargo (note the missing Cargo.toml at the end).  The build will provide the exact same (expected error) behavior as when running it within the project root directory.
Additional information
~Passing a path with a trailing '/' will result in failure.  It is unclear whether this is a result of improper input sanitization, or whether the config.cwd value is being improperly handled on output.  In either case this needs to be resolved before this PR is merge-ready.~
(the above issue appears to have been a side effect of local corruption of my rustup toolchain, and unrelated to this change)
Because a struct Config gets created before command line arguments are processed, a config will exist with the actual cwd recorded, and it must then be replaced with the new value after command line arguments are processed but before anything tries to use the stored cwd value or any other value derived from it for anything.  This change effectively creates a difficult-to-document requirement during cargo initialization regarding the order of events.  For example, should a setting stored in a config file discovered via cwd+ancestors search be wanted before argument processing happens, this could result in unpleasant surprises in the exact use case this feature is being added to fix.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) soon.
Please see the contribution instructions for more information.

The note about the trailing slash under "additional information" is in question.  Subsequent retesting seems to suggest something else is going on.  I am continuing to look into it.

FYI I modified the following line in the PR description because it would cause github to close the issue when this PR is merged which we probably don't want to have happen

Fixes #2930 for many, possibly all impacted, by switching to this new cli argument.

:umbrella: The latest upstream changes (presumably #11029) made this pull request unmergeable. Please resolve the merge conflicts.

Ping @compenguy. Just checking in to see if you are still interested in working on this, or if you had any questions we could help with.

Ping @compenguy. Just checking in to see if you are still interested in working on this, or if you had any questions we could help with.

I'm still on this.  It's mostly held up on getting the cli tests implemented, which I'm not entirely clear how to get started on, but if my day job can cut me a break at some point I should be able to put in the time to figure it out.

Thanks for the response and take your time!
Ed already set up a good lazy load mechanism in #11029 for you to move on. So basically we could have a directory changed before Config being loaded. You may use this method to assert the Config has yet been loaded (it was removed anyway but can be restored).
For writing tests, you could follow here or existing tests. You could create a new place like tests/testsuite/change_directory.rs to put your tests. I have no preference to functional or UI tests, but functional tests might be easier because you'll need to assert things before and after changing the directory.
I'd suggest adding tests for at least the following scenarios:

Multiple nested .cargo/config.toml.
cargo check for build commands. Also its interaction with Cargo workspaces
cargo new for non-build commands.
cargo fix, as it has a special management of subprocesses.
cargo help <alias> to make sure aliases loaded correctly

You also need to add a doc for the new flag. You can find the doc of global options here. Note that every time you modify docs for cargo commands, you'll need to do this to rebuild manpages. I usually just run bash ./src/doc/build-man.sh from the project root of Cargo.
I am only suggesting, so please say it out loud if you have a better idea! And feel free to ask anything :)

And in case this is a concern, most of our tests are calling into the cargo binary so you don't havee to worry about mucking with the current process's working directory.

Hi @compenguy, just checking in to see if you are still working on this issue. If not, may I handle the remaining parts? Thank you.

I remember us talking about this during a cargo team meeting but can't find a reference to it, so going ahead and doing a quick poll
@rfcbot merge

@rfcbot concerns bikeshed-flag-name
#10098 mentions other tools like make and git choosing -C as the short form of this flag. However, only make adopts the long form --directory. Randomly googling some other tools, go CLI only have the short -C, whereas mecurial uses --cwd.
Personally I am happy with -C but not sure about the usefulness of the long form --directory.

* Do we want to support multiple `-C` specified as how `git` and `make` do? I have no idea if that is a useful behaviour.

It seems to be that it would be pretty safe and non-breaking to add that later if it came up as being useful.  I can't think of a way that adding that later would adversely impact any existing users.

* As mentioned [here](https://github.com/rust-lang/cargo/pull/10952#issuecomment-1342893986), could you document this new flag? Could also do in a follow-up PR.

Now that I think about it, it probably would have been better to update the docs in a follow-up PR, because it adds a lot of noise to the diff, but I tried to keep the commits tidy and separated so filtering by commit is still pretty clean.  At the moment, the first commit adds the new behavior, the second ensures the invariant (adding the assert!()), the third changes the doc template, and the very last commit is all the automated changes to the docs.

@rfcbot concerns bikeshed-flag-name
#10098 mentions other tools like make and git choosing -C as the short form of this flag. However, only make adopts the long form --directory. Randomly googling some other tools, go CLI only have the short -C, whereas mecurial uses --cwd.
Personally I am happy with -C but not sure about the usefulness of the long form --directory.

I personally don't care too much what the long form is: --directory is fine, as is --cd. I'd also be fine with this being short-only.

Am not a fan of short flag names, especially if there is no long-form version.
I would be happy with --directory.

I don't have a strong opinion about the long name. --cwd or --cd seems more explicit as to what it does, and would lean slightly towards that, but other options seem fine. -C seems to be the most common for a short flag.
A quick survey:

make: -C or --directory
git: -C
Meson: -C
ninja: -C
hg: --cwd
Gradle: -p, --project-dir?
Julia: --project?
Swift: -working-directory?
dub: --root
msbuild: passed as first argument
npm: n/a
Ivy: n/a?
Maven: n/a?
Bazel: n/a?
Buck: n/a?
go: n/a?
nuget: n/a?
sbt: n/a?
dart: n/a?

--cwd/--cd sounds better to me for its explicitness (and way shorter). Although --directory is also widely used in make and dart, it seems a bit common in other different scenarios regarding directories. Just trying to avoid the same situation like https://github.com/rust-lang/cargo/issues/11554. If people feel good with --directory, I can go with that as well.

We've talked about the flag name a bit on Zulip. Still have yet reached a consensus on the long flag name. @epage gave a good suggestion that the majority of the Cargo team and others thumb it up:

-C alone for now, deferring any bike shedding to an issue as -C alone has enough precedence and some progress is better than no progress

To move forward, @compenguy, could you remove the long flag --directory and keep -C only?

Thank you!
@rfcbot resolved bikeshed-flag-name

As the rfcbot was more used as a straw poll, this was pre-approved before the PR was made, and the flag situation has significant buy-in on zulip, I'm going to merge now rather than wait the 10 days
(I updated the title and description)
@bors r+

:pushpin: Commit 6510cc5c4bbf2fc9c0daf3288d2510ac2366fc4f has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit 6510cc5c4bbf2fc9c0daf3288d2510ac2366fc4f with merge 74043673e6824c09daf8fdf3cad9681914f4283e...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 74043673e6824c09daf8fdf3cad9681914f4283e to master...


Raise exception for bad session names

Current coverage is 78.32% (diff: 90.47%)

Merging #14 into master will increase coverage by 0.21%

@@             master        #14   diff @@
==========================================
  Files             8          8
  Lines           795        812    +17
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            621        636    +15
- Misses          174        176     +2
  Partials          0          0

Powered by Codecov. Last update 0814211...430c9d9


Integrate numeric "focus score" focusing tool into main dencam code

We have a script in the utilities that demonstrates this functionality.  Seems plausible that we want to integrate this into the video preview page of the UI.  Might be a good idea to have BJ test out the utility first and confirm that he thinks this would be a good thing to have in the field.
Note: the implementation in the utility depends on OpenCV.  It is very likely that we don't want to add such a heavy dependency to the main DenCam code.  Hence, either we don't add this feature or we work out how to compute the variance of the Laplacian (the only thing OpenCV is being use to do) with lighter-weight tools (seems quite possible that this is feasible).

Now that Alex has made some great improvements to the utility that demonstrates this functionality, we should have BJ try it out and have him assess whether this tool would be a good thing to use in the field for focusing actual DenCams in situ.  If he thinks yes, then we should go ahead and integrate it into the appropriate page of the DenCam UI.

I've been working on integrating the focus score tool into the main dencam tool, but I am running into issues initializing a camera for the respective screen (ex. preview page = picamera, focus tool = opencv). This issue mostly has to do with the button handling being done using threading. If I try to initialize the cameras in the buttons class, there's no guarantee that the thread finished before the screen has updated, thus causing the application to crash because it's trying to access a camera that hasn't been initialized yet.


[BUG] Remove os.system calls from example notebooks

Is your feature request related to a problem? Please describe.
We should remove 'os.system' calls and other bash code from the example notebooks, and replace with python functions as much as possible.
We currently have calls in our example notebooks like os.system("wget http://files.grouplens.org/datasets/movielens/ml-25m.zip") . The problem with is that we don't ship wget on the container we install, and since we don't check the return code of the os.system call this fails silently.
We can directly download these files in python fairly easily - and provide progress bars on how this works. For an example see https://github.com/benfred/implicit/blob/master/implicit/datasets/_download.py , which is used to download movielens here https://github.com/benfred/implicit/blob/ba673a914819c98e144a7cbe691b26a654cbfe26/implicit/datasets/movielens.py#L38

Done with #709


Retry logic for 403 actions

While integrating gdrivefs to a test suite, I noticed that it doesn't handle rate limit exceeding errors in the exponential waiting fashion (https://developers.google.com/drive/api/v3/handle-errors#exponential-backoff). Would it make sense if I contribute a similiar (maybe without using funcy.retry, so no dependencies would be introduced) decorator to the example below? So that this would be handled for all use cases (which is very important, if you are making simultaneous requests [e.g during CI, 10 workers], without a retry logic it just fails but with this it succeeds on one of the tries).
https://github.com/iterative/dvc/blob/63f32936b20c23abc32e9dba1aba19ab5db804e9/tests/remotes/gdrive.py#L19-L47

Yes, I completely agree there should be proper retry logic with exponential backoff. This is done for s3fs and gcsfs, for example. dvc's implementation would be fine too (since it is meant for gdrive); but elsewhere we have found that the number of conditions that do need a retry easily grows with time!


Since latest update lots of performance problems editing complicated notebooks

We have written the needed data into your clipboard because it
Issue Type: Performance Issue
The performance upgrade appears to have moved all the rendering to a background thread, which gives the illusion of UI interaction but few things work until CPU goes back to 0 on a complicated notebook. The new shortcut keys for the Markdown cells also seems to be causing a lot of issues elsewhere. I cannot type in pre-existing code cells in a notebook but have to copy the code out, edit it elsewhere and then paste it back in, and the Ctrl-C/Ctrl-V shortcut keys don't even work for paste.
Azure Data Studio version: azuredatastudio 1.33.0 (6783aa6967e8c6d086b6f04e5fbf527659981389, 2021-10-26T17:28:15.782Z)
OS version: Windows_NT x64 10.0.17763
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (8 x 2095)

GPU Status
2d_canvas: unavailable_softwaregpu_compositing: disabled_softwaremultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: disabled_offrasterization: disabled_softwareskia_renderer: enabled_onvideo_decode: disabled_softwarevulkan: disabled_offwebgl: unavailable_softwarewebgl2: unavailable_software

Load (avg)
undefined

Memory (System)
20.00GB (5.27GB free)

Process Argv

Screen Reader
no

VM
100%

 was too large to send. Please paste.

Hi @dullroar, thank you for filing an issue. Couple of clarifying questions:

Would you be able to share the notebook where you encountered this issue: "I cannot type in pre-existing code cells in a notebook"?
Is this happening with every pre-exisiting notebook that you open?
Just to confirm, are you referring to the Incremental Grid Rendering feature when you say performance upgrade? And are you seeing the issues when the feature is turned on?

Lucy:

This is internal notebooks documenting data quality issues in our CRM system so sadly, no. I will say where I see it most is in fairly large notebooks, but there is nothing fancy going on. A text cell stating "This is an attribute on a CRM entity, following are one or two queries showing data issues," and then the code cell(s) and result(s). All results are grids, not charts. None of the result sets are overly large, it is just there are quite a few cells and cell types in each notebook. The most problematic .ipynb file with results in it is 3MB.
Yes, although at "some point" when editing a new notebook, past a certain x of complexity (size? cell count? don't know), will start slowing down, too.
I am referring to the release that just came out, azuredatastudio 1.33.0 (6783aa6, 2021-10-26T17:28:15.782Z from the info I pasted above, apparently. To my knowledge I am not on any insider release. I upgraded to this release because the release notes promised performance improvements. The prior release without the problems I am experience would just freeze for a couple minutes when opening one of these files, but then everything worked pretty well. Now, the usability has gone down dramatically. :(

Hi @dullroar,
The fix for "I cannot type in pre-existing code cells in a notebook" is now available in Insiders, which can be downloaded here: https://github.com/microsoft/azuredatastudio#try-out-the-latest-insiders-build-from-main.
Can you try out the latest Insiders build and see if you run into any issues typing in pre-existing code cells?
For the performance issues, can you check if the Incremental Grid Rendering feature is on? You can find this by going to Settings and searching for Incremental Grid Rendering. Turning this feature off should give you the same performance as the previous release (1.32).

For the Markdown keyboard shortcuts, could you explain the specific shortcuts that are not working and what issues they are causing?

I am now on the Insiders build and can state the "cannot type in pre-existing code cells in a notebook" still exists there.
I turned off Incremental Grid Rendering, and indeed it goes back to the old behavior of taking minutes to render and become responsive, but then it works quite nicely. I prefer that. I can always go get some more water while waiting for a notebook to open. :)
It isn't the Markdown keyboard shortcuts that aren't working - sorry if I was unclear. It is Ctrl-C/Ctrl-V isn't working by keyboard shortcut in the code cells when the symptoms in #1 are occurring, but mouse commands (select all, Copy, Paste) still work. So sometimes, since it can take minutes for a code cell to become editable, I will use the mouse to copy out the code in the cell, edit it somewhere else (koff Notepad++ koff) and then use the mouse to paste it back into the notebook code cell. That works, but is the definition of "clunky" and obviously not desired behavior.

Can you confirm the version of ADS Insiders you're using (Help -> About -> Copy)

Version: 1.33.0 (user setup)
Commit: 6783aa6967e8c6d086b6f04e5fbf527659981389
Date: 2021-10-26T17:28:15.782Z
VS Code: 1.58.0
Electron: 12.0.9
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.17763

Ah, that's still the stable release. Could you try following the instructions here to get the insiders build and try that? https://github.com/Microsoft/azuredatastudio#try-out-the-latest-insiders-build-from-main
(they can be installed side-by-side so you don't have to uninstall the stable build or anything first)

I am confused. I downloaded the release from that page, installed it, and started it from the installer wanting to launch it. Do I have to remove the system-installed release first?

Odd, which insiders installer did you use?

azuredatastudio-windows-user-setup-1.34.0-insider.exe is what I launched from the Downloads folder.

Logged out, logged back in, started ADS, confirmed in Task Manager the path to the executable was in my AppData. It still looks like the same About info, though:
Version: 1.33.0 (user setup)
Commit: 6783aa6967e8c6d086b6f04e5fbf527659981389
Date: 2021-10-26T17:28:15.782Z
VS Code: 1.58.0
Electron: 12.0.9
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.17763

Also confirmed ADS is in my PATH and that it is pointing to AppData, and that there isn't another ADS in my path (like from a system install).

How are you launching the insiders build? You should have two entries in the start menu :

Or you can also

Launch from a cmd window with azuredatastudio-insiders
Browse to %LOCALAPPDATA%\Programs\Azure Data Studio - Insiders and launch azuredatastudio-insiders.exe

@dullroar FYI I updated the title to focus specifically on the "can't type in the cell" issue.
Once we've verified the fix with you then we can see if you're still running into perf issues and if so open a separate issue for that.

Thanks @Charles-Gagnon @lucyzhang929 for your help on diagnostics.

Thanks for the confirmation @dullroar - so it sounds like the "can't type in code cell" issue is resolved in the insiders?
Would you mind making a new issue for the perf issues with any details you can provide? We can start helping with taking a look at that separately then.

@Charles-Gagnon I would consider this issue resolved. May I tag you when I open a new performance-related ticket?

Feel free to tag me @dullroar. I can assign it out 😄


feat: support for new ICD hosting model

Description
Add support for the IBM Cloud Database new hosting model, including bumping minimum memory requirements from 1G to 4G.
Release required?

[ ] No release
[ ] Patch release (x.x.X)
[x] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content
Reference: Cloud Database Hosting Models
The host model switching section has details of the migrations that the service is rolling out. All instances will have to be switched and the time line for the migrations is outlined.
During the transition both existing and new models are supported.
Users of this module should consider the changes being implemented by the service. It is recommended that new deployments use the shared or isolated compute models to start, by specifying the host_flavour. It is recommended that existing deployments make updates to control the upgrade process by specifying shared or isolate compute configurations and the resource required to run the service, by adding the host_flavour.
Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

The only change during upgrade is the group block and results in an update in-place.
This is expected and the exact details vary by the existing deployment.
existing deployment with less then 4G memory: The group memory block will update in-place to 4G of memory.
existing deployment with 4G plus memory: No change.
existing deployment adding host_flavor (as below): The group memory block will update in-place to the flavor chosen.
The upgrade test is safe to skip.
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185: Terraform used the selected providers to generate the following execution
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185: plan. Resource actions are indicated with the following symbols:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   ~ update in-place
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:  <= read (data resources)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185: Terraform will perform the following actions:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   # module.icd_rabbitmq.data.ibm_database_connection.database_connection will be read during apply
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   # (depends on a resource or a module with changes pending)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:  <= data "ibm_database_connection" "database_connection" {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + amqps         = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + analytics     = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + bi_connector  = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + cli           = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + deployment_id = "crn:v1:bluemix:public:messages-for-rabbitmq:eu-de:a/abac0df06b644a9cabc6e44f55b3880e:f2d90f99-37e7-4779-a909-411f96edf766::"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + emp           = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + endpoint_type = "private"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + grpc          = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + https         = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + id            = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + mongodb       = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + mqtts         = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + mysql         = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + ops_manager   = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + postgres      = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + rediss        = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + secure        = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + stomp_ssl     = (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + user_id       = "admin"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + user_type     = "database"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:     }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   # module.icd_rabbitmq.ibm_database.rabbitmq_database will be updated in-place
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   ~ resource "ibm_database" "rabbitmq_database" {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         id                        = "crn:v1:bluemix:public:messages-for-rabbitmq:eu-de:a/abac0df06b644a9cabc6e44f55b3880e:f2d90f99-37e7-4779-a909-411f96edf766::"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         name                      = "rabbitmq-upg-vqirdq-rabbitmq"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         tags                      = []
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         # (19 unchanged attributes hidden)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       - group {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           - group_id = "member" -> null
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           - cpu {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               - allocation_count = 0 -> null
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           - disk {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               - allocation_mb = 1024 -> null
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           - members {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               - allocation_count = 3 -> null
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           - memory {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               - allocation_mb = 1024 -> null
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:       + group {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           + group_id = "member"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           + cpu {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               + allocation_count = 0
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           + disk {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               + allocation_mb = 1024
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           + host_flavor {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               + id = "multitenant"
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           + members {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               + allocation_count = 3
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:           + memory {
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:               + allocation_mb = 4096
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:             }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:         # (3 unchanged blocks hidden)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:     }
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185: Plan: 0 to add, 1 to change, 0 to destroy.
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185: Changes to Outputs:
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   ~ hostname                   = "f2d90f99-37e7-4779-a909-411f96edf766.bpb68u2f0gvaqgd0n64g.private.databases.appdomain.cloud" -> (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185:   ~ port                       = 32495 -> (known after apply)
TestRunCompleteUpgradeExample 2024-06-28T11:18:42Z command.go:185: ╷

/run pipeline

/run pipeline

/run pipeline

/run pipeline


Model examples/now-lambda-reason as a separate package

This makes examples/now-lambda-reason a separate esy package.
The following works:
% cd examples/now-lambda-reason
% esy i
info install 0.4.3
info fetching: done
info installing: done
% esy b
info esy build 0.4.3
info building @opam/lambda-runtime@link:../../lambda-runtime.opam
info building @opam/lambda-runtime@link:../../lambda-runtime.opam: done
info building @opam/now-lambda@link:../../now-lambda.opam
info building @opam/now-lambda@link:../../now-lambda.opam: done
% esy x basic
Fatal error: exception Failure("Could not find runtime API env var: AWS_LAMBDA_RUNTIME_API")

This also removes the devDependency on @opam/reason at the root esy.json.

I think I'm fine with the single top level esy.json package for now. I'll revisit this later.


configuring rate limit for the API does not take effect

Bug Description
Configuring rate limit for the API does not take effect
I'm access the details API in the sleep container，even if it is greater than 10 times per minute, it can still be accessed
The logs thrown by ratelimit are duplicates with no useful information
/ $ curl -I http://details.boook:9080/details/0
HTTP/1.1 200 OK
content-type: application/json
server: envoy
date: Fri, 24 Sep 2021 08:38:03 GMT
content-length: 178
x-envoy-upstream-service-time: 8

i rarely use EnvoyFilter, so how do I locate problems?
apiVersion: v1
kind: ConfigMap
metadata:
  name: ratelimit-config
  namespace: default
data:
  config.yaml: |
    domain: productpage-ratelimit
    descriptors:
      - key: PATH
        value: "/details/0"
        rate_limit:
          unit: minute
          requests_per_unit: 10

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-details
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: details
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ratelimit
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            domain: productpage-ratelimit
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: rate_limit_cluster
                timeout: 10s
              transport_api_version: V3
    - applyTo: CLUSTER
      match:
        cluster:
          service: ratelimit.default.svc.cluster.local
      patch:
        operation: ADD
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          load_assignment:
            cluster_name: rate_limit_cluster
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                     socket_address:
                      address: ratelimit.default.svc.cluster.local
                      port_value: 8081

ratelimit image: envoyproxy/ratelimit:40393342
Version
# ~/istio-1.11.1/bin/istioctl version
client version: 1.11.1
control plane version: 1.10.0
data plane version: 1.10.0 (9 proxies)

Additional Information
NONE
Affected product area

[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Control Plane Revisions

Is this the right place to submit this?

[X] This is not a security vulnerability
[X] This is not a question about how to use Istio

the details of the configuration
    filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        accessLog:
        - name: envoy.access_loggers.file
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            logFormat:
              textFormat: |
                [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%
            path: /dev/stdout
        httpFilters:
        - name: envoy.filters.http.ratelimit
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            domain: productpage-ratelimit
            failureModeDeny: true
            rateLimitService:
              grpcService:
                envoyGrpc:
                  clusterName: rate_limit_cluster
                timeout: 10s
              transportApiVersion: V3
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

do you config the rate limit actions according to https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/ ?

do you config the rate limit actions according to https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/ ?

thank you for your reply.
i updated the description of the issue. please have a look again

more than ten requests can return http code 429
/ $ curl -I http://httpbin.boook:8000/ip
HTTP/1.1 429 Too Many Requests
cbcccbbb: true
content-length: 18
content-type: text/plain
date: Sun, 26 Sep 2021 09:27:45 GMT
server: envoy
x-envoy-upstream-service-time: 1

proxy config listener
[root@10-20-11-81 bookinfo]# istioctl pc listener httpbin-66cdbdb6c5-zjfwq.boook -oyaml | grep -5 /ip
            typeUrl: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              descriptors:
              - entries:
                - key: PATH
                  value: /ip
                token_bucket:
                  fill_interval: 120s
                  max_tokens: 1
                  tokens_per_fill: 1

i don't know what went wrong, and I want to fix it.

@xichengliudui The rateLimits.actions is also needed to define the descriptors so that envoy can understand what PATH means. You can add a configPatch in that EnvoyFilter.
  - applyTo: HTTP_ROUTE
    match:
      context: SIDECAR_INBOUND
      routeConfiguration:
        vhost:
          route:
            action: ROUTE
    patch:
      operation: MERGE
      value:
        route:
          rateLimits:
          - actions:
            - requestHeaders:
                descriptorKey: PATH
                headerName: :path

@pangsq thanks, i'm going to try it now.

Thank you very much. Now Ratelimit is working fine

The requestHeaders name felt a bit strange and made me ignore it...
Why not requesPath

The requestHeaders name felt a bit strange and made me ignore it...
Why not requesPath

Besides :path, there are some other special headers that can be used in requestHeaders, like :method,:authority,:scheme,host. All the attributes can be represented as headers, so maybe no need to add a requestPath or other configs.

/close

@xichengliudui  As i know, the source_cluster does not mean where the request is from, but represents what service-cluster the server instance is.
For example, if you have a deployment called client, and a deployment called server in namespace default, then you config a ratelimit descriptor: source_cluster: server.default.   You will find the requests from client to server is limited. I think the source_cluster is not designed for limiting requests from some origin cluster.
Code in go-control-plane:
// The following descriptor entry is appended to the descriptor:
//
// .. code-block:: cpp
//
//   ("source_cluster", "<local service cluster>")
//
// <local service cluster> is derived from the :option:`--service-cluster` option.
type RateLimit_Action_SourceCluster struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields
}

Code in envoy: ( it is clearly that source_cluster is local_service_cluster)
bool SourceClusterAction::populateDescriptor(RateLimit::DescriptorEntry& descriptor_entry,
                                             const std::string& local_service_cluster,
                                             const Http::RequestHeaderMap&,
                                             const StreamInfo::StreamInfo&) const {
  descriptor_entry = {"source_cluster", local_service_cluster};
  return true;
}

thank you for your answer.
does rate limit currently have the ability to limit source services?
e.g
service A -> service C    // 60 requests per minute
service B -> service C    // 100 requests per minute

user-agent or x-envoy-downstream-service-cluster can be used to represent which cluster the request is from.
But istio does not enable AddUserAgent in HttpConnectionManager , you may config EnvoyFilters for "service A" and "service B" (or one EnvoyFilter in istio-system for all workloads).

user-agent or x-envoy-downstream-service-cluster can be used to represent which cluster the request is from. But istio does not enable AddUserAgent in HttpConnectionManager , you may config EnvoyFilters for "service A" and "service B" (or one EnvoyFilter in istio-system for all workloads).

@pangsq  我很少接触 envoy，谢谢解答

@pangsq
i used x-envoy-downstream-service-cluster to set the rate limit, but the actual effect was different from what I predicted
after I added x-envoy-downstream-service-cluster, it not only didn't take effect, but also affected the API rate limit set before
Is there any method to debug?
spec:
  workloadSelector:
    labels:
      app_mesh: httpbin
  configPatches:
  - applyTo: HTTP_ROUTE
    match:
      context: SIDECAR_INBOUND
      routeConfiguration:
        vhost:
          route:
            action: ROUTE
    patch:
      operation: MERGE
      value:
        route:
          rateLimits:
          - actions:
            - requestHeaders:
                descriptorKey: CLIENTCLUSTER
                headerName: "x-envoy-downstream-service-cluster"
            - requestHeaders:
                descriptorKey: PATH
                headerName: ":path"

spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.local_ratelimit
        typed_config:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
          value:
            descriptors:
            - entries:
              - key: PATH
                value: /ip
              token_bucket:
                fill_interval: 120s
                max_tokens: 5
                tokens_per_fill: 5
            - entries:
              - key: CLIENTCLUSTER
                value: sleep.test
              token_bucket:
                fill_interval: 120s
                max_tokens: 5
                tokens_per_fill: 5
            ......
            stat_prefix: http_local_rate_limiter
            token_bucket:
              fill_interval: 60s
              max_tokens: 10
              tokens_per_fill: 10

i checked EnvoyFilter and saw that the configuration had been set up successfully
istioctl pc listener httpbin-75f6f79b96-r2mwv.ratelimit-test -ojson
},
                                                "route": {
                                                    "cluster": "InboundPassthroughClusterIpv4",
                                                    "timeout": "0s",
                                                    "rateLimits": [
                                                        {
                                                            "actions": [
                                                                {
                                                                    "requestHeaders": {
                                                                        "headerName": "x-envoy-downstream-service-cluster",
                                                                        "descriptorKey": "CLIENTCLUSTER"
                                                                    }
                                                                },
                                                                {
                                                                    "requestHeaders": {
                                                                        "headerName": ":path",
                                                                        "descriptorKey": "PATH"
                                                                    }
                                                                }
                                                            ]
                                                        }
                                                    ],
                                                    "maxStreamDuration": {
                                                        "maxStreamDuration": "0s",
                                                        "grpcTimeoutHeaderMax": "0s"
                                                    }
                                                },

                           "httpFilters": [
                                {
                                    "name": "envoy.filters.http.local_ratelimit",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                        "typeUrl": "type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit",
                                        "value": {
                                            "descriptors": [
                                                {
                                                    "entries": [
                                                        {
                                                            "key": "PATH",
                                                            "value": "/ip"
                                                        }
                                                    ],
                                                    "token_bucket": {
                                                        "fill_interval": "120s",
                                                        "max_tokens": 5,
                                                        "tokens_per_fill": 5
                                                    }
                                                },
                                                {
                                                    "entries": [
                                                        {
                                                            "key": "CLIENTCLUSTER",
                                                            "value": "sleep.test"
                                                        }
                                                    ],
                                                    "token_bucket": {
                                                        "fill_interval": "120s",
                                                        "max_tokens": 5,
                                                        "tokens_per_fill": 5
                                                    }
                                                }
                                            ],
                                            "filter_enabled": {
                                                "default_value": {
                                                    "numerator": 100
                                                },
                                                "runtime_key": "local_rate_limit_enforced"
                                            },
                                            "filter_enforced": {
                                                "default_value": {
                                                    "numerator": 100
                                                },
                                                "runtime_key": "local_rate_limit_enforced"
                                            },
                                            "response_headers_to_add": [
                                                {
                                                    "append": false,
                                                    "header": {
                                                        "key": "has-been-rate-limit",
                                                        "value": "true"
                                                    }
                                                }
                                            ],
                                            "stat_prefix": "http_local_rate_limiter",
                                            "token_bucket": {
                                                "fill_interval": "60s",
                                                "max_tokens": 10,
                                                "tokens_per_fill": 10
                                            }
                                        }
                                    }
                                },
                              ]