Shuttle Express jog dial support

One of the other features that I really liked in Chilipeppr was the ability to use a Shuttle Express USB jog dial for machine shuttle.  @lordmundi wrote the code specifically to communicate via the Serial Port JSON Server, which is used by Chilipeppr.  Take a look at the project here:  https://github.com/lordmundi/shuttleCP
Not sure how easy it would be to change it to use web calls instead of JSON.
All this being said, I think there are plans for a jogging mode/interface in upcoming grbl releases.

It should be pretty easy to convert my codebase over to something that works with cnc.js, as my code is a standalone program that is just reading the jog dial and then sending G-code over a websocket to command the machine.  If your code has a websocket server that can accept commands, then it should be pretty trivial to port.

It should be pretty easy to convert my codebase over to something that works with cnc.js, as my code is a standalone program that is just reading the jog dial and then sending G-code over a websocket to command the machine.  If your code has a websocket server that can accept commands, then it should be pretty trivial to port.

This may take a while because I don't have the Shuttle Express Jog Dial. It will be great if someone can provide support for the porting. :smiley:

I can provide support. And, we could probably get some folks to chip in
some money to get you a jog dial.
FG
Sent from my Android phone
On Jan 4, 2016 3:55 AM, "Cheton Wu"<EMAIL_ADDRESS>wrote:

This may take a while because I don't have the Shuttle Express Jog Dial.
It will be great if someone can provide support for the porting. [image:
:smiley:]
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-168627421.

Just did a quick search and found the jog dial sold by Amazon. Is this the one that's supported by ShuttleCP?
http://www.amazon.com/gp/product/B0032Y0OH0?psc=1&redirect=true&ref_=ox_sc_act_title_1&smid=A1YG1VZVF48PM9

yeah, that's it.  i'd be happy to donate $20 toward you getting one.
I should also mention that most of my work was to get it to read the jog
dial under linux and that was running on a different machine than my
chilipeppr instance.  If you have the jog dial running on the same machine
as your cnc.js instance, its possible it might be way easier.  For example,
i think there is some way to get the jog dial to simply reproduce key
strokes when you hit the various buttons and things (I think this is most
certainly how people use it in windows). If you had keyboard shortcuts for
all your jog motions, it's possible all you would need to do is setup the
jog dial to use those key-strokes just like folks do with video editing and
things.
That would be a more fool-proof way to integrate things... i just didn't do
that with shuttleCP because my jog dial wasn't on the same machine as
Chilipeppr and chilipeppr didn't have keyboard shortcuts for everything.
FG
On Mon, Jan 4, 2016 at 9:09 AM, Cheton Wu<EMAIL_ADDRESS>wrote:

Just did a quick search and found the jog dial sold by Amazon. Is this the
one that's supported by ShuttleCP?
http://www.amazon.com/gp/product/B0032Y0OH0?psc=1&redirect=true&ref_=ox_sc_act_title_1&smid=A1YG1VZVF48PM9
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-168701530.

I would also be happy to chip in some money to get you a unit!

Thanks for your kindly help. The jog dial looks interesting and the price is affordable for me, so I'd like to buy one myself. I will keep you updated on progress once I received the package.

Send me your PayPal address and I'll send over $20. We need more CNC apps
to support how dials and get away from keyboards! :)
FG
Sent from my Android phone
On Jan 4, 2016 9:21 PM, "Cheton Wu"<EMAIL_ADDRESS>wrote:

Thanks for your kindly help. The jog dial looks interesting and the price
is affordable for me, so I'd like to buy one myself. I will keep you
updated on progress once I received the package.
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-168881599.

I created a donation button at the bottom of the README (https://github.com/cheton/cnc.js#donate). Thanks for your support on the hardware investment.

Ok, I just sent a donation. Let me know if you didn't get it.
FG
Sent from my Android phone
On Jan 5, 2016 2:34 AM, "Cheton Wu"<EMAIL_ADDRESS>wrote:

I created a donation button at the bottom of the README (
https://github.com/cheton/cnc.js#donate). Thanks for your support on the
hardware investment.
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-168934357.

I just received your payment. Thank you for making a donation.

I placed the order on Amazon on Jan 4. It will be delivered to Shipito's warehouse, and then forwarded to Taiwan. The expected arrival date might be in late January. Also thank @meikrantz for your donation.

Cool. Definitely look into using it as keyboard shortcuts first as I think
that is better and more in line with how other programs use that hardware.
I'm not sure, but I think the manufacturer has utilities to map the
keystrokes on their website.
FG
Sent from my Android phone
On Jan 5, 2016 11:36 PM, "Cheton Wu"<EMAIL_ADDRESS>wrote:

I placed the order on Amazon on Jan 4. It will be delivered to Shipito's
warehouse, and then forwarded to Taiwan. The expected arrival date might be
in late January. Also thank @meikrantz https://github.com/meikrantz for
your donation.
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-169233194.

So, before I started to integrate your shuttleCP, the first thing I can do is to provide a configuration UI to configure keyboard shortcuts that map to certain buttons (e.g. jog motions) on cnc.js. This would be the most easiest way to control the UI by ShuttleExpress jog dial.

yep, i think so... keys to switch the active axis, to switch the jog
increment, and to move positive or negative on the selected axis.  then,
for the jog wheel, you'll need to configure it somehow to just send the key
over and over at different speeds or maybe something else... not sure.  but
for all the other buttons and the main rotary wheel it should be axis
select, speed select, and move forward and move backward.
FG
On Wed, Jan 6, 2016 at 10:15 PM, Cheton Wu<EMAIL_ADDRESS>wrote:

So, before I started to integrate your shuttleCP, the first thing I can do
is to provide a configuration UI to configure keyboard shortcuts that map
to certain buttons (e.g. jog motions) on cnc.js. This would be the most
easiest way to control the UI by ShuttleExpress jog dial.
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-169543803.

Found an interesting CNC Simulator at http://cncsimulator.info/screenshots.
It provides support for ShuttleExpress as well: http://cncsimulator.info/blog/?p=413

Very cool. So are there any utilities to configure the keys for Linux?
FG
Sent from my Android phone
On Jan 9, 2016 2:57 AM, "Cheton Wu"<EMAIL_ADDRESS>wrote:

Found an interesting CNC Simulator at http://cncsimulator.info/screenshots
.
It provides support for ShuttleExpress as well:
http://cncsimulator.info/blog/?p=413
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-170211343.

Added some keyboard shortcuts to v0.15.2:
! - Feed Hold
~ - Resume
ctrl + alt + command + h - Homing
ctrl + alt + command + u - Unlock
ctrl + alt + command + r - Reset
ctrl + alt + command + = - Switch Jog Distance: 1, 0.1, 0.01, 0.001, or custom
ctrl + alt + command + [ - Jog Backward
ctrl + alt + command + ] - Jog Forward
ctrl + alt + command + x - Select/Deselect X Axis
ctrl + alt + command + y - Select/Deselect Y Axis
ctrl + alt + command + z - Select/Deselect Z Axis
My Shuttle Express Settings just look like the following:

Button 2: ctrl + alt + command + x
Button 3: ctrl + alt + command + y
Button 4: ctrl + alt + command + z
Button 5: ctrl + alt + command + =
Turn Jog Left: ctrl + alt + command + [
Turn Jog Right: ctrl + alt + command + ]

For the Shuttle Wheel support, it might need a mechanism to count the number of keystrokes per second to determine Shuttle Zone (-7 to +7) and G1 parameters like feedrate and distance. I will try to find a better way to add Shuttle Wheel support to the web interface.

Awesome.
For the wheel, you could just fake it. That's what I'm doing in that
shuttleCP. Basically setting some parameters for how far to move for each
update and then continually sending them. This could be a configurable
parameter in the jog dial settings. IIRC, I have one at the top of my code.
This is great! Can't wait to see a video of it working.
FG
Sent from my Android phone
On Jan 24, 2016 3:44 AM, "Cheton Wu"<EMAIL_ADDRESS>wrote:

Added some keyboard shortcuts to v0.15.2
https://github.com/cheton/cnc.js/releases/tag/v0.15.2:
! - Feed Hold
~ - Resume
ctrl + alt + command + h - Homing
ctrl + alt + command + u - Unlock
ctrl + alt + command + r - Reset
ctrl + alt + command + = - Switch Jog Distance: 1, 0.1, 0.01, 0.001, or
custom
ctrl + alt + command + [ - Jog Backward
ctrl + alt + command + ] - Jog Forward
ctrl + alt + command + x - Select/Deselect X Axis
ctrl + alt + command + y - Select/Deselect Y Axis
ctrl + alt + command + z - Select/Deselect Z Axis
My Shuttle Express Settings just look like the following:

Button 2: ctrl + alt + command + x
Button 3: ctrl + alt + command + y
Button 4: ctrl + alt + command + z
Button 5: ctrl + alt + command + =
Turn Jog Left: ctrl + alt + command + [
Turn Jog Right: ctrl + alt + command + ]

For the Shuttle Wheel support, it might need a mechanism to count the
number of keystrokes per second to determine Shuttle Zone (-7 to +7) and G1
parameters like feedrate and distance. I will try to find a better way to
add Shuttle Wheel support to the web interface.
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-174273251.

Okay, had a chance to test this out this evening.  It works, but performance is not that great.  Buttons select the appropriate axis, and highlight it in the GUI, which is nice.  The problem is with buffering.  You have to turn the jog wheel very slowly, or it skips quite a bit.  That would be okay with fine movements, but without shuttle wheel support for large movements it's somewhat limiting.
Good start though!  Nice to actually have the ShuttleXpress make the machine move again!

I haven't implemented the buffering mechanism, but only use a simple flag canJog to check if it needs to send commands to Grbl.
'JOG_FORWARD': () => {
    let { port, activeState } = this.props;
    let canJog = (!!port && (activeState === ACTIVE_STATE_IDLE));

    if (canJog) {
        let distance = this.getJogDistance();
        let jog = {
            'x': () => this.jog({ X: distance }),
            'y': () => this.jog({ Y: distance }),
            'z': () => this.jog({ Z: distance })
        }[this.state.selectedAxis];

        jog && jog();
    }
}

Indeed, current implementation will lose steps if current state is not Idle, that is originally developed to prevent accidentally ShuttleXpress touches while running a G-code file. That will be solved in a later version to make smooth movement, and the shuttle wheel support will be provided as well.

The latest update v0.15.4 should has resolved the jog wheel performance issue, and it also includes shuttle wheel support.
https://github.com/cheton/cnc.js/wiki/User-Guide#shuttlexpress-settings
ShuttleXpress Settings
Buttons

Button 2 - Select/Deselect X Axis
ctrl + alt + command + x
Button 3 - Select/Deselect Y Axis
ctrl + alt + command + y
Button 4 - Select/Deselect Z Axis
ctrl + alt + command + z
Button 5 - Switch Jog Distance (1, 0.1, 0.01, 0.001, or a custom value)
ctrl + alt + command + =

Jog Wheel

Turn Jog Left
ctrl + alt + command + [
Turn Jog Right
ctrl + alt + command + ]

Shuttle Wheel
Adjust the keystroke repeat rate to 10 times per second for all Shuttle Zones except the Shuttle Zone 0.

Shuttle Zone -7
ctrl + alt + shift + 7
Shuttle Zone -6
ctrl + alt + shift + 6
Shuttle Zone -5
ctrl + alt + shift + 5
Shuttle Zone -4
ctrl + alt + shift + 4
Shuttle Zone -3
ctrl + alt + shift + 3
Shuttle Zone -2
ctrl + alt + shift + 2
Shuttle Zone -1
ctrl + alt + shift + 1
Shuttle Zone 0
ctrl + alt + command + 0
Shuttle Zone 1
ctrl + alt + command + 1
Shuttle Zone 2
ctrl + alt + command + 2
Shuttle Zone 3
ctrl + alt + command + 3
Shuttle Zone 4
ctrl + alt + command + 4
Shuttle Zone 5
ctrl + alt + command + 5
Shuttle Zone 6
ctrl + alt + command + 6
Shuttle Zone 7
ctrl + alt + command + 7

Hi @lordmundi,
According to your shuttlecp.c at #L201, there is an OVERSHOOT factor for distance calculation. May I know if it is necessary while using the shuttle wheel?
Here is my code:
https://github.com/cheton/cnc.js/blob/master/src/web/components/widgets/axes/JogPad.jsx#L93

this is fantastic!!! got any videos of it working?  it's not real till we
see a video.  :)
for the overshoot, the reason is because of the acceleration / deceleration
profiles.  If you time it such that the next command begins as the previous
one ends, then the controller will ramp up, get to top speed, then start
ramping down and then eventually hit the desired mark and stop.  Then the
next command would come and it would do that again.  So you would get this
"rarr...rarr...rarr...rar...." sound (sorry to use such technical terms
here).  The idea is to get the next command in before it started
decelerating so that it would continue on at top speed and not decelerate
until the commands stopped coming in when you let go of the wheel.  I
experimented and found that an overshoot of about 6% was giving me good
results.
That's all it was.  If that doesn't make sense, let me know and I'll try to
explain it differently.
These great keystrokes and additions should make code like mine not
needed... this is really the better way to do things.  Plus, this should
work for any OS i believe.
FG
On Mon, Feb 1, 2016 at 5:32 AM, Cheton Wu<EMAIL_ADDRESS>wrote:

Hi @lordmundi https://github.com/lordmundi,
According to your shuttlecp.c at #L201
https://github.com/lordmundi/shuttleCP/blob/master/shuttlecp.c#L201,
there is an OVERSHOOT factor for distance calculation. May I know if it is
necessary while using the shuttle wheel?
Here is my code:
https://github.com/cheton/cnc.js/blob/master/src/web/components/widgets/axes/JogPad.jsx#L93
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-177927682.

I just connected it to my homemade CNC machine, and it sounds so horribl just like what you mentioned.
(rarr...rarr...rarr... :scream_cat: :scream: )  XDDD

However, giving a small amount of overshoot factor (e.g. 1.06~1.2) did not help much for my test. I may need to figure out where I can improve to eliminate deceleration between subsequent commands.

haha... yeah, i think it took me a bit to calculate just how far to command
for each "cycle" of the software, which for you will depend on the
keystroke frequency I suppose.  Then, you just need to tune that amount to
have a certain amount of overshoot which is dependent on the speed.
Feel free to steal any of my code that helps.
FG
On Mon, Feb 1, 2016 at 9:54 AM, Cheton Wu<EMAIL_ADDRESS>wrote:

I just connected it to my homemade CNC machine, and it sounds so horribl
just like what you mentioned.
(rarr...rarr...rarr... [image: :scream_cat:] [image: :scream:] ) XDDD
[image: CNC] https://www.youtube.com/watch?v=hm6ola-UEIw
However, giving a small amount of overshoot factor (e.g. 1.06~1.2) did not
help much for my test. I may need to figure out where I can improve to
eliminate deceleration between subsequent commands.
—
Reply to this email directly or view it on GitHub
https://github.com/cheton/cnc.js/issues/20#issuecomment-178039791.

I guess a buffering delay solution plus the overshoot factor might work by accumulating each small steps and use a large flush interval, may be a 0.5s time interval or longer. I will make more tests to find a way that can significantly reduce frequent deceleration.

I have an interesting finding. I eventually discovered that the sharp sound during acceleration and deceleration can be significantly reduced with a lower feed rate (mm/min) and a higher acceleration speed (mm/sec^2).
My NEMA 23 (57mm) 3V stepper motor can run in full speed without missing steps with following max rate and acceleration settings:
$110=5000.000 (x max rate, mm/min)
$111=5000.000 (y max rate, mm/min)
$112=500.000 (z max rate, mm/min)
$120=50.000 (x accel, mm/sec^2) // Grbl's default value is 10.000 (mm/sec^2)
$121=50.000 (y accel, mm/sec^2)
$122=50.000 (z accel, mm/sec^2)

After several tests, I found my optimal values by setting the maximum feed rate from 1500 to 800 (mm/min) and increasing the acceleration speed from 10 to 50 (mm/sec^2), and never heard the humming noise. For example:
> G91 G1 F200 X1.33  # Shuttle Zone 1
> G90
   :   :   :
> G91 G1 F800 X4  # Shuttle Zone 7
> G90
ok

Since the settings may show different results between different hardwares, I may provide a popup menu for configuring min/max feed rate value (e.g. 200~800) for Shuttle Wheel to meet different needs.

The sample test code: ShuttleZone.jsx
https://github.com/cheton/cnc.js/blob/master/src/web/components/widgets/axes/ShuttleZone.jsx#L30

so does that mean you are adjusting your grbl config parameters?  or are you adjusting the max feed rate being commanded in the GCODE?
if you have an amount of overshoot that you can configure, then that would give the user one single "knob" to adjust to get the motion of their machine smooth without too much extra travel.

May I know your Grbl system settings? The overshoot factor didn't help much with my CNC machine,
so I adjusted both the x/y/z acceleration speed (in Grbl) and the max feed rate (e.g. 800 mm/min) sent with the G-code, and finally get the expected result. For my test, the timing control is handled by web browser, which is different from ShuttleCP, I guessed it might be related to some timing accuracy issues between web browser and the server.
I'd also like to add overshoot factor as a configurable option, it should help with different scenarios.

I haven't had a chance to do more testing, but wanted to share my grbl settings.  In order to get smooth movement on my machine, I greatly increased the max rate and acceleration.  Looking forward to getting back to the shop to further test the shuttle!
$110=10000.000 (x max rate, mm/min)
$111=10000.000 (y max rate, mm/min)
$112=1500.000 (z max rate, mm/min)
$120=300.000 (x accel, mm/sec^2)
$121=300.000 (y accel, mm/sec^2)
$122=150.000 (z accel, mm/sec^2)

The latest update v0.15.6 should have fixed the issue of humming noise when scrolling the Shuttle Wheel. You can configure feed rate range, repeat rate, and distance overshoot by clicking the Edit button at the top of the Axes widget.
I also made an update to the wiki: Reduce humming sound when accelerating and decelerating
Let me know if you have any suggestions for further improvements.

Finally had a chance to get in the shop and do some testing with 0.15.8 and ShuttleXpress.  It works great!  Haven't tried tweaking the overshoot and acceleration parameters, but movement is very smooth.

It's great to hear that. :smile:

Hello, everyone,
I'm really sorry to bring up this old issue again. But I have some general questions about the Shuttle Express...
I have CNCJS running on a Raspberry 3.
Am I right in assuming that I simply plug the ShuttleExpress into the server USB port and don't have to open the browser window from the client PC? Do I need any drivers to control the Shuttle?
Or does it only work under Windows in the browser with the ShuttleExpress software? Like this:

I already tried to check it on Facebook but I don't think anyone understands me :)
I hope someone has an answer.
Regards Christoph

According to my reading of the code, the Shuttle Express support is in the client-side - browser - code, not in the server code.  The Shuttle Express feeds simulated keystrokes to the browser-based UI, which then interprets them and sends control commands to the server over the network connection. Plugging it into the computer that runs the CNCjs server - unless the browser is running on the same computer - would not be effective.
I expect that the reason why you are getting little response on the Facebook page is because very few CNCjs users have a Shuttle Express.  I certainly don't have one.

@slimline33
I agree to @MitchBradley . I have a ShuttleExpress. One of the most important thing is, it doesn't send its keystrokes unless you configure it to do so. As far as my knowledge, it supports windows and mac only (I mean its configuration software). That means, that software always runs in background while you use shuttleexpress from windows or mac client and maps your inputs to relevant keystrokes. For Linux (client), I had no success with my little efforts but there might be some short of workaround to map it with keystrokes or some efforts are required to do so.
In short, you can use shuttleexpress with windows and mac running browsers (as clients) while your server part is in RPi3. I hope this is helpful.

Hello @MitchBradley and @karoria . Thank you so much for your answers.
That's what I suspected. But I wonder why in CNCJS settings there is a possibility to set the feedrate and the repeat.
What surprised me is that @Lordmundi described (https://github.com/cncjs/cncjs/issues/20#issuecomment-168703235) that he runs the SHUTTLE EXPRESS on the same machine as CNCJS. I think he means that he also controls the machine via the servers browser.
Unfortunately I have my problems with the Windows keys under Windows 10 when working with the Shuttle Express via the client. For example he wants to open Microsoft OneDrive for me and so on. I would not like to disable the key shortcuts under Windows 10. Who knows if this will remain until the next update.

just to clear up any confusion, I'm not running CNCJS, but I did write some code that read the shuttle express on linux and output commands to some other cnc utilities.  I think you could adapt it if you want.  That project is here: https://github.com/lordmundi/shuttleCP

Hi @lordmundi
Thank you for your answer.
I already know your code. But I did not understand the context of this issue. I assumed that @cheton had implemented parts from your code in CNCJS. I must have misunderstood that!


Implement Application Error Page

Description
Implement Application Error Page.
Ticket: 86b16rzhu
Type of change
New feature (non-breaking change which adds functionality)
How Has This Been Tested?

[x] Test A - Test the page appear correctly on browsers (Safari, Edge, Chrome, Firefox).

@wei4r This PR should be created with all the features done on the expedition, no need to create a PR for each feature in the expedition you are working on


[Bug] 'AutoSlim' object has no attribute 'channel_cfg'

Describe the bug
Refer to Pruning-Split Checkpoint(Optional) for the following command:
$ python ./tools/model_converters/split_checkpoint.py \
>   configs/pruning/autoslim/autoslim_mbv2_supernet_8xb256_ci10.py \
>   output/epoch_49.pth \
>   --channel-cfgs output/subnet_497881198.yaml,output/subnet_297081551.yaml,output/subnet_197465826.yaml
load checkpoint from local path: output/epoch_49.pth
Traceback (most recent call last):
  File "./tools/model_converters/split_checkpoint.py", line 69, in <module>
    main()
  File "./tools/model_converters/split_checkpoint.py", line 60, in main
    algorithm.channel_cfg[i])
  File "/home/tanghuayang/venv_torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 948, in __getattr__
    type(self).__name__, name))
AttributeError: 'AutoSlim' object has no attribute 'channel_cfg'

Refer to document when you retrain a slimmable model, your command will be like --cfg-options algorithm.channel_cfg=cfg1,cfg2,cfg3
so --cfg-options algorithm.channel_cfg is missing in your command.

mentioned in the documentation:
For example, when you retrain a slimmable model, your command will be like --cfg-options algorithm.channel_cfg=cfg1,cfg2,cfg3.
 The order of them should be the same.
and there is no algorithm.channel_cfg in the example given.

Hi, tanghy2016. Maybe you can refer to this example.
There are five steps to complete the autoslim pipeline.

After step2, you will get some YAML files, which represent different network structure.
In step3,  you can choose some models with different flops such as 200M flops, 300M flops and 500M flops, and retrain them on imagenet.
Note that the YAML files used in step4 should be the same as in step3.

Yes, in my experiment, I have gone to step 4, and as you can see from our execution command, it contains 3 YAML file inputs, and these 3 files are the output of step 2

Hi, @tanghy2016 . You can try the following commands, which changed the comma to a space.
python ./tools/model_converters/split_checkpoint.py \
   configs/pruning/autoslim/autoslim_mbv2_supernet_8xb256_ci10.py \
   output/epoch_49.pth \
   --channel-cfgs output/subnet_497881198.yaml output/subnet_297081551.yaml output/subnet_197465826.yaml

By the way, why is the accuracy of step 1 relatively low in my two experiments?
The first experiment used the CIFAR10 dataset, scaled to 224, and used mobilenet-v2. After step 1, the accuracy rate was only about 15%.
The second experiment uses the 2-category data set prepared by myself, which is also scaled to 224. Using mobilenet-v2, after step 1, the accuracy rate is only about 50%.
The configuration files of the two experiments refer to autoslim_mbv2_supernet_8xb256_in1k.py, and no other changes have been made except to modify the path of the dataset.

same error: AttributeError: 'AutoSlim' object has no attribute 'channel_cfg'

Hi, @tanghy2016 . You can try the following commands, which changed the comma to a space.
python ./tools/model_converters/split_checkpoint.py \
   configs/pruning/autoslim/autoslim_mbv2_supernet_8xb256_ci10.py \
   output/epoch_49.pth \
   --channel-cfgs output/subnet_497881198.yaml output/subnet_297081551.yaml output/subnet_197465826.yaml

same error: AttributeError: 'AutoSlim' object has no attribute 'channel_cfg'

May you should change --channel-cfg to --channel-cfgs.

I use --channel-cfgs in my execution command, do you have time to reproduce my experiment? My configuration is as follows:
# copy autoslim_mbv2_supernet_8xb256_in1k.py to autoslim_mbv2_supernet_8xb256_ci10.py
# modify the configuration in autoslim_mbv2_supernet_8xb256_ci10.py as follows:

_base_ = [
    '../../_base_/datasets/mmcls/cifar10_bs16.py',
    '../../_base_/schedules/mmcls/cifar10_bs128.py',
    '../../_base_/mmcls_runtime.py'
]

model = dict(
  # ...
  head=dict(
    # ...
    num_classes=10,
    # ...
  )
  # ...
)

# 将configs/_base_/datasets/mmcls/cifar10_bs16.py中的:
dict(type='RandomCrop', size=32, padding=4),
# 修改为:
dict(type='RandomResizedCrop', size=224),
# 以符合mobilenet v2的输入要求

Refer to issues 138
# add ```optimizer_config = None``` into ```configs/pruning/autoslim/autoslim_mbv2_supernet_8xb256_ci10.py```

_base_ = [
    '../../_base_/datasets/mmcls/cifar10_bs16.py',
    '../../_base_/schedules/mmcls/cifar10_bs128.py',
    '../../_base_/mmcls_runtime.py'
]

+ optimizer_config = None

Hi tanghy2016, do you set retraining to True in algorithm config?

algorithm = dict(
	# ...
	retraining=True
	# ...
)

By the way, why is the accuracy of step 1 relatively low in my two experiments? The first experiment used the CIFAR10 dataset, scaled to 224, and used mobilenet-v2. After step 1, the accuracy rate was only about 15%. The second experiment uses the 2-category data set prepared by myself, which is also scaled to 224. Using mobilenet-v2, after step 1, the accuracy rate is only about 50%. The configuration files of the two experiments refer to autoslim_mbv2_supernet_8xb256_in1k.py, and no other changes have been made except to modify the path of the dataset.

Hi, tanghy2016. The official codebase didn't release the code about training on CIFAR10. We have not attempted to train AutoSlim on the CIFAR10 dataset.
According to the accuracy you reported, we conclude that the supernet failed to converge. Can you find some examples of MobileNetv2 trained on CIFAR10? You can try to modify our supernet to adapt it to CIFAR10 dataset.

CIFAR10 is used in the paper and results are given

Although the results on CIFAR10 are given in the paper, there are still many things that have not been released, such as the model structure searched on CIFAR10.
Maybe you can follow the 32x32 to continue the next steps to verify the accuracy.
The result of the first step on imagenet is only 50%~60%, but the final result can reach more than 70%.

Hi tanghy2016, do you set retraining to True in algorithm config?
algorithm = dict(
	# ...
	retraining=True
	# ...
)

Thank you very much, the problem has been solved

Although the results on CIFAR10 are given in the paper, there are still many things that have not been released, such as the model structure searched on CIFAR10. Maybe you can follow the 32x32 to continue the next steps to verify the accuracy. The result of the first step on imagenet is only 50%~60%, but the final result can reach more than 70%.

The experimental results show that the accuracy rate of subnet is 77%, which is far from the paper. ಥ_ಥ

Although the results on CIFAR10 are given in the paper, there are still many things that have not been released, such as the model structure searched on CIFAR10. Maybe you can follow the 32x32 to continue the next steps to verify the accuracy. The result of the first step on imagenet is only 50%~60%, but the final result can reach more than 70%.

The experimental results show that the accuracy rate of subnet is 77%, which is far from the paper. ಥ_ಥ

Is the size of the weight files obtained by split_checkpoint.py all the same?


[LayoutConstraints] Unable to simultaneously satisfy constraints.

I added the component to my project and it works pretty good. The only problem are these constraint errors. Anybody has the same issue and knows if what's causing it?
	Probably at least one of the constraints in the following list is one you don't want.
	Try this:
		(1) look at each constraint and try to figure out which you don't expect;
		(2) find the code that added the unwanted constraint or constraints and fix it.
	(Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints)
(
    "<NSAutoresizingMaskLayoutConstraint:0x79ea45b0 h=-&- v=-&- Fusuma.FSVideoCameraView:0x7b971e50.width == UIView:0x7baba980.width - 14   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x79ea46d0 h=-&- v=-&- Fusuma.FSVideoCameraView:0x7b971e50.height == UIView:0x7baba980.height - 91   (active)>",
    "<NSLayoutConstraint:0x79eb5b60 UIView:0x79eb6090.width == UIView:0x79eb6090.height   (active)>",
    "<NSLayoutConstraint:0x79ebb460 V:|-(50)-[UIView:0x79eb6090]   (active, names: '|':Fusuma.FSVideoCameraView:0x7b971e50 )>",
    "<NSLayoutConstraint:0x79ebb530 H:|-(0)-[UIView:0x79eb6090]   (active, names: '|':Fusuma.FSVideoCameraView:0x7b971e50 )>",
    "<NSLayoutConstraint:0x79ebcf60 H:[UIView:0x79eb6090]-(0)-|   (active, names: '|':Fusuma.FSVideoCameraView:0x7b971e50 )>",
    "<NSLayoutConstraint:0x79ebd040 V:[UIView:0x79eb6170]-(0)-|   (active, names: '|':Fusuma.FSVideoCameraView:0x7b971e50 )>",
    "<NSLayoutConstraint:0x79ebd120 V:[UIView:0x79eb6090]-(0)-[UIView:0x79eb6170]   (active)>",
    "<NSLayoutConstraint:0x7bad3920 UIButton:0x7bad3780.height == 45   (active)>",
    "<NSLayoutConstraint:0x7babb020 H:[UIView:0x7baba7e0]-(0)-|   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb0d0 H:|-(0)-[UIView:0x7baba7e0]   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb430 UIView:0x7baba980.leading == UIView:0x7baba7e0.leading   (active)>",
    "<NSLayoutConstraint:0x7babb4c0 V:|-(0)-[UIView:0x7baba980]   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb530 UIView:0x7baba980.trailing == UIView:0x7baba7e0.trailing   (active)>",
    "<NSLayoutConstraint:0x7babb5a0 UIView:0x7baba980.bottom == UIView:0x7baba7e0.bottom   (active)>",
    "<NSLayoutConstraint:0x7babb7e0 V:[UIButton:0x7bad3780]-(0)-|   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babba90 V:[UIView:0x7baba7e0]-(0)-[UIButton:0x7bad3780]   (active)>",
    "<NSLayoutConstraint:0x79ebacd0 'UIView-Encapsulated-Layout-Height' UIView:0x7babaab0.height == 600   (active)>",
    "<NSLayoutConstraint:0x79ebac60 'UIView-Encapsulated-Layout-Width' UIView:0x7babaab0.width == 600   (active)>"
)

Will attempt to recover by breaking constraint
<NSLayoutConstraint:0x79eb5b60 UIView:0x79eb6090.width == UIView:0x79eb6090.height   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKit/UIView.h> may also be helpful.
2017-01-24 19:46:26.985780 Bringme[83100:3182779] [LayoutConstraints] Unable to simultaneously satisfy constraints.
	Probably at least one of the constraints in the following list is one you don't want.
	Try this:
		(1) look at each constraint and try to figure out which you don't expect;
		(2) find the code that added the unwanted constraint or constraints and fix it.
	(Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints)
(
    "<NSAutoresizingMaskLayoutConstraint:0x79eb14c0 h=-&- v=-&- Fusuma.FSCameraView:0x7b9c1bd0.width == UIView:0x7bab3770.width - 14   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x79eb15e0 h=-&- v=-&- Fusuma.FSCameraView:0x7b9c1bd0.height == UIView:0x7bab3770.height - 91   (active)>",
    "<NSLayoutConstraint:0x7b987260 UIView:0x7b9b1200.height >= 100   (active)>",
    "<NSLayoutConstraint:0x7b982b30 V:|-(50)-[UIView:0x7b9b7f60]   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b9826e0 H:[UIView:0x7b9b7f60]-(0)-|   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b97f840 H:|-(0)-[UIView:0x7b9b7f60]   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b97ec00 V:[UIView:0x7b9b1200]-(0)-|   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b97da30 V:[UIView:0x7b9b7f60]-(0)-[UIView:0x7b9b1200]   (active)>",
    "<NSLayoutConstraint:0x7bad3920 UIButton:0x7bad3780.height == 45   (active)>",
    "<NSLayoutConstraint:0x7babb020 H:[UIView:0x7baba7e0]-(0)-|   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb0d0 H:|-(0)-[UIView:0x7baba7e0]   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb210 V:|-(0)-[UIView:0x7bab3770]   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb2a0 UIView:0x7bab3770.trailing == UIView:0x7baba7e0.trailing   (active)>",
    "<NSLayoutConstraint:0x7babb310 UIView:0x7bab3770.leading == UIView:0x7baba7e0.leading   (active)>",
    "<NSLayoutConstraint:0x7babb380 UIView:0x7bab3770.bottom == UIView:0x7baba7e0.bottom   (active)>",
    "<NSLayoutConstraint:0x7babb7e0 V:[UIButton:0x7bad3780]-(0)-|   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babba90 V:[UIView:0x7baba7e0]-(0)-[UIButton:0x7bad3780]   (active)>",
    "<NSLayoutConstraint:0x7bab7d60 UIView:0x7b9b7f60.height == UIView:0x7b9b7f60.width   (active)>",
    "<NSLayoutConstraint:0x79ebacd0 'UIView-Encapsulated-Layout-Height' UIView:0x7babaab0.height == 600   (active)>",
    "<NSLayoutConstraint:0x79ebac60 'UIView-Encapsulated-Layout-Width' UIView:0x7babaab0.width == 600   (active)>"
)

Will attempt to recover by breaking constraint
<NSLayoutConstraint:0x7b987260 UIView:0x7b9b1200.height >= 100   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKit/UIView.h> may also be helpful.
2017-01-24 19:46:26.987236 Bringme[83100:3182779] [LayoutConstraints] Unable to simultaneously satisfy constraints.
	Probably at least one of the constraints in the following list is one you don't want.
	Try this:
		(1) look at each constraint and try to figure out which you don't expect;
		(2) find the code that added the unwanted constraint or constraints and fix it.
	(Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints)
(
    "<NSAutoresizingMaskLayoutConstraint:0x79eb14c0 h=-&- v=-&- Fusuma.FSCameraView:0x7b9c1bd0.width == UIView:0x7bab3770.width - 14   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x79eb15e0 h=-&- v=-&- Fusuma.FSCameraView:0x7b9c1bd0.height == UIView:0x7bab3770.height - 91   (active)>",
    "<NSLayoutConstraint:0x7b982b30 V:|-(50)-[UIView:0x7b9b7f60]   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b9826e0 H:[UIView:0x7b9b7f60]-(0)-|   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b97f840 H:|-(0)-[UIView:0x7b9b7f60]   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b97ec00 V:[UIView:0x7b9b1200]-(0)-|   (active, names: '|':Fusuma.FSCameraView:0x7b9c1bd0 )>",
    "<NSLayoutConstraint:0x7b97da30 V:[UIView:0x7b9b7f60]-(0)-[UIView:0x7b9b1200]   (active)>",
    "<NSLayoutConstraint:0x7bad3920 UIButton:0x7bad3780.height == 45   (active)>",
    "<NSLayoutConstraint:0x7babb020 H:[UIView:0x7baba7e0]-(0)-|   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb0d0 H:|-(0)-[UIView:0x7baba7e0]   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb210 V:|-(0)-[UIView:0x7bab3770]   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babb2a0 UIView:0x7bab3770.trailing == UIView:0x7baba7e0.trailing   (active)>",
    "<NSLayoutConstraint:0x7babb310 UIView:0x7bab3770.leading == UIView:0x7baba7e0.leading   (active)>",
    "<NSLayoutConstraint:0x7babb380 UIView:0x7bab3770.bottom == UIView:0x7baba7e0.bottom   (active)>",
    "<NSLayoutConstraint:0x7babb7e0 V:[UIButton:0x7bad3780]-(0)-|   (active, names: '|':UIView:0x7babaab0 )>",
    "<NSLayoutConstraint:0x7babba90 V:[UIView:0x7baba7e0]-(0)-[UIButton:0x7bad3780]   (active)>",
    "<NSLayoutConstraint:0x7bab7d60 UIView:0x7b9b7f60.height == UIView:0x7b9b7f60.width   (active)>",
    "<NSLayoutConstraint:0x79ebacd0 'UIView-Encapsulated-Layout-Height' UIView:0x7babaab0.height == 600   (active)>",
    "<NSLayoutConstraint:0x79ebac60 'UIView-Encapsulated-Layout-Width' UIView:0x7babaab0.width == 600   (active)>"
)

Will attempt to recover by breaking constraint
<NSLayoutConstraint:0x7bab7d60 UIView:0x7b9b7f60.height == UIView:0x7b9b7f60.width   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKit/UIView.h> may also be helpful.

I just installed the pod to try it out and I have exactly the same issue. Any fix so far?

PR-156

Here is a recent solution for Cordova Apache people:
https://storksnestblog.wordpress.com/2018/10/16/solving-obscure-keyboard-fail-in-cordova/


Reading hdf5 files

Hello,
Thanks a lot for making the dataset available. I'm really looking forward to trying to work with it.
I have just started looking at the data and am having trouble reading the images from cropped_256.hdf5. Perhaps this is just due to my own unfamiliarity with hdf5, but it seems like the images are 1 dimensional and that dimension does not seem necessarily divisible by 256.
I am reading the images in as follows:
import numpy as np

with h5py.File('cropped_256.hdf5','r+') as hdf5:
    img = np.array(hdf5['bioscan_dataset']['3652597.jpg'])
    print(img.shape)
    print(img.shape / 256)

and I get the following output:

(7633,)
29.81640625

I have tried with a few different images and I get similar results. For now I'll use the zip files so this probably isn't a barrier for many, but if I am doing something wrong in the reading of the hdf5 data, maybe a how-to in the readme would be a good idea.
Thanks again for the dataset!

Hi,
Please use the utilts.py from code repository, and the function which reads from hdf5 file.
Below is a sample code, adjust it accordingly.
file_name = 'path_to_hdf5_file/cropped_256.hdf5'
hdf5 = h5py.File(file_name, 'r')
hdf5_dataset = hdf5['bioscan_dataset']
image_names = list(hdf5_dataset.keys())
for img_name in image_names:
image = read_from_hdf5(hdf5_dataset, img_name, saved_as_binary=True)
image.show()
Best
/Zahra

Zahra Gharaee
Postdoctoral Research Fellow
Vision and Image Processing Lab
East Campus (EC4),University of Waterloo
200 University Avenue West
Waterloo, ON, Canada N2L 3G1

From: Roberta Hunt @.>
Sent: Friday, September 29, 2023 7:31 AM
To: zahrag/BIOSCAN-1M @.>
Cc: Subscribed @.***>
Subject: [zahrag/BIOSCAN-1M] Reading hdf5 files (Issue #3)
Hello,
Thanks a lot for making the dataset available. I'm really looking forward to trying to work with it.
I have just started looking at the data and am having trouble reading the images from cropped_256.hdf5. Perhaps this is just due to my own unfamiliarity with hdf5, but it seems like the images are 1 dimensional and that dimension does not seem necessarily divisible by 256.
I am reading the images in as follows:
import numpy as np
with h5py.File('cropped_256.hdf5','r+') as hdf5:
img = np.array(hdf5['bioscan_dataset']['3652597.jpg'])
print(img.shape)
print(img.shape / 256)
and I get the following output:
(7633,)
29.81640625
I have tried with a few different images and I get similar results. For now I'll use the zip files so this probably isn't a barrier for many, but if I am doing something wrong in the reading of the hdf5 data, maybe a how-to in the readme would be a good idea.
Thanks again for the dataset!
—
Reply to this email directly, view it on GitHubhttps://github.com/zahrag/BIOSCAN-1M/issues/3, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB6XMTOGYXWVWPRHJLVFBUDX42WRNANCNFSM6AAAAAA5MKRE4Y.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

Works like a charm. Thanks so much for sorting that out!


⚠ Bofrost.nl has degraded performance

In 1ce4e8b, Bofrost.nl (https://www.bofrost.nl) experienced degraded performance:

HTTP code: 200
Response time: 1141 ms

Resolved: Bofrost.nl performance has improved in c0623f2.


fix: augment vue rather than @vue/runtime-core

Problem: after https://github.com/nuxt/nuxt/pull/28446, the mixing of module augmentation causes issues for modules downstream like vue-to-print which cause typescript intellisense to break completely (e.g. in vscode).
Solution: this PR removes augmentations of @vue/runtime-core in favour of only augmenting vue (the new recommendation), which should fix issues when other packages (like vue-to-print) are only augmenting vue (see https://github.com/nuxt/nuxt/pull/28542)

Hey @siaikin ! Thanks for your library! Recent nuxt upstream changes broke the types. Could we please get a new release with this fix 🙏🏼

Thank you very much for your pr🥳.
I'm sorry that I didn't reply soon, because of the jet lag, I just got up now🥱.
I will check your pr and merge it immediately.
发自我的 iPhone
在 2024年8月23日，04:21，miguelrk @.***> 写道：

Hey @siaikinhttps://github.com/siaikin ! Thanks for your library! Recent nuxt upstream changes broke the types. Could we please get a new release with this fix 🙏🏼
—
Reply to this email directly, view it on GitHubhttps://github.com/siaikin/vue-to-print/pull/8#issuecomment-2305562991, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF42KDAYWVKZY6VHNXXA2MLZSZB47AVCNFSM6AAAAABM6JSAGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBVGU3DEOJZGE.
You are receiving this because you were mentioned.Message ID: @.***>

:tada: This PR is included in version 1.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[ADAP-505] [Bug] v1.5 breaks models with descriptions of BigQuery struct columns

Is this a new bug in dbt-bigquery?

[X] I believe this is a new bug in dbt-bigquery
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
I am not able to add descriptions to STRUCT-columns in DBT v1.5.
Expected Behavior
I should be able to add descriptions to STRUCT-columns and their nested columns, as I was able to do in versions prior to 1.5.
Steps To Reproduce
# models/bugged.sql
{{ config(materialized='table') }}

SELECT
  STRUCT( "complete" AS status ) AS nested

# models/bugged.yml
version: 2

models:
  - name: bugged
    columns:
      - name: nested
        description: "Describe the nested struct"

Run dbt run --full-refresh --select bugged
Relevant log output
08:35:55  Running with dbt=1.5.0
08:35:56  Unable to do partial parsing because saved manifest not found. Starting full parse.
08:36:01  Found 1 models, 0 tests, 0 snapshots, 0 analyses, 0 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
08:36:01
08:36:03  Concurrency: 16 threads (target='dev')
08:36:03
08:36:03  1 of 1 START sql table model my_dataset.bugged ............................ [RUN]
08:36:07  Unhandled error while executing target/run/my_model/models/bugged.sql
400 PATCH https://bigquery.googleapis.com/bigquery/v2/projects/my-project/datasets/my_dataset/tables/bugged?prettyPrint=false: Policy tags can only be attached to a leaf field.
08:36:07  1 of 1 ERROR creating sql table model my_dataset.bugged ................... [ERROR in 3.55s]
08:36:07
08:36:07  Finished running 1 table model in 0 hours 0 minutes and 5.80 seconds (5.80s).
08:36:07
08:36:07  Completed with 1 error and 0 warnings:
08:36:07
08:36:07  400 PATCH https://bigquery.googleapis.com/bigquery/v2/projects/my-project/datasets/my_dataset/tables/bugged?prettyPrint=false: Policy tags can only be attached to a leaf field.
08:36:07
08:36:07  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

### Environment

```markdown
- OS: MacOS Ventura 13.3.1 (22E261)
- Python: Python 3.11.3
- dbt:

Core:
  - installed: 1.5.0
  - latest:    1.5.0 - Up to date!

Plugins:
  - bigquery: 1.5.0 - Up to date!

### Additional Context

Probably related to https://github.com/dbt-labs/dbt-bigquery/issues/673, but that is a feature request for data_type and contract enforcment, this bug-issue is a breaking change.

Markdown removed the link to the possibly related issue, so adding it here for links: https://github.com/dbt-labs/dbt-bigquery/issues/673

It appears this config is also required to reproduce the error:
# dbt_project.yml
models:
    my_model:
      +persist_docs:
        columns: true

Just found out we have the same issue.
This is a huge breaking change as we have a lot of nested columns documented.

#687
This one is related

Thanks for reaching out @bendiktv2 and @StoryFabian !
This is the same as https://github.com/dbt-labs/dbt-bigquery/issues/687, so I'm going to close this in favor of that one.
There is a PR to fix this that you can track along with: https://github.com/dbt-labs/dbt-bigquery/pull/688


README.md broken link

The "install the operators" phrase in the README.me file points to the What is SageMaker page, it's been redirected to an unhelpful link.

Hi @dferguson992, thanks for reporting the issue. Looks like this PR: https://github.com/awsdocs/amazon-sagemaker-developer-guide/pull/189 changed the name of pages leading to broken links. Please refer to this AWS docs page while we work on fixing it: https://docs.aws.amazon.com/sagemaker/latest/dg/kubernetes-sagemaker-operators.html.
On a side note, are you starting a new project or are you an existing user of this controller?
The reason I ask this is because if this is a new project, I highly recommend using ACK framework based sagemaker-controller which has the latest features and is the new version of this operator. Here is a tutorial to get started: https://aws-controllers-k8s.github.io/community/docs/tutorials/sagemaker-example/


Ivy and directives - "get" of undefined

After updating to @ngneat/until-destroy I've noticed that our tests no longer work when @UntilDestroy is used on a directive.
Error:
 FAIL  apps/play/src/app/one.directive.spec.ts
  ● Test suite failed to run

    TypeError: Cannot read property 'get' of undefined

       7 |   selector: '[playOne]'
       8 | })
    >  9 | export class OneDirective {
         |                          ^
      10 |   constructor() {
      11 |     interval(1000).pipe(untilDestroyed(this)).subscribe(console.log);
      12 |   }

      at decorateDirectiveJIT (../../node_modules/@ngneat/until-destroy/bundles/ng:/@ngneat/until-destroy/lib/until-destroy.ts:77:65)
      at decorateDirective (../../node_modules/@ngneat/until-destroy/bundles/ng:/@ngneat/until-destroy/lib/until-destroy.ts:103:5)
      at ../../node_modules/@ngneat/until-destroy/bundles/ng:/@ngneat/until-destroy/lib/until-destroy.ts:116:7
      at Object.__decorate (../../node_modules/tslib/tslib.js:95:99)
      at Object.<anonymous> (src/app/one.directive.ts:9:26)
      at Object.<anonymous> (src/app/one.directive.spec.ts:1:1)

Basic code:
import { Directive } from '@angular/core';
import { UntilDestroy, untilDestroyed } from '@ngneat/until-destroy';
import { interval } from 'rxjs';

@UntilDestroy()
@Directive({
  selector: '[playOne]'
})
export class OneDirective {
  constructor() {
    interval(1000).pipe(untilDestroyed(this)).subscribe(console.log);
  }
}

Basic reproduction: https://github.com/intellix/ngneat-take-until-directive-repro/commit/64efd59bfd061fe41223aa8ed5bfd5bbc484cd3e
Probably related: https://github.com/ngneat/until-destroy/issues/86

It's not working because Ivy is disabled:

If I run ngcc then it works:

Please, search among other issues before opening a new one...
There already exist issues with the same problem.
Let's respect each other, I can't always waste my time for cloning each repo and testing it locally... :disappointed:

Apologies for wasting your time. It's literally a brand new project installed with Angular 9. I didn't even know there was a private ivyEnabled flag you could import to check for that.
Why would a newly installed Angular 9 project not use ivy during tests and why would I need to opt into that? It's enabled by default but not in tests?

https://github.com/ngneat/until-destroy/issues/54#issuecomment-586980049


node: stress_process: specifiy -schema replication(strategy=NetworkTopologyStrategy) by default

Currently stress creates the keyspace with SimpleReplication which is deprecated and inhibits tablets.
This change passes the strategy=NetworkTopologyStrategy replication option by default, if it is not otherwise specified.

For example, see the test_refresh_and_restart_after_compaction_strategy_change and test_reshard_after_compaction_strategy_and_smp_change dtests:
        node1.stress(["write", "n=0", "no-warmup", "-schema", "replication(factor=1)", "-rate", "threads=1"])

@fruch what puthin version do we use ccm ci?
So we need backward compatibility?

@fruch what puthin version do we use ccm ci? So we need backward compatibility?

as the CI says, we support python3.8-3.12 (same compatibility of the python-driver that use ccm in it's tests)
so no usage of walrus operator

@bhalevy
few comments:

soon we'll need to extract the c-s usage out of ccm - i.e. once c-s won't be part of unified package - and it would be packaged on it's own
lets raise an issue in java-tools repo (until c-s would be on it's own repo) to do this on it's own by default

@bhalevy
also, why aren't we fixing the dtest to do what we expect ? wouldn't it be easier to reason about ?

@bhalevy
also, why aren't we fixing the dtest to do what we expect ? wouldn't it be easier to reason about ?

Not all dtest specify the schema for c-stress as they don't care.
We can do the wrapping in dtest rather than in ccm, but currently we still call node.stress(), that already manipulates the stress command line args, so it seems like the right place to do so.

fixed the match and replace algorithm
and searching for "NetworkTopologyStrategy" in opt

@bhalevy
few comments:

soon we'll need to extract the c-s usage out of ccm - i.e. once c-s won't be part of unified package - and it would be packaged on it's own
lets raise an issue in java-tools repo (until c-s would be on it's own repo) to do this on it's own by default

https://github.com/scylladb/scylla-tools-java/issues/400

BTW, for dtest I'm not sure we need to use cassandra-stress at all.
We could other tools, or just provide a simple load-generator in python that would use the python driver.
I don't think there's anything in particular in scylla-stress that we need in dtest...

Seen a problem in e.g. test_nodetool_snapshot_during_major_compaction.
 Each option/command can be specified at most once.

Closing now that it's the default in cassandra-stress


Remove unnecessary goroutine

As title.

Codecov Report

Merging #16080 (cc249d9) into main (ee5e1c4) will increase coverage by 0.00%.
The diff coverage is 50.00%.

@@           Coverage Diff           @@
##             main   #16080   +/-   ##
=======================================
  Coverage   44.06%   44.07%
=======================================
  Files         683      683
  Lines       82358    82357    -1
=======================================
+ Hits        36295    36301    +6
+ Misses      40176    40160   -16
- Partials     5887     5896    +9

Impacted Files
Coverage Δ

services/pull/check.go
25.51% <50.00%> (-0.52%)
:arrow_down:

modules/indexer/stats/db.go
42.85% <0.00%> (-7.15%)
:arrow_down:

modules/git/repo_base_nogogit.go
82.85% <0.00%> (-2.86%)
:arrow_down:

modules/git/utils.go
62.50% <0.00%> (-2.78%)
:arrow_down:

models/error.go
39.05% <0.00%> (-0.48%)
:arrow_down:

services/pull/pull.go
42.92% <0.00%> (ø)

models/issue_comment.go
45.97% <0.00%> (+0.43%)
:arrow_up:

modules/git/commit.go
60.16% <0.00%> (+4.97%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3ef6a6...cc249d9. Read the comment docs.

Queue.pushfunc may block.
Are you sure that this is actually unnecessary?
I remember leaving this in because of this potential.

Queue.pushfunc may block.
Are you sure that this is actually unnecessary?
I remember leaving this in because of this potential.

The implementation should keep this function not block I think. We may add a comment on the interface method.
And I think for block condition, PushFunc is the same as Push, but we used Push many places with no goroutine.
Also I checked the implementation of levelqueue and redisqueue, they should not block the function.

They can all block - just hopefully and usually for not very long. The point of the goroutine was because I think in this particular instance we didn't want any delay to the calling function - I can't quite remember which one that was - and because I suspect there was a potential chance of a deadlock in the original pre-Queue implementation as PRs.
There could still be a deadlock potential here. IIRC the handler for this Queue calls this function. If you're using a channel or persistable channel then if the channel is full and there's no more boosts available you could have the worker for the queue blocked waiting for a worker to pop work off the queue.
So if that's the case then we need this or we need to adjust the queue.push implementations for channel etc to do a select send and goroutine if blocked.

🚀


Reflection does not work if proto contains import

What version of gRPC and what language are you using?
v 1.51.3
What operating system (Linux, Windows,...) and version?
OS: Linux
What runtime / compiler are you using (e.g. python version or version of gcc)
gcc 11.3.0
What did you do?
I have tried to separate interface description in several files from examples, that is located here: https://github.com/grpc/grpc/blob/master/examples/protos/helloworld.proto
and modify example: https://github.com/grpc/grpc/tree/master/examples/cpp/helloworld
to generate required files from several proto files instead of one (Please, see attached file helloworld.tar.gz
)
What did you expect to see?
$ ./grpc_cli ls <IP_ADDRESS>:50051 helloworld.Greeter -l
filename: helloworld.proto
package: helloworld;
service Greeter {
rpc SayHello(helloworld.HelloRequest) returns (helloworld.HelloReply) {}
}
$
What did you see instead?
$ ./grpc_cli ls <IP_ADDRESS>:50051 helloworld.Greeter -l
Service or method helloworld.Greeter not found.
$

Sorry, I can't download anonymous tar.gzs from the internet. Can you provide a git branch with your reproduction?

I made a repo that demostrates the problem: https://github.com/gwenya/grpc-reflection-with-import
As far as I can tell, it only happens when the service contains an rpc definition for which both the request message and the reply message are imported from another proto.

@drfloob I provided a GitHub repo demonstrating the issue, can you look into this?

Sorry about that, the bot should not have closed this, that's interesting. I've reopened the issue, but I do not have cycles to investigate at the moment. Thank you for providing a repro! That will be helpful.
If anyone in the community does have cycles to investigate more deeply, it's worth looking into whether any of the other gRPC implementations support the behavior you're after, or if this is maybe a protobuf limitation.

Hi @drfloob - I ran into this issue too in python and I assume the issue is similar in C++. I believe some implementations, at least Java, implement the reflection service by providing hooks into the gRPC server definition from internal classes.
https://github.com/grpc/grpc-java/blob/master/services/src/main/java/io/grpc/protobuf/services/ProtoReflectionService.java#L89
During this process any dependent service classes are loaded explicitly.
The python implementation seems to be decoupled and purely uses the protobuf pool to resolve services. This means that

list_services returns a list of services defined by the user rather than inspecting the server. A bit annoying but not the end of the world
Any protos that haven't been imported won't be in the pool. In the repro case, the FooService type that is queried during reflection is in a proto that is not imported.

The current workaround is for the user to import the proto as a no-op
import server_pb2
import server_pb2_grpc

# Request protos defined in different file, but we need to import server_pb2 itself
# to enable protobuf machinery.
_ = server_pb2.DESCRIPTOR

...

server_pb2_grpc.add_Service_to_server(MyService(), server)

This ensure the service is in the pool. While improving the ability for reflection service to introspect the server could be good, IMO the simple way to fix this is to have the grpc code generator (this repo) add the no-op import to the _grpc stub
import grpc
import pojos_pb2 as pojos__pb2
import server_pb2 as server__pb2 # Actually unused, just to register in pool

...

This is not unlike how the python protoc plugin generates imports to register external messages to the pool without actually using them, to register in the pool
import other_pb2 as other__pb2 # Actually unused, just to register in pool

I assume similar applies to C++, except perhaps an initializer needs to be referenced explicitly to ensure it's linked into the final program.


Documentation added to clock.py

Type of change

[ ] Added new project
[ ] Bug fix
[ ] New features
[X] Enhanced documentation

Changes proposed in this pull request

Documentation added

Checklist

[X] The code is Python 3 compatible
[X] I have added relevant comments in my file to make code understandable

Thanks, @Tark-pea for the updations 🚀


Add support to designate more than one Spring Cloud Function in AWS adapter

When using spring-cloud-function-adapter-aws, a Spring Boot application can hold only one Spring Cloud Function that can be invoked by AWS Lambda. The function.name property restricts us from creating more than one Spring Cloud function for AWS Lambda within the Spring Boot app. This means if I have to create another Spring Cloud Function for AWS, I have to create a new Spring Boot app.
Is it possible to specify the Spring Cloud Function in the Handler configration of the Lambda as <Fully_Qualified_Class_Name>::<Spring_Cloud_Function_name>?

Reopened here https://github.com/spring-cloud/spring-cloud-function/issues/238


Better tracker integration precision-recall curve

See issue below, apply for the titled plot
See reference

The refactoring was done. Are we not integrating with tracker?
@idomic

I think @neelasha23 is referring to adding log_precision_recall to the tracker, I think it's a valid point. But I'm unsure if we should tackle it soon.

yes that's what I meant!


denops#server#restart() does not work properly

Hello!
After executing call denops#server#restart() in Denops, when I execute :DenopsHello, it exits with the following error. This error also occurs when I run any Denops plugin other than DenopsHello.
function denops#request[1]..denops#server#request の処理中にエラーが検出されました:
行    4:
E605: 例外が捕捉されませんでした: The server is not ready yet

My current environment is as follows.
-OS Manjaro Linux
DISTRIB_ID=ManjaroLinux
DISTRIB_RELEASE=21.3.0
DISTRIB_CODENAME=Ruah
DISTRIB_DESCRIPTION="Manjaro Linux"

Vim Neovim

NVIM v0.8.0-dev
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compilation: /usr/bin/cc -DNVIM_TS_HAS_SET_MATCH_LIMIT -DNVIM_TS_HAS_SET_ALLOCATOR -O2 -g -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wdouble-promotion -Wmissing-noreturn -Wmissing-format-attribute -Wmissing-prototypes -Wimplicit-fallthrough -Wsuggest-attribute=pure -Wsuggest-attribute=const -Wsuggest-attribute=malloc -Wsuggest-attribute=cold -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/home/coma/neovim/build/config -I/home/coma/neovim/src -I/home/coma/neovim/.deps/usr/include -I/usr/include -I/home/coma/neovim/build/src/nvim/auto -I/home/coma/neovim/build/include
Compiled by coma@ComaBook

Features: +acl +iconv +tui
See ":help feature-compile"

      システム vimrc: "$VIM/sysinit.vim"
       省略時の $VIM: "/usr/local/share/nvim"

Run :checkhealth for more info

Denops commit hash is f967ee0730804649d340ec409d5e97c76dacf817(HEAD).
Thanks!

It is denops-helloworld's implentation problem.
It should check the server is ready like ddc.vim/ddu.vim.

But denops-helloworld is the minimal example program.
It must support deonps#server#restart()?  I don't know.

I'm not sure but maybe starting denops server just after stopping doesn't work and we need some relaxation interval between stop and start in denops#server#restart()

@Comamoca Could you try #197?


Add default values per Tiller installation

There are often cases where per installation metadata (or other values) are needed in a chart which are linked to the cluster which the chart is being installed into. For example; region, clustername, tillername, tillerversion, etc.
Is it possible to have such values in the current Helm?
If not, could we discuss adding, low priority, default values.yaml when installing/running Tiller. This could be an option to tiller, and therefore helm init. These values could either be set with --set, --set-string, --set-file, and/or --values. This would match the helm command.
The best (or starter) option would be to pass a path to a YAML file (configmap or otherwise) to read default values from for each helm install or helm upgrade.
My exact use case it to simply have region and clustername set for each install. I am using Helm with weaveworks/flux. This has a compontent called Helm Operator which watches for a CRD (FluxHelmRelease). A FluxHelmRelease can have values supplied, but copy and pasting these for each cluster and AWS region we deploy to would be cumbersome .
Thanks, Alex

potentially related to https://github.com/helm/helm/issues/2055.

FYI TillerVersion is available via the .Capabilities object. See https://docs.helm.sh/developing_charts#predefined-values for more info

Although clustername is useful, it is just an example. In the template we currently have access to the top level variable .Values. To make this change explicit we could introduce a new top level variable; for example .TillerValues, etc.
An example usage I have in mind is in a chart template:
{{ default .TillerValues.Region .Values.Region }}

The use case I had for this I have instead implemented in flux.
The general use case for this will go away if there is a tillerless helm.
For those interesting in default values when using flux, take a look at https://github.com/weaveworks/flux/pull/1468


Add inline widget to display variables by clicking on them

Up vote!
It's better to show variable content while hovering the cursor, not until click.


Consent URL API vs OIDC consent collection

Problem description
Telefónica created a PR in API backlog regarding a new API named Consent URL.
It is DT's position that Consent URL API is not needed because at the same point in time Consent URL API would be used by the API consumer the API consumer can alternatively request an access token which also leads to consent being collected.
Possible Outcome

Does ICM recommend creating a consent API?
Do we need something else than an API?
Do we need nothing?

Additional Considerations

CAMARA identity and consent management is based on OIDC and CIBA.
The OIDC parameter prompt is mandatory to implement by all OpenId providers.
Delegated Consent should/must be avoided.
Consent collection MUST be controlled by the API provider.
Consent collection in done according to Section <IP_ADDRESS> of the OIDC Core 1.0 spec
Regarding consent collection CIBA also refers to Section <IP_ADDRESS> of the OIDC Core 1.0 spec.
It is important to remark that in cases where personal user data is processed by the API, and users can exercise their rights through mechanisms such as opt-in and/or opt-out, the use of 3-legged access tokens becomes mandatory. This measure ensures that the API remains in strict compliance with user privacy preferences and regulatory obligations, upholding the principles of transparency and user-centric data control.

@AxelNennker What is missing in above list is the mangement of the consent, for which the API provider is responsible.
This should consist of showing the end user what consent has been given for which API consumer, at which date, for which purpose, including the exact details of the consent.  The API provider is also responsible for providing the user an option to withdraw the consent, and this should be as easy as how the consent was collected. As soon as the consent is withdrawn, all processing of the personal data should cease immediately.
In the ideal case, the end user should only give once consent for a case, and the text presented to the end user should inform him about both the processing that the API Provider will do, and the processing the API Consumer will do. It will also need to specify the purpose, so the end user can make an informed decision.
If you combine the above with the current specification, there are several issues that can occur. To name a couple:

When the end user withdraws user consent (in the interface that the API Provider is offering), the API Consumer is not notified that the consent has been withdrawn, and he will continue the processing with the already shared data (which should stop if you ask me).
When the end user witdraws consent, the API Consumer is not aware, and will trigger the next consent question (again) when    he calls the API again. For the end user this will look really weird, I have just witdrawn consent, and now I am asked the same question again ?
When the end user has a subscription which is terminated at the API Consumer, the consent issued in the past to the API Provider is still in the consent database with the API Provider. This data point should be removed, or at least it should be marked that the subscription does not exsist anymore (and the data point should be removed eventually). Currently there is no interface for the API Consumer to let the API provider know that the consent is no longer relevant, so this will lead to a lot of unnecessary pollution in this database.

What I think is necessary to solve this:

the API Consumer must also be check the current status of any consent given in the past, so they can cease any processing they are doing based on the consent.
the API Consumer must also be able to remove the consent (or signal that it is no longer required, or is withdrawn)

Problem description
Telefónica created a PR in API backlog regarding a new API named https://github.com/camaraproject/APIBacklog/pull/67.

It is DT's position that Consent URL API is not needed because at the same point in time Consent URL API would be used by the API consumer the API consumer can alternatively request an access token which also leads to consent being collected.

Possible Outcome

Does ICM recommend creating a consent API?
Do we need something else than an API?
Do we need nothing?

I fully agree that an Authorization Code or CIBA authentication flow can be initiated at any time, depending on the use case and the application's capabilities, to capture consent. In fact, consent capture doesn't need to happen precisely when the API is consumed.
The key benefit of the Consent URL API is that it offers a significant advantage by enabling a fully decoupled, out-of-band mechanism for consent capture. This API allows the application to fully control the user experience (and I'm not talking about the look and feel of the consent capture page here, which would be provided by the Operator). The application can generate a consent URL from the Operator, which the user must authenticate with. However, the application has the flexibility to deliver this URL to the user through the most appropriate channel or medium, tailored to the user's current context.
Importantly, this API does NOT delegate consent capture to a third party but rather empowers the third party (the application) to present the Operator's consent capture URL at the most opportune time and place. The actual consent capture occurs within the Operator's secure environment, ensuring the user's authentication with the Operator.
In uses cases for CAMARA, the Authorization Code flow has been defined to be used with network authentication (a.k.a. silent authentication), where user interaction was minimal. Partners have often raised concerns about the user experience, especially when the user is directed to an Operator portal for authentication or consent within the Authorization Code flow. Furthermore, the Authorization Code flow assumes the availability of a front-end device for the user, which isn't always the case.
While CIBA does provide an out-of-band method for user authentication and consent capture, it relies on the Operator reaching out to the user through channels under the Operator’s control, like SMS or push notifications (If the Operator Management application exists and is available to the target user). This approach can be limiting.
The Consent URL API adds a layer of flexibility to the consent collection process, allowing applications to integrate and customize the consent experience more seamlessly without intending to replace existing CAMARA OIDC flows. Instead, it enhances these flows by providing more options for when and how consent is captured.

Additional Considerations

CAMARA identity and consent management is based on OIDC and CIBA.

👍agree

The OIDC parameter prompt is mandatory to implement by all OpenId providers.

👍agree

Delegated Consent should/must be avoided.

👍100% agree. Consent URL API is not intended to delegate consent as explained above.

Consent collection MUST be controlled by the API provider.

👍agree

Consent collection in done according to Section <IP_ADDRESS> of the OIDC Core 1.0 spec

That's what's currently defined for Auth code Flow, yes. But Consent URL API would be an analogous procedure triggered independently of Auth Code Flow itself (or CIBA).

Regarding consent collection CIBA also refers to Section <IP_ADDRESS> of the OIDC Core 1.0 spec.

Same. 👍agree

It is important to remark that in cases where personal user data is processed by the API, and users can exercise their rights through mechanisms such as opt-in and/or opt-out, the use of 3-legged access tokens becomes mandatory. This measure ensures that the API remains in strict compliance with user privacy preferences and regulatory obligations, upholding the principles of transparency and user-centric data control.

👍agree
For me it is not about "Consent URL API vs OIDC consent collection" but "Consent URL API + OIDC consent collection"

Why not just use the following OIDC authoriztation code flow request?
  GET /authorize?
    prompt=consent
    &response_type=code
    &scope=openid%20profile%20email
    &client_id=s6BhdRkqt3
    &state=af0ifjsldkj
    &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb HTTP/1.1
  Host: server.example.com

Hi, @jpengar could you provide a diagram for this API consent url ?
Is a use case could be to ask the user the consent before iniating a CIBA callflow ?

During the TSC meeting we agreed that we should have a deadline for this ICM discussion, and that deadline would be the next ICM meeting.
I guess if we still don't agree then this goes back to TSC to decide what to do with this API proposal.

Hi, @jpengar could you provide a diagram for this API consent url ? Is a use case could be to ask the user the consent before iniating a CIBA callflow ?

The API consumer can use OIDC authorization code flow to ask for consent even if they plan to use CIBA later.
The consent record would be registered in the API provider's consent DB.
Even for a CIBA-only API provider following the OIDC standard is a good idea, I think.
  GET /authorize?
    prompt=consent
    &response_type=code
    &scope=openid%20profile%20email
    &client_id=s6BhdRkqt3
    &state=af0ifjsldkj
    &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb HTTP/1.1
  Host: server.example.com

Why not just use the following OIDC authoriztation code flow request?
  GET /authorize?
    prompt=consent
    &response_type=code
    &scope=openid%20profile%20email
    &client_id=s6BhdRkqt3
    &state=af0ifjsldkj
    &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb HTTP/1.1
  Host: server.example.com

As previously stated in https://github.com/camaraproject/IdentityAndConsentManagement/issues/215#issuecomment-2462553039, the current CAMARA definitions only consider network-based authentication in the Authorization Code Flow. In response to your proposed OIDC request with prompt=consent, CAMARA does not prevent the Operator from using additional authentication methods to authenticate the user who owns the network-authenticated device as part of the in-band consent capture process when consent is needed. However, network-based authentication will still be carried out. In the event that the user authenticated in the consent capture process does not correspond with the network-authenticated device, the authentication server must return an error. Therefore, this will require the execution of your proposed request from the user's consumption device. The consent URL API provides the flexibility to present the obtained consent URL in the optimal location for the user at that time.

@diegogonmar During the TSC meeting on 2024-12-05 you seem to put the focus of Consent URL API on scenarios that include a Target Device.
For a target device with no input-output capabilities (e.g. dog tracker with SIM) I imagine the following scenario.

user buys target-device from vendor
user buys target-device in MNO-shop in person and shop-agent adds target device to user's MMO account for which the user knows their MNO-login-credentials
At home user visits vendor's website and creates a user account their
user enters the target-device's phone number into vendors site
The vendor's website redirects the user's browser to the aggregator's website starting an OIDC authorization code flow with signed request = {
"prompt": "consent",
"login_hint":  "tel:<MSISDN-of-target-device>",
... scope, redirect_url, client_id, response_type, ...
}
The aggregator redirects to the MNO's authorization server
The user authenticates using their MNO-login-credentials
The MNO asks for consent for CAMARA DeviceLocation API for the MSISDN-of-target-device
The user consents
The MNO creates a OIDC code
MNO redirects back
API consumer gets code and retrieves access token
vendor's website uses it to track the target device

FWIW, I too do not believe that we need a separate consent URL mechanism. Operators should be able to lean on existing 3-legged user-present user-absent flows to gather consent from subscribers.
cc: @gmuratk @mengan @ramtmo

Thanks for example @AxelNennker, quite useful. The agreement in TSC was to move forward to create a focussed sub-team within ICM to discuss the most optimal solution. I understood that DT will evolve and enter into details of the OIDC based solution, right?
Your bullet list is fine, we need to enter into details. Few challenges were raised time ago, when talking about network auth: https://github.com/camaraproject/IdentityAndConsentManagement/issues/215#issuecomment-2427132149. Sadly no answer was provided, so those points were not addressed in that discussion, and some apply to the bullet list provided. Maybe you can elaborate a solution giving answer to those points.
Below I comment a couple of aspects regarding you example, you can check that some were already included in the challenges of the other thread
Regarding:
The vendor's website redirects the user's browser to the aggregator's website starting an OIDC authorization code flow with signed request = {
"prompt": "consent",
"login_hint": "tel:",
... scope, redirect_url, client_id, response_type, ...
}
The aggregator redirects to the MNO's authorization server
The user authenticates using their MNO-login-credentials
You propose login_hint to be something more than a hint but a "must authenticate user owner of this msisdn?". If so, override happens only with prompt=consent? Does login_hint take precedence on network_auth in case of mismatch between network-authenticated MSISDN and the one provided in login_hint?
If login_hint sent and consent already given, token is issued without user/password authentication? Based on what rules?
All of this should be addressed otherwise there is no interoperability, as currently CAMARA ICM instructs Operator to perform network auth in AuthCode. This proposal seems to override that behavior, but unless clear rules are defined (when to be overriden, how?), there will be no interoperability.
Regarding:
API consumer gets code and retrieves access token
Is this token issued for the person that used their credentials (may own N msisdns) or for the msisdn set in login_hint? Location API that will be used for this example is device based, so token issued for a person is useless. If accepted is risky! because consent was given to a given MSISDN, if token is issued for a person, may call Location API with a different MSISDN. Maybe you propose token issued for msisdn, so login is done for the msisdn although user/password is forced to collect consent?
This can be solved, but the whole mechanism need to be defined, don't you think? There are plenty of grey zones, where each operator may act different, e.g.: ignoring login_hint is perfectly possible and makes sense in case of Network Auth default behavior
So, rather than having a high-level example, would you make a full proposal addressing these and other points/challenges that may arise? I would appreciate a solution proposal to properly iterate on the points to be solved, taking group decisions.


Incorrect declaration file emit for recursive type

I'm having an issue where using the code directly works fine, but using the compiled code together with the emitted declaration file does not work.
TypeScript Version:  2.4.2, 2.5.3, 2.6.2, 2.7.2, 2.8.0-dev.20180224

Search Terms: recursive declaration
Code
export type RecordTypeToJson<
    T extends { _brand: string; _recordType: { [name: string]: any } }
> = ({
    record: { [TName in keyof T["_recordType"]]: RecordTypeToJson<T["_recordType"][TName]> };
} & { [other: string]: any })[T["_brand"]];
export interface Record<T> {
    _brand: "record";
    _recordType: T;
}
export type Smth<T> = any;
export abstract class Expression {
    toJson<TColumns>(this: Smth<TColumns>): RecordTypeToJson<Record<TColumns>> {
        return null!;
    }
}

export function test() {
    return class extends Expression {};
}

Expected behavior: Declaration file output:
export declare type RecordTypeToJson<T extends {
    _brand: string;
    _recordType: {
        [name: string]: any;
    };
}> = ({
    record: {
        [TName in keyof T["_recordType"]]: RecordTypeToJson<T["_recordType"][TName]>;
    };
} & {
    [other: string]: any;
})[T["_brand"]];
export interface Record<T> {
    _brand: "record";
    _recordType: T;
}
export declare type Smth<T> = any;
export declare abstract class Expression {
    toJson<TColumns>(this: Smth<TColumns>): RecordTypeToJson<Record<TColumns>>;
}

export declare function test(): {
    new (): {
        toJson<TColumns>(this: any): RecordTypeToJson<Record<TColumns>>;
    };
};

Actual behavior: test function is wrong:
[...]

export declare function test(): {
    new (): {
        toJson<TColumns>(this: any): { [TName in keyof TColumns]: ({
            record: any;
        } & {
            [other: string]: any;
        })[TColumns[TName]["_brand"]]; };
    };
};

Which throws error:
[ts] Type 'TColumns[TName]["_brand"]' cannot be used to index type '{ record: any; } & { [other: string]: any; }'.

Playground Link: Not useful since playground does not show declaration file output
Related Issues: idk, couldn't find anything

@DanielRosenwasser This is just a complex way of exposing our current behavior for recursive type aliases, simplified:
type A<T extends {fake: string, real: any}> = ({
    a: A<T["real"]>
} & { [index: string]: any })[T["fake"]]; // Somehow not circular, thanks to the delay caused by the index, yet not an error, thanks to the index signature

// type Q = A<{ fake: "a", real: Q /*any circularly referential type will do*/}>; // clearly circular; uncomment, causes alias above to be marked circular, in addition to itself

Except in this case, instead of instantiating a type alias, which causes us to emit the circularity error, we encounter the issue while instantiating a class (instead of an alias), so there's no circularity stack and we just print any when the type circulates. The circular type has some interesting constraints placed on its by its structure that aren't preserved when we attempt to reprint it this way, causing the error. We can't print it like the OP indicates because that'd require not eagerly evaluating the top level of the RecordTypeToJson type into its resulting record mapped-type (and also having an alias symbol on the top level indexed access).


Context Menu of sidebar doesn't work in letter icon

A followup issue to #604 .
We fallback to initial letter of the realm in case the icon isn't available, in that case the context menu doesn't work. We should probably include an event listener for context in that case too.

Hmm, good catch, we should make sure the context-menu works in every case.

@zulipbot claim


Fix modal backdrop overlap in sticky container

Description
There was a reported issue here that the modal backdrop overlaps the modal when it's used in a sticky container.
Motivation & Context
The issue mentioned renders the modal completely non-interactive once launched; The user cannot interact or even dismiss the modal
Type of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Refactoring (non-breaking change)
[ ] Breaking change (fix or feature that would change existing functionality)

Checklist

[x] I have read the contributing guidelines
[x] My code follows the code style of the project (using npm run lint)
[ ] My change introduces changes to the documentation
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] All new and existing tests passed

Live previews

https://deploy-preview-38564--twbs-bootstrap.netlify.app/docs/5.3/components/modal/

Related issues
Closes https://github.com/twbs/bootstrap/issues/38561

@julien-deramond makes sense, I just noticed the chronological order of the z-indexes variables. Thing is, I guess this leaves us with the situation that we either give it a 1010 (but accepting the fact that sticky elements will sit on top of the backdrop - which wouldn't make much sense IMO) or maybe let go of this element and give the modal a low opacity background. Would you go for something as drastic as this though? :thinking:


Add rempl server to plugin for simplify integration

add rempl server to plugin deps
start rempl server when building is starting
add plugin options for disallow to start rempl server or override rempl server config
add plugin option for auto-open UI after building is started

closed by 459ac1b


[2.6.3] Failed to find find system chart fleet

Rancher Server Setup

Rancher version: 2.6.3
Installation option (Docker install/Helm Chart): Docker install
Proxy/Cert Details: No proxy, using Let's Encrypt for cert

Information about the Cluster
The issue appears to be related to steps the Rancher container takes at boot time, not with the downstream clusters. Including anyway

Kubernetes version: 1.19.*
Cluster Type (Local/Downstream): Downstream

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Clusters are both imported and Rancher-created (RKE1)

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
From an Ubuntu shell, I'm using  docker logs -f $containerName, so root

Describe the bug

I searched for a while and found similar issues but they don't appear to be the same.
As best as I can tell, rancher-system-charts doesn't appear to have a fleet Helm chart, which some component of Rancher appears to expect. So instead of 2.6.3 finishing booting and logs settle down, we see an endless [ERROR] loop in the container.
This upgrade was performed following the usual steps which have been successful before today. I carefully read the 2.6.3 release notes, and performed the step that applied to me
Presumably something is stale, I guess a Helm repo maybe, but I'm not sure how to fix.
To Reproduce
Upgrade a single node/Docker Rancher container from v2.6.0 -> 2.6.3
docker logs -f $containerName
Result
After starting Rancher container and following the Docker logs, the following messages loop every 5 seconds:
2022/01/11 21:02:24 [ERROR] available chart version (0.3.500) for fleet is less than the min version (100.0.2+up0.3.8)
2022/01/11 21:02:24 [ERROR] Failed to find system chart fleet will try again in 5 seconds: no chart name found
2022/01/11 21:02:29 [ERROR] available chart version (0.3.500) for fleet is less than the min version (100.0.2+up0.3.8)
2022/01/11 21:02:29 [ERROR] Failed to find system chart fleet will try again in 5 seconds: no chart name found
(repeats over and over)

Expected Result
After Rancher container finishes booting, the logs should 'settle' and not loop [ERROR]s
Screenshots
The error seems to start fairly early in the boot process

I imagine it's b/c this gets called relatively early.
Additional context=
Command: docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 -v /rancherdata:/var/lib/rancher rancher/rancher:v2.6.3
(that's the command I ran on the Ubuntu host to run Rancher)

If I understand right, Rancher installs depend on rancher/charts.
I speculate my Rancher container might have a stale repo, so maybe I need to run helm repo update somewhere.
(Why Rancher wouldn't run that itself, I'm not sure)
I'm not sure where I'd run such a command, assuming that hypothesis is right (I tried the Rancher local cluster - no dice)

I see the same in an environment where I have Kubernetes 1.19.

For me, setting .Values.useBundledSystemChart to true (docs) did the trick.
I'm running v2.6.4-rc17 now from the https://releases.rancher.com/server-charts/latest helm repo ref https://github.com/rancher/rancher/issues/36259#issuecomment-1084434561

Exact same error with 2.6.4

Shame to hear it's still an issue with 2.6.4 :/
Have you tried @ddelange 's suggestion? https://github.com/rancher/rancher/issues/36114#issuecomment-1084518479

@onpaws Thanks for the link, i got #37039 on my other cluster when upgrading to 2.6.4 xD.
Is the suggestion possible with the docker installation?

I think so, NB I haven't tried, but based on this looks like you should be able to pass useBundledSystemChart

@onpaws Ah yes, thanks. I tried that, now I just see #37039 ...


Means to customize FormatOptions

Hi,
I'd like to customize the template output for toml, but because FormatOptions and toml::FormatOptions (same for yaml and json5) are non_exhaustive and only implement the Default trait, it looks like I can't instantiate these structs easily. Right now, the only way I can think of is creating a new trait and implementing it for these structs.
If I'm not missing something, would you consider adding each one a function new allowing the user to pass parameters. I realize that maybe it would lead to a breaking change if you add new fields to these structs. Maybe a builder pattern?
Thanks!

Mh I'm not sure I understand. The intended way is for you to use the Default impl with the struct init syntax:
let options= toml::FormatOptions {
    indent: 4,
    .. Default::default()
};

What's wrong with that? But I agree it could be explained better in the docs.

Ah, I had completely forgotten about this way. Perfect, thanks! (and sorry for the noise)


Add Slits Ophyd Classes

Adds Ophyd classes to represent slits, as well as specific classes for the I24 Slits 04 Virtual Motors, and slit simulator.

As discussed, @dan-fernandes, if you could drop me a message or add me as a reviewer when you're ready for another opinion I'll check it lines up with what we're doing in MX, thanks


Make sql server autostart

https://github.com/openpsych/django/issues/38 was caused by the sql server not automatically starting after the system restarted. It should do that, so we avoid errors in the future.

This has now been done.


Use external javascritp library

Hi guys. Thanks for this great project.
Curently i am using
sound = __new__ (Audio('./tones/ogg/65.ogg'))
sound.play()

for adding and playing audio file.
Now i want to use external library like howler.js.
The documantation says that i should add the sound and play like this:
var sound = new Howl({
  src: ['sound.mp3']
});
sound.play();

I tried to add like this:
sound = __new__ (Howl({ src: ['./tones/ogg/65.ogg']}))
sound.play()

but for some reason it doesnt work.
I have included howler.js in script in html file.

Hi,
Preferably use the followng Python syntax:
sound = __new__ (Howl({ 'src': ['./tones/ogg/65.ogg']}))
sound.play()

So with quotes around src, since it's a string literal (I'd expect).
Alternatively, if you like the (somewhat loose) quoteless JS syntax better, you may use a pragma or a command line switch, as described here:
http://sterlicht.alwaysdata.net/transcrypt.org/docs/html/special_facilities.html#keeping-your-code-lean-pragma-jskeys-and-pragma-nojskeys

Thanks for your anwser.
For some reason it still doesn't work.
I will try to figure it out.

Maybe it helps to take a look at the generated JavaScript.
I am curious what the cause of this is.

I found the problem. The problem is that i used old js library and was looking at new documentation. Instead of using src i had to use urls. Thanks!

OK, thanks for the feedback!


feat(rpc/rest/zmq): Added quorum listextended RPC

Issue being fixed or feature implemented

Introduction of RPC quorum listextended.
This change was requested by Platform since the already existing RPC quorum list doesn't return all required data.
quorum listextended behaves exactly like quorum list, but with a more enriched reply.
Data returned per quorum:

quorumHash
creationHeight
quorumIndex
minedBlockHash

Note: I chose to create a new one, since the reply models differs a lot.
array of strings for quorum list compared to array of objects for quorum listextended.
What was done?

How Has This Been Tested?

Breaking Changes

Checklist:

[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added or updated relevant unit/integration/functional/e2e tests
[ ] I have made corresponding changes to the documentation

For repository code-owners and collaborators only

[x] I have assigned this pull request to a milestone

Release notes need to be added

Please implement some functional tests for this. I would expect that to be relatively easy

Please implement some functional tests for this. I would expect that to be relatively easy

@PastaPastaPasta Done

Would it not make more sense for this to be a map from quorumHash -> data instead of a list of data blobs?

It's the same after all

Details

@PastaPastaPasta Done

@PastaPastaPasta Yup, ACK


cl21 cl22: lost fix - Khronos Bug 15691 kernel_preprocessor_macros test encodes file name and path in the SPIR-V binaries

for branches cl21+ because requires spirv support

All PRs merged, closing.


Vehicle get stucks after aborting a mission

The steps to cause it are:

Load a mission onto the vehicle via the OCU
Click Start Mission
Very quickly click Abort

Some notes:

It is independent of mission.
It doesn't matter if it is the same mission or not

I tried to reproduce the bug using the simulation but without success.
The set up I tried was:

Launch all nodes except Jaus Ros Bridge.
Simulate the Jaus Ros Bridge interface:

Load the mission
Execute the mission
Abort the mission as quick as posible. (I also tried it with 1 sec of delay between executing and aborting command)

IMHO I think we could find the error in how the Jaus Ros bridge processes the communication and if we are in manual or mission mode.

@ChrisScianna this should no longer occur, but I'd like visual confirmation from you before closing this ticket.

The work here is done, but this ticket cannot be closed until it gets validated on the water.


vimL function must not be called in a lua loop callback

Hi,
I tried setting this up this plugin for lua testing, but I'm not able to make it work. I'm gettings this error:
Error executing luv callback:
...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:14: The coroutine failed with this message: ...nux64/share/nvim/runtime/lua/vim/treesitter/language.lua:17: E5560: vimL function must not be called in a lua loop callback
stack traceback:
        [C]: in function 'error'
        ...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:14: in function 'callback_or_next'
        ...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:40: in function <...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:39>

Here's how I produce it:
Use this minimal.lua:
vim.cmd([[set runtimepath=$VIMRUNTIME]])
vim.cmd([[set packpath=/tmp/nvim/site]])

local package_root = "/tmp/nvim/site/pack"
local install_path = package_root .. "/packer/start/packer.nvim"

local function load_plugins()
  require("packer").startup({
    {
      "wbthomason/packer.nvim",
      "nvim-lua/plenary.nvim",
      "nvim-treesitter/nvim-treesitter",
      "antoinemadec/FixCursorHold.nvim",
      "rcarriga/neotest",
      "rcarriga/neotest-plenary",
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. "/plugin/packer_compiled.lua",
    },
  })
end

_G.load_config = function(is_initial)
  if is_initial then
    vim.cmd([[runtime plugin/nvim-treesitter.lua]])
    vim.cmd([[TSUpdateSync lua]])
  end

  require("neotest").setup({
    adapters = {
      require("neotest-plenary"),
    },
  })
end

if vim.fn.isdirectory(install_path) == 0 then
  vim.fn.system({ "git", "clone", "https://github.com/wbthomason/packer.nvim", install_path })
  load_plugins()
  require("packer").sync()
  vim.cmd([[autocmd User PackerCompileDone ++once lua load_config(true)]])
else
  load_plugins()
  load_config()
end

Clone neotest : git clone https://github.com/rcarriga/neotest && cd neotest
Open up events_spec.lua file with minimal config above: nvim -u /path/to/minimal.lua tests/unit/client/events_spec.lua
Try running tests for whole file: :lua require("neotest").run.run(vim.fn.expand("%"))

Screenshot:

System info:

OS: Manjaro Linux
Neovim version: Tried both  v0.7.0 and latest nightly, behaves the same

Thanks for the report, this was due to a recent change. Should be working in latest master, let me know if not :smile:

Working great now, thanks!

Ack! Apologies. This didn't come up in my testing for some reason.

Actually just saw this was happening silently when I tested which is why it ran so fast... It just wasn't running :laughing: Still runs about twice as fast though after this fix so still massive win!

The fastest code is the code that doesn't run 😆