[question] how to specify user group

Environment:

Jib version: 3.3.0
Build tool: Gradle
OS: Mac Big Sur

Description of the issue:
how to specify user group to run the container as non-root user.
but the specified user belongs to root group.(gid=0).
Expected behavior:
# build.gradle
jib {
    from {
        image = "adoptopenjdk:11-jre"
    }
   to {
     container {
        users: "1000:100"
     }
   }
}

# /etc/group in the base image.
...
users:x:100:

I expected the result of command.(app/logs/hoge.log is created by app execution user.)
I have no name!@myapp:/$ ls -l /app/logs/
total 0
-rw-r--r-- 1 1000 users   0 Sep 12 09:10 hoge.log

But, the actual result is the below
I have no name!@myapp:/$ ls -l /app/logs/
total 0
-rw-r--r-- 1 1000 root   0 Sep 12 09:10 hoge.log

id command also returns gid=0.
Steps to reproduce:

set "users" property in the build.gradle
docker build (using jib)
docker run and docker exec -it [container-id] /bin/bash

jib-gradle-plugin Configuration:
jib {
    from {
        image = "adoptopenjdk:11-jre"
        ...
    }
   to {
     container {
        users: "1000:100"
        ...
     }
   }
}

Additional Information:

@hiwayama thanks for the question! I would suggest taking a look at the README for specifying jib.container.user. The format should generally look like:
jib {
  container {
      user = '{myuser}:{mygroup}'
  }
}

Thank you for the quick response.
I'm sorry. I wrote the configuration file wrong.
I could specify uid/gid.
I have no name!@df24c8ba3e28:/$ id
uid=1000 gid=100(users) groups=100(users)

I noticed that this situation occuered on k8s cluster, so I reopened this issue.
Do you have any solutions?
k8s node version: v1.20.11
# kubectl exec
kubectl exec -it {java-based-image} -- id
uid=1000 gid=0(root) groups=0(root)

# docker run
$ docker run --rm -it --entrypoint id {java-based-image}
uid=1000 gid=100(users) groups=100(users)

the primary group ID of the containers will be root(0).

I'm sorry. It's the k8s spec.
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/


Migrate from AppAuthentication to Azure.Identity

The AppAuthentication library has been superseded by Azure.Identity (see notice), which provides a much wider range of auth provides.
I think the ServiceEndpoint class should just expose a TokenCredential property instead of relying on connection string parsing. It also provides better discoverability for auth properties.

We'll consider adding a constructor for TokenCredential in the future release.
Thank you for mentioning us.


Deprecate IF NOT EXISTS clause in database creation

As noted in https://github.com/influxdata/influxdb/issues/5707 InfluxDB itself does not support this syntax anymore and will raise a syntax error if attempted.
I swapped the default value of the parameter and added a E_USER_DEPRECATED notice when it's still being explicitly used.
Refs #67

Any chance of merging this into a new release?

Thanks a lot!


Could not open capture device. Please cError alsaaudio name 'alsaaudio_soundcard' is not defined

Thanks for the detailed instructions on issues  and I'm sorry if I missed what to do if this error occurs.
I copied in some of the command outputs  just in case it is needed.
Here's the error when starting supersid.py
Could not open capture device. Please cError alsaaudio
name 'alsaaudio_soundcard' is not defined
Traceback (most recent call last):
File "/home/sid/supersid/supersid/sampler.py", line 719, in __init__self.capture_device = alsaaudio_soundcard(
I can play test sounds from the usb with terminal commands.
speaker-test -c2 -twav -l7 -D plughw:3,0
In supersid config file
[Capture]
Audio = alsaaudio
Device = plughw:CARD=S3,DEV=0
Format = S24_3LE
PeriodSize = 1024
Channels = 1
Original error starting supersid.py
Could not open capture device. Please cError alsaaudio
name 'alsaaudio_soundcard' is not defined
Traceback (most recent call last):
File "/home/sid/supersid/supersid/sampler.py", line 719, in init
self.capture_device = alsaaudio_soundcard(
sid@raspberrypi:~ $ cat /proc/asound/cards
0 [vc4hdmi0       ]: vc4-hdmi - vc4-hdmi-0
vc4-hdmi-0
1 [vc4hdmi1       ]: vc4-hdmi - vc4-hdmi-1
vc4-hdmi-1
2 [Headphones     ]: bcm2835_headpho - bcm2835 Headphones
bcm2835 Headphones
3 [S3             ]: USB-Audio - Sound Blaster Play! 3
Creative Technology Ltd Sound Blaster Play! 3 at usb-0000:01:00.0-1.1, full spe
(supersid-env) sid@raspberrypi:~/supersid/supersid $ python3 -u sampler.py -s=96000 -m=alsaaudio -d="plughw:CARD=S3,DEV=0" -f=S24_3LE -p=1024 2>&1
Accessing 'plughw:CARD=S3,DEV=0' at 96000 Hz via alsaaudio format S24_3LE, channels 1...
alsaaudio device 'plughw:CARD=S3,DEV=0', sampling rate 96000, format S24_3LE, channels 1, periodsize 1024
alsaaudio 'plughw:CARD=S3,DEV=0' at 96000 Hz
96000 <class 'numpy.int64'> read from alsaaudio 'plughw:CARD=S3,DEV=0', shape (96000, 1), format S24_3LE, channel 0, duration 1.00 sec, peak freq [16171] Hz
[0 0 0 0 0 ... -19968 -24832 -29184 -26368 -18432],
Vector sum -34562048,

It appears I was not in the python virtual environment when starting the sid program.  Currently seems to be working.  Although my signal looks strange.Have t mess with antenna


e2e test added

#142

@UlisesGascon check the skipped test in https://github.com/UlisesGascon/NodeGoat/pull/1/commits/b5eb5f6eacf7cbc158a31f8d8fb54d70661ec03d

Thanks for this awesome work @KoolTheba! ⭐


Update hal_flash itf functions for compatibility

hal_flash itf functions now get the hal_flash struct pointer
as first parameter to allow customization for external memories.
This will be required once core gets PR #162 merged in.

The pull request was made against master, should've been made against develop.
I did pull in the change anyway to develop, as it applied cleanly.
Thanks for doing the work, Fabio


i18n(zh-cn): Update image-service-reference.mdx

Description (required)
Update image-service-reference.mdx
Related issues & labels (optional)

#9240
Suggested label: i18n

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

File
Note

zh-cn/reference/image-service-reference.mdx
Localization changed, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


Possibility of similar identicon collision

Hi there,
My question is that, what's the maximum possible number of identicons pydenticon can produce?
As I can see that pydenticon uses 7 colors to fill in tiles and have one options to toggle the colors also.
Methinks that this way, maximum possible identicons will be (7 * 2) * (2 ** 15), as it uses 15 tiles only because of the vertical axis symmetry.
This number is mere 458752 which is too less compared to total number of users on GitHub.
So this way it should be producing similar identicons and collisions are inevitable.
Correct me, if I'm wrong somewhere.

Well, most of those parameters are really configurable. When you are initialising the generator, you can pick your own dimensions, digest algorithm (this can limit the size of your identicons), as well as possible foregrounds and background (background is always just one, so it does not affect the number of possible combinations).
I think that with 5x5 size, 7 foreground colours, and 1 background colour, the number will be more likely to be something like (215 - 1) * 7 + 1 (215 for the number of 1/0 combinations for 15 fields, minus the one that is all zeros, times 7 possible foreground colours, plus the one that is all zeros. If I am not mistaken. So that'd be 229370. I hope I haven't missed anything in the calculation here.
As a side-note, due to the algorithm, you can have maximum of 256 foreground colours (one byte of entropy for that).

Also keep in mind that your collisions will probably happen much before you managed to exhaust all 229370 combinations (due to hashing algorithm chosen and data).

OK, got this.
Thanks for the reply :)


roll engine to version d90223fadfd219af23d4a4c260a78fb21be3fe99

It contains the following change :
d90223fad Roll Dart to version 1be785ae2ddb1754a184cd638ab719e94d86b4e9 (#5985)

Since this never landed, I suppose it will be superseded by https://github.com/flutter/flutter/pull/20427


recognize your work a bit more

@wlandau-lilly @shikokuchuo I can't help but notice that on the website, only Jeroen is named, why not you two? :smile_cat:

The above discussion is a bit dated. Given R-Consortium acceptance of the R-Multiverse project, I'd like to propose that all 4 of us are included on this page. We are also all listed in the (draft) governance document at https://github.com/r-multiverse/r-multiverse.github.io/pull/16.

We now have a "team" page: http://r-multiverse.org/team.html
@maelle and @jeroen, please feel free to edit https://github.com/r-multiverse/r-multiverse.github.io/blob/main/team.qmd in a PR if/when you are ready.


Pack pdb file if it exists when type project

My thinking is it would be nice when used with SourceLink.

I did not RTFM


attempt to remove unnecessary axioms

#51
Please check my change carefully and see if it makes sense.

after thinking a bit, I think we don't even need functional extensionality, if we always use equivalence between Prop.

@Ailrun if you feel good, please go ahead and merge it. the next step is cumulativity and the corresponding lemmas for contexts and substitutions (with potential adjustments). The latter part can go in parallel with the realizability proof. I think @Antoine-something can also help get things done in par. maybe you can work out the lemmas for contexts and substitutions? realizability is a bit more difficult so it can be left for Clare.


Changed background color of "Advanced Search" checkboxes for visibility

Changed the background color of the checkboxes for "Advanced Search" for visibility, since the foreground property does not appear to work for checkbox color.
Before: http://i.imgur.com/CWN2Pgo.png
After: http://i.imgur.com/V1aGiaF.png

I've merged this via command line due to the conflict, I'm not sure why github hasn't reflected it.

I've merged this via command line due to the conflict, I'm not sure why github hasn't reflected it.

It seems the change on line 228 (current line number in the main branch) didn't get merged, which was the removal of the background attribute for the "None" checkbox. This might be why github is saying there are unmerged changes.

It seems the change on line 228 (current line number in the main branch) didn't get merged, which was the removal of the background attribute for the "None" checkbox. This might be why github is saying there are unmerged changes.

Ah yes, that was probably it. I've removed the attribute.

Ah yes, that was probably it. I've removed the attribute.


clean up tags for repo (?)

why do we have a "1.0.0-ea3.1" tag and a "v1.0.0-ea3.1" tag? can we delete the one without a "v" prefix?

Yes, @grechaw and I need to coordinate that. Last attempt failed.

Possible to clean this up today?

Whoops. I already shipped it in bugtrack. Got led astray by an email w/link to it....

oh ok gotcha. what about the npm travis branches - just curious how they are being used?

Those are supposed to be fleeting. They hang around when the overnight travis build fails to finish. I just delete them periodically.


Content Disappears at 320 CSS Pixel Viewport Width

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.92.2
OS Version: 14.6.1 (23G93)

Steps to Reproduce:  Certain content, such as the "Search" icon and "Worspace"control text, disappears when the page is viewed at a 320 CSS pixel width, making it inaccessible to users with low vision who need to resize text.

Resize the VSCode window to a viewport width of 320 CSS pixels.
Observe that specific content, like the "Search" icon and "Worspace" control text, disappears.

User Impact: Users with low vision who need to resize text will not be able to access this content.
Expected Behavior: The page should reflow without requiring horizontal scrolling, ensuring all content and functionality are available at 320 CSS pixels.
Actual Behavior: Content is cut off or disappears, requiring horizontal scrolling or leading to loss of functionality.
Recommendation: Ensure that pages reflow properly without requiring two-dimensional scrolling at 320 CSS pixels. Moving content to an accessible show/hide control, like a hamburger menu, would maintain accessibility.

/duplicate https://github.com/microsoft/vscode/issues/227626


Global component that doesn't need being imported

I'm struggling with the case of having a series of predefined components ready to use in my mdx:
Some sample text <Highlight>and some highlighted text</Highlight>

In this case, I want Highlight to be some kind of "global" component and avoid the need of
import { Highlight } from '...'

I'm trying with substitutions and globals, but without success. It loos like globals is for global modules, not components.

 That's just up to client, since bundler doesn't fail on undefined components.
<Component components={{
  Highlight
}}/>

Closing the issue. Sorry.


Subtract nonbonded

I made some updates on data/md.py.

Scale chargeProd and epsilon to exclude nonbonded interactions
When nonbonded interactions are updated with updateParametersInContext(), an error could raise if the set of non-excluded pairs change (https://github.com/openmm/openmm/issues/3738).

Improve interpretation by changing variable names
I changed some variables in to improve the interpretation.

Attached file is a jupyter notebook and the input file used to check that the reference potential energy can be recovered by subtracting and/or adding coulomb and LJ potentials using different methods in md.py.
test.zip

Codecov Report

Merging #124 (ce6fccc) into master (ce6fccc) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head ce6fccc differs from pull request most recent head 869c6bf. Consider uploading reports for the commit 869c6bf to get more accurate results

It looks like the test failure is due to the missing OE license.

This looks like a good solution!
The license won't be used in PRs contributed from forks, so @yuanqing-wang will have to test this locally. We generally open PRs from a branch to avoid this issue.

This looks great! Thanks!


Url in example throws 404

The URL https://raw.githubusercontent.com/wafflespeanut/paperclip/master/openapi/tests/pet-v2.yaml in the CLI example throws 404
URL to the Example: https://paperclip.waffles.space/cli.html

Oh yes, the new URL is actually https://raw.githubusercontent.com/wafflespeanut/paperclip/master/tests/pet-v2.yaml and the change lives in book/cli.md
Thanks for finding this!

I'll raise a PR

I've assigned it to you!


add class_getitem to QuerySet and Manager

Hi, I'm a primary maintainer of django-stubs https://github.com/typeddjango/django-stubs.
I'll never find time/enough motivation to actually write DEP for type-checking, so I thought I'll just submit a PR with methods I need to support it, to start a discussion.

@mkurnikov Thanks, however all new features require an accepted ticket. Type-checking is a big topic so also some discussion on the mailing list and probably a DEP would be required. Opening a PR is not a good way to start a discussion.


Add Kerberos support for HDFS file system

This PR addresses https://github.com/tensorflow/tensorflow/issues/5369.
It also fixes an issue in DeleteDir.

Can one of the admins verify this patch?

Jenkins, test this please


grt: RC parasitic estimates wrong with read_guides

Describe the bug
There are issues with RC parasitic estimates when using read_guides.
Expected Behavior
RC parasitic estimates should be the same, whether the data was sourced from the odb or read_guides
Environment
Git commit: 53c8d2fa59bcde425bdf566f48cad543b999c6a2
kernel: Linux 5.19.14-200.fc36.x86_64
os: Fedora Linux 36 (Workstation Edition)
cmake version 3.22.2
gcc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)
clang version 14.0.5 (Fedora 14.0.5-1.fc36)

To Reproduce
The following test case shows the issue:
# openroad parasitics.tcl
...
[DEBUG GRT-0001] net net_M3
(14.85, 9.99)  1 -> (14.85, 9.99)  1 l=0.00
(14.85, 9.99)  2 -> (14.85, 9.99)  2 l=0.00
(14.85, 9.99)  3 -> (14.85, 89.91)  3 l=79.92
(14.85, 89.91)  2 -> (14.85, 89.91)  2 l=0.00
(14.85, 89.91)  1 -> (14.85, 89.91)  1 l=0.00
[DEBUG GRT-0001] net_M3:0 -> net_M3:0 via 1-1 r=0.017
[DEBUG GRT-0001] net_M3:1 -> net_M3:1 via 2-2 r=0.017
[DEBUG GRT-0001] net_M3:2 -> net_M3:3 79.92u layer=3 r=2.625 c=11.761
[DEBUG GRT-0001] net_M3:4 -> net_M3:4 via 2-2 r=0.017
[DEBUG GRT-0001] net_M3:5 -> net_M3:5 via 1-1 r=0.017
[DEBUG GRT-0001] net_M3:4 -> inv_M3_2/A (14.60, 89.67) 0.49u layer=2 r=0.015 via_res=0.017 c=0.072
[DEBUG GRT-0001] pin inv_M3_2/A -> to grid 0u layer=2 r=0.015 via_res=0.017 c=0.072
[DEBUG GRT-0001] net_M3:1 -> inv_M3_1/Y (14.67, 9.75) 0.42u layer=2 r=0.012 via_res=0.017 c=0.061
[DEBUG GRT-0001] pin inv_M3_1/Y -> to grid 0u layer=2 r=0.012 via_res=0.017 c=0.061
 Pi model c2=3.06e-17 rpi=29.5 c1=3.06e-17 max_r=29.5

The connectivity appears wrong. An example of what is expected (parasitic estimates calculated directly after global routing, ie not using read_guides:
[DEBUG GRT-0001] net net_M3
(14.85, 9.99)  1 -> (14.85, 9.99)  2 l=0.00
(14.85, 9.99)  2 -> (14.85, 9.99)  3 l=0.00
(14.85, 9.99)  3 -> (14.85, 89.91)  3 l=79.92
(14.85, 89.91)  3 -> (14.85, 89.91)  2 l=0.00
(14.85, 89.91)  2 -> (14.85, 89.91)  1 l=0.00
[DEBUG GRT-0001] net_M3:0 -> net_M3:1 via 2-3 r=0.017
[DEBUG GRT-0001] net_M3:1 -> net_M3:2 79.92u layer=3 r=2.625 c=11.761
[DEBUG GRT-0001] net_M3:2 -> net_M3:3 via 3-2 r=0.017
[DEBUG GRT-0001] net_M3:3 -> inv_M3_2/A (14.60, 89.67) 0.49u layer=2 r=0.015 via_res=0.017 c=0.072
[DEBUG GRT-0001] pin inv_M3_2/A -> to grid 0u layer=2 r=0.015 via_res=0.017 c=0.072
[DEBUG GRT-0001] net_M3:0 -> inv_M3_1/Y (14.67, 9.75) 0.42u layer=2 r=0.012 via_res=0.017 c=0.061
[DEBUG GRT-0001] pin inv_M3_1/Y -> to grid 0u layer=2 r=0.012 via_res=0.017 c=0.061
 Pi model c2=5.56e-15 rpi=2.5e+03 c1=7.02e-15 max_r=2.72e+03
 Pi model c2=5.56e-15 rpi=2.5e+03 c1=7.02e-15 max_r=2.72e+03

I'm guessing it has something to do with setting seg.final_layer.
Test case:  read_guides_issue-2.tar.gz

Thanks @eder-matheus this fixes the issue I was hitting.


Would it support different themes per frame?

Hi, I looked at the open/closed issue and could not find anything related to it, is there a possibility so each frame has its individual theme?

Is that something Emacs itself can do?

Seems like it is not possible, I tried using different emacsclients to achieve it, but still not possible. I would assume it is not supported right now


Restyle doesnt work anymore after running StyleGAN Nada

After getting restyle to work, I had much fun using it for a few months.
Then I heard about StyleGAN Nada and wanted to try it out... so I downloaded the repo and tried it out... first time running it worked but after that it didnt anymore... I did not install anything new and there arent any error messages.
The problem is that both restyle and nada (which depends on restyle) dont work anymore because they are stuck loading both the pSp and the e4e encoder inside the models folder. After a bit of research I found out that they are stuck loading the StyleGan2 Generator which is stuck at importing the FusedLeakyReLU, fused_leaky_relu and upfirdn2d. It doesnt crash/throw any errors and seems to load forever.
I am pretty frustrated right now because I cant use restyle anymore...
Does anyone know why this happens and how I can fix it?

I think I have actually come across this issue myself. What I believe is happening is an issue with the compilation of the CUDA extensions and ninja. For some reason, sometimes the files become locked and the script becomes stuck forever.
I believe the following issue my help you solve your issue:
https://github.com/eladrich/pixel2style2pixel/issues/80#issuecomment-778637066
If not let me know and I will be happy to try and help you understand the source of the issue

I dont have a torch_extensions folder... My .cache folder contained some torch stuff but deleting it didnt change anything.
Thats my Traceback:
Traceback (most recent call last):
File "toonify_ffhq.py", line 16, in
from models.psp import pSp
File "C:\Users\msk4x\Documents\Projekte\restyle-encoder-main\models\psp.py", line 8, in
from models.stylegan2.model import Generator
File "C:\Users\msk4x\Documents\Projekte\restyle-encoder-main\models\stylegan2\model.py", line 7, in
from models.stylegan2.op import FusedLeakyReLU
File "C:\Users\msk4x\Documents\Projekte\restyle-encoder-main\models\stylegan2\op_init_.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "C:\Users\msk4x\Documents\Projekte\restyle-encoder-main\models\stylegan2\op\fused_act.py", line 13, in
os.path.join(module_path, 'fused_bias_act_kernel.cu'),
File "C:\Users\msk4x\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "C:\Users\msk4x\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\cpp_extension.py", line 1306, in _jit_compile
baton.wait()
File "C:\Users\msk4x\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\file_baton.py", line 42, in wait
time.sleep(self.wait_seconds)
KeyboardInterrupt

Notice that the stack trace points to the exact same issue in the link I provided. The torch extensions folder should be stored somewhere so it is surprising you are unable to find it by following the steps I provided there.
Anyway, can you try running the following to find the files:
find / -name "fused"

Also, you can run your script in debug and dive into the load function that loads these extensions. From there you can see the path to the extensions and go to delete the lock file:
https://github.com/yuval-alaluf/restyle-encoder/blob/cb2010a6fc18af2d370e8886ebd3b38408406c9e/models/stylegan2/op/fused_act.py#L9-L16

Works!!! Thank you so much I am so happy right now :) Windows Search somehow didnt find the folder and your command didnt work but with a bit of research I found another command and found the folder inside AppData/Local

Will try StyleGAN Nada again later but if it stops working I now know the fix :)

Ah. Didn't know you were working on windows which explains why the command didn't work.
I'm happy to see you were able to solve the issue 😄


Import from GitHub failed

Description
Trying to migrate from GitHub https://github.com/wekan/wekan to locally installed Gitea that uses SQLite, with all issues, wiki, etc, using my GitHub API key. It did work some months ago, but not anymore.
I can not test this on try.gitea.io because it works too too slowly, try.gitea.io does not load in browser, and I would prefer not to add my GitHub API key there.
Error:
Migrating from https://github.com/wekan/wekan failed.

NzgxNzU4ODk1OA for <IP_ADDRESS>:60492, 200 OK in 0.7ms @ user/task.go:17(user.TaskStatus)
2022/11/09 21:43:48 ...ervices/task/task.go:56:handle() [E] Run task failed: GetTagCommit[v6.48]: context deadline exceeded
2022/11/09 21:43:49 [636c02f5] router: completed GET /user/task/1?_csrf=9VnsDWeT-qdHbn5J_SQHDqeTxoM6MTY2ODAyMjU3NzgxNzU4ODk1OA for <IP_ADDRESS>:60492, 404 Not Found in 0.7ms @ user/task.go:17(user.TaskStatus)

Gitea Version
1.17.3
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots

Git Version
2.37.2
Operating System
Ubuntu 22.10
How are you running Gitea?

Downloaded release for amd64

mkdir gitea
cd gitea
./gitea web

Database
SQLite

You can change the default timeouts here but I'm not really sure if that's the problem here:
https://github.com/go-gitea/gitea/blob/5a6cba4cf43f9e5be3ca4a3ffd8bfa351c92acbd/custom/conf/app.example.ini#L2130-L2138
Just a note: Looks like you are migrating all releases of that repo. The last releases all contains 500mb files and there are ~40 of them... That's a lot of storage and traffic...

@KN4CK3R
Thanks! It seems that when I unchecked migrating releases, it is continuing migrating issues, PRs etc fine.


User session status header

Users should see the status of their session when navigating the app

if a user is unauthenticated, a login navigation control should be displayed
if a user is authenticated, their user name should be display instead
a control for logging out should also be visible

I hadn't thought of this but it's a great idea. It could be the kind of thing where on a computer you always see it but on mobile you have to go to your account page then log out.

@tlr3552 I would like to take 2nd pass at this with some more focus on the UI design of the header.
@NoahWil5on could you please create a UI wireframe/mockup of a this header? The header should be a universal directive; meaning it applied by default to all views (unless otherwise specified in the requires for a given view).

@NoahWil5on @thunder033 I can (and should) include the updated changes to this issue with the switch-over to AM issue #70


Dropdown menu editor bug when hovering on MacOS

Godot version
4.0.dev.20220118
System information
macOS Monterey - 12.1 - M1 - Vulkan
Issue description
When hovering any but the first dropdown menu they glitch and the first one is showed. See the gif:
(I'm not pressing anything, just clicking to open the menu and moving the cursor)

This doesn't happen if the window is not at the screen's 0,0 position.

Steps to reproduce

Open the Godot editor
Open the top menu dropdowns
Hover over the options

Minimal reproduction project
No response

Duplicate of #53912, will be fixed by #56825


Bind flags as prefixed env variables

Signed-off-by: Hector Fernandez<EMAIL_ADDRESS>
Summary

There is an increase in the amount of flags used in our commands. Therefore we should think about binding these flags as environment variables that would help us when using the commands.
closes https://github.com/sigstore/cosign/issues/2330
Release Note

Documentation

Codecov Report

Merging #2346 (88d16d7) into main (ef9cf9d) will decrease coverage by 0.16%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2346      +/-   ##
==========================================
- Coverage   29.87%   29.70%   -0.17%
==========================================
  Files         134      134
  Lines        8289     8335      +46
==========================================
  Hits         2476     2476
- Misses       5482     5528      +46
  Partials      331      331

Impacted Files
Coverage Δ

cmd/cosign/cli/attach.go
0.00% <0.00%> (ø)

cmd/cosign/cli/attest.go
0.00% <0.00%> (ø)

cmd/cosign/cli/attest_blob.go
0.00% <0.00%> (ø)

cmd/cosign/cli/clean.go
0.00% <0.00%> (ø)

cmd/cosign/cli/completion.go
0.00% <0.00%> (ø)

cmd/cosign/cli/copy.go
0.00% <0.00%> (ø)

cmd/cosign/cli/dockerfile.go
0.00% <0.00%> (ø)

cmd/cosign/cli/download.go
0.00% <0.00%> (ø)

cmd/cosign/cli/generate.go
0.00% <0.00%> (ø)

cmd/cosign/cli/generate_key_pair.go
0.00% <0.00%> (ø)

... and 14 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Incorrect PID tracking when used in conjunction with python-daemon.

I'm using pid module in conjunction with python-daemon.
Although I don't have a simple example yet, the way python-daemon works is by providing a DaemonContext in which you can pass a file locking context manager. Normally I would use lockfile.pidfile but it doesn't feature staleness detection and pid seems to do everything correctly (almost).
The following code is contrived and hasn't been tested, but explains the situation I'm seeing.
from daemon import DaemonContext
from pid import Pidfile

# Using detach_process, DaemonContext roughly performs the following:
# 1. fork() [exit parent]
# 2. setsid()
# 3. fork() [exit parent]
# 4. override signal handlers
# 5. pidfile.__enter__()

with DaemonContext(detach_process=True, pidfile=Pidfile(pidname="test", piddir="/tmp")):
    pass

The problem is that entering the context manager at step 5, the Pidfile context gathered information about the process environment before step 1 in the above comment, and overrides the signal handler if overriding TERM.
I am trying to brainstorm a way to provide a DaemonContext friendly pid context manager. I can subclass Pidfile and implement DaemonContextPidfile, which reinitialises all variables in __init__() when  __enter__() is called. Or perhaps the process environment should be gathered only when the __enter__() has been called.
The simpler solution is:
with DaemonContext(detach_process=True):
    with Pidfile(pidname="test", piddir="/tmp/"):
        pass

Which is perfectly valid, except Pidfile() may tread on an existing SIGTERM handler.
Perhaps we can supply a SIGTERM handler to Pidfile which decorates the passed signal handler to ensure Pidfile.close() is called before executing the handler that is passed in.
Please let me know if you think any of these ideas seem viable.
Thanks.

@nnathan sure, let me recap to make sure I understand correctly. I see two patches here:

Allow to specify a custom function for sigterm_noop_handler
Have a way to run the "setup" code from __init__() inside __enter__()

Both seem good idea's to me.
As for (2) it would be easier if we just require using PidFile as a context manager but I kinda like that using it as a context manager is optional. It would be fairly easy to just add another flag to __init__() so that we can allow to lazily initialize the setup code, if possible we can do that in the create() method so it will work for context decorators and 'normal' code.

Yep, I think you have a good understanding of what I'm suggesting.
I think (1) is trivial if we allow a  register_term_signal_handler_function which is assigned a signal handler function and either creates a custom function which calls close() before calling register_term_signal_handler_function().
I think (2) is a little more nuanced but approachable in a few different ways. We can have __enter__() just call __init__() again (essentially doing it twice), or supply a lazily flag to notify which only determines process environment in the create() method.
The last thing I want to do is change the current behaviour of the code as it works really well. I see the above as simply features that provide a superset of functionality to remain compatible with daemon context managers.

Since python-daemon already sets a SIGTERM signal, would it not be enough to just set register_term_signal_handler=False ?

Created a 'lazy' branch and pushed a patch for this in there.
See https://github.com/trbs/pid/commit/b233bfebdce258a835e9654de77bb78015006761
Could you take a lot / test this and see if this works as expected ?

Thanks, I'll do some thorough testing and get back to you in a few days. (I was working on a similar approach but was seeking something more pythonic, the approach you laid out seems the path of least resistance.)

If you have any suggestions for improvements or better ways, I'm all ears ! :)
Thanks for wanting to test this. Things like these pidfile's can be very tricky and I don't want to break things for people.
Although I was thinking that the 'lazy-er' approach might make more sense as the default... since like with python-daemon environment variables might change between creating the Pidfile object and executing the create() method.

Hi, I managed to test the new pid module. It works well with python-daemon. I tried all combinations of lazy and detach_process (double fork option) to ensure that pid behaves as expected.
I agree that the lazy approach should be the default. It makes sense  that you want the most accurate assessment of the process environment prior to checking/acquiring a lock. As such I set that to the default, and along the way I had to create and fix some test cases (which I've submitted in PR https://github.com/trbs/pid/issues/5).
Lastly, I moved the logging setup to _setup() since I believe that the logging stream handle may be gone from the time you import the module to the time you instantiate PidFile(). This matters when used as a "passed" context manager (in the case of python-daemon) and for non-context manager use of PidFile() where between instantiation and check/acquire you may fork and close of all file handles/streams (including logging).

If you have any suggestions for improvements or better ways, I'm all ears ! :)

I honestly couldn't find a more elegant and beautiful way to approach the delayed setup. I think you've done a good job with the current situation. Also with the tests you wrote, this should vastly help should we see a refactor in the future.

Oh, I totally forgot to test the SIGTERM handler. I'll get back to you with results shortly.

After working on the code a little, my mind is a little clearer.

I cannot find a situation where we want lazy setup to NOT be the default behaviour. We should be interrogating the process environment at the time of creating or checking a lock.

 We will still need a flag to determine internally if _setup() has been called.

If there is some extreme case where the user needs to control when _setup() takes place, then they can explicitly call _setup().

What do you think of the above assessment? If you agree, I can do the legwork in my existing PR to fix this.

I agree with your assessment. Cannot think of any other reason as well.
Let's remove the protected status from _setup() and just call it setup() to indicate it's new and improved role :)
I added register_term_signal_handler='auto' in the lazy branch. After rechecking this we should safely be able to detect whether or not we need to install the handler. According to documentation at https://docs.python.org/2/library/atexit.html we need this when a handler is not explicitly installed.
This should make sure we do the 'correct' thing per default. Previously we where overriding a registered function if it was set by default (causing the issues with python-daemon) which was not very nice of PidFile. Since we only needed when no other handler is defined we might as well leave it alone if that has happened already.
The user can still specify True or False explicitly to do otherwise. I also added a check to see if register_term_signal_handler is a callable and if it is use the callable as the signal handler function. (same functionality as before just without the extra variable)

I've made lazy behaviour default (see PR https://github.com/trbs/pid/pull/6).
A few observations about the automatic signal handler which I think we should rectify:

pid should not tread on SIG_IGN. This means the caller had explicitly set it after python started but before the PIDFile functionality was invoked. Specifically SIG_IGN has the behaviour of ignoring TERM signals altogether (discarding them). Therefore the process will not terminate and that is perfectly fine for the behaviour for pid.

For same reason as (1) I think pid should not tread when signal.getsignal(SIGTERM) == None. I think a None return value may occur if a process interfaces with C which may set the Signal handler but isn't represented as a callable object in Python (I will test this and see soon).

Again I'm sure about the functionality of (1) but not of (2).
I don't want to step on any toes, but these are the tests I envision:
def test_pid_default_term_signal():
    signal.signal(signal.SIGTERM, signal.SIG_DFL)

    with pid.PidFile():
        assert callable(signal.getsignal(signal.SIGTERM)) is True

def test_pid_ignore_term_signal():
    signal.signal(signal.SIGTERM, signal.SIG_IGN)

    with pid.PidFile():
        assert signal.getsignal(signal.SIGTERM) == signal.SIG_IGN

def test_pid_custom_term_signal():
    def _noop(*args, **kwargs):
        pass

    signal.signal(signal.SIGTERM, _noop)

    with pid.PidFile():
        assert signal.getsignal(signal.SIGTERM) == _noop

def test_pid_unknown_term_signal():
    # Not sure how to properly test this when signal.getsignal returns None
    #  - perhaps by writing a C extension which might get ugly
    #
    # with pid.PidFile():
    #     assert signal.getsignal(signal.SIGTERM) == None

    pass

Lastly, I would like to add some (very short to the point) documentation for the front page describing the new behaviour going forward and a short note on signal handling.

Agreed on SIG_IGN and None. I should have looked into signalmodule.c and read the Python documentation better. I've changed that and added your tests. I disabled test_pid_unknown_term_signal test because this indeed seems a hard one to test. I was thinking that we might be able to use ctypes (on Linux/OSX) to set the handler (or call PyOS_setsig directly). But reading the signalmodule.c code it seems the handlers needs to be set before loading the signal module for the first time.
Thanks for wanting to write a little documentation about this.
Lets change the version number to 2.0.0, a major version increase, to indicate the impact of the changes. If you want we can add an AUTHORS file ? Your name or nick should be mentioned for all the work you put into this ! if you want of course :)

Lets change the version number to 2.0.0, a major version increase, to indicate the impact of the changes. If you want we can add an AUTHORS file ? Your name or nick should be mentioned for all the work you put into this ! if you want of course :)

:+1:
You're welcome to attribute as a (minor) contributor. I am honoured for the mention.
I think with my last pull request and a few other touch ups (such as bumping changelog), I think we can go forward to 2.0.0. Thanks again for putting in the legwork. I really found pid very pleasant with the combined effort to make the behaviour more desirable (instead of me going off on my own with a custom fork).

Secretly I dream of a day in which we can delete this repository and Python will ship with a well testing and full featured PidFile/lockfile implementation :-)

A very big THANK YOU ! :+1:


Integration with React

Can this Library be integrated with React Js or one needs to use drf-client only???

You can this library directly with the raw JS WS api that is in the browser: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket.

You can this library directly with the raw JS WS api that is in the browser: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket.

Thanks man this is great library you came up with. I shall testdrive it over weekend and see if I can rely upon it in my projects

Thanks man this is great library you came up with. I shall testdrive it over weekend and see if I can rely upon it in my projects


always_run is deprecated

[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4.

This is what I get on Ansible <IP_ADDRESS>

Unfortunately, Ansible doesn't leave me a lot of wiggle room here—if I switch to check_mode, then the role breaks for everyone on Ansible < 2.2.0 (which is still a lot of people). So I'm going to leave it until 2.3.0 is out.

2.2.1 is here, the deprecation notice is still here... still waiting until 2.3.

2.3.0-rc1 is out... getting closer!

Ansible 2.3.1 is the latest version (checked right now).

Ah yes, time to fix it :)

Oh, this was fixed already — see https://github.com/geerlingguy/ansible-role-mas/blob/master/tasks/main.yml#L24 (updated in https://github.com/geerlingguy/ansible-role-mas/commit/7d6f40b66b2b00f0ca3f68c6929a54433d24f956)


ViewResolver chain 规则

ViewResolver chain 规则

	protected void initStrategies(ApplicationContext context) {
...
		initViewResolvers(context);
...
	}

在解析的时候
	/**
	 * Resolve the given view name into a View object (to be rendered).
	 * <p>The default implementations asks all ViewResolvers of this dispatcher.
	 * Can be overridden for custom resolution strategies, potentially based on
	 * specific model attributes or request parameters.
	 * @param viewName the name of the view to resolve
	 * @param model the model to be passed to the view
	 * @param locale the current locale
	 * @param request current HTTP servlet request
	 * @return the View object, or {@code null} if none found
	 * @throws Exception if the view cannot be resolved
	 * (typically in case of problems creating an actual View object)
	 * @see ViewResolver#resolveViewName
	 */
	protected View resolveViewName(String viewName, Map<String, Object> model, Locale locale,
			HttpServletRequest request) throws Exception {

		for (ViewResolver viewResolver : this.viewResolvers) {
			View view = viewResolver.resolveViewName(viewName, locale);
			if (view != null) {
				return view;
			}
		}
		return null;
	}


SubarrayDescription should support taking telescope positions as EarthLocation

Please describe the use case that requires this feature.
To support actual telescopes, it would be better to support getting EarthLocation instances for the SubarrayDescription telescope positions.

Connected to: https://github.com/cta-observatory/ctapipe/issues/1979


404 redirect

in reference to issue #14

Added 404 page
Added redirection to blog posts

When redirecting to a previous blog post, does not check to see if blog post exists

For the default layout, changed the css and js links to have absolute paths (so the 404 page can inherit the styles wherever)

It seems like you got this to work with {{ site.baseurl }}, yeah? I'm going to go ahead and merge. @mkalish let me know if you have concerns.


Why this detected by VMProtect?

I try do test this on one programm with VMP, and this can detected from kernel. How do resolve this problem?

Why it should not be detected by commercial protection software? How to resolve? Figure out what it uses to detect VBox, if this is yet another Oracle VMM bug, like https://github.com/hfiref0x/VBoxHardenedLoader/issues/39, find a root of problem, submit it as bugreport to Oracle and hope for the fix... in about few(?) years.

Thanks for the answer.
Maybe you can tell? I am very interested in one question about Kernel and Windows.
I am using Windows 10, enable Windows debug mode.
I am trying to open the application that loads the driver.
Instead of working normally, my Windows hangs. In the event log, I find that an error has occurred: Kernel Power 41.
This happens when Windows debugging is enabled. I do not know who could ask this, no one knows and cannot give an answer. Thanks!

Configure your system to capture memory dump and disable automatic restart. Then you can use windbg to analyze this dump and figure out what is wrong.

I've tried this already. WinDbg tells me that the exception was in the driver that starts to run. Can I share a dump for you?

I don't see how this is related to this repository, sorry.


Can this loader use a PNG as a texture ?

The title is self-explanatory for what I want.

This library just loads the .obj file (including texture coordinates). Loading the actual textures needs an image loading library (like libpng), along with setting up the textures in OpenGL.


Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...)

Examine performance of individual data accessor methods of MemorySegmentIndexInput, as compared to the ByteBufferIndexInput. Specifically getByte, getShort, and getBytes.
This issue has been filed to raise awareness of a seemingly perf degradation of MemorySegmentIndexInput when used heavily in decompression, see https://github.com/dacapobench/dacapobench/issues/264. I've not yet investigated the low-level segment guards, but maybe there is something here (or in the VarHandles) that is not optimising?

I've also opened https://github.com/mikemccand/luceneutil/issues/267 to understand why our nightly benchmarks didn't notice this.  @uschindler maybe you have an idea!

The issue is analyzed already: The benchmark is wrong. It closes and opens indexes all the time. Due to the additional safety applied on closing indexes (to not crush the JVM with SIGSEGV), the JVM signals to all other threads that they need to go into safepoint to do the thread-local handshake.
The benchmakr should be rewritten to use a single IndexReader instance with many threads and not open a separate one for short time all the time in search threads. This causes thread-local-handshake havoc.

So this issue is not in Lucene. The slowdown comes from incorrect use of Lucene. I am waiting for the confirmation by them.

Thanks for your quick reply @uschindler. Your analysis makes sense to me.

Thanks for your quick reply @uschindler. Your analysis makes sense to me.

If they don't respond, I will rewrite their benchmark to only open the index once and then hammer it with different threads. If this is then identical speed the issue seen is confirmed to be caused by the thread-local handshakes.
Although this is incorrect use of Lucene, we should still try to investigate with Hotspot people why the handshakes/safepoints always lead to some kind of deoptimization in all threads accessing MemorySegments. This is not necessarily bad in normal workloads, but could lead to slowdowns in NRT use cases with high index refresh frequencies.
We could also add an option to MMapDirectory so you can make it ready for purely single-threaded use. I am not sure if this is useful in practise, but we could easily allow this. If another thread then reads from an index file it will trigger IllegalStateException.

We have the confimation: https://github.com/dacapobench/dacapobench/issues/264#issuecomment-2084907716
I think we can close this issue.
Anyways we should talk to the Hotspot people to figure out how we can improve the safepoint/thread-local handshake in a way that it does not get so expensive for other threads. To me it looks like on every close, all methods accessing MemorySegments get deoptimized, but I haven't looked into this.

Thanks for jumping on this so quickly @uschindler.

They identified that deoptimizations are indeed the cause of all this. When you trigger a thread-local handshake by closing an IndexInput, all code in other threads which runs at the same time and accessing memory segments gets deoptimized somehow. I have no idea how it enters optimized mode again, but this seems to cause the slowdown for that benchamrk.
I am not sure if we can open issues about this in OpenJDK to optimize this. Why does the code need to be deoptimized when we enter a safepoint/handshake? It just traps with a segv and enters safepoint.

Hi,
I reopened this to make a further investigation.
@dsmiley  talked to me at Berlinbuzzwords and he also commented on #13146. He has seen a major slowdown on Apache Solr with Java 21 and using Lucene 9.8 (or like that). The issue there was partly 3x slower queries (as far as I understood) in an environment where many many Solr Cores with man many softcommits (IndexReader repoens for NRT) are happening at the same time. So think of about 1000 indexes on the same JVM and all of them are repopen indexes every second.
There is seems to be the problem that closing one IndexInput seems to deoptimize every other MemorySegment although not related / belonging to the same shared Arena.
I try to write a JMH benchmark without Lucene code to reproduce the issue and measure how large the issue is (depending on how often you close an IndexInput and how many parallel threads are using other MemorySegments, also those not tied to same Indexinput).
My idea is:

Have many threads working on "hot" MemorySegments allocated with different shared Arenas.
Have a few threads closing those Arenas.

Theoretically, the benchmark should not slow if the few threads close Arenas which are not bound to and hot workers.
What we have seen in dacapo bench is that it seems to affect all MemorySegments negatively, not only those which are affected by the close. of Arena.
Once I have the benchmark I will open issue at OpenJDK.
In the meantime, we could "preserve" the old Lucene 9.1 (the version without MemorySegments) in the "misc" module of Lucene as LegacyMMapDirectory. This would allow people to use it, but it is strinly discouraged. It won't work anymore with Java 24 (or around that time).

Practically speaking for Lucene, a lot of the issues we've seen with this seem to be around checksum IndexInput instances, which are opened and closed a lot at least under the SegmentReader ctor (as opposed to longer-lived MemorySegmentIndexInput that are used for searching, etc). This leads to a lot of closing for checksum IndexInput where with READ_ONCE, iiuc, there's little benefit to using MemorySegment -- or at least Arena.ofShared() -- at all.
In a toy program written to illustrate this issue (jdk only) I've been able to improve things somewhat (~6x, fwiw) by naively protecting access to Arena.close() by a singleton 1-permit semaphore. Not a solution by any means, but further indication that there are definitely issues in high-concurrency environments.

improve things somewhat (~6x, fwiw)

Uhm, ~6x seems a lot more than just a "somewhat" to me!  It's spooky that such a workaround (forcing only one thread to .close() at a time) can yield such gains in current JVMs...
Is this performance degradation serious enough that we should fix FSDirectory.open's defaults to not use memory segments for readonce cases?

The problem is that Lucene at moment needs shared arenas, as we do not know beforehand if it's is used by multiple threads.
In general the observation helps. So it looks like the thread local handshake causes issues with deoptinization if multiple threads cause a thread local handshake at same time
Why not simply use the following workaround and measure it's effect:
Add a static CLOSE_LOCK = new Object() (it must be static) and synchronize the Arena.close() on it.
If this brings back performance, we can report this to jdk. At some point we can remove the lock when jdk version is greater xy.

But yes. For read once we could use a confibed arena. In that case it will throw exception when accessed by multiple threads. In addition maybe IndexInput.clone() should throw exception and not allow clones.

FYI, I had not much time to work on this. It would be great if we could open a bug report at OpenJDK and inform Maurizio about this issue.
I played a bit around with it - but that having a global lock on Arena#close() is helping is incredible! It may help the JDK devs to figure out what goes wrong!
Actually on all platforms, the thread-local handshake should not cause slowdown on "reader" threads. Only the actual Arena#close() call may be delayed or hang (actually not really different to a global lock as proposed). In no case it should cause a deoptimization on the other threads reading from MemorySegment!
To me this looks like an issue that theres a race inside the thread local handshakes so when multiple threads trigger this at same time, the JVM somehow gets into a state where it can only "escape" by deoptimizing all reader threads.
If we can create a simple benchmark without Lucene code showing this, we may help them to fix this quite soon! @magibney do you have some sample benchmark only using FileChannel#map on shared arenas and closing them with multiple threads? I was about to write something. If the performance of such benchmark restores when we synchronize, we have a proof whats going wrong.
I hope I can spend some time to work on this this week, but maybe @magibney has something already. I told @dsmiley on berlinbuzzrords that I will help!

I sent an email to Maurizio, maybe he has an idea. This would help us to open a good issue. Maybe he already sees the problem. He has best overview on the internals when closing shared arenas.

I also posted to panama-dev mailing list.

Haven't yet put together proper benchmarks, but I'm happy to share the toy program that I was using to evaluate the issue. I'm also not exactly sure that this is written to best target exactly the behavior that's problematic, but it seems to get pretty close, and with some tweaks can probably help guide in the right direction.
It's a single java class, no dependencies.
Invoked as
java -cp [classpath_root] org.apache.lucene.demo.ConcurrentMmapOrArenaClose [bool_mmap] [n_threads] [n_idle_threads] [n_reps] [bool_semaphore]

where:

[bool_mmap] if false uses Arena, if true uses legacy mmap
[n_threads] number of threads concurrently opening and closing mmaps/shared arenas
[n_idle_threads] number of threads that hold a region open but idle, closing only once all [n_threads] have completed
[n_reps] the number of times that each of [n_threads] will open/close mmap/shared arena
[bool_semaphore] for Arena, should access to Arena.close() be protected by a global single-permit semaphore?

A couple of things I've observed:

[n_idle_threads] has a relatively small impact, but does have some impact
[n_reps] is mostly to control the total amount of time to be long enough to be significant, but not so long that one gets tired of waiting for it to complete
[bool_semaphore] seems to make a fairly consistent ~6x difference, even across various settings of [n_threads]
the threads in [n_threads] are very busily opening/closing, so it's an artificial case of course. The concern is mostly that as [n_threads] varies it seems to scale very badly (esp. as compared to legacy/mmap)
I did try Arena.ofConfined(); unsurprisingly it is very fast (but I didn't bother leaving that nor making that configurable because it's not really an option for Lucene atm).

For read once we could use a confined arena

If this is an option I think it would likely have a heavy impact. Again, anecdotally from real-world experimentation in highly concurrent contexts, these READ_ONCE cases seemed to predominate in thread dumps. And the flip side of scaling badly with increased number of shared Arena.close() is that we should achieve an outsized impact from any reductions we can make (e.g., via READ_ONCE) to the number shared Arena.close() that are necessary!

Add a static CLOSE_LOCK = new Object() (it must be static) and synchronize the Arena.close() on it.

Ah right, that's simpler. I initially used a semaphore because I was experimenting with varying the number of permits, but there was no benefit to even low levels of concurrency, so I quickly fell back to single-permit, not thinking that that's equivalent to just synchronizing on a global lock 😄

Hi,
actually when looking at the code I don't understand what you are measuring. David said that reads are slow, but you actiually never read from the memory segment.
You only measure that it slows down closing, which is expected with shared arenas (as the thread-local handshake is needed). So It is actually expected that actually CLOSING takes longer, but what the others have observed is that also reads in totally unrelated threads get slower. So to me the test case here is not showing the full problem that the benchmark project and David Smileys issue.
So I think I need to write my own benchmark with JMH. Actually there should be 2 groups of threads:

On thread randomly open and close memory segments. Thoe will definitely be slower!
Another thread reading from another MemorySegment that was also mmapped but not closed. This one should stay fast. The others have seen slowdowns on reads there.

Add a static CLOSE_LOCK = new Object() (it must be static) and synchronize the Arena.close() on it.

Ah right, that's simpler. I initially used a semaphore because I was experimenting with varying the number of permits, but there was no benefit to even low levels of concurrency, so I quickly fell back to single-permit, not thinking that that's equivalent to just synchronizing on a global lock 😄 (EDIT: also, having read the jdk list email: the global lock doesn't make the issue go away entirely, but it seems to mitigate it by a constant factor -- scaling with concurrency is still far worse than linear though)

OK. But having a slowdown for Arena.close() is expected. What about other threads doing purely reads from MemorySegment?
Uwe

Actually your benchmark has no warmup, I would expect it to be slow as the MemorySegment shared code is quite heavy.

Ah ok! I was indeed evaluating something different -- my concern was around closing per se, particularly the contribution of multiple concurrent closes to locking IndexWriter, and increased hard-commit times (apparently due to READ_ONCE checksum IndexInput)?
Even if I'm basically measuring "close throughput", the behavior still seems unusual.
Yeah, re: warmup ... this is a toy program, not a benchmark; I was sharing it just to be more transparent about what I was looking at. Possible mitigations/workarounds with READ_ONCE and synchronizing access to Arena.close() still may apply -- just something to investigate with real benchmarks. (A bit naively here, but fwiw: I'd expect the lack of warmup to if anything make the longer-running invocations skew to look better, whereas what we're seeing is the opposite).
I also have questions re: "having a slowdown for Arena.close() is expected". I was unaware of this, but I guess it's a question of degrees. How much worse is expected? All the numbers below with a grain of salt (no warmup measuring something that may not be directly as important), but just so there are some actual numbers here ...
Without global lock on Arena.close(), concurrency -> duration:
2: duration: 7ms
4: duration: 13ms
8: duration: 29ms
16: duration: 66ms
32: duration: 251ms
64: duration: 801ms
128: duration: 3585ms
256: duration: 12977ms
512: duration: 53724ms
1024: duration: 309576ms

with global lock on Arena.close(), concurrency -> duration:
2: duration: 10ms
4: duration: 18ms
8: duration: 30ms
16: duration: 52ms
32: duration: 112ms
64: duration: 267ms
128: duration: 749ms
256: duration: 2410ms
512: duration: 9180ms
1024: duration: 43172ms

I also have questions re: "having a slowdown for Arena.close() is expected". I was unaware of this, but I guess it's a question of degrees. How much worse is expected? All the numbers below with a grain of salt (no warmup measuring something that may not be directly as important), but just so there are some actual numbers here ...

The slower shared close is expected because that makes it different from the old code which could possibly crash the JVM. It will be slower and theres nothing we can do against it without having the risk to crash the JVM. The whole idea of the shared segments is:

we don't want to slow down reads by always checking on each method invocation if the MemorySegment is closed. In multithreaded environments if one thread unmaps the ByteBuffer forcefully (thats what the usafe code is doing), another thread will segmentation fault, because the read form the ByteBuffer will hit an unmapped segment. This is because we cannot have an expensive check on every access.
So we slow down the actual close by making it use some intrernal JVM magic: The close call will trigger an so called thread-local handshake. Basically it will signal to all threads that they should go into safepoint. Because all other threads cannot do this at the same time, the close() have to wait for all threads to do this. Depending on multithreaded load this may take some time. This ensures that before actually unmapping the memory segment, we know that all threads have "halted". This halting is done by some crazy magic using internal sigsegv at some specific points (e.g., when it exits a method or when it has a loop continuation.
When all threads have halted, we can unmap the memory segment. And then all threads in the safepoint will make sure that they check the flag "was arena closed" before proceeding. This code will thor IllegalStateException("already closed").

Basically the thread local handshakes replace a volatile check on every read in every thread by some mechanism that the closing thread announces "i want to close arena, please all threads stop when you are working with memory segments". It then has to wait that all threads went into that state. Then it closes segment and then all threads can sping again after they have verified that the memory segment is still alive.
So basically we shift an expensive check per access to memory segment to a waiting time in close.
The issue that was observed is that obviously the "ask all other threads to go into thread local handshake" seems to have some effect on their performance.
But yes: When we do close() too often it gets expensive also for reading threads (and this is what others and David Smiley have seen).
The explanation here is a bit complicated and may be not fully correct (as theres a lot of magic involved). But basically the idea is to shift the expensive "check on every read from IndexInput" to "make close slower to allow all threads to finish work before we do close".
In Lucene with MappedByteBuffer we do neither (we just close the ByteBuffer's backing buffer and risk crashing the JVM). So the new code will be slower on highly concurrent environments. It depends on number of threads doing closes.
To your benchamrk: Without warmup this is in infact nonsense, as the whole magic is here happening in C2 compiler which requires a lot of warmup time. What you measure is a lot of interpreter logic which may cause loger waiting times on close() than with C2 where it should be highly optimized.

In short: When you reduce the number of expensive close() calls on shared segments, reduce the number of times all threads need to go into thread-local handshakes.
Next to use confined segments for single-threaded reads (like checksumming), we may also think of combining multiple files that are always open at same time to use a single arena (because all files are opened/closed at same time).
I think what you see with the global lock is just that the number of thread-local handshakes goes down, because you close all memory segments in one go and not trigger multiple handshakes at same time.
I'd like to go the route suggested by @ChrisHegarty and use confined arenas for read-once. I think we can relatively simple implement this. Bugs will be found by our tests easily. In additon read-once segments should never be cloned.
But still we should create a real JMH benchmark that emulates a real use case: Many threds are reading from various segments and others closing a subset of them.

Thanks so much Uwe for the detailed explanation!
Confined arena sounds great for READ_ONCE -- in fact I was obliquely suggesting this also with my initial comment, but I wasn't sure whether there might be some reason this would not work for Lucene; glad to hear there's no such reason! And in fact I expect this may make an outsized contribution to the overall Arena.close() situation, based on what I've seen in stack traces (where threads seem to actually get stuck in Arena.close() seems mainly to be these one-off instances).

Hi,
it looks like the underlying deoptimization issue is explained by Jorn Vernee: https://mail.openjdk.org/pipermail/panama-dev/2024-July/020532.html

I've read the various github threads, and I think I have a good idea of
what's going on. Just to recap how shared arena closure works:

We set the arena's isAlive = false (more or less)
We submit a handshake from the closing thread to all other threads
During that handshake, we check whether each thread is accessing the
arena we are trying to close.

Unfortunately, this requires deoptimizing the top-most frame of a
thread, due to: JDK-8290892 1. But note that this is a 'one off'
unpacking of the top-most frame. After which the method finishes running
in the interpreter, and then goes back to executing compiled code.

If we find a thread accessing the same arena, we make it throw an
exception, to bail out of the access.
After the handshake finishes, we know that either: 1) no other thread
was accessing the arena, and they will see the up-to-date isAlive =
false because the handshake also works as a means of synchronization. Or

if they were accessing the arena, they will get an exception.

After that it's safe to free the memory.

So, if shared arenas are closed very frequently, as seems to be the case
in the problematic scenarios, we're likely overwhelming all other
threads with handshakes and deoptimizations.
Addressing JDK-8290892 would mean that we only need to deoptimize
threads that are actually accessing the arena that is being closed.
Another possible interim improvement could be to only deoptimize threads
that are actually in the middle of accessing a memory segment, not just
all threads. That's a relatively low-hanging fruit I noticed recently,
but haven't had time to look into yet. I've filed:
https://bugs.openjdk.org/browse/JDK-8335480
This would of course still leave the handshakes, which also require a
bunch of VM-internal synchronization between threads. Shared arenas are
really only meant for long-lived lifetimes. So, either way, you may want
to consider ways of grouping resources into the same shared arena, to
reduce the total number of closures needed, or giving users the option
of indicating that they only need to access a resource from a single
thread, and then switching to a confined arena behind the scenes (which
is much cheaper to close, as you've found).
Jorn

Here is some response I sent: https://mail.openjdk.org/pipermail/panama-dev/2024-July/020535.html
Actually the ideas I have:

Use READ_ONCE in combination with single threaded confined Arenas
Possibly group multiple files into one arena. One Idea that came into my mind: MMapDirectory groups files belonging to the same segment together and uses a single Arena for them. Behind the senes it keeps a refcount for each group of files. So whenever a file is opened the refcount for that group increases and when it is closed it decreases. When the refcount goes to 0 the Arena is closed.

Here is some response I sent: https://mail.openjdk.org/pipermail/panama-dev/2024-July/020535.html

Thanks @uschindler - this is very helpful.

Actually the ideas I have:

Use READ_ONCE in combination with single threaded confined Arenas. Also disallow cloning of such IndexInputs (because thats mainly done for multi-threaded access).

A quick PR to see what breaks with this - #13535.  Good new is that nothing seems to break.

Possibly group multiple files into one arena. One Idea that came into my mind: MMapDirectory groups files belonging to the same segment together and uses a single Arena for them. Behind the scenes it keeps a refcount for each group of files. So whenever a file is opened the refcount for that group increases and when it is closed it decreases. When the refcount goes to 0 the Arena is closed.

Seems reasonable. Will have to benchmark to see how much it will improve things in practice.

Possibly group multiple files into one arena. One Idea that came into my mind: MMapDirectory groups files belonging to the same segment together and uses a single Arena for them. Behind the scenes it keeps a refcount for each group of files. So whenever a file is opened the refcount for that group increases and when it is closed it decreases. When the refcount goes to 0 the Arena is closed.

This seems really promising as well. I may try to pick this up unless anyone else is already looking into it? This seems like the right thing to do semantically even, in addition to the potential performance benefits.

Possibly group multiple files into one arena. One Idea that came into my mind: MMapDirectory groups files belonging to the same segment together and uses a single Arena for them. Behind the scenes it keeps a refcount for each group of files. So whenever a file is opened the refcount for that group increases and when it is closed it decreases. When the refcount goes to 0 the Arena is closed.

This seems really promising as well. I may try to pick this up unless anyone else is already looking into it? This seems like the right thing to do semantically even, in addition to the potential performance benefits.

I had no time to look into this. It should be doable with a instance ConcurrentHashMap, not sure where to place this instance (as it is also using those providers). So the problematic part is where to place this (as it cannot be in MMapDirectory where it should belong as it uses Java 21 preview apis).
You can give it a try, but i have the feeling it gets tricky on the actual implementation. We may need to add some abstraction into MMapDirectory instances (the ConcurrentHashMap) with the refcounter and a pointer to some "hidden" payload (the areana) which is picked by the private MemorySegmentIndexInputProvider.
We can work together. If you have a better idea, tell me.

I extended the Arena interface to track refCount for https://github.com/apache/lucene/pull/13555 -- curious what you think of this approach?

We also have another PR doing the same than @magibney's: https://github.com/apache/lucene/pull/13570
We also got some updates on the JDK front. Next to the fixes here (use confined segments for READONCE and grouping shared arenas) there are a few suggestions:
I have the feeling that Elasticsearch does not see the issue like Solr does under high load because the concurrency is different. @ChrisHegarty Is it true that Elasticsearch refreshes indexes in a sepate thread, so all close() calls are from a few threads only? In large Solr the installations the problem is mainly that every core looks like having its own threads that refresh indexes. Maybe it would be a good idea in Apache Solr to somehow have a maintenance thread which is responsible for refreshing indexes. The original patch by @magibney already showed that a static singleton lock around Arena#close looks like it helps. This in fact makes all closes sequential.
@dsmiley Maybe that's another solution to prevent this issue from happening: Introduce a global "maintenance task" in Solr which is responsible to reopen indexes. We could include that also in the SearcherManager tool? (cc @mikemccand).
I think with all those fixes here we have fixed most of the overhead.

Yeah: https://github.com/openjdk/jdk/pull/20158#issuecomment-2228916752

What does "refreshing indexes" mean exactly?  Solr does not have special threads for opening/closing indexes; they will typically happen on the common Jetty request thread pool.  If the suggestion is intended to ensure that only one thread open/close an index, I don't see why that would be a "maintenance task" but I do see it could easily be done simply by ensuring only one thread does this work on-demand.  But I'd be concerned indexes would open slowly if you have many.  Any way; feels like a work-around to the JDK bug and not something desirable in its own right.

It is not a JDK bug, it is expected behaviour.

Basically the idea is to close indexes asynchronous in a single thread doing this to not block main threads.

To be clear: With raising number of threads, the process of closing an indexinput gets slower. With recent Lucene improve,ments this is mitigated a bit by grouping files together and read READONCE files with a confined arena, but for all other IndexInputs which are in multithreaded use, the closing thread has to wait for the handshake on ALL other threads, so it does not scale well when there are many threads. This is needed for safety to ensure no other thread is accesssing the index while we are closing. There is no way to work around that as it is the required safety applied by the JVM (that we are currently missing in Lucene, this is why you can crush the whole VM when you close an IndexReader while threads are accessing it).
Because of this, the JDK maintainers recommend to possibly do the closing of indexes asynchronously in production environments. So basically in the normal processing of Solr, reopen the index as usual, but when actually closing the old IndexReader defer this to a single-threaded helper. This might also be a good idea for Elasticsearch.
This would allow to open the fresh indexes as soon as possible, but the old segments get closed in a delayed way. We can't implement this inside Lucene as we have no access to thread pools, it is up to the user to possibly close indexes async.

Solr already takes care to ensure a SolrCore is closed while nobody is accessing it via its refcount mechanism on the SolrCore.  I understand the JVM doesn't know that, and needs to take precautions.  That it's O(threads) means the Solr project should prioritize taming its 10k-bounded Jetty thread pool for this new reason.

basically reopening IndexReaders (closing old segments, open new segments). Basically a Solr softcommit, but also applies to commits

Woah; per commit! I thought you just meant the equivalent of opening and closing a SolrCore, which has an index -- somewhat rare.  Are you saying that, ideally, Solr should close the previous IndexReader (tied to a Searcher that will close) asynchronously?  And you suggested by one thread... does it need to be one thread for the whole JVM (this worries me) or one per core?

the JDK maintainers recommend to possibly do the closing of indexes asynchronously in production environments

It's okay if closing a SolrCore is a bit slower.  In practice, this means a core UNLOAD request would take longer thus deleting a collection takes longer (albeit is already parallelized per replica).  Shutting down Solr would take longer; okay.

Hi David,
sorry for late response. In fact there problem is quite simple. Every IndexInput#close() costs time in the thread calling this. This time is linear to the number of threads (in plain Java 19-21 at moment).
With the fixes by JDK people it will change a bit:

It will still be somehow linear to the total number of threads, but gets faster when not all threads are actually inside a MemorySement read
The will be no overhead anymore on consuming threads that don't use MemorySegments; at the moment it deoptimizes all threads.

So with the JDK changes to be applied to Java 22+ (and maybe also 21) the overhead will be almost zero for threads querying Lucene. The work for closing only minimally changes (if at all).
So the best we can do is: Don't concurently close IndexInputs in the main threads because on heavy loaded Solr clusters this can take a lot of time. So the idea is to do all "maintenance" on Lucene IndexSearchers/IndexReaders in a small & separate pool of threads. Actively reopen a DirectoryReader (after commit or softcommit) can be done in main thread, but closing the old one can easily be deferred to a small maintenance thread pool.
There is no way around this, as the ByteBuffer based MMap will no lobger be possible with later Java versions, as sun.misc.Unsafe will be decommissioned in Java 24 latest. So the new code and its additional safety we have to live with. And that's good. My suggestion would be to to also rethink Solr's thread usage. 10.000 threads is incredibly wrong! Possibly use virtual threads. If you do this, the problem will also go away.
Uwe

P.S.: The code optimizations were committed for JDK 23 already. Backport is under discussion.

Thanks for your detailed response!

The will be no overhead anymore on unrelated threads that don't use MemorySegments; at the moment it deoptimizes all threads.

Much of the fundamental tasks of a search engine access the index and those will use the MemorySegment API.  Won't the "deoptimize" part still be a problem?

Thanks for your detailed response!

The will be no overhead anymore on unrelated threads that don't use MemorySegments; at the moment it deoptimizes all threads.

Much of the fundamental tasks of a search engine access the index and those will use the MemorySegment API. Won't the "deoptimize" part still be a problem?

No, see benchmarks. In fact only the top frame is deoptimized, but once the processing can go on the top frame is replaced by the orptimized code again. This is a typical VM operation that happens under other circumstances, too.

I set a breakpoint in a debugger on IndexInput.close while running a Solr test and it's called in lots of places separately from closing an IndexReader.  Lucene FieldsIndexWriter.finish reads a file it just finished writing, for example.  If the underlying problem is more closely associated to the IndexInput and not the IndexReader, shouldn't IndexInput.close be internally async, and thus be a Lucene level change?
I also set a breakpoint on IndexReader.close.  Lucene's SpellChecker.swapSearcher is invoked when the searcher changes (e.g. from a commit) via Solr's SpellCheckerListener.  Hopefully the only effect leaving this is commits being slower.

I set a breakpoint in a debugger on IndexInput.close while running a Solr test and it's called in lots of places separately from closing an IndexReader. Lucene FieldsIndexWriter.finish reads a file it just finished writing, for example.

This ChecksumIndexInput should be opened with READONCE, so it is confined. So there's no cost on closing:
https://github.com/apache/lucene/blob/cc3b412183c7acbb4a6c498e9babffba4e7d1f28/lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/FieldsIndexWriter.java#L123
Uwe

All ChecksumIndexInput are automatically opened using READONCE in main branch as those cannot get consumed by multiple threads. In the backport it's manually done; if we missed one place please open issue.

I filed an issue in Solr: https://issues.apache.org/jira/browse/SOLR-17375

Noticing a related issue in OpenSearch https://github.com/opensearch-project/OpenSearch/issues/16180


Serve stale

This is a feature that I wanted because of spotty support of stale-while-revalidate.
If get returns serveStale then we should hang on to the content and serve it while the cache is being updated with new content.
This is kind of inspired by this blog post that you linked:
https://www.nginx.com/blog/benefits-of-microcaching-nginx/
Would you be interested in merging something like this into this library?

Thanks for the PR
is it the same as stale-while-revalidate?
why you said spotty support of stale-while-revalidate? can explain why you need this, what kind of problems are you facing? 🙂

Thanks for the note!
Sorry I shouldn't have said spotty support. Just that stale-while-revalidate headers don't really work for my situation because I want to serve the same stale cached content to multiple clients. Definitely a "memory stale-while-revalidate" is the right idea.
Let me try again:
I am developing the website for a news organization with a fair amount of traffic, and the index pages (homepage and similar) are complicated and fairly slow to generate. Basically, I don't ever want a client to have to wait for the page to generate: I always want to serve a cached copy.
I implemented cacheable-response and that helps a lot, but once the ttl expires, the servers get slammed with requests that can't be fulfilled until the page is generated again (similar to the behavior seen after step 2 in that NGINX blog post with the improved avg load time but big spikes as cache expires).
My solution here is to keep the cached content around past the TTL and continue to serve that while also triggering a cache refresh in the background.
My intent is to configure it something like: get: (...) => ({ttl: 60 * seconds, serveStale: 2 * hours, ...}
Do you have a similar situation? How are you solving it?

I like the idea although it looks be confuse since stale-while-revalidate was created for doing that job.
Why not just be possible serve stale cache copy respecting the already stale-while-revalidate value?
Then you don't need an specific setting for this, just it works according with your revalidation expectations, always sync with the HTTP layer

I want to always have a cached copy ready, instead of having the first user after TTL expiration to wait for the SSR to complete.
Any ideas how to accomplish this? Thanks!

Closing in favour of https://github.com/Kikobeats/cacheable-response/pull/80


Should be "professional" not "technician"

as per Microsoft's own naming

Hi Craig
Microsoft’s certification program for these people is called the Microsoft Certified Professional program. There is info on it everywhere. No-one calls database people technicians. Technicians generally repair hardware. I think when referring to a certified person, the naming should align directly with Microsoft’s own naming for these people. Yesterday I asked a room full of people about the term technician. None thought it was appropriate.
https://en.m.wikipedia.org/wiki/Microsoft_Certified_Professional
Regards,
Greg
Dr Greg Low
1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 4913 fax
SQL Down Under | Web: www.sqldownunder.com

From: Craig Guyer<EMAIL_ADDRESS>Sent: Thursday, February 1, 2018 12:54:48 AM
To: MicrosoftDocs/sql-docs
Cc: Greg Low; Mention
Subject: Re: [MicrosoftDocs/sql-docs] Should be "professional" not "technician" (#377)
Thank you @greglowhttps://github.com/greglow Can you please point me to what Microsoft source you are referencing? We reviewed some internal style guides and they did not indicate the need for a language update but it could very well be one source is behind others, it would not be the first time. Thank you for your time!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/sql-docs/pull/377#issuecomment-361996698, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEkhazEuTqa00QvtXyc6_YttFwp-Z0xkks5tQJrYgaJpZM4Rx2YT.

Thanks @greglow .  I do not think either options in anyway affects the readability or knowledge gained from the content.  We also do not have an official take on the term or this usage of either of those words and since the proposed changes are consistent, I will go ahead and merge them.


Cookie monster not workng on new beta

it not workin on new beta 1.9

You're too fast!  It just came out!  Give me time >_>

I don't really have an idea, how large the necessary changes are. Do you have any approximation on how long it might take?

The update kind of came at the wrong timing, since I was in the middle of a cold and I have other Christmas related things to worry about.  I plan to finish it tomorrow or more likely Christmas Day.

The beta script is in a state where I think I can close this bug.

Ouch, almost forgot this. Thanks for the update! Tbh, playing without Cookie Monster was rather painful (resp. exhausting).
Two general questions:
Can we talk about improvements now, too? There are quite a few new features which could use Cookie Monster support.
Is it save to assume that porting will be rather simple and fast once the beta goes live?

I guess you can talk a bit about improvements, but I really have no idea if and when Orteil goes to live.  Nor do I know what gameplay changes he will do before then.
It's pretty simple to port over as beta and live are on different branches, so there should be no problems really.

I'll just post my suggestions and than we will see :blush:
According to Orteil we can expect an update pretty soon.

We'll see if Orteil can keep his word....


Prepare for public release

Clean out OAO-specific deploy and operational stuffs
setup.py enhancements
move Flask app to nerium_app package

This is looking really nice 🔥

I added lots of reviewers, because I'm hoping it's ready for upload to PyPi/Docker Hub (well, I pushed to Docker Hub already, because I only needed an image for that).


Add a light color theme

I noticed the app can feel a bit too dark on glossy monitors, which produces a lot of reflects on the screen.
Adding a light/solarized theme would make it less reflective and users would have the choice a theme.
It also is a good opportunity of cleaning up SCSS files and reducing the amount of CSS present in components.

It could be useful to add a hotkey shortcut to toggle between themes.


Improve handling of complex mail flow and DKIM/DMARC/SPF

I was investigating the DKIM related issues with Agency 1, which seem to be do mostly to their more complex mail flow configuration.
Agency 1 mail flow is more complex than just office365 to the internet.  They route their mail back through what looks like an on-prem Forcepoint email gateway, which is what actually signs their mail as seen by recipient.  This is based on me looking at some inbound email from people at Agency 1.  This uses selector agency1seg2048a._domainkey.agency1.gov. and it seems to work as expected.
However, EXO is also configured with DKIM signing of email being sent to the forcepoint gateway.  They have multiple domains in EXO, some of which have dkim "disabled" and others have it "enabled".  The actual mail samples I saw seem to use DKIM signing using a a selector associated with the domain "agency1365.onmicrosoft.com", which is marked as enabled in the config.  This is apparently what microsoft does when the custom domain dkim policy is not enabled (as in this example).  This mail is purely internal to Agency 1 infrastructure between O365 and their on-prem email gateway (which also appears to validate DKIM).
We probably at least need to add some documentation indicating this behavior for situations with more complicated mail flow.  In theory, we could detect mail routing configs that send outbound email not directly to the destination and use that to influence our messages.

See my comment on #41


x/telemetry: improvements to the front page at telemetry.go.dev

The front page of https://telemetry.go.dev is still largely a placeholder. We should improve it prior to the Go 1.23 release, which includes support for opt-in telemetry. At that time, people will want to know more about telemetry, as well as view telemetry data that has been collected thus far.
There is a lot of room for improvement, since the page is currently just a bunch of links. We discussed this in our team meeting, and had a few ideas for how it could be improved. More ideas are welcome.
Minimum requirements

The content of the Overview section must remain prominently displayed.
We should filter the links to recent data, or paginate them (but the full set of data must remain accessible, for example at an /all url).
We should show a summary of recent data. For example, we could choose a selection of all charts, and generate them based on data merged from the last week of reports. This may require some pre-processing on the worker.

Other ideas

Show timelapse data. For example, a chart of go/gopls version usage by week would be interesting.
Add a download form that allows specifying a date range, and downloading a zip file with all telemetry data in that date range.
Show a chart of uploads by day/week, to provide info about opt-in rate
Incorporate some way to view stack counter data. For example, we'd be interested in showing recent gopls crashes.

CC @golang/telemetry @ansaba @h9jiang

It'd be nice to stick to static content, so I'd avoid the download form. People can script downloads easily enough.
The others all sound good. I agree with the minimum requirements, and let's focus on doing those before anything else.
Thanks!

We should show a summary of recent data. For example, we could choose a selection of charts, and generate them based on data merged from the last week of reports. This may require some pre-processing on the worker. will require merging of data and extra work.
It the work seems like a Stretch goal, an easier way would be Filter on date range and provide Next pagination on individual date page.

It the work seems like a Stretch goal, an easier way would be Filter on date range and provide Next pagination on individual date page.

@ansaba just to make sure I understand the suggestion: you mean rather than merge the last week of charts, just show the most recent day's chart, with a pagination to go back/forward in time. Agree we can do that if we don't have time to merge data. Longer term it would be nice to see the prior week's data, since the upload happens once a week and so the past week of data represents active users.

We've deployed the CLs above to https://telemetry.go.dev. There is of course more that we could do, but we'll probably stop here. The home page is styled more like a subpage of go.dev, and has some aggregate charts for the last week. The chart viewer itself has a dynamic floating menu that makes it easier to click around to different charts (which will be more important as we add more collection).
For true exploration of the data, we'd be better served by ingesting telemetry data into a database or perhaps cloud monitoring, where it can be dynamically queried and charted. That is a project for another day.


Access to adlist connection refused.

When i update Gravity on my Pihole web interface i get this ;
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
[✗] Status: Connection Refused
[✗] List download failed: no cached list available
How to resolve, Thanks

Hi @Ragziesoft thank you for your question.
$ curl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

... works for me. So it's definitely available and online.

So could you guide me on how do i resolve it in my case.

Hi @Ragziesoft
Works great from here.
I don't know what to suggest.

Is there a way to download the adlist separately and save it in cached list?

Try the non github link

Okay success


JS: Refactors Store.load and Store.dump API to enable more options

CC @ludovicm67 @BenjaminHofstetter The no_transaction and unchecked both enabled seems to provide a ~20% speedup when loading data (actual speedup is quite dependant on the actual RDF data)


Write up scan server example for Blog post

Is your feature request related to a problem? Please describe.
We currently don't have great documentation for using scan servers.
Describe the solution you'd like
A blog post should be written up and published for using scan servers and the ScanSelector.
https://github.com/apache/accumulo/blob/2.1/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java#L130
https://github.com/apache/accumulo/blob/main/core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerSelector.java#L150

I tried to keep this comment (https://github.com/apache/accumulo/pull/2665#issuecomment-1113302935) up to date during development and testing. It could be used as the foundation for a blog post.


[Event Request] Codeunit 57 "Document Totals" - OnBeforePurchaseCalculateTotalsNoRounding

Describe the request
In Codeunit 57, please include the OnBeforePurchaseCalculateTotalsNoRounding event within the PurchaseCalculateTotalsNoRounding procedure.
Actual Code:
procedure PurchaseCalculateTotalsNoRounding(var TempCurrentPurchaseLine: Record "Purchase Line"; var VATAmount: Decimal; var TempTotalPurchaseLine: Record "Purchase Line"; var TaxAreaCode: Code[20])
var
PurchaseLine: Record "Purchase Line";
begin
Clear(TempTotalPurchaseLine);
PurchaseLine.SetRange("Document Type", TempCurrentPurchaseLine."Document Type");
PurchaseLine.SetRange("Document No.", TempCurrentPurchaseLine."Document No.");
if PurchaseLine.FindSet() then
repeat
if ((PurchaseLine."Tax Group Code" = '') or
((PurchaseLine."Tax Area Code" = '') and (TaxAreaCode <> '')))
then begin
TempTotalPurchaseLine.Amount += (PurchaseLine."Line Amount" - PurchaseLine."Inv. Discount Amount");
TempTotalPurchaseLine."Inv. Discount Amount" += PurchaseLine."Inv. Discount Amount";
TempTotalPurchaseLine."Amount Including VAT" += (PurchaseLine."Line Amount" - PurchaseLine."Inv. Discount Amount");
end else begin
TempTotalPurchaseLine.Amount += PurchaseLine.Amount;
TempTotalPurchaseLine."Inv. Discount Amount" += PurchaseLine."Inv. Discount Amount";
TempTotalPurchaseLine."Amount Including VAT" += PurchaseLine."Amount Including VAT";
end
until PurchaseLine.Next() = 0;
VATAmount := TempTotalPurchaseLine."Amount Including VAT" - TempTotalPurchaseLine.Amount;
end;
Required code with a new event request:
procedure PurchaseCalculateTotalsNoRounding(var TempCurrentPurchaseLine: Record "Purchase Line"; var VATAmount: Decimal; var TempTotalPurchaseLine: Record "Purchase Line"; var TaxAreaCode: Code[20])
var
PurchaseLine: Record "Purchase Line";
IsHandled: Boolean;
begin
IsHandled := false;
OnBeforePurchaseCalculateTotalsNoRounding(TempCurrentPurchaseLine, VATAmount, TempTotalPurchaseLine, TaxAreaCode, IsHandled);
if IsHandled then
exit;
Clear(TempTotalPurchaseLine);
PurchaseLine.SetRange("Document Type", TempCurrentPurchaseLine."Document Type");
PurchaseLine.SetRange("Document No.", TempCurrentPurchaseLine."Document No.");
if PurchaseLine.FindSet() then
repeat
if ((PurchaseLine."Tax Group Code" = '') or
((PurchaseLine."Tax Area Code" = '') and (TaxAreaCode <> '')))
then begin
TempTotalPurchaseLine.Amount += (PurchaseLine."Line Amount" - PurchaseLine."Inv. Discount Amount");
TempTotalPurchaseLine."Inv. Discount Amount" += PurchaseLine."Inv. Discount Amount";
TempTotalPurchaseLine."Amount Including VAT" += (PurchaseLine."Line Amount" - PurchaseLine."Inv. Discount Amount");
end else begin
TempTotalPurchaseLine.Amount += PurchaseLine.Amount;
TempTotalPurchaseLine."Inv. Discount Amount" += PurchaseLine."Inv. Discount Amount";
TempTotalPurchaseLine."Amount Including VAT" += PurchaseLine."Amount Including VAT";
end
until PurchaseLine.Next() = 0;
VATAmount := TempTotalPurchaseLine."Amount Including VAT" - TempTotalPurchaseLine.Amount;
end;
[IntegrationEvent(false, false)]
local procedure OnBeforePurchaseCalculateTotalsNoRounding(var TempCurrentPurchaseLine: Record "Purchase Line"; var VATAmount: Decimal; var TempTotalPurchaseLine: Record "Purchase Line"; var TaxAreaCode: Code[20]; var IsHandled: Boolean)
begin
end;
Additional context
This new event is necessary to enable subscription for implementing custom code and to provide the option to bypass the base code when needed.
Internal work item: AB#559361

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 27959.


Update for processing order

Adding VNet, Peering and SE preference

@malopMSFT : Thanks for your contribution to the Azure documentation! The author, @jimdial, has been notified to review your proposed change.

#sign-off

@malopMSFT: I'm sorry - only the author of this article, @jimdial, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

#sign-off Fixes MicrosoftDocs/azure-docs/#5404

@malopMSFT : Thanks for your contribution to the Azure documentation! The author, @jimdial, has been notified to review your proposed change.


Bug(Frontend): clicking projects on navbar will load nothing

useFilter would check the queries and pass them to the project.search function.
https://github.com/privacy-scaling-explorations/maci-rpgf/blob/04b5a592f56a9621bdc4fee8182fca66d5c34dc0/src/features/projects/hooks/useProjects.ts#L23
Q1: why click on projects on the navbar automatically add these queries? ➡ these might be the initial queries, was set because of the function setFilters
Q2: why we already set type Filter.seed to number here
https://github.com/privacy-scaling-explorations/maci-rpgf/blob/04b5a592f56a9621bdc4fee8182fca66d5c34dc0/src/features/filter/types/index.ts#L11
but it's still a string instead of a number?
The above FilterSchema is checking the type while calling project.search api, so we still need to make sure we pass correct type of queries to server.
https://github.com/privacy-scaling-explorations/maci-rpgf/blob/04b5a592f56a9621bdc4fee8182fca66d5c34dc0/src/features/filter/hooks/useFilter.ts#L64
toURL set everything to string.

Fixed in #26
(@kittybest if I'm mistaken, please re-open)


runtime/pprof: test fails intermittently on windows/amd64

??? which test case? why testing.Main cannot capture the condition?
ok  	runtime	37.654s
?   	runtime/cgo	[no test files]
ok  	runtime/debug	0.155s
exit status<PHONE_NUMBER>
FAIL	runtime/pprof	8.979s
?   	runtime/race	[no test files]
ok  	sort	0.190s

See http://build.golang.org/log/dfb6e28b125b9068674f188c0658155d89c924fb and https://storage.googleapis.com/go-build-log/b284db5e/windows-amd64-gce_39c49307.log

Dup of #10231 ?
Alex

Yup, merged into #10231.


Rewrite some parts of extension, adding a Page class

The extension needs to have some parts rewritten. Specifically, a separate Page class, unique to this extension, needs to be added to represent each separate Page in the resulting output, rather than having this extension's code spread around extending the other stock Asciidoctor classes.
This should simplify the code significantly, making it easier to understand, and will make it much easier to resolve open issues (even resolve some of them without any work on them specifically).
This issue needs to be completed before most of the other open issues can be worked on.

See also my sponsor page.

When will it be released? I am looking forward to it 😄

When will it be released? I am looking forward to it 😄

There is no timeline or completion date for this issue. If you want to encourage development here, please consider sponsoring.


Adding Sub array sum in Basic DSA questions

This is a program of printing the sum of all the possible sub array of given size.
#forHacktoberFest
#opensource

Go Ahead


[Question] Deploy without need for S3 bucket

Is your feature request related to a problem? Please describe.
I tried to deploy via docker-compose and saw that the spinup failed due to missing S3 data in the ENV variables. According to the documentation S3 is used for storing user profile images. Since I don't have an AWS account at the moment but enough storage on the host, I wonder if this could be avoided
Describe the solution you'd like
Deploy isolated to a host, with all data stored locally either in named volumes or host mounted folders

Related to https://github.com/AmruthPillai/Reactive-Resume/issues/830

@Gymnae - Should be resolved with above ENV variable value put to false


Add warnings about text/binary mode files.

Resolves #2725.
@sigmavirus24 I'm wondering if we should add a warning to super_len for whenever we decide on the length of the file using os.fstat, if the file has not been opened in binary mode. It should be easy enough to do. Thoughts?

@Lukasa do we want to add that warning in super_len and then make it a hard exception in 3.0.0?

@sigmavirus24 Yeah, probably.

♥ thanks brosephs @sigmavirus24 @Lukasa

@Lukasa do you want me to send that separately or do you want to do that as part of this PR?

Ah, I forgot about that. Let's do it here.

 It's strictly a subclass of DeprecationWarning, which I think is ok too.

Excellent work @Lukasa . Thanks!