Fix an edge case random number problem

This PR fixes an edge case indexing bug. Problem arose when probabilities of cell options going into shared_tools.random_pick were not exactly sum(probs)==1 due to rounding or some other cause. If the random number generated in this case was between 1 and the sum(probs), the returned index would be 9 into a (9,) shaped array, and triggering an index error (this is an example of the error).
Now, the returned idx is limited to the length of the vector. This is a slight fudge, i.e., making the probability of the last cell slightly larger than it would otherwise be, but it does not appear to be an issue.

Codecov Report

Merging #118 (a21e4e6) into develop (b50147b) will decrease coverage by 0.12%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop     #118      +/-   ##
===========================================
- Coverage    80.36%   80.24%   -0.13%
===========================================
  Files           11       11
  Lines         1966     1969       +3
===========================================
  Hits          1580     1580
- Misses         386      389       +3

Impacted Files
Coverage Δ

pyDeltaRCM/shared_tools.py
48.48% <0.00%> (-1.52%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b50147b...a21e4e6. Read the comment docs.

outdated change due to changes in #133


Logged in admins do not lose admin status when it is changed + deleted users aren't kicked out

Describe the bug
When an admin removes the admin status of itself, the status does not update until he logs out and logs in again. This also happens when an admin updates the status of another admin that is currently still logged in -> could pose large security problem. Deleted user's also aren't kicked out and can still book desks.
To Reproduce
Steps to reproduce the behavior:

Go to user management overview as an admin using the production server
Click on the admin checkbox of the logged in admin
The admin should now no longer be an admin and should not be able to access the user management page
However the logged in user can now just reclaim his admin priviledges

Expected behavior
The change in admin status should immediately change so that logged in previous admins can not just reclaim their admin status immediately. Deleted user's should also be kicked out when their account is deleted.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Windows 10
Browser: Chrome
Version [e.g. 22]

Smartphone (please complete the following information):

Device: -
OS: -
Browser: -
Version: -

Additional context
Add any other context about the problem here.

Das geht eigentlich schon wir schmeißen ihn nur nicht raus. Wenn der admin sich wieder neu anmeldet ist er keiner mehr.

One can not change it's own access rights


Opencv intellisence fails

I am writing a dockerfile and I have RUN pip install opencv-python
Pip will install the opencv as sudo in the /usr/local/lib/python3.6/site-packages/cv2
I cannot get the autocomplete feature.

Here is my tracing output:
[Info  - 12:02:54 AM] Pylance language server 2021.1.3 (pyright 61c53da4) starting
[Info  - 12:02:54 AM] Server root directory: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist
[Info  - 12:02:54 AM] No configuration file found.
[Info  - 12:02:54 AM] Setting pythonPath for service "vscode": "/usr/bin/python3"
Search paths found for configured python interpreter:
  /usr/lib/python3.6
  /usr/lib/python3.6/lib-dynload
  /home/vscode/.local/lib/python3.6/site-packages
  /usr/local/lib/python3.6/dist-packages
  /usr/lib/python3/dist-packages
[Error - 12:02:54 AM] stubPath /home/vscode/typings is not a valid directory.
[Info  - 12:02:54 AM] Assuming Python version 3.6
[Info  - 12:02:54 AM] Assuming Python platform Linux
[Info  - 12:02:54 AM] Searching for source files
[Info  - 12:02:54 AM] Found 4249 source files
[Info  - 12:02:54 AM] Background analysis(1) root directory: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist
[Info  - 12:02:54 AM] Background analysis(1) started
Background analysis message: setConfigOptions
Background analysis message: setTrackedFiles
Background analysis message: markAllFilesDirty
Background analysis message: setFileOpened
Background analysis message: getSemanticTokens
[BG(1)] parsing: /home/vscode/x.py (15ms)
[BG(1)] parsing: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/stdlib/3/builtins.pyi [fs read 2ms] (128ms)
[BG(1)] binding: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/stdlib/3/builtins.pyi (48ms)
[BG(1)] binding: /home/vscode/x.py (0ms)
[BG(1)] parsing: /usr/local/lib/python3.6/dist-packages/cv2/__init__.py [fs read 0ms] (8ms)
[BG(1)] binding: /usr/local/lib/python3.6/dist-packages/cv2/__init__.py ...
[BG(1)]   parsing: /usr/local/lib/python3.6/dist-packages/cv2/data/__init__.py [fs read 0ms] (0ms)
[BG(1)]   binding: /usr/local/lib/python3.6/dist-packages/cv2/data/__init__.py (0ms)
[BG(1)] binding: /usr/local/lib/python3.6/dist-packages/cv2/__init__.py (3ms)
[BG(1)] parsing: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/stdlib/2and3/_typeshed/__init__.pyi [fs read 1ms] (13ms)
[BG(1)] binding: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/stdlib/2and3/_typeshed/__init__.pyi (2ms)
[BG(1)] parsing: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/third_party/2and3/typing_extensions.pyi [fs read 0ms] (6ms)
[BG(1)] binding: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/third_party/2and3/typing_extensions.pyi (2ms)
[BG(1)] parsing: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/stdlib/3/typing.pyi [fs read 2ms] (38ms)
[BG(1)] binding: /home/vscode/.vscode-server-insiders/extensions/ms-python.vscode-pylance-2021.1.3/dist/typeshed-fallback/stdlib/3/typing.pyi (12ms)
Background analysis message: getSemanticTokens
Background analysis message: analyze
[BG(1)] analyzing: /home/vscode/x.py ...
[BG(1)]   checking: /home/vscode/x.py (2ms)
[BG(1)] analyzing: /home/vscode/x.py (2ms)
Background analysis message: resumeAnalysis

Solution:
Opencv IntelliSense fails if it is installed in /usr/local/lib/pythonX/site-packages. I have to install it using pip install --user which is not common in many Dockerfiles.
It works fine if it is installed under $HOME/.local/lib/pythonX/site-packages
How to reproduce the issue:

make sure to deactivate conda and other environments.
sudo pip3 install opencv-python
make sure it is installed under  /usr/local/lib/pythonX/site-packages : python -c "import cv2;print(cv2.__file__)"
open vscode and see IntelliSense fails

fix:
sudo pip uninstall opencv-python
pip install opencv-python --user

PS. I know we should never use sudo pip but sometimes this is unavoidable especially in containers and other provided images. There is something wrong with opencv package being in this directory for intellisense.
Related issue:
https://github.com/microsoft/pylance-release/issues/138

I am running into the same issue. Except doing
sudo pip uninstall opencv-python
pip install opencv-python --user

Doesn't solve my issue.
I don't know how to get tracing info. If someone can show me, I can share

Check where your opencv is installed.
for tracing see the related issue. Someone from pylance team shared a link

https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#filing-an-issue has some instructions for trace logging.

I am on macOS. I see a folder called cv2 here
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
I also see a folder call cv2 here
/Users/<user>/Library/Python/3.8/lib/python/site-packages

I'm most interested in trace logs if you can provide them, thanks.

I am thinking my sys.path is a bit messed up. anyway here is the log
log.txt

That's the "Python" output window; I'm looking for the "Python Language Server" output window when you have logLevel set to Trace.
This is mainly to look for log lines that show that we aren't loading our bundled native stubs.

I am getting it from here? It says Python Language Server on the top right?

My mistake, thanks.

what is your output for this?
import cv2
print(cv2.__file__)

Something is wrong with your python

Yes, we're failing to even run python to get the normal info we need (sys.path, the python version, etc). That protobuf pth file appears to be causing issues. If we can't call python, we probably won't be able to resolve much of anything. I don't know if your issue is related to the OP.

thanks all - I figured it out. I had a python2 package in my PYTHONPATH. And when I was testing I only reloaded the window. But you have to close the entire process to reset env for vscode.

closing this issue since I was related to opencv being installed in a different location than the interpreter.

The next release of Pylance will have a partial stub for opencv, and this should help fix issues where we couldn't match the native stub to the installed user version of opencv.


Failed redirection to calling screen after profile update action

After redirecting & making changes in the profile page, the application should redirect back to the page from where the profile action was called. Currently, it redirects back to the dashboard, and the project for which the changes were made is lost. User needs to trace back from browser to reach-back the project searched from the list.

Yeah I agree with this.


logging Variable order on puppet 3.x with Ruby 1.8

All Variable vaule has be changed ,  and will not affect the order?
Test steps are as follows

define node.pp

node 'hostname' {
    logging = > {
        'level' = > info,
        'to_files' = > false,
        'to_syslog' = > false,
        'files' = > {
            path = > '/var/log/filebeat',
            keepfiles = > '7',
            name = > 'filebeat.log',
        },
    },
}

Corresponding to the generated files

# more /etc/filebeat/filebeat.yml
……
  logging:
    to_syslog: false
    to_files: false
    level: info
    files:
      path: /var/log/filebeat
      name: filebeat.log
      keepfiles: 7

Change file order

node 'hostname' {
    logging = > {
        'files' = > {
            path = > '/var/log/filebeat',
            keepfiles = > '7',
            name = > 'filebeat.log',
        },
        'level' = > info,
        'to_files' = > false,
        'to_syslog' = > false,
    },
}

Didn't change anything with puppet

because no value change*

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for monster-agent-test-172-16-14-237
Info: Applying configuration version '1460095419'
Notice: Finished catalog run in 0.19 seconds

Change to_files value , Puppet Changed, but order has not changed.

node 'hostname' {
    logging = > {
        'files' = > {
            path = > '/var/log/filebeat',
            keepfiles = > '7',
            name = > 'filebeat.log',
        },
        'level' = > info,
        'to_files' = > false,
        'to_syslog' = > false,
    },
}

+++ /tmp/puppet-file20160408-25677-10ar15d-0	2016-04-08 14:08:19.505687725 +0800
@@ -19,7 +19,7 @@

   logging:
     to_syslog: false
-    to_files: false
+    to_files: true

     files:
       path: /var/log/filebeat

Wish you a good day, and i'm looking forward to your reply.

What behavior are you expecting here that it's not doing?

This is not a problem, I retry testing is work when I clean test environment.  :-)


upgrade vulcan-subscribe to use example-forum for posts & categories

Hi @SachaG, I noticed that the refactoring that consolidated vulcan:posts and vulcan:categories into example-forum in 6a9f53e and 62dffa6a9d91b0d1351062ae8ccaf68692dd3748 didn't update the vulcan-subscribe module to work with it. I hope this patch should fix it.

@SachaG why is a core package referencing example-forum?

This is leftover code from before, I don't use that package myself and I didn't create it so I haven't really been maintaining it…


minor speed improvements

Task/Issue URL: https://app.asana.com/0/392891325557410/561256581080450
Tech Design URL:
CC:
Description:

When browsing on the same site there is no need to re-inject the scripts
Show progress at the earliest opportunity after the user decides to navigation (ie start showing progress as soon as the decision is considered allowed)
Drop content blocking scripts for duckduckgo.com

Steps to test this PR:
Test 1

Search for cnn https://duckduckgo.com?q=cnn
Expected: site should be more responsive to load
Expected: content blocking scripts have not been injected (use Web Inspector to confirm)
Expected: Site has an A grade
Expected: Tab has correct favicon in Tabs view

Test 2

Follow link to cnn.com
Expected: Progress bar starts immediately
Expected: Appropriate site grade is displayed

Test 3

Click on a link on cnn.com
Expected: loading seems more responsive to load
Expected: site grade and privacy details are appropriate

Internal references:
Software Engineering Expectations
Technical Design Template

@brindy Speaking of small performance improvements, have you considered tweaking JavascriptLoader's load(script:withReplacements:into:forMainFrameOnly:injectionTime:) method? What I have in mind is:
for (key, value) in replacements {
    js = js.replacingOccurrences(of: key, with: value)
}

where, how I understand, you process string and replace all pre-set (and pre-determined) values. I figured out that adding .literal option wouldn't hurt:
js = js.replacingOccurrences(of: key, with: value, options: .literal)

That change reduced time to process CNN site by 100ms on my iPhone SE. I briefly re-checked the output of that code and everything seems to be correctly replaced.
Just my $0.02 :)

Thanks @bwaresiak - I'll take a look against the 5S. Sounds like it wouldn't do any harm anyway.


Fluorine considered metal

According to https://github.com/mcs07/MolVS/blob/26179ca8c74ad6c92452d22bca9031e7091beed8/molvs/metal.py#L33
Fluorine is a metal. Ouch!
It seems that 'r' is missing, and that's actually Francium, not Fluorine.

Oops, that's an embarrassing one...


(PUP-3807) Sort puppet config output

Sort puppet config output based on the key.

CLA signed by all contributors.


GetCurrentUserInfo is throw before loading info completed

There are some strange behavior in file user.service.ts
getCurrentUserInfo(): UserInfo | undefined {
  let currentUserInfo;
  let isInitialized = false;

  this.userInfo.subscribe(userInfo => {
    currentUserInfo = userInfo;
    isInitialized = true;
  }).unsubscribe();

  if (!isInitialized) {
    throw new Error('Cannot get current UserInfo: userInfo is not available yet');
  }

  return currentUserInfo;
}

Sometimes when I test my code, it shows the Cannot get current UserInfo: userInfo is not available yet exception. I think the userInfo.subscribe should be synchronous.
To make sure, I try to add some delays in getInitialUserInfo() and this error is reproduce.

It's the expected behavior, as it's the current value. The method might not be clear (sorry!) You should depend on this method when you want to use the current userInfo locally, which might be outdated. If you want the latest userInfo, please create a new method that call the API directly. (maybe rename + comment the old one to prevent confusion).


Prevent tests failing to run with no GUI

If you try to run the Iris tests via the command-line with no GUI available at all (for example when using SSH) then the tests will fail:
======================================================================
ERROR: Failure: RuntimeError (could not create GdkCursor object)
----------------------------------------------------------------------
Traceback (most recent call last):
  ...
  File "/.../lib/iris/tests/__init__.py", line 73, in <module>
    import matplotlib.pyplot as plt
  ...
  File "/.../matplotlib-1.3.1-py2.7-linux-x86_64.egg/matplotlib/backends/backend_gtk.py", line 58, in <module>
    cursors.MOVE          : gdk.Cursor(gdk.FLEUR),
RuntimeError: could not create GdkCursor object

It looks like pyplot tries to set a GUI cursor thing at import time, assumedly because it defaults to using the GTKagg backend, before we change the backend to Agg for the purposes of the Iris tests.
This proposed fix rather crudely just captures the runtime error on pyplot import and sets the use matplotlib flag false, so that no visual tests are run in such cases.
Alternatively, I guess, we could set the backend as standard to Agg before importing pyplot, and not do this later on (i.e. on __init__.py L125).

i can see the motivation for this, but I don't like the implementation
I would rather investigate other options, rather than capturing more exceptions in a try block.
Is the intent here to be able to run graphical tests without a display (possible, I think) or just skip them?
i think the 'switch backend' approach is better

i think the 'switch backend' approach is better

Thanks @marqh - I agree and my most recent commit demonstrates that approach to fixing this.

Those random failures are worrying. Is there an issue which relates?

Is there an issue which relates?

I don't believe there is.


some questions about the implement of CSPN

Hi， thank you for you brilliant code, it's very helpful.
I have a question about the sample and stitch steps of model PENet_C2_train in the  model, sample operation in line 921 and stitch operation 965. Where can I read the paper of it, I noticed that https://github.com/JUGGHM/PENet_ICRA2021/issues/16 ,https://github.com/JUGGHM/PENet_ICRA2021/issues/59 some issue called it DE-CSPN++, but I did not find the paper. Is it a variant of RA-CSPN?

the second question is about the propagation step (forward) of function CSPN and CSPNAccelerate, I notice that not only did you input the guide and depth (hn), but you also included the original depth  (h0) as input, and the center guide is used in the the original depth (h0),  which is consistent with the original CSPN paper (the Eq.1 of 3.1).
However，the CSPN code did not do so ( or maybe i didn't find it).
I'm just curious how this two implements are done affects the outcome, and what experience do you have with this

Hello, have you solved your problem? I have the same question.

Hello, have you solved your problem? I have the same question.

No，I got some answer of my second question in my own experiment， take original depth (h0) into the CSPN will slightly improve the accuracy
------------------ 原始邮件 ------------------
发件人: @.>;
发送时间: 2023年2月13日(星期一) 中午1:25
收件人: @.>;
抄送: @.>; @.>;
主题: Re: [JUGGHM/PENet_ICRA2021] some questions about the implement of CSPN (Issue #60)
Hello, have you solved your problem? I have the same question.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

No，I got some answer of my second question in my own experiment， take original depth (h0) into the CSPN will slightly improve the accuracy
…
------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年2月13日(星期一) 中午1:25 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [JUGGHM/PENet_ICRA2021] some questions about the implement of CSPN (Issue #60) Hello, have you solved your problem? I have the same question. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
我们能加一个联系方式吗？

加我的微信号把，chuis1012
------------------ 原始邮件 ------------------
发件人: @.>;
发送时间: 2023年2月13日(星期一) 下午3:34
收件人: @.>;
抄送: @.>; @.>;
主题: Re: [JUGGHM/PENet_ICRA2021] some questions about the implement of CSPN (Issue #60)
No，I got some answer of my second question in my own experiment， take original depth (h0) into the CSPN will slightly improve the accuracy
…
------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年2月13日(星期一) 中午1:25 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [JUGGHM/PENet_ICRA2021] some questions about the implement of CSPN (Issue #60) Hello, have you solved your problem? I have the same question. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
我们能加一个联系方式吗？
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


[Feature] LexicalMap

Apply a StringReplace of a certain function on lexical patterns
LexicalMap[f, expr, lp]

LexicalMap[ToUpperCase, "This is cool", TextType["Adjective"] | TextType["Verb"]]

(* expected output *)
"This IS COOL"

And of course with a levelspec: LexicalMap[f, expr, lp, pos]

Wait, level spec doesn't make any sense, expr is a string..

Simple enough to implement.

This already works with StringReplace, though I like the design of Map.
LexicalMap[ToUpperCase, "This is cool", TextType["Adjective"|"Verb"]]

vs
StringReplace["This is cool", LexicalPattern[match:TextType["Adjective"|"Verb"] :> ToUpperCase[match]]]

I've implemented it, but some longstanding behavior has reared its head. Note what gets replaced around "is cool" the spacing is inconsistent. an adjective text type should match just the text type and the spaces, but only bind to the word.

Maybe setting the MatchTrim option in the StringReplace UpValue?


Update cache action versions

Description of change
updated github cache action to the latest version
Type of change
Add an x to the boxes that are relevant to your changes.

[ ] Bug fix (a non-breaking change which fixes an issue)
[X] Enhancement (a non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Fix

How the change has been tested
run in this PR
Change checklist
Add an x to the boxes that are relevant to your changes.

[X] I have followed the contribution guidelines for this project
[X] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes

Should we update the cache action version for the examples workflow to 2.1.6 too, or convert it to use the new cache action?

I will look into it, sorry for the delay.


Signed integer support missing?

Unless I am missing something, the documentation mentions supporting reading signed integers; but there's no ReadInt16/ReadInt32 method in the client.

Hi Stephen,
wouldn't simply using a cast work in this case? This is how I usually do it:

read a 16-bit, signed integer:

var  reg uint16
err, reg = client.ReadRegister(0x1000, modbus.HOLDING_REGISTER)
if err != nil { ... }
fmt.Printf("unsigned value: %v, signed value: %v\n", reg, int16(reg))

write a 16-bit, signed register:

var setpoint int16
setpoint = -100
err = client.WriteRegister(0x1000, uint16(setpoint))
Hope this helps,
-Simon


updated getting started guides to point to runtime and designer statu…

…s for .NET Conf

Docs Build status updates of commit 30bfb29:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-edge/webview2/gettingstarted/winforms.md
:white_check_mark:Succeeded
View

microsoft-edge/webview2/gettingstarted/wpf.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


feat: clear button for ion-select for multiple

It would be nice for have select and/or clear all button on multi select.
I found one old ticket https://github.com/ionic-team/ionic/issues/12366 which also address this feature.
Current Version : 5.4.15 does not have this feature
Thanks

Thanks for the feature request. Can you provide your use case for how you would use this feature?

You dont have to open again ion-select to unselect one by one (which one you dont want).
But clear icon cant be added in ion-select like we have in searchbar.
It would be great to have in ion-select popup, when you use ion-select as a multiple select. first item will be select all/unselect all.

Alternatively you can wrap the ion-icon inside a div and make a custom button/use ion button with icon only and strip the rest with CSS

Thanks for the feature request. After discussing this with the team, we have decided not to add this. ion-select is intended for simple list selection. If there is a need for a select all/none button because there are many select options, we recommend using a modal instead. Using a modal allows for more customization such as select all/none and searching. Thanks!


s3 list_object_versions ignores Prefix on DeleteMarkers.

Hi,
after deleting some objects in a bucket with versioning, when I search with client.list_object_versions(Bucket='myBucket', Prefix='MyKey') only objects with the key "MyKey" should appear in the result. For "Versions" this is true, for "DeleteMarkers" the prefix is ignored.
If I do the same with MinIo the expected result appears.

It looks like the error is on my side. I'll search a bit further and if it's an error I'll open the error again.

Unfortunately there was my error and the previous mentioned bug.
And now i have also some code to show the issue.
``
from moto import mock_s3
import boto3
def create_objects(client, bucket_name):
    for pos in range(2):
        client.put_object(Bucket=bucket_name, Key=f'obj_{pos}', Body=f'This is object {pos}'.encode('utf-8'))

    for pos in range(2):
        client.put_object(Bucket=bucket_name, Key=f'hist_obj_{pos}', Body=f'This is history object {pos}'.encode('utf-8'))
        for hist_pos in range(2):
            client.put_object(Bucket=bucket_name, Key=f'hist_obj_{pos}',
                              Body=f'This is history object {pos} {hist_pos}'.encode('utf-8'))

    for pos in range(2):
        client.put_object(Bucket=bucket_name, Key=f'del_obj_{pos}', Body=f'This is deleted object {pos}'.encode('utf-8'))
        client.delete_object(Bucket=bucket_name, Key=f'del_obj_{pos}')

if __name__ == "__main__":
    BUCKET_NAME = "tests_bucket"
    s3_mock = mock_s3()
    s3_mock.start()
    # client = boto3.client("s3", endpoint_url="http://localhost:5000")
    client = boto3.client("s3")
    client.create_bucket(Bucket=BUCKET_NAME)
    client.put_bucket_versioning(Bucket=BUCKET_NAME, VersioningConfiguration={
        'MFADelete': "Disabled",
        'Status': "Enabled"})
    create_objects(client, BUCKET_NAME)

    objs = client.list_object_versions(Bucket=BUCKET_NAME)
    hist_objs = client.list_object_versions(Bucket=BUCKET_NAME, Prefix="hist_obj")
    del_objs = client.list_object_versions(Bucket=BUCKET_NAME, Prefix="del_obj_0")

Thanks for raising this and for providing a test case @FZetting - marking it as a bug :+1:


Update network-requirements.md

added IPs to portal access and siem agent sections removed third party dlp integration- approved by Lilach Tofach Zohar removed a domain in the commercial customers section

updated from the private repo instead
#please-close


Introduce login with Microsoft

Introduces a second oauth login
README describes how to turn it on and configure it
Access the auth token via User#microsoft_graph_credential.oauth_token to interact with Microsoft Graph API
If microsoft account does not have a full name, login/signup will fail. Future work might allow a user to provide these missing details in hostedgpt.
Future work: refactor out common code + routes with google auth

Sequence flow:

@drnic I hoped to finish reviewing today but I didn't make it. I should be able do this tomorrow.

Thanks for chasing down all these details! It looks great, merging now.


add --port parameter for rasa interactive

Proposed changes:

Added --port parameter for the user to give any port for Rasa interactive learning.  This parameter will be used to start core and followed by interactive learning.
close https://github.com/RasaHQ/rasa/issues/4709

Status (please check what you already did):

[x ] made PR ready for code review
[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[x] reformat files using black (please check Readme for instructions)

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the PR! We'll give it a review as soon as possible.

Thanks for the PR! We'll give it a review as soon as possible.

Hi @erohmensing ,  This is the first time I am trying to contribute for an issue.  Not sure if I have followed all the standards correctly.

@pnandhini11 so far things look pretty good! A few changes I can see here:

reformat files using black (please check (Readme)[https://github.com/RasaHQ/rasa_nlu#code-style] for instructions)
update the changelog

@pnandhini11 so far things look pretty good! A few changes I can see are still necessary:

reformat files using black (please check Readme for instructions)
update the changelog

For this PR, I don't think an automated test or updates to the documentation is necessary.

Do I have reformat and checkin again?

What do you mean? For reformatting, you should run black via the instructions, and then push any changes that are made to the files.

@erohmensing  Hi, I am using windows system and not able to run pytype from windows. Can you help me solve this check?  Also you mentioned automated test not required but build failed in test as well.

I think this is a good place for your reviewer @ArjaanBuijk to step in to help you get the tests passing 🙂

Hi @erohmensing , thanks for the reviews so far.
Hi @pnandhini11 , I will have a look at the tests to check why they fail.

Hi @pnandhini11 ,
A few things need to be fixed before your update can be merged. The tests fail, and you can view what tests by clicking on the Details link next to the Tavis CI - Pull Request:

This brings you to the page where it lists the tests that fail, and you can click to inspect the details:

The best way to make these changes and then verify that all is OK, is to run the tests locally. It is described in the README file how to do this. See these sections:

Type Checking
Running the tests

I had a look at the failing tests and these functions need to be updated:

rasa/cli/arguments/interactive.py:

line 44, in set_interactive_arguments: Function add_port_param was called with the wrong arguments [wrong-arg-types]
Expected: (parser: argparse.ArgumentParser)
Actually passed: (parser: argparse._ArgumentGroup)
line 67, in set_interactive_core_arguments: Function add_port_param was called with the wrong arguments [wrong-arg-types]
Expected: (parser: argparse.ArgumentParser)
Actually passed: (parser: argparse._ArgumentGroup)

tests/cli/test_rasa_interactive.py::test_interactive_core_help

tests/cli/test_rasa_interactive.py::test_interactive_help

Let me know if it is clear what to do.

@pnandhini11 , pease see my comments about the failing tests.

Hi @ArjaanBuijk , @erohmensing ,
Thanks for the reply.  I couldn't use Pycheck as I am having windows system and no pycheck for windows yet.  Thats why I needed your help.
Have a question on this  :
(Copy from your previous message )
rasa/cli/arguments/interactive.py:
line 44, in set_interactive_arguments: Function add_port_param was called with the wrong arguments [wrong-arg-types]
Expected: (parser: argparse.ArgumentParser)
Actually passed: (parser: argparse._ArgumentGroup)
line 67, in set_interactive_core_arguments: Function add_port_param was called with the wrong arguments [wrong-arg-types]
Expected: (parser: argparse.ArgumentParser)
Actually passed: (parser: argparse._ArgumentGroup)
My Question :  I am not sure why to use _ArgumentGroup to pass the parameters to add_port_param method.  I have made it using (parser: argparse.ArgumentParser) because to be inline with existing code followed in the entire repository.
You can check the existing argument coded in interactive.py :
def add_skip_visualization_param(parser: argparse.ArgumentParser):
parser.add_argument(
"--skip-visualization",
default=False,
action="store_true",
help="Disable plotting the visualization during interactive learning.",
)
Let me know how to go about it now. Thanks in advance.

@erohmensing @ArjaanBuijk  any updates to my previous comment?

@pnandhini11 , sorry for delay in response. I will be able to review the issues later today and will then get back to you on them.

@wochinge ,
Is there a reason that rasa/cli/arguments/interactive.py is so different from rasa/cli/arguments/shell.py ?
I see two ways to implement the --port parameter for rasa interactive:

Just add the --port parameter as @pnandhini11 has implemented.
from rasa/cli/arguments/interactive.py call add_server_arguments , as is done in rasa/cli/arguments/shell.py

Is there a reason that rasa/cli/arguments/interactive.py is so different from rasa/cli/arguments/shell.py ?

shell is just speaking to a trained model. interactive is speaking to trained model, but also training a model and also visualizing things.

Just add the --port parameter as @pnandhini11 has implemented.
from rasa/cli/arguments/interactive.py call add_server_arguments , as is done in rasa/cli/arguments/shell.py

How about a middle course and

extract the current port argument declaration from add_server_arguments to a add_port function
calling add_port in add_server_arguments and set_interactive_arguments

Maybe we can also de-duplicate set_interactive_arguments and set_interactive_core_arguments a bit

Hi @pnandhini11 , would you be able to implement the functionality using an add_port function as @wochinge proposes? I like that approach a lot.

Hi @pnandhini11 , would you be able to implement the functionality using an add_port function as @wochinge proposes? I like that approach a lot.

Let me work on this and get back to you.

@pnandhini11 What's the state? Can we somehow help you?

Hi @wochinge , I made the updates as discussed but got busy with my work. And missed updating the status. Will do it by this weekend.

@pnandhini11 No rush, I was just curious about the status 👍

@ArjaanBuijk @wochinge Made changes in rasa\cli\arguments\interactive.py to call add_server_argument instead of add_port method i created last.
Check and let me know if i understood your idea correctly.
By mistake i deleted rasa\cli\interactive.py and added again in this commit. Didn't do any code changes in it.

@pnandhini11 How is it going?

wrapped up in https://github.com/RasaHQ/rasa/pull/5837


Darksidewalletd tests

Hi @pacu  .  I grabbed your branch, and rebased it on top of latest dev.   There's a test of functionality that we dropped.  So I deleted that test.
If you want to review this rebase and merge it into your PR that'll bring it up to date.
Regarding the removal of checkpoints, we'll need to discuss that with @AloeareV .

Regarding the removal of checkpoints, we'll need to discuss that with @AloeareV .

what do you mean by this?
the darksidewalletd tests need the 663150 checkpoint I added unless Zingo-lib now grabs treeStates all the way and don't need the stored now anymore. that could work

🎉


[core] Partitions system table support time travel (#4511)

pick: https://github.com/apache/paimon/pull/4511

There are test failure


Github Sync fails with OpenSSL error

Trying to setup the Github Sync fails with the following error:
Unable to proceed because Github Downloader - InOutError Could not initialize OpenSSL library
OpenSSL library in Fedora Workstation 37 is installed, everything else seems to work.
Is there a way to write a log?

Hmm, thats an interesting one. I suspect there may be an SSL incompatibility here. Its not unusual when working with SSL !
You might see a more detailed error report if you run from the command line. Where I can, I drop things like that to console.
But I suspect I might need to do some debugging here.  Is yours a fresh install or an upgrade from previous versions ?  Has it worked with previous versions of Fedora ?
Thanks for the report !
Davo

No Chlorine8477, I have not been able to replicate the issue here. Ran it up on a F 37 vm, it synced against github just fine for me. I even tried updating the OS thinking maybe a new libssl was an issue, still fine.  Have you updated there ?
Now, I "believe" I have trapped out all the possible token or network errors (and give you a more useful error message) but I guess we have to assume I have missed one. I cannot think what else it could be. Maybe I get you a binary with a bit more debugging going on in that section of the code and you run it, show me the results ?
Which widget set are you using, gtk2 or Qt5 ?
Davo
Chlorine ? Its a poisonous gas !

Good morning David,
I assure you, I am not really toxic, but it makes people weary ;)
To the matter at hand: I am running tomboy-ng on GTK2 on a very updated Fedora Workstation 37, fresh out of the box.
BUT it might be the 64-bit version, I will try the 32 bit version later today.
strace did not really output anything interesting btw....
I'll get back to you later

I sort of doubt its a 32/64 bit issue although I do know that is handled differently on Fedora than the deb based systems I am familiar with. Its an I/O error and, as near as I tell, the httpclient unit itself does not raise an EInOutException. That suggests to me that the problem relates, some how, to being unable to open libssl. It also would seem unlikely that you could have a libssl problem and be otherwise unaware of it ! Most unlikely...
Did your strace indicate that libssl was opened OK ?
Even less likely is a failure to save a downloaded file, perhaps because of write protection. I am pretty sure the error occurs before download, not after. But ...
A mystery indeed.
Davo
Warning, don't upload to here an strace with having a good look first, I suspect it could show your github token !

Hey Chlorine, any further thoughts on this ?
Davo

One way or another, this issue seems to be a stale one.
Thanks for your report.
Davo

In case anyone has trouble with this on ubuntu - this solved it for me (not sure if it's the right way to go, but it seems to work now)
sudo apt-get purge openssl
sudo apt-get install libssl-dev

Indeed, cleverfrog, installing libssl-dev is a solution for linux users. Lazarus, when it cannot find a suitable version openssl tries the non-versioned version, which is provided by the -dev package (actually just a symlink without any version number pointing the the installed version).
I currently have a version out, built with FPC323 that directly solves this problem.  Hope to release it soon.
thanks,
Davo

This matter addressed in the new, 0.40 release of tomboy-ng.
Thank you for you input.
David


Generating C# code from the Open API spec results in code that fails desrialize

Summary
Generating C# API from the Open API spec results in code that fails to deserialize results because certain DateTime values are occasionally null.
fail: Example.ArgoCommands.ArgoListCommand[0]
      Failed with Exception
      Newtonsoft.Json.JsonSerializationException: Error converting value {null} to type 'System.DateTime'. Path 'items[5].spec.volumeClaimTemplates[0].metadata.creationTimestamp', line 3043, position 39.
       ---> System.InvalidCastException: Null object cannot be converted to a value type.
         at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
         --- End of inner exception stack trace ---
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)

Diagnostics
What Kubernetes provider are you using?
OpenShift 4.7
What version of Argo Workflows are you running?
Argo 3.0.4
As this is more of a serialization issue, its easiest to just show an example of a real workflow spec where this occurs.  We don't specify meta information like creationTimestamp in our spec anyway, its added automatically.  So this is one example:
      volumeClaimTemplates:
      - metadata:
          creationTimestamp: null
          name: workdir-volume
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 64Gi
          storageClassName: basic
        status: {}

We have also seen this in:
  status:
    conditions:
    - status: "True"
      type: Completed
    estimatedDuration: 117
    finishedAt: "2021-05-14T13:57:12Z"
    message: 'runtime error: invalid memory address or nil pointer dereference'
    nodes:
      migration-wft-bkznt:
        children:
        - migration-wft-bkznt-51186898
        displayName: migration-wft-bkznt
        estimatedDuration: 117
        finishedAt: null
        id: migration-wft-bkznt
        message: Workflow operation error
        name: migration-wft-bkznt
        phase: Error
        startedAt: "2021-05-14T13:57:12Z"
        templateName: migration
        templateScope: local/
        type: Steps

Our current workaround is to post process the generated code with commands like:
sed -i \
  's/public DateTime FinishedAt/public DateTime? FinishedAt/g' \
  "./src/${PACKAGE_NAME}/Model/IoArgoprojWorkflowV1alpha1NodeStatus.cs"

But this is definitely a hack.  Is there a way to mark these status related date fields nullable so that they will deserialize properly?

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

I think we do something similar for Java. Maybe check out that SDK?

@alexec i am having trouble following the java sdk generation, so i have a few questions if you have time to answer them.
Specifically, there are some seemingly arbitrary choices for name qualifiers to strip:
curl https://raw.githubusercontent.com/argoproj/argo/$(VERSION)/api/openapi-spec/swagger.json \
  | sed 's/io.argoproj.workflow.v1alpha1.//' \
  | sed 's/io.k8s.api.core.v1.//' \
  | sed 's/io.k8s.apimachinery.pkg.apis.meta.v1.//'  \
  > swagger.json

Which leaves some argo names shortened (ie: Workflow) while others remain long (ie: IoArgoprojEventsV1alpha1EventSource.  Is this desirable?  Was the failure to strip event classes an oversight?
It also does some restructuring to put the client out as a top level package (--invoker-package io.argoproj.workflow).  Again this appears to be convenience, but could easily lead to clients generated for other languages to differ in uncomfortable/surprising ways.
In order to integrate with the kubernetes GenericClient (which seems like an intended path) the top level model objects must extend from IKubernetesObject. I assume this was never added because its an external API, and maybe there is some simple config to tell the generated to add that as a parent class. But my solution is to leverage the partial class nature of the model objects with more after the fact monky patching:
>&2 echo "add the IKubernetesObject interface to model object to work with k8s.GenericClient"
cat > "./src/${PACKAGE_NAME}/Model/AddIKubernetesObject.cs" <<EOF
namespace ${PACKAGE_NAME}.Model
{
    using k8s;
    public partial class IoArgoprojWorkflowV1alpha1CronWorkflow : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1CronWorkflowList : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1Workflow : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1WorkflowList : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1WorkflowEventBinding : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1WorkflowEventBindingList : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1WorkflowTemplate : IKubernetesObject
    {
    }
    public partial class IoArgoprojWorkflowV1alpha1WorkflowTemplateList : IKubernetesObject
    {
    }
}
EOF

This feels wrong, is there a better way?
Finally, back to the original point of date values, the java code switches to joda which has a constructor which accepts null and turns that into current date

Parameters:
instant - the datetime object, null means now

So to mimic this, i could create a new Time datatype that accepts null during de-serialization, but this will just result in lies like: finishedAt: <now> when in reality it is finishedAt: null which could surface as harmful bugs, right?  I think it would make more sense to mark the fields nullable, wouldn't it?  Then the generation could would (in theory) generate the proper DateTime? in C#.  It could conscientiously be limited to those datetime values which actually allow null (if there are any that don't), but I definitely don't have the depth of knowledge to determine which would be which other than experimentally.

Is this desirable? Was the failure to strip event classes an oversight?

No, I don't think it is desirable. It should be fixed, this is a new feature which is probably why it was overlooked.
I wonder if this could be better fixed in the generation of the swagger? Have you taken a look at how this works today?
I'm not much of an expert on this I'm afraid?

I wonder if this could be better fixed in the generation of the swagger?

If it does get fixed in the swagger generation, it would ensure all generated code has common feel, so yeah, this would be ideal.

Have you taken a look at how this works today?

I have not.  Right now i am still trying to figure out the code generation side.  Cant get nullable to work, tried:
sed \
  's/^\(.*\"io.k8s.apimachinery.pkg.apis.meta.v1.Time\"\).*/\1: {"nullable": true,/g' \
  src/argo-workflows-${ARGO_VERSION}/api/openapi-spec/swagger.json" \
  > /tmp/customized_swagger.json

but failed:
--- src/argo-workflows-3.0.4/api/openapi-spec/swagger.json      2021-05-14 04:40:59.000000000 +0000
+++ /tmp/customized_swagger.json        2021-05-21 15:04:29.815840000 +0000
@@ -11996,7 +11996,7 @@
         }
       }
     },
-    "io.k8s.apimachinery.pkg.apis.meta.v1.Time": {
+    "io.k8s.apimachinery.pkg.apis.meta.v1.Time": {"nullable": true,
       "description": "Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON.  Wrappers are provided for many of the factory methods that the time package offers.",
       "type": "string",
       "format": "date-time"

Exception in thread "main" org.openapitools.codegen.SpecValidationException: There were issues with the specification. The option can be disabled via validateSpec (Maven/Gradle) or --skip-validate-spec (CLI).
 | Error count: 1, Warning count: 4
Errors:
        -attribute definitions.io.k8s.apimachinery.pkg.apis.meta.v1.Time.nullable is unexpected
Warnings:
        -attribute definitions.io.k8s.apimachinery.pkg.apis.meta.v1.Time.nullable is unexpected

So had to settle for the --import-mappings generator option:
  --import-mappings io.k8s.apimachinery.pkg.apis.meta.v1.Time=System.DateTime? \

From a deserialization side this will prevent any of the null date time values from failing to deserialize, but it will also allow for creation of (possibly) invalid request objects since it was applied to all time objects.  However i think this is the best I can do here.

I just ran into this as well. In my case it threw an exception when it tried to serialize the Workflow's status.startedAt property. It makes sense that startAt would be null if I Just submitted a new workflow. Will explore @lucastheisen 's workaround of using --import-mappings


🐛 [BUG] : Register, Complete Profile, Language Steps

Describe the bug
Test all possible scenarios like if user terminate app in any of the steps.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Fix here #413 as well


Support embedding Effect without "leaking"

Also consider Otel

It's kinda hard to tell from just the title and description what it is this issue should cover. Are you asking for the ability to create Effect-based super-types like Stream does for Effect?

It's kinda hard to tell from just the title and description what it is this issue should cover. Are you asking for the ability to create Effect-based super-types like Stream does for Effect?

that's to better support bundling effect into third party libraries like contentlayer

Addressed in https://github.com/Effect-TS/effect/pull/1564

Example project by @tim-smart 🙏
https://github.com/tim-smart/openai-client-demo


Duplicate CassandraCQLClient classes

Both the cassandra and cassandra2 bindings have a CassandraCQLClient class. They both need to exist because they handle connecting to different versions of Cassandra. They have the same package and class name: com.yahoo.ycsb.db.CassandraCQLClient.
In bin/ycsb cassandra-cql and cassandra2-cql bindings point to the same class: https://github.com/brianfrankcooper/YCSB/blob/master/bin/ycsb#L53-L54. This is probably an issue as the two classes act differently, and only one of them will ever get used regardless of the binding specified, likely comes down to ordering in the classpath or maybe even one of them gets kicked out during the build.
This may not be an issue if you are using a single module build of either cassandra or cassandra2, but will most likely affect users that use a release of the entire project.

This is confusing, and I'm definitely game to change it. However, the classpath loading logic should select the proper binding. From the full distribution, running cassandra-cql adds these entries:
/cassandra-binding/conf
/cassandra-binding/lib/antlr-2.7.7.jar
/cassandra-binding/lib/antlr-3.2.jar
/cassandra-binding/lib/antlr-runtime-3.2.jar
/cassandra-binding/lib/avro-1.4.0-cassandra-1.jar
/cassandra-binding/lib/cassandra-all-1.2.9.jar
/cassandra-binding/lib/cassandra-binding-0.5.0.jar
/cassandra-binding/lib/cassandra-driver-core-1.0.3.jar
/cassandra-binding/lib/cassandra-thrift-1.2.9.jar
/cassandra-binding/lib/commons-cli-1.1.jar
/cassandra-binding/lib/commons-codec-1.2.jar
/cassandra-binding/lib/commons-lang-2.6.jar
/cassandra-binding/lib/commons-logging-1.1.1.jar
/cassandra-binding/lib/compress-lzf-0.8.4.jar
/cassandra-binding/lib/concurrentlinkedhashmap-lru-1.3.jar
/cassandra-binding/lib/guava-13.0.1.jar
/cassandra-binding/lib/high-scale-lib-1.1.2.jar
/cassandra-binding/lib/httpclient-4.0.1.jar
/cassandra-binding/lib/httpcore-4.0.1.jar
/cassandra-binding/lib/jackson-core-asl-1.9.2.jar
/cassandra-binding/lib/jackson-mapper-asl-1.9.2.jar
/cassandra-binding/lib/jamm-0.2.5.jar
/cassandra-binding/lib/jbcrypt-0.3m.jar
/cassandra-binding/lib/jetty-6.1.22.jar
/cassandra-binding/lib/jetty-util-6.1.22.jar
/cassandra-binding/lib/jline-1.0.jar
/cassandra-binding/lib/json-simple-1.1.jar
/cassandra-binding/lib/libthrift-0.7.0.jar
/cassandra-binding/lib/log4j-1.2.16.jar
/cassandra-binding/lib/lz4-1.1.0.jar
/cassandra-binding/lib/metrics-core-2.0.3.jar
/cassandra-binding/lib/netty-3.5.9.Final.jar
/cassandra-binding/lib/servlet-api-2.5-20081211.jar
/cassandra-binding/lib/servlet-api-2.5.jar
/cassandra-binding/lib/slf4j-api-1.6.4.jar
/cassandra-binding/lib/slf4j-log4j12-1.7.2.jar
/cassandra-binding/lib/snakeyaml-1.6.jar
/cassandra-binding/lib/snappy-java-1.0.5.jar
/cassandra-binding/lib/snaptree-0.1.jar
/cassandra-binding/lib/stringtemplate-3.2.jar
/conf
/lib/core-0.5.0.jar
/lib/HdrHistogram-2.1.4.jar
/lib/jackson-core-asl-1.9.4.jar
/lib/jackson-mapper-asl-1.9.4.jar

Whereas cassandra2-cql has:
/cassandra2-binding/conf
/cassandra2-binding/lib/cassandra2-binding-0.5.0.jar
/cassandra2-binding/lib/cassandra-driver-core-2.1.8.jar
/cassandra2-binding/lib/guava-14.0.1.jar
/cassandra2-binding/lib/metrics-core-3.0.2.jar
/cassandra2-binding/lib/netty-buffer-4.0.27.Final.jar
/cassandra2-binding/lib/netty-codec-4.0.27.Final.jar
/cassandra2-binding/lib/netty-common-4.0.27.Final.jar
/cassandra2-binding/lib/netty-handler-4.0.27.Final.jar
/cassandra2-binding/lib/netty-transport-4.0.27.Final.jar
/cassandra2-binding/lib/slf4j-api-1.6.4.jar
/conf
/lib/core-0.5.0.jar
/lib/HdrHistogram-2.1.4.jar
/lib/jackson-core-asl-1.9.4.jar
/lib/jackson-mapper-asl-1.9.4.jar

Yes, OK. After looking closely at the classpath building I see that there actually shouldn't be an issue here as the correct module jar is chosen for the classpath based on the binding name. OK, I'm cool, why fix what's not broken?


KSU UAV data pipeline planning

This is a followup from call on 9/6/2016 including:

Mark Lucas, KSU, UAV data preprocessing + db storage and archiving
Geoff Morris, KSU, sorghum geneticist
Jesse Poland, KSU, more focused on wheat
Kevin Wang, KSU, ag engineer & data processing, e.g. NDVI + height from images
@pless, WashU
@smarshall-bmr, MAC scanner operator
myself and @jdmaloney, NCSA

...focused around setting up a pipeline between KSU and UI. Currently the data flow at KSU is as follows:

captured data is uploaded to a central server
data is processed to generate database entries + do file renaming
images are georeferenced
raw data is archived into folder structure
two primary DB tables store a) metadata about the images and b) UAV flight data including number of images, coverage, time, etc.

After discussion, current plan is generally:

I will modify the monitor service used at MAC and soon to be used at Danforth to support SQL query as a way to fetch list of files to transfer
Mark and I will coordinate to determine SQL queries necessary to:

get new list of images that have not yet been sent to NCSA
get metadata associated with those images

my monitor service will periodically query the DB for new files + metadata and transfer to NCSA

Other topics that arose include:

is there consideration of other data in the future for TERRA, e.g. wheat?
what is the 'hello world' extractor for UAV data?
@pless will be in Illinois in early October
does KSU have a Globus setup already? @jdmaloney thought the beocat Globus endpoint might be enough. What user will we use?

Regarding the SQL queries:
I plan to modify my code to use a Python SQL library like: http://www.tutorialspoint.com/python/python_database_access.htm …then I would imagine the logic can be:

Execute GetFileList SQL query to get a database cursor on a set of result rows – this should return a list of files and if possible their timestamps
Iterate across files and execute GetFileMetadata on each one (?) to fetch that file’s metadata
Bundle current file + metadata into a JSON file and queue for sending
Repeat for next file until we reach them all, and record the latest file we sent
Rerun GetFileList every x minutes, with query only returning files newer than the latest file from last time

So on KSU end the GetFileList and GetFileMetadata queries are the critical pieces. Pseudo:

GetFileList = select all filename, timestamp from files_table where timestamp > LAST_SEEN_TIMESTAMP
GetFileMetadata = select all ??? from file_info where filename = CURRENT_FILE

Thanks Max. I will work on getting example queries to you over the next few days.
There are two tables for the UAS image data sets:
uas_run: which stores information about each flight.
uas_images: which stores information about all of the image files associated with each flight.
For the GetFileList query, there is no timestamp indicating when the file was created stored in the uas_run table itself. However, there is a record ID which is an auto-incremented running number assigned to each row which could be used instead: i.e. instead of LAST_SEEN_TIMESTAMP we could used LAST_SEEN_RECORD_ID.
Regarding the GetFileMetadata query, there is an  flight_id stored in the uas_run table that is also stored for each metadata record in the uas_images table. This identifies all metadata records stored for each flight. So the query would for GetFileMetadata for a specific flight  would be to select all rows from uas_images where uas_images.run_id=uas_run.flight_id.
The image file sets are stored as tar files in a dedicated folder on our server.

One question regarding the use of Globus Connect: I got myself set up to use Globus Connect yesterday, and I have tested transferring files between my computer and the Beocat server. In order to do this, I was told that I had to have an account on Beocat.
Does this mean that anyone who wants to retrieve files from Beocat also has to have an account on Beocat? Or is it sufficient to share the folder containing the UAS files with a specific GLOBUS username?

@jdmaloney can you weigh in on Globus question above?
@marklucascork feedback re: SQL looks good - incrementing field is easier than a timestamp, and we should be able to build a JSON object from a set of metadata rows pretty readily.

@marklucascork Short answer is it depends.
It is possible to set up a Globus Share on an existing endpoint that does not require users to have valid auth on the underlying system.  This is what we do with the terraref endpoint that we host here at NCSA.  We can add individual Globus ID's to have read-only or read/write permissions on the directory being shared and they never have to have any NCSA credentials.
I don't believe this sharing feature is a free feature in Globus, but if KSU is paying Globus it may have access to this ability.  So I think the best thing may be to ask the admins on the system.  If they tell you they can create a shared endpoint, then that would work great and we'll get you a Globus ID to give permissions to.
Hope that helps, but please feel free to let me know if that doesn't clear something up, or if more questions arise.

Thanks @jdmaloney. I have aske our administrators about whether our Globus Connect endpoit supports sharing with valid Globus User ID.

Here are example SQL queries required for retrieving our UAS data:
A: GetFileList query
SELECT record_id,flight_id FROM uas_run WHERE record_id > 0;
The above example retrieves information about all entries in the uas_run table. If you wish to retrieve entries newer than the last one retrieved, replace 0 with the record_id of the last data set retrieved:
SELECT record_id,flight_id FROM uas_run WHERE record_id > LAST_SEEN_RECORD_ID;
B: GetFileMetadata
SELECT * FROM uas_images WHERE flight_id = 'uas_20160107_065204_20160107_065728’;
The above example will retrieve all metadata records for the flight_id uas_20160107_065204_20160107_065728. In order to get data for any other flight, simply use the flight_id(s) retrieved from the GetFileList query above.
The image files themselves are stored on Beocat in the folder /bulk/jpoland/images/uas. Each tar file contains the image files and any raw data associated with the flight e.g. log files.
I am hoping that we can use the Globus Share facility to share these with you. If not, you will need to set up an account on Beocat. I will send instructions on how to do this if this turns our to be necessary.
Please let me know if you have any questions.

@marklucascork this looks great. I'll work on getting SQL query support implemented in the pipeline monitor next week.

I tried sharing the uas folder on Beocat and found that sharing is not enabled at KSU. I verified this with the Beocat system administrators.
I believe that this means that you will need to get an account on Beocat in order to access the files that you need. I may also need to give you a read-only account on the database. However, we could probably work around this by invoking a script that queries the database using e.g. my credentials.
Instructions for getting a Beocat account:

The first thing that you need to do to get access is to apply for an EID. The link to the application page is http://eid.ksu.edu
Once you receive your EID, then you need to apply for an account on Beocat (the KSU cluster). The application page for Beocat is https://account.beocat.ksu.edu/user

Let me know when you have established your account on Beocat and we will take it from there.

This is a stale issue with no comments in last 2 months. Please create new issues if work remains or put this content in the documentation.


Scraper of Mobility reports v 2.0

Google recently published a mobility report with time-series in CSV format. You can download it on their website.
That means there's no need for a PDF file parser anymore. Due to that, I plan to change the concept of this repository.
Here are my points that I propose to implement here:

Archive the PDF parser as part of the great history of this repository.
Automatically download to this repository all available files (including PDF) on Google and Apple sites. If there are no problems with Google reports, the Apple website parser needs to be rewritten, because my ad-hoc solution does not work, unfortunately.

Make a simple visualization app for this data (for example, using Bokeh library).

Feel free to offer your suggestions here.
Thank you!

Great! I will pull this into balefire.info for USA.  The color shading was resolved as an FYI. I am using D3 and D3Plus for my visuals.  The drawback is my visuals are USA obviously.  I can look into the possibility of doing a global as it would involve mostly ignoring/reducing the data merges.

I got the google data in the system.  It is pretty interesting. I do need to sit down with it some time but there are some pretty telling Pearson coefficients correlating with it.  I also see that confirmed cases per 10k is higher when mobility data is higher. What I really should do is assess the log of that two weeks after to see if there is a correlation there. The graphs suggest that is the case.  I will put a screen shot below showing 4/11 and one plot of AK. As an FYI, the university is doing a short article on the tool next week so hope to get more info on our data out there.

@ladew222 Cool! I hope that I compile a summary file from Google and Apple reports in the next 2-3 days.

Wow cool.

I've recently made a couple of updates, so I summarize what's been done here:

Everything related to PDF parser now is in the directory "scraper v 1.0".
Apple report is now automatically downloaded to the repository every day. But with Google data now a little problem: if the CSV download is okay, the ability to download the PDF is now disabled, because the structure of Google webpage has significantly changed. But I think that's not a critical problem.
Also now automatically generated summary reports from Google and Apple data, which I mentioned above. They are available here. But some points should be noted here:

the matching of subregions from Google data with cities from Apple data needs to be further improved. Currently, they are matched as they are in the original data.
with the U.S. data is a serious problem because they are quite heterogeneous. So far, the cities are in the "sub_region_1" column. I think it is probably even better to remove the detailed breakdown by counties for the US from the summary report.
It is appropriate to adjust the baseline for Apple data for a longer period that intersects with the baseline Google period (e.g. January 13 to February 6). This is a rough approach, but I think it would be better than just taking the baseline for January 13th.

Google Sheets are now updated automatically.

Also, it is necessary to think about the view of data visualization app, which would provide simple answers about the mobility situation in a particular region.

Cool. Here is the choropleth of residential mobility as it is now if you havent seen it.

Wow, looks nice! But I couldn't reproduce this picture in your dashboard( I got it like this:

Maybe, I didn't press some button or checkbox?

My fault. It looks like Google does not have significant enough data for that metric. Retail and Recreation has the fuller map.

Got it, thanks! I suggest adding the ability to make a breakdown by states, it will allow us to see the picture throughout the United States.

Makes sense.  I will add that in.

On Apr 21, 2020, at 1:12 PM, ActiveConclusion<EMAIL_ADDRESS>wrote:
Got it, thanks! I suggest adding the ability to make a breakdown by states, it will allow us to see the picture throughout the United States.

Are you thinking about a map by states?

Yes

Last week's Update Digest:

The problem with downloading Google PDF reports fixed (I fixed this problem a week ago, just didn't write here).
Apple has added more regions/cities to their report. The main problem with it is that cities and subregions go without country names, but I have already fixed this issue (it was a challenging issue for me).
With the addition of new data from Apple, there are now huge problems with the merging of reports, the scale of which I have not even assessed yet.

Latest updates:

Recently, Google Sheets with US detailed data crashed due to a large amount of data for one sheet. My apologies for everyone who used this spreadsheet as the source, currently you can use CSV version of this report. Maybe, I will reformat this Google Sheets (e.g. split the states by tabs), or, unfortunately, it will have to be abandoned forever.

Merging of Apple and Google reports significantly improved. I finally have made a matching table of subregions of Apple and Google. Also, I've split the summary report into several:

Report by regions (without US counties)
Report by countries (only totals)
Report for the US only

If someone sees errors in the matching table, please create the issue immediately.

Also, I think it's a good idea to add a geo-type column to Google data (such as in Apple report).

I haven't written anything here in a while, but I should have. So, point by point:

Until today, the parser has been working successfully for a long time automatically without my intervention. But strange things are happening on the Apple website today, so I predict the problems tonight.
I haven't fixed the problem with Google Sheets for the US yet, but there's already some progress.
Lately, I've been actively processing OpenSky COVID-19 Flight Dataset. I hope that within 2-3 days I will put my results in a separate repository. The main problem is that I do not understand the quality of data in this dataset and how to evaluate it. But it is what it is. If all goes well, I will also add these data to the merged reports.


Added ability to explicitly set the edit mode and force changes.

See issue #138.

@mindscratch see PR #245


adding custom page for paraspatle-jpg

Description 📃
This pull request is to add a new contributor to the repository, the change adds a new page to the hosted website which shows some info about user.

Type of Changes 📝

[x] Added Feature (adding a non-existing new feature)

Checklist ✅

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code or work. 🔎
[x] I have written comments in the code whenever felt necessary. 🖊
[x] I am contributing to this project for the first time. 🆕
[x] My changes generate no new warnings. ⚠

Screenshots 📸

Original
Updated

insert original screenshot
insert updated screenshot

Looks good to me 🙌  ! Thanks for the contribution ✨


[typescript] Incompatible Route and Location types

Version
3.1.6
Reproduction link
http://shorturl.at/ixJQ5
Steps to reproduce
this.$router.push({
    ...this.$route,
    params: {
        newParam: true
    }
})

With typescript, it breaks.
What is expected?
Route and Location types are compatible.
What is actually happening?
As Route now have the name parameter with string | null | undefined it breaks the navigation, as Location will only accept string | undefined as name.

Commit causing the issue:
https://github.com/vuejs/vue-router/commit/8f831f28f33bb766d5baaabc8b2daff259f3d45b

The types are indeed not compatible: null can exist as a value of Route but when providing a Location with methods like push and replace, null is not valid name. FYI, there are a few other things that cannot be passed like fullPath. In general, spreading $route to a push/replace is a bad idea because it's a normalised location and it contains more information than a Location


Unable to add "sub" claim to JWT token.

While migrating an asp.net  project to .net core  I have ran across the following issue:
when building a token using the following code:
//...
			var claims = new ClaimsIdentity(additionalClaims);
//...
			var tokenDescriptor = new SecurityTokenDescriptor {
				Audience = "https://localhost:44369",
				Issuer = "https://localhost:44369",
				Subject = claims ,
				Expires = DateTime.UtcNow.AddHours(_tokenConfig.TimeoutHours),
				SigningCredentials = new SigningCredentials(new SymmetricSecurityKey(_tokenConfig.Key), SecurityAlgorithms.HmacSha256Signature)
			};

			//Token specific claims
			tokenDescriptor.Subject.AddClaim(new Claim(ClaimTypes.Name, user.Email, ClaimValueTypes.String));
			tokenDescriptor.Subject.AddClaim(new Claim("my.random.claim", user.Email));
			tokenDescriptor.Subject.AddClaim(new Claim("sub", user.Email));

			var token = tokenHandler.CreateJwtSecurityToken(tokenDescriptor);

the line : tokenDescriptor.Subject.AddClaim(new Claim("sub", user.Email));   has no effect.
Then in the controller action:
if (User is ClaimsPrincipal claimsUser) {
	string email = claimsUser.Claims.Where(c => c.Type == "sub").Select(c => c.Value)
	  .SingleOrDefault();
}

All other defined claims from above are present in claimsUser.Clamis, except for "sub".
I noticed that in https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/blob/05e02b5e0383be40e45c667c12f6667d38e33fcc/src/System.IdentityModel.Tokens.Jwt/ClaimTypeMapping.cs#L50
{ JwtRegisteredClaimNames.NameId, ClaimTypes.NameIdentifier },
 { JwtRegisteredClaimNames.Sub, ClaimTypes.NameIdentifier },

both  RegisteredClaimNames are mapped to the same ClaimType, so in the reversed dictionary only the first one will be present, but I'm not sure that means anything.

@vllama I believe you are experiencing "claimMapping".
Inbound the JwtSecurityTokenHandler uses this mapping: https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/blob/93334a8e94e66d3df36f2850a03549ed1bfd4234/src/System.IdentityModel.Tokens.Jwt/ClaimTypeMapping.cs#L41, you should find a claim of type ClaimTypes.NameIdentifier. There will be a property on the claim with the short name sub.
You can turn this off by setting: JwtSecurityTokenHandler.DefaultMapInboundClaims = false, in your startup code.
There are other ways to do the same thing.

Thanks, I was able to modify the library that was looking for sub in the meantime to use the NameId, so that worked as well.

@vllama as a note the JsonWebTokenHandler has similar functionality as the JwtSecurityTokenHandler but does not perform any mapping.


Update Select to use ListboxSection instead of Section

The latest version of React Aria Components deprecates the generic Section component, in favour of "specific components for each parent collection component, e.g. ListBoxSection and MenuSection".
This affects our Select component, which uses Section. We will need to refactor this component to properly support ListBoxSection alongside #500.

@Supriya-Arora tagging you for visibility, but I'm happy to take a first pass at this.


 math constants

Document the C++ 2020 header
P0631R8
Introduced in: VS 2019 16.5

moved to quest


Steps to reproduce/run repository

Hi, I am trying to reproduce your results and run the full code on raw data. Could you please post some steps on how to start and what is the data flow between different files? Thanks a lot

I am trying to reproduce for ADHD 200 dataset. I cannot see dataloaders and preprocessing files for it. Can you please help me with the process?

Hi, the code should be runnable with the provided Synthetic data. Synthetic data can be downloaded from https://drive.google.com/file/d/17ju4HiZOX7MtdKxKv1aNXISdQkCg-qwq/view?usp=sharing Please note that the synthetic data are provided only to facilitate running the code and release of the original data used in the paper is out of the authority of the author. Please refer to the corresponding data resource.  And the entry of code is train_val_test/train.py

Hi, thanks for replying. The synthetic data provided that is in the google drive folder is for ADHD 200 dataset or for others like ADNI?
Also, it seems like the ADHD dataloader is missing from the codebase, to convert raw data to synthetic data. Could you please upload it?
Thanks a lot

The synthetic data provided is just example code to facilitate running the code quickly. Sharing ADHD or ADNI is out of our autority. Sorry. So, the current runnable dataloader can be converted to load any data once you have real data to replace the synthetic data.

Hi again, even with using the synthetic data, the data_choose method is incomplete as it is looking for the adhd and adhd_val functions. I believe that you missed uploading those files. Could you please verify.

I see the problem. Sorry for not making it clear. Could you check if the code loads the config file train_val_test/config/config.yaml, which specifies the data and parameters for the network. And I just use the adni dataloader to load the synthetic data.
data: 'adni'
data_param:
train_data_param:
data_path: /data/datasets/SyntheticDataForKernelTransformer/train_data.npy
label_path: /data/datasets/SyntheticDataForKernelTransformer/train_label.pkl
...
If the code does not automaticaaly load the config.yaml file, you may try to explicitly specify it by using -config  XXX/train_val_test/config/config.yaml when running the code.
I will make this clearer later.
Thanks for pointing out this.
Jianjia

Hi, thanks for replying and seeing the problem. The train.py is able to run with the synthetic data provided. But it is still unclear if this synthetic data belongs to the ADNI dataset or the ADHD-200 dataset. I want to request you to please upload the ADHD dataloader as well so I can recreate the synthetic data for it. I do have the original dataset downloaded with all the files so I only need the dataloader which is missing from the project. Please upload files related to the ADHD dataset if possible.
Thanks


Create Seedlist.rst and update Basicusage.rst for API Servers

What current issue(s) does this address, or what feature is it adding?
Adds Seedlist.rst. Follows neo-cli but is neo-python specific. I referenced this when building my RPC server.
Updates Basicusage.rst with port description and specific instructions for Windows WSL (Ubuntu) users.
*Note this PR was re-created from this previous PR for better comparison and merging

@ixje Okay, I recreated it. Thank you for the feedback.

Coverage remained the same at 80.26% when pulling e51cf173b50ff531c85c04f8b7eadad2f3ea60d6 on jseagrave21:patch-2 into 6104077d18bfc48409302d94b0e5d49b17cbca74 on CityOfZion:development.

I edited the API section of Basicusage.rst to give better instructions for running an API server. I did not include an exhaustive list of API commands as seen in the NEO API Reference or example of how to query. If this would be useful for neo-python servers, and not redundant, I would be happy to add it as well.

Great work!

Thank you! Please let me know if more documentation would be helpful. I don't want to be redundant but I would also like it to be easy for someone to read and understand Neo-python instead of overloading Discord.


Add pipeline statistics

Objective
It's useful to have access to render pipeline statistics, since they provide more information than FPS alone. For example, the number of drawn triangles can be used to debug culling and LODs. The number of fragment shader invocations can provide a more stable alternative metric than GPU elapsed time.
See also: Render node GPU timing overlay #8067, which doesn't provide pipeline statistics, but adds a nice overlay.
Solution
Add RenderStatistics resource, which provides information about CPU & GPU elapsed time, as well as pipeline statistics (primitives count, shader invocations, etc) per each render pass.
Add MeasuredRenderPass, which derefs to regular RenderPass and records timing and pipeline statistics. Note: only labeled passes are recorded.
TrackedRenderPass now internally uses MeasuredRenderPass.
TODO

Maybe store spans instead of elapsed time.
Disable statistics when the required features aren't available.

Changelog

Add RenderStatistics resource
Update 3d_scene example to print statistics to console

Disable statistics when the required features aren't available.

Can you say more about this? It sounds like we should block on fixing that before merging this PR to avoid unexpected problems, but I'm not fully sure I understand.

Can you say more about this? It sounds like we should block on fixing that before merging this PR to avoid unexpected problems, but I'm not fully sure I understand.

I have addressed that. Now when timestamp queries are unsupported, no GPU timing will be recorded. Same for pipeline queries. CPU timings are always recorded though, since they don't require any features.
Also, I have written the docs.
I have the following question: should I keep the RenderStatistics resource, or perhaps they should be stored as diagnostics instead? I would imagine having diagnostics like render.main_opaque_pass_3d.gpu_time. This would give us smoothing basically for free. But I see a few problems with this approach:

How to generate diagnostic UUIDs, given there can be any number of render passes (at least one for each shadow caster, for instance)
How to assign diagnostic instants? They can either share one instant at which they became available, or I could try to approximate instants given GPU timestamps and the instant when the frame was submitted.
Missing ergonomics for nested diagnostics (a render pass would have, like, 7 separate diagnostics, and there's no way to easily iterate through all of them).

I would say it's simpler to leave render statistics as a resource, and a future PR could address copying those into diagnostics and/or providing a GUI overlay, like in #8067.

I've gated all recording functionality behind RenderStatisticsPlugin, which is disabled by default.
Additionally, now I reuse buffers instead of creating them every frame. There are 2 query sets, and 2 buffers (one for resolving the query set, and the other for cpu-readback) per frame-in-flight (of which there are 3, apparently).
This makes performance of statistics recording basically negligible (given there are only a couple render passes). I ran many_sprites benchmark and saw no measurable regressions.

Don't think we'll have time to finish the impl and get this reviewed on top of all the other rendering work going on atm, so I'm moving this to the 0.13 milestone. Would still love to get this in eventually though.

@LeshaInc this looks neat but has been dormant for a while. Would you like me to put this work up for adoption?

@LeshaInc this looks neat but has been dormant for a while. Would you like me to put this work up for adoption?

This was mostly blocked on #9266, now that it's merged I could honestly finish it myself in the nearest future. All that's left is integrating the new nested diagnostics, resolving merge conflicts, and adding statistics for the new passes.

Awesome, I'm glad to hear it :) Ping me if you need anything else.

@LeshaInc let me know if you need any help on this. I'd love to get both pipeline statistics, but also timing info.
For now a raw diagnostics output should be the minimal implementation the PR should handle, but in the future it'd be nice to compute a histogram or some sort of smoothed output over multiple frames, as I did in https://github.com/bevyengine/bevy/pull/8067.
I also think it's worth looking into wgpu-profiler. It gives us tracy integration (iirc), and handles a lot of the implementation details already.

I've updated everything to store pipeline statistics & timing info in diagnostics, instead of a separate resource. All you need to enable gathering all of this info is to add RenderDiagnosticsPlugin and LogDiagnosticsPlugin.
Here's how the diagnostics look (3d_scene example)
render/main_opaque_pass_3d/clipper_invocations                         :   74.000000   (avg 74.000000)
render/main_opaque_pass_3d/clipper_primitives_out                      :   74.000000   (avg 74.000000)
render/main_opaque_pass_3d/elapsed_cpu                                 :    0.062421ms (avg 0.065678ms)
render/main_opaque_pass_3d/elapsed_gpu                                 :    0.237975ms (avg 0.233948ms)
render/main_opaque_pass_3d/fragment_shader_invocations                 :<PHONE_NUMBER>00   (avg<PHONE_NUMBER>00)
render/main_opaque_pass_3d/vertex_shader_invocations                   :   92.000000   (avg 92.000000)
render/shadows/elapsed_cpu                                             :    1.017897ms (avg 1.053214ms)
render/shadows/elapsed_gpu                                             :    0.049878ms (avg 0.049476ms)
render/shadows/shadow pass point light 0 -x/clipper_invocations        :   62.000000   (avg 62.000000)
render/shadows/shadow pass point light 0 -x/clipper_primitives_out     :    2.000000   (avg 2.000000)
render/shadows/shadow pass point light 0 -x/elapsed_cpu                :    0.065137ms (avg 0.067303ms)
render/shadows/shadow pass point light 0 -x/elapsed_gpu                :    0.004264ms (avg 0.004489ms)
render/shadows/shadow pass point light 0 -x/fragment_shader_invocations:    0.000000   (avg 0.000000)
render/shadows/shadow pass point light 0 -x/vertex_shader_invocations  :   68.000000   (avg 68.000000)
render/shadows/shadow pass point light 0 -y/clipper_invocations        :   74.000000   (avg 74.000000)
render/shadows/shadow pass point light 0 -y/clipper_primitives_out     :   74.000000   (avg 74.000000)
render/shadows/shadow pass point light 0 -y/elapsed_cpu                :    0.065375ms (avg 0.066253ms)
render/shadows/shadow pass point light 0 -y/elapsed_gpu                :    0.004936ms (avg 0.004821ms)
render/shadows/shadow pass point light 0 -y/fragment_shader_invocations:    0.000000   (avg 0.000000)
render/shadows/shadow pass point light 0 -y/vertex_shader_invocations  :   92.000000   (avg 92.000000)

(I've stripped irrelevant info and sorted the lines, since LogDiagnosticsPlugin currenty logs them in hashmap order)
As you can see, there's info for all render passes, including elapsed CPU (recording) and GPU time, and pipeline statistics. They can be easily accessed from DiagnosticsStore programmatically, if needed.
To record diagnostics in render nodes, you have to:

Get the diagnostics recorder from RenderContext.let diagnostics = render_context.diagnostic_recorder();

It will be an opaque type implementing RenderDiagnostics trait. It's methods will be no-op unless RenderDiagnosticsPlugin is present. This is done to avoid writing #[cfg(feature = "diagnostics")] everywhere.
Begin the span:let time_span = diagnostics.time_span(render_context.command_encoder(), "shadows");

Here command encoder can also be replaced with a render pass, or a compute pass.
End the span. You have to provide the same encoder, which was used to begin the span.time_span.end(render_context.command_encoder());

There are two kinds of spans:

Time spans can be used in both top-level encoders, as well as render/compute pass encoders. They only collect elapsed time.
Pass spans can only be used inside passes, and they provide more info, such as number of shader invocations. Their support is more limited, so in case they are unsupported, only CPU time will be recorded.

Now I'd like to address a few points.
@JMS55

For now a raw diagnostics output should be the minimal implementation the PR should handle, but in the future it'd be nice to compute a histogram or some sort of smoothed output over multiple frames, as I did in https://github.com/bevyengine/bevy/pull/8067.

I agree, a visual overlay would be nice. I think it should work for all diagnostics, not just render ones. I imagine some kind of tree view, where you can expand individual diagnostics and look at their plots, histograms, etc. However,  bevy_ui is still far from being capable of such complex UI. I think an unofficial visualizer built on top of bevy_egui could be made as a temporary solution.

I also think it's worth looking into wgpu-profiler. It gives us tracy integration (iirc, and it would be awesome to see both CPU/GPU timing in tracy at once), and handles a lot of the implementation details already.

I agree it's best for the ecosystem to use an established crate. However, wgpu-profiler doesn't record pipeline statistics, which is what I was interested in to begin with. I see a few ways to move forward

Keep render profiling inside bevy. The internals are less than 1000 lines long, including the integration with bevy_diagnostic. A future PR could add tracy integration, which shouldn't be too hard.
Keep the same abstract API, but replace the implementation with wgpu-profiler (could be done in a future PR). This requires adding support for pipeline statistics to it, which would take some time to get reviewed, merged, and released.

In both cases, I think it's worth for bevy to have it's own API for recording rendering spans, with integration with TrackedRenderPass, bevy_diagnostic, and other shenanigans. The underlying implementation could always be changed without breaking the API.

Nice progress: seems very useful. Is this ready to come out of draft now?

One consideration is we would want to lock the GPU to base memory/clock speed in order to get stable profiling.
Other than that, once this PR is no longer in draft, I'd be happy to review it and hopefully get this merged early into the 0.14 development cycle :)

@LeshaInc 0.13 is done, lets work on this now :). Can you rebase this? Once done I'll review it.

@JMS55 done

Should this live in bevy_dev_tools? @alice-i-cecile

I think so: it's valuable for discovery, and to ensure that other diagnostics can be built based on the public API.

Btw I'm unfortunately busy with IRL stuff, sorry for the delay on reviewing this. I still want to get this in as soon as I can.

@LeshaInc if you can get CI passing I'll merge this in for you :) I think you may need to adjust some imports.


Some SVG freezes entire process

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?
Yes, latest version used. The outputs are same.
What are the steps to reproduce?
Take this image: https://dentalfin.netlify.app/_nuxt/img/purpleBg.0f9eb3f.svg
Try to convert the image to WebP by using 192 DPI in sharp constructor.
The process will hang and go on indefinitely. It uses 100% CPU.
What is the expected behaviour?
The image should be converted to WebP.
What is the output of running npx envinfo --binaries --system?
npx: installed 1 in 1.784s

  System:
    OS: macOS 11.4
    CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    Memory: 112.60 MB / 8.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.17.3 - /usr/local/opt/node@14/bin/node
    npm: 6.14.13 - /usr/local/opt/node@14/bin/npm

Attaching SVG here:
purpleBg.0f9eb3f.svg.zip

Is there a way to specify timeout or just cancel the operation?

Btw, this happens with 72 DPI as well and with jpeg output as well.

rsvg-convert purpleBg.0f9eb3f.svg > out.jpeg works fine. It takes a few seconds but at least gives output.

I think this is due to issue https://gitlab.gnome.org/GNOME/librsvg/-/issues/698, given that this SVG image contains an feMorphology with a large radius. This was fixed in librsvg 2.50.4 and 2.51.1.
The pre-built binaries provided by sharp >= 0.28.0 were built against librsvg 2.51.0.

Seems possible. We are using debian librsvg which is 2.44 and for rsvg-convert, we are using homebrew version which gives 2.50.7.
Btw, shouldn't sharp have some sort of timeout to avoid such issues.

I have upgraded library but still it seems to be taking forever. I checked with rsvg-convert after upgrade. Following is output
time rsvg-convert -d 192 purpleBg.0f9eb3f.svg > out.jpeg

real    0m21.178s
user    0m18.445s
sys     0m0.382s

When used with sharp, this does not work in 21 sec.

The forthcoming sharp v0.29.0 will provide librsvg v2.51.4 (or later) that include the upstream fix. The sample file provided takes about 18 seconds to render locally.
Perhaps you could check/block any untrusted input for the use of the expensive feMorphology filter.


[acti_func] implement quick gelu

Implemented quick gelu function. Please note that quickGeluPrime which is calculate derivate of quickGelu function is not yet implemented.

Signed-off-by: hyeonseok<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2625. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.


Remove version: Stackers.Stack version 3.25.1

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://binaries.getstack.app/builds/prod/win/x64/Stack Setup 3.25.1-x64.exe
Response status code does not indicate success: 530 ().
(Automated message - build 896)


Topred's gain level tweaks

Gain levels recommended by Topred L.  I have been testing these for a couple weeks with very good results.
For Auto or fixed level 3, this sets the following levels.  Auto/fixed levels other than 3 remain as default.  The radio uses level 3 for all but very strong signals.
BW: 1
Weak: 2
PGA: 6
Mix: 3
LNA: 6
LNAs: 3

Hi @xawen,
Maybe I missed something but I have following values instead of the one you list above
(default values on radio start, in AUTO3 mode)
BW: 4
Weak: 0
LNA: 7

I start in FM, here what I get when turning on the radio without touching anything.
FW is last commit of main branch: 37f9b1e.
Don’t you get the same thing?

Ah, I think I know the problem.  Do you have the register editor enabled in Makefile?  When reg edit is enabled, the code that sets the BW and W defaults is disabled - this is in BK4819_SetFilterBandwidth().
The reason is that function gets called every time you change frequency, change VFO, etc....  The reg editor disables it so that any changes you make with reg editor don't revert as you use the radio.

Hooo That's right many thanks!
I was afraid of having broken something 😅


[Bug] AdvancedCauseOfDeath shows up as "Unknown" on LethalStats

The mod is perfectly updated to use Coroner 2.0.0's new AdvancedCauseOfDeath.
However, the LethalStats website cannot comprehend the new causes, and displays them as "Unknown".

Solved! Thanks for letting me know :)


Background-size: cover on site-header

Consider using background-size: cover on site header.

Will need to finish #38 first.

@parkjc Has this been done? If not, what is the advantage of this enhancement?

@fjumawan #38 is now closed, so we can look at this. It's basically just specifying that the header image to stretch and/or crop the header image to fit all screen sizes as needed. Can be adjusted in the css. Let me know if you need a hand.

@billymeinke Will look into this.

@billymeinke looks like this has been done.

Great, thanks @fjumawan!

@billymeinke No problems! Wast trying to figure out how to do it and I bumped into it. :)


typst watch doesn't work when editing the watched file with (neo)vim

Description
The watch command does not seem to work with (neo)vim anymore.
The issue seems to come from the fact that when saving a file, vim will rename the old file and write a new file with the old name. However, for notify-rs this causes a Remove event to be triggered on that file, and removed files are implicitly unwatched on some platforms according to their docs:

On some platforms, if the path is renamed or removed while being watched, behaviour may be unexpected. See discussions in #165 and #166. If less surprising behaviour is wanted one may non-recursively watch the parent directory as well and manage related events.

When running the code through gdb, after saving a first time, the program just waits for events there, and saving the file again does not trigger events.
Note: the oldest commit where this saving behavior caused an issue was e1d76960eb67443fbe58e5af0cc6da2facee26a1, trying to unwatch the dependencies, some of which were already unwatched, caused a failed to unwatch something.typ error to happen.
steps to reproduce

create a typst file, let's say test.typ
watch the file with typst watch test.typ
edit the file in vim or neovim
save the file (with :w), this one should work
try saving again, and the modifications are not picked up by typst

Vim's saving method can be changed to copy the new file into the old file by executing the command :set backupcopy=yes, in this case the typst watch command works as expected.
Reproduction URL
No response
Operating system
Linux
Typst version

[X] I am using the latest version of Typst

$ echo main.typ | entr -c typst compile /_

I am also having this issue with typst and neovim.

@alerque i'm curious, what is SILE ? i couldn't find anything about it.

I guess for Typst it's a bit different because built-in watch can profit from incremental compilation. If understand the entr command correctly, it will also pick up direct dependencies of main.typ and not e.g. nested imports or local packages.
Unfortunate that the new approach has this issue because watching individual files instead of the root directory is much nicer in principle (less unnecessary recompilations for files that don't affect the document even though they are in the working dir and it correctly picks up imports from packages).

Yes, this level of workaround does loose the benefit of incremental compilation.
Entr can watch anything you tell it to. For example one approach is to tell it to watch everything tracked by git: git ls-files | entr .... Of course you could also just pass the output of find or more specialized inputs. For example SILE has a feature that cat output a list of every file used while rendering a document: font files, libraries, images, includes, everything. That feature can was originally added to pass to make to know when a document needs re-rendering but can be adapted for use by entr as a watch list.

this issue stems from the fact that vim create a new file and move it to replace the old one.
the library used to watch is notify, which afaik doesn't get when that happens (maybe there is a workaround).
watching the parent directory instead of the file would work but maybe there is a cleaner fix.

Unfortunately I cannot currently test any potential fix as I'm on macOS and don't have a Linux machine available. If anybody on Linux wants to give fixing this a shot, I'd appreciate it.

I have the some issue using version:0.6
typst watch main.typ

@ayoubeimhamdi By rights your Makefile should either by a Justfile (since you're just using it as a task runner) or maybe try including dependencies ;-)
.PHONY: all
all: main.pdf

DEPS != find . -type f -iname "*.typ"

main.pdf: main.typ $(DEPS)
	@typst compile $<

Now make will understand when you PDF does not need rebuilding. It still won't get you incremental rendering updates.
Back to topic:
@laurmaedje Given that many other tools are able to catch changes to files when Neovim and other apps that do write→rename sequences (including entr as discussed in this issue) there is probably a way to do it. You probably just need to add more filesystem events to your notify loop including creation of files by the given name.

I want to quickly chime in to point out the rust project watchexec.
Basically, the command line app behaves similar to entr.
However, they also provide libraries for custom file watches and various other subfeatures of the main app.
Maybe this is a suitable building block for typst watch?

Removing the deleted files from the previous compilation's dependencies in the watch function seems to work. That way, when calling watch_dependencies, if the files are in the current compilation's dependencies and still exist, they will be considered as not watched and be watched again.
(I am not very familiar with rust, especially the borrow checker part, so there might be a better way to do this)
loop {
    let mut removed = HashSet::new(); // will contain paths to the removed files
    let mut recompile = false;
    for event in rx
        .recv()
        .into_iter()
        .chain(std::iter::from_fn(|| rx.recv_timeout(timeout).ok()))
    {
        let event = event.map_err(|_| "failed to watch directory")?;

        if matches!(event.kind, notify::EventKind::Remove(notify::event::RemoveKind::File)) {
            let path = &event.paths[0];
            removed.insert(path.clone());

            // Tries to remove the watch if it still exists
            // I don't know if watching the same file multiple times could be problematic on some
            // platforms, so this may not be necessary
            match watcher.unwatch(&path) {
                Err(notify::Error { kind, .. }) => match kind {
                    notify::ErrorKind::WatchNotFound => {},
                    _ => return Err("failed to unwatch file".into()),
                }
                _ => {}
            }
        }

        recompile |= is_event_relevant(&event, &output);
    }

    if recompile {
        // Retrieve the dependencies of the last compilation.
        let previous: HashSet<PathBuf> = world
            .dependencies()
            .filter(|path| !removed.contains(*path)) // remove the deleted files from the previous deps
            .map(ToOwned::to_owned)
            .collect();

        // Recompile.
        compile_once(&mut world, &mut command, true)?;
        comemo::evict(10);

        // Adjust the watching.
        watch_dependencies(&mut world, &mut watcher, previous)?;
    }
}

@tretre91 Nice, that seems like a good solution!


Move to done-ssr

This is really a donejs project.

Need to update done-css to point use the doneSsr global instead of canSsr.

https://github.com/donejs/css/issues/30 is needed to complete this.