InstallSSHKey - "Host key verification failed." after installing ssh key

Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: InstallSSHKey@0
Environment

Agent - Hosted or Private:

If using Hosted agent, provide agent queue name: Hosted Ubuntu 1604

Issue Description
Installed ssh key using the InstallSSHKey task, in azure pipelines, after that executed a bash script with git clone, which requires ssh connection. While executing the git clone the error ocurrs: Host key verification failed.
Task logs
steps:
- task: InstallSSHKey@0
  displayName: 'Install an SSH key'
  inputs:
    hostName: bitbucket.org
    sshPublicKey: '****'
    sshKeySecureFile: '****'

InstallSSHKey Task Execution log
2019-04-09T15:23:54.5592273Z ##[section]Starting: Install an SSH key
2019-04-09T15:23:54.5830187Z ==============================================================================
2019-04-09T15:23:54.5830341Z Task         : Install SSH Key
2019-04-09T15:23:54.5830385Z Description  : Install an SSH key prior to a build or release
2019-04-09T15:23:54.5830449Z Version      : 0.141.2
2019-04-09T15:23:54.5830489Z Author       : Microsoft Corporation
2019-04-09T15:23:54.5830571Z Help         : [More information](https://go.microsoft.com/fwlink/?linkid=875267)
2019-04-09T15:23:54.5830618Z ==============================================================================
2019-04-09T15:23:58.8106207Z [command]/usr/bin/ssh-agent
2019-04-09T15:23:58.9108151Z SSH_AUTH_SOCK=/tmp/ssh-HL0mWERJf86I/agent.3324; export SSH_AUTH_SOCK;
2019-04-09T15:23:58.9108629Z SSH_AGENT_PID=3325; export SSH_AGENT_PID;
2019-04-09T15:23:58.9108909Z echo Agent pid 3325;
2019-04-09T15:23:58.9150245Z [command]/usr/bin/ssh-add -L
2019-04-09T15:23:59.0096737Z The agent has no identities.
2019-04-09T15:23:59.0219293Z [command]/usr/bin/ssh-add /home/vsts/work/_temp/azure_***
2019-04-09T15:23:59.0277641Z Identity added: /home/vsts/work/_temp/azure_*** (/home/vsts/work/_temp/azure_***)
2019-04-09T15:23:59.0300383Z [command]/usr/bin/ssh-add
2019-04-09T15:23:59.0688825Z ##[section]Finishing: Install an SSH key

failed script task log:
2019-04-09T15:24:08.4138732Z ##[section]Starting: Command Line Script
2019-04-09T15:24:08.4141199Z ==============================================================================
2019-04-09T15:24:08.4141271Z Task         : Command Line
2019-04-09T15:24:08.4141336Z Description  : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
2019-04-09T15:24:08.4141386Z Version      : 2.148.0
2019-04-09T15:24:08.4141446Z Author       : Microsoft Corporation
2019-04-09T15:24:08.4141494Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2019-04-09T15:24:08.4141571Z ==============================================================================
2019-04-09T15:24:08.5390094Z Generating script.
2019-04-09T15:24:08.5407340Z ========================== Starting Command Output ===========================
2019-04-09T15:24:08.5434583Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/8120cc21-4164-4d08-8b2b-62ccd9d812ca.sh
2019-04-09T15:24:08.5492005Z /home/vsts/work/_temp/8120cc21-4164-4d08-8b2b-62ccd9d812ca.sh
2019-04-09T15:24:08.5493445Z known_hosts
2019-04-09T15:24:08.5501239Z + echo /home/vsts/work/_temp/8120cc21-4164-4d08-8b2b-62ccd9d812ca.sh
2019-04-09T15:24:08.5501768Z + GIT_SSH_COMMAND='ssh -vvv'
2019-04-09T15:24:08.5501965Z + ls /home/vsts/.ssh
2019-04-09T15:24:08.5502464Z + cat /home/vsts/.ssh/known_hosts
2019-04-09T15:24:08.5502552Z + git clone git@bitbucket.org:*****/*****.git
2019-04-09T15:24:08.5504672Z bitbucket.org
2019-04-09T15:24:08.5507284Z Cloning into '*****'...
2019-04-09T15:24:11.0754875Z Host key verification failed.
2019-04-09T15:24:11.0755302Z fatal: Could not read from remote repository.
2019-04-09T15:24:11.0755398Z
2019-04-09T15:24:11.0755462Z Please make sure you have the correct access rights
2019-04-09T15:24:11.0755523Z and the repository exists.
2019-04-09T15:24:11.0877593Z ##[error]Bash exited with code '128'.
2019-04-09T15:24:11.0904033Z ##[section]Finishing: Command Line Script

Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs

Apparently the host signature is not being added to ~/.ssh/known_hosts file.
executing the command ssh-keyscan bitbucket.org > ~/.ssh/known_hosts solves the issue.

@mjscosta I believe the task is expecting you to provide the host signature via the hostName field rather than just bitbucket.org
See: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/install-ssh-key?view=azure-devops for instructions

@joshmgross Indeed, I've missunderstand the docs, thanks. It is now working. I'll close the issue.

@mjscosta I am having the same issue. Maybe I misunderstood the docs just like you did. 😅
What did you have to change from your initial task commands?

@mjscosta I am also curious what you did to solve it, @sney-kqa did manage to resolve it?
Install private SSH key
   - task: InstallSSHKey@0
    displayName: 'Install an SSH key'
    inputs:
      knownHostsEntry: bitbucket.org
      sshPublicKey: 'xxx'
      sshKeySecureFile: 'id_rsa'

The key works when I attempt it on our private build server
C:\agent\_work\2\s>ssh -T<EMAIL_ADDRESS>logged in as xxx

You can use git or hg to connect to Bitbucket. Shell access is disabled

But still getting a host key verification failed error when trying a
git pull git@bitbucket.org:org/repo.git $(Build.SourceBranch)
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Is there a reason the persistCredentials property on checkout can't help us???
- checkout
  persistCredentials: true


Visual form for Windows

Hi,
I have worked on a visual application for Windows to help the users to operate your programmer more easily.
Its was building using Lazarus and its already working. Just need to be checked on Linux.
I want to colaborate with this visual form in your project, let me know if you like it.
Below there's a screenshot that shows how it is:

Hello, this looks great, thank you for your work.
Is this publicly available? I'd like to try it out.
I can put the link to your work into readme, for others to try and use.

Thanks a lot.
This application is not publicly available yet.
I was just waiting for your answer, because the code is very related to your (actually it needs your executable).
I can create a new repository of source code and executable for others to use it, or if you want, you can add to this repository too.
I look forward to your reply.

Did you move any further with this application?

Did you move any further with this application?

The application seems work well.
I was just waiting for your permision to upload it in a new repository.
That is because actually in the current state of the project it still needs to use your executable in background.

I'd love to see it public and add link to your project to the project description.

Thanks, check here my project.
And please, let me know if there's something wrong.


a11y: SfSelect with v-focus

Related issue
Closes #1068
Scope of work
a11y improvements for SfSelect

awesome @justyna-13 🥳 🎉


az acs dcos browse navigates to localhost as well as the proxy

Running az acs dcos browse -n [] --resource-group [] opens two browsers. One goes to the DCOS portal (which is expected), but the other navigates to the proxy that was set and fails with this error:
dial tcp <IP_ADDRESS>:63037: getsockopt: connection refused

weird, let me look at the code, it shouldn't do that ...

Ah, I see the problem, filed:
https://github.com/Azure/azure-cli/pull/1702


old wireguard plugin

hello. i have upgrade opnsense without a problem but wireguard now is integrate in a core system.
in plugin i have a old version of os wireguard plugin.
how can remove this?

https://github.com/opnsense/plugins/issues/3812#issuecomment-1940617594

ok. thank you!


Added QR code component.md

Added QR code component
Title and Issue number

Close #<issue_no>

Checklist:

[] I have mentioned the issue number in my Pull Request.
[] I have commented my code, particularly in hard-to-understand areas
[] I have created a helpful and easy to understand README.md

Please sir my PR request accept karlo.
On Mon, 10 Oct, 2022, 5:00 pm Pranjay Poddar, @.***>
wrote:

@.**** requested changes on this pull request.
Follow PR template
—
Reply to this email directly, view it on GitHub
https://github.com/pranjay-poddar/Dev-Geeks/pull/858#pullrequestreview-1135867175,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHQRVCMB5PJ326OHZ7IPUTWCP433ANCNFSM6AAAAAARBEGQSM
.
You are receiving this because you authored the thread.Message ID:
@.***>

Closing as part of Hacktoberfest 2022 and now outdated.


Different result in android vs ios

I'm trying to run the following query and I get different results in android vs ios. Is it maybe, because of the COALESCE not working in android? It does in IOS.
What I'm trying to solve is a problem where I want to create a third column where every row in the column, is a SUM of two other nullable columns, and since I can't add NULL, I want to replace all NULL values by zero, using coalesce (I also tried ifnull, which is the same as coalesce with 2 values and also does not work in android).
SELECT * FROM (select *, coalesce(X, 0) + coalesce(Y, 0) as F from Z ORDER BY F DESC) group by T ORDER BY F DESC;

Interesting, are you using moor/ffi or moor_flutter?


update recent addition to 5.0 syntax

motivation: support 5.0 clients
changes: return value explicitly

see https://github.com/swift-server/swift-service-lifecycle/pull/129

@ktoso right I remember that dicussion. it does means we need to remove the<EMAIL_ADDRESS>file so that SwiftPM will happily satisfy the dependency causing potential issues to downstream projects

Hah I see, we missed to remove the file, here's a PR to rectify that: https://github.com/apple/swift-log/pull/259

Superseded by https://github.com/apple/swift-log/pull/259


Test positivity ratio is incorrect on main page

See screenshot: For instance test positivity ratio in Karnataka is shown as 6% while there were almost 50.000 new cases and number of tests were 1.9L. So positivity rate should be around 25%

Table shows the overall TPR

@bee-rickey how do you calculate overall TPR?


BREAKING CHANGE - Can't resolve some modules in 13.0.34

After rebuilding node_modules our app couldn't be build anymore. While building you get the following log output errors:
Module not found: Error: Can't resolve '@nativescript/core/data/observable-array' in '/Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript-community/ui-pager/angular/fesm2015'
Did you mean 'index.js'?
BREAKING CHANGE: The request '@nativescript/core/data/observable-array' failed to resolve only because it was resolved as fully specified
(probably because the origin is strict EcmaScript Module, e. g. a module with javascript mimetype, a '*.mjs' file, or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
resolve '@nativescript/core/data/observable-array' in '/Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript-community/ui-pager/angular/fesm2015'
  Parsed request is a module
  using description file<EMAIL_ADDRESS>(relative path: ./fesm2015)
    resolve as module
      looking for modules in /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules
        existing directory /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript/core
          using description file<EMAIL_ADDRESS>(relative path: .)
            using description file<EMAIL_ADDRESS>(relative path: ./data/observable-array)
              /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript/core/data/observable-array is not a file
      /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript-community/ui-pager/angular/fesm2015/node_modules doesn't exist or is not a directory
      /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript-community/ui-pager/angular/node_modules doesn't exist or is not a directory
      /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript-community/ui-pager/node_modules doesn't exist or is not a directory
      /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript-community/node_modules doesn't exist or is not a directory
      /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/node_modules doesn't exist or is not a directory
      looking for modules in /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules
        existing directory /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript/core
          using description file<EMAIL_ADDRESS>(relative path: .)
            using description file<EMAIL_ADDRESS>(relative path: ./data/observable-array)
              /Users/Shared/Entwicklung/xxx/nativescript-xxx-app/node_modules/@nativescript/core/data/observable-array is not a file
      /Users/Shared/Entwicklung/xxx/node_modules doesn't exist or is not a directory
      /Users/Shared/Entwicklung/node_modules doesn't exist or is not a directory
      /Users/Shared/node_modules doesn't exist or is not a directory
      /Users/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
 @ ./src/app/dashboard/dashboard.module.ts 4:0-71 10:134-145
 @ ./src/app/app-routing.module.ts 41:28-70
 @ ./src/app/app.module.ts 26:0-56 189:12-28
 @ ./src/main.ts 6:0-45 10:69-78

The same error appears also for the packages:

@nativescript/core/profiling

Because we had
"@nativescript-community/ui-pager": "^13.0.33"
the breaking changes were updated automatically.
We fixed the issue with specifing the previous version with:
"@nativescript-community/ui-pager": "13.0.33"
Let me know, if you need any more information.

+1
Ran into the same issue today. Using specific versions (13.0.33) does help. Thank you @ishiharas!


Document guarantees around drop and panicing

Currently, when a panic occurs during a regular (i.e., non-unwinding) drop of a local variable, the remaining local variables are still going to have their drop called as usual: https://play.rust-lang.org/?gist=1e2b47513bbfebb71cb6d4e05eca822d&version=stable&mode=debug
Furthermore, if a panic occurs during the drop of a struct with members, the member's drop is also still going to be executed: https://play.rust-lang.org/?gist=69c5e0e922f96dbdc939dad6453ecc6c&version=stable&mode=debug
It would be nice to have such guarantees spelled out explicitly somewhere. Given that this is externally visible behavior, it is probably covered by the stability guarantee, but still -- these guarantees will be really important for providing safe stack pinning APIs.
I am not sure what would be a good place for this to be documented, maybe somewhere in https://doc.rust-lang.org/stable/reference/?
Cc @pythonesque
(Moved here from https://github.com/rust-lang/rust/issues/50765)

Leaving a link here to an assumption we make in generator optimizations:
https://github.com/rust-lang/rust/pull/60840#issuecomment-492435720

Thanks! I feel this is somewhat separate; your assumption is more on the level of MIR semantics that drop shim generation. But it's probably sufficiently closely related.
For a different discussion, see https://github.com/rust-lang/rust/issues/60611 and https://github.com/rust-lang/rust/issues/60822.

Cc https://github.com/rust-lang-nursery/nomicon/issues/135

@Gankro

would especially appreciate if someone could properly write down [...] what on earth happens with double-panics and panic-in-destructors.

playground:
struct T(A, B, ManuallyDrop<C>);
impl Drop for T {
    fn drop(&mut self) {
        let _d = D;
        panic!();
        unsafe { ManuallyDrop::drop(&mut self.2) };
    }
}

struct U(T);

fn main() {
    std::panic::catch_unwind(|| {
      let _e = E;
      let _f = F;
      let _t = U(T(A, B, ManuallyDrop::new(C)));
    });
    // x
}

prints:
D
A
B
F
E

That is, when a destructor panics:

The values in the Drop::drop stack frame that panics are dropped in reverse order.
The fields of the type whose Drop::drop panics are then dropped
The panic escapes Drop::drop, and unwinding continues

The type whose Drop::drop impl droped is left in a partially-dropped state. All of its fields have been dropped, but its Drop::drop impl does not finish running. E.g. in the case above, the destructor of the ManuallyDrop field was not called. All types containing it are also therefore in a partially-dropped state. Attempting to use these types in this state without re-initializing them is probably going to lead to UB.
We do not guarantee that program execution will reach // x above, e.g., if panic=abort, or if the user unplugs their computer. We do however guarantee that , if program execution reaches // x above, then all these destructors have run in precisely this order, and unsafe code can rely on that to build safe abstractions.

Also see the discussion at https://github.com/rust-lang/rust/pull/60766#discussion_r288687259.


[BUG] When building the documentation a warning regarding missing file is issued

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the FastAPI documentation, with the integrated search.
[X] I already searched in Google "How to X in FastAPI" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to FastAPI but to Pydantic.
[X] I already checked if it is not related to FastAPI but to Swagger UI.
[X] I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
The issue can be reproduced by running the build command for the documentation
`python ./scripts/docs.py build-all`

Description
When running the above command, the following output is generated: (part)
Building docs for: tr
Building docs for: uk
Building docs for: zh
WARNING:mkdocs.structure.nav:A relative path to 'index.md' is included in the 'nav' configuration, which is not found in the documentation files
Successfully built docs for: zh
Successfully built docs for: tr
Successfully built docs for: ko
Successfully built docs for: az
Operating System
Windows
Operating System Details
No response
FastAPI Version
0.73.0
Python Version
Python 3.9.4
Additional Context
Every documentation containing a mkdocs.yml file expectes at least the index.md file to be present.
The azerbaijani translation does not contain an index.md file, causing the above mentioned warning when building the documentation.
Even though it's just a warning, it does not mention which language is affected making it hard detect actual issues, while testing the documentation.
Two possible solutions

would be to remove the azerbaijani translation entirely, since it does not actually contain any translated documents.
Or, probably a better, temporary work around would be to include the englisch version of the index.md in the azerbaijani docs folder until a proper translation is added.

I'd like to offer a PR fix for solution number 2.

I'd like to propose PR #4524 as fix for this issue


Investigate compile time of serde crate

We were doing so well for a few rustc releases. Building the current master branch of serde:

compiler
time in seconds

1.12.0
29.5

1.13.0
8.3

1.14.0
7.2

1.15.0
6.1

1.16.0
5.8

1.17.0
6.2

1.18.0
5.8

1.19.0
5.8

But since then it has been going up and up:

compiler
time in seconds

1.20.0
6.2

1.21.0
6.7

1.22.0
6.9

1.23.0
6.4

1.24.0-beta
8.6

1.25.0-nightly
8.6

Will need to look into what makes the current code take 50% longer to build with a modern compiler compared to an old compiler.

Building the same commit (59017aa19b48a2fff795f4e1915dbb596ac427db) with some newer compilers:

compiler
time in seconds

1.24.0
8.6

1.25.0
7.7

1.26.0
8.5

1.27.0
8.5

1.28.0
8.5

1.29.0
8.1

1.30.0
8.0

1.31.0
8.2

1.32.0
7.6

1.33.0-beta
7.6

1.32.0-nightly
7.8

One thought I had today about the compile time of the serde crate is that it very frequently nowadays has the derive feature enabled, which means that the compile time of serde is actually that of serde_derive plus serde itself. That dependency edge can add up to ~20s just to compile, magnifying the compile time much more than the code in serde itself.
There's not really a solution to this today per se. A workaround is to meticulously only use serde_derive in all your crates, and then hopefully some crates only depend on serde to compile faster before serde_derive finishes. Practically though this is unlikely to bring much benefit. Actually solving this would require extensive compiler changes, and I'm not even sure what those would look like!
Put another way though the compile time of serde I don't see as too too important, but serde_derive, which almost everyone using serde needs, is more important because it takes longer to compile than serde itself.
One "quick" way to improve the comiple time of the serde_derive crate is to make the crate itself an empty crate which immediately calls into another crate. Cargo can't pipeline compilation of procedural macros, so procedural macros fully wait for their dependencies to finish (in this case syn). Codegen for syn often takes awhile though (even 40% of time in a debug build) and that's opportunity for parallelism which isn't currently being taken advantage of. If serde_derive were a tiny crate that simply called serde_derive_impl::expand(the_inputs...) then that would enable pipelining, and the compile time of serde_derive itself would be near instantaneous (it's just a link step). That'd improve compile times modestly I think, but it's no slam dunk by any means.

Thanks, I can see how that would help. That is a neat visualization.
It is unfortunate that most people would probably perceive it the opposite way if we make this change -- more crates = more bloat = perceived as slow, even if it builds end-to-end faster than the current setup.

Before splitting serde_derive:

:tada: After:

Here's another data point of a different graph I've seen:

That's got a lot of dependencies pruned, but the entire compilation of serde_derive could fit within the codegen time of syn. That may happen when syn has all of its features enabled, perhaps? In any case I'm getting similar 1-2s wins locally in my graphs, but I think the wins get magnified on smaller machines and in larger compilation graphs. For example the compilation here could probably save at least 10-20s with a pipelining opportunity.
Anyway, as I already wrote on the forums, there's a feature request on rustc to enable this pipelining and I agree that in general users (even serde) shouldn't have to, in the long term, use a crate organization like this just to get better compile times.

This would be virtually solved everywhere if serde was part of stdlib. It would maybe increase marginally the compile time for the std, but it would improve compile times for most projects.
The only thing would be how to make the macros usage compile faster. Would watt be a good solution that can become pŕoduction ready? Can we leave it as an external experiment for now and eventually replace it in stdlib too? Why isn't it used in serde yet?
I'm using the issue #1025 to focus the debate around putting it into std, but it may require a new issue.

I have a bunch of observations/questions here!
To me, it seems another issue is that the code that uses serde_derive is slow to compile. Specifically, it seems that JSON serialization turns out to be a non-trivial part of rust-analyzer's overall compile time.

I can't measure that directly, but the following two high-level benches nod in that direction:
First, if I run RUSTC_BOOTSTRAP=1 cargo build -Z timings --release -p rust-analyzer --bin rust-analyzer, I get the following output:

cargo_metadata and especially lsp-types are among the heaviest dependencies (rust-analyzer's own crates are noticeably bigger, but they contain a lot of domain logic). Both cargo_metadata and lsp-types are mostly struct and enum definitions with derive(serde).
Second, if I run cargo llvm-lines --release -p rust-analyzer --bin rust-analyzer, I get the following:
  326920 (100%)  9611 (100%)  (TOTAL)
   12281 (3.8%)   196 (2.0%)  core::result::Result<T,E>::map
   10576 (3.2%)    49 (0.5%)  <serde_json::value::de::MapDeserializer as serde::de::MapAccess>::next_key_seed
   10080 (3.1%)    63 (0.7%)  serde_json::value::de::visit_array
    9487 (2.9%)    88 (0.9%)  <serde_json::value::de::SeqDeserializer as serde::de::SeqAccess>::next_element_seed
    7987 (2.4%)    50 (0.5%)  serde_json::value::de::visit_object
    6981 (2.1%)    48 (0.5%)  serde_json::value::de::<impl serde::de::Deserializer for serde_json::value::Value>::deserialize_struct
    6921 (2.1%)    77 (0.8%)  <serde_json::value::de::MapDeserializer as serde::de::MapAccess>::next_value_seed
    6534 (2.0%)    54 (0.6%)  <serde_json::value::ser::SerializeMap as serde::ser::SerializeMap>::serialize_value
    4273 (1.3%)     1 (0.0%)  <lsp_types::_::<impl serde::de::Deserialize for lsp_types::TextDocumentClientCapabilities>::deserialize::__Visitor as serde::de::Visitor>::visit_map
    3808 (1.2%)    28 (0.3%)  alloc::raw_vec::RawVec<T,A>::grow_amortized
    3692 (1.1%)    26 (0.3%)  alloc::raw_vec::RawVec<T,A>::allocate_in
    3395 (1.0%)    56 (0.6%)  serde_json::value::de::<impl serde::de::Deserializer for serde_json::value::Value>::deserialize_option
    3192 (1.0%)    56 (0.6%)  alloc::raw_vec::RawVec<T,A>::current_memory
    2877 (0.9%)    49 (0.5%)  <serde_json::value::de::BorrowedCowStrDeserializer as serde::de::Deserializer>::deserialize_any
    2757 (0.8%)    49 (0.5%)  <serde_json::value::de::MapKeyDeserializer as serde::de::Deserializer>::deserialize_any
    2624 (0.8%)   148 (1.5%)  <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
    2538 (0.8%)    51 (0.5%)  serde::de::Visitor::visit_string
    2520 (0.8%)    28 (0.3%)  core::alloc::layout::Layout::array
    2388 (0.7%)     1 (0.0%)  <lsp_types::_::<impl serde::de::Deserialize for lsp_types::TextDocumentClientCapabilities>::deserialize::__Visitor as serde::de::Visitor>::visit_seq
    2078 (0.6%)     1 (0.0%)  lsp_types::_::<impl serde::ser::Serialize for lsp_types::ServerCapabilities>::serialize
    2020 (0.6%)     1 (0.0%)  rust_analyzer::run_server

It seems to be that sered-using crates are both heavy to compile themselves, as well as add downstream monomorphisations to the consumers.

With this in mind, I have the following questions (extremely vague, mind you!):

Do we understand how much of this perceived cost is essential? Hypothetically, if I remove the serde and just write JSON serialization/deserialization "by hand", will I be able to save much in terms compilation time?
This should have been a question about dynamic-dispatch based serializtion, but I figured I should just check how fastre to compli cargo-metadata would be with miniserde!


Add 22.3.0

Has a click fix

@jonashaag You should have the necessary permission, feel free to push on your own to master.

Wasn't aware!


Password manager

@MonishRaman

update index.html file it iis not showing on live server

@MonishRaman do changes


Improve debug info when a relay is required and one is not available

$ sudo nexctl nexd peers ping
HOSTNAME                                      WIREGUARD ADDRESS     LATENCY     PEERING METHOD     CONNECTION STATUS
<hostname>     <ip>           -           none               x Unreachable

This was observed on a host behind symmetric NAT in a VPC with no relay available. In this scenario, we know that connectivity won't work, and we know why this is the case. We should add output here to give clear instructions on what to do next.
Similar helpful output should be provided in the nexd peers list output.
A link to the relevant docs should be included.  https://docs.nexodus.io/user-guide/discovery-and-relay/
The log info could be improved as well.

{"level":"info","ts":"2023-11-22T13:40:36.953Z","caller":"nexodus/nexodus.go:1202","msg":"Symmetric NAT is detected, this node will be provisioned in relay mode >

This sounds like the node will turn into a relay. It should ber clear that this node requires a relay to be present in the VPC. We could also add log messages to note when a relay is required and not present.

https://docs.nexodus.io/user-guide/discovery-and-relay/
This doc should also be improved to make it clear that this is necessary for any user of the system with a VPC that needs a relay.

https://docs.nexodus.io/user-guide/discovery-and-relay/#silent-onboarding
Update this doc to include the instructions for when using packages. For example, when using the rpm:
Install/run just like you did on your other machines, but edit /etc/sysconfig/nexodus
It should include:
NEXD_ARGS="--service-url https://try.nexodus.io/ relay"


Inheritance case

If you use inheritance, then this check still triggers.
We have some code kinda like so:
class my_network::config (
  Hash $parameter_one,
  String $parameter_two,
) inherits my_network {

  # More code

The check triggers on this. I assume because of the "inherits" part.
Might be because of something else. I just assume it is because it doesn't handle inherits.

I can't read apparently.


Revamp logical dimensions

This PR changes how we handle dimensions. All dimensions created before the search is started are now registered in a logical dimension. Logical dimensions are groups of actual dimension that form a (possibly) tiled dimension in the original code.
One of the side effect is that the user now exclusively deals with logical dimensions when interacting with the builder. All the magic to convert the indexes/access patterns/nestings on logical dimension to actual dimensions is hidden in the builder.
This allowed us to split the size of dimensions in two types.

The old ir::Size type is simplified to remove the tiling information. It now corresponds to the size of a logical dimension.
An ir::PartialSize type is added, it corresponds to the size of an actual dimension. Later, this type will be modified to depend on the decisions exposed in the domain.

bors r+


Adding an option to strip a path from the source map path before prepending a prefix.

When using the sourceMappingURLPrefix the default behaviour to remove leading dots is sometimes not sufficient. For example:
sourcemaps.write('../../path/to/maps', {
        sourceMappingURLPrefix: 'https://asset-host.example.com/assets'
})

would result in a url of the form sourceMappingURL=https://asset-host.example.com/assets/../path/to/maps/helloworld.js.map
This change adds an option to specify a path which is first stripped from the unprefixed path, before then adding the prefix, e.g.
sourcemaps.write('../../path/to/maps', {
        sourceMappingURLPrefix: 'https://asset-host.example.com/assets',
        sourceMappingURLPrefixStripPath: '../../path/to'
})

which would result in a url of the form sourceMappingURL=https://asset-host.example.com/assets/maps/helloworld.js.map

Coverage remained the same at 100.0% when pulling 563d07020a0799403663cc839d9b7e8d0fb649bc on cjwainwright:prefix-strip-path into 8349c0c9c13305aa46e59d842400380a2ca27356 on floridoo:master.

This is now possible through the sourceMappingURL option, which allows full control over the whole URL.


ARM MPU _arch_buffer_validate allowing reads to kernel memory

Reproducible with this PR:
https://github.com/zephyrproject-rtos/zephyr/pull/8863
Run tests/kernel/mem_protect/syscalls on sam_e70_xplained.
The check to _arch_buffer_validate on a buffer defined with __kernel is not failing as it ought to be.
Not reproducible on frdm_k64f, so this seems specific to ARM MPU.

The MPU is protecting the memory if I call libc strlen on the same string, so this is definitely a problem with the validation function and not the MPU configuration itself.

@andrewboie submitted a PR that fixes this; the test you indicate is now successful after re-basing PR over #8868 .

thanks so much!!


Zip product

Closes #48
@limx0 care to look this over?

Codecov Report

Merging #50 into master will increase coverage by 0.11%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #50      +/-   ##
=========================================
+ Coverage   97.48%   97.6%   +0.11%
=========================================
  Files           9       9
  Lines         994    1043      +49
=========================================
+ Hits          969    1018      +49
  Misses         25      25

Impacted Files
Coverage Δ

streams/tests/test_core.py
99.43% <100%> (+0.04%)
:arrow_up:

streams/core.py
98.49% <100%> (+0.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3611d00...122e472. Read the comment docs.

@ordirules @danielballan

Although I am more than happy to write whatever the consensus is, I'd personally rather have a whole bunch of nodes which do things that may overlap a little rather than have nodes which have many flags. This way the user only has to remember more nodes rather than having to remember more flags.
I'd like to get this in today if possible.

We discussed about it off PR. Merging combine_latest with zip_latest is not a good idea, since it would force adding a buffer to combine_latest. I agree with the current PR and see its value.
However, we might consider discussing more complex scenarios for this stream before discussing merging. Let's say we have 3 streams:
Stream 1 : ----1----------2---------------------3-
Stream 2 : --------I--II-----III-------------------
Stream 3 : ---------------------------a----b------

What should be the output of this? Should it be:
(2, III, a), (2, III, b), (3 III, b)

or should it be recursive, like:
(2, I, a), (2, II, b), (2, III, a), (2, III, b), (3, III, b)

I would be tempted to say the former (which as far as I can tell is what the code is currently doing), since the latter could be done with (I think...):
s4 = s1.zip_latest(s2)
s4.zip_latest(s3)

So my vote is we need to write two tests that output the two scenarios I mentioned, then I'm okay with it.
So something like:
s1  = Stream()
s2 = Stream()
s3 = Stream()
s_simple = s1.zip_latest(s2, s3)
s_recursive = s3.zip_latest(s1.zip_latest(s2))

L_simple = s_simple.sink_to_list()

L_recursive = s_recursive.sink_to_list()

s1.emit(1)
s2.emit('I')
s2.emit("II")
s1.emit(2)
s2.emit("III")
s3.emit('a')
s3.emit('b')
s1.emit(3)

Any other thoughts? Note, I haven't tested this, and output may not be correct but this is the idea.

I'm confused, which is the lossless stream in your example?

I updated it to clarify. The output was incorrect. Also shortened. Please read above.

After discussion with @CJ-Wright, to clarify, this routine makes an assumption. Only one stream is considered to be the lossless stream, and thus buffered. In s1.zip_latest(s2, s3), the lossless stream is always s1. This is like combine_latest with emit_on=[s1] and an additional lossless buffer for s1. Once all streams have emitted at least once, the behaviour is exactly the same as s1.combine_latest(s2, s3, emit_on=[s1]).
The case of multiple lossless buffers is not considered, but @CJ-Wright will post an issue. It will require recursion through the buffers.

I'm going to merge this, everything passes and such. If we find the name super distasteful we can always change it again.


Outdated Comment About cosmos-sdk's Error Handling

This comment says cosmos-sdk does not handle ErrorGasOverflow same as ErrorOutOfGas.
https://github.com/CosmWasm/wasmvm/blob/d40163d45f8f750bca50a07245118951e9f475bc/api/callbacks.go#L70-L73
But, the present version of cosmos-sdk looks treating ErrorGasOverflow same as ErrorOutOfGas.
https://github.com/cosmos/cosmos-sdk/blob/3c82a32f9b2d8773114282b1977061cfc587dc26/baseapp/baseapp.go#L591-L594

but this needs to be periodically verified, in case they do start checking for this type

Thanks for checking on this.
I looked at the code you linked and the function it calls but I only see ErrorOutOfGas there. Can you point me to where ErrorGasOverflow is handled?


state/cache: implement get_many/set_many

This PR implements batching save and get in State.
As a result of this change, _build_tree() is faster by 70% - 90% on the first run, and 50% faster on repeated runs with this PR. You can find the results below for the mnist dataset.
This should dramatically improve the performance for dvc add --no-relink, and other places where we build trees and compute hashes. Real-world impact is mixed and depends on individual commands, but all of the commands show improvements from 4-5 % to 20%, and in some cases,  even 70% or more.
This also brings back support for core.checksum_jobs to concurrently hash multiple files at once.
This is done by partitioning the files to hash into two buckets: a) files that are smaller than 1MB and, b) files that are larger than 1MB.
Files smaller than 1MB are hashed sequentially, and larger files are hashed in a ThreadPoolExecutor.
Benchmark
main
$ rm -rf .dvc/tmp
$ time dvc-data build dataset/
object e42412b82dcab425ce9c7e2d0abdfb78.dir

________________________________________________________
Executed in   15.16 secs    fish           external
   usr time   10.63 secs  270.00 micros   10.63 secs
   sys time    3.18 secs   44.00 micros    3.18 secs

$ time dvc-data build dataset/
object e42412b82dcab425ce9c7e2d0abdfb78.dir

________________________________________________________
Executed in    3.03 secs    fish           external
   usr time    2.73 secs  277.00 micros    2.73 secs
   sys time    0.27 secs   49.00 micros    0.27 secs

PR
$ rm -rf .dvc/tmp
$ time dvc-data build dataset/
object e42412b82dcab425ce9c7e2d0abdfb78.dir

________________________________________________________
Executed in    3.31 secs    fish           external
   usr time    2.48 secs  246.00 micros    2.48 secs
   sys time    0.69 secs   32.00 micros    0.69 secs

$ time dvc-data build dataset/
object e42412b82dcab425ce9c7e2d0abdfb78.dir

________________________________________________________
Executed in    1.37 secs    fish           external
   usr time    1.21 secs  275.00 micros    1.21 secs
   sys time    0.14 secs   35.00 micros    0.14 secs

DVC-bench results on an MNIST dataset

Test
main (in s)
PR (in s)
△

test_add-add
87.2241
62.7932
28%

test_checkout_copy-checkout
31.0993
29.1070
6%

test_checkout_copy-checkout-noop
6.5580
5.4686
16%

test_checkout_copy-checkout-update
7.5199
6.2810
16%

test_data_status-data-new
21.3120
15.9083
25%

test_data_status-data-noop
5.4173
4.1119
24%

test_data_status-data-changed
5.3719
4.1306
23%

test_data_status-data-changed-noop
5.3541
4.1927
21%

test_diff-diff
22.6914
18.3842
19%

test_diff-diff-changed
6.5643
5.3981
17%

test_diff-diff-changed-noop
6.5565
5.3985
17%

test_diff-diff-noop
6.6704
5.3392
20%

test_import_url-import-url
131.6878
87.0350
34%

test_status-status
20.6743
6.0155
71%

test_status-status-changed
6.0105
3.1429
48%

test_update-import-url
130.6300
86.5345
33%

test_fetch-fetch
37.0039
26.7890
27%

See https://github.com/iterative/dvc-bench/actions/runs/10245232542.

@skshetry what is the status of this? what is the context for this?

what is the context for this?

At the moment, we save each state entries one at a time, and each individual save is within their own transaction. If you have a lot of files, this can be as costly as doing copy/symlink/reflink. (On my machine, it takes 1872 us for a __setitem__, 2103 us for a reflink).
(See https://github.com/iterative/dvc/issues/10491, https://github.com/iterative/dvc/issues/6227 and tens of other issues).
The idea is to try to save all the entries at once, at least for all files within a directory at a certain level. (We already assume that the dataset directory that user's are tracking are shallow and each level has tons of files. This goes with what we see in real world with sharding datasets into different directories like train/0/, train/1 etc.).
But to take advantage of this, we need to also change how we build objects for a tree (which happens in build_tree). Instead of doing everything inside a for-loop, we need to separate the operation into multiple steps, with all of them doing batching that includes:

getting all saved hashes from State
figuring out files to hash (if they are missing from State).
hashing all files at once
saving all hashes back to State.
building the tree object.

"hashing all files at once" will also gives us ability to concurrently hash files. We can even split this into two stages: hash small files (< 1MB) sequentially, and hash larger files in a ThreadPoolExecutor. (Note that we already have to gather fs.info() of all the files while building the object which already has size information).

what is the status of this?

That's a matter of priority that we set. If it gets prioritized, I can work on this, initially focusing on this and improving performance for dvc add.

which happens in build_tree

Do you mean build_entry (https://github.com/iterative/dvc-data/blob/ffa6839e35cdd193da469605978eb8e0946433ee/src/dvc_data/index/build.py#L19)?

I don't see that much time is spent in build_tree.

Both do similar work, but one builds the tree object, while other builds the index.
build_tree is for non-index based operations. That is used in dvc add, dvc commit and dvc repro. In dvc repro, it's even worse because we rebuild objects many times.
build_entries is for index-based operations which is used by data status, dvc checkout, etc.

Is it hashfile.build._build_tree then? I saw index.save.build_tree initially, but that doesn't make sense to me.

@skshetry yes, I was looking into the disckcache and build_tree recently. It all makes sense to me if it's true that disckcache can't handle many parallel operations (may be with some SQLite changes). If there is not obvious way to check this (I was about to change dvc-data to add TQDM first and make it parallel to play a bit with SQLite versions / settings) - it's fine to go this path and do batching (+ also making it parallel in terms of hashes).
DiskCache is really advertising multithreading, really good performance, etc - so it's surprising to see that it's just getting stuck on a "basic" parallel inserts so much. If this is about transactions - can we check if we can run everything in a single transaction then (and if it fails - start over? or do partial commits if possible each 1000?)
Are there some SQLite settings we can tune / disable that we don't need since we are not a web app?
Does DisckCahe create any indecies? Can you share the structure of the DB underneath?

DiskCache is fast for web applications like django apps, etc. as a replacement for redis.
On those web applications, you have a very different workload, where you are unlikely to save millions of rows, or read 10s of millions of rows at once.
Using sqlite in parallel does not work that well. There is python thread overhead, and on read, returning rows has to be converted to python objects which is going to be blocked by GIL. On write, it is going to be serialized on sqlite3 side.

If this is about transactions - can we check if we can run everything in a single transaction then (and if it fails - start over? or do partial commits if possible each 1000?)

Yes, using transactions help (reduces as much by 10x), but you will still have to call Connection.execute a million times. We can do better by using executemany.

Are there some SQLite settings we can tune / disable that we don't need since we are not a web app?

I tried changing some settings but nothing helped when I worked on dvc-data-rs.

Does DisckCahe create any indecies? Can you share the structure of the DB underneath?

Yes, it does. I find it useful for our read-heavy workload.
You can see the structure here: https://github.com/skshetry/dvc-data-rs/blob/4be70284f7c7be1a11e0d8c250a7ea3df4cdeb42/src/state.rs#L43-L66.

Is it hashfile.build._build_tree then? I saw index.save.build_tree initially, but that doesn't make sense to me.

Sorry, yes it is _build_tree.
https://github.com/iterative/dvc-data/blob/ffa6839e35cdd193da469605978eb8e0946433ee/src/dvc_data/hashfile/build.py#L90

Yes, using transactions help (reduces as much by 10x), but you will still have to call Connection.execute a million times. We can do better by using executemany.

no doubt about that (though we might loose a bit since it will be serials - first read, then md5, then write) vs all in parallel.
just to be mindful about the complexity of changes if we can do 10-15x with simple changes - I would do that first. That's a huge difference and can unblock many people and we can do it "quick".

(though we might loose a bit since it will be serials - first read, then md5, then write) vs all in parallel.

Parallelizing a large function that does a lot of things, and has different characteristics (IO-bounded + CPU-bounded) does not work well as we have found out in dvc. It has tons of overheads, hides a lot of things, makes it difficult to optimize and most importantly, does not scale linearly. Also, not everything is parallelizable as we see with sqlite3.
If you look into our internals, we have slowly moved away from doing that, and instead, we have added batching support to critical APIs like fs.info(), transfer, etc.
Our plan was/is to optimize those APIs if needed (maybe by dropping to C extensions or looking for a different solutions or maybe using asyncio/threads in those individual functions).

Regarding "loose a bit" thing, if it's going to be faster than running it in parallel, then we won't lose anything. Running some computations to check what files to hash are going to be cheaper than going to sqlite3, or running on another thread with GIL.

Codecov Report
Attention: Patch coverage is 88.78505% with 48 lines in your changes missing coverage. Please review.

Project coverage is 67.50%. Comparing base (081a1d8) to head (2e097a4).
Report is 15 commits behind head on main.

Files
Patch %
Lines

src/dvc_data/hashfile/build.py
68.51%
20 Missing and 14 partials :warning:

src/dvc_data/json_compat.py
64.28%
5 Missing :warning:

src/dvc_data/hashfile/hash.py
78.94%
3 Missing and 1 partial :warning:

src/dvc_data/compat.py
81.81%
1 Missing and 1 partial :warning:

src/dvc_data/hashfile/state.py
96.96%
2 Missing :warning:

src/dvc_data/hashfile/db/__init__.py
75.00%
0 Missing and 1 partial :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #522      +/-   ##
==========================================
+ Coverage   62.98%   67.50%   +4.51%
==========================================
  Files          62       64       +2
  Lines        4342     4693     +351
  Branches      740      780      +40
==========================================
+ Hits         2735     3168     +433
+ Misses       1448     1338     -110
- Partials      159      187      +28

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks @skshetry! Do you have any examples or benchmarks to show the overall improvement and in what scenarios it should be better? Are there any scenarios where you expect it to be worse?

Thanks @skshetry! Do you have any examples or benchmarks to show the overall improvement and in what scenarios it should be better?

See the description above (at the very end) and https://github.com/iterative/dvc-bench/actions/runs/10252645782.

Are there any scenarios where you expect it to be worse?

INSERT OR REPLACE INTO is usually slower than what diskcache does if there are existing keys. Diskcache first selects key and either inserts or updates the row.
This is likely due to unique index constraints that we have on (key, raw).
And now _build_file/_build_tree goes through more code and stages, so in small microbehchmarks, they may be slightly slower. But this is unlikely to show up in real-world.
(Check out the above benchmarks).


undefined method `validate_unsupported' for #ManageIQ::Providers::Openstack::CloudManager::Template:0x000000153ed6a8

When doing SSA of Template(image)
http://paste.openstack.org/show/478820/

Fixed by the following:
https://github.com/ManageIQ/manageiq/pull/5528

@roliveri hm, sure you linked correct pr? I have a fix in #5434, should I close it?

@Ladas - I did paste the wrong PR. I intended to paste:
https://github.com/ManageIQ/manageiq/pull/5439
but you're right,
https://github.com/ManageIQ/manageiq/pull/5434
is required as well.
I was unaware that PR hadn't been merged. If no one has any issues with it, I can merge it now.


[MRG] Refactor shorthand parser

Addresses #46 – parsing of shorthands is now more robust, so that column names with spaces, colons, parentheses, and other characters works.
Additionally, I moved the shorthand parsing to the _finalize() method, rather than using a traitlet observer. Due to the refactoring, I was also able to remove the specialization of type within channels, as well as observers of shorthand and type.
The result is about 70 fewer lines of code overall, excluding the additional tests that are part of this PR.

One potential issue here: the altair output will now potentially differ depending on whether data is passed as a dataframe or a URL. For example, if a dataframe has a column named sum(foo), this will be passed through the shorthand. On the other hand, if a URL is provided which points to data with a column named sum(foo), this will be treated as an aggregate over column foo, because there's no way for Altair to know what the URL contains.
I wonder if it might be better to remove the column validation procedure, so that output is consistent regardless of the data source.

OK – it's simplified now and doesn't use any fancy column matching. Note that if you have a field that looks like sum(foo), you can leave the shorthand blank and specify it via field instead.

@ellisonbg – I think this is ready to go. I'd merge myself, but I don't want to conflict with anything you might be working on.

Great, merging!


puts output is interleaved between different fibers

why output these? sometime is right.
MacOS

Hi,
Please next time provide code that we can copy and paste. Now I have to type all of that to try it... It would also help if you explained what was your expected output, though in this case I guess you don't want the output interleaved.
puts(obj) first prints object, then prints a newline. It might happen that these calls interleave between fibers.
There are several solutions to this:

Do nothing: use mutex or something similar, don't assume IOs are fiber-safe (they are not)
We could make the top-level puts and print methods fiber-safe
We could have a flag set on IOs to make them fiber-safe. STDOUT would be marked as fiber-safe by default.

I'd start with option 2 for now, as I'm not sure 3 is generally applicable.

There are 2 writes to STDOUT but there is no guarantee that each write will write everything it's supposed to. Well, in this example this is a single byte write.

this is how we fixed Logger.

Having special care for ::puts but not for IO#puts could be inconsistent.

Interesting. IO#write could check whether a lock is set and use it?

@ysbaddaden 3. Yes, we could make IO::FileDescriptor have a flag fiber_safe or similar. Then all writes (and reads?) would synchronize using a mutex. I don't know if this could be generally applied to all IOs because IO is almost abstract, but at least having it in IO::FileDescriptor and maybe other IOs provided in the standard library could be useful.
In a way I'd like to think of that as how IO in Ruby is super practical: buffered by default, but you can control that sync; it supports encoding and you can set it, etc.

1000.times do |i|
    spawn do
        puts i
    end
end

sleep

@asterite thanks, can you provide that right example with sync?

@jefurry
mutex = Mutex.new

1000.times do |i|
  spawn do
    mutex.synchronize do
      puts i
    end
  end
end

sleep

@asterite thanks.


Adding Diagnostics Source events/insight to Caching

As a consumer of diagnostics data we would like to be able to know when a cache hits and misses occur. Having this data will allow us to start realizing/showing "case and effect" with cache hits and misses. I would imagine that as a very minimum we would find out when hits occur, what was the key, the data, how long it took, and the source of the data (in memory vs not), and when misses occur, what was the key, how long it took, and the source of the data (in memory vs not), etc. There might be more data that would be interesting, but we can flush that out in the issue.
@Tratcher @Eilon

Moving to backlog.

cc @blakepell


Get CI back running

tl;dr Pinning versions to those available in August 2022 and using older github-actions  fixed the docker image and book build. This is not a longterm solution. This needs some discussion how to proceed here.

#100 Trying to get CI running, almost wasted effort, since changing the wrong  workflow :roll_eyes:
#101 Fixing changes from #100, now using bionic as base image which corresponds to lrose-core 18.04 version
#103 fixes GLIBC issue with newer actions
#104 Checking and Pinning dependencies to get book build running again, forcing repos2docker version to 2022.02.0

Fixed:

container/image builds again on main and was pushed to https://github.com/openradar/erad2022/pkgs/container/erad2022
closed all dependabot PR's until root cause of https://github.com/actions/checkout/issues/1590 is identified and mitigated
pinned Py-Art to 1.12.7 and jupyter-book to 0.12.3 make the linkchecker work again
fixed links in wradlib-notebooks, linkchecker works again!
fixed pinnings and versions, get book building again

Issues:

[x] https://github.com/actions/checkout/issues/1590, need to use checkout v3
[x] sphinx 4.5.0 is introduced by pip installing (pyart and/or sphinx-pythia-theme) which in turn break book-build, fixed by pinning jupyter-book
[x] force juypter-repo2docker version to 2022.02.0, this fixes remaining build issues
[ ] The above will only work until github restricts actions to use node20, see https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/, that means we are bound to upgrade our base_image to include newer GLIBC anyway.

@mgrover1 #100, #101, #103 and #104 iteratively fixed the CI.
This just forced the CI, docker image etc back to versions around August 2022. That seemed to work, at least the rendered book contains the nicely rendered notebooks. See above this is no longterm solution.
@mike-dixon @leavesntwigs @DanielMichelson
If there is any movement forward on how to add lrose and baltrad to conda-forge this would really help for long-term maintenance of such repo here.

I'll leave the issue open for a while just that everyone who gets directed to this repo immediately sees what's going on.

Thanks for all the work here @kmuehlbauer - I think this is a reasonable solution.


[ai-text-analytics] characterCount -> graphemeCount

Rename characterCount to graphemeCount in DocumentStatistics for consistency with use of "grapheme" in response objects.

/azp run js - textanalytics - tests


[SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/.

Describe the problem you faced
Getting Failed to delete the hudi jar from /usr/lib/hudi/./ location after upgrading the EMR on EKS to 6.15.0 which has in built hudi 0.14.0 and Spark 3.4.1 version.
The application fails at the start of the spark application with the below error message,
Files  local:///usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar from /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar to /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
23/12/19 10:11:38 WARN JavaUtils: Attempt to delete using native Unix OS command failed for path = /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar. Falling back to Java IO way
java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingUnixNative(JavaUtils.java:177) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:113) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:94) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1232) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$14(SparkSubmit.scala:436) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.map(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractTraversable.map(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.deploy.SparkSubmit.downloadResourcesToCurrentDirectory$1(SparkSubmit.scala:427) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$17(SparkSubmit.scala:452) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at scala.Option.map(Option.scala:230) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:452) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1001) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
Exception in thread "main" java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:150)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:121)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:94)
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1232)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$14(SparkSubmit.scala:436)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at org.apache.spark.deploy.SparkSubmit.downloadResourcesToCurrentDirectory$1(SparkSubmit.scala:427)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$17(SparkSubmit.scala:452)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:452)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1001)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

To Reproduce
Steps to reproduce the behavior:
Submit a spark job in EMR on EKS with spark.jars=/usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
Expected behavior
Spark application to run without any errors
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : 0.14.0

Spark version : 3.4.1

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : yes

@Lakshmi-Holla12 Interesting. Not sure why spark is trying to delete the jar from that location. Can you try putting hudi spark jar in some different location and point that to it.

@Lakshmi-Holla12 Were you able to resolve this issue?

Hi @ad1happy2go, I'm still experiencing the issue with EMR 6.15. By default, Spark jar is located at /usr/lib/hudi, which I am currently using. During the spark-submit, I provide the following parameters --jars /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar and --py-files /usr/lib/hudi/<my src code path>. However, I'm not sure why Spark is trying to delete the files. I've tried a few steps to resolve the issue but to no luck:

I attempted putting the Hudi Spark jar in a different location and ran into the same error.
I removed the --jars parameter during Spark submit, this time it looks for --py-files and throws the same error with not able to delete /usr/lib/hudi/./.
I tried setting this configuration spark.hadoop.fs.s3a.deletion.enable, false. This was meant to disable the deletion of temporary files by the S3A filesystem. However, I'm still facing the same issue.
I also tried setting spark.hadoop.io.native.lib.available, false. This was to disable the use of native Unix OS commands for deletion. However, I'm still facing the same issue.

@CTTY Any insights on this. thanks in advance.

Hi @ad1happy2go @CTTY,
I have been testing EMR on EKS with different EMR versions: emr-6.15.0, emr-6.14.0, emr-6.13.0, and emr-6.11.0.
I found that emr-6.11.0 works fine with Spark 3.3.2 and hudi 0.13.0.
However, emr-6.14.0 and emr-6.13.0, which use Spark 3.4.1 and hudi 0.13.1, started to fail with the same error message: "java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.13.1-amzn-2.jar".
Similarly, emr-6.15.0, which uses Spark 3.4.1 and hudi 0.14.0, also encountered the same issue with the error message: "java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar".
I am not sure what changes in Spark 3.4.1 could be causing the deletion of these files.

I think we need to understand why Spark is trying to delete Hudi jars here, can you share the previous log so we can see what Spark is doing before this exception?
On the other hand, I wouldn't use /usr/lib/hudi/ to put your own script as the entire directory is owned by root user there and you may run into some weird permission issue


Check for dependency on spring-boot-starter-webflux for SpringBoot thin goal

Open Liberty does not currently support SpringBoot 2.0.x applications that depend on spring-boot-starter-webflux. We should detect that dependency and issue an error when the thin goal is executed.

@anjumfatima90 Is the support for spring-boot-starter-webflux in Open Liberty now?

Yes, In <IP_ADDRESS>

Closing since this is no longer an issue.


Enable Sentry for Develop Deployments

... to catch errors thrown on mainnet and testnet

it's already done, I just fixed the SENTRY_ENVIRONMENT variable in /etc/counterparty-api.yml files


[Event Request] Codeunit 90 OnPostPurchLineOnAfterUpdatePurchLineBeforePost

we Need a similar trigger as the trigger OnPostSalesLineOnAfterTestUpdatedSalesLine from codeunit 80 as well in codeunit 90:
PROCEDURE PostPurchLine
...
TestPurchLine(PurchHeader,PurchLine);
UpdatePurchLineBeforePost(PurchHeader,PurchLine);
OnPostPurchLineOnAfterUpdatePurchLineBeforePost(VAR Purchline;VAR EverythingInvoiced);  <-- New Event trigger
IF "Qty. to Invoice" + "Quantity Invoiced" <> Quantity THEN
EverythingInvoiced := FALSE;

Another event already implemented for same purpose
IF "Qty. to Invoice" + "Quantity Invoiced" <> Quantity THEN
EverythingInvoiced := FALSE;
OnPostPurchLineOnAfterSetEverythingInvoiced(PurchLine,EverythingInvoiced);


Make maxTextFieldLength configurable

Close NCI-Agency/anet#1299
Add to the dictionary in anet.yml:
  maxTextFieldLength: 250

@gjvoosten The bootstrap text area widget in ANET is not the most user friendly. It comes up with a very small box that the user should manually resize (on certain browsers, not on IE11). To properly guide the users to enter a certain amount of text, we manually inject a  style setting the height of that box, see here:
      longName:
        label: Task description
        placeholder: Enter an effort description, example ....
        componentClass: textarea
        style:
          height: 400px

It would be good if the max number of characters is also defined there - per field. We could then maybe make the height a functon of that max number of chars.

note: the above assumes https://github.com/NCI-Agency/anet/commit/fa99c5e5f0bac2b441a3a411dbef0c72d85a3516#diff-35c967556ebcba61cb279a22390568f1


Channel, direct message or private group not found

When trying to erase all messages from a user we get this result:
Channel, direct message or private group not found
We used:
slack-cleaner --token [TOKEN] --message --channel gossip  --user [USER] --perform
Any ideas?

Shot in the dark here, but is gossip one of your channels? If not, you might want to change it to a channel that you actually have. If you want to delete messages from all the public channels, you might try not using the channel option at all and go with:
slack-cleaner --token <TOKEN> --message --user <USER> --perform

@hukevinxiaochen this returns a slack-cleaner: error: one of the arguments --channel --direct --group --mpdirect is required :'( but it would be a nice option


docs: update examples

Current coverage is 81.12% (diff: 100%)

No coverage report found for master at 6b2f86d.

Powered by Codecov. Last update 6b2f86d...6355aa7


[Adaptive Card iOS - AdaptiveCard.BackgroundImage.FillMode.Cover.Json]: Voice Over is not conveying the name for the 'Date Picker' button.

Target Platforms
iOS
SDK Version
1.0 (2.4.0-beta.<IP_ADDRESS>)
Application Name
Adaptive Cards
Problem Description
Test Environment:
Device: iPhone 11
iOS: 16.3.1
Application: Adaptive Cards
App Version: 1.0 (2.4.0-beta.<IP_ADDRESS>)
Screen Reader: Voice Over
Repro Steps:

Launch Adaptive Cards iOS application.
Home screen will appear.
Activate Voice over
Navigate to "V1.2" control and select it.
Navigate to 'Test' control and activate it.
Navigate to 'AdaptiveCard.BackgroundImage.Fill.Jsons' and activate it.
The Card will open.
Navigate to the 'Date Picker' button and observe the issue.

Actual Result:
While navigating to the 'Date Picker' and 'Combo box', voice over is not conveying the name for the button. It is announcing as 'Opens the date picker button' for date picker and 'Button' for combo box.
Expected Result:
Screen Reader should convey the name for both buttons and it should announce as 'Date Picker button double tap to open date picker' and 'Choice combo box' for the combo box.
User Impact:
It will impact the screen reader user as they will not able to know about the actual use of the control if the screen reader does not announce the proper name of description.
WCAG Reference:
https://www.w3.org/WAI/WCAG21/Understanding/name-role-value.html
"Have feedback to share on Bugs ? Please help fill Trusted Tester Bug Feedback (office.com) “
Screenshots
https://user-images.githubusercontent.com/95913748/229021485-9a6a749d-e20e-41c7-9348-610bfc8521be.mov
Card JSON
NA

Sample Code Language
No response
Sample Code
No response

closing this issue, the app wasn't built correctly perhaps, only cards under Scenario should be tested, not under Elements. I will make note of this and fix it.

Adding A11yTTValidated tag as per the above comment.


Not able to create a file from view

view_name = lookml.View('test_n')
lookml.File(view_name)
Traceback (most recent call last):
File "", line 1, in
File python3.8\3.8.10\lib\site-packages\lookml\lib\project.py", line 47, in init
self.name = self.path.split('/')[-1]
TypeError: 'NoneType' object is not callable

https://github.com/llooker/pylookml/blob/25f4b026944d0bf7dc932fdb4ce0d1a60f34403f/lookml/tests/scenarios/scenario_gen_from_schema_test.py#L69

Hi @subbaraocherukuri ,
lookml.File Function accepts string (path of the file) as argument instead of View object.

The code for this would be:
view = lookml.View('test_n')
my_file = lookml.File('test_n.view.lkml')
my_file + view


Add support for multi-cluster runners

Use cases involving multiple clusters like Cross Cluster Search (CCS) or Cross Cluster Replication (CCR) require various additional runners in order for tracks to properly set them up,
We should implement dedicated runners in Rally for;

Configuring 'local' and 'remote' cluster (these terms are relative) seeds
Add support for 'local' and 'remote' clusters to 'follow' indices

Additionally, all current runners are written only to execute in the context of a single Elasticsearch client for the first cluster passed in via target-hosts (i.e. default). This present a problem for when users would like operations to execute across multiple clusters. We should consider adding a new operation (e.g. all-clusters) that will run a given operation on all clusters without requiring us to modify each existing runner.
Relates;
https://github.com/elastic/rally/pull/1563
https://github.com/elastic/rally-tracks/pull/293

Upon review, multi-cluster benchmarks are almost always going to involve complex enough setups and architectures that it will be difficult to abstract cleanly via generalised multi-cluster runner(s). Our docs suggest as much too.
Closing this for now.


19689 - Fixing color styles except gray.

*Issue:19689 *https://github.com/bcgov/entity/issues/19689
Description of changes:
fixed all the tailwind colors except gray
created follow up ticket for dealing with gray color ticket #19782
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the namex license (Apache 2.0).

/gcbrun

Temporary Url for review: https://bcregistry-btr-dev--pr-117-x7ztcgpb.web.app

/gcbrun

Temporary Url for review: https://bcregistry-btr-dev--pr-117-x7ztcgpb.web.app


Set date of publishing new version

Literally just changing the date.

Merging overriding CI as this is touching only the spec.


More descriptive Docker Cloud hostnames

So on Docker Cloud, the agent gets a hostname based on /etc/hostname, which is pretty good, but a bit opaque for the administrator, who is used to seeing the node cluster name. I wonder if it's possible to prepend the node cluster if it's detected? Maybe try to find it through the Docker Cloud API? Not sure if you'd consider that an edge case or something that should be done in a downstream container, though.

If you are using AWS EC2 as your node provider for Docker Cloud, you can connect Datadog to AWS, and the nodes will be tagged with their node_cluster_name. That does the job for us, at least 👍

This can probably be closed, considering this aspect of Docker Cloud is shutting down on May 21st.


Windows Workflow Foundation

Windows Workflow Foundation was a Microsoft technology that provided an API, an in-process workflow engine, and a rehostable designer to implement long-running processes as workflows within .NET applications. The last version of WF was released as part of the .NET Framework version 4.5 in 2012. It was not brought forward into .NET Core. A workflow was a series of distinct programming steps or phases. Each step is modeled in WF as an Activity. The .NET Framework provided a library of activities. Custom activities can also be developed for additional functionality. Activities could be assembled visually into workflows using the Workflow Designer, a design surface that ran within Visual Studio. The designer could also be hosted in other applications. Encapsulating programming functionality into the activities allowed the developer to create more manageable applications; each component of execution can be developed as a Common Language Runtime object whose execution will be managed by the workflow runtime.
links:

https://en.wikipedia.org/wiki/Windows_Workflow_Foundation
https://andreioros.com/blog/windows-workflow-foundation-net-core/

Initial release:	November 21, 2006; 17 years ago
last release: August 15, 2012; 11 years ago
this framework has since been abandoned - it is not part of .NET 5 onwards
In 2019, Microsoft re-emphasized that .Net Core is the future of .Net and that WF will not be a part of it, forwarding interested parties to the open-source community project.

So this one is a tricky aspect.  .NET Framework is not dead and although there's a ton of different products that are being abandoned in Core and .NET 6+, many are still on life support with .NET framework with a commitment from Microsoft to still support and bundle .NET Framework with Windows.
Until it is actually killed off, it is not "dead" and therefore not ready for the graveyard. Frequently this means extended maintenance support has an announced end.


HHH-9679 - Fix hibernate-enhance-maven-plugin buid script

Already reviewed by @whitingjr

Thanks Luis!  I got this applied.  It seemed ok locally.  The CI run just picked it up : http://ci.hibernate.org/view/ORM/job/hibernate-orm-master-h2/815/console


Revert "feat: exposes the ability to set the provider_visibility in DA"

Description
This reverts commit 51797eb741fab45c875787218ee0711bf3f053f9.
Release required?

[ ] No release
[x] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content
reverts the ability to expose  provider_visibility as mq cloud service doesn't yet support provider_visibility set to private
Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

:tada: This PR is included in version 0.1.1 :tada:
The release is available on:

GitHub release
v0.1.1

Your semantic-release bot :package::rocket:


What is security vulnerability 47833 about?

Sorry for selecting "Feature Request", this is obviously a "Question" but it also obviously doesnt belong on stackoverflow...
So saftey started reporting that click is vulnerable. As im depending on flask, which requires click <8.0.0, i need to know if its safe for me to ignore this, so i need to know what exactly this is about and/or how to mitigate.

I have no idea what that tool is, but I don't think we have any CVEs and Snyk doesn't show anything either: https://snyk.io/advisor/python/click#security
Considering that the site you screenshotted (https://pyupio.github.io/safety-db/ so others don't have to type the URL manually) doesn't show any details when clicking on <8.0.0 and have a bunch of nonsense for other package names, I wouldn't give it a second thought...

Uh, my apologies, i understood that pyup works in a way that they sync with the vendors.. Nevermind, i will ping them then and see if we somehow can find out what this is about.

it seems that the reported error is legit, see https://github.com/pyupio/safety-db/issues/2354
Is there any plan to fix this in the version 7 ? It would be nice to be forced to migrate to v8. Thanks!

Only the latest feature release receives fixes. Currently, that's the 8.1.x branch.


Error with call_common_event

By replacing the common event call (using the RM command) by its RME counterpart in my system, I have this error.
I'm not sure what's the cause and it's not a problem for me since I can still use the basic RM command, I just wanted to let you know.
Also, common events called with the RME command seem to be called too late? See picture attached

Yeah, that's something we already know.
call_common_event(id) will call the specified common event whenever it will be possible, usually at the end of the event.
In v1.4 there will be a command include_common_event(id) which will do the same as the classic event command : call the common event immediately.

My bad did'nt see the first problem aha.
Can you give us more insight on what triggers the error ?
Usually this means that you have an error in your script call.

There's a lot going on in my system so I'll have to investigate. I have currently no idea on what may trigger the error - but I guess it is related to the difference between call_common_event(id) and the classic event command, as the latter causes no issue.
I'll update you if I somewhat manage to be more specific!

Maybe you are using me. As the event is finished with call_common_event(id), me might be nil at this time.

As there's no news about this. I consider this issue closed.
Feel free to comment if you have any news.


Incremental training

Hello！
I've finished an engine， en->fr.  I try to translate some sentences，and use the “ ./MMT add ”command to modify the translation results. When I translate this sentence again。I did get the result of the modification.
But，I try to translate a similar sentence, or a word，I did not get the ideal result, the translation of some words is still the original translation results。
So I want to know, every time I via “./mmt add“  command to modify results, how to make changes to the engine。 Does the engine go back to train，Or is it just like a dictionary that stores the translation results of this sentence?
Also, if I need to do an incremental update on my previous engine, is there an introduction?
Thanks

Hi @sdlmw ,
the steps you followed are 100% correct. To add new data to the engine you can use:

mmt add to add a single contribution
mmt import to import an entire TMX file

The way however the adaptation works is searching for sentences similar to the one you submitted. So there is the chance that if you ask for a translation of a sentence that is not that similar to the one you have added (but has only very few words in common), MMT won't pick it up for adaptation. This is done to avoid few de-contextualized samples make the engine unstable. You should provide more samples from a context (i.e. a TMX with many samples of the words you are focusing on) to start see a more robust adaptation.
I hope this helps, if you have any other questions don't hesitate to write them here!
Davide

Thank you for your reply, I think I have understood, thank you！@davidecaroselli


Integrated validation into graphalytics-core

Added ability for core to perform automatic validation of output after each benchmark. The validation is performed according to the rules specified in the Graphalytics specifications (i.e., correct handling of floating-point precision, equivalence, etc.). Validation can be enabled using the flag "benchmark.run.validation-requried" and the reference output should be placed in directory "benchmark.run.validation-directory" in files named "[dataset name]-[algorithm name]" (e.g., wiki-Talk-PR).

Can one of the admins verify this patch?

ok to test

Build finished.


Alt interface branch goes to quantization dev main branch

Thanks for contributing to TVM!   Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

Man I am so good at this.  As the only reviewer I approve my pull request.


Add .DS_Store to .gitignore for Mac developers

Macs have a .DS_Store file, this causes git to ignore it, it should never be in a repo.

Probably, it should be placed in ~/.gitconfig, ~/.gitignore.
[core]
    excludesfile = ~/.gitignore


[Feature] Create certificate attach feature for profile page

Create file attach feature for contact form.

Closing this task since it is completed.


💡 [REQUEST] - How to use minicpmv2-6's fewshot and multi-image infer feature with VLLM?

起始日期 | Start Date
No response
实现PR | Implementation PR
No response
相关Issues | Reference Issues
No response
摘要 | Summary
How to use minicpmv2-6's fewshot and multi-image infer feature with VLLM?
基本示例 | Basic Example
How to use minicpmv2-6's fewshot and multi-image infer feature with VLLM?
缺陷 | Drawbacks
How to use minicpmv2-6's fewshot and multi-image infer feature with VLLM?
未解决问题 | Unresolved questions
No response

For multi-image input, try it with:
from transformers import AutoTokenizer
from PIL import Image
from vllm import LLM, SamplingParams

MODEL_NAME = "openbmb/MiniCPM-V-2_6"
# Also available for previous models

image = Image.open("xxx.png").convert("RGB")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
llm = LLM(
    model=MODEL_NAME,
    trust_remote_code=True,
    gpu_memory_utilization=1,
    max_model_len=2048
)

messages = [{
    "role":
    "user",
    "content":
    # Number of images
    "(<image>./</image>)" + \
    "\nWhat is the content of this image?"
}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Single Inference
inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
        # Multi images, the number of images should be equal to that of `(<image>./</image>)`
        # "image": [image, image]
    },
}

# 2.6
stop_tokens = ['<|im_end|>', '<|endoftext|>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

sampling_params = SamplingParams(
    stop_token_ids=stop_token_ids,
    use_beam_search=True,
    temperature=0,
    best_of=3,
    max_tokens=64
)

outputs = llm.generate(inputs, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

For icl, you need to reconstruct the prompt.


composer run timeout

Hi!
There is problem with composer run command. After 5 minutes php server terminated with message:
[Symfony\Component\Process\Exception\ProcessTimedOutException]
The process "php -S localhost:8888 -t web web/index_dev.php"
exceeded the timeout of 300 seconds.

This is because of composer's default timeout (set to 300s by default https://getcomposer.org/doc/03-cli.md#composer-process-timeout).
Is there any way to disable\increase this timeout?

try setting the COMPOSER_PROCESS_TIMEOUT env var on the command line, like so:
COMPOSER_PROCESS_TIMEOUT=600 composer run

of relevance in case you're on Windows: http://stackoverflow.com/questions/24950813/how-to-use-composer-process-timeout

Hey, any way to set up unlimited composer process directly in composer.json ?


[USDU-305][USDU-345] Convert Imported PointInstancer transforms to Unity transform space & Adds Testcase for Importing Instance USD Files and their generated GameObject Axis direction

[USDU-345]
[USDU-305]
Purpose of this PR
Ticket/Jira #:

Unity is of Left handed orientation, while many DCCs are of Right handed orientation
We need a test case to test for correct Z-Axis placement when Importing a instancer USD file
This PR also includes the fix that Jeremy has suggested in [USDU-305]
Testing
Functional Testing status:

I created a simple USD file that generates a model that has 8 unique vertex positions
The test imports this model, and runs through all its vertex points
if the imported mesh's vertex point is not included in the expected vertex points (ones defined in the usd file) it will fail the test
Do note we expect the vertex point to have its Z-Axis value to be flipped to compensate for the Z-Axis difference between Unity and DCCs - What is of 'positive' value on a DCC like Blender will be turned negative in Unity, but this is to import the object as-is from the DCC
Performance Testing status:

N/A
Overall Product Risks

N/A
Complexity:

Min
Halo Effect:

Additional information
Note to reviewers:

Reminder:

[ ] Add entry in CHANGELOG.md (if applicable)

Thanks for the reviews! @vickycl


maximum iteration count reached

I achieved this warning message after 1 particle was left in the Lagrangian integration for many steps. Why does this happen?  How is it handled?

With latest code, this is still an issue for SPNA_Lagrangian_demo.ipynb

That's pretty strange. I think I ran the exact same notebook and didn't get the error. What is the time interval between steps? I think the SPNA one is pretty short.

Yes, I didn't watch it run long enough. I will investigate it this weekend. Thanks for catching that.


Unable to encode Sentry error - {:invalid, {nil, "users"}}

Environment

Elixir version (elixir -v): 1.5.3
Erlang/OTP version (erl): 20.1
Sentry version (mix deps): 6.0.5
Operating system:

Description
Error when sending tuple error
Unable to encode Sentry error - {:invalid, {nil, "users"}}
Expected Behavior
Encode tuple properly and send error to sentry
Actual Behavior
Can't encode tuple error so does not send error to sentry
Steps to Reproduce
Send a tuple as an error

Can you give a short example? A tuple isn't encodable as JSON, but it should be handled in most cases.

Encountering the same issue. It looks like Sentry completely relies on the Jason library for encoding params, which by default can't encode tuples.

Unable to encode Sentry error - %Protocol.UndefinedError{description: "Jason.Encoder protocol must always be explicitly implemented", protocol: Jason.Encoder, value: {:DOWN, #Reference<0.1321247442.30408705.69669>, :process, #PID<0.11679.690>, :normal}}

I would've expected Sentry to automatically take care of this, as it turns out a large number of errors were never sent to sentry and we didn't find out.
For now we are just going to implement the Jason.Encoder protocol manually for tuples.

@sheharyarn thanks for adding more context to this issue 🙂
I understand the expectation that these cases should be handled automatically (and that's always the goal), but unfortunately it is more complicated than implementing the Jason.Encoder protocol within Sentry for other types.  Different users of Sentry may have their own preferred encoding of tuples, structs, or other types, and implementing them within Sentry may conflict with that, or be cause for surprise within someone else's expectations.
There is potential for improvement as far as notifying when Sentry has failed for whichever reason, but I've held off.  The relative risks around ending up in an infinite loop of the failure repeating being the chief reason, but I also have concerns around noise and generating a helpful event that leads to solving the root problem.
For me, it would be helpful to see how Sentry reaches the point of trying to encode an unencodable value.  Is it happening via Sentry.Plug, Sentry.LoggerBackend, event metadata, or something else?

Hey @mitchellhenke. Thanks for the quick reply. We're using all three but most of the time we pass error tuples and other structs as part of the metadata when manually sending an event via Sentry.capture_message/2:
with {:ok, result} <- do_something() do
  result
else
  {:error, error} ->
    Sentry.capture_message("[error] Something failed", %{error: error})
    {:error, error}
end

In cases like that where it's a term of any type, it's difficult for Sentry to accommodate for the reasons mentioned above.  error could be just about anything, e.g. an anonymous function, which can't be encoded.
If the structure of the term isn't important, passing%{error: inspect(error)} is an option, as a binary should always be accepted.

@sheharyarn due to the way protocols get dispatched, I don't think including Jason.Encoder protocol implementations can fit within Sentry.
I unfortunately am a bit stumped on how else to make this more ergonomic on your end, because ideally these cases are more obvious ahead of time or there's a better way to handle it when it does happen.

Hey @mitchellhenke, thanks for the update. For now, we've started calling inspect on the passed attributes in our custom logger/sentry wrapper so sentry captures all events, but lost the ability to expand/collapse maps in the sentry dashboard (which is not that big of an issue).
Another solution we're looking at is recursively walking the error metadata map and converting all present tuples to lists and structs to regular maps before passing it to the capture_message method.

To wrap this up, I think the best solution is to lean on Jason (or whichever JSON lib), and direct users to implement protocols for values that can't be by default or use inspect().
#372 returns more information in the form of why sending an event failed, including JSON encoding failures, to help diagnose these issues.


How to add sitemap.xml

I recently learned about sitemap.xml.  Can zzo theme generate sitemap.xml?

I follow the instruction found at the Hugo forum and it works fine with Zzo. You might also want to manage robots.txt as mentioned in the same article.

I customized a sitemap.xml and the end result looks like  this

Nice. One suggestion. In robots.txt you have:

Disallow: /notes/motto/

However, sitemap.xml contains:

https://www.thisfaner.com/notes/motto/motto-010/

and others. It generates confusion. You should exclude them from your sitemap.xml.

@szpak I want to use the following two methods to filter, but was unsuccessful
{{ range where (where .Site.RegularPages "Section" "!=" "gallery") "Section" "!=" "motto"}}

{{ range where (where .Site.RegularPages "Section" "!=" "gallery") "CurrentSection" "!=" "motto"}}

What's wrong?
notes
|---motto
        |--- _index.md
        |--- motto01.md

@fandean
Have you checked what value (.Site.RegularPages "Section") has?
So,  you want to exclude the motto folder in sitemap.xml files, right?
I think "Section" "!=" "motto"}} was not successful because the section is defined with the root folder.(in our case, the notes is the section name). I don't know why the section name is not changed even though we have _index.md file in motto.
As hugo document says, https://gohugo.io/content-management/sections/

By default, all the first-level directories under content/ form their own sections (root sections).
If you need a specific template for a sub-section, you need to adjust either the type or layout in front matter.

So, How about set type in your notes/motto markdown files and filter it like this
{{ range where (where .Site.RegularPages "Section" "!=" "gallery") ".Type" "!=" "motto"}}

I think you can get an idea

Thank you for your suggestion, I use other ways to filter the motto


Funnel Chart Direction

What problem does this feature solve?
The funnel chart always points downwards if sort is set as "none". I need the funnel to point upwards without sorting.
A "direction" option in series-funnel could have the default direction as "down", which could also be set as "up".
What does the proposed API look like?
{
...
type: "funnel",
sort: "none",
direction: "up",
....
}

I am afraid I do not get your point. Could you show what the effect like in a picture?

Hi,
I have the same "problem" :)
Here is my Funnel chart. I don't want to sort (or, sort on the label names), and I'd like "the pointy tip" to be up rather than down.


[Suggestion] Oredictionifier upgrade

I'd like to see an upgrade for digital miners that allows you to convert all items into one specific type as it mines, rather than having to let it clutter the inventory of the device and change it later, it could fit a maximum of 8-16, each adding 1 more oredictionary filter

This is why the Oredictionificator block is a part of Mekanism. I'd rather leave the functionality the way it is now.


No updates in 2 years? Project is abandoned?

@JoshMarler and/or @nick-thompson if you've abandoned this, might I suggest throwing it into archive mode so people understand (and if an active fork emerges, point to it) that this might be subject to bitrot from lack of maintainance.


OCPBUGS-38183: Update CSO deployment file

What this PR does / why we need it: Updated CSO deployment in HyperShift
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes # OCPBUGS-38183
Checklist

[x] Subject and description added to both, commit and PR.
[x] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/retest-required

/test e2e-aws

/test e2e-aks

/test e2e-aws

/test e2e-kubevirt-aws-ovn-reduced

/retest-required

/retest-required

/test e2e-aws

/jira refresh

/approve
/lgtm

/jira backport release-4.17


x/build: set GOTRACEBACK=2 on all builders?

Some of the bugs are fairly hard to reproduce, and without
the system function stacks, it's difficult to see the problem.
For example, #15145.
I propose that we set GOTRACEBACK=2 on all builders.

Yes, let's just do this.

It's now spelled GOTRACEBACK=system.  (2 still works, but is deprecated.)


Correct broken link to Github sample

Link to sample on GitHub was broken, now corrected.

Hi @LeeDumond, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, DNFBOT;

@LeeDumond, Thanks for signing the contribution license agreement so quickly! Actual humans will now validate the agreement and then evaluate the PR.
Thanks, DNFBOT;

@LeeDumond  as mentioned, can you make this fix at https://github.com/Microsoft/Docs-1?


When querying large amount of data, points returned in more than one series

I'm using Grafana to create a dashboard using influxdb. I have around 300 events per second stored in my influx database and if i use a query in Grafana that includes more than a couple minutes of data from one series, I get the data returned but in multiple series with the same name. This causes Grafana to create multiple lines with the same names and completely throws off the graph. This is the query that causes the issue:
q=SELECT+counter1+FROM+%22clickLogger_clickMetric2%22++where+time+%3E+now()+-+3m+
The response that Grafana is getting is
{
	results: [
		{
			series: [
				{
					name: "clickLogger_clickMetric2",
					columns: [
						"time",
						"counter1"
					],
					values: [...]
				},
				{
					name: "clickLogger_clickMetric2",
					columns: [
						"time",
						"counter1"
					],
					values: [...]
				}
			]
		}
	]
}

If I change the query to
q=SELECT+counter1+FROM+%22clickLogger_clickMetric2%22++where++time+%3E+now()+-+3m+and+time+%3C+now()+-+2m
only one series is returned and everything works fine. It seems like if there are too many points returned in the query, the influx response is splitting them into multiple series - not ideal behavior.

closing in favor of #3242