No report is generated Allure

Hi!!
Installed the plugin "Allure for Bamboo" on the Bamboo server according to the instructions - https://docs.qameta.io/allure/latest/#_bamboo. But when I try to generate a report, I get an error:
> Failed to execute the build 'JST-IS-JOB1-110'
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'io.qameta.allure.bamboo.AllureReportTask': Unsatisfied dependency expressed through constructor argument with index 0 of type [io.qameta.allure.bamboo.AllureSettingsManager]: : No qualifying bean of type [io.qameta.allure.bamboo.AllureSettingsManager] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {}; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [io.qameta.allure.bamboo.AllureSettingsManager] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {}
>         at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray (ConstructorResolver.java:749)
>         at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor (ConstructorResolver.java:185)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor (AbstractAutowireCapableBeanFactory.java:1139)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance (AbstractAutowireCapableBeanFactory.java:1042)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean (AbstractAutowireCapableBeanFactory.java:504)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean (AbstractAutowireCapableBeanFactory.java:476)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean (AbstractAutowireCapableBeanFactory.java:342)
>         at sun.reflect.GeneratedMethodAccessor36.invoke (Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke (Unknown Source)
>         at java.lang.reflect.Method.invoke (Unknown Source)
>         at com.atlassian.plugin.osgi.spring.DefaultSpringContainerAccessor.createBean (DefaultSpringContainerAccessor.java:97)
>         at com.atlassian.plugin.module.ClassPrefixModuleFactory.createModule (ClassPrefixModuleFactory.java:35)
>         at com.atlassian.plugin.module.PrefixDelegatingModuleFactory.createModule (PrefixDelegatingModuleFactory.java:88)
>         at com.atlassian.bamboo.plugin.descriptor.AbstractBambooModuleDescriptor.createModule (AbstractBambooModuleDescriptor.java:30)
>         at com.atlassian.bamboo.plugin.descriptor.AbstractBambooModuleDescriptor.access $ 000 (AbstractBambooModuleDescriptor.java:17)
>         at com.atlassian.bamboo.plugin.descriptor.AbstractBambooModuleDescriptor $ 2.call (AbstractBambooModuleDescriptor.java:104)
>         at com.atlassian.bamboo.plugin.BambooPluginUtils.callUnsafeCode (BambooPluginUtils.java:132)
>         at com.atlassian.bamboo.plugin.descriptor.AbstractBambooModuleDescriptor.getModule (AbstractBambooModuleDescriptor.java:113)
>         at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks (TaskExecutorImpl.java:285)
>         at com.atlassian.bamboo.task.TaskExecutorImpl.execute (TaskExecutorImpl.java:122)
>         at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call (ExecuteBuildTask.java:74)
>         at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build (DefaultBuildAgent.java:215)
>         at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl $ 1.call (BuildAgentControllerImpl.java:139)
>         at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl $ 1.call (BuildAgentControllerImpl.java:130)
>         at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor (CustomVariableContextImpl.java:221)
>         at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild (BuildAgentControllerImpl.java:129)
>         at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent $ 1.run (DefaultBuildAgent.java:137)
>         at com.atlassian.bamboo.utils.BambooRunnables $ 1.run (BambooRunnables.java:51)
>         at com.atlassian.bamboo.security.ImpersonationHelper.runWith (ImpersonationHelper.java:31)
>         at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority (ImpersonationHelper.java:20)
>         at com.atlassian.bamboo.security.ImpersonationHelper $ 1.run (ImpersonationHelper.java:52)
>         at java.lang.Thread.run (Unknown Source)
> Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [io.qameta.allure.bamboo.AllureSettingsManager] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {}
>         at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoSuchBeanDefinitionException (DefaultLis

What is the problem ? Thanks for the help !

@dragichv You must be using the "old" approach of Allure report generation. Can you provide more details on how you set up your job? If you used the "Allure Report" task within the job - this is the old way of configuration. And the documentation does not list it anymore, while the Bamboo plugin itself still contains it though.

Version of the plugin "Allure for Bamboo" - 1.4
Version "allure-commandline.zip" - 2.3.5
The plugin configuration is as follows (attach)
"You must use the" old "approach to generating Allure reports." - what do you mean. Do not get to describe in more detail how to do this?
Thanks for the answer!

@dragichv According to your first screenshot, you're trying to build the report using the Allure Report task in a job, which is broken in the latest Allure Bamboo plugin. The new approach is described in the docs, you don't need to add an additional task in the job, you just need to specify the Allure artifacts to be picked up in the job by specifying the correct pattern like **/allure-results/** and then tick the Enable Allure Building box in the plan configuration on "Miscellaneous" tab. Please refer the new docs to configure it right.

@smecsia - I don't believe the specifying of the Allure artifacts to be picked up is described in the docs (not these ones anyway - https://docs.qameta.io/allure/latest/#_bamboo) - does the configuration of the artifact make a difference?

I also do not quite understand where this is described in the documentation, without using a separate task

@robertgates55 you're right, it is not described at the moment. I've recently sent a PR to fix this.

Great thanks - I can confirm that the following config has resolved the issue for me:

how it looks to me for allure 2.2.1 (Something went wrong with the generation of the Allure report.) Here are some details:
java.lang.RuntimeException: The Allure executable could not be found by the name allure-2.2.1), if I installed version 2.3.5 and specified the path in the Path policy

@dragichv Allure Bamboo is currently using the approach of building the report on Bamboo server, not on the agents. So you don't need to specify the requirements for the agents.
Please refer the docs to specify the Allure executable capability and make sure that the Allure executable is available on your Bamboo server (as specified by the path in the capability). Capability has to be named properly, executable label should contain the version you'd like to use, e.g. "Allure 2.3.5", where "2.3.5" is the version you want to execute.
You can also let the plugin to download the executable automatically for you. To do this, you need to tick the "Download if no executable present' box in the configuration.
My suggestion for your particular case would be to:

Remove all Allure executables from Bamboo server
Remove all Allure capabilities from the Bamboo Server configuration
Reinstall Allure Bamboo plugin
Open the Bamboo Server capabilities list and make sure that it contains the "Allure Report" capability (any version)
Open "Allure Report" admin menu and make sure that "Download if no executable present" box is checked
Open your plan configuration and make sure that you have enabled "Enable Allure Building" checkbox on the "Miscellaneous" tab of your plan
Make sure you've specified the Allure artifacts to be picked up by your job
Run your job and verify that "Allure Report" tab is showing the built report

In case of any problems, refer the Bamboo server logs.

@smecsia откатил плагин до версии 1.3. Для формирования использую отдельную таску через ваш плагин. Сам командлайн закинул на сервер бамбу, создал executable. Сборка артифакта проводится на агентах, затем артифакт расшаривается и во втором stage формируется отчет. Спасибо за помощь !!!


python3Packages.cxxheaderparser: 1.3.3 -> 1.3.4

Diff: https://github.com/robotpy/cxxheaderparser/compare/1.3.3...1.3.4
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 337422 run on x86_64-linux 1

4 packages built:

python311Packages.cxxheaderparser
python311Packages.cxxheaderparser.dist
python312Packages.cxxheaderparser
python312Packages.cxxheaderparser.dist


Fixed prefix script commands with ./scripts

@roryabraham
Details
This PR addresses an issue where the npm install command fails on Windows machines using WSL due to the script path not being recognized correctly. The changes involve prefixing script commands with ./ to ensure compatibility across both macOS and Linux-based systems (including WSL).
Fixed Issues
$
PROPOSAL:
Tests

On a Windows machine using WSL, run npm install in the project directory
Verify that the installation completes without the "sh: 1: scripts/postInstall.sh: not found" error
Verify that the [postInstall](    "postinstall": "scripts/postInstall.sh", ) script and other scripts run correctly
On a macOS machine, run npm install and verify it still works as expected

[x] Verify that no errors appear in the JS console

Offline tests
QA Steps

Clone the repository on a Windows machine with WSL
Run npm install and verify it completes successfully without any "not found" errors for scripts
Repeat the process on a macOS machine to ensure compatibility is maintained
Verify that no errors appear in the JS console

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

I have read the CLA Document and I hereby sign the CLA

Verify signed commits / verifySignedCommits- this is failing, so what do I need to do?
@dangrous
Shall I recommit after adding GPG keys?

yes, you have to set up commit signing and rebase

getting some errors while rebase, quickly solving it

@rory all signed, please verify

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

Reassure Performance Tests is creating a conflict, so this needs to be fixed at my end or what... @roryabraham

nope, that failure appears unrelated in this case. Thanks for the contribution.

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/roryabraham in version: 9.0.36-1 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@roryabraham @dangrous Could you help with QA steps?
Steps doesn't seem to be validated by Applause QA.

no qa needed here @kavimuru, you're all set

🚀 Deployed to production by https://github.com/grgia in version: 9.0.36-2 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


PEBB module ignores duration

The PEBB module is ignoring the duration key, and will only perform a transfer test for as long as it takes for the transfer to finish. This makes the log_interval key useless and doesn't allow the user to determine the sustained data-transfer rates between host and device. Consider this configuration file:
actions:

name: h2d-d2h
device: all
duration: 100000
module: pebb
log_interval: 1000
device_to_host: true
host_to_device: false
block_size: 10240000
.
According to the RVS specification this should result in 100 INFO messages giving the average bandwidth over a 1 second window (the log_interval) for each device being measured. Instead, it only gives one RESULT line per device. The result is required output, but the time series of the test is never executed.

Please try running with -d 3 switch to allow INFO messages to be displayed.

I have used the -d 3 option. Moreover I have attempted to run using variable durations. Do you have a config script that will run with longer duration than a single copy? Maybe this is an issue with parameter passing.

It seems you are missing parallel: true
When parallel is missing it is looking for count . If not found, the default is 1 and test exits quickly.

Just a small clarification: if parallel:false test will run until duration expires of number of test transfers reaches count (whichever comes first).
We may change interpretation of count so that it repeats test (as if there were multiple actions in .conf file)
As per duration I have to double check.


Add BatchIteratorProjector implementation to SysUpdateProjector

Codecov Report

Merging #5056 into batch-iterator will increase coverage by 0.13%.
The diff coverage is 73.33%.

@@                 Coverage Diff                  @@
##             batch-iterator    #5056      +/-   ##
====================================================
+ Coverage             82.29%   82.43%   +0.13%
- Complexity            11018    11039      +21
====================================================
  Files                  1331     1331
  Lines                 45131    45136       +5
  Branches               5659     5659
====================================================
+ Hits                  37142    37208      +66
+ Misses                 5318     5269      -49
+ Partials               2671     2659      -12

Impacted Files
Coverage Δ
Complexity Δ

...crate/operation/projectors/SysUpdateProjector.java
77.77% <73.33%> (-6.84%)
5 <4> (+2)

...crate/executor/transport/TransportShardAction.java
77.5% <0%> (-12.5%)
8% <0%> (-1%)

...src/main/java/io/crate/jobs/NestedLoopContext.java
56.52% <0%> (-8.7%)
6% <0%> (-1%)

...te/action/job/BatchConsumerRowReceiverAdapter.java
68.75% <0%> (-6.25%)
6% <0%> (-1%)

...nce/sys/check/node/DiskWatermarkNodesSysCheck.java
66.66% <0%> (-3.71%)
7% <0%> (-1%)

.../planner/projection/builder/ProjectionBuilder.java
84.61% <0%> (-3.08%)
19% <0%> (-1%)

...executor/transport/TransportShardDeleteAction.java
66.03% <0%> (-1.89%)
10% <0%> (ø)

...ql/src/main/java/io/crate/metadata/TableIdent.java
89.28% <0%> (-1.79%)
28% <0%> (-1%)

...crate/analyze/relations/AbstractTableRelation.java
88.46% <0%> (-1.29%)
32% <0%> (ø)

sql/src/main/java/io/crate/metadata/Schemas.java
89.21% <0%> (-0.99%)
38% <0%> (-1%)

... and 35 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f74c72...c76731a. Read the comment docs.


Add Tibshelf Community School

Official domain:
https://www.tibshelf.derbyshire.sch.uk/
Relevant IT course lasting longer than 1 year:
https://www.tibshelf.derbyshire.sch.uk/page/?title=Computer+Science&pid=78
Page with recognition of .sch.uk domain name in email addresses:
https://www.tibshelf.derbyshire.sch.uk/page/?title=How+to+get+in+touch&pid=99
Student email login portal:
https://outlook.com/students.tibshelf.derbyshire.sch.uk

@djam90 First stage of verification passed, official domain is tibshelf.derbyshire.sch.uk as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@djam90 Thank you for your patience while waiting for a review and processing your request. Pull request merged.


Transformation limits

Find a easy way to limit transformations. For example, without a scale limit, it is so easy for user to get lost.

Consider also a need for Grid to be enabled only within limits.

Related to issue https://github.com/taataa/tapspace/issues/28 where matrices turn singular because there is no limits.


Add django-compat as a dependency

Hi,
I mentioned this briefly in #231, and thought I'd raise it one more time in case it's helpful:
Would it make sense to specify that django-compat as a required dependency in setup.py, so it's installed with django-hijack?

Without the entry in setup.py, users have to find and install the package when trying to follow the installation docs 🙂

Hi @sondrelg,
Thanks for reaching out. You make a valid point, however, we dropped the compat dependency in v4, which is close to its final release. Therefore, it is correct, that you don't find the dependency in the main branch.
Best,
Joe


[BUG] Winget installing/upgrading ignores different architectures

Please confirm these before moving forward

[X] I have searched for my issue and have not found a work-in-progress/duplicate/resolved issue.
[X] I have tested that this issue has not been fixed in the latest (beta or stable) release.
[X] I have checked the FAQ section for solutions
[X] This issue is about a bug (if it is not, please use the correct template)

Describe your issue
The Microsoft.DotNet.DesktopRuntime packages necessitate installing both the x86 and x64 version, since I have applications for both architectures on my computer. Winget reports these by appending (x86) to package names. However, Winget doesn't realize this and tries to upgrade without specifying an architecture (i.e. with the default architecture, which is x64).
Steps to reproduce the issue

Make sure you have Winget install x64 packages by default
Try to install/upgrade the x86 version of Microsoft Windows Desktop Runtime

WingetUI Log
Welcome to WingetUI Version 3.0.0
               Version Code 3
C:\Users\aaron\AppData\Local\WingetUI\CachedLangFiles\lang_en.json
Loaded language locale: en
Api auth token: gt2apwnzcu0a68sv6i18a3gptxl2vi9ewgaerp0mxhtxxbh0xbgrqy8wi3puwdv7
Starting update check
Api running on http://localhost:7058
Lang files were updated successfully
WingetUI is up to date
Downloaded icons and screenshots successfully!
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: .NET Tool
█ Enabled: True
█ Found: True
█ Fancye exe name: dotnet tool
█ Executable path: C:\Program Files\dotnet\dotnet.exe
█ Call arguments: tool
█ Version:
█   8.0.200
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: Winget
█ Enabled: True
█ Found: True
█ Fancye exe name: winget.exe
█ Executable path: C:\Users\aaron\AppData\Local\Microsoft\WindowsApps\winget.exe
█ Call arguments:
█ Version:
█   v1.8.532-preview
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: Pip
█ Enabled: True
█ Found: True
█ Fancye exe name: pip
█ Executable path: C:\Users\aaron\scoop\apps\python\current\python.exe
█ Call arguments:  -m pip
█ Version:
█   pip 24.0 from C:\Users\aaron\scoop\apps\python\current\Lib\site-packages\pip (python 3.12)
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: Npm
█ Enabled: False
█ THE MANAGER IS DISABLED
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: Chocolatey
█ Enabled: False
█ THE MANAGER IS DISABLED
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: Scoop
█ Enabled: True
█ Found: True
█ Fancye exe name: scoop
█ Executable path: C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe
█ Call arguments: -NoProfile -ExecutionPolicy Bypass -Command scoop
█ Version:
█   Current Scoop version:
█   v0.3.1 - Released at 2022-11-15
█
█   'extras' bucket:
█   34255a59f psiphon3@181: Fix hash (Closes #12947)
█
█   'main' bucket:
█   33536582e symfony-cli: Update to version 5.8.12
█
█   'nonportable' bucket:
█   a04f561 10x-np: Update to version 1.0.134
█
█   'versions' bucket:
█   f89d9a454 zig-dev: Update to version 0.12.0-dev.3283
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▀▀▀▀▀▀▀▀▀▀▀▀▀ MANAGER LOADED ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ Name: PowerShell
█ Enabled: True
█ Found: True
█ Fancye exe name: powershell.exe
█ Executable path: C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe
█ Call arguments: -NoProfile -Command
█ Version:
█   Name                           Value
█   ----                           -----
█   PSVersion                      5.1.26040.1000
█   PSEdition                      Desktop
█   PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
█   BuildVersion                   10.0.26040.1000
█   CLRVersion                     4.0.30319.42000
█   WSManStackVersion              3.0
█   PSRemotingProtocolVersion      2.3
█   SerializationVersion           <IP_ADDRESS>
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
LoadComponentsAsync finished executing. All managers loaded. Proceeding to interface.
Unknown source!
Unknown source!
Upgradable scoop package not listed on installed packages - id=gpg4win
Upgradable scoop package not listed on installed packages - id=inkscape
Caching admin rights for process id 3156
Starting check for updates wait interval with waitTime=43200

Package Managers Logs
Starting package update operation for package id=Microsoft.DotNet.DesktopRuntime.7with Manager name=Winget
Given installation options are <InstallationOptions: SkipHashCheck=False;InteractiveInstallation=False;RunAsAdministrator=False;Version=;Architecture=;InstallationScope=;InstallationScope=;CustomParameters=;RemoveDataOnUninstall=False>
Process Executable     : C:\Program Files\WingetUI\Assets\Utilities\gsudo.exe
Process Call Arguments : "C:\Users\aaron\AppData\Local\Microsoft\WindowsApps\winget.exe" update --id Microsoft.DotNet.DesktopRuntime.7 --exact --accept-source-agreements --silent --disable-interactivity --accept-package-agreements --force --include-unknown
Working Directory      : C:\Users\aaron
    |    /
    | No installed package found matching input criteria.
Process Exit Code      : -1978335212
Process End Time       : 2024/3/14 15:10:54

Relevant information
No response
Screenshots and videos
No response

Please note that you can set with which architecture a package is installed and updated from the "Installation Options" for that package. However, I will try to default an autodetect behaviour for winget

Hello,
I have been looking to this issue and it seems that the packages Microsoft.DotNetRuntime{6,7,8} do not have architechture-specific versions. Therefore, this issue cannot be fixed from my side.
There are packages who are listed as Publisher.PackageName.x86 and Publisher.PackageName.x64, and then WinGet properly detects the architecture. However, this does not seem to be the case

Could you close the issue as not planned instead of completed? Thanks!


AVRO-3602 Support Map(with non-String keys) and Set in ReflectDatumReader

Support Map(with non-String keys) and Set in ReflectDatumReader
Jira
My PR addresses the following AVRO-3602 issues
Tests
My PR adds the following unit tests:
TestReflectDatumReader.testRead_PojoWithSet
TestReflectDatumReader.testRead_PojoWithMap

@thiru-mg You should also close the respective JIRA after merging PRs.
Since the PR does not introduce backward incompatible changes it should probably be ported back to branch-1.11 too ?!

@martin-g @thiru-mg I created backport MR for 1.11 branch

@yurybubnov A separate MR is not really needed. We could easily cherry-pick the squashed commit from master/1.12.0.
Let me do it!


Removed an explicit __exact lookup.

refs #23940

Hah, that should have been the first thing I tried! Nice work

Should we merge this? Do we need a test? Is there additional work to do to fix the ticket (since "refs" was used in the commit message)?

From @jarshwah investigation it seems to fix the reported issue.
Maybe we could track the whole field/lookup/transform collision checks in another ticket and add a test to make sure the reported case works correctly.

Sounds good. Feel free to backport this to 1.8 as well.

Updated the patch with a test, should be good for a final review.


cudatoolkit 11.1 doesn't exist?

On running
conda install pytorch==1.10 torchvision torchaudio cudatoolkit==11.1 -c pytorch -c conda-forge
I get the following error
PackagesNotFoundError: The following packages are not available from current channels:
  - cudatoolkit==11.1
In the instructions, it says "Finding the above took a lot of trial and error. The difficulty was finding a pytorch and cudatoolkit combination which works with pytorch3d (required later). The above worked for me."
Looking it up, I can't even find a cudatoolkit 11.1.  There's an 11.3 and an 11.7, neither of which seem to end up working in the notebook.  Any ideas?

I may have just found a solution.  Leaving this up for anyone else who struggles with this problem.  On this page it lists cudatoolkit-11.1.1 which is installing for me now.  Command:
conda install pytorch==1.10 torchvision torchaudio cudatoolkit==11.1.1 -c pytorch -c conda-forge

I had an issue with that command as well, this worked for me:
conda install pytorch==1.10 torchvision torchaudio cudatoolkit=11.1 -c pytorch -c  conda-forge
According to the pytorch install command on the website, the cudatoolkit version should be a single = (instead of double) and that worked for me. Same result, it also installed 11.1.1

This worked for me. Thanks @thegrandmasterflash
conda install pytorch==1.10 torchvision torchaudio cudatoolkit==11.1.1 -c pytorch -c conda-forge

Thanks all

Not working any more, sadly. Looks like cudatoolkit has been taken down.

Not working anymore. Both -
conda install pytorch==1.10 torchvision torchaudio cudatoolkit=11.1.1 -c pytorch -c conda-forg  and
conda install pytorch==1.10 torchvision torchaudio cudatoolkit=11.1.1 -c pytorch -c conda-forg

For me, it's also not working with other cudatoolkit versions and it's actually a bit weird:
As one can simply look up, the package does exist, e.g., here https://anaconda.org/nvidia/cudatoolkit or here https://anaconda.org/conda-forge/cudatoolkit
However, I get an error that it cannot be found (no matter what version) when I run conda install. Also when searching it via  conda search
❯ conda search -f cudatoolkit -c nvidia   --override-channels
Loading channels: done
No match found for: cudatoolkit. Search: *cudatoolkit*

PackagesNotFoundError: The following packages are not available from current channels:

  - cudatoolkit

Current channels:

  - https://conda.anaconda.org/nvidia/osx-64
  - https://conda.anaconda.org/nvidia/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

or it only finds older dev-versions
❯ conda search -f cudatoolkit -c conda-forge   --override-channels
Loading channels: done
No match found for: cudatoolkit. Search: *cudatoolkit*
# Name                       Version           Build  Channel
cudatoolkit-dev                  9.2          py36_2  conda-forge
cudatoolkit-dev                  9.2          py37_0  conda-forge
cudatoolkit-dev                 10.0               1  conda-forge
cudatoolkit-dev                 10.0               2  conda-forge
cudatoolkit-dev                 10.0          py36_0  conda-forge
cudatoolkit-dev             10.1.243      h0b31af3_1  conda-forge
cudatoolkit-dev             10.1.243      h0b31af3_2  conda-forge
cudatoolkit-dev             10.1.243      h0b31af3_3  conda-forge
cudatoolkit-dev             10.1.243      h1de35cc_0  conda-forge

Has anyone an idea why this happens?

I can't help with that, but I can say that if you're trying to get DiscoDiffusion running locally, the easiest way is to install "Visions Of Chaos".  It's a free windows binary that allows an easy entry with GUI to get all the different diffusion tools working easily.

conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -c conda-forge this was worked for me but installing 11.1.1 instead of 11.1? do you guys know why is that


[BUG] Some of the docs is out-of-date

Describe the bug
e.g.:
Each volume has a dedicated controller, which is called the Longhorn Engine and runs as a container.

https://longhorn.io/docs/1.0.2/concepts/#11-the-longhorn-manager-and-the-longhorn-engine

Fixed.


Using colmap sfm

Thanks for your amazing work and sharing of the project!  I use data captured by phone for reconstruction,

the camera poses were estimated by colmap, but the reconstruction results were not good

I thinked the results is rendered by incorrect camera poses, so I use the hloc for sfm, and write a script to convert the hloc-sfm output files to cams_meta.npy. But the  reconstruction results including using the ngp_fox dataset were still not good. Can you help me analyze the problem?

Since the incorrect camera poses. The problem has been solved.


[FEATURE] Hybrid request does not return inner_hits for nested objects.

Is your feature request related to a problem?
Yes, I'm experiencing a problem when I use the hybrid search plugin in OpenSearch v2.11.0. Specifically, when I include the "inner_hits" parameter in my query for nested objects, I do not receive any inner hits in the response. This is causing frustration as my system requires this level of detail for optimal operation.
What solution would you like?
I would like the hybrid search plugin to be updated to include the functionality to correctly return inner hits from nested queries. Ideally, this would function seamlessly as it does in standard OpenSearch queries. This improvement would allow me and other users to fully utilize the power of the hybrid search plugin.

Can you please share more details for us to understand your request better: index mapping, query example, expected response?

I removed vectors values, do you need them also?
Index mapping :
{
  "mappings": {
    "properties": {
      "chunks": {
        "type": "nested",
        "properties": {
          "embedding": {
            "type": "knn_vector",
            "dimension": 1536,
            "method": {
              "name": "hnsw",
              "space_type": "cosinesimil",
              "engine": "nmslib",
              "parameters": {
                "ef_construction": 128,
                "m": 24
              }
            }
          },
          "payload": {
            "index": "true",
            "norms": "false",
            "store": "true",
            "type": "text"
          },
          "length": {
            "type": "integer"
          },
          "id": {
            "type": "text"
          }
        }
      }
    }
  },
  "settings": {
    "index": {
      "knn": true,
      "number_of_shards": 5,
      "number_of_replicas": 1
    }
  }
}

Document example:
{
    "chunks": [
        {
            "id": 1,
            "length": 173,
            "payload": "Text 1 example",
            "tokens": 256,
            "embedding": [...]
        },
        {
            "id": 2,
            "length": 173,
            "payload": "Text 2 example",
            "tokens": 256,
            "embedding": [...]
        },
        {
            "id": 3,
            "length": 173,
            "payload": "Text 3 example",
            "tokens": 256,
            "embedding": [...]
        }
    ]
}

request:
{
    "_source": false,
    "query": {
        "hybrid": {
            "queries": [
                {
                    "nested": {
                        "path": "chunks",
                        "query": {
                            "knn": {
                                "chunks.embedding": {
                                    "vector": [...],
                                    "k": 10
                                }
                            }
                        },
                        "inner_hits": {
                            "size": 10,
                            "_source": {
                                "includes": [
                                    "chunks.payload",
                                    "chunks.id"
                                ]
                            }
                        }
                    }
                },
                {
                    "bool": {
                        "must": [
                            {
                                "nested": {
                                    "path": "chunks",
                                    "query": {
                                        "simple_query_string": {
                                            "query": "*",
                                            "fields": [
                                                "chunks.payload"
                                            ],
                                            "default_operator": "and"
                                        }
                                    }
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}

response:
{
    "took": 18,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "index_name",
                "_id": "doc_id_1",
                "_score": 1.0,
            }
        ]
    }
}

expected response:
{
    "took": 17,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1,
        "hits": [
            {
                "_index": "index_name",
                "_id": "doc_id_1",
                "_score": 1,
                "inner_hits": {
                    "hsr_chunks": {
                        "hits": {
                            "total": {
                                "value": 3,
                                "relation": "eq"
                            },
                            "max_score": 0.7954481,
                            "hits": [
                                {
                                    "_index": "index_name",
                                    "_id": "doc_id_1",
                                    "_nested": {
                                        "field": "hsr_chunks",
                                        "offset": 0
                                    },
                                    "_score": 0.7954481,
                                    "_source": {
                                        "payload": "Text 1 example",
                                        "id": 1
                                    }
                                },
                                {
                                    "_index": "index_name",
                                    "_id": "doc_id_1",
                                    "_nested": {
                                        "field": "hsr_chunks",
                                        "offset":1
                                    },
                                    "_score": 0.7949572,
                                    "_source": {
                                        "payload": "Text 2 example",
                                        "id": 2
                                    }
                                },
                                {
                                    "_index": "index_name",
                                    "_id": "doc_id_1",
                                    "_nested": {
                                        "field": "chunks",
                                        "offset": 2
                                    },
                                    "_score": 0.75225127,
                                    "_source": {
                                        "payload": "Text 3 example",
                                        "id": 3
                                    }
                                }
                            ]
                        }
                    }
                }
            }
        ]
    }
}

This issue is also biting me.
We have nested property which stores attachments on a document. We use the inner_hits today to reflect when the query was found in one of the attachments. However, in trying to implement a hybrid search which combines a simple_query_string with a neural_sparse search, we're losing the inner_hits, which means we cannot identify when the search came from our nested search.

@dswitzer can we try 2 text queries with hybrid search and see if inner hits are coming or not. Reason I am asking this is for vector search there are improvements which are doing in 2.12 and 2.13 version relates to nested fields with vectors.
Ref: https://github.com/opensearch-project/k-NN/issues/1447
Ref: https://github.com/opensearch-project/k-NN/issues/1065

@navneet1v The issue persist even if it contains query with non-vector fields only.
The issue with hybrid search with inner_hits is that, the innerHit result does not get generated at all.

@heemin32 thanks for confirming it. Can you please share the example on this issue on what and how you tested it.

Create Index
PUT /my-hybrid
{
  "mappings": {
    "properties": {
      "chunks": {
        "type": "nested",
        "properties": {
          "embedding": {
            "type": "knn_vector",
            "dimension": 3,
            "method": {
              "name": "hnsw",
              "space_type": "cosinesimil",
              "engine": "nmslib",
              "parameters": {
                "ef_construction": 128,
                "m": 24
              }
            }
          },
          "payload": {
            "index": "true",
            "norms": "false",
            "store": "true",
            "type": "text"
          },
          "length": {
            "type": "integer"
          },
          "id": {
            "type": "text"
          }
        }
      }
    }
  },
  "settings": {
    "index": {
      "knn": true
    }
  }
}

Add doc
PUT /_bulk?refresh=true
{ "index": { "_index": "my-hybrid", "_id": "1" } }
{ "chunks": [{"id": 1, "length": 173, "payload": "Text 1 example", "tokens": 256, "embedding": [1, 1, 1]}, {"id": 2, "length": 173, "payload": "Text 2 example", "tokens": 256, "embedding": [2, 2, 2]},{"id": 3,"length": 173,"payload": "Text 3 example","tokens": 256,"embedding": [3, 3, 3]}]}

Query
GET /my-hybrid/_search
{
	"_source": false,
	"query": {
		"hybrid": {
			"queries": [
				{
					"nested": {
						"path": "chunks",
						"query": {
							"simple_query_string": {
								"query": "*",
								"fields": [
									"chunks.payload"
								],
								"default_operator": "and"
							}
						},
						"inner_hits": {
              "size": 10,
              "_source": {
                "includes": [
                  "chunks.payload",
                  "chunks.id"
                ]
              }
            }
          }
				}
			]
		}
	}
}

@Kovsonq @dswitzer what is the main use case for those inner hits returned in the result? How critical is the score information for that use case?
I spent some time checking what can be done for inner hits and our limitations. We can include an inner hits section in the response, similar to what's done for other queries in OpenSearch. The only limitation I'm seeing is with the scores. Inner hits have their own logic for retrieving scores; at a high level, they run a light version of the search again during the Fetch phase. At this point, the score normalization process for the hybrid query has been completed, and scores are updated in the query result section of the response. Scores added for inner hits will not be normalized but will be in raw form and scale. This means that, depending on the query, scores can be unbounded and will not correlate with the main hits in the query results (as those are normalized).

@martin-gaievski,
My primary use case is to just be able to highlight the matching terms. The score of the inner hits does not matter much to me, because I'm just using it to highlight keyword matches.

@martin-gaievski,
The primary use case for inner_hits in OpenSearch is to retrieve detailed matching information from nested objects within documents. This is particularly useful in scenarios where documents have complex structures with nested fields, and there is a need to understand which specific parts of these documents match the query criteria.
In the context of nested objects, score information for inner hits is important because it allows users to identify the most relevant chunks or sub-documents within a larger document. When a hybrid search is performed, having access to the scores of inner hits enables users to rank and prioritize these nested sections effectively.
Scenario: we need to return the top 20 most relevant nested documents (not parent documents) for the query.

@Kovsonq
I'm still not 100% understand why you need normalized scores in a final list of results. If we enable inner_scores without normalized scores, then relative order of child documents will be present in the final result list. As the inner_hits is passed at the sub_query level those hits for child documents will be local to that sub-query anyway, not global for all hybrid query.
If you need to retrieve information about child documents with normalized scores then I feel those child document should be modeled as top level (-> parent) documents.

After doing deep dive for this request I can conclude that we need more time and some additional mechanisms (most likely include core OpenSearch) to implement this feature correctly.
Simplistic approach where inner hits are given per sub-query doesn't work and may provide false positives. Example scenario:

hybrid query has two sub-queries, one text match, second is neural query. user specify inner hits for match query
one document has low score in match query, say it's in position 12. At the same time same document has much better score in neural query - something like 0.95, position 2.
after doing normalization final position of that document is 3.
inner hits for the document will have information collected for match query
In result user may have false impression that high final position of the document in due to hits in match, but in reality it's neural that contributed the most.
In other words, we need an inner hits concept at the high level hybrid query, not at the level of sub-query

I'm also trying to do the same, it seems also that the normalization isn't being applied correctly for hybrid search on nested fields as well. I've verified for normalizing using all of the values of the nested field, using the highest value of the nested field for each doc, using the sum of the values of the nested field. The normalization just doesn't come out correctly.
For context my use case is to run hybrid search on chunks of documents and ideally I wouldn't need to create a new document in opensearch for every chunk that I want to index.
I believe this is a common use case, it would be super AMAZING if we could get this support!

Is there any blocking issue to support this feature? cc: @martin-gaievski

@yuye-aws yes, there are fundamental blockers for inner hits: the process is split into two parts, first run at the shard level and doesn't have access to normalized scores and combined order of documents, second part is at the fetch phase and it's also at the shard level. Second item has additional problem of query and fetch phases not communicating with each other directly.

@martin-gaievski Thanks for your prompt reply. Although I do not have much context for the inner hits and hybrid query, it really seems to be a tricky problem to resolve. Is there any existing info for me to get more knowledge? (Like PR https://github.com/opensearch-project/neural-search/pull/776)

Still really excited to have this support! We're waiting for this to switch over to OpenSearch, it has everything else we need, but to hack around this to create our own implementation using just the top level docs is too messy.


[DO NOT MANUAL MERGE] Release Qualification

Trigger release qualification jobs
none

@silverdev: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth-default

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-auth

prow/daily-e2e-rbac-no_auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-no_auth-default

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-auth

prow/daily-e2e-rbac-no_auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-no_auth-default

prow/daily-e2e-rbac-no_auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-auth

prow/daily-e2e-rbac-no_auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-no_auth-default

prow/daily-e2e-rbac-no_auth.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-no_auth

prow/daily-e2e-rbac-auth-default.sh
6259eb1f9395829697875972e4edf2401c8314ab
link
/test daily-e2e-rbac-auth-default

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


CC-2 - Okta Login

I have it working, but not fully tested nor is it very pretty code. Needs some FE developer help.
Mostly stolen from Scotty's demo work here: https://codesandbox.io/s/missionhub-okta-test-oknqp?file=/src/App.tsx

@jbirdjavi is the API code already in stage for testing against?

@wrandall22 Yes, but it didn't have the correct secrets settings until just now.

For some reason it's trying to redirect me to the admin console: https://signon-admin.okta.com/admin/app/oidc_client/instance/0oa1oqxpke9BXTbhp0h8#tab-general. Maybe I don't have the right ids?

This is on staging now @jbirdjavi do you know what the QA process is?

I don't really. Brian Funkhouser is product for CC but he's not available much this summer. Maybe Nikki Neary has an idea?
My thoughts on things to QA (feel free to add):

Can log in an existing account
Can create a new account and login
Can still log in via FB if that was already active before creating new account
Logout (logs out from Okta too?)

The only one that doesn't seem to work well (in stage at least) is

Can still log in via FB if that was already active before creating new account

It seems the Facebook application is not set up correctly to handle the stage site.

@wrandall22 I found the settings in the FB app so FB login is now working on staging.


Using pre-run Freesurfer

I recognize that this concept has been addressed in poldracklab/fmriprep#1182 and I have read that thread prior to posting here. I have previously posted here on this topic but I'll refresh it for the current error. I have the most current version of fmriprep (updated today).
I have run Freesurfer (BIDS/Freesurfer v 6.0) using a longitudinal pipeline for 29 participants. The output from that is currently in OUTDIR/freesurfer. I now am trying to fmriprep. I've run on a single subject in order to assess the usability of the prior freesurfer run. The process completed and I received the reports for the subject. However, none of the freesurfer derivatives are present. A careful inspection of the logs reveals that it finds the pre-run Freesurfer directory but doesn't utilize it.
Here is the error message:

RuntimeError: Command:
recon-all -autorecon1 -noskullstrip -openmp 4 -subjid sub-01 -sd /out/freesurfer -notalairach -nonuintensitycor -nonormalization
Standard output:
ERROR: It appears that this subject ID is an existing
base/template from longitudinal processing (-base):
sub-01
If you are trying to re-run a -base template you
need to pass the -base and all -tp flags:
   \' -base <templateid> -tp <tpNid> ... -all \'

   (Instead of -all you can pass other flags, such
   as -autorecon2 -autorecon3 to run only parts.)

Standard error:
Return code: 1

I am attaching the log file with the error. Here's the commands for fmriprep:
docker run -i --rm \
-v $INDIVDIR:/data:ro \
-v $OUTDIR:/out \
-v /data/BL2/BIDS_derivatives/code/license.txt:/opt/freesurfer/license.txt \
poldracklab/fmriprep \
/data /out \
participant \
--longitudinal \
--low-mem \
--n_cpus 1 \
--output-space template T1w fsaverage \
--work-dir /scratch \
--use-aroma \
--participant_label "sub-$subject" \
--write-graph

Any help would be appreciated
crash_log.txt

@pettitta I'm just seeing this right now. I'm curious if this solution works for allowing FMRIPrep/Nipype to identify that Freesurfer has already been run and correctly import the expected folders?

Was this ever addressed/documented? Thanks for your patience.

I'll close this and welcome anyone who reopens it because it wasn't correctly addressed.

This isn't done, as it's about using longitudinal FreeSurfer. #21 should cover that.

Let's then redirect everyone to the longitudinal FS issue you linked.


For AzureOpenAI, may need another depolyment id?

I am using Azure OpenAI, and I have no problem with the Visual chatGPT studio, but when I am trying to use turbo chat, the API returns 404 resource not found. I noticed that I can only have one model for each deployment in the Azure AI Studio (the depolyment ID I put in Visual Studio Options uses TextDavinci003), so may be I need have another deploymentId for GPT_3_5_Turbo? (If so maybe there will have to be two API instances - one for main and one for turbo chat?) Anyway, seems a difference between AzureOpenAI and OpenAI.

Ok, I understand the problem, however I can't test it myself since I don't have access to Azure OpenAI yet.
But from what I've noticed, it really seems to me that a different deployment is needed for Turbo Chat. Or it can be related with the API Version.
Can you perform this test, for example by changing your deployment model to gpt-3.5-turbo or creating a new deployment with this model and testing to see if it works?
@Rabosa616 , because you showed to me how to implement the connection to azure, how have you been using the tool? Did you have a problem similar to @oliu321 ?

@jeffdapaz Ok, I just did some test so here is what I found:

I am not 100% sure whether you will need a different deployment for Turbo Chat, but probably it's the way to go because in the Azure OpenAi, one deployment can only use one model (and it's a bijection so one model can only be used in one deployment - which makes little sense to me but it's what it is)

I saw a model which is called code-davinci-002 in the model list, may be use that for the coding (but keep the gpt-35-turbo for turbo chat?)

I created a deployment just for gpt-35-turbo model, now there is no more 404 but the URL in the calls to Azure OpenAI was wrong so it will still fail: Azure expects: https://[Azure Open Ai Resource Name].openai.azure.com/openai/deployments/gpt35turbofordev/completions?api-version=2022-12-01 , but your extension is using (an extra chat): https://[Azure Open Ai Resource Name].openai.azure.com/openai/deployments/gpt35turbofordev/chat/completions?api-version=2022-12-01

It seems if I change values in the tools/options (for example change the Azure Deployment ID) I will have to restart visual studio for new values to be loaded - seems it was cached somewhere and cached value would be used without a restart of visual studio

Another issue related to this, now I set the deployment gpt35turbofordev (which is my name for gpt-35-turbo model) but I tried to use the chat functionality (not the turbo one) actually the call to the Azure OpenAPI seems to be also working but: if I put a string "test" in the request box, then response will be an ever increasing infinite (seems to be) {"test", "test", "test", ...} and the extension will hang and the whole visual studio would be frozen.

@oliu321 & @jeffdapaz I will check it next week, this week I have no capacity at all, sorry for that.

@oliu321 , follow my answers:
1 - Probably, but it won't be just that. See what I say in "3.
2 - OpenAI discontinued this model, so I recently removed it from the extension options. I believe it will soon be removed from Azure as well.
In my opinion for coding the best template is TextDavinci003. And yes, for chat Turbo use the gpt-3.5-turbo, or GPT-4 if you prefer.
3 - I believe that now the problem is related to what I had said about the API version.
I'm using a library to communicate with OpenAI/Azure, and in its repository I found this issue below. Read the last comment:
https://github.com/OkGoDoIt/OpenAI-API-dotnet/issues/81
Can you confirm wich version your deployment are using? Maybe I need add in the option the deployment specific for the Turbo Chat and another options to set the API version for Azure.
4 - It's because I can't get when the options was changed. It's only will be changed when create a new "chatGPT object" instance. Maybe close the chat window and open again do the trick.
About your test on other tool window, I need to see. When we make a generic request, OpenAI responds with random responses, however with me blocking VS never happened.
@Rabosa616 , no problem. Very thanks!

API version I was using 2022-12-01 which was suggested by Azure (they generates sample code for you) I didn't see a way to specify API version in the Azure AI Studio, but if you want to add new version for turbo chat like 2023-03-15-preview
I will check more and let you know. Thanks.

@oliu321,
If I prepare a version with these changes and share the installer here, can you install and test if our suggestions will solve the problem?
If they solve, I can publish it on the marketplace for everyone, knowing that it is the "final solution" for Azure.

@jeffdapaz
Yes I can and thanks.

@oliu321 ,
ok, I will prepare these changes.

@oliu321 ,
Here you can get the VSIX file with my modifications.
You will see the new options "Azure Turbo Chat Deployment ID" and "Azure Turbo Chat API Version".
The API version option is not required, so you can test with and without setting the version to see how works.
Tell me something later.

@jeffdapaz
Just tested it, I will have to set both "Azure Turbo Chat API Version" to 2023-03-15-preview and "Azure Turbo Chat Deployment ID" to the gpt35 deployment and then it works. Setting "Azure Turbo Chat API Version" to 2022-12-01 or leaving it blank won't work.
Thanks, good enough for me for now.

@oliu321,
Thanks for the feedback!
I will publish on marketplace a new version with these changes.


Build brew docker image for x86 and arm64

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[ ] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

When opening the brew repository with VSCode on am M1 Mac, it prompted me to reopen the window in a devcontainer. I agreed, then that devcontainer failed to startup, saying there was no such image. Attempting to pull manually resulted in this:
❯ docker pull ghcr.io/homebrew/brew:latest
latest: Pulling from homebrew/brew
no matching manifest for linux/arm64/v8 in the manifest list entries

This PR will build the linux/arm64 image in parallel with the amd64 image, allowing arm-based macs (and any other arm64-based system) to use the brew container.

Looks like the https://github.com/docker/setup-qemu-action is not whitelisted. Could someone add that to the repository settings? It is created and published by the official Docker team / namespace.
https://github.com/Homebrew/brew/actions/runs/11167051891

This PR will build the linux/arm64 image in parallel with the amd64 image, allowing arm-based macs (and any other arm64-based system) to use the brew container.

Homebrew does not support linux/arm64 so we shouldn't be building this image either, sorry.

When opening the brew repository with VSCode on am M1 Mac, it prompted me to reopen the window in a devcontainer.

This devcontainer exists purely for GitHub Codespaces. If there's a way to disable this alert: we should do that, instead.


Bug: cannot connect with mullvad

cannot connect with mullvad

Is this urgent?

[ ] Yes
[x] No

What VPN service provider are you using?

[ ] PIA
[x] Mullvad
[ ] Windscribe

What's the version of the program?
See the line at the top of your logs
Running version latest built on 2020-05-03T16:29:27Z (commit 89187b6)

What are you using to run the container?

[ ] Docker run
[x] Docker Compose
[ ] Kubernetes
[ ] Docker stack
[ ] Docker swarm
[ ] Podman
[ ] Other:

Extra information

Logs:
=========================================
============= PIA container =============
========== An exquisite mix of ==========
==== OpenVPN, Unbound, DNS over TLS, ====
===== Shadowsocks, Tinyproxy and Go =====
=========================================
=== Made with ❤  by github.com/qdm12 ====
=========================================
Running version latest built on 2020-05-03T16:29:27Z (commit 89187b6)
📣  New HTTP control server, see https://github.com/qdm12/private-internet-access-docker#http-control-server
🔧  Need help? https://github.com/qdm12/private-internet-access-docker/issues/new
💻  Email<EMAIL_ADDRESS>☕  Slack? Join from the Slack button on Github
💸  Help me? https://github.com/sponsors/qdm12
2020-05-04T08:01:12.600Z	INFO	IPtables version: v1.8.3
2020-05-04T08:01:12.658Z	INFO	TinyProxy version: 1.10.0
2020-05-04T08:01:12.665Z	INFO	ShadowSocks version: 3.3.4
2020-05-04T08:01:12.672Z	INFO	OpenVPN version: 2.4.8
2020-05-04T08:01:12.678Z	INFO	Unbound version: 1.9.6
2020-05-04T08:01:12.679Z	INFO	Settings summary below:
OpenVPN settings:
|--Network protocol: udp
|--Verbosity level: 1
|--Run as root: no
|--Target IP address: <nil>
|--Custom cipher:
|--Custom auth algorithm:
Mullvad settings:
 |--User: [redacted]
 |--Country: germany
 |--City: frankfurt
 |--ISP: 31173
 |--Port:
System settings:
|--User ID: 1000
|--Group ID: 1000
|--Timezone:
|--IP Status filepath: /ip
DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: enabled
 |--Block malicious: enabled
 |--Block surveillance: disabled
 |--Block ads: disabled
 |--Allowed hostnames:
  |--
 |--Private addresses:
  |--<IP_ADDRESS>/8
  |--<IP_ADDRESS>/8
  |--<IP_ADDRESS>/12
  |--<IP_ADDRESS>/16
  |--<IP_ADDRESS>/16
  |--::1/128
  |--fc00::/7
  |--fe80::/10
  |--::ffff:0:0/96
 |--Verbosity level: 1/5
 |--Verbosity details level: 0/4
 |--Validation log level: 0/2
 |--IPv6 resolution: disabled
Firewall settings:
 |--Allowed subnets: <IP_ADDRESS>/24, <IP_ADDRESS>/8
TinyProxy settings: disabled
ShadowSocks settings: disabled
2020-05-04T08:01:12.681Z	INFO	openvpn configurator: checking for device /dev/net/tun
2020-05-04T08:01:12.682Z	INFO	openvpn configurator: writing auth file /etc/openvpn/auth.conf
2020-05-04T08:01:12.682Z	INFO	routing: detecting default network route
2020-05-04T08:01:12.683Z	INFO	routing: default route found: interface eth0, gateway <IP_ADDRESS>, subnet <IP_ADDRESS>/16
2020-05-04T08:01:12.683Z	INFO	firewall configurator: accepting all traffic
2020-05-04T08:01:12.691Z	INFO	Launching standard output merger
2020-05-04T08:01:12.692Z	INFO	routing: adding <IP_ADDRESS>/24 as route via eth0
2020-05-04T08:01:12.695Z	INFO	routing: adding <IP_ADDRESS>/8 as route via eth0
2020-05-04T08:01:12.697Z	INFO	firewall configurator: clearing all rules
2020-05-04T08:01:12.708Z	INFO	firewall configurator: blocking all traffic
2020-05-04T08:01:12.720Z	INFO	firewall configurator: creating general rules
2020-05-04T08:01:12.732Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.736Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.740Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.744Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.748Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.751Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.754Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.758Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.761Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.765Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.768Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.772Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.776Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.779Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.783Z	INFO	firewall configurator: allowing output traffic to VPN server <IP_ADDRESS> through eth0 on port udp 1197
2020-05-04T08:01:12.790Z	INFO	firewall configurator: accepting input and output traffic for <IP_ADDRESS>/16
2020-05-04T08:01:12.796Z	INFO	firewall configurator: accepting input traffic through eth0 from <IP_ADDRESS>/24 to <IP_ADDRESS>/16
2020-05-04T08:01:12.799Z	INFO	firewall configurator: accepting output traffic through eth0 from <IP_ADDRESS>/16 to <IP_ADDRESS>/24
2020-05-04T08:01:12.803Z	INFO	firewall configurator: accepting input traffic through eth0 from <IP_ADDRESS>/8 to <IP_ADDRESS>/16
2020-05-04T08:01:12.806Z	INFO	firewall configurator: accepting output traffic through eth0 from <IP_ADDRESS>/16 to <IP_ADDRESS>/8
2020-05-04T08:01:12.810Z	INFO	openvpn: starting
2020-05-04T08:01:12.810Z	WARN	http server: restartOpenvpn function is not set, waiting...
2020-05-04T08:01:12.810Z	INFO	openvpn configurator: starting openvpn
2020-05-04T08:01:12.812Z	WARN	http server: restartUnbound function is not set, waiting...
2020-05-04T08:01:12.815Z	INFO	openvpn: Mon May  4 08:01:12 2020 OpenVPN 2.4.8 armv7-alpine-linux-musleabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb  7 2020
2020-05-04T08:01:12.815Z	INFO	openvpn: Mon May  4 08:01:12 2020 library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-05-04T08:01:12.821Z	INFO	openvpn: Mon May  4 08:01:12 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:12.821Z	INFO	openvpn: Mon May  4 08:01:12 2020 UDP link local: (not bound)
2020-05-04T08:01:12.822Z	INFO	openvpn: Mon May  4 08:01:12 2020 UDP link remote: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:12.822Z	INFO	openvpn: Mon May  4 08:01:12 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-05-04T08:01:12.924Z	INFO	openvpn: Mon May  4 08:01:12 2020 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1557', remote='link-mtu 1558'
2020-05-04T08:01:12.924Z	INFO	openvpn: Mon May  4 08:01:12 2020 WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'
2020-05-04T08:01:12.924Z	INFO	openvpn: Mon May  4 08:01:12 2020 [de-fra-004.mullvad.net] Peer Connection Initiated with [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:19.011Z	INFO	openvpn: Mon May  4 08:01:19 2020 NOTE: setsockopt TCP_NODELAY=1 failed
2020-05-04T08:01:19.012Z	INFO	openvpn: Mon May  4 08:01:19 2020 TUN/TAP device tun0 opened
2020-05-04T08:01:19.013Z	INFO	openvpn: Mon May  4 08:01:19 2020 /sbin/ip link set dev tun0 up mtu 1500
2020-05-04T08:01:19.014Z	INFO	openvpn: Mon May  4 08:01:19 2020 /sbin/ip addr add dev tun0 <IP_ADDRESS>/16 broadcast <IP_ADDRESS>
2020-05-04T08:01:19.016Z	INFO	openvpn: Mon May  4 08:01:19 2020 /sbin/ip -6 addr add fdda:d0d0:cafe:1197::1006/64 dev tun0
2020-05-04T08:01:19.018Z	INFO	openvpn: Mon May  4 08:01:19 2020 Linux ip -6 addr add failed: external program exited with error status: 2
2020-05-04T08:01:19.018Z	INFO	openvpn: Mon May  4 08:01:19 2020 Exiting due to fatal error
2020-05-04T08:01:19.078Z	ERROR	openvpn: exit status 1
2020-05-04T08:01:19.078Z	INFO	openvpn: starting
2020-05-04T08:01:19.078Z	INFO	openvpn configurator: starting openvpn
2020-05-04T08:01:19.078Z	ERROR	close |0: file already closed
2020-05-04T08:01:19.084Z	INFO	openvpn: Mon May  4 08:01:19 2020 OpenVPN 2.4.8 armv7-alpine-linux-musleabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb  7 2020
2020-05-04T08:01:19.085Z	INFO	openvpn: Mon May  4 08:01:19 2020 library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-05-04T08:01:19.091Z	INFO	openvpn: Mon May  4 08:01:19 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:19.091Z	INFO	openvpn: Mon May  4 08:01:19 2020 UDP link local: (not bound)
2020-05-04T08:01:19.091Z	INFO	openvpn: Mon May  4 08:01:19 2020 UDP link remote: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:19.092Z	INFO	openvpn: Mon May  4 08:01:19 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-05-04T08:01:19.188Z	INFO	openvpn: Mon May  4 08:01:19 2020 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1557', remote='link-mtu 1558'
2020-05-04T08:01:19.189Z	INFO	openvpn: Mon May  4 08:01:19 2020 WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'
2020-05-04T08:01:19.189Z	INFO	openvpn: Mon May  4 08:01:19 2020 [de-fra-005.mullvad.net] Peer Connection Initiated with [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:25.688Z	INFO	openvpn: Mon May  4 08:01:25 2020 NOTE: setsockopt TCP_NODELAY=1 failed
2020-05-04T08:01:25.690Z	INFO	openvpn: Mon May  4 08:01:25 2020 TUN/TAP device tun0 opened
2020-05-04T08:01:25.691Z	INFO	openvpn: Mon May  4 08:01:25 2020 /sbin/ip link set dev tun0 up mtu 1500
2020-05-04T08:01:25.695Z	INFO	openvpn: Mon May  4 08:01:25 2020 /sbin/ip addr add dev tun0 <IP_ADDRESS>/16 broadcast <IP_ADDRESS>
2020-05-04T08:01:25.700Z	INFO	openvpn: Mon May  4 08:01:25 2020 /sbin/ip -6 addr add fdda:d0d0:cafe:1197::1005/64 dev tun0
2020-05-04T08:01:25.705Z	INFO	openvpn: Mon May  4 08:01:25 2020 Linux ip -6 addr add failed: external program exited with error status: 2
2020-05-04T08:01:25.706Z	INFO	openvpn: Mon May  4 08:01:25 2020 Exiting due to fatal error
2020-05-04T08:01:25.768Z	ERROR	openvpn: exit status 1
2020-05-04T08:01:25.768Z	INFO	openvpn: starting
2020-05-04T08:01:25.768Z	ERROR	close |0: file already closed
2020-05-04T08:01:25.769Z	INFO	openvpn configurator: starting openvpn
2020-05-04T08:01:25.781Z	INFO	openvpn: Mon May  4 08:01:25 2020 OpenVPN 2.4.8 armv7-alpine-linux-musleabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb  7 2020
2020-05-04T08:01:25.781Z	INFO	openvpn: Mon May  4 08:01:25 2020 library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-05-04T08:01:25.790Z	INFO	openvpn: Mon May  4 08:01:25 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:25.790Z	INFO	openvpn: Mon May  4 08:01:25 2020 UDP link local: (not bound)
2020-05-04T08:01:25.790Z	INFO	openvpn: Mon May  4 08:01:25 2020 UDP link remote: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:25.790Z	INFO	openvpn: Mon May  4 08:01:25 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-05-04T08:01:25.895Z	INFO	openvpn: Mon May  4 08:01:25 2020 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1557', remote='link-mtu 1558'
2020-05-04T08:01:25.895Z	INFO	openvpn: Mon May  4 08:01:25 2020 WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'
2020-05-04T08:01:25.896Z	INFO	openvpn: Mon May  4 08:01:25 2020 [de-fra-001.mullvad.net] Peer Connection Initiated with [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:27.156Z	INFO	openvpn: Mon May  4 08:01:27 2020 AUTH: Received control message: AUTH_FAILED,[TOO_MANY_CONNECTIONS] This Mullvad account is already used by the maximum number of simultaneous connections
2020-05-04T08:01:27.156Z	INFO	openvpn: Mon May  4 08:01:27 2020 SIGUSR1[soft,auth-failure] received, process restarting
2020-05-04T08:01:37.157Z	INFO	openvpn: Mon May  4 08:01:37 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:37.157Z	INFO	openvpn: Mon May  4 08:01:37 2020 UDP link local: (not bound)
2020-05-04T08:01:37.158Z	INFO	openvpn: Mon May  4 08:01:37 2020 UDP link remote: [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:37.260Z	INFO	openvpn: Mon May  4 08:01:37 2020 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1557', remote='link-mtu 1558'
2020-05-04T08:01:37.261Z	INFO	openvpn: Mon May  4 08:01:37 2020 WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'
2020-05-04T08:01:37.261Z	INFO	openvpn: Mon May  4 08:01:37 2020 [de-fra-001.mullvad.net] Peer Connection Initiated with [AF_INET]<IP_ADDRESS>:1197
2020-05-04T08:01:38.451Z	INFO	openvpn: Mon May  4 08:01:38 2020 AUTH: Received control message: AUTH_FAILED,[TOO_MANY_CONNECTIONS] This Mullvad account is already used by the maximum number of simultaneous connections
2020-05-04T08:01:38.452Z	INFO	openvpn: Mon May  4 08:01:38 2020 SIGUSR1[soft,auth-failure] received, process restarting
...

Configuration file:
version: "3.7"
services:
  pia:
    build: https://github.com/qdm12/private-internet-access-docker.git
    image: qmcgaw/private-internet-access
    container_name: pia
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun
    network_mode: bridge
    init: true
    ports:
#      - 8888:8888/tcp
#      - 8388:8388/tcp
#      - 8388:8388/udp
      - 7878:7878
      - 6789:6789
      - 8989:8989
      - 9191:9191
#    command: --auth-nocache
    environment:
      - VPNSP=mullvad
      - USER=[redacted]
      - PASSWORD=
      - ENCRYPTION=
      - PROTOCOL=udp
      - COUNTRY=Germany
      - CITY=Frankfurt
      - ISP=31173
      - PORT=
#      - NONROOT=no
      - DOT=on
#      - BLOCK_MALICIOUS=on
#      - BLOCK_NSA=off
#      - UNBLOCK=
#      - FIREWALL=on
      - EXTRA_SUBNETS=<IP_ADDRESS>/24,<IP_ADDRESS>/8
#      - TINYPROXY=on
#      - TINYPROXY_LOG=Critical
#      - TINYPROXY_USER=
#      - TINYPROXY_PASSWORD=
#      - SHADOWSOCKS=on
#      - SHADOWSOCKS_LOG=on
#      - SHADOWSOCKS_PASSWORD=
    restart: always

Host OS:
Raspbian buster

Hi thanks for reporting.
The problem comes from IPv6 it seems:
/sbin/ip -6 addr add fdda:d0d0:cafe:1197::1006/64 dev tun0
Linux ip -6 addr add failed: external program exited with error status: 2
Exiting due to fatal error

Do you have IPv6 enabled in your network?
I also added in the latest build a few minutes ago (build):

tun-ipv6 line to the openvpn config which was missing in comparison with the latest mullvad openvpn files
the ip6tables to the docker image, maybe that can allow ip -6 addr add .... to work

Thanks for your response!
Adding the following to docker-compose.yml fixed the issue.
sysctls:
      - net.ipv6.conf.all.disable_ipv6=0

My bad for not checking the faq!

Well thanks for reminding me of it :smile: I completely forgot it myself despite having written it!


bin_age_csv missing in util/init

Hi all,
I'm an R guy. I was trying bucky and bumped into an error early on when running:
./bmodel make_input_graph -d 2020-10-01
I solved it by adding "bin_age_csv" to the bucky/util/__init__.py  file as follows:
# flake8: noqa  # pylint: disable=missing-module-docstring
from .util import TqdmLoggingHandler, _banner, cache_files, dotdict, estimate_IFR, bin_age_csv

Cheers

Thanks for the report,
This should now be fixed in devel by https://github.com/mattkinsey/bucky/commit/5437645d24cd6ee0cb46f9b84ce7943903e949fd


Attach disk with docker images to micro-vms on boot

What does this PR do?
This PR optimizes micro-vm setup time by attaching a second disk to each VM on boot, and mounting it at a specified location.
This disk contains the docker images required for system-probe tests. This approach eliminates the need for sharing the docker images with each micro-vm and performing docker load, thus significantly reducing the tests jobs time.

Motivation

Additional Notes
Corresponding test-infra-definitions PR:
https://github.com/DataDog/test-infra-definitions/pull/387
https://datadoghq.atlassian.net/browse/EBPF-243

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

should we consider using memory optimized machine types?

We are not memory-constrained at the moment.

Looks like this failed?
error: Process exited with status 1: running " sudo virsh vol-create ci-22271412-4670-kernel-matrix-testing-22271412-global-pool /tmp/volume-ci-22271412-4670-kernel-matrix-testing-22271412-global-pool-docker-arm64.qcow2.xml":
    error: Failed to create vol from /tmp/volume-ci-22271412-4670-kernel-matrix-testing-22271412-global-pool-docker-arm64.qcow2.xml
    error: inaccessible backing store volume /home/kernel-version-testing/docker-arm64.qcow2: No such file or directory

Looks like this failed?
error: Process exited with status 1: running " sudo virsh vol-create ci-22271412-4670-kernel-matrix-testing-22271412-global-pool /tmp/volume-ci-22271412-4670-kernel-matrix-testing-22271412-global-pool-docker-arm64.qcow2.xml":
error: Failed to create vol from /tmp/volume-ci-22271412-4670-kernel-matrix-testing-22271412-global-pool-docker-arm64.qcow2.xml
error: inaccessible backing store volume /home/kernel-version-testing/docker-arm64.qcow2: No such file or directory

Yeah the job for arm64 docker disk in ami-builder failed due to docker rate limiting. Will try again today.


[SPARK-32468][SS][TESTS][FOLLOWUP] Provide "default.api.timeout.ms" as well when specifying "request.timeout.ms" on replacing "default.api.timeout.ms"

What changes were proposed in this pull request?
This patch is a follow-up to fill the gap in #29272 which missed to also provide default.api.timeout.ms as well.  #29272 unintentionally changed the behavior on Kafka side timeout which is incompatible with the test timeout. (default.api.timeout.ms gets default value which is 60 seconds, longer than test timeout.)
Why are the changes needed?
We realized the PR for SPARK-32468 (#29272) doesn't work as we expect. See https://github.com/apache/spark/pull/29272#issuecomment-668333483 for more details.
Does this PR introduce any user-facing change?
No, as it only touches the tests.
How was this patch tested?
Will trigger builds from Jenkins or Github Action multiple time and confirm.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31639/
Test PASSed.

Let's make sure Jenkins builds pass multiple times, and Github Action passes.

Test build #127015 has started for PR 29343 at commit 4929439.

retest this, please

cc. @zsxwing as he reviewed #29272
also cc. @HyukjinKwon as he indicated me the Kafka tests become flaky

Test build #127017 has started for PR 29343 at commit 4929439.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31641/
Test PASSed.

retest this, please

Test build #127018 has started for PR 29343 at commit 4929439.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31642/
Test PASSed.

Test build #127015 has finished for PR 29343 at commit 4929439.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127015/
Test PASSed.

retest this, please

Test build #127020 has started for PR 29343 at commit 4929439.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31644/
Test PASSed.

Test build #127017 has finished for PR 29343 at commit 4929439.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127017/
Test PASSed.

retest this, please

Looks good given the description and explanation.

Thanks for working on this @HeartSaVioR.

Test build #127018 has finished for PR 29343 at commit 4929439.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127018/
Test PASSed.

Test build #127020 has finished for PR 29343 at commit 4929439.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127020/
Test PASSed.

retest this, please

Test build #127028 has started for PR 29343 at commit 4929439.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31651/
Test PASSed.

Had a look, the explanation and solution makes sense. Thank you guys helping me out!

I am going to merge this unblock other PRs.

Merged to master.

Thanks for the quick reviews & reaction!

Test build #127028 has finished for PR 29343 at commit 4929439.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127028/
Test PASSed.