OpenShift Virtualization Live Migration Volume Offline?

I am using the CSI driver 2.4.1 with OpenShift 4.12.37, OpenShift Virtualization 4.12.10, and NimbleOS <IP_ADDRESS>0. I have configured the live migration settings and am using the ReadWriteMany access mode for Block volumes. The setup appears to work correctly, however, after a live migration, the CSI appears to make my volume Offline and the VM moves to the Paused state. When I manually set the volume Online, the VM resumes successfully.
Here's my steps:

Create a VM with a ReadWriteMany PVC. VM starts successfully.
Perform a live migration
Observe the volume is set to Offline and the VM is paused

Any ideas on how to prevent the volume from being set offline?

Could you elaborate on how this VM was created? Have you considered these steps when setting up your OS images?

Yes, I have completed the steps you mentioned. That being said, my VM disk was NOT created from the openshift-virtualization-os-images. It's an export of a VM we had setup in RedHat Virtualization. My steps to get here are a bit convoluted:

Export disk from RHV
Import disk as a PVC ("pvc1") to OpenShift. Note that this was done with the 2.3.0 of the CSI driver and used the RWO access mode.
Upgrade to CSI driver 2.4.1.
Clone pvc1 to a new PVC ("pvc2") with the RWX access mode.
Create a new VM using pvc2.

This won't work as we don't support RWO to RWX transformation. We're sort of treating this as a bug at the moment and it will hopefully be fixed in the next version a few months out.
You can however fix this if you're handy with REST APIs. You need to set multi_initiator: true on the backend volume. I believe there's a Nimble CLI way to do this as well but I'm not sure.

Could I export my RWO volume and re-import it as RWX?

Once RWO, RWX is a no go unfortunately. The same bug applies.

I just updated to CSI driver 2.4.2 and have the same problem.
I've figured out how to set multi_initiator, however, it appears that this parameter only applies to iSCSI volumes. I'm using FC and getting this error when setting the flag:
curl -k -H 'X-Auth-Token: REDACTED' https://my-nimble-host:5392/v1/volumes/my-vol-id -d '{"data": {"multi_initiator": true}}' -X PUT
{"messages":[{"code":"SM_http_bad_request","severity":"error","text":"The request could not be understood by the server."},{"code":"SM_unexpected_arg","severity":"error","arguments":{"arg":"multi_initiator"},"text":"Unexpected argument 'multi_initiator'."}]}

I did notice that volumes I manually create on the Nimble have multi_initiator set to true.
I tried to import the disk to a new PVC w/ RWX mode, but multi_initiator was still false on the newly-created Nimble volume too.
virtctl image-upload pvc rwx-test-2 --size=64Gi --image-path=rhel8-template.img.gz --block-volume --storage-class=nimble-san-sc --access-mode ReadWriteMany

It looks like https://github.com/hpe-storage/csi-driver/pull/40/files should create new volumes w/ multi_initiator set to true.

This is very strange. I copied and pasted your curl command into my environment and I can flick multi_initiator back and forth true/false no problem on an FC array with a blank unattached volume. Does your volume have any connections to it? I added a dummy initiator to my volume and that didn't matter.

There are no connections on my volume. I get the same error whether or not the volume is online. I can toggle the online state using curl, so it's not an obvious dumb error on my part.
What do you mean by "blank unattached volume"? I have a PVC (rwx) bound to the PV, however, the VM is turned off so the PVC isn't in active use.
I've got a Nimble with NimbleOS <IP_ADDRESS>0-1048557-opt in case that matters.

I updated my array. No change. Can we dig out some logs what's actually happening on your cluster?
The CSP logs should reveal some clues, oc logs -nhpe-storage deploy/nimble-csp.

Also now confirmed on OCP 4.14 that the following combination indeed set multi_initiator: true on the volume. Using CSI Operator v2.4.2.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
  name: hpe-standard-fc
parameters:
  csi.storage.k8s.io/controller-expand-secret-name: hpe-backend-fc
  csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage
  csi.storage.k8s.io/controller-publish-secret-name: hpe-backend-fc
  csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage
  csi.storage.k8s.io/fstype: xfs
  csi.storage.k8s.io/node-publish-secret-name: hpe-backend-fc
  csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage
  csi.storage.k8s.io/node-stage-secret-name: hpe-backend-fc
  csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage
  csi.storage.k8s.io/provisioner-secret-name: hpe-backend-fc
  csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage
  description: Volume created by the HPE CSI Driver for Kubernetes
  destroyOnDelete: "true"
  accessProtocol: fc
provisioner: csi.hpe.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-first-fc-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 32Gi
  storageClassName: hpe-standard-fc
  volumeMode: Block

I updated to OpenShift 4.14; same behavior.
I'm starting to think this is a Nimble problem. I created a brand-new volume using the Nimble UI, and I can't seem to get multi_initiator set to true. I've tried various data protection options and access options, but nothing seems to make multi_initiator set to true.

Yeah, I think so too. Nimble support should be able to add some clarity here.

I created a ticket, but think I beat them to the punch :)
Turns out iSCSI has to be enabled, even for FC-only configurations. Once I ran group --edit --iscsi_enabled yes on the Nimble, I can confirm that new PVCs have multi_initiator set correctly. I'm going to assume that a software update at some point turned iSCSI off since my very old volumes are multi-initiator.
A quick test showed that live migration is working now. Thanks for the help!

Thanks for confirming. What an obscure finding. We need to document this.


Add GET listings api call

Coverage increased (+9.5%) to 58.613% when pulling af568294e077125d9075b9d5221b851409777a9b on cpacia:master into c8014ce771a521d017facc7ecb15c61894fcac0b on OpenBazaar:master.

@cpacia this looks good to me.


RHELSQ-4656: Conditional skip for container embedding test

This pull request includes:

[ ] adequate testing for the new functionality or fixed issue
[ ] adequate documentation informing people about the change such as

[ ] submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

A small head's up: I made a change to the same test script (container embedding) in https://github.com/osbuild/osbuild-composer/pull/3398, so if that one gets merged first, we'll need to resolve the conflict here.  Happy to add the same condition if this PR gets merged ahead of it.

I can see the packer job failing, I guess it needs https://github.com/osbuild/osbuild-composer/pull/3400 ?

The packer job failure is unrelated, let's merge this.

The packer job failure is unrelated, let's merge this.

Thanks. I was waiting for the nightly CI job to complete before we can merge. https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/843026534


⚠ ClamAV has degraded performance

In 35e7248, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 66 ms

Resolved: ClamAV performance has improved in 1c1c287.


$ in ID or other Attributes causes problems

id="X$PageBody$MainContent$ctl00$UserName"
I believe the $ in the ID or other Attrib causes problems
Tried with both the getfromid and QuerySelector and QueryAll with just basic //input with no $ in query and it wont return the element
Oh and in console removed $ from ID and it works fine, tested.

Hmm, that's odd, the query string is just passed through, but will have a look

Hmm, that's odd, the query string is just passed through, but will have a look

By the way thank you for doing this project, esp this early on in the game, it will help so many people, your work is invaluable and I sincerely appreciate it.

Not seeing that here.
Evaluate (XPath) will work, but QuerySelector won't $ is used for EndsWith
<input id="$test">
Works
document.evaluate("//input[@id='$test']",document).iterateNext()
Doesn't
document.querySelector("input#$test")
'input#$test' is not a valid selector
You can see that in DevTools console

thank you for doing this project

No problem at all, it's been good fun


chore: bump public call depth

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#5845  👈
#5839
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @fcarreiro and the rest of your teammates on  Graphite


Text areas should use native mouse cursor

VSCode Version: 1.51.1
OS Version: macOS Big Sur

Steps to Reproduce:

Hover over any text area/text field and see the following:

This is because of: https://github.com/microsoft/vscode/blob/f74e473238aca7b79c08be761d99a0232838ca4c/src/vs/base/browser/ui/mouseCursor/mouseCursor.css#L10-L14
I'm not sure why a custom bitmap cursor is used on macOS, but IMO is a downgrade over the native cursor, and the blurriness is more pronounced as I use a larger cursor size.
See the improved version after just using cursor: text in custom CSS:

In the fixed version, both contrast and clarity are improved since it is not a bitmap and is black with a white border around it

Does this issue occur when all extensions are disabled?: Yes

This is always coming back since years.

when we used the default OS cursor, people complained that they cannot see it.
so we added our custom cursor
but then macOS Catalina shipped with support for dark themes, which improved their native cursor on dark background, so we removed our custom cursor
then people complained again and wanted back our custom cursor

I think this needs to be an option, it appears we aren't pleasing everybody regardless what we do.
The latest addition via https://github.com/microsoft/vscode/pull/96947#issue-413229918


Improve inspect/collect/gamete cursor states and highlight circles

This PR makes minor improvements to the cursor and organism highlight in relation to the show/hide gamete (based on discussion with Michael):

collect cursor is shown when collect button is active (regardless of state of show/hide gametes)
if inspect is active and we are hiding gametes, red inspect cursor is shown
if inspect is active and we are showing gametes, organism highlight is green circle
if inspect is active and we are hiding gametes, organism highlight is red circle
if collect is active and either show or hide gametes is active, organism highlight is blue circle

Looks great :+1:


Feature Request: Add subtitles to Dubs

Feature Request: Add subtitles to Dubs.
It would be a neat feature, if we had the option to show subtitles for dubbed content.

I am with you bro

you can do it if you use source 3/4
Subtitle: Your language
Check Use Dub Stream.
But if you play dub with English subtitle, it was kinda disoriented because the sub sentences is different with what dub says.

Whatever bro you must start a YouTube channel for tutorials please


Alignmentsieve (blacklist, tn5 shift, minqual, rm mito reads) + prettier qc report

What problem is the PR solving / What's new?
If genomepy manages to download a blacklist, it makes use of it.
What did change
WIP

Lots of changes, biggest change being that we now can make use of alignmentsieve. This means minimum mapping quality, tn5 shifts, mitochrondiral and blacklist removal. Another new "cool" feature is that our QC reports look much better (if I may say so).

whoah, you merged blacklist into blacklist, instead of develop into blacklist!

Erhm what? I just pulled remote blacklist into local blacklist.


Fix some tests broken with CC

Fixes for:

Ivy publishing failures with Java 9+
JavaModuleExecutionIntegrationTest
most of WindowsResourcesIntegrationTest and WindowsResourceParallelIntegrationTest
ide.visualstudio tests

@bot-gradle test ACC

I've triggered the following builds for you. Click here to see all build failures.

All ConfigCache Tests for Pull Request Feedback (Trigger) (Pull Request Feedback)
ConfigCache Java21 Adoptium Macos Aarch64 (Trigger) (Ready for Release)
ConfigCache Java21 Adoptium Windows Amd64 (Trigger) (Ready for Release)

@bot-gradle test ACC

I've triggered the following builds for you. Click here to see all build failures.

All ConfigCache Tests for Pull Request Feedback (Trigger) (Pull Request Feedback)
ConfigCache Java21 Adoptium Macos Aarch64 (Trigger) (Ready for Release)
ConfigCache Java21 Adoptium Windows Amd64 (Trigger) (Ready for Release)

@bot-gradle test ACC

I've triggered the following builds for you. Click here to see all build failures.

All ConfigCache Tests for Pull Request Feedback (Trigger) (Pull Request Feedback)
ConfigCache Java21 Adoptium Macos Aarch64 (Trigger) (Ready for Release)
ConfigCache Java21 Adoptium Windows Amd64 (Trigger) (Ready for Release)

@bot-gradle test ACC

I've triggered the following builds for you. Click here to see all build failures.

All ConfigCache Tests for Pull Request Feedback (Trigger) (Pull Request Feedback)
ConfigCache Java21 Adoptium Macos Aarch64 (Trigger) (Ready for Release)
ConfigCache Java21 Adoptium Windows Amd64 (Trigger) (Ready for Release)

This is a good chunk of fixes that reduces the noise from CC tests on CI. I'll continue fixing the remaining tests in next PRs.

The merge queue build has failed. Click here to see all failures.

The merge queue build has failed. Click here to see all failures.


LinkedList.add crashs on web with a cast error

hi,
a user of my get_it_mixing reported a strange cast error when he runs the app on the web.
I could reproduce this in the most basic setup where adding an item to a linked list throws an exception on the web but not on other platforms.
You could find the repro project here
https://github.com/escamoteur/linked_list_web_error
[√] Flutter (Channel stable, 2.0.5, on Microsoft Windows [Version 10.0.18363.1500], locale de-DE)
@mraleph

It happens also in pure Dart on dart pad

@escamoteur I first simplified this example to:
import 'dart:collection';

class SomeType extends LinkedListEntry<SomeType>{}
final _watchList = LinkedList<LinkedListEntry<SomeType>>();

void main() {
  var entry = SomeType();
  try{
    _watchList.add(entry);
  }catch(e){
    print(e);
  }
}

Then thought you don't add an object of type LinkedListEntry, but of SomeType. So this example fixed it for me.
import 'dart:collection';

class SomeType extends LinkedListEntry<SomeType>{}
final _watchList = LinkedList<SomeType>();

void main() {
  var entry = SomeType();
  try{
    _watchList.add(entry);
    print('no problem');
  }catch(e){
    print(e);
  }
}

But _watchEntry extends LinkedListEntry too

Slightly simplified example:
import 'dart:collection';
import 'package:flutter/foundation.dart' as _;

class _WatchEntry<S, T>
    extends LinkedListEntry<_WatchEntry<Object, Object?>> {}
final _watchList = LinkedList<_WatchEntry<Object, Object?>>();

void main() {
  _WatchEntry<Object, Object?> entry = _WatchEntry<bool, bool?>();
  try{
    _watchList.add(entry);
  } catch(e){
    print(e);
  }
  print("Done");
}

This crashes with a type error on dartpad, but if I remove the unused import it succeeds.
The type error that gets printed out is:
Expected a value of type 'LinkedList<_WatchEntry<Object, Object>>?', but got one of type 'LinkedList<_WatchEntry<Object, Object?>>'

The fact that _WatchEntry<Object, Object> is a type that doesn't appear anywhere in this program suggests to me that this is a compiler bug.
cc @sigmundch @rakudrama

cc @nshahan
Thanks everyone for helping us narrow this down to a smaller repro!
The removal of the unnecessary import was a good clue! Dartpad for flutter uses DDC, whereas dartpad for non-flutter uses dart2js. I didn't know this, but the dartpad UI seems pretty dynamic and switches the backend depending on whether flutter is imported.
Locally, this program shows the error in ddc:
import 'dart:collection';

class A<S, T> extends LinkedListEntry<A<Object, Object?>> {}

void main() {
  LinkedList<A<Object, Object?>>().add(A<bool, bool?>());
}

And removing dart:collection:
class Entry<E extends Entry<E>>{
  L<E>? _list;
}
class L<E extends Entry<E>> {
  add(E entry) { entry._list = this; }
}

class A<T> extends Entry<A<Object?>> {}

void main() {
  L<A<Object?>>().add(A<bool?>());
}

is enough to trigger the failure.

a small tweak to remove some of the indirection:
class Entry<E extends Entry<E>>{
  foo(L<E>? input) {}
}
class L<E extends Entry<E>> {}

class A<S> extends Entry<A<Object?>> {}

void main() {
  A<bool>().foo(L<A<Object?>>());
}

This fails with the error:  Expected a value of type 'L<A<Object>>?', but got one of type 'L<A<Object?>>'
An odd piece here is the type L<A<Object>>. You cannot write that type in the program (you get a compile-time error highlighting that A<Object> doesn't comply with the bound of L). So this is likely coming from a place where DDC may be dropping the ? accidentally.

The output of DDC seems to be initializing the base type of A<S> incorrectly:
dart.setBaseClass(repro8.A$(S), repro8.Entry$(repro8.A$(core.Object)));

as opposed to:
dart.setBaseClass(repro8.A$(S), repro8.Entry$(repro8.A$(dart.nullable(core.Object))));

quick update - it seems that whenever we emit this setBaseClass we are omitting the nullability here: https://github.com/dart-lang/sdk/blob/master/pkg/dev_compiler/lib/src/kernel/compiler.dart#L870

Here is a smaller repro illustrating the semantic difference:
class I<T> {}
class F<E extends F<E>> extends I<E> {}
class A<T> extends F<A<Object?>> {} // extending F forces DDC to deferr the superclass and trigger the bug

void main() {
  print(A<bool>() is I<A<Object?>>);
  print(A<bool>() is I<A<Object>>);
}

This prints true/false in the VM and dart2js, but true/true in DDC


Adds JSONAPI version 1 to README.md

I'm learning iOS and I want to use this, but it was hard to figure out what version of the jsonapi format it is using. I had to check the tests to match the format to conclude whether to use this lib or not. I maintain other jsonapi libs (Ruby, JS) since before RC1 and the flux since then up to v1 was really confusing for newcomers.
This line on the README should help other people I believe.

You're right, this should be added now that JSON:API v1 is stable. Thanks!


Ensure final row of NFTs are displayed in sidebar

Only full rows were being pushed into the set of rows to display, and the last row was being ignored if it wasn't full.
Before – only 2 zorbs appearing in sidebar

After – all zorbs listed

Thanks for fixing this @Robinnnnn 🙏🏻 Somehow I misslook this stuff


feature(bindings): Python bindings with full public methods for account/account_manager

Description of change
Bind all pub methods for classes AccountManager, AccountInitialiser, AccountHandle, SyncedAccount, AccountSynchronizer, Transfer, and all related/used/returned structures.
Examples to use the binded lib:
# Copyright 2021 IOTA Stiftung
# SPDX-License-Identifier: Apache-2.0

import iota_wallet

# Create a AccountManager
manager = iota_wallet.AccountManager(
    storage='Stronghold', storage_path='./storage')
manager.set_stronghold_password("password")
manager.store_mnemonic("Stronghold")

# First we'll create an example account and store it
client_options = {'node': 'https://api.lb-0.testnet.chrysalis2.com'}
account_initialiser = manager.create_account(client_options)
account_initialiser.alias('alias')
account = account_initialiser.initialise()

# Update alias
account.set_alias('the new alias')

# Get unspent addresses
unspend_addresses = account.list_unspent_addresses()
print(f'Unspend addresses: {unspend_addresses}')

# Get spent addresses
spent_addresses = account.list_spent_addresses()
print(f'Spent addresses: {spent_addresses}')

# Get all addresses
addresses = account.addresses()
print(f'All addresses: {addresses}')

# Generate a new unused address
new_address = account.generate_address()
print(f'New address: {new_address}')

# List messages
messages = account.list_messages(5, 0, message_type='Failed')
print(f'Messages: {messages}')

Links to any relevant issues
N/A
Type of change
Choose a type of change, and delete any options that are not relevant.

Enhancement (a non-breaking change which adds functionality)

How the change has been tested

Tested w/ examples by the built libs.

Change checklist
Tick the boxes that are relevant to your changes, and delete any items that are not.

[X] I have followed the contribution guidelines for this project
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[X] I have checked that new and existing unit tests pass locally with my changes

We've been having some issues with macOS lately. I'm not sure why 🤔

We've been having some issues with macOS lately. I'm not sure why

Just found this issue: https://zulip-archive.rust-lang.org/131828tcompiler/94754possibleincrementalbugonmacOS.html

We've been having some issues with macOS lately. I'm not sure why

A

@rajivshah3 do you know how to solve the coverage issue?

I realized the code coverage we're currently using is quite inaccurate for some reason. I'm planning to switch it to LLVM source-based coverage but I haven't had time lately

Just found this issue: https://zulip-archive.rust-lang.org/131828tcompiler/94754possibleincrementalbugonmacOS.html

NIce find! Will create a PR for this now


[DO NOT MERGE] Ensure Warden Failing Build Category 2

This build should also fail.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Codecov Report

Merging #4559 into master will decrease coverage by 3.89%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4559      +/-   ##
==========================================
- Coverage    53.4%    49.5%    -3.9%
==========================================
  Files       10535     6332    -4203
  Lines      220638   177160   -43478
==========================================
- Hits       117836    87710   -30126
+ Misses     102802    89450   -13352

Impacted Files
Coverage Δ

...afactory/azure/mgmt/datafactory/models/__init__.py
50.67% <0%> (-49.33%)
:arrow_down:

...ights/azure/applicationinsights/models/__init__.py
51% <0%> (-49%)
:arrow_down:

...e/cognitiveservices/vision/face/models/__init__.py
51.11% <0%> (-48.89%)
:arrow_down:

...services/language/textanalytics/models/__init__.py
51.16% <0%> (-48.84%)
:arrow_down:

...iveservices/search/entitysearch/models/__init__.py
51.25% <0%> (-48.75%)
:arrow_down:

azure-eventgrid/azure/eventgrid/models/__init__.py
51.25% <0%> (-48.75%)
:arrow_down:

...ration/azure/mgmt/datamigration/models/__init__.py
51.27% <0%> (-48.73%)
:arrow_down:

...ure/mgmt/recoveryservicesbackup/models/__init__.py
51.46% <0%> (-48.54%)
:arrow_down:

...ce/azure/mgmt/containerinstance/models/__init__.py
51.8% <0%> (-48.2%)
:arrow_down:

...tiveservices/search/videosearch/models/__init__.py
52.08% <0%> (-47.92%)
:arrow_down:

... and 4345 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 063af3e...cfb9f6a. Read the comment docs.


Sign Language Translator

Develop an system that uses computer vision to recognize and interpret sign language gestures in real-time .
It must be a 2 way translator, that is,

It must convert the Sign gestures to text and Synthesized speech (optional)
It must also convert spoken language into visual sign language representations. (here primary language can be considered as English)

[!NOTE]
For your reference, I've already worked on the different variation of the project, where user can upload the document (or) provide You tube video link (or) provide audio/video , where it'll transcribe and map them to different sign languages
Check : https://github.com/SaiNivedh26/ACCESSIBLE-AI-Digital-content-to-sign-language-converter

@smartfellow1234 Proceed! Please star the repo if you find it helpful.
And Make sure the implementation should perfectly aligned with the issue.

I have some additional features For this to include
AutoCorrection
AutoSuggestion - Natural Language Processing
Translator - Sign to Speech integrating 100 Languages
Male and Female Voice
Can you please assign me this @UppuluriKalyani?

proceed @rithvikreddybasani !


Report menu and camera fixes

Can’t report players without filling out required fields
Dropdown open buttons no longer animate when not allowed
Fixed weird tween effect with the “game or player” when switching to
the report page
Disabled selectors can no longer be used by clicking them in the
center
Camera zoom level persists between modes, preventing players from
getting into third person when first person is forced

Should we just close this? It seems like it will just be here forever otherwise.


update permissions for annarozin ,remove itaimarko

Tagging @daniel-beck
Adding maintainer for WhiteSource plugin:
https://github.com/jenkinsci/whitesource-plugin
Description

Submitter checklist for adding or changing permissions

Always

[x] Add link to plugin/component Git repository in description above

For a newly hosted plugin only

[ ] Add link to resolved HOSTING issue in description above

For a new permissions file only

[x] Make sure the file is created in permissions/ directory
[x] artifactId (pom.xml) is used for name (permissions YAML file).
[x] groupId / artifactId (pom.xml) are correctly represented in path (permissions YAML file)
[x] Check that the file is named plugin-${artifactId}.yml for plugins

When adding new uploaders (this includes newly created permissions files)

[ ] Make sure to @mention an existing maintainer to confirm the permissions request, if applicable
[ ] Use the Jenkins community (LDAP) account name in the YAML file, not the GitHub account name
[ ] Make sure to @mention the users being added so their GitHub account names are known if they require GitHub merge access (see below).
[x] All newly added users have logged in to Artifactory at least once

Reviewer checklist (not for requesters!)

[ ] Check this if newly added person also needs to be given merge permission to the GitHub repo (please @ the people/person with their GitHub username in this issue as well). If needed, it can be done using an IRC Bot command
[ ] Check that the $pluginId Developers team has Admin permissions while granting the access.
[ ] In the case of plugin adoption, ensure that the Jenkins Jira default assignee is either removed or changed to the new maintainer.
[ ] If security contacts are changed (this includes add/remove), ping the security officer (currently @daniel-beck) in this pull request. If an email contact is changed, wait for approval from the security officer.

There are IRC Bot commands for it

@annarozin Please read the PR template and follow the instructions.

@annarozin Please read the PR template and follow the instructions.

Done, please see my changes in "Description"

Hi @annarozin since you are not changing the security contacts, you don't need to tag Daniel. You do need to tag one of the contributors that is already listed in the file (most likely edoshor) so we have their OK to you being added to the list. We would also need confirmation from itaimarko to remove them from the list.

Hi @slide itaimarko is no longer working at WhisteSource same for edoshor, what is the best way to continue from here?
As you can see I'm a contributor in WhiteSource repo and I have a user in Artifactory.

Hi @slide itaimarko is no longer working at WhisteSource same for edoshor, what is the best way to continue from here?
As you can see I'm a contributor in WhiteSource repo and I have a user in Artifactory.

Adding people is different from removing people. I can see that you have commit access. Generally we don't remove people from the access list without their permission, can you change the PR so that is just adds you for now?

@slide done

This was a quick abandoned plugin takeover, no emails to current maintainers and nothing.

This was a quick abandoned plugin takeover, no emails to current maintainers and nothing.

In the past, we've done similar when the person already had commit access to the repository. We need to have a better, documented process for plugins developed by companies. They often go through people leaving and coming on board quickly which is different from a completely community maintained plugin. I agree it is not ideal, but there is precedent.


AttrinuteError when trying to install a VM extension that is already installed

Description
Outline the issue here:
I've installed an extension using the Azure cli with vm extension set. I then tried to install the extension again with the same version number. The follow error is shown:

Environment summary
Install Method: How did you install the CLI? (e.g. pip, interactive script, apt-get, Docker, MSI, nightly)
Answer here: curl -L https://aka.ms/InstallAzureCli | bash
CLI Version: What version of the CLI and modules are installed? (Use az --version)
Answer here:
azure-cli (2.0.9)
acr (2.0.7)
acs (2.0.9)
appservice (0.1.9)
batch (3.0.2)
billing (0.1.2)
cdn (0.0.5)
cloud (2.0.5)
cognitiveservices (0.1.5)
command-modules-nspkg (2.0.0)
component (2.0.6)
configure (2.0.9)
consumption (0.1.2)
core (2.0.10)
cosmosdb (0.1.9)
dla (0.0.9)
dls (0.0.9)
feedback (2.0.5)
find (0.2.5)
interactive (0.3.5)
iot (0.1.8)
keyvault (2.0.7)
lab (0.0.7)
monitor (0.0.7)
network (2.0.9)
nspkg (3.0.0)
profile (2.0.7)
rdbms (0.0.4)
redis (0.2.6)
resource (2.0.9)
role (2.0.7)
sf (1.0.4)
sql (2.0.6)
storage (2.0.9)
vm (2.0.9)
Python (Linux) 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609]
OS Version: What OS and version are you using?
Answer here:
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial
Shell Type: What shell are you using? (e.g. bash, cmd.exe, Bash on Windows)
Answer here:
bash

What extension are you trying to install?

This has been fixed


Save instance of results object on to disk

Sometimes at the end of long optimization runs, we would like to store the objects to disk for future use.
One immediate use-case is https://github.com/scikit-optimize/scikit-optimize/issues/180
For pickling, lambdas etc, I suggest we backport cloudpickle (https://github.com/apache/spark/blob/master/python/pyspark/cloudpickle.py) to skopt.externals, unless others have better ideas.

This is a duplicate of #190. Should we move there or close that one?

Ah I forgot about that. I'll close this.


chore: upgrade dependencies

We are having some issues preventing us from updating some of our core dependencies due to the html_template version being a little bit behind.

Thanks. Rolled to pub.dev as 0.8.6


AttributeError: 'ElectronicEnergy' object has no attribute 'electronic_integrals'

Informations

Qiskit version: Qiskit Nature 0.5.2
Python version: 3.9
Operating system: Linux, Ubuntu 22.04

What is the current behavior?
return FermionicOp.from_polynomial_tensor(self.electronic_integrals.second_q_coeffs())

AttributeError: 'ElectronicEnergy' object has no attribute 'electronic_integrals'
Steps to reproduce the problem
We can get any ElectronicEnergy object from qiskit_nature.second_q, for my case, it was from the WEST software package.
effective_hamiltonian = QDETResult(filename='west.wfreq.save/wfreq.json')
H_FCI = effective_hamiltonian.heff.make_fermionic_operator()
print(ElectronicEnergy.second_q_op(H_FCI))
To run this, you must have ran wfreq.x on a machine with quantum espresso and WEST
What is the expected behavior?
Produce a FermionicOperator object
Suggested solutions
It appears that self._electronic_integrals exists, but not self.electronic_integrals

Like this, we cannot reproduce this issue. Please either submit a fully working code to reproduce this issue or properly point us to the source where the ElectronicEnergy object is being constructed. Looking at the source code of the ElectronicEnergy class here, I would not expected _electronic_integrals to even exist. Only the public attribute is ever defined.

Looking over this stale issue again, it seems like you are misusing the ElectronicEnergy.second_q_op instance method:

print(ElectronicEnergy.second_q_op(H_FCI))

While this works in Python, this is not correct usage. Instead, you would need to create and instance of ElectronicEnergy and then call its .second_q_op method. For example:
electronic_energy = ElectronicEnergy(...)
print(electronic_energy.second_q_op())

I suspect that whatever code you used to generate an ElectronicEnergy it actually produces the legacy object and you are trying to use it like the new but different object.
You can get your code to work by one of these means:

correctly use the legacy object and checks its code for the documentation (since this is not published via the HTML docs)
update your code to the new objects

I strongly suggest the latter. To do so, you can follow our migration guide.


import System error

mv mirah-0.0.11.dev-java.gem ..
cd -
mkdir -p dist
mkdir -p build
Compiling Ruby sources
MirahCommand.java:10: '.' expected
import System;
             ^
MirahCommand.java:10: ';' expected
import System;
              ^
2 errors
Compiling Mirah sources

import System should be removed from ./build/org/mirah/MirahCommand.java

Fixed by #166.


Restarting the app leads to a duplication of the scheduled test list

Due to the current way the scheduled Exercise List is persistent in the DB, the list is duplicated on every restart.
The full implimentation is only for testing purpose, but we should create a hotfix do solve this for now

Solved


⚠ Bofrost.lu has degraded performance

In 0730bef, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1781 ms

Resolved: Bofrost.lu performance has improved in 04e6704 after 40 minutes.


Add libcamera autofocus options

Libcamera has some controls options for focus that are useful for some new cameras that exists.
Arducam 16Mp and 64Mp cameras with autofocus are examples of this.
This commit add some of this options for focus to the command line.

Thank you for this work.  Most of it looks reasonable, but I'd like to address a few minor details in the code.

@njhollinghurst might have a few further comments.

This one is almost ready to merge.
The style checker has picked out some issues that need correcting:
https://github.com/raspberrypi/libcamera-apps/actions/runs/3514039845/jobs/5906783036
You can ignore the warnings about the std::map definitions, and most of the others are likely asking for spaces to tabs conversions.

Thanks for the fixups!
We don't use merge commits on our tree, but  I'll fix this locally and push your changes shortly.


Add a backend:bundle command to the CLI

Feature Suggestion
Right now the only simply way to deploy a production version of the backend is by using the backend:build-image command. It creates a temporary folder outside of the repo and copies over the distribution versions of each backend package in the monorepo and then runs the Docker build within that package. This ensures that the resulting backend image only contains backend dependencies and built code. It also avoid installation of development dependencies.
Because the backend:build-image command is operating both on the host machine and docker, it can't be used inside a docker build itself, or in build environments that run exclusively in docker containers and don't support docker in docker. There is a lower-level build-workspace command that can be used to replicate this setup, but the backend:build-image command also adds proper docker build caching of yarn install as well as automatic detection and building of all dependencies of the backend within the monorepo, both which are very tedious and verbose to replicate manually within a Dockerfile.
We should aim to restructure the backend production build process to preferably generate usable build output isolated to just the dist/ dir. It should also allow for yarn install --production to be run afterwards, with installation limited to only the involved packages.
Possible Implementation
Possibly something similar to how the current build-workspace and backend:build-image commands work, but move the output to a dist/ dir instead? It's tricky because we want to ensure that the node_modules structure is the same as during development.
Context
This is brought up daily in Discord #support 😅

I've started looking at speeding up our glacial docker builds in GitHub Actions. Starting with yarn cache.

It creates a temporary folder outside of the repo [...]
backend:build-image command also adds proper docker build caching of yarn install

How can I save and restore the cache you mention? I can't find where the docker build caching of yarn install is set up.
Is this moot once you're figured out how to get the backend:bundle to work?

@fidelcoria-aa see our repo for inspiration on  how we do the caching - it's a bit involved to get it as efficient as possible. https://github.com/backstage/backstage/blob/master/.github/workflows/ci.yml#L32

@freben, that isn't doing docker build. We already do yarn caching like you have there.
The problem is the docker build does not use the yarn cache.
I also already set up:
  - uses: docker/setup-buildx-action@v1
  - uses: actions/cache@v2
    env:
      cache-base:  v1-${{ runner.os }}-cache-docker
    with:
      path: /tmp/.buildx-cache
      key: ${{ env.cache-base }}-${{ github.sha }}
      restore-keys: |
        ${{ env.cache-base }}-

But  how to get the backstage build to use the cache... still digging around. maybe you can help
how can I get the same effect as:
  - name: Docker build
    uses: docker/build-push-action@v2
    with:
      push: true
      tags: |
        ${{ steps.getTag.outputs.shaImageTag }}
        ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:latest
      cache-from: type=local,src=/tmp/.buildx-cache
      cache-to: type=local,dest=/tmp/.buildx-cache

when I have to do a run: yarn workspace @runway/backend build-image

@fidelcoria-aa Is this for building the backend only? and are you calling backstage-cli backend:build-image along with a Dockerfile that includes an ADD for skeleton.tar such as the on in this repo? https://github.com/backstage/backstage/blob/master/packages/backend/Dockerfile
The skeleton.tar contains all the necessary files to do a cached yarn install at the docker cache level. When it comes to using yarn cache itself that's not really possible with regular docker build tooling afaik. It would require mounting of host directories into the build containers. That's unless you use a base image that already contains some of the packages which you rebuild occasionally.
For speeding up the builds I would generally suggest a local caching NPM proxy if that's a possibility, as that can be quick enough that it doesn't really matter much whether you're fetching packages over the network or build cache.

@Rugvip is working on this very thing - is there something you would like to share here?

Sorry to pollute this PR, this could've been a discord or a new issue.

Is this for building the backend only? and are you calling backstage-cli backend:build-image along with a Dockerfile that includes an ADD for skeleton.tar such as the on in this repo?

yes and yes. We have the latest version of backstage-cli as well. But the yarn install during docker build is way too slow to be using a cache. (When a docker build is repeated in the same GH Action Job there is caching and faster build).
Yeah I thought about mounting host directories but was looking for a more compact option. I'm trying this out now.  I'll open a new issue if I can't get it to work.


Teleport with leashes

Would be really nice to be able to teleport entities that u leashed with u.

partially implemented in 66ffddafd4a356a16c33c43b169de3057d4057bb

@SooStrator1136 finished the implementation in 445b1da5b343eed8eb2a11663d14b65c86842faf


ViewContextChanged fix

@CryZe, does this fix #2306?

Yeah that does indeed seem to fix the bug. Thank you 🎉


New cop idea: Layout/EmptyLineAfterLeadingRequires

Is your feature request related to a problem? Please describe.
I often see these two patterns of empty lines at the beginning of ruby files, and I feel it would be better if the style of empty lines were consistent:
Pattern 1
# frozen_string_literal: true

require 'foo'
bar
baz

Pattern 2
# frozen_string_literal: true

require 'foo'

bar
baz

Describe the solution you'd like
How about adding a new cop like Layout/EmptyLineAfterLeadingRequires ?
# bad
require 'foo'
bar
baz

# good
require 'foo'

bar
baz

The implementation itself is probably easy, so if it looks good, I will create a pull request.
Describe alternatives you've considered
I once considered whether this cop should also target other requires not located at the beginning of the file, but it seems that it would be better not to do so.

Will this cop apply to require_relative as well?

Good point. It certainly seems like it should apply to require_relative as well as require.

Should this apply to only leading requires or all requires? I vote for all requires.
And please do open a PR.

One example of a problem we might have with applying requires to other than the top of the file is the following case:
a

require 'some_file_for_b'
b

c

In the above code, the absence of a blank line indicates that require and the following code are related.
However, this is lost if the rule of always having an empty line is applied like this:
a

require 'some_file_for_b'

b

c

I feel that require appearing in the middle of a file is often placed just before a specific related process like this. What do you think about this? Would it be more profitable to regard them as edge cases and create a cop like Layout/EmptyLineAfterRequires?

Why not something like this?
Layout/EmptyLineAfterRequires:
  Description: 'Checks for an empty line after require statements.'
  Enabled: true
  VersionAdded: '<<next>>'
  EnforcedStyle: top_level_only
  SupportedStyles:
    - all_requires
    - top_level_only
  RequireMethods:
    - require
    - require_relative
    # e.g. rubocop-rails could extend to add require_dependency, for apps using classic autoloader


Accept max in-flight requests on the connection

This PR adds support to max in-flight requests. The option is set to null now so we can test it out, I want to understand the impact of more or less concurrent requests before we commit to a default value.

@ianwsperber @Nevon I addressed all the comments, please take another look


Connection Issues with Posh-SSH 3.0 and RSA 2048 Keys on Linux 2023 OS

Hello everyone,
I am experiencing connection issues when using Posh-SSH 3.0 to establish an SSH connection with a Linux machine running the 2023 operating system. The connection fails because the OS does not allow RSA 2048 keys for authentication, likely due to security policies that favor stronger key algorithms.
Here are the details:
Environment: Using Posh-SSH 3.0 on a Windows machine.
Target OS: Linux 2023 OS.
Key Type: The SSH key currently being used is an RSA key with a bit length of 2048.
Error Message: Permission denied (publickey).
I would like to know if there is a workaround to enable the use of RSA 2048 keys on the Linux 2023 OS, for establishing the connection.
Any insights or suggestions would be greatly appreciated!

Can you upgrade to the latest version of Posh-SSH and test?

Thanks. Latest version resolved the problem.


Pulse drawer v2, mpl plotter and interface

Summary
This adds a part of new pulse drawer (5/5). See #4643
This PR implements one of plotters and common interface of the pulse drawer v2.
Details and comments
The detail and the design concept are written in the module docstring.

Mpl2DPlotter class is added. This is one of the plotter APIs of the v2 pulse drawer. This takes a canvas object and generate a 2D image from them.
Several minor updates to generators and layouts to cleanup the appearance of output image.

You can test image generation with following code with this branch:
from qiskit import QuantumCircuit, transpile, schedule
from qiskit.visualization.pulse_v2 import interface, stylesheet
from qiskit.test.mock import FakeAlmaden

qc = QuantumCircuit(2)
qc.h(0)
qc.cx(0, 1)
qc.measure_all()
qc = transpile(qc, FakeAlmaden())
sched = schedule(qc, FakeAlmaden())

interface.draw(program=sched, style=stylesheet.IqxStandard(), backend=FakeAlmaden())

You will get this image with above code.

This is much cleaner than the existing drawer's output.

In addition, the wall time to generate above image is:

v2 drawer: 143 ms
v1 drawer: 660 ms

This performance improvement is thanks to the waveform truncation and consecutive data points compression.
We need to write reno after sufficient usability test until official release (I don't think I did enough test with this drawer, because it has lots of new features), add import to the top level visualization module (also add this to API docs), and support parametrized schedule visualization with #5073 .

When can we replace schedule.draw with this?

@lcapelluto thanks for reviewing.
Now the drawer supports visualization of parametric pulse with unbound parameters. I also fixed a bug of sequential truncation along the same channel and updated unittests.
This is example of generated image:
from qiskit import QuantumCircuit, transpile, schedule, pulse
from qiskit.circuit import Parameter

from qiskit.visualization.pulse_v2 import interface, stylesheet
from qiskit.test.mock import FakeAlmaden

with pulse.build(FakeAlmaden()) as cx10:
    pulse.play(pulse.GaussianSquare(3200, Parameter('amp_cx'), 128, 2688), pulse.DriveChannel(0))
    pulse.play(pulse.GaussianSquare(3200, 0.1, Parameter('sigma_cx'), 2688), pulse.ControlChannel(1))

qc = QuantumCircuit(2)
qc.h(0)
qc.cx(0, 1)
qc.measure_all()
qc.barrier(0, 1)
qc.cx(1, 0)
qc.barrier(0, 1)
qc.measure_all()
qc.add_calibration('cx', (1, 0), cx10)

qc = transpile(qc, FakeAlmaden())
sched = schedule(qc, FakeAlmaden())

interface.draw(program=sched, style=stylesheet.IQXStandard(), backend=FakeAlmaden())

I think this PR is ready for the final review.

PR appears to be failing CI. Once its passing we can get it in 👍 .

Thanks. I just removed file save method because the file save method is called only by interface (i.e. top level draw, because draw doesn't return plotter class) and managing file save API arguments of different plotters with draw arguments may be difficult (or make arguments messy). The function returns a figure object and user can save it in file if the figure object has a method to save itself (matplotlib Figure has .savefig).


[SPARK-19937] Collect metrics of block sizes when shuffle.

What changes were proposed in this pull request?
Metrics of blocks sizes(when shuffle) should be collected for later analysis. This is helpful for analysis when skew situations or OOM happens(though maxBytesInFlight is set).
This pr proposes to:

Store the distribution of sizes in MapStatus and count the block sizes in ranges
[0, 1k), [1k, 10k), [10k, 100k), [100k, 1m), [1m, 10m), [10m, 100m), [100m, 1g), [1g, 10g), [10g, Long.MaxValue).
Record the average and the max of block sizes. Because when block sizes is over 2000, HighlyCompressedMapStatus is returned and only average size is recorded, thus inaccuracy exists.

Test build #74449 has started for PR 17276 at commit 648ceaa.

Test build #74448 has started for PR 17276 at commit 430ec95.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74448/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74449/
Test FAILed.

Test build #74509 has started for PR 17276 at commit d0932ed.

Test build #74515 has started for PR 17276 at commit e2e56d3.

Test build #74515 has finished for PR 17276 at commit e2e56d3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74515/
Test FAILed.

Test build #74509 has finished for PR 17276 at commit d0932ed.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74509/
Test FAILed.

Test build #74605 has started for PR 17276 at commit 2ccde1f.

Test build #74607 has started for PR 17276 at commit 91e338b.

Test build #74605 has finished for PR 17276 at commit 2ccde1f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74605/
Test FAILed.

Merged build finished. Test FAILed.

Test build #74607 has finished for PR 17276 at commit 91e338b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74607/
Test FAILed.

Merged build finished. Test FAILed.

Test build #74665 has started for PR 17276 at commit 7cd290d.

Test build #74665 has finished for PR 17276 at commit 7cd290d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74665/
Test PASSed.

Merged build finished. Test PASSed.

@squito
Would you mind help comment on this when have time ? :)

Test build #74917 has started for PR 17276 at commit 0e85332.

Test build #74917 has finished for PR 17276 at commit 0e85332.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74917/
Test FAILed.

Test build #74931 has started for PR 17276 at commit 7ac639f.

Test build #74938 has started for PR 17276 at commit e6091b6.

Test build #74939 has started for PR 17276 at commit a88e12e.

Test build #74931 has finished for PR 17276 at commit 7ac639f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74931/
Test FAILed.

Merged build finished. Test FAILed.

Test build #74938 has finished for PR 17276 at commit e6091b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74939/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74938/
Test PASSed.

Test build #74962 has started for PR 17276 at commit f7ff868.

Test build #74962 has finished for PR 17276 at commit f7ff868.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74962/
Test PASSed.

Merged build finished. Test PASSed.

Test build #74981 has started for PR 17276 at commit d86f985.

@squito
Thanks a lot for your comments and I will think and do the test carefully :)

Test build #74981 has finished for PR 17276 at commit d86f985.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74981/
Test PASSed.

Test build #75032 has started for PR 17276 at commit b2ba192.

Test build #75034 has started for PR 17276 at commit 1720b5e.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75032/
Test FAILed.

Test build #75039 has started for PR 17276 at commit c9276c2.

Test build #75034 has finished for PR 17276 at commit 1720b5e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75034/
Test PASSed.

Test build #75039 has finished for PR 17276 at commit c9276c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75039/
Test PASSed.

Merged build finished. Test PASSed.

@jinxing64 do you mind closing this pr for now (or marking as [WIP] at least)?

@squito oh, I feel sorry if this is disturbing. I will mark it as wip.

no worries, I'm just not sure when to look again, with all the notifications from your commits. Committers tend to think that something is ready to review if its passing tests, so its helpful to add those labels if its not the case.

Test build #75098 has started for PR 17276 at commit 6a96c3b.

Test build #75098 has finished for PR 17276 at commit 6a96c3b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75098/
Test FAILed.

Test build #75145 has started for PR 17276 at commit 4f992fc.

Test build #75146 has started for PR 17276 at commit c58cb7e.

Test build #75145 has finished for PR 17276 at commit 4f992fc.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75145/
Test FAILed.

Test build #75146 has finished for PR 17276 at commit c58cb7e.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75146/
Test FAILed.

Merged build finished. Test FAILed.

Test build #75163 has started for PR 17276 at commit 0efa348.

Test build #75163 has finished for PR 17276 at commit 0efa348.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75163/
Test FAILed.

Test build #75220 has started for PR 17276 at commit c26ea56.

Test build #75220 has finished for PR 17276 at commit c26ea56.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75220/
Test PASSed.

Test build #75229 has started for PR 17276 at commit 8801fc6.

@squito
Thanks a lot for taking time looking into this pr.
I updated the pr. Currently just add two metrics: a) the total size of underestimated blocks size, b) the size of blocks shuffled to memory.
For a), executor use maxBytesInFlight to control the speed of shuffle-read. I agree with your comment another metric that may be nice to capture here is maximum underestimate. But think about this scenario: the maximum is small, but thousands of blocks are underestimated, thus maxBytesInFlight cannot help avoid the OOM during shuffle-read. That's why I proposed to track the metrics of total size of underestimated blocks size;
For b), currently all data are shuffled-read to memory. If we add the feature of shuffling to disk when memory shortage, we need to evaluate the performance. I think another two metrics need to be taken into account: the size of blocks shuffled to disk(to be added in another pr) and task's running time(already exist). The more data shuffled to memory, the better performance; The shorter time cost, the better performance.
I also added some log for debug in ShuffleWriter, including the num of underestimated blocks and the size distribution.

Test build #75229 has finished for PR 17276 at commit 8801fc6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75229/
Test PASSed.

Test build #75238 has started for PR 17276 at commit cf5de4a.

@mridulm
Thanks a lot for taking time looking into this and thanks for comments :)

I changed the size of underestimated blocks to be partitionLengths.filter(_ > hc.getAvgSize).map(_ - hc.getAvgSize).sum
I added a method genBlocksDistributionStr and call it from ShuffleWriters, thus avoid duplicate codes

Test build #75239 has started for PR 17276 at commit 873129f.

Test build #75238 has finished for PR 17276 at commit cf5de4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75238/
Test PASSed.

Test build #75239 has finished for PR 17276 at commit 873129f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75239/
Test PASSed.

@jinxing64
If the intent behind these metrics is to help with SPARK-19659, it would be good to either add it as part of SPARK-19659 or subsequently (once the feature is merged).
This ensures that the metrics added are actually relevant to the existing spark core, and not a future expected evolution of the code - for example, the review of SPARK-19659 might significantly change its design/implementation : making some of these either irrelevant or require other more informative metrics to be introduced.
I am unclear about the intention btw - do you expect shuffle reads to be informed by metrics from the mapper side ? I probably got that wrong.

@mridulm
Sorry for late reply.  I opened the pr for SPARK-19659(https://github.com/apache/spark/pull/16989) and make these two PRs independent. Basically this pr is is to evaluate the performance(blocks are shuffled to disk) and stability(size in MapStatus is inaccurate and OOM can happen) of the implementation proposed in SPARK-19659.
I'd be so thankful if you have time to  comment on these two PRs.

Test build #76454 has started for PR 17276 at commit 873129f.

Test build #76454 has finished for PR 17276 at commit 873129f.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76454/
Test FAILed.

@mridulm @squito
Thanks a lot for taking time review this pr.
I will close it for now and make another one if there is progress.


[DOC] Awareness Attribute Decommission

What do you want to do?

[ ] Request a change to existing documentation
[X] Add new documentation
[ ] Report a technical problem with the documentation
[ ] Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
We have introduced a feature to decommission an awareness attribute from the cluster in the event of zonal failures. With decommissioning an awareness attribute, nodes having those attributes will be kicked out of the cluster and will stay out until the attribute is recommissioned back. Hence, these decommission/recommission APIs serves as an andon cord to cut-off inter-zone replication traffic.
We need to document a brief summary of the feature and all the new apis.
Details on api can be found in the https://github.com/opensearch-project/OpenSearch/issues/3639.
Which versions are affected?
2.4.0
What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

High Level Proposal - https://github.com/opensearch-project/OpenSearch/issues/3402
Design Proposal - https://github.com/opensearch-project/OpenSearch/issues/3917
APIs RFC - https://github.com/opensearch-project/OpenSearch/issues/3639
Meta Issues - https://github.com/opensearch-project/OpenSearch/issues/3957

Major PRs -

https://github.com/opensearch-project/OpenSearch/pull/4084
https://github.com/opensearch-project/OpenSearch/pull/4261
https://github.com/opensearch-project/OpenSearch/pull/4590
https://github.com/opensearch-project/OpenSearch/pull/4586
https://github.com/opensearch-project/OpenSearch/pull/4320
https://github.com/opensearch-project/OpenSearch/pull/4604

@Naarcha-AWS any updates on this? Let me know if more details are required

@Naarcha-AWS any updates?

I'll account for these as well with the 2.5 updates for #1811.


Inactivates link when Y is out of Element bounds

There is an issue with active link. When we scroll up to some section – correspondent link becomes active, but when we scroll away from that section – link still remains active (unless some other link become active).

Also top and height variables renamed to more consistent elemTopBound and elemBottomBound correspondently.

Thanks for the PR will look into it tomorrow.

You are welcome :) Any comments?

yeah sorry about the delay, currently sick as fuck, but will look into it asap.

Get well soon!

Will look into it this weekend :)

Merged!


KAFKA-5184, KAFKA-5173; Call closeSasl from MultipleListenersWithSameSecurityProtocolBaseTest

Also:

Refactor the code to make it easier to reason about
Add an assert that may possibly help us narrow down how KAFKA-5184
can happen (it seems impossible).

@rajinisivaram, can you please take a look? These failures are happening quite often, so we need to figure them out ASAP.

@ijuma Sorry, I was traveling etc. So haven't had time to look into the failures yet. I haven't had a chance to look at this PR yet, but noticed there was the JAAS configuration failure even with this PR. The only way that could occur is if we are leaving some threads behind in some tests which load Configuration. I will take a look at the test changes in the last few weeks and also dump threads at the time of failure to narrow that one down. I think this PR makes the code a lot neater, so that is very good. And obviously closeSasl needs to be added to the test. I will try and narrow down the threads being left behind in tests and come back to this PR later today. Hope that is ok.

@rajinisivaram, thanks for the update. Yes, I noticed that failure too. I pushed a few additional changes. Something else that could be related is the addition of the SaslSslAdminClientIntegrationTest. The tearDown for the client was not happening in sequence.
Doing a thread dump sounds like a good idea.
And yes, I realised that you have been travelling, so no worries. :) Thanks for helping with this!

All tests passed, will run them again just in case.

retest this please

@ijuma These changes look good to me. Unfortunately, there was another failure even with these changes. This should definitely still be merged though since it has made the code a lot neater.

Thank @rajinisivaram, I merged this to trunk. Annoying that it doesn't fix the underlying issue. Hopefully, we'll track it down in your PR with additional logging.


tidy up zjson

There has been recent discussions about changing zjson to make it more ergonomic.
The zng data format has richly typed records as well as a deterministic column order, whereas json objects do not have a deterministic column order or typing beyond strings, float64's, object, arrays, and null.  Thus, if you want to transmit zng data in json and preserve its type structure, you need to encode all this in a layer above json.
A different question is should we have the option to not impose the complexity of zng on a client?  Just throw it out.  This is trivial to implement: currently the search endpoint returns zjson for both "zjson" and "ndjson" requests.  We can change the ndjson output to use the ndjson writer instead of the zjson writer.  Then a user could get vanilla ndjson without the richness of zng.  So, we can basically already support this with a trivial change.
Regarding the zjson design, the current approach roughly follows that of the underlying zng model of embedding type information in the stream: type definitions declare arbitrarily complex and nested data types, and values are sent referencing the type information recursively with small-integer type identifiers.  In this approach, a sequence of ndjson objects represents a typed value and if the type definition hasn't been previously seen, it is included in the json object the first time it is encountered.
For example, consider this zng:
#0:record[a:string,b:record[x:int32,y:ip]]
0:[hello;[2;<IP_ADDRESS>;]]
0:[world;[4;<IP_ADDRESS>;]]
0:[goodnight;[4;<IP_ADDRESS>;]]
0:[gracie;[4;<IP_ADDRESS>;]]

This encodes in ndjson (with zq, but not with zqd presently):
{"a":"hello","b":{"x":2,"y":"<IP_ADDRESS>"}}
{"a":"world","b":{"x":4,"y":"<IP_ADDRESS>"}}
{"a":"goodnight","b":{"x":4,"y":"<IP_ADDRESS>"}}
{"a": "gracie","b":{"x":4,"y":"<IP_ADDRESS>"}}

i.e., you no longer know that x was an integer (it is now a javascript number) or that y is an IP address.  Certain apps or users may want the simplicity of this so let's just support it.
What we're trying to do, though, as a strategic direction for the company is to differentiate from the json world (and log analytics systems) and make our app's ux aware of rich data types and structure.  i.e., zeek experts get to see the zeek data presented in the column order they are used to and see familiar structures like IP address, network ports, sets of IP addresses, vectors of values, and so forth (and I would argue these attributes of structured data presentation for semi-structured data will have impact far beyond zeek).  So, zjson to the rescue:
{
  "id": 24,
  "type": [
    { "name": "a", "type": "string" },
    { "name": "b", "type": [ {  "name": "x",  "type": "int32" },  { "name": "y",  "type": "ip" }  ]  }
  ],
  "values": [ "hello",  [  "2",  "<IP_ADDRESS>"  ]  ]
}
{  "id": 24,  "values": [  "world",  [  "4", "<IP_ADDRESS>" ]  ] }
{  "id": 24,  "values": [  "goodnight",  [  "4", "<IP_ADDRESS>" ]  ] }
{  "id": 24,  "values": [  "gracie",  [  "4", "<IP_ADDRESS>" ]  ] }
}

This is admittedly more complex than json but this is because json and javascipt don't support rich typings so we have to bite the bullet one way or another (unless we want abandon our aspirations and retreat to simple javascript types in the our app).
One proposal is that zjson should change so that the types are zipped into the values so each line of ndjson is fully self-describing.  This could look something like this:
 [  { "name": "a",  "type": "string", "value":  "hello"},
      {name: "b", "type":"record",  value: [ {"name": "x", "type": "int32", "value": "2" },
                                                                    { "name": "y", "type": "ip", "value": "<IP_ADDRESS>" ]  ]
 [  { "name": "a",  "type": "string", "value":  "world"},
      {name: "b", "type":"record",  value: [ {"name": "x", "type": "int32", "value": "4" },
                                                                    { "name": "y", "type": "ip", "value": "<IP_ADDRESS>" ]  ]
 [  { "name": "a",  "type": "string", "value":  "goodnight"},
      {name: "b", "type":"record",  value: [ {"name": "x", "type": "int32", "value": "4" },
                                                                    { "name": "y", "type": "ip", "value": "1<IP_ADDRESS>" ]  ]
 [  { "name": "a",  "type": "string", "value":  "graciie"},
      {name: "b", "type":"record",  value: [ {"name": "x", "type": "int32", "value": "4" },
                                                                    { "name": "y", "type": "ip", "value": "1<IP_ADDRESS>" ]  ]

Perhaps there is a better approach than this that conveys the data as "tidy data"?  And for existing tooling/apps that expect richly structured tidy data in json format?  I'm not sure how to head in this direction.
Another observation is that a trivial javascript function in the client could zip zjson into the latter form.  Whether the backend supports the zipped format or the client does the zipping is pretty trivial in terms of implementation.
So whether the client zips or the backend zips, the data is informationally equivalent and we're talking about ergonomics and ease of use of the API.  Perhaps we do both on the backend?  They are so close together and pretty trivial to do both.
Another point is that if we are strategically attempting to make our client understand richly structured data (at the intersecton of semi-structured json and schema-rigid tables) then I would think it would be important that the client have abstractions that understood the record type as a first-class entity and the grouping of records by type, etc.  As the client gets more complicated and more aware of the richly structured data types, it seems like this would be helpful.  And if so, and if we switched to the zipped format only, then there would need to logic to detect the type signature of each zipped record so that they could be type collated in this fashion.
Whether or not we change this approach, I noticed there is one glaring problem in the current zjson design/implementation: type definitions for zng records are nicely decomposed into recursive definitions (as above) but type definitions for zng arrays, zng sets, and zng unions must be parsed from the zng type string. This was just a poor implementation decision and reflects our lack of clarity a year ago when this code was written.  Also, it appears that type aliases are simply dropped in zjson (aliases are important as people add type extensions with semantics defined outside the scope of zng and want client support for such things, as in the "logical types" of parquet and avro).
So the problem is with this:
#0:record[a:string,b:array[record[x:int32,y:int32]]]
0:[hello;[[2;3;]]]
0:[world;[[4;5;][6;7;]]]

which is currently encoded in zjson as follows:
{
  "id": 25,
  "type": [
    {
      "name": "a",
      "type": "string"
    },
    {
      "name": "b",
      "type": "array[record[x:int32,y:int32]]"
    }
  ],
  "values": [ "hello", [  [ "2", "3" ]  ]  ]
}
{
  "id": 25,
  "values": [  "world", [ [ "4", "5" ],  [ "6", "7" ] ] ]
}

Here, the type string array[record[x:int32,y:int32]] would have to be parsed and decoded by the client.  Yuck!  We got this design right for the record type above but not for array, set, and union.  A fix could be to represent these container types with JSON object instead of string as follows:
{
  "id": 25,
  "type": { "record", "fields": [
    {
      "name": "a",
      "type": "string"
    },
    {
      "name": "b",
      "type": { "type:": "array", "of": { "type": "record", "fields": [ { "name":"x", "type":"int32" }, { "name": "y", "type":"int32" } ] } }
    }
   }
 ...

The question also came up as whether we should have a single request-response API endpoint without streaming ndjson, though this seems like an orthogonal point.   The advantages of streaming are:

the app can get progress updates of a long running query and report to the user
the app can get warnings from a query and report to a user in a timely fashion allowing them to abort
flow-control works from the client consuming a large streamed results all the way back to the scanner reading the data from storage or perhaps from other work nodes over the network (for example if you suspend zapi, all the right things happen and no one runs out of memory)
the receiver-driven data flow model works well with the streaming model as we don't have a model in the current implementation to stop a query at page boundaries and resume the query when the client asks for the next page
streaming queries are needed when the results won't fit in memory and, for example, the client is streaming the results to local disk.

That said, for ergonomics again, it could make sense to have a simple request/response search endpoint (in addition to the streaming endpoint) that wraps all the results, warnings, stats etc into a single json response making it easy to do simple queries with existing tooling.  This would be in addition to retaining the streaming search endpoint.  This request/response endpoint could orthogonally offer zipped or regular json encoding as discussed above.  The same behavior could be implemented on the client-side behind a client API (e.g., inside of zealot or inside of a zq python module).

@jameskerr if zealout had a zng.record class then it wouldn't matter if the data came zipped up or not, and you could just query methods on the record to get at fields, types, do introspection, etc.  You could also ask zealot to give zng.records oor vanilla javascript objects following the ndjson format if the client didn't care about the richness of zng.  (Of course the brim client cares about the rich data types but maybe other users of the zealot library wouldn't.)

I haven't heard any comments or follow-up on this.  My proposal is to create new tasks for the following:

fix the zjson type encoding problem for array/sets/union and make sure zealot can either handle it or make a brim task to coorodniate with this backend change
change zqd to return vanilla ndjson instead of zjson when format=ndjson in the search request and add an option to choose between encapsulating the ndjson in the streaming protocol or sending ndjson as raw lines per application/x-ndjson
add an option to zqd to give the entire results/stats/etc as a single json reply (good for curl | jq), i.e., not ndjson

@mccanne sorry for the radio silence. Thanks for revisiting the zjson format again. I think your proposals all make sense. I forgot about the array/set/union encoding issues, so I'm glad those were discovered again and issues made.
I think the current zjson model is great for all the reasons you've mentioned above. These new ergonomic changes will make the data even more portable.
As a user, if I want rich types, streaming responses, and no-bloat payloads, I download a library (zealot, python, zapi) and I'm on my way. If I want a quick and dirty fetch to do an ad hoc chart on https://observablehq.com/, codepen, or curl | jq, then I am able to do that too. I dig it.

@jameskerr Cool!  Thanks for closing the loop here.

I've confirmed with @mccanne that this issue has served its purpose because it led to the creation of #1273 and the issues below it. Closing this one.


"Save and Exit"/"Publish" actions in wizard

Description

PR adds more functionality to the footer of the wizard with the "Save and Exit" and "Publish" buttons. "Cancel" was also edited as I noticed an unnecessary alert pop-up with it in the Wizard frontpage step.
Related issues

Fixes #66
Type of change

[x] New feature (non-breaking change which adds functionality)

Changes Made

modified save/publish alerts
modified the conditionals for the actions of cancel/save/exist buttons
added a patch operator to set folder published key to true

Testing

[x] Needs testing (start an issue or do a follow up PR about it)

Mentions

Should this be PR to develop branch instead of master? How did we wanted this to flow? From https://github.com/CSCfi/metadata-submitter-frontend/blob/master/CONTRIBUTING.md#submitting-changes it is develop?

Should this be PR to develop branch instead of master? How did we wanted this to flow? From https://github.com/CSCfi/metadata-submitter-frontend/blob/master/CONTRIBUTING.md#submitting-changes it is develop?

Oh yea, definitely.

Also as pointed out by @saulipurhonen. Publishing here errors with: "detail: "Request contains '/published' key that cannot be updated to folders."
This is because of this line in the backend:
https://github.com/CSCfi/metadata-submitter/blob/b928909fbfc7635f711703e2999cb2539c81a33a/metadata_backend/api/handlers.py#L358
The line should be changed to: allowed_paths = ["/name", "/description", "/metadataObjects", "/published"] to work

@csc-jm @saulipurhonen we can enable published there, but would be nice to keep track of this workflow.
We did remove it for a reason just don't remember why. I will enable it.

Tested with "/published" path on backend and seems to work great.


Indicator Performance Tracking Table (IPTT) Layouts

There are two variations of the Indicator Performance Tracking Table (IPTT): Program Indicator Overview and Periodic Target Overview. Which report gets created depends on choices made on the IPTT landing page. (See #119).
Above the table

We always display the date range from the start date of the first time/target period needed to the end date of the last time/target period needed. (Periods will always start at the beginning of a month and end at the end of a month.)
We always display the program name.

IPTT -- Program Indicator Overview

This table organizes data by unit of time, and can show any combination of program indicators, regardless of their target frequencies. Note that with the exception of Life of Program, we only display Actuals -- no target or % met.
Sorting: The indicators should be sorted by Level, then Number, then Name.
IPTT -- Periodic Target Overview

This table organizes data by periodic target. It can only show indicators with one target frequency at a time.
Sorting: The order of single-indicator tables should use the same sorting defaults as the Program Indicator Overview report: Level, then Number, then Name.
Table column headers and contents

NO. -- Indicator number (if entered)
1.1. Links to Indicators page open to that indicator with data/evidence table exposed (link not included in exports)
INDICATOR
2.1. Indicator name -- links to Indicators page open to that indicator with data/evidence table exposed (link not included in exports)
Gear icon -- Pops open indicator configuration modal. (Icon not included in exports.)
LEVEL
UNIT OF MEASURE
CHANGE -- N/A, + or -
C / NC -- Cumulative or Non-cumulative
BASELINE -- Baseline value or N/A
LIFE OF PROGRAM
9.1. Target
9.2. Actual
9.3. % Met
PERIOD DATA (Quarter 1, Semi-annual period 2, Midline, etc.)
10.1. Target (Only in Periodic Target Overview report)
10.2. Actual (Both types of reports)
10.3. % Met (Only in Periodic Target Overview report)

UI detail
If it's not difficult to do, an on-hover row highlight color helps the eye travel across the table. The highlight color is #CCEFEE

@jennyhavah based on our conversation earlier today, I am not going to be developing the "EVENT-BASED" targets report and will take out that option from the dropdown.

For LoP indicators, can we display the phrase "Not applicable" in the C/NC column? It came across like there was missing data, when it's simply that we don't currently offer any C/NC choices for LoP target.

Sorting: The order of single-indicator tables should use the same sorting defaults as the Program Indicator Overview report: Level, then Number, then Name.

Doesn't appear to be implemented yet. The default sorting order is very important. (This is independent of any manual sorting which is in a different story.)

Change "Actuals" plural to "Actual" in the data column headers.

INDICATOR
2.1. Indicator name -- links to Indicators page open to that indicator with data/evidence table exposed (link not included in exports)

Not done yet. Is this easy or hard?

Please make the following UI tweaks in time for the demo. (It needs a lot of polishing, but the following items will make it look a lot less busy and broken up.)

The page title is not supposed to be on top of the filter panel, pushing it down. See the mockup: It is positioned above the date range.
Please change the background color of the page to #F5F5F5.
Please change the header row colors to #F5F5F5.
Please change the Life of Program columns to #F5F5F5.

When a baseline is marked N/A, then an "N/A" should show up in the Baseline column of the IPTT.

The title that shows up underneath the table is the "table caption" and used by screen readers.

That's fine. I noted the page title issue in a ticket Paul will work on.

I see the updated order in the IPTT table rows, but not the other places I mentioned. My top priority is to make the order of options in the filter panel consistent with that of the table itself.
If you like, I can move the other places to a separate ticket.

Let's create another story for updating the level order in other places. This issue is getting too long. I updated the level sorting issue in the IPTT table as well as in the filter form.

New ticket created to capture remaining level-related work: #296


Move to maintained suds fork now that suds-jurko is no longer maintained

Since the suds-jurko library that this project uses is no longer active and its Bitbucket repo is getting shutdown this year has any thought been given to moving to the actively maintained suds fork at https://github.com/suds-community/suds?
Bitbucket says:

June 1, 2020: users will not be able to use Mercurial features in Bitbucket or via its API and all Mercurial repositories will be removed.

Thanks for creating/maintaining such a useful library!

Yes.  I have switched to suds-community.  Look for it in the next release.
Thank you for pointing it out.

Thanks for the response. I'll keep an eye out for the next release.


[bug] line comments within stream comments

If line comments are present within stream comments /* and */ the result generated in wrong because line comments within are converted to /* and */
/*
// foo
*/

is turned into:
/*
/* foo */
*/

I'm not entirely clear on in what context this is occurring, but it doesn't reproduce under Sass or SCSS.


Add methods for detecting retriable and fatal Ably exceptions

Added helper methods that will enable us to detect fatal and retriable errors. This should allow us to implement appropriate retry mechanisms for certain Ably operations (like presence enter and leave).
Open questions:

should we only look at the code property or maybe also at the statusCode?
how should we classify exceptions that have codes that doesn't start with either 4 or 5?

I don't think there's any case that's not a 4xx or a 5xx. I think I'd prefer that there's an error logged in that case - even if statusCode == 0.

I don't think there's any case that's not a 4xx or a 5xx. I think I'd prefer that there's an error logged in that case - even if statusCode == 0.

@paddybyers I've created an issue for this (https://github.com/ably/ably-asset-tracking-android/issues/931) as this is a bit tricky to do right and seems a bit out of scope of this PR 😉


Create basic test cases for old components (retroactive technical debt)

Closed. New bugs have been opened


More-specific automator policies

As container solutions, I'd like more specific control over what the automator is doing, so that I can push images willy-nilly without having them all be deployed.
In more detail, they are pushing images to quay for feature-branches, which then the automator just deploys. They move a tag to indicate the latest master. Because of this, just deploying the "newest" available image is not good enough. It might be good if the automator had a "track tag" mode.

For more information, the setup that I like to use is:

Tag
Meaning

0.3.4
A tag, released by github

latest
The same as the latest released tag

snapshot
The latest commit on master

518ad9e0f9e45...
A particular commit

So as developers, we can test specific commits to make sure they do what we want them to do.
And ideally, we would then perform a release, which would create a new tagged version and move the latest tag to that. Then the staging environments would see that change and use the new image according to whatever rule is in place.
Rules could be:

Just latest
Just snapshot
Any new tag
NPM style tags. E.g. ~0.3.1 for any new patch, ^0.3.1 for any new minor or * for any major
Other style tags. E.g. 0.3.*, 0.*, * could mean the same thing as above.

Yeah, we really do the dumbest thing that works, which is to look for the most recent image. There are many completely reasonable practices that will trip that up :(
Following a tag might not be too difficult: the trick is that for Kubernetes to notice it's a new image, it needs to have a different name. But we could figure out which unique image name is aliased by the tag, and use that.
The version selectors would be pretty cool. I'm not sure how many people go from version number to version number, rather than commit to commit, mind you.

Based on @philwinder's comments above, I'd say we could do a "tag matcher".
e.g. fluxctl automate -s my-service --tag=~0.3.1 default being --tag=*. Then to track snapshot you could do fluxctl automate -s my-service --tag=snapshot, and it would only deploy new images with the matching tag.
Afaict that should cover roughly all use cases. Though, possibly a wild-card tag matcher would allow non-semver matching, e.g. master-* like we use.

Example use case: I want to automatically update to the most recent version of the mongo:3 container. But I sure as hell don't want to run the tag mongo:3-windowsservercore!! ;-)

Note, this would have prevented us from breaking dev flux by pushing a random image to quay repo.

Yes, I'm seriously for this. I think the ability to deploy semver tags would be the ideal vehicle for production CI
For development / other workflows a simple wildcard would be ideal master-* would be ideal.

TODO:

[ ] Core functionality (in #589)
[ ] CLI for updating/removing tag globs
[ ] Web UI for end users

 Flux: enable automated releases for versions other than "latest"


HiveError: Cannot read, unknown typeId: 34. Did you forget to register an adapter? when Removing an Adapter

Steps to Reproduce

Create custom Adapter MainClass, create custom Adapter Property which is a property of A. Register
Write data
Remove adapter B
Read old data.

Reading fails with message HiveError: Cannot read, unknown typeId: 34. Did you forget to register an adapter?
Based on the docs, reading old data after removing a property should be safe, but it throws
Found another possible solution in this comment
But we added Hive.ignoreTypeId(N) and now the error has changed, but it is still failing
The new error is RangeError: Not enough bytes available.
Code sample
// Initialization
Future<void> initializeHive() async {
  await Hive.initFlutter();

  Hive.registerAdapter(MainClassAdapter());
  Hive.registerAdapter(PropertyAdapter());
  ...
}

//  Classes definition
@HiveType(typeId: 1)
class MainClass {
  MainClass({
    required this.property,
  });
...
  @HiveField(N)
  final Property property;

  @HiveType(typeId: 2)
class Property {
  Property({required this.name});

  @HiveField(0)
  final String name;
}

// After writing some data with this structure,
// we removed Property from the registerAdapter calls, and form the MainClass definition

But we started getting HiveError: unknown typeId when reading old data

// Then, we tried with

// Initialization
Future<void> initializeHive() async {
  await Hive.initFlutter();

  Hive.registerAdapter(MainClassAdapter());
  Hive.ignoreTypeId(2);
  ...
}

// But now we get RangeError: Not enough bytes available.

Version

Platform: iOS, Android
Flutter version: 3.7.0
Hive version: hive: 2.2.3, hive_flutter: 1.1.0, hive_generator: 2.0.0

I think you should add adapterName like that

Could you elaborate on the need for that extra parameter for a solution here? It seems that if not set a default one is created, that is already unique to each adapter

Any update on this?!

deleteFromDisk() and open again

Try add ignoreTypeId before registerAdapter
Hive.ignoreTypeId(2);
Hive.registerAdapter(MainClassAdapter());

I'm also facing this issue with Flutter version 3.16.9

@minoesteban Did you find a solution finally ?


__perform had a bug where it was only ever calling the first event

bug fix
Description
We accept multiple events routing to multiple reflexes, but somehow a bug slipped in that was only catching the first event
Why should this be added
Obvious bug
Checklist

[x] My code follows the style guidelines of this project
[x] Checks (StandardRB & Prettier-Standard) are passing

Yes, I think so.
I actually just added documentation for multiple reflex events per element yesterday. Before that, you had to look at the code or assume it worked - and by the time you got there, it’d been broken for an indeterminate amount of time.
In short, this is not something a lot of folks have been asking for and just being able to support mouseenter+mouseleave is precisely perfect. I have observed an awkward tendency for developers to fall down rabbit holes where the laws of plausible vs worth it get temporarily suspended. 😺
I am happy to tell people that want three click handlers on one element to just use javascript.


'unwrap' depth of the let_value is unclear

let_value description mentions behavior similar  to future unwrap.  While the spec does not explicitly state depth of unwrapping, port of sudoku solver implies that the depth is not limited.

let_value takes a callable that returns a sender that sends T. It returns a sender that sends T.
There is one level of unwrapping there, if you view it as unwrapping. It is better thought of as delegating the completion to the completion of the nested work.
I think we have avoided anywhere in the spec using any heuristics about this. let_value always returns a sender, never a value.

@LeeHowes , did I get it wright that if the nested work returns some more work (i.e. returns a sender that sends another sender nut not a value), spec does not guarantee that to work ?

If the callback returns a sender that sends a sender, then let_value returns a sender that sends a sender.
There's no magic here. Senders aren't special, they are values. The T in Lee's comment can well just be a sender; the algorithm doesn't care.

@griwes , thanks for the clarification.
In this case sudoku solver (ported by @kirkshoop and mentioned in the spec) seems to rely  on the behavior of libunifex  version of let_value which is not part of the spec. Which seems to me controversial.

I'm sorry, I'm not seeing what you mean. Can you elaborate where you're seeing something that disagrees with the spec?
In the test code specifically, I can only see a single layer of unwrapping per let_value; note that there's two nested let_values, so the effect is that we "unwrap" twice.

I don't think that's inconsistent. Take the example code you linked to:
          let_value(just(42),
              [&](int& x) {
                return let_value(async(context, [&] { return x / 2; }), [&](int& y) {
                  return async(context, [&] { return x + y; });
                });
              })),

That is structurally similar to:
          let_value(just(42),
              [&](int& x) {
                return let_value(just(42), [&](int& y) {
                  return just(42);
                });
              })),

Which types as:
let_value(sender_of<int>, let_value(sender_of<int>, sender_of<int>(int&)))

That inner let_value channels the int so that is the same type as:
let_value(sender_of<int>, sender_of<int>(int&));

and the whole thing is a sender_of<int>. There's no double unwrapping going on is there? It's just that each let_value is a function from T to sender_of<T'>.

@anton-potapov is there still something unclear here after the explanations? Can we close this issue?


contentFiles not working for newer project formats

Do you want to request a feature or report a bug?
bug
What is the current behavior?
Currently when installing Akavache into new project format projects the AkavacheSqliteLinkerOverride files doesn't get added
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
Create a project using the PackageReference format and install Akavache (any version) you'll see the file doesn't get added
What is the expected behavior?
That AkavacheSqliteLinkerOverride  gets added and transformed
What is the motivation / use case for changing the behavior?
iOS links out SQLitePersistentBlobCache and then the static caches just use InMemoryCache
Which versions of Akavache, and which platform / OS are affected by this issue? Did this work in previous versions of Akavache? Please also test with the latest stable and snapshot (https://www.myget.org/feed/Akavache/package/nuget/Akavache) versions.
Anything that's new project format
Other information (e.g. stacktraces, related issues, suggestions how to fix)
I messed with this a little bit but couldn't figure out how to make it work...
I read through this
https://docs.microsoft.com/en-us/nuget/schema/nuspec#including-content-files
And added the content files in the specified location
/cs/any/AkavacheSqliteLinkerOverride.cs.pp
But it doesn't want to work
I read through this issue
https://github.com/NuGet/Home/issues/4803
and in there they reference
https://www.nuget.org/packages/ContentFilesExample/
Which also has a pp file included in the content that doesn't add into a destination project.  The best I was able to get working is if you don't use a pp file and just a cs file but even then I was only able to get it to add into netstandard projects and when I tried to add into a xamarin project the file didn't get copied into it.
It sounds like from the answers the solution is to add it via init.ps1 or a targets file but I might not be following the conversation right.

The plan going forward for this is to allow users to move files from under the contentFiles folder into their project through a gesture in Visual Studio. Currently the include of these files is written out to the auto generated nuget props file in the obj folder which allows them to be removed or overridden.
My suggestion for copying config XML files to the user project is to handle this in an init.ps1 script, or possibly in a .targets file.

I'm guessing this
https://github.com/dotnet/sdk/issues/1100
is a blocker for the pp files

Init.ps1 still works (this was the vector for that nuget typosquatting exploit) but it executes everytime a solution is opened. It only works on Windows and when using Visual Studio for Windows. The other PowerShell options such as install.ps1 no longer work.

The best workaround i've found so far is to just include a cs file (no pp) into  contentfiles/any/any
Then it'll get added into the netstandard projects at least but not the head projects.
we could probably add a content readme into
contentfiles/any/xamarinios10
contentfiles/any/monoandroid7
which I'm hoping work since contentfiles adds based on TFM
That just warns about it getting linked out with a sample of the files to add
On a side note this isn't a breaking change for the 6.0 release ... 5.0 has the same issue when interacting with the new project formats

The prayer to the linker gods was from a time before options like https://developer.xamarin.com/guides/cross-platform/advanced/custom_linking/ existed. Wonder if we can package up a linker configuration definition into a nuget package?

Yes; we can release without resolving this as long as we update the README up around the top with some instructions/link to this GitHub issue.

I haven't been able to find a way to package anything up :-/  this one seems a bit special because it's the whole assembly being linked out since the only interaction the library has outward is via reflection..   Like even just calling an empty init method into the assembly fixes it
Maybe contentFiles supports a linker build action and will include a file.. I'll try it in a few just to see :-x

I'm wondering if it's worth just punting on this whole idea of inserting the linked file... Mainly because the issue isn't really with SqlPersistentBlobCache getting linked out the issue is with the DLL just not being included at all because of the static analysis of nothing calling into the DLL..    If you create an empty INIT method inside Akavache.SqlLite3 that literally does nothing and an outside library called into it it has the same affect as the Linked in file.
Having to set the ApplicationName in order to use the static BlobCache properties is already init code that has to be added.    Plus all of this is only really relevant if you're just relying on the static Cache variables.
I feel like just adding some Init call into Akavache.SqlLite3 is more explicit and just kills a few birds with one stone... Something like
namespace Akavache.SqlLite3
public void InitSqlLiteStoreForGlobalCaches(string ApplicationName)
{
BlobCache.ApplicationName = ApplicationName
//maybe even stick the batteries call here and add some properties on the method to include their own batteries
}


client.UserId is always 0

I can't get the current userId because this value started showing 0 instead of the current user ID

client.UserId is non-zero:

after a successful login
or when resuming an existing session (where a user was previously successfully logged-in)