com.amazonaws.AWSCognitoIdentityErrorDomain error 0

I followed all the steps in the README for S3TransferUtilitySampleSwift but I am repeatedly getting the following error: Error: The operation couldn’t be completed. (com.amazonaws.AWSCognitoIdentityErrorDomain error 0.) for both uploadData function and downloadData functions.

Hello @jonnokim17
Could you please provide the full stack trace? Have you set up the IAM permissions correctly and allowed unauthenticated access?
Thanks,
Rohan

Below is the callstack I get when attempting todownload a file from my server's aws bucket. I am not using the upload image functionality within the app. What I want to achieve is to be able to download a file from my bucket securely, so that users of my app cannot have access the files. I understand that AWS encrypts/decrypts the files for free, but using a GET request in my app to retrieve the file would NOT prevent someone from downloading the files.. I was hoping using AWSS3TransferUtility would resolve the issue.
["0   S3TransferUtilitySampleSwift        0x0000000103fbb82b _TFFC28S3TransferUtilitySampleSwift22DownloadViewController5startFCSo8UIButtonT_U1_FGCSo7AWSTaskCSo32AWSS3TransferUtilityDownloadTask_GSqPs9AnyObject__ + 1035", "1   S3TransferUtilitySampleSwift        0x0000000103fbbb9e _TTRXFo_oGCSo7AWSTaskCSo32AWSS3TransferUtilityDownloadTask__oGSqPs9AnyObject___XFo_oGS_S0___iGSqP___ + 30", "2   S3TransferUtilitySampleSwift        0x0000000103fbbd41 _TPA__TTRXFo_oGCSo7AWSTaskCSo32AWSS3TransferUtilityDownloadTask__oGSqPs9AnyObject___XFo_oGS_S0___iGSqP___ + 81", "3   S3TransferUtilitySampleSwift        0x0000000103fbbd90 _TTRXFo_oGCSo7AWSTaskCSo32AWSS3TransferUtilityDownloadTask__iGSqP___XFdCb_dGS_S0___aGSqPs9AnyObject___ + 64", "4   AWSCore                             0x0000000104120b28 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 104", "5   AWSCore                             0x00000001040c350e __30+[AWSExecutor defaultExecutor]_block_invoke_2 + 190", "6   AWSCore                             0x00000001040c3ebe -[AWSExecutor execute:] + 94", "7   AWSCore                             0x00000001041210b2 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke.126 + 50", "8   AWSCore                             0x000000010412044d -[AWSTask runContinuations] + 765", "9   AWSCore                             0x000000010411fe1a -[AWSTask trySetError:] + 282", "10  AWSCore                             0x00000001041222e9 -[AWSTaskCompletionSource setError:] + 89", "11  AWSCore                             0x0000000104120dff __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke_2 + 271", "12  AWSCore                             0x0000000104120c31 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 369", "13  AWSCore                             0x00000001040c350e __30+[AWSExecutor defaultExecutor]_block_invoke_2 + 190", "14  AWSCore                             0x00000001040c3ebe -[AWSExecutor execute:] + 94", "15  AWSCore                             0x00000001041210b2 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke.126 + 50", "16  AWSCore                             0x000000010412044d -[AWSTask runContinuations] + 765", "17  AWSCore                             0x000000010411fe1a -[AWSTask trySetError:] + 282", "18  AWSCore                             0x00000001041222e9 -[AWSTaskCompletionSource setError:] + 89", "19  AWSCore                             0x0000000104120dff __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke_2 + 271", "20  AWSCore                             0x0000000104120c31 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 369", "21  AWSCore                             0x00000001040c350e __30+[AWSExecutor defaultExecutor]_block_invoke_2 + 190", "22  AWSCore                             0x00000001040c3ebe -[AWSExecutor execute:] + 94", "23  AWSCore                             0x00000001041210b2 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke.126 + 50", "24  AWSCore                             0x000000010412044d -[AWSTask runContinuations] + 765", "25  AWSCore                             0x000000010411fe1a -[AWSTask trySetError:] + 282", "26  AWSCore                             0x00000001041222e9 -[AWSTaskCompletionSource setError:] + 89", "27  AWSCore                             0x0000000104120dff __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke_2 + 271", "28  AWSCore                             0x0000000104120b28 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 104", "29  AWSCore                             0x00000001040c350e __30+[AWSExecutor defaultExecutor]_block_invoke_2 + 190", "30  AWSCore                             0x00000001040c3ebe -[AWSExecutor execute:] + 94", "31  AWSCore                             0x00000001041210b2 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke.126 + 50", "32  AWSCore                             0x000000010412044d -[AWSTask runContinuations] + 765", "33  AWSCore                             0x000000010411fe1a -[AWSTask trySetError:] + 282", "34  AWSCore                             0x00000001041222e9 -[AWSTaskCompletionSource setError:] + 89", "35  AWSCore                             0x0000000104120dff __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke_2 + 271", "36  AWSCore                             0x0000000104120c31 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 369", "37  AWSCore                             0x00000001040c350e __30+[AWSExecutor defaultExecutor]_block_invoke_2 + 190", "38  AWSCore                             0x00000001040c3ebe -[AWSExecutor execute:] + 94", "39  AWSCore                             0x00000001041210b2 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke.126 + 50", "40  AWSCore                             0x000000010412044d -[AWSTask runContinuations] + 765", "41  AWSCore                             0x000000010411fe1a -[AWSTask trySetError:] + 282", "42  AWSCore                             0x00000001041222e9 -[AWSTaskCompletionSource setError:] + 89", "43  AWSCore                             0x0000000104120dff __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke_2 + 271", "44  AWSCore                             0x0000000104120c31 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 369", "45  AWSCore                             0x00000001040c350e __30+[AWSExecutor defaultExecutor]_block_invoke_2 + 190", "46  AWSCore                             0x00000001040c3ebe -[AWSExecutor execute:] + 94", "47  AWSCore                             0x00000001041210b2 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke.126 + 50", "48  AWSCore                             0x000000010412044d -[AWSTask runContinuations] + 765", "49  AWSCore                             0x000000010411fe1a -[AWSTask trySetError:] + 282", "50  AWSCore                             0x00000001041222e9 -[AWSTaskCompletionSource setError:] + 89", "51  AWSCore                             0x0000000104120dff __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke_2 + 271", "52  AWSCore                             0x0000000104120c31 __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke + 369", "53  Foundation                          0x00000001044f2237 __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 7", "54  Foundation                          0x00000001044f1f3b -[NSBlockOperation main] + 101", "55  Foundation                          0x00000001044f06f7 -[__NSOperationInternal _start:] + 627", "56  Foundation                          0x00000001044ec47c __NSOQSchedule_f + 198", "57  libdispatch.dylib                   0x000000010807e05c _dispatch_client_callout + 8", "58  libdispatch.dylib                   0x000000010805c94f _dispatch_queue_serial_drain + 221", "59  libdispatch.dylib                   0x000000010805d669 _dispatch_queue_invoke + 1084", "60  libdispatch.dylib                   0x000000010805fec4 _dispatch_root_queue_drain + 634", "61  libdispatch.dylib                   0x000000010805fbef _dispatch_worker_thread3 + 123", "62  libsystem_pthread.dylib             0x00000001084e35a2 _pthread_wqthread + 1299", "63  libsystem_pthread.dylib             0x00000001084e307d start_wqthread + 13"]
I'm not sure what IAM Permissions is.. but I did set public access Read object to YES. Thanks in advance!

You need to give permissions for the Cognito Unauth role for the cognito pool id which you are using so that you can upload or download. Have you set the correct configuration in info.plist?
Thanks,
Rohan

Thanks for the reply Rohan.
Here's a screenshot on my info.plist. Am I missing something?

Here's a setup for Identity Pool via AWS console:

hey @rohandubal - do you have any updates on this? Thank you.

Hi @jonnokim17,
Have you updated the bucket and keys in Constants.swift?
https://github.com/awslabs/aws-sdk-ios-samples/blob/master/S3TransferUtility-Sample/Swift/S3BackgroundTransferSampleSwift/Constants.swift
This section covers the steps for setting up access.
http://docs.aws.amazon.com/mobile/sdkforios/developerguide/cognito-auth-aws-identity-for-ios.html

seeing this as well

There are two different things here: identityPoolId and poolId. The poolId you have currently is actually your identityPoolId. Your poolId is shorter, something like this: us-east-1_*******. About that length. Find the poolId by going to the General Settings of your Users Pool in Cognito. It's at the top. Identity pool ID is given in the Federated Identities part of Cognito.

This issue has been closed because of inactivity. Please open a new issue in https://github.com/aws/aws-sdk-ios/issues if you are still encountering problems.


Display Back Matter media types in a human-readable format

As a user involved in the governance, risk, and compliance of systems and components exploring controls, I want the media type of the back matter resources to be displayed in a human readable format when possible.
Acceptance Criteria

Back matter media types have human-readable names for the common types (OSCAL documents, PDFs, HTML document, etc)
If the media type doesn't match to something well-known, the exact value from the media-type field is displayed
If the media type isn't specified, whichever heuristic is used to determine a media type results in a human readable name (such as using the file extension)
Product Owner agrees with the list of recognized types.

Implementation Notes

Per the specification, this is supposed to be an IANA registered media type; however, it seems that some things don't comply with this

Our default catalog seems to not specify media-type for many documents; however we seem to detect many of them as PDFs or "Unknown"
Some of the media types, such as application/oscal.content+json is not formally registered as an IANA media type

Find an (the?) authoritative source for media types and recognize those.
Enumerate the official list of recognized types and associate with human-friendly names. Consider icons (not required)

Core list plus OSCAL extensions.

@brian-easyd We have an awkward issue with this because of the very small number of back matter resources in existing documents that specify a media-type. So in most cases, the viewer today falls back to Unknown or a guess based off the file extension.
Determining Media Type
graph TD
    A([Displaying Media Type]) --> B{Is `media-type` present?}
    B -->|Present| C
    B -->|Absent| D[Use file extension to guess media type]
    D --> E{Can media type\n be determine from\nextension}
    E -->|Media Type Found| C[Convert to friendly-name\nusing lookup]
    E -->|Not Found| Fallback([Display 'Unknown Content'])
    C --> Z([Display friendly name for\nmedia-type])

Friendly names for MIME types
The IANA registrations do not necessarily contain a friendly name. This lists how we will represent specific media types and classes of media types in the viewer.
image/

Name
Display in Viewer

gif
GIF

heic
HEIC Image[^1]

jpeg
JPEC

png
PNG

tiff
TIFF

vnd.adobe.photoshop
Photoshop PSD

svg+xml
SVG

Fallback: "Image"
application/

Name
Display in Viewer

json
JSON

oscal.content+json
OSCAL (JSON)

xml[^2]
XML

oscal.content+xml
OSCAL (XML)

octet-stream
Binary data

msword
Microsoft Word (.doc)

application/vnd.openxmlformats-officedocument.wordprocessingml.document
Microsoft Word (.docx)

vnd.ms-powerpoint
Microsoft PowerPoint (.ppt)

vnd.openxmlformats-officedocument.presentationml.presentation
Microsoft PowerPoint (.pptx)

vnd.ms-excel
Microsoft Excel (.xls)

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Microsoft Excel (.xlsx)

pdf
PDF

pem-certificate-chain
PEM Chain

pkcs7-signature
PKCS7 Signature

pgp-signature
PGP Signature

ogg
OGG

rtf
Rich Text (RTF)

Fallback: "Binary Data"
text/

Name
Display in Viewer

html
HTML

xml[^2]
XML

oscal.content+xml
OSCAL (XML)

Fallback: "Text Content"
example/
We will always report as "Unknown Type". Should never appear in real-world documents.
No special handling for specific types

Registry
Display in viewer

audio/*
Audio

font/*
Font

message/*
Message

model/*
Model

video/*
Video

[^1]: Using "Image" is intentional; while HEIC is becoming a more common format, many users do not recognize it as an image type
[^2]: XML appears in both text/ and application/ but application/ is recommended. We will display both as "XML" (and the corresponding OSCAL extension) because they are functionally the same to users.

@kylelaker first I want to say that I love this level of planning! Overall this is great and very near what I had in mind. I'd like to offer the following feedback:

In the flos diagram, where media-type is present, we then have the question of "Recognized" (Y/N). If Recognized, the rest of your flow is valid. If not recognized, we should simply display "Not Recognized".
I'd actually like to see all images represented as "Image (GIF)", "Image (JPG)", etc. Typically the user will care more about whether it's an image (yes or no), and won't care about the image format.
When we get into AR, audio and video are potentially valid media types, and should at least be displayed as "Audio" or "Video" with the type in parens if know -similar to audio above. Some auditors prefer to record interviews (although often the org or individual staff members don't love that), and may take a video of a screen interaction or a social engineering test.

Otherwise, I think this is great for an MVP. I eventually see our tool recognizing output from other tools, like Nessus scanner output. But we can address those down the road.


Allow driver mode to be supplied via command line

If this looks right, could you please suggest how to test this?

It's on the right track. I'm going to merge it and make a few changes from there (easier than modifying the fork!).


elastic search container doesn't work right after start

After starting qps-infra elk container doesn't work as expected. As a hotfix we should:

run any test
restart elk

so need a way to determine valid solution to register exchange in rabbitmq right after startup. It should resolve elk issue

Update username, password in definitions.json for RabbitMQ if you are not using default qpsdemo/qpsdemo user


MQTT camera displays previous image received

Home Assistant release with the issue:

0.90.1
Last working Home Assistant release (if known):
N/A
Operating environment (Hass.io/Docker/Windows/etc.):

Hass.io on Debian
Component/platform:

MQTT Camera
https://www.home-assistant.io/components/camera.mqtt/
Description of problem:
The MQTT camera component shows the image received previous to the image last sent from the MQTT-client.
I initially noticed this behavior from a camera but could easily replicate the issue by using mosquitto_pub to send two different images like this with the following results:
mosquitto_pub -h hassio -p 1883 -u USERNAME -P PASS -t whatever/topic/here -f 1.png
Result: No image shown in HA
mosquitto_pub -h hassio -p 1883 -u USERNAME -P PASS -t whatever/topic/here -f 2.png
Result: 1.png shown in HA
mosquitto_pub -h hassio -p 1883 -u USERNAME -P PASS -t whatever/topic/here -f 1.png
Result: 2.png shown in HA
...and so on
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
camera:
  - platform: mqtt
    name: snapshots
    topic: whatever/topic/here

Traceback (if applicable):

Additional information:

I tried to reproduce this error. Can you please try this again but refresh in your browser after publishing a new message/image? Does this show the correct image?

I missed a detail when describing the issue. I had only tested this from the dev-states (more-info dialog).
When displaying the camera as an entity in the standard lovelace view it works fine - the image refreshed to the correct one after whatever set interval the frontend use.
If the camera is viewed in a more-info dialog however, the behavior is like described in the original post.
I simultaneously looked at the camera as an entity in lovelace and in the more-info dialog, and they were out of sync with each other.


build(snap): update snap build pipeline patch

This change updates the snap build pipeline optimization
patch which when used in combination with the following
command:
$ snapcraft prime --use-lxd
...significantly reduces the snap build time when run
from the Jenkins pipeline. It accomplishes this by removing
the non-edgex-go parts from snapcraft.yaml. Additionally,
the 'snapcraft prime' command causes the snapcraft to prime
the snap, but stop before the actual binary snap package is
created.
Note - once this patch is actually put into place in
the pipeline builds, future updates to this patch will
accompany any future changes to snap/snapcraft.yaml.
Signed-off-by: Tony Espy<EMAIL_ADDRESS>PR Checklist
Please check if your PR fulfills the following requirements:

[NA] Tests for the changes have been added (for bug fixes / features)
[NA] Docs have been added / updated (for bug fixes / features)

If your build fails due to your commit message not passing the build checks, please review the guidelines here: https://github.com/edgexfoundry/edgex-go/blob/master/.github/Contributing.md.
What is the current behavior?
This PR adds a patch which can be used to optimize the Jenkins pipeline builds (see commit message for more details).
Does this PR introduce a breaking change?

[ ] Yes
[x] No

New Imports

[ ] Yes
[x] No

I'm closing this as I worked with @bnevis-i to update the patch in his recent secrets-config PR:
https://github.com/edgexfoundry/edgex-go/pull/2808


Need to make an Architectural choice to decide, whether to define Store Policy and expose it to consumers

Need: Endorsements are of different nature, Integrity or Integrity and Privacy Protected, hence each type of Endorsement may need a different protection, Hence the need for different type of stores. This requires a Policy to be defined.

Policy layer has a bearing on the API defined in the vts.proto.
Should we have a single API AddEndorsementRequest() where the API.

Issue now tracked by:
https://github.com/veraison/services/issues/86


[auto-merge] branch-22.04 to branch-22.06 [skip ci] [bot]

auto-merge triggered by github actions on branch-22.04 to create a PR keeping branch-22.06 up-to-date. If this PR is unable to be merged due to conflicts, it will remain open until manually fix.

FAILURE - Unable to auto-merge. Manual operation is required.
{'message': 'Pull Request is not mergeable', 'documentation_url': 'https://docs.github.com/rest/reference/pulls#merge-a-pull-request'}

Please use the following steps to fix the merge conflicts manually:
# Assume upstream is NVIDIA/spark-rapids remote
git fetch upstream branch-22.04 branch-22.06
git checkout -b fix-auto-merge-conflict-5192 upstream/branch-22.06
git merge upstream/branch-22.04
# Fix any merge conflicts caused by this merge
git commit -am "Merge branch-22.04 into branch-22.06"
git push <personal fork> fix-auto-merge-conflict-5192
# Open a PR targets NVIDIA/spark-rapids branch-22.06

IMPORTANT: Before merging this PR, be sure to change the merging strategy to Create a merge commit (repo admin only).
Once this PR is merged, the auto-merge PR should automatically be closed since it contains the same commit hashes

I will take care of the conflict


cmd/cue: improve error messages for get go

Originally opened by @myitcv in https://github.com/cuelang/cue/issues/701
As a follow up to #621 (and in preparation for #646) we should ensure we have good error messages for cue get go:

in case a dependency is not available/cannot be resolved (i.e. properly surfacing go/packages errors)
in case a target package cannot be properly type checked. Including:

syntax errors
type checking errors

Dropping the now-defunct v0.4.x milestone from this issue, but leaving the zGarden label such that we come round to considering what milestone this should sit in.


Unable to edit or replace deployment 'userAssignedIdentity': previous deployment from '10/16/2024 4:36:41 AM' is still active (expiration time is '10/23/2024 4:36:39 AM'). Please see https://aka.ms/arm-deploy-resources for usage details.

Hi Team,
I am trying to deploy azure service of module managed identity. I have this structure mentioned below. Getting deployment error "Unable to edit or replace deployment 'userAssignedIdentity': previous deployment from '10/16/2024 4:36:41 AM' is still active (expiration time is '10/23/2024 4:36:39 AM'). Please see https://aka.ms/arm-deploy-resources for usage details." when deploying on Azure Portal. Can you please help to fix the issue to make it work with this modular approach structure and any changes if i need to make??
// **managedidentity.bicep** file which calls avm module declared in object

@description('Specifies the inpit values for ManagedIdentity')
param managedidentity object

module userAssignedIdentity 'br/public:avm/res/managed-identity/user-assigned-identity:0.4.0' = {
  name: managedidentity.deploymentName
    params: {
    name: managedidentity.name
    location: managedidentity.location
    lock: {
      kind: managedidentity.lock.locklevel
      name: managedidentity.lock.lockname
    }
    tags: managedidentity.tags
  }
}

// **main.bicep file =**
@description('Specifies the inpit values for ManagedIdentity')
param managedidentity object
// Creation of ManagedIdentity

module userAssignedIdentity 'managedidentity.bicep' = {
  name: managedidentity.deploymentName
  scope: resourceGroup(managedidentity.resourceGroup)
    params: {
   managedidentity: managedidentity
  }
}

// **.bicepparam file = Input values**
using ./main.bicep file

param managedidentity = {
  deploymentname: 'userAssignedIdentity'
  name: 'mi-sc-1991-d-o-011'
   resourceGroup: 'module'
  location: 'Sweden Central'
  tags: {
    Organization: 'Asia'
    Environment: 'dev'
  }
  lock:{
     lockname: 'lock-sc-1897-d-o-02'
     locklevel: 'CanNotDelete'
  }
}

Hey @Mrunamii218,
the issue is this line name: managedidentity.deploymentName in each module deployment. This name is given to the deployment in Azure, but because there is no dependency in between the 2 deployments that use this name, they're executed at the same time - hence leading to the conflict.
You should not have the issue if you use 2 different deployment names.


collect() and union() skipping null values

Repro is with Zed commit 4dc6236.
While helping a community Slack user I bumped into the following surprising result:
$ zq -version
Version: v1.14.0-2-g4dc62369

$ echo '1 2 null 4' | zq -z 'collect(this)' -
[1,2,4]

I would have expected the null value to still be in the array created.
Presumably due to the same effect, here collect() is not even returning an array.
$ echo 'null null' | zq -z 'collect(this)' -
null

union() has similar behaviors.
$ echo '1 2 null 4' | zq -z 'union(this)' -
|[1,2,4]|

$ echo 'null null' | zq -z 'union(this)' -
null

#5041 was opened to change the behavior so collect() would preserve these values, but then we discussed it as a team and have decided to hold off on merging anything yet. Some things that were touched on in the discussion:

There was consensus that if we change this for collect() we should do the same for union() so they're consistent.

One reason for hesitation on merging #5041 is that this change is significant enough that it might break some programs, e.g., those of a particular community zync user that has some particularly large/complex Zed that relies heavily on collect() at times. If Zed were more mature it would probably be appropriate to do a careful/slow transition, e.g., maintain the current behavior with some kind of flag that becomes a new default, then surface a warning to users whose programs are relying on that behavior because we intend to swap the default in a future version, since this would give them time to understand and react to the change. Instead we discussed helping them to audit their programs for possible exposure and also giving a heads-up to the wider community via Slack before switching the behavior. But then we identified other behaviors with aggregate functions we might want to consider, so we concluded that we should gather up our thoughts on all those changes and then figure out a transition plan.

As an example of "other behaviors", @mccanne cited avg() as an example, since here null values are currently treated as "not present", whereas if other aggregate functions like collect() and union() are changed to start preserving the null values, users might now expect a different behavior from avg() too, e.g., treating them as "present" and implying some meaningful value such as 0, which would change the result.

$ zq -version
Version: v1.14.0-2-g4dc62369

$ echo '1 3 5 7' | zq -z 'avg(this)' -
4.

$ echo '1 null 3 5 7' | zq -z 'avg(this)' -
4.

@mattnibs pointed out that if we did toggle the behavior, it would be easy for users to get back the current behavior using the where clause, such as can be seen with his branch from #5041:

$ zq -version
Version: v1.14.0-3-ga33df054

$ echo '1 2 null 4' | zq -z 'collect(this)' -
[1,2,null,4]([(int64,null)])

$ echo '1 2 null 4' | zq -z 'collect(this) where this != null' -
[1,2,4]

In the time since this issue was opened, the language has been evolving into SuperSQL (currently at commit 4084e01) and so mimicking SQL becomes our typical starting position when picking behaviors. In this regard, it does seem like preserving the null values would indeed still be the way to go.
For instance, we don't yet have support for the SQL aggregate function by the name ARRAY_AGG(), but this is what provides the functional equivalent to what we've traditionally done with collect(). As we can see using DuckDB to run through the same examples shown above, the null values are preserved.
$ duckdb --version
v1.1.3 19864453f7

$ cat data.json
{"foo":1}
{"foo":2}
{"foo":null}
{"foo":4}

$ duckdb -c 'select array_agg(foo) from data.json'
┌─────────────────┐
│ array_agg(foo)  │
│     int64[]     │
├─────────────────┤
│ [1, 2, NULL, 4] │
└─────────────────┘

$ cat nulls.json
{"foo":null}
{"foo":null}

$ duckdb -c 'select array_agg(foo) from nulls.json'
┌────────────────┐
│ array_agg(foo) │
│     json[]     │
├────────────────┤
│ [NULL, NULL]   │
└────────────────┘

As for the example with avg() noted above, it does seem to skip them there, i.e., the calculation is still based on only the actual numeric values shown.
$ cat numbers.json
{"foo": 1}
{"foo": null}
{"foo": 3}
{"foo": 5}
{"foo": 7}

$ duckdb -c 'select avg(foo) from numbers.json '
┌──────────┐
│ avg(foo) │
│  double  │
├──────────┤
│      4.0 │
└──────────┘

As for the concerns about whether this change could break users' programs, we're already making a lot of other significant changes as part of the transition to SQL, so the direction things are headed we're likely going to just have to publish some docs to help guide users through modifying their programs, so this would likely just be covered as part of that.


cloning flutter projects problems

Has anyone ever cloned flutter_architecture_samples => "https://github.com/brianegan/flutter_architecture_samples" and was able finally to go through it and his mind still fine ???
please tell me how, the "package get" fails due to versions incompatibility of git and the driver, I tried switching to "channel dev" as a solution from a closed issue about the same problem but did not work also

@goondeal It's not a flutter repo, and I think the problems are in particular example you trying to run. What exactly are you doing? Please provide your flutter doctor -v output. What is the fail log?
Also, I would advise you to post it to StackOverflow as well as help there may come faster.

Hello @goondeal !
Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thank you for your contribution.
P.S. If anyone still experience the same problem, please file a new issue with the information required in the template.
All system setups can be slightly different so it’s always better to open new issues and reference related issues.


Visibility modes, closes #322

Added visibility modes toggle buttons on deployment
Choices added for the modes public, auth and private

The choices are ordered: more public --> more private, I figured that auth proxy is more public than private since auth proxy still allows external traffic if it is authenticated while private only allows internal traffic.
Tooltips explaining what the different modes do

Let me know if i have misunderstood the modes or should rewrite the descriptions, this is how i figured they work:

Mode
Description

public
Your deployment is publicly accessible on the internet.

auth
Your deployment is private and reachable on the internet through an authentication proxy, this limits the allowed traffic.

private
Your deployment is private and not reachable on the internet. Only internal traffic is allowed in (traffic between deployments).

Amazing work! Tooltips look perfect 👍 LGTM


Maximum supported frequency of riscv

Hi,
Could you please let me know what maximum supported frequency of riscv in Xilinx device(like artix or virtex ultrascale, ultrascale plus)?
Thank you so much,
Duc

Hi,
Here are some artix7 numbers :
https://spinalhdl.github.io/NaxRiscv-Rtd/main/NaxRiscv/performance/index.html
I don't have any other number :)
Regards
Charles


I have already downloaded the CUDA version of torch, why does it automatically uninstall my CUDA when I pip install autogluon and turn it into the CPU version?

Bug Report Checklist

[ ] I provided code that demonstrates a minimal reproducible example.
[ ] I confirmed bug exists on the latest mainline of AutoGluon via source install.
[ ] I confirmed bug exists on the latest stable version of AutoGluon.

Describe the bug

Expected behavior

To Reproduce

Screenshots / Logs

Installed Versions

# Replace this code with the output of the following:
from autogluon.core.utils import show_versions
show_versions()

Hi @psv666, thank you for opening the issue. The latest version of AutoGluon requires PyTorch v2.1, while your previous installation was v2.0. Therefore, the old PyTorch version (2.0) needs to be uninstalled before the new version (2.1) required by AutoGluon can be installed.


Copyright details not filled in the LICENSE.

At line 190 in LICENSE copyright details have not been filled in.

That's intentional. It's part of the "APPENDIX: How to apply the Apache License to your work." section of the license with accompanying instructions just above it:

attach the following boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include the brackets!)

Apache's own LICENSE file for httpd has the same thing: http://svn.apache.org/viewvc/httpd/httpd/trunk/LICENSE?view=markup.


HHH-9419: Remove after get with optimistic lock fails

Do not perform a version check on a non-existing entry when entry was removed in the same session after being optimistically locked
Unit test provided

Cherry-picked and pushed.
Thanks!
Gail


Safari Extension

Mostly requires rewriting chrome's manifest.json to safari format.

Are there any new plans for the Safari Extension? :/

Still only 3%?

I'm not going to pay apple for a dev license and an apple device to publish there.
Safari add-on is not happening.

Thanks for the clarity, totally understand your point of view.
Perhaps a consideration: putting a price on the extension could earn you back the dev license + time invested. You'd need a significant amount of Safari users though..

... a lot more than 3%.


support large file upload and download

we have a limitation on event payload sizes to upload and download. provide an option to split and upload large file as chunks.

large file support available now.
if event size is more than 1Mb it will considered as large file.


fix: build SVT_AV1 for aarch64 or arm64

What do you think @fdintino

would be great if @fdintino could review this PR.
I'm facing errors installing pillow-avif-plugin with arch arm64

Second this - @fdintino should review this PR, would be great if it makes it possible to install piloow-avif-plugin in ARM servers.

I'd also really like to see this merged--I'll be deploying to gravitron and need arm64. Thank you!

Thanks!


haskellPackages.dhall-lsp-server: Apply patch relaxing lens bound

Description of changes

Things done
Fixing the dhall-lsp-server build by applying a patch from an upstream PR relaxing the restrictive upper bound on lens.

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

@dalpd Hey, sorry for not getting to this for a while.
Would you be able to rebase this on the current haskell-updates branch and make sure it is still correctly compiling?
Also, if you'd really like to make sure that dhall-lsp-server continues to work, it'd be great if you could add yourself as a maintainer.  That way you'd get pinged if it ever starts breaking again:
https://github.com/NixOS/nixpkgs/blob/04ba740f89b23001077df136c216a885371533ad/pkgs/development/haskell-modules/configuration-hackage2nix/main.yaml#L148-L169

@ofborg build dhall-lsp-server

Alright, looks like this is building on Linux at least, thanks!
cc @Gabriella439 about this as well just in case


Fix incorrect background jobs number

The jobs -l command displays a "pwd now :..." line when a job was launched from a different directory than the current one that cause powerlevel9k to display an incorrect number of background jobs. This change fixes it.

Hey @cbourgeois - Great catch, and thanks for submitting a fix! We really appreciate your contribution!


Rename box does not respect editor font

Try to rename any variable in TS, notice that the rename box is not respecting the font in the editor.
This is like this since forever but not sure if there is an issue already for this

Unsure if it should be like that at all? @bgashler1 Ideas? Answers?

Comparing this to search/replace I don't think we should do it. Please reopen when and explain why this shouldn't be like search/replace or quick outline/

Arguments that this should respect the code font:

lives more connected to the source code than search replace
always has the variable name as the content
visually not appeleaing to have two different fonts so close to each other

Thinking about it, I would vote that search / replace and quick outline also respect editor fonts. What is the reasoning for those to have different fonts?
QuickOpen makes sense to have different font when displaying titles of files, but not when displaying variable names imho.
Feel free to leave closed if you do not think the arguments are strong enough.
@bpasero @stevencl opinions welcome

I have to admit I had never really noticed that before. I don't feel very strongly about it. Consistency is good though, so if it was a simple, inexpensive change to make I would be for it. If it was to change, I'd expect it to change in search too.

Sorry I let this one slip by me in not replying quickly.  I also agree with Steven that consistency is key.
I tend to lean towards not having the monospaced code font in the input box.  The reason why is that the input box is a UI overlay, and UI has its own non-monospaced font.  We're not editing inline—we're merely entering something as input to the overlay which then makes the change on our behalf.

Then it makes sense to keep this closed, thanks for feedback.

so how did this change then?

I just saw it and I love it. It always felt weird to me that it was non-monospace, regardless of the other input boxes. The same goes for the find & replace.

I did not change it, but I do like it. Thanks misterious contributor

Sorry, reverted. IMO consistency goes over personal preference

Consistency goes both ways.
You seem to be on the side of widget consistency: most text input widgets show non-monospace font, so this one should to. Examples are in the search and git viewlets' input boxes, quick open, find and replace. Counter examples are the debug evaluate input box and watch expressions, both of which use monospace font, imo correctly.
I don't agree with that type of consistency.
I feel content consistency is more important. Meaning: wherever code is always shown, monospace should be used; non-monospace otherwise. This already is very consistent nowadays: editor, output, suggest widget, parameter hints, hover widget; they all show monospace font. It is even consistent with quick open and the search and git viewlets' input boxes, since you do not always write code in there; you might write plain text, so that is non monospace.
This leads me to point out where this consistency is not followed: find all references, peek definition, search viewlet contents and, finally, the rename widget. All of these are examples in which I don't feel we're doing the right thing: they should all show monospace font.

That's a very compelling argument @joaomoreno, I agree.

I think we agree on consistency, I just disagree about making that change in small increments and starting with a random widget. If we can establish and formulate a rule then we should go by it.

Assigning this to @stevencl to coordinate this effort across all the areas @joaomoreno mentioned above

I would also say that writing RegExps in the search box is much easier with a monospaced font.


updated NLB Matrix

updated NLB Matrix

@srveit not sure what is wrong on this. normal edit so not sure what caused this


Wiesemann-Theis.NetworkOPC version 4.87

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/115065&drop=dogfoodAlpha

@wingetbot run


Feature/upgrade code

Make code more readable and upgrade old C code

The code has passed all the original tests, it can be merged now.


Dockerize

Need to make it easier for frontend folks to develop UI.
We should have a docker container:

Running wasmd chain with right RPC and REST ports exposed
Automatically deploys contracts onto chain
Has a set of accounts created with balances

Reopening.


Delegate quantization to mapbox-vector-tile

Connects to https://github.com/mapzen/tilequeue/issues/82
Depends on https://github.com/mapzen/TileStache/pull/141
@zerebubuth could you review please?

:+1:


Fix useRemarkSync param type

Initial checklist

[x] I read the support docs
[x] I read the contributing guide
[x] I agree to follow the code of conduct
[x] I searched issues and couldn’t find anything (or linked relevant results below)
[x] If applicable, I’ve added docs and tests

Description of changes
It looks like https://github.com/remarkjs/react-remark/pull/18 by @ChristianMurphy  introduced the incorrect type for the 2nd parameter of the useRemarkSync() function:

https://github.com/remarkjs/react-remark/pull/18/files#r1493819195

It seems incorrect to me because of A) the name and B) the onError property not being used.

 Please update your lock file with `npm install` before continuing.
npm ERR!
npm ERR! Invalid: lock file's type-fest@0.[8](https://github.com/remarkjs/react-remark/actions/runs/7950859747/job/21703629598?pr=74#step:5:9).1 does not satisfy type-fest@0.[13](https://github.com/remarkjs/react-remark/actions/runs/7950859747/job/21703629598?pr=74#step:5:14).1
npm ERR! Missing<EMAIL_ADDRESS>from lock file
npm ERR! Missing<EMAIL_ADDRESS>from lock file
npm ERR! Missing<EMAIL_ADDRESS>from lock file

Welcome @karlhorky 👋
I appreciate your enthusiasm and the PR, but also chill.
Multiple passive aggressive @ mentions is a bit much.
It's a type, documentation, I welcome improvements, but is not breaking or a blocker for your usage of the library.
It is also the weekend, take it down a notch.

To your CI/CD comment, I think npm has been tested with the legacy version of peer dependencies. The error look like npm's new handling of peer dependencies.
This could either by resolved by adding the legacy peer flag in CI, or focusing on getting #39 wrapped up, which upgrades all the dependencies to newer and more compatible versions.

but also chill.
Multiple passive aggressive @ mentions is a bit much

I'm sorry. Was definitely not my intention to cause any stress or be aggressive and I'd like to understand more to improve. I'll try reaching out privately on Twitter/X.

This could either by resolved by adding the legacy peer flag in CI

Added a step in https://github.com/remarkjs/react-remark/pull/74/commits/0607f1034824825d5f7752afd5b9c1bb98f1c26e
(Made it a separate step to make it easier to delete later)

Glad to see see again @karlhorky!
The changes look good to me.

Glad to help, thanks for the reviews! If there's anything else I should do to get this merged, let me know :)


Extra node left on a newly drawn area when the final or first node is clicked

When drawing a new area, it works if the user hits 'ESC' with the correct undo/redo state.
However, if you click the final or starting node in the area to complete the draw gesture, an extra node is left in the history to undo.

Fixed in 8b27ce1cce0e59608407c17fe63d7b93047a1ce1. The issue was that the _finish and _cancel methods were redundant in the area drawing mode. I got rid of the cancel state and ensured that the 'finish' call also did a call to history.pop() to remove the offending extra draw node left dangling.


While using controlled pagination and selection toggleAllRowsSelected works only for the current page

Description
Using both controlled pagination and selection with the autoResetSelectedRows: false and using a custom row identifier using the getRowId prop, I want to control the selection in all the pages, for example, from an external button to deselect all the selected items.
The problem is using the toggleAllRowsSelected(false) function, doesn't deselect all the elements, but deselect only the element on the currently visible page.
Example
Example here
Steps To Reproduce
Steps to reproduce the behaviour:

Open the example
As you can see there are 2 default items selected, one on the first page and one on the second
Go to page one
Click the button "Deselect all" (it will execute toggleAllRowsSelected)
Go to page two, you will see the element with index 18 selected

Expected behaviour (Recommended)
I expect the "Deselect all button" to deselect all the items on all pages!
Desktop

OS: MacOS Big Sur 11.2.1
Browser Chrome
Version 88.0.4324.192

Hi @tannerlinsley, first of all, great talk at the React Summit last week 👍
I hit the same issue, is there any workaround for deselecting all rows?

Thanks @ajkl2533, I've noticed from your code the stateReducer prop 🤦

Did anyone find a solution to this problem ??? I am currently facing this.  @ajkl2533

Hi @thisisgurkaran I didn't find anything better than dispatching custom action.

Hi all! I'm facing the same issue.
I have a table with controlled pagination and an external button that handles the rows selection of all pages. The button click triggers the toggleAllRowsSelected() but it doesn't work as expected because deselect only the rows of the current visible page.
Exactly like the @emberist example.

Hey @GiorgioCapaccio, the example provided by @ajkl2533 fix the issue!

Is there an updated solution that works with Table v8?


Fix issue 11959 - Set private symbols declared in version(unittest) blocks

A check in D-Scanner is planned for this.

@wilzbach the style target fails with:
make -C ../druntime -f posix.mak MODEL=64 DMD=../dmd/generated/linux/release/64/dmd OS=linux BUILD=release
make[1]: *** ../druntime: No such file or directory.  Stop.
posix.mak:469: recipe for target '../druntime/generated/linux/release/64/libdruntime.a' failed
make: *** [../druntime/generated/linux/release/64/libdruntime.a] Error 2

I think this a config problem on the Buildkite's side.


fix(deps): upgrade java core version

PR summary
Upgrade java core to the most recent version
PR Checklist
Please make sure that your PR fulfills the following requirements:

[ ] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

Current vs new behavior

Does this PR introduce a breaking change?

[ ] Yes
[ ] No

Other information

:tada: This PR is included in version 0.18.4 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Allow H264 headers to specify a framerate

This is merely a number in the H264 SPS headers - it may or may not actually be true. The codec has no idea when frames will actually be delivered to it.
It's optional whether to supply this number, and in most cases, unless you specifically need a value in the headers, we would recommend omitting it and driving the camera however you like.
Signed-off-by: David Plowman<EMAIL_ADDRESS>

getting assert
  File "picamera2.py", line 1506, in start_recording
    self.start_encoder(encoder, output, pts=pts, quality=quality)
  File "picamera2.py", line 1465, in start_encoder
    self.encoder.start()
  File "encoders/encoder.py", line 187, in start
    self._start()
  File "encoders/v4l2_encoder.py", line 135, in _start
    fcntl.ioctl(self.vd, VIDIOC_STREAMON, typev)
ProcessLookupError: [Errno 3] No such process

I think I got that when I asked it for high framerates. Could you post a minimal code snippet that goes wrong?

hmm, you are absolutely right, 30 is highest I can specify, but I can record at higher than that...

Though I don't know for sure, I'm guessing that the codec needs to be set to the officially correct "profile" and "level". I think the V4L2 driver handles those, but it looks to me that it may be the Python to V4L2 layer that isn't exporting all that stuff.
It looks like it defaults to level 4.0, which is capped at 245760 macroblocks/second. For your resolution, the limit there would work out at 31fps.  To get 60fps, it will need level 4.2. I'll have to go and look into that.

yes, combination of this PR and level change does give requested FPS using ffmpeg -i. Saying that, it doesn't exactly work like picamera1: picamera1 would gives 39.93 for 10sec of recording at 40fps (expected - frame drops) where with proposed changes it does say exactly what you specify - for example 40.0fps.
It's is definitely better for my purposes than no fps in headers, but I would hesitate to merge this PR as is. Maybe as you suggested it is better to allow application to set V4L2_CAP_TIMEPERFRAME if it's not possible to set correct one for library...
The profile setting does look valid change as a separate PR

Hi, glad that seems to be working. I didn't really understand the issue about the 40fps vs 39.93fps. As far as I know, the framerate gets set in the headers right at the start and never changes, so I can't see where that number is coming from. I'm pretty sure it's nothing to do with frame drops, perhaps there's some strange rounding going on somewhere?

Note that this PR now requires the latest version of python-v4l2.
@chrisruk I couldn't persuade flake8 to like the order of the imports... any ideas?

@davidplowman Hi, just wondering, did you try isort on it?

@davidplowman Hi, just wondering, did you try isort on it?

Ah, I hadn't realised I can run isort for myself to "fix" my file. Thanks! Bizarrely it seems to have moved the final two symbol imports from v4l2 onto one line... I can't say I'm not a bit baffled why this is "better". But if it works for flake8, then it works for me!

@RolandasRazma Are you able to create one of these short 39.93fps h.264 bitstreams so that I could download it and have a look what's in there? Also, what camera are you using?
I'm still struggling with the fact that (AFAIK) there is no timing information in an h.264 bitstream except that one time-per-frame number which gets set when the codec starts. Another theory may be that when you ask the camera for 40fps you don't quite get it because the camera has to quantise everything to whole pixel lines, maybe that's why you get a different number?
It's true that the legacy code would create variable framerate streams by default, but this was changed because certain applications (notably gstreamer, I believe) could not handle them. So now they always claim 30fps unless you say otherwise. (Even without these patches, I'm seeing that the headers have the "fixed_frame_rate" flag set to 1 so these do not appear to be variable framerate streams.)

hmm, maybe I'm doing something wrong. Without changes picamera2 ffmpeg -I video.h264 -vf vfrdet -an -f null - give me VFR:0.797954 (312/79) min: 39999 max: 40000 avg: 39999 with changes VFR:0.000000 (0/391) for same camera/settings
In my case it make no difference, but have nagging feeling that someone else will stumble upon this

hmm, maybe I'm doing something wrong. Without changes picamera1/picamera2 ffmpeg -I video.h264 -vf vfrdet -an -f null - give me VFR:0.797954 (312/79) min: 39999 max: 40000 avg: 39999 with changes VFR:0.000000 (0/391) for same camera/settings
Without changes FPS reported by ffmpeg as 30 on picamera2 (39.93 on picamera1 with sps_timing turned on) With changes as 40 Requested FPS: 40
In my case it make no difference, but have nagging feeling that someone else will stumble upon this
Can try to make short videos, in all permutations. I'm using imx219 (camera v2)

I was trying to find out what that vfrdet does. There's actually very little code there, it's just looking at the pts (timestamp) values, which of course begs the question where they come frame.
I tried ffprobe test.h264 -show_entries frame=pkt_pts_time -of csv=p=0 but they're all reported as N/A. So I have no idea what ffmpeg is doing. Is it trying to reverse engineer some supposed duration based on bitrate? Or maybe ffmpeg doesn't really handle the case of frames not having timestamps (I've seen this sort of behaviour elsewhere), so slightly unpredictable things happen?
If you could post a file where ffprobe reports 39.93 that would be interesting. For the rest, I'm not really seeing anything "unexpected" in the h.264 bitstreams, so it's a question of finding out what ffmpeg is really telling us.

import picamera
import time

camera = picamera.PiCamera()
camera.resolution = (1280, 720)
camera.framerate = 90
camera.start_recording('picamera1.h264', format='h264', level='4.2', sps_timing=True)
camera.wait_recording(1)
time.sleep(1)
camera.wait_recording(1)
camera.stop_recording()

picamera1.h264.zip
ffmpeg -i picamera1.h264

87.81 fps
ffmpeg -i picamera1.h264 -vf vfrdet -an -f null -

VFR:0.397129 (83/126) min: 13333 max: 48000 avg: 13333
ffprobe picamera1.h264 -show_entries frame=pkt_pts_time -of csv=p=0

Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 87.81 fps, 90 tbr, 1200k tbn, 180 tbc
N/A
N/A
...

Thanks for posting that file. Looking at that h264 bitstream, all I see is that it records a framerate of exactly 90fps, and there's no material difference between the h264 headers in any of the files that I can detect, they all seem "fair enough" to me.
So it remains all a bit of a mystery exactly what ffmpeg is telling us, and I don't think I believe those marginally different values - I think the "39.93fps" stream should be played at 40fps, and the "87.81fps" stream is in reality 90fps. I can certainly believe there are in general difficulties with raw h264 bitstreams because they don't have "proper" timestamps, having seen such things elsewhere. In summary, therefore, I'm somewhat inclined to accept this PR until such time as we can understand what, if anything, is actually wrong (at which point I'm very happy to come back to it). What do you think?

for sure this is best we can get to, but you will have questions about this...
Maybe naming could be more distinct from FPS like "sps_framerate" (similar to enable_sps_timing") or even "timeperframe" to copy v4l2.

Ah yes, whoops! Let me fix that quickly...

what is release process with apt (it is now recommended)?

I am unfortunately out of the office for a couple of weeks, but I'll try and do a release as soon as I can after that. If you needed something sooner could you just check out the "next" branch and use that directly for now (just remember to delete it again once the release shows up).

no rush, just wondering how release process work in general


Update jQuery to 3.5.1

This is an enhancement or feature.
Summary
Updates jQuery and replaces jquery-3.4.1.js with jquery-3.5.1.js.
Context
Closes #2712

Also consider having the Travis CI build fail gracefully when API keys are unavailable!
Some info: https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions

Travis CI makes encrypted variables and data available only to pull requests coming from the same repository (“internal pull requests”). These are considered trustworthy, as only members with write access to the repository can send them.

Tests and audits pass:

Running npm install && npm run build:js modifies some files, should those be updated as well?

@magneticflux- Yes, run npm run build:js and commit assets/js/main.min.js so the compiled version is included. As far as I can tell upgrading to jQuery 3.0.5 didn't break anything.
@iBug care to run some quick tests to see if things look good to you?

I did a quick search and it looks like there's no worries (but haven't tested it yet).
jQuery 3.5 is a security update that contains only one breaking change regarding how htmlPrefilter handles multiple self-enclosed XHTML tags. I don't see anything depending on that behavior so we can safely expect everything proceeds as normal.


privateCert.additionalHosts is not honored

With values.yaml like:
pingaccess-admin:
  enabled: true
  privateCert:
    generate: true
    additionalHosts:
      - myhost-pingaccess-admin.usa.ping-multi-cluster.com
      - myhost-pingaccess-admin

The cert is created with only the main domain. Alternate names is not populated according to additionalHosts.
Thinking this may be due to code somewhere here: https://github.com/pingidentity/helm-charts/blob/master/charts/ping-devops/templates/pinglib/_yamlSnippets.tpl#L58
but, not confirmed.


ProactiveAgent Spec Document

Output: Create a document (markdown is fine) outlining the ProactiveAgent library goal capabilities and design. This can then be repurposed easily in our presentations/demo

includes #7


v2.6.5 Release Task - Run Image-Scanning

Rancher Backend Engineering Task
This is a Rancher backend engineering task for (https://github.com/rancher/rancher/issues/37315) for the rancher engineering release captain:

[ ] Go to rancher/image-scanning and do the thing!

Completed successfully.


feat: add etcd store package

@starsz How did you set TTL for these session data?

@starsz How did you set TTL for these session data?

https://github.com/api7/etcdstore/pull/1/files#diff-10b0efd3f5b9f5a91fbd0ca256e6360a255e0c939ea0031df6ebe6fb19f8c353R95-R105
Use lease to set the ttl

@starsz How did you set TTL for these session data?

https://github.com/api7/etcdstore/pull/1/files#diff-10b0efd3f5b9f5a91fbd0ca256e6360a255e0c939ea0031df6ebe6fb19f8c353R95-R105
Use lease to set the ttl

Sorry, I didn't notice it.


feat(pixels): add my new pixel (38,22)

Checklist

[x ] I ran npm test locally and it passed without errors.

[x ] I only edited the _data/pixels.json file.

[x ] I entered the username in the pixels.json that I'm also using to create this pull request.
[x ] I acknowledge that all my contributions will be made under the project's license.

Adding my pixel to the grid.

It took, I think, four tries, but I learnt a lot.
Thank you, Dominik, for all of your help!


systemd-resolved cannot be fully disabled because /etc/resolv.conf is managed

Affected Puppet, Ruby, OS and module versions/distributions

Puppet: all
Ruby: all
Distribution: Ubuntu 20.04 (client)
Module version: 3.0.0, 3.1.0

How to reproduce (e.g Puppet code you use)
class demo {
  class { 'systemd':
    manage_resolved => true,
    resolved_ensure => 'stopped',
  }

  file { '/etc/resolv.conf':
    ensure => 'file',
  }
}

What are you seeing
Catalog compilation fails with with a duplicate resource declaration error.
What behaviour did you expect instead
File[/etc/resolv.conf] should not be managed when systemd-resolved is managed to the stopped state.
Output log

Duplicate declaration: File[/etc/resolv.conf] is already declared at (.../modules/systemd/manifests/resolved.pp, line: 77); cannot redeclare (.../modules/demo/manifests/init.pp, line: 7) (file: .../modules/demo/spec/fixtures/modules/demo/manifests/init.pp, line: 7, column: 3) on node ...

Any additional information you'd like to impart
Just stopping systemd-resolved on a system that has had it configured completely breaks name resolution.  Something else needs to manage /etc/resolv.conf afterward, or at least to update it once, but the module interferes with that.
For context, on domain-joined Ubuntu 20.04, systemd-resolved needs to be disabled in order for sssd to authenticate users against Active Directory.
Here is the RSpec test used to demonstrate the issue:
# frozen_string_literal: true

require 'spec_helper'

describe 'demo' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts.merge(systemd_internal_services: { 'systemd-resolved.service' => 'enabled' }) }

      it { is_expected.to compile }
    end
  end
end

Agree it would be nice to get this or someway of safely exiting systemd-resolved once it is deployed
Background is that  if you stop
systemd  stop systemd-resolved

Then both of the possible targets of /etc/resolv.conf are left behind i.e

/run/systemd/resolve/stub-resolv.conf
/run/systemd/resolve/resolv.conf

so setting use_stuf_file => false at the same time to restore symlink to /run/systemd/resolve/resolv.conf
may seem like a good idea until you reboot and /run/systemd/resolve is not there.
In a non puppet world it would be
rm -f /etc/resolv.conf
systemctl restart NetworkManager

but that is very OS dependent of course.
How about something like
exec{'install_working_resolve_conf':
     command  => `cp /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf`
     onlyif         => `resolv.conf  is a symlink one of  /run/systemd/resolve/stub-resolv.conf or  /run/systemd/resolve/resolv.conf`
}

all a bit fiddly.

On the very fine point I agree when the service is stopped /etc/resolv.conf should not be managed as that is only going to end badly.

It feels to me like we should document that resolv.conf gets left behind and it's up to the user of this module to find some other content. As @traylenator correctly states: the replacement is not always the same. Some may end up with a static resolv.conf file, others may use NetworkManagers and there are probably a few more options out there. Trying to solve it here will just end poorly.


fix(bufferTime): inner intervals will now clean up properly

adds marble tests around bufferTime
adds maxFrames property to VirtualTimeScheduler that will limit the execution of tests

attn @kwonoj @trxcllnt @steveorsomethin @mattpodwysocki... This PR has changes in it to VirtualTimeScheduler/TestScheduler that allow developers to test "unlimited" observables, in that it defaults to max out execution at 750 frames (75 character marbles). This can be set via rxTestScheduler.maxFrames = 1000 (or whatever number you pick) at any point during the test.

merged with 4ef41b0


Please allow translation of 'Vertex'

This is 'Knoten' in German:

The translation key for this one is geometry.vertex:

sorry, my fault.


Add SL as codeowners for snaps-owned controllers

Explanation

Whenever the version of a package in this monorepo is bumped, all dependencies on that package in other workspace packages are also bumped. This happens frequently in release PRs.
The Snaps team is listed as codeowners of permission-controller, notification-controller, and rate-limit-controller, so when any changes to these packages occur in a PR, the Snaps team must approve that PR. This means that if a dependency on a package is bumped in any of these controllers within a release PR, then the Snaps team must approve that release — even if none of those controllers are included in the release. This creates an obstacle to creating releases quickly.
To fix this, this commit adds the Shared Libraries team as codeowners of these controllers as well. This way, they can ask the Snaps team to approve only if it is necessary.
Changelog

(N/A)
Checklist

[x] I've updated the test suite for new or updated code as appropriate
[x] I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
[x] I've highlighted breaking changes using the "BREAKING" category above as appropriate

I feel like this defeats the purpose of us being codeowner. If the main obstacle is dependency changes, maybe add the shared libraries team to just the package.json?

I feel like this defeats the purpose of us being codeowner. If the main obstacle is dependency changes, maybe add the shared libraries team to just the package.json?

I'm a little bit fuzzy on the reasoning for not using workspace:^ versions in this repo. Is the upside large enough that it is worth having to consider changes like this?
I will also pose the question though, have we been holding back releases by not reviewing these fast enough? I feel like we've been pretty consistent in keeping up with reviews where we are needed, but willing to be convinced otherwise.
If we have and speed is a necessity here, I would agree with Maartens suggestion. But I will also mention that we have had mistakes being made in the past by exclusively changing package.json (for example flip-flopping between devDeps and deps 😅).

I'm a little bit fuzzy on the reasoning for not using workspace:^ versions in this repo. Is the upside large enough that it is worth having to consider changes like this?

Using workspace:^ versions made it impossible for clients to test changes locally, because Yarn does not resolve these versions when using yarn link. This error is demonstrated here: https://github.com/MetaMask/core/pull/1623. Switching to static versions solved this problem.

I will also pose the question though, have we been holding back releases by not reviewing these fast enough? I feel like we've been pretty consistent in keeping up with reviews where we are needed, but willing to be convinced otherwise.

No — y'all have been doing a good job of responding so far, but I know it is always late over there and so if there is a chance that we have to get a release out more quickly than usual then I was trying to give us a way to work around it in case of you were offline.
That said, I agree that perhaps adjusting the codeowners is not the right approach here. It is not the end of the world if we have to wait an extra day for an approval anyway. And if that does indeed end up becoming an issue maybe there is another approach we can take, such as limiting to certain files. I'll close this, then.


Support model pickling

Hi,
When I try to dump a model using pickle, I get the following error:
RuntimeError: Pickling of "pinocchio.libpinocchio_pywrap.Model" instances is not enabled (http://www.boost.org/libs/python/doc/v2/pickle.html)
Here is a short snippet to reproduce the error:
import pickle as pkl
import pinocchio as pin

model = pin.Model()
pkl.dump(model, open('model.pkl', 'wb'))

Would it be possible to include this feature ?
Thanks,
Robin

Solved by #958.

Hi! Sorry to "re-open" this already closed issue but why can we pickle pin.Model() but not pin.GeometryModel()?
Thank you!


UE5

Can you please make it available for Unreal Engine 5?

UE5pre1: It's currently working without problems. just put on the plugin folder.

Thanks, I'll try it!

My pleasure. I'd really love to know if there's any direct tool on unreal
to capture mocap animations and body tracking with normal webcams (not
running the exe ThreeDPoseTracke and VMCtoMOP). Thanks!
Il giorno dom 6 mar 2022 alle ore 11:50 maz0uzi @.***>
ha scritto:

Thanks, I'll try it!
—
Reply to this email directly, view it on GitHub
https://github.com/HAL9HARUKU/MopUE4/issues/8#issuecomment-1059938650,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGGF4CPGLOORK3MTKCOP2E3U6SEWZANCNFSM5OTA62PA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID:
@.***>

--
Francesco Cappilli
MSc. Environmental Engineer
P  Please consider the environment before printing this email
Confidentiality Notice – This e-mail message including any attachments is
for the sole use of the intended recipient and may contain confidential and
privileged information pursuant to Legislative Decree 196/2003 and the
European General Data Protection Regulation 679/2016 – GDPR-. Any
unauthorized review, use, disclosure or distribution is prohibited. If you
are not the intended recipient, please contact the sender by reply e-mail
and destroy all copies of the original message.

@fpibbs NNEngine is interesting

That's really powerful thanks, it's what I was searching for!

I got it working in UE5 editor, but it didin't work in runtime
don't know what's the problem

Does it support Metahuman?

Will it work in Unreal 5.1 as well?

Please upgrade the plug-in so that even the character's finger can move in Unreal Engine


Parse Hypothesis strageies for schema inference, validation, and synthesis

Is your feature request related to a problem? Please describe.
Pandera's new synthesis feature is very attractive, but is broken if the constraints are non-trivial (e.g. pa.Check.str_matches(complicated_regex)).  Related discussion here.
Describe the solution you'd like
Ideally, we could use complex schema for both validation and synthesis. In order for that to be possible, Pandera needs to generate data more efficiently, which means not using rejection sampling. Luckily, Hypothesis has solved the efficient synthesis problem, but Hypothesis strategies can't be used for validation.
It would be great if Pandera could parse a Hypothesis data_frames strategy into Schemas for validation:
my_schema = pa.infer_schema(hypothesis.extra.pandas.data_frames(...))
The Schema, then, could use hypothesis for efficient data generation.
Describe alternatives you've considered
I've tried disabling Hypothesis's health checks but Hypothesis literally ran out of random bytes to parse into your dataframe, and there's not really anything that [Hypothesis] can do about that
Additional context
We were attracted to Pandera over Great Expectations due to the ability to create hypothesis strategies directly from schemas. This saves us the labor of maintaining separate validation schemas and Hypothesis strategies.

hey @crypdick, yes getting data synthesis right is pretty challenging and I'd like to smooth out the rough edges. Just so I get a better sense of the issue, can you provide an example schema that's causing the health check issues?

@cosmicBboy sure! I had to obfuscate sensitive details but hopefully this gives you a rough idea.
import re

import pandera as pa

# simplified for security
IMG_URL_REGEX = re.compile(
    r"""
    ^s3://   # prefix
    [-a-z0-9]+/
    [A-Za-z0-9-]+/
    20[1-2][0-9]+/  # years 2010-2029
    [A-Za-z0-9-_,.]+/  # including _, commas, period
    v[0-9]/  # versions v0-v9
    2[4-5]/
    [0-9]+/
    [0-9]+  # file name
    [.]png$  # extension, escape dot
    """,
    re.X,
)

valid_ids = {str(i) for i in range(1, 1000)}   # placeholder for more complicated logic

is_valid_img_url = pa.Check.str_matches(IMG_URL_REGEX)
is_not_empty_series = pa.Check(lambda series_: len(series_) > 0, name="Series not empty")
is_valid_id = pa.Check.isin(valid_ids)

# class IDs joined by |
# this later gets filtered to ensure each element is a valid ID (invalid IDs throw Exceptions downstream)
numeric_str_delim_by_pipes = re.compile(
    r"""[0-9]{1,4}  # in reality, this is more complicated
        (\|[0-9]{1,4})*  # 0 or more IDs
        """,
    re.X,
)

grouped_by_s3uri_schema = pa.DataFrameSchema(
    columns={
        "img_url": pa.Column(
            pa.String,
            allow_duplicates=False,
            nullable=False,
            checks=[is_valid_img_url, is_not_empty_series],  # most restrictive check first
        ),
        "labels": pa.Column(
            pa.String,
            checks=[
                pa.Check.str_matches(numeric_str_delim_by_pipes),
                pa.Check(
                    lambda str_delim_pipes: all([id_ in valid_ids for id_ in str_delim_pipes.split("|")]),
                    element_wise=True,
                ),
            ],
        ),
    }
)

It would be great if Pandera could parse a Hypothesis data_frames strategy into Schemas for validation:
my_schema = pa.infer_schema(hypothesis.extra.pandas.data_frames(...))
The Schema, then, could use hypothesis for efficient data generation.

This sounds complicated 😅 (I did sort of look into this when building out the data synthesis functionality but found it way over my head, I'm not sure if there's a straightforward way of parsing a multi-part strategy)
Note that rejection sampling only occurs after the first check in the checks list, so inefficiencies in the first check would not be due to rejection sampling. Testing the code out locally it looks like img_url checks are leading to data_too_large and large_base_example, and labels checks are  leading to filter_too_much issues.
I think it's up to pandera to make data generation as easy as possible and providing users with an interface to customize data synthesis using the hypothesis API.
Another potential solution: custom checks and strategies
The extensions API was designed to give users access to the hypothesis API to make custom data synthesis strategies. This lets you register custom checks into the Check namespace and associate it with a data-generation strategy. You can effectively combine multiple checks into one, or implement any hypothesis strategy you want.
You can do something like:
import re
from typing import Optional

import hypothesis
import hypothesis.strategies as st
import pandas as pd
import pandera as pa
import pandera.extensions as extensions

# simplified for security
IMG_URL_REGEX = re.compile(
    r"""
    ^s3://   # prefix
    [-a-z0-9]+/
    [A-Za-z0-9-]+/
    20[1-2][0-9]+/  # years 2010-2029
    [A-Za-z0-9-_,.]+/  # including _, commas, period
    v[0-9]/  # versions v0-v9
    2[4-5]/
    [0-9]+/
    [0-9]+  # file name
    [.]png$  # extension, escape dot
    """,
    re.X,
)

NUMERIC_STR_DELIM_REGEX = re.compile(
    r"""[0-9]{1,4}  # in reality, this is more complicated
        (\|[0-9]{1,4})*  # 0 or more IDs
        """,
    re.X,
)

VALID_IDS = [str(i) for i in range(1, 1000)]

# Define custom url strategy and check
def url_strategy(
    pandas_dtype: pa.PandasDtype,
    strategy: Optional[st.SearchStrategy] = None,
    *,
    check_regex,
):
    if strategy is None:
        # replace this with more efficient Hypothesis strategy.
        return st.from_regex(check_regex, fullmatch=True)
    raise pa.errors.BaseStrategyOnlyError(
        "'url_strategy' must be a base strategy"
    )

@extensions.register_check_method(
    statistics=["check_regex", "synthesis_regex"],
    strategy=url_strategy,
    supported_types=pd.Series,
)
def valid_url(pandas_obj, *, check_regex, synthesis_regex=None):
    """Url regex check that separates check and data synthesis regex."""
    return pandas_obj.str.match(check_regex, na=False)

# Define custom label strategy and check
def labels_strategy(
    pandas_dtype: pa.PandasDtype,
    strategy: Optional[st.SearchStrategy] = None,
    *,
    valid_ids,
):
    if strategy is None:
        # replace this with more efficient Hypothesis strategy
        return st.lists(
            st.sampled_from(VALID_IDS), unique=True, min_size=1
        ).map("|".join)
    raise pa.errors.BaseStrategyOnlyError(
        "'labels_strategy' must be a base strategy"
    )

@extensions.register_check_method(
    statistics=["valid_ids"],
    strategy=labels_strategy,
    supported_types=pd.Series,
)
def valid_labels(pandas_obj, *, valid_ids):
    # combines the regex match check and the valid_ids check
    valid_ids = set(valid_ids)
    return pandas_obj.str.match(NUMERIC_STR_DELIM_REGEX) & (
        pandas_obj.map(lambda x: all(id_ in valid_ids for id_ in x.split("|")))
    )

schema = pa.DataFrameSchema(
    columns={
        "img_url": pa.Column(
            pa.String,
            allow_duplicates=False,
            nullable=False,
            checks=[
                pa.Check.valid_url(url_regex=IMG_URL_REGEX),
                pa.Check(
                    lambda series_: len(series_) > 0, name="Series not empty"
                ),
            ],
        ),
        "labels": pa.Column(
            pa.String,
            checks=pa.Check.valid_labels(valid_ids=VALID_IDS),
        ),
    }
)

@hypothesis.given(schema.strategy(size=5))
def test_schema(df):
    print(df)
    # test something

Phew! thanks for bearing with me. All that said, I did find some inefficiencies in the way the dataframe strategy was being constructed: #400 << this PR should actually make the img_url health checks go away and the code as you had it in your sample code snippet should work. Feel free to pull the feature/399 branch and try it out locally!
For the label generation code I'd recommend using the labels_strategy + valid_labels approach that I implemented above, since it generates data by sampling the set of valid ids instead of from the regex, which guarantees that | delimited strings contain valid ids

Other types of solutions that would be fairly heavy lifts re: going into the guts of the pandera.strategies module:

supporting composite strategies as a way of chaining multiple checks together.
providing an interface for "multi-check resolution", which basically aggregates the statistics of multiple checks and constructs a single strategy from them.

@cosmicBboy Tyvm for the detailed answer! When I copy-paste your code, I get a TypeError: url_strategy() missing 1 required keyword-only argument: 'check_regex'.

woops! just edited the code snippet , should work now

@cosmicBboy you beat me to it :) ty for pointing me to the extensions docs, I hadn't noticed this feature before

It would be great if Pandera could parse a Hypothesis data_frames strategy into Schemas for validation:
my_schema = pa.infer_schema(hypothesis.extra.pandas.data_frames(...))
The Schema, then, could use hypothesis for efficient data generation.

This sounds complicated 😅 (I did sort of look into this when building out the data synthesis functionality but found it way over my head, I'm not sure if there's a straightforward way of parsing complex multi-part strategies)

Ah, yeah.  As a Hypothesis core dev I wouldn't try this; complex strategies like data_frames() often have all the relevant structure inside @composite strategy functions and pulling it back out of the closures is totally infeasible.

A better approach, at least for element-wise checks, would be to get our "efficient filter rewriting" https://github.com/HypothesisWorks/hypothesis/issues/2701 done - I think it would be reasonably simple to support numeric bounds and string regex patterns for an initial pass, and that would probably cover a lot of your use-cases.

Thanks for the pointers @Zac-HD! Also, just wanted to say I'm a big fan of hypothesis, it's made a huge difference in the way I test DS/ML code (indeed, pandera strategies simply wraps hypothesis for convenience)
The current implementation of check strategy chaining in pandera does heavily use filter, really as a first pass to offer this functionality. I'll look into using composite and map more to make strategy chaining more efficient

@cosmicBboy the solution you posted seems to break down when used with indexes. In particular, if you edit the  "img_url": pa.Column(...) part to
    index=pa.Index(
        pa.String,
        allow_duplicates=False,
        nullable=False,
        name="img_url",
        checks=[
            is_valid_img_url,
            is_not_empty_series,
        ]
    ),

Then run my_schema.strategy().example() we get a ValueError: Length mismatch: Expected axis has 7 elements, new values have 0 elements. However, when we use it in @given it works fine.

@crypdick do you have a stacktrace of the error? this is definitely a bug

@cosmicBboy sure, here you go:
  File "/home/richard/.config/JetBrains/PyCharm2021.1/scratches/scratch_2.py", line 85, in <module>
    print(multihot_dataset_schema.strategy().example())
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py", line 319, in example
    example_generating_inner_function()
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py", line 307, in example_generating_inner_function
    @settings(
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/hypothesis/core.py", line 1163, in wrapped_test
    raise the_error_hypothesis_found
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/pandera/strategies.py", line 117, in set_pandas_index
    df_or_series.index = index
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/pandas/core/generic.py", line 5154, in __setattr__
    return object.__setattr__(self, name, value)
  File "pandas/_libs/properties.pyx", line 66, in pandas._libs.properties.AxisProperty.__set__
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/pandas/core/generic.py", line 564, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/home/richard/src/DENDRA/venv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 226, in set_axis
    raise ValueError(
ValueError: Length mismatch: Expected axis has 6 elements, new values have 0 elements

hey @crypdick #410 should fix the issue! there was a bug leading to the length mismatch where the generated df and index would not have the same length.
Will be merging into master probably by tomorrow. In the meantime, you can circumvent this bug by providing a value for the size parameter: my_schema.strategy(size=<int>).example()

Sounds great! Cc @daavidstein

@crypdick let me know if you come across any other problems relating to this issue! we can reopen it if needed


package naming conflict

django_sendfile uses the same package namespace as http://code.google.com/p/pysendfile/. Maybe a better one would be "django_sendfile".

Any news on this? I tried to come up with a namespace transitioning scheme but none of them seem to work without at least providing a sendfile/__init__.py from within the django-sendfile package.


keep yapping

It should "build" now, but it's doesn't work because python dependency is not there

thx


Add context for failed requests to DOs

Along with the error, print the DO binding and HTTP method and path.

cc/ @xtuc, @will118


MAINT: Add citation to actions that use eggNOG and/or Diamond

Some eggNOG/Diamond actions are missing the citations field.
This PR adds the eggNOG publication to the bibliography and cites the corresponding actions.
I also added a Diamond citation to actions that used it but did not cite it.
Closes #119

### Merge after
- [ ] bokulich-lab/q2-types-genomics#73
- [ ] #109
- [ ] #116
- [ ] https://github.com/bokulich-lab/q2-moshpit/pull/118

If you wanna see how the citations render, run the following.
Set env
# For linux:
# export MY_OS="linux"
# For mac:
export MY_OS="osx"
wget "https://data.qiime2.org/distro/shotgun/qiime2-shotgun-2023.9-py38-"$MY_OS"-conda.yml"
conda env create -n q2-shotgun --file qiime2-shotgun-2023.9-py38-osx-conda.yml
rm "qiime2-shotgun-2023.9-py38-"$MY_OS"-conda.yml"

conda activate q2-shotgun
conda remove q2-types-genomics q2-types q2-moshpit
pip install git+https://github.com/qiime2/q2-types.git
git clone git@github.com:bokulich-lab/q2-types-genomics.git
git clone git@github.com:bokulich-lab/q2-moshpit.git
cd q2-types-genomics
gh pr checkout 73
pip install .
cd ../q2-moshpit
gh pr checkout 120
pip install -e .

Test it
qiime moshpit <name-of-action> --citations

Actions modified:

[ ] build-custom-diamond-db
[ ] fetch-eggnog-db
[ ] fetch-diamond-db
[ ] fetch-eggnog-proteins
[ ] fetch-ncbi-taxonomy
[ ] build-eggnog-diamond-db
[ ] eggnog-diamond-search
[ ] eggnog-annotate

Same here @VinzentRisch  🙏🏼 . If you want to see the true diff
git diff fetch_ncbi_iss_107 eggnog_cit

@VinzentRisch thank you 🙏🏼

@misialq it looks like its not really working :( ⬆

@misialq it looks like it works! but you have to include the keywords in the PR message. It does not work with comments.

Ready to merge @misialq 🚀


Support for testing "outer" config

I'm using a Phinx as a part of the Nette framework.
There is a Dependency Ijection contaier, where I can setup the Phinx configuration directly without the 'phinx.yml' file, using the setConfig method.
Unfortunately the Test command force to use the file.
https://github.com/cakephp/phinx/blob/26753bdb66827383781eb0033b34cb467da1d12c/src/Phinx/Console/Command/Test.php#L62
It will be nice to have a possibility to check the connection with this configuration.
Everything that I need is to encapsulate this line like this:
if($this->hasConfig() === false) {
    $this->loadConfig($input, $output);
}

Looking quickly, all the other commands make a call to $this->bootstrap($input, $output);, which then does:
        if (!$this->hasConfig()) {
            $this->loadConfig($input, $output);
        }

So yeah, no reason to have a similar if check within the test command.


Exception during run.bat script execution

2019-12-13 15:01:11 ERROR o.d.m.e.l.WordVectorSerializer:2525 - Can't instantiate saved TokenizerFactory: org.deeplearning4j.text.tokenization.tokenizerfactory.DefaultTokenizerFactory

It could be probably related to recent replacement of deprecated similarityToLabel method


Random bot trading fails

error saying no logs in txn receipt; is this happening bc the trade is invalid? If so, we should catch bad trades earlier & supply a better error message.

Closing; this is likely resolved and if not will come up again with the new fuzz bot pipeline.


Replace unwrap in doc examples

To complete one of the items on the list in #467, this PR replaces any unwrap calls in documentation examples by ?.

Tests fail with:
---- src/transport/smtp/mod.rs - transport::smtp (line 33) stdout ----
error[E0601]: `main` function not found in crate `rust_out`
Error:   --> src/transport/smtp/mod.rs:32:1
   |
1  | / #![allow(unused)]
2  | | extern crate lettre;
3  | | use std::error::Error;
4  | | #[cfg(any(feature = "native-tls", feature = "rustls-tls"))]
...  |
22 | | Ok(())
23 | | }
   | |_^ consider adding a `main` function at the crate level

Well, that was an ugly commit chain as I found the right way to test all of the combinations, but the tests are now passing.
@amousset please have another look when you are able. I'm happy to make any changes you need.

Thanks, looks good to me!


[Bug] Infinite loop while using xgrammar with specific grammar

Env
xgrammar == 0.1.6
CUDA == 12.4
torch == 2.4.0
transformers==4.46.1

Issue
Hitting a infinite loop using the specific grammar provided in the code. (WIP: Trying to debug and understand the error)
Context
We (@antferdom) are researching different gramar-based decoding techniques for code generation. In this specific setting we're trying to generate a DSL called UVL. A reiterative problem is that is indent-sensitive so context-dependent, if It can be fixed I believe it could unlock Python codegen as well.
Errors
When leaving generation, no error is triggered rather than an infinite execution. When the processs is stopped with SIGINT (CTRL + C) the error output is:
File "/root/ebnf_xgrammar.py", line 110, in <module>
    generated_ids = model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2215, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3223, in _sample
    next_token_scores = logits_processor(input_ids, next_token_logits)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/logits_process.py", line 104, in __call__
    scores = processor(input_ids, scores)
  File "/usr/local/lib/python3.10/dist-packages/xgrammar/contrib/hf.py", line 90, in __call__
    self.matchers[i].fill_next_token_bitmask(self.token_bitmask, i)
  File "/usr/local/lib/python3.10/dist-packages/xgrammar/matcher.py", line 211, in fill_next_token_bitmask
    self._handle.fill_next_token_bitmask(bitmask.data_ptr(), list(bitmask.shape), index)

Code snippet for reproducibility
# from: https://xgrammar.mlc.ai/docs/how_to/ebnf_guided_generation.html#try-out-via-hf-transformers
import xgrammar as xgr

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig, set_seed
import random

r1 = 2 #random.randint(0, 1e5)
set_seed(r1)
print(f"seed: {r1}")
device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "meta-llama/Llama-3.2-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.float32, device_map=device
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
config = AutoConfig.from_pretrained(model_name)

messages = [
    {"role": "system", "content": """
You are a helpful AI assistant for creating sintactically correct expressions similar to these example:
features
INDENT
    SmartWatch
    INDENT
        mandatory
        INDENT
            Functionalities
            INDENT
                mandatory
                INDENT
                    FitnessMonitor
                    SleepTracker
                    VibrateAlert
                    INDENT
                        mandatory
                        INDENT
                            Call
                            Notification
                        DEDENT
                    DEDENT
                DEDENT
            DEDENT
            Sensors
            INDENT
                mandatory
                INDENT
                    Pedometer
                    Accelerometer
                DEDENT
                optional
                INDENT
                    HeartRateSensor
                DEDENT
            DEDENT
            Connectivity
            INDENT
                mandatory
                INDENT
                    BT40
                DEDENT
            DEDENT
        DEDENT
    DEDENT
DEDENT
     """},

    {"role": "user", "content": """
    Create a new one representing a computer
     """},
]
texts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer(texts, return_tensors="pt").to(model.device)

tokenizer_info = xgr.TokenizerInfo.from_huggingface(tokenizer, vocab_size=config.vocab_size)
grammar_compiler = xgr.GrammarCompiler(tokenizer_info)

uvl_base = r"""
root ::= "\n features \n\t" feature ws
feature ::= reference "\n"( ("\t")+ group)?
group    ::= ( "or " | "alternative " | "optional " | "mandatory ") ("\n" ("\t")+  feature)+
reference ::= [a-zA-Z_] [a-zA-Z_0-9]*
ws ::= [ \t\n]
"""

compiled_grammar = grammar_compiler.compile_grammar(uvl_base)

xgr_logits_processor = xgr.contrib.hf.LogitsProcessor(compiled_grammar)

import datetime
init_time = datetime.datetime.now()

generated_ids = model.generate(
    **model_inputs, max_new_tokens=500, logits_processor=[xgr_logits_processor]
)
generated_ids = generated_ids[0][len(model_inputs.input_ids[0]) :]
print(tokenizer.decode(generated_ids, skip_special_tokens=False))

print(f"generation elapsed time :{datetime.datetime.now()-init_time}")

Hi @roG0d, thanks for reporting the issue!
The current support for grammar with left recursion can be problematic. To fix this, note that CFG with left recursion can be converted into an equivalent CFG without left recursion (a tutorial can be found here.
We will support automatically eliminating left recursion in XGrammar later.

Hi @roG0d, thanks for the bug report.  Sorry for the late response as I just finished my travel.
After digging into your script I think that is mainly because there is too much non-determinism in the grammar (that means, for the same input string, there can be multiple interpretations according to the grammar). To much nondeterminism means multiple (maybe exponential) parsing stacks, which increases overhead at runtime. (Check out the "Why is it hard to accelerate general CFGs?" section in our blog.
Specifically in your grammar,
feature ::= reference "\n" ( ("\t")+ group )?
group ::= ( "or " | "alternative " | "optional " | "mandatory ") ("\n" ("\t")+  feature)+
reference ::= [a-zA-Z_] [a-zA-Z_0-9]*

group and feature overlap because or, alternative, etc. can be interpreted as reference or group keywords. So for this input
feature1
	or
		feature2

Currently, you can try rewriting your grammar to reduce such nondeterminism.
We are also looking into ways to alleviate this issue through automatic optimization. Please stay tuned!


Mount Points Should Replace on Longest String

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Define a mount point of /mnt/x
Define a second mount point of /mnt/x/y
Create a spark job that reads or writes to /mnt/x/y
Observe that the qualified name of the entity contains y due to the accelerator matching to /mnt/x.

Expected behavior
We should not see /y and instead should see the string that mnt/x/y represents.
Logs
N/A
Screenshots
N/A.
Desktop (please complete the following information):

OS: N/A
OpenLineage Version: 0.14.1
Databricks Runtime Version: 10.1
Cluster Type: Job
Cluster Mode: Standard
Using Credential Passthrough: No

Additional context
N/A

Cc: @atulkaila

Addressed with #101 and #108.

Closed with release of 2.1.1


luci-app-advanced-reboot: bugfix: add ACL json

Signed-off-by: Stan Grishin<EMAIL_ADDRESS>

Do legacy apps even need the .acl_depends in the controller? If they do not, should I remove it from all my other legacy apps?

@stangri - acl_depends annotations are strongly encouraged in master so LuCI knows which applications are available in which authentication state and to allow for fine grained multi user configuration in the long run.
Missing ACL annotations will lead to bugs like https://github.com/openwrt/luci/issues/4071

you should move all commands that allow write access to the system in any way (more general, any commands with side effects) into the write scope
Just to confirm, if I'm only using dd to read (dd if=... without of=) should it still be in the write section? Is there any way to limit the scope of dd in the ACL file to explicitly exclude of= (I mean without the helper executable)?

Also, I've asked in a PM on the forum, but this looks like a better place -- what do I need to run to append the template.pot with the localizable resources from the JSON file? And while we're on the subject, since it's way more than just UCI access, should the localizable text be different from "Grant UCI access for luci-app-advanced-reboot"?

Where can I see the script running on build servers to confirm how the pot templates are generated from strings in acl.d json?

Just to confirm, if I'm only using dd to read (dd if=... without of=) should it still be in the write section? Is there any way to limit the scope of dd in the ACL file to explicitly exclude of= (I mean without the helper executable)?

Hm, nope - not with the currently supported fileglob syntax. I plan to eventually add support for full regex patterns, then you can specify something like ^dd if=[^ =]+ but right now it needs a helper executable to restrict things.

Also, I've asked in a PM on the forum, but this looks like a better place -- what do I need to run to append the template.pot with the localizable resources from the JSON file?

You mean the descriptions? They should be covered since 0bc29706408c4cf8746917a6e4d4045397adca64 in master without any additional steps. I think this change isn't backported to 19.07 - however the only user facing code using these descriptions is luci-app-acl which is not available in 19.07 yet.

Just to confirm, if I'm only using dd to read (dd if=... without of=) should it still be in the write section? Is there any way to limit the scope of dd in the ACL file to explicitly exclude of= (I mean without the helper executable)?

Hm, nope - not with the currently supported fileglob syntax. I plan to eventually add support for full regex patterns, then you can specify something like ^dd if=[^ =]+ but right now it needs a helper executable to restrict things.

I'll give it a go and ping you for review when ready, thanks.

Also, I've asked in a PM on the forum, but this looks like a better place -- what do I need to run to append the template.pot with the localizable resources from the JSON file?

You mean the descriptions? They should be covered since 0bc2970 in master without any additional steps. I think this change isn't backported to 19.07 - however the only user facing code using these descriptions is luci-app-acl which is not available in 19.07 yet.

Oh, bummer, I use the latest stable SDK to build my packages and I currently pull the luci.mk and build directory from the openwrt-19.07 branch in luci before each build (which includes generating the pot template). Should I pull them from master branch of luci repo instead?

Uhm, I'll backport the JSON translation stuff to 19.07, thats probably easier, even if nothing is using it yet.


Allow FrameInfo to specify preset locations

Seems to be a missing feature compared to AviSynth's Subtitle

ASS style makes this possible through the Alignment field.


[FR] Extend JSONPath syntax to get object names

Hello community,
At the moment we don't have a way to get JSON object names using JSONPath syntax. But some programs send statistics as objects that contain other objects rather than arrays. For example, this is the NGINX Unit statistics output:
{
    "connections": {
        "accepted": 1067,
        "active": 13,
        "idle": 4,
        "closed": 1050
    },

    "requests": {
        "total": 1307
    },

    "applications": {
        "wp": {
            "processes": {
                "running": 14,
                "starting": 0,
                "idle": 4
            },

            "requests": {
                "active": 10
            }
        },
        "mw": {
            "processes": {
                "running": 15,
                "starting": 0,
                "idle": 3
            },
            "requests": {
                "active": 12
            }
        }
    }
}

In this case we need to iterate over the applications object and get the name for each application object to use it as the metric label.
It seems that this problem should be solved by the exporter, not the application itself, because we can't control the output of each API. See also the case with another API in #250.
There is a practice to extend the JSONPath syntax to get the object names, parent elements, etc. For example, to get the name of an object, a tilde (~) is typically used. This syntax is used by the widely used JSONPath-Plus JS library and by the yaml-jsonpath Go library created by VMWare.
Using this syntax we could get the metrics for applications running by NGINX Unit like this:
---
modules:
  default:
    metrics:
      - name: "unit_app"
        type: "object"
        valuetype: "gauge"
        path: "{ .applications.* }"
        labels:
          app: "{ @~ }"
        values:
          processes_running: "{ .processes.running }"
          processes_starting: "{ .processes.starting }"
          processes_idle: "{ .processes.idle }"
          requests_active: "{ .requests.active }"

I think we don't need to switch from the current library or send a syntax-extending patch to the upstream, we need to add it just for this exporter.
For now, anyone experiencing this issue will need to fork the project just to add this feature and maintain it themselves. Or this can also be solved by creating an exporter that uses JMESPath. But simply extending the existing syntax seems preferable.

+1 to this!

Yes, this is very useful and necessary!

+1 - I have a similar use case