Pycodestyle checker

Todo

[x] Implement pycodestyle job to selene github-actions.
[ ] Consider pycodestyle checks.
[ ] Fix pycodestyle issues.

@yashaka i have fixed most common pycodestyle issues.
Need to consider the most difficult: do we ignore them in the whole selene project or do we fix them?

E402 module level import not at top of file
E731 do not assign a lambda expression, use a def

E402
I have discovered that /selene/selene/__init__.py uses some strange documentation with imports between it.
Does it still need in the same way?
E731
Lambdas - do we need them in assign?
Report: https://github.com/aleksandr-kotlyar/selene/runs/1665430347?check_suite_focus=true
/home/runner/work/selene/selene/selene/__init__.py:55:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:79:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:90:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:112:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:123:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:219:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/common/predicate.py:77:1: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/selene/common/predicate.py:90:1: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/selene/core/entity.py:1315:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/support/shared/config.py:184:13: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/selene/support/shared/config.py:375:9: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/tests/unit/common/test_predicate.py:56:1: E402 module level import not at top of file

@yashaka i have fixed most common pycodestyle issues.
Need to consider the most difficult: do we ignore them in the whole selene project or do we fix them?

E402 module level import not at top of file
E731 do not assign a lambda expression, use a def

E402
I have discovered that /selene/selene/__init__.py uses some strange documentation with imports between it.
Does it still need in the same way?
E731
Lambdas - do we need them in assign?
Report: https://github.com/aleksandr-kotlyar/selene/runs/1665430347?check_suite_focus=true
/home/runner/work/selene/selene/selene/__init__.py:55:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:79:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:90:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:112:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:123:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/__init__.py:219:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/common/predicate.py:77:1: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/selene/common/predicate.py:90:1: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/selene/core/entity.py:1315:1: E402 module level import not at top of file
/home/runner/work/selene/selene/selene/support/shared/config.py:184:13: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/selene/support/shared/config.py:375:9: E731 do not assign a lambda expression, use a def
/home/runner/work/selene/selene/tests/unit/common/test_predicate.py:56:1: E402 module level import not at top of file

E402 can't be fixed for now, because we have to refactor too much to make it work...
We have to ignore it, unless we break down entities to its own modules and find the way to have no conflicts in imports...

E402 can't be fixed for now, because we have to refactor too much to make it work...
We have to ignore it, unless we break down entities to its own modules and find the way to have no conflicts in imports...

E402 can't be fixed for now, because we have to refactor too much to make it work...
We have to ignore it, unless we break down entities to its own modules and find the way to have no conflicts in imports...

Okay, i understood. Let's move E402 discuss to #265. I will ignore E402 check for now.

E402 can't be fixed for now, because we have to refactor too much to make it work...
We have to ignore it, unless we break down entities to its own modules and find the way to have no conflicts in imports...

Okay, i understood. Let's move E402 discuss to #265. I will ignore E402 check for now.

Regarding lambdas - i love them :+) And don't agree with PEP8 that they shouldn't be used when assigning. In some specific cases lambdas are more handy then defs, mainly in situations when we need to implement "curryable" functions (fn that returns fn that returns fn, etc.)
But in some cases we can remove them, and leave defs... I have to check all such cases...

Regarding lambdas - i love them :+) And don't agree with PEP8 that they shouldn't be used when assigning. In some specific cases lambdas are more handy then defs, mainly in situations when we need to implement "curryable" functions (fn that returns fn that returns fn, etc.)
But in some cases we can remove them, and leave defs... I have to check all such cases...

Regarding lambdas - i love them :+) And don't agree with PEP8 that they shouldn't be used when assigning. In some specific cases lambdas are more handy then defs, mainly in situations when we need to implement "curryable" functions (fn that returns fn that returns fn, etc.)
But in some cases we can remove them, and leave defs... I have to check all such cases...

Understood. Should i wait for it or just ignore E731 for now for making PR with pycodestyle checker viable?

I completed this issue. See PR #267 .

I completed this issue. See PR #267 .

Resolved in #267

Resolved in #267


Request to add country in streams API

Streams json api outputs:
{"channel":"1001Noites.br","url":"https://cdn.jmvstream.com/w/LVW-8155/ngrp:LVW8155_41E1ciuCvO_all/playlist.m3u8","http_referrer":null,"user_agent":null,"status":"timeout","width":1280,"height":720,"bitrate":1601536,"frame_rate":29.97,"added_at":"2022-08-13T04:41:45Z","updated_at":"2023-01-01T01:33:07Z","checked_at":"2023-01-01T01:33:07Z"},
requesting to add a tag based on country origin to this like so:
{"channel":"1001Noites.br","country":"BR", "url":"https://cdn.jmvstream.com/w/LVW-8155/ngrp:LVW8155_41E1ciuCvO_all/playlist.m3u8","http_referrer":null,"user_agent":null,"status":"timeout","width":1280,"height":720,"bitrate":1601536,"frame_rate":29.97,"added_at":"2022-08-13T04:41:45Z","updated_at":"2023-01-01T01:33:07Z","checked_at":"2023-01-01T01:33:07Z"},
tag "country":"BR" for Brazil, US for USA, etc.
Created a parser of sorts to parse the streams, would like to parse only US results.
Here is the link to my parser: https://kaos-inc.ga/dev/streams
As you can see, they are output for use with a Kodi addon I'm overhauling, the named channels are sorted and counted on the left and the unnamed channels are sorted/counted on the right.

I don't think that's necessary since you can always get this information from channels.json by channel ID or just use the ID itself, which already contains the country code (1001Noites.br/CNN.us).


Maintenance: publish layers & increment version during main release process

Bug description
At the moment the release process for the utilities and layers loosely works as follows:
graph LR
    A[Make Release] --> B(Version)
    B --> C(Publish)
    C -->|Triggers| D[Publish Layers]
    D --> E(Layer Published)
    E -->|Manual commit| H(Update layer version in docs)
    C -->|Manual trigger| F[GitHub Release]
    F --> G(Docs Published using tag)
    G -->|Tag from here| C

Looking at the diagram above we can observe that the Layer publishing workflow and the release one diverge from the publishing one. This is a problem because the following happens:

During versioning & publishing (1st workflow) we increment the packages version & push a tag (i.e. v.1.3.0)
Subsequently, the layers are published (2nd workflow) and we manually have to increment the Layer version (i.e. `3) in the docs
Finally, we make a release using the GitHub UI (3rd workflow) which uses the tag created at step 1

This means that by the time we make the GitHub release and release the docs, the tag that was created at step 1 is outdated and that the commit that updates the Layer version in the docs won't be taken in account.
This is what happened in the last release (v1.3.0) which should correspond to Layer version 3 but instead shows version 2 (from before the process above) that corresponds to v1.2.1 of the library.
Expected Behavior
The tag that is used to release a version & publish docs should already include the commit that updates the Layer version in the docs.
Current Behavior
Layer version in the docs is not synced with latest & with release version.
Possible Solution
There are three options that I could think of.
Option 1
Publish workflow optimistically increments by one the Layer version in the docs at some point BEFORE versioning.
graph LR
    A[Make Release] --> B(Version)
    B --> L(Opimistically update docs with new version)
    L --> M(Push new tag)
    M --> C(Publish)
    C -->|Triggers| K[Publish layers]
    K --> E(Layer Published)
    C -->|Manual trigger| F[GitHub Release]
    F --> G(Docs Published using tag)
    G -->|Tag from here| C

In this scenario the layers are published at the same step as now, but the Make Release workflow optimistically bumps the layer version in the docs before making the release. This way the commit that increments the layer version in the docs will have been made before the tag is pushed.
This option is the simpler to implement and hinges on the fact that Layers are always and exclusively published as part of a full release train.
Option 2
Layers + docs update are published AFTER versioning has happened BUT BEFORE the tags is pushed.
graph LR
    A[Make Release] --> B(Version)
    B --> K[Publish layers]
    K --> L(Update docs with new version)
    L --> M(Push new tag)
    M --> C(Publish)
    C -->|Manual trigger| F[GitHub Release]
    F --> G(Docs Published using tag)
    G -->|Tag from here| C

In this scenario, the Publish Layers action is triggered as intermediate step between npm run version & npm run publish (here). This way the layer publishing process
This option will require us to run the commands currently run in the postversion command in each package.json file (git push && git push --tags) after the docs have already been updated with the new Layers version.
This will work as long as the Publish Layers workflow is able to use the same context as the Make Release one & be able to see the that the version in the package.json files has been bumped.
Option 3
Leave all workflow as they are and manually update the version in the docs BEFORE starting the release process.
This step requires no changes in the pipelines but revolves around us maintainers following the process & making no errors.
If we are to follow this, we'll have to document the release SOP and make sure it's followed.
Steps to Reproduce
See above.
Environment

Powertools version used: N/A
Packaging format (Layers, npm): N/A
AWS Lambda function runtime: N/A
Debugging logs: N/A

Related issues, RFCs
#826

Chat with @rubenfonseca as he's just finished integrating our release with Layer version bump in the docs + ARM64 too, since we're now making ~168 Layer deployments per release

Seems that one of the workflows introduced in the PR that closed this issue has a circular dependency: https://github.com/awslabs/aws-lambda-powertools-typescript/actions/runs/4271035677


[placeholder] Provide metadata to images using LABEL

Will use this to link all PR's for all images that use ENV to deliver metadata to OpenShift/Kubernetes.
The proposal and all discussion is here:
https://github.com/mfojtik/origin/blob/metadata_proposal/docs/proposals/metadata.md
Note that some of the labels might change, depending on the discussions about label standardization.  However, we can't wait for that, so we should switch to use LABELs we have defined now.

@jhadvig @mnagy @soltysh @bparees @smarterclayton PTAL
This was a lot of manual work/changes and I could make a mistake, can you please double-check that the metadatas are correct?

@bparees I disabled the docker downgrade in the php Jenkins job as we need to have newer docker in place in order to recognize the LABEL instruction... I think the test job does not need to have this downgrade in place as it is not pushing anything anywhere... For the 'push' job, we have to investigate if the issue is still there, even with the newer Docker package.


Fix some typos and add more docs

Only some small things I noticed. After that, we are ready to go :)

Thank you!


Can yaml with comments be converted to properties and properties with comments be converted to yaml format?

Is your feature request related to a problem? Please describe.
Yes, I'm looking for a solution to convert yaml files containing comments to properties files with comments, and vice versa.
Describe the solution you'd like
I would like to see support for converting yaml files containing comments to properties files with comments, and vice versa. This would be a useful feature for managing and updating properties files with comments, and vice versa. It would be great if this feature could be added to the existing software.
Describe alternatives you've considered
I have considered manually converting the files, but this is time-consuming and error-prone. I have also searched for existing tools that can do this, but have not found anything that fits my needs so far.
Additional context
I am currently using the software for managing properties files, but I also have some yaml files that contain comments and are used for other purposes. It would be great if I could easily convert these files back and forth between yaml and properties file formats with their respective comments intact.

See eemeli/dot-properties for a .properties parser & stringifier that's capable of handling comments. Using that you should be able to put together a conversion tool relatively easily.

Before submitting this issue, I was researching your dot-properties project, and I was thinking of combining it with your yaml project to create a conversion tool. I thought that both of these projects were developed by you, so I wanted to ask if you have already implemented a similar functionality. If not, I will study and combine them myself. Thank you for your projects.


fix(autocmds): error occurs when bigfile size is nil

I'm using neovim
When vim.g.bigfile_size == nil, then an error will occur: attempt to compare nil with number.

After this fix, the comparison will not be made, so no error will occur.

that option is always set

I updated neovim to v0.10.0 and I get the exact same error message. If I set vim.g.bigfile_size to some value, the error is gone

@msvechla that can only happen if you're not really using LazyVim

Hmm you are right, something weird was happening. I checked my config against the starter again and this line was missing:
 spec = {
    -- add LazyVim and import its plugins
    { "LazyVim/LazyVim", import = "lazyvim.plugins" },

Afterwards everything started working again and LazyVim also got updated. Thanks for the help!


Pass along label/placeholder/registry for custom multiselect widgets.

Reasons for making this change
Inconsistency between properties passed to custom select widgets. When widget is for a question of type string with an enum, the custom widget receives props for label, placeholder, and registry. When widget is used on a multi-select field, these properties are missing.
Sample Schema:
{
	type: 'object',
	properties: {
		selectQuestion: {
			title: 'Select Question',
			type: 'string',
			enum: ['foo', 'bar', 'baz', 'qux'],
			enumNames: ['foo', 'bar', 'baz', 'qux']
		},
		multiSelectQuestion: {
			title: 'Multi-Select Question',
			type: 'array',
			uniqueItems: true,
			items: {
				type: 'string',
				enum: ['foo', 'bar', 'baz', 'qux'],
				enumNames: ['foo', 'bar', 'baz', 'qux']
			}
		}
}

Sample UI Schema:
{
	selectQuestion: {
		'ui:widget': 'select'
	},
	multiSelectQuestion: {
		'ui:widget': 'select'
	}
}

Checklist

[ ] I'm updating documentation

[ ] I've checked the rendering of the Markdown text I've added

[X] I'm adding or updating code

[ ] I've added and/or updated tests
[ ] I've updated docs if needed
[X I've run npm run cs-format on my branch to conform my code to prettier coding style

[ ] I'm adding a new feature

[ ] I've updated the playground with an example use of the feature

Thanks! The default MultiSelectWidget doesn't do anything with label/placeholder though -- do you think there is something we should do to change that?


Song randomizer?

What feature do you want to get added on the base engine?
A script to randomize the songs in a week for like mod random picks.

To test your sight, and reliability, please select the option of what should NOT be requested.
Useful feature

Failed test

To test your sight, and reliability, please select the option of what should NOT be requested.


added MaterialInstance::duplicate(...)

It's now possible to duplicate a MaterialInstance.

#3977 #3992


Missing PFX encryption requirement for custom domains

For custom domains on API Management layers, it seems the password is not decrypted correctly if the PFX was exported using AES256_SHA256 encryption algorithm. It is, however, if we use TripleDES_SHA1.
I guess this is a static assumption on the azure portal side - if so, this should be added to the documentation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d02a5991-da24-06a2-0c24-d012d2e07399
Version Independent ID: 64acd893-43bb-81af-d14d-0b7553e93daf
Content: Configure custom domain name for Azure API Management instance - Azure API Management
Content Source: articles/api-management/configure-custom-domain.md
Service: api-management
GitHub Login: @vladvino
Microsoft Alias: apimpm

@gmassari-88, Thank you for your feedback! We will review and update as appropriate.

@gmassari-88 I've tried generating a PFX using AES256_SHA256 and uploaded it to APIM without any issues. Could you share the exact error that you are seeing?
Also, here is the encryption details of the PFX that I used by running this command - openssl pkcs12 -info -in <path/to/pfx/file> -noout
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 2048, PRF hmacWithSHA256

Specific error as extracted from ActivityLogs says Password Incorrect, quote down here.
"errorMessage": "{
     \"error\":{
          \"code\":\"InvalidParameters\",
          \"message\":\"Invalid parameter: Specified password for the certificate associated with Proxy is incorrect.\",
          \"details\":null,
          \"innerError\":null
     }
}"

Cert details, from the same openssl command you mentioned:
MAC: sha256, Iteration 2000
MAC length: 32, salt length: 20
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 2000, PRF hmacWithSHA256
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 2000, PRF hmacWithSHA256

Now, before everyone here claims PEBKAC, I had the pass stored in a clip from Powershell, and was able to use that same cert and pasted password to test the import locally. I also pasted the unencripted password into azure portal.
If it makes any difference, I exported the cert like so:
$unsecPass = [Call a custom cmdLet to create a plain text random string password based on diceware] | clip;
$certPass = Convertto-SecureString -String $unsecPass -AsPlainText -Force;
Get-ChildItem cert:\localmachine\my\[Insert thumbprint here] |
   Export-PfxCertificate -FilePath $home\Desktop\myCert.pfx -ChainOption BuildChain -CryptoAlgorithmOption AES256_SHA256 -Password $certPass;

@gmassari-88 I can confirm that the AES256_SHA256 encryption that you are using isn't supported. You will have to use TripleDES_SHA1 for now.
We have assigned this issue to the content author to update accordingly.

API Management now supports custom domain certificates encrypted with the AES256_SHA256 algorithm: https://github.com/Azure/API-Management/releases/tag/release-service-2021-08 Please, allow a few weeks for the release to roll out.
#please-close


⚠ Jenkins Plugins Artifactory has degraded performance

In 74a3fa1, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1435 ms

Resolved: Jenkins Plugins Artifactory performance has improved in ca6ce08 after 25 minutes.


I'm not receiving code after phone input

I have tried many phone formats, but I do not get an sms code.

Possibly this issue https://github.com/343max/eero-client/issues/18


Polymorphic relations

What's the best way to handle polymorphic relations with MikroORM?
Method 1
This almost seems to work but I don't know how to make the entity parameter for the OneToOne decorator be dynamic.
@Entity()
export class Plan implements UuidEntity<Plan> {
  @PrimaryKey()
  uuid = v4()

  @OneToOne(() => GaragePlan) // How can I make this return type dynamic?
  node!: IdentifiedReference<DeckPlan | GaragePlan>

  @Property()
  type!: string

  @BeforeCreate()
  async do() {
    const node = await this.node.load()
    this.type = node.type
  }

  constructor(node: GaragePlan | DeckPlan) {
    this.node = Reference.create(node)
  }
}

@Entity()
export class DeckPlan implements UuidEntity<DeckPlan> {
  @PrimaryKey()
  uuid = v4()

  // Other properties...

  @Property({ persist: false })
  type = 'DECK_PLAN'
}

@Entity()
export class GaragePlan implements UuidEntity<GaragePlan> {
  @PrimaryKey()
  uuid = v4()

  // Other properties...

  @Property({ persist: false })
  type = 'GARAGE_PLAN'
}

Method 2
I'd rather not use this method.
@Entity()
export class Plan implements UuidEntity<Plan> {
  @PrimaryKey()
  uuid = v4()

  @OneToOne(() => GaragePlan)
  garagePlan?: IdentifiedReference<GaragePlan>

  @OneToOne(() => DeckPlan)
  deckPlan?: IdentifiedReference<DeckPlan>

  @Property()
  type!: string
}

// Same DeckPlan & GaragePlan as in Method 1

Method 3..?

This is not currently supported. I can imagine this done via single table inheritance (#33), where you would have all of Plan, GaragePlan and DeckPlan entities stored in one table - Plan being the common ancestor with the type as discriminator column. GaragePlan and DeckPlan would extend the (abstract) Plan entity.
Otherwise this can't work, how would the ORM know what entity to load (which table to look into) based just on the PK? Also I think indexes and referential integrity check could not work this way.
What you could do now on the other hand is to use JSON column for GaragePlan/DeckPlan objects, and implement custom type that would handle picking the right child class for you. This would mean you would have only one table, also you could not have any relations inside your "child entities" (they would not be entities). If you want to rather have it split across 3 tables, method 2 is the way to go.
Something like this could work:
export class NodeType extends Type {

  convertToJSValue(value: any): any {
    if (value.type === 'GARAGE_PLAN') {
      return new GaragePlan(value);
    }

    if (value.type === 'DECK_PLAN') {
      return new DeckPlan(value);
    }

    throw ValidationError.invalidType(NodeType, value, 'database');
  }

  getColumnType() {
    return 'json';
  }

}

export class DeckPlan {
  // Other properties...
  type = 'DECK_PLAN'
}

export class GaragePlan {
  // Other properties...
  type = 'GARAGE_PLAN'
}

@Entity()
export class Plan implements UuidEntity<Plan> {
  @PrimaryKey()
  uuid = v4();

  @Property({ type: NodeType })
  node!: DeckPlan | GaragePlan;

  constructor(node: GaragePlan | DeckPlan) {
    this.node = node;
  }
}

Thank you, will try a JSON column. Looking forward to #33


ObjectDisposedException on iOS

I get ObjectDisposedException when I try to navigate from one page to another.
I am using Xamarin Forms. The navigation is made as so: Navigation.PushAsync(newPage, true);
Exception: Exception type: System.ObjectDisposedException
The object was used after being disposed.
at System.Net.Sockets.Socket.EndAccept (System.Byte[]& buffer, System.Int32& bytesTransferred, IAsyncResult asyncResult) [0x00000] in :0
at System.Net.Sockets.Socket.EndAccept (IAsyncResult result) [0x00000] in :0
at Mtb.Server.Proxy.LogProxy.VS_IncomingOutput (IAsyncResult result) [0x00000] in :0

@mduchev - im going to assume you have moved on from this issue. If not, if there is some way you can reproduce it and show that it only happens when the toasts plugin is been used, please comment back here and we can reopen the issue.


Merge branch 'main' of https://github.com/spring-guides/gs-rest-service

@rsi1 Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.


HTML code not consistent on First Steps

At https://doc.babylonjs.com/babylon101/first there is sample code (above) that matches the playground and below there is an HTML template with a couple differences. Specifically,
var camera = new BABYLON.ArcRotateCamera("Camera", Math.PI / 2, Math.PI / 2, 2, BABYLON.Vector3.Zero(), scene);
var camera = new BABYLON.ArcRotateCamera("Camera", Math.PI / 2, Math.PI / 2, 2, new BABYLON.Vector3(0,0,5), scene);

and
var sphere = BABYLON.MeshBuilder.CreateSphere("sphere", {}, scene);
var sphere = BABYLON.MeshBuilder.CreateSphere("sphere", {diameter:2}, scene);

The top code seems better.

I will make this change and also look for any other inconsistencies on this page.

done


route-detect: init at 0.8.0

Find authentication (authn) and authorization (authz) security bugs in web application routes
https://github.com/mschwager/route-detect
Related to https://github.com/NixOS/nixpkgs/issues/81418
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 341070 run on x86_64-linux 1

2 packages built:

route-detect
route-detect.dist


Images: Update database functions to use transactions

Following on from https://github.com/lxc/lxd/pull/11217 as part of https://github.com/lxc/lxd/pull/11180 we need to load image profiles before running the placement scriptlet.
In order to do this properly and efficiently we should do it in as few DB transactions as possible, and in order to achieve this, a refactor of several of the image management functions was required to work within an existing transaction.
This has also presented the opportunity to introduce context support to these functions for proper cancellation as well.

jenkins: test this please


Update multiubuntu.md

Corrected syntax for karmor logs command.
Purpose of PR?:
Fixes # Corrected syntax for karmor logs command in documentation.
Does this PR introduce a breaking change?
No.
If the changes in this PR are manually verified, list down the scenarios covered::
Additional information for reviewer? :
Mention if this PR is part of any design or a continuation of previous PRs
Checklist:

[ ] Bug fix. Fixes #
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] PR Title follows the convention of  <type>(<scope>): <subject>
[ ] Commit has unit tests
[ ] Commit has integration tests

@noahabbott123 please sign-off the commits as well.


Brattan/head in ready beh 34

Things the PR does go here

Makes the robot look forward during initial

Pre-PR Checklist:
Have you

[ ] Updated NUbook if necessary (add link to NUbook PR here)
[ ] Added/updated tests for your changes, including regression tests for bug fixes
[ ] Updated relevant module READMEs
[ ] Added/modified documentation directives in relevant code
[ ] Added a descriptive title and relevant labels to the PR

Title of the PR needs changing, see https://nubook.nubots.net/guides/general/git#making-a-pull-request


__non_webpack_require__ regression since version 4.22.0

Bug report

What is the current behavior?
The problem is that Webpack, starting from version 4.22.0, is now parsing the entire
typeof __non_webpack_require__ !== 'undefined'

expression during the build process, causing the above line to be reduced to true.
If the current behavior is a bug, please provide the steps to reproduce.
This bug originates from Mozilla's PDF.js repository. In https://github.com/mozilla/pdf.js/pull/10170#issuecomment-431697032 we made a build of the viewer with a Webpack version higher than 4.21.0, the last working version. Since 4.22.0 this is broken. In https://github.com/mozilla/pdf.js/pull/10170#issuecomment-431697032 the difference between the output of version 4.21.0 and 4.22.0 is highlighted. The original source code line is here: https://github.com/mozilla/pdf.js/blob/81f5835cd7024827e0671ccd66525df98fb15703/src/display/api.js#L56

What is the expected behavior?
The expected, and previous output, is that __non_webpack_require__ is replaced with require as
typeof require !== 'undefined'

According to the documentation, this should still work; see https://webpack.js.org/api/module-variables/#non_webpack_require-webpack-specific

Other relevant information:
webpack version:  4.21.0 (last working version), 4.22.0 (first broken version), latest (still broken)
Node.js version: latest
Operating System: Linux/Windows
Additional tools:

Thank you for fixing this so quickly!


Typo "Azure function"→"Azure Function"

https://docs.microsoft.com/en-us/azure/digital-twins/how-to-integrate-time-series-insights

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good to me! #sign-off

Looks good to me! #sign-off

Partially correct; changed "an Azure Function" to "a function in Azure" per internal style guidance.
Thanks.

Partially correct; changed "an Azure Function" to "a function in Azure" per internal style guidance.
Thanks.


Automated cherry pick of #4317: fix: hide worker manager watchdog log

Cherry pick of #4317 on release/2.13.
#4317: fix: hide worker manager watchdog log

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.13@17e897c). Click here to learn what that means.
The diff coverage is n/a.

@@              Coverage Diff               @@
##             release/2.13   #4318   +/-   ##
==============================================
  Coverage                ?   6.87%
==============================================
  Files                   ?     520
  Lines                   ?   88194
  Branches                ?       0
==============================================
  Hits                    ?    6062
  Misses                  ?   81629
  Partials                ?     503

Flag
Coverage Δ

#aFlag
6.87% <ø> (?)

Impacted Files
Coverage Δ

pkg/appsrv/workers_watchdog.go
29.62% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17e897c...1441b70. Read the comment docs.

/lgtm
/approve


 Unless you change the types too to BigInt.. idk? LGTM?

integer limits the duration of the moderation cases to roughly 24 days 18 hours, and there are many cases where mods want to apply an action that lasts for months. bigint is suitable for this.
As for ranges, we check that in the commands.

This feels like a literal... "hey let's drop the limit and thats all", instead of a "hey lets support BigInts, literally"... Unless you change the types too to BigInt.. idk? LGTM?

integer limits the duration of the moderation cases to roughly 24 days 18 hours, and there are many cases where mods want to apply an action that lasts for months. bigint is suitable for this.
As for ranges, we check that in the commands.

I'm talking about the JS type bigint, not the db one. Those values are still interpreted as a number in JS, and stored as a String on the DB (which I assume then gets converted to a bigint by the DB)

I'm talking about the JS type bigint, not the db one. Those values are still interpreted as a number in JS, and stored as a String on the DB (which I assume then gets converted to a bigint by the DB)

Those values are interpreted as a string, and sent as a string. The transformer deserializes the string into a number, and the number into a string.

Those values are interpreted as a string, and sent as a string. The transformer deserializes the string into a number, and the number into a string.


Added Few FAQ's

Added Few FAQ's
Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the CODEOWNERS.
House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[X] Tested Locally

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d915c01d98157463d89f2b--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65dd16f879de9e438c627bed--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65df4a7fbc1b5420746a79fb--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65df4aab36767b2203975660--harness-developer.netlify.app


add uni-snd : low level sound layer

Ok this is a bit involved and should be considered a first draft.
It only covers the lower uni-snd layer and not its integration into the unrust engine. I hacked a sound example to demonstrate how to use the uni-snd library, not how to properly integrate it in unrust.
There's a SoundDriver that has a very different behavior in native and web.
On the native side, it starts a detached thread and sends data to the audio interface, pulling it from the provided SoundGenerator.next_value function.
Since the SoundGenerator is owned by the thread, you cannot keep a reference and call methods on it. instead, you have to use the SoundDriver.send_event method to send messages through a channel.
On the web side, the frame function should be called every frame so that it keeps the audio buffer filled.
Right now, stereo mode is hardcoded and the sound generator should alternatively provide the left and right channels values.

Everything looks find for me, so do you want me to merge now ?

One more thing, i still think printing error as an error handing is not good. How about we add an error_status() method to let the user to query there are any error ?

Seems all good to me. It's been used successfully here : http://roguecentral.org/~jice/viper2/?cart=celeste

LGTM, however i cannot hear any sound from your link. anyway i would merge it now.

you have to press X or C to start the game and ear some sound :)

WOW..... that's impressive.

thanks :)


Improve performance by reordering constructors

Milan Straka reports the following improvements from reordering constructors in the containers package:

The order of constructors of IntSet and IntMap matters when considering
performance.  Currently in GHC 7.0, when type has 3 constructors, they
are matched from the first to the last -- the best performance is
achieved when the constructors are ordered by frequency.
On GHC 7.0, reordering constructors from Nil | Tip | Bin to Bin | Tip | Nil
improves the containers_benchmark

by 9.5% on x86 and by 8% on x86_64 for IntMap
by 11% on x86 and by 9% on x86_64 for IntSet
The performance should never decrease for any architecture.

is this ticket a todo for possibly improving performance?

It is. I did make a quick attempt of reordering the constructors, without seeing any gain (if I recall correctly.) I left the issue here to remind myself to look into it in depth in the future. It could be that GHC changed how it lays out branches in the code generator such that it no longer uses the data definition order, but the ordering of the case statement branches.

Cool. I may have a go at looking at this too sometime in the near term. I assume that the relative freq in the note means #(bin)> #(tip) > #(nil)
?

Yes that's right.

The current order of constructors is
Empty | BitmapIndexed | Leaf | Full | Collision

I think that position of Empty is a bit dubious – I'd expect that both BitmapIndexed and Leaf nodes are encountered more frequently.
But I also wonder whether GHC is really privileging the first constructors all that much. In this C-- for $wsize (generated with GHC 9.2.2), an Empty node with tag 1 has to pass through three branches before returning to the loop.
       cxF6: // global
           _sm2t::I64 = I64[Sp + 8];
           _cxFf::P64 = R1 & 7;
           if (_cxFf::P64 < 4) goto uxGj; else goto uxGl;
       uxGj: // global
           if (_cxFf::P64 < 3) goto uxGk; else goto cxFc;
       uxGk: // global
           if (_cxFf::P64 < 2) goto cxFa; else goto cxFb;
       cxFa: // global
           R1 = _sm2t::I64;
           Sp = Sp + 16;
           call (P64[Sp])(R1) args: 8, res: 0, upd: 8;
       cxFb: // global
           _sm2w::P64 = P64[R1 + 6];
           _sm2B::P64 = _sm2w::P64;
           _sm2A::I64 = I64[_sm2w::P64 + 8];
           _sm2z::I64 = 0;
           _sm2y::I64 = _sm2t::I64;
           goto cxFu;
       cxFu: // global
           if (%MO_S_Lt_W64(_sm2z::I64,
                            _sm2A::I64)) goto cxFD; else goto cxFI;
       cxFD: // global
           _sm2E::P64 = P64[(_sm2B::P64 + 16) + (_sm2z::I64 << 3)];
           I64[Sp - 16] = cxFB;
           R3 = _sm2y::I64;
           R2 = _sm2E::P64;
           P64[Sp - 8] = _sm2B::P64;
           I64[Sp] = _sm2A::I64;
           I64[Sp + 8] = _sm2z::I64;
           Sp = Sp - 16;
           call $l$wgo1_gpfo_info(R3,
                                  R2) returns to cxFB, args: 8, res: 8, upd: 8;
       cxFB: // global
           _sm2A::I64 = I64[Sp + 16];
           _sm2B::P64 = P64[Sp + 8];
           _sm2z::I64 = I64[Sp + 24] + 1;
           Sp = Sp + 16;
           _sm2y::I64 = R1;
           goto cxFu;
       cxFI: // global
           R1 = _sm2y::I64;
           Sp = Sp + 16;
           call (P64[Sp])(R1) args: 8, res: 0, upd: 8;
       cxFc: // global
           R1 = _sm2t::I64 + 1;
           Sp = Sp + 16;
           call (P64[Sp])(R1) args: 8, res: 0, upd: 8;
       uxGl: // global
           if (_cxFf::P64 < 5) goto cxFd; else goto cxFe;
       cxFd: // global
           _sm2L::P64 = P64[R1 + 4];
           _sm2Q::P64 = _sm2L::P64;
           _sm2P::I64 = I64[_sm2L::P64 + 8];
           _sm2O::I64 = 0;
           _sm2N::I64 = _sm2t::I64;
           goto cxFV;
       cxFV: // global
           if (%MO_S_Lt_W64(_sm2O::I64,
                            _sm2P::I64)) goto cxG4; else goto cxG9;
       cxG4: // global
           _sm2T::P64 = P64[(_sm2Q::P64 + 16) + (_sm2O::I64 << 3)];
           I64[Sp - 16] = cxG2;
           R3 = _sm2N::I64;
           R2 = _sm2T::P64;
           P64[Sp - 8] = _sm2Q::P64;
           I64[Sp] = _sm2P::I64;
           I64[Sp + 8] = _sm2O::I64;
           Sp = Sp - 16;
           call $l$wgo1_gpfo_info(R3,
                                  R2) returns to cxG2, args: 8, res: 8, upd: 8;
       cxG2: // global
           _sm2P::I64 = I64[Sp + 16];
           _sm2Q::P64 = P64[Sp + 8];
           _sm2O::I64 = I64[Sp + 24] + 1;
           Sp = Sp + 16;
           _sm2N::I64 = R1;
           goto cxFV;
       cxG9: // global
           R1 = _sm2N::I64;
           Sp = Sp + 16;
           call (P64[Sp])(R1) args: 8, res: 0, upd: 8;
       cxFe: // global
           R1 = _sm2t::I64 + I64[P64[R1 + 3] + 8];
           Sp = Sp + 16;
           call (P64[Sp])(R1) args: 8, res: 0, upd: 8;


Multi-network for Remote contract import.

Is your feature request related to a problem? Please describe.
Currently remote cross contract import only supports OCI. The community wants more sources of contracts.

Network
Direct path

Describe the solution you'd like
I should be able to import via alternative methods
Describe alternatives you've considered
N/A
Additional context
https://discord.com/channels/897514728459468821/1177272905780973568

Currently, OCI works like this:
voting = "ghcr.io/eigerco/nebula/contracts/voting:v0.2.0"

This is also valid:
voting = { reference = "ghcr.io/eigerco/nebula/contracts/voting:v0.2.0", digest="sha" }

Improvements can be done:
#[derive(Debug, Serialize, Deserialize, Clone)]
#[serde(untagged)]
pub enum Contract {
    NoDigest(String),
    WithDigest {
        digest: Option<String>,
        reference: String,
    },
  Network(todo!()),
  Fs(PathBuf)
}


Support for pip --trusted-host argument enhancement

This also changes the following behaviors:

extra-index-url is now supported without specifying index-url
trusted-host and extra-index-url can specify both a single value and a collection as pip allows that

trusted-host added for both dependency and plugin installs
fixes #341, connected to #341

@mriehl  review please

Coverage increased (+0.06%) to 72.119% when pulling 09644a66adba0b8751b0803b5e93cd7a448a17d3 on arcivanov:issue_341 into 3db1079f491e14d79fdf52477cc8b8a4f43622ba on pybuilder:master.

LGTM :+1:


Fix a few warnings

This PR fixes two warnings I was seeing in dev builds: missing viewer prop for GlobalMap and duplicate keys in the EventSection component.

Fails

:no_entry_sign:

Missing Test Files:

src/lib/Scenes/Map/__tests__/MapRenderer-tests.tsx

If these files are supposed to not exist, please update your PR body to include "#skip_new_tests".

  Generated by :no_entry_sign: dangerJS

Warnings

:warning:

Missing Test Files:

src/lib/Scenes/Map/__tests__/MapRenderer-tests.tsx

If these files are supposed to not exist, please update your PR body to include "#skip_new_tests".

  Generated by :no_entry_sign: dangerJS


Patch library match dialog

still need to add the missing dependency in the future but it works for now.

I wouldnt consider this a solution :D

Postponed the real fix for home.
Just updating the cdk to 2.9 fixes the issue.
Structures work well and I added a ParserTest in PR #1727


⚠ Bofrost.be has degraded performance

In 5df3106, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1658 ms

Resolved: Bofrost.be performance has improved in aca2a17.


about LIKE Condition 。。。

mysql field uid = 1,2,13,14,12
medoo "city[~]" => "2"
to: 2,12
4 .i want it 2, no 12

you need to open debug select sql，check your sql condition。
RRRena

在 2016年12月21日，08:54，Ono Naoki<EMAIL_ADDRESS>写道：
mysql field uid = 1,2,13,14,12
2 medoo "city[~]" => "2"
3 to: 2,12,
4 i want it 2, no 12
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Why not just "city" => 2?


Download Error

Version
maybe v1.12.2
Platform
Windows 11
Steps to reproduce
一部の動画URLを入力するとエラーになってしまい、動画がダウンロードされません。

Details
ダウンロードされてほしい。
Checklist

[X] I have looked through existing issues to make sure that this bug has not been reported before
[X] I have provided a descriptive title for this issue
[X] I have made sure that this bug is reproducible on the latest version of the application
[X] I have provided all the information needed to reproduce this bug as efficiently as possible
[X] I have sponsored this project
[ ] I have not read any of the above and just checked all the boxes to submit the issue

Duplicate of #469 (also 403 error).


Add PostgreSQL support

@Tibo-lg Good catch! I added the missing files

@shehzanmaredia Sounds good, I will make a GitHub issue about the generics refactoring.

This is the issue I made: https://github.com/lava-xyz/sibyls/issues/19


MenuFlyout isOpen property is not respected

Problem Description
Trying to only open MenuFlyout once a specific condition is met, I thought that I could use the isOpen = {condition} property but the flyout does not seem to respect it. For example, if I set isOpen = {false} the flyout still opens when I right click on the item it is attached too.
Steps To Reproduce
<GridViewItem>      <Grid></Grid>     <MenuFlyout         isOpen = {false}     >          <MenuFlyoutItem text = "some flyout item"/>     </MenuFlyout> </GridViewItem>
I would expect running this code would create a flyout that never opens? But it does open whenever the onRightTapped event is activated
Expected Results
Very possible I am incorrectly using the MenuFlyout? I would like there to be a way to ideally create a flyout programmatically without having to add it to the XAML tree, but having the MenuFlyout only open once a condition is met seemed like the next best option
Environment
npx react-native info

This is on par with how MenuFlyout in xaml behaves when used as a ContextFlyout:
https://docs.microsoft.com/en-us/uwp/api/Windows.UI.Xaml.Controls.MenuFlyout?view=winrt-22000

I think the first step would be to figure out how you would do what you want in XAML and then we can see how it translates to react-native

I think in XAML you wouldn't use a ContextFlyout, but instead you'd use AttachedFlyout, and call ShowAt programmatically in code behind in response to some event


Having multiple resource estimation tests still causes nondeterministic failures related to concurrency

All I did was have two resource estimation tests. And then pretty often (>10% of the time) I get failures like this one:
 Unhandled exception. System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
  ---> Microsoft.Quantum.Diagnostics.AllowAtMostNCallsCA on [...]bin/Debug/netcoreapp3.1/D:\a\1\s\submodules\QuantumLibraries\Standard\src\Diagnostics\Allows.qs:line 0
    at Test.test_repro on test/test.qs:line 0

I've transferred this issue from qsharp-language to qsharp-runtime and updated the label to bug. We need to investigate whether the resource estimator is expected to support parallel access to these data structures. Our other local simulator targets support parallel execution, so resource estimator likely should too.

+1, I've also encountered this in my recent work.


Add logit-ssl-drain service to PaaS manifest

Since we create new applications when we deploy using blue/green deployment, we need to specifiy the service that sends logs so it's applied on deployment.

You can preview this change here:
Built with commit 03e73b8ec2073517910671106766db61c5c3300d
https://deploy-preview-1105--govuk-design-system-preview.netlify.com


Updated .gitignore for windows/mac users

Description
.DS_Store is a MacOS directory cache file and thumbs.db is a windows directory cache file. Sometimes Mac/Windows users accidentally push these files which may interfere with other Mac/Windows users.
Types of changes

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[X] I have read the CONTRIBUTING document.

@sidhyatikku thank you for your PR. These files should be part of your global .gitignore, so there's no need to add it to individual projects.
Here's an example of how to do it https://coderwall.com/p/vz6ymw/how-to-globally-gitignore-ds_store
Thanks!


Updating README to include IFTTT as Airflow user

We, @IFTTT, use Airflow to monitor and schedule all our Data Pipelines and ETLs.

Thanks for your contribution.
See my comments on #1731 and how the submitter corrected it on #1733
In addition to a call for references, we also want to acclimate potential future contributors with the apache commit guidelines. This is required before I can merge your change.

I resolved this on merge but please adhere to our guidelines for future commits. Welcome to the community!


Packages compatibility issues between Microsoft.Bot.Connector and Microsoft.Azure.Management.ResourceManager

Hi,
I'm trying to use the Microsoft.Azure.Management.ResourceManager package on my bot and it's failing because the one of the dependencies of the ResourceManager packages requires the Microsoft.Rest.ClientRuntime package to be of version 2.1.0 or higher.
The issue is that the Microsoft.Bot.Connector package constrains that same package (Microsoft.Rest.ClientRuntime) to be between versions 1.8.2 && 2.0.0.
Is there a plan to update the Microsoft.Bot.Connector dependencies to the latest versions?
Thanks.
Ez.

Hi @dandriscoll, any idea on this?

Hi @ejadib, unfortunately we don't have plans to update to the new version of ClientRuntime at the moment. Microsoft.Rest.ClientRuntime has breaking changes between version 1 and version 2, and we are currently using features in version 1. That said, it would be nice to upgrade, but I can't promise a timeline at this point.

My 2 cents: I did a little spike to workaround this issue leveraging ILMerge to include the required Microsoft.Rest.Client.Runtime clases inside the Bot.Connector assembly, thus, removing the offending dependency.
The downside here is that then I was required to recompile the Bot.Builder project using the new assembly.
This last recompile step could be avoided if this is done on the Bot.Connector build process since it should be re-signed after merge (ILMerge does it provided the .snk file)
I've used the following command line to merge both assemblies:
ILMerge.exe Microsoft.Bot.Connector.dll Microsoft.Rest.ClientRuntime.dll
/out:Microsoft.Bot.Connector.Merged.dll
/lib:<path to Newtonsoft.Json.dll>
/lib:<path to System.Net.Http.Formatting.dll>
/lib:<path to System.Web.Http.dll>
/internalize:exlcudes.txt

Notice the /internalize switch which makes all classes internal avoiding conflicts with the new version of the package. However, Bot.Builder uses two public classes added to the excludes.txt file:
Microsoft.Rest.ValidationException
Microsoft.Rest.ValidationRules

Hope this helps!


Removed the RK_Inst enum type and associated code from SymbolicValue.

This is a follow-up to https://github.com/apple/swift/pull/19040.

@swift-ci please test tensorflow

GPU test has a flaky test: https://ci-external.swift.org/job/swift-PR-TensorFlow-Linux-gpu/85/
Mac test finished: https://ci-external.swift.org/job/swift-PR-TensorFlow-macOS/62/console


Convert 'Cobenefits' field to free text+picklist

17-item drop down plus free text entry field to allow users to select SDGs or enter their own cobenefit.

@ccornick this isnt a picklist already so for me to accept this into development you need to provide the picklist values for me.


Remove Ubuntu 12.04 testing in Travis + integration tweaks

12.04 goes EOL on 4/28. http://fridge.ubuntu.com/2017/03/15/ubuntu-12-04-precise-pangolin-reaches-end-of-life-on-april-28-2017/
The other changes are super minor tweaks to the testing cookbooks and the removal of a duplicate line in the Travis YAML that tripped up my YAML checks

This avoids trying to add opensuse testing for now, but does remove ubuntu 12.04 and perform a tiny bit of cleanup


[Bug Report] Unable to train Mamba SAE

Describe the bug
Error running with Mamba: 'HookedMamba' object has no attribute 'W_E'.
Code example
cfg = LanguageModelSAERunnerConfig(
    model_name="state-spaces/mamba-2.8b",
    model_class_name="HookedMamba",
    ...
)
sae = SAETrainingRunner(cfg).run()

Full code: https://github.com/joelburget/mamba-sae/blob/2f87fb99660516c47121aa7a0f65d8944c42778b/hyperparam_sweep.py
Traceback (most recent call last):
  File "/workspace/mamba-sae/hyperparam_sweep.py", line 70, in <module>
    sae = SAETrainingRunner(cfg).run()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 82, in __init__
    self._init_sae_group_b_decs()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 170, in _init_sae_group_b_decs
    layer_acts = self.activations_store.storage_buffer.detach()[:, 0, :]
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 376, in storage_buffer
    self._storage_buffer = self.get_buffer(self.half_buffer_size)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 545, in get_buffer
    refill_batch_tokens = self.get_batch_tokens(
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 410, in get_batch_tokens
    return torch.stack(sequences, dim=0).to(self.model.W_E.device)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'HookedMamba' object has no attribute 'W_E'. Did you mean: 'W_K'?

Full output
root@39393c1a11a5:/workspace/mamba-sae# wandb agent PEAR-ML/mamba-sae-sweep/i0mhj40q
wandb: Starting wandb agent 🕵
2024-10-01 15:28:00,877 - wandb.wandb_agent - INFO - Running runs: []
2024-10-01 15:28:01,254 - wandb.wandb_agent - INFO - Agent received command: run
2024-10-01 15:28:01,254 - wandb.wandb_agent - INFO - Agent starting run with config:
	learning_rate: 0.0012748220954754614
	sparsity_penalty: 0.0922662545654348
2024-10-01 15:28:01,259 - wandb.wandb_agent - INFO - About to run command: /usr/bin/env python hyperparam_sweep.py --learning_rate=0.0012748220954754614 --sparsity_penalty=0.0922662545654348
Resolving data files:   0%|                                                          | 0/37 [00:00<?, ?it/s]
2024-10-01 15:28:06,273 - wandb.wandb_agent - INFO - Running runs: ['ffh3gcsk']
Resolving data files: 100%|█████████████████████████████████████████████████| 37/37 [00:04<00:00,  8.95it/s]
wandb: Currently logged in as: joelb (PEAR-ML). Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.18.2
wandb: Run data is saved locally in /workspace/mamba-sae/wandb/run-20241001_152811-ffh3gcsk
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run dauntless-sweep-4
wandb: ⭐ View project at https://wandb.ai/PEAR-ML/mamba-sae-sweep
wandb: 🧹 View sweep at https://wandb.ai/PEAR-ML/mamba-sae-sweep/sweeps/i0mhj40q
wandb: 🚀 View run at https://wandb.ai/PEAR-ML/mamba-sae-sweep/runs/ffh3gcsk
Run name: 12288-L1-0.0922662545654348-LR-0.0012748220954754614-Tokens-3.000e+07
n_tokens_per_buffer (millions): 4.194304
Lower bound: n_contexts_per_buffer (millions): 0.004096
Total training steps: 7324
Total wandb updates: 73
n_tokens_per_feature_sampling_window (millions): 4194.304
n_tokens_per_dead_feature_window (millions): 20971.52
We will reset the sparsity calculation 7 times.
Number tokens in sparsity calculation window: 4.10e+06
Using Ghost Grads.
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
Moving model to device:  cuda
Resolving data files: 100%|████████████████████████████████████████████████| 37/37 [00:00<00:00, 114.64it/s]
Traceback (most recent call last):
  File "/workspace/mamba-sae/hyperparam_sweep.py", line 70, in <module>
    sae = SAETrainingRunner(cfg).run()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 82, in __init__
    self._init_sae_group_b_decs()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 170, in _init_sae_group_b_decs
    layer_acts = self.activations_store.storage_buffer.detach()[:, 0, :]
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 376, in storage_buffer
    self._storage_buffer = self.get_buffer(self.half_buffer_size)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 545, in get_buffer
    refill_batch_tokens = self.get_batch_tokens(
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 410, in get_batch_tokens
    return torch.stack(sequences, dim=0).to(self.model.W_E.device)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'HookedMamba' object has no attribute 'W_E'. Did you mean: 'W_K'?

System Info
root@39393c1a11a5:/workspace/mamba-sae# cat requirements.txt
torch>=2.2.0
transformers[sentencepiece]>=4.39.2
accelerate>=0.27.2
datasets>=2.15.0
wandb
sae-lens[mamba]
root@39393c1a11a5:/workspace/mamba-sae# uname -a
Linux 39393c1a11a5 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@39393c1a11a5:/workspace/mamba-sae# python --version
Python 3.10.12

Checklist

[x] I have checked that there is no similar issue in the repo (required)

I was also able to repro with tutorials/mamba_train_example.py after a fresh clone:

Fixing a couple other errors first
Error 1
root@39393c1a11a5:/workspace/SAELens# python3 tutorials/mamba_train_example.py
Run name: 131072-L1-1.2e-05-LR-0.0004-Tokens-3.000e+08
n_tokens_per_buffer (millions): 0.524288
Lower bound: n_contexts_per_buffer (millions): 0.004096
Total training steps: 73242
Total wandb updates: 732
n_tokens_per_feature_sampling_window (millions): 524.288
n_tokens_per_dead_feature_window (millions): 2621.44
We will reset the sparsity calculation 73 times.
Number tokens in sparsity calculation window: 4.10e+06
Using Ghost Grads.
Traceback (most recent call last):
  File "/workspace/SAELens/tutorials/mamba_train_example.py", line 57, in <module>
    SAETrainingRunner(cfg).run()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 57, in __init__
    self.model = load_model(
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/load_model.py", line 39, in load_model
    HookedMamba.from_pretrained(
TypeError: HookedMamba.from_pretrained() got an unexpected keyword argument 'center_writing_weights'

Diff
diff --git a/tutorials/mamba_train_example.py b/tutorials/mamba_train_example.py
index d76a673..a055581 100644
--- a/tutorials/mamba_train_example.py
+++ b/tutorials/mamba_train_example.py
@@ -52,6 +52,7 @@ cfg = LanguageModelSAERunnerConfig(
         "fast_ssm": True,
         "fast_conv": True,
     },
+    model_from_pretrained_kwargs={}
 )

Error 2
Traceback (most recent call last):
  File "/workspace/SAELens/tutorials/mamba_train_example.py", line 58, in <module>
    SAETrainingRunner(cfg).run()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 66, in __init__
    self.activations_store = ActivationsStore.from_config(
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 69, in from_config
    return cls(
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 218, in __init__
    raise ValueError(
ValueError: pretokenized dataset has context_size 1024, but the provided context_size is 128.

Diff
--- a/tutorials/mamba_train_example.py
+++ b/tutorials/mamba_train_example.py
@@ -27,7 +27,7 @@ cfg = LanguageModelSAERunnerConfig(
     l1_coefficient=0.00006 * 0.2,
     lr_scheduler_name="cosineannealingwarmrestarts",
     train_batch_size_tokens=4096,
-    context_size=128,
+    context_size=1024,
     lr_warm_up_steps=5000,
     # Activation Store Parameters
     n_batches_in_buffer=128,
@@ -52,6 +52,7 @@ cfg = LanguageModelSAERunnerConfig(
         "fast_ssm": True,
         "fast_conv": True,
     },
+    model_from_pretrained_kwargs={}
 )

 SAETrainingRunner(cfg).run()

root@39393c1a11a5:/workspace/SAELens# python3 tutorials/mamba_train_example.py
Run name: 131072-L1-1.2e-05-LR-0.0004-Tokens-3.000e+08
n_tokens_per_buffer (millions): 4.194304
Lower bound: n_contexts_per_buffer (millions): 0.004096
Total training steps: 73242
Total wandb updates: 732
n_tokens_per_feature_sampling_window (millions): 4194.304
n_tokens_per_dead_feature_window (millions): 20971.52
We will reset the sparsity calculation 73 times.
Number tokens in sparsity calculation window: 4.10e+06
Using Ghost Grads.
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
Moving model to device:  cuda
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:00<00:00, 254.39it/s]
Traceback (most recent call last):
  File "/workspace/SAELens/tutorials/mamba_train_example.py", line 58, in <module>
    SAETrainingRunner(cfg).run()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 82, in __init__
    self._init_sae_group_b_decs()
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 170, in _init_sae_group_b_decs
    layer_acts = self.activations_store.storage_buffer.detach()[:, 0, :]
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 376, in storage_buffer
    self._storage_buffer = self.get_buffer(self.half_buffer_size)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 545, in get_buffer
    refill_batch_tokens = self.get_batch_tokens(
  File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 410, in get_batch_tokens
    return torch.stack(sequences, dim=0).to(self.model.W_E.device)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'HookedMamba' object has no attribute 'W_E'. Did you mean: 'W_K'?

Thanks for flagging this. This might be an issue with MambaLens or we
weren't testing something we needed to during changes. Will follow up
shortly.
On Tue, Oct 1, 2024, 9:56 PM Joel Burget @.***> wrote:

I was also able to repro with tutorials/mamba_train_example.py after a
fresh clone:
Fixing a couple other errors first
Error 1
@.***:/workspace/SAELens# python3 tutorials/mamba_train_example.py
Run name: 131072-L1-1.2e-05-LR-0.0004-Tokens-3.000e+08
n_tokens_per_buffer (millions): 0.524288
Lower bound: n_contexts_per_buffer (millions): 0.004096
Total training steps: 73242
Total wandb updates: 732
n_tokens_per_feature_sampling_window (millions): 524.288
n_tokens_per_dead_feature_window (millions): 2621.44
We will reset the sparsity calculation 73 times.
Number tokens in sparsity calculation window: 4.10e+06
Using Ghost Grads.
Traceback (most recent call last):
File "/workspace/SAELens/tutorials/mamba_train_example.py", line 57, in
SAETrainingRunner(cfg).run()
File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 57, in init
self.model = load_model(
File "/usr/local/lib/python3.10/dist-packages/sae_lens/load_model.py", line 39, in load_model
HookedMamba.from_pretrained(
TypeError: HookedMamba.from_pretrained() got an unexpected keyword argument 'center_writing_weights'
Diff
diff --git a/tutorials/mamba_train_example.py b/tutorials/mamba_train_example.py
index d76a673..a055581 100644
--- a/tutorials/mamba_train_example.py
+++ b/tutorials/mamba_train_example.py
@@ -52,6 +52,7 @@ cfg = LanguageModelSAERunnerConfig(
"fast_ssm": True,
"fast_conv": True,
},

model_from_pretrained_kwargs={}
)

Error 2
Traceback (most recent call last):
File "/workspace/SAELens/tutorials/mamba_train_example.py", line 58, in
SAETrainingRunner(cfg).run()
File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 66, in init
self.activations_store = ActivationsStore.from_config(
File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 69, in from_config
return cls(
File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 218, in init
raise ValueError(
ValueError: pretokenized dataset has context_size 1024, but the provided context_size is 128.
Diff
--- a/tutorials/mamba_train_example.py
+++ b/tutorials/mamba_train_example.py
@@ -27,7 +27,7 @@ cfg = LanguageModelSAERunnerConfig(
l1_coefficient=0.00006 * 0.2,
lr_scheduler_name="cosineannealingwarmrestarts",
train_batch_size_tokens=4096,

context_size=128,

context_size=1024,
lr_warm_up_steps=5000,
Activation Store Parameters
n_batches_in_buffer=128,
@@ -52,6 +52,7 @@ cfg = LanguageModelSAERunnerConfig(
"fast_ssm": True,
"fast_conv": True,
},
model_from_pretrained_kwargs={}
)

SAETrainingRunner(cfg).run()
@.***:/workspace/SAELens# python3 tutorials/mamba_train_example.py
Run name: 131072-L1-1.2e-05-LR-0.0004-Tokens-3.000e+08
n_tokens_per_buffer (millions): 4.194304
Lower bound: n_contexts_per_buffer (millions): 0.004096
Total training steps: 73242
Total wandb updates: 732
n_tokens_per_feature_sampling_window (millions): 4194.304
n_tokens_per_dead_feature_window (millions): 20971.52
We will reset the sparsity calculation 73 times.
Number tokens in sparsity calculation window: 4.10e+06
Using Ghost Grads.
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be deprecated in transformers v4.45, and will be then set to False by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
Moving model to device:  cuda
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:00<00:00, 254.39it/s]
Traceback (most recent call last):
File "/workspace/SAELens/tutorials/mamba_train_example.py", line 58, in
SAETrainingRunner(cfg).run()
File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 82, in init
self._init_sae_group_b_decs()
File "/usr/local/lib/python3.10/dist-packages/sae_lens/sae_training_runner.py", line 170, in _init_sae_group_b_decs
layer_acts = self.activations_store.storage_buffer.detach()[:, 0, :]
File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 376, in storage_buffer
self._storage_buffer = self.get_buffer(self.half_buffer_size)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 545, in get_buffer
refill_batch_tokens = self.get_batch_tokens(
File "/usr/local/lib/python3.10/dist-packages/sae_lens/training/activations_store.py", line 410, in get_batch_tokens
return torch.stack(sequences, dim=0).to(self.model.W_E.device)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1688, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'HookedMamba' object has no attribute 'W_E'. Did you mean: 'W_K'?
—
Reply to this email directly, view it on GitHub
https://github.com/jbloomAus/SAELens/issues/311#issuecomment-2387059513,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQPMYZYCAPWVQTOFQAZTMU3ZZMEABAVCNFSM6AAAAABPF3A52SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBXGA2TSNJRGM
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Play,Pause or Stop status is not read correctly

Hello,
the status value (play, pause, stop) "media.state" is not read out correctly. LibreELEC / KODI v18 LEIA

yes if the file is playing the adapter shows paused. Same issue here.

ok


Running a view function from a non-existent design doc gives status 200 instead of 404

Hello.
For your dockerized 2.0.0, when you try to run a view function from a non-existent design document, the http response header gives a status of 200 (OK) and the response body with is a json with zero rows like this:
{
"total_rows": 0,
"offset": 0,
"rows": []
}
According to the 2.0.0 documentation, it should return a 404 (Not Found).  I suspect you may be using an older build as the most recent Couchdb sources that I compiled natively (i.e, not inside a container) does not have this error.

Building from the latest couchdb sources should fix this issue.


flesh out http identifier documentation

Try and clarify some things around identifiers.

⭐ ✒  Huge improvement!

:star:


Implement support for Uint8ClampedArray

This commit implements support for binding APIs that take
Uint8ClampedArray in JS. This is pretty rare but comes up in a
web-sys binding or two, and we're now able to bind these APIs instead
of having to omit the bindings.
The Uint8ClampedArray type is bound by using the Clamped marker
struct in Rust. For example this is declaring a JS API that takes
Uint8ClampedArray:
use wasm_bindgen::Clamped;

#[wasm_bindgen]
extern {
    fn takes_clamped(a: Clamped<&[u8]>);
}

The Clamped type currently only works when wrapping the &[u8], &mut [u8], and Vec<u8> types. Everything else will produce an error at
wasm-bindgen time.
Closes #421

Ah and this commit also performs a drive-by fix to bind all slice types as &mut [T] instead of &mut [u8] and &[T] for everything else. In all cases for web-sys we have to pessimistically assume that the input is being modified as we don't actually know if it will be modified or not.


Rancher Agent 0.60 on Windows Nano 2016

Rancher Versions:
Server:1.4.2
healthcheck:
ipsec:
network-services:
scheduler:
kubernetes (if applicable):
**Docker Version:17.03.0-ee-117.03.0-ee-1
**OS and where are the hosts located? (cloud, bare metal, etc):Windows Nano 2016
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
**Environment Type: (Cattle/Kubernetes/Swarm/Mesos)Cattle
**Steps to Reproduce:Rancher Agent 0.6.0 fails to start.
[nano-server1]: PS C:\Users\Administrator\Documents> get-content C:\ProgramData\rancher\agent.log
time="2017-03-20T11:12:44+11:00" level=error msg="Error while downloading error: Get /scripts/api.
crt: unsupported protocol scheme """
time="2017-03-20T11:12:44+11:00" level=error msg="registration failed. err: Get /scripts/api.crt:
unsupported protocol scheme """
Trying to get rancher agent to start on Windows 2016 Nano, have stopped firewall etc etc docker running is high version
[nano-server1]: PS C:\Users\Administrator\Documents> docker version
Client:
Version:      17.03.0-ee-1
API version:  1.26
Go version:   go1.7.5
Git commit:   9094a76
Built:        Wed Mar  1 00:49:51 2017
OS/Arch:      windows/amd64
Server:
Version:      17.03.0-ee-1
API version:  1.26 (minimum version 1.24)
Go version:   go1.7.5
Git commit:   9094a76
Built:        Wed Mar  1 00:49:51 2017
OS/Arch:      windows/amd64
Experimental: false

@calloes Can you check if your rancher server is reachable from your 2016 windows nano server by the IP you put in the registration url?

Definitely i have rancheros boxes on the same network that work fine, from the nano server i can hit 443 of my rancher server, also the github download of the agent works fine.

I am wondering if the registration url is unreachable(the url is wrong, untrusted certs(self signed certs) from the agent.

Can you share the command you used to register the agent? Please remove any sensitive info.

I have a similar problem:
Rancher Server

OS: CentOS 7.3.1611
Rancher Server Version: v1.6.4
Docker version: 17.06.0-ce

Rancher Host:

OS: Windows Server 2016
Agent version: 0.6.0
Docker version: 17.03.1-ee-3

Environment:

Using a corporate proxy (both http and https), authenticated, with self-signed certificate.

I'm running into the same issue trying to register a Windows 2016 Server as rancher host/agent. @calloes / @linusyong could you figure out a solution/workaround?

I did some digging, here is what I found out: When the agent(.exe) starts it GETs the URL passed to -register-service which returns a bash script defining environment some variables, e.g. CATTLE_URL. If that first GET fails the CATTLE_URL is not defined and when the agent tries to GET CATTLE_URL + '/scripts/api.crt' that effective URL is /scripts/api.crt which the GO runtime cannot deal with.
In my case the first GET failed due to firewall rules. In order to fix that all I had to add static hostname-IP-mappings in C:\Windows\System32\drivers\etc\hosts pointing to the internal IP of the reverse proxy in front of the rancher service et vóila it works.
@calloes, @linusyong please see C:\ProgramData\rancher for agent|panic.log and make sure etc\Cattle\api.crt is not empty (which is another indicator for the service not being reachable).

Closing as stale.


[AMORO-3189] Support loading mixed catalog without ams url on Spark

Why are the changes needed?

Close #3189.
Brief change log

Support loading MixedFormatSparkCatalog without the AMS URL
Support loading MixedFormatSparkSessionCatalog without the AMS URL
Support loading SparkUnifiedCatalog without the AMS URL
Support loading SparkUnifiedSessionCatalog without the AMS URL

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

Thanks for the review! @Aireed


Track runtime variables in control-flow builders

Summary
This adds support for all the new classical runtime variables through
the control-flow builder interface.  In particular, most usefully it
automatically manages the scoping rules for new declarations and inner
variable accesses, and ensures that its built scopes automatically close
over any variables used within them (including making sure nested scopes
do the same thing).
The builder API is factored out a little into an explicit interface
object, with QuantumCircuit getting an explicit implementation of
that.  This is done because the number of separate API methods we would
have needed to pass around / infer was getting overly large, and this
allows us to just use standard virtual dispatch to automatically do the
right thing.
Python doesn't have a way to have an object implement an
interface other than by structural (duck) typing, so to avoid name
leakage and collisions, we instead make QuantumCircuit's
implementation a friend class that handles the inner state on its
behalf.
Not everything control-flow-builder related is factored out into the API
because it didn't seem overly useful to do this, especially when the
overridden behaviour would just have been to throw exceptions.
Details and comments
Depends on #10963 and #10974
Close #10938

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

418 of 445 (93.93%)  changed or added relevant lines in 17 files are covered.
11 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.04%) to 87.107%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

qiskit/circuit/classical/types/ordering.py
18
19
94.74%

qiskit/circuit/controlflow/_builder_utils.py
3
4
75.0%

qiskit/circuit/controlflow/control_flow.py
10
11
90.91%

qiskit/circuit/controlflow/for_loop.py
5
6
83.33%

qiskit/circuit/controlflow/if_else.py
5
6
83.33%

qiskit/circuit/controlflow/switch_case.py
4
5
80.0%

qiskit/circuit/controlflow/while_loop.py
4
5
80.0%

qiskit/circuit/classical/expr/constructors.py
5
7
71.43%

qiskit/circuit/store.py
33
35
94.29%

qiskit/circuit/classical/expr/expr.py
31
35
88.57%

Files with Coverage Reduction
New Missed Lines
%

crates/qasm2/src/expr.rs
1
93.76%

crates/qasm2/src/lex.rs
3
91.41%

qiskit/circuit/controlflow/builder.py
7
92.06%

Totals

Change from base Build<PHONE_NUMBER>:
0.04%

Covered Lines:
61101

Relevant Lines:
70145

💛  - Coveralls


Dead link https://www.visualstudio.com/en-us/products/team-explorer-everywhere-vs.aspx

Redirects to https://docs.microsoft.com/en-us/vsts/java/Docs/tools/eclipse/ which is also dead
Found in docs/git/share-your-code-in-git-eclipse.md

Link in the source doc is fixed


recognize cuda cu and cuh extensions

at present the bazel plugin cannot deal with cu/cuh file extensions for cuda.  Bazel itself has this problem (I will solve that problem upstream, I have some patches for it) but the intelij plugin also doesn't seem to like it all the way.
Is it possible to fix or outline how to add these file extensions to this plugin?

CUDA language support (or any language support other than BUILD / Skylark) isn't something that's provided by the Bazel plugin.
This is a feature request for CLion itself -- closing, as it's a dupe of https://youtrack.jetbrains.com/issue/CPP-866.

@brendandouglas I want to clarify , I did not mean syntax highlighting, for instance, I mean if there are some file extension checks being done for any purposes, that the right things are happening.
Code like this, from blaze/cpp/CFileExtensions.java is a potential source of issue, like in bazel itself:
  static final ImmutableSet<String> C_FILE_EXTENSIONS = ImmutableSet.of("c");
  static final ImmutableSet<String> CXX_FILE_EXTENSIONS =
      ImmutableSet.of("cc", "cpp", "cxx", "c++", "C");

  static final ImmutableSet<String> CXX_ONLY_HEADER_EXTENSIONS =
      ImmutableSet.of("hh", "hpp", "hxx");

OK, thanks for clarifying. Reopening.

The following trivial changes seemed to improve awareness of the files greatly:
--- a/cpp/src/com/google/idea/blaze/cpp/CFileExtensions.java
+++ b/cpp/src/com/google/idea/blaze/cpp/CFileExtensions.java
@@ -23,10 +23,10 @@ final class CFileExtensions {
   // See https://bazel.build/versions/master/docs/be/c-cpp.html#cc_binary.srcs
   static final ImmutableSet<String> C_FILE_EXTENSIONS = ImmutableSet.of("c");
   static final ImmutableSet<String> CXX_FILE_EXTENSIONS =
-      ImmutableSet.of("cc", "cpp", "cxx", "c++", "C");
+      ImmutableSet.of("cc", "cpp", "cxx", "c++", "C", "cu");

   static final ImmutableSet<String> CXX_ONLY_HEADER_EXTENSIONS =
-      ImmutableSet.of("hh", "hpp", "hxx");
+      ImmutableSet.of("hh", "hpp", "ipp", "hxx", "inl", "cuh", );
   private static final ImmutableSet<String> SHARED_HEADER_EXTENSIONS = ImmutableSet.of("h", "inc");

   static final ImmutableSet<String> SOURCE_EXTENSIONS =

Hi, CUDA is natively supported by CLion now, could you consider making this change? Thanks!

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.


Metadata Submitter backend API - OpenAPI specs

Adresses #13
Description

View at: http://editor.swagger.io/?url=https://raw.githubusercontent.com/CSCfi/metadata-submitter/c76d7d711ed5cfdb59b494e740193c6110bfe25a/docs/specification.yml
For error messages we can also use: https://tools.ietf.org/html/rfc7807
For pagination we can use: https://tools.ietf.org/html/rfc5988 in the header the next, see https://developer.github.com/v3/#pagination
Related issues

Fixes #13
Type of change

[x] Documentation
[x] New feature (non-breaking change which adds functionality)

Changes Made

Testing

[x] Tests do not apply

Mentions

@csc-jm @otahontas what do you think about it ?
We can also extend the schemas endpoint where we will add new schemas or update them when we will support custom schemas.

/validate has nothing in the request body?
/submit and /validate respond the same exact way?
Other than that all looks good to me

@csc-jm and @otahontas good points addressed them in new commits, let me know of other changes.


Spikes when large line is used

Describe the bug
When the line width is large, little cursor movements generate spikes.
To Reproduce

Select maximum line width
keep pressing, do small movements

Expected behavior
A roundish shape should be drawn.
Instead, these spikes appear on the screen:

There is some extra context semi-related to this issue that might be useful:
If the code is modified to ensure that every stroke is configured with LineJoin::Round and LineCap::Round, the lines will generally look more like they have been drawn with a circular brush (with a potential performance cost due to the extra triangles needed for that roundness).
However, even with a round brushstroke, spikes can still appear, and I believe the root cause there is https://github.com/nical/lyon/issues/891. One workaround is to split the path whenever that issue would come up, although I haven't tested if that would tank performance.

Thanks for finding out the root cause! I'll probably implement the workaround you have suggested.


Make heading relative in RelativeTargetPose

Makes the heading input relative to the current heading in RelativeTargetPose action space.

Please add a changelog entry since this is an interface change.


Tends to prefer radio button on forms which have checkbox and radio in same form.

For example, a form I was testing has this behaviour
<form id="pageForm">

When you click the below radio, all the input elements in the div are disabled, if they are enabled and vice versa.

    Click to change: <input id="toggleElement" type="radio" name="toggle" onchange="toggleStatus()" />

        This is our example div block.

        Sample Text Box: <input type="text" name="name" />

        Sample Checkbox : <input type="checkbox" name="participate" />

        Sample Radio : <input type="radio" name="bookEarly" />

        Sample Select: <select name="sampleSelect">
                            <option>Option 1</option>
                            <option>Option 2</option>
                        </select>

</form>

but with a lot more options, that's just to get the idea with a made up bit of code.
But for some reason form filler prefers radio buttons over checkboxes, so I found it difficult to test the other elements.

I tried your sample code and don't see a problem. On every occasion the single radio buttons (toggle and bookEarly) were always checked; the checkbox (participate) alternated between checked and un-checked; and the drop down value was randomly selected. Can you please clarify what you were expecting form filler to do?

that's not a great example.
I can say the exact use case actually, it was a long list of checkboxes on a survey form that I was testing on.
At the bottom was a radio which when activated would disable the checkboxes. For some reason, despite them all being in the same form, it would select the radio button instead of the other options most of the time.

The extension uses Math.random() to determine which radio buttons (or checkboxes or dropdown items) to select. It could be that Math.random() coincidentally selected the same radio button. Is this problem consistency reproducible?


Minimal permissions

When running the stack deployment, you can tell CloudFormation to use a particular set of permissions, using the --cloudformation-execution-policies parameter during the CDK bootstrap. This PR includes an AWS IAM policy document with the required permissions to do so.

Also, unrelated to those permissions, we have the permissions for an AWS user to build the AMI with Packer and run the CDK bootstrap. Those permissions are the ones in agent-aws-stack-tester-permissions.json. These are used by the agent-aws-stack-tester AWS user, which is the one that deploys the stack in CI. Those permissions are the same permissions described in our public docs.

Also updated the stack version to 0.1.6 and the dependencies.

Tagging @DamjanBecirovic and @bogyo210 to check it out as well.


WIP: auto-tagging for all resources

@kapilt - I just want to confirm that I am doing the right thing.  I need to look up all resources that support tags and add in this?
actions.register('auto-tag-user', AutoTagUser)

So the work is in determining which resources support tags?

i'd like to get the registries doing event notification and subscription on type registration, so generics like this or cloudwatch can be introspect and attach to relevant types. right now cloudwatch is using the resource metaclass to watch at class construction time.

@kapilt - do you know what I introspect for tags or is that what I have to figure out?  I was looking at the boto3 library to see if I could determine which resources tagging supports, but I did not see an obvious sentinel to indicate tagging (I was looking for a consistent create_tags method which was not present for some resources)

@scotwk I'm working on a PR now that I think fulfills what you're trying to do. I'm working on specifically adding this for cloudfront and vpc though as per my need.

re the registry subscriber, also need to introduce post/final style events, else too many ordering concerns.

@scotwk introspection should be on presence of tag action to add autotag.

@kapilt - is this what you were thinking?  I added in the ability for events (aka hooks) into the registry.  The only one being used right now is the 'final' hook, but I wanted to make it extensible so I also included an event during register and unregister.  These can be removed easily if you think its YAGNI.
The one hook I added will run after all actions are registered and it adds in the auto-tag-user action to anything with a tag action
This is going to have conflicts with #1315 but they should be easily resolved (I'm thinking it will be mainly around the fact that we both removed the AutoTagUser stuff from each of the resources using it).

@kapilt - thanks for the reviews.  I think it's good to go.

Here is a list of resources that support tagging: https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/supported-resources.html#supported-resources-tagging-console

@scotwk I pulled down this PR, the auto-tag-user didn't work for rds-snapshot or app-elb

@JohnTheodore - what issue are you seeing?
$ custodian schema rds-snapshot.actions
load resources
rds-snapshot:
  actions: [auto-tag-user, delete, invoke-lambda, mark-for-op, notify, put-metric,
    region-copy, remove-tag, restore, tag, unmark]

$ custodian schema app-elb.actions
load resources
app-elb:
  actions: [auto-tag-user, delete, invoke-lambda, mark-for-op, modify-listener, notify,
    put-metric, remove-tag, set-s3-logging, set-shield, set-waf, tag]

@kapilt - This is ready to go.  I have resolved the merge conflicts with master and also handled the two comments from your last review.  This is a few months old, so perhaps a new review is in order since the surrounding code has changed a bit.

I pulled the latest code and rds-snapshot works great with auto-tag-user now.
I'll test several other services and message this thread with any issues. thanks for the great PR, this makes my life easier!

@kapilt - done

Request reopening as the following resources are taggable but custodian is not supporting tagging:

waf
waf-regional
Elasticsearch (aka es)

@FireballDWF - this PR was to add auto-tag-user to anything that supports the tag action.  It was not to add the tag action everywhere.  I think it would be more appropriate to open a new issue for any resources missing the tag action


Tweaks break InCallService

I have some tweaks installed and working, when I active InCallService on bootstrap, I can’t make calls with the phone app.
I was trying AudioRecorder, I thought that was the problem but I uninstalled it and the issue persist, so I de activate InCallService on bootstrap, and know works.
Can I inject tweaks on InCallService?

some tweaks have to update for ios16 A12+

Seems like injection into com.apple.InCallService breaks making calls from the Phone app from starting.

some tweaks have to update for ios16 A12+

Seems like injection into com.apple.InCallService breaks making calls from the Phone app from starting.

maybe crash by some tweak,  any crash log?

some tweaks have to update for ios16 A12+

Seems like injection into com.apple.InCallService breaks making calls from the Phone app from starting.

maybe crash by some tweak, any crash log?

I add a krashlog in the link below
https://drive.google.com/drive/folders/1cCvCxXS4gZU0u960PQZDrxoRPmRPoBAc
thanks

some tweaks have to update for ios16 A12+

Seems like injection into com.apple.InCallService breaks making calls from the Phone app from starting.

maybe crash by some tweak,  any crash log?

Don't have any tweaks that would modify it

some tweaks have to update for ios16 A12+

Seems like injection into com.apple.InCallService breaks making calls from the Phone app from starting.

maybe crash by some tweak, any crash log?

I add a krashlog in the link below
https://drive.google.com/drive/folders/1cCvCxXS4gZU0u960PQZDrxoRPmRPoBAc
thanks

Doesn't look like any tweaks are messing with it, only one that concerns me is 'SpringBoardHome'


Flaky withdraw e2e test

The npm e2e test occasionally fails with
 ● execute a withdraw and a deposit in succession

    ContractFunctionExecutionError: The contract function "finalizeWithdrawalTransaction" reverted with the following reason:
    OptimismPortal: proven withdrawal finalization period has not elapsed

    Contract Call:
      address:   0x6509f2a854BA7441039fCE3b959d5bAdd2fFCFCD
      function:  finalizeWithdrawalTransaction((uint256 nonce, address sender, address target, uint256 value, uint256 gasLimit, bytes data))
      args:                                   ({"nonce":"1766847064778384329583297500742918515827483896875618958121606201292619776","sender":"0x976EA74026E726554dB657fA54763abd0C3a0aa9","target":"0x976EA74026E726554dB657fA54763abd0C3a0aa9","value":"1000000000000000000","gasLimit":"21000","data":"0x","withdrawalHash":"0x3c154e4d0dcefb2cb5473ba79c5dbbccd3efd61b79a64c309e06a99f498f7d1c"})

    Docs: https://viem.sh/docs/contract/estimateContractGas
    Version<EMAIL_ADDRESS>
Example from CI: https://app.circleci.com/pipelines/github/celo-org/optimism/1882/workflows/c8d6dc1b-9f24-4fac-9025-2a27052b9584/jobs/34786
Trying it out locally, it only fails for me if place a sleep between the node startup and the test:
--- a/op-e2e/celo/run_all_tests.sh
+++ b/op-e2e/celo/run_all_tests.sh
@@ -25,6 +25,8 @@ for _ in {1..10}; do
   sleep 0.2
 done

+sleep 300
+
 ## Run tests
 echo Start tests
 failures=0

Increasing the l2OutputOracleSubmissionInterval to prevent the L2OO from falling behind does not solve the issue, which is expected since that should not matter when fault proofs are turned on (which is the case for the devnet).
The flakyness started with the celo8 rebase.
@ezdac knows more about how viem deals with the withdrawal finalization period.

Some insight into how viem / our e2e testsuite deals with the withdrawal on L1
The part affected by the error in CI seems to be the withdraw flow on L1:
First the OptimismPrortal.proveWithdrawalTransaction( ) is called:
https://github.com/celo-org/optimism/blob/3d7ab0789da70fcd4ddd9472e7c723cbf5033ee5/packages/contracts-bedrock/src/L1/OptimismPortal.sol#L304-L308
and the block-timestamp at the time of the call is persisted as the proven withdrawals timestamp.
The e2e-testsuite uses viem, which calculates the theoretical wall-time until the L1 block of the withdrawals finalization period is elapsed based on the OptimismPortal.finalizationPeriodSeconds field.
It then waits that amount of time (+ some grace period) and calls the OptimismPortal.finalizeWithdrawal( ) function.
The contract itself checks the passed time based on the block-timestamp and reverts with the condition that the finalization period hasn't in fact elapsed:
https://github.com/celo-org/optimism/blob/3d7ab0789da70fcd4ddd9472e7c723cbf5033ee5/packages/contracts-bedrock/src/L1/OptimismPortal.sol#L339-L346
So to me it seems likely that there is a gradually increasing gap in produced blocks measured in wall-time vs. the theoretical production of L1 blocks when assuming constant block production time.


RTL Support example: object doesn't support property or method 'remove'  error

getting an error object doesn't support property or method 'remove' from line 10395 in ui-grid-unstable.js
the related method:
s.rtlScrollType = function rtlScrollType() {
    if (rtlScrollType.type) {
      return rtlScrollType.type;
    }

    var definer = angular.element('
A
')[0],
        type = 'reverse';

    document.body.appendChild(definer);

    if (definer.scrollLeft > 0) {
      type = 'default';
    }
    else {
      definer.scrollLeft = 1;
      if (definer.scrollLeft === 0) {
        type = 'negative';
      }
    }

    definer.remove();
    rtlScrollType.type = type;

    return type;
  };

this happens once you start scrolling in the RTL Support example

@SoundLogic I'm not seeing this error. Are you getting it in a certain browser?

Yes IE 11, windows 8.1
Jordan Pollard

On Jun 2, 2015, at 11:00, Brian Hann<EMAIL_ADDRESS>wrote:
@SoundLogic I'm not seeing this error. Are you getting it in a certain browser?
—
Reply to this email directly or view it on GitHub.

@SoundLogic Should be fixed with next build.


chore: bdd-spring-1585585879 to 0.0.1

chore: Promote bdd-spring-1585585879 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment