Refactor Key Management in @web5/crypto, @web5/dids, and @web5/credentials

Design Guidelines

The key manager interface must be passed as an argument to all public API methods that require key material. e.g.

DID Creation
Data Signing

MVP should include an in-memory Key Manager implementation.
Consumers of our SDKs should be able to provide their own KeyManager implementations if desired.

Feature Proposal Tasks:

[ ] Propose API surface design prior to implementation. This can be done by creating a draft PR for a respective feature and providing prototypal code or proposing the design in the comments
[ ] Approval Required from a minimum of 2 people

Implementation Tasks:

[ ] Produce/update API reference documentation for any/all methods added to the public API surface. documenting private API methods is optional
[ ] Produce/update relevant example usage
[ ] Test coverage must be provided for all public API methods.
[ ] Tests that consume shared test vectors must all be passing

Proposal
General Design Choices

Use JSON web key (JWK) as the key format.

Conversion utilities provided to convert between JWK and (1) Web Crypto API's CryptoKey and (2) raw Uint8Array keys.

Use IANA JSON Object Signing and Encryption (JOSE) naming conventions for algorithm, curves, etc. whenever they exist.  JOSE style names used when a standard name is draft/unpublished.
All I/O that interacts with private or secret keys is done via reference using a KeyIdentifier type, which is a string value.  Concrete implementations can use any string as the key identifier (e.g. JWK thumbprint, UUID generated by hosted KMS, etc.).
Provide a base crypto interface definition that concrete implementations conform to.
Any implementation of the crypto interface can be passed as an argument to all public API methods that involve key material (e.g., DID creation, VC signing, general data signing/verification, etc.).

Base Interface
Define a CryptoApi interface that all concrete implementations implement/extend:
export interface CryptoApi {

  // Key Generation Operations
  generateKey(options: Web5Crypto.GenerateKeyOptions): Promise<Web5Crypto.KeyIdentifier>;

  // Key Derivation Operations
  deriveBits(options: Web5Crypto.DeriveBitsOptions): Promise<Uint8Array>;
  deriveKey(options: Web5Crypto.DeriveKeyOptions): Promise<Web5Crypto.KeyIdentifier>;

  // Key Wrapping Operations
  wrapKey(options: Web5Crypto.WrapKeyOptions): Promise<Uint8Array>;
  unwrapKey(options: Web5Crypto.UnwrapKeyOptions): Promise<JsonWebKey>;

  // Signature Operations
  sign(options: Web5Crypto.SignOptions): Promise<Uint8Array>;
  verify(options: Web5Crypto.VerifyOptions): Promise<boolean>;

  // Hashing Operations
  digest(options: Web5Crypto.DigestOptions): Promise<Uint8Array>;

  // Cipher Operations
  encrypt(options: Web5Crypto.EncryptOptions): Promise<Uint8Array>;
  decrypt(options: Web5Crypto.DecryptOptions): Promise<Uint8Array>;

  // Import/Export Operations
  importKey(options: Web5Crypto.ImportKeyOptions): Promise<Web5Crypto.KeyIdentifier>;
  exportKey(options: Web5Crypto.ExportKeyOptions): Promise<JsonWebKey>;
}

Concrete Implementations
At a minimum, develop and publish two implementations of the CryptoApi interface:

InMemoryCrypto:

isomorphic implementation (Node.js, web browser, and React Native)
prefers the Web Crypto API cryptographic algorithm implementations but substitutes @noble or node:crypto libs if Web Crypto lacks the required functionality
uses an in-memory key store (Map)

AwsKmsCrypto:

implementation for back-end development in a Node.js runtime environment
limited to the algorithms made available by existing AWS services (initially only AWS KMS).
all private / secret key material is secured by AWS infrastructure

Consumers of our SDKs can develop their own CryptoApi implementations, if desired.  Concrete examples include an implementation for Google Cloud KMS, one that persists un-extractable keys in a web browser, or one that offers both in-memory and local key stores depending on whether keys are ephemeral or long-lived.
Usage Examples
Key Generation
In-memory:
import { InMemoryCrypto } from `@web5/crypto`;

const crypto = new InMemoryCrypto();

const dataEncKeyId = crypto.generateKey({ algorithm: { name: 'A256CTR' }});
console.log(dataEncKeyId) // vDNSXebfrWOE2fzXhJXbRH28tZhuv2FEd9XYow-SpRA

AWS KMS:
import { AwsKmsCrypto } from `@web5/crypto`;
import { KMSClient } from '@aws-sdk/client-kms';

const kmsClient = new KMSClient({ region: 'us-east1' })
const crypto = new AwsKmsCrypto({ kmsClient });

const dataEncKeyId = crypto.generateKey({ algorithm: { name: 'A256CTR' }});
console.log(dataEncKeyId) // vDNSXebfrWOE2fzXhJXbRH28tZhuv2FEd9XYow-SpRA

Overall awesome stuff!
Major pieces of feedback/questions:
First, related to

Encode raw Uint8Array keys to JWK and decode JWKs to raw Uint8Array keys

What's the motivation for supporting raw keys vs only using JWKs?
Related to encryption. I think it's important to make key rotation first class. Cloud managers typically provide some form of automatically doing this, but it really depends on the use case. How do you envision key rotation being facilitated by the CryptoAPI?
Related to naming, I'm curious why CryptoAPI? Did you consider smaller and more focused interfaces?
Lastly, should there be special provisions in order to handle large streams of data (or large files) that need to be signed or encrypted?

Does digest do just hashing, or HMAC? Will there be AlgorithmIdentifiers for hashing functions? Those don't need keys, so it stands out amongst the other key using operations

web5-kt had separate Crypto, KeyManager, and algorithm classes (e.g. Secp256k1). Seems like CryptoApi combines all 3 into one? I like the simplicity of it, but I think there'll be a lot of duplicate code in CryptoApi implementations. Will there be any effort to extract JWK/JOSE, algorithms, or validation to common code?

Having one AlgorithmIdentifier type to cover all the different and incompatible algs feels like we're dooming implementers to have validation at the top of every function. At least KDF could be separate from the encrypt/sign algs

Great questions and feedback @andresuribe87 and @tomdaffurn.

First, related to

Encode raw Uint8Array keys to JWK and decode JWKs to raw Uint8Array keys

What's the motivation for supporting raw keys vs only using JWKs?

The proposal is to only use the JWK key format in the CryptoApi methods.  The suggested "conversion utilities" would make it easy to convert other key formats (e.g., CryptoKey, Uint8Array, etc.) to/from JWK.  These will likely be needed internally when the underlying KMS or cryptographic algorithm implementations use another format.  For example, an implementation that uses AWS KMS will have to convert ASN.1 DER keys to JWK, like web5-kt does in AwsKeyManager.  Another example would be an in-memory or local key store implementation that uses the Web Crypto API for signing, verification, encryption, etc. will have to convert CryptoKey to JWK.

Related to encryption. I think it's important to make key rotation first class. Cloud managers typically provide some form of automatically doing this, but it really depends on the use case. How do you envision key rotation being facilitated by the CryptoAPI?

Good question and a topic that deserves more consideration.  How do we balance the simplicity and consistency of an API that provides a set of common crypto operations with more fine-grained configuration and advanced functionality that is only available in some scenarios (e.g., HSM backed)?  What should be considered a "common" operation?  Will consumers of our SDKs use them to manage key rotation or will they implement these policies directly in the underlying KMS?
I used the Web Crypto API as inspiration, but it obviously doesn't account for features like key rotation.  Additionally, if something like JWK thumbprint is used as a key identifier, how would that work post-key rotation?  We should consider this and other questions to arrive at a consensus.

Related to naming, I'm curious why CryptoAPI? Did you consider smaller and more focused interfaces?

This was partly inspired by the Web Crypto API, but yes other options were considered and I think both @mistermoe and I have prototyped half a dozen versions over the past few months.  In general, the approach was to provide a single API that exposed a common set of cryptographic algorithms including digital signature, cipher, hashing, and key derivation.
A few mentions of other options considered:

Multiple implementations that are operation specific (KeyGenerator, Signer, Cipher, Hasher, etc.) which would be used depending on the need.  For example, a use case that involves JWS might only need the Signer for creating and then later verifying a signature whereas JWE might need both a Signer and Cipher.  Does this improve the developer experience or is easier/intuitive to use a single API to access all of the common crypto operations?
A single API that implements KeyGenerator, Signer, Cipher, Hasher, etc. interfaces such that the API could be used in a unified way or a developer could pull in one of the smaller modules for a specific need.  This would be straightforward to implement in TypeScript but not sure if its possible in Kotlin.
Use a dot-delimited approach (crypto.aes.encrypt(), crypto.secp256k1.sign(), crypto.hmac.sign()).  You still end up having to pass properties to these methods (curves, length, initialization vector, nonce, salt, etc.), so it isn't clear that this buys you much when IDE intellisense .

Lastly, should there be special provisions in order to handle large streams of data (or large files) that need to be signed or encrypted?

That's an insightful question and one I've been thinking about a lot lately.  The current Web Crypto API supports only one-shot encryption and hashing.  Good read on the subject here and here.  A draft community report still under consideration by W3C and browser vendors here.

The suggested "conversion utilities" would make it easy to convert other key formats (e.g., CryptoKey, Uint8Array, etc.) to/from JWK.

Is this something that needs to be exposed as part of the public API? It seems like it's a lower-level implementation detail. In general, I'm a big fan of making public APIs as slim as possible. Smaller public APIs means a smaller surface that needs to be maintained, implemented, and less choice for developers (which translates into less time learning how to use the API). Going from private -> public is much easier than going from public -> private, in case it's something that we really want to expose to the public.

A single API that implements KeyGenerator, Signer, Cipher, Hasher, etc. interfaces such that the API could be used in a unified way or a developer could pull in one of the smaller modules for a specific need. This would be straightforward to implement in TypeScript but not sure if its possible in Kotlin.

Just noting that my strong preference is to have smaller interfaces like the ones mentioned above. It's possible in most languages that I'm familiar with, including Kotlin, golang and C++. I would also slims down the dependencies that are needed, as devs can pull in only what they need, instead of the whole bundled API. Another benefit is that this conversation can be separated into smaller pieces. For instance, I think we should include encryption until we have a rock solid story of how we're going to support key-rotation (I believe it's crucial that we make sure our design promotes the correct way of doing encryption, which include key-rotation).

@frankhinek a couple of additional use cases that I think are important to address.
There is often times where applications need to know whether the private key associated with a public key is available in the instance that implements CryptoAPI. For such cases, it would be important to add a method that can address that. One possible solution is to include containsKey(keyAlias) as part of the API. This assumes that users have a way of going from public key -> key alias. So you would need a way to either:

tell CryptoAPI which keyAlias to use when generating or
have another method in CryptoAPI that does public key -> key alias.

(2) is what we've implemented in kotlin as shown in https://github.com/TBD54566975/web5-kt/blob/bbef724240451e143e5ba72d829d25c371e440dd/crypto/src/main/kotlin/web5/sdk/crypto/KeyManager.kt#L64
But I actually think we should implement (1). Let me elaborate why below.
A second, more advanced, use case that's specific to DidIon (and possibly DidWeb). When a user boots up an application for the n-th time, they will likely want to know whether their instance of CryptoApi contains all the keys necessary to sign and manage a given did. Let's focus on ION for a second. This means that the application will need to know whether the latest recovery and latest update keys are present. The problem here arises when I do updates. An update to the DID document changes the update key. But, anchoring the update might fail. This introduces race conditions that aren't fun to deal with. From the perspective of a user of the CryptoAPI, I believe that this can be fixed if we can pass in arbitrary keyAliases to use (option (1) above), and additionally having a method to rename a keyAlias atomically (or have the ability to have multiple aliases for a single key).
An alternative is to expose transaction functionality with commits and rollbacks, but that sounds very nasty.

@frankhinek many props for always taking the time to thoroughly explain your proposals and the rationale behind decsions.
Re:

ah ok so, if AwsKmsCrypto wanted its verify to work the same way as InMemoryCrypto.verify works, since verifying doesn't necessitate calls to a KMS, i suppose we could just create an instance of InMemoryCrypto inside AwsKmsCrypto and call that instance's verify function?

ah ok so, if AwsKmsCrypto wanted its verify to work the same way as InMemoryCrypto.verify works, since verifying doesn't necessitate calls to a KMS, i suppose we could just create an instance of InMemoryCrypto inside AwsKmsCrypto and call that instance's verify function?

Yes, that's certainly an approach that could be taken.  Ideally implementers have enough of the basic structure and reference implementations in place that they can extend or modify to meet specific use case requirements.

A few thoughts:

byte encoding is not specific enough. for ecdsa keys there are compressed/uncompressed serialization formats, I know RSA has a few different types too. we should provide encoding format recommendations for each key type (or a default practice)
verification needs to add audience as an optional field to make use of the aud claim JWTs have


Our Custom File dbzResizableTitleWindow extends Spark TitleWindow is not resizable

dbzResizableTitleWindow.as : https://paste.apache.org/xorru
dbzResizableTitleWindow is in DBIZLib
In many Setups we are using PickList_Simple which is also in DBIZLib
PickList_Simple's opening and closing tag is controls:dbzResizableTitleWindow
So the PickList_Simple Should be resizable.

I made several changes culminating in commit d8aeacf5d43f366b6646f87edfbc5937ea7be220.
Spark Panel now uses the PanelSkin so hopefully your custom skin will now work better.


Avro enum type, with symbols values not generated in HTML template

Describe the bug
We are using AsyncApi 2.0.0 specifications with avro file as $ref
Given this declaration bloc in avro file :
{
      "type": "enum",
      "name": "WH_INBOUND_TYPE",
      "symbols" : ["Unit", "Inner", "Master", "Pallet"]
    }

We expected to see an HTML bloc rendering as :

but actually displayed as :

How to Reproduce
Here is the Gist with the test.avsc and the asyncapi.yaml definitions : https://gist.github.com/M3lkior/e282c8ebd3ac94759d4f58ebefdb8f49
Use the asyncapi generator (0.5.0 or 1.0.0-rc.11) command, with the<EMAIL_ADDRESS>:
ag asyncapi.yaml @asyncapi/html-template -o testdir

Expected behavior
The expected display behavior :

Ok thanks @derberg . the doc attribute is flagged as optional in the Avro Spec :)

Very sorry @derberg; I came back to my specification after having deepened the avro spec and my avro schema was wrong,
After refactoring my schema, the generation for enum type is good !

@M3lkior no worries mate


fix(versions): update Activiti/activiti-cloud-runtime-bundle-service versions into develop

UpdateBot pushed maven dependency: org.activiti.dependencies:activiti-dependencies to: 7.1.72

UpdateBot commands:
updatebot push-version --kind maven org.activiti.dependencies:activiti-dependencies 7.1.72 org.activiti.api:activiti-api-dependencies 7.1.48 org.activiti.core.common:activiti-core-common-dependencies 7.1.50 org.activiti.build:activiti-parent 7.1.28


Collapse arrows/signs

The older Bootstrap (v. 3.) included glyphicon where arrows/signs (+/-) where located.
In v. 4 it would be nice to have some 'default' arrows/signs to easily configure so it is easy to see that it is a collapse and which state it is in.

i made a pen a view days ago to explain how easy it is to make some sweet effects by only using pseudo classes
https://codepen.io/gijsbotje/pen/XaoYGY
it might be what you are looking for
(keep in mind that this is not pretty and was only for demonstration purposes (although i use the plus minus variant a lot))

@gijsbotje cool thanks. I will have a look at it.

The above simple way would be a good idea to show in the doc. Just to show how easily it can be done.


Bug: Header Increment does not recognize setext-style headings

Describe the Bug
The "Header Increment" rule does not seem to recognize setext-style headers as headers.
How to Reproduce
Steps to reproduce the behavior:

Create an Obsidian document with headers as below
Lint the file
"Header 3" is incorrectly changed to an H1

Example to reproduce issue with
Header 1
========

Header 2
--------

### Header 3

Expected Behavior
Header 1
========

Header 2
--------

### Header 3

Actual Behavior
Header 1
========

Header 2
--------

# Header 3

Device

[ ] Desktop
[X] Mobile

Thanks for reporting this. I don't think that this rule is designed for the style mentioned. Could you link to the style information for the set-text style headings as all headings the linter deals with are the atx/starting hashtag style?

It looks like the equals sign right below the text is considered to be a part of tge heading. I guess we could swap how we identify headers, but that is a lot more extensive than what is mentioned here.

A minus sign below text means an H2. It is unclear as to what the standard format for this is.

https://daringfireball.net/projects/markdown/syntax#header

Markdown supports two styles of headers, Setext and atx.

Setext-style headers are “underlined” using equal signs (for first-level headers) and dashes (for second-level headers). For example:

This is an H1

This is an H2
Any number of underlining =’s or -’s will work.

Gotcha. I was aware of its existence, but was not aware of its actual rules. These rules predate when I started work on the plugin. I can check to see if anything can be done to fix this, but it may be something that is listed as not supported. However that will be determined by what info I have access to in the underlying libraries.
Thanks for clarifying that it is any number of - or = signs.

Here is a question that would need to be answered before I could try to address this issue:

What should happen when an H1 or H2 needs incrementing as the style will be ambiguous?

I can retain the same style until it becomes an H3, but that means going from an H3 to an H2 would not be setext style
I could make this a drop down, but it is looking more and more like I should extract style settings to be something outside of rules and have them passed into rules as params to help allow consistency across different rules and reduce option duplication

Hm. That’s a really good question. I think a setting in the Rule of whether to use setext or atx headers is the answer. Either way it should recognize setext headers, but unless that option is set, it wouldn’t create them.

Yeah it is something that would need to be added. Since this is something that needs to be addressed in all heading rules and not just this one rule, I am going to get to it once I get through the current list of features to work on. I will put it at the bottom of the queue for things to work on above the backlog content.


Update Pages “home”

Automatically generated by Netlify CMS

👷 Deploy Preview for chic-shortbread-66a66b processing.

Name
Link

🔨 Latest commit
579e3bc3f3acc8947e9c3578e8625a758e9bd70b

🔍 Latest deploy log
https://app.netlify.com/sites/chic-shortbread-66a66b/deploys/6553f2771ddb2500088e245b


rm gocode

https://github.com/ddddddO/gtree/issues/229

Codecov Report
All modified lines are covered by tests :white_check_mark:
:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
see 1 file with indirect coverage changes
:loudspeaker: Thoughts on this report? Let us know!.


Typo - DBCC CheckINdent instead of DBCC CheckIdent.

Reporting a typo on the page: The word CHECKINDENT is used instead of CHECKIDENT.

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fa4daf87-623e-2295-5a9d-e8715f75118a
Version Independent ID: 802c7527-6f54-07e1-2f26-4bbf4a29e4e3
Content: Using IDENTITY to create surrogate keys - Azure Synapse Analytics
Content Source: articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-tables-identity.md
Service: synapse-analytics
Sub-service: sql-dw
GitHub Login: @XiaoyuMSFT
Microsoft Alias: xiaoyul

Thanks for the feedback and bringing this to our notice . At this time we are reviewing the feedback and will update the document as appropriate .

@himanshusinha-msft - Can you please change the author to emtehran (MS alias), mstehrani (Github login)?

#assign-julieMSFT

@himanshu-mishra I will make these changes.

#https://github.com/MicrosoftDocs/azure-docs/issues/77480

@pillaianil Thank you for your contribution to Microsoft Docs. I have made the change and you will see it later today. I will go ahead with closing this issue. The @notification mechanism continues to work even after an item is closed, in case further communication is needed. We hope to hear from you again.

#please-close


Hangouts Chat does not work with Okta

Hello,
Coming from Franz, I have an issue with Hangouts Chat with Fredi (issue that was not present with Franz)
I cannot log in my G Suite account for Google Hangouts Chat
My company is using Okta
Expected Behavior
Be able to access my company' Gsuite account for Hangouts Chat
Current Behavior
My company login page is displayed but nothing append when I click on 'Log in'
Possible Solution
I just downloaded the Franz recipe for Hangouts Chat, imported it into Ferdi using the 'Custom Services' and it works as expected
Steps to Reproduce (for bugs)

Add the Hangout Chat and try to configure a Gsuite account behind an Okta login' service

Context

Your Environment

Ferdi Version used: 5.5.0 (x64)
Operating System and version: Windows 10 Pro (1903)

Im having this issue too, this is the error i see on the dev tools:
devicefingerprint:94 Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://chat.google.com') does not match the recipient window's origin ('https://<my-company-name>.okta.com').
window.onload @ devicefingerprint:94
load (async)
(anonymous) @ devicefingerprint:93

possibly related: https://github.com/getferdi/recipes/pull/137

same issue here 🤕


Upgrade to latest LS protocol

Taken over from @MikhailArkhipov's PR #2144

Upgrade to latest LS protocol since we need new CompletionItem field
Also
Fixes #2113
Fixes #2000

@d3r3kk
Please use the PR template.

We can likely improve our vscode-mock loader to catch these problems. Added issue #2165.

@DonJayamanne LOL, I've just taken @MikhailArkhipov's PR and usurped it to fix the missing type in vscode-mock. I'll see if I can make some sense of what you are asking :man_shrugging:.

@DonJayamanne please have a look, I'll submit when the builds run clean.


The demo notebook fails in colab due to not being able to install pycolmap

Here is the log that I get on the first step:
/content/Hierarchical-Localization
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Obtaining file:///content/Hierarchical-Localization
  Preparing metadata (setup.py) ... done
Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (2.0.0+cu118)
Requirement already satisfied: torchvision>=0.3 in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (0.15.1+cu118)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (1.22.4)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (<IP_ADDRESS>)
Requirement already satisfied: tqdm>=4.36.0 in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (4.65.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (3.7.1)
Requirement already satisfied: plotly in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (5.13.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (1.10.1)
Requirement already satisfied: h5py in /usr/local/lib/python3.10/dist-packages (from hloc==1.3) (3.8.0)
ERROR: Could not find a version that satisfies the requirement pycolmap>=0.3.0 (from hloc) (from versions: none)
ERROR: No matching distribution found for pycolmap>=0.3.0
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.3/15.3 MB 72.9 MB/s eta 0:00:00

[2023/04/28 11:58:23 hloc WARNING] pycolmap is not installed, some features may not work.

It installs fine on my local machine, so maybe it's a Python version issue? python --version gives Python 3.10.11 in the colab notebook.

Similar issue. I can't install pycolmap to the project
ERROR: Could not find a version that satisfies the requirement pycolmap (from versions: none)
ERROR: No matching distribution found for pycolmap

Sorry I somehow missed this - I just published a new version of pycolmap, with wheels for 3.10 now available at https://pypi.org/project/pycolmap/#files

Thanks. Everything is working again. 👍

It works now, thank you!


Current version in maven repository?

Ahoi, is there a repository containing the latest version? Central doesn't return anything and your internal repository seems to be 4.x latest, while the current branch shows 5.x beta.
Any plans on updating that?

Never mind, I saw the latest beta release in the tomp2p repo.


Add option flag to parameters

In certain situations, parameters of type Option have a special meaning, usually symbolizing the possible absence of a value in a serialized form. To allow deserialization in these situations, it is required to introspect a parameter's type and in case it is of type Option, allow for the absence of that parameter in the actual deserialized value.
One such example is JSON typeclass derivation. Typically, an option parameter of value None may be omitted in the serialization of objects and will be equivalent to json "null" during deserialization. Unfortunately, it is currently impossible to query the type of a parameter in magnolia, hence making it impossible to deserialize JSON objects that do not set fields for every Option parameter of their case case class.
This patch adds a boolean field "option" to parameters, which is set if the parameter is of type option. See here for an example use-case (or this diff for more context).
It may also be helpful to expose parameters' types instead of the the specific option query proposed, however I'm not sure how a Type from the macro realm can be passed along to userland. I'd be open to suggestions to work on that too.

what do you think of this proposal in general? Anything that needs clarification or refactoring?

Sorry it's taken me so long to notice this PR. For some reason I've not been getting notifications from Magnolia... (and I haven't been proactively checking!)
I think it's essential we support these sorts of derivations. Though I'm wondering whether there is a way to do this which doesn't special-case Option; and I completely get your point about generalizing it to other types (but not having a good way to represent these). There would be a tiny additional runtime performance cost for each isOption field for every Param object.
So I'm going to think about this some more. Ideally, the special casing for Options would be pushed out into a special chained typeclasses for Options, along the lines of,
implicit def option[T: Typeclass]: Typeclass[Option[T]] = ...

Not sure if that would apply here...

I'm not sure I quite understand what you mean by chained typeclasses for Options. In the scenario described, we'd like to know if a field of a case class is an option so that we can modify the serialization of the case class itself (not the field). E.g.
case class A(x: Option[Int])

// without option field serialization, this would fail since the fiel "x" is not defined:
"{}".parseJson.convertTo[A]

// with option field serialization, this works
"{}".parseJson.convertTo[A] == A(None)

I don't see why Option should be treated as special... it feels to me like the annotations are more appropriate for customisations.

My reasoning was that some data formats (such as JSON)  allow sending null values by omitting fields entirely. To be able to deserialize them we need to inspect those omittable values in the parent object, before dereferencing them.
Such omittable fields could very well be defined with annotations, however I think using option types to represent such fields is very convenient and also a fairly common pattern in Scala.

If you need to special case a typeclass then pattern match on it :shrug:

also, if you want to special case a type why not just compare against the TypeName?
I may do something similar in an example json library to distinguish between "null and missing" during decoding.

Happy to reopen if necessary, but I think the solution @fommil describes should work, so I'll close this.


Call setImageInputs function again, and the images are not showing

I set the images on viewDidLoad() function. But in other function I want to replace those images, when I do this, the images are not showing...
Something like this:
    override func viewDidLoad() {
        super.viewDidLoad()
        slideshow.setImageInputs([ImageSource(imageString: "img1")!, ImageSource(imageString: "img2")!, ImageSource(imageString: "img3")!, ImageSource(imageString: "img4")!])
}

@IBAction func changePhoto(){
     slideshow.setImageInputs([AlamofireSource(urlString: "https://thumbs.dreamstime.com/z/flysch-rocks-barrika-beach-sunset-58426273.jpg")!, AlamofireSource(urlString: "https://thumbs.dreamstime.com/z/man-surfboard-beautiful-foggy-beach-boy-running-golden-sunrise-daytona-florida-58532550.jpg")!, AlamofireSource(urlString: "https://thumbs.dreamstime.com/z/woman-putting-mask-her-face-black-cloak-sitting-ground-58291716.jpg")!])
 }

Hi @dasoga, thank you for the report. I'm marking it as a duplicate with #17 so please follow up on the status there.


User Management listing page sorting fails

Tried to sort the listing by Role, got an error message

Go to User Management listing page 2. Try to sort users by role

We closed 2110 and disabled sorting roles in UM page. Created a new ticket for future product owner to allow sorting my roles in UM page in future :)


Airflow: Armada Deferrable Operator undefined object attribute 'poll_interval'

Version 0.5.5 of the Armada Deferrable Operator for Airflow raises an exception due to an undefined object attribute poll_interval:
[2023-09-13, 09:05:01 UTC] {taskinstance.py:1598} ERROR - Trigger failed:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/triggerer_job_runner.py", line 537, in cleanup_finished_triggers
    result = details["task"].result()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/triggerer_job_runner.py", line 615, in run_trigger
    async for event in trigger.run():
  File "/home/airflow/.local/lib/python3.10/site-packages/armada/operators/armada_deferrable.py", line 263, in run
    poll_interval=self.poll_interval,
AttributeError: 'ArmadaJobCompleteTrigger' object has no attribute 'poll_interval'
[2023-09-13, 09:05:01 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
airflow.exceptions.TaskDeferralError: Trigger failure

Closing - this is fixed in commit 291ef41a on Sep 18, 2023.


Training Cycles Just Predicting "O"

Hi,
I've kicked off NeuroNER and the initial randomised output does attempt to predict the labels all OK however subsequent training cycles target higher accuracy by just predicting the default "O" labels.
Is there some option to force ignoring of "O"'s that I'm missing here even if/when the data is possibly skewed?
nz-police-cleaned.txt
Starting epoch 0
Training completed in 0.00 seconds
Evaluate model on the train set
processed 37679 tokens with 295 phrases; found: 37476 phrases; correct: 5.
accuracy:   0.25%; precision:   0.01%; recall:   1.69%; FB1:   0.03
Address: precision:   0.00%; recall:   0.00%; FB1:   0.00  7452
Area: precision:   0.02%; recall:   8.33%; FB1:   0.03  6048
BankAccount: precision:   0.00%; recall:   0.00%; FB1:   0.00  1314
Company: precision:   0.00%; recall:   0.00%; FB1:   0.00  403
Conviction: precision:   0.00%; recall:   0.00%; FB1:   0.00  397
Event: precision:   0.06%; recall:   1.56%; FB1:   0.12  1662
Person: precision:   0.13%; recall:   1.92%; FB1:   0.25  1495
PhoneNumber: precision:   0.14%; recall:   1.85%; FB1:   0.26  723
Rank: precision:   0.00%; recall:   0.00%; FB1:   0.00  1119
Tribe: precision:   0.00%; recall:   0.00%; FB1:   0.00  1893
Vehicle: precision:   0.00%; recall:   0.00%; FB1:   0.00  6299
Website: precision:   0.00%; recall:   0.00%; FB1:   0.00  2486
WorkPlace: precision:   0.00%; recall:   0.00%; FB1:   0.00  6185
Starting epoch 1
Training completed in 122.82 seconds
Evaluate model on the train set
processed 37679 tokens with 629 phrases; found: 0 phrases; correct: 0.
accuracy:  94.19%; precision:   0.00%; recall:   0.00%; FB1:   0.00
Address: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Area: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
BankAccount: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Company: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Conviction: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Event: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Person: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
PhoneNumber: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Rank: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Tribe: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Vehicle: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Website: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
WorkPlace: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Evaluate model on the valid set
processed 37679 tokens with 320 phrases; found: 0 phrases; correct: 0.
accuracy:  97.53%; precision:   0.00%; recall:   0.00%; FB1:   0.00
Address: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Area: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Company: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Conviction: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Event: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Person: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
PhoneNumber: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Tribe: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Vehicle: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Evaluate model on the test set
processed 37679 tokens with 295 phrases; found: 0 phrases; correct: 0.
accuracy:  97.82%; precision:   0.00%; recall:   0.00%; FB1:   0.00
Address: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Area: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
BankAccount: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Company: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Conviction: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Event: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Person: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
PhoneNumber: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Rank: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Tribe: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
Vehicle: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
WorkPlace: precision:   0.00%; recall:   0.00%; FB1:   0.00  0

As per a previous response - let it run for an extra long time i.e. set patience to 100+ or whatever works.
The score will eventually diverge from the really good score e.g. 97% for just predicting "O"s and then start predicting the proper labels with scores of 94%...


Does it also work for ESP8266?

Hi,
I find your tool great. Can I use it for ESP8266?
Is the merging independent of the CPU type?
Thanks!

Hi @jsponz,
It's just merging binaries together, it's not particularly linked to ESP32. I'm also using to merge bin from STM32 and it works just as good.
So give it a try for ESP8266, and let us know if it worked for you :)

Hi,
I have tested it but I have an issue:

Thanks!

I answer myself :-)

I test the output later at home :-)

It works perfect!!! Thanks a lot!!!

@jsponz Thanks for the update :)


Update directives.md

Added assertion on ref directive examples. RefCallback can be undefined, so added optional chaining in the second example.

Is there something I have to do here still or can this only be done by maintainers?

cc: @arthurevans

Hi @Christian24. Sorry for the delay. A lot of us were either on vacation or sick the week after the release and we dropped the ball here.  Looks good to me but I'm going to ask @kevinpschaaf to take a look.

I don't know what the CI bot's problem with this is, but I'm going to give up and merge it...


Doc Enhancement: Track and alert on last-updated topics

It would be good to see an aggregated list of all topics, sorted by last-updated date.
With such a list, we could see which pages are most outdated and give them special attention to review.
This could be automated, but being able to check this manually would be sufficient.

Fixed in #1737.


Deleting select historical records

Not sure if this is related to the situation raised in #254, but the app had a runaway save problem and created a ton of nearly duplicate historical entries which need to be cleared.
Running a routine that found each of the duplicate historical objects from Model.history.first().instance. And then using .delete() seems to delete the original record rather than the history record.
Is it possible to do this through the queryset manager?

Have you tried Model.history.first().delete()? instance is a re-constituted original model. Any kind of database work done on instance is not done on the history table, it's done on the original.

I'm using this app and, initially, I wanted something more "field-oriented" but now I like to have the full info in each historical record.
The only annoying thing is that each.save() will create a new record no matter if there was any change...
I'm not sure if there would be a big impact of adding that check (if all old fields are already loaded in memory I guess it's ok) but maybe something useful would be a management command to eliminate duplicate records (or django admin action since this is well admin-oriented).
I wouldn't mind having that in crontab to cleanup database (instead of doing the check directly) and it would solve also issues like this one.
Would that be of interest?

@fopina exactly – we've had this feature request a few times, but checking for dirty fields on save would definitely be a lot slower. I'm open to having a management command that deletes duplicate records that users can run periodically. Feel free to put up a PR for that.

@fopina @rossmechanic To clean up a dirty model history I ended up building a function like that, but it was quite messy and I'm sure there are better ways to do it. What I've done in the meantime is had programs check for fields worth saving being changed and otherwise use save_without_historical_record (instead of save), but a cleanup crontab would be great. Could be quite memory/process intense thugh.


Updage website links

You are now on https://www.oakeninnovations.com/ , correct? Various links in the repo may need an update :wink:

We are.
On Tue, Apr 30, 2019, 11:29 AM Dan Shields<EMAIL_ADDRESS>wrote:

You are now on https://www.oakeninnovations.com/ , correct? Various links
in the repo may need an update 😉
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Oaken-Innovations/tesla-tollbooth-acorn/issues/1, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAOG4ETOHIRKY7TI6VF7VBDPTBXVZANCNFSM4HJOHV7Q
.


--temperature and --top_p ignored when using inference server

Temperature is always 0 and top_p is always 1. Where can that be changed?
I'm successfully using KoboldCPP as an inference server.

Please set do_sample to True first.  Sampling doesn't occur otherwise.  It's described in the UI and FAQ and is normal for HF transformers inference, so we go by that.   If you still have issues let me know, but will close for now.

Thank you, that solved it. Interestingly, I get much more factual results with temp=0.1 and top_p=0.3.

@wypiki It might be just anecdotal, it's easy to be persuaded by sampling a few cases after you switched.


Wont link to JS

ignore the last picture

Hey Zach
It looks like you're .js file is on the same level as your .html and that there isn't an a scripts folder? And I think the .css file is the same. Yet you reference both as if they were inside other folders.
If you change the css, does that have any effect on the page? If not, might be an indication that your references are incorrect.
Let me know if that helps!

Yes that worked once i put them into separate folders. Thank you again for the help.


Opensora-hpcai: support modelarts training

Adds # (feature)

Support modelarts training of Opensora
Add msrun launch script to common module mindone/utils/modelarts

Before submitting

[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline?
[x] Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
documentation guidelines
[x] Did you build and run the code without any errors?
[x] Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
[x] Did you write any new necessary tests?

for common shell scripts, we should put them in mindone/tools/. instead of mindone/mindone/utils

modelarts related scripts are moved to mindone/tools


Typo Fix: Capitalization

Motivation
Typographical mistake
Modifications
Capitalization
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (/ no)
The public API: (/ no)
The schema: (/ no /)
The default values of configurations: (/ no)
The wire protocol: (/ no)
The rest endpoints: (/ no)
The admin cli options: (/ no)
Anything that affects deployment: (/ no /)

Documentation
For contributor
For this PR, do we need to update docs?
No, this PR updates the docs, but past versions are affected.
For committer
For this PR, do we need to update docs?

If yes,

if you update docs in this PR, label this PR with the doc label.

if you plan to update docs later, label this PR with the doc-required label.

if you need help on updating docs, create a follow-up issue with the doc-required label.

If no, label this PR with the no-need-doc label and explain why.

Hi, other files with the same problem have been updated. Thanks for taking the time to review my changes!


Enable click events on tracks and custom modals

Description

What was changed in this pull request?

Tracks and plugin tracks can now implement onMouseClick(), which allows them to react to click events on the corresponding track. Furthermore they can use the method showCustomTrackDialog to open modals and display custom data in them (e.g. in response to a click on the track).

Why is it necessary?

I want to be able to display additional data in a modal when a certain part of a track is clicked. showCustomTrackDialog expects a react component with props as input. That react component can live in the plugin track repository. This makes it very flexible which the data we can show in that modal and how we present it.
Here is an example (implemented in a plugin track):
onMouseClick(){
  if(!this.mouseClickData) return;

  const chr = this.mouseClickData.chrName;
  const pos = this.mouseClickData.from - this.mouseClickData.chrOffset;

  const props = {
    chr: chr,
    pos: pos,
    dataFetcher: this.variantDetailFetcher // the data shown will be loaded from an external source
  }

  const data = {
    title: `Variant ${chr}:${format(',')(pos)}`,
    bodyComponent: VariantDetails, // this is a react component
    bodyProps: props
  };
  this.showCustomTrackDialog(data);
}

Here is a demo:
https://gyazo.com/9bfe2138895c3f414ed631c2ded706e0
Fixes #___
Checklist

[x] Set proper GitHub labels (e.g. v1.6+, ignore if you don't know)
[ ] Unit tests added or updated
[ ] Documentation added or updated
[ ] Example(s) added or updated
[ ] Update schema.json if there are changes to the viewconf JSON structure format
[x] Screenshot for visual changes (e.g. new tracks or UI changes)
[x] Updated CHANGELOG.md

I know that there is already the app.click event that you use for the annotation tracks, but as far as I can tell this only works when clicking on a pixi graphics object (using its mouseup method). Since I need to display a lot of elements that are clickable, I can't create separate graphics object for each of them. Therefore, I capture track-wide clicks and figure out afterwards, if something of interest has been clicked (like we do it for getMouseOverHtml).

What happens if two tracks which are overlayed and try to display a custom dialog.
Also would be good to update the docs with this functionality.

Sorry, just coming back to this now. I added a paragraph to the docs.
If two tracks are overlayed and you click on a region where both tracks open a dialog, only one would open. I don't know if there is a simple solution to this problem, but it seems to me that this would be rare issue.

That's fair. Another way to do this would be to get the return values from each track's click handler synchronously and then create a dialog from the returned values.
I actually implemented something like that in the resgen-flavored higlass a while ago. Here's the code:
https://github.com/reservoirgenomics/resglass/blob/main/app/scripts/HiGlassComponent.js#L4553
Each track has a synchronous click handler that returns some data. The contents of the app.click event get populated with those return values. Because the tracks return data synchronously rather than subscribing to the app.click event, an outside caller (i.e. application) can subscribe and get notified of click events and then process data from them. Resgen uses this for annotations so that when you click on an annotation, the resgen viewer will show its details in the sidebar.
In this case, you could have tracks implement a clickDialog function that would return a React component. In HiGlassComponent you could call that function alongside the track's click function and build up a list of components to display. That would make it possible to display values from multiple overlaid tracks simultaneously in one dialog.
What do you think?
If you're OK with this approach, I could create a PR for the click handler that's in resglass and then you could stack a PR on top of that with the dialog code?
I think the dialog code is super useful, by the way, we need something like that for the pileup plugin. Event handling is tricky though and I'd prefer we did it in a way that's robust and can handle cases that don't happen often (like having multiple tracks wish to display some dialog data).

Great! I'll try to get a PR up this weekend.

Here it is: https://github.com/higlass/higlass/pull/1098
It's still missing tests but it should provide you something to work off of.


[BUG] File names with non ascii characters results in octal escape sequence output

Is there an existing issue for this?

[X] I have searched the existing issues

Does this issue exist in the latest version?

[X] I'm using the latest release

Describe the bug?
I have a file called bad-char-é.txt
When I do a get changed files character is not decoded correctly

To Reproduce

Create a file called bad-char-é.txt
Modify the file
Run tj-actions/changed-files on the branch, example workflow

name: workflow
on: [pull_request]
jobs:
  job-1:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Get changed files
        id: changed-files
        uses<EMAIL_ADDRESS>      - name: List all changed files
        run: |
          for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
            echo "$file was changed"
          done

What OS are you seeing the problem on?
all
Expected behavior?
Filename should be passed out as bad-char-é.txt
Relevant log output
Run<EMAIL_ADDRESS>  with:
    token: ***
    separator:
    files_separator:

    files_ignore_separator:

    sha: 9b9e2bd41fe87a55f0318f0e239e34664a847454
    since_last_remote_commit: false
    use_fork_point: false
Run # "Set base sha..."
  # "Set base sha..."
  if [[ -n "" ]]; then
    echo "::set-output name=base_sha::"
  elif [[ "false" == "true" && "" != "0000000000000000000000000000000000000000" ]]; then
      echo "::set-output name=base_sha::"
  fi
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
Run # "Calculating the previous and current SHA..."
  # "Calculating the previous and current SHA..."
  bash $GITHUB_ACTION_PATH/diff-sha.sh
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    GITHUB_SERVER_URL: https://github.com
    GITHUB_REPOSITORY: nathangiusti/Deserialize-Power-BI-Report
    GITHUB_BASE_REF: main
    GITHUB_HEAD_REF: bug-test
    GITHUB_ACTION_PATH: /home/runner/work/_actions/tj-actions/changed-files/v18.2
    INPUT_SHA: 9b9e2bd41fe87a55f0318f0e239e34664a847454
    INPUT_BASE_SHA:
    INPUT_TOKEN: ***
    INPUT_PATH:
    INPUT_USE_FORK_POINT: false
changed-files-diff-sha
  Resolving repository path...
  Setting up 'temp_changed_files' remote...
  No 'temp_changed_files' remote found
  Creating 'temp_changed_files' remote...
  Getting HEAD SHA...
  remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
  From https://github.com/nathangiusti/Deserialize-Power-BI-Report
   * [new branch]      main       -> main
   * [new branch]      main       -> temp_changed_files/main
  Verifying commit SHA...
Run<EMAIL_ADDRESS>  with:
    files-separator:

    excluded-files-separator:

    base-sha: 70e93c14ca1f49f991518b9fa41c8b19daca84fb
    sha: 9b9e2bd41fe87a55f0318f0e239e34664a847454
    include-deleted-files: true
    separator: |
    files-from-source-file-separator:

    excluded-files-from-source-file-separator:

    follow-symbolic-links: true
    strip-top-level-dir: true
/usr/bin/git rev-parse --show-toplevel
/home/runner/work/Deserialize-Power-BI-Report/Deserialize-Power-BI-Report
/usr/bin/git diff --diff-filter=D --name-only 70e93c14ca1f49f991518b9fa41c8b19daca84fb 9b9e2bd41fe87a55f0318f0e239e34664a847454

Warning: No match found for specified patterns. Ensure that subdirectory patterns a prefixed with "**/". See: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet
Run bash $GITHUB_ACTION_PATH/entrypoint.sh
changed-files
0s
Run for file in "bad-char-\303\251.txt"; do
  for file in "bad-char-\303\251.txt"; do
    echo "$file was changed"
  done
  shell: /usr/bin/bash -e {0}
bad-char-\303\251.txt was changed

Anything else?
https://github.com/nathangiusti/Deserialize-Power-BI-Report/runs/5575418851?check_suite_focus=true
Example of the bad run
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@nathangiusti  Sorry for the delay this should be resolved now

@jackton1
Do I need to use the quotepath argument for this to work?

@nathangiusti Yes


Analytics View - Filter with created date

Hi Team,
When we try to create a new view with the filter criteria as 'created date' along with the date its taking the timestamp field also and  timestamp is taken as 18:30:00.000Z which is getting fixed with the created date field.  Along with the time its taking the 'N-1' date as created date.

@manojpaidimarri Thank you for the feedback.
Please note this is a documentation forum so our support options are limited.  For product-related issues please move this issue here.
What exactly does the N-1 refer to?  Is the created date off by one day or do you need the time portion removed for easier filtering?

Closing issue now.


Add cosign signing step

Description
Changes proposed in this pull request:

...
...
...

Related issue(s)

kyma-project/test-infra#4119

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

/retest

/retest


Calculation of Alignment and Uniformity

Hi, thank you for the interesting paper.
Im trying to calculate the alignment and uniformity metrics you provided in the paper, and need some clarification on how you came to your results.
Could you specify on which STS task you calculated these metrics? I saw that you added a custom evaluation in sts_filter.py, but there only the STSFilterBenchmarkEval is really used. Does that mean that the metrics are only calculated on this task? Also, you add the alignment and uniformity together over the train, dev and test set here. Is the correct way of going about this, or should this actually be calculated over the combined dataset, like the spearman correlation which is used for reference?
Could you maybe provide the source code used to calculate the metrics?

Unfortunately, it is quite a long time ago, so I only remember some of the evaluation details of this specific experiment. Yes, the analysis is conducted on the averaged STS results (just it is done in papers like SimCSE). I don't see a problem with using train+test data for this experiment since the model is trained on random Wikipedia sentences.


Update marvel configuration.json

Just changing the example text on the private_key variable to avoid confusion.


Images render too large

When rendering images in a jupyter notebook (from matplotlib for example), these images look much bigger in vscode than in nbviewer or github’s jupyter viewer.
Example (see the image at the end):
https://nbviewer.jupyter.org/github/david-cortes/contextualbandits/blob/master/example/online_contextual_bandits.ipynb
https://github.com/david-cortes/contextualbandits/blob/master/example/online_contextual_bandits.ipynb
VSCode version: 1.60.0
Jupyter extension version: v2021.8.2031190714
OS: linux (debian)

Thanks for filing this issue.
I've tested this, and nbviewer has a max width, hence no matter what kind of a monitor you have the width of the notebook is restricted.
Same applies to the GitHub notebook viewer.
The notebooks in VS Code dont have a width restriction,a s VS Code provides editing of notebooks (nbviewer & github viewer are readonly and built purely for vewing of notebooks - i.e. a readonnly experience).
You can however try the centered layout, View: Toggle Centered Layout.

Please do let me know if those options work.


Create a fresh release

The last release is too old https://github.com/Noelware/docker-manifest-action/releases/tag/0.3.1
Could you please make a fresh one?
@auguwu ^

I haven’t added new features, so I haven’t made a new release? I’m assuming a “fresh” release is just for dependency updates?


How to send Embed?

I couldn't send embed, it say "Invalid Form Body". (Code: 50035)
POST | https://discord.com/api/v9/channels/channelid/messages | 400
Can you write how to send embed? Let others benefit as well.

I posted this on Discussions.


Support for ARM

Hi @noopkat , great work on the library.
I recently shifted to arm and wanted to know any similar workarounds

Hi there!
Thanks for your kind words. ARM is some I definitely want AVRGirl to support in the future. I don't know of any current NodeJS libraries which support ARM either, I am sorry to say!

That's Sad.
Can a native C++ module be implemented and just call it from the JS files?
Or what could be the alternative work around?

Mbed have their CLI. Though implemented in python. Was thinking the same way though a universal for most architectures.
https://docs.mbed.com/docs/mbed-os-handbook/en/5.2/getting_started/blinky_cli

ARM support would be possible with JavaScript for sure, I just have to brush up on the different communication protocols it requires, and then implement them in Node 😄 It's not impossible, it'll just take time 😄
Calling python as an exec process from Node would be a hack to get around this for now.

I wanna chip in 🙌🙌🙌.
What's the work around?
On Mon, Feb 13, 2017, 20:36 Suz Hinton<EMAIL_ADDRESS>wrote:

ARM support would be possible with JavaScript for sure, I just have to
brush up on the different communication protocols it requires, and then
implement them in Node 😄 It's not impossible, it'll just take time 😄
Calling python as an exec process from Node would be a hack to get around
this for now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/noopkat/avrgirl/issues/2#issuecomment-279463696, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGOuFzKda0vb8JHhqKIpjDA5UTqsPJLAks5rcJSYgaJpZM4L4PPd
.


oauth for warper bot user

set up authentication method for the warper application to use oauth as a separate bot user

okay this has been added.
The admin would need to specify an existing oauth'd / mediwiki provider bot user e.g.
http://commons.wikimedia.beta.wmflabs.org/wiki/User:Chippy_warper_bot
in the application.yml
mediawiki_bot_user: Chippyy warper bot
if non wiki users are warping the map, the bot user will change the page on behalf of them.


pubsub: make poll wait for messages

Instead of returning immediately if no messages are available,
poll now waits for messages.
Fixes #2108.
@davidtorres @haakonringberg

@davidtorres PTAL

Changes Unknown when pulling ce4799029827e4a532bce840b1b106a80dd0e436 on pongad:pull-wait into ** on GoogleCloudPlatform:master**.

LGTM - this seems to fulfill the requested change.


All records just disappeared in the UI

Description
have the following error in the console:
120Error: <rect> attribute width: A negative value is not valid. ("-2.1666666666666665")

Python, Logfire & OS Versions, related packages (not required)
No response

reported this in slack as well.
I have no query, so this should "just work".

I believe this was caused by backend ingest problems at the time. The browser console errors about <rect> are old an unrelated.


Suppress unnecessary_cast to Function when the cast is allowing violation of avoid_dynamic_calls

The pattern for allowing a dynamic call when the avoid_dynamic_calls is enabled is to explicitly cast to dynamic or Function in the expression that receives the dynamic call. Such a cast should suppress the unnecessary_cast diagnostic. When we cast to Function to allow the dynamic call, we end up with an unnecessary cast.
void foo(Function callback) {
  argument.something(); // avoid_dynamic_calls - this is a correct diagnostic, we want the dynamic function call to be obvious
  (argument as Function).something(); // unnecessary_cast - this should be suppressed
}

Idiomatic usage of the lint is to keep references as Object? throughout as much of the code as possible, and introduce the dynamic only within the expression where a dynamic call is made. We assume backends will optimize a cast from Object? to dynamic. Neither dynamic or Object? is more specific than the other so we lose nothing when we change all dynamic to Object?. Function is more specific than Object?, so we would lose information changing all Function to Object or Object? just to be allowed to cast.
I think we should allow specifically casting a Function to Function when avoid_dynamic_calls is enabled.

A perhaps better option could be to allow as Function


How to deploy clickhouse along with clickhouse-keeper using the clickhouse operator?

Currently the docs still point to Zookeeper and also says Zookeeper is required while Clickhouse 22.3 says Clickhouse-Keeper is production ready! I do see some k8's files for Clickhouse-Keeper but that is sort of implying to run Clickhouse-Keeper cluster separately just like ZK.
I am looking to run clickhouse cluster along with clickhouse keeper embedded into some of the Clickhouse nodes in the cluster. Would be great to have an example using k8's directly or through clickhouse operator?

It's a bit heavy on configuration but it is possible. I'm hoping the operator will get support for this pretty soon but can be manually added.
Here's what I did to get it working:

In the replicaServiceTemplate you'll need to expose ports for zookeeper (2181) as well as raft (9444)
In the serviceTemplate you'll just need to also expose 2181
Set your zookeper host to the LoadBalancer/ClusterIP that the operator creates (and that you have exposed port 2181 on)
You then need to customize each pod so that the pods you want contain a <keeper_server> configuration that has the correct <server_id> for that pod as well as the raft configuration pointing to all the other instances that will be running clickhouse keeper

I haven't fully tested this yet, but initially it seems to work. Sorry for the bad formatting here, but thought I'd share it quickly so others can at least play around with this. The value for the Zookeeper server is set as below:
- host: service-posthog-clickhouse
   port: 2181

This host is generated from the serviceTemplate service-template (the one with generateName: service-{chi}).
I based this off of the example yaml from here. It can use some cleanup (like creating the config file in an init container instead with a shared volume) but I haven't gotten to that yet.
{{- if .Values.clickhouse.enabled }}
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: {{ template "posthog-plural.name" . }}-clickhouse
spec:
  defaults:
    templates:
      serviceTemplate: service-template
      replicaServiceTemplate: replica-service-template
  configuration:
    users:
      {{- template "clickhouse.passwordValue" . }}
      {{ .Values.clickhouse.user }}/networks/ip:
        {{- range $.Values.clickhouse.allowedNetworkIps }}
        - {{ . | quote }}
        {{- end }}
      {{ .Values.clickhouse.user }}/profile: default
      {{ .Values.clickhouse.user }}/quota: default
      {{- if .Values.clickhouse.backup.enabled }}
      {{ .Values.clickhouse.backup.backup_user }}/networks/ip: "<IP_ADDRESS>/0"
      {{ template "clickhouse.backupPasswordValue" . }}
      {{- end}}
      {{- if .Values.clickhouse.additionalUsersConfig }}
      {{- .Values.clickhouse.additionalUsersConfig | toYaml | nindent 6 }}
      {{- end}}
    profiles:
      {{- merge dict .Values.clickhouse.profiles .Values.clickhouse.defaultProfiles | toYaml | nindent 6 }}

    clusters:
      - name: {{ .Values.clickhouse.cluster | quote }}
        templates:
          podTemplate: pod-template
          clusterServiceTemplate: cluster-service-template
          {{- if and (.Values.clickhouse.persistence.enabled) (not .Values.clickhouse.persistence.existingClaim) }}
          dataVolumeClaimTemplate: data-volumeclaim-template
          {{- end }}
        layout:
          {{- toYaml .Values.clickhouse.layout | nindent 10 }}

    settings:
      {{- merge dict .Values.clickhouse.settings .Values.clickhouse.defaultSettings | toYaml | nindent 6 }}

    files:
      events.proto: |
        syntax = "proto3";
        message Event {
          string uuid = 1;
          string event = 2;
          string properties = 3;
          string timestamp = 4;
          uint64 team_id = 5;
          string distinct_id = 6;
          string created_at = 7;
          string elements_chain = 8;
        }
      keeper_config.xml: |
        <clickhouse>
            <include_from>/tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml</include_from>
            <keeper_server incl="keeper_server">
                <path>/var/lib/clickhouse-keeper</path>
                <tcp_port>2181</tcp_port>
                <four_letter_word_white_list>*</four_letter_word_white_list>
                <coordination_settings>
                    <!-- <raft_logs_level>trace</raft_logs_level> -->
                    <raft_logs_level>information</raft_logs_level>
                </coordination_settings>
            </keeper_server>
        </clickhouse>

    zookeeper:
      nodes:
      {{- if .Values.clickhouse.externalZookeeper }}
        {{- toYaml .Values.clickhouse.externalZookeeper.servers | nindent 8 }}
      {{- end }}

  templates:
    podTemplates:
      - name: pod-template
          {{- if .Values.clickhouse.podAnnotations }}
        metadata:
          annotations: {{ toYaml .Values.clickhouse.podAnnotations | nindent 12 }}
          {{- end }}
        {{- if .Values.clickhouse.podDistribution }}
        podDistribution: {{ toYaml .Values.clickhouse.podDistribution | nindent 12 }}
        {{- end}}
        spec:
          {{- if .Values.clickhouse.affinity }}
          affinity: {{ toYaml .Values.clickhouse.affinity | nindent 12 }}
          {{- end }}
          {{- if .Values.clickhouse.tolerations }}
          tolerations: {{ toYaml .Values.clickhouse.tolerations | nindent 12 }}
          {{- end }}
          {{- if .Values.clickhouse.nodeSelector }}
          nodeSelector: {{ toYaml .Values.clickhouse.nodeSelector | nindent 12 }}
          {{- end }}

          {{- if .Values.clickhouse.securityContext.enabled }}
          securityContext: {{- omit .Values.clickhouse.securityContext "enabled" | toYaml | nindent 12 }}
          {{- end }}

          {{- if .Values.clickhouse.image.pullSecrets }}
          imagePullSecrets:
            {{- range .Values.clickhouse.image.pullSecrets }}
            - name: {{ . }}
            {{- end }}
          {{- end }}

          containers:
            - name: clickhouse
              image: {{ template "posthog.clickhouse.image" . }}
              env:
              - name: KEEPER_SERVERS
                value: {{ .Values.clickhouse.layout.replicasCount | quote }}
              - name: RAFT_PORT
                value: "9444"
              command:
                - /bin/bash
                - -c
                - |
                  HOST=`hostname -s` &&
                  DOMAIN=`hostname -d` &&
                  if [[ $HOST =~ (.*)-([0-9]+)-([0-9]+)$ ]]; then
                      NAME=${BASH_REMATCH[1]}
                      ORD=${BASH_REMATCH[2]}
                      SUFFIX=${BASH_REMATCH[3]}
                  else
                      echo "Failed to parse name and ordinal of Pod"
                      exit 1
                  fi &&
                  if [[ $DOMAIN =~ (.*)-([0-9]+)(.posthog.svc.cluster.local)$ ]]; then
                      DOMAIN_NAME=${BASH_REMATCH[1]}
                      DOMAIN_ORD=${BASH_REMATCH[2]}
                      DOMAIN_SUFFIX=${BASH_REMATCH[3]}
                  else
                      echo "Failed to parse name and ordinal of Pod"
                      exit 1
                  fi &&
                  export MY_ID=$((ORD+1)) &&
                  mkdir -p /tmp/clickhouse-keeper/config.d/ &&
                  {
                    echo "<yandex><keeper_server>"
                    echo "<server_id>${MY_ID}</server_id>"
                    echo "<raft_configuration>"
                    for (( i=1; i<=$KEEPER_SERVERS; i++ )); do
                        echo "<server><id>${i}</id><hostname>$NAME-$((i-1))-${SUFFIX}.${DOMAIN_NAME}-$((i-1))${DOMAIN_SUFFIX}</hostname><port>${RAFT_PORT}</port></server>"
                    done
                    echo "</raft_configuration>"
                    echo "</keeper_server></yandex>"
                  } > /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml &&
                  cat /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml &&
                  /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml

              ports:
                - name: http
                  containerPort: 8123
                - name: client
                  containerPort: 9000
                - name: interserver
                  containerPort: 9009
              {{- if .Values.clickhouse.persistence.enabled }}
              volumeMounts:
              {{- if .Values.clickhouse.persistence.existingClaim }}
                - name: existing-volumeclaim
              {{- else }}
                - name: data-volumeclaim-template
              {{- end }}
                  mountPath: /var/lib/clickhouse
              {{- end }}

              {{- if .Values.clickhouse.resources }}
              resources: {{ toYaml .Values.clickhouse.resources | nindent 16 }}
              {{- end }}
            {{- if .Values.clickhouse.backup.enabled }}
            - name: clickhouse-backup
              image: {{ template "posthog_backup.clickhouse.image" . }}
              imagePullPolicy: {{ .Values.clickhouse.backup.image.pullPolicy }}
              command:
                - /bin/bash
                - -c
                - /bin/clickhouse-backup server
              {{- with .Values.clickhouse.backup.env }}
              env:
                {{- toYaml . | nindent 16 }}
              {{- end}}
              ports:
                - name: backup-rest
                  containerPort: 7171
            {{- end }}

    serviceTemplates:
      - name: service-template
        generateName: service-{chi}
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
            - name: clickhouse-keeper
              port: 2181
          type: {{ .Values.clickhouse.serviceType }}
      - name: cluster-service-template
        generateName: service-{chi}-{cluster}
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: ClusterIP
          clusterIP: None
      - name: replica-service-template
        generateName: service-{chi}-{shard}-{replica}
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
            - name: clickhouse-keeper
              port: 2181
            - name: raft
              port: 9444
          type: ClusterIP

    {{- if and (.Values.clickhouse.persistence.enabled) (not .Values.clickhouse.persistence.existingClaim) }}
    volumeClaimTemplates:
      - name: data-volumeclaim-template
        spec:
          {{- if .Values.clickhouse.persistence.storageClass }}
          storageClassName: {{ .Values.clickhouse.persistence.storageClass }}
          {{- end }}
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: {{ .Values.clickhouse.persistence.size | quote }}
    {{- end }}

{{- end }}

@DavidSpek That looks interesting. Is it part of a larger helm chart you could share? Thanks!

@spoofedpacket Along the way we discovered some issues with the original solution I posted above, but I have now edited it. It's part of our helm chart for deploying PostHog using Plural. The current setup is working well and we are using it to run our PostHog production deployment. The template file for the ClickHouse instance can be found here, with the values being set here.
Some interesting things to note. To allow for increasing the amount of shards for scaling, we are using dedicated pod and service templates for the ClickHouse-Keeper replicas see here and here,  and adding the ClickHouse-Keeper configuration file along with the special templates are being set for the 3 replicas of the first shard here.

@spoofedpacket @alexvanolst I've since made a small helm chart that allows you to easily deploy clickhouse using the operator with support for using the built-in clickhouse keeper. See https://github.com/pluralsh/module-library/tree/main/helm/clickhouse

@DavidSpek according to
https://github.com/pluralsh/module-library/blob/main/helm/clickhouse/templates/clickhouse_instance.yaml#L282
is your embedded clickhouse-keeper installation stable now?
did you compare performance for zookeeper vs clickhouse-keeper?

@DavidSpek roger that. Anyway, thank you for your efforts! ;-)

@Slach Would you be open to a contribution that applies a similar configuration in the operator? There it would be possible to do some more advanced logic in terms of distributing the keeper instances across the nodes?

Yes we open to contributions,
kind: ClickHouseKeeper should be implements as a separate CRD
Each instance of clickhouse-keeper shall be deployed as separate statefulset (to allow separate manage)
and could be link inside kind: ClickHouseInstallation and kind: ClickHouseInstallationTemplate CRDs

@Slach Sorry for not getting back to you quicker about this. While I see the value of having a separate CRD for ClickHouseKeeper, would you also be open to adding the functionality for running an embedded ClickHouse Keeper within a ClickHouse installation? For me the main benefit of ClickHouse Keeper is less maintenance and resource overhead since it doesn't require dedicated pods.

@DavidSpek embedded clickhouse-keeper will restrict your scalability
because you need less clickhouse-keeeper instances, then clickhouse-server instances
for example, typical installations clickhouse-server
two replicas per each shard, replicas could be in different DC
and usual clickhouse-keeper installation
1 or 3 instances to quick quorum
odd number of keeper instances per datacenter, to avoid split brain situation

@Slach I am aware of that, and also the fact that in almost all cases you wouldn't want to scale Raft past 5 or 7 nodes due to performance issues. However, the operator could implement some smart logic in terms how many clickhouse-keeper nodes to run and how those are spread across the regular clickhouse nodes. As a quick example:
If nodes < 3, run 1 keeper
If nodes >= 3 and <= 5, run 3 keepers
And then some more comparisons can be done in terms of if the keepers can be split nicely across shards or replicas. It could even take the affinity rules into account so the nodes running keeper are in separate AZs.


Segfault when sopel bot connects

So, I just migrated our matterircd to a docker setup using the latest 42wim/matterircd.
It seems to work fine to connect from irssi.
Things seemed to be working for the past ~12 hours, however (since just now) when I connect my sopel bot (https://github.com/sopel-irc/sopel) the entire server segfaults:
irc_1  | time="2020-10-11T12:40:46Z" level=info msg="New connection: <IP_ADDRESS>:56240" module=matterircd
irc_1  | panic: runtime error: invalid memory address or nil pointer dereference
irc_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0xd0 pc=0x85a019]
irc_1  |
irc_1  | goroutine 58 [running]:
irc_1  | github.com/42wim/matterircd/mm-go-irckit.CmdPrivMsg(0xa4c6c0, 0xc0003a60d0, 0xc0003d8000, 0xc0003a25f0, 0xc0003ca2d0, 0x1)
irc_1  |        /go/src/github.com/42wim/matterircd/mm-go-irckit/server_commands.go:412 +0x999
irc_1  | github.com/42wim/matterircd/mm-go-irckit.commands.Run(0xc0003b80c0, 0xa4c6c0, 0xc0003a60d0, 0xc0003d8000, 0xc0003a25f0, 0x0, 0x0)
irc_1  |        /go/src/github.com/42wim/matterircd/mm-go-irckit/commands.go:55 +0x135
irc_1  | github.com/42wim/matterircd/mm-go-irckit.(*server).handle.func1(0xc0003a60d0, 0xc0003d8000, 0xc0003a25f0)
irc_1  |        /go/src/github.com/42wim/matterircd/mm-go-irckit/server.go:362 +0x7a
irc_1  | created by github.com/42wim/matterircd/mm-go-irckit.(*server).handle
irc_1  |        /go/src/github.com/42wim/matterircd/mm-go-irckit/server.go:361 +0xe5
mattermost-docker-ifm_irc_1 exited with code 2

Hmmm, in the log of the bot I see this:
[2020-10-11 12:53:55,870] sopel.irc.backends   INFO     - Connection accepted by the server...
[2020-10-11 12:53:55,870] sopel.irc            INFO     - Connected.
[2020-10-11 12:53:55,876] sopel.coretasks      INFO     - Joining 1 channels (with JOIN throttle OFF); this may take a moment.
[2020-10-11 12:53:55,891] sopel.bot            ERROR    - Unexpected error (__new__() missing 1 required positional argument: 'version') from matterircd at 2020-10-11 12:53:55.891745. Message was: matterircd 0.20.2-dev o o debugmo
de false
Traceback (most recent call last):
  File "/home/sopel/.local/lib/python3.8/site-packages/sopel-7.0.6-py3.8.egg/sopel/bot.py", line 606, in call
    exit_code = func(sopel, trigger)
  File "/home/sopel/.local/lib/python3.8/site-packages/sopel-7.0.6-py3.8.egg/sopel/coretasks.py", line 228, in parse_reply_myinfo
    bot._myinfo = MyInfo(*trigger.args[0:3])
TypeError: __new__() missing 1 required positional argument: 'version'

For now I reverted to the 0.19.1 image which doesn't give this issue for me (the sopel bot still raises the error, but matterircd doesn't crash because of it ;) )

Should be fixed in master, please let me know if still an issue.


training error

Thank you for the great work.
I followed the usage guide in the github but some errors occurred. Any code not changed.
The following error occurred when copying pretrained model to './pretrained' folder  and executing 'python main.py'.
"OSError: file ./pretrained/config.json not found"
'config.json' file was not found in your github.  When I copied the 'config.json' from original Bert, it was cleared.
After that, I got an error "OSError: file ./pretrained/pytorch_model.bin not found"
Please give me some tips to solve it.
Thanks in advance.
Seungkwon.

Sorry, I forgot to share about pretrained models.
Actually, for preparing "pytorch.bin", you have to convert original tf model to pytorch model using huggingface's convert script.
but it's too much bother to convert model.
Therefore I opened my "pretrained" folder to the public.
This is the link.
Thank you for letting me know.

Thank you for the kind response


POST requests fall into infinite loops with Cookie header being set

POST requests fall into infinite loops with Cookie header being set.
Expected outcome
Well, should be one request only...
Actual outcome
ARC generates infinite number of POST requests.
Versions
App: <IP_ADDRESS>-stable
Chrome: 50.0.2661.75
Platform: Linux (-)
Steps to reproduce

Build a POST request.
Set Cookie header.
Fire the request.

On a side note, man, how come you released such a major upgrade with sooo many problems without possibility to rollback...?

It was such a helpful nice application to use and now... ehh, sigh...

Hi,
Thank you for issue report.
I can't reproduce the problem. Can you clarify what's happening? Are you sure that the server have not closed loop in redirection? Maybe you can paste some logs from the logger (left nav > bottom of the panel > more button).

Trying to reproduce the problem and can not do it because of another problem, I can't get rid of Cookies now, they are attached to all my requests, no matter what I do.

I checked that there are no cookies in the browser.
No cookies in the Raw headers form.

And yet I always get the attached Cookies to the requests, restarted the app and the browser, nothing helps.
Any suggestions?

YOu can clear cookies in settings. If you disable cookie collection this mechanism will not work at all (cookies won't be added to the requests automatically and won't be stored).
Not that it is working as a browser would work in cookie management.

OK, thanks! Finally got rid of them!
And the original problem with infinite requests loop is still there. It doesn't seem to have anything to do with Cookies though, so my initial suggestion is wrong.
The problem happens when the server is trying to redirect the client with 302, the Location header is set to a different URL of course.

So I guess the app is doing well because it is redirecting the request to new URL. If you have an infinite loop in the server (e.g. redirecting from one URL to another and then back to original one) then you have this issue. The app is applying server request.
However there should be an error thrown in this cases like chrome's Error 310 (net::ERR_TOO_MANY_REDIRECTS).

I think I finally got to the root of the problem.
Let's see what's happening step by step.

I do a POST request to the login form with user credentials.
The server responds with 302 redirect to success page and sets session cookies. Apparently the app ignores these cookies (with Collect Cookies enabled in the Settings)!
Next the app gets to the success page and since there are no session cookies set, the server responds with 302 redirect back to the login page.
App does the same original POST request to the login page and authenticates me again!
That's it, I'm in the infinite loop now.

So, questions are the following:

Why cookies ignored in responses with redirect.
Why the application keeps sending the same POST request when with 302 the client should not follow the redirect automatically for POST/PUT/DELETE requests.

Hi,
Thanks for detailed information. It's helpful.
As for the cookies, the app is following session management spec with cookie match algorithms included in the spec. As per test conducted on cookie storage it works as intended. However if there's an issue there it would be helpful if you could add an information of:

what's the domain and path parameter of the cookies set by server. If not set then the request url's domain and path
what's the redirection domain and path
Cookies probably were stored in the session store but not included in the response since the domain or path did not matched (at leas according to the spec).

As for a redirection:

RFC 1945 and RFC 2068 specify that the client is not allowed
to change the method on the redirected request.  However, most
existing user agent implementations treat 302 as if it were a 303
response, performing a GET on the Location field-value regardless
of the original request method. The status codes 303 and 307 have
been added for servers that wish to make unambiguously clear which
kind of reaction is expected of the client.

If most user agents actually do this I think I can make this exception in the app. However right now it follows the spec.
Thanks to this issue I just found another issue:

The different URI SHOULD be given by the Location field in the response. Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s).

Now the transport library only follows redirection only when the Location header is present in the response. It should also check the response body for the URL.

I have a similar problem, and what was causing the problem is that:

the cookie is set by the server and is url-encoded. For example, + is %2B
the cookie is then put in the ARC storage decoded.
when it's sent again to the server, it uses "Cookie: xxx+xxx" instead of url-encoding it. That means the server will convert it to a space, and it doesn't match the original cookie.

@elipoultorak I think it is a separate issue here for cookie parsing and sending.

The 302 status issue has been resolved in referenced issue. Soon it will be released in beta and then stable channel.


SARAALERT-897: Changing LDE when updating monitoring status for HoH results in changing monitoring status of all dependents that are in continuous exposure

Description
Jira Ticket: SARAALERT-897
Changing LDE when updating monitoring status for HoH results in changing monitoring status of all dependents in both workflows that are in continuous exposure when it should only update LDE (not monitoring status), and only update the LDE of dependents in the exposure workflow.
(Bugfix) How to Replicate

Find/create a HoH with dependents in both workflows that have continuous exposure turned on and that are being actively monitored.
Change the monitoring status of the HoH to be "not monitored" and select to apply the monitoring status change to "This monitoree Only"
Select "Yes, household members that are no longer being exposed to a case" when asked about updating the LDE for all household members who have continuous exposure turned on.
Notice that the dependents with continuous exposure turned on have been closed out as well.

There is a video attached to the JIRA ticket showing how to recreate the bug as well
(Bugfix) Solution
Modified the update_status method in patients_controller.rb to have a separate conditional for this specific case when updating the LDE of dependents in the exposure workflow so that only the LDE is updated and not the monitoring status.
Important Changes
Please list important files (meaning substantial or integral to the PR) along with a list of the general changes that should be highlighted for reviewers.
patients_controller.rb

added an additional conditional to check if LDE needs to be updated for dependents in the exposure workflow with continuous monitoring on

MonitoringStatus.js

Modified variable for changing LDE from using apply_to_group_cm_only to apply_to_group_cm_exp_only

Testing
This fix was tested on the following browsers (submitter must check all those that apply):

[x] Chrome
[ ] Firefox
[ ] Safari
[ ] IE11

Please describe the tests needed to verify your changes. Provide instructions for reproducing these tests. List any relevant details for your test configuration.
Test all combinations between the radio buttons in the monitoring status modal!

Another note - this option shouldn't even be shown when the HoH does not have any dependents in Exposure with continuous exposure turn ON I think - I can check with Kelly

Looks like the the static analysis is failing - probably needs some rubocop fixes.

Looks like the the static analysis is failing - probably needs some rubocop fixes.

it's failing on master too...

Looks like the the static analysis is failing - probably needs some rubocop fixes.

it's failing on master too...

When this is merged https://github.com/SaraAlert/SaraAlert/pull/424, rebase over master


Adding shallow clone feature to cap 3

How about tests and checking the CONTRIBUTION notes, sorry as it stands I can't merge this (I know it's frustrating for such a simple change to have to jump through hoops, but after your contribution we have to maintain the code until the end of time!)

+1
fixing tests would be great.

I will update this when I get some time.

Sorry @pwaterz the tests all failed as a result of this, could you take another look please? https://travis-ci.org/capistrano/capistrano/builds/69349041

I have cleaned up the code, looks like someone else already merged some code into master though.

Opened a new pull request to clean some things up https://github.com/capistrano/capistrano/pull/1476


Docker build error in windows

Can anyone say why the highlighted(yellow) warnings displayed while building the docker?

I have created a windows docker and created a linux container inside windows docker.
command for docker build: (cmd)
docker build –f Dockerfile -t pensive_jones .
command for docker run: (cmd)
docker run pensive_jones testing HostName=xxx.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=DY898xsafsfWxqUT6ytgRSAFSDGSDF/tUZU=
@barnstee @hansgschossmann @marcschier @sebader @mregen @rangedsp

the yellow marked lines are not from the docker build, but from the docker run command.
publisher does use a configuration file to load the nodes to publish (this is one option). this file is not there, so you either need to provide it or use iothub direct method calls to configure publisher. pls check out the readme of the repo and let me know if the description is not good enough.

Thank you! I tried by giving this option --pf="" while using docker run command. Even then I am getting the same warning. Could you help me on the same?

the permission denied issue you see in the log output, should go away with this PR: https://github.com/Azure/iot-edge-opc-publisher/pull/131

Nice to see. How about publishernodes.json missing warning? Please help me in this.

This is actually not a warning but a information. Are you planning to use a preconfigured .json file? If yes, then most likely your configuration is not correct.
From: Norman03<EMAIL_ADDRESS>Sent: Freitag, 13. Juli 2018 16:11
To: Azure/iot-edge-opc-publisher<EMAIL_ADDRESS>Cc: Hans Gschoßmann<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Azure/iot-edge-opc-publisher] Docker build error in windows (#130)
Nice to see. How about publishernodes.json missing warning? Please help me in this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fiot-edge-opc-publisher%2Fissues%2F130%23issuecomment-404844851&data=02|01|johanng%40microsoft.com|fbd24227f52a4ce09cdb08d5e8ca7092|72f988bf86f141af91ab2d7cd011db47|1|0|636670878524176513&sdata=%2FGYDkuT4jvGHnJ5ha0u1X5mYHkvrMHc7pQtfl1kQvi8%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHxels0I-frNFt0UcMrDVSnmuOB1Z79Yks5uGKprgaJpZM4VMkG5&data=02|01|johanng%40microsoft.com|fbd24227f52a4ce09cdb08d5e8ca7092|72f988bf86f141af91ab2d7cd011db47|1|0|636670878524186521&sdata=GJytZrSGkdhJDLnCl3NU5BHMySUKA%2FngFtZ1%2B0JWk8I%3D&reserved=0.

I went through my json file and tested by running in native windows application. The application run successfully. But when I run in docker container I use the same json file I get this error.
Can you please put the exact commands you used to run in docker container?

I think now i understand. You need to add a -v mapping option to your docker run. With the latest it should look something like: docker run -v d:\iotedge:/appdata iot-edge-opc-publisher ....
Assuming your publishednodes.json is in d:\iotedge

Thank you! @hansgschossmann
I will check the -v option and reply back in next 2 hours. Can you please suggest me few links for creating a docker linux container?

Great. -v works fine. Let me post the exact steps I followed to run the opcpublisher application in the docker container. I hope this helps you to update in the readme.md

No activity for a while. Closing for now. Pls reopen if required.


No equivalent of join() in PackedStringArray

Godot version
b1063a7
System information
W10
Issue description
join function is rather important for String arrays, but strangely, it's missing in the new PackedStringArray. It was there in PoolStringArray
Steps to reproduce

var array = PackedStringArray(["hello", "world"])
var hellow = array.join()
You expected "hello world", but it was me, Nonexistent function!
:(

Minimal reproduction project
No response

This function now is reversed in Godot 4.0
PackedStringArray(req_godot).join('.') -> '.'.join(PackedStringArray(req_godot))

Most of the languages I used used the old way, i.e. Array.join(). Maybe we could allow both options or at least document in PackedStringArray how to join it.


MOD: version.sh

What problem does this PR solve?
upgrade version to 2.4.3 in version.sh
What is changed and how it works?
Check List
Tests

No code

./run-all-tests profile=scala-2.11


Improvement of the process

It is possible to link to your file paths in the repository within the list of programming languages in the README.md.
This list can help with adding new languages

Yeah, good idea @BDadmehr0.
Do you want to work on this issue ?

Do you want to work on this issue ?

link to  file paths ?

Yes.

yes working on it

Good


Temporary force 9.3 version due to RHEL-36249 (devel-ng)

Forced version 9.3 for AlmaLinux, Rocky, EuroLinux due to:
https://issues.redhat.com/browse/RHEL-36249

Looks good.


[core-lro] Handle non-string status

Because REST-level clients don't do typed serialization/deserialization.

API change check
API changes are not detected in this pull request.


Fix the unsafe repo error which was introduced by the CVE-2022-24765

@snvector Thanks a lot for this pull request. It is in line with the official workaround provided by GitHub in https://github.com/actions/checkout/issues/766.  I am however still deciding on whether it is not better to just fix the git version to 2.34.1-r0 till the Github team fixed the issue on the actions ecosystem level. @snvector and @jlecoeur you can give your opinion on how we should fix this in https://github.com/rickstaa/action-create-tag/issues/10#issuecomment-1104409863.

Your fix seems to work. I will mix this into the master branch after the comments above are resolved.

Without tag: https://github.com/rickstaa/action-test-repo/runs/6172963791?check_suite_focus=true
With tag:  https://github.com/rickstaa/action-test-repo/runs/6173032045?check_suite_focus=true

@snvector I will merge this already since a lot of my repos are waiting on this fix. Feel free to comment if there was a reason you also added
git config --global safe.directory /github/workspace

I will push an extra commit if this line is required.


Handout vs. Jupyter Notebooks

could you provide more clarification on without hidden state that supports any text editor as written in handout's README?

There are two parts:

you can Handout in any text editor, it is the same as editing any plait test script file
the script does not have a hidden state as opposed to Jupyer notebook

As for the state in Jupyter notebooks you can refer to a presentation by Joel Grus, for example.

@epogrebnyak that presentation was convincing to me, hidden states can be real pain.
handout is a great alternative, what else do we have in handout that jupyter fails at?

@code-monk08: the list of planned features is as follows, the more important one is discussion of output formats  in #9.
#25 is a bit about interactive use, #12 deal with Sweave family of tools.  Also a  good TOC here.
Please feel free to contribute your workflow example or feature request.

@code-monk08 As @epogrebnyak pointed out, the main differences are that you can use your own Python editor of IDE rather than the web editor of Jupyter that doesn't seem suitable for power users, at least to me. The other is that there is no hidden state and communication only goes from the Python script to the rendered handout, not the other way around. Handout being a library also means that you can import your resulting script from other files, although I'm not sure if this happens often. What else?  :)

What else?  :)

I think this is already in discussion but adding in more options where one can customize the output HTML even further will be nice..
also I have feature wise implementation ideas.. I will comment those on respective issues.


fix(sqs): remove binary payload variant

Current dependencies on/for this PR:

main

PR #53   👈

This comment was auto-generated by Graphite.


flutter test -d should give a warning/error

Use case
I often mistakenly run things like flutter test -d chrome[^1], with the expectation that my test platform will be Chrome. Instead of erroring or telling me what I did wrong, the tests will be executed with the default platform without a problem. I'll admit that this has tricked me into thinking that my tests passed on Chrome before...
[^1]: That's the correct flag for flutter run, but for flutter test it should be --platform=chrome.
Proposal
Instead, if flutter test receives an invalid -d flag, it should output a suggestion to try the --platform flag. I'd also argue that it shouldn't run the tests at all in that scenario.

-d (--device-id) is defined as a global option in the tool. Perhaps it would be sufficient to move this to the individual commands that actually use this option. See usesTargetOption as an example.


Adding video player

Reconceived this in a much simpler way, now is much more declarative and much simpler.

Looks good!


integration-cli: log error when starting registry

I had troubles figuring out why a test registry was always failing (before realizing a 401), the log I added states clearly why so I thought it was helpful to everyone
Also do not hardcode http status code..
Signed-off-by: Antonio Murdaca<EMAIL_ADDRESS>

LGTM if janky is happy

LGTM