Leap year and general date validation for isDate().

This is a follow up from #418. The one gotcha is that it doesn't accept null as valid which the Date function does. However, this works just fine when I test the function by itself; it may just be an artifact of the testing suite.
Let me know if you'd like to see more tests.

I'm a little bit wary of merging this considering how lenient Date.parse() is and how few tests there currently are. More tests would definitely be appreciated. The ISO8601 tests are a good start: https://github.com/chriso/validator.js/blob/b20a8db53b112eea86dfed9b9d6efabb4cde63de/test/validators.js#L2134

@chriso Just thought I'd let you know that the ISO 8601 dates in the test suite mismatch in some places with the JavaScript definition of valid date.
For example, these are all valid ISO 8601 dates but invalid JavaScript dates:
'2009123'
'2009-123'
'2009-222'
'2009-W01-1'
'2009-W51-1'
'2009-W511'
'2009-W33'
'2009W511'
'2009-05-19 14'
'2009-W21-2'
'2009-W21-2T01:22'
'2009-139'
'20090621T0545Z'
'2007-04-05T24:00'
'2010-02-18T16:23:48,444'
'2010-02-18T16:23:48,3-06:00'
'2010-02-18T16:23.4'
'2010-02-18T16:23,25'
'2010-02-18T16:23.33+0600'
'2010-02-18T16.23334444'
'2010-02-18T16,2283'
'2009-05-19 143922.500'
'2009-05-19 1439,55'

And these are valid JavaScript dates but invalid ISO 8601 dates:
'200905'
'2009-'
'2009-05-19 14:'
'200912-01'

I've updated my tests with ISO 8601 dates accordingly. I'm sure I need more, but I'd love suggestions as to what type of tests to add.

Awesome, thanks. Going by these Date.parse() docs, some RFC 2282 tests might be helpful.
Not sure if this is a valid timezone offset or not, but isDate() isn't consistent here with Date.parse/new Date:
> validator.isDate('2008-01-01 00:00:00 GMT+0030')
false
> new Date('2008-01-01 00:00:00 GMT+0030')
Tue Jan 01 2008 08:30:00 GMT+0900 (KST)
> new Date(Date.parse('2008-01-01 00:00:00 GMT+0030'))
Tue Jan 01 2008 08:30:00 GMT+0900 (KST)

I borrowed some RFC 2822 tests from around the web - I figured it'd be slightly better than trying myself - and I removed tests and checks for non-string types.

LGTM, thanks!


min gzip version of vue.js in the github asset tarball

What problem does this feature solve?
Some distribution systems require to install a specific version from a source tarball (e.g. buildroot). Unfortunately, the current vue.js tarballs available on github do not contain the gziped version of vue.js. Thus, vue-2.6.11/dist/vue.min.js is three times bigger than https://vuejs.org/js/vue.min.js
What does the proposed API look like?
Either gzip dist/vue.min.js before distributing the tarball or provide the min gzip version as an additional file inside the dist/ directory.

My bad, the file is already in the correct size (~90KB). It is during the transfer than it saves some bytes. I close this irrelevant feature request.


Connection with server lost loop

I'm running syncplay server on linux with TLS enabled running on master branch. At some point, Windows clients running 1.6.4a and other lower versions started getting into this disconnect loop that looks like this:
[21:34:51] Attempting to connect to [REDACTED]:8999
[21:34:52] Attempting secure connection
[21:34:52] Successfully reached [REDACTED] ([REDACTED])
[21:34:52] Connection with server lost, attempting to reconnect
[21:34:52] Attempting secure connection
[21:34:52] Connection with server lost, attempting to reconnect
[21:34:52] Attempting secure connection
[21:34:53] Connection with server lost, attempting to reconnect
[21:34:53] Attempting secure connection
[21:34:53] Connection with server lost, attempting to reconnect
...

This only starts happening an indeterminate amount of time after starting up the server (less than 24 hours).
Looking at syncplay server's stdout, augmented by printing out the non-utf-8 string referenced by the error message:
https://gist.github.com/zehric/fba67e7deb8eb2da27e9f45a70024a11
Any idea what's going on here? This is difficult to debug because the issue only starts happening some long period of time after starting up the server, and I'm not even sure if the client drop error has anything to do with the connection loop.

Hi, thanks for using Syncplay. From your gist, line 6 b'\x13BitTorrent protocol\x00\x00\x00\x00 [...] it appears that something else is interfering with port 8999, either in the server or in the clients.
I'd advise you to check what is running on that port on your systems and to switch to another port number. If you are running your own server, you can pick basically any free port.

It turns out that the port number was indeed the problem. Since 8999 is the default port and a lot of torrent clients use that port, they seem to somehow be interfering with Syncplay. I feel like this issue should be kept active though, since Syncplay should be made to be more resistant to other apps sending extraneous data to that port, otherwise DoS can happen pretty easily, whether intentional or not.

I think the relevant line of code is https://github.com/Syncplay/syncplay/blame/master/syncplay/protocols.py#L44 self.dropWithError(getMessage("line-decode-server-error"))
Does anyone want to comment on if they think this line should be removed?

After digging a bit it seems that once you connect to modern torrent network your IP/port gets passed around for a good while. This would perfectly explain observed behavior. Though there are few things of note to consider:

I think it really happen if both torrent client and Syncplay server are only run periodically - it's not really possible for both of them to use the same port at the same time. IMHO this alone makes observed behavior fairly unlikely. Can you confirm that was the case @zehric ?
Syncplay indeed should be resistant to receiving moderate amounts of garbage data. Dropping clients like described in the log you pasted would imply that it at least tried to deal with it correctly. That would basically make this a performance issue.

That said I think it's either a very large volume of garbage data (it normally can handle over 100 concurrently connected clients even on very low end hardware and with no restarts for weeks), or the server is doing something very wrong when dropping clients like this (like leaking crazy amounts of memory, spawning threads without ever closing them etc.). Latter scenario seems likely given how it only occurs after server runs for some time.
@zehric - what was the ballpark number of dropped clients in the stdout? Even just order of magnitude should be useful information.

Some additional info that might be useful is that sometimes waiting long enough does result in clients being able to connect. I'd say ballpark around 50, sometimes less and sometimes more, connection retries would happen before the client successfully connected. Though it hasn't happened since I changed the port so I'm going off of memory here.
After a day of running, I'd say I probably saw less than 50 dropped clients in the stdout.

My syncplay server should be running 24/7 on my raspberry pi at home, so it's not only running periodically. My torrent client is running on my desktop computer, and that does run only periodically, yes.
Port 8999 was exposed to WAN so my friends could connect to my home network. Changing the internal port but not the external one did not solve the issue, so my suspicion is that it's actually external torrent clients sniffing port 8999 that's causing the issue.

This issue isn't a huge priority of mine to solve since it was resolved by just changing the port number, so I'm not too keen on trying to replicate the issue for now. If it comes up again I'll look into it.

I've added a link to this issue from https://syncplay.pl/guide/trouble/ so that others can see how to resolve this issue if it occurs for them. Given that this does not seem to be a commonly occurring problem I will be closing this issue. Thanks for your assistance in helping us better understand this issue and how to resolve it.

I don't wanna open this issue, but how would I enable the "debug mode" that @zehric posted above in a gist?
Appreciate the help.


Payload exploration

What
This PR contains a still-in-progress exploration of having contracts expose a payload function to facilitate signing in wallets and other downstream systems. To make development as fast as possible, I've replicated/modified auth functionality in this crate rather than working in soroban-auth. The final implementation of course would be in soroban-auth.
Earlier discussions of this idea used names like f_payload but in general those names don't comply with the 10 character symbol requirement. The universal payload function is the reaction to that issue. The signature of payload is almost certain to change as this exploration develops.
This also incorporates an idea from @dmkozh about putting all signatures into a single argument at the end of the argument list. Exploration on this part is very preliminary, and is almost certain to change entirely.
cc @leighmcculloch who requested this PR
Why
Making signing easier is a huge piece of our work in the last several weeks.
Known limitations
Too many to enumerate

Added the missing pieces described above and fixed a couple of bugs. A wallet or other downstream system would use this as follows:

Wallet receives request to call Contract::f(x, y, z...)
Wallet calls Contract::has_sig(f). If false, stop here and just submit the request as received. If true, continue to step 3.
Wallet checks that the last parameter to f is an empty map (there might be other things that could be done at this step but this is reasonable for starters)
Wallet calls Contract::payload(f, [args excluding the last parameter which is an empty map], [])
Wallet takes the result of payload and requests the appropriate signatures using the Identifier and SaltedSignaturePayload in each value, then packages those signatures back into a map using the Symbol in each key
Wallet submits Contract::f(x, y, z...) with the last parameter (which was an empty map) replaced by the map of signatures produced in step 5

I know that this flow is pretty complex, but I think it covers an enormous number of cases including interesting cases that involve functions that forward some-but-not-all signatures to other functions. See liquidity_pool_router_with_payload deposit for context on how that works.
This is basically concept-complete at this point so feedback on the idea would be great. Sorry that the implementation is largely hideous, but I'm confident it could be cleaned up.

I forgot to mention that the wallet needs to do some verification on the payloads it receives before requesting signatures. Let me think about exactly what needs to be checked.


refactor: patching ContainerImage to add repositories

This is a followup of what was done in #303
Most notably:

rename the image_already_exists to find_image to reflect what it does - it used to return bool, now it returns the image if found.
only get the image from Pyxis once - no need to get it a second time with the repository in the filter, we can just search through the repositories in the image that we got from the first request
only include the repositories field in the patch payload
for consistency, create the same two repo entries when adding a new repository like we do when creating a brand new image

It seems this mostly covers https://issues.redhat.com/browse/RELEASE-1285 as well - I just need to add tests for pyxis.patch.

I did some testing - I created an image. Then I ran the script again with the same arguments - it skipped the creation. Then I ran it again, but with a different repository - it added the repository. Then I ran it again with repo1 and repo2 and it skipped everything both times. The image I created is here: https://pyxis.stage.engineering.redhat.com/v1/images/id/6746e22dee93ab8a1d8d7d16


Issue with no-duplicate-imports

I don't believe this rule should includeExports, because the following should not be considered an error:
import { foo } from 'bar'

export * from 'bar'

Sure thing.

Thanks :)

5.3.1
standard 7.0.1


DataFetcherResult support for data + errors

In graphql-java v7.0, the DataFetcherResult class has been introduced to follow the GraphQL specification:

If the data entry in the response is not null, the errors entry in the response may contain any errors that occurred during execution. If errors occurred during execution, it should contain those errors.

Unfortunately, when I try to use it with graphql-java-tools 4.3.0, the feature won't work, because the schema parser tries to map the Java types with the defined schema types and it would complain about DataFetcherResult, since it's not a defined type:
public DataFetcherResult<Iterable<User>> createUsers(List<User>)
It would fail with this error:
... Constructor threw exception; nested exception is com.coxautodev.graphql.tools.SchemaClassScannerError: Unable to match type definition (ListType{type=TypeName{name='User'}}) with java type (graphql.execution.DataFetcherResult<java.lang.Iterable<com.test.User>>): Java class is not a List or generic type information was lost: graphql.execution.DataFetcherResult<java.lang.Iterable<com.test.User>>

Is this feature would be available in a coming version?  I saw the master branch is no longer compatible with graphql-java v7.0, so I guess the first part would be to fix this ...
From what I understand the change would be very trivial, we just need to invoke getData() on a DataFetcherReult before calling typeClassMatcher.match.

Once v8.0 merges in, can we get a PR in for this?

Just a note on this, it looks like this feature isn't supported by BatchedExecutionStrategy since that execution strategy is deprecated, so you can't have both at once.


Hugo: close dropdowns on escape key

add eventlistener for escape key in dropdowns

For https://linear.app/usmedia/issue/CUE-322

Approved.
Extra note:
public close(trigger: HTMLElement, target: HTMLElement) {
this.isOpen = false;
const input: HTMLInputElement = this.getInput(target);
    requestAnimationFrame(() => {
        target.classList.remove('is-open');
        trigger.classList.remove('is-active');
        // create custom event for closing autocomplete panel?

        requestAnimationFrame(() => {
            window.setTimeout(() => {
                target.classList.remove('is-active');
            }, 400);
            input.value = '';
        });
    });
}

Whenever I press escape in the browser it errors on input being null (input.value). It's in the menu.ts (widgets) on 111. Should be a null checking.

@JorindeUsMedia Please could you re-review after the recent force push?

Imported as https://review.gerrithub.io/c/cue-lang/cuelang.org/+/1172307

Closed via 7efea6b38c7e9f1afe0ba996cdbc8b86bb29fe94


Chroma returns the same document more than once when use as a retriver

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
import bs4
from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
blog_docs = loader.load()

from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=300,
    chunk_overlap=50)

splits = text_splitter.split_documents(blog_docs)

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(documents=splits,
                                    embedding=OpenAIEmbeddings())

retriever = vectorstore.as_retriever()

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(documents=splits,
                                    embedding=OpenAIEmbeddings())

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
docs = retriever.get_relevant_documents("What is Task Decomposition?")

print(f"number of documents - {len(docs)}")
for doc in docs:
  print(f"document content - `{doc.__dict__}")

the printed values are
document content - {'page_content': 'Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', 'metadata': {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, 'type': 'Document'}
document content - {'page_content': 'Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', 'metadata': {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, 'type': 'Document'}
document content - {'page_content': 'Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', 'metadata': {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, 'type': 'Document'}
document content - {'page_content': 'Resources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.', 'metadata': {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, 'type': 'Document'}
as you can see 3 documents are the same.
I checked and splits contain 52 documents, but the value of
res = vectorstore.get()
res.keys()

len(res['documents'])

is 156, so I think each document is stored 3 times instead of 1.
Error Message and Stack Trace (if applicable)
No response
Description
I'm trying to use chroma as retriever in a toy example and except to get different documents when 'get_relevant_documents' is applied. instead, I'm getting the same document 3 times
System Info
langchain==0.2.1
langchain-community==0.2.1
langchain-core==0.2.3
langchain-openai==0.1.8
langchain-text-splitters==0.2.0
langchainhub==0.1.17
Linux
Python 3.10.12
I'm running on Colab

Hello @amirhagai,
In the code above you run the following fragment 3 times:
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
That means you add documents to vectorestore 3 times.
I don't know if there is any mechanism which prevent saving of duplicates, but if you want to reset state of your vectorstore please run: vectorstore.delete_collection()

Thanks for the clarification!
It actually appears twice as I mistakenly copy paste the same cell twice :)
But I do see that when I run the cell more than once the DB add items to itself, instead of creating new instance. does it the excepted behavior?
Just to make things clear each time I run this cell -
"""
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(documents=splits,
embedding=OpenAIEmbeddings())
"""
Chroma add items. and changes the retrieval even if it's not redefined.
The docs say that the function "Create a Chroma vectorstore from a list of documents."
https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.chroma.Chroma.html
Thanks again :)

I had the same weird behavior while learning about Langchain.
I don't know if it's expected behavior, but I think it's weird and should not be like that. It's confusing because, as @amirhagai mentioned, they are two different instances of "Chroma" wrapper. But internally, they refer to the same chroma collection, which is the default collection: _LANGCHAIN_DEFAULT_COLLECTION_NAME (defined as "lanchain").
Also (and also mentioned by @amirhagai) the docstring and documentation says "Create a Chroma vectorstore" which reinforces the idea of a "new and fresh collection"
I think that at least for classmethod from_text and from_documents, which expect to be intance factories, it should be called delete_collections automatically.

@amirhagai  @mariano22

Use from langchain_chroma.vectorstores import Chroma instead of from langchain_community.vectorstores import Chroma. because  langchain_chroma.vectorstores is maintained regularly.

The behavior of vectorstore = Chroma.from_documents(documents=splits,                                  embedding=OpenAIEmbeddings())  is correct as expected. It will be creating a collection of name  _LANGCHAIN_DEFAULT_COLLECTION_NAME.

If you want to create a separate collection, please specify name of the collection as it is a necessary parameter while defining the chroma client. Example:
vectorstore_1 = Chroma.from_documents(documents=splits, embedding = embedding, collection_name="col")
vectorstore_2 = Chroma.from_documents(documents=splits, embedding = embedding, collection_name="sol")

You can also store them in your local directory since these are currently in your RAM and will be lost as soon as you stop the kernel. Example:
vectorstore = Chroma.from_documents(documents=splits, embedding = embeddings, collection_name="sol", persist_directory="my_dir")

However, I was not able to find a classmethod that returns vectorstore from persisted collection without insering new document. @eyurtsev Please quote me if I am wrong on this.

Thanks, @spike-spiegel-21, for the clarifications.
The confusion came from the fact that the static method specifies a way of creating a vector store, but it's unclear if the collection already exists if it will load it and not overwrite it.
I think it would be more intuitive if .from_collection performs a delete_collection in case the collection name already exists.
But if you don't think the same, at least I would suggest clarifying in the tutorial and, especially, in the documentation: https://api.python.langchain.com/en/latest/vectorstores/langchain_chroma.vectorstores.Chroma.html#langchain_chroma.vectorstores.Chroma.from_documents
(I had to read the code to understand the behaviour or supposing by what I was observing)


Fixing README.md example for addons/info

Issue:
What I did
Fixed a minor syntax error in addon/info/README.md
How to test
Markdown error no testing needed.
Does this need a new example in the kitchen sink apps?
No
Does this need an update to the documentation?
No
If your answer is yes to any of these, please make sure to include it in your PR.

Rebased to stay up to date. Caught this when setting up the config for my company. <3


bug(yaml-parse): attempting to use network policies as resources results in error

Attempting to use a network policy as a managed resource by operator-builder results in the following error:
FATA[0001] failed to create API: unable to inject the resource to "config.operatorbuilder.io/v1": unable to inject resource into .workload/v1alpha2/workload.yaml, error processing file resources/network-policy.yaml; error unmarshaling yaml, yaml: line 14: did not find expected '-' indicator

Here is the workload config:
---
name: tanzu-namespace
kind: StandaloneWorkload
spec:
  api:
    domain: platform.cnr.vmware.com
    group: tenancy
    version: v1alpha2
    kind: TanzuNamespace
    clusterScoped: true
  companionCliRootcmd:
    name: tanzu-ns-ctl
    description: Manage Tanzu Namespaces
  resources:
    # - resources/namespace.yaml
    # - resources/limit-range.yaml
    # - resources/resource-quota.yaml
    - resources/network-policy.yaml

Here is the network policy object:
# NOTE: code markers were added before functionality for complex data types existed.  these are not functional
#       or even syntactically correct but for documentation/visualization purposes only.
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: "tanzu-network-policy"
  namespace: "tanzu-namespace" #+operator-builder:field:name=namespace,type=string
spec:
  podSelector:
    #+operator-builder:field:name=networkPolicy.targetPodLabels,type=string,default="",description=`
    #Labels on pods within this namespace that the network policy applies to.`
    matchLabels: ""  # this is actually a map[string]string but used to build the project for now
  ingress:
    - from:
        - namespaceSelector:

            #+operator-builder:field:name=networkPolicy.ingress.namespaceLabels,type=string,default="",description=`
            #Labels on namespaces that the ingress network policy applies to for pods which match
            #field networkPolicy.targetPodLabels.`
            matchLabels: ""  # this is actually a map[string]string but used to build the project for now

        - podSelector:

            #+operator-builder:field:name=networkPolicy.ingress.podLabels,type=string,default="",description=`
            #Labels on pods that the ingress network policy applies to for pods which match field
            #networkPolicy.targetPodLabels.`
            matchLabels: ""  # this is actually a map[string]string but used to build the project for now

    #+operator-builder:field:name=networkPolicy.ingress.ports,type=[]string,default={},description=`
    #Allowed ingress ports on matches of namespaces from field networkPolicy.ingress.namespaceLabels and
    #for pods from field networkPolicy.ingress.podLabels for pods which match field networkPolicy.targetPodLabels.`
    ports: [] # this is actually an []map[string]string but used to build the project for now
  egress:
    - to:
        - namespaceSelector:

            #+operator-builder:field:name=networkPolicy.egress.namespaceLabels,type=string,default="",description=`
            #Labels on namespaces that the egress network policy applies to for pods which match
            #field networkPolicy.targetPodLabels.`
            matchLabels: ""  # this is actually a map[string]string but used to build the project for now

        - podSelector:

            #+operator-builder:field:name=networkPolicy.egress.podLabels,type=string,default="",description=`
            #Labels on pods that the egress network policy applies to for pods which match field
            #networkPolicy.targetPodLabels.`
            matchLabels: ""  # this is actually a map[string]string but used to build the project for now

    #+operator-builder:field:name=networkPolicy.egress.ports,type=[]string,default={},description=`
    #Allowed egress ports on matches of namespaces from field networkPolicy.egress.namespaceLabels and
    #for pods from field networkPolicy.egress.podLabels.`
    ports: []  # this is actually an []map[string]string but used to build the project for now

Closing.  This was incorrect YAML.


Limit /drip size and delay

See: https://twitter.com/G04TSE/status/631499475706580992

PR @ https://github.com/Runscope/httpbin/pull/242


[WIP] Updating to Lightning 2.0

This is an effort to update the pytorch lightning dependency (see #168 )
A lot has changed since the LNNP module was written (current version is 1.6.4) and I expect this update would increase compatibility with some functionalities we seek (AKA torch compile/torchscript training).

The resume_from_checkpoint option in train.py has disappeared:

    trainer = pl.Trainer(
        strategy=DDPStrategy(find_unused_parameters=False),
        max_epochs=args.num_epochs,
        accelerator="auto",
        devices=args.ngpus,
        num_nodes=args.num_nodes,
        default_root_dir=args.log_dir,
        #resume_from_checkpoint=None if args.reset_trainer else args.load_model,
        callbacks=[early_stopping, checkpoint_callback],
        logger=_logger,
        precision=args.precision,
        gradient_clip_val=args.gradient_clipping,
        inference_mode=False
    )

Trying to see how one is supposed to replace it

From LightningDeprecationWarning: Setting Trainer(resume_from_checkpoint=) is deprecated in v1.5 and will be removed in v1.7. Please pass Trainer.fit(ckpt_path=) directly instead.
So it could be something like this:
if not args.load_model:
   trainer.fit(model, data)
else:
   trainer.fit(model, data, ckpt_path=load_model)

I found this issue in which they discuss why Lighting does not want to support testing during the training loop:
https://github.com/Lightning-AI/lightning/issues/9254
They also describe the trick that is being used now of adding the test dataloader as a validation dataloader so you can use it during training.
However, I do not understand why we are paying such a high price in performance to test during training. @PhilippThoelke , you implemented this AFAIK, would you provide some insights?

Testing during training was useful since it can be difficult to estimate model performance on val loss, when val loss is also used to adjust the learning rate (e.g. through the ReduceLROnPlateau schedule). This is particularly important for fast prototyping and architecture development. I wasn't aware that this slows down training by 6x, that is insane! How much of that is actually due to the testing though, and how much are improvements in Lightning 2.0?
I don't know if that lr scheduler is still relevant for torchmd-net, but a 6x efficiency decrease is never a viable solution.
Possibly related: #27

TBH I do not really know what amount of the speedup is due to other improvements in lighting.
It makes sense to me that not testing is the main cause of the speedup simply because one is not going over a whole dataset every few epochs, which requires reading a bunch of stuff from disk, etc.
I understand the fast prototyping argument. But we are currently going against the lighting devs.
I will try to recreate the multiple-dataloader hack (looking at the docs/github issues I do not see any other workaround) so the functionality is still provided and I can measure if this is indeed the cause of the slowdown.
Well I guess I can also run the baseline without testing during training -.-
If that turns out to be the case then perhaps we can print a warning in the "test during training" case to explain that you are paying a high price for it hehe

I would drop it if it really makes such a huge difference. I don't see how
validation is not a problem and test costs 6 times.
On Tue, Aug 8, 2023 at 7:17 PM Raul @.***> wrote:

TBH I do not really know what amount of the speedup is due to other
improvements in lighting.
It makes sense to me that not testing is the main cause of the speedup
simply because one is not going over a whole dataset every few epochs,
which requires reading a bunch of stuff from disk, etc.
I understand the fast prototyping argument. But we are currently going
against the lighting devs.
I will try to recreate the multiple-dataloader hack (looking at the
docs/github issues I do not see any other workaround) so the functionality
is still provided and I can measure if this is indeed the cause of the
slowdown.
Well I guess I can also run the baseline without testing during training
-.-
If that turns out to be the case then perhaps we can print a warning in
the "test during training" case to explain that you are paying a high price
for it hehe
—
Reply to this email directly, view it on GitHub
https://github.com/torchmd/torchmd-net/pull/210#issuecomment-1670010960,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3KUOT6EZJEZKBHRM2IURLXUJYC7ANCNFSM6AAAAAA3IM2GXM
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Ok since I do not know yet how to reproduce the trick in latest Lighting I ran current main with and without testing during validation to compare:

Note that I was not trying to benchmark when I noticed this, I was checking correctness. Thus this train example must not be the most representative one (it does not fully occupy the GPU), but still...
Testing is carried out every 10 epochs in the orange line.
Roughly it is a x3 slowdown if you test every 10 epochs. Without testing I see a x3 speedup between Lightning 1.6.3 and 2.0.4.
I get testing being super slow because the way it is set up in this example it just goes over the whole dataset each time. Besides being just a lot of forward passes it requires reading the whole thing from disk.
With this I know believe that:

Updating Ligthing is very worth it.
We should keep the testing functionality for prototyping.
We should include a warning explaining that test-during-training can be expensive.

I managed to bring back the functionality.
As far as I can tell the only way to reproduce the test-during-train trick is to reload the dataloader every epoch.
This does not seem to have a profound effect in performance.
Sadly the performance when test-during-training is enabled does not look better than before.
I will test some more to make sure these performance numbers are not a fluke, but functionality-wise this should be ready to go!

I get that testing during training does only matter when you are prototyping, and in fact it helped in my case to stop training after few epochs when some hyperparameter change did not provide any improvement. If we are changing this, I have two suggestions:

Report validation MAEs on both energies and forces on top of (separate) validation loss(es), which are MSEs. This can give a better sense of how the training is going.
Even when no testing is performed during training, I would suggest in that case to provide test errors when the training is finished by default. Then people would not need to run inference separately on the test set to report performances.

Another option is, when the training is finished, look for the checkpoint with lowest validation loss and run test with this one. I think this is what happens in MACE, for example.

Thanks for your insights!
I am convinced test-during-training has some useful cases, luckily I was able to cook it in again so I believe it is best to leave it there as an option at least for now. If you want to go go fast simply do not set the "test_interval" option.

Another option is, when the training is finished, look for the checkpoint with lowest validation loss and run test with this one. I think this is what happens in MACE, for example.

AFAIK this is exactly what torchmd-train does:
https://github.com/torchmd/torchmd-net/blob/dca66796d00680a79a7b7c85d6704d30d15dc84c/torchmdnet/scripts/train.py#L179-L184

Report validation MAEs

As you say, the values reported as "val_loss_y/dy" are mse_loss. I can add another reported value called "val_l1_loss_y/dy". It should have no effect whatsoever on performance.

reporting l2 and l1 loss on validation would be useful
On Mon, Aug 14, 2023 at 9:09 AM Raul @.***> wrote:

Thanks for your insights!
I am convinced test-during-training has some useful cases, luckily I was
able to cook it in again so I believe it is best to leave it there as an
option at least for now. If you want to go go fast simply do not set the
"test_interval" option.
Another option is, when the training is finished, look for the checkpoint
with lowest validation loss and run test with this one. I think this is
what happens in MACE, for example.
AFAIK this is exactly what torchmd-train does:
https://github.com/torchmd/torchmd-net/blob/dca66796d00680a79a7b7c85d6704d30d15dc84c/torchmdnet/scripts/train.py#L179-L184
Report validation MAEs
As you say, the values reported as "val_loss_y/dy" are mse_loss. I can add
another reported value called "val_l1_loss_y/dy". It should have no effect
whatsoever on performance.
—
Reply to this email directly, view it on GitHub
https://github.com/torchmd/torchmd-net/pull/210#issuecomment-1676793285,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3KUORMEYVYBDAC6ZJGN3TXVHFL3ANCNFSM6AAAAAA3IM2GXM
.
You are receiving this because you commented.Message ID:
@.***>

I noticed there was no "on_test_epoch_end" member in LNNP. Meaning that the trainer.test line in train.py effectively just discarded all the work it did as far as I can tell. Also there is no logger attached to the test run, so nothing is written to disk about it.
I wrote one so that test losses are actually reported to the terminal like this:

This is ready to merge on my part.
@AntonioMirarchi, could you rerun some training with this PR to double check?
cc @raimis please review

Test failed for some reason. Can you take a look?

It was just an overprotective numerical check. Just retriggering the CI has fixed it.

In the end is this instrinsically faster or was it just the effect of testing during training?

This small, not representative example, I am using is def faster beyond the test thing.
Although every test I have run turns out faster, I would not expect real life runs to be much better. For instance, the ET-SPICE.yaml test, removing test-during-training, is just about 5% faster. YMMV.
I would expect to see improvements when epochs are really short, for instance.

Hi, Thanks for the efforts to test-during-training. I wonder can we use test-during-training now? If yes, could you please give us an example? Thank you very much!
P.S. I do not always need to test-during-training, but in some project for specific research purpose I have to...

Hi, The behavior is just like before this PR via the "test_interval" option. However, setting it to -1 will skip test during training.
Check out this doc page https://torchmd-net.readthedocs.io/en/latest/torchmd-train.html


Fix complex expression parsing

Related Issue
Fixes https://github.com/microsoft/AdaptiveCards/issues/4967
Description
In some cases, a complex binding expression might use the } character within a string literal within the expression. In those cases, the template engine would mistakenly identify the character as the end marker of an ${ } expression block, resulting in the expression not parsing nor evaluating.
This fixes that problem by solely relying on the Adaptive Expression library's logic to do the string interpolation.
How Verified
Verified manually in adaptivecards-designer-app
Microsoft Reviewers: Open in CodeFlow

/azp run


Installing Cloudflare Argo Tunnel

Looking to setup this fabulous fork.
Have a question - How do you install Cloudflare Argo tunnel in a Container-Optimized OS in google Cloud?
According to their documentation ->  https://cloud.google.com/container-optimized-os/docs/concepts/features-and-benefits
"Limitations
Container-Optimized OS has limited or no support for some of the common features you may be familiar with on other operating systems:
Container-Optimized OS does not include a package manager; as such, you'll be unable to install software packages directly on an instance. However, you can use CoreOS toolbox to install and run debugging and admin tools in an isolated container.
Container-Optimized OS does not support execution of non-containerized applications.
The Container-Optimized OS kernel is locked down; you'll be unable to install third-party kernel modules or drivers.
Container-Optimized OS is not supported outside of the Google Cloud Platform environment."
Any thoughts on this?  Thanks so much for your work in making this!
Thanks!

you can run the steps
Step 2.3: Setup Cloudflare argo tunnel

Download and install Cloudflare argo tunnrl from here: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation
Run cloudflared tunnel login locally and select your domain to get the cert.
Run cloudflared tunnel create bwd
Run cloudflared tunnel route dns bwd bwd.example.com

on any pc oder vm. and take doe XXXXXXXXXXX.json file from there.


error message with docker run: "Did not receive the pod started notification"

"from time to time", I have this message from the runtime. This may happen or not without changing anything, so it maybe a race somewhere. It would also help if the message was more meaningful, does that mean qemu had problems starting, ...?
Also this message is repeated twice, could it be that the runtime prints that on stdout/err but shouldn't?
$ docker run --runtime=cc busybox ls /
Did not receive the pod started notification
docker: Error response from daemon: oci runtime error: Did not receive the pod started notification.

Reproducing this one may be hard. Once thing for sure is that I didn't have a well setup pause binary.

I can confirm I have seen this issue in the past with incorrectly setup pause binary.

Maybe #182 will resolve this.

That sounds correct, with #182 we won't go far enough to trigger this.


day 1

https://practice.geeksforgeeks.org/problems/triangle-number/1?utm_source=youtube&utm_medium=collab_striver_ytdescription&utm_campaign=pattern_3

add problem statement in a comment or attach a link of a problem with code


rationalizes build-cross and dist Makefile targets

build-cross now depends on build, and uses the git tag if the checked out branch correlates with a tag
dist now depends upon build-cross

This produces friendly artifacts with a make dist like so:
$ find _dist -maxdepth 1 -type f
_dist/acs-engine-a1af5fbc14aee06f19a9d4863dcc05fe7d437cab-darwin-amd64.tar.gz
_dist/acs-engine-a1af5fbc14aee06f19a9d4863dcc05fe7d437cab-darwin-amd64.zip
_dist/acs-engine-a1af5fbc14aee06f19a9d4863dcc05fe7d437cab-linux-amd64.tar.gz
_dist/acs-engine-a1af5fbc14aee06f19a9d4863dcc05fe7d437cab-linux-amd64.zip
_dist/acs-engine-a1af5fbc14aee06f19a9d4863dcc05fe7d437cab-windows-amd64.tar.gz
_dist/acs-engine-a1af5fbc14aee06f19a9d4863dcc05fe7d437cab-windows-amd64.zip

Or, for a release, when a release tag is checked out:
$ find _dist -maxdepth 1 -type f
_dist/acs-engine-v0.7.0-darwin-amd64.tar.gz
_dist/acs-engine-v0.7.0-darwin-amd64.zip
_dist/acs-engine-v0.7.0-linux-amd64.tar.gz
_dist/acs-engine-v0.7.0-linux-amd64.zip
_dist/acs-engine-v0.7.0-windows-amd64.tar.gz
_dist/acs-engine-v0.7.0-windows-amd64.zip

Release note:

NONE

@jchauncey FYI


Add be_a_subclass_of matcher.

I've added a be_subclass_of matcher to the builtins.
I realise that it is possible to use the be matcher for this, like so expect(String).to be < Object. This however, feels rather cryptic. I feel that expect(String).to be_subclass_of Object is clearer.
I've included a specification, as well as a cucumber feature based off of the be_kind_of feature.
Let me know if there's anything I can do to make this easier to merge.

Thanks for the well written PR, @thomasfedb.  I do have some concerns about this, however:

expect(klass).to be_a_subclass_of(klass) passes for any class even though we would never say a class is a subclass of itself, since classes are in their own ancestors list.
expect(Array).to be_a_subclass_of(Enumerable) also passes, but I wouldn't normally speak of a class subclassing its mixins.
In ruby it seems like a code smell to specify that a class is a subclass of another class in its tests.  It goes against the spirit of duck typing, IMO.
I can't think of a single time I've ever wanted this or heard of any users wanting this (before your PR).

Given my concerns above and the fact that it's literally 3 lines of code to define it in projects that want it:
RSpec::Matchers.define :be_a_subclass_of do |ancestor|
  match { |klass| klass.ancestors.include?(ancestor) }
end

...my instinct is to not merge this and instead to leave it to the few users who want it to define it in their projects.

Hey @myronmarston
The first two points I can resolve quite simply I believe. Something along the lines of actual != expected && expected.kind_of?(Class). Or perhaps better to raise an exception if the matcher is instantiated with a non-Class. Let me know if you think it's worth going ahead to resolve.
With regards to your concern about the use of such a matcher being a smell, I do think there are valid cases for its use. Particularly in situations involving factory objects, decorators, and the like, especially given that Ruby is a reasonably metaprogramming-happy language.
For example, I worked on some code recently that dynamically created classes, and wished to assert that they descended from a particular other class depending on the details of their creation. I've certainly seen such assertions in code I've worked in, and found them reasonable in their context.
I think that if you are including is_kind_of then is_subclass_of isn't a particularly odd divergence. A lot of them time in Ruby we just want our object to #quack, but sometimes you need to be sure you're dealing with a Duck.

Thanks for following up, @thomasfedb.  I'm still on the fence about this but you've brought me a bit closer to being in favor.  I'm curious what other RSpec core folks think... /cc @rspec/rspec ?

The first two points I can resolve quite simply I believe. Something along the lines of actual != expected && expected.kind_of?(Class). Or perhaps better to raise an exception if the matcher is instantiated with a non-Class. Let me know if you think it's worth going ahead to resolve.

The actual != expected check makes sense for the expect(klass).to be_a_subclass_of(klass) case.  For the module case, I'm on the fence about what the semantics should be.

I think that if you are including is_kind_of then is_subclass_of isn't a particularly odd divergence. A lot of them time in Ruby we just want our object to #quack, but sometimes you need to be sure you're dealing with a Duck.

be_kind_of and be_subclass_of are quite different, IMO.  The former operates on objects, and specifies something about it's class.  The latter operates only on classes, and specifies something about it.  I see it as different because in most cases be_subclass_of is a tautology (e.g. class User < ActiveRecord::Base already states User is a subclass of ActiveRecord::Base, so expect(User).to be_a_subclass_of(ActiveRecord::Base) is redundant).  be_kind_of doesn't run into the same redundancy.
You mention wanting this for a special case of dynamically creating classes, and in that case I can see it serving a purpose but that feels like a special case to me.  I don't see this matcher ever being useful for the "normal" way of defining classes.
Anyhow, I'm still curious what others think.  I'm going to comment on the diff now but don't worry about addressing it until we've decided if this is going to be included or not.

Thanks for your diff comments @myronmarston, I'll wait to see what others think before resolving them.
I do agree that there are many cases, eg: class User < ActiveRecord::Base where there is no need to be including expect(User).to be_a_subclass_of(ActiveRecord::Base) in specifications.
There are however, cases where an class's hierarchy is a product of less declarative, procedural code, and these are valid situations to check. I submitted this against rspec-expectations itself, less because I think this matcher will be used in every single codebase, but more because it is a basic matcher that probably fits in here best.

@myronmarston Do you know if any other contributors have an opinion on this?

Nope, they would comment here with their opinion.  People have been busy as of late.
@rspec/rspec -- we'd love some of your opinions about this :).

I'm not convinced this is general-enough purpose.

I'm :-1: on adding this as a core feature. I can't think of a case where I've ever wanted to check that something is a subclass of something else.

Seems like there's a general consensus that this doesn't belong in rspec given the reasons outlined above and how easy it is for users to add to the few projects that would need it.  I'm going to close this.  Thanks for suggesting it, though, @thomasfedb -- RSpec is moved forward by contributions like these and I hope you'll contribute again.


Search API is missing some useful criterias, e.g. isNull(), isNotNull, contains(Collection items)

For some advanced search queries, the above mentioned methods would be very useful.
Example:
Attribute<Integer> foo = cache.getSearchAttribute("foo");
Attribute<Integer> bar = cache.getSearchAttribute("bar");
Attribute<Integer> categories = cache.getSearchAttribute("categories");
Query query = cache.createQuery().includeValues();
query.addCriteria(foo.isNull());
query.addCriteria(bar.isNotNull());
query.addCriteria(categories.contains(Arrays.asList(1,2,3)));

Why there is no search api in 3.x ?? I think it was one of the best features

Searching over what's in a Cache, i.e. what currently happens to be the
hotset based on the cache access pattern, has no semantic really.
Say, what's the average salary of all the employees currently in the Cache?
This means nothing. It only means something is the cache knows about the
complete population of Employees. Which is a guarantee a Cache will never
give you.
On Tue, Nov 17, 2015 at 4:43 AM Marco Del Percio<EMAIL_ADDRESS>wrote:

Why there is no search api in 3.x ?? I think it was one of the best
features
—
Reply to this email directly or view it on GitHub
https://github.com/ehcache/ehcache3/issues/190#issuecomment-157321379.

Well actually the search api was useful for querying data where ehcache could provide an in-memory dataset. Something like H2 yet with the advantages of ehcache.
There are other products providing in-memory dataset querying but I thought that ehcache was a good candidate for that, if this feature is removed in v3 then we better go for a different product

Totally in agreement here... that's indeed another product than ehcache.
Ehcache "only" provides a Caching API.
As for an in-memory datagrid that provides you with a search API (and
possibly more), well... there are other products out there today, and
probably more so in the future ;)
On Tue, Nov 17, 2015 at 1:09 PM Marco Del Percio<EMAIL_ADDRESS>wrote:

Well actually the search api was useful for querying data where ehcache
could provide an in-memory dataset. Something like H2 yet with the
advantages of ehcache.
There are other products providing in-memory dataset querying but I
thought that ehcache was a good candidate for that, if this feature is
removed in v3 then we better go for a different product
—
Reply to this email directly or view it on GitHub
https://github.com/ehcache/ehcache3/issues/190#issuecomment-157456123.

I see,
It's a shame I thought it was quite promising, there're several articles comparing ehcache with other in-memory datagraid products and since we didn't need synchronization with different nodes
ehcache was quite good in it.

Well, actually it sorts of got really bad with the new topologies supported
with 3.0.
Let me explain: OSS Ehcache only supported brute forcing in order to
search, which works fine on "small" heap caches. With the offheap store now
being open sourced, one could easily have multiple 10GB of data in a cache
on low end commodity hardware. Coupled with the fact that deserialization
would be required to evaluate against each attribute, well, latency starts
looking very bad.
Long story short, if you know all the data in a cache is all you need to
query, you probably could implement the logic yourself outside of the cache
pretty quickly. But performance would degrade rapidly as the dataset grows.
If you don't worry about that cool, but as library developers we can't
possibly take that approach. It'd be a disservice to our users.
Now we are working on something else to cover these larger dataset use
cases. It won't happen through the Ehcache API though.
Hope this makes some sense.
On Tue, Nov 17, 2015 at 1:25 PM Marco Del Percio<EMAIL_ADDRESS>wrote:

I see,
It's a shame I thought it was quite promising, there're several articles
comparing ehcache with other in-memory datagraid products and since we
didn't need synchronization with different nodes
ehcache was quite good in it.
—
Reply to this email directly or view it on GitHub
https://github.com/ehcache/ehcache3/issues/190#issuecomment-157460025.


rebrand: bullet_train -> flagsmith

@dabeeeenster  you may(especially ) want to take a look at the version and company name part of it.


@conda-forge-admin, please update circle

Hi, I updated the circle-ci deploy key.

Oops. Missed you beat me here. :) Thanks for taking a look.


Release/dedupe

@alexanderdean about overriding existing jars: to overwrite existing asset all following conditions should be truth:

Wrong (previous) version in Build.scala (very likely to happen)
Same wrong version in TRAVIS_TAG (unlikely, but still possible)
override: true in release.yml

All three these bullets make me pretty confident about safety of published instances.

Thanks @chuwy - that gives me a lot of confidence too...

Switching over to the release PR...


How I can know the width and height of every element in Giphy Grid View ?

If I want to add Giphy to image or video
with the same width and height , which parameter allow to me know them ?

Hi @amalnaami
gif.images!.fixedWidth.width
gif.images!.fixedWidth.height

@bazookon  Thanks


Compilation error if first line in function def is val or var definition while using Scala 3 syntax

Sample code
@main def main() =
  val a = "World"
  println(s"Hello $a")

Results in error
sample.scala:2:3 expected (If | While | Try | DoWhile | For | Throw | Return | ImplicitLambda | SmallerExprOrLambda)
  val a = "World"
  ^

But following works:
@main def main() = {
  val a = "World"
  println(s"Hello $a")
}

Also this works:
@main def main() =
  println
  val a = "World"
  println(s"Hello $a")

Ammonite version Ammonite REPL & Script-Runner, 2.5.5-12-e1748830 running on Mac.
JVM info
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

Seems like a small edge case, feel free to ask for more info if needed.

I have now sneaking suspicion that this is works as expected with current scala 3 support, cloning for now, need to investigate a bit more.


Get Audio bytes when ReceiveUnmixedMeetingAudio is set to true

I am working on a bot that is supposed to attend group calls/online meetings. The bot needs to access audio spoken by each participant that will be send to transcription service at the end of the meeting. In CallHandler.cs we can add an event handler OnAudioMediaReceived(object sender, AudioMediaReceivedEventArgs e), where we can get the audio buffers for each speaker. My question is, where can I find the byte array against each speaker. When event handler is fired e.Buffer.Data always returns '0's while there are audio bytes in  e.Buffer.UnmixedAudioBuffers[0].Data. So I am creating a byte array out of it and adding that to a generic list which will be parsed once the meeting is ended, to generate the transcript:
byte[] managedByteArray = new byte[e.Buffer.UnmixedAudioBuffers[0].Length];
                  int length = (int)e.Buffer.UnmixedAudioBuffers[0].Length;
                  Marshal.Copy(e.Buffer.UnmixedAudioBuffers[0].Data, managedByteArray, 0, length);
                  this.AudioByteArrayList.Add(managedByteArray);

I just want to make sure that am I getting the audio bytes in a correct way? as I am not getting desired results from speech to text service ( I am using cognitive speech service provided by Microsoft for transcription) .

Yes this is the correct way (from there you can pack the samples into wav files if you wish to listen to the audio) maybe you have to setup you cognitive service to inform it that you are sending pcm samples

I have tried to create a wav file out of it but no able to run it as it simply says:

My code to create audio file simply accepts the list of byte arrays that I just showed in the question. Following is the code to generate the wave file out of it:
private string CreateAudioFile()
        {
            string startupPath = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase);
            string rootPath = startupPath.Replace("file:\\", "");
            string filePath = rootPath.Replace("bin\\x64\\Debug", "speakerAudio" + fileNameCount + ".wav");
                return filePath;
        }

        private void SaveDataToAudioFile(string filePath, List<byte[]> data)
        {
            using (var stream = new FileStream(filePath, FileMode.Append))
            {
                foreach(byte[] byteArray in data)
                {
                    stream.Write(byteArray, 0, byteArray.Length);
                }

            }
        }

You are missing the wav file header pleas checkout https://github.com/microsoft/BotBuilder-RealTimeMediaCalling/issues/19#issuecomment-311433357

Regarding,

maybe you have to setup you cognitive service to inform it that you are sending pcm samples
Where can we specify that we are sending PCM samples? as it might be expecting wav samples

There aren't wav samples because wav is a file format as a container for pcm data but I don't know how to setup the cognitive service I think that's something on the cognitive service site

Hi @1fabi0,
Thanks for your help and responses, I am able to create an audible wav file out of received audio buffers. But facing an issue in a particular scenario while converting speech to text. When participant doesn't speak and is not muted as well, we are still getting audio buffers although those doesn't contain any voice/sound in it. Wav file is created but when this file is sent to Speech-to-text service, service times out as there seems to be a silence timeout limit for Microsoft speech to text service. So, is it possible to discard any silent buffers when participant is not mute?

Audio Packages have a IsSilence flag so you should check this flag if it is set to false/to true. If you still receive audio packages(IsSilence is false) you may have to check for signatures on the receiving packages and filter these packages out but this can be difficult if the packages you receive are different

@1fabi0  The isSilence = true  flag  happens only initially  before the actual connection of the speakers.

I had a similar problem with channel meetings but as I described check for patterns/signatures in the packages where the IsSilence flag is false ideally packages that are just silent have the flag is silence = true


python311Packages.roadtools: init at 0.0.1

Description of changes
Azure AD tooling framework
https://github.com/dirkjanm/ROADtools
Related to https://github.com/NixOS/nixpkgs/issues/81418

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 296738 run on x86_64-linux 1

24 packages built:

python311Packages.roadlib
python311Packages.roadlib.dist
python311Packages.roadrecon
python311Packages.roadrecon.dist
python311Packages.roadtools
python311Packages.roadtools.dist
python311Packages.roadtx
python311Packages.roadtx.dist
python311Packages.selenium-wire
python311Packages.selenium-wire.dist
python311Packages.signxml
python311Packages.signxml.dist
python312Packages.roadlib
python312Packages.roadlib.dist
python312Packages.roadrecon
python312Packages.roadrecon.dist
python312Packages.roadtools
python312Packages.roadtools.dist
python312Packages.roadtx
python312Packages.roadtx.dist
python312Packages.selenium-wire
python312Packages.selenium-wire.dist
python312Packages.signxml
python312Packages.signxml.dist


Blog Explaining ZLC to ZAC

As we work with the TSC on Zowe.org changes we should have a blog to explain the ZLC to ZAC transition - some ideas:

One thing fun about open source is the freedom to change and adapt.
The community formed the TSC earlier this year and it is now well established.
Zowe is a collection of teams (we call squads) and the TSC is made up on squad leaders who work cross squad to advance Zowe.
We saw the need for the TSC to speed technical implementation - we've seen great progress in on time deliveries of releases and more rigor in the dev process.
Many capabilities in Zowe require cross squad coordination and the TSC is there to make that happen.
Open source has a history of developer lead innovation. The TSC aligns with many open source project.
Great developers know they can't code and listen to the market simultaneously.
Listening and advising is now the role of the Zowe Advisory Council.
Open to all - intended to the voice of Zowe consumers and stakeholders.

Thanks for creating this issue Bruce.  I took the action item from our last ZAC meeting to kick this off with a draft document we can all start contributing to collectively, and went ahead and wrote the attached draft before noting that you had written the above bullets.  So please do not take my draft's omission of those bullets as a rejection of them.  I just ran out of Zowe time today
Who is Zack.docx
!

Peter, excellent start.  I've reviewed the existing content and made a few suggestions as comments (attached here).
Who.is.Zack.docx

One observation - maybe some acknowledgement needed of TSC operating under the policies and practices of the Linux Foundation to explain the project is part of a bigger org?
Some words off the top of my head:
ZAC membership and governance
The Zac is open to all interested parties of Zowe's future. It is a group to focus on "what to build next" for the good of the z/OS platform. Your participation can be in live meetings (put link to the calendar) or by raising issues for consideration (put link to github). Like any Product Management and Development relationship the more fact-based case to make to assist with work prioritization the better! The ZAC is intended to run on a consensus basis - there is no voting process involved. The group's purpose is to describe the needs of the consumers of Zowe in a way to provide the TSC clear objectives for future releases. Come join the ZAC team and assist in shaping the future of Zowe.

With this new version I have incorporated all feedback thus far provided   ...hopefully to satisfaction.
Who.is.Zack.docx

New version
Who.is.Zack oct12.docx

Who.is.Zack.oct12 - update for 2023.docx

Although I can assign this @PWFlyMan there are other github issues that I try to assign you and your name does not appear as a valid option

Although I can assign this @PWFlyMan there are other github issues that I try to assign you and your name does not appear as a valid option

Do I need to be set as a Collaborator?

Bruce, I turned on tracking and made some suggestions in the attached, including a stab a new title that might serve as click-bait for mainframe leaders who like to get involved, even if they don't know what ZAC is.
Who.is.Zack.oct12.-.update.for.2023 (1).docx

I reviewed the document and I don't have specific comments, thank you

Medium darft here https://medium.com/@armstrob/seeking-leaders-to-help-shape-the-future-of-the-mainframe-using-open-source-573bc3c6f0c2
Need some help with ideas for images to include? How to be sure to file as Zowe Vendor Neutral category and to get the review process going


ACMEv2(?): Intermediary (chain) placed in fullchain.pem twice.

Hi!
For some reason, when issuing my wildcard certificate (ACMEv2), it appears fullchain.pem ends up with the same intermediary certificate (chain.pem) twice.
I notice ^M line -endings in the first iteration of the intermediary in fullchain.pem, which do not exist in the chain.pem. However, upon opening cert.pem, it seems the intermediary is already there, with the ^M line-endings.
I suspect that with ACMEv2, the contents of chain.pem already comes from Let's Encrypt (hence being stored in cert.pem).
Daniël

Please check your dehydrated version. This should be fixed in master a while back.

Yeah, Lukas just showed me on IRC as well. I was using 0.6.1, cloned the repo to make a fix, then tested with the same 0.6.1 release on a different machine that had my hook scripts. I need more caffeine.


cmake: autoconf host regex for windows is -pc- instead of *-win32

…so we also match for MSYS host environment

x86_64-pc-msys is the string for native msys. vs2019 toolchain resets AUTOCONF_HOST than to x86_64-pc-win32. in both cases we got windows results, but paths must be MSYS compatible.

Now I see. In both cases we use a native win32 cmake. Not sure if this works to build msys executables. Have you tried?
The -pc- is the vendor field and almost arbitrary. See https://wiki.osdev.org/Target_Triplet

Now I see. In both cases we use a native win32 cmake. Not sure if this works to build msys executables. Have you tried?

Yep, worked in my cases.

Ok. Then we should test for AUTOCONF_BUILD being msys, right? Because we always use a native Cmake...

indeed, would make sense!

please recheck?!


Using custom device fields with other types (e.g. Boolean)

Hello,
this project currently does not allow the usage of custom field types other than text:
device_cfs = netbox.extras.custom_fields.filter(type="text", content_type_id=23)

Why exactly are you filtering for only text-type fields? Are there any disadvantages in removing the type filter?
device_cfs = netbox.extras.custom_fields.filter(content_type_id=23)

Thanks for your quick reply - we have some Boolean custom fields which we'd have to include too. Let's call the field "monitor" and we'd only want to include all the items that have the monitor field set to true.

I think that we are talking about 2 different things. The custom field construction that you mention is only used for the hostgroup name generation.
Corrent me if i'm wrong but you want to filter your devices with a custom field 0 / 1 value. You can easily do this as follows in the config file:
nb_device_filter = {"cf_monitor": True}
Is this correct?

It was a configuration issue on our side. Everything works as expected - but thanks for the heads-up!


get haddock working

Apologies for the slightly vague bug report, but after spending a little while investigating already, I'd like to get an idea of what's actually supposed to work before digging in more depth.
There's code in Documentation.hs and AtPoint.hs that appears to be intended to include haddock docs in the hover information.
However I never see any, even with really trivial test cases. Is this something that is expected to work or is working for others?
One thing I noticed is that there are references in Documentation.hs to things happening when Opt_Haddock or Opt_KeepRawTokenStream are enabled, but I don't see them enabled anywhere.

@hsenag afaik haddock comments are shown on hover now, feel free to reopen if it is not the case


Fix fit(), then transform() bug

Fixing bug in the fit() then transform() workflow, referenced in #89

Hello @msedwar! Thanks for submitting the PR.

In the file umap/umap_.py, following are the PEP8 issues :

Line 48:80: E501 line too long (84 > 79 characters)
Line 105:80: E501 line too long (97 > 79 characters)
Line 122:13: E303 too many blank lines (2)
Line 201:80: E501 line too long (88 > 79 characters)
Line 206:80: E501 line too long (82 > 79 characters)
Line 215:80: E501 line too long (81 > 79 characters)
Line 225:80: E501 line too long (80 > 79 characters)
Line 248:80: E501 line too long (80 > 79 characters)
Line 444:80: E501 line too long (87 > 79 characters)
Line 475:80: E501 line too long (82 > 79 characters)
Line 522:80: E501 line too long (80 > 79 characters)
Line 593:80: E501 line too long (82 > 79 characters)
Line 1110:80: E501 line too long (80 > 79 characters)
Line 1168:80: E501 line too long (80 > 79 characters)
Line 1172:80: E501 line too long (80 > 79 characters)
Line 1176:80: E501 line too long (82 > 79 characters)
Line 1177:80: E501 line too long (80 > 79 characters)
Line 1178:80: E501 line too long (84 > 79 characters)
Line 1186:80: E501 line too long (80 > 79 characters)
Line 1190:80: E501 line too long (80 > 79 characters)
Line 1191:80: E501 line too long (82 > 79 characters)
Line 1210:9: E131 continuation line unaligned for hanging indent
Line 1211:9: E131 continuation line unaligned for hanging indent
Line 1212:9: E131 continuation line unaligned for hanging indent
Line 1213:9: E131 continuation line unaligned for hanging indent
Line 1214:9: E131 continuation line unaligned for hanging indent
Line 1215:9: E131 continuation line unaligned for hanging indent
Line 1216:9: E131 continuation line unaligned for hanging indent
Line 1217:9: E131 continuation line unaligned for hanging indent
Line 1237:80: E501 line too long (80 > 79 characters)
Line 1285:80: E501 line too long (84 > 79 characters)
Line 1287:80: E501 line too long (82 > 79 characters)
Line 1288:80: E501 line too long (81 > 79 characters)
Line 1362:80: E501 line too long (81 > 79 characters)
Line 1379:80: E501 line too long (86 > 79 characters)
Line 1419:80: E501 line too long (92 > 79 characters)
Line 1555:80: E501 line too long (82 > 79 characters)
Line 1588:80: E501 line too long (81 > 79 characters)
Line 1591:80: E501 line too long (85 > 79 characters)

Coverage remained the same at 94.518% when pulling 9f9870d990eb140009e3948ace2b7343a4c33c2e on msedwar:transform-fix into 70d8c08376ebae6248fed85e8bd410c5130587a2 on lmcinnes:master.


Import tag broken inside script tag?

This syntax should be valid according to Jade spec:
    script
      include script.js

However instead of including the contents of script.js you just get this syntax in your HTML:
<script>include sellpy_import/_search.js
</script>

Any ideas where I should start looking to fix this? I can probably get a pull request up for it if I can just get some rough idea where to look.

I believe this was fixed as of June with this commit: https://github.com/syrusakbary/pyjade/commit/8b161f4e6052ec8a1c2eae5f30c78dce411c7137
The jade
script
  include script.js

will now produce the django (for example) template
<script>{% include "script.js" %}
</script>

which django would process the way you want, while script. would produce the literal
<script>include script.js
</script>


Update messages

Type of change

[ ] Breaking change
[ ] Non-breaking change
[x] Cosmetic change

Description
Jira link: Jira title
Provide a short description of the change. For example: what parts of the code changed, is there any new behaviour...etc.
Attachments or Screenshots
Add any screenshots (if relevant) here

Did we get given welsh updates as well?

No. I spoke to Janos and he said to write comments in the stories if Welsh translations are required. These will be done as separate stories.


Question: How can I get the title and the artist name?

Hi, I would like to ask how can I get the name and artist name only?

I'm trying to get only the title and subtitle but I don't know how can I get them from the raw format, is there a way to get it?
thank you in advance.

@Lolimipsu, hello!
    out = await shazam.recognize_song("data/dora.ogg")
    print(out)

    serialized = Serialize.full_track(out)
    print(serialized.track.title)
    print(serialized.track.subtitle)

or
out["track"]["title"]
out["track"]["subtitle"]

or
out["track"]["title"]
out["track"]["subtitle"]

ohh so this is how I should do it, thank you!

ohh so this is how I should do it, thank you @dotX12  !


v0.18.0

@MaximeHeckel ok to publish this?

This has been tested and it's working. Those are minor changes.


Retrying a workflow that uses a DAG template and a loop does not retry the failed task/step.

Summary
What happened/what you expected to happen?
Retrying a workflow that uses a DAG and a withParam does not retry the failed task/step.
https://user-images.githubusercontent.com/6024271/150692962-9353b9a2-a251-46a4-864b-5ea0ef3e3fb8.mov
A quick fix that seems to work is to set the retry limit to 0 but it causes other issues in the collected output.
      retryStrategy:
        limit: "0"

The output that is collected in the final step is duplicated and repeated based on the number of retries done.
From the recording, each output is repeated twice in the final array. I expect three outputs but I get six as it was retried twice.
https://user-images.githubusercontent.com/6024271/150692930-8d91a688-4e0b-410e-8401-abe97ef5a7b3.mov
What version of Argo Workflows are you running?
3.2.4
Diagnostics
Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: retry-barebones
spec:
  entrypoint: main
  templates:
    - name: main
      inputs:
        parameters:
          - name: numbers
            value: |
              ["one", "two", "three"]
      dag:
        tasks:
          - name: whalesay-before-process
            template: whalesay
            arguments:
              parameters:
                - name: message
                  value: "HELLO ALL"

          - name: process
            template: process
            withParam: "{{inputs.parameters.numbers}}"
            dependencies:
              - whalesay-before-process
            arguments:
              parameters:
                - name: message
                  value: "{{item}}"

          - name: whalesay-after-process
            template: whalesay
            dependencies:
              - process
            arguments:
              parameters:
                - name: message
                  value: "Root: {{tasks.process.outputs.parameters.output}}"
                  # On retry there are extra values in the output, duplicated multiple times based on retry count

    - name: process
      # this fixes the behaviour
      retryStrategy:
        limit: "0"
      inputs:
        parameters:
        - name: message
      outputs:
        parameters:
          - name: output
            valueFrom:
              parameter: "{{tasks.step2.outputs.parameters.output}}"
      dag:
        tasks:
          - name: step0
            template: whalesay
            arguments:
              parameters:
                - name: message
                  value: "{{inputs.parameters.message}}"

          - name: step1
            template: whalesay
            dependencies:
              - step0
            arguments:
              parameters:
                - name: message
                  value: "{{inputs.parameters.message}}"
                - name: always-pass
                  value: "false"

          - name: step2
            template: whalesay
            dependencies:
              - step1
            arguments:
              parameters:
                - name: message
                  value: "{{tasks.step1.outputs.parameters.output}}"

    - name: whalesay
      inputs:
        parameters:
        - name: message
        - name: always-pass
          value: "true"
      outputs:
        parameters:
          - name: output
            valueFrom:
              path: /tmp/output.txt
      script:
        command: [ python ]
        image: python:alpine
        imagePullPolicy: IfNotPresent
        source: |
          import random
          import os
          msg = '{{inputs.parameters.message}}'
          print(msg)

          if '{{inputs.parameters.always-pass}}' == 'true':
            a = -1
          else:
            a = random.randint(1,10)

          if a > 0 and a <= 5:
            raise Exception(msg)

          with open('/tmp/output.txt', 'w') as o:
            o.write('random' + msg + ' : ' +str(a))

What Kubernetes provider are you using?
What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
Emissary
# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

I'll look into this.

e58859d795