Dumping updates and data validation

This PR adds the latest iteration of updates related to running the full dump after dependencies have been updated. It also adds data validation to the processors.
Dump and content updates:

PublicationType tags are added to Publication nodes, following updates in INDRA
A boolean is added to Publication, Evidence nodes and indra_rel relationships which is true if Publication/Evidence is retracted or if a statement has at least one Evidence from a retracted source
Add a helper function to get the proper Neo4j boolean values from a Python boolean or condition and use it in all places that generate boolean metadata
Put locations for processor files in their own files in indra_db and pubmed sub modules to avoid circular imports
A typo for a key previously lead to indra evidence jsons in Evience nodes not being validated, the typo is now fixed (see assert_valid_node())
Beliefs are set in the statement json of indra_rel relations. Previously, this was only done for the belief property of the indra_rel relations
Various improvements to logging and comments

Data validation:

The data validation checks that a value and its type are as expected for a given Neo4j data type as specified in the ingestion tsv file header. See the documentation for more info on Neo4j data types.
The data validation is done at dump time and is added in assert_valid_node(), which is called from both validate_nodes() and validate_relations().
Tests are added for the data validator

todo:

[x] Process and dump nodes edges from indra_db (currently being re-run)
[ ] Fix missing data for Publication nodes

I already fixed the bioregistry error in https://github.com/biopragmatics/bioregistry/pull/1030 but the autorelease broke because another data source stopped working... fixing now!


How to unlock?

I can lock but my button cannot reverse...

I'm not fully understand your question, can you elaborate it a bit?


[DSIP-19] Support sagemaker connections in the connection center, as well as external connections to the connection center in sagemaker tasks

issue: https://github.com/apache/dolphinscheduler/issues/14338
Related issues: [DSIP-19][Feature] Add connection center feature for DS https://github.com/apache/dolphinscheduler/issues/10283
Mail: https://lists.apache.org/thread/xl6pb3sbrt0ffrf1fltcph39s9w1pjlx

@xdu-chenrj You may try something like @Order to avoid some exceptions related to concurrency. see: https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api-test/dolphinscheduler-api-test-case/src/test/java/org/apache/dolphinscheduler/api/test/cases/ProjectAPITest.java


exampleDirectiveWithModel

Can someone please help me to use dragular with dragular-model, without DOM manipulation using queryselector and angular.element?
http://luckylooke.github.io/dragular/#/docs/exampleDirectiveWithModel
i get errors when i am using dragular-item with dragular-model?

Issue: When I am loading a list dynamically example $scope.models.alarmRelay1List.push(alarm), and when I am dragging it to the container, it is dragging the desired element. But when I finally drop it, its dropping the next element, instead of the first element. This happens when I am using Dragular-model.
If I am using DOM manipulation, Dragular is working fine.
Also it is coming only for the list I am using push(), other lists are dragging the correct element.

Please try to edit this fiddle to reproduce your problem, thanks.

Closed as inactive


Fixes #1365 - Always export service ports in app.ports field

...even if there are no dynamic app ports to assign.
Also:

cleanup some methods without side effects that had a () call convention.
improve documentation about --local_port_min and --local_port_max

I implemented your suggestions

Can you please rebase?

rebased

:+1:


What is the difference between Build on save and pressing ctrl-o b?

Build on save works irrespective of what's set as the Build command, even gobbledygook.

The default build command pulp build --no-psa --json-errors fails for ctrl-o bwith Error running PureScript build command: Error: Didn't find JSON output at Object.exports.error ...

On the command line, pulp build --no-psa --json-errors fails with Error: Unrecognised arguments: --json-errors

pulp documentation says parameters to be passed through to psc compiler must be separated from build options with --.
Thus pulp build --no-psa -- --json-errors works for ctrl-o-b and command line.
I have pulp version 10.0.0, psc version 0.10.2 and<EMAIL_ADDRESS>Thank you
Rolland

pulp build --no-psa -- --json-errors
..works for me too
So I guess this could be a typo or missing change in the plugin's settings (?)

I think this is happening because of the recent changes in pulp's 0.10 release with regards to command line arguments: https://github.com/bodil/pulp/blob/master/CHANGELOG.md#1000

That certainly explains that. Thank you.
What does Build on save use as a build command? It seems to work irrespective of what is in the build command. I'm probably missing something.

@rollandjb Build on save uses the running psc-ide-server instance to recompile the currently edited module. It's a different kind of build though. As an example consider the following module structure:
C -> B -> A
here C depends on B and B depends on A.
You now make a change inside B that causes type errors inside B and C.
Build on Save will only report the type errors inside B though and not the ones in C.
The "full build" through pulp can also do bundling etc. Build on save is primarily used for near instant feedback in the editor.

Thanks for reporting, guess I'll update the default (and break it for anyone that hasn't updated pulp!)

@nwolverson No problem!
@kRITZCREEK Thanks for the explanation.

Updated this default - if you find your build command now fails for the opposite reason, upgrade pulp or set an explicit build command.


[Bug]: The name parameter is not set to the Name property in MongoIdentityRole class

The name parameter is not set to the Name property in MongoIdentityRole class
public MongoIdentityRole(string name, TKey key)
{
	InitializeFields();
	Id = key;
	Name = Name;
}

Good spot, merged and fixed.

Releases
https://github.com/alexandre-spieser/AspNetCore.Identity.MongoDbCore/releases/tag/2.1.1


match wrapper: move match tree evaluating into filter methods

Commit Message: match wrapper: move match tree evaluating into filter methods
Additional Description:
The http filter match wrapper will inject a match tree to the filter chain manager. And before the filter chain manager applies the filter, the match tree will be evaluated.
  // Code snippet of match tree evaluating of filter chain manager.
  for (; entry != decoder_filters_.end(); entry++) {
    (*entry)->maybeEvaluateMatchTreeWithNewData(
        [&](auto& matching_data) { matching_data.onRequestHeaders(headers); });

    if ((*entry)->skipFilter()) {
      continue;
    }

What this PR does? This PR created delegating filter and move match tree evaluating into methods of this delegating filter.
  Envoy::Http::FilterHeadersStatus decodeHeaders(Envoy::Http::RequestHeaderMap& headers,
                                                 bool end_stream) override {
    match_state_.evaluateMatchTree([&headers](Envoy::Http::Matching::HttpMatchingDataImpl& data) {
      data.onRequestHeaders(headers);
    });
    if (match_state_.skipFilter()) {
      return Envoy::Http::FilterHeadersStatus::Continue;
    }
    return decoder_filter_->decodeHeaders(headers, end_stream);
  }

What are the benefits?

More clean filter chain. All the code of match tree evaluating can be moved.
Only the filter with match wrapper pays the additional overhead. Now we always do some checks for every filter.
We can manage the match wrapper in the same way as a normal http filter.

This is still a draft PR for discussion.
Risk Level: Mid.
Testing: Waiting.
Docs Changes: N/A.
Release Notes: N/A.
Platform Specific Features: N/A.

Hi, cc @snowp here is the initial design and implementation. Does this is a possible approach to make the implementation of match wrapper more intuitive?

Originally the idea was to have a generic way for filters to consume match data via the MatchWrapper - this way any filter could benefit from matching through it without having to implement matching. That said, I'm not sure if that's used that much in practice so there might be some value in a simpler skip implementation? I'm not using this often enough to have a sense of this myself, so I'd defer to @envoyproxy/api-shepherds

Yeah, current API is a little complex. The filter developers need implement a onMatchCallback to consume match result. And this onMatchCallback may called in various phases. May be a simpler implementation, for example, match condition A, then create filter witch config A, match condition B, then create filter witch config B, would be more easy for developers to understand and use.

cc @envoyproxy/api-shepherds Hi, folks, do you have any suggestion to this PR? I'd like to get more inputs.

This makes sense to me. cc @snowp for additional thoughts. Thank you!

I think this pr is ready for a quick review (or some suggestions). cc @mattklein123 @snowp

Originally I thought you were talking about moving away from ExtensionWithMatcher but this just seems like an improved implementation :)

This is also I want. But this way means some big behaviour changes of match wrapper.

/retest

cc @snowp Hi, could you take a look again? Thanks. 😄

/retest

Hi, @snowp, comment is updated. PTAL. Thanks. 🌷


openssl 3.0 getrandom syscall only random provider

For most users and really for most new linux distributions that will ship version 3.0 it will be extremely cool to have a random provider that uses only the getrandom syscall for all entropy needs, without any further hops or better said without any other possible points of complexity and failure. and that this provider is selectable as a default at build time.
I had an ENGINE that did this but it appears all this approach is dead  There is currently no reason to use anything else than getrandom on recent linux kernels, it now has an extremely high throughput measured in GB/s and recent changes make it even faster.
Most people are not bound to FIPS requirements neither are mandated to use the NIST-approved algorithms, the syscall overhead of this is probably irrelevant as well.

This is already the default, if the kernel is recent enough.

This is already the default, if the kernel is recent enough.

Maybe I've misunderstood what @crrodriguez is asking for. But what is the default is to use getrandom for seeding our own RNG. IIUC @crrodriguez is asking for getrandom to be used directly without going through our own RNG.

It should be quite straight forward to write such a provider:
https://www.openssl.org/docs/man3.0/man7/provider.html
https://www.openssl.org/docs/man3.0/man7/provider-rand.html
It could be written by anyone as a 3rd party provider. It wouldn't need to be incorporated into the OpenSSL source to be useful.

This is already the default, if the kernel is recent enough.

Maybe I've misunderstood what @crrodriguez is asking for. But what is the default is to use getrandom for seeding our own RNG. IIUC @crrodriguez is asking for getrandom to be used directly without going through our own RNG.

Correct, using getrandom as the only source.. So distributions only have to be watching ONE CSPRNG there are of course other components beside openssl that do this, but their impact is very limited and no very far reaching. could be in theory just patched with configure option to do that.
This is less invasive than what for example openBSD did at some point patching out every single component to use arc4random_* API..later they just created a fork of opennsl that does just that..

The idea is use getrandom for everything, except when the FIPS mode is ON at the kernel level so openssl  uses its own l FIPS approved RNG or uses the kernel DBRNG instead. Using the kernel DBRNG probably has a larger overhead and not worth trying,

The OpenSSL RNG can be used for OpenSSH. Changing this is a high-impact request. The RNG on my x86 platform (and others) uses the built-in hardware randomizer and is very efficient. That becomes single-source, so I am wondering about the purpose also.

The OpenSSL RNG can be used for OpenSSH.
Yes,  openssh offers a compile-time option to do this.

Changing this is a high-impact request.

Yes, it is

The RNG on my x86 platform (and others) uses the built-in hardware randomizer and is very efficient.

all this entropy sources are used by the linux kernel, mixed into the random pool.

That becomes single-source,

I am not arguing for a single source, but to have a radom provider that can be used by default system-wide that only gets its data from the kernel.

so I am wondering about the purpose also.

The purpose is to put the eggs in one basket.. trust but watch this single basket, if there is a problem, it is fixed in the kernel faster than anywhere else, reducing the work of distribution maintainers, developers etc..

The OpenSSL RNG can be used for OpenSSH.

Yes, but openssh offers a compile-time option to do this, it is actually a fallback, openBSD developers prefer arc4random API above anything else.

Changing this is a high-impact request.

Yes, it is.

The RNG on my x86 platform (and others) uses the built-in hardware randomizer and is very efficient.

all this entropy sources are used by the linux kernel, mixed into the random pool.

That becomes single-source,

I am not arguing for a single source, but to have a radom provider that can be used by default system-wide that only gets its data from the kernel.

so I am wondering about the purpose also.

The purpose is to put the eggs in one basket.. trust but watch this single basket, if there is a problem, it is fixed in the kernel faster than anywhere else, reducing the work of distribution maintainers, developers etc..

I don't see how this is of general benefit. First, I am not a maintainer on a Linux platform. Second, I need to be able to change the randomization source independently of OpenSSL on ia64 and do not have control over it, because the system RNG is supplied by another project. This will more likely isolate OpenSSL than further integrate it. I am also unsure how this impacts Linux VMs on non-Linux hypervisors, particularly when the hypervisor is a completely different operating system or different distro than the VM.
For hardware RNGs, the single threaded nature of the RNG is not really a problem because the x86 RNG is "likely" fast enough, but a socket-based RNG is going to become a bottleneck - it already is in some of the applications using OpenSSL, which is why there was pressure to move elsewhere. My concerns are not abating.

The OpenSSL RNG can be used for OpenSSH.

Yes, but openssh offers a compile-time option to do this, it is actually a fallback, openBSD developers prefer arc4random API above anything else.

Changing this is a high-impact request.

Yes, it is.

The RNG on my x86 platform (and others) uses the built-in hardware randomizer and is very efficient.

all this entropy sources are used by the linux kernel, mixed into the random pool.

That becomes single-source,

I am not arguing for a single source, but to have a radom provider that can be used by default system-wide that only gets its data from the kernel.

so I am wondering about the purpose also.

The purpose is to put the eggs in one basket.. trust but watch this single basket, if there is a problem, it is fixed in the kernel faster than anywhere else, reducing the work of distribution maintainers, developers etc..

I don't see how this is of general benefit. First, I am not a maintainer on a Linux platform. Second, I need to be able to change the randomization source independently of OpenSSL on ia64 and do not have control over it, because the system RNG is supplied by another project. This will more likely isolate OpenSSL than further integrate it. I am also unsure how this impacts Linux VMs on non-Linux hypervisors, particularly when the hypervisor is a completely different operating system or different distro than the VM.
For hardware RNGs, the single threaded nature of the RNG is not really a problem because the x86 RNG is "likely" fast enough, but a socket-based RNG is going to become a bottleneck - it already is in some of the applications using OpenSSL, which is why there was pressure to move elsewhere. My concerns are not abating.

You are not the target audience of this, nothing Im proposing will affect your use case in anyway. There will be no new linux disributions shipping openSSL 3.x on ia64 because it is a deceased architecture.
Linux on all hypervisors Im aware of (KVM, Xen, Hyper-v,  vmware)  either should or already obtain enough seeding material from the host CPU, host RNG whatever it is (RDRAND/RDSEED, a hw_random device) or use a seed from UEFI, bootloader or from implementing a hw_random or virtio-rng driver.    Once seeded, the current incarnation of the linux CSPRNG stops blocking and keeps producing output forever, there is nothing else to do afterwards.

The easiest way to do this would be to build OpenSSL with getrandom() as it's seed source and then write a passthrough "DRBG" that just calls its parent.  This will involve extra copies but would be easy.
Writing a RNG would be slightly more effort but still fairly easy.  The providers/implementations/rands/seeding directory has all the code to grab random bytes, it just needs wrapping with a provider RAND.
I have thought of doing this a couple of times, but time and prioritisation keep getting in the way.

@zx2c4 same issue here unfortunately.. userspace RNG with default API not using getrandom as only source.

Marking as inactive, to be closed at the end of 3.4 dev, barring further input

Now getrandom is implemented in the VDSO https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7a3fad30fd8b4b5e370906b3c554f64026f56c2f .
This makes the syscall overhead disappear and outperform anything available out there so far (after setup cost of course) all the supporting code will be soon added to libc
I urge you to reconsider your approach and there is now no real reason to have an userspace CSPRNG at least on linux.

Implementing a getrandom(2) provider would be straightforward.  I'll add a help wanted flag to this since it's something the core team is unlikely to get to doing it in a timely fashion.
The CSPRNGs cannot be removed because they are a FIPS 140 requirement and FIPS support is important to the project.

Implementing a getrandom(2) provider would be straightforward. I'll add a help wanted flag to this since it's something the core team is unlikely to get to doing it in a timely fashion.
The CSPRNGs cannot be removed because they are a FIPS 140 requirement and FIPS support is important to the project.
Not using the DRBGs (outside of FIPS mode) seems worthwhile.

Yes, I am only talking about !FIPS_Mode()


Startup warnings from Armeria

20:39:52.542 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'removeProject' of the class 'com.linecorp.centraldogma.server.internal.api.ProjectServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName]
20:39:52.546 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'listRepositories' of the class 'com.linecorp.centraldogma.server.internal.api.RepositoryServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName]
20:39:52.548 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'normalizeRevision' of the class 'com.linecorp.centraldogma.server.internal.api.RepositoryServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]
20:39:52.549 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'createRepository' of the class 'com.linecorp.centraldogma.server.internal.api.RepositoryServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName]
20:39:52.549 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'removeRepository' of the class 'com.linecorp.centraldogma.server.internal.api.RepositoryServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]
20:39:52.550 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'patchRepository' of the class 'com.linecorp.centraldogma.server.internal.api.RepositoryServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName]
20:39:52.554 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'listFiles' of the class 'com.linecorp.centraldogma.server.internal.api.ContentServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]
20:39:52.556 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'getFiles' of the class 'com.linecorp.centraldogma.server.internal.api.ContentServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]
20:39:52.557 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'commit' of the class 'com.linecorp.centraldogma.server.internal.api.ContentServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]
20:39:52.558 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'listCommits' of the class 'com.linecorp.centraldogma.server.internal.api.ContentServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]
20:39:52.559 [main] WARN com.linecorp.armeria.server.AnnotatedHttpServices - Some path variables of the method 'getDiff' of the class 'com.linecorp.centraldogma.server.internal.api.ContentServiceV1' do not have their corresponding parameters annotated with @Param. They would not be automatically injected: [projectName, repoName]

HttpApiRequestConverter consumes the variables and then inject a Java object into the methods which are raising those warnings. So, actually the methods don't need the variables, but it's hard for Armeria to know whether a variable is already consumed or not. Let me have some time to think about this. :-)

Perhaps we could add a method like this to RequestConverterFunction:
default Set<String> referredParamNames(Class<?> expectedResultType) {
    return ImmutableSet.of();
}

so that a converter exports the names of the params it accesses. I didn't think much about how this will work with BeanRequestConverterFunction, but it's probably possible. What do you think?

It has been fixed in Armeria-side some time ago.


Maintain Changelog

A changelog would allow everyone to keep track of what changes from version to verison.
It doesn't have to be fancy just file the tracks the chages thats it.
here a good example: https://github.com/nginxinc/kubernetes-ingress/blob/master/CHANGELOG.md

Apart from the keep-track-of changes aspect at the moment, I can't see at first glance which version the chart currently has and which traefik version it ships. (except going to the command line)

Fixed by ba98e364cf82d3c0d179b321aa668d53dc10e030
We now create a tag for each version.

Fixed by ba98e364cf82d3c0d179b321aa668d53dc10e030
We now create a tag for each version.


Is it possible to setup swagger ti cxf without spring ?

Hi
This is only a question. I am trying to integrate swagger into project. We are currently using CXF wihout  Spring (we have javax.inject). I succedd in setup uri to api etc ..., but swagger was not managed to find resource packages to scan and json contained only version and uri, without any api.
If this is possible can you please direct me to some example, tutorial or sources ?
Many thanks in advance

For general support questions, it's better to use the google group. Please start a thread there and provide additional information such as the version of swagger-core and CXF you're using, plus the method you use to configure swagger-core (web.xml, BeanConfig and so on).


Feat/terminate ci

Fixes #656

When dynamo DB was not deployed into the claudie namespace, the testing-framework failed, which also failed CI after ~37min (the same time it took testing-framework to show an error). I would consider this a sufficient test.


the_geom guessed as double instead of Geometry

Here's the input file: https://gist.githubusercontent.com/rafatower/f2b6c2918a66749d5898/raw/5a345f34cb57c41f506af54a335f9389dbfc69f0/untitled_table_3%2520(2).csv
And here's the failure:
true: Errored importing data from /tmp/imports/20150918-26933-1dz6hjn/untitled_table_3%25202.csv:
false: Sequel::DatabaseError: PG::Error: ERROR:  invalid input syntax for type double precision: ""
LINE 4:           AND the_geom != ''
                                  ^

It happens around the georeferencer.rb and column.rb classes.

Filed a ticket to osgeo.org
You can see it here

@rafatower
On purpose of making CartoDB more robust against this error:
I've found that the method call that causes the issue is this one:
https://github.com/CartoDB/cartodb/blob/master/services/importer/lib/importer/column.rb#L58
raise "empty column #{column_name}" if empty?
In context:
def geometrify
  job.log 'geometrifying'
  raise "empty column #{column_name}" if empty?
  convert_from_wkt                                  if wkt?
  convert_from_kml_multi                        if kml_multi?
  convert_from_kml_point                        if kml_point?
  convert_from_geojson_with_transform if geojson? && @from_geojson_with_transform
  convert_from_geojson                           if geojson?

  cast_to('geometry')
  convert_to_2d
  job.log 'geometrified'
  self
end

I'm trying to find a way to universally check if a string is empty or a number 0 so to modify the subsequent call in https://github.com/CartoDB/cartodb/blob/master/services/importer/lib/importer/column.rb#L221-L228
def records_with_data
  @records_with_data ||= db[%Q{
    ELECT #{column_name} FROM "#{schema}"."#{table_name}"
    WHERE #{column_name} IS NOT NULL
    AND #{column_name} != ''
    LIMIT 1
  }]
end

so that != '' doesn't make the code break in case that the_geom is interpreted as a number.

geometrify is only called on certain columns, right? there's an overlap between that code and this ticket: #4730

@rafatower
After our chat, it's still not clear for me what we should do in this case.
Should we:

Add a deprecated warning and monitor this code
Remove this code

Any insight is greatly appreciated :smile:

For the moment, just make it robust. I'll add the deprecation note in my branch, since it is there where the supposedly unique usage is.

Does that mean that PR #5743 can be merged? (Not today, on Monday)


[SPARK-44662] Perf improvement in BroadcastHashJoin queries with stream side join key on non partition columns

What changes were proposed in this pull request?
On the lines of DPP which helps DataSourceV2 relations when the joining key is a partition column, the same concept can be extended over to the case where joining key is not a partition column.
In this PR, the keys available in the BroadcastHashJoinExec are pushed down to the DataSourceV2 scans in form of a SortedSet structure.
For non partition columns, the DataSources like iceberg have max/min stats on columns available at manifest level, and for formats like parquet , they have max/min stats at various data structure levels. The passed SortedSet can be used to prune using ranges at both driver level ( manifests files) as well as executor level ( while actually going through chunks , row groups etc at parquet level)
If the data is stored as Columnar Batch format , then it would not be possible to filter out individual row at DataSource level, even though we have keys.
But at the scan level, ( ColumnToRowExec) it is still possible to filter out as many rows as possible , if the query involves nested joins. Thus reducing the number of rows to join at the higher join levels.
Attaching link to a presentation which outlines the idea:
Broadcast Keys pushdown
SPIP : SPIP-44662
Why are the changes needed?
There is scope of improvement in the performance of Inner and Left Semi join queries when using BroadcastHashJoin
Does this PR introduce any user-facing change?
No
How was this patch tested?
Ran TPCDS suite using iceberg as DataSource.
Converted many of the existing Spark Query tests to also run using iceberg as data source.
Will be adding more unit tests.

Will be closing this PR and creating a new one as I have renamed the branch on which this PR has been created


Traited Render System and Reload tracking

Depends on #170

This is a bit of a bigger PR that adds the following:

Use traits for binding to the pipeline. This means we can bind like this:vec![Binding::texture_cube("CubeMap", Stage::Fragment, cubemap)],

instead of this:vec![Binding::texture_cube("CubeMap", Stage::Fragment, &cubemap.uniform)],

This is achieved with the traits in dotrix_core::providers::*

Add a reload system to signal when an asset needs to be rebound to the pipeline.
There are two kinds of reload, ReloadKind::Update and ReloadKind::Reload the first is when data was updated but not the length of the buffer or dimensions of the texture, the second is when these are changed and when a rebind is required.

This rebind is done automatically during a bind of the pipeline. Which is now a no-op if no rebind has been signalled as required. We can therefore just bind everyframe and let the underlying system handle checking for the need to rebinding.

Also a dotrix_derive that adds derive macros to help add reloadable and other traits

Comments are requested as this is kinda far reaching in its effects and we can maybe discuss alternative methods and techniques

Wow! Going to check it out!

I have a few ideas on improvements already but will have to see if they pan out the way I think they will


Fix isValidIP in iptable-start.sh and add unit test for it.

Signed-off-by: Serguei Bezverkhi<EMAIL_ADDRESS>Closes: #13562

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: sbezverk
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @incfly

Can we add some tests to this ? even though it's bash this is simple function so should be possible

@sbezverk: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
15553cd322ebb74d5073496d5cbc37b6fab8298a
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn what would trigger test? please suggest.

Can probably add a make target? not sure how/if we have other tests like this

RE: testing... quickest way to create a test would probably be to add a flag to the script to "test" itself and a function in there to call the util functions with a set of params and check results.  That could be invoked under make lint target or something existing.
Staying with bash, an alternative would be to pull out the utils into another file and have some go test code somewhere that uses exec to source and exec util functions and check results.

@tiswanso re creating new util functions in separate file while staying with bash.
I think we can workaround by adding a top level function for all the default intialization in iptable-start.sh.
And then outside
case $1 in
   test-case)
   do_nothing # just to get those util functions defined in the bash to be sourced.
   *)
    do_all_default_setup_as_today

Could we continue making progress on this fix?

@incfly I will complete either over the weekend or Monday the latest, on training this week..

I think we should use these 2 regex, they are not ideal but I think it is as close as we could get using bash without making regex completely unreadable.
For testing we could create something like make test-scripts where we could source bash based scripts and then run some sort of unit tests against functions defined in scripts.
@incfly @tiswanso what do you think?

Thanks for the fix. Let's ensure this time we have it well tested. I can help to look into what can be done to add a test target. Shouldn't be hard, I think.

@incfly @tiswanso tests/scripts/scripts_test.sh can be used as a common place to test bash functions. Now we just need to add start of this script to one of CI jobs, what do you think about ci/circleci: shellcheck?

PASS
ok  	istio.io/istio/pkg/version	0.034s
# Execute bash shell unit tests scripts
./tests/scripts/scripts_test.sh
Running IPv6 and IPv4 addresses validation functions...
Test has been completed successfully...

Now it runs as a part of circleci: test

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: linsun, sbezverk
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [linsun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Vadliation.sh does not to include in debian pkg. Ways to reproduce
make sidecar-deb
dpkg-deb -c  $GOPATH/out/linux_amd64/release/istio-sidecar.deb

drwxr-xr-x 0/0               0 2019-04-29 10:15 ./
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./usr/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./usr/local/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./usr/local/bin/
-rwxr-x--- 0/0        25132952 2019-04-29 10:15 ./usr/local/bin/envoy
-rwxr-x--- 0/0            2189 2019-04-25 16:40 ./usr/local/bin/istio-node-agent-start.sh
-rwxr-xr-x 0/0           21341 2019-04-29 10:10 ./usr/local/bin/istio-iptables.sh
-rwxr-x--- 0/0        18887544 2019-04-24 14:38 ./usr/local/bin/node_agent
-rwxr-x--- 0/0        40033110 2019-04-24 14:38 ./usr/local/bin/pilot-agent
-rwxr-xr-x 0/0            4479 2019-04-26 10:15 ./usr/local/bin/istio-start.sh
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./usr/share/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./usr/share/doc/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./usr/share/doc/istio-sidecar/
-rw-r--r-- 0/0             138 2019-04-29 10:15 ./usr/share/doc/istio-sidecar/changelog.gz
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./var/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./var/lib/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./var/lib/istio/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./var/lib/istio/envoy/
-rw-r----- 0/0             174 2019-04-25 16:40 ./var/lib/istio/envoy/envoy_bootstrap_drain.json
-rw-r----- 0/0            8588 2019-04-25 16:40 ./var/lib/istio/envoy/envoy_bootstrap_tmpl.json
-rw-r----- 0/0            3562 2019-04-25 16:40 ./var/lib/istio/envoy/sidecar.env
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./lib/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./lib/systemd/
drwxr-xr-x 0/0               0 2019-04-29 10:15 ./lib/systemd/system/
-rw-r----- 0/0             250 2019-04-25 16:40 ./lib/systemd/system/istio-auth-node-agent.service
-rw-r----- 0/0             222 2019-04-25 16:40 ./lib/systemd/system/istio.service

I tried that on your branch.

@costinm very surprised, how it could break anything if 1. it runs perfectly fine in my test bed, 2. all CI tests are passing?!?!

So mesh expansion asks users to install debian package on their VM. I tried to build the debian package on your branch, and it does not include the validations script, which means that by the time debian package are started, iptables-start.sh will fail to load validation.sh.
The reason why no test fails is because we don't have e2e test for mesh expansion, but I'm adding one. And you can see because of the ip format parsing issue, I have to change to 1.1.3 debian version to get test passing temporarily.

@incfly please check the latest revision and let me know if it is still missing.

/lgtm
Looks good to me. I'll let @costinm to do another pass/sign off.
@costin we have an offline discussion and go for the -t another bash cmd option in a single file.

@costinm could you please check PR and if you are ok, please remove hold.

fyi, this successful e2e-simple run is on k8s ipv6 (KDC DinD) run with 13563, 13117, and some testcase fix diffs (https://github.com/tiswanso/istio/tree/simpletest_ipv6):
https://circleci.com/gh/istio/cni/1828

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: costinm, incfly, linsun, sbezverk
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [costinm,linsun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold cancel

/test all

@sbezverk: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-simpleTests.sh
cbffecce564692500351ce0c29fde0b237a23279
link
/test e2e-simpleTests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e-simpleTests

/retest

@sbezverk: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
cbffecce564692500351ce0c29fde0b237a23279
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Cast MediaPlayers go to Unavailable after a while

Home Assistant release with the issue:

0.78.3
Last working Home Assistant release (if known):
0.77.x (I think)
Operating environment (Hass.io/Docker/Windows/etc.):

Hassio on RP3
hassOS 1.10 & hassOS 1.11
Component/platform:

cast
Description of problem:
After a couple of days, mediaplayer become unavailable.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
discovery:
  ignore:
    - samsung_tv

Traceback (if applicable):
18-09-27 03:08:18 ERROR (Thread-784) [pychromecast.socket_client] Error reading from socket.
2018-09-27 03:08:18 WARNING (Thread-784) [pychromecast.socket_client] Error communicating with socket, resetting connection
2018-09-27 03:08:18 ERROR (Thread-784) [pychromecast.socket_client] Failed to connect, retrying in 2.0s
2018-09-27 03:08:22 ERROR (Thread-784) [pychromecast.socket_client] Failed to connect. No retries.
2018-09-27 03:09:05 ERROR (Thread-886) [pychromecast.socket_client] Error reading from socket.
2018-09-27 03:09:05 WARNING (Thread-886) [pychromecast.socket_client] Error communicating with socket, resetting connection
2018-09-27 03:09:05 ERROR (Thread-886) [pychromecast.socket_client] Failed to connect. No retries.

Additional information:
I have 7 google casts in the house (1x home, 2x mini, 2x audio, 2x chromecasts) and they're shown in my frontend.  I've always had reading from socket errors in the logs, but it's occasional and it always resolved itself after a while.
Since updating to 0.78.x after a couple of days, my mediaplayers will eventually come up as Unavailable and stay like that.  All automations around the media players never fire (e.g. automatic sleep timers fail and podcasts are played all night!).
A reboot fixes the issue, but only for a couple of days.

Duplicate as #16686

Sorry, I did search honest!

It did I'm afraid.

@nelbs please open a new issue for your case. Chromecast component is complex, the previous fix only address particular configuration.
When you open the issue, please provide your debug log, config, and what type of Chromecast device involved. Chromecast, Google Home, or Smart TV, etc.


Refactor ExtData to remove "little" cfio

MAPL_ExtData on the development branch of MAPL is currently in a state of confusion. It is using the pFIO server for data access but this is being called from MAPL_CFIO and ExtData itself is using the old "little" cfio layer for file metadata information. This layer is difficult to maintain and not flexible.

This is done for as far as ExtData is concerned.


Add DataTables plugin

https://github.com/ypnos-web/cakephp-datatables
The plugin consists of a Component and a Helper to easily use jQuery DataTables in a CakePHP app.
The component provides a finder wrapper that processes AJAX requests by DataTables.
The Helper provides an easy method of setting up the table in a template.

I'd say some tests would be nice to ensure no regressions can happen.

Yes, I agree.
Right now I finished an initial stable version that is pretty feature complete.
Tests come next before I add new code to master for the obvious reason.
You can wait for the tests. From my perspective I think it also makes sense not to wait hence the PR.

We'll wait for tests :)

@ypnos-web Any update?

what about accepting the plugins witch not have tests, by adding a note to the end of it's name.
just an idea

That's not acceptable IMO. The tests help to see early on if the plugin is still compatible with the future minors and majors, making it easier to avoid regressions.
We owe that to the people deciding for a plugin and starting to use it.
This list is supposed to outline awesome plugins - and that is an important piece of this :)
Please help us keeping this high standard of quality, and making it easy for people to trust into this awesome list.

you are right

The status is that I have plans for the plugin, including the tests, but it will take until summer before I get to it. Until then status quo (only bugfixes, tests still missing).
I fully agree with having a strict policy for all entries.

Closing the PR for now in that case. It's already been open for almost a year. You are welcome to make a new PR when you have tests done.


Using. KMS key in AWS

If there are secrets stored in AWS secrets manager with or without a KMS key and I have an EKS cluster created with that KMS key, does KES know when to use the key automatically or is there some kind fo setting in the external secret?
David

Correct me if I am wrong.  The KMS key added to the cluster at cluster creation time is or encrypting the passwords in the cluster - etcd.  The question I am asking is about using a custom KMS key to store passwords in Secrets Manager.  If no KMS key is used when storing a password in Secrets Managerthen AWS uses its own default key.  If customer key us used to encrypt the password in Secrets Manager, how does KES know to use the custom key when fetching the secret value? Does the policy for the ExternalSecret role need to vie modified to mention the custom key.  Something like the following?  I found this in YAML form in issue Add required IAM permissions to Readme #116.  This should allow access to decrypt using the keys which the secrets are encrypted and granting access to a sub path of values in this case hello-service.  Is this correct?
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Decrypt",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "arn:aws:kms:us-east-1:0123456789AB:key/a0495691-1f7a-4d94-b27d-58a20e2d1aef"
},
{
"Sid": "Password",
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue"
]
"Resource": [
"arn:aws:secretsmanager:us-east-1:0123456789AB:secret:/hello-service/*"
]
}
]
}


New beta UsesFeatures targets 0 devices upon distributing app

I might be misunderstanding something here, but i think this is a bug :)
Bug
When installing the new GeoLocator plugin, it UsesFeatures (location etc.). When distributing to Google Play Store, the apk then targets 0 units.
Even with these in AssenblyInfo, it does not work.
[assembly: UsesFeature("android.hardware.location", Required = false)]
[assembly: UsesFeature("android.hardware.location.gps", Required = false)]
[assembly: UsesFeature("android.hardware.location.network", Required = false)]
Version Number of Plugin: <IP_ADDRESS>-beta
Device Tested On: -
Simulator Tested On: -
Expected Behavior
The apk to be able to be deployed to Google Play and through that installed to devices
Actual Behavior
No devices can install from Google Play
Steps to reproduce the Behavior
Add Geolocator beta to app, publish on store.

Because it adds android.feature.location.GPS and android.feature.NETWORK to function list, while the correct ones are android.hardware.location.GPS and android.hardware.location.NETWORK :)


swastik pattern(cpp)

Hacktoberfest 2021

@vishalb52 I'm closing the PR as it is not fulfilling the requirement...you can send a PR again after maintaining all the steps and eligibility criteria for a valid PR...


Firefox: Add backup

https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles


Add build_args also to ct_s2i_build_as_df function.

Add build_args also to ct_s2i_build_as_df function.
We have to functions to build image. This one and ct_test_app_dockerfile
In case of --ulimit is defined in s2i_args variable we will add this parameter to docker/podman command line.
This pull request blocks https://github.com/sclorg/s2i-nodejs-container/pull/425.

[test]

[test]

[test]

I have tested this updated PR with s2i-nodejs-container repo and it works
[test]

[test]


Fix typo in tutorial

Thanks for the PR! This was actually intended though to show off how you can add trait bounds to the Item type of a future, rather than specify a particular type which the Error = syntax is doing. I wonder though if perhaps it's confusing to bound the type by the Future trait which is also being discussed? Maybe something like Clone would be a bit more ubiquitous and less confusing?

Ah, indeed! I suppose my change wouldn't actually compile, would it :P
I agree that using a different trait would be less confusing. But then again, I wonder how important it is to show off this particular aspect of the Rust grammar, as it's not specific to the futures-rs crate.
In any case, I'll leave this decision to you guys. Great work, by the way!


db_download not working for EMBL

Hello,
The db_download function is not working for the EMBL database, presumably because they changed the URL. The output is pasted below for the mammal download script in the README:

crabs db_download --source embl --database 'mam*' --output embl_mam.fasta --keep_original yes

Thanks,
Mike
downloading sequences from EMBL
--2023-05-21 09:29:10--  ftp://ftp.ebi.ac.uk/pub/databases/ena/sequence/con-std_latest/std/STD_mam*dat.gz
=> ‘.listing’
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... <IP_ADDRESS>
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|<IP_ADDRESS>|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/databases/ena/sequence/con-std_latest/std ... done.
==> PASV ... done.    ==> LIST ... done.
[    <=>                                                                                ] 68,057      25.8KB/s   in 2.6s

2023-05-21 09:29:14 (25.8 KB/s) - ‘.listing’ saved [68057]
Removed ‘.listing’.
No matches on pattern ‘STD_mam*dat.gz’.
Combining all EMBL downloaded fasta files...
formatting intermediary file to CRABS format
0it [00:00, ?it/s]
found 0 sequences with incorrect accession format
written 0 sequences to embl_mam.fasta

Hello @mikeallen-eco,
Could you please try capitalising your entry, i.e., ‘—database MAM*’? I believe this will solve your issue. Please let me know if this doesn’t work and I’ll look into it further.
I’ll try to get the code modified, so that it will capitalise the users entry automatically.
Best regards,
Gert-Jan

That worked. Thank you!
Mike

Hi @gjeunen,
Is it possible that this work-around for downloading sequences from the EMBL database no longer works?
I have tried the following two commands:
singularity exec /PATH_TO_CONTAINER/crabs_0.1.7.sif crabs db_download --source embl --database 'VRT*' --output embl_vrt.fasta --keep_original yes
and
singularity exec /PATH_TO_CONTAINER/crabs_0.1.7.sif crabs db_download --source embl --database 'vrt*' --output embl_vrt.fasta --keep_original yes
However, although the job finishes running, both result in an empty '.fasta' file. My associated error file contains a single line stating:
0it [00:00, ?it/s]
Could something be wrong with the link used to query the files?
Thanks for any input!
Cheers,
Thomas


Make the wait for a server connection as a phase

Also, looks like the build failed for some reason.

Current coverage is 64.32% (diff: 100%)

Merging #901 into master will decrease coverage by 0.09%

@@             master       #901   diff @@
==========================================
  Files           735        735
  Lines         34852      34853     +1
  Methods           0          0
  Messages          0          0
  Branches       5307       5307
==========================================
- Hits          22451      22419    -32
- Misses        10831      10865    +34
+ Partials       1570       1569     -1

Powered by Codecov. Last update a15c624...654f9e1


Correct model name in OpenAI integration docs

This is a small correction of the code sample to set a custom OpenAI context. The identifier GPT-4o mini causes an error message (Invalid model).
Testing
Ran mkdocs serve and verified changes.

Thanks @SH4DY


Using the recording status flip/policy API with a bot that joins the call via joinURL

I have an issue which is a subset of #444  - it is still unaddressed there.
My use case is that I want to query a recording bot into a meeting manually and then notify the users that the call is now being recorded via the OnRecordingStatusFlip API. I can already record users, so this is really only about having the notification and policy compliance (if the users are assigned the policy).
Is this possible at all? How would it work with external/guest users? Currently I am still getting exception: Code: 8506 Message: API call not allowed for non-Compliance Recording call.

It's not a subset of that issue it's exact the same issue and it's not possible only policy joined bot's are allowed to flip the recording status right now


about the computation of variance

Hi, these days I find an old paper Estimating the Mean and Variance of the Target Probability Distribution to support the idea about considering mean and variance as two independent output of the network. I think that Eq. 11-14 may be the reason why you regress the images (though I don't know why).

Besides, I'm still wondering that what is the difference between taking the variance as a direct output and computing the variance based on the sampled results. Do they actually have the same meaning (or same value)? Thanks!

you regress the images (though I don't know why).

Simply because I am interested in a regression problem. In the NIPS paper you can also find the case of classification.

what is the difference between taking the variance as a direct output and computing the variance based on the sampled results. Do they actually have the same meaning (or same value)?

Again according to my understanding of the NIPS paper, they do not have the same meaning, the former being aleatoric uncertainty and the latter being epistemic.

Sorry that maybe I misuse the word. I mean that Eq. 11-14 may be the reason why you use an encoder-decoder framework.

Besides, in combined/model.py line 103~109, it seems only aleatoric uncertainty takes part in the loss function. So why is this a combination of both two uncertainties? Does 'Combining Aleatoric and Epistemic Uncertainty in One Model' mean that we can get two kinds of uncertainties of the model, rather than considering both of them in the loss function to see how do these two uncertainties affect the model? (Although the predictive uncertainty for pixel y is the combination of aleatoric and epistemic, i.e. Eq. 9)

So why is this a combination of both two uncertainties?

Only aleatoric uncertainty takes part in the loss function, since epistemic uncertainty estimation is simply done by using dropout at test time. I sample 20 nets, and compute mean and variance of their predictions. In my understanding this is the way of combining the two uncertainties.

Hi, I have read another paper Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding, in which there is a Bayesian SegNet with an encoder-decoder architecture. Encoder encodes features and decoder upsamples them with dropout layer inserted. So I wonder that is your encoder-decoder architecture just for the specific network (SegNet for example), or is it designed to get mean and variance of the output? Thanks!

So I wonder that is your encoder-decoder architecture just for the specific network (SegNet for example), or is it designed to get mean and variance of the output?

Hi, sorry but again I am not completely sure about the meaning of your question. I also design an encoder-decoder for regression purposes. The decoder has two heads to regress mean as well as variance in the case of aleatoric uncertainty.

I'm sorry that maybe I don't make my purpose clear. Put simply, how can I implement predictive mean and predictive variance in a classification task without encoder-decoder architecture?

Oh ok, got it.
I think you should first define whether your are interested in 1) Heteroscedastic (Aleatoric) uncertainty or 2) epistemic uncertainty.

You should check section 3.3 of the paper. I have not read carefully that part.
I think you can obtain it by performing dropout at test time and averaging as suggested by equation (3).

In any case i suggest to study carefully the whole paper. Consider contacting the authors as well, they can help you better I guess.
Best,
P.

Thank you for your suggestion. For your suggestion 2, the epistemic uncertainty can be easily captured by performing dropout and averaging the outputs, but modeling aleatoric uncertainty in classification also needs to output predictive mean as well as predictive variance just like regression task (please see Eq. 6), so I'm interested in your method about using an encoder-decoder architecture to do this, and also wondering that whether I can get mean and variance without it. I think in this step, both classification task and regression task are nealy the same. After getting mean and variance, we can use it to disturb logit vector to get the form of loss attenuation of a classification scenario, i.e. Eq. 12 (I'm pretty clear about this part).
So in a word, I just want to know how to 'use a single network to transform the input x, with its head split to predict both y^ as well as σ^2'? Thanks!

I think in this step, both classification task and regression task are nealy the same

Not so sure about this.

So in a word, I just want to know how to use a single network to transform the input x, with its head split to predict both y^ as well as σ^2'?

For regression I have implemented it in model.py in the aleatoric-uncertainty folder. I guess there are other ways of doing that. For classification, honestly I don't know.


Add subscribe initial position for consumer cli

Motivation
In some case, users expect to consume messages from beginning similar to the option --from-beginning of kafka consumer CLI.
Modifications
Add --subscription-position for pulsar-client and pulsar-perf.

/pulsarbot run-failure-checks


The get_ports_list() function does not work in the reference device.

The get_ports_list() function does not work in the reference device.
The duplicate code is as follows

import gdsfactory as gf

test =gf.Component()

gc = gf.components.grating_coupler_te()
gc1 = test.add_ref(gc)

ports = gc1.get_ports_list()

The error message is as follows:

Environment:

i added some docs about this into the migration guide


Check how Mathlab shifts zero frequencies to zero (normalization)

https://www.mathworks.com/help/matlab/ref/fftshift.html
If possible, apply useful features to this crate.

https://www.gaussianwaves.com/2015/11/interpreting-fft-results-complex-dft-frequency-bins-and-fftshift/
But this is probably not in the responsibility of this crate but in higher level APIs.


To Do List App feature

Fully implemented the todo list app

Hi @dazehinn,

Good job so far!
There are some issues that you still need to work on to make your project better but you are almost there!
Highlights
✔ Gitflow was correctly used
✔ Good design
✔ Routers implemented
✔ App functional
Required Changes ♻
♻ While you configured the linters correctly, you disabled all of them so please do the code without them.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.


A Little Better APP

a app.py support (https://hf-mirror.com/Lykon/dreamshaper-8-lcm/tree/main) , (https://huggingface.co/Lykon/dreamshaper-7/tree/main) . (https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7)
have an dropdown to chose the model

from future import annotations
import os
import random
import time
import gradio as gr
import numpy as np
import PIL.Image
import torch
from diffusers import AutoPipelineForText2Image
from diffusers import LCMScheduler
import torch
import os
import torch
from concurrent.futures import ThreadPoolExecutor
import uuid
temp_model="114"
DESCRIPTION = '''# Latent Consistency Model
Distilled from Dreamshaper v7 fine-tune of Stable Diffusion v1-5 with only 4,000 training iterations (~32 A100 GPU Hours). Project page
'''
if torch.cuda.is_available():
DESCRIPTION += "\nRunning on CUDA 😀"
elif hasattr(torch, 'xpu') and torch.xpu.is_available():
DESCRIPTION += "\nRunning on XPU 🤓"
else:
DESCRIPTION += "\nRunning on CPU 🥶 This demo does not work on CPU."
MAX_SEED = np.iinfo(np.int32).max
CACHE_EXAMPLES = torch.cuda.is_available() and os.getenv("CACHE_EXAMPLES") == "1"
MAX_IMAGE_SIZE = int(os.getenv("MAX_IMAGE_SIZE", "1024"))
USE_TORCH_COMPILE = os.getenv("USE_TORCH_COMPILE") == "1"
"""
Operation System Options:
If you are using MacOS, please set the following (device="mps") ;
If you are using Linux & Windows with Nvidia GPU, please set the device="cuda";
If you are using Linux & Windows with Intel Arc GPU, please set the device="xpu";
"""
device = "mps"    # MacOS
device = "xpu"    # Intel Arc GPU
device = "cuda"   # Linux & Windows
"""
DTYPE Options:
To reduce GPU memory you can set "DTYPE=torch.float16",
but image quality might be compromised
"""
DTYPE = torch.float16  # torch.float16 works as well, but pictures seem to be a bit worse
def load_model(model_path):
global pipe
pipe = AutoPipelineForText2Image.from_pretrained(model_path, variant="fp32",requires_safety_checker=False)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
cuda0 = torch.device('cuda:0')
pipe = pipe.to(cuda0, dtype=torch.float32,non_blocking=True)
return pipe
def randomize_seed_fn(seed: int, randomize_seed: bool) -> int:
if randomize_seed:
seed = random.randint(0, MAX_SEED)
return seed
def save_image(img, profile: gr.OAuthProfile | None, metadata: dict, root_path='./'):
unique_name = str(uuid.uuid4()) + '.png'
unique_name = os.path.join(root_path, unique_name)
img.save(unique_name)
# gr_user_history.save_image(label=metadata["prompt"], image=img, profile=profile, metadata=metadata)
return unique_name
def save_images(image_array, profile: gr.OAuthProfile | None, metadata: dict):
paths = []
root_path = './images/'
os.makedirs(root_path, exist_ok=True)
with ThreadPoolExecutor() as executor:
paths = list(executor.map(save_image, image_array, [profile]*len(image_array), [metadata]*len(image_array), [root_path]*len(image_array)))
return paths
def generate(
prompt: str,
model_path:str,
seed: int = 0,
width: int = 512,
height: int = 512,
guidance_scale: float = 2.0,
num_inference_steps: int = 15,
num_images: int = 4,
randomize_seed: bool = False,
profile: gr.OAuthProfile | None = None,
) -> PIL.Image.Image:
global temp_model
global pipe
if model_path != temp_model:
load_model(model_path)
temp_model = model_path
cuda0 = torch.device('cuda:0')
pipe = pipe.to(cuda0, dtype=torch.float32,non_blocking=True)
seed = randomize_seed_fn(seed, randomize_seed)
torch.manual_seed(seed)
start_time = time.time()
result = pipe(
prompt=prompt,
width=width,
height=height,
guidance_scale=guidance_scale,
num_inference_steps=num_inference_steps,
num_images_per_prompt=num_images,
lcm_origin_steps=50,
output_type="pil",
).images
paths = save_images(result, profile, metadata={"prompt": prompt, "seed": seed, "width": width, "height": height, "guidance_scale": guidance_scale, "num_inference_steps": num_inference_steps})
print(time.time() - start_time)
return paths, seed
examples = [
"portrait photo of a girl, photograph, highly detailed face, depth of field, moody light, golden hour, centered, extremely detailed,  award winning photography",
"Self-portrait oil painting, a beautiful cyborg with golden hair, 8k",
"Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",
"A photo of beautiful mountain with realistic sunset and blue lake, highly detailed, masterpiece",
]
with gr.Blocks(css="style.css") as demo:
gr.Markdown(DESCRIPTION)
gr.DuplicateButton(
value="Duplicate Space for private use",
elem_id="duplicate-button",
visible=os.getenv("SHOW_DUPLICATE_BUTTON") == "1",
)
with gr.Group():
with gr.Row():
prompt = gr.Text(
label="Prompt",
show_label=False,
max_lines=1,
placeholder="Enter your prompt",
container=False,
)
run_button = gr.Button("Run", scale=0)
result = gr.Gallery(
label="Generated images", show_label=False, elem_id="gallery",
)
with gr.Accordion("Advanced options", open=False):
seed = gr.Slider(
label="Seed",
minimum=0,
maximum=MAX_SEED,
step=1,
value=0,
randomize=True
)
randomize_seed = gr.Checkbox(label="Randomize seed across runs", value=True)
with gr.Row():
width = gr.Slider(
label="Width",
minimum=256,
maximum=MAX_IMAGE_SIZE,
step=32,
value=512,
)
height = gr.Slider(
label="Height",
minimum=256,
maximum=MAX_IMAGE_SIZE,
step=32,
value=512,
)
with gr.Row():
guidance_scale = gr.Slider(
label="Guidance scale for base",
minimum=2,
maximum=14,
step=0.1,
value=2.0,
)
num_inference_steps = gr.Slider(
label="Number of inference steps for base",
minimum=1,
maximum=8,
step=1,
value=4,
)
with gr.Row():
num_images = gr.Slider(
label="Number of images",
minimum=1,
maximum=8,
step=1,
value=2,
visible=True,
)
with gr.Row():
data = ['D://lcm//model//lcm-v8', 'D://lcm//model//lcm-v7', 'D://lcm//model//lcm-fast']
model_path = gr.Dropdown(
data,
label="Model",
)
# with gr.Accordion("Past generations", open=False):
#     gr_user_history.render()

gr.Examples(
    examples=examples,
    inputs=prompt,
    outputs=result,
    fn=generate,
    cache_examples=CACHE_EXAMPLES,
)

gr.on(
    triggers=[
        prompt.submit,
        run_button.click,
    ],
    fn=generate,
    inputs=[
        prompt,
        model_path,
        seed,
        width,
        height,
        guidance_scale,
        num_inference_steps,
        num_images,
        randomize_seed,
    ],
    outputs=[result, seed],
    api_name="run",
)

if name == "main":
demo.queue(api_open=False)
# demo.queue(max_size=20).launch()
demo.launch(server_name="<IP_ADDRESS>")


SP Week of May 17

TEMPEST

[ ] Use function we wrote to read in all loggers
[ ] Merge with ports.csv to link to tree codes / plots
[ ] QA/QC: look for problems with that merge (missing tree codes, logger codes etc.)
[ ] QA/QC: flag out of bounds - negative values and values over 1.0
[ ] Email Roy for his sapflow troubleshooting process and updates

ESS-DIVE

[ ] If we have consensus on term format, update terms

Synthesis

@stephpenn1 so writing another paper is something you'd like to do. 👍  Here are some possibilities to think about, and we can discuss further:

Restart the Rs-Js work. I guess the analysis is 80% done here, but it is a little tricky because there are lots of different angles (as you know).
Lead the first synthesis paper. I think the analytical work is 50% done here, but I would heavily take charge of that—or work in concert with you. It would be focused and tight, focusing on the meta-analysis of three separate response variables for water addition and separation.
Something like fortedata! For example, build a compassdata package and a companion Shiny data dashboard, and write a paper describing them, kind of like Jeff's paper.


add log directory and ignore log files

What changes were proposed in this pull request?

Why are the changes needed?

Will break the compatibility? How if so?

Does this PR introduce any user-facing change?

How was this patch tested?

Checklist

[ ] I've run the tests to see all new and existing tests pass.
[ ] If this Pull Request resolves an issue, I linked to the issue in the text above.
[ ] I've informed the technical writer about the documentation change if necessary.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.luliwjc seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

LGTM


Build Search option for Rnplay web version

Would be really nice to have the search api integrated on rnplay.org
As i understand the functionality shall be there from the API. it shall be a quick one.

I agree this would be great to have. Any interest in helping out?

I'll try to. help me out with the endpoint url for search. i guess only the UI work might be needed. I'm no expert at rails (work on Java/python/node and JS mostly) . but from what i understand It shall be all front-end. i am at the reactiflux channnel , we can probably continue the conversation there.

@saurshaz Any interest in this still?

OK, thanks!
El jue., 31 dic. 2015 a las 18:46, Saurabh SHarma<EMAIL_ADDRESS>escribió:

@jsierles https://github.com/jsierles interest yes, short of time to
devote guys.
—
Reply to this email directly or view it on GitHub
https://github.com/rnplay/rnplay-web/issues/35#issuecomment-168227266.

OK, thanks!
El jue., 31 dic. 2015 a las 18:46, Saurabh SHarma<EMAIL_ADDRESS>escribió:

@jsierles https://github.com/jsierles interest yes, short of time to
devote guys.
—
Reply to this email directly or view it on GitHub
https://github.com/rnplay/rnplay-web/issues/35#issuecomment-168227266.

Going to close this for now. Feel free to open again later if someone starts on this!


DO NOT MERGE: cardano-crypto-class: Bump vector dependency

Just making a PR for a point release.

This should not have been a release, it could have been taken care of by a simple revision.
Regardless of this fact the version bump was incorrect, it should have been a patch release: <IP_ADDRESS> and the upper bound could have been completely removed, rather than set to < 0.14

Release was made closing


chore(deps): bump Go dependencies to latest version

#297  👈
dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @sergeyberezansky and the rest of your teammates on  Graphite

Merge activity

Jul 31, 10:46 AM EDT: @sergeyberezansky started a stack merge that includes this pull request via Graphite.


Layout Issue

@javivelasco @patrick-jones
Am trying out http://react-toolbox.com/#/components/layout. It works fine mostly but if I set the NavDrawer to pinned mode and try to resize the screen see below

The drawer disappears but the panel does not resize to fill the screen.

+1 I'm having the same problem.

May you try if it works now?

This is solved


Needs a dark mode

My eyes hurt

Styles are in yukon\web\style

It might be possible to copy styles from the webpage of Yukon (yes it runs in a web browser too natively) and then to use a plugin that converts the page to dark mode. This saves thinking on what styles to use.


Update Issue - 62275194

default description
URL tested: http://ci.bsstag.com/welcome
Open URL on Browserstack

Property
Value

Browser
Chrome 76.0

Operating System
Windows 10

Resolution
1024x612

Screenshot Attached
Screenshot URL

Click here to reproduce the issue on Browserstack

Updated Screenshot Attached


Handling Labels with Forward Slashes

These were all single labels which were restored as nested labels. They used the date format:

2015 Registrations
2/8
3/1
3/22
etc

i'm encountering this, too. anyone got a workaround or fix?

Gmail has considers / to be the divider for label hierarchies. GYB assumes that labels with a / in their name should also have their parent label created and does this by default. I recognize this is inconvenient to users who have labels like this with a / in their name but the alternative of not creating these parent labels is equally as inconvenient and is also more prevalent.
Closing issue as won't fix.


feat: support calibrate local timestamp

Description

support calibrate local timestamp

General Checklist

[x] Added new tests to cover change, if needed
[x] Build succeeds using Swift Package Manager
[x] All unit tests pass
[ ] Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
[ ] Documentation update for the change if required
[x] PR title conforms to conventional commit style
[ ] If breaking change, documentation/changelog update with migration instructions

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.72%. Comparing base (63e1df1) to head (85999b1).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
+ Coverage   95.66%   95.72%   +0.05%
==========================================
  Files          35       35
  Lines        1986     1986
==========================================
+ Hits         1900     1901       +1
+ Misses         86       85       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


manually set tlast=1 when sending the last sample

I am using AxiStreamSource to write to the DUT, I am sending 8 samples of 12 bits each and I want to set tlast high only after all the samples are written, how can I do it?

tlast is for framing.  So, send your samples as a "block" of 8 samples with a single call to send, and tlast will get set on the last sample.

I have a list containing around 5000 values. The following is how I am writing.
for i in range (0,len(data_list)-1,1)
write_data=data_list[i]
await axi_source.write(write_data)
In here after 8 samples tlast goes high, but I want it to be high after at 4999th data. How can I do that?

What about await axi_source.send(data_list)?  If you want to use tlast for framing, don't use read() and write(), instead use send() and recv().

It’s still the same. tlast is going high for each 8 samples. I checked the axis.py file
Isn’t it that write method calls send method?
Are both same ?


add Simplified Chinese translation for README

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: WWILLV sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for taking the time to contribute this localization pull request. However, we are not able to accept contributions to this historical reference repository at this time.


Help with sticky keys

Hello.
I've looked at some of the examples but cannot figure out how to add sticky keys.  Can anyone help?
Specifically here's what I'm trying to do:
I want to remap some keys on my homerow so that when I press and release L-shift, and then press 'd', I get '{', 'f' would give '[', 'g' would give '(', 'h' would give ')', 'j' would give ']', 'k' would give '}', etc.
Can anybody help me out with this mapping?
Thanks a lot!
Adam

Please use new setting by this private.xml:
https://github.com/tekezo/Files/blob/master/Karabiner/private.homerow_sticky/private.xml
How to use private.xml:
https://pqrs.org/osx/karabiner/document.html#privatexml

Wow! Thanks so much for your help.  Much appreciated!


stage/instantiate output folder requirement

To elaborate on the section Requirement of empty folder  from https://github.com/iterative/ldb-resources/issues/2 .
For the data-centric competition, I eventually need to instantiate the train and val splits in a format suitable for the training.
With the current behavior of stage/instantiate, I need to do:
$ cd competition
$ ldb stage ds:numerals-train train
$ cd train
$ ldb instantiate --format tensorflow-inferred
$ cd ..
$ rm -r train/.ldb_workspace  # Don't need it and I would need to update the training code to ignore the folder

And repeat the same steps for val split.
What I would like to do is to have a single .ldb_workspace at the top level of my project.
By having output folder as a requirement for instantiate instead of stage, I would be able to run:
$ ldb stage ds:numerals-train
$ ldb instantiate competition/train  --format tensorflow-inferred
$ ldb stag ds:numerals-val --force
$ ldb instantiate competition/val --format tensorflow-inferred

This should be addressed by alowing INSTANTIATE to specify a folder
https://github.com/iterative/ldb/pull/262

Tried https://github.com/iterative/ldb/pull/262 , solved


Crash editing js functions

Version: 1.20.5
Builder: Vite 5.4 (with sass-embedded 1.80)
Node 22.4.1
How: From blank canvas, add a Function an click: "edit function"
React code:
<div className="" style={{ height: 600 }}>
  <JdmConfigProvider>
    <DecisionGraph value={graph} onChange={(val) => setGraph(val)} style={{ height: 600 }} />
  </JdmConfigProvider>
</div>

Graph object when clicked "Edit Function"
{
    "nodes": [
        {
            "type": "functionNode",
            "content": {
                "source": "import zen from 'zen';\n\n/** @type {Handler} **/\nexport const handler = async (input) => {\n  return input;\n};\n"
            },
            "id": "4be5b4e1-6100-4720-ac5e-d4e481ecc5bf",
            "name": "function1",
            "position": {
                "x": 280,
                "y": 160
            }
        }
    ],
    "edges": []
}

Error:
Uncaught TypeError: Cannot read properties of undefined (reading 'variabletype_fromJson')
    at _VariableType.fromJson (chunk-YJYMPD56.js?v=24283913:92894:22)
    at createVariableType (chunk-YJYMPD56.js?v=24283913:93050:25)
    at chunk-YJYMPD56.js?v=24283913:193618:19
    at commitHookEffectListMount (chunk-RY6NLCXT.js?v=24283913:16915:34)
    at commitPassiveMountOnFiber (chunk-RY6NLCXT.js?v=24283913:18156:19)
    at commitPassiveMountEffects_complete (chunk-RY6NLCXT.js?v=24283913:18129:17)
    at commitPassiveMountEffects_begin (chunk-RY6NLCXT.js?v=24283913:18119:15)
    at commitPassiveMountEffects (chunk-RY6NLCXT.js?v=24283913:18109:11)
    at flushPassiveEffectsImpl (chunk-RY6NLCXT.js?v=24283913:19490:11)
    at flushPassiveEffects (chunk-RY6NLCXT.js?v=24283913:19447:22)

And then:
The above error occurred in the <Ra> component:

    at Ra (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:193578:13)
    at Suspense
    at Xr (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:191238:17)
    at div
    at div
    at http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:191311:47
    at div
    at http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:191289:67
    at Lo (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:184872:21)
    at ReactFlowProvider (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:92081:28)
    at div
    at http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:191328:15
    at MotionWrapper (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:34266:5)
    at ProviderChildren (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:34371:5)
    at ConfigProvider (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:34651:27)
    at Na (http://localhost:3000/node_modules/.vite/deps/chunk-YJYMPD56.js?v=24283913:193751:10)
    at div
    at div
    at div
    at RuleEngine (http://localhost:3000/src/Pages/RuleEngine/index.jsx?t=1729246768034:30:41)
    at RenderedRoute (http://localhost:3000/node_modules/.vite/deps/chunk-ZXSMJV3H.js?v=24283913:4062:5)
    at Routes (http://localhost:3000/node_modules/.vite/deps/chunk-ZXSMJV3H.js?v=24283913:4571:5)
    at div
    at div
    at App (http://localhost:3000/src/App.jsx?t=1729246768034:50:20)
    at RenderedRoute (http://localhost:3000/node_modules/.vite/deps/chunk-ZXSMJV3H.js?v=24283913:4062:5)
    at Routes (http://localhost:3000/node_modules/.vite/deps/chunk-ZXSMJV3H.js?v=24283913:4571:5)
    at Router (http://localhost:3000/node_modules/.vite/deps/chunk-ZXSMJV3H.js?v=24283913:4514:15)
    at BrowserRouter (http://localhost:3000/node_modules/.vite/deps/react-router-dom.js?v=24283913:554:5)
    at MQTTContextProvider (http://localhost:3000/src/Context/mqttConnector.jsx:43:27)
    at SocketContextProvider (http://localhost:3000/src/Context/socketContext.jsx:27:27)
    at ConsoleContextProvider (http://localhost:3000/src/Context/consoleContext.jsx:23:51)
    at LoginContextProvider (http://localhost:3000/src/Context/loginContext.jsx:23:39)
    at CurrentObjsContextProvider (http://localhost:3000/src/Context/currentObjsContext.jsx:24:45)
    at FiltersContextProvider (http://localhost:3000/src/Context/filtersContext.jsx:38:39)
    at UserContextProvider (http://localhost:3000/src/Context/userContext.jsx:23:41)

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://reactjs.org/link/error-boundaries to learn more about error boundaries.

I have the same issue.

I'm also experiencing this with webpack. Has anyone been able to get around it?

Hi, the issue is likely caused by WASM not being initialized, but being invoked. I've created a PR to solve this.
However we would still strongly suggest that you utilise web assembly as this enables auto-complete and type checking. This can be done differently depending on your bundler. For vite, this is done in straight forward way using https://github.com/gorules/editor/blob/master/vite.config.ts#L8. And then initialising it: https://github.com/gorules/editor/blob/master/src/main.tsx#L26.

Hi, the issue is likely caused by WASM not being initialized, but being invoked. I've created a PR to solve this.
However we would still strongly suggest that you utilise web assembly as this enables auto-complete and type checking. This can be done differently depending on your bundler. For vite, this is done in straight forward way using https://github.com/gorules/editor/blob/master/vite.config.ts#L8. And then initialising it: https://github.com/gorules/editor/blob/master/src/main.tsx#L26.

This worked smoothly, thank you @stefan-gorules , you are doing a very good job on gorules, this framework has a bright future.


Runtime Crash at "__swift_instantiateConcreteTypeFromMangledName"

Description
DiscordBM crashes at __swift_instantiateConcreteTypeFromMangledName.
Reproduction

Clone DiscordBM.
On CLI run BOT_TOKEN=aaaaaa swift test --filter IntegrationTests.DiscordClientTests.testGateway.
Or on Xcode add BOT_TOKEN as an env var with any value, and run the testGateway() test.

Stack dump

Expected behavior
No crash.
Environment

macOS 14 + Swift 5.10 + Xcode 15.4.
macOS 15 + Swift 6 + Xcode 16.

Not reproducible on Linux.
Additional information
No response

I believe this could be the same issue: https://github.com/apple/swift/issues/74333

I tried a bunch and i noticed it i comment out this line in PartialApplication declaration, it fixes the problem, at least partially:
    public var flags: IntBitField<DiscordApplication.Flag>?

So IntBitField<DiscordApplication.Flag>? must be causing the problem.

Using typealiases doesn't help.
removing the optional sign doesn't help.
Adding explicit Codable conformance doesn't help, but Xcode will show a line in the decoder as the source of the crash instead: self.flags = try container.decodeIfPresent(IntBitField<DiscordApplication.Flag>.self, forKey: PartialApplication.CodingKeys.flags)
Doing stuff like let uint = try container.decodeIfPresent(UInt.self, forKey: PartialApplication.CodingKeys.flags); self.flags = uint.map { .init(rawValue: $0) } in the decoder does not help.

CC: @al45tair @mikeash

Which commit should I be using with beta 3? I tried 7865b015f9135d8e1799a0f406be0174e4b04d85 (which is current ToT for main) and got a bunch of "error: pattern that the region based isolation checker does not understand how to check. Please file a bug." (Filed rdar://131471942 internally for that.) I tried ff804c560d14e5993261ad1bed923766c1b52265 which is just before mentions of Swift 6 and that crashes with a stack overflow. Then tried be95b07e3cc9db075d201f14051589676dd99785 and that seems to work fine. It's possible I need to more closely match the macOS build, but before I do that I want to make sure I'm targeting a commit that's known to build and crash (in the appropriate way!) with beta 3.

I would also really appreciate it if there are any workarounds to this so I can apply so earlier Swift versions can also continue to work properly 🙂.

Still an issue in Xcode 16 beta 4.

Still an issue on Xcode 16 beta 2.
And still getting frequent user reports about this issue which I have 0 idea how to even work-around.
Pretty frustrating.
I don't even know if looking around in the Swift repo is any good considering this only happens on macOS and Xcode contains a fork of the Swift repo, not the repo itself :/
Not that I'm any good at compiler work, but could have tried.
cc @tbkka @mikeash @hborla

@mikeash not sure what "commit" but any of the toolchains bundled with the Xcode versions I've mentioned should not have any trouble reproducing the issue.

Sorry, I meant which commit in the DiscordBM repository should I use to try to reproduce this?

Ah hmm really any recent commit. Just checkout main branch.

I don't recall what trouble I had previously, but I tried again (with the latest main and Xcode 16.1b2) and I think I've reproduced the issue. The crash is a bit different from what you show, but it seems to be the same basic thing. The problem is that the init(from:) method on Gateway.Event uses an enormous amount of stack space, enough to run out of stack and then crash on the stack guard page. I'm getting a crash in ___chkstk_darwin, but I think your crash in __swift_instantiateConcreteTypeFromMangledName is the same thing, it just happens to be that call which hits the stack guard page in your run.
The cause of the enormous stack frame appears to be the enum Payload type. There are issues with stack usage by large enums with many cases that have associated values. Marking it as indirect, as in indirect public enum Payload: Sendable, mitigates that and allows the tests to run without crashing for me. Hopefully that can work for you as well.

@mikeash thank you, I'll take that workaround.

To be clear that enum's size is 100% legitimate. It's an enum with a case for each different type of Discord payloads, so the size of the enum is not really under my control.
Also would be nice to have a compile error instead of a runtime crash, at least.

Also ... why crash on macOS but not Linux? Just curious.

Yeah, not casting blame here, just describing the root of the issue. Something about how the stack frame layout code works causes it to make a separate area for each enum case's payload, rather using the same space for all of them. This is especially troublesome in debug builds, release builds may stand a better chance at using stack space more efficiently.
Stack space is an odd thing where most systems just sort of handwave the fact that the stack is finite and not particularly large. We just sort of assume it's going to be big enough, without any particular guarantee that it is. Usually you need a lot more recursion to run into the limit, though. This function is a really extreme case where a single function call eats up the whole stack.
I'd guess your Linux is running with a large stack, and so it manages not to run off the end.

Is this in debug builds?   I had to deal with a similar problem working with enums that had many payload cases:  The debug builds would use an enormous amount of stack space, but release builds were quite reasonable.  Here's the basic problem:
Consider this:
func f(e: MyEnum) {
  switch e {
  case a(let aa):
     let ab = aa
     let ac = ab
  case b(let bb):
     let bc = bb
     let bd = bc
  case c(let cc):
     ... etc ...
  }
}

In a debug build, the stack frame for this function will have a separate stack slot for every one of the above variables -- aa, ab, ac, bb, bc, bd, etc, etc.   That is necessary to ensure that you can inspect any one of them at any time in the debugger.  In a release build, the compiler will do some additional optimizations:  It will overlay the variables for each case so that aa/ab/ac use the same stack area as bb/bc/bd, and so on  This dramatically reduces the amount of stack space required, but means that none of the variables in the switch can be inspected after the end of the switch (because the debugger doesn't know which overlaid variables are really valid).
One way I found to work around this is to rewrite the above in a somewhat roundabout way:
func f(e: MyEnum) {
  switch e {
  case a(let aa):
    {
       let ab = aa
       let ac = ab
    }()
  case b(let bb):
    {
       let bc = bb
       let bd = bc
    }()
  case c(let cc):
     ... etc ...
  }
}

That is, each complex case handler gets put into a separate closure that's invoked immediately.  In Debug builds, these closures end up as separate functions with separate stack frames.  That avoids the main function having to reserve stack space for every path in the switch.  In release builds, these closures get inlined and then optimized, so the final result in that case is the same as in the first version above.
Does this help?

@mikeash

Yeah, not casting blame here, just describing the root of the issue.

No worries, I didn't notice any 🙂.

@tbkka

Is this in debug builds?

Hmm i don't recall trying on macOS with a release build, but macOS debug builds would crash while Ubuntu debug or release builds were both fine.

Is this the same crash?
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff206b190e __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff206e05bd pthread_kill + 263
2 libsystem_c.dylib 0x00007fff20635406 abort + 125
3 libcoreclr.dylib 0x0000000103ac9f7e mono_post_native_crash_handler + 14
4 libcoreclr.dylib 0x0000000103a6dffa mono_handle_native_crash + 458
5 libcoreclr.dylib 0x00000001039c46cf mono_sigsegv_signal_handler_debug + 335
6 libsystem_platform.dylib 0x00007fff20725d7d _sigtramp + 29
7 ???<PHONE_NUMBER>00000000 0 + 0
8 libswiftCore.dylib 0x00007fff2cdf42dd swift::Demangle::__runtime::Demangler::demangleSymbolicReference(unsigned char) + 141
9 libswiftCore.dylib 0x00007fff2cdf12a8 swift::Demangle::__runtime::Demangler::demangleType(__swift::__runtime::llvm::StringRef, std::__1::function<swift::Demangle::__runtime::Node* (swift::Demangle::__runtime::SymbolicReferenceKind, swift::Demangle::__runtime::Directness, int, void const*)>) + 168
10 libswiftCore.dylib 0x00007fff2cdd75a4 swift_getTypeByMangledNameImpl(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadataswift::InProcess const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTableswift::InProcess const* (swift::TargetMetadataswift::InProcess const*, unsigned int)>) + 516
11 libswiftCore.dylib 0x00007fff2cdd4d6d swift::swift_getTypeByMangledName(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadataswift::InProcess const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTableswift::InProcess const* (swift::TargetMetadataswift::InProcess const*, unsigned int)>) + 477
12 libswiftCore.dylib 0x00007fff2cdd4f9b swift_getTypeByMangledNameInContext + 171
13  HSTracker                       0x000000010231aa81  __swift_instantiateConcreteTypeFromMangledName (in HSTracker) (/<compiler-generated>:0)
14  HSTracker                       0x00000001025b8a9e  closure #1 in BattlegroundsSession.battlegroundsGameMode.setter (in HSTracker) (<stdin>:0)
15  HSTracker                       0x000000010263b259  thunk for @escaping @callee_guaranteed () -> () (in HSTracker) (/<compiler-generated>:0)
16 libdispatch.dylib 0x00007fff20535623 _dispatch_call_block_and_release + 12
17 libdispatch.dylib 0x00007fff20536806 _dispatch_client_callout + 8
18 libdispatch.dylib 0x00007fff20542b4f _dispatch_main_queue_callback_4CF + 940
19 com.apple.CoreFoundation 0x00007fff208158d8 CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE + 9
20 com.apple.CoreFoundation 0x00007fff207d7b32 __CFRunLoopRun + 2755
21 com.apple.CoreFoundation 0x00007fff207d69ac CFRunLoopRunSpecific + 563
22 com.apple.HIToolbox 0x00007fff28a211f3 RunCurrentEventLoopInMode + 292
23 com.apple.HIToolbox 0x00007fff28a20f55 ReceiveNextEventCommon + 587
24 com.apple.HIToolbox 0x00007fff28a20cf3 _BlockUntilNextEventMatchingListInModeWithFilter + 70
25 com.apple.AppKit 0x00007fff22fe0ad2 _DPSNextEvent + 864
26 com.apple.AppKit 0x00007fff22fdf2a5 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1364
27 com.apple.AppKit 0x00007fff22fd15c9 -[NSApplication run] + 586
28 com.apple.AppKit 0x00007fff22fa57cc NSApplicationMain + 816
29  HSTracker                       0x0000000102644369  main (in HSTracker) (AppDelegate.swift:19)
30 libdyld.dylib 0x00007fff206fbf3d start + 1

Reported on MacOS 11.7.0 only as far as I have heard so far. Most other users do not seem to be affected. Code was compiled with Xcode 16.1.

That block of core on #14 is the block seen here:
            let modified = _battlegroundsGameMode != newValue
            _battlegroundsGameMode = newValue
            if modified {
                DispatchQueue.main.async {
                    self.updateSectionsVisibilities()
                    if #available(macOS 10.15, *) {
                        Task.detached {
                            await self.updateCompositionStatsVisibility()
                        }
                    }
                    self.update()
                    self.updateScaling()
                }
            }

I decompiled the code and matched the address and it was inside the block being dispatched on the main thread and it is roughtly this:
int _$s9HSTracker20BattlegroundsSessionC21battlegroundsGameModeAA08SelectedbeF0OvsyyScMYccfU_(int arg0, int arg1, int arg2, int arg3) {
    rcx = arg3;
    r13 = arg0;
    HSTracker.BattlegroundsSession.updateSectionsVisibilities(arg0, arg1, arg2, rcx);
    rdi = 0xa;
    rsi = 0xf;
    rdx = 0x0;
    if ((Swift._stdlib_isOSVersionAtLeast(rdi, rsi, rdx) & 0x1) != 0x0) {
            r14 = &stack[-56] - (*(*(___swift_instantiateConcreteTypeFromMangledName(_$sScPSgMD, 0xf, 0x0, rcx) - 0x8) + 0x40) + 0xf & 0xfffffffffffffff0);

VM region near:
VM Regions Near 0:
-->
__TEXT 102315000-1030fa000 [ 13.9M] r-x/r-x SM=COW /Applications/HSTracker.app/Contents/MacOS/HSTracker

and
VM Region Summary:
ReadOnly portion of Libraries: Total=905.7M resident=0K(0%) swapped_out_or_unallocated=905.7M(100%)
Writable regions: Total=673.9M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=673.9M(100%)

                            VIRTUAL   REGION
REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Accelerate framework 256K 2
Activity Tracing 256K 1
CG backing stores 3240K 6
CG image 4100K 1
CoreAnimation 3904K 35
CoreGraphics 12K 2
CoreUI image data 340K 3
Foundation 16K 1
Image IO 48K 6
Kernel Alloc Once 8K 1
MALLOC 231.1M 82
MALLOC guard page 48K 12
MALLOC_LARGE (reserved) 128K 1 reserved VM address space (unallocated)
MALLOC_NANO (reserved) 384.0M 1 reserved VM address space (unallocated)
SQLite page cache 128K 2
STACK GUARD 56.1M 26
Stack 23.7M 26
VM_ALLOCATE 78.1M 53
VM_ALLOCATE (reserved) 8192K 1 reserved VM address space (unallocated)
__DATA 21.3M 448
__DATA_CONST 24.5M 279
__DATA_DIRTY 1462K 175
__FONT_DATA 4K 1
__LINKEDIT 502.7M 13
__OBJC_RO 70.3M 1
__OBJC_RW 2496K 3
__TEXT 403.0M 449
__UNICODE 588K 1
libnetwork 128K 8
mapped file 416.9M 48
shared memory 752K 13
=========== ======= =======
TOTAL 2.2G 1701
TOTAL, minus reserved VM space 1.8G 1701

Also forgot to say that this crash appears to have started after I upgraded from Xcode 15.3 to 16.1

@mikeash Thanks for pointing it out. I compared the previous application bundles and it was present until the most recent version, so I need to see what went wrong.

I reviewed my project.pbxprof changes and I don't see anything that should have caused the library from being included. I have added a manual link with library as optional per one Stackoverflow post suggestion and it seems to be packaging now. Wonder if this is an Xcode bug

I rebuilt the application with Xcode 15.3 and the problem went away. I tried building with 16.1 and 16.2 beta 3 and embedding libswift_Concurrency.dylib but the problem persisted until I build with 15.3 again.
Reported the issue to Apple via https://feedbackassistant.apple.com/feedback/15937972

@mikeash I downgraded to Xcode 15.3 and I verified the with otool -l that the @rpath is now correct but I am still getting a crash:
8 libswiftCore.dylib 0x00007fff2cc6c2dd swift::Demangle::__runtime::Demangler::demangleSymbolicReference(unsigned char) + 141
9 libswiftCore.dylib 0x00007fff2cc692a8 swift::Demangle::__runtime::Demangler::demangleType(__swift::__runtime::llvm::StringRef, std::__1::function<swift::Demangle::__runtime::Node* (swift::Demangle::__runtime::SymbolicReferenceKind, swift::Demangle::__runtime::Directness, int, void const*)>) + 168
10 libswiftCore.dylib 0x00007fff2cc4f5a4 swift_getTypeByMangledNameImpl(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 516
11 libswiftCore.dylib 0x00007fff2cc4cd6d swift::swift_getTypeByMangledName(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 477
12 libswiftCore.dylib 0x00007fff2cc4cf9b swift_getTypeByMangledNameInContext + 171
__swift_instantiateConcreteTypeFromMangledName (in HSTracker) (/<compiler-generated>:0)
closure #1 in BattlegroundsSession.battlegroundsGameMode.setter (in HSTracker) (<stdin>:0)
thunk for @escaping @callee_guaranteed () -> () (in HSTracker) (/<compiler-generated>:0)

Here's the output of otool -l:
Load command 49
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name @rpath/libswift_Concurrency.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 0.0.0
compatibility version 1.0.0

Any ideas of what else might be wrong? I don't know how to confirm I have fixed the crash without access to a system with the issue


Update README.md

http://developer.parrot.com/docs/bebop/ 404s

Looks like it's back up for me now? I did think I opened it last week and it was broken, so I guess they must have fixed it. Feel free to reopen if it still 404s for you.