CLI and Web endpoint produce different swagger.json outputs when routing convention is configured through an IApplicationModelConvention is delayed

I'm a maintainer of the JsonApiDotNetCore library (JADNC) and working on an OpenAPI Specification integration using Swashbuckle.
In JADNC we configure the routing through the use of an IApplicationModelConvention. We register our routing convention not in the ConfigureServices(IServiceCollection) method but in the Configure(IApplicationBuilder) method of the Startup. Here is the gist of that:
public class Startup
{
    private static Action<MvcOptions> _postMvcOptionsConfiguration;
    public void ConfigureServices(IServiceCollection services)
    {
        /* other code ommitted for brevity */
        services.AddMvcCore().AddMvcOptions(options =>
        {
            _postMvcOptionsConfiguration?.Invoke(options);
        }).AddApiExplorer();
        services.AddSwaggerGen(options => { /* ... */ })
        /* other code ommitted for brevity */

    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {

        /* other code ommitted for brevity */

        // Here we can retrieve any service and register it as a convention which is what we do in JADNC, but keeping the example simple here
        // var routingConvention = app.ApplicationServices.GetService<CustomRoutingConvention>();
        _postMvcOptionsConfiguration = options => options.Conventions.Insert(0, new CustomRoutingConvention());

        app.UseSwagger(options =>
        {
            options.RouteTemplate = "docs/{documentName}/openapi.json";
        });
        /* other code ommitted for brevity */
    }
}

This works fine when accessing the OAS document on the docs/{documentName}/openapi.json endpoint. However, when generating the OAS document using dotnet swagger tofile I noticed the Configure(IApplicationBuilder) isn't being called, and as a result the desired routes don't make it to the document.
Eg the following difference will arise (snippet):
  "paths": {
    "/api/v1/articles/{id}": {
      "get": {
        "tags": [
          "Articles"
        ],
        "responses": {
          "200": {
            "description": "Success"
          }
        }
      }

From CLI:
  "paths": {
    "/{id}": {
      "get": {
        "tags": [
          "Articles"
        ],
        "responses": {
          "200": {
            "description": "Success"
          }
        }
      }

I've added a minimal repro case
For JADNC it's undesired to move the registration of the routing convention to ConfigureServices(IServiceCollection) because it depends on services that are registered in the DI container. See JsonApiRoutingConvention.

You can tell the CLI tool to use an identical host setup as your application with the following approach:
https://github.com/domaindrivendev/Swashbuckle.AspNetCore#use-the-cli-tool-with-a-custom-host-configuration

You can tell the CLI tool to use an identical host setup as your application with the following approach:
https://github.com/domaindrivendev/Swashbuckle.AspNetCore#use-the-cli-tool-with-a-custom-host-configuration

Thanks for the quick response.
I tried that and ensured the SwaggerHostFactory is actually called by the CLI, but Startup.Configure(IApplicationBuilder ) still isn't called. I've updated the repo to demonstrate this.

Thanks for the quick response.
I tried that and ensured the SwaggerHostFactory is actually called by the CLI, but Startup.Configure(IApplicationBuilder ) still isn't called. I've updated the repo to demonstrate this.

My bad - you're absolutely correct. With it's current implementation, the CLI tool builds an IHost or IWebHost and corresponding service container, and then uses that to retrieve an instance of ISwaggerProvider, but falls short of actually spinning up the application behind the scenes, and as a result (and something I'm only learning now) does not invoke the startup Configure method. I need to dig deeper to find an alternative approach that works - I'd prefer not have to actually spin up the app behind the scenes but it's starting to look like that may be neccessary in some shape or form.

My bad - you're absolutely correct. With it's current implementation, the CLI tool builds an IHost or IWebHost and corresponding service container, and then uses that to retrieve an instance of ISwaggerProvider, but falls short of actually spinning up the application behind the scenes, and as a result (and something I'm only learning now) does not invoke the startup Configure method. I need to dig deeper to find an alternative approach that works - I'd prefer not have to actually spin up the app behind the scenes but it's starting to look like that may be neccessary in some shape or form.

Thanks for the elaboration. I'm looking into a workaround along the direction of executing the bit of code earlier that normally runs in Startup.Configure. Ideally I'd do this only when the code is being called by Swashbuckle CLI. Is there some environment variable being set that allows me to detect this?

Thanks for the elaboration. I'm looking into a workaround along the direction of executing the bit of code earlier that normally runs in Startup.Configure. Ideally I'd do this only when the code is being called by Swashbuckle CLI. Is there some environment variable being set that allows me to detect this?

Is there any traction on this?  I have other customizations that I need to include in the CLI-generated document.
Basically, this needs to be invoked:
app.UseSwagger(x =>
{
    x.PreSerializeFilters.Add(SwaggerDocCustomizer.ModifySwaggerDoc);
});

Apparently this doesn't work in dotnet 6 minimal hosting configuration either:
Program.cs
..
var app = builder.Build();
app
    ...
    .UseEndpoints(b => {
        b.MapVersionedODataRoute("odata", "api/odata", modelBuilder).Select().Count();
        //every /odata/ route and schema is excluded from the generated swagger.json although these lines are run
    });

@domaindrivendev Just tuning in on this discussion to ensure that we understand the implications of any changes that may or may not be underway.
We are actually really happy that the CLI does not invoke the startup Configure method. In our setup, we have an appsettings.json file that does not in itself allow for the CLI to build the service collection due to missing configuration. We thus use appsettings.Development.json to run the Swashbuckle CLI, by setting ASPNETCORE_ENVIRONMENT=Development. However, this mode also causes our startup Configure method to seed our databases with development related data, for ease of use. We wouldn't want that to happen when building the swagger.json.
In this case, I guess we would need to maintain two sets of app settings files, such as appsettings.Development.json and appsettings.Swagger.json, or would it be possible to use a cleverer approach (similarly to what @maurei is asking for)?

Entity Framework faced the same problem for running migrations. How they solved that is to start your app, but first subscribe to runtime-level events. The event handler terminates the app after services have been registered and the needed info has become available. It's all described at https://andrewlock.net/exploring-dotnet-6-part-5-supporting-ef-core-tools-with-webapplicationbuilder/.


Add "any-file-contents" rule to check if at least one file in a given list has content specified by regex

Motivation
The file-contents rule passes if all the files discovered include the content specified via a regular expression. However, for our use case we need to check whether the content is included in at least one of the discovered files.
Proposed Changes
Add any-file-contents rule, which mimics the functionality of the file-contents rule. However, the any-file-contents rule features a globsAny option, instead of globsAll.
To reduce code duplication, the fileContents method (which implements the logic of the file-contents rule) is extended with an optional boolean parameter (called any). If this parameter has the value true, then the rule passes if and only if there is at least one file with content matching the regex. Otherwise, the rule passes if and only if all files include content matching the regex.
Test Plan

it returns passes if requested file contents exists in exactly one file
it returns passes if requested file contents exists in two files
it returns fails if the requested file contents does not exist in any file
it returns failure if no file exists with failure flag enabled
it should handle broken symlinks

Seems like some tests are failing @drgs https://github.com/philips-forks/repolinter/runs/8064809542?check_suite_focus=true#step:13:374


PARQUET-879: Use same configure command for Thrift

I'm trying this locally (CMake 2.8.x) due to the failure in #244


Move code from edx-platform/openedx/core/apidoc.py to edx-drf-extensions

Some bits may stay in edx-platform if they are truly specific to edx-platform.

(In progress)

See https://github.com/edx/api-doc-tools/pull/13

Now that #13 has merged, this issue is complete.
Next, in #11, we will cut a release and install it into https://github.com/edx/edx-platform.

Now that #13 has merged, this issue is complete.
Next, in #11, we will cut a release and install it into https://github.com/edx/edx-platform.


Connection lost on preview

Hi!
It's a promising tool, I could connect my devices, but when I trying to click on the Start Preview button, the connection lost.
I could use saki4510t's plugin, but it doesn't support Explosure settings. Unfortunatly the two plugin is incompatible for build.

Hello @TomXD,
It is been a while since I had time to work on this project and it caught my attention that the best is to public archive it.
About your issue, it is common that with new Android versions, fixes need to be made in order for the plugin to work.
Please feel free to fork the project and continue improving it.


[SPARK-22914][DEPLOY] Register history.ui.port

What changes were proposed in this pull request?
Register spark.history.ui.port as a known spark conf to be used in substitution expressions even if it's not set explicitly.
How was this patch tested?
Added unit test to demonstrate the issue
Please review http://spark.apache.org/contributing.html before opening a pull request.

Can one of the admins verify this patch?

CC @vanzin


Fix access to unpublished objects via API

I want to merge this change because it fixes access to unpublished objects via API. When user send query to Product or Collection node, server checks user's permissions with PublishedQuerySet.visible_to_user method. When user send query to ProductVariant node, server checks visibility of variant's related product.
In addition, that PR fixes similar access leak in graphql.product.resolvers.resolve_product_variants method.
Please note that permission mechanizm requires user object from info.context. It is always provided by GraphQL engine when you executing query from web. However, when you use schema.execute, please provide context_value to ensure proper working of permission mechanizm.

Fixes #3522 .
Screenshots

Pull Request Checklist

[ ] Privileged views and APIs are guarded by proper permission checks.
[ ] All visible strings are translated with proper context.
[ ] All data-formatting is locale-aware (dates, numbers, and so on).
[ ] Database queries are optimized and the number of queries is constant.
[ ] Database migration files are up to date.
[ ] The changes are tested.
[ ] The code is documented (docstrings, project documentation).
[ ] GraphQL schema and type definitions are up to date.
[ ] Changes are mentioned in the changelog.

@Kwaidan00 Please resolve conflicting files.


automate coverity scan uploads

It looks like coverity scan might be gone. We will need to replace it with some other static analysis tool. Need to investigate which one.

Clang Static Analyzer was pretty decent when I was using it years ago.

sigh, I'll see if I can find something else, codeclimate is beautiful but doesn't support C/C++

Another one I found on Twitter: http://fbinfer.com/
Not sure how easy any of these will be to integrate with Bazel.

Looks that Coverity Scan is back.

@PiotrSikora yeah I saw. I'm going to change this title to track automating coverit uploads. We need to figure out how to do that. Potentially we can have a CI job that just runs on master commit.

LGTM reached out and apparently supports C/C++ in beta and willing to help out: https://lgtm.com/
I may play with it over the next week or so to see how it works, however, all these tools aren't perfect by any means.

Hi guys. I tried adding Envoy to our C/C++ beta, but it's not entirely clear what the best way to build your code is. It'd be great if you could give me some pointers — happy to help! Here's an example on how we set up rsyslog for lgtm analysis: https://github.com/rsyslog/rsyslog/pull/2548.

I think we can just do https://github.com/envoyproxy/envoy/tree/master/ci#coverity-scan-build-flow on master commits. Will move this into 1.7.0 milestone. If anyone in the community wants to help with this, that would be great.

I'm going to close this as I think we are going to invest in LGTM/semmle.


Unable to load libcurdart.so.10.1, libcuda.so.1

System information

OS Platform and Distribution:  Amazon Linux 2 AMI on tc2 micro EC2 instance
TensorFlow installed from (source or binary):  binary
TensorFlow version: 2.3.1
Python version: 3.7.9
Installed using virtualenv? pip? conda?: pip
CUDA/cuDNN version: Unknown/does not exist
GPU model and memory: None

Describe the problem
When attempting to run Tensorflow on an EC2 instance, the following error occurs:
$ python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))" 2020-12-02 23:53:45.322002: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2020-12-02 23:53:45.322135: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2020-12-02 23:53:46.627216: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-12-02 23:53:46.627358: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303) 2020-12-02 23:53:46.627417: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ip-172-31-19-184.us-east-2.compute.internal): /proc/driver/nvidia/version does not exist 2020-12-02 23:53:46.627871: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-12-02 23:53:46.634896: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency:<PHONE_NUMBER> Hz 2020-12-02 23:53:46.635090: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3a194e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-12-02 23:53:46.635120: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version tf.Tensor(-963.18164, shape=(), dtype=float32)
As you can see, the commands still work.  However, there is about a 3-5 second delay before it finally executes while it searches for the CUDA files.
Provide the exact sequence of commands / steps that you executed before running into the problem
Spin up a tc2 micro instance on EC2 using the Amazon Linux 2 AMI
Connect to EC2 instance
sudo yum groupinstall "Development Tools"
sudo yum install python37
curl -O https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py --user
Pip install numpy and pandas as usual
pip install --no-cache-dir tensorflow
Any other info / logs
The instance does not contain a GPU, being a simple tc2 micro CPU instance.
Attempting to fix with sudo yum install libcudart10.1 (as recommended in other issues) returns the following error:
No package libcudart10.1 available. Error: Nothing to do
A GPU is unnecessary.  Acceptable solutions would prevent it from returning the errors by either:
-Preventing it from trying to load libcudart and going straight to CPU execution
-Installing the necessary files so it doesn't return errors before going to CPU execution
If all else fails, the installation still seems to work, albeit with major lag time as tensorflow searches for nonexistent files.  My primary concern is eliminating this lag time, NOT getting a GPU to work.

@intelligentgoldfish,
TensorFlow 2.x packages support both CPU and GPU. You are facing the warnings as TensorFlow is look for the GPU drivers and can safely ignore those warnings.
Alternatively if you don't need GPU support, you can install the TensorFlow CPU only package using the below command
pip3 install tensorflow-cpu

Thanks!

Sounds good.  Thank you!


Bootstrap new account, AWS and Azure

Do we have any guidance here for how to bootstrap the AWS account? There's a repo bootstrap-aws that will create a s3 bucket and dynamodb state table automatically
Originally posted by @amaanibn-nasar1-nhs in https://github.com/nhs-england-tools/repository-template/pull/112#discussion_r1315498112

Consider:

Recommended method of bootstrapping accounts
What should be included in it
Where it all lives and how it's executed
Documentation

Include CCoE in making decision on this piece of work


[edge compute] - Update the EndpointStatusInspect API operation

Update the EndpointStatusInspect API operation to include information about Edge stacks that can be retrieved by the Edge agent.

as mentioned in specs, I will add the GET /api/endpoints/:id/status the key: Stacks which will be an array of the following struct:
{
    EdgeStackID   EdgeStackID
    Version            int
}

This requires when inspecting the endpoint to fetch its edge stacks, which means something like the following:
for each stack, get its edge groups.
for each group:
if group is static, check if this endpoint is part of the group its endpoints array
if group is dynamic:
get the endpoint tags (from its own TagIDs array and from its group)
check if endpoint has at least one of the group tags (OR relation)
check if endpoint has all the tags (AND)
that's a loop, inside a loop, inside a loop. I don't like this, however, I don't like adding another relation (like we did with tags), but that might be the way we need to go.
In that case, adding the endpoints relation to an edge group will reduce it to check if the group has the endpoint id (like in the static case), but it's still O(n^3), and it means to refactor all the places where an endpoint changes its tag, an edge group is changing, an endpoint group is changing...
adding a relation stack -> endpoints just moves this algorithm to another place.

@chiptus yes, we're hitting limitations of the key/value store here and this is where having a relational database would have been helpful.
IMO it would be better to have this pre-computed and store inside another relational resource such as edge_stacks_endpoints. This resource could only be here for internal usage (bolt implementation but no HTTP API). And yes, that requires updating the mappings when an operation might change it.
The algorithm above is already quite heavy, imagine having 15.000 endpoints polling every 60 seconds.

ok, will add a new resource:
EdgeSTacksEndpoints {
   EdgeStackID int
   Endpoints: map[EndpointID]bool
}

it will need to be updated in these cases:

edgestack_create
edgestack_update
edgestack_delete (delete the relation object as well)
edgegroup_update (update the related edge stacks)
edgegroup_delete (remove the group from related stacks and update the related edge stacks)
endpoint_create (assign an endpoint to its matching stacks, based on tags)
endpoint_update (if tags were changed, edge group might change and thus edge_stack should be updated)
endpoint_delete (remove endpoint from the related stack)
endpoint_group_update (if tags were changed, the group endpoints' tags were changed and thus endpoints need to be reassigned)
endpoint_group_delete - same

anything I missed? maybe this should be in another PR?

I believe this can be done inside this PR.

edgegroup_update (update the related edge stacks)
[ ]  ~edgegroup_delete (remove the group from related stacks and update the related edge stacks)~ can't delete an edgegroup when it's used

maybe like in delete, we won't let the user update an edge group when it's being used?

@chiptus ideally updating an Edge group would automatically add edge endpoints to edge stacks and Portainer would directly propagate the stacks to these endpoints.

as the question of this issue is: for an edge endpoint give me the related edge stacks we have decided to go the opposite direction: endpointID -> edgeStacks
EndpointEdgeStacksRelation {
   EndpointID int
   EdgeStacks: map[EdgeStackID]bool
}

it will need to be updated in these cases:

[ ] edgestack_create
[ ] edgestack_update
[ ] edgestack_delete (delete the relation object as well)
[ ] edgegroup_update (update the related edge stacks)
[ ] edgegroup_delete (remove the group from related stacks and update the related edge stacks) can't delete an edgegroup when it's used
[ ] endpoint_create (assign an endpoint to its matching stacks, based on tags)
[ ] endpoint_update (if tags were changed, edge group might change and thus edge_stack should be updated)
[ ] endpoint_delete (remove endpoint from the related stack)
[ ] endpoint_group_update (if tags were changed, the group endpoints' tags were changed and thus endpoints need to be reassigned)
[ ] endpoint_group_delete - same

anything I missed? maybe this should be in another PR?

shared functions:
endpoints.findEdgeStacks(endpoint Endpoint) EdgeStackID[]

based on the tags, groupID, and endpointID - find related edge groups

tags = Union(endpoint.Tags, endpointGroup.Tags)
find edge groups:

static: EdgeGroup.Endpoints.includes(endpoint.ID)
dynamic partial: Intersection(tags, EdgeGroup.Tags) != 0
dynamic full: tags are contained in edgeGroup.Tags

find related edge stacks

edgestacks.findRelatedEndpoints(edgeStack) endpointID[] :

for each group:

find related endpoints:

static: return edgeGroup.Endpoints
dynamic partial:

for each endpoint (in db.Endpoints):
tags = Union(endpoint.Tags, endpointGroup.Tags)
endpoint is related if Intersection(edgeGroup.Tags, tags) != 0

dynamic full:
endpoint is related if tags are contained in edgeGroup.Tags

endpoint_create

create a relation object
find related stacks (using findEdgeStacks)

endpoint_update

update relation object with related stacks

endpoint_delete

delete relation object

edgestack_create
for each related endpoint (using findRelatedEndpoints):
add edgeStack to endpoint relation object
edgestack_update
find old related endpoints (before update) and new (after update).
if oldEndpoint is not related - remove edgeStack from the relation object
for each new endpoint - add edgeStack to relation object
edgestack_delete
find related endpoints (before delete)
remove edgestack from relation object
edgegroup_update

find related endpoints (old - before update and new)
for each endpoint - new and old:
update edge_stacks (like in endpoint_update)

edgegroup_delete

can't delete an edgeGroup when an edgeStack is related to it, so if it's deletable, it means that no endpoint is related to an edge stack through it.

endpoint_group_create
a new endpoint group can be related to a stack through a dynamic edge group (that is related to its endpoints through its tags)

for each endpoint in the group:

update endpoint relation if needed (like in endpoint_update)

endpoint_group_update

if endpoints weren't changed, do nothing
for each endpoint in old and new endpoints:

update endpoint relation

endpoint_group_delete

for each of its endpoints:

update endpoint relation

function tests:
endpointgroup_endpoint_add

create edge endpoint
create endpoint group and edge group with the same tags (don't add the endpoint)
create edge stack
edit the endpoint group and add the endpoint from step 1

Closed via #3764


Implement Docker functionality

@gtklocker would you like to help with this?

@dimkarakostas contact @carolinux to consider Ansible as an alternative


use get_user_model method to allow custom auth user models overrides

Fix for issue #1

Thank for your efforts ;)


Webview -> Ctrl + c stopped to work on Text Area on Webview

VSCode Version: 1.47.2 (17299e413d)
OS Version: Windows 10

Steps to Reproduce:

Install sfdx cli (Salesforce CLI)
Create a Dev Account on Salesforce to get a DevHub (https://developer.salesforce.com/signup)
Authenticate the Dev Hub using sfdx force:auth:web:login -a NAME
Set the Dev Hub as your defaultdevusername using sfdx force:config:set defaultusername=NAME -g
Now Install the Salesforce SOQL Editor extension
Create an sfdx project using the commnad sfdx force:project:create -n PROJECT_NAME
Open this project with VS Code.
Wait a few seconds until the extension is Activated. It activates onCommand or when it Identifies the file called sfdx-project.json
Open the extension with the command SFDX: SOQL Editor
Write anything on the Text Area, then try to copy and paste it.
Verify that the text was not copied.

Obs: When the app is running outside of VSCode, copying the text with Ctrl+c works.

Did this work before you updated to 1.47.2?

Did this work before you updated to 1.47.2?

Yes. Before updating it was the Ctrl+V that was bugged. It was duplicating stuff. Now the Ctrl+c stopped haha.

@mjbvz Is this issue related with the fix #101958

the CodeStream extension is also affected by this. It worked with 1.47.1, but not in 1.47.2. For us, it doesn't seem limited to textareas within the the webview, attempting to copy any text inside the webview fails.

I got the same problem
vscode version: 1.47.2

Does this reproduce in the latest VS Code insiders build?

I confirm that my issue was fixed. I copied and pasted the text inside the Webview with no problem

I have just installed the insiders build and when using the insiders instance of VSCode I am no longer experiencing this issue.
I am currently seeing this issue with 1.47.2 using the RestClient extension.
I am unable to copy small chunks of data from the response body which is shown using a WebView.
The extension has an option to show the response in an untitled document, and when I enable this the copying works again.

@AllanOricil  Did you test on insiders? Which operating system?

@mjbvz I confirm the issue is fixed in the Insiders version. I was able to copy something from the text area and then paste it.

Closing as this is fixed in 1.48 then. Should be out in early august


Indicate Value min/max/step values in the UI

Is your feature request related to a problem? Please describe.
The UI for the Values doesn't show min / max and step restrictiosn. If you set a value outside of the range, the command will report an error.
Describe the solution you'd like
Indicate min / max and step in the UI next to the Value somewhere. Or validate the value client side before allowing it to be sent (but I might want to force an invalid value for some debug purpose?). Or do this for select Values, like wake up interval.
Describe alternatives you've considered
N/A
Additional context
I was setting the wake up interval for a battery device (ZSE41). I usually use 7 days (604,800 seconds), most of my devices all this. I tried with a Zooz and got an error flash in the UI. When I checked the value it was set to 86400 seconds instead.
I checked the node dump and it lists a min and max value, for this device, as well as a step value of 60 seconds. 7 days is out of range (so would non-multiples of 60). It would be preferable to know this prior to setting an invalid value.
    {
      "id": "60-132-0-wakeUpInterval",
      "nodeId": 60,
      "toUpdate": false,
      "commandClass": 132,
      "commandClassName": "Wake Up",
      "endpoint": 0,
      "property": "wakeUpInterval",
      "propertyName": "wakeUpInterval",
      "type": "number",
      "readable": false,
      "writeable": true,
      "label": "wakeUpInterval (property)",
      "default": 43200,
      "stateless": false,
      "commandClassVersion": 2,
      "min": 3600,
      "max": 86400,
      "step": 60,
      "list": false,
      "value": 86400,
      "lastUpdate":<PHONE_NUMBER>980,
      "newValue": 604800
    },

@kpine I was going to implement this but I noticed that some values already report this in the hint. It could be a bit redundant in such cases.
Suggestions? cc @AlCalzone

Those hints you mention exist only for configuration parameters, and they should only exist in cases that cannot be modeled with min/max/step, e.g. in the case here with one value outside the "normal" range.
Respecting the min/max/step where they exist is a good idea in general.

Respecting the min/max/step where they exist is a good idea in general.

I already respect them on UI the problem here is that actually I don't show them. My idea was to add it to the hint but then I saw some hints already have them so I was wondering what's the best choice here

Fixed by https://github.com/zwave-js/zwave-js-ui/commit/1107718f674e222aa1b4efdfe4a397b68c37341d


A cleaner pull request to add $graphlookup

I've done refactoring according to your suggestions, and added support to startWith expressions.

did you take a look at the current code?


about stable block and plastic block

Hi, @yaoyao-liu
why can't I find  stable block and plastic block in your modified_resnet file. Did I miss something.
Thanks.

Hi @David-Xia,
Thanks for your interest in our work.
We forward two different kinds of blocks using the following file:
https://github.com/yaoyao-liu/class-incremental-learning/blob/main/adaptive-aggregation-networks/utils/process_fp.py
If you have further questions, feel free to ask.
Best,
Yaoyao


Migrate to critical or penthouse directly?

Now that penthouse has migrated to Puppeteer, it might be worthwhile to use it (or critical) instead of some of critical's sub-dependencies.
Benefits

Solve #1 for free
Viewport (width x height) comparison
Configurable Viewport

Hesitations

Still entails a large dependency download
Big "overhead" for only viewport calculation

maybe not that important to most people?
(generally) not much excess/waste without viewport

As @nekr suggested, could declare either as an optional|peer dependency & dynamically install at runtime.
To do this, would have to detect if width or height options are defined, check that the package is installed, then run separate code block.

Could either perform & await installation
or tell user to manually install & exit.


Fix installer (SDL2.dll and etc. missing after using installer)

I have already tried reinstalling, should i just use the regular version?

I'll work on fixing this issue, it seems the installer forgets to include the required .DLLs. I recommend just using the normal version for now.

Fixed in the newest version:
https://github.com/sam-astro/Z-Sharp/releases


Multilingual dataset?

Hi team, Is your training dataset multilingual or only in English and Chinese?

See https://github.com/01-ai/Yi/issues/25


Support BigPipe streaming in Varnish VCL template

See:

BigPipe environment requirements
Using BigPipe with Varnish
BigPipe Varnish documentation: Surrogate-Control header value's future evolution?
Documentation from BigPipe module

For Docs, the process to test BigPipe on Drupal VM is currently:

Set drupalvm_webserver: nginx in config.yml. (Apache 2.4's mod_proxy_fcgi can't (AFAICT) support streaming/no buffering).
Add php_output_buffering: "Off" in config.yml.
Add nginx_ppa_use: true in config.yml (I might make this the default for Drupal VM).
Make the following changes inside nginx-vhost.conf.j2 (I'd like to make these toggled based on something like a drupalvm_bigpipe_enable flag, maybe):
1. Set gzip off; in the gzip configuration area.

Another important note: Apache's mod_proxy_fastcgi doesn't support streaming for BigPipe, so testing can only be done against Nginx at this time (unless you want to go back to using crufty old mod_php), and for Nginx, you have to set fastcgi_buffering to off.
This setting would need to be added inside https://github.com/geerlingguy/drupal-vm/blob/master/provisioning/templates/nginx-vhost.conf.j2#L29-L38 — should we default it to on or off though...?

And for Varnish, it seems the only change required would be:
sub vcl_backend_response {
  # Disable buffering only for BigPipe responses
  if (beresp.http.Surrogate-Control ~ "BigPipe/1.0") {
    set beresp.do_stream = true;
    set beresp.ttl = 0s;
  }
}

One issue fixed upstream: https://github.com/geerlingguy/ansible-role-php/issues/99 — you can now set php_output_buffering to Off to disable it.

For testing (quicker than doing everything inside Drupal), I'm trying to just verify that Nginx is passing through the stream correctly... so far it seems to not be doing so.
Put this script (named test.php) inside the drupal folder:
<?php
header('Content-type: text/html; charset=utf-8');
header('X-Accel-Buffering: no');
echo 'Begin ...
';
for ($i = 0; $i < 3; $i++) {
  echo $i . '
';
  flush();
  sleep(1);
}
echo 'End ...
';
?>

Then run it within Drupal VM using PHP directly: php /var/www/drupalvm/drupal/test.php. It should show a progressive output with no buffer.
Then load it via http://drupalvm.dev/test.php. Currently with the configuration I'm testing, I can't get Nginx to flush it immediately; it caches then flushes the buffer at the end.

And I was monitoring the packets being sent from Nginx to PHP-FPM using tcpdump -nn -i any -A -s 0 port 9000; when I made a request, I could see the packets being sent realtime to Nginx, so Nginx is definitely buffering here... grr.

It gets more annoying... somewhere, something is buffering something, and it's getting annoying figuring out where this buffering is happening.
If I run curl localhost/test.php inside the VM, I see the test script output render live (flushing after each chunk, as long as it's 1024 bytes or more—if it's < 1024 bytes, the whole response is chunked and flushed at the end... this is with Nginx or Apache mod_php5/prefork).
If I run curl http://drupalvm.dev/test.php from outside the VM (with or without the default firewall rules in place), then curl doesn't print anything until the entire response is returned from Apache.
So there's some funkiness going on at the Apache layer (the minimum of 1024 bytes), plus some strange behavior with requests that go through the VM from the host OS.
Next up, tested with BigPipe on the Drupal 8.1.0 site, with the results:

Test from inside VM (`curl 'http://drupalvm.dev/' -H 'Cookie: SESSxyz=XYZ'``): works; I can see the bigpipe placeholdering!
Test from outside VM (same exact command): doesn't work (nothing returned until the whole page is returned).

So far, besides the config.yml changes in the OP above, the varnish VCL change is the only other thing required. Tested and working so far.

I'm going to rebuild Drupal VM in VirtualBox and see if the same issue persists. I'm guessing it's something with VMware's network adapter/NAT...

Works great in VirtualBox! VMware Fusion--... I wonder what's causing the #networkingfail

A few notes, as well (just because I want to make sure to remember these points, even though they're more obvious in hindsight):

BigPipe won't make placeholdered content load asynchronously, like you can do via ESI (server side) or HInclude (client side); there is only one PHP process and one connection between client/server. And it's synchronous.
BigPipe doesn't make your page load faster—but it makes the perceived load faster when you have bits of content that take a longish amount of time to render.
BigPipe is not helpful for brochureware sites or sites where there is little or no authenticated traffic (this should be obvious, but I think many people don't realize most websites aren't Facebook :) ).

At this point it's just a documentation issue; I'm going to document the two ways to use BigPipe and put a big fat bold warning about not trying to use it with VMware Fusion... though I want to get to the bottom of why Fusion buffers things and doesn't allow streaming!

What... the...? I just recreated the VM again with VMware, and now everything is working fine. So strange. Looks like it must've been some strange networking issue on my Mac, maybe from VMware, maybe from the network I was on when I brought up that VM originally... maybe the VPN software on the Mac I was using, who knows.
But now I can get BigPipe working with Nginx/Varnish/Apache on Drupal VM on VMware Fusion or VirtualBox, reliably. Time to add docs!

I'm hoping to post two blog posts—one on PHP and HTTP request streaming/buffering in general (to expand a bit on the environment requirements on Drupal.org and PHP streamed responses in general), and the other on BigPipe and Drupal VM.


Add recognition for ESR versions of Firefox

Pretty self-explanatory. This was needed for Talon to recognize the Firefox binary from the stable Debian repository, which is an extended support release version.

@fidgetingbits any reason not to merge this one? I saw you approved


Can't resolve '@mui/material/generateUtilityClass'

Untill now everything worked well. Today I updated packages and I got this error:
./node_modules/@mui/lab/Masonry/masonryClasses.js:1:0
Module not found: Can't resolve '@mui/material/generateUtilityClass'
I'm using NextJS framework.
This is my code:
import Masonry from '@mui/lab/Masonry';

// ...
<Masonry columns={{ xs: 1, sm: 2, md: 4 }} spacing={2}>
    {[...Array(31)].map((item: null, i: number) => (
         <img src={`/galeria/realizacie/${i}.jpg`} alt={String(i)} draggable={false} />
     ))}
</Masonry>
// ...

+1

I think its due to them moving internal functions between packages around that are dependant on each other. check what version of @mui/base you get. all "base", "styles", "material" and systems" I think needs to be part of the same release to guarantee that it will work: https://github.com/mui/material-ui/releases

I think its due to them moving internal functions between packages around that are dependant on each other. check what version of @mui/base you get. all "base", "styles", "material" and systems" I think needs to be part of the same release to guarantee that it will work: https://github.com/mui/material-ui/releases
at least that solved the same error in our environment.

This is my dependency list:
"dependencies": {
"@emotion/cache": "latest",
"@emotion/react": "latest",
"@emotion/server": "latest",
"@emotion/styled": "latest",
"@fontsource/roboto": "^4.5.8",
"@mui/icons-material": "latest",
"@mui/lab": "^5.0.0-alpha.105",
"@mui/material": "latest",
"axios": "^1.1.3",
"next": "latest",
"nodemailer": "^6.8.0",
"nookies": "^2.5.2",
"react": "latest",
"react-dom": "latest",
"react-google-recaptcha-v3": "^1.10.0",
"react-responsive-carousel": "^3.2.23",
"sass": "^1.55.0"
}

Upgrading @mui/material from 5.10.10 to 5.10.11 fixed the problem for me

Upgrading @mui/material from 5.10.10 to 5.10.11 fixed the problem for me

Yes... When I replaced "@mui/material": "latest" with "@mui/material": "^5.10.11", the issue went gone. Thanks for your note.

Upgrading @mui/material to the lastest version works

Upgrading @mui/material from 5.10.10 to 5.10.11 fixed the problem for me

Thank you so much!

how can I update? I dont know how to.

how can I update? I dont know how to.

You can change a version in the package.json file and then use yarn on npm i. You can use npm-check-updates tho.


dep add: avoid long lines in :aliases

Please answer the following questions and leave the below in as part of your PR.

[x] This PR corresponds to an issue with a clear problem statement.

https://github.com/babashka/neil/issues/224

[x] I have updated the CHANGELOG.md file with a description of the addressed issue.

I consider this work to be covered by the changelog entry for https://github.com/babashka/neil/issues/215

[x] I have considered whether I should add more tests covering the code I've changed.

New tests have been added for the new function babashka.neil/value->assoc-in-pairs, and a test for babashka.neil/add-alias-str, covering the case where the :nrepl alias produced very long lines in :extra-deps

Behavior before this PR
Given this deps.edn
{:aliases
 {:dev {}}}

, neil dep add nrepl would produce this deps.edn:
{:aliases
 {:dev {}
  :nrepl {:extra-deps {nrepl/nrepl {:mvn/version "1.1.2"} cider/cider-nrepl {:mvn/version "0.49.0"} refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}}
          :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color" "--middleware" "[cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor]"]}}}

Behavior after this PR
With the same deps.edn, neil dep add nrepl would produce this deps.edn:
{:aliases
 {:dev {}
  :nrepl {:extra-deps {nrepl/nrepl {:mvn/version "1.1.2"}
                       cider/cider-nrepl {:mvn/version "0.49.0"}
                       refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}}
          :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color" "--middleware" "[cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor]"]}}}

I think I'm happy with the code now.


Getting "MISSING_MESSAGE: No messages were configured on the provider" when generating static pages

Description
I'm still using the page folder in next js and when using a component in _app.tsx that renders for every page (like a navbar or footer for instance) and when I use getStaticProps with getStaticPaths (e.g  folderA/[id].tsx) . I'm getting errors in the console such as "MISSING_MESSAGE". It would be harmless as it eventually load and works in dev, but I realized I was getting a lot of those while building my static pages when deploying to prod, and does not work at all in a deployed environment(getting a 500 from Next js and my page does not load at all ). I'm happy to provide more logs and screenshot if need be.
Mandatory reproduction URL (CodeSandbox or GitHub repository)
https://codesandbox.io/p/sandbox/next-intl-bug-template-pages-krm37f
Reproduction description
Steps to reproduce:

Open reproduction
Go to URL test/whatever
See error in console: MISSING_MESSAGE: No messages were configured on the provider

Expected behaviour
No error should popup in console nor during static pages build time.

I can unfortunately not reproduce the bug, navigating to https://krm37f-3000.csb.app/test/whatever works fine for me. I don't think anything was changed in the reproduction.
Please also see other related issues: https://github.com/amannn/next-intl/issues?q=is%3Aissue+MISSING_MESSAGE+is%3Aclosed (esp. https://github.com/amannn/next-intl/issues/87)

Ah my bad, I think I did not pas the right link for the reproduction but it seems to be related to the fallback:true option in the static props. Maybe you could add that to your doc ?


chore: bump serverless-offline from 8.5.0 to 8.7.0

Fixes: https://github.com/ucsf-ckm/zorgbort/security/dependabot/13

Ref: https://github.com/dherault/serverless-offline/pull/1368


Can't seem to run the cell with magic %spark.conf and %md

Hi,
It seems like the magic command %spark.conf is not recognised as cell that can be run.
Any plans to support this?
Also found that the cell with %md is also ignored.

Hi, is the spark.conf a legitimate kernel? Afaik it may refers to a method of spark instead of a kernel that runs code. If the same code works in the original web notebook, would you like to provide a screenshot on the code cell and the expected result?

Hi, I guess it is your cell language setting mismatches with the interpreter in magic command that causes the code not run as expected. I didn't mention in the readme but the magic command does not actually control how vscode identifies the language of the corresponding cell, you will have to also select the language in the bottom right corner of the cell to desired one:  Then try run again with your code.
If this is not the case that you have encountered, is the spark.conf a legitimate imterpreter? Afaik it may refer to a method of spark instead of a kernel that runs code. If you have set the correct language of the cell, didn't solve the problem, and the same code works in the original web notebook, would you like to provide a screenshot on the code cell and the expected output?

For md, i did login to the Zeppelin UI, the markdown cell is rendered correctly.
Is this an expected output when i run a markdown cell?
eg:

For spark.conf:
here is a sample of what i run in the regular zeppelin notebook, this sets some custom configs for the spark kernel. This does not have any output, however, i can set specific spark configs easily.
%spark.conf
spark.hadoop.fs.s3a.access.key {spark.ecs.user}
spark.hadoop.fs.s3a.secret.key {spark.ecs.password}
spark.kubernetes.executor.podNamePrefix g-plots-35p
zeppelin.k8s.interpreter.container.image my-python-interpreter-image
zeppelin.k8s.spark.container.image spark-interpreter image

@flowy0 Let me know if you have more questions, otherwise this issue will be closed in 3 days.

For md, yes it is expected. One of the differences between Zeppelin UI and VSCode cell is that in VSCode, the magic command is treated as part of the markdown context when redering, while Zeppelin UI automatically hides the magic command when rendering.
For spark.conf, could you try setting the language mode of the cell to any language except for markdown? Despite the code being highlighted weird, can it be run anyway now? From my end:  FYI shortcut to run a cell: ctrl+enter.
The code seems like a shell script. I will add support to shell in the language mode in the next version.

After I changed it to run as Python, it seems to be fine and take effect. Thanks so much!


Resort to a separate protocol for failable initializer

It is effectively impossible to conveniently implement ObjectMapper for NSManagedObject subclasses with the initializer being part of the protocol. So I'm repeating the same trick of separating the part of the protocol with the initializer into a protocol of its own.

Thanks for the contribution, but I don't want to modify the framework in this way for the sake of CoreData.

Notice, the exactly same problem will happen for each and every class that must derive from another class and has "delicate" initialisation requirements. It is similarly difficult to adopt OM for Realm objects. It is effectively impossible to  adopt OM unless you have control over the implementation. The proposed changes allow adopting it via an extension.

I do agree that ideally ObjectMapper should be usable in just an extension.
On second pass, I do like your approach to the problem (sorry, I think i reviewed this too quickly the first time). I'm going to put a bit more thought into it and get back to you.

This is really appreciated. :smile:

@andrebraga I made a few updates and fixed the tests on this branch. All that has now been merged into master. I'm sure why Github did not close this automatically...
Thanks for your addition! 👍

Yay, thanks! With the class cluster work my score for PR acceptance is looking great :smile: :smile:
Get the word out to Realm people, I think they'll love this!


Upgrade dependency jest-haste-map>walker to 1.0.8

Version
27.4.5
Steps to reproduce
Getting a security vulnerability  https://github.com/advisories/GHSA-jgrx-mgxx-jf9v
Expected behavior
Should be fixed by upgrading jest-haste-map dependency walker to version 1.0.8
Actual behavior
Getting a security vulnerability  https://github.com/advisories/GHSA-jgrx-mgxx-jf9v
Should be fixed by upgrading jest-haste-map dependency walker to version 1.0.8
Additional context
No response
Environment
.

1.0.8 is in semver range, so you can update in your own project. Feel free to send a PR bumping here if you want, but it's not needed to resolve the warning on your side


Support serde 0.7

Closes #55

Since power comes with responsibility, need a review? :)

@nox That would be great!  Also, do you know when servo plans on moving to serde-0.7?

There is a Webrender workweek ongoing so we are waiting until it finishes to not make their work harder. So next week probably, when we will do a rustup.

Apart from these few remarks, LGTM!

As soon as the lights go green I'll merge!


add caution admonitions to make the warning / tip more easy to catch

hey @marklawlor I missed this one when passing style from parent to child so I think adding the notice above the code will make it more easier to note

Cheers :+1:


[fix] Boot User VoyagerServiceProvider

This fixes user boot at VoyagerServiceProvider, when running php artisan voyager:admin<EMAIL_ADDRESS>It happens when using another table name for users.
It wasn't using the config from voyager.user.namespace.

I'm okay with this if someone else can test it.  @marktopper maybe?

Has anyone had a chance to test this? @marktopper? @akazorg? @abdgad?


[Issue 3473]:Animation support for varying image sizes

** >> Make sure you describe your PR to the README Change Log section! << **
This PR changes (delete as applicable)

Nothing, it's a bug fix

Describe the changes below:
This patch allows the support for an animation to contain varying sizes of images and correctly anchoring them during the animation.

The problem with doing it like this is that it ignores frames that have custom pivot points, such as those exported from Texture Packer where the pivot has been positoned carefully in order to prevent what you're seeing here.

I think it'd be better to have the Animation component call the Texture component so it handles custom pivots. I'll have a look at it shortly.


Pass extra data from client to the server

I need to pass extra data from client to the server view. In plain datatable plugin I used to initialize the datatable passing this option:
'ajax': {
    'url': '/api/endpoint/new',
    'data': function(d) {
        d.name = 'Joe';
}

Now if I use the same syntax but with datatableview I get an error:
datatableview.auto_initialize = false;
datatableview.initialize($('.datatable'), {
'ajax': { // Error
    'url': '/api/endpoint/api',
    'data': function(d) {
      d.name = 'Joe';
    }
},
// 'ajax': '/api/endpoint/new', <-- works fine

The error is:
DataTables warning: table id=DataTables_Table_0 - Ajax error.
For more information about this error, please see http://datatables.net/tn/7

Firebug:
"NetworkError: 404 Not Found - http://<IP_ADDRESS>:8000/api/endpoint/[object%20Object]?
sEcho=1&iColumns=2&sColumns=codice%2Cdescrizione&iDisplayStart=0&iDisplayLength=10&
mDataProp_0=0&sSearch_0=&bRegex_0=false&bSearchable_0=true&bSortable_0=true&
mDataProp_1=1&sSearch_1=&bRegex_1=false&bSearchable_1=true&bSortable_1=true&sSearch=&
bRegex=false&iSortingCols=0&hey=Joe&_=1475762591663"

Please note the part http://<IP_ADDRESS>:8000/api/endpoint/[object%20Object].
I'm using datatable view at this commit: b10292c9be43aae26675d64e4e51bde01b9155cf. What did I do wrong?

Followup. I found that preXhr event can be used to inject data before the Ajax request to the server.
datatableview.initialize($('.datatable')
    .on('preXhr.dt', function(e, settings, data) {
        data.foobar = 'baz';
    });
);

I still wonder if it's a the correct way to do that or not, but works.

Regarding the first comment: for the code at that commit, that name of that parameter is ""sAjaxSource" instead of "ajax", but that also get overriden by what datatableview puts in. In other words, it does not seem possible to override it. See line 106 at https://github.com/pivotal-energy-solutions/django-datatable-view/blob/b10292c9be43aae26675d64e4e51bde01b9155cf/datatableview/static/js/datatableview.js

In the current version, there are two ways you can accomplish this, depending on where it is more useful to hook into the datatableview code.
datatableview.defaultDataTableOptions is an object which is extended on a per-object basis with the select and options object provided to datatableview.initialize(), so you can add a callback like this directly to the default set:
// The 'ajax' entry in this object does not exist yet, but if you supply a partial set of options
// like I do below, it will be extended later when the init code wants to add in the url, etc.
//
// Be aware that you're modifying the default options for all tables that initialize on this page!
// Also, be sure you're doing this after the datatableview.js has been included, but before any
// datatables are initialized (e.g., probably don't want to put this in document.onReady)
datatableview.defaultDataTableOptions.ajax = {
    data: function(d){
        d.name = 'Joe';
        return d;
    }
};

Alternatively, and if you need per-table controls, you can provide a datatableview.finalizeOptions(datatable, options) callback, which is called after datatableview.initialize() is completely done setting up the built-in options from the template-rendered table.  You can inspect the table object to make sure you changing the right one, etc.
$(function(){
    datatableview.finalizeOptions = function(datatable, options){
        // options.ajax is already fully formed in this scenario.
        options.ajax.data = function(d){
            d.name = 'Joe';
        };
        return options;  // Always return the options
    };


Any Inference Speed Record in CPU Mode

I am looking for a semantic segmentation model that could run fast in CPU mode.
I appreciated your post offers a very detailed analysis on the runtime and accuracy for variants of MobileNetv2-UNet.
However, I found that the mobile devices you did experiments on are all embedded with GPU device. I am more concerned with the runtime and accuracy performance on CPU mode (hopefully the model could run at FPS >= 10).
Did you do any similar analysis before? Or are you confident that the model (MobileNetv2-UNet) could give a speed of at least 10FPS with a normal Macbook Air? (8GB memory, i7 CPU)
FYI, I tried your model (image size = 224, 224) in my laptop and found that the inference speed is ~0.3s. Not sure how I could optimize the speed

I used another lighter UNet from https://github.com/TianzhongSong/Person-Segmentation-Keras/blob/master/README.md and could run inferences at over 10 FPS in TensorFlow JS with it in the browser. It was also around 10 FPS on my MacBook Air using just the CPU with the python code calling TensorFlow.

I used another lighter UNet from https://github.com/TianzhongSong/Person-Segmentation-Keras/blob/master/README.md and could run inferences at over 10 FPS in TensorFlow JS with it in the browser. It was also around 10 FPS on my MacBook Air using just the CPU with the python code calling TensorFlow.

Great to hear that! Did you simply use the lighter UNet or combine it with MobileNetv2?
Do you simply and directly run the code in repo and achieve such FPS?

Yes, same speed as the original code.
I only make a simple modification to obtain better masks: https://github.com/TianzhongSong/Person-Segmentation-Keras/issues/4
Laurent

Hi Laurent,
Many thanks for your information. I notice the repo you mentioned doesn't have a pretrained model. Would you mind sharing me your model weights? Thank you!

here's one I trained for 20 epochs.

trained on 20 epochs on his dataset.
persondata_unet.h5.zip

Thanks for your great help and prompt reply, Laurent!
I will give it a try.

trained on 20 epochs on his dataset.
persondata_unet.h5.zip

thanks for your weights file, but it confused me that my trained file which is 38.3MB is much bigger than yours, and when i load your weights file, this error occured-"You are trying to load a weight file containing 23 layers into a model with 21 layers." . really need your help~! thank you!

My layers are defined as is:
def create_unet(self):
s = Input(self.sz)
c1 = Conv2D(8, 3, activation='relu', padding='same') (s)
c1 = Conv2D(8, 3, activation='relu', padding='same') (c1)
p1 = MaxPooling2D() (c1)
c2 = Conv2D(16, 3, activation='relu', padding='same') (p1)
c2 = Conv2D(16, 3, activation='relu', padding='same') (c2)
p2 = MaxPooling2D() (c2)
c3 = Conv2D(32, 3, activation='relu', padding='same') (p2)
c3 = Conv2D(32, 3, activation='relu', padding='same') (c3)
p3 = MaxPooling2D() (c3)
c4 = Conv2D(64, 3, activation='relu', padding='same') (p3)
c4 = Conv2D(64, 3, activation='relu', padding='same') (c4)
p4 = MaxPooling2D() (c4)
c5 = Conv2D(128, 3, activation='relu', padding='same') (p4)
c5 = Conv2D(128, 3, activation='relu', padding='same') (c5)
u6 = Conv2DTranspose(64, 2, strides=(2, 2), padding='same') (c5)
u6 = Concatenate(axis=3)([u6, c4])
c6 = Conv2D(64, 3, activation='relu', padding='same') (u6)
c6 = Conv2D(64, 3, activation='relu', padding='same') (c6)
u7 = Conv2DTranspose(32, 2, strides=(2, 2), padding='same') (c6)
u7 = Concatenate(axis=3)([u7, c3])
c7 = Conv2D(32, 3, activation='relu', padding='same') (u7)
c7 = Conv2D(32, 3, activation='relu', padding='same') (c7)
u8 = Conv2DTranspose(16, 2, strides=(2, 2), padding='same') (c7)
u8 = Concatenate(axis=3)([u8, c2])
c8 = Conv2D(16, 3, activation='relu', padding='same') (u8)
c8 = Conv2D(16, 3, activation='relu', padding='same') (c8)
u9 = Conv2DTranspose(8, 2, strides=(2, 2), padding='same') (c8)
u9 = Concatenate(axis=3)([u9, c1])
c9 = Conv2D(8, 3, activation='relu', padding='same') (u9)
c9 = Conv2D(8, 3, activation='relu', padding='same') (c9)
outputs = Conv2D(1, 1, activation='sigmoid') (c9)
self.model = Model(inputs=[s], outputs=[outputs])
print(self.model.summary())
self.model.compile(optimizer = Adam(), loss = 'binary_crossentropy', metrics
= [mean_iou])
On Fri, Sep 20, 2019 at 6:22 PM Chancetc<EMAIL_ADDRESS>wrote:

trained on 20 epochs on his dataset.
persondata_unet.h5.zip
https://github.com/akirasosa/mobile-semantic-segmentation/files/3601616/persondata_unet.h5.zip
thanks for your weights file, but it confused me that my trained file
which is 38.3MB is much bigger than yours, and when i load your weights
file, this error occured-"You are trying to load a weight file containing
23 layers into a model with 21 layers." . really need your help~! thank you!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/akirasosa/mobile-semantic-segmentation/issues/46?email_source=notifications&email_token=AABEQOO3DNFC2J3XEYZ3G53QKT2EJA5CNFSM4IVSOKL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HGUCY#issuecomment-533621259,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABEQOIF6562NPDHLNI6SITQKT2EJANCNFSM4IVSOKLQ
.

My layers are defined as is: def create_unet(self): s = Input(self.sz) c1 = Conv2D(8, 3, activation='relu', padding='same') (s) c1 = Conv2D(8, 3, activation='relu', padding='same') (c1) p1 = MaxPooling2D() (c1) c2 = Conv2D(16, 3, activation='relu', padding='same') (p1) c2 = Conv2D(16, 3, activation='relu', padding='same') (c2) p2 = MaxPooling2D() (c2) c3 = Conv2D(32, 3, activation='relu', padding='same') (p2) c3 = Conv2D(32, 3, activation='relu', padding='same') (c3) p3 = MaxPooling2D() (c3) c4 = Conv2D(64, 3, activation='relu', padding='same') (p3) c4 = Conv2D(64, 3, activation='relu', padding='same') (c4) p4 = MaxPooling2D() (c4) c5 = Conv2D(128, 3, activation='relu', padding='same') (p4) c5 = Conv2D(128, 3, activation='relu', padding='same') (c5) u6 = Conv2DTranspose(64, 2, strides=(2, 2), padding='same') (c5) u6 = Concatenate(axis=3)([u6, c4]) c6 = Conv2D(64, 3, activation='relu', padding='same') (u6) c6 = Conv2D(64, 3, activation='relu', padding='same') (c6) u7 = Conv2DTranspose(32, 2, strides=(2, 2), padding='same') (c6) u7 = Concatenate(axis=3)([u7, c3]) c7 = Conv2D(32, 3, activation='relu', padding='same') (u7) c7 = Conv2D(32, 3, activation='relu', padding='same') (c7) u8 = Conv2DTranspose(16, 2, strides=(2, 2), padding='same') (c7) u8 = Concatenate(axis=3)([u8, c2]) c8 = Conv2D(16, 3, activation='relu', padding='same') (u8) c8 = Conv2D(16, 3, activation='relu', padding='same') (c8) u9 = Conv2DTranspose(8, 2, strides=(2, 2), padding='same') (c8) u9 = Concatenate(axis=3)([u9, c1]) c9 = Conv2D(8, 3, activation='relu', padding='same') (u9) c9 = Conv2D(8, 3, activation='relu', padding='same') (c9) outputs = Conv2D(1, 1, activation='sigmoid') (c9) self.model = Model(inputs=[s], outputs=[outputs]) print(self.model.summary()) self.model.compile(optimizer = Adam(), loss = 'binary_crossentropy', metrics = [mean_iou])
…
On Fri, Sep 20, 2019 at 6:22 PM Chancetc @.***> wrote: trained on 20 epochs on his dataset. persondata_unet.h5.zip https://github.com/akirasosa/mobile-semantic-segmentation/files/3601616/persondata_unet.h5.zip thanks for your weights file, but it confused me that my trained file which is 38.3MB is much bigger than yours, and when i load your weights file, this error occured-"You are trying to load a weight file containing 23 layers into a model with 21 layers." . really need your help~! thank you! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AABEQOO3DNFC2J3XEYZ3G53QKT2EJA5CNFSM4IVSOKL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HGUCY#issuecomment-533621259>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABEQOIF6562NPDHLNI6SITQKT2EJANCNFSM4IVSOKLQ .

really appreciate your help! i'll have a try, thanks a lot!


feat(frontend): Add regtest to IcSendBtcNetwork

Motivation
The component displaying the network name wasn't taking into account regtest.
Changes

Add a condition to check testnet instead of falling back to testnet if not mainnet.

Tests
Works locally in #2528

@AntonioVentilii-DFINITY @peterpeterparker please review

LGTM, tks!
I think for ETH we have some util similar to this, like isEthereumTestnet, or something. Maybe you can create a similar one at some point? I imagine you will use this comparison quite a lot

I'll create it when I need it then, thanks!


Update repositories.txt

@ArduinoBot

@ArduinoBot

Hello @ollprogram. The automated system here only operates on open pull requests. Since your submission was already accepted and the pull request merged, the bot won't pay any more attention to this pull request.
But this human is happy to provide assistance. Was there something you needed from the bot?

@per1234 Hello, I'm sorry for this. I just have changed my latest release in order to remove the warnings of the arduino bot to have a better library. And I thought it was possible to retry the analyze of my repository by mentionning the bot. I wanted to see if I removed the warnings. But it's not very important. Thanks for your answer.

I thought it was possible to retry the analyze of my repository by mentionning the bot.

It is not possible. The bot here only checks the library at the time of the submission.

I wanted to see if I removed the warnings.

The checks are ran every time the Library Manager system indexes a new release of your library. The results are shown on the dedicated logs page for your library:
https://downloads.arduino.cc/libraries/logs/github.com/ollprogram/SimpleShiftRegisterController/
However, it is better to catch problems before you make a release. You can run the same checks locally on your computer using the Arduino Lint tool:
https://arduino.github.io/arduino-lint/latest/
Better yet, you might consider setting up an automated check in your library repository for compliance with the Library Manager requirements. This could run on every commit and pull request to ensure such violations are caught immediately.
Arduino provides a GitHub Actions action that makes this very easy to do by adding a simple YAML file to the repository:
https://github.com/arduino/arduino-lint-action#readme
This action uses the same Arduino Lint tool as the Library Manager system.
It is in use in all the official Arduino library repositories (example).


[SPARK-36837][BUILD] Upgrade Kafka to 3.1.0

What changes were proposed in this pull request?
This PR aims to upgrade Apache Kafka client library from 2.8.1 to 3.1.0 to support Java 17 officially.

https://issues.apache.org/jira/browse/KAFKA-13273 (Add support for Java 17)

Why are the changes needed?
Apache Kafka 3.1.0 has the following improvements and bug fixes including client side.

https://downloads.apache.org/kafka/3.1.0/RELEASE_NOTES.html
https://downloads.apache.org/kafka/3.0.0/RELEASE_NOTES.html

The following is the notable accumulated breaking changes at Apache Kafka 3.0+

KAFKA-12554: Refactor Log layer

KIP-405: Log layer refactor https://docs.google.com/document/d/1dQJL4MCwqQJSPmZkVmVzshFZKuFy_bCPtubav4wBfHQ/edit#

KAFKA-12945: Remove port, host.name and related configs in 3.0

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48090/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48090/

Test build #143581 has started for PR 34089 at commit d0c9ed4.

Test build #143581 has finished for PR 34089 at commit d0c9ed4.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143581/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48091/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48091/

Test build #143588 has started for PR 34089 at commit 7a581fc.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48099/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48099/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48099/

Test build #143588 has finished for PR 34089 at commit 7a581fc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Personally I'm in favor of holding on upgrade for major version till a couple of bugfix versions based on the major version are released. There is around 6 months for Spark 3.3.0 to be released, and we can let early-adopters to experiment with Kafka 3.0.0 (even 3.0.x) clients in the meanwhile.

Yep, I'm also considering to wait for Kafka 3.0.1 due to KAFKA-13322, @HeartSaVioR .
(Thanks, @dongjinleekr )

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49257/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49257/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49257/

This PR is a draft because we are waiting for Apache Kafka 3.0.1 release.

This PR is ready for review. Could you review this please, @viirya, @HeartSaVioR , @HyukjinKwon ?

Also, cc @LuciferYang for Java 17.

Thank you, @viirya . Sure, I'll keep this PR for a while .

Thank you, @LuciferYang .

Merged to master for Apache Spark 3.3.

Sorry for the post-review. I'm not a PMC member so I have no idea about the discussion history in private@ (apologize if there were discussions and decisions about this), but have we discussed thoughtfully about dependency upgrades?
IMHO, when we deal with major upgrade, I'd like to see thoughtful experiments on production usage (at least stage) for while before getting into it. Kafka 3.1 was released just 2 weeks ago while Kafka 3.0 was released around 4 months ago (which is still a bit short term to stabilize), so I'd feel safer on 3.0.x.
(If there was no release on 3.0.x, I'm not sure what happens. We may want to see why they didn't go through bugfix versions and just go through new minor version. Did they figure out any critical/blocker issues about Kafka 3.0 after release? If then isn't it normal to have a bugfix version instead?)
One question: Have anyone apply Kafka 3.1 (not about cluster but about client) in stage/production and track the stability of the streaming queries? We are jumping from Kafka 2.8.0 to 3.1.0, one major and another minor.

@HeartSaVioR thank you for bring this up for discussion!  I also have similar concerns since this is a major version upgrade for the kafka client.

Has anyone run a spark streaming job using the Kafka 3.1 client for any meaningful amount of time and load?
Is there a reason to update 3.1 in a hurry?  Is there something in the 2.8.x line of clients that is deficient?  Not against updating but just wanted to know if there was any special reasons.

Thank you for your interest, @HeartSaVioR and @jerrypeng .

Apache Spark main code is using Apache Kafka client library only. You don't need to link this PR to Kafka 3.1 server installation.

Do you guys think new Apache Kafka Client 3.1 is unstable than Apache Kafka Client 2.8 already?

This PR aims to promote Apache Kafka Client 3.1 testing in Apache Spark 3.3 timeframe (next two months).

https://spark.apache.org/versioning-policy.html
March 15th 2022 | Code freeze. Release branch cut.
Late March 2022 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.
April 2022 | Release candidates (RC), voting, etc. until final release passes

As you asked, we are testing now. So, could you participate the testing together for the community instead of asking delaying or reverting without any evidence?

It sounds to me as the phases are switched.
If we were to test the dependency against the stage/production to make sure we don't break anything, testing on stage/production should be performed "before" we push to master branch. It is totally fine if the PR is submitted and stayed for a while on testing phase (it would be nice if we share the progress), but it seems odd to me if we push the change first and say "we will test the change".
Let me quote my comment again:

Personally I'm in favor of holding on upgrade for major version till a couple of bugfix versions based on the major version are released. There is around 6 months for Spark 3.3.0 to be released, and we can let early-adopters to experiment with Kafka 3.0.0 (even 3.0.x) clients in the meanwhile.

I didn't point out any ticket there. It wasn't about KAFKA-13322. I raised general concern about upgrading major release.
Everyone tend to be very cautious about the dependency version upgrade (bugfix versions tend to be OK). Flink uses Kafka 2.8.1 in master branch, which was upgraded from 2.4.1 in Nov 2021 (while Kafka 3.0 was already available) due to Scala 2.12. The latest version of Kafka client listed in Beam is 2.5.1 if I understand correctly. Confluent doesn't even announce Confluent platform backed by Kafka 3.1.
If we strongly believe we should take a step on Kafka 3.0 earlier than any others, let's build a strong rationalization on main purpose to upgrade, and ask for Kafka community to release 3.0.1. Otherwise, 2.8.1 seems to be the thing we want to go (it's already 2.8.1.)

Hi, Apache Kafka community, @ijuma  @dajac , @hachikuji , @cmccabe , @mjsax
@HeartSaVioR (One of Apache Spark committer) and @jerrypeng seems to have a serious concern about the Apache Kafka community blessed 3.1.0. Here they are currently suggesting to skip Apache Kafka 3.1.0 again. As you see in this PR's history, Apache Spark community already skipped Apache Kafka 3.0.0 via @HeartSaVioR 's request.
Please note that the question is only about Kafka Client here.

How unstable is Apache Kafka 3.1.0 Client?
Is it worst than 3.0.1 Client?
When is 3.0.1 ETA because @HeartSaVioR wrote like ask for Kafka community to release 3.0.1.?
Or, do we need to skip all Apache Kafka 3 client and stick to use 2.8.x in the whole year 2022 (Apache Spark 3.3.0)?

Thank you, @ijuma . Apache Spark schedule is planned like the following.

2022-03-15: Code freeze. Release branch cut.
2022-03-31: QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.
April 2022: Release candidates (RC), voting, etc. until final release passes

During the QA period, some reverting activities including Apache Kafka 3.1.0 might happen at branch-3.3.

In general, we are expecting several RCs and Apache Spark 3.3.0 is expected around DATA+AI Summit 2022 which is on JUNE 27-30, 2022.

Sounds good. Given your timeline, I think 3.1.x makes sense. We should be able to release 3.1.1 in early March.

I don't understand at all. My concern is general one, not to claim and blame some project/version.
The problem we have now is, when we upgrade to Kafka by 2.8.1 to 3.x and found blocker issue(s) (anything, e.g. perf regression) in Kafka client later, we have no way to replace Kafka artifact in runtime to mitigate. We will have to just wait for Kafka community to fix the issue and release a new version. Wouldn't it be problematic already? Will we downgrade and release bugfix version immediately just due to the fact our dependency has serious issue?
When someone upgrades the version, they are exposed on the benefits on new versions but also exposed on the risks on new bugs. The new minor version may contain some risks, the new major version may contain more if the project respects semver. We seem to take the risks but the risks are technically propagated to end users in their production. That said, we should build enough rationalization on upgrading dependency. I concern less on minor version upgrade but major version upgrade is completely different story.

What are the benefits on upgrading Kafka to 3.1 (or even 3.0) from 2.8.1 in terms of Spark's view?
What are the risks on not upgrading Kafka to 3.1 (or even 3.0)?

For example, upgrading Kafka to 2.5 was very reasonable because Spark dealt with hanging issue on offset fetch via new feature of AdminClient. The benefits clearly overweighted the risks.
Kafka's amazing guarantee, being able to talk with old client and new broker (and vice versa) is not only helping us to consider Kafka minor version upgrade on client as safe one. It also works for opposite way. We don't need to rush on upgrading Kafka client unless we figure out the pains on older client communicates with newer broker.
We are going to release a new "minor" version of Spark, not major version, which means end users will expect certain amount of risks on upgrading but not too much like upgrading major version. We have to take control the overall risks on upgrading Spark from 3.2 to 3.3.
@ijuma
I see lots of bug tickets having affected versions as 2.x but having resolved versions as 3.0.x and 3.1.0. Does it mean we no longer be able to expect new bugfixes in 2.8.x?

@HeartSaVioR Whoever uses Spark can define the kafka-clients version they want to use, right?

I see lots of bug tickets having affected versions as 2.x but having resolved versions as 3.0.x and 3.1.0. Does it mean we no longer be able to expect new bugfixes in 2.8.x?

Yes and that's why you should not stick to older versions.
Kafka bumped the major version mostly due to changes related to KRaft that do not affect clients (which is what Spark uses). The client related changes with compatibility impact are changes in default (acks and idempotence), so people have full control over them via their client configuration.

@HeartSaVioR Whoever uses Spark can define the kafka-clients version they want to use, right?
How? Spark releases a single artifact of kafka data source so the only way to mitigate is injecting it in runtime, and it would require binary compatibility on kafka-clients. Does Kafka guarantee binary compatibility between majors and minors for kafka-clients?

The client related changes with compatibility impact are changes in default (acks and idempotence), so people have full control over them via their client configuration.

Could you please elaborating more on this? Does this mean Kafka 3.0 has different default values on functionality related configurations? If then it sounds like end users using Kafka 3.0 should understand what is going on before moving on, and the end users of Spark will have to do this in technically "minor version upgrade".

One mitigated way for this is having different artifacts for Kafka 2.x and 3.x for a while in Spark. This sounds to me the only way for giving end users the control of selecting Kafka version.

Hi, @HeartSaVioR . In the worst case, we can revert it from branch-3.3 if there occurs an issue. I'm wondering why you want us to skip the community-wide Apache Kafka 3.1 testing. Without this patch, do you want all the community downstream develop their own patch for Apache Kafka 3.1 and use it?

How? Spark releases a single artifact of kafka data source so the only way to mitigate is injecting it in runtime, and it would require binary compatibility on kafka-clients.

Right, so if you build your app by including Spark via a Maven/Gradle dependency, you can specify a kafka-clients dependency on your project to override the version used. Is there a reason why this is not possible with Spark?

Does Kafka guarantee binary compatibility between majors and minors for kafka-clients?

Not always, but the question is whether there is binary compatibility for the APIs that Spark uses. I suspect the answer is yes for 3.0, but it would be good to verify (as I am not deeply familiar with Spark's code). This could be done by compiling Spark with Apache Kafka 3.1.0 (as it after this PR) and then using it with kafka-clients 2.8.x at runtime.

Could you please elaborating more on this? Does this mean Kafka 3.0 has different default values on functionality related configurations? If then it sounds like end users using Kafka 3.0 should understand what is going on before moving on, and the end users of Spark will have to do this in technically "minor version upgrade".

Yes. @dongjoon-hyun Does Spark set any Apache Kafka configs itself (things like acks, timeouts, etc.) or does it leave it to the user to do that?

Right, so if you build your app by including Spark via a Maven/Gradle dependency, you can specify a kafka-clients dependency on your project to override the version used. Is there a reason why this is not possible with Spark?

It is possible, but that is not something we will test hard before releasing. Choosing the default must be done like we ourselves have a production directly affected by this, because that is what end users are facing. If we can do opposite, like compiling Spark with Kafka 2.8.1 and using it with kafka-clients 3.1.x at runtime, I would feel safer. For sure, I wouldn't recommend unless they really need it, and say "it's your own risk".

Could you please elaborating more on this? Does this mean Kafka 3.0 has different default values on functionality related configurations? If then it sounds like end users using Kafka 3.0 should understand what is going on before moving on, and the end users of Spark will have to do this in technically "minor version upgrade".

Yes.

OK this backs up my concern. This is something we totally missed. I don't see any effort in this PR about thinking hard on the possible impacts we will get.
That is why I concern more on major releases; everyone struggling with semver knows that bumping major version is one of chance in years to make breaking changes, so we break minors to majors in major release. Spark community is not aiming to release the major version, so end users tend to not expect any breaking changes. And we technically do.

No, @ijuma . In the production, they prefer to use their own proved set in general.

Yes. @dongjoon-hyun Does Spark set any Apache Kafka configs itself (things like acks, timeouts, etc.) or does it leave it to the user to do that?

To @HeartSaVioR . It would be great if you share the result of your hard thinking instead of underestimate the community PR's contribution. For now, I don't see any hard evidence supporting your concern here.

I don't see any effort in this PR about thinking hard on the possible impacts we will get.

As you suggested here, could you initiate a discussion email on dev mailing list including your suggestions (https://github.com/apache/spark/pull/34089#issuecomment-1034216715), @HeartSaVioR ?

I don't think this is the right process. I shouldn't be someone who needs to prove the possible risks on other's proposal. If someone introduces the change, it's up to someone to explain the benefits and risks, and persuade about the change to the community despite the possible risks. I (as a committer) could ask, and reject the proposal if it doesn't make sense.
What I have been figured out must be done in prior to merge this. Does it make sense?
That said, I can initiate the discussion thread, but if I do, it's on behalf of authors and approvers on this PR. Not a duty of myself.

@ijuma
Easier way for you to understand this situation would be, Zookeeper releases their 4.0 version weeks ago, and Kafka is going to adopt in Kafka 3.2. Would it work for Kafka community with a single PR?

Again, I have been saying we haven't constructed a good process on upgrading dependencies. I am not inclined to blame someone, since the process has been done totally valid way in terms of BYLAWS. I'm blaming the process.

I think this discussion has been a bit confusing because it started as a stability concern and then moved to compatibility. So, let's address them separately.
On the stability front, 3.0.1 and 3.1.1 should both be stable and likely stabler than older releases. We haven't made any significant architectural changes in 3.x and a bunch of bugs have been fixed.
On the compatibility front, there are two potential issues I can think of:

Default configuration changes. This KIP has the details on the potential compatibility issues. Some of them are edge cases, but the issue with old kafka clusters may not be so uncommon. There is an easy way to avoid this issue, but it does require the customer to set a config.

Binary compatibility issues if an application uses both Spark and Kafka clients directly and they use one of the removed methods. This is somewhat unlikely, but including for completeness.

If the expectation for Spark users is that a minor release is a drop-in replacement and no action is expected from users, then I agree that the above poses a problem. The approach when it comes to these things vary from project to project. Since it takes a long time to go from one major to another major, the bar is usually a little lower.
For example, Kafka dropped support for Scala 2.11 and added support for ZK 3.5.7 in Apache Kafka 2.5. We did a KIP for the former (Scala 2.11 upgrade), but did not do a KIP for the latter (ZK 3.5.x upgrade) as our analysis indicated that it was a compatible upgrade given how Kafka uses ZK. Even though it was a .7, it did have a bunch of critical bugs in it still, but they would probably not have been found and fixed if we had delayed adoption.

Thank you for your patience, @ijuma . I'm also still confused here.
To @HeartSaVioR , I'm not sure what are you talking by the following.

Again, I have been saying we haven't constructed a good process on upgrading dependencies (especially "major version").

To be clear,

I've been working on this since September 2021 and have been trying to respect all the community opinions including yours so far.
During that period, I also tested this PR with Apache Kafka 2.8.1 of course and there was no user-facing API breaking change in my perspective. So, I merged this after getting the official approval.
Also, as I mentioned here, there is no harm for us to test Apache Kafka 3.1 and this is not a one-way option because we still can revert in branch-3.3. I'd love to receive more constructive feedbacks like specific bug reports via JIRA, so that we can fix them together and move forward.

For the configuration changes, we can handle it as a documentation-level efforts.

Thanks for the detailed inputs, @ijuma .

If the expectation for Spark users is that a minor release is a drop-in replacement and no action is expected from users, then I agree that the above poses a problem.

Seems like it is not a drop-in replacement which remains my concern on upgrade.
@dongjoon-hyun
We are releasing a new minor version, NOT a new major version, which end users easily expect that upgrading is "drop-in replacement". From what I understand is, Spark community is struggling very hard about NOT bringing the behavioral changes in minor version to respect the semantic versioning. That is why whenever we bring the change we make a safeguard against config and the default value is "keep the existing behavior" in many cases, unless we release a major version or the change is to solve the correctness issue.
When we evaluate bumping a new major version of dependency, we must be fully aware of the breaking changes the bumping will bring to us. The breaking changes on config are one of things we totally missed. We didn't know about this till @ijuma explained to us. This is a huge hole in the process. We are saying we can fix that, but what if I wasn't here and we just let it go?
After this PR we force our users to use Kafka client 3.1 (end users wouldn't know the change before they look into their new dependency tree), without knowing about the details what are the benefits they will gain and what are the possible risks they will also get. This is a non-trivial problem.

For example, Kafka dropped support for Scala 2.11 and added support for ZK 3.5.7 in Apache Kafka 2.5. We did a KIP for the former (Scala 2.11 upgrade), but did not do a KIP for the latter (ZK 3.5.x upgrade) as our analysis indicated that it was a compatible upgrade given how Kafka uses ZK. Even though it was a .7, this ZK version did have a bunch of critical bugs in it still, but they would probably not have been found and fixed if we had delayed adoption.

This is a good example, ZK 3.5.x upgrade didn't go through KIP (in other words, dependency upgrade tends to go through KIP), because the analysis indicated that it was a compatible upgrade given how Kafka uses ZK. Should we do the analysis and share the result before moving on, instead of simply checking with existing tests and say "OK we are good to go"?
The testing in staging/production, should have been done before merging, because doing post-review is NOT a "comfort" action, and PR author also is NOT that comfortable from post-review comments. "Let us do this first and leave the remaining on post-reviewing because we can always revert it", works technically, but everyone is not comfortable with this.

Is it true? I don't think you can expect that with Apache Spark. You need to re-build your apps with Apache Spark 3.3 artifacts.

We are releasing a new minor version, NOT a new major version, which end users easily expect that upgrading is "drop-in replacement".

Yes, users may have to rebuild the apps, but they don't expect the "runtime" issues or "runtime" breaking changes. They simply upgrade the version, and try to build, and if the build succeeds, they will say "OK it's good".

So, if you hit any n Apache Kafka 3.1 on Apache Spark 3.3 (master) branch, could you elaborate specifically?

They don't expect the "runtime" issues or "runtime" breaking changes.

It is not going to be productive if we are going to defend the change. I'm not in favor of post-reviewing just because of this.
The fact is, no one knew about the breaking changes. The analysis was done in very high level, but we soon figured out there are more, thanks for folk in the Kafka community. It is not important whether the breaking changes are minors or not. We had to try to find all things and evaluate the risks before moving on. We would be pretty much confident if we have a requirement on consulting with Kafka community on upgrading versions. Whether we enforce this to minor version upgrade or only major version upgrade is another story.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
This is great in terms of stability, but there is no silver bullet. This is a trade off between possible data loss vs performance.
https://cwiki.apache.org/confluence/display/KAFKA/An+analysis+of+the+impact+of+max.in.flight.requests.per.connection+and+acks+on+Producer+performance
I see the conclusion in the analysis, "We don't understand the behavior of acks=all and acks=1 across different workloads and across the entire latency spectrum. We should leave the default as is.", and the default, has changed.

It seems that you are using the breaking change in more broader way.
When Apache Spark changes the default configuration, we write a migration guide for it. We can add it to our migration guide. In addition, we can override from Spark side like we did for Hadoop with spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version.

I see the conclusion in the analysis, "We don't understand the behavior of acks=all and acks=1 across different workloads and across the entire latency spectrum. We should leave the default as is.", and the default, has changed.

Anything else you want to add?

I'm sorry, but again, it is not on my business.
Does Kafka have migration guide from Kafka 2 vs 3? If they provide it, could you please go through with thoughtful reviews? If they don't have, could we please construct the way how we can ensure we tell everything on breaking changes?

Well, when I read them (3.0/3.1) before, it was about mostly Kafka server part. Please don't assume that I didn't read it.

I'm sorry, but again, it is not on my business because I'm in favor of leaving the version as it is.
Does Kafka have migration guide from Kafka 2 vs 3? If they provide it, could you please go through with thoughtful reviews? If they don't have, could we please construct the way how we can ensure we tell everything on breaking changes?

If you didn't read it, you had better read it instead of ignoring it.
If you are saying that I missed something mistakenly. I can catup if you can give me some evidence.

Again, please don't underestimate or look down the other community member's efforts. We can make a mistake, but we works together to make a progress instead of sitting on the Stone Age.

I don't have enough background on Kafka but here are the summary from what I understood by reading comments here:

This PR was reviewed by multiple committers.
Kafka is one of the main dependency in Structured Streaming so it might need some more extra care and reviews. For example, issues such as KAFKA-13322 might cause a non-trivial impact.

For the former, this PR was reviewed by multiple committers when it was merged so I don't think there is a particular problem.
For the latter, I agree that we might need some more extra care for major dependency changes such as Parquet. In my experience at the Spark community, we put extra care of the major version upgrade when a library has many issues after last few upgrades. Might be good to document and clarify this somewhere.
Since there were a couple of post-reviews and concerns here (https://github.com/apache/spark/pull/34089#issuecomment-1033397906 and https://github.com/apache/spark/pull/34089#issuecomment-1033406639), it might be great to have some feedback from Kafka maintainer(s) about the upgrade, stability and safety.

Thank you, @HyukjinKwon . Yes, we asked here (https://github.com/apache/spark/pull/34089#issuecomment-1034004605) and got some answers (https://github.com/apache/spark/pull/34089#issuecomment-1034027323 and https://github.com/apache/spark/pull/34089#issuecomment-1034093422) from Kafka maintainers.

I think it wouldn't happen to pull Kafka community to ask for the details on migration if I didn't concern it, but it was addressed in any way, so fair enough.
I figured out KIP-679 is zero mentioned in "Notable changes" in Kafka 3.0 which feels me less comfortable to simply rely on Notable changes, but it would be never figured out if we didn't hear this from Kafka community and I didn't look into the details, so I'll consider this as like "I wasn't aware of".
As a general comment, since Kafka 3.0 brings breaking changes I'd document the version changes into SS migration guide (we didn't), linking Notable changes in Kafka doc, and explicitly mention "please contact Kafka community for details of changes". At least it helps us to route end users' concerns to the Kafka community, keep us be transparent with changes of Kafka and leave our stance as just a one of usages of Kafka client. I don't see anyone actively participating on both communities, so for me it seems to be the only valid strategy we can take as of now.
Another general comment is, I'd make sure we guarantee downgrading to Kafka 2.8.1 (in runtime or even with different set of artifacts) and give end users freedom to choose Kafka 2 vs 3. We simply consider like there is no demand for end users to stick with Kafka 2.8.1. If we were trying to provide separate Kafka data source artifacts for Kafka 3, I wouldn't concern at all.
It's up to all others to take my general comment or ignore it. I wouldn't mind.

I figured out KIP-679 is zero mentioned in "Notable changes" in Kafka 3.0 which feels me less comfortable to simply rely on Notable changes

This oversight was fixed recently, the website will be updated soon. Like software, documentation can have bugs on occasion. Generally, it's fine to rely on the notable changes section. I am also happy to provide input directly (as I did here) whenever it's helpful for the Spark community.

Thank you always, @ijuma . As a member of Apache Spark PMC, I've been grateful to you always for your helps in JIRA and PRs.

@ijuma Thanks for the update. Much appreciated on your active feedbacks.
Btw, I see the mentioning is about a bug on default value of idempotent not properly accounted. It doesn't mention about performance implication of changing default value of ack/idempotent. If that is the Kafka community's call to consider it as minor/trivial then I'll rely on that, but I'd wonder the reason the action is taken opposite way with the conclusion of analysis.

Thank you for informing that, @ijuma !


[WIP] Cypress/E2E: Upgrade cypress to 9.1.0 with<EMAIL_ADDRESS>

Summary
Upgrade cypress to 9.1.0 with<EMAIL_ADDRESS>Release Note
NONE

thank you for this @saturninoabril i have been seeing this for a very long time 😉


Is there a minimum amount of time servers must be audited before an Application Control Policy can be switched to Enforce?

Hello,
We are running a pilot to use Azure Adaptive Application Control.
All of our the 'File-type protection mode' options within policies are currently set to audit with no option to switch to enforce.

There is an old issues raised about a comment stating that 'Enforce protection mode is disabled until further notice' - https://github.com/MicrosoftDocs/azure-docs/issues/43146
We can see no guidance as to whether we need to wait or do something specific within the policies to enable enforce.
Is there a process to follow? Is it disabled? Does it require a minimum audit period?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8d5df7a3-8abc-026a-7bee-2861ba4a2e38
Version Independent ID: 9c91b1ee-7874-a992-4e52-c621e914da08
Content: Adaptive application controls in Azure Security Center
Content Source: articles/security-center/security-center-adaptive-application.md
Service: security-center
GitHub Login: @memildin
Microsoft Alias: memildin

UserVoice for the same thing from 2019 - https://feedback.azure.com/forums/347535-azure-security-center/suggestions/37338196-enable-enforcement-on-adaptive-application-contr

@mtwelve Thanks for reaching out. We will investigate and update the thread accordingly.

@mtwelve This will be coming back next year. No Timeline as of now.


Vulnerability CVE-2018-1002105 in Spring Cloud Kubernetes Core 1.0.0.RELEASE

After upgrading to the Greenwich.RELEASE of the Spring Cloud package, the corresponding BOM (http://central.maven.org/maven2/org/springframework/cloud/spring-cloud-dependencies/Greenwich.RELEASE/spring-cloud-dependencies-Greenwich.RELEASE.pom) is bringing v1.0.0.RELEASE of spring-cloud-kubernetes-core.
Upgrading has triggered the dependency vulnerability plugin we are using to reveal that this version of spring-cloud-kubernetes-core is vulnerable to CVE-2018-1002105 (https://www.twistlock.com/2018/12/04/cve-2018-1002105-critical-k8s-vulnerability/).
I've scoured the repos and noticed there's a v1.0.1.RELEASE. Does it contain a fix? If not, is there a fix in plan? At the moment, I'm trying to find my way around the BOM to change the version, but it's still imposing its version over what I'm setting. I'm using Gradle and I've tried both exclusion of dependency and adding it separately with the new version and trying to overwrite the rule, but nothing worked. Could someone also please advise over how I can make it use a specified version instead of the one in the BOM?

I don't see why Spring Cloud Kubernetes would be vulnerable to a Kubernetes CVE. @spencergibb @ryanjbaxter do you have any info perhaps?

It was a very odd thing to encounter, since it seemed a server-side issue from the vulnerability description on NIST, but this the output of the OWASP plugin run:
spring-cloud-kubernetes-core-1.0.0.RELEASE.jar: ids:(org.springframework.cloud:spring-cloud-kubernetes-core:1.0.0.RELEASE, cpe:/a:kubernetes:kubernetes:1.0.0) : CVE-2015-7528, CVE-2016-1905, CVE-2016-7075, CVE-2018-1002105
CVE-2018-1002105 has a CVSS score of 7 (High vulnerability).
Just a wild guess here: perhaps this version of the API is somehow linked to a vulnerable Kubernetes version:

Kubernetes v1.10.0-1.10.10 (fixed in v1.10.11)
Kubernetes v1.11.0-1.11.4 (fixed in v1.11.5)
Kubernetes v1.12.0-1.12.2 (fixed in v1.12.3)

?

@iocanel do you perhaps have any idea? I don't see why a server side Kubernetes API issue would affect clients like us...

As the author of the kubernetes-client I don't see how that affects either the client itself or spring-cloud-kubernetes.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.