[FEATURE]: Add support for logs and subroute modules

Should this plugin support logs and subroute modules ?
My configuration like this:
  "apps":{
    "layer4":{"servers":{"stream":{
      "listen":[":443"],
      "logs":{"default_logger_name":"log0"},
      "routes":[
        { "terminal":true,
          "match":[{"tls":{"sni":["MyDomain.com"]}}],
          "handle":[{"handler":"subroute","routes":[
          { "handle":[{"handler":"tls"}]},
          { "handle":[{"handler":"proxy","upstreams":[{"dial":["unix//dev/shm/websocket"]}],"proxy_protocol":"v2"}],
            "match":[{"http":[{"header":{"Upgrade":["websocket"]},"path":["/websocket"]}]}]},
          { "handle":[{"handler":"proxy","upstreams":[{"dial":["local:80"]}]}]}]}]}]
    }}},
    "http":{"servers":{
      "srv1":{
        "allow_h2c":true,
        "listen":[":80"],
        "logs":{"default_logger_name":"log1"},

error log:
Mar 27 04:55:27  caddy[15504]: 2022/03/26 20:55:27.015        ERROR        watcher        applying latest config        {"config_file": "/etc/caddy/Caddyfile.json", "error": "loading new config: loading layer4 app module: decoding module config: layer4: json: unknown field \"logs\""}

Mar 27 04:57:50  caddy[15504]: 2022/03/26 20:57:50.015        ERROR        watcher        applying latest config        {"config_file": "/etc/caddy/Caddyfile.json", "error": "loading new config: loading layer4 app module: decoding module config: layer4: json: unknown field \"terminal\""}

Mar 27 04:58:33  caddy[15504]: 2022/03/26 20:58:33.015        ERROR        watcher        applying latest config        {"config_file": "/etc/caddy/Caddyfile.json", "error": "loading new config: loading layer4 app module: provision layer4: server 'srv0': route 0: position 0: loading module 'subroute': unknown module: layer4.handlers.subroute"}

Logs module makes the layer4 plugin output pretty logs by apart logs from program's kernel log .
And will subroute make caddy more secure by filter sni before actions ?

There has already been an issue for supporting subroute, see #20. Maybe I can try to implement this if there's no other volunteer :)

And will subroute make caddy more secure by filter sni before actions ?

No, subrouting doesn't add any security. It is just convenient for reducing repetition in some cases by nesting certain routing logic. You can do SNI routing without subroutes. They are orthogonal features.

And will subroute make caddy more secure by filter sni before actions ?

No, subrouting doesn't add any security. It is just convenient for reducing repetition in some cases by nesting certain routing logic. You can do SNI routing without subroutes. They are orthogonal features.

I mean routes module can only process handles parallelly. So with subroute module, layer4 plugin can filter wrong or empty sni without leak certs, and stream current one to different servers by uri or headers which I couldn't find the way routes can be done.

As for logs, what do you want logged that isn't already logged?

By now, layer4's log is mixed with admin's log, which like this journalctl -u caddy
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.011        INFO        watcher        config file changed; reloading        {"config_file": "/etc/caddy/Caddyfile.json"}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.012        INFO        using provided configuration        {"config_file": "/etc/caddy/Caddyfile.json", "config_adapter": ""}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.017        WARN        admin        admin endpoint disabled
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.019        WARN        http        server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server        {"server_name": "srv1", "http_port": 80}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.020        DEBUG        tls.cache        added certificate to cache        {"subjects": ["Domain.com"], "expiration": "2022/06/02 23:59:59.000", "managed": true, "issuer_key": "acme.zerossl.com-v2-DV90", "hash": "xxxxxxxxxxxxxxx", "cache_size": 1, "cache_capacity": 10000}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.020        DEBUG        layer4        listening        {"address": "tcp/[::]:443"}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.020        DEBUG        http        starting server loop        {"address": "[::]:80", "http3": false, "tls": false}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.021        INFO        tls.cache.maintenance        started background certificate maintenance        {"cache": "0xc000111ab0"}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.024        INFO        tls.cache.maintenance        stopped background certificate maintenance        {"cache": "0xc000110700"}
Mar 27 08:52:17  caddy[15504]: 2022/03/27 00:52:17.024        INFO        autosaved config (load with --resume flag)        {"file": "/var/lib/caddy/.config/caddy/autosave.json"}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.070        DEBUG        tls.handshake        choosing certificate        {"identifier": "Domain.com", "num_choices": 1}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.071        DEBUG        tls.handshake        default certificate selection results        {"identifier": "Domain.com", "subjects": ["Domain.com"], "managed": true, "issuer_key": "acme.zerossl.com-v2-DV90", "hash": "xxxxxxxxxxxxxxxxxxxxx"}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.071        DEBUG        tls.handshake        matched certificate in cache        {"subjects": ["Domain.com"], "managed": true, "expiration": "2022/06/02 23:59:59.000", "hash": "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.515        DEBUG        layer4.handlers.tls        terminated TLS        {"remote": "remote_ip:port", "server_name": "Domain.com"}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.516        DEBUG        layer4.handlers.proxy        dial upstream        {"remote": "remote_ip:port", "upstream": "<IP_ADDRESS>:80"}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.918        DEBUG        http.handlers.reverse_proxy        selected upstream        {"dial": "<IP_ADDRESS>:8005", "total_upstreams": 1}
Mar 27 08:52:37  caddy[15504]: 2022/03/27 00:52:37.927        DEBUG        http.handlers.reverse_proxy        upstream roundtrip        {"upstream": "<IP_ADDRESS>:8005", "duration": 0.008124022, "request": {"remote_ip": "<IP_ADDRESS>", "remote_port": "47644", "proto": "HTTP/2.0", "method": "POST", "host": "MyDomain.com", "uri": "f85c72d0", "headers": {"X-Forwarded-Proto": ["http"], "X-Forwarded-Host": "MyDomain.com", "Content-Type": ["application/grpc"], "User-Agent": ["grpc-go/1.44.1-dev"], "Te": ["trailers"], "X-Forwarded-For": "remote_ip"}}, "headers": {"Content-Type": ["application/grpc"]}, "status": 200}

So if layer4's log can output like configurations below:
  "logging":{"logs":{
    "default":{
      "level":"DEBUG","encoder":{"format":"console","level_format":"upper","time_format":"wall_milli"},
      "exclude":["http.log.access.log0","layer4.log.access.stream0"]},
    "log0":{
      "level":"DEBUG","encoder":{"format":"json","level_format":"upper","time_format":"wall_milli"},
      "writer":{"filename":"/tmp/caddy.log","output":"file"},
      "include":["http.log.access.log0","layer4.log.access.stream0"]}}},
  "apps":{
    "http":{"servers":{
      "srv0":{
        "listen":[":80"],
        "logs":{"default_logger_name":"log0"},
    "layer4":{"servers":{"stream0":{
      "listen":[":443"],
      "logs":{"default_logger_name":"stream0"},

Makes all access actions together from layer4 to https in one file "caddy.log". And keeps caddy's admin log in system journal.
But it seems http.handlers.*(like http.handlers.file_server , http.handlers.reverse_proxy , http.handlers.rewrite and http.log.error.log1 ) still in system journal even I've defined "writer":{"filename":"/tmp/caddy.log","output":"file"},"include":["http.log.access.log0"] ?

include and exclude map directly to the logger name, so if you want to include http.handlers.* in another logger, then put http.handler in that logger.
layer4 doesn't have access logs right now, but you can just move layer4 logs to another logger by including layer4, etc.

include and exclude map directly to the logger name, so if you want to include http.handlers.* in another logger, then put http.handler in that logger's include.
layer4 doesn't have access logs right now, but you can just move layer4 logs to another logger by including layer4, etc.

It works !
Thank you so much for help!
Now I splitted http and layer4's log from tls and admin's log by "include":["http","layer4"] and "exclude":["http","layer4"]

I mean routes module can only process handles parallelly.

Can you explain what you mean by this? Each handler in a route is called sequentially in turn, not in parallel. And each route is evaluated sequentially as well.
(Thanks Francis, for answering questions about logging.)

Can you explain what you mean by this? Each handler in a route is called sequentially in turn, not in parallel. And each route is evaluated sequentially as well.
(Thanks Francis, for answering questions about logging.)

Sorry for my pool expression.
I mean with subroute, I can upstream with both sni & uri, but now, routes module can only choose one matcher which either tls's sni or http's header.
Seems a graph maybe clear:
Route:
|
----->If SNI: domain A------>Then Dial: Server1
|
----->If SNI: domain B------->Then Dial: Server2
|
----->If HTTP Header: URI A------>Then Dial: Server3
|
----->If HTTP Header: URI B------>Then Dial: Server4

parallelly shoud be they have the same priority?
Subrouter:
|
------>If SNI: domain A
                       |
                       -----------IF HTTP Header: URI A------>Then Dial: Server1
                       |
                       -----------IF HTTP Header: URI B------>Then Dial: Server2
|
------>If SNI: domain B
                       |
                       -----------IF HTTP Header: URI A------>Then Dial: Server3
                       |
                       -----------IF HTTP Header: URI B------>Then Dial: Server4

You can have more than one matcher for each route, so you can do "if SNI && if HTTP" in one matcher.
Either way, I'm pretty sure we can close this as a duplicate of https://github.com/mholt/caddy-l4/issues/20 at this point, we know that you're looking for layer4.handlers.subroute which that issue asks for, and your log question has been answered.

You can have more than one matcher for each route, so you can do "if SNI && if HTTP" in one matcher.

May I ask how to write configurations for && in route ? I've searched for days.

Either way, I'm pretty sure we can close this as a duplicate of #20 at this point, we know that you're looking for layer4.handlers.subroute which that issue asks for, and your log question has been answered.

This is an AND:
"match": [
	{
		"sni": [],
		"http": []
	}
]

This is an OR:
"match": [
	{
		"sni": []
	},
	{
		"http": []
	}
]

This is an AND:
"match": [
	{
		"tls": {},
		"http": []
	}
]

This is an OR:
"match": [
	{
		"tls": {}
	},
	{
		"http": []
	}
]

The layer4 match docs on caddyserver.com don't seem to mention this, but the HTTP app does it the same way, and explains:
https://caddyserver.com/docs/json/apps/http/servers/routes/match/

The matcher sets which will be used to qualify this route for a request (essentially the "if" statement of this route). Each matcher set is OR'ed, but matchers within a set are AND'ed together.

You can also read through the code to understand how it's implemented (it's decently commented) https://github.com/mholt/caddy-l4/blob/master/layer4/matchers.go
But I guess in this case you might need a subroute because you'd need to use the tls handler to terminate TLS before you can match http things, I think.

Yes, I've read both
https://caddyserver.com/docs/json/apps/http/servers/routes/match/
https://caddyserver.com/docs/json/apps/http/servers/routes/handle/
and tried configurations like below while only server2 works.
        { "handle":[
            {"handler":"tls"},
            {"handler":"proxy","upstreams":[{"dial":["server1"]}],"proxy_protocol":"v2"}],
          "match":[{
            "tls":{"sni":["DomainA"]},
            "http":[{"path":["URI1"]}]}]},
        { "handle":[
            {"handler":"tls"},
            {"handler":"proxy","upstreams":[{"dial":["server2"]}],"proxy_protocol":"v2"}],
          "match":[
            {"tls":{"sni":["DomainB"]}},
            {"http":[{"path":["URI2"]}]}
          ]
        },

Matcher with DomainA&URI1 does not work, and all DomainB's connections was streamed to server2.
And there's another question: the layer4 module cannot reconizge grpc/http2 connections with similar configuration like below, but there's no upstream record in logs :
        { "handle":[{"handler":"tls"}]},
        { "handle":[
            {"handler":"proxy","upstreams":[{"dial":["<IP_ADDRESS>:8003"]}]}
          ],
          "match":[{"http":[{"header":{"Content-Type":["application/grpc*"]}}]}
          ]
        { "handle":[
            {"handler":"proxy","upstreams":[{"dial":["<IP_ADDRESS>:8003"]}]}
          ],
          "match":[{"http":[{"path":["/grpc"]}]}
          ]
        },

Both configurations does not work.
The only way to upstream gRPC is using layer4 to proxy all other connections (incl gRPC) to caddy server and then use reverse_proxy by match Content-Type or path.
example:
          { "handle":[{
              "handler":"reverse_proxy",
              "flush_interval":-1,
              "upstreams":[{"dial":gRPC Server"}],
              "transport":{"protocol":"http","versions":["h2c"]}}],
            "match":[{"header":{"Content-Type":["application/grpc*"]},"path":["/grpc"]}]},

So what's the difference between caddy server and layer4 in gRPC ?
May I upstream gRPC connections by using layer4 module?

I believe the same question is posted in a new issue #51, and with #52 merged, I'll consider this issue finished.


warning: ‘fmpz_mod_mat_mul’ accessing 40 bytes in a region of size 32

Noticed in the CI that there's something wrong about the mat_mul operations (https://github.com/flintlib/python-flint/actions/runs/10402835971/job/28808112409?pr=182)
For example
 In function ‘__pyx_pf_5flint_5types_12fmpz_mod_mat_12fmpz_mod_mat_36_pow’,
    inlined from ‘__pyx_pw_5flint_5types_12fmpz_mod_mat_12fmpz_mod_mat_37_pow’ at src/flint/types/fmpz_mod_mat.c:11474:13:
src/flint/types/fmpz_mod_mat.c:1330:51: warning: ‘fmpz_mod_mat_mul’ accessing 40 bytes in a region of size 32 [-Wstringop-overflow=]
 1330 |     #define compat_fmpz_mod_mat_mul(C, A, B, ctx) fmpz_mod_mat_mul(C, A, B)
      |                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~
src/flint/types/fmpz_mod_mat.c:11676:7: note: in expansion of macro ‘compat_fmpz_mod_mat_mul’
11676 |       compat_fmpz_mod_mat_mul(__pyx_v_res->val, __pyx_v_res->val, __pyx_v_tmp->val, __pyx_v_self->ctx->val);
      |       ^~~~~~~~~~~~~~~~~~~~~~~
src/flint/types/fmpz_mod_mat.c:1330:51: note: referencing argument 1 of type ‘fmpz_mod_mat_struct[1]’
 1330 |     #define compat_fmpz_mod_mat_mul(C, A, B, ctx) fmpz_mod_mat_mul(C, A, B)
      |                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~
src/flint/types/fmpz_mod_mat.c:11676:7: note: in expansion of macro ‘compat_fmpz_mod_mat_mul’
11676 |       compat_fmpz_mod_mat_mul(__pyx_v_res->val, __pyx_v_res->val, __pyx_v_tmp->val, __pyx_v_self->ctx->val);
      |       ^~~~~~~~~~~~~~~~~~~~~~~
src/flint/types/fmpz_mod_mat.c:1330:51: warning: ‘fmpz_mod_mat_mul’ reading 40 bytes from a region of size 32 [-Wstringop-overread]
 1330 |     #define compat_fmpz_mod_mat_mul(C, A, B, ctx) fmpz_mod_mat_mul(C, A, B)
      |                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~

Here we have:
cdef extern from "flint/fmpz_mod_mat.h":
    ctypedef struct fmpz_mod_mat_struct:
        fmpz_mat_t mat
        fmpz_t mod
    ctypedef fmpz_mod_mat_struct fmpz_mod_mat_t[1]

and in the C source: https://github.com/flintlib/flint/blob/05340d2a7349761aff2cd3370810c93e2174ce46/src/fmpz_mod_types.h#L36
typedef fmpz_mat_struct fmpz_mod_mat_struct;

typedef fmpz_mod_mat_struct fmpz_mod_mat_t[1];

So maybe we want instead
cdef extern from "flint/fmpz_mod_mat.h":
    ctypedef fmpz_mat_struct fmpz_mod_mat_struct:
    ctypedef fmpz_mod_mat_struct fmpz_mod_mat_t[1]

oh i see there's this change in 3.1.0 versus earlier versions of flint -- so maybe that's the issue?

It is probably better to correct the struct definition but I don't know if that would break under older Flint. It might be that we can just make it an opaque struct. I assume we don't need to access the mat or mod members directly anywhere.
I have been seeing warnings like that for a long time coming from different parts of the codebase but I don't see them now when building with latest flint. The CI job you point to is using Flint 3.0.1 though.

We're not able to do two different struct definitions for different flint, right?

We can have different struct definitions but it needs to be done in the C preprocessor just like the other #define:
https://github.com/flintlib/python-flint/blob/5445684db1481b806e8027499fa591c7508f6f03/src/flint/flintlib/fmpz_mod_mat.pxd#L21-L30
From Cython's perspective the struct would be opaque but we could also use the preprocessor to define macros to access the elements if needed.

so could we do:
cdef extern from "flint/fmpz_mod_mat.h":
    ctypedef struct fmpz_mod_mat_struct_new:
        fmpz_mat_t mat
        fmpz_t mod
    ctypedef fmpz_mod_mat_struct_newfmpz_mod_mat_new_t[1]

    ctypedef fmpz_mat_struct fmpz_mod_mat_struct_old:
    ctypedef fmpz_mod_mat_struct fmpz_mod_mat_old_t[1]

Then use #define fmpz_mod_mat_struct_old fmpz_mod_mat_struct etc?

That might work.
I was imagining something more like
cdef extern from *
    """
    #if FLINTVER < 3.1
    # define compat_fmpz_mod_mat_struct ...
    """
    ctypedef compat_fmpz_mod_mat_struct

If we need something to be different at the C level for different Flint versions it needs to be handled by the C preprocessor and it all needs to be opaque from Cython's perspective.


Update postcode content on the mailing list

Trello card
https://trello.com/c/ML8z1D9w/7053
Context
We ran user research on the mailing list sign up, and changed some of the content on the postcode step.
Changes proposed in this pull request
Update the content below the question on the postcode page.
Guidance to review

@emmaachesongray In the designs, we changed the question to **What's** your UK postcode (optional)? - have we moved away from this now?


When to update 5.3

Hi @1157457050,
We have a PR open to compile the plugin for 5.3: #1207. You can download the binaries from CI to test it now, or wait until it is included with the next release.
In the future, please post questions to the community forum. We reserve Github for confirmed bugs and feature requests.


FPT_RVR_EXT.1 Platform Firmware Recovery

Document consequences of selections.
"the secure local update mechanism described in FPT_TUD_EXT.4" invokes FPT_TUD_EXT.4

Done


Startup.Configure() is different in API template

When creating the project from the API template, the Startup.Configure method is as follows:
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseAuthorization();

        app.UseEndpoints(endpoints => {
            endpoints.MapControllers();
        });
    }

Whereas in this tutorial it is:
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
// The default HSTS value is 30 days. You may want to change this for
// production scenarios, see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
        app.UseHttpsRedirection();
        app.UseMvc();
    }

Which one is correct?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 583ae3e7-5915-f22b-e54c-6c100cf2a7da
Version Independent ID: 95f2549e-29f9-e31a-b4d0-432b971643ff
Content: Tutorial: Create a web API with ASP.NET Core
Content Source: aspnetcore/tutorials/first-web-api.md
Product: aspnet-core
Technology: aspnetcore-tutorials
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Select the 3.0 version


[zstd] Fix clang-cl flags

[x] Changes comply with the maintainer guide
[x] SHA512s are updated for each updated download
[x] The "supports" clause reflects platforms that may be fixed by this new version
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

@FrankXie05 Is there a reason this is not getting reviewed ?

Thanks!


Integration Tests: Resolve Test Dependency on Machine Deletion to Prevent Inconsistent Results

Description:
Currently, the integration tests have dependencies where one test relies on the successful completion of another, specifically regarding machine deletion. If the deletion of a machine is not fully completed before the execution of the next test, it can cause inconsistencies, such as in the ListMachine test case.
In the ListMachine test, the current expectation is that the machine count should be 1. However, if the previous test has not finished deleting a machine, the list may include more than one machine, causing the test to fail.
To ensure reliable and independent test execution, this dependency issue needs to be resolved. The tests should be refactored to ensure that each test can run independently without relying on the state changes (such as machine deletion) from the previous tests.
Ref: https://github.com/ironcore-dev/libvirt-provider/actions/runs/10451444191/job/28937878022?pr=370

this issue is solving in refactoring of integration tests. PR is here https://github.com/ironcore-dev/libvirt-provider/pull/345


Function to flatten multi-level columns

Current coverage is 87.04% (diff: 100%)

No coverage report found for master at 880d5ed.

Powered by Codecov. Last update 880d5ed...ae71515


All resources are grouped under / - not by resource.

Instead of showing all the resources under the root / path, let's group those by resourceName. Like the pet store example.

Hours looking why they are not splited and I see this.
Anyone has the solution to get this? Group the resources by resourceName?
Thanks!

Hi @rantav and @mserralta ,
For my current project, i needed this kind of resources hierarchy in the resulted swagger html, so i created a patch on my fork.
We've use this patch for about a month (seems to be stable up to now), i've just push it on my github account to share with you if you need it.
https://github.com/lounagen/flask-restful-swagger/commit/88d3aeb9726fa56630431528f21a53d1afbc54a8
See the attached screenshot, associated to this code sample:
@swagger.resource(
    hierarchy_id='interfaces'
)
class InterfaceListView(Resource):
 [... my flask restful swagger payload for this class...]

@swagger.resource(
    hierarchy_id='interfaces'
)
class InterfaceView(Resource):
 [... my flask restful swagger payload for this class...]

@swagger.resource(
    hierarchy_id='devices'
)
class DeviceListView(Resource):
 [... my flask restful swagger payload for this class...]

@swagger.resource(
    hierarchy_id='devices'
)
class DeviceView(Resource):
 [... my flask restful swagger payload for this class...]

[... and so on with some
![flask_restful_swagger_hierarchy_screenshot](https://cloud.githubusercontent.com/assets/6803621/5782252/e8e58aaa-9db9-11e4-9da1-959b82b606d1.png)
other classes ...]

If someone need it, i can add a 'todo.py' sample integration file with such sample.
Side effect of this pretty swagger hierarchy 'like the petstore website' is that the generated json is now splitted into multiple json (one per resource hierarchy).
To keep backward compatibility in the patch below, i added a new endpoint to print all the resources in a flat json (the actual behaviour, so it should be transparent for existing apps)
Hope that could be usefull!
ps @rantav : great job you've done, i was new to flask and patch your project was a very interesting way to learn flask quickly :-)

This is awesome, thank you!

Yes. Awesome. I forgot to say we are using your fork and it is what we were looking for.
Thanks

We have been using this for a very long time now and it's awesome!
We noticed that the json version is broken (http://localhost:8080/api/spec.json). Any idea how to best fix this? @lounagen @mserralta

I am not sure if I understand what you mean. We have
{swaggerVersion": "1.2", "apiVersion": "1.0"}
apiVersion is defined by yourself in .docs() method.
SwaggerVersion we kept in 1.2 as we are still using this swagger specification. So I think we do not have anything broken. What do you mean by "json version"?

Instead of http://localhost:8080/api/spec.html we are accessing http://localhost:8080/api/spec.json (json instead of html). Instead of the correct sub pages we get the response back I posed above. Is this working for you?

@mserralta Instead of http://localhost:8080/api/spec.html we are accessing http://localhost:8080/api/spec.json (json instead of html). We are not seeing the sub pages listed and only get the response without any meaningful content (see what I posted above). Is this working for you?

@lukas-gitl Oh I see. You're right. In api/spec.json I only get information about my defined models and, instead, the resources are defined in /api/spec/_/resource_list.json where in every listed URL are defined the final endpoint.
At least this is how we work with...

@mserralta Ah/api/spec/_/resource_list.json is great. And then you can just access the resource as listed in there (without json ending). Thank you!


Need help running Camera-based object detection (3d-yolo, SMOKE) using Python

We appreciate you go through Apollo documentations and search previous issues before creating an new one. If neither of the sources helped you with your issues, please report the issue using the following form. Please note missing info can delay the response time.
System information

Google Colab Notebook
Python 3.10
Model files were pulled from Apollo r<IP_ADDRESS> docker image

Steps to reproduce the issue:

Load the models using the respective frameworks (3d-yolo -> PaddlePaddle, SMOKE -> PyTorch)
Load the images from Waymo Open Dataset (Waymo-open-dataset-tf-2-11-0==1.5.1) into tensors
Run inference

Supporting materials (screenshots, command lines, code/script snippets):

3d-yolo

Here I'm loading 3d-yolo's inference model using PaddlePaddle

def initModel(path_to_model: str, model_file_name: str) -> list:
    # Check if paddlepaddle is installed correctly
    paddle.utils.run_check()

    # Set the device to cpu
    paddle.set_device('cpu')

    paddle.enable_static()
    place = paddle.CPUPlace()
    exe = paddle.static.Executor(place)

    [inference_program, feed_target_names, fetch_targets] = paddle.static.load_inference_model(
        path_prefix = path_to_model,
        executor = exe,
        model_filename = model_file_name + '.pdmodel',
        params_filename = model_file_name + '.pdiparams'
    )
    return [inference_program, feed_target_names, fetch_targets, exe]

Here I'm loading the images from Waymo Open Dataset as tensors

def imageByteToTensor(dataframe, yieldInterval = 20):
  imgs = []
  orignal_img = []
  i = 1
  for _, row in dataframe.iterrows():
    if i % yieldInterval == 0:
      yield [tf.convert_to_tensor(imgs), orignal_img]
    image_dict = v2.CameraImageComponent.from_dict(row)
    image_tensor = tf.image.decode_image(image_dict.image, channels=3, dtype=tf.dtypes.float32)
    image_tensor = tf.image.resize(image_tensor, [800, 1440])
    imgs.append(image_tensor / 255)
    orignal_img.append(image_dict.image)
    i += 1

Here I'm running the inference

x = imageByteToTensor(images, 1)
tensor, imgs = next(x)
res = executor.run(program, feed={target_names[0]: tensor.numpy()}, fetch_list=targets)

Here is what the object targets contain which I presume is the format of res. There's currently no documentation on what these tensors mean, and I've been trying to look for documentation on it. Could any of the developers be kind enough to shed some light? Much appreciated!

[var save_infer_model/scale_0 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_1 : LOD_TENSOR.shape(-1, 50, -1, 128).dtype(float32).stop_gradient(False),
var save_infer_model/scale_2 : LOD_TENSOR.shape(-1, 50, 90, 32).dtype(float32).stop_gradient(False),
var save_infer_model/scale_3 : LOD_TENSOR.shape(-1, 50, 90, 48).dtype(float32).stop_gradient(False),
var save_infer_model/scale_4 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_5 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_6 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_7 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_8 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_9 : LOD_TENSOR.shape(-1, 50, 90, 16).dtype(float32).stop_gradient(False),
var save_infer_model/scale_10 : LOD_TENSOR.shape(-1, 50, -1, 4).dtype(float32).stop_gradient(False),
var save_infer_model/scale_11 : LOD_TENSOR.shape(-1, 50, 90, -1, 4).dtype(float32).stop_gradient(False),
var save_infer_model/scale_12 : LOD_TENSOR.shape(-1, 50, 90, 64).dtype(float32).stop_gradient(False),
var save_infer_model/scale_13 : LOD_TENSOR.shape(-1, 50, 90, 64).dtype(float32).stop_gradient(False),
var save_infer_model/scale_14 : LOD_TENSOR.shape(-1, 64, 200, 360).dtype(float32).stop_gradient(False)]

SMOKE

Here I'm loading smoke_libtorch_model.pth using PyTorch

model = torch.jit.load('/content/drive/MyDrive/fyp/models/smoke_libtorch/smoke_libtorch_model.pth')
model.eval()

Problem here is that model.forward() takes in 2 arguments as seen in print(model.code()). So, what should Ks and down_ratios be set to? Much appreciated!

def forward(self,
    input: Tensor,
    argument_2: Tuple[Tensor, Tensor]) -> Tuple[Tensor, Tensor]:
  heads = self.heads
  backbone = self.backbone
  Ks, down_ratios, = argument_2
  _0 = (backbone).forward(input, )
  _1 = (heads).forward(_0, Ks, down_ratios, )
  return (_1, _0)

can i say there are some problems here
3d-yolo -> PaddlePaddle   // it's use tensorRT, but not PaddlePaddle
SMOKE -> PyTorch  // it's pytorch version

and be sure SMOKE need 3 inputs
https://github.com/ApolloAuto/apollo/blob/446c9c6d481fef9853ab9f167a0e934092058244/modules/perception/camera/lib/obstacle/detector/smoke/smoke_obstacle_detector.cc#L96-L101
You can get more details in
https://github.com/ApolloAuto/apollo/blob/446c9c6d481fef9853ab9f167a0e934092058244/modules/perception/camera/lib/obstacle/detector/smoke/smoke_obstacle_detector.cc#L313

Hi @daohu527,
Thank you for pointing it out and please do correct me as I'm a student who is new to all of this!
I have to apologise for the mix up, I used the 3d-r4 model instead of the 3d-yolo model.
Based on apollo_deploy.yml, I understand that 3d-r4-half is using Caffe framework and has 3 outputs (cls_pred, label, confidence). And there are files model and params in the docker image that I'm able to load into PaddlePaddle, and hence the targets object.

As for SMOKE, once again thank you for pointing me to where I should be looking at. After connecting some of the dots, I landed on libtorch_obstacle_detector.cc, should I use the same values?
https://github.com/ApolloAuto/apollo/blob/446c9c6d481fef9853ab9f167a0e934092058244/modules/perception/inference/onnx/libtorch_obstacle_detector.cc#L69-L74
I also noticed that values are pulled from a map
https://github.com/ApolloAuto/apollo/blob/446c9c6d481fef9853ab9f167a0e934092058244/modules/perception/inference/onnx/libtorch_obstacle_detector.cc#L110-L125
Thank you!

I think these will help you
    fake_input = torch.zeros([1, 3, 640, 960]).cuda() # apollo use 640 960
    K = torch.tensor([[[1, 0, 0], [0, 1, 0], [0, 0, 1]]]).float().cuda()
    down_ratio = torch.tensor([[10, 8.4375]]).float().cuda()
    cam_info = (K, down_ratio)
    output = model(fake_input, cam_info)


[modal] Incorrect logo displayed when retiring raw TCO2

Describe the bug
When retiring raw TCO2 (e.g. the Vlindr mangrove project) the UI displays the USDC logo instead of a reasonable logo for the TCO2:
To Reproduce
Steps to reproduce the behavior:

Purchase an individual project
Go to Portfolio and retire the project
Notice the incorrect logo

Expected behavior
A reasonable logo should be provided rather than the USDC logo for spending TCO2
Screenshots

Desktop (please complete the following information):

OS: Windows 10
Browser: Firefox
Version: 121.0

Additional context
Add any other context about the problem here.

@0xAurelius I think this is related to #1963 and the designs for these modals have been updated to reflect the required changes, we have just not had dev bandwidth to assign and have the work completed yet.
I'll review this issue in more detail early next week and make sure there are not additional requirements to solve for.

Super super quick fix so I went n shipped it https://github.com/KlimaDAO/klimadao/pull/2060
Hide the icon, which is how the other modals work (create listing, edit listing etc.)


Introduce preferred_username_claims setting in cuttlefish config style

This should be backported to stable because it documents something that's yet to ship in 3.11.10.


add docstring examples to maxent and quartile, update notebook to show how to fix parameters

Codecov Report

Merging #139 (ba8d558) into main (08713cb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #139   +/-   ##
=======================================
  Coverage   78.46%   78.46%
=======================================
  Files          18       18
  Lines        2080     2080
=======================================
  Hits         1632     1632
  Misses        448      448

Impacted Files
Coverage Δ

preliz/maxent.py
78.94% <ø> (ø)

preliz/quartile.py
88.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@OriolAbril could you help me with this?
I would like to show the code and plots for the examples in the docstring of maxent and quartile, and only the plots for https://preliz.readthedocs.io/en/latest/api_reference.html#module-preliz.distributions.continuous and https://preliz.readthedocs.io/en/latest/api_reference.html#module-preliz.distributions.discrete

Thank you @OriolAbril!


Rebar compile umbrella tests

The obvious: This branch generalizes some infrastructure in rebar_compile_SUITE so that tests can be written once and then run in both umbrella and plain application mode.  Existing cases are backwards compatible, but some have been updated to be "dual use".
The less obvious:  This branch adds a test that reveals multiple problems in MIB compilation.

Note: This branch adds 2 failing tests.  Travis CI is reporting this correctly.

Thanks, can you rebase.
@talentdeficit have you seen this?

no i missed this. i largely rewrote the compiler test suite with the branch that landed a few days ago so a rebase might be tough. we should definitely land the mib tests though and try to adapt this work if it still makes sense

This will never merge.  Closing this PR while I prepare a new one with just the important bits.


Faster workspace execution

I'm addressing multiple issues here:

Persistent outputs are slower because they write to disk. It's just a bad default.
The default LIMIT is annoying and no longer necessary if we don't persist the table. (#151)
Each SQL box created 6 VertexToEdgeAttribute operations per input vertex attribute. With 100 attributes and 20 SQL boxes this was 12,000 operations on every setAndGetWorkspace request. That takes around 3 seconds. That's long enough that you start clicking on more things, creating an increasing backlog of requests. In my test I got 20–30 second response times easily.

For the third issue 4 of the 6 VertexToEdgeAttribute operations were unnecessarily caused by the defaultTableName code. That's easy to fix. But the other two are unavoidable if we execute the box.
So I've added a "BoxCache" to avoid unnecessary box executions. This is a huge improvement. Even if you only moved a box, we used to execute all the boxes in the workspace. But with the cache we don't execute anything and it's super fast. When you change something, we only execute that one box.
I went with this universal solution rather than try to improve the SQL box code, because it's not the only box that scales linearly with the number of attributes. E.g. a filter box will pull over all attributes. It's a bit faster than the SQL box, but I still saw 1–2 second latencies with 100 attributes and 20 filter boxes. The cache fixes this for all boxes.
Do you see any potential issues with it?

Thanks!


License missing

Could you please indicate the license for this library? I guess like the app using it it will be Apache-2.0, but this repo doesn't state it. Thanks in advance!

Indicated.

Thanks!


[Open Question] BaseMvRxViewModel first class coroutine support

We've already made MvRx work great with rxjava by adding things like disposeOnClear and execute.
I think we could add first class support for coroutines by:

Making BaseMvRxViewModel a CoroutineScope
Making an execute on Deferred<T> which would look something like (untested code):

    protected suspend fun <T> Deferred<T>.execute(reducer: S.(Async<T>) -> S) {
        setState { reducer(Loading()) }
        try {
            val value = await()
            setState { reducer(Success(value)) }
        } catch (e: Exception) {
            setState { reducer(Fail(e)) }
        }
    }

@hellohuanlin @elihart @BenSchwab @ATizik

It's preferable to avoid Deferred, maybe it can look like this?
protected suspend fun <T> (suspend ()->T).execute(reducer: S.(Async<T>) -> S) {     setState { reducer(Loading()) }     try {         val value = this()         setState { reducer(Success(value)) }     } catch (e: Exception) {         setState { reducer(Fail(e)) }     } }
Also when this issue is resolved, additional execute extensions can be added

@ATizik Why is it preferable to avoid Deferred?

@gpeal
In idiomatic coroutine code instead of
fun foo(): Deferred<Int> = TODO()
it's better to use
suspend fun foo(): Int = TODO()
Here is more info

@ATizik Makes sense. Some things like the retrofit coroutines adapter will return Deferred<T> though.

@gpeal Retrofit now support suspend functions 😄

it would be great to do this once the new Jetpack provided ViewModel coroutine scope is out of beta.
It seems reasonable to provide an extension to both Deferred<T> and (suspend ()->T)`.
I think we should also have the function return a Job so the user can cancel it if needed. In the try/catch block do we need to handle CancellationException any differently if the job is canceled, and we don't want it to trigger the fail state?
I'd like to see us make the basic case of running basic processing in the viewmodel as simple as possible. Something like
class MyViewModel {
     init {
           async {
                   lookupSomeData()
          }.execute {
               copy(myData = it)
          }
    }
}

@gpeal Hello, the problem I have  with MvRx is that it uses RxJava as a dependency. If you are also using coroutines in your app, then this increase your apk/bundle size. What's your thoughts on this?

It would be nice to have a version with only coroutines and no Rx for that reason, but I think it may be non trivial, since the whole subscription system is based on rxjava so it isn't as simple as swapping out the state store.
Can you say how large the APK size increase is after the RxJava dependency with R8/Proguard properly minifying it?

In progress with the work on 2.0


Feature request:  time limit individual sub-test rather than entire run_dljc

Having a 30 minute time limit on run_dljc is a problem for the larger suites like boofcv and jmonkeyengine.  I think it would be better to have a shorter limit on each individual sub-test.  Or at least have a way to set different limits for different suites.

I will add this, hopefully next week, by moving the time limit functionality from the integration test directly into DLJC. I'll probably allow 30 minutes for any given command invoked by DLJC by default, with an option to override per-project.

Done. As of 03c653f, there is now a 900 second (15 minute) time limit per DLJC-subcommand, and the overall time limit has been removed.
That time limit can be adjusted per-project by adding a "timelimit" entry for that project in corpus.json.


Update azure-boost-series.md

Including new VM sizes in public preview

@mattmcinnes - can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

#sign-off


Thank you

I just randomly decided to search for "HaloTV Archive" and came across https://csullender.com/halotv and I couldn't believe my eyes. You actually saved the videos and posted the archive online!!!
I had a video on your website 19 years ago (wow). It got seen by the Official Xbox Magazine editor who subsequently did an interview and article about myself, my clan, and the tricks we had discovered and posted in the video. I thought it was lost to time, but thanks to you, I now have a copy of the video. It's not as clear nor as good as I remembered lol but that's 19 years of technology advancements for you.
Thank you so much for having the foresight to save this stuff and make it available in this modern era.

That's awesome! Glad you were able to find it! If I remember correctly, the videos were typically transcoded so that likely accounts for some of the loss in quality.


Fixed Bug In Button By Lakhan06 (Files Not Linked Properly)

Button Fixed Successfully

PR For #441

@Souravvmishra Thank you!


Getting distilled images and testing on them

Hello!
I'm wondering what is the correct way to get distilled images and test performance on them as well check performance on a normal dataset. I'm confused since there are many parameters and I've already read the advanced docs. So, am I right that to get distilled data, for example on Cifar10, I need to run
python main.py --mode distill_basic --dataset Cifar10 --arch AlexCifarNet  --distill_lr 0.001

Then, to test the network on a normal full dataset I need to set --mode train, and if I want to test distilled images I need to set --mode distill_basic --phase test (and also set some params as it was in getting distilled data like --n_epochs 400)?
And then what it means if I write --mode train --phase test?

Sorry, I need to clarify: what is the correct way to train a net on distilled data and then test on a normal dataset?


Add a trouble-shooting section for building the plugin

I just got bitten by this :)

@s12v Would you mind reviewing this? Thanks

Thanks @mumoshu!


⚠ Bofrost.de has degraded performance

In 3094efb, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1416 ms

Resolved: Bofrost.de performance has improved in b91052b.


Add jsx-pascal-case to complete list of rules

The jsx-pascal-case isn't in the example complete config.

Thanks!


Doesn't work on the MS Store version

Doesn't work on the MS Store version, the OS doesn't allow to override the js file.

I'm also using the osx app store slack app. I followed the steps and I see the them apply to the sidebar and the enter chat section at the bottom, but the large main window that shows the chat history remains white, unchanged from stock style.

The file location is different for the MS Store version. I was able to get things working at this path: C:\Program Files (x86)\Slack\resources\app.asar.unpacked\src\static

The MS Store version will not be installed in C:\Program Files as I know


Add is_device_copyable

This is added for compatibility with libraries that expect this type trait. I decided to not make it always true in case that breaks less code.

Thank you!
Have you checked whether it actually works with code that expects to specialize it?
SYCL is very vague w.r.t how its sycl namespace works. In AdaptiveCpp, sycl lives in the namespace hipsycl (adaptivecpp in future versions). The SYCL header then sets an alias.
With this implementation, I'm not sure if it would work if users attempted to specialize like so:
namespace sycl {

template<>
struct is_device_copyable<MyType> : public std::true_type {};
} //sycl

because the declaration of is_device_copyable does not live in ::sycl.
I suspect you may have to declare it in ::sycl instead of ::hipsycl::sycl, which would be a bit unfortunate since it would be the only thing that we pollute the global ::sycl namespace with.

Ah, this was tested on specializations of the form:
template<>
struct sycl::is_device_copyable<MyType> : std::true_type {};

which is what oneDPL uses, which does work.
The form you provided does not work and would indeed require us to pollute the ::sycl namespace.

Interesting. DPC++ seems to declare it in sycl::_V1: https://github.com/intel/llvm/blob/a1355e8717c4c10113b9f3b0e2e214e02c70c421/sycl/include/sycl/detail/is_device_copyable.hpp#L46
Not sure if it makes a difference that _V1 is an inline namespace, but it's quite possible then that DPC++ has the same issue.
The form that you mentioned is what is used as example by the spec: https://registry.khronos.org/SYCL/specs/sycl-2020/html/sycl-2020.html#_is_device_copyable_type_trait
However, I'm not seeing any mention in the spec that the form I used is illegal, and the synopsis for is_device_copyable suggests that it should be declared in ::sycl.
As a first stab at this, we could merge your implementation, and simultaneously ask for clarification of the spec.
There's no SYCL WG meeting scheduled before january, but we could open an issue in KhronosGroup/SYCL-Docs to get some opinions from other people from the SYCL working group.

I'm pretty sure the inline namespace allows the DPC++ implementation to work because that's the primary use-case for them, and it does work with that form in testing.


Upgrade to NU5 mainnet version of librustzcash

Motivation
librustzcash won't have a tagged version until after the zcashd 4.7.0 release. So we need to split its version upgrade out of the NU5 testnet 2 work in #2982.
After #2982, Zebra and zcash_script will depend on a git version of the following shared Zcash crates:

zcash_history
zcash_primitives
zcash_note_encryption
zcash_script

We need to update to the NU5 mainnet versions of these crates.
We should also upgrade to the newest released versions of these crates:

orchard
incrementalmerkletree
equihash
halo2

Tasks

Wait until the zcashd team publishes an updated version of librustzcash
Make sure https://github.com/zcash/librustzcash/pull/459 is in that release (if not, upgrade our fork, as in #2982)

zcash_script

Upgrade zcash_script to the latest orchard and its dependencies (published crate)
Upgrade zcash_script to the latest librustzcash and its dependencies (published crate), removing the git versions in Cargo.toml
Tag a zcash_script beta release

Zebra

Upgrade Zebra to the latest versions of orchard and its dependencies (published crate)
Upgrade Zebra to the latest version of librustzcash and its dependencies (published crate), removing the git versions in Cargo.toml
Upgrade Zebra to the latest version ofzcash_script (git tag)
Remove the exceptions for librustzcash and its dependencies in deny.toml
Merge the Zebra upgrade PR along with the PR for ticket #1841

Hey team! Please add your planning poker estimate with ZenHub @conradoplg @dconnolly @gustavovalverde @jvff @oxarbitrage @teor2345 @upbqdn


Allow primitives in place of a prop definition

class Thing extends DefineClass {
  static define = {
    age: 13
  }
}

needs to setup the type

This should turn into:
class Thing extends DefineClass {
  static define = {
    age: {
      type: types.check(Number),
      default: 13
    }
  }
}


v1.0 backports 2018 05 08

Backports:

https://github.com/cilium/cilium/pull/4006
https://github.com/cilium/cilium/pull/4019
https://github.com/cilium/cilium/pull/4034

test-me-please

Hit #4042.

test-me-please

Hit this:
https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/3028/testReport/junit/k8s-1/7/K8sValidatedServicesTest_CNP_Specs_Test/
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:312
Unable to connect from productpage-v1-127201421-fbmf3 to details:9080/health: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"wget\": executable file not found in $PATH": unknown

Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/k8sT/Services.go:324

Weirdly this is not failing on master, however master has other refactoring in this test. Will retry just to see if it was a transient error.

test-me-please

My guess at this point is that the container for productpage has considerably changed between v1.5.0 and v1.6.0, and the latter no longer contains wget. Furthermore, the earlier container is probably cached in the latest vagrant dev images used on master for builds, so it may not need to fetch/update to the latest container. However, the v1.0 vagrant image doesn't have this, so it fails.
https://hub.docker.com/r/istio/examples-bookinfo-productpage-v1
If so, the fix would be to pin the bookinfo example images we use to known good versions.

Update: Posted #4059 to fix the above issue, will need to backport that also to get this series passing CI.

test-me-please


[WIP][Concurrency] introduce options to wait operations

Similar to https://github.com/apple/swift/pull/37885 [Concurrency] Introduce TaskOptionRecord for spawn options but for all the wait operations.
This is a bit more tricky and I'm still tracking down all spots this needs slight fixes.
Resolves rdar://78931812

Resolves rdar://78931812


slider tick spacing alignment incorrect

Bug, feature request, or proposal:
Bug
What is the expected behavior?
Tick spacing should be consistent
What is the current behavior?
There is a bigger gap between the second to last and last tick
What are the steps to reproduce?
It works fine until you have more than 8 ticks. This slider will show the issue
<md-slider min="0" max="10" tickInterval="1"></md-slider>
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Angular 4.2.6
Angular Material 2.0.0-beta.8
Screenshot
From your demo site

Confirmed, doesn't seem to just be the last tick either, it slowly gets more and more off as you move toward the end of the slider

I believe this is an issue with how chrome handle's the background-size property. It seems that if it comes out to some fractional pixel value it slowly accumulates error each time the background is repeated.
Evidence for this:

I do not see the problem when the slider track length is evenly divisible by the number of ticks (e.g. 1000px and 10 ticks).
I do not see the problem in Firefox at all regardless of the track length.

We may want to consider using a different strategy for drawing ticks or filing a bug with chrome to see if we can get them to change how background-size works. As a temporary workaround if you see this issue in your app, I suggest setting the slider size so that the track is evenly divisible by the number of ticks.

Confirmed that this is fixed in the MDC-based slider (which uses an entirely different style/implementation for ticks)


chapter02 button_pigpio.py and button_gpiozero.py use different debounce times

In chapter02, button_gpiozero.py uses debounce time of 0.1 seconds. button_pigpiozero.py uses debounce time of 10000 microseconds. I think the idea of chapter 2 is to execute the same functionality using pigpio and gpiozero, so this difference in debounce time can be confusing for readers.
10000 microseconds == 0.01 seconds != 0.1 seconds

Hi Eric,
Thank you for bringing this discrepancy to our attention.
I have confirmed that there is a type-o on Page 64 where I state that the debounce time is measured milliseconds when it is actually meant to be microseconds.
I have added an Errata section at the following link discussing the misprint and implications.
https://github.com/PacktPublishing/Practical-Python-Programming-for-IoT/tree/master/errata
I hope you are enjoying the book.
Thanks again,
Gary.


Cannot assign requested address in AWS

I have implemented a websocket server using Spring framework and Netty SocketIO. While development I used the host address as localhost and the server was working perfectly fine.
When I'm trying to deploy my code to AWS, I changed the host address to my instance's public IP address. But I'm getting the following error and I'm unable to start my server
Caused by: java.net.BindException: Can't assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:484)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1080)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:197)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:350)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)

Should I be using any other IP address instead of the public IP address of my Instance?

@sunilgulabani it works! thank you!!

thanks bro..

@pablojr I was able to resolve this issue. When we create SocketIO Server with hostname as "localhost", application is only accessible from local environment. I changed the hostname parameter value to "<IP_ADDRESS>" and it worked seamlessly. Rest AWS security group need to be expose for port which we used.

can you please tell me how to do that i am facing same issue but do not know how to do that. ??Your help wil be appreciated with bundle of thanks.

@sunilgulabani it works! thank you!!

can you please tell me how to do that i am facing same issue but do not know how to do that. ??Your quick help will be highly appreciated...


chore: release 1.2.282

:robot: I have created a release *beep* *boop*
1.2.282 (2024-01-23)
Bug Fixes

cleanup: panic: runtime error: invalid memory address or nil pointer dereference (#5937) (33465ab)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/werf/werf/releases/tag/v1.2.282 :sunflower:


Add roles and people to news edition links

Trello: https://trello.com/c/3GT17tkH/211-associate-content-with-ministers-l
This makes these two link types available as edition links so that they
can be used by Content Publisher.
It also updates the descriptions to not be email specific since these
aren't only used for email.
These links do seem a bit fishy as they all used to link to both aspects
of a role appointment without the link to the role appointment. It would
probably make more sense that these linked to a role appointment and
these were expanded. Though this would be a more substantial change.

Whitehall passes against this branch: https://github.com/alphagov/whitehall/pull/4370


No option for disabling autofill on both fields when one is filled in range datepicker

As simple as that. How can I disable autocompletion in the datepicker range type when the one of them is clicked?

My fault. Here it is. https://bootstrap-datepicker.readthedocs.io/en/latest/options.html#keepemptyvalues


Add easier support for callbackWaitsForEmptyEventLoop

🚀 Feature Proposal
In order to make AWS works properly with mixed callback/async invocation, one might need to modify context.callbackWaitsForEmptyEventLoop.
Nowadays, it is possible with a code like this:
const proxy = awsLambdaFastify(app)

exports.handler = function(event, context, callback) {
  context.callbackWaitsForEmptyEventLoop = false
  return proxy(event, context, callback)
}

But it would be great if it can directly provided as option. See below.
Note that in the implementation checking of the property should be done with options.hasOwnProperty or similar, since both truthy and falsy values are acceptable.
Motivation
To make typical use case easier to manage and without boilerplate code.
Example
exports.handler = awsLambdaFastify(app, {callbackWaitsForEmptyEventLoop: false})

cool idea... landed in v1.0.4


connectivity: check node taints when determining single cluster mode

When checking whether to enable single node connectivity tests, also
check the individual nodes and exclude the master node from the nodes
where connectivity test pods can be scheduled. Otherwise the multi-node
connectivity test will later fail to schedule pods due to unsatisfied
affinity/anti-affinity:
0/2 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
Fixes #167

Superseded by #171 to allow CI runs (currently not possible from forks).


question about the wave velocity in this system

Just saw the paper in Science Advances. Really fantastic work!!
What is the wave velocity with respect to the mesh size and the time step? I guess the device is 3D-printable.

Thanks for your interest in the work!
In the paper we use a "normalized" wave speed. I suppose you could translate this into units of dx/dt if you wanted, but I'm not sure how useful it is to do that since our choice of dx and dt are somewhat arbitrary. We chose the relative wave speeds in the two materials to approximate those of air and polymer (see the citation). The simulation shown in the paper is really a proof of principle to demonstrate the physical dynamics of the wave RNN. Note also that the simulation in the paper is 2D, which also means that it would be challenging to 3D print a working version of that exact design shown in the figure.
In general, the constraint in these types of simulations is that dx, dy, and dt satisfy the CFL stability condition. You could also train a model using exact wave speeds for the materials, e.g. c = 331.0 m/s in air, while picking suitable values for dx and dt. We've previously done this in wavetorch and it works fine.


Fix: test_compute_interpolated_distance_regression and numerical column error

Hi @MUCDK @giovp ,
When running generate_gt_temporal_data I encountered an error because adata[day] wasn't categorical but was a float. So in the code I cast it as a category. Not sure why this happens now. But in the end I created the file and it should help the tests pass. But as I said I am not sure what I should do for the temporal key.
After this PR, ".A" PR will also be ready to merge. Then the "prepare" PR

That’s fine. I think when we generated the data we didn’t have the restriction for the temporal key to be of categorical type. But it’s true that now we do check for it. So all good! Thanks for this fix


Worker doesn't communicate with controller - slurm

Hello,
Here is what I'm running on a CentOS system, I'm using the request package to get my public IP:

from parsl.channels import SSHChannel
#from parsl.channels import SSHInteractiveLoginChannel
from parsl.providers import SlurmProvider
from parsl.launchers import SrunLauncher
from parsl.providers import KubernetesProvider
#from parsl.launchers import SimpleLauncher

from parsl.config import Config
from parsl.executors.ipp import IPyParallelExecutor
from parsl.executors.ipp_controller import Controller

# This is an example config, make sure to
#        replace the specific values below with the literal values
#          (e.g., 'USERNAME' -> 'your_username')

    #SDSC
from requests import get
ip = get('https://api.ipify.org').text
config_sdsc = Config(
    executors=[
        IPyParallelExecutor(
            label='comet_ipp_multinode',
            provider=SlurmProvider(
                'debug',
                channel=SSHChannel(
                    hostname='comet.sdsc.xsede.org',
                    username='mmadany',     # Please replace USERNAME with your username
                    password=,
                    script_dir='/home/mmadany/parsl_scripts',    # Please replace USERNAME with your username
                ),
                launcher=SrunLauncher(),
                scheduler_options='',     # Input your scheduler_options if needed
                worker_init='source /home/mmadany/.bashrc; conda activate parsl',     # Input your worker_init if needed
                walltime="00:10:00",
                init_blocks=1,
                max_blocks=1,
                nodes_per_block=1,
            ),
            controller=Controller(public_ip=ip,port_range="62000,65000"),    # Please replace PUBLIC_IP with your public ip
        )

    ],
)

import parsl
parsl.clear()
parsl.load(config_sdsc)

from parsl.app.app import python_app, bash_app

@bash_app
def generate(outputs=[]):
    touch /home/mmadany/fromparsl_test
    return "test write success"

print(generate())

and this is what I get from the engine job log:

(parsl) [mmadany@comet-ln2 026]$ tail -f engine_logs/parsl.auto.1560064011.8623514.log
2019-06-09 00:05:25.330 [IPEngineApp] Loading url_file 'runinfo/026/ipengine.acb21e24-9e1f-446e-bba3-3bd8729d5a0d.json'
2019-06-09 00:05:25.360 [IPEngineApp] Registering with controller at tcp://<IP_ADDRESS>:63105
2019-06-09 00:05:30.366 [IPEngineApp] CRITICAL | Registration timed out after 5.0 seconds

The correct address to use isn't necessarily the address that comes back from that ipify.org service - it depends on the network setup of your system.
Parsl has a module parsl.addresses with a few different ways of finding a suitable address. See below. There isn't one which works in all situations though.
What you need is an address that a slurm worker node can connect to your submitting system where you are running parsl. From looking at your setup, I guess that you are running on some personal computer somewhere on a university network; it might be that the kind of network connection this needs isn't even possible due to firewalls, and in that case you might do better to run on a comet login node.
import parsl.addresses

ip = address_by_query() # makes the same api.ipify.org call that you have already

ip = address_by_route() # finds a public address by looking for a default route to google

ip = address_by_hostname() # uses the system local hostname rather than an ip

ip = address_by_interface(interface_name) # uses the ip address assigned to a named network interface

Hi @benclifford, thanks for your response.
I'm hesitant to use the login node because:

I'd like to try and configure other clusters
it goes down sometimes
my initial installation attempt of parsl on the node failed due to a libc incompability

I can confirm the port is open now, and I added a check in my worker_init:
`worker_init='echo "hello comet"; source /home/mmadany/.bashrc; conda activate parsl; nc -z -v -w5  '
and I can see in the port is open in stdout:

[mmadany@comet-ln3 ~]$ cat parsl_scripts/parsl.auto.1560310181.9417555.submit.stdout
hello comet
Connection to   port [tcp/*] succeeded!
Found cores : 24
Found nodes : 1
Done

now the output in the run_engine logs is:

2019-06-11 20:41:36.665 [IPEngineApp] Loading url_file 'runinfo/054/ipengine.9b0589ef-5cc0-4fa2-988b-419f9f0c08ba.json'
2019-06-11 20:41:36.695 [IPEngineApp] Registering with controller at tcp://
2019-06-11 20:41:36.744 [IPEngineApp] Starting to monitor the heartbeat signal from the hub every 3010 ms.
2019-06-11 20:41:36.749 [IPEngineApp] Completed registration with id 3
2019-06-11 20:41:42.768 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (1 time(s) in a row).
2019-06-11 20:41:45.778 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (2 time(s) in a row).
2019-06-11 20:41:48.787 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (3 time(s) in a row).
2019-06-11 20:41:51.798 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (4 time(s) in a row).
2019-06-11 20:41:54.808 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (5 time(s) in a row).
2019-06-11 20:41:57.818 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (6 time(s) in a row).
2019-06-11 20:42:00.828 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (7 time(s) in a row).
2019-06-11 20:42:03.838 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (8 time(s) in a row).
2019-06-11 20:42:06.848 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (9 time(s) in a row).
2019-06-11 20:42:09.857 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (10 time(s) in a row).
2019-06-11 20:42:12.868 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (11 time(s) in a row).
2019-06-11 20:42:15.878 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (12 time(s) in a row).
2019-06-11 20:42:18.888 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (13 time(s) in a row).
2019-06-11 20:42:21.898 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (14 time(s) in a row).
2019-06-11 20:42:24.908 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (15 time(s) in a row).
2019-06-11 20:42:27.917 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (16 time(s) in a row).
2019-06-11 20:42:30.928 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (17 time(s) in a row).
2019-06-11 20:42:33.938 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (18 time(s) in a row).
2019-06-11 20:42:36.948 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (19 time(s) in a row).
2019-06-11 20:42:39.957 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (20 time(s) in a row).
2019-06-11 20:42:42.968 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (21 time(s) in a row).
2019-06-11 20:42:45.978 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (22 time(s) in a row).
2019-06-11 20:42:48.988 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (23 time(s) in a row).
2019-06-11 20:42:51.998 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (24 time(s) in a row).
2019-06-11 20:42:55.008 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (25 time(s) in a row).
2019-06-11 20:42:58.018 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (26 time(s) in a row).
2019-06-11 20:43:01.028 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (27 time(s) in a row).
2019-06-11 20:43:04.038 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (28 time(s) in a row).
2019-06-11 20:43:07.048 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (29 time(s) in a row).
2019-06-11 20:43:10.058 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (30 time(s) in a row).
2019-06-11 20:43:13.068 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (31 time(s) in a row).
2019-06-11 20:43:16.078 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (32 time(s) in a row).
2019-06-11 20:43:19.088 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (33 time(s) in a row).
2019-06-11 20:43:22.098 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (34 time(s) in a row).
2019-06-11 20:43:25.108 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (35 time(s) in a row).
2019-06-11 20:43:28.118 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (36 time(s) in a row).
2019-06-11 20:43:31.128 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (37 time(s) in a row).
2019-06-11 20:43:34.138 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (38 time(s) in a row).
2019-06-11 20:43:37.148 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (39 time(s) in a row).
2019-06-11 20:43:40.157 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (40 time(s) in a row).
2019-06-11 20:43:43.168 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (41 time(s) in a row).
2019-06-11 20:43:46.178 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (42 time(s) in a row).
2019-06-11 20:43:49.188 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (43 time(s) in a row).
2019-06-11 20:43:52.198 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (44 time(s) in a row).
2019-06-11 20:43:55.208 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (45 time(s) in a row).
2019-06-11 20:43:58.218 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (46 time(s) in a row).
2019-06-11 20:44:01.228 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (47 time(s) in a row).
2019-06-11 20:44:04.237 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (48 time(s) in a row).
2019-06-11 20:44:07.248 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (49 time(s) in a row).
2019-06-11 20:44:10.258 [IPEngineApp] WARNING | No heartbeat in the last 3010 ms (50 time(s) in a row).
2019-06-11 20:44:10.258 [IPEngineApp] CRITICAL | Maximum number of heartbeats misses reached (50 times 3010 ms), shutting down.
ERROR:root:Invalid alias: The name clear can't be aliased because it is another magic command.
ERROR:root:Invalid alias: The name more can't be aliased because it is another magic command.
ERROR:root:Invalid alias: The name less can't be aliased because it is another magic command.
ERROR:root:Invalid alias: The name man can't be aliased because it is another magic command.

Hi @MatthewBM ,
From your log paste above, it does look like an ipengine is able to connect back to the ipcontroller. If you haven't updated your test script already, the issue might be that you invoke a bash_app and then don't wait for it's result. Parsl programs exit unless you explicitly wait for the results of apps, and this would also terminate the ipcontroller to which workers connect resulting in workers dying due to missed heartbeats.  Secondly, bash_apps are special python functions that are written like they would return a bash script. When execution is complete bash_apps will return the exit code. Could you try updating your script like this :
@bash_app
def generate(outputs=[]):
    return """touch /home/mmadany/fromparsl_test
    """

@python_app
def platform_check():
    import platform
    return platform.uname()

gen_future = generate()
print("Waiting for ", gen_future) # You will print a
print("generate is now done with exit code: ", gen_future.result())

# For python app:
print("Python app ran on: ", platform_check().result())

As for config, here's what I just tested with from a remote node, please note that I'm using our new HighThroughputExecutor rather than the IPyParallelExecutor which is on its way to getting deprecated.
from parsl.config import Config
from parsl.channels import SSHChannel
from parsl.launchers import SrunLauncher
from parsl.providers import SlurmProvider
from parsl.executors import HighThroughputExecutor
from parsl.addresses import address_by_query

config = Config(
    executors=[
        HighThroughputExecutor(
            label='Comet_HTEX_multinode',
            address=address_by_query(),
            worker_logdir_root='/home/yadunand/parsl_workdir', # <--- Update here
            provider=SlurmProvider(
                'debug',
                channel=SSHChannel(
                    hostname='comet.sdsc.xsede.org',
                    username='yadunand',                                  # <--- Update here
                    script_dir='/home/yadunand/parsl_scripts',   # <--- Update here
                ),
                launcher=SrunLauncher(),
                # string to prepend to #SBATCH blocks in the submit
                # script to the scheduler
                scheduler_options='',
                # Command to be run before starting a worker, such as:
                # 'module load Anaconda; source activate parsl_env'.
                worker_init='source activate parsl_0.8.0a0_py3.6',    # <--- Update here to how you setup your env
                walltime='00:10:00',
                init_blocks=1,
                max_blocks=1,
                nodes_per_block=2,
            ),
        )
    ]
)

Hi @yadudoc ,
I don't have worker_logdir_root as an option for HighThroughputExectutor.
I tried commenting it out and running what you sent, and got this from stderr:

>
> (base) [mmadany@comet-ln3 parsl_scripts]$ cat parsl.auto.1560317655.2157164.submit.stderr
> nc: connect to host port 63105 (tcp) failed: Connection refused
> 0: Traceback (most recent call last):
> 0:   File "/home/mmadany/miniconda3/envs/parsl/bin/process_worker_pool.py", line 476, in <module>
> 0:     os.makedirs(os.path.join(args.logdir, args.uid))
> 0:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 210, in makedirs
> 0:     makedirs(head, mode, exist_ok)
> 0:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 210, in makedirs
> 0:     makedirs(head, mode, exist_ok)
> 0:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 210, in makedirs
> 0:     makedirs(head, mode, exist_ok)
> 0:   [Previous line repeated 1 more time]
> 0:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 220, in makedirs
> 0:     mkdir(name, mode)
> 0: PermissionError: [Errno 13] Permission denied: '/home//user'
> 1: Traceback (most recent call last):
> 1:   File "/home/mmadany/miniconda3/envs/parsl/bin/process_worker_pool.py", line 476, in <module>
> 1:     os.makedirs(os.path.join(args.logdir, args.uid))
> 1:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 210, in makedirs
> 1:     makedirs(head, mode, exist_ok)
> 1:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 210, in makedirs
> 1:     makedirs(head, mode, exist_ok)
> 1:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 210, in makedirs
> 1:     makedirs(head, mode, exist_ok)
> 1:   [Previous line repeated 1 more time]
> 1:   File "/home/mmadany/miniconda3/envs/parsl/lib/python3.6/os.py", line 220, in makedirs
> 1:     mkdir(name, mode)
> 1: PermissionError: [Errno 13] Permission denied: '/home/user'
> srun: error: comet-14-02: task 0: Exited with exit code 1
> srun: error: comet-14-03: task 1: Exited with exit code 1
>

I haven't been able to get the HighThroughPut worker to connect, but I'm seeing the controller communicating fine with this script, but it just hangs:

config_sdsc_ipy = Config(
    executors=[
        IPyParallelExecutor(
            label='comet_ipp_multinode',
            provider=SlurmProvider(
                'debug',
                channel=SSHChannel(
                    hostname='comet.sdsc.xsede.org',
                    username='mmadany',     # Please replace USERNAME with your username
                    script_dir='/home/mmadany/parsl_scripts',    # Please replace USERNAME with your username
                ),
                launcher=SrunLauncher(),
                scheduler_options='',     # Input your scheduler_options if needed
                worker_init='echo "hello comet"; source /home/mmadany/.bashrc; conda activate parsl; nc -z -v -w5 pestilence.crbs.ucsd.edu 63105',     # Input your worker_init if needed
                walltime="00:10:00",
                init_blocks=1,
                max_blocks=1,
                nodes_per_block=1,
            ),
            controller=Controller(public_ip=ip,port="63105"),    # Please replace PUBLIC_IP with your public ip
        )

    ],
)

from parsl.app.app import python_app, bash_app
import parsl
parsl.clear()
parsl.load(config_sdsc_ipy)

@bash_app
def generate(outputs=[]):
    return """touch /home/mmadany/fromparsl_testfile
    """

gen_fut = generate()
print(" Wait: ", gen_fut)
print(" Finished: ", gen_fut.result())

Hi @yadudoc
I'm realizing I'm not grasping the abstractions surrounding bash apps, data, and cluster configs. I'm going to do the binder tutorials and come back if I have further questions.


downloading models

Excuse me, I find out that the link (https://www.dropbox.com/s/6zejmxu0aur3568/hamer_demo_data.tar.gz) to retrieve the models now is closing, how can I handle it, thank you!

+1 the file is too popular!
Link temporarily disabled

This can happen when the link has been shared or downloaded too many times in a day.

Check back later and we’ll open access to more people.

Yeah, I had that too

for you guys trying to download, there is the hugginbg face link, that could be used to download the model
try using this command
git lfs clone https://huggingface.co/spaces/geopavlakos/HaMeR

Thanks for pointing this out! This should be only temporary, but we have moved the model to a location that should be always accessible. Please run again the fetch_demo_data.sh command.

Thanks a lot.


[NEW] Support "Message read receipt"

General information

PR that implements it in our back-end: https://github.com/RocketChat/Rocket.Chat/pull/9717
Property of the message to look into: unread (Boolean)
Settings to be verified if feature is enabled:

Message_Read_Receipt_Enabled
Message_Read_Receipt_Store_Users

Assets
Check.zip
Color when not read by everyone: #999999
Color when read by everyone: #1D74F5
Description
When the setting Message_Read_Receipt_Enabled is enabled, the Message objects are going to have a property called unread. The app needs to map this value and show a "check" information on the right side of the timestamp of the message. When feature is disable, this check image is not displayed.
If the setting Message_Read_Receipt_Store_Users  is enabled, when user long press the message, we're going to have a new option called Information that will display only the message in a new screen and below the message there will be a list of users that read the message.

Closed by https://github.com/RocketChat/Rocket.Chat.Android/pull/1338


Re-implement policy evaluation in hyades API server

Current Behavior
Currently the dependency track application performs policy evaluation of components of a project after the Vulnerability analysis is complete for all the components of the project. This is also a sequential and heavy task if triggered for a potfolio. This means that policies are evaluated sequentially for every project and every component in the portfolio. Thus if there are 10 policies to go by, and 150 components in a project and 1000 projects in a portfolio, we can picture how long we are looping around the policy evaluation to complete.

Proposed Change

To speed up the policy evaluation for project and portfolio levels, we could leverage the partitions on kafka topics in two ways to advantage us:

One component being evaluated for multiple policies at the same time.
Multiple components for a given project being evaluated for a given policy
Mutliple projects being evaluated in parallel for a policy

If a policy needs the most recent vulnerability info of a component then we can achieve this by triggering policy evaluation of a component right after the vulnerability analysis of the component is complete. This would still be faster updates on policy rather than waiting for vulnerability analysis of all components to complete before doing policy evaluation.
I will attach a diagram as well for further clarity

Per discussion it seems like the best fit is to keep the policy evaluation on the hyades api server side. We could make it multi threaded so that the response on UI is faster. But would need to gauge the impact on performance of the application in terms of db.

@mehab Can we close this issue, as https://github.com/DependencyTrack/hyades-apiserver/pull/54 has been merged?


[Pulsar Client Cpp] memory leak when receive async

Describe the bug
Currently when the consumer invoke receiveAsync(), there will a callback push to pendingReceives_ , https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/lib/ConsumerImpl.cc#L644, but if where has no more message in the topic, the callback in  pendingReceives_ will not pop anymore, and lead to memory leak.
To Reproduce
Steps to reproduce the behavior:

invoke consumer.receiveAsync() in a for loop, and don't produce any message to the same topic

Firstly it's true, not like the producer's send operation, consumer's receive operation has no timeout. So calling receiveAsync infinitely without any limit while no messages could be received will make pending receives in memory growing infinitely.
However, you can add the limit for the number of pending receives from a client side:
#include <pulsar/Client.h>
#include <mutex>
#include <condition_variable>

using namespace pulsar;

struct PendingReceiveTracker {
    std::mutex mutex_;
    std::condition_variable cond_;

    size_t numPendingReceives_ = 0;
    const size_t maxNumPendingReceives_;

    PendingReceiveTracker(size_t maxNumPendingReceives) : maxNumPendingReceives_(maxNumPendingReceives) {}

    void beforeReceiveAsync() {
        std::unique_lock<std::mutex> lock(mutex_);
        while (numPendingReceives_ >= maxNumPendingReceives_) {
            // TODO: you can add other handler instead of wait here
            cond_.wait(lock);
        }
        numPendingReceives_++;
    }

    void completeReceiveAsync() {
        std::unique_lock<std::mutex> lock(mutex_);
        if (numPendingReceives_ == 0) {
            return;
        }
        numPendingReceives_--;
        cond_.notify_all();
    }
};

int main() {
    Client client("pulsar://localhost:6650");
    Consumer consumer;
    client.subscribe("my-topic", "consumer-1", consumer);  // ignore the returned value check here

    PendingReceiveTracker tracker(1000);
    for (int i = 0; i < 100000; i++) {
        // When the pending receives achieve 1000, this method blocks here until any message is received.
        // You can also change the strategy, see the TODO in beforeReceiveAsync()
        tracker.beforeReceiveAsync();
        consumer.receiveAsync([&tracker](Result result, const Message& msg) {
            if (result == ResultOk) {
                tracker.completeReceiveAsync();
                // TODO: process `msg`
            } else {
                // TODO: handle error
            }
        });
        // do other things here...
    }

    client.close();
}

As for a timeout limit, it's a little more complicated that a timer is needed.
Let's go back to the timeout case. Send timeout means some specific messages' associated responses are not received during the timeout. Users can choose to reproduce the same messages again if they don't care message duplication.
However, unlike the send timeout, receive timeout is not related to any message. So when you encounter the timeout with N pending receives, what will you do? Clear these pending receives? OK, then you'll add more pending receives again.  Clear and then re-add the pending receives is not much different with make these receives continue pending. The only behavior that make much difference is stop receiving, like triggering an alarm.
A simpler way is just calling receive() with a timeout. IMO, receiveAsync() doesn't has any more significant benefit than receive(Message& msg, int timeout) except waiting for multiple messages at the same time. C++ client doesn't provide something like Java's CompletableFuture, so you need to play tricks with the callback.

I think the issue is similar to #8300 . Bur unlike the Java Client, C++ API only exposes the receive callback so there's no case like a future of receive is cancelled but the associated pending receive is still in the queue.
The C++ client's receive logic is more simple:

Add a pending receive when receiveAsync is called.
Remove a pending receive when an incoming messages is received by ClientConnection, while the callback of receiveAsync is called.
Clear all pending receives and complete all associated callbacks when the consumer is closed.

If you want to cancel a pending receive, the only way is to close the consumer. So adding a receive count limit is needed.
By the way, if you don't want to introduce the conditional variable and notify/wait, a simple way is:
    std::atomic_int numPendingReceives{0};
    for (int i = 0; i < 100000; i++) {
        if (++numPendingReceives >= 1000) {
            // TODO: wait for a few milliseconds? or sending alarms?
        }
        consumer.receiveAsync([&numPendingReceives](Result result, const Message& msg) {
            numPendingReceives--;
            if (result == ResultOk) {
                // TODO: process `msg`
            } else {
                // TODO: handle error
            }
        });
        // do other things here...
    }

@BewareMyPower thanks for your detail reply, I think client side should avoid these risk which will cause application unstable but not depend on some workaround on application size.

@ltamber The cost of flexibility is the complexity. It's not like sendAsync vs. send, receive(Message&, int) is easy and enough to use in most cases. The only case to use receiveAsync is to wait multiple messages.
If you're using Java client, when you choose receiveAsync, the logic will still be complicated.
Let's start with a simple client code:
        try (PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build()) {
            Consumer<byte[]> consumer = client.newConsumer()
                    .topic("my-topic")
                    .subscriptionName("my-sub")
                    .subscribe();
            for (int i = 0; i < 100000/* just to represent a large number */; i++) {
                consumer.receiveAsync().whenComplete((msg, e) -> {
                    if (e == null) {
                        // TODO: process `msg`
                    } else {
                        // TODO: handle error
                    }
                });
            }
        } catch (PulsarClientException e) {
            e.printStackTrace();
        }

The issue is the same. If there're no messages, the pending receives queue will grow infinitely until OOM.
Then you need to manage these futures, e.g.
            ArrayBlockingQueue<CompletableFuture<Message<byte[]>>> queue = new ArrayBlockingQueue<>(1000);
            for (int i = 0; i < 100000; i++) {
                CompletableFuture<Message<byte[]>> future = consumer.receiveAsync();
                queue.put(future); // it blocks when the queue is full
                future.whenComplete((msg, e) -> {
                    if (e == null) {
                        // TODO: process `msg`
                    } else {
                        // TODO: handle error
                    }
                    queue.poll(); // just to release a space
                });
            }

So it's like the first C++ example. The code looks more simple just because ArrayBlockingQueue has already encapsulated the conditional variable. Also,
As for the second C++ example, instead of using queue.put(future):
                if (!queue.offer(future)) {
                    // TODO: handle the queue is full error to make sure the next queue.add won't fail
                    queue.add(future);
                }

You can just wait for a while, or after #8326 , you can call cancel here to remove all pending receives:
                if (!queue.offer(future)) {
                    queue.forEach(futureInQueue -> futureInQueue.cancel(true));
                    queue.add(future);
                }

The code looks more simple but it's just because Java standard library is more strong. You still have to handle the corner cases from a client side instead of relying on the library.

Closed as stale. The development of the C++ client has been permanently moved to http://github.com/apache/pulsar-client-cpp. Please open an issue there if it's still relevant.


tests: Add mptcpd sockaddr utils unit test.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.6%) to 48.617%

Totals

Change from base Build<PHONE_NUMBER>:
0.6%

Covered Lines:
967

Relevant Lines:
1989

💛  - Coveralls


Create 2023-01-15-netdata-functions.md

A blog to announce Netdata functions.

Does date need to be updated


Add required Bazel version (4.0.0)

After trying to compile on Debian bullseye, it was pointed out to me (see https://github.com/google/lyra/issues/10) that Bazel 4.0.0 is required for the build commands to work correctly. This PR mentions this requirement in the README.

@googlebot I signed it!


rtupdate

rtupdate asumes interface is eth0, so getting an error:
root@server:/# sudo rtupdate
Device "eth0" does not exist
Tried to manually edit rtupdate to the correct interface:
root@rtor:/# sudo rtupdate
Must be run using sudo

No need for looking up the ip address so I have removed that, and you should not get that error if you use the latest version.
You should run using a sudo not directly as root.


Add step to set SiteURL.

Added a step to include setting SiteURL in config.json

Also set "SiteURL" to the full base URL of the site (e.g. "https://mattermost.example.com").

The mattermost instance won't start if the SiteURL isn't set. I figured it would be easier to read ahead of time instead of reading the logs to figure out what went wrong.

Thanks @src-r-r for the pull request!
Per the CONTRIBUTING.md file displayed when you created this pull request, we need to add you to the list of approved contributors for the Mattermost project.
Please help complete the Mattermost contribution license agreement?
This is a standard procedure for many open source projects. Your form should be processed within 24 hours and reviewers for your pull request will be able to proceed.
Please let us know if you have any questions.
We are very happy to have you join our growing community! If you're not yet a member, please consider joining our Contributors community channel to meet other contributors and discuss new opportunities with the core team.

/check-cla

@src-r-r Thanks for this PR and the contribution. Can you please sign the CLA? thanks again


Tweak body interrupting for better performance of DefaultHead Middleware

[info] Benchmark                                     Mode  Cnt      Score      Error   Units
[info] InterruptWhenBench.interruptWhenIOBench      thrpt    5  14981.354 ±   47.508  ops/ms
[info] InterruptWhenBench.interruptWhenStreamBench  thrpt    5  11910.674 ± 1005.696  ops/ms

  def stream: fs2.Stream[IO, String] =
    fs2.Stream.emit("foo").repeat

  @Benchmark
  def interruptWhenStreamBench: IO[Unit] =
    stream.interruptWhen(fs2.Stream(true)).compile.drain

  @Benchmark
  def interruptWhenIOBench: IO[Unit] =
    stream.interruptWhen(IO.pure[Either[Throwable, Unit]](Right(()))).compile.drain

Yay, we have shed all of these bikes! Joking apart, thanks for reviewing this, folks.


toggle.sg-download: init at 2016-02-08

Initial packaging of a basic toggle.sg download script.
Things done:

[X] Tested using sandboxing (nix-build --option build-use-chroot true or nix.useChroot on NixOS)
Built on platform(s)

[X] NixOS
[ ] OS X
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[X] Fits CONTRIBUTING.md.

Looks ok to me


Create subpackage unix

Based on the discussion in #7 I would like to suggest a subpackage unix. The suggestion is to have functions in this package, which have the same names as the respective unix commands. This would allow to keep the API of script clean and high level.
At the same time this would provide an easy start for people, how are familiar with the unix commands.
Maybe some of the high level commands in the scriptpackage would even decide to use the unix low level commands for their implementation.

Very interesting idea! I'll think about this.

The suggestion is to have functions in this package, which have the same names as the respective unix commands.

I totally understand the reasoning behind this suggestion and it makes sense to me. However, I don't think I want to implement it directly in script, for at least two reasons:

I want script to be usable by people who aren't already familiar with the Unix commands (though obviously the problem domain is much the same, so many of the names will be similar). So I want the column filter to be named Column(), not Cut() for example, because the latter makes very little sense unless you already know about the Unix cut command. If you're fresh out of school and doing your first DevOps job, I want you to be able to grab script and start being productive without having to learn the traditional coreutils commands first.

I don't want there to be two different ways to do the same thing. That's a sure-fire recipe for a confusing and hard-to-maintain API. After all, if you copy the Unix name, you must copy the Unix API, and that's different to the existing script APIs. Translating from one to the other, I think you'll find if you try this to any great extent, is fraught with difficulty.

Now, I'm not saying a unix library like this is a bad idea, or that it shouldn't happen. I'm just saying I won't implement it as part of the script library. Anyone would be welcome (indeed, encouraged) to write their own library which has the Unix command names and APIs, and uses the script library as a backend to implement them (or, indeed, not).


Set FRR mode as the default METALLB_BGP_TYPE

We only support the FRR BGP mode d/s,
setting it here as default. This shouldn't
be changed by the users.
Signed-off-by: Ori Braunshtein<EMAIL_ADDRESS>

/lgtm
/approve

/override ci/prow/operator-e2e


How to use prompt turning(language_prompt_v2) in custom datasets?

Hi, Thanks for your sharing this nice work!
I would like to confirm the following question:

When I follow the setting "SOLVER.WEIGHT_DECAY 0.25  SOLVER.BASE_LR 0.05 SOLVER.TUNING_HIGHLEVEL_OVERRIDE language_prompt_v2", I can not get a nice result. There is still a big gap compared to full_model turning<1shot,3shot,5shot,10shot,...,full data>.
When I only use name in my custom dataset, I can not achieve the effect in the paper.  When I use manul prompt, can you give me some suggestions?
There are some automated ways to set prompt? Like NLP?
I find 4 language prompt turning way in the code, can you tell me the difference between them?

Finally, thanks  for this great work!

Hi, thank you for raising this great question!

Difference between 4 kinds of prompt tuning ways

v2 is the most "literal" prompt tuning, in that it will freeze both the language model and the rest of the model; add a bias embedding vector to the prompt embedding; and we will just tune the bias vector (https://github.com/microsoft/GLIP/blob/e496d64141086fbd5f064ee6f4a9b7427f0be101/maskrcnn_benchmark/modeling/rpn/vldyhead.py#L914). Since we are only optimizing this bias vector, v2 usually requires some hyper-parameter search, with usually larger learning rate and weight decay. Please consider tuning the learning rate with respect to your batch size.
v1 is a more relaxed prompt tuning, in that we will tune the whole language model (BERT) during prompt tuning. We used v1 mainly as a way to test our idea, as it is easier to optimize. You can just use mostly the same hyper-parameters as fine-tuning for v1. Try setting SOLVER.LANG_LR between 1e-5 to 5e-5.
If v2 does not work for you out of the box, we recommend trying v1 first and tune v2; conceptually, v1 and v2 should have similar performance if the hyper-parameters of v2 are tuned right.
v3 is v1 + unfreezing a few more linear layers as in linear probing (e.g., the centerness branch); v4 is v2 + unfreezing a few more linear layers. Practically, we find them to be similar to v1 & v2.

Prompt tuning performance could also depend on your custom dataset.
For some datasets that are very different from the pre-training dataset, we do find that prompt tuning could underperform full-tuning (you could check the detailed per-dataset stats in the paper Appendix, Table 14 & 15).

Manually prompts matters the most for zero-shot evaluation
Could you share the statistics of your custom dataset? My experience is that as long as the class names make sense, with a moderate amount of data, the model should be able to learn well without good manual prompts.

In fact, prompt tuning is the "automated way to set prompt" embeddings from my understanding. It's similar to the soft prompt approach in Lester et al.

@bushou-yhh hi, I'm just wondering if your result of prompt tuning a vl model on Disdrone dataset is comparable to the normal finetune process in pure visual modality, do you have any clues?

hi, I'm just wondering if your result of prompt tuning a vl model on Disdrone dataset is comparable to the normal finetune process in pure visual modality, do you have any clues?

In my practice, VLMs work much better than pure vision models (with the same settings, no patch, etc.), GLIP's pre-training to get a good concept that transfer to Visdrone works well, and at the same time GLIP can do open-vocabulary OD, which is worth exploring (But I don't do aerial imagery anymore,  the knowledge is probably outdated)

@bushou-yhh Thanks for your infomation! so are you using VLM as pretrained model to finetune visual tasks or finetuning as a grounding task?