[BUG]  Audit log support kafka

What is the bug?
A clear and concise description of the bug.
In source  code，Kafka can used for audit log, but I couldn't find how to use it in the following configuration. How can I configure to use Kafka as the audit log?

plugins.security.audit.type: kafka
plugins.security.audit.config.bootstrap_servers: ...
plugins.security.audit.config.topic_name: ...

The above configuration will result in unknown settings error
How can one reproduce the bug?
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

What is the expected behavior?
A clear and concise description of what you expected to happen.
What is your host/environment?

OS: [e.g. iOS]
Version [e.g. 22]
Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.

[Triage] Hi @arvinsg, thanks for filing this issue. This seems like a question better handled by the implementation experts over on the OpenSearch forum. Here is a link: https://forum.opensearch.org/.
I am going to close this issue since I think this is just a configuration issue but please open a new issue if you find a bug in the code.

I'd like for this issue to be re-opened as it seems the configuration directives described in src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java#L54 seems to missing from src/main/java/org/opensearch/security/support/ConfigConstants.java#L41. Preventing configuration of this sink.

This change was added at the inception of security plugin: https://github.com/opensearch-project/security/commit/beef34004d22b232c5e309c1728b20e889e19adc. One of the team members will investigate this and get back.

I'd like for this issue to be re-opened as it seems the configuration directives described in src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java#L54 seems to missing from src/main/java/org/opensearch/security/support/ConfigConstants.java#L41. Preventing configuration of this sink.

Hi @fazir83, this issue was closed because configuration questions are generally best handled on the forum. This issue does not discuss a specific code bug or reproduction steps but is more of a "how to" type issue. We generally suggest heading to the forum for this type of support.

Working around the issues with the configuration checking
[dev]$ git diff src/main/java/org/opensearch/security/support/ConfigConstants.java
diff --git a/src/main/java/org/opensearch/security/support/ConfigConstants.java b/src/main/java/org/opensearch/security/support/ConfigConstants.java
index a17678ea..3936e51a 100644
--- a/src/main/java/org/opensearch/security/support/ConfigConstants.java
+++ b/src/main/java/org/opensearch/security/support/ConfigConstants.java
@@ -204,6 +204,10 @@ public class ConfigConstants {
     // Log4j
     public static final String SECURITY_AUDIT_LOG4J_LOGGER_NAME = "log4j.logger_name";
     public static final String SECURITY_AUDIT_LOG4J_LEVEL = "log4j.level";
+
+    // Kafka
+    public static final String SECURITY_AUDIT_KAFKA_TOPIC_NAME = "kafka.topic_name";
+    public static final String SECURITY_AUDIT_KAFKA_BOOTSTRAP_SERVERS = "kafka.bootstrap_servers";

     // retry
     public static final String SECURITY_AUDIT_RETRY_COUNT = "plugins.security.audit.config.retry_count";
[dev]$ git diff src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java
diff --git a/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java b/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java
index 36ef7709..2c640862 100644
--- a/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java
+++ b/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java
@@ -1718,6 +1718,22 @@ public final class OpenSearchSecurityPlugin extends OpenSearchSecuritySSLPlugin
                 )
             );

+            // Kafka
+            settings.add(
+                Setting.simpleString(
+ ConfigConstants.SECURITY_AUDIT_CONFIG_DEFAULT_PREFIX + ConfigConstants.SECURITY_AUDIT_KAFKA_BOOTSTRAP_SERVERS,
+                    Property.NodeScope,
+                    Property.Filtered
+                )
+            );
+            settings.add(
+                Setting.simpleString(
+ ConfigConstants.SECURITY_AUDIT_CONFIG_DEFAULT_PREFIX + ConfigConstants.SECURITY_AUDIT_KAFKA_TOPIC_NAME,
+                    Property.NodeScope,
+                    Property.Filtered
+                )
+            );
+
             // Kerberos
settings.add(Setting.simpleString(ConfigConstants.SECURITY_KERBEROS_KRB5_FILEPATH, Property.NodeScope, Property.Filtered));
             settings.add(

Doesn't even work.
Even if you were configuring it with the help of the routers, it would not run. As both AuditMessageRouter and KafkaSink seems to be not working correctly.
plugins.security.audit.routes.AUTHENTICATED: kafka
plugins.security.audit.endpoints.kafka:
  type: kafka
  config:
    bootstrap_servers: "<IP_ADDRESS>:19092"
    topic_name: "test_topic"
    client_id: "tester_os"

For one, the KafkaSink is unable to correctly load the Producer due to a classLoader issue, as per this stackoverflow
Secondly, if you were to work around that using the suggestions in the above stackoverflow:
[dev]$ git diff src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java
diff --git a/src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java b/src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java
index e67ed665..7907ec21 100644
--- a/src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java
+++ b/src/main/java/org/opensearch/security/auditlog/sink/KafkaSink.java
@@ -57,8 +57,8 @@ public class KafkaSink extends AuditLogSink {
             }
         }

- producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, LongSerializer.class.getName());
- producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
         topicName = sinkSettings.get("topic_name");

         // map path of
@@ -76,6 +76,7 @@ public class KafkaSink extends AuditLogSink {
             this.producer = AccessController.doPrivileged(new PrivilegedExceptionAction<KafkaProducer<Long, String>>() {
                 @Override
                 public KafkaProducer<Long, String> run() throws Exception {
+ Thread.currentThread().setContextClassLoader(null);
                     return new KafkaProducer<Long, String>(producerProps);
                 }
             });

You would run into the same issues described in #4397 (JavaSecurityPolicy errors)
And working around that seems to indicate some issues in AuditMessageRouter:
[2024-06-04T20:08:11,205][TRACE][o.o.s.a.r.AuditMessageRouter] [snip] Setting up routes for endpoint AUTHENTICATED, configuration is kafka
[2024-06-04T20:08:11,205][ERROR][o.o.s.s.DynamicConfigFactory] [snip] Could not dispatch event: class org.opensearch.security.auditlog.config.AuditConfig to subscribing class class org.opensearch.security.auditlog.impl.AuditLogImpl
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.Map (java.util.HashMap$Node and java.util.Map are in module java.base of loader 'bootstrap')
    at org.opensearch.security.auditlog.routing.AuditMessageRouter.lambda$enableRoutes$2(AuditMessageRouter.java:125) ~[opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:541) ~[?:?]
    at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1850) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
    at org.opensearch.security.auditlog.routing.AuditMessageRouter.enableRoutes(AuditMessageRouter.java:155) ~[opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.auditlog.impl.AuditLogImpl.enableRoutes(AuditLogImpl.java:92) ~[opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.auditlog.impl.AbstractAuditLog.onComplianceConfigChanged(AbstractAuditLog.java:138) ~[opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.auditlog.impl.AuditLogImpl.setConfig(AuditLogImpl.java:86) ~[opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517) [eventbus-java-3.3.1.jar:?]
    at org.greenrobot.eventbus.EventBus.postToSubscription(EventBus.java:440) [eventbus-java-3.3.1.jar:?]
    at org.greenrobot.eventbus.EventBus.postSingleEventForEventType(EventBus.java:421) [eventbus-java-3.3.1.jar:?]
    at org.greenrobot.eventbus.EventBus.postSingleEvent(EventBus.java:394) [eventbus-java-3.3.1.jar:?]
    at org.greenrobot.eventbus.EventBus.post(EventBus.java:275) [eventbus-java-3.3.1.jar:?]
    at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:314) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:572) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:561) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:556) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:540) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:533) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:286) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$10(ConfigurationRepository.java:441) [opensearch-security-<IP_ADDRESS>-SNAPSHOT.jar:<IP_ADDRESS>-SNAPSHOT]
    at java.base/java.lang.Thread.run(Thread.java:833) [?:?]

Where AuditMessageRouter.java seems to be sending the entry from the router config expecting it to be a list. But setting it as a list in opensearch.yml returns a settings exception from opensearch-core
Like I have said before, I am not a Java developer, but I really do think this is not a case of needing some help configuring, I think this part of the codebase has been unchanged drastically since floragunns SecurityGuard
This should either be fixed, or the dead code of "KafkaSink" should be removed as to not confuse people on unsupported ways of configuring, or the configuration should be changed to not include "Kafka" as a supported appender (unless specified you need to recompile opensearch-core with the correct libraries needed) or the audit feature should be fixed to support kafka as a first class sink.
Or, if I am indeed wrong, which I really hope I am, it would be very much helpful for the KafkaSink to be documented on how to implement, because even reading through the source code, two people have been unable to do so, hence my apprehension against closing this issue due to "configuration issues".

It is still a problem. Related problem to it:
https://github.com/opensearch-project/security/issues/4397

[Triage] Thanks @willyborankin for taking a look. Since this is a confirmed issue, I will mark as triaged.


Version change from 9.1.2 to 9.1.3

Changes

[x] Changed package version to 9.1.3.

Pending release references

#115

Regards.

Codecov Report

Merging #116 (42711dd) into master (99b7e0f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #116   +/-   ##
=======================================
  Coverage   91.87%   91.87%
=======================================
  Files         254      254
  Lines        5352     5352
  Branches      857      857
=======================================
  Hits         4917     4917
  Misses        268      268
  Partials      167      167

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


LB instance targets are not reachable for sometime after they are restarted.

Rancher Version: Build from master
rancher/dns:v0.10.5
rancher/healthcheck:v0.1.0
rancher/net:v0.7.0
rancher/metadata:v0.6.3
rancher/plugin-manager:v0.2.1
rancher/scheduler:v0.1.2
Steps to reproduce the problem:
Create a LB service with target services.
Attempt any of the following actions:

Restart 1 instance of the target services.
Stop  1 instance of the target services and wait for it to start.
Deactivate/Activate environment.

Access LB.  Haproxy is configured correctly with the backend ip address, but the backend container is not accessible via ip sec network for a while. It's up and running (I've checked docker status), it is just not reachable by its ip address.
@leodotcloud the way to reproduce it is - you don't even need an lb. Just start any client service with "curl" installed, then create nginx service. Test nginx service by:
curl <IP_ADDRESS>:80

from the client container. Restart nginx container, continue curling the nginx ip from curl. Even when nginx container is up and running (which happens immediately), the ip address is not accessible for a while.
Once debugged, see if this slowness is acceptable. If yes, let @sangeethah know if longer timeouts have to be configured in validation tests

@sangeethah Is this issue still present?

@leodotcloud , yes this issue is still reproducible in automation runs.
After a instance is restarted , LB will not direct traffic to the restarted instance for the first few hits of LB ip.

Tested with rancher-server -v1.4.0-rc5.
Following automation tests relating to Lb instance restart does not report this issue anymore:
1.Restart 1 instance of the target services.
2.Stop 1 instance of the target services and wait for it to start.
3.Deactivate/Activate environment.

@leodotcloud , when testing with rancher-server -v1.4.1-rc1 , I see this issue starting to happen again.

@sangeethah As discussed offline, in your setup the ipsec version is v0.8.1. The fix went in v0.8.3. The catalog is pointing to the wrong tag.


Attachemnt in format '.patch' are changed to '.patch.bin' during download.

My Setup

My OS name and version: Windows 7
My current app version: 2.8

[x] I have tested with the latest application version
[x] I can simulate the issue easily

Description
Attachments, which are ended with .patch during download change to .patch.bin

I'm unable to recreate this. Is it just any file renamed to *.patch that doesn't download?
I've tested this and it seems to be working.
I'm closing this issue for now, but can you reopen it if it's still a problem. - It could have/still be a problem with the rc app, and not electron


Improve lintPython execution time

Use Mypy daemon to improve lintPython performance.
Related to: universe issue-11585

Explore pyre-check

Checkout mypyc


Provide context menu for directories

Feature suggestion
Detailed Description

Provide access to directory context menu when navigating files (e.g. on auto-mounted device).
Context

It is currently not possible, from the app, to enqueue an entire directory for music playback. When using the main Kodi UI, I can do that with the context menu on a directory, but the app does not have a context menu on directories.

Isn't that what the top left icon on the remote d-pad does (the one to the left of the up arrow)?
At least going into Music->files->tapping there brings up the context menu.

That's when using the app as a remote and looking at the TV screen.
I am talking about the part of the app that locally displays menus and lets you navigate without even looking at the TV screen.

Yeah I found that, it's not very discoverable though. Moreover, it doesn't help when I already have some songs queued and now want to queue another whole directory.


Add option to add custom url to workspace

Describe what your pull request addresses

[ ] #8
[ ] This just adds the option to the UI it doesn't actually work yet.

Marked as a draft PR (work in progress); looks good so far. Do you want to merge to main?
If so, remove the draft status, let me know, and I'll finish it up.
@superdav42

Second there is a remnant of a URL input field in the app bar. Left by mistake?

Yes, I just removed it.
I haven't made it actually fetch the content yet. I was going to wait until Lars PR is merged since it'll probably change many things related to that. This is why it's in draft.

This is ready for review now. Content is loaded from the provided URL.


Keycloak OIDC - Session for the currently logged admin is expiring before time making the user search to fail

What kind of request is this:
bug
Steps to reproduce:

Login as Admin
Stay logged in for some time
Enable Keycloak OIDC
Try searching for users in the Ember Members input fields (Cluster Members, Project Members, Edit Cluster)

Result:
I am seeing these after some time (or if the admin has been logged for some time)  and try to do a user search from ember forms:
user-65l85 is the local admin
2021/06/21 19:29:10 [DEBUG] Triggering auth refresh on user-65l85
2021/06/21 19:29:10 [DEBUG] Skipping refresh for user-65l85 due to max-age

API returns:
{"baseType":"error","code":"ServerError","message":"invalid character 'B' looking for beginning of value","status":500,"type":"error"}
Expected Result:
Session shouldn't expire before time and the search users fields should continue to work
Other details that may be helpful:
The search works again if I log out and log back in.
I also tried to reproduce this with OpenLDAP, the session stays and the user keeps
Environment information

Rancher version: 2.6 master-head (06/21/2021) 4d18400
Installation option: HA RKE1

Related issue: https://github.com/rancher/rancher/issues/33036

Rancher version: 2.6 master-head (07/08/2021) cf5a75f
Is the refresh token request on-demand?
Keycloak seems to have a 30 minutes SSO Session Idle of 30 minutes by default.
If I stay longer that that time idle in the current logged in session without searching and then attempt a user/group search I hit the 500 described in this issue not only with admin but with any user that's logged in and idle.
Rancher has a auth-user-session-ttl-minutes  that defaults to 960 or 16 hours.
If I match Keycloak's SSO Session Idle with Rancher's session TTL I won't hit this issue. But then using defaults makes refresh token fix not relevant if it's on-demand if we hit the SSO Session Idle first we will end with a 500 request and the user will have to request a new session (log out and log in again).
Unless there's a way to request Keycloak's Refresh Token before we hit the SSO Session Idle the session Idle will be bumped automatically for the users instead of trying to request a new refresh token on-demand. As far as I understand from Keycloak's Docs.
Refresh token requests will also bump the idle timeout.

Rancher version: 2.6 master-head (07/14/2021) __
Principal search is now returning an empty body. But the Rancher server logs show the panic below.
/v3/principals?action=search
2021/07/14 19:21:28 [ERROR] Error during subscribe websocket: close sent
2021/07/14 19:21:36 [ERROR] [keycloak oidc]: received error from keycloak: Get "https://<KEYCLOAK_IP>:8443/auth/admin/realms/amp/users?search=tes": oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_grant","error_description":"Token is not active"}
2021/07/14 19:21:36 [ERROR] Panic serving api request:
goroutine 57311 [running]:
runtime/debug.Stack(0xc00539d2c0, 0x489dc00, 0x8328690)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/rancher/norman/api.(*Server).ServeHTTP.func1(0x5b5dd40, 0xc0098eaf50)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/api/server.go:175 +0x85
panic(0x489dc00, 0x8328690)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloak(0xc002431ec8, 0xc001fa8500, 0x43, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:196 +0x265
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).searchPrincipals(0xc002431ec8, 0xc008bb7a60, 0x3, 0x0, 0x0, 0xc00532d680, 0x0, 0x1, 0xc009bb0900, 0x8c1, ...)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:51 +0x1d7
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).SearchPrincipals(0xc0001b87e0, 0xc008bb7a60, 0x3, 0x0, 0x0, 0x52be9e2, 0x5, 0xc008d5b0c8, 0x17, 0xc005a6e9d4, ...)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:92 +0x30e
github.com/rancher/rancher/pkg/auth/providers.SearchPrincipals(0xc008bb7a60, 0x3, 0x0, 0x0, 0x52be9e2, 0x5, 0xc008d5b0c8, 0x17, 0xc005a6e9d4, 0xb, ...)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:183 +0x135
github.com/rancher/rancher/pkg/auth/principals.(*principalsHandler).actions(0xc000107450, 0xc00ace3d8b, 0x6, 0xc003f3a0e0, 0xc0013a4a20, 0x0, 0x0)
	/go/src/github.com/rancher/rancher/pkg/auth/principals/handler.go:58 +0x41a
github.com/rancher/norman/api.handleAction(0xc003f3a0e0, 0xc0013a4a20, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/api/server.go:263 +0x62
github.com/rancher/norman/api.(*Server).handle(0xc0009d4f20, 0x5b5dd40, 0xc0098eaf50, 0xc0074e2200, 0x40b8ec, 0x4d8a520, 0xc0098eaf50)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/api/server.go:251 +0xde
github.com/rancher/norman/api.(*Server).ServeHTTP(0xc0009d4f20, 0x5b5dd40, 0xc0098eaf50, 0xc0074e2200)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/api/server.go:180 +0x7d
github.com/rancher/apiserver/pkg/middleware.ContentType.func1(0x5b4dcc0, 0xc0023c02a0, 0xc0074e2200)
	/go/pkg/mod/github.com/rancher/apiserver@v0.0.0-20210519053359-f943376c4b42/pkg/middleware/content.go:33 +0x93
net/http.HandlerFunc.ServeHTTP(0xc001dcf590, 0x5b4dcc0, 0xc0023c02a0, 0xc0074e2200)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/apiserver/pkg/middleware.Gzip.func1(0x5b5ce40, 0xc00e16d500, 0xc0074e2200)
	/go/pkg/mod/github.com/rancher/apiserver@v0.0.0-20210519053359-f943376c4b42/pkg/middleware/gzip.go:60 +0x1e3
net/http.HandlerFunc.ServeHTTP(0xc001dcf5a8, 0x5b5ce40, 0xc00e16d500, 0xc0074e2200)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/rancher/pkg/auth/requests.authHeaderHandler.ServeHTTP(0x5af2fa0, 0xc001dcf5a8, 0x5b5ce40, 0xc00e16d500, 0xc0074e2200)
	/go/src/github.com/rancher/rancher/pkg/auth/requests/filter.go:64 +0x93f
github.com/rancher/rancher/pkg/rbac.NewAccessControlHandler.func1.1(0x5b5ce40, 0xc00e16d500, 0xc0074e2100)
	/go/src/github.com/rancher/rancher/pkg/rbac/user_based.go:33 +0x1c7
net/http.HandlerFunc.ServeHTTP(0xc008939458, 0x5b5ce40, 0xc00e16d500, 0xc0074e2100)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/steve/pkg/auth.ToMiddleware.func1.1(0x5b5ce40, 0xc00e16d500, 0xc0074e2000)
	/go/pkg/mod/github.com/rancher/steve@v0.0.0-20210520191028-52f86dce9bd4/pkg/auth/filter.go:167 +0x1c7
net/http.HandlerFunc.ServeHTTP(0xc0023c0180, 0x5b5ce40, 0xc00e16d500, 0xc0074e2000)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00267c780, 0x5b5ce40, 0xc00e16d500, 0xc008ea3e00)
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00267d440, 0x5b5ce40, 0xc00e16d500, 0xc008ea3c00)
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
github.com/rancher/rancher/pkg/multiclustermanager.(*DeferredServer).Middleware.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea3c00)
	/go/src/github.com/rancher/rancher/pkg/multiclustermanager/deferred.go:113 +0x9d
net/http.HandlerFunc.ServeHTTP(0xc000b47f60, 0x5b5ce40, 0xc00e16d500, 0xc008ea3c00)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00267c3c0, 0x5b5ce40, 0xc00e16d500, 0xc008ea3a00)
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00267db00, 0x5b5ce40, 0xc00e16d500, 0xc008ea3700)
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
github.com/rancher/rancher/pkg/api/steve/aggregation.(*aggregationHandler).Middleware.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea3700)
	/go/src/github.com/rancher/rancher/pkg/api/steve/aggregation/aggregation.go:52 +0x6e
net/http.HandlerFunc.ServeHTTP(0xc000b47f80, 0x5b5ce40, 0xc00e16d500, 0xc008ea3700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00267c300, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
github.com/rancher/rancher/pkg/api/steve/proxy.NewProxyMiddleware.func1.1(0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/src/github.com/rancher/rancher/pkg/api/steve/proxy/proxy.go:75 +0x65
net/http.HandlerFunc.ServeHTTP(0xc000b47fa0, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/rancher/pkg/api/steve/proxy.RewriteLocalCluster.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/src/github.com/rancher/rancher/pkg/api/steve/proxy/proxy.go:42 +0xf2
net/http.HandlerFunc.ServeHTTP(0xc0026b0048, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/rancher/pkg/websocket.websocketHandler.ServeHTTP(0x5af2fa0, 0xc0026b0048, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/src/github.com/rancher/rancher/pkg/websocket/handler.go:35 +0xa2
github.com/rancher/apiserver/pkg/middleware.ContentTypeOptions.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/pkg/mod/github.com/rancher/apiserver@v0.0.0-20210519053359-f943376c4b42/pkg/middleware/cache.go:47 +0x131
net/http.HandlerFunc.ServeHTTP(0xc0026b0060, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/rancher/pkg/auth.SetXAPICattleAuthHeader.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/src/github.com/rancher/rancher/pkg/auth/server.go:180 +0x2ac
net/http.HandlerFunc.ServeHTTP(0xc0026b0078, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/rancher/pkg/auth/audit.auditHandler.ServeHTTP(0x5af2fa0, 0xc0026b0078, 0x0, 0xc0026ae280, 0x5b5ce40, 0xc00e16d500, 0xc008ea3500)
	/go/src/github.com/rancher/rancher/pkg/auth/audit/filter.go:57 +0x53c
github.com/rancher/steve/pkg/auth.ToMiddleware.func1.1(0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/go/pkg/mod/github.com/rancher/steve@v0.0.0-20210520191028-52f86dce9bd4/pkg/auth/filter.go:167 +0x1c7
net/http.HandlerFunc.ServeHTTP(0xc00257e690, 0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/dynamiclistener/server.wrapHandler.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/go/pkg/mod/github.com/rancher/dynamiclistener@v0.3.1-0.20210616080009-9865ae859c7f/server/server.go:193 +0x95
net/http.HandlerFunc.ServeHTTP(0xc002589ad0, 0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/rancher/dynamiclistener.HTTPRedirect.func1(0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/go/pkg/mod/github.com/rancher/dynamiclistener@v0.3.1-0.20210616080009-9865ae859c7f/redirect.go:20 +0xa4
net/http.HandlerFunc.ServeHTTP(0xc00294a9a8, 0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/usr/local/go/src/net/http/server.go:2069 +0x44
net/http.serverHandler.ServeHTTP(0xc001b5d960, 0x5b5ce40, 0xc00e16d500, 0xc008ea2b00)
	/usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc009abd360, 0x5b71080, 0xc007f1b3c0)
	/usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3013 +0x39b

Rancher version: `2.6 master-head (07/29/2021) 94bca99
Originally I was testing this with a low value in this Keycloak setting: SSO Session Idle
This allowed me to observe the panic above because the sessions were getting expired as expected and the panic was thrown.
On the latest master build tested (see first line of this comment) using the same setting I no longer see the panic in Rancher instead the user gets a 401 response and gets redirected back to the login page as the session is expired.
{"baseType":"error","code":"Unauthorized","message":"invalid token","status":401,"type":"error"}
That validated the panic fix. But the Keycloak setting to use for testing the Session refresh should be Access Token Lifespan
Citing dev conversation:

... a proper test here would be to turn the access token lifespan to less than the session idle time (its currently set to 20 minutes).

This allowed me to observe the searches working after the Token Lifetime time is passed as it triggers the token refresh that allows the API request to continue working.
I've opened a separate issue for token refresh improvements:
https://github.com/rancher/rancher/issues/33852


tilt-schema.json link needs to be updated

https://github.com/Transparency-Information-Language/schema/blob/master/documentation/markdown/tilt-schema-properties-meta.md

What exactly do you mean by that?

https://github.com/Transparency-Information-Language/schema/blob/master/documentation/out/tilt-schema.json
this link isn't working anymore


fix: update button

:tada: This PR is included in version 1.2.21 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Added geostationary orbit creation method.

The method is supposed to be called with
-An attractor and
-Attractor's rotational velocity or period
-Hill radius (optional)
Issue: #514

Thanks and congratulations @hrishikeshgoyal for your first contribution! :clap:


Update color scheme for any tab/pane on the fly

Description of the new feature/enhancement

I'd like to be able to set the color scheme for any tab or pane I have open in Windows Terminal with a command. If I'm temporarily opening multiple instances of the same profile, I'd like to distinguish them by setting the color scheme for that instance to any of the available color schemes I have. The alternative to this would be to go into settings and duplicate profiles, but with different color schemes etc.
I don't believe this has been raised here so far. If it's possible, I have not found it in any documentation yet.
Proposed technical implementation details (optional)

I think this is not yet possible because I don't think are Windows Terminal commands from within Windows Terminal yet. I know you can run wt.exe with some options to configure how a new instance of Windows Terminal is launched.
Assuming there will be a single executable for commands in the future from within Windows Terminal, something like:
<executable.exe> --color "Campbell"

would be nice.
Or even:
wt-color "Campbell"

So, colortool can't quite yet read color schemes that you have installed in the terminal settings, but I think you can usually find a colortool compatible version of most colorshemes for the Terminal.
Then the workflow with colortool is just colortool -x Campbell.
Then there's also #4472 for "run commands in the current window", and #5401 for "add a shortcut action to switch to a specific color scheme", which will then pair well with #2046/#5400 for "I want to be able to run an action in the current window" (from the command palette). All those together will combine to let you do something like Ctrl+shift+P, "Change colorscheme", (pick the colorscheme), similar to how vscode works.

@zadjii-msft thanks for all the details and links!
I think colortool would help me to start.
#4472/#5401 is more ideal, and picking colorscheme from a command palette would be most ideal.
I have some custom schemes in Terminal settings that would be nice to use instead of figuring out how to hook them up with colortool.

lets see how it goes

The idea of having the colortool read schemes from the Terminal settings was suggested once before in issue https://github.com/microsoft/terminal/issues/3687#issuecomment-558175555.

Okay, since the above solutions sound good, I'll close this in favor of the linked threads. Thanks!


display custom columns

Part of https://github.com/uwcirg/cosri-patientsearch/issues/143
allow configurations of custom columns.
A config, FHIR_REST_EXTRA_PARAMS_LIST is added to allow specification of additional FHIR resource(s) needed for each patient.
For instance, tried, for DCW:
FHIR_REST_EXTRA_PARAMS_LIST = os.getenv("FHIR_REST_EXTRA_PARAMS_LIST", ["QuestionnaireResponse?_elements=resourceType,authored,subject&_sort=-authored"])
DASHBOARD_COLUMNS = [
        {
            "label": "MRN",
            "expr": "$.identifier[?(@.system=='https://hl7.org/mrn')].value",
            "sorting": False
        },
        {
            "label": "First Name",
            "expr": "$.name[0].given[0]",
        },
        {
            "label": "Last Name",
            "expr": "$.name[0].family",
        },
        {
            "label": "Birth Date",
            "expr": "$.birthDate",
        },
        {
            "label": "Last Assessed",
            "expr": "$.resources[?(@.resourceType=='QuestionnaireResponse')].authored",
             "sorting": False,
        },
    ]

Note, I tried _revinclude parameter to include additional resources, example.  However, including such parameter returns a flat array of Patient resources and other resources, which I fear may mess up paging.  Another issue with using _revinclude or _include parameters to query for additional FHIR resources for the patient is that one cannot seem to sort the additionally included resource(s).  This is problematic if sorting the additional results is important (e.g. for DCW, to obtain last assessed date, one needs to sort the questionnaireResponses by authored date in descending order).
As a consequence, it seems the frontend needs to separately query for additional needed resources.
Also noting that sorting by MRN nor Last Assessed doesn't seem feasible via FHIR query.

We decided to populate these fields, have them all sortable except for "last assessed", since that's very tricky.


Chapter 4 Opening Scan doesn't work after completing the section once and skipping to it again

When skipping to it again, seems like the first scan only allows you to scan at the outpost location rather than the marked one. After doing so it doesn't fullfil the goal either and doesn't continue.

video of scan working on riki but not on requested location
https://streamable.com/u7vd8g


fix pausing without app instance

On android, pause event can come in early, before the app instance is
created.
fixes #6400

I'm not very familiar with how we handle pausing, but does App know how to deal with starting while it is paused? Because the previous code always terminated the app if it wasn't started yet, but with the changes the app will be able to start while it is paused.

@matham I don't know much about it either, but it seems to me that when paused, the whole kivy app is stuck in this loop: https://github.com/Fak3/kivy/blob/074e14ed4b4a9b7dccf3d28996c47b875cd5b51a/kivy/core/window/window_sdl2.py#L485 so the app instance will not be created until it is unpaused.

Having same issue, could you put your code as PEP8 so Travis stop fail ?
My App close itself sometimes for this reason...

I have both an on_pause and on_resume function in my app class which work fine normally, so why is this issue still persisting? Should I basically restart everything (as in reschedule, reload screen, reset variables) within my on_resume function?

Thank you Fak3!!!!!!! This bug was driving me crazy. My app closed without any error and I did not have any Idea on how to fix. I applyed this patch and now it works. It is unreasonable to not have fixed this more than a year after this `patch.
--- a/kivy/core/window/window_sdl2.py	2020-08-06 15:43:26.000000000 +0200
+++ b/kivy/core/window/window_sdl2.py	2020-10-04 18:25:42.000000000 +0200
@@ -10,13 +10,13 @@
     - support scrolling
     - clean code
     - manage correctly all sdl events

 '''

-__all__ = ('WindowSDL', )
+__all__ = ('WindowSDL2', )

 from os.path import join
 import sys
 from kivy import kivy_data_dir
 from kivy.logger import Logger
 from kivy.base import EventLoop
@@ -226,17 +226,15 @@
             self.dispatch('on_memorywarning')

         elif action == 'app_willenterbackground':
             from kivy.base import stopTouchApp
             app = App.get_running_app()
             if not app:
-                Logger.info('WindowSDL: No running App found, exit.')
-                stopTouchApp()
-                return 0
+                Logger.info('WindowSDL: No running App found, pause.')

-            if not app.dispatch('on_pause'):
+            elif not app.dispatch('on_pause'):
                 Logger.info(
                     'WindowSDL: App doesn\'t support pause mode, stop.')
                 stopTouchApp()
                 return 0

             self._pause_loop = True
@@ -245,13 +243,14 @@
             # on iOS, the did enter foreground is launched at the start
             # of the application. in our case, we want it only when the app
             # is resumed
             if self._pause_loop:
                 self._pause_loop = False
                 app = App.get_running_app()
-                app.dispatch('on_resume')
+                if app:
+                    app.dispatch('on_resume')

         elif action == 'windowresized':
             self._size = largs
             self._win.resize_window(*self._size)
             # Force kivy to render the frame now, so that the canvas is drawn.
             EventLoop.idle()
@@ -704,17 +703,14 @@
     def do_pause(self):
         # should go to app pause mode (desktop style)
         from kivy.app import App
         from kivy.base import stopTouchApp
         app = App.get_running_app()
         if not app:
-            Logger.info('WindowSDL: No running App found, exit.')
-            stopTouchApp()
-            return
-
-        if not app.dispatch('on_pause'):
+            Logger.info('WindowSDL: No running App found, pause.')
+        elif not app.dispatch('on_pause'):
             Logger.info('WindowSDL: App doesn\'t support pause mode, stop.')
             stopTouchApp()
             return

         # XXX FIXME wait for sdl resume
         while True:
@@ -729,14 +725,15 @@
                 EventLoop.quit = True
                 break
             elif action == 'app_willenterforeground':
                 break
             elif action == 'windowrestored':
                 break
-
-        app.dispatch('on_resume')
+
+        if app:
+            app.dispatch('on_resume')

     def _update_modifiers(self, mods=None, key=None):
         if mods is None and key is None:
             return
         modifiers = set()


Upgrade container and dependency versions

Upgrade container and dependency versions
Containers:

AWS LocalStack 0.10.8 to 0.12.16
Cassandra 3.11.6 to 4.0.0
ClickHouse <IP_ADDRESS> to <IP_ADDRESS>
Confluent Platform 5.5.1 to 6.2.0 (Kafka 2.5 to 2.8)
Couchbase 6.5.1 to 7.0.0
Elasticsearch 7.7.0 to 7.13.4
Google Cloud SDK 257.0.0 to 350.0.0
Grafana 7.1.3 to 8.0.6
InfluxDB 1.4.3 to 2.0.7
Keycloak 12.0.4 to 15.0.0
MariaDB 10.3.6 to 10.6.3
MinIO 2020-03-25 to 2021-07-12
MongoDB 4.2.0 to 5.0.0
MySQL 5.7.22 to 8.0.25
Neo4j 4.2.1 to 4.3.2
PostgreSQL 12.7 to 13.3
Prometheus 2.10.0 to 2.28.1
Pulsar 2.6.1 to 2.8.0
RabbitMQ 3.8.17 (same version but locked to 3.8.x now)
Redis 4.0.12 to 6.2.4
Vault 1.4.0 to 1.8.0
VoltDB 8.3.3 to 9.2.1

SDK / Clients:

aws-java-sdk-s3 1.11.890 to 1.12.37
camel-spring-boot-starter 3.6.0 to 3.11.0
clickhouse-jdbc 0.2.4 to 0.3.1-patch
couchbase-client 1.4.13 to 3.2.0
kafka-clients 2.6.0 to 2.8.0
keycloak-core 12.0.4 to 15.0.0
mariadb-java-client 2.7.1 to 2.7.3
minio 7.1.4 to 8.3.0
mysql-connector-java 8.0.22 to 8.0.26
neo4j-ogm-core 3.2.19 to 3.2.25
postgresql 42.2.18 to 42.2.23
pulsar-client 2.6.2 to 2.8.0

Dependencies:

equalsverifier 3.5 to 3.7
maven 3.6.1 to 3.8.1
maven-gpg-plugin 1.6 to 3.0.1
maven-javadoc-plugin 3.2.0 to 3.3.0
maven-wrapper 0.5.5 to 0.5.6
spring-boot 2.4.1 to 2.5.3
spring-cloud 2020.0.0 to 2020.0.3
spring-cloud-gcp 1.2.6 to 1.2.8
testcontainers 1.15.1 to 1.16.0

thanks for bumping version to latest, we would need some time to review it tho =)

thanks for bumping versions to the latest, we would need some time to review it tho =)

No pressure, took already long enough to get everything working and into one minimal commit ;)
One test run takes 90 minutes locally, and then GitHub fails, man... those maven logs are even harder to read in that "console".

yes, there is some room for improvement in the project build process, as new modules keep coming... and build time same as log size keep growing.

yes, there is some room for improvement in the project build process, as new modules keep coming... and build time same as log size keep growing.

Absolutely, the build / test logs are far too big, and especially the Kafka test takes too much time, the biggest one for me locally.
I will disable Kafka tests temporarily to see if all others work, even with the same command line from the logs I still can see no problem locally.

thanks for bumping versions to the latest, we would need some time to review it tho =)

Docker was designed without any security concept, file permissions of mounted volumes are a real mess :(
Everything is working now, you can checkout and test for yourself if you want.
When I find time I may try out if user namespace remapping would improve the current test solution.

Looking forward to seeing this released.

Looking forward to seeing this released.

Thank you, me too, maybe you can ask for a review, ll tests are green :)

summoning @Aloren

Please rebase the branch to develop.

Please rebase the branch to develop.

Done, can you try yourself and resolve all conversations if possible?
Please squash before accepting and merging.

@reneleonhardt Okay. Please next time don't make such big PRs, it is very difficult to review. Please refer to https://github.com/Playtika/small-pull-request-manifesto

@reneleonhardt Okay. Please next time don't make such big PRs, it is very difficult to review. Please refer to https://github.com/Playtika/small-pull-request-manifesto

I didn't know about your manifesto, ok.
You squashed with the pre-filled test (all commit messages), keep in mind that I updated this PR description manually every time, so you could have copied the end result from there (i.e. Keycloak 15.0.0 instead of 13.0.1).


HKey -  slice out of bounds error - with dmap put

When calling dmap.put with the embedded client, I am getting an out-of-bounds error in hkey.go(actually in the hasher, but this is the root of the error), changing the code to
func HKey(name, key string) uint64 {
	tmp := name + key
	return hashFunc.Sum64([]byte(tmp))
}

resolves the error.

Hi @mattdurham
Thank you for reporting this.
Which Olric version are you using? How can I reproduce this error? I have never seen this error. The current version of the HKey function uses unsafe package to reduce memory consumption. Removing this will increase the GC pressure. So I need the input you use to reproduce the problem.

More interesting, it fails if I have the debugger attached but succeeds if not.

I think it's a behavior on M1, my x64 box exhibited no bad behavior. Alright if I submit a PR to revert to slower method if its arm arch/os osx?

I managed to reproduce the error on a Macbook with an M1 processor.
As you can see, there is a bug in cespare/xxhash. We should report this bug to the author.
You can also use a different hash implementation by implementing the hasher.Hasher interface. config.Config has an Hasher field.
Configuration: https://github.com/buraksezer/olric/blob/master/config/config.go#L258
The default hasher implementation: https://github.com/buraksezer/olric/blob/e0a16290aa590b4bdf441176ca6b9510421eb763/hasher/hasher.go#L21
What do you think? By the way, I would like to be happy to learn more about your use case, if it's possible to talk about it publicly.
Here is the output:
=== RUN   TestClusterClient_Delete
2022/05/25 00:52:26 [INFO] Olric 0.5.0-beta.3 on darwin/arm64 go1.18.1 => olric.go:313
2022/05/25 00:52:26 [DEBUG] memberlist: Using dynamic bind port 49369
2022/05/25 00:52:26 [INFO] Join completed. Synced with 0 initial nodes => discovery.go:59
2022/05/25 00:52:26 [INFO] Routing table has been pushed by <IP_ADDRESS>:49368 => operations.go:86
2022/05/25 00:52:26 [INFO] The cluster coordinator has been bootstrapped => discovery.go:42
2022/05/25 00:52:26 [INFO] Memberlist bindAddr: <IP_ADDRESS>, bindPort: 49369 => routingtable.go:413
2022/05/25 00:52:26 [INFO] Cluster coordinator: <IP_ADDRESS>:49368 => routingtable.go:414
2022/05/25 00:52:26 [INFO] Node name in the cluster: <IP_ADDRESS>:49368 => olric.go:374
2022/05/25 00:52:26 [INFO] Olric bindAddr: <IP_ADDRESS>, bindPort: 49368 => olric.go:380
2022/05/25 00:52:26 [INFO] Replication count is 1 => olric.go:382
    olric_test.go:110: A new cluster member has been created: <IP_ADDRESS>:49368
panic: runtime error: slice bounds out of range [::11] with capacity 0

goroutine 41 [running]:
github.com/cespare/xxhash/v2.Sum64({0x140000298f0, 0xb, 0x0})
	/Users/buraksezer/go/pkg/mod/github.com/cespare/xxhash/v2@v2.1.2/xxhash_other.go:41 +0x3f0
github.com/buraksezer/olric/hasher.xxhasher.Sum64({}, {0x140000298f0, 0xb, 0x0})
	/Users/buraksezer/go/src/github.com/buraksezer/olric/hasher/hasher.go:28 +0x3c
github.com/buraksezer/olric/internal/cluster/partitions.HKey({0x14000026894, 0x6}, {0x140000268a0, 0x5})
	/Users/buraksezer/go/src/github.com/buraksezer/olric/internal/cluster/partitions/hkey.go:37 +0x90
github.com/buraksezer/olric/internal/dmap.(*DMap).put(0x140001ce200, 0x14000192600)
	/Users/buraksezer/go/src/github.com/buraksezer/olric/internal/dmap/put.go:357 +0x6c
github.com/buraksezer/olric/internal/dmap.(*Service).putCommandHandler(0x140001682d0, {0x1013ffa48, 0x140000ed880}, {{0x14000026880, 0x34, 0x40}, {0x14000094de0, 0x4, 0x4}})
	/Users/buraksezer/go/src/github.com/buraksezer/olric/internal/dmap/put_handlers.go:62 +0x3f4
github.com/buraksezer/olric/internal/server.Handler.ServeRESP({0x14000110a30, 0x140001108a0}, {0x1013ffa48, 0x140000ed880}, {{0x14000026880, 0x34, 0x40}, {0x14000094de0, 0x4, 0x4}})
	/Users/buraksezer/go/src/github.com/buraksezer/olric/internal/server/handler.go:68 +0x4b0
github.com/buraksezer/olric/internal/server.(*ServeMux).ServeRESP(0x1400011c088, {0x1013ffa48, 0x140000ed880}, {{0x14000026880, 0x34, 0x40}, {0x14000094de0, 0x4, 0x4}})
	/Users/buraksezer/go/src/github.com/buraksezer/olric/internal/server/mux.go:73 +0x11c
github.com/tidwall/redcon.handle.func2(0x140000ed880, 0x140001579d0)
	/Users/buraksezer/go/pkg/mod/github.com/tidwall/redcon@v1.4.5/redcon.go:429 +0x3b8
github.com/tidwall/redcon.handle(0x140001579d0, 0x140000ed880)
	/Users/buraksezer/go/pkg/mod/github.com/tidwall/redcon@v1.4.5/redcon.go:442 +0x94
created by github.com/tidwall/redcon.serve
	/Users/buraksezer/go/pkg/mod/github.com/tidwall/redcon@v1.4.5/redcon.go:378 +0x53c

Debugger finished with the exit code 0

Thank you @mattdurham for the update. Please don't hesitate to ask questions about the design of Olric and potential integration problems. We also have a Discord channel, it's good for instant communication: https://discord.com/invite/ahK7Vjr8We

type fixedhasher struct{}

func (m fixedhasher) Sum64(key []byte) uint64 {
	return xxhash.Sum64([]byte(string(key))) // use xxhash.Sum64([]byte(string(key))) replace xxhash.Sum64(key) can fixed this problem
}


tox: Should the py26 testenv be kept ?

As a potential contributor, the first thing I do when starting to work on a project is run the test suite.
When I clone tower-cli and run tox I end up with:
docs: commands succeeded
flake8: commands succeeded
ERROR:   py26: commands failed
py27: commands succeeded
py34: commands succeeded
py35: commands succeeded
py36: commands succeeded

They are 208 occurrences of the same failure: ValueError: zero length field name in format.
(ie. prefix = urljoin(prefix, "{}/".format(CUR_API_VERSION)))
It happens because the format expression has been used without indices. Expression that has become valid in py27+ and py32+.
So should the py26 tox testenv be kept?

In case it shouldn't be kept I made #605 that removes it.


renaming function argument from grad to jac

for the sake of consistency with the other optimisers it would be good if the argument for the gradient in _minimize_trustregion_constr would be called 'jac' instead of 'grad'

It'd say it's 50:50 here. It's in a private function, so it shouldn't matter if it's renamed. But then again I'm not 100% that it's worth the churn. @Karrenbelt would you be interested in making a PR?

Actually, I think there would be issues renaming it to jac because that's also used for the jacobian of the constraints. I'd say it's better to leave it as is.

Won't fix, in a private function.


Increase controller resyncPeriodSeconds

The resyncPeriodSeconds in manifests/controller.yaml is currently set to 5 which means kubepost applies each resource in postgres every 5s.
This seems quite a bit excessive to me and causes unnecessary load on the Postgres server and log output.
Is there a reason it needs to be this short instead of something around 5min - 15min?

This will be a priority in the near future. Currently the goal is to create a new CRD that will be used by the operator to configure itself.


Updated link to docs

Signed-off-by: Kyle Davis<EMAIL_ADDRESS>Description
Adds a more specific URL to the docs link in the README
Check List

[x] Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

D'oh. Looks like it's a dupe of #78

closing this since it is already merged.


Benchmarks, stress test, memleak tests

It'd be good to put some hard numbers out, especially for memleaks and the speed of resolving relations is complex machines.

Closed by https://github.com/pancsta/go-libp2p-pubsub-benchmark


Add gradle imports for latest version

Please add import lines for the latest alpha versions as well in the README file.
testImplementation 'com.nhaarman.mockitokotlin2:mockito-kotlin:2.0.0-alpha04'

9d642a26188b545c1ef83615cea42d4bf2ee5985


SoC format

For some reason my SoC is not availiable, so it returns “Unknown”. Would it be an idea to let it return a number to make the Gauge Card happy?

I think it depends on your car what is shown here, my returns an None as well -> hence, not a number. I don't see a universal fix here that would work for everyone, you can however create a template sensor in HA.


Add docker ps filter for image

Makes it possible to filter containers by image, using --filter=image=busybox or --filter=image=8c2e06607696. This is related to #13365, and for the image filter, it's actually ask sometimes on irc 🐸.
This felt a little too simple to implement, so maybe I'm missing something :sweat_smile:.
Signed-off-by: Vincent Demeester<EMAIL_ADDRESS>

what about containers that have that image the distant history?  e.g. the image in question was used to make another image and that's 2nd image is the one used to create the container.

Ah :stuck_out_tongue_closed_eyes:, this would make it way more complicated to implement then :sweat_smile:, and I'm not sure how you would handle from a user point of view.
Given that it was implement
And I have 3 images : base -> middle -> final
When I'm doing ``--filter=image=middle``
Then should it filter all containers based on middle and final, or just those with final ?

I think the simpler the better (just filter with the image used when starting the container). And I don't really see a use case for distant history — I might be wrong though.

I only see (potential) value is we walk the ancestry tree, the easy case I can pretty much grep :-( Not convinced.

@icecrime On the other hand I'm not convinced on the grep part :blush: (although I like grep a lot :smiley_cat:). There is a few cases where I wouldn't find grep useful:

Same or common part of the name shared by some images and some containers :

$ docker ps -a | grep busybox
5c33c99ac89f        busybox         […]                         toto
5ba009755259        maven           […]                         busybox1
2aff61195a5a        maven           […]                         busybox

The ability to chain it. Let's say I want to stop all container based on the busybox images : with the filter it would be docker ps -aq --filter=image=busybox | xargs docker stop, without docker ps -a | grep busybox | awk '{ print $1 }' | xargs docker stop (but as seen in the previous point I could get false positive).

I do agree these are probably edge cases and might not happen a lot in real life or not that much but it felt so easy to implement.
I also do see potential on the "walk the ancestry tree" but I'm not sure how it would look in a user point of view. Question like :

Do I always want to walk the ancestry tree (and this, filtering on a base image would return all container running these or images based on it) or not ?
If not, how would I do to specify it needs to walk the tree or not ? (like --filter=image=debian&children)
Is there a real use case for that ? I don't know but maybe.

I only see the ancestry usecase as being interesting so no need for a flag to say "look at entire history", it should do it automagically.

I can see use cases for taking ancestry when filtering, but I really wonder if this is what I would expect if I filter name=foo. I think I would expect to only get containers started from foo. Perhaps another flag to include ancestry?

sorry name=foo should obviously be image=foo in my previous comment :-)

the reason I'm not sure "image=foo" is that interesting is because there isn't much of a difference between:
docker run -ti ubuntu some-command

and
docker build -f Dockerfile-FROM-ubuntu+some-command ...
docker run ...

I guess it might be a little useful to have: image=foo and ancestor=foo but I wonder how quickly people will opt for the ancestor variant once they realize they really want to look for more than the immediate parent image.

but I wonder how quickly people will opt for the ancestor variant once they realize they really want to look for more than the immediate parent image.

True, in many cases, either immediate parent or "uses" an image are what you're looking for. From a UX perspective though, I'm really against calling the "ancestor" filter "image", because it doesn't reflect the intent and will definitely lead to a lot of support questions / invalid bug reports.
We can opt for not implementing immediate parent (for now) and only implement ancestor search once people start asking for it. Perhaps an easier name for the filter, would "inherits" be a good name? --filter inherits=ubuntu:14.04
To the filtering itself, what kind of searches should be supported? A quick list;

image
image:tag
image:tag@digest
short-id
full-id

I definitely +1 to more filters.
Grep is nice, but not particularly nice in this case.

Collective review
@duglin @calavera @tonistiigi
We would prefer the tree-traversal implementation instead of just filtering with a direct parent image name.
Because in the inspect output, Image refers to the direct parent image, we should not call the filter image, and we suggest ancestor instead (less ambiguous).
As for answering https://github.com/docker/docker/pull/14570#issuecomment-121414249, we should support all of the input forms. They will have to be mapped to a full image ID anyways.
Let's leave the "direct-parent-only" implementation for another time, as we seem to agree the tree-traversal is more useful.

SGTM, thanks!

SGTM too, I'll work on that then :wink:

Updated the PR with an implementation of the ancestor filter :wink:.

I think we need doc updates.

@duglin took care of your comments. Implementation with a map feels easier to read (less for loops :smile_cat:).
I was waiting for a few reviews before updating docs :stuck_out_tongue_closed_eyes:, that will be my next task :wink:.

LGTM

looks good, just one question.

Oh I though I pushed the docs (which are quite small) but didn't.. :sweat_smile:. /cc @thaJeztah

Updated with to take care of @duglin and @thaJeztah comments, I think :sweat_smile:.

docs LGTM, as discussed with @vdemeester,  examples will be added in a filtering chapter/section https://github.com/docker/docker/pull/14570#discussion_r37805119

@vdemeester LGTM

Moving to docs review.

LGTM per my previous comment https://github.com/docker/docker/pull/14570#issuecomment-134394517

ping @moxiegirl, looks like we merged the "follow up" before the one it's "following up" :smile:

@thaJeztah yay :sweat_smile: :stuck_out_tongue_closed_eyes:

This is why I should leave the merging to others (@thaJeztah) ...

@moxiegirl counting that as a LGTM, merging!

:tada: :wink:


Issue 164: Add setup and install script

Closes #164
On this PR the following changes take place:

Add setup.sh script

checks user OS
installs homebrew for MacOS
sets pyenv to manage python versions/envs
installs python >= 3.10
installs direnv to handle dev local env
installs poetry to manage deps
installs dependencies

Modify TOML pyproject to be poetry compliant
create .envrc file to be used by direnv to set local dev environment
Modify README to instruct use of new script

We aren’t switching to poetry I thought ….

Please remove poetry.

We aren’t switching to poetry I thought ….

Oh, I did not know it was decided, my bad, will remove in a couple of hours (not home ATM)

I’m also more broadly confused about install scripts.  This looks like an excellent candidate install script, I’m uncertain if we should actually be managing a users python environment

butyeah, maybe make a PR to remove the poetry bits as its own thing and we'll merge it int pronto.
I'm open to using poetry, but when I looked into it, it seemed to lack the support for flexible app entry point definitions that vanilla setup-tools pyproject.toml supports, and that seems important

butyeah, maybe make a PR to remove the poetry bits as its own thing and we'll merge it int pronto.
I'm open to using poetry, but when I looked into it, it seemed to lack the support for flexible app entry point definitions that vanilla setup-tools pyproject.toml supports, and that seems important

would you mind giving me more context? Perhaps I can look into it, poetry in my experience has been quite a good tool to use!

I’m uncertain if we should actually be managing a users python environment

I think given this creates a venv with an appropriate python version and installs the project deps in there, we should be fine. (if someone wants to explore or do something else they can create a venv on their own anyway)

@cartazio ping to remind for review when you are available

seemed to lack the support for flexible app entry point definitions that vanilla setup-tools supports

You can use poetry scripts definition to create arbitrary entry points afaik

hey @bjornaer will the bash script exit if user is running windows?

@VictorOdede yes. I added first a MacOS setup support, I'm happy to expand towards other OS' but would rather do that on another PR.

hey @bjornaer will the bash script exit if user is running windows?

@VictorOdede yes. I added first a MacOS setup support, I'm happy to expand towards other OS' but would rather do that on another PR.

the mac setup looks great. just wondering what's stopping you from adding windows/linux support here?

hey @bjornaer will the bash script exit if user is running windows?

@VictorOdede yes. I added first a MacOS setup support, I'm happy to expand towards other OS' but would rather do that on another PR.

the mac setup looks great. just wondering what's stopping you from adding windows/linux support here?

A lot of change all at once; it's easier to review one script rather than 3, since they will probably have different steps I was thinking on having separate files for each OS (and setup.sh as the entry point)

Hey @bjornaer multiple scripts in the PR isn't really a problem as long as they all work. I think it would be better to merge this with Windows and Linux support so that we don't have to open new tickets for each OS

Hey @bjornaer i'm just gonna merge this in and then create new tickets for the windows and linux scripts and you could pick those up if you want. Thanks for this useful contribution😊

Hey @bjornaer i'm just gonna merge this in and then create new tickets for the windows and linux scripts and you could pick those up if you want. Thanks for this useful contribution😊

My bad for the delay on responding @VictorOdede , as you might guess other things in life caught my attention haha, I did start working on the other two versions of the setup script, when you create the issues I'll open PRs for them then!

Hey @bjornaer i'm just gonna merge this in and then create new tickets for the windows and linux scripts and you could pick those up if you want. Thanks for this useful contribution😊

My bad for the delay on responding @VictorOdede , as you might guess other things in life caught my attention haha, I did start working on the other two versions of the setup script, when you create the issues I'll open PRs for them then!

Hey that's understandable. I've already opened the new tickets so you can continue working on it. However, we are yet to triage the new tickets so we haven't assigned bounties yet


Want to add the function of filling replacement templates in order.

Feature Request

Q
A

New Feature
yes

BC Break
no

Proposal
Want to add the function of filling replacement templates in order.
It is better to support array
for example
 sprintf( 'http://xx.com/{id}/look/{pid}', $id, $pid);

@yinbug What would this feature do that is not yet covered by

UriTemplate feature
Uri component package

The array of the official example is a key value pair. I want to leave out the key names and fill them in order.
This is just an idea of mine.
Official example:
`
 '42', 'hotel' => 'Rest & Relax'];

$uriTemplate = new UriTemplate($template);
$uri = $uriTemplate->expand($params);
// $uri is an League\Uri\Uri instance.

echo $uri, PHP_EOL;
`

Fill in order:
`
expand($params);
`

Still I fail to see the added value. Why can't you just fill in the names of the arguments ?

Maybe I want to be lazy. I don't want to input so many parameters.


REPORT - Adding ReadMe file

Application guideline will be added to ReadMe File.

We shouldn't add it to readme file. Readme is only for basic setup and information about the project.


Disclaimer about permitted file formats in the instructions.

While doing some testing today I realized that Orvium still shows a red warning while uploading anything else then .docx. We should add a disclaimer that this warning can be ignored in case of .odt and .zip to our instructions on this website. Otherwise people might get confused or feel insecure about using .odt or latex.
That's how it looks in Orvium:

Can you report that to the Orvium devs? That seems problematic even if we have the disclaimer here on this site.

Actually I just saw that there are new buttons in the settings for odt and zip:

I've now enabled them.


Release v2.11.2

This is an auto-generated release PR for version 2.11.2.

closing, will point specifically to 2.11.2 backend for e2e tests


Fix WL zone deployment by pipeline

Problem
The install_workloadzone.sh is not called by the 02 wlzone pipeline leveraging self hosted agent rather then deployer VM.
Solution
Move the install_workloadzone.sh call out of the  if [ -f /etc/profile.d/deploy_server.sh ]; then
Tests

Notes

@ZsoltParoczi , we are working internally on an overhaul of the pipeline code. Some of the items mentioned here are already part of the new changes being made. Would request you to move this PR to draft state until you have a chance to review the new changes.


Allow custom compiled scripts

With Kotlin we are able to place script plugins into the buildSrc dir. Those plugins become precompiled.
At this point, this is the only usable way in which we can develop custom script plugins which use other Gradle plugins. Just placing a .gradle.kts file and applying it via apply(from = "...") is flawed, as the file cannot use any type-safe accessors for the Gradle model.
In this repository, we prepare the buildSrc project for those plugins.
In library repositories, we may define custom script plugins and place them under buildSrc/src/main/kotlin. For example, custom-jar-magic.gradle.kts:
plugins {
    java
}

tasks.jar {
    exclude("**/*.properties")
}

Then, such plugins become available to the target projects. For example, spine-lib/build.gradle.kts:
plugins {
    `custom-jar-magic`
}

Later we will migrate the most (or all) the script plugins in this repository (which are currently in Groovy) into precompiled script plugins.

@armiol, PTAL.


Possible to extend the "secondary_info" line to include entity attributes

As the title suggests would it be possible (or is there a plan) to extend the secondary_info of an entity to allow additional attributes of that entity?
Or even better to allow an attribute of any entity at all?
My use case is showing battery levels and/or calendar dates in the google calendar entity on the frontend under the entity.
I can do it now using the custom-ui by andrey but it doesn't seem to work in lovelace.

We are not planning on implementing this feature. it has been discussed before here: https://github.com/home-assistant/home-assistant-polymer/pull/2089
Decision is final.


refactor: remove incorrect typescript references in comments and code

Description
Housekeeping: clearly some stuff was left behind by copying from another template.
And a few trailing spaces have been removed by standard IDE linting.

/rtm


added error messages are uninformative

in https://github.com/ploomber/jupysql/pull/400 we added some extra information whenever a query raises an exception. however, I've noticed that in many cases, these error aren't informative. Here are a few examples, I purposely added errors to my SQL queries:
In [4]: %sql select2 * from penguins.csv
Running query in 'duckdb://'
UsageError:
Looks like there is a syntax error in your query.Possible reason:
Syntax Error in select2 * from penguins.csv: Required keyword: 'expression' missing for <class 'sqlglot.expressions.Mul'> at Line 1, Column 14

Original error message from DB driver:
(duckdb.ParserException) Parser Error: syntax error at or near "select2"
LINE 1: select2 * from penguins.csv
        ^
[SQL: select2 * from penguins.csv]
(Background on this error at: https://sqlalche.me/e/14/f405)

If you need help solving this issue, send us a message: https://ploomber.io/community

In [5]: %sql zelect * from penguins.csv
Running query in 'duckdb://'
UsageError:
Looks like there is a syntax error in your query.Possible reason:
Syntax Error in zelect * from penguins.csv: Required keyword: 'expression' missing for <class 'sqlglot.expressions.Mul'> at Line 1, Column 13

Original error message from DB driver:
(duckdb.ParserException) Parser Error: syntax error at or near "zelect"
LINE 1: zelect * from penguins.csv
        ^
[SQL: zelect * from penguins.csv]
(Background on this error at: https://sqlalche.me/e/14/f405)

If you need help solving this issue, send us a message: https://ploomber.io/community

In [6]: %sql  * from penguins.csv
Running query in 'duckdb://'
UsageError:
Looks like there is a syntax error in your query.Possible reason:
Syntax Error in * from penguins.csv: Invalid expression / Unexpected token at Line 1, Column 6

Original error message from DB driver:
(duckdb.ParserException) Parser Error: syntax error at or near "*"
LINE 1: * from penguins.csv
        ^
[SQL: * from penguins.csv]
(Background on this error at: https://sqlalche.me/e/14/f405)

If you need help solving this issue, send us a message: https://ploomber.io/community

Our custom errors come right after "looks like there is..." but as you can see they're either cryptic or non-informative. We should fix this and ensure that whenever we are adding more context, we're sure that it'll help the user.

fixed in https://github.com/ploomber/jupysql/pull/705


Question

Could you add an addon for audiobooks please. im a big fan of your addons there very helpful but im missing some kind off tool for audiobooks. maybe docker-booksonic-air.
sorry if this is not the way to ask but i didn't know another way.

Hi, I've pushed a version, could you please test if everything works and if it functions as you would expected ? thanks

It's a bit long to start but should work

Wow thats fast thanks. I’ll test it little bit I’m on my way home right now.

yes it works nice. first run takes a while but after then it runs smooth. thanks your the best

Thanks! did you already restart it? To test for persistence. Thanks!

Yes no issues. It works great maybe i shoud mention i don’t use the smb option. I have all my drives mounted in the hassio/share folder. im running home assistant supervised on debian. I ran it first in docker but then home assistant gets unsupported i know thats no real issue but i like this better. Thank you 🙏

Thanks for the precisions. I'll close this as it works! Enjoy!


Fix StackScript typo

Closes https://github.com/linode/developers/issues/117
See https://github.com/linode/python-linode-api/blob/master/linode/objects/init.py#L8

Oops, thanks


Why is the output garbled

From the screenshot, it seems that you are using the vicuna version of the videochat model. Please ensure that the weights of your vicuna-7b-v0 are loaded correctly.
Load the correct weight you can refer to https://github.com/OpenGVLab/Ask-Anything/issues/224#issuecomment-2301333331


feat: Add LICENSE

Required for packaging on conda-forge, and should be included for general use.

When the sdist is built during packaging, the LICENSE file gets picked up, and then when the wheel is built from the sdist it is included in the wheel metadata directory.

MIT is chosen to match the stated license in the setup.py metadata:

https://github.com/iris-hep/func_adl/blob/5f23cfb72a841fcb905af8b66017f8b6851abde3/setup.py#L76

Required for https://github.com/ssl-hep/ServiceX_frontend/issues/352

Thanks @gordonwatts. Would you be able to make a v3.3.2 patch release for this to get picked up by https://github.com/ssl-hep/ServiceX_frontend/issues/352?

@gordonwatts Though this week is busy, could you make a new patch release?


v0.13.0 as-installed emits interrupt! with STATE, which doesn't match repo

I've got svd2rust 0.13.0 installed:
> svd2rust -h
svd2rust 0.13.0 ( )
Generate a Rust API from SVD files

USAGE:
    svd2rust [FLAGS] [OPTIONS]
...

It emits the following interrupt macro:
macro_rules! interrupt {
    ($Name:ident, $handler:path,state: $State:ty = $initial_state:expr) => {
        #[allow(unsafe_code)]
        #[deny(private_no_mangle_fns)]
        #[no_mangle]
        pub unsafe extern "C" fn $Name() {
            static mut STATE: $State = $initial_state;
            let _ = $crate::Interrupt::$Name;
            let f: fn(&mut $State) = $handler;
            f(&mut STATE)
        }
...

...but the docs and the repo refer to locals.

I'm not sure I understand what this issue is for, the macro you are showing is in interrupt.rs#L126
The exported interrupt! macro is an artifact from earlier versions, since svd2rust generates it for you now depending on target.

@thenewwazoo

but the docs

says that the interrupt! macro looks different on Cortex-M. Quoting the docs

NOTE The interrupt! macro on Cortex-M device crates is closer in syntax to the exception! macro. This documentation doesn't apply to it. For the exact syntax of this macro check the documentation of the device crate you are using.

And the device crate properly document the macro. See the stm32f30x for example.

Oh, I see now. I found the "closer in syntax" note to be confusing - the distinction here is between the (apparently historical?) macro in the svd2rust crate itself and the macro it generates. I might reword that as:

NOTE This documentation is for a macro exported only from the svd2rust crate. The interrupt! macro generated by svd2rust for Cortex-M device crates is different from this macro, and mirrors the exception! macro. For that macro, see the documentation for the generated device crate.


Use --config-file instead of --repo-config-file when --no-git is set

The task always uses --repo-config-file, making it impossible to use a custom config file in combination with the --no-git switch. When --no-git is set, one must use --config-file.

Awesome find! Thank you for fixing this bug

Happy to help :)


Create Concept for Linting as a General Mechanism in the Camunda Modeler (API etc.)

[ ] linting is a general concern of Camunda Modeler with generic API
[ ] linting must work with all file types (BPMN, DMN, forms)
[ ] elements that have linting errors can be addressed (e.g. <id>#<property>)
[ ] command format of linting errors and warnings that is used by all linters (bpmnlint, form linter, etc.)
[ ] lint errors contain information about the error and respective element (and property)
[ ] all necessary input for linting is the file itself and set of rules (additional input like execution platform must be contained in file)
[ ] different sets of rules exist for different execution platforms (e.g. with Cloud 1.0 and 1.1 there will be different sets of rules for linting forms)

Child of https://github.com/camunda/camunda-modeler/issues/2323

In order to keep the linting feature generic the idea was that linting happens on XML/JSON basis, meaning the diagram/form will be exported first and than imported again by the respective linter. Following that approach might have an impact on performance as import and export are rather expensive operations. I conducted a couple of tests using bpmnlint. The results are the following:

No Export/Import
Export and Import

Small Diagram (69 Elements)
< 1 ms
< 100 ms

Medium Diagram (545 Elements)
< 1 ms
200 - 250 ms

Large Diagram (1089 Elements)
< 1 ms
300 - 400 ms

The following diagrams were used:
Small Diagram

Medium Diagram

Large Diagram

Conclusion
There might be a noticeable impact on performance. I will implement a prototype of the linting based on XML/JSON and see if it feels responsive enough. Linting should ideally happen in a non-blocking way. The delay between a change in the diagram/form and seeing the linting results could be short enough to be imperceivable especially with small to medium-sized diagrams/forms.


Change the LogStore conf key name

Right now the LogStore conf key is "delta.logStore.class", but this is not getting picked up because we require spark conf to start with "spark.". So we change this to "spark.delta.logStore.class".

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Cannot update labels if entity is loaded more than once in same session

It is not possible to update labels if we reload the entity after changing them on OGM v2.0.5.

Load entity
Update entity's labels
Load entity again
Save --> FAIL

I would expect to keep track of labels until the session is cleaned and regardless the number of times the entity is reloaded during the exact same session.
I've modified a test at PizzaIntegrationTest to verify that.
@Test
public void shouldMarkLabelsAsDirtyWhenExistingCollectionUpdated() {
    Pizza entity = new Pizza();
    List<String> labels = new ArrayList<>();
    labels.add("TestLabel1");
    labels.add("TestLabel2");
    entity.setLabels(labels);
    session.save(entity);
    session.clear();

// Problem is found on this block
    entity = session.load(Pizza.class, entity.getId());
    entity.getLabels().clear();
    entity.getLabels().add("TestLabel3");

    session.load(Pizza.class, entity.getId());  // This will cause the problem

    session.save(entity);
    session.clear();
// End

    labels = session.load(Pizza.class, entity.getId()).getLabels();
    assertEquals(1, labels.size());  // FAIL - labels are the old 1 and 2 values
}

java.lang.AssertionError:
Expected :1  (TestLabel3)
Actual   :2  (TestLabel1 and TestLabel2)

@jdorleans This was intentional. If lack of what you describe above is a show-stopper, we can look at doing something different.

Closing. If this is still an issue please reopen.


gRPC doesn't build on manylinux1 (sched_getcpu not found)

I am building gRPC v1.21.0 from source, on different platforms. When trying with manylinux1 (from dockcross), I get the following error:
[  2%] Building CXX object CMakeFiles/grpc_plugin_support.dir/src/compiler/csharp_generator.cc.o
/work/build/manylinux1-x64_CFLAGS2/grpc/grpc/src/grpc/src/core/lib/gpr/cpu_linux.cc: In function ‘void init_num_cpus()’:
/work/build/manylinux1-x64_CFLAGS2/grpc/grpc/src/grpc/src/core/lib/gpr/cpu_linux.cc:40:20: error: ‘sched_getcpu’ was not declared in this scope
   if (sched_getcpu() < 0) {
                    ^
/work/build/manylinux1-x64_CFLAGS2/grpc/grpc/src/grpc/src/core/lib/gpr/cpu_linux.cc: In function ‘unsigned int gpr_cpu_current_cpu()’:
/work/build/manylinux1-x64_CFLAGS2/grpc/grpc/src/grpc/src/core/lib/gpr/cpu_linux.cc:69:26: error: ‘sched_getcpu’ was not declared in this scope
   int cpu = sched_getcpu();
                          ^

In the past, I fixed that by forcing the definition of GPR_MUSL_LIBC_COMPAT (as can be seen here). But that is not a proper fix.
I have seen that tools/run_tests/artifacts/artifact_targets.py sets manylinux from the CFLAGS:
environ['CFLAGS'] = '-DGPR_MANYLINUX1=1'

But I don't know how that is related to my musl issue above, and setting CFLAGS did not seem to solve this issue.
I created a small repo showing how I build everything here. Here is a way to reproduce the issue:

Build gRPC with cmake -Bbuild/default -S. to convince yourself that it works.
Get dockcross for manylinux1-x64:

    $ docker run --rm dockcross/manylinux1-x64 > ./dockcross-manylinux1-x64

Try to build with dockcross and see the error described above:

    $ ./dockcross-manylinux1-x64 cmake -Bbuild/manylinux1-x64 -S.

Try with CFLAGS set and see that the error stays:

    $ ./dockcross-manylinux1-x64 bash -c 'export CFLAGS="-DGPR_MANYLINUX1=1"; cmake -Bbuild/manylinux1-x64 -S.'

@gnossen: Would you have an advice about that? Am I missing something?

It looks like your flag isn't getting plumbed through. We define GPR_CPU_POSIX on manylinux. This file mentioned in your error should basically be filtered out by the preprocessor if the flag gets plumbed through correctly.
Try adding the flags to the CMAKE_FLAGS variable used in the grpc external project. Like so
list(APPEND CMAKE_ARGS
    "-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_INSTALL_PREFIX}"
    ...
    "-DCMAKE_CXX_FLAGS=-DGPR_MANYLINUX1=1"
    )

That will get you past your undefined symbol. But it looks like there are still some other problems with the build:
_gRPC_CPP_PLUGIN-NOTFOUND: program not found or is not executable
--grpc_out: protoc-gen-grpc: Plugin failed with status code 1.
gmake[5]: *** [gens/src/proto/grpc/status/status.grpc.pb.cc] Error 1
gmake[4]: *** [CMakeFiles/grpc++_error_details.dir/all] Error 2
gmake[4]: *** Waiting for unfinished jobs....
_gRPC_CPP_PLUGIN-NOTFOUND: program not found or is not executable
_gRPC_CPP_PLUGIN-NOTFOUND: program not found or is not executable
--grpc_out: protoc-gen-grpc: Plugin failed with status code 1.
--grpc_out: protoc-gen-grpc: Plugin failed with status code 1.

CC @nicolasnoble

Right, so I was not setting the flag properly, thanks for that! And I also get the grpc_cpp_plugin issue, but I don't get it because the file is there in:
./build/manylinux1-x64/grpc/grpc/src/grpc-build/grpc_cpp_plugin

Oh wait, I believe that's related to the other fix I was doing in the older version (see here). Let me try that.

Yes, I believe that's it: this branch builds successfully with:
docker run --rm dockcross/manylinux1-x64 > ./dockcross-manylinux1-x64
./dockcross-manylinux1-x64 cmake -Bbuild/manylinux1-x64 -S.

I think that's related to a "known peculiarity" (as in: I cannot imagine a scenario where anybody would want CMake to behave like this, but it is documented :sweat_smile:):

[CMAKE_CROSSCOMPILING] will be set to true by CMake if the CMAKE_SYSTEM_NAME variable has been set manually (i.e. in a toolchain file or as a cache entry from the cmake command line). In most cases, manually setting CMAKE_SYSTEM_NAME will only be done when cross compiling, since it will otherwise be given the same value as CMAKE_HOST_SYSTEM_NAME if not manually set, which is correct for the non-cross-compiling case. In the event that CMAKE_SYSTEM_NAME is manually set to the same value as CMAKE_HOST_SYSTEM_NAME, then CMAKE_CROSSCOMPILING will still be set to true.

So my workaround is to point to a fork of gRPC, which applies this patch:
     #if cross-compiling, find host plugin
-    if(CMAKE_CROSSCOMPILING)
+    if(CMAKE_CROSSCOMPILING AND NOT CMAKE_SYSTEM_NAME STREQUAL CMAKE_HOST_SYSTEM_NAME)
        find_program(_gRPC_CPP_PLUGIN grpc_cpp_plugin)
     else()
        set(_gRPC_CPP_PLUGIN $<TARGET_FILE:grpc_cpp_plugin>)

i.e. CMAKE_CROSSCOMPILING AND NOT CMAKE_SYSTEM_NAME STREQUAL CMAKE_HOST_SYSTEM_NAME.
@nicolasnoble @gnossen: Do you have an opinion about that? I would see two ways to patch it:

The way I did it above, but I don't like it because for every if(CMAKE_CROSSCOMPILING), one will have to wonder if it applies to manylinux or not.
At the top of the root CMakeLists.txt, "fix" CMAKE_CROSSCOMPILING (i.e. disable it if CMAKE_SYSTEM_NAME STREQUAL CMAKE_HOST_SYSTEM_NAME). The disadvantage of this is that it is changing the official cmake behavior, but it sounds like a documented bug to me anyway.

I'm thinking that this could be solved in a more elegant way from the manylinux toolchain, actually. I opened an issue in the dockcross repo.
Regarding -DGPR_MANYLINUX1, I see the following lines in your build:
#elif defined(GPR_MANYLINUX1)
// TODO(atash): manylinux1 is just another __linux__ but with ancient
// libraries; it should be integrated with the `__linux__` definitions below.

I am not sure exactly what it would imply to "integrate with the __linux__ definitions. Do you think it is feasible to remove this GPR_MANYLINUX1 variable? Or is it somehow required to know that it is a manylinux build, in which case the manylinux toolchain should probably provide a MANYLINUX1 variable instead of you setting GPR_MANYLINUX1?

I can't say I fully understand what the TODO here means either. Unfortunately, the author has since left the team.
If we were to get rid of GPR_MANYLINUX1, what would the alternative be in your mind? I see two approaches to configuring a native build.
@nicolasnoble Please feel free to correct/disagree on anything I'm about to say.
The first is prescriptive. The user tells the compiler exactly which libc and system features to expect. This is what we're doing with GPR_MANYLINUX1. We do use this in our official manylinux wheel build. I don't particularly have a problem with this, because it allows us to use the standard setuptools facilities to compile our C extension.
The second is exploratory. An initial probing step is executed to determine which features are available on the platform on which the artifact is being built. CMake, Autoconf, and configure scripts are prime examples of this type. Since you're using Cmake, I would have expected this to "just work". I think the problem here with sched_getcpu is it was introduced in glibc 2.6, but we have no preprocessor version check on it.
TL;DR I think we should:

Leave GPR_MANYLINUX1 as it is.
Make platform detection smarter for sched_getcpu and any other assumptions that fall apart on older version of glibc.

Leave GPR_MANYLINUX1 as it is.
Make platform detection smarter for sched_getcpu and any other assumptions that fall apart on older version of glibc.

Well a third way would be to set MANYLINUX1 in the manylinux toolchain directly, so you can do if (MANYLINUX1), just like you do if (IOS), if (ANDROID) or if (MINGW). To me, the question is the following:
"Is manylinux just an old linux, or does it need to be handled differently?"
And in my understanding, it could actually be considered like another platform, just like Android: manylinux has some limitations because e.g. of the old glibc, so it is legit for a build system to do some if (MANYLINUX1) in some cases. In which case I would try to set MANYLINUX1 directly at the manylinux toolchain level. Does that make sense to you?

In my opinion, manylinux1 legitimately is "just an old Linux". It's based off of Centos 5.11 with a very limited set of shared object libraries installed, all with old ABIs.
With that said, it certainly is a valid choice to treat it as its own platform. In fact, it's very similar to what we're doing right now. But I'm not holding out hope that the manylinux maintainers will be responsive to the idea.

Good news: both PRs got merged into dockcross, meaning that in the next release, we will have:

MANYLINUX1 or MANYLINUX2010 defined in the toolchain (see here)
CMAKE_CROSSCOMPILING will work as we expect, i.e. will be FALSE for manylinux (see here).

No action needed then, but it will allow us to remove GPR_MANYLINUX1 in favor of MANYLINUX1, and it will automatically fix the issue I was experiencing with CMAKE_CROSSCOMPILING :+1:.
Closing this issue, and I'll try to open a PR once MANYLINUX1 can be used.

@JonasVautherin Amazing! Thank you!


Experiment with converters as renderers

Conceptually, renderers are a more general case of converters, being essentially a function from a document to any type. With the idea from https://github.com/CondeNast/atjson/issues/285 of adding additional safeguards and guarantees around renderer implementations, writing converters as renderers could help ensure that the converter satisfies some useful properties such as handling all the possible annotations in the source document.
(Written by @colinarobinson ❤)

This is an idea I had for this: https://gist.github.com/tim-evans/0893e956afc460cc53b9e0387d3e6ba3
I'm not sure how to extrapolate this code into a renderer, though...

here's an approach at how the renderer version of this might look in a simple case: https://gist.github.com/colinarobinson/69027d2e48a056c3f145e67ad8587043


switch to use major.minor.patch "properly"

anything what changes API models sufficiently to need a fix in dandi-cli would need a major release.  Would give us more space for all the .minor.patch annotation and practice to be semver pure ;-)
WDYT @satra and @jwodder

Technically, before version 1, semver allows API-breaking changes on any release.  See item 4 of the spec:

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

I understand that with 0.x all bets are off. But since we already have components depending on schema, I think it would not hurt and only make sense to try to stick to semver and thus jump to schema-1.x with the first breaking change. Library is indeed in greater flux, and not as widely used, so we could keep 0.x for longer.
Wdyt @satra ?

i would keep 0.x till we have tested the schema + publish + supporting multiple schemas for a bit. we just have to be careful after publish is in though.

As we are getting into a business of supporting multiple ".patch" versions in https://github.com/dandi/dandischema/pull/60/files#diff-fca559c50b570e7d5a8bc3aace5e144a9cf0810d05f15879e7da7d4569bcedb5R10 :
I think we should sooner than later start to just rely on semver and define ALLOWED_VALIDATION_SCHEMAS there as minimal version(s) in each 0.MINOR series we support, so there 0.4.4 would state that we support any schema in 0.4 starting with 0.4.4 (so 0.4.5, 0.4.6 etc would all be alright without explicitly listing them).
Then we can continue improving schema in backward compatible way (e.g. relaxing restrictions, or adding new optional fields -- although that might make "invalid" previously version "valid" with a new one...). Idea: may be should start adding "addedIn" field when a new property added and validate for that (it would be useful even regardless so people later could see why they need to start using a new schema).
Then new 0.MINOR would be the one introducing backward incompatible changes (making field mandatory, removing a field, etc). If we are not to support multiple 0.MINOR versions, here it could then be just a single version here instead of a list.

at this point our min version practically speaking is 0.4.4. at some point also  sooner than later we need to fix the min version and then progress with strategic updates.
i'm fine with moving to 0.5.0 for both schema release and dandischema library with the next release. however that means minor is tied to schema, not anything in the library.

in my comment above I was talking about schema version.  Not yet sure if trying to sync up schema and library version is worth the effort.
A little analysis:  we went with minor boost to 0.5.0 since we made schema non-backward compatible

by e.g. making some fields mandatory, such as
 class Participant(DandiBaseModel):
     """Description about the sample that was studied"""

-    identifier: Optional[Identifier] = Field(nskey="schema")
+    identifier: Identifier = Field(nskey="schema")
...
 class BioSample(DandiBaseModel):
     """Description of the sample that was studied"""

-    identifier: Optional[Identifier] = Field(nskey="schema")
-    sampleType: Optional[SampleType] = Field(
-        None, description="OBI based identifier for the sample used", nskey="dandi"
+    identifier: Identifier = Field(nskey="schema")
+    sampleType: SampleType = Field(
+        description="OBI based identifier for the sample used", nskey="dandi"
     )

most of the data in dandiarchive has 0.4.x version of metadata.
dandi-api, if upgrades version of dandischema and uses dandischema's validation, it might result in validation failures on 0.4.4 metadata, even though with https://github.com/dandi/dandischema/pull/60/files#diff-fca559c50b570e7d5a8bc3aace5e144a9cf0810d05f15879e7da7d4569bcedb5R10 adding ALLOWED_VALIDATION_SCHEMAS we might be claiming that we support validation of 0.4.4 ... What should we do about  that @satra ?  I guess we could sweep through archive and see if 0.4.4 metadata we have validates against perspective release of dandischima with schema-0.5.0 (e.g. as of #60), but I think we need to think up some more flexible approach (eh -- somewhat the same issue as with pynwb schema validations but there at least the entire schema is available and could be validated against...) -- might be worth starting an issue to have a dedicated discussion

good point.
indeed anything that's non-compliant at the asset level will require a migration. worthwhile checking if there are assets that are non-compliant at this point. also worthwhile checking if the new version fails to extract metadata in the cli. for example, i'm not completely sure sampletype is being considered.
let's use the move to 0.5.0 then a bit more harshly to make sure we can do validation against 0.4.4 and 0.5.0, even if we later set min version to 0.5.0. this will mean encoding changes between the pydantic models in some manner. i'll have to think about that one.


Aliquot steps added

Related Issue

Info about Issue or bug

fix #3023
Describe the changes you've made
A clear and concise description of what you have done to successfully close your assigned issue. Any new files? or anything you feel to let us know!
Checklist:

[ ] My code follows the style guidelines of this project.
[ ] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] I have made corresponding changes to the documentation.
[ ] My changes generate no new warnings.

Screenshots

Original
Updated

original screenshot
updated screenshot

@ShrutiOmer  I had opened the issue before ur Issue So I request to kidly close the isue #3000


AudioStreamSynchronized never sets playing to false or emits the finished signal.

Tested versions

Reproducible in: 4.3.dev5.official [89f70e98d]

System information
Windows 10 - Godot 4.3.dev5.official
Issue description
When using an AudioStreamPlayer along with the newly introduced AudioStreamSynchronized the playing variable is never properly set to false, and finished is never called when the stream stops playing. This is odd considering that, well, you'd assume you'd be told when the stream stops playing (since it does stop, it just doesn't tell you).
Steps to reproduce

Create an AudioStreamPlayer and attach a new AudioStreamSynchronized to it.
Add as many streams to the AudioStreamSynchronized as you want (so far I've tested 1 stream, 3 streams of the same length, and 2 streams with different length and they all reproduced the bug).
Play the AudioStreamPlayer and wait for the stream to end.
Audio will stop playing and get_playback_position will return 0, however neither playing nor finished were used properly.

Minimal reproduction project (MRP)
audiostreamsynchronized-mrp.zip

I started looking into this, seems like it's due to AudioStreamPlaybackSynchronized::mix always returning the passed in p_frames value, rather than the actual mixed frame count.  Because of this, AudioServer::_mix_step can't tell when to update that playback's state.  I tried changing it to use the max return value from each of the synchronized streams, which seems to fix it for this class.  However, it looks like AudioStreamPlaylist would have a similar issue, and maybe AudioStreamInteractive (I'm not really sure how that class is meant to be used).
Anyone more familiar with the audio code know if the above change sounds right, and how we want to handle the other classes?  I could test out my change some more and submit a PR, but I'm not sure if this should be part of a larger change.

Not sure when this happened but I was testing the stable Godot 4.3 release today and it seems this problem has been fixed somewhere along the development line, so I'm going to close this issue.