Observable.Using does not propagate exceptions from resource .Dispose method to stream

Bug
System.Reactive 5.0.0, Windows .NET 5.0
There are two error-handling cases with Observable.Using:

Exception is thrown when creating the resource (i.e. in provided resource factory method). In this case, exception is propagated to the resulting stream and can be handled in reactive way by using methods like .Catch. Works as expected.
Exception is thrown when calling Dispose method of the resource. If this one throws, exception will be unhandled, crashing the application regardless of onError handlers.

Reproduction snippet:
static async Task Main(string[] args)
{
    var source = Observable.Interval(TimeSpan.FromMilliseconds(50));
    var withResource = Observable.Using(
        () => Disposable.Create(() =>
        {
            Console.WriteLine("Disposing");
            throw new InvalidOperationException("HA");
        }),
        _ => source.Take(10));

    withResource.Subscribe(Console.WriteLine, ex => Console.WriteLine("Caught error"));
    await Task.Delay(10000);

    Console.WriteLine("Exiting application");
}

Expected onError handler to work here but exception is thrown as unhandled, crashing the application.
Is this intended behavior? Seems a bit inconsistent given that clearly measures were taken to ensure proper exception handling in resource creation (see System.Reactive.Linq.ObservableImpl.Run method).
I understand that allowing Dispose method to throw is bad design and will of course fix problem in my code, but still, accidents happen.

The same happens with the Finally operator, so it might be related.
Observable
    .Timer(TimeSpan.FromMilliseconds(100))
    .Finally(() => throw new ApplicationException("Oops!"))
    .Subscribe(_ => { }, ex => Console.WriteLine(ex.Message),
        () => Console.WriteLine("Completed"));

Thread.Sleep(1000);

Outcome: Unhandled exception (Fiddle)
The exception thrown in the Finally lambda is not handled by the Subscribe:onError handler.

The onError handler would have to be called after the downstream called Dispose on its subscription, thus signalling that it's no longer interested in notifications. Not sure about the exact nature of the contract and the guidelines in this case, but calling onError might both be bad style and also break existing code. Also, a general guideline says that Dispose should never throw, though I am aware that it does in the real world. I'd suggest you try make Dispose non-throwing if possible.

Not sure about the exact nature of the contract and the guidelines in this case, but calling onError might both be bad style and also break existing code.

@danielcweber according to this document:

When an observer issues an Unsubscribe notification to an Observable, the Observable will attempt to stop issuing notifications to the observer. It is not guaranteed, however, that the Observable will issue no notifications to the observer after an observer issues it an Unsubscribe notification.

This is in par with your "might be bad style" argument. But in the code posted by the @ArXen42 in the initial post, there is no explicit request for unsubscription. What happened is just that the source sequence completed. The Using operator could invoke the Dispose method of the resource before propagating the OnCompleted notification to its observers, and so be in a position to emit an OnError notification instead, without breaking the Rx contract.

The order of execution on an OnCompleted notification is pretty much the same for any operator, as you can see in the Sink class. It could be overridden for specific operators, although this would probably concern even more than just Using and Finally and soon create an uneven experience throughout all operators. I would still go with the guideline of "Dispose should never throw".
@bartdesmet any thoughts?

@danielcweber as I am reading here, there is indeed a .NET Framework guideline to avoid throwing from Dispose. But this guideline is not always followed, even by the .NET standard libraries. For example the Task.Dispose method is documented that it may throw an InvalidOperationException.
Also the Finally operator when subscribed returns a disposable subscription that may throw.
var sequence = Observable.Never<int>().Finally(() => throw new Exception());
IDisposable subscription = sequence.Subscribe();
try { subscription.Dispose(); }
catch (Exception ex) { Console.WriteLine(ex.Message); }

This program prints: Exception of type 'System.Exception' was thrown. (Fiddle)
Is seems surprising to me that the "by design" behavior of the Using and Finally operators is to crash the process, if the user-supplied action fails. Why are these operators different from other operators that invoke user-supplied actions (like the Do or the Start for example), that propagate the errors of those actions to the observers?

The Finally operator behaves a little bit different to its System.Interactive counterpart. Infact OnDispose would be a more appropriate name, since it is directly called, when the subscription is disposed. This can even be during OnNext, see this fiddle. So what happens in your example:
Observable
    .Timer(TimeSpan.FromMilliseconds(100))
    .Finally(() => throw new ApplicationException("Oops!"))
    .Subscribe(_ => { }, ex => Console.WriteLine(ex.Message),
        () => Console.WriteLine("Completed"));

Thread.Sleep(1000);

The timer completes and sends OnCompleted downstream to the Subscribe operator.
The Subscribe operator will then dispose the subscription, to the Finally operator.
The Finally operator will call the user supplied action.
The action throws an exception.

At that point, the Finally operator cannot call OnError, because the subscription is already canceled, no one can and wants to recieve the exception. Basically it would simply be swallowed. Second, it could happen that OnError is executed concurrently to OnNext or OnCompleted (see the fiddle above). This would violate the IObservable contract.

Thanks @quinmars for the example and the explanation. It helped me understand why the Finally behaves this way from a technical perspective, and indeed OnDispose or OnUnsubscribe would be a more descriptive name for it. Do you know why it was decided to have this behavior? To me this behavior makes it a less useful operator. Essentially it can only be used for invoking fail-proof actions.
To be honest when I started using the Finally operator, I was thinking of it as a convenient alternative to the Do operator, combining the onError and onCompleted actions to one. I hadn't thought about the implication arising from the requirement to invoke the finallyAction on unsubscription too. I was not anticipating to dispose my subscriptions very often, and I hadn't realized that the built-in Rx operators dispose subscriptions all the time.

I just found out that the Using and Finally are not the only operators that can throw application-crashing unhandled exceptions. All operators that interact with IEnumerable<T>s can do the same too, in case the IEnumerator<T>.Dispose method fails. Like for example the Zip and the SequenceEqual operators. Below is a minimal demonstration of this behavior:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Threading;

class Foo : IEnumerable<int> // Enumerable that throws on Dispose
{
    public IEnumerator<int> GetEnumerator() => new Enumerator();
    IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();
    private class Enumerator : IEnumerator<int>
    {
        public int Current => default;
        object IEnumerator.Current => this.Current;
        public bool MoveNext() => false;
        public void Reset() { }
        public void Dispose() => throw new Exception();
    }
}

static class Program
{
    public static void Main()
    {
        var source = Observable.Return(1).ObserveOn(Scheduler.Default);

        var sequence = source.SelectMany(_ => new Foo());
        //var sequence = source.Concat(new Foo().ToObservable());
        //var sequence = source.Zip(new Foo());
        //var sequence = source.SequenceEqual(new Foo());

        sequence.Subscribe(_ => { }, ex => Console.WriteLine(ex.Message),
            () => Console.WriteLine("Completed"));

        Thread.Sleep(1000);
    }
}

Outcome: Unhandled exception (Fiddle)
It seems that the Rx library as a whole is based on the assumption that disposables never fail. 👀

It seems that the Rx library as a whole is based on the assumption that disposables never fail.

It isn't. It's just that on the edges of entering and leaving the hot observable (that is, everything that happens before an observer is ready to observe, and everything that happens after it has unsibscribed/observed a termination notification), you are dealing with ordinary code flow, along with all the terms and conditions that apply when something throws.  You cannot expect exceptions to end up on OnError before a certain point in the subscription, nor after a certain point after unsubscription/termination of the sequence. I'm not saying the library couldn't refactored that way though. Maybe the AsyncRx project is a place to rethink these kind of things. It's right here in this repo and currently lacking a test suite.

@danielcweber this is the wrong place to tell my opinion about the AsyncRx project, but honestly it terrifies me for the sheer number of undocumented nuances that I expect it to have. The Rx is already full of nuances, and adding asynchrony in the mix can only make the number go up. To be clear I have zero familiarity with the project. I'm just expressing my speculations here.


sky_engine reexports dart libraries that have missing _embedder.yaml mappings

@Hixie @stereotype441
I first saw this while debugging dart-lang/dartdoc#2143.  The example I found is in dart:html, whose source code is copied directly from:   https://github.com/dart-lang/sdk/blob/master/sdk/lib/html/dart2js/html_dart2js.dart.  It reexports promiseToFuture from dart:html_common, which isn't available in sky_engine.
I do not understand how Flutter can possibly work under these conditions (Why are there no compilation errors?  Why are all analysis errors here always squelched?) but it clearly does.... so I'm filing this issue in part to ask:

Is this an actual problem (meaning, it is a bug that all of our tools seem to ignore this situation), or are situations like this considered "normal"?
If it isn't a problem, is ignoring this situation in tools the correct way to handle it?  Or should we be working around this in a different way?

For example, a combination of factors tend to result in dartdoc and the analyzer ignoring this under most circumstances and just silently hiding things.   You'll notice that in the documentation for dart:html at https://api.dart.dev/dev/2.8.0-dev.20.0/dart-html/dart-html-library.html includes promiseToFuture, but https://api.flutter.dev/flutter/dart-html/dart-html-library.html does not.

I believe the reason this works is because sky_engine isn't actually used by anything besides the analyzer and dartdocs. The actual platform code is already present in the platform dill file produced by the engine, and the supported dart libraries are defined in https://github.com/flutter/engine/blob/master/lib/snapshot/libraries.json

@jonahwilliams @devoncarew That makes sense.  Though,  why is dart:html  available in sky_engine but not the platform file?

Flutter and Flutter for Web are different platforms with different available sets of dart SDK libraries. dart:html was added to sky engine because the web team wants web plugin authors to be able to get analysis results.
There is more context in https://github.com/flutter/flutter/issues/39998

Thanks.  From reading this my general approach will be to continue to ignore and/or bypass analysis errors generated from sky_engine in dartdoc, and I'll follow up in #39998 if need be.


Bump framework version

$subject

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/7973343342

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/7973343342
Status: success


[Table] Fixed paddings according to guidelines

Checkbox column should have 24px padding next to text column.

Toolbar should have 14px padding to icon on the right side

@kybarg Could you remove the extra commit?

I wish we have added some visual regression tests for the Table component.
That looks good otherwise 👍

@oliviertassinari

@kybarg Could you remove the extra commit?

Done 😄 My bad, too many branch switching lately)

@kybarg On a roll! :+1:

@mbrookes thank you ☺


feat: expose json schema for functional content

PR raised by @guillaume-goulet https://github.com/AmadeusITGroup/otter/pull/245


Suggestion: Use a python formatter (such as black)

I would strongly advise to use a formatter. The code base is rather inconsistent and could benefit from a formatter (i.e. black. In the same vein it would also be a good idea to set up some github actions to check for formatting issues automatically (and maybe even add tests).
It's way easier if the code base would use more conventional code formatting and makes it easier for new people to modify and read the code.
The main problem is that the incorporation of a formatter would create many merge conflicts with existing PRs. I would suggest to either finish all PRs in one go and merge them, or format the code base gradually.

I do not see the need for an auto-formatter for the codebase. While I do agree that the formatting could be improved slightly, there is nothing I can see that would make it difficult to read or contribute to. This code is likely never going to be fully PEP-8 compliant, and I see no real benefit in enforcing such rules.
If this project gains significantly more traction in the future, I might add some formatting conventions in order to maintain some level of consistency from external contributions. However, in its current state, I don't see much value in enforcing strict style rules on such a small project.


boxxy: 0.2.7 -> 0.3.4, add figsoda as a maintainer

Diff: https://github.com/queer/boxxy/compare/v0.2.7...v0.3.4
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 216514 run on x86_64-linux 1

1 package built:

boxxy

This seems to have broken aarch64-linux, I will take a look

Couldn't fix it, might be an upstream issue, marked broken on aarch64-linux


Fix Deferred

@enumag Why did you close this?

@kelunik Because it didn't fix the error I was experiencing anyway and only made it worse... Although I do know that it is technically correct to resolve with a Promise... Reopen and merge it if you wish.

Thanks!


Update MD file on new branch 2017-22-18_13-11-00

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestCon.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Unrecognized Kafka Sink settings

Guidelines
I tried to install the neo4j-streams 4.1.0 to publish data from a Kafka topic to Neo4j database. I copied the .jar file to the /plugins repository and I modified the neo4j conf file to add the appropriate parameters.
When I restart the neo4j database I have the following WARN :
2021-11-08 16:34:20.967+0000 WARN  Unrecognized setting. No declared setting with name: streams.sink.topic.cypher.TEST-TOPIC
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: kafka.ssl.truststore.location
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: streams.sink.enabled
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: kafka.bootstrap.servers
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: neo4j.database
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: kafka.ssl.keystore.location
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: kafka.group.id
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: kafka.security.protocol
2021-11-08 16:34:20.968+0000 WARN  Unrecognized setting. No declared setting with name: kafka.ssl.keystore.password

It seems that the paramters related to neo4j streams connector are not recognized by the neo4j database.
I tried to push the following file to the noe4j database :
[{
 "id": 42,
 "properties": {
   "title": "Answer to anyting",
   "description": "It depends."
 },
{
 "id": 43,
 "properties": {
   "title": "Answer to anyting",
   "description": "It depends."
 }
}],

I used this cypher tempalte :
streams.sink.topic.cypher.TEST-TOPIC=MERGE (n:Label {id: event.id}) ON CREATE SET n += event.properties

I use the SSL security protocol to connect to Kafka.
Versions

OS: RedHat 8.2 (Ootpa)
Neo4j: 4.2.3
Neo4j-Streams: 4.1.0

Hi @aissaelouafi,
you can ignore those warnings. That's definitely not the reason why the plugin is not working.
Could you please share the entire neo4j.conf so that i can review it? Even just the subset of properties related to Kafka and Streams would be enough.
How are you sending the events? Are you using a console producer?
Regards,
Mauro

Hi @mroiter-larus,
Thanks a lot for your comment.
This is the neo4j conf file we use :
kafka.bootstrap.servers=xxxxxx:9092
streams.sink.enabled.to.dbpov2=true
streams.sink.errors.log.enable=true
streams.sink.errors.log.include.messages=true
streams.sink.topic.cypher.mykafka-topic.to.dbpov2=MERGE (n:Label {id: event.id}) ON CREATE SET n += event.properties

kafka.security.protocol=SSL

kafka.ssl.truststore.location=./kafkasink.jks
kafka.ssl.truststore.password=test1234

kafka.ssl.keystore.location=./kafkasink.jks
kafka.ssl.keystore.password=test1234
kafka.group.id=neo4j

dbms.jvm.additional=-Djavax.net.debug=ssl:handshake

I send the events using a python-kafka producer. We use the entreprise version of neo4j.

Hi @aissaelouafi,
The configuration seems to be ok.
Do you have any exception in the Neo4j logs? If not, try adding the following:
streams.sink.enabled=true

Could you please also check if the topic mykafka-topic exists in Kafka? If not, try checking if Kafka auto.create.topics.enable is enabled or disabled. If it is disabled you should find a WARNING into the Neo4j logs which warns you that you have to manually create the topics you need.
Regards,
Mauro

Hi @mroiter-larus,
The topic exist in Kafka. But I have this warning, I think it's related to the ssl certificates :
javax.net.ssl|WARNING|03|Finalizer|2021-11-15 15:26:48.624 CET|SSLSocketImpl.java:500|SSLSocket duplex close failed (
"throwable" : {
  java.net.SocketException: Socket is not connected
        at java.base/java.net.Socket.shutdownOutput(Socket.java:1570)
        at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownOutput(BaseSSLSocketImpl.java:232)
        at java.base/sun.security.ssl.SSLSocketImpl.deliverClosedNotify(SSLSocketImpl.java:651)
        at java.base/sun.security.ssl.SSLSocketImpl.closeNotify(SSLSocketImpl.java:627)
        at java.base/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:550)
        at java.base/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:485)
        at java.base/sun.security.ssl.BaseSSLSocketImpl.finalize(BaseSSLSocketImpl.java:275)
        at java.base/java.lang.System$2.invokeFinalize(System.java:2121)
        at java.base/java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87)
        at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)}

)

Hello @mroiter-larus,
I just found these messages as well, I think that I have an issue with the .jar file
2021-11-16 12:03:49.246+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.serializers.KafkaAvroDecoder` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/serializer/Decoder
2021-11-16 12:03:49.247+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.serializers.AbstractKafkaAvroSerializer` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/utils/VerifiableProperties
2021-11-16 12:03:49.249+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.formatter.AvroMessageReader` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/common/MessageReader
2021-11-16 12:17:09.529+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.formatter.AvroMessageFormatter` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/common/MessageFormatter
2021-11-16 12:17:09.536+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/utils/VerifiableProperties
2021-11-16 12:17:09.537+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.serializers.KafkaAvroDecoder` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/serializer/Decoder
2021-11-16 12:17:09.538+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.serializers.AbstractKafkaAvroSerializer` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/utils/VerifiableProperties
2021-11-16 12:17:09.540+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.confluent.kafka.formatter.AvroMessageReader` from plugin jar `/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar`: kafka/common/MessageReader

I dont know if you have an idea about this issue.
Regards,
Aissa

Hi @aissaelouafi,
I suppose you've an issue with the security configuration. Please check this guide about Kafka security setup: https://docs.confluent.io/platform/current/security/security_tutorial.html#security-tutorial.
However, you can try disabling security, just to be sure that it is the problem.
Mauro

@aissaelouafi,
in attachment you will find a .zip file which contains all the necessary to setup a working Dockerized environment (Neo4j + Zookeeper + Kafka) with SSL encryption enabled. Just unpack the .zip file and run docker-compose up -d to start it up. Once it will be up and running you can connect to the Kafka broker container and run a kafka-console-producer to send some events into the topics.
Hope it helps!
Regards,
Mauro
Github_509.zip

Hi @mroiter-larus,
Thanks a lot for your time and for the answer.
I'll try it on my laptop but I have the issue with the ne4oj enterprise version 4.2.3 and Kafka 2.7 I dont know if there is some dependencies between neo4j version and kafka version with the neo4j streams plugin.
Regards,
Aissa

@aissaelouafi,
here you can find a compatibility table between Confluent Platform and Apache Kafka Compatibility so that you can choose the proper  Docker's image version for Confluent Kafka and Zookeeper in order to reproduce the environment you need. In your case, Confluent platform 6.1.x uses Apache Kafka 2.7.x.
About the Docker's image for Neo4j, i've already choose the 4.2.3 versions.
Mauro

@mroiter-larus
Thanks again, I just tried the docker it's running and working well.
On the other side, I have this warning :
2021-11-17 14:36:20.832+0000 WARN  [o.n.k.a.p.GlobalProcedures] The procedure 'streams.sink.config' is not on the allowlist and won't be loaded.
2021-11-17 14:36:20.832+0000 WARN  [o.n.k.a.p.GlobalProcedures] The procedure 'streams.sink.status' is not on the allowlist and won't be loaded.
2021-11-17 14:36:20.832+0000 WARN  [o.n.k.a.p.GlobalProcedures] The procedure 'streams.sink.stop' is not on the allowlist and won't be loaded.
2021-11-17 14:36:20.832+0000 WARN  [o.n.k.a.p.GlobalProcedures] The procedure 'streams.sink.start' is not on the allowlist and won't be loaded.
2021-11-17 14:36:20.832+0000 WARN  [o.n.k.a.p.GlobalProcedures] The procedure 'streams.sink.restart' is not on the allowlist and won't be loaded.
2021-11-17 14:36:20.832+0000 WARN  [o.n.k.a.p.GlobalProcedures] The procedure 'streams.consume' is not on the allowlist and won't be loaded.

I can't see any logs related to kafka sink when I produce to the kafka topic. I dont know if I need to add some specific procedures to the allowed list or not.
Regards,
Aissa

@aissaelouafi
You just need to add the following property in your neo4j.conf file:
dbms.security.procedures.whitelist=streams.*

The equivalent for Neo4j Docker container will be:
NEO4J_dbms_security_procedures_whitelist: "streams.*"

Please read the Neo4j Streams documentation here for further details about Neo4j Streams procedures.
Regards,
Mauro

@mroiter-larus
I just add, the warning has disappeared but I can't receive any data from the topic, it's so strange.
I can't see any log related to streams procedure when I produce data to the kafka topic as well.
I added the kafka key and value deserializer as well :

dbms.security.procedures.allowlist=bloom.*,gds.*,apoc.*,streams.*

kafka.bootstrap.servers=xxx:9092
streams.sink.enabled=true
streams.sink.enabled.to.dbpov2=true
streams.sink.errors.log.enable=true
streams.sink.errors.log.include.messages=true
streams.sink.topic.cypher.my-kafka-topic.to.dbpov2=MERGE (n:Label {id: event.id}) ON CREATE SET n += event.properties

kafka.enable.auto.commit=true
kafka.key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
kafka.value.deserializer=org.apacge.kafka.common.serialization.ByteArrayDeserializer
kafka.security.protocol=SSL

kafka.ssl.truststore.location=/apps/neo4j/neo4j-enterprise/conf/kafka-neo4j.truststore.jks
kafka.ssl.truststore.password=xxxx
kafka.ssl.keystore.location=/apps/neo4j/neo4j-enterprise/conf/kafka-neo4j.keystore.jks
kafka.ssl.keystore.password=xxxx

#kafka.ssl.endpoint.identification.algorithm=HTTPS
kafka.group.id=test
kafka.group.client.id=neo4j

It's strange because I can't see any logs related to the plugins.
Thanks a lot of your help.

@aissaelouafi
A couple of questions:

did you get some exception on the Kafka side? Please check the Kafka logs and share it if you find any errors
could you please check the status of the Sink module? Just run CALL streams.sink.status() from the Neo4j browser. If the response is "STOPPED" then try running: CALL streams.sink.restart(). The status should be "RUNNING"

Watch out at the following two properties:

kafka.group.client.id property should be: kafka.client.id
dbms.security.procedures.allowlist should be dbms.security.procedures.whitelist

Mauro

I tried also to consume the topic manually using the procedure, but I get this error :

I tried using :
CALL streams.consume('my-kafka-topic') YIELD event RETURN event

@aissaelouafi
A couple of questions:

did you get some exception on the Kafka side? Please check the Kafka logs and share it if you find any errors
could you please check the status of the Sink module? Just run CALL streams.sink.status() from the Neo4j browser. If the response is "STOPPED" then try running: CALL streams.sink.restart(). The status should be "RUNNING"

Watch out at the following two properties:

kafka.group.client.id property should be: kafka.client.id
dbms.security.procedures.allowlist should be dbms.security.procedures.whitelist.
Did you fix the two properties that I've reported to you?
Please add the following properties (see here for further details):
streams.check.apoc.timeout=20000
streams.check.apoc.interval=5000
streams.sink.poll.interval=5000

then restart the Neo4j server and check again the Sink status.
Mauro

@aissaelouafi,
a little suggestion about the Failed to load ... from plugin jar "/apps/neo4j/neo4j-enterprise/plugins/neo4j-streams-4.1.0.jar" warnings: ensure that the user which is running Neo4j, has the permission to access the Neo4j plugins folder.
Mauro

@mroiter-larus
I added the following parameters, but the status is still UNKNOWN I think that there is an issue with the jar file, but I can see the streams procedure in the browser :
streams.check.apoc.timeout=20000
streams.check.apoc.interval=5000
streams.sink.poll.interval=5000

I tried also to add the streams procedure to this parameter dbms.security.procedures.whitelist but it's not working, It's working with this param dbms.security.procedures.allowlist.
For the permissions, I use the root user to start the neo4j service, the owner of neo4j folder and plugins folder it's neo4j user. The others plugins are working well, we have issues with only the kafka streams plugin.
As you can see, I can see the whole functions related to the streams plugin, but the status is still UNKNOWN :

Regards,
Aissa

@aissaelouafi
you're right about the dbms.security.procedures.whitelist...it is deprecated since Neo4j 4.3. So, it's ok to use dbms.security.procedures.allowlist. My bad!
However, i can't understand why the plugin is not starting. As you could see, it works perfectly in the test environment i've shared. I've also tested it with Confluent platform 6.1.3 (Apache Kafka 2.7) and it works also in this case. So the Neo4j Streams JAR should be ok.
The issue is somehow related to the configuration.
I'll investigate and i'll keep you posted.
In the meantime i would suggest you to try with a fresh install of Neo4j (just to avoid any issue in your current Neo4j settings).
Regards,
Mauro

@mroiter-larus
Thanks a lot for your time and your help.
Unfortunately I can't try with a fresh install of neo4j because it's used by other teams for development and environment.
I think it's related to the configuration as well. I hope that we can found the issue soon.
Thanks a lot again for your help.
Regards,

@aissaelouafi
With fresh install i was thinking about a new Neo4j instance. Not to the same you're now working with.
Mauro

@mroiter-larus
I tried again with your docker environment. The status was unknown but when I restarted the Kafka broker, the status has changed to started, I'll maybe try to restart the Kafka on the other side.
Also, I have some questions related to the docker environment, I still can't see data in the neo4j database. So I have a couple of questions regarding your neo4j configuration :

Why you use the ByteArrayDeserializer the Kafka key and value deserializer ? In my case, I send json data. Should I use the JsonSerializer as follow :

NEO4J_kafka_key_deserializer: org.apache.kafka.common.serialization.JsonDeserializer
NEO4J_kafka_value_deserializer: org.apache.kafka.common.serialization.JsonDeserializer

How you send data from the broker using the Kafka-producer ? In my case I send Json data using the JsonSerializer :
kafka-console-producer --broker-list localhost:9092 --topic testtopic --property value.serializer=custom.class.serialization.JsonSerializer < data.json

Should I create manually the database dbpov2 ?

Thanks a lot for your time.
Regards

@mroiter-larus
I think I have a good news regarding the issue of UNKNOWN status of the sink plugin.
We need to have a separate conf file for the neo4j streams plugins.
I created a seperate conf file called streams.conf and I can see that the status is RUNNING.
I have some exceptions related to the SSL certificates I'll check that.
Regards,
Aissa

@aissaelouafi
I'm sorry, I just assumed that you already had your configuration in the streams.conf file. From Neo4j Streams 4.0.7 all the Neo4j Streams and Kafka settings has to be moved from neo4j.conf to streams.conf (You have to create this file manually). This is reported in the official Neo4j Streams documentation here.
About your questions:

Neo4j Streams support two tpye of deserializers: ByteArrayDeserializer and KafkaAvroDeserializer. (Please see here)
I send events via kafka-console-producer with the following:

kafka-console-producer --bootstrap-server localhost:9092 --topic testtopic

Then i send events one by one in json format as they were string, for example:
{ "id": 42, "properties": { "title": "Answer to anyting", "description": "It depends." }}

But if you want to send events from file you can collect the json string events such as the one above, and placing them one per line.
I would suggest not to use --broker-list because it is deprecated (use --bootstrap-server option instead)
3. If you want to write/read data into/from a non-default Neo4j database you have to manually create it first.
Hope this helps.
Cheers,
Mauro

@mroiter-larus
It's working fine on my laptop, thanks a lor for your help.
I still have an SSL exception on the other server, but I'll look for it.
2021-11-19 13:26:56.437+0000 WARN  [s.StreamsRouterConfigurationListener] [neo4j/2d8c2716] Cannot retrieve valid topics because the following exception,
next attempt is in 300000 ms:
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ~[neo4j-streams-4.1.0.jar:?]
        at streams.kafka.KafkaAdminService$start$1.invokeSuspend(KafkaAdminService.kt:29) [neo4j-streams-4.1.0.jar:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [neo4j-streams-4.1.0.jar:?]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [neo4j-streams-4.1.0.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [neo4j-streams-4.1.0.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [neo4j-streams-4.1.0.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [neo4j-streams-4.1.0.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [neo4j-streams-4.1.0.jar:?]
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:352) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:295) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:290) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:402) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:484) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:340) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:170) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.common.network.Selector.poll(Selector.java:483) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540) ~[neo4j-streams-4.1.0.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196) ~[neo4j-streams-4.1.0.jar:?]
        at java.lang.Thread.run(Thread.java:834) ~[?:?]

@mroiter-larus
I have a last question about the configuration file, I want to use the following group : GROUP-ID because it's the group declared by Kafka.
But the plugin add the neo4j database name as a suffix  of the groupe-id, for example : -dbpov2
So the group become : GROUP-ID-dbpov2, So I have this exception :
2021-11-19 15:08:58.106+0000 ERROR [s.StreamsSinkConfigurationListener] [dbpov2/d26d4715] Not authorized to access group: GROUP-ID-dbpov2
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: GROUP-ID-dbpov2

How can I fix this issue on the conf file ?

@aissaelouafi,
This happens because dbpov2 is not the default database. If you set the following
dbms.default_database=dbpov2
kafka.group.id=GROUP-ID

the plugin will not add the suffix -dbpov2. But i'm not sure this could be a solution in your case. It depends if you were using the neo4j default database or if you're going to use it.
When you get that exception?

@mroiter-larus
Thanks a lot it's working fine right now.
I have one question about dealing with multiples kafka topics, it's possible to manage multiple kafka group id ?
I mean how can we define a kafka group id per topic.
for example, if we have two kafka topic topic_a and topic_b :
streams.sink.topic.pattern.node.topic_a.to.db_2=Person{!id}
streams.sink.topic.pattern.node.topic_b.to.db_2=City{!id}

It's possible to specify a kafka.group.id by topic or should we use the same kafka.group.id for all topics ?
Regards,
Aissa

@aissaelouafi,
Currently Neo4j Streams does not support the creation of multiple consumer groups. Every consumer created by the Neo4j Streams plugin will belong to the Kafka consumer group defined by the kafka.group.id parameter.
You could migrate to Kafka Neo4j connector, and increasing the tasks.max you would have that the single consumer would be splitted into multiple tasks (consumers that will read concurrently from the topic partitions): if you have 10 partitions with tasks.max set to 5, each task will receive 2 partitions to read from.
However this is very dangerous, because parallelization could cause Neo4j locks. Please read the following documentation for more information:

https://neo4j.com/labs/kafka/4.0/architecture/sinkconsume/#_important_information_about_batch_imports_into_neo4j
Take a look at the neo4j.batch.parallelize parameter in the table here

I'm closing this for lack of feedback, feel free to reopen it in the future


Added exceptSelect function for Query

Q
A

Is bugfix?
no

New feature?
yes

Breaks BC?
no

Tests pass?
yes

I've got it. The case is a table with 100s of columns and the need to select everything except "status", "is_deleted".

Personally I both like and don't like this addition. I've been in situation like that and wished there was a method like that but that was only once in like 5 years.

@yiisoft/core-developers, @yiisoft/reviewers opinions?

I do not consider use case to be common enough. If it would SQL native syntax suported it long ago.
Having a database table with so much columns, so you need exceptSelect feature, usually indicates a poor database structure design.
Also such functionality can be easily composed into an ActiveQuery behavior or trait and be applied at will. Also building a list of fields can be moved into a separated helper.

I thought about you said, if I build a list , i must update it when i changed sql structure,then, every table just have 20 columns,but i meet this problem in many projects,every time i must write same code to resolve it

And,maybe it is not very common,but I think it is necessary. Anyway I really hope you can accept this feature ,so I can never write same code again,thanks

Ok,thank you to review the code,i will rename variables tomorrow

@Inocence

I thought about you said, if I build a list , i must update it when i changed sql structure,then, every table just have 20 columns,but i meet this problem in many projects,every time i must write same code to resolve it

And,maybe it is not very common,but I think it is necessary. Anyway I really hope you can accept this feature ,so I can never write same code again,thanks

I think your can use your own repo with this behaviour just now and use it in all projects.
For example: I have repo bscheshirwork/yii2-cubs

Ok, I got it,thanks.

We currently have addSelect() to add columns to the select clause, maybe we can have a removeSelect() that works complementary to that.

https://github.com/yiisoft/yii2/blob/376006a2d5048764ebad70a3fd1077ceb4989352/framework/db/Query.php#L568-L597
Have only one parameter $columns and independent of table structure or database driver.
Revere of this will expect same signature.
i.e. something like
public function removeSelect($columns, ?array $alias = null)

Can I will use removeSelect for query of \yii\redis\ActiveRecord model?

Additional expects
$query
->addSelect(["*", "CONCAT(first_name, ' ', last_name) AS full_name"])
->removeSelect(["*", "CONCAT(first_name, ' ', last_name) AS full_name"])

still $query is unchanged

i think i should change the function to make it works like addSelect()

I like what @cebe proposed.

Won't make it into 2.0.


Misaligned Checkbox and not able to select all rows in IR

Thanks for the plugin,
what could be the problem that:

the checkbox and container is misaligned in an interactive report
the selecting all rows only works when the "Select with click on row" feature is enabled ?
(see image)

I'm not able to reproduce this on apex.oracle.com - it works there.
But i have 2 APEX 22.1.3 instances on 19c where i experience this behaviour.

Hi,
can you please share a version of plugin that you use (1.0 or 1.1) and the settings that you have configured for those plugin instances? Is there any custom CSS code on that page?

Hi,
Plugin Version is 1.1.0 on both instances, both APEX Versions are 22.1.3 on 19c.
plugin_1.png:
newly created app, only 1 page, no custom css/js involved. selecting all only works when "select with click on row" is checked.
Dev console shows that there is no event listener present for the header checkbox (for rows the listener is present)

plugin_2:
older app but refreshed theme, no css/js involved.also no event listener on the header checkbox, misalignment can be solved checking "IR Limit column width" and resizing the browser window. same settings as in plugin_1.png

I have prepared a patch version. Now after changes I'm not able to reproduce any of those bugs in any environment. Please check if this solution works for you as well.

hi,
thanks - seems to work now !


Can't load .n64 files

Even though the file chooser for ROMs says it allows .n64 files, when I try to open one it says it's "Not a valid .z64 file". This is because the endianness of a .n64 file is different from a .z64 file, which is not accounted for in your code. Please do so ASAP.

The latest commit allows conversion of little-endian ROMs using the converter under File->Convert V64/n64...
You can test it using the development builds:
https://github.com/dh4/cen64-qt#development-builds
Let me know if you have any issues using it.


allow a binary name to be passed to make build

While working on promtool and prometheus I often went back and forth commenting out binaries in the .promu.yml file in order to only rebuild the binary I wanted to rebuild.
In this pr: https://github.com/prometheus/promu/pull/77 I added the option to pass a binary name to promu's build function. This is controlled by setting the BINARIES var.
For example:
$make build BINARIES=promtool
>> building binaries
 >   promtool

In this PR we simply pass BINARIES to promu. If it's not set, promu will build all binaries from the YAML file as normal.
$make build
>> building binaries
 >   prometheus
 >   promtool

rebased off master, my local master branch was out of date so some tests were failing

Hmm, not sure if we want this. So for local dev what I do is go install ./cmd/prometheus and then do prometheus. Now this caches the build and significantly speeds up compilation.

hey @Gouthamve I agree that go install is the best way to go for quick builds when making changes, but you want to run make build if you want everything to be clean so I think having this option is of some use :)
This change has also been merged to alert manager here: https://github.com/prometheus/alertmanager/pull/990

Aah, yep. :+1: from me
@brian-brazil?

I don't mind one way or the other, as long as the Makefiles are kept the same across all our Go projects.

@brian-brazil would you like me to make this change to all the go projects in the prometheus org then?
I only opened PR's for prometheus and alertmanager as they were the only two I found that built more than one binary via promu.

Any change made to one should be made to all.
Having 15 hand-crafted Makefiles is not going to be very maintainable, so we should be keeping them consistent.

@brian-brazil okay I will make the same change to the following repos:
blackbox_exporter
promu
statsd_exporter
node_exporter
snmp_exporter
prom2json
consul_exporter
mysqld_exporter
collectd_exporter
haproxy_exporter
pushgateway
memcached_exporter
graphite_exporter
influxdb_exporter

I already made this change in alertmanager. I think that list is complete, I didn't see any others using promu for building.

I'm not sure it makes sense to touch all these projects and add support for that just because the rare need to only build promtool without prometheus. The build make target wraps a single command, what's the issue of executing promu directly instead of using make if you want to only build promtool?

@grobie agreed, that's why I only opened this PR and the other for alertmanager, those are the only two building multiple binaries. The only issue I see with executing promu directly if you just want to build one of the two binaries rather than both, is that instead of using make we're now advertising two tools to people for building prometheus org go binaries instead of just make.
Up to you guys.

unless we have a request for this I don't think we should add it.

With go 1.10's build caching, I think the need for this is greatly lessened.

@cstyan would you mind closing for now and will revisit if needed.


planning prompt example

I need you to create a timetable for me given the following facts:

my plane takes off at 6:30am
I need to be at the airport 1h before take off
it will take 45mins to get to the airport
I need 1h to get dressed and have breakfast before we leave
The plan should include when to wake up and the time I need to get into the vehicle to get to the airport in time for my 6:30am flight , think through this step by step.

You are an expert travel agent who is very careful with accounting for time when planning activities.

Mixtral seems responsive to system prompt


Crash after hiding post

Steps to reproduce:
-Hide a post (either by long clicking and choosing hide or clicking the hide icon from the post)
-Quickly swipe to another subreddit while the red “undo” button is active

Fixed!


Long delay on 'click' action

I am using 'click on coordinates' inside the while loop, but there are about 1s delay between iterations. Can it be reduced somehow? The browser is in touch mode if that's matter

Please write to premium email.


[Enhancement] Throw compile error when _GLIBCXX_USE_CXX11_ABI=0

What type of PR is this：

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Problem Summary(Required) ：

Modifying SinkBuffer::buffer is protected by mutex, but getting size of buffer isn't protected to gain better performance. std::list::size before gcc5 traverses the whole list, which is not thread-safe.
Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch

[x] 2.5
[x] 2.4
[x] 2.3
[x] 2.2

run starrocks_admit_test

run starrocks_admit_test

@Mergify backport branch-2.2 branch-2.3 branch-2.4 branch-2.5


Upgrade direct dependencies

Issue #, if available:
Description of changes:
This PR upgrades most direct dependencies to their latest versions. I stayed within k8s 1.25 for the k8s deps.
Testing performed:
build successful & tests pass
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/ok-to-test

Updated for k8s v1.25.13. Can this be  merged? There is not much exciting change here.

/run-e2e -c 4.18

/run-e2e -c 4.17

@vishesh92 a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.18
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

@vishesh92 a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.17
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

@hrak Just started e2e test runs. Let's merge it once they are complete.

Test Results : (tid-328)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.18
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr297-sl-328.zip
[PASS] When testing machine remediation Should replace a machine when it is destroyed
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
[PASS] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified control plane offering is not found [TC7]
[PASS] When the specified resource does not exist Should fail due to the specified template is not found [TC6]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the specified disk offering is not found
[PASS] When the specified resource does not exist Should fail due to the compute resources are not sufficient for the specified offering [TC8]
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is not customized but the disk size is specified
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is customized but the disk size is not specified
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade worker machine due to insufficient compute resources
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade control plane machine due to insufficient compute resources
[PASS] When testing subdomain Should create a cluster in a subdomain
[PASS] When testing Kubernetes version upgrades Should successfully upgrade kubernetes versions when there is a change in relevant fields
[PASS] When testing multiple CPs in a shared network with kubevip Should successfully create a cluster with multiple CPs in a shared network
[PASS] with two clusters should successfully add and remove a second cluster without breaking the first cluster
[PASS] When testing app deployment to the workload cluster [TC1][PR-Blocking] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing app deployment to the workload cluster with slow network [ToxiProxy] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing resource cleanup Should create a new network when the specified network does not exist
[PASS] When testing app deployment to the workload cluster with network interruption [ToxiProxy] Should be able to create a cluster despite a network interruption during that process
[PASS] When testing with custom disk offering Should successfully create a cluster with a custom disk offering
[PASS] When testing affinity group Should have host affinity group when affinity is pro
[PASS] When testing affinity group Should have host affinity group when affinity is anti
[PASS] When testing with disk offering Should successfully create a cluster with disk offering
[PASS] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] Should successfully scale machine replicas up and down horizontally

Ran 28 of 29 Specs in 8313.549 seconds
SUCCESS! -- 28 Passed | 0 Failed | 0 Pending | 1 Skipped
PASS

Test Results : (tid-329)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.17
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr297-sl-329.zip
[PASS] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] Should successfully scale machine replicas up and down horizontally
[PASS] When testing with disk offering Should successfully create a cluster with disk offering
[PASS] When testing app deployment to the workload cluster [TC1][PR-Blocking] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing subdomain Should create a cluster in a subdomain
[PASS] When testing machine remediation Should replace a machine when it is destroyed
[PASS] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
[PASS] When testing resource cleanup Should create a new network when the specified network does not exist
[PASS] When testing app deployment to the workload cluster with network interruption [ToxiProxy] Should be able to create a cluster despite a network interruption during that process
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified control plane offering is not found [TC7]
[PASS] When the specified resource does not exist Should fail due to the specified template is not found [TC6]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the specified disk offering is not found
[PASS] When the specified resource does not exist Should fail due to the compute resources are not sufficient for the specified offering [TC8]
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is not customized but the disk size is specified
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is customized but the disk size is not specified
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade worker machine due to insufficient compute resources
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade control plane machine due to insufficient compute resources
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When testing Kubernetes version upgrades Should successfully upgrade kubernetes versions when there is a change in relevant fields
[PASS] When testing app deployment to the workload cluster with slow network [ToxiProxy] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time
[PASS] When testing with custom disk offering Should successfully create a cluster with a custom disk offering
[PASS] When testing multiple CPs in a shared network with kubevip Should successfully create a cluster with multiple CPs in a shared network
[PASS] When testing affinity group Should have host affinity group when affinity is anti
[PASS] with two clusters should successfully add and remove a second cluster without breaking the first cluster

Summarizing 1 Failure:

[Fail] When testing affinity group [It] Should have host affinity group when affinity is pro
/jenkins/workspace/capc-e2e-new/test/e2e/common.go:331

Ran 28 of 29 Specs in 9051.873 seconds
FAIL! -- 27 Passed | 1 Failed | 0 Pending | 1 Skipped
--- FAIL: TestE2E (9051.89s)
FAIL

Test Results : (tid-329) Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8 Kubernetes Version: v1.27.2 Kubernetes Version upgrade from: v1.26.5 Kubernetes Version upgrade to: v1.27.2 CloudStack Version: 4.17 Template: ubuntu-2004-kube E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr297-sl-329.zip
Summarizing 1 Failure:

[Fail] When testing affinity group [It] Should have host affinity group when affinity is pro
/jenkins/workspace/capc-e2e-new/test/e2e/common.go:331

I'm going to need some help with this test failure. I'm not entirely sure it is related to the changes.
• [SLOW TEST:432.775 seconds]
When testing multiple CPs in a shared network with kubevip
/jenkins/workspace/capc-e2e-new/test/e2e/shared_network_kubevip_test.go:27
  Should successfully create a cluster with multiple CPs in a shared network
  /jenkins/workspace/capc-e2e-new/test/e2e/shared_network_kubevip.go:59
------------------------------
When testing affinity group
  Should have host affinity group when affinity is pro
  /jenkins/workspace/capc-e2e-new/test/e2e/affinity_group.go:68
STEP: Getting a CloudStack client secret
STEP: Creating a CloudStack client
STEP: Creating a namespace for hosting the "affinity-group" test spec
INFO: Creating namespace affinity-group-5cb2u3
INFO: Creating event watcher for namespace "affinity-group-5cb2u3"
INFO: Creating the workload cluster with name "affinity-group-fm0sv5" using the "affinity-group-pro" template (Kubernetes v1.27.2, 3 control-plane machines, 2 worker machines)
INFO: Getting the cluster template yaml
INFO: clusterctl config cluster affinity-group-fm0sv5 --infrastructure (default) --kubernetes-version v1.27.2 --control-plane-machine-count 3 --worker-machine-count 2 --flavor affinity-group-pro
INFO: Applying the cluster template yaml to the cluster
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/affinity-group-fm0sv5-md-0 created
cluster.cluster.x-k8s.io/affinity-group-fm0sv5 created
machinedeployment.cluster.x-k8s.io/affinity-group-fm0sv5-md-0 created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/affinity-group-fm0sv5-control-plane created
cloudstackcluster.infrastructure.cluster.x-k8s.io/affinity-group-fm0sv5 created
cloudstackmachinetemplate.infrastructure.cluster.x-k8s.io/affinity-group-fm0sv5-control-plane created
cloudstackmachinetemplate.infrastructure.cluster.x-k8s.io/affinity-group-fm0sv5-md-0 created

INFO: Waiting for the cluster infrastructure to be provisioned
STEP: Waiting for cluster to enter the provisioned phase
INFO: Waiting for control plane to be initialized
INFO: Waiting for the first control plane machine managed by affinity-group-5cb2u3/affinity-group-fm0sv5-control-plane to be provisioned
STEP: Waiting for one control plane node to exist
INFO: Installing a CNI plugin to the workload cluster
clusterrole.rbac.authorization.k8s.io/kindnet created
clusterrolebinding.rbac.authorization.k8s.io/kindnet created
serviceaccount/kindnet created
daemonset.apps/kindnet created

INFO: Waiting for control plane to be ready
INFO: Waiting for the remaining control plane machines managed by affinity-group-5cb2u3/affinity-group-fm0sv5-control-plane to be provisioned
STEP: Waiting for all control plane nodes to exist
INFO: Waiting for control plane affinity-group-5cb2u3/affinity-group-fm0sv5-control-plane to be ready (implies underlying nodes to be ready as well)
STEP: Waiting for the control plane to be ready
STEP: Checking all the the control plane machines are in the expected failure domains
INFO: Waiting for the machine deployments to be provisioned
STEP: Waiting for the workload nodes to exist
STEP: Checking all the machines controlled by affinity-group-fm0sv5-md-0 are in the "<None>" failure domain
INFO: Waiting for the machine pools to be provisioned
STEP: Getting a CloudStack client secret
STEP: Creating a CloudStack client
STEP: Listing all machines
STEP: affinity-group-fm0sv5-control-plane-jstr9 is in host ref-trl-5533-k-M7-remoteRequest-kvm1 (429a9e48-241d-413d-8095-9ba7954d9779)
STEP: affinity-group-fm0sv5-control-plane-br7fn is in host ref-trl-5533-k-M7-remoteRequest-kvm1 (429a9e48-241d-413d-8095-9ba7954d9779)
STEP: affinity-group-fm0sv5-md-0-9ph8l is in host ref-trl-5533-k-M7-remoteRequest-kvm3 (580e782c-480e-4f20-832f-d7c18312b8b4)
STEP: affinity-group-fm0sv5-md-0-5fjh7 is in host ref-trl-5533-k-M7-remoteRequest-kvm2 (f48baeea-caaf-4e8b-b255-cef5f435f793)
STEP: Dumping logs from the "affinity-group-fm0sv5" workload cluster
Failed to get logs for machine affinity-group-fm0sv5-control-plane-ffrlf, cluster affinity-group-5cb2u3/affinity-group-fm0sv5: error creating container exec: Error response from daemon: No such container: affinity-group-fm0sv5-control-plane-ffrlf
Failed to get logs for machine affinity-group-fm0sv5-control-plane-hb8s2, cluster affinity-group-5cb2u3/affinity-group-fm0sv5: error creating container exec: Error response from daemon: No such container: affinity-group-fm0sv5-control-plane-hb8s2
Failed to get logs for machine affinity-group-fm0sv5-control-plane-zgzhd, cluster affinity-group-5cb2u3/affinity-group-fm0sv5: error creating container exec: Error response from daemon: No such container: affinity-group-fm0sv5-control-plane-zgzhd
Failed to get logs for machine affinity-group-fm0sv5-md-0-5c9c997966-gnht5, cluster affinity-group-5cb2u3/affinity-group-fm0sv5: error creating container exec: Error response from daemon: No such container: affinity-group-fm0sv5-md-0-5c9c997966-gnht5
Failed to get logs for machine affinity-group-fm0sv5-md-0-5c9c997966-hrpkd, cluster affinity-group-5cb2u3/affinity-group-fm0sv5: error creating container exec: Error response from daemon: No such container: affinity-group-fm0sv5-md-0-5c9c997966-hrpkd
STEP: Dumping all the Cluster API resources in the "affinity-group-5cb2u3" namespace
STEP: Deleting cluster affinity-group-5cb2u3/affinity-group-fm0sv5
STEP: Deleting cluster affinity-group-fm0sv5
INFO: Waiting for the Cluster affinity-group-5cb2u3/affinity-group-fm0sv5 to be deleted
STEP: Waiting for cluster affinity-group-fm0sv5 to be deleted
STEP: Deleting namespace used for hosting the "affinity-group" test spec
INFO: Deleting namespace affinity-group-5cb2u3
STEP: Getting a CloudStack client secret
STEP: Creating a CloudStack client

• Failure [343.292 seconds]
When testing affinity group
/jenkins/workspace/capc-e2e-new/test/e2e/affinity_group_test.go:27
  Should have host affinity group when affinity is pro [It]
  /jenkins/workspace/capc-e2e-new/test/e2e/affinity_group.go:68

  affinity-group-fm0sv5-md-0-5fjh7 is deployed in a different host: ref-trl-5533-k-M7-remoteRequest-kvm2 when affinity type is pro

  /jenkins/workspace/capc-e2e-new/test/e2e/common.go:331

  Full Stack Trace
  sigs.k8s.io/cluster-api-provider-cloudstack-staging/test/e2e.CheckAffinityGroup(0xc00058c500, {0xc0010123a8, 0x15}, {0x2654374, 0x3})
  	/jenkins/workspace/capc-e2e-new/test/e2e/common.go:331 +0x4e5
  sigs.k8s.io/cluster-api-provider-cloudstack-staging/test/e2e.executeTest({0x29f9d20, 0xc000058148}, {0xc000d66a50, {0xc000648afb, 0x4c}, {0x2a08a08, 0xc000abd8c0}, {0xc000648a90, 0x2a}, 0x0, ...}, ...)
  	/jenkins/workspace/capc-e2e-new/test/e2e/affinity_group.go:111 +0x60e
  sigs.k8s.io/cluster-api-provider-cloudstack-staging/test/e2e.AffinityGroupSpec.func2()
  	/jenkins/workspace/capc-e2e-new/test/e2e/affinity_group.go:69 +0x97
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0x0?)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:113 +0xb1
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc00019b860?)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:64 +0x125
  github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0x48?)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/it_node.go:26 +0x7b
  github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc00063e690, 0xc000c2b9f0?, {0x29db980, 0xc0000aa900})
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:215 +0x2a9
  github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc00063e690, {0x29db980, 0xc0000aa900})
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:138 +0xf2
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0002d4b00, 0xc00063e690)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:200 +0xf1
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0002d4b00)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:170 +0x1b6
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0002d4b00)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:66 +0xc5
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc000184c40, {0x7f877a71eaa8, 0xc00019b6c0}, {0x265b87d, 0x8}, {0xc000131040, 0x2, 0x2}, {0x29fbbc0, 0xc0000aa900}, ...)
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/suite/suite.go:79 +0x4e5
  github.com/onsi/ginkgo.runSpecsWithCustomReporters({0x29de6a0?, 0xc00019b6c0}, {0x265b87d, 0x8}, {0xc000131020, 0x2, 0x267d76e?})
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:245 +0x189
  github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters({0x29de6a0, 0xc00019b6c0}, {0x265b87d, 0x8}, {0xc000093f50, 0x1, 0x1})
  	/root/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:228 +0x1be
  sigs.k8s.io/cluster-api-provider-cloudstack-staging/test/e2e.TestE2E(0x2779540?)
  	/jenkins/workspace/capc-e2e-new/test/e2e/e2e_suite_test.go:100 +0xba
  testing.tRunner(0xc00019b6c0, 0x277ce18)
  	/usr/local/go/src/testing/testing.go:1446 +0x10b
  created by testing.(*T).Run
  	/usr/local/go/src/testing/testing.go:1493 +0x35f

@hrak The failure is an intermittent one. Have seen that test fail sometimes on other PRs as well.

/approve

/lgtm


Proposal: further refactor widgets

Following the recent PRs on refactor the widgets, here is a proposal to further refactor widgets.
See the files changed tab for an example.
Here are the proposed changes:

Create WidgetInput components so for example same WidgetInputAudio component can be shared between WidgetAudioClassifcation, WidgetAudioToAudio, ASRWidget etc. All WidgetInput components need to export bindable object that implements Widget Input interface.
Create WidgetOutput components so for example same WidgetOutputClassification component can be shared between WidgetAudioClassifcation, WidgetTextClassifcation etc. All WidgetOutput components need to export bindable object that implements Widget Output interface.
And each individual widgets is just a set of WidgetInput and WidgetOutput component without any functions (all functions). See old WidgetAudioClassifcation vs proposed WidgetAudioClassifcation
WidgetWrapper will do more. validateExample, applyInputSample, get api inference output are directly implemented only once in WidgetWrapper

would love to hear your opinions @SBrandeis @julien-c @coyotte508 @gary149 @Pierrci
if this proposal is no good, I will implement all the remaining output validations (see this comment) and that should be the end of the widgets sprint (can also work a bit on the widget docs)

Reviewing this today!

Thanks. It is a proposal. So I only implemented fkr AudioClassification widget. And if/when we agree on decisions, will implement for all widgets

Thanks. It is a proposal. So I only implemented fkr AudioClassification widget. And if/when we agree on decisions, will implement for all widgets

// WidgetWrapper
<script>
    export let WidgetInputComponent;
    let widgetInput;

    export let WidgetOutputComponent;
    let widgetOutput:

    let error = ""

    function validateExample(){
        widgetInput.validateExample() && widgetOutput.validateExample()
    }

    function callApi(){
        reqBody = widgetInput.getReqBody();
        output = callInferenceApi(reqBody)
        widgetOutput.renderOutput(output)
    }
</script>

<WidgetHeader/>

<WidgetInputComponent bind:widgetInput/>
<WidgetOutputComponent bind:widgetOutput/>

<WidgetFooter/>

@huggingface/inference, we probably won't need a validateOutput function anymore?

oh indeed. It would make sense to use @huggingface/inference then rather than trying to reinvent validation here. I will look into it


Access history of deployments

From @sebgoa on October 10, 2016 14:21
Deployments have history.
$ kubectl rollout history deployments ghost
deployments "ghost"
REVISION    CHANGE-CAUSE
1       <none>

In each deployment we should display the history like we do exec and logs.
selecting the history should show a 'table' of the different history with their cause.
And swiping left one of those should trigger a rollback to that revision.
A button at the bottom should trigger a rollback to the last revision (kubectl rollout undo...)
underneath it fetches the replica sets that manages the matching pods and switches between them.
Copied from original issue: skippbox/cabin-sw#88

From @kemcake on October 10, 2016 15:44
This could be a really nice feature 👍 being able to rollback easily

it is an annotation:
I1011 11:11:09.486096   52027 round_trippers.go:318] GET https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/deployments/ghost 200 OK in 54 milliseconds
I1011 11:11:09.486136   52027 round_trippers.go:324] Response Headers:
I1011 11:11:09.486153   52027 round_trippers.go:327]     Content-Type: application/json
I1011 11:11:09.486195   52027 round_trippers.go:327]     Date: Tue, 11 Oct 2016 09:11:09 GMT
I1011 11:11:09.486222   52027 round_trippers.go:327]     Content-Length: 1034
I1011 11:11:09.486339   52027 request.go:891] Response Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"ghost","namespace":"default","selfLink":"/apis/extensions/v1beta1/namespaces/default/deployments/ghost","uid":"8b255dd5-8f92-11e6-bfdf-064b8000021e","resourceVersion":"72673","generation":2,"creationTimestamp":"2016-10-11T09:10:24Z","labels":{"run":"ghost"},"annotations":{"deployment.kubernetes.io/revision":"1","kubernetes.io/change-cause":"kubectl run ghost --image=ghost --record"}},"spec":{"replicas":1,"selector":{"matchLabels":{"run":"ghost"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"run":"ghost"}},"spec":{"containers":[{"name":"ghost","image":"ghost","resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","securityContext":{}}},"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":1,"maxSurge":1}}},"status":{"observedGeneration":2,"replicas":1,"updatedReplicas":1,"availableReplicas":1}}
I1011 11:11:09.487301   52027 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.7 (darwin/amd64) kubernetes/a2cba27" -H "Authorization: Basic Zm9vYmFyOkZkS1BTdXdR" https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/replicasets?labelSelector=run%3Dghost
I1011 11:11:09.519533   52027 round_trippers.go:318] GET https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/replicasets?labelSelector=run%3Dghost 200 OK in 32 milliseconds
I1011 11:11:09.519598   52027 round_trippers.go:324] Response Headers:

kubectl --v=99 rollout history deployments ghost

From @kemcake on October 11, 2016 9:14
ok kubernetes.io/change-cause key, not present if none

From @kemcake on October 11, 2016 9:8
Do you know where should I get the "change cause" ? And how can I get a deployments with a change-cause different than "none" ?

ah no, my bad there is a rollback API to deployments
POST https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/deployments/ghost/rollback

From @kemcake on October 11, 2016 9:18
yes that's already what I did :)

if you check the trace of:
kubectl --v=99 rollout history deployments ghost

you will see that it gets the deployments, and then it gets the list of replicasets that have the matching pod labelselector.
 GET https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/replicasets?labelSelector=run%3Dghost

The history is built by printing the list of replicasets and the annotation that they contain if present.
The roll back will be a bit tricky, as we need to scale the replicasets up and down.

From @kemcake on October 11, 2016 17:13
Also @sebgoa

A button at the bottom should trigger a rollback to the last revision (kubectl rollout undo...)

But, tapping on the last revision in the list should do the job, do we need a button for that ? Or maybe a button clearly saying "Rollback to last revision" ?

From @kemcake on October 11, 2016 9:49
we receive this:
I1011 11:47:36.716904   24048 request.go:891] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"expected pointer, but got invalid kind","code":500}

on the command line, while it seems to work
deployment "ghost" rolled back

and in the request body of a rollback you can pass a revision number:
I1011 11:21:54.622256   52203 request.go:558] Request Body: {"kind":"DeploymentRollback","apiVersion":"extensions/v1beta1","name":"ghost","rollbackTo":{"revision":1}}
I1011 11:21:54.622334   52203 round_trippers.go:299] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.3.7 (darwin/amd64) kubernetes/a2cba27" -H "Authorization: Basic Zm9vYmFyOkZkS1BTdXdR" -H "Accept: application/json, */*" https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/deployments/ghost/rollback
I1011 11:21:54.676258   52203 round_trippers.go:318] POST https://<IP_ADDRESS>:443/apis/extensions/v1beta1/namespaces/default/deployments/ghost/rollback 201 Created in 53 milliseconds

no you are right. I tested what you did, it works fine.
The view is a bit "lame" but not sure what to do to make it a bit more "fun"

yes +1

From @kemcake on October 13, 2016 13:8

Way better ha ?

From @kemcake on October 12, 2016 15:11
True, I'll add some icon or color to make it more lively

tested, that works.


Modern Web Part browse back JavaScript fails to trigger onInit

Note: Related Item #3095 was prematurely closed
Category

[ ] Question
[ ] Typo
[x] Bug
[ ] Additional article idea

Expected or Desired Behavior
When browsing Back to a page, the JavaScript should run again when clicking back from an item in a Modern Web Part.
JavaScript should run on page load and again when navigating back from an Announcement or Calendar item.
Observed Behavior
JavaScript does not run when the page is returned to using the Back Browse feature (Backspace Button or Click to Go Back browsing arrow).  This occurs on a Modern Communications Site when navigating back from a Calendar or Announcement item.  I have tried mapping to the unload event with no luck.
The Standard apps provided in the spfx-40-fantastics (Rich Text Editor) and Script Editor web part solutions also exhibit this bug.
The result is that JavaScript solution developed using the onInit promise of the SPFx framework will fail under these circumstances.
Steps to Reproduce
This is reproducible using a new Communications Site and the Build your first SharePoint Framework tutorial documentation.  https://docs.microsoft.com/en-us/sharepoint/dev/spfx/extensions/get-started/build-a-hello-world-extension
Create an announcement item or calendar item first.  Now from the page where the SPFx Extension is running, click on the new Announcement or calendar item.  Use the Back button (or hit backspace) to go back to the page.  JavaScript will not trigger.

So again - I don't think there is anything incorrect here.  OnInit fires when the extension is initialized.  It isn't initialized during partial-page navigates.
If you want to trigger something on a navigate, you should hook onto the OnNavigate event.


Can't use borrowing and a HashMap

Deriving deserialize for a structure if failing and I've narrowed it down to this:
#[derive(Deserialize)]
struct Inner<'a> {
    username: Option<std::collections::HashMap<&'a[u8], &'a[u8]>>,
}

it fails with:
   Compiling playground v0.0.1 (file:///playground)
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'de` due to conflicting requirements
 --> src/main.rs:6:10
  |
6 | #[derive(Deserialize)]
  |          ^^^^^^^^^^^
  |
note: first, the lifetime cannot outlive the lifetime 'de as defined on the impl at 6:10...
 --> src/main.rs:6:10
  |
6 | #[derive(Deserialize)]
  |          ^^^^^^^^^^^
  = note: ...so that the types are compatible:
          expected serde::de::SeqAccess<'_>
             found serde::de::SeqAccess<'de>
note: but, the lifetime must be valid for the lifetime 'a as defined on the impl at 6:10...
 --> src/main.rs:6:10
  |
6 | #[derive(Deserialize)]
  |          ^^^^^^^^^^^
  = note: ...so that the types are compatible:
          expected serde::Deserialize<'_>
             found serde::Deserialize<'_>

Reproduction: http://play.rust-lang.org/?gist=60a0b51a951cd4ed27b9d804fff0000f&version=stable&mode=debug&edition=2015

The structure works when it's:
#[derive(Deserialize)]
struct Inner<'a> {
    username: Option<std::collections::HashMap<String, String>>,
}

I believe this needs to be written with a #[serde(borrow)] attribute to indicate that the 'de lifetime of the Deserialize impl must outlive 'a.
#[derive(Deserialize)]
struct Inner<'a> {
    #[serde(borrow)]
    username: Option<HashMap<&'a [u8], &'a [u8]>>,
}

@dtolnay that was it! hopefully this ticket is searchable and helps the next person 🌟


Add CSV import without translation.

For #19627, this is a temporary solution in the meantime - prior to implementing a full-blown Spreadsheet resource class that can work with CSV files.

@akien-mga Updated

@akien-mga FYI: I have updated this so that it no longer breaks compatibility.

Thanks!


examples/udp is using NuttX's CONFIG_ENDIAN_BIG for host OS program

the same issue as https://github.com/apache/incubator-nuttx-apps/issues/150

Fixed by #156, let's close this issue now.

no, https://github.com/apache/incubator-nuttx-apps/pull/156 doesn't fix this.


Show values of terminals in errors, instead of names, when appropriate

So, first thing would be to define "apropiate" .
Something like :

If terminal comes from a simple string and string length is less than some integer, except from debug mode in that case show full string
If terminal comes from union of strings and total length less than some int... same a before.
If terminal comes from anon regex it would be great to show them in debug mode  or at least show them as anon_regex_n
if terminal is an anon mix strings and regex show something like anon_mix_n

Is there other kinds of terminals?

Even when not in debug mode, /[A-Z]+\-\d+/ is still more informative than __ANON_17. But if you stick with ANON, then can we get them numbered by their line and column in the grammar? So __ANON_17 would be __ANON_L45_C23 or similar. Actually, that labeling scheme for anonymous terminals is probably a good idea regardless of what we do with error messages.

@charles-esterbrook That's a nice idea!

Is someone working on this right now? Otherwise I will give it a try.

Go ahead. I didn't get started yet, and I don't think anyone else did.


fix CI issues

This PR:

fixes issues with CI tests

Checklist

[x] Update changelog in CHANGELOG.md.

this is a WIP until i've fixed all the issues - i don't want to mark it as draft though as i need CI to run each time I push changes.

OK now it works :D


Rp2040SdioSetup.ino with "SdError 0X1E, 0X0"

I really appreciate the SDIO expansion for Rp2040. This is exactly what I need.
However, while both Rp2040SdioSetup.ino and bench.ino run on one of my raspberry pi picos flawlessly, I struggle with another. I am pretty sure it is NOT a wiring issue.
Rp2040SdioSetup outputs:
begin() failed
Do not reformat the SD.
SdError: 0X1E,0X0
What does this error code mean? what could be the issue?

This is a timeout while waiting for data from from the SD.
It is at one of three places in the RP2040  SDIO driver.
libraries\SdFat\src\SdCard\Rp2040Sdio\PioSdioCard.cpp (3 hits)
	Line 385:         sdError(SD_CARD_ERROR_READ_TIMEOUT);
	Line 465:           sdError(SD_CARD_ERROR_READ_TIMEOUT);
	Line 493:         sdError(SD_CARD_ERROR_READ_TIMEOUT);

I am pretty sure it is NOT a wiring issue.

It is could be a configuration problem. Do these match your wiring?
#define RP_CLK_GPIO <your CLK>
#define RP_CMD_GPIO <your CMD>
#define RP_DAT0_GPIO <Your DAT0>

Another possibility is that the wiring/socket distorts the signals since the timing is critical for SDIO.
I can't help much since I can't look at your problem with a scope or logic analyzer.

thank you for your reply.
Unfortunately, my scope is too slow for a meaningful analysis. I can see 30 MHz clock pulses, but the edges are flattened out very much by the bandwidth of the scope.
Your guess regarding a distortion is probably right. I have modified an existing pcb that used to use SPI for addressing the SD card. I changed the control lines for SDIO control and ended up with relatively long wires (10 cm, maybe) without proper return paths. Probably, there is lots of cross talk between the lines and other logic.
With that in mind, it probably does not make much sense to dig deeper with a logic analyzer. It is likely, that the cause is not software, but hardware.
I will include SDIO control in the next pcb revision and see, if it works.


OLS-431: Refactor evaluate model code a bit

Description
Refactor evaluate model a bit
Type of change

[x] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library
[ ] Bump-up library or tool used for development (does not change the final image)

Related Tickets & Documents

Related Issue #OLS-431

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.01%. Comparing base (32c2c70) to head (f79c9e8).
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1165      +/-   ##
==========================================
- Coverage   95.46%   93.01%   -2.45%
==========================================
  Files          56       56
  Lines        2380     2378       -2
==========================================
- Hits         2272     2212      -60
- Misses        108      166      +58

see 5 files with indirect coverage changes

/approve


Updated dependencies

There are some other dependencies that I have included in my project that I have not included on this pull-request, some of them are interesting like React-Native-Material-UI but I'm thinking probably it would be better to remove dependencies and boilerplate from this project to make it more barebones, since there are thing like redux-thunk that I'm not using and in fact I've not been able to find any usage on the template default project...

I am planning to add react-scripts as a dependency and then we just need to upgrade react-scripts and everything else will be taken care of by it.
Since now react-native web has a babel plugin, so maybe we would not need to eject scripts and would be optional for the end user.
I am experimenting with react-scripts, give me a couple of days.
Regarding redux thunk, plan was to add an example but that got lost somewhere. I'll remove it in next release because most people won't need it.

I am planning to add react-scripts as a dependency and then we just need to upgrade react-scripts and everything else will be taken care of by it.

"You don’t need to install or configure tools like Webpack or Babel. They are preconfigured and hidden so that you can focus on the code."
Seems pretty nice... :-D

Any update on the react-scripts thing?

I have just updated all the dependencies except the React-related ones, because you are working on your branch using react-scripts. I have also updated the webpack configuration to match it.

with react-scripts, all the dependencies will be up to date (most of the times) and since react-native-web now has a babel plugin so I have not ejected scripts and config which means that webpack config files are hidden by default which makes sense for most of the beginners. What we can do is, eject the config and scripts on a different branch and merge this P.R. in that and all the advanced users can make use of that branch (or a separate template on npm). Ideas and suggestions are welcome. 😄

Obviously, if react-scripts makes all the hard work, then this is a bit useless :-P I like the advanced users, thought. In fact, one of my ideas was to have barebones projects that don't add any external module (redux, react-router, etc...) and just print a "hello world" message on screen... but can do it in all the platforms with just a single click :-D Then it can be used as basis to build other more advanced templates by using it as dependency. For example, at my work we have added redux-offline and react-native-material-ui and seems to be for us like a cool basis to start working on all our future projects :-) But obviously, it doesn't make sense to add this dependencies to re-start or being installing them each time we create a new project...

Other option is to merge this, and keep working with react-scripts and other similar big changes on a diferent branch to prepare a next 0.5.0 version :-)

This makes sense to me. We can merge this in master as of now.

In fact, one of my ideas was to have barebones projects that don't add any external module (redux, react-router, etc...) and just print a "hello world" message on screen... but can do it in all the platforms with just a single click :-D Then it can be used as basis to build other more advanced templates by using it as dependency.

This can be part of the cli, we can ask the user about the kind of template they want and init their project with that template.

This makes sense to me. We can merge this in master as of now.

Ok, let's go for it :-)

This can be part of the cli, we can ask the user about the kind of template they want and init their project with that template.

So instead of install the templates with react-native cli, do it with a custom cli command specific for re-start templates (that are a superset of standard react-native templates)? Besides the execution of install scripts to configure the package.json file of the created projects (that probably there could be some hack for that), what drawbacks has the standard mechanims?


Get data about what devices each facility is using (like who is using mobile) for workflows (like submitting test results)

Have this in app insights


Start script without npm

Hi,
I want to start my app without npm but it doesn't work.
I have a code like this in index.js:
module.exports = async function (req, res) {
  ...
}

And starting it with npm works fine.
But if I try something like:
const srv = micro(async function (req, res) {
  ...
})
srv.listen(3000)

And starting it with node index.js I get an error: Unexpected token function
I'm using node v7.3.0.
Thanks

if you use the micro bin then it will automagically apply async-to-gen.
So you need to use that or start node with node --harmony-async-await index.js

You'll have to add this to the top of the file:
require('async-to-gen/register')
const micro = require('micro')

Then async-to-gen will convert the code. You could also transpile it using async-to-gen index.js > compiled.js and then start it using node compiled.js.

I use this solution:
npm install --save async-to-gen

And then I add in index.js
require('async-to-gen/register')

In this way I could start it with node index.js
Thanks guys, good work!


2022-10-06-BSC-Token-Hub

BSC Token Hub exploit with over $120 Million funds lost

/qualitycheck

So, It needs to pass every statement? And facts should be verifiable from sources?

So, It needs to pass every statement? And facts should be verifiable from sources?

Sometimes, the bot gets things wrong. Someone from the wiki team will go through everything manually. That said, you can help the bot and the team by adding markdown links directly in the failed claims.

/qualitycheck

Sources have been added, for the failed claims, and some details omitted. When this PR will be merged, I would continue on other incidents 👏🏻

@evgenydmitriev Done 👍🏻

/qualitycheck

There is links to the sources inside the report, covering most failed claims

/payout


fix: Promises in Table buttons

Description
Support for promises in Table buttons
Fixes #11122
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

TODO

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

/ok-to-test sha=73716d7

/ok-to-test sha=7e69439

/ok-to-test sha=7e69439

/ok-to-test sha=4d73896

/ok-to-test sha=4d73896

/ok-to-test sha=4d73896


Flatbuffer binary encoding of existing json spec?

Would a flatbuffer binary encoding of the existing json protocol spec for the matrix events be considered?
I experimented with encoding the existing json schema as binary using flatbuffers, and it did appear to work alright for m.room.member_example.json. I presume it would be fine for other messages, and that the flatbuffers schemas could be automatically generated with a tool from the matrix-spec repo (Meaning potentially no need to maintain a separate binary protocol specification).
I published my test here - https://github.com/matrix-hacks/matrix-flatbuffers
... I believe this would allow clients and servers to completely ditch the need for json parsing/encoding which relieves clients of a lot of their code size and complexity, while increasing message handling performance significantly -- all relevant for minimal/embedded clients written in C and other low level languages.
Flatbuffers already has language bindings for Go (and python and C++ and javascript and many others).

I should also point out, the flatbuffer binary message can be fully roundtripped back to json. i.e. I can parse m.room.member_example.json into binary form, then take that binary form and convert it back out into the same json it started out as.


Multiple cases in switch statement

Hi,
The metrics are not reliable when the code uses a switch case in this way:
switch (value)
{
case 1:
//do some stuff
break;
case 2:
case 3:
case 4:
case 5:
case 6:
default:
//default stuff
break;
}
Instead of counting 2 'real' branches, the metrics reflect 7.
Perhaps this can be improved by not counting nested cases and default.
Regards

After some reading, I learned that switch case statement should not be used due they are prone to errors. Therefore metrics are quite correct.
Regards