Remove unwanted KubeArmor service running in kube-system namespace

Feature Request
Short Description
Remove KubeArmor services running in kube-system namespace.
Is your feature request related to a problem? Please describe the use case.
Basically, these are auto-generated services by kubebuilder  and currently they are of no use.
To see those service,
[linuzz@fedora ~]$ sudo microk8s kubectl get svc -n kube-system | grep kubearmor
kubearmor                                       ClusterIP   <IP_ADDRESS>   <none>        32767/TCP                8d
kubearmor-policy-manager-metrics-service        ClusterIP   <IP_ADDRESS>   <none>        8443/TCP                 8d
kubearmor-host-policy-manager-metrics-service   ClusterIP   <IP_ADDRESS>   <none>        8443/TCP                 8d

Describe the solution you'd like
Clean up those services.
Describe alternatives you've considered

Hey @daemon1024 , since I don't have permission to add cleanup tag over enhancement tag. So,  can you do it.

/assign

I don't have permission to add cleanup tag over enhancement tag. So, can you do it.

enhancement should be fine.

kubearmor                                       ClusterIP   <IP_ADDRESS>           32767/TCP

This is an important service, we connect to this for logs/telemetry.

kubearmor-policy-manager-metrics-service        ClusterIP   <IP_ADDRESS>           8443/TCP                 8d
kubearmor-host-policy-manager-metrics-service   ClusterIP   <IP_ADDRESS>           8443/TCP

I would like to understand what potential usecase would this have before proceeding to remove it. We might consider leveraging those usecases as well if they deem fit.

Hey @daemon1024 , as you said that below service

kubearmor ClusterIP <IP_ADDRESS> 32767/TCP

is used in providing logs and telemetry.
But, I deleted all 3 services from my cluster.

kubearmor ClusterIP <IP_ADDRESS> 32767/TCP
kubearmor-policy-manager-metrics-service ClusterIP <IP_ADDRESS> 8443/TCP 8d
kubearmor-host-policy-manager-metrics-service ClusterIP <IP_ADDRESS> 8443/TCP

Then also karmor cli provides me logs and telemetry.  I don't understand why so ?
At present time these all are services running in my cluster
kubectl get svc   -n kube-system
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
kube-dns                    ClusterIP   <IP_ADDRESS>    <none>        53/UDP,53/TCP,9153/TCP   17d
metrics-server              ClusterIP   <IP_ADDRESS>   <none>        443/TCP                  17d
kubernetes-dashboard        ClusterIP   <IP_ADDRESS>   <none>        443/TCP                  17d
dashboard-metrics-scraper   ClusterIP   <IP_ADDRESS>    <none>        8000/TCP                 17d

The kubearmor service here is part of the https://github.com/kubearmor/kubearmor-relay-server, if you check the selector it would have kubearmor-app: kubearmor-relay

But, I deleted all 3 services from my cluster.
Then also karmor cli provides me logs and telemetry. I don't understand why so ?

What you are doing right now is essentially connecting to kubearmor directly. This only works because you are using kArmor on the same node. Also if there are multiple kubearmor in your cluster ( due to multiple nodes) you won't be getting the aggregated feed. To know more about the relay server you can check out the README.

Yeah, that could be the reason. Ok, I will also try to check it by installing both(k8s cluster and karmor cli) of them in diffferent nodes.

Basically there are 3 deployments,
kubectl get deploy -n kube-system | grep kubearmor
kubearmor-relay                 1/1     1            1           17d
kubearmor-policy-manager        1/1     1            1           17d
kubearmor-host-policy-manager   1/1     1            1           17d

kubearmor-relay and
kubearmor ClusterIP <IP_ADDRESS> 32767/TCP

are related to each other.
Similarly,

kubearmor-policy-manager deployment and
kubearmor-policy-manager-metrics-service ClusterIP <IP_ADDRESS> 8443/TCP 8d     service

are related to each other.
And,

kubearmor-host-policy-manager         deployment
kubearmor-host-policy-manager-metrics-service ClusterIP <IP_ADDRESS> 8443/TCP            service

aree related to each other.

What does deployment kubearmor-host-policy-manager  and kubearmor-policy-manager  does. What is their use?

And both deployment uses kubebuilder images, i.e. gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
https://github.com/kubearmor/KubeArmor/blob/4faeb7f825c175ac96600c1ba0b58085b2765bd0/deployments/microk8s/kubearmor.yaml#L215

Hey @daemon1024 , I think controllers defined for deployments kubearmor-host-policy-manager and kubearmor-policy-manager will handle policy validation as well as internal logic. There is no need of services corresponding to  those deployment.
I have also tried to proof it.
Basically I ran the policy in presence of deployment  kubearmor-host-policy-manager and its service kubearmor-policy-manager-metrics-service.
policy.yaml
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-group-1-proc-path-block
  namespace: multiubuntu
spec:
  severity: 5
  message: "block the sleep command"
  selector:
    matchLabels:
      group: group-1
  process:
    matchPathss:
      - path: /bin/sleep # try sleep 1 (permission denied)
  action:
    Block

Then it return below errors(because the above policy has matchPathss instead of matchPath:
linuzz@hp:~/go/src/github.com/kubearmor/KubeArmor/examples$ kubectl apply -f multiubuntu/security-policies/ksp-group-1-proc-path-block.yaml
error: error validating "multiubuntu/security-policies/ksp-group-1-proc-path-block.yaml": error validating data: ValidationError(KubeArmorPolicy.spec.process): unknown field "matchPathss" in com.kubearmor.security.v1.KubeArmorPolicy.spec.process; if you choose to ignore these errors, turn validation off with --validate=false

After that I deleted the service kubearmor-policy-manager-metrics-service, and repeated the above scenario and got same errors as above.
So, I  don't think there is any role of the kubearmor-policy-manager-metrics-service` service.
What do you think?

@daemon1024 , any thoughts  ??

@viveksahu26 The Validation that you tried out right now is the validation performed by k8s based on the CRD specification and not by the controller.
You can see here what the controller validates for
https://github.com/kubearmor/KubeArmor/blob/bf6087fd0bc04315ccc3afd9babd72f215ec77ac/pkg/KubeArmorPolicy/controllers/kubearmorpolicy_controller.go#L123
Like here it checks if we mandate an allow action to be used with owner only
Thanks a lot for spending time to analyse things here. Really appreciate it.

@viveksahu26 The Validation that you tried out right now is the validation performed by k8s based on the CRD specification and not by the controller.

Yes, it's CRD specs.

@daemon1024 , Do you want me to research more on above services functionality. Or should I remove it. As I couldn't see any role of those services.

Found an interesting docs  on image(gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0) used by deployments(kubearmor-host-policy-manager and kubearmor-policy-manager). kube-rbac-proxy is a small HTTP proxy for a single upstream, that can perform RBAC authorization against the Kubernetes API using SubjectAccessReview.SubjectAccessReview.

@daemon1024 , Any thoughts on this ??


fix(ngMessages): don't crash when first transcluded message is removed

Please check if the PR fulfills these requirements

[x] The commit message follows our guidelines: https://github.com/angular/angular.js/blob/master/CONTRIBUTING.md#commit-message-format
[x] Tests for the changes have been added (for bug fixes / features)

Other information:
Under specific circumstances, ngMessages would go into an infinite loop and crash the
browser / page:

At least two ngMessage elements are wrapped inside another element (e.g. ngTransclude)
The first message is currently visible
The first message is removed (e.g. when the whole ngMessages element is removed by an ngIf)

When a message is removed, it looks for a previous message - in this specific case it would misidentify
the second message for a previous message, which would then cause the first message to be marked as the
second message's next message, resulting in an infinite loop, and crash.
This fix ensures that only messages with an index lower than the starting message are considered
previous elements.
Fixes #14183
@gkalpak you triaged the original issue, do you mind reviewing this?

I believe this fix could result in different behavior, since now the order of messages is determined by the order of registration and not their place in the DOM. E.g.:
<ng-messages for="someObj">
  <include-messages>

  </include-messages>
  <ng-message when="error3">...</ng-message>
  <ng-message when="error4">...</ng-message>
</ng-messages>

--
@Narretz, can you confirm that this is indeed tha case ?
If yes, can we avoid it ?

@gkalpak Thanks for looking into this. As far as I can see, I don't think the order is compromised. I've added a test to make sure it doesn't happen. Does the test reflect the case you had in mind? You can also add another test, don't be shy ;>
I've also fixed a memory leak with ngTransclude in the test that happens when ngIf destroys the ngMessages container. If only one message is currently visible, ngMessages deregisters this message and then tries to render the second message. The second message never gets its destroy handler triggered, but somehow holds onto the transclusion scope, which can't be collected then. I'm not sure if that's an issue in the real world, though.

The problem becomes obvious when removing one of the messages. Here is a failing testcase:
  it('should properly detect the previous message, even if it was registered later',
    inject(function($compile, $rootScope, $templateCache) {
      $templateCache.put('include.html', '
A
');
      var html =
          '
' +
            '

' +
            '
B
' +
            '
C
' +
          '
';

      element = $compile(html)($rootScope);
      $rootScope.$apply('items = {b: true, c: true}');

      expect(element.text()).toBe('B');

      var nodeB = element[0].querySelector('[ng-message="b"]');
      angular.element(nodeB).remove();
      $rootScope.$apply('items.a = true');

      expect(element.text()).toBe('A');
    })
  );

(Removing the element like this is kind of contrived, but I am pretty sure the same issues could be reproduced with ngIf and a remote template for example.)

Thanks for the test!
This is definitely a problem. I've pushed a new commit that makes this and the original test pass, but it only works if there's just one level of nesting when you remove the element directly:
        '
' +
          '
' +
            '
Fill in the text field.
' +
            '
Extra error message.
' +
          '
' +
        '
'

More levels of nesting still produce the same problem:
   '
' +
      '
' +
        '
' +
          '
Fill in the text field.
' +
          '
Extra error message.
' +
        '
' +
        '
Different error message.
' +
      '
' +
    '
'

It's quite difficult to maintain a notion of previous element, when the nesting becomes more complex.
But the problem will only happen if someone removes a single message from the DOM, something that shouldn't be necessary. The original issue - removing the complete ngMessages element with an ngIF - is already fixed by removing all message nodes when the ngMessages element is destroyed.
So we basically two options - just fix the original issue, or improve the querying for previousNodes. Maybe a top-down search is better then searching from the comment element itself.

TBH, I'm still not sure what caused the original issue. I read your description in the commit message:

When a message is removed, it looks for a previous message - in this specific case it would misidentify the second message for a previous message, which would then cause the first message to be marked as the second message's next message, resulting in an infinite loop, and crash.

But I haven't looked into why/how can the 2nd message be identified as previous to the 1st one (is it because it becomes the ctrl.head) ? And even then, why does the 1st message become the next for the 2nd one ?
Maybe we could fix that logic.

So the problem is in this function: https://github.com/angular/angular.js/blob/aa077e81129c740041438688dff2e8d20c3d7b52/src/ngMessages/messages.js#L444
When you have DOM like this:
'' +
'' +
'Fill in the text field.' +
'Extra error message.' +
'' +
''
and the "required" message element gets removed, findPreviousMessage looks upwards in the DOM to find a previous message. Since required has no previousSibling, the wrapper element is the next prevNode: https://github.com/angular/angular.js/blob/aa077e81129c740041438688dff2e8d20c3d7b52/src/ngMessages/messages.js#L459. The code then assigns the last child of the wrapper as the prevNode, and since this is the "extra" message it is mis-identied as a previous node. Later, the "required" message is set as the next message for the "required" message: https://github.com/angular/angular.js/blob/aa077e81129c740041438688dff2e8d20c3d7b52/src/ngMessages/messages.js#L485
The problem is really that the nesting inside ngMessages can be arbitrarily complicated, but this wasn't really planned, because you should only use ng-message and ng-message-include. I can understand if someone has an ng-transclude, but more nesting shouldn't really be necessary.
Searching for previous nodes can definitely be improved, but I'm not sure if it's worthwhile. One possibility is to start the search from the ngMessages element, and stop when you find the passed in the comment element.

From a quick look, it seems that the actual bug is on messages.js#L455: It shouldn't look into the comment element's parent, because it has already looked in there (since it started at comment.
I haven't tried it, but I believe that initializing parentLookup with comment.parentNode on messages.js#L446 should fix the bug. E.g.:
var parentLookup = [comment.parentNode];

This fix works if there's only one level of nesting. It's the same as doing:
             if (prevNode.childNodes.length &&
                prevNode !== comment.parentNode && // this is new
                parentLookup.indexOf(prevNode) == -1
              ) {
               parentLookup.push(prevNode);
               prevNode = prevNode.childNodes[prevNode.childNodes.length - 1];
             } else {
               prevNode = prevNode.previousSibling || prevNode.parentNode;
             }

But both fixes don't work if there's a DOM like this:
    '
' +
      '
' +
        '
' +
          '
Fill in the text field.
' +
          '
Extra error message.
' +
        '
' +
        '
Different error message.
' +
      '
' +
    '
'

This is probably a very unusual case, but this will also cause an infinite loop, so we should probably fix it. Maybe merge the patch now and then work on this later?

Taking a more careful look, the problem seems to be that once we traverse all previousSiblings, we go to the parentNode and from there we start at the last childNode (which is essentially a nextSibling to the original comment.
Basically, if we reach at a parentNode after having examined one of its childNodes, we should never go back in, since it is guaranteed that we won't find an actual "previous message" in there.
This is how findPreviousMessage should look like:
function findPreviousMessage(parent, comment) {
  var prevNode = comment;
  var parentLookup = [];
  while (prevNode && prevNode !== parent) {
    var prevKey = prevNode.$$ngMessageNode;
    if (prevKey && prevKey.length) {
      return messages[prevKey];
    }

    // dive deeper into the DOM and examine its children for any ngMessage
    // comments that may be in an element that appears deeper in the list
    if (prevNode.childNodes.length && parentLookup.indexOf(prevNode) === -1) {
      parentLookup.push(prevNode);
      prevNode = prevNode.childNodes[prevNode.childNodes.length - 1];
    } else if (prevNode.previousSibling) {
       prevNode = prevNode.previousSibling;
    } else {
      prevNode = prevNode.parentNode;
      parentLookup.push(prevNode);
    }
  }
}

Taking a more careful look, the problem seems to be that once we traverse all previousSiblings, we go to the parentNode and from there we start at the last childNode (which is essentially a nextSibling to the original comment.

That's what I was trying to explain before. But it's probably easier experienced than explained
And your solution seems to work!

I'Ve updated the PR with your fix, PTAL

I left a couple of minor comments (plus the errors caught by Travis - good job Travis).
LGTM otherwise :+1:

I fixed up the tests, so that they actually fail without the fix, and put them into their own describe. I think that's it. This issue already made my shortlist for "worst issue of 2016"

LGTM
(And I also think this issue was fun :smiley:)


[5.1 04/24][TypeChecker] Improve contextual mismatch diagnostics for key path

Description
Detect situations where key path doesn't have capability required
by the context e.g. read-only vs. writable, or either root or value
types are incorrect e.g.
struct S { let foo: Int }
let _: WritableKeyPath<S, Int> = \.foo

Here context requires a writable key path but foo property is
read-only.
Reviewed By: @DougGregor
(cherry picked from commit 94977ee17573146d6f058b76c7d9c81d43b7aab2)

@swift-ci please test


Add video by Eric Narro

Link to suggestion
https://www.youtube.com/watch?v=kv7rqHgX20w
Targeted section
Talks and videos
Describe your suggestion
Library of the week 13 : Polars with Python ⏳ 15 min - A video that presents Polars with Python by @enarroied. Article supplied with the video in this page.
Checklist

[X] checked if your suggestion is not already listed in Awesome Polars list.
[X] checked if it can fit in an existed section or suggest a new section and explain why your item does not fit into one and what would be the interest of your new section.
[X] checked when it's a repository if it's maintained (archived repository will be rejected).

@enarroied your video will be added to the Awesome Polars List !


MB-6399 Setup benchmarking tool with puppeteer

Description
This PR does the initial setup for benchmarking the document viewer page. Puppeteer v7 is imported since v1 is quite old. The time is calculated using the browser performance api and parsing all the log entries for navigation and resource events. The script works locally and as well as with the S3 bucket though some setup is needed.
Reviewer Notes
Is there anything you would like reviewers to give additional scrutiny?
Setup
Add any steps or code to run in this section to help others prepare to run your code:
Locally
make db_bandwidth_up // to create documents
make server_run
make office_client_run
node puppeteer/benchmarking.js // to run the benchmarking script

S3
Uncomment from .envrc file:
export STORAGE_BACKEND=s3

Terminal
// reload configs
direnv allow

// upload documents to s3
aws-vault exec transcom-gov-dev -- make db_bandwidth_up

// key in your aws OTP
aws-vault exec transcom-gov-dev -- make server_run

// run client
make office_client_run

// run script
node puppeteer/benchmarking.js

Example of result

Code Review Verification Steps

[ ] Request review from a member of a different team.
[ ] Have the Jira acceptance criteria been met for this change?

References

Jira story for this change
this api is used.

Warnings

:warning:

It looks like you are attempting to bypass a linter rule, which is not within
security compliance rules.
** Contains a rule that is not in the permitted eslint list. You are free to disable only: (
no-underscore-dangle
,prefer-object-spread
,object-shorthand
,camelcase
,react/jsx-props-no-spreading
,react/destructuring-assignment
,react/forbid-prop-types
,react/prefer-stateless-function
,react/sort-comp
,import/no-extraneous-dependencies
,import/order
,import/prefer-default-export
,import/no-named-as-default
) **
Please remove the bypass code and address the underlying issue. cc: @transcom/Truss-Pamplemoose

Messages

:book:
:link: MB-6399

  Generated by :no_entry_sign: dangerJS against f3149de49bd432d4cec8b6f21ef48a1078668b1b

Have we considered using Cypress to take these benchmarks instead of Puppeteer? I just ask because we already have Cypress in the project and I believe it can be used to achieve everything here. Their docs have an example of using Cypress to take performance metrics: https://github.com/cypress-io/cypress-example-recipes/blob/master/examples/testing-dom__performance-metrics/cypress/integration/spec.js

I believe so but we came across this article by Cypress saying it isn't meant to be used for performance testing. One of the key reasons why we decided on Puppeteer is that we are able to throttle the network connection. We also noticed that Puppeteer v1 was already included in the project but decided to pull in v7 since it got a bit frustrating to look at the api.

We also have the cypress-audit plugin set up which we are only using for a11y testing right now, but it can also be used to generate Lighthouse performance reports: https://github.com/mfrachet/cypress-audit/blob/master/docs/lighthouse.md

Yea, I believe will end up using lighthouse later on for a more detailed report page performance.

I don't know if these satisfy your use case but I wanted to make sure we were considering the tools we already have before adding a new one.

Yes, feel free to checkout our doc on gathered tools. @duncan-truss can probably add some more comments.

I'm getting an error. Looks like it can't find the user ID?
node puppeteer/benchmarking.js
(node:43994) UnhandledPromiseRejectionWarning: TimeoutError: waiting for selector `button[value="9bda91d2-7a0c-4de1-ae02-b8cf8b4b858b"]` failed: timeout 30000ms exceeded
    at new WaitTask (/Users/moncef/projects/mymove/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:509:34)
    at DOMWorld.waitForSelectorInPage (/Users/moncef/projects/mymove/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:420:26)
    at Object.internalHandler.waitFor (/Users/moncef/projects/mymove/node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:31:77)
    at DOMWorld.waitForSelector (/Users/moncef/projects/mymove/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:313:29)
    at Frame.waitForSelector (/Users/moncef/projects/mymove/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:841:51)
    at Page.waitForSelector (/Users/moncef/projects/mymove/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1280:33)
    at /Users/moncef/projects/mymove/puppeteer/benchmarking.js:41:14
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:43994) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:43994) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Does this work only with certain browsers? Can we specify which browser to use? I launched the office app in Safari.

Oh, it's because the user was not created due to a known bug with make db_bandwidth_up that I just fixed. Could you please rebase with master?

Oh, it's because the user was not created due to a known bug with make db_bandwidth_up that I just fixed. Could you please rebase with master?

@monfresh just pushed up merge from master

Does this work only with certain browsers? Can we specify which browser to use? I launched the office app in Safari.

By default, puppeteer uses chromium. I think the other supported browser is Firefox.

Was the goal for this PR to only have a quick initial script in place? If so, do we have follow up stories to test the individual files, for both the TOO and TIO? Currently, from what I can tell, this only loads the first image in the doc viewer. We also want to test the performance of the other 2 files.

Was the goal for this PR to only have a quick initial script in place? If so, do we have follow up stories to test the individual files, for both the TOO and TIO? Currently, from what I can tell, this only loads the first image in the doc viewer. We also want to test the performance of the other 2 files.

Yup, we have other stories to test all the files. We're just making sure the initial setup is good. @duncan-truss already got a huge head start on that in his PR.


AD Groups and User's names showing up as Hex instead of text

I'm having 2 issues with Active directory authentication:

I get an "invalid credentials" error when trying to use the LDAP Security Groups, but can login without issue when that is set to false.

After configuring AD authentication, when users login, their first and last names show up as Hex strings instead of the proper text.

Further info on issue 1, the error I'm receiving application-side is:
[ERROR] LDAP group lookup for user "local variable 'ldap_user_dn' referenced before assignment" has failed

I got issue 2 figured out, ended up having to add a .decode('utf-8') statement on the first and last names and email address.
Still working on issue 1, it looks like the models.py is only setup to do group security for ldap, and not AD.


Add documentation in the wiki for what we currently have

In order to people start using all the different extensions that are part of the WebAPIContrib, they need to be able to know that the extension exist, what it does and how to use it.
We can start by documenting in the Wiki what we currently have: formatters. handlers, filters, IoC helpers etc.

I think issue #120 will take care of this.


Xcode 13 Build Issues

Hi @willm132, which version of OrderedDictionary are you using? This issue should be fixed as of 4.0.1 AFAIK.


Default markdown template missing

It points to http://scripting.com/code/pagepark/defaultmarkdowntemplate.txt which is currently returning a 404.

OK -- I'm going to fix the default in the distribution, but in the meantime, you can fix it locally by adding this line to the top-level config.json.
 "urlDefaultMarkdownTemplate": "http://scripting.com/code/pagepark/templates/markdown/template.txt",

Also I made the change on my primary PP server and it;'s fixed there.
http://lucky.wtf/test.md
Thanks for uncovering this, as you can tell I don't do a lot of Markdown. ;-)

Looks good!


Occidental Technologies missing from merchant list.

This company was recently approved and merged to the main monero-site.
Here is the merge pull request.
However, upon searching the listed merchants with a ctrl + F (Occidental Technologies), the name is not found on the listing under the services section.

I'm not sure why the listing is not showing up, can someone give me some info as to why?

The server hosting getmonero still needs to be updated with the newest code. Please don't open a new issue just to make a question when there is already a dedicated issue and a dedicated pull request. You could have commented under any of the two.


PIP-271: Add broker health check status into prometheus metrics #20147

Motivation:
Broker metrics don't have anything to indicate health of the broker (to indicate if broker is active). In Prometheus broker metrics which are used for monitoring, it will be useful if metrics also show the broker health. This way, Prometheus can automatically scrape the broker state and can be used for monitoring purposes. So we need such metric to capture broker health.
Goals:
This PIP adds support to include the broker health status in the broker operability metrics.
Sample:
When we hit "/metrics" endpoint, a part of the output looks like below. Notice the "pulsar_health" metric which is added as a result of this PIP. The status "1" says that broker is active, "0" is inactive, "-1" unknown.
Output

# TYPE pulsar_active_connections gauge
pulsar_active_connections{cluster="standalone", broker="localhost", metric="broker_connection"} 2
# TYPE pulsar_connection_closed_total_count gauge
pulsar_connection_closed_total_count{cluster="standalone", broker="localhost", metric="broker_connection"} 0
# TYPE pulsar_connection_create_fail_count gauge
pulsar_connection_create_fail_count{cluster="standalone", broker="localhost", metric="broker_connection"} 0
# TYPE pulsar_connection_create_success_count gauge
pulsar_connection_create_success_count{cluster="standalone", broker="localhost", metric="broker_connection"} 2
# TYPE pulsar_connection_created_total_count gauge
pulsar_connection_created_total_count{cluster="standalone", broker="localhost", metric="broker_connection"} 2
# TYPE pulsar_health gauge
pulsar_health{cluster="standalone", broker="localhost", metric="broker_connection"} 1

Approach:
A new metric called "brk_health" is added into the BrokerOperabilityMetrics. This metric is updated at a fixed rate from the BrokerService.
BrokerOperabilityMetrics.java

 Metrics getConnectionMetrics() {
        Metrics rMetrics = Metrics.create(getDimensionMap("broker_connection"));
        rMetrics.put("brk_connection_created_total_count", connectionTotalCreatedCount.longValue());
        rMetrics.put("brk_connection_create_success_count", connectionCreateSuccessCount.longValue());
        rMetrics.put("brk_connection_create_fail_count", connectionCreateFailCount.longValue());
        rMetrics.put("brk_connection_closed_total_count", connectionTotalClosedCount.longValue());
        rMetrics.put("brk_active_connections", connectionActive.longValue());
        rMetrics.put("brk_health", healthCheckStatus);
        return rMetrics;
    }

We schedule a periodic health check job at a fixed rate in the BrokerService. This job updates the broker health check metric in the BrokerOperabilityMetrics stats at the frequency configured in the broker configs.
BrokerService.java

protected void initializeHealthChecker() {
        ServiceConfiguration config = pulsar().getConfiguration();
        if (config.getHealthCheckMetricsUpdateTimeInSeconds() > 0) {
            int interval = config.getHealthCheckMetricsUpdateTimeInSeconds();
            statsUpdater.scheduleAtFixedRate(this::checkHealth,
                    interval, interval, TimeUnit.SECONDS);
        }
    }

    public CompletableFuture<Void> checkHealth() {
        return internalRunHealthCheck(TopicVersion.V2, pulsar(), null).thenAccept(__ -> {
            this.pulsarStats.getBrokerOperabilityMetrics().recordHealthCheckStatusSuccess();
        }).exceptionally(ex -> {
            this.pulsarStats.getBrokerOperabilityMetrics().recordHealthCheckStatusFail();
            return null;
        });
    }

No new API is needed as we already have a "healthCheck" API in Admin module which provides the necessary functionality. However we don't make a REST call to this API as it could be costly. Instead we add a helper function "internalRunHealthCheck" in the Admin module which piggy backs on the existing functionality in the Admin module.
Configuration Changes:
This PIP gives option to dynamically switch on/off the broker health check metric using "healthCheckMetricsUpdateTimeInSeconds" config. Setting it to -1 will disable the metric. We can also configure the frequency of the metric update using this config. By default it is set to value "-1" which effectively disables it.
broker.conf

healthCheckMetricsUpdateTimeInSeconds=-1

Link to vote: https://lists.apache.org/thread/c9y3b4z6jotdcbwt5qcvht4ngcpzrzzo

Few comments before you continue with implementation:

Please refactor out the core logic in BrokerBase.internalRunHealthCheck into it's own HealthCheckService.
Please refrain from adding more code to BrokerService - this class is 3600 lines - what good reason do you have adding more lines to this class? HealthCheckService sounds likes a reasonable place to host the scheduling part.
Please don't user scheduleAtFixedRate since you can overload the internal queue of the executor service. You can use scheduleAtFixedDelay.
You intend to place rMetrics.put("brk_health", healthCheckStatus); in getConnectionMetrics() but as far as I know, this metric as nothing to do with metrics of connections. Can you please find / create a more suitable place?

healthcheck attribute is another attribute which gets refreshed by regular task same as other tasks like backlog-check, stats-check, inactive-topic, etc. also, this PIP doesn't introduce any new service or implementation rather it only calls existing admin-api on regular interval to refresh the attribute. so, it follows the existing task pattern and don't think it requires  a new service class here.
I agree with point #3 and #4 though.

VOTE has passed and this PIP has been approved


Revert use of smart pointers

e.g. commits
d0c6ff80892433c3c37eeefe2540ec2716387541
3483059b9024abd59962c7678a6e05430b09a9a4
0ccd51a972d290be31a9642a5ef3b01a5c8bc37e
change use of plain-old pointers, to make use of C++11 smart pointers.
While I'm not sure Delaunay algorithm is the quickest algorithm to begin with anyway,
the changes were made without considerations to performance (e.g. speed).
And I'd guess it's slower to use them.

I really don't know if the smart pointers make things slow.
I suspect it's the precise arithmetic. -- But it's better to use the smart pointers than to have sloppy new/delete discipline.


chore: custom email notification payload update

PR summary
notification payload changes for custom email- bypass subscription
Fixes: <! -- https://github.ibm.com/Notification-Hub/planning/issues/10427 -->
PR Checklist
Please make sure that your PR fulfills the following requirements:

[x] The commit message follows the Go Commit Message Guidelines.
[x] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)

PR Type

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] New tests
[ ] Build/CI related changes
[ ] Documentation content changes
[ ] Other (please describe)

What is the current behavior?
bypass subscription for custom email is not present
What is the new behavior?
notification payload changes for custom email- bypass subscription
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

:tada: This PR is included in version 0.2.8 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Tut React Native admonitions. #12495

Summary

Test plan

Perfect. Thanks for helping. (As I wrote already, just ignore the idea on all versions. Let's keep it simple.)
@SimenB This is ready to be merged. Please take a look.

All versions add @mrazauskas

👍


Upgrade 5.1 workflows to 5.1.1

In the interest of testing the readiness of the upcoming 5.1.1 release, I've
switched the GitHub action workflows to target 5.1.1~rc1 https://github.com/ocaml/opam-repository/pull/24874
It would be nice to move the OCurrent runners in multicoretests-ci to target those too.

OCaml 5.1.1 has now been released so it would be natural to move workflows to use that over rc1: https://github.com/ocaml/opam-repository/pull/24917

Closing as all 5.1 workflows now use 5.1.1


⚠ Hartman Foundation has degraded performance

In 8621e86, Hartman Foundation (https://gordonhartman.com/) experienced degraded performance:

HTTP code: 200
Response time: 2568 ms

Resolved: Hartman Foundation performance has improved in c52a9b2.


[WIP] Add Azure AD Integration with Quatrix tutorial

Hi @ktoliver! Thanks, I've just updated the PR and signed the CLA. Please proceed with the further steps.

Hi @v-hagamp, thanks for the comments above. I've added the Next steps section, I'm not sure how to add image borders,  though.

@mfuterko I don't see this application was deployed in the Azure AD gallery, if the application is not existing in the gallery, then weren't create the document for it directly.
Will you please add Quatrix into the Azure AD gallery first?

@mfuterko,
Microsoft Application Network Portal https://aka.ms/AzureADListYourApp : Use this invitation and make sure that you open the invitation in the new browser InPrivate window. You can then submit the request in our portal for listing the application in the gallery. (Note: Save and close option will create DRAFT of the request.) Please refer to FAQ’s on our site and attached document for guidance.
Publicly documented process to submit application request: aka.ms/AzureADAppRequest
Microsoft Application Network Portal for ISVs to track apps: aka.ms/AzureADListYourApp
Azure AD App integration.pdf
Thanks,
Hamsika

#please-close


Paginate Infinite Scroll: Load Items Initially

I'm following the Firebase-util Paginate Infinite Scroll demo, and was wondering if there is a way to load an initial number of items, instead of loading 0 items to begin with.
When I click on my button, I'm loading in the next 16 items. Is there a way I can load 16 items on the initial load, and then another 16 upon clicking my button, etc etc?
My HTML and JS looks like this:
index.html

Items loaded: {{scrollItems.length}}

<button ng-click="scrollItems.scroll.next(16)">Load Next 16</button>

app.js
app.controller('ServiceListController', ['$scope', '$firebaseArray', '$scrollArray',
  function($scope, $firebaseArray, $scrollArray) {
    var servicesRef = new Firebase('https://fbutil.firebaseio.com/paginate');
    $scope.scrollItems = $scrollArray(servicesRef, 'number');
}])

app.factory('$scrollArray', ['$firebaseArray', function($firebaseArray) {
  return function(ref, field) {
    var scrollRef = new Firebase.util.Scroll(ref, field);
    var list = $firebaseArray(scrollRef);
    list.scroll = scrollRef.scroll;

    return list;
  }
}])

Here's a demo of this in action.
Any help with this is appreciated. Thanks in advance!

Add next to the scroll ref:  javascriptlist.scroll = scrollRef.scroll.next(10);

Yep. Just call next().

@michaeldim When I add next to the initial scroll ref, it prevents me from clicking the button and loading more items.

@realph I say all of this with the upmost love and sincerity; I've been in your shoes many times myself and have the greatest respect for the frustration of navigating through new technologies--particularly when they aren't working as expected.
You need to be more detailed, more persistent and more thorough here. Please respect the community's time and recognize that debugging the problem is your purview.  Once you've reduced it to a known error, if you are having trouble finding the solution, we would love to help.
If you're looking for techniques to debug the issue, an extremely effective one is to produce an mcve. If that doesn't resolve the problem en tránsito, post the mcve with some steps to reproduce. At that point I'm happy to help you come up with solutions.

Thank you for the advice @katowulf. I'll try and explain issues better in the future and follow such best practices as to not drive the community insane.
On this particular issue, would you recommend trying to do a bit of work on my end and explain it better along with an mcve in this thread, or open a new thread completely?
Thanks once again. While it is frustrating for me, I can imagine it also being frustrating for you – so apologies for the constant hammering of posts, especially when I'm not being detailed with my explanations.

@realph How about this: http://plnkr.co:80/v7M0vF5ziTK9MLXxsDp3


Support unsigned encrypted emails

Please search to check for an existing issue.

I have, and found #1999 and #1743, but neither seem to exactly address what I want, I'm not sure whether #2087 addresses this particular issue, but it doesn't seem to be the case.
Expected behavior
I want the ability to separately sign and encrypt emails. I like being able to encrypt by default (kuddos you guys for finally supporting opportunistic encryption with OpenKeyChain and PGP/MIME!), but I would like to disable signing by default, even if encrypted.
Being able to encrypt without signing supports confidentiality with plausible deniability, which I think is an important mode of communication.
Actual behavior
At the moment, it seems that one can

Send unencrypted and unsigned emails
Send unencrypted and signed emails
Send encrypted and signed emails

The option to send encrypted but unsigned emails is missing.
Steps to reproduce

Enable auto encryptions for known keys
Send an email for a known key
No option to not sign the message (except by disabling crypto entirely)
Any encrypted message is sent with a signature.

Environment
K-9 Mail version: 5.203
Android version: not relevant
Account type (IMAP, POP3, WebDAV/Exchange): not relevant

See #1961


Support remote auto-configs based on user's geolocation (IP)

This can make CensorTracker work for different countries with censorship.

Fixed in #303


Blazor server side: dotnet run fails with InvalidOperationException on MacOS

Describe the bug
dotnet run fails with the follwing exception on MacOS for blazorserverside projects
System.InvalidOperationException: Could not load the embedded file manifest 'Microsoft.Extensions.FileProviders.Embedded.Manifest.xml' for assembly 'Microsoft.AspNetCore.Components.Server'.
   at Microsoft.Extensions.FileProviders.Embedded.Manifest.ManifestParser.Parse(Assembly assembly, String name)
   at Microsoft.Extensions.FileProviders.ManifestEmbeddedFileProvider..ctor(Assembly assembly)
   at Microsoft.AspNetCore.Components.Server.ConfigureStaticFilesOptions.PostConfigure(String name, StaticFileOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsManager`1.<>c__DisplayClass5_0.<Get>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
   at Microsoft.Extensions.Options.OptionsManager`1.Get(String name)
   at Microsoft.Extensions.Options.OptionsManager`1.get_Value()
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware..ctor(RequestDelegate next, IWebHostEnvironment hostingEnv, IOptions`1 options, ILoggerFactory loggerFactory)
--- End of stack trace from previous location where exception was thrown ---

To Reproduce
Steps to reproduce the behavior:

Get the latest preview4 sdk from core-sdk
dotnet new blazorserverside
dotnet run

Expected behavior
The app runs successfully
Additional context
Note: This doesn't happen for client-side blazor projects.

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview4-011216
 Commit:    19c844a865

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.14
 OS Platform: Darwin
 RID:         osx.10.14-x64
 Base Path:   /usr/local/share/dotnet/sdk/3.0.100-preview4-011216/

Host (useful for support):
  Version: 3.0.0-preview4-27613-07
  Commit:  5c329c85ae

cc @danroth27 @rynowak

Reassigning it to @pranavkm as he already has a PR out to fix this.

Changes are in. Leaving open for verification.

Same problem in linux
SDK de .NET Core (reflejando cualquier global.json):
 Version:   3.0.100-preview4-011223
 Commit:    118dd862c8

Entorno de tiempo de ejecución:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /snap/dotnet-sdk/36/sdk/3.0.100-preview4-011223/

Host (useful for support):
  Version: 3.0.0-preview4-27615-11
  Commit:  ee54d4cbd2

.NET Core SDKs installed:
  3.0.100-preview4-011223 [/snap/dotnet-sdk/36/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview4-19216-03 [/snap/dotnet-sdk/36/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0-preview4-27615-11 [/snap/dotnet-sdk/36/shared/Microsoft.NETCore.App]

Is there any workaround?

@kattunga Yes, unfortunately this is a known issue with the Preview 4 release. It will be fixed in the next preview in a couple of weeks. There isn't a known workaround at this time, but if we come up with one we will update the known issues doc accordingly. The client-side Blazor templates should work fine though.

While it does not give the exception on startup in the latest nightly preview it still does not work. The exception now occurs on the "blazorserverside" template project as soon as one clicks on the counter:
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
An unhandled exception has occurred while executing the request.
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Extensions.FileProviders.Embedded.Manifest.ManifestFileInfo.EnsureLength()
at Microsoft.Extensions.FileProviders.Embedded.Manifest.ManifestFileInfo.get_Length()
at Microsoft.AspNetCore.StaticFiles.StaticFileContext.LookupFileInfo()
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

@tn-5, we are aware of this issue. A fix for this has already been merged https://github.com/aspnet/AspNetCore/pull/9527. It will be available on an upcoming build.

thx

@ajaybhargavb "upcoming build" ... does that mean Pre5, or sooner than that?

@guarded it will be released with preview 5, but you can try out a nightly build.
Daniel Roth

I also got this when trying to run Blazor Server Side (preview4) on an Azure Kubernetes service

I'm getting the same error even on client side on Mac.
https://pastebin.com/C6YJhsZM

@simonziegler You can get builds of Preview5 from https://github.com/aspnet/AspNetCore/blob/master/docs/DailyBuilds.md


SDA-4329 | feat: Validation for user creating account-roles

$ ginkgo --focus 43067 tests/e2e
Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages:
Ginkgo CLI Version:
2.11.0
Mismatched package versions found:
2.17.1 used by e2e
Ginkgo will continue to attempt to run but you may see errors (including flag
parsing errors) and should either update your go.mod or your version of the
Ginkgo CLI to match.
To install the matching version of the CLI run
go install github.com/onsi/ginkgo/v2/ginkgo
from a path that contains a go.mod file.  Alternatively you can use
go run github.com/onsi/ginkgo/v2/ginkgo
from a path that contains a go.mod file to invoke the matching version of the
Ginkgo CLI.
If you are attempting to test multiple packages that each have a different
version of the Ginkgo library with a single Ginkgo CLI that is currently
unsupported.
Running Suite: e2e tests suite - /home/akanni/OCP-Repository/rosa/tests/e2e
Random Seed:<PHONE_NUMBER>
Will run 1 of 61 specs
SSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 61 Specs in 177.783 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 60 Skipped
PASS
Ginkgo ran 1 suite in 2m58.778614988s
Test Suite Passed

/lgmt

/lgmt

/lgtm


Small fixes for widgets

Two small fixes for widgets to process 'Complete' acknowledgements in python3 and to handle DynamicMaps with zero-kdims.

Is this a different message from 'Ready'? Or are we still planning to change the 'Complete' message to 'Ready'?

We'll switch from Ready to Complete in the widgets once we implement the widget manager.

Ok then. Merging.


The server install wipes out user directory contents like dependencies

Say that I were doing something:
mvn  dependency:copy-dependencies@my-exec  liberty:dev
in order to copy my dependencies into place somewhere in target/liberty/wlp/usr (this use case is discussed more in #705).
In this case the 'install-server' performed by dev mode would delete and wipe out the contents of my 'usr' dir, because I had not previously already installed Liberty.
A more experienced user could workaround this by doing:
mvn  liberty:create dependency:copy-dependencies@my-exec  liberty:dev
so that by the time we get to liberty:dev, we have already done an install and have no need to do a new one.
But this is a bit redundant since liberty:dev otherwise basically includes liberty:create (plus everything else it includes).
Maybe the target/liberty/wlp/ contents should be merged or at least the target/liberty/wlp/usr contents preserved?
Is there any case where a user could depend on the existing behavior?   Where a clean would be desired?     I'm thinking in terms of usr just having dependencies but it could have full server dirs with logs, workarea, etc.

@chyt  any outlook for this one?   Thanks.

I think we've discussed this a bit off Github. I'm fine with doing a copy-merge, or having liberty:create retain the contents usr folder (the user can clear that as well with mvn clean).


[Feature] Add translated nested object return in the template

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report
[ ] Performance issue
[X] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior
We cannot return a nested object translated based on a key in the template directly
Expected behavior
We can receive a translated object by passing a key (and your parameters) on our template.
What is the motivation / use case for changing the behavior?
If we are implementing a smart/dumb pattern, sometimes we need to pass a set of labels/placeholders to our dumb component. To have a clean dumb component (and follow the design pattern) we need to pass a value/object of values to our child
Imagine the following scenario:
signin.component.html (smart component) Today
....
  <app-signin-form class="signin-form"
    [value]="valueSigninForm$ | async"
    [placeholders]="getTranslateProperty(t('signinForm.placeholders'))"
    [labels]="getTranslateProperty(t('signinForm.labels'))"
    [errors]="getTranslateProperty(t('signinForm.errors'))"
    (changed)="onChangedSigninForm($event)">
  </app-signin-form>
....

signin.component.ts (smart component) Today
....
  getTranslateProperty(key: string) {
    return this.translocoService.translateObject(key, null, this.translocoService.getActiveLang());
  }
....

signin-form.component.ts (dumb component) Today
....
  @Input() value: SigninForm;
  @Input() placeholders: SigninFormPlaceholders;
  @Input() labels: SigninFormLabels;
  @Input() errors: SigninFormFieldErrors;

  @Output() changed: EventEmitter<SigninForm>;
....

signin-form.model.ts Today
import { FormFieldErrorMessage } from<EMAIL_ADDRESS>
export type SigninFormFields = 'email' | 'password';
export type SigninForm = Partial<Record<SigninFormFields, string>>;
export type SigninFormPlaceholders = Partial<Record<SigninFormFields, string>>;
export type SigninFormControls = 'submit';
export type SigninFormLabels = Record<(SigninFormFields & SigninFormControls), string>;
export type SigninFormFieldErrors = Partial<Record<SigninFormFields, FormFieldErrorMessage>>;

The desired implementation is that in **. Ts ** of the smart component, we do not need the getTranslateProperty method, replacing it with a direct reference in the template. So, we can modify the actual implementation with
signin.component.html (smart component) Desired
....
  <app-signin-form class="signin-form"
    [value]="valueSigninForm$ | async"
    [placeholders]="t('signinForm.placeholders', true)"
    [labels]="t('signinForm.labels', true)"
    [errors]="t('signinForm.errors', true)"
    (changed)="onChangedSigninForm($event)">
  </app-signin-form>
....

signin.component.ts (smart component) Today
....
// We remove this method because is not necessary anymore.
//  getTranslateProperty(key: string) {
//    return this.translocoService.translateObject(key, null, this.translocoService.getActiveLang());
//  }
....

@itayod ok, thank you for your response 👍
Have a good week o/


Download or export session transcript information by date

Good morning!
Is there any way to download or export session transcript information by date?
Every morning, I download it from the PVA portal, and I need to download it by clicking many lists, and if we have a lot of sessions, it would be excruciating.
It would be good if I could download it by date, but I can't find a way in your document.
Kind regards,
Mina

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fded0a67-519d-8aca-c6d0-c44038c14b22
Version Independent ID: b9c0832f-3cdb-8029-7171-66df4bb8f517
Content: Download chatbot sessions - Power Virtual Agents
Content Source: power-virtual-agents/analytics-sessions.md
Service: power-virtual-agents
GitHub Login: @iaanw
Microsoft Alias: iawilt

Thanks for your comment, @minalove. This issue has been assigned to the doc author for review.

@minalove Thanks for your message! The best place for product requests is the Power Virtual Agents forum.


Adding CI

Adding travis based CI to ensure the tex files compile properly!

After hours of bugfixes, the CI finally works!


Error Copilot Setup

Error when try :Copilot setup
Error detected while processing function copilot#Command[19]..5[13]..<SNR>41_JsonBody:
line    3:
E474: Attempt to decode a blank string
E474: Failed to parse

cannot confirm. just tested it with the newest version and it worked flawlessly

cannot confirm. just tested it with the newest version and it worked flawlessly

I'm using it on Kali KDE, and everything goes fine, just the copilot has that error and I can't open Git Status menu.
Maybe I don't know how to use :Copilot log, but getting no response when I run it.
I'll try update my system packages and do a fresh install of this repository.
Thanks for your reply!

Nothing changed, same error...

sorry i was sleeping and didn't hat the time to answer sooner, I was about to start my kali VM to test it.. I'm happy that you figured out yourself and giving us a follow up how you ended up fixing it.


Update Greetings

👨🏾💻 In this pull request, I have:

Created a rails API called hello-rails-back-end
Setup the database with Postgres
Created an API endpoint that pulls data for 5 different greetings
Link to hello-react-front-end pull request

Link to frontend pull request

Hi @enaburekhan do you still have the same issue when running the app with the ruby version from the gem file? (2.7.2) I'm asking because I'm not able to reproduce the error on my, machine. The server runs correctly on my end

@SunnySpark, I had to upgrade the ruby version to 3.0.0. Because I don't have 2,.7.2 installed on my machine. Maybe that could be the problem. I'm sorry about this. Kindly resubmit so it can be approved.


Definition of done and ready: check consistency with CSE recent guidelines

https://github.com/microsoft/code-with-engineering-playbook/tree/master/team-agreements/definition-of-done
https://github.com/microsoft/code-with-engineering-playbook/tree/master/team-agreements/definition-of-ready

@meulta Can you point to the recent guidelines?

I think that just making sure it is in sync with https://cwcwiki.com/wiki/Definition_of_Done (internal link) would be great.

@meulta should we close this issue - or how do you feel about the above?

Yes we can close this!

Yes we can close this!


Missing editor functions when creating Markdown file on all versions after 1.11.8

Gitea version (or commit ref):
Git version: git version 2.17.1
Operating system: Ubuntu 18.04
Database (use [x]):

[ ] PostgreSQL
[ ] MySQL
[ ] MSSQL
[ x] SQLite

Can you reproduce the bug at https://try.gitea.io:

[x ] Yes (provide example URL)
[ ] No
[ ] Not relevant

Log gist:

Description
When creating a  markdown file in version 1.12.1, the editor functions are missing. This affects all versions after 1.11.8
Screenshots
Its working in version 1.11.8

Missing in versions after 1.11.8

Closing as dupe of #12064


Integrate URI Resolver Extension Wrappers + Remove All Plugins

After more digging, I think the problem is much simpler, and can be solved by this existing PR: https://github.com/polywrap/toolchain/pull/1528

also, how did you fix the wasm memory errors when resolving with many wrappers?

wouldn't it be better if the wasm files are generated when the client is built? that way we don't push it (and need to re-build it if modified)? i am ok with both ways tho but just wondering if you thought about this

@cbrzn yes definitely, I think this should be something that is done through the CLI. For now it's just done within this custom script, but this should be easier to do with a single command.

also, how did you fix the wasm memory errors when resolving with many wrappers?

@cbrzn upgrading to node > 17 helped a lot. In previous versions of node, WebAssembly.Memory instances aren't freed from memory very aggressively, leading to the heap blowing up over time. In order to help with memory related problems in the browser, we have plans to add a limits section to the client config, so that we can limit (for example) the number of parallel invocations of a given wrapper, effectively load-balancing invocations.


Step 3 failing on WSL with error "gpg: keyserver receive failed: No dirmngr"

It would be great if you just add this link: https://github.com/Microsoft/WSL/issues/3286#issuecomment-402594992 or copy comment text to https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest#cli-fails-to-install-or-run-on-windows-subsystem-for-linux
Because it solves issue and there is no reason to force people search it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fe1526a9-7f22-504e-484c-0f3edc11b178
Version Independent ID: c5dec108-efcd-aa77-04b5-803b6b894dba
Content: Install the Azure CLI on Linux with apt
Content Source: docs-ref-conceptual/install-azure-cli-apt.md
Product: azure
GitHub Login: @sptramer
Microsoft Alias: sttramer

@akkensheen - As the linked issue states, this was "Fixed in 1809". The correct solution here is to update to a more recent version of Windows (and WSL), as described in the troubleshooting section.


Allow ordered spatial predicates in the distributed tree

This PR introduces ordered traversal in the distributed settings.
Motivation
Thermal radiation using raytracing. Instead of spitting out all the intersections, we process them on the fly accumulating information within a rank.
Semantics
I would like to have a discussion on what it actually means. In a distributed setting we can only guarantee the ordered traversal within a rank, but not among the ranks as the intersected ranks will be processed in parallel.
In addition, imagine a situation where the rank bounding boxes overlap, and the actual rank subdomain are non-convex. We can't truly perform a global ordered traversal in that case.
The only case where we can guarantee ordered results would be the case where subdomain boxes do not overlap. But I think this is already good.

I have not put a test yet as we need to agree on the semantics first.

I would like to have a discussion on what it actually means. In a distributed setting we can only guarantee the ordered traversal within a rank, but not among the ranks as the intersected ranks will be processed in parallel.

We should do a reduce on the minimum intersection distance once the search is done.

We should do a reduce on the minimum intersection distance once the search is done.

I'm not sure I understand. Can you please elaborate?

If I understood correctly what you are saying we have multiple processors that will report a first hit of the ray.  To get a single hit, we need to find the hit with the smallest distance to the origin of the ray along the ray direction and filter out the other ones. So we should a reduce min on the results and keep the one that matches the min.

@Rombur I'm not sure why you are talking about the first hit. If we were talking only about the one nearest thing, that would be one thing. But we are talking about processing all the intersections, but in order. However, it would be impossible to do in the distributed setting. Imagine non-convex subdomains, or not simply connected, in 1D. Lets say | 0 | 1 | 0 | 1 | 0 | 1 | are MPI ranks of subdomains that a ray intersects. What does it mean to have ordered traversal in this case?


Added support for repo URLs prefixed with ssh://

From the Git Book:

To clone a Git repository over SSH, you can specify ssh:// URL like this:
$ git clone<EMAIL_ADDRESS>Or you can use the shorter scp-like syntax for the SSH protocol:
$ git clone user@server:project.git

Instead of depending on the url to start with "git@", the checkout type and git service are now determined using NSURL's resourceSpecifier and scheme properties without manipulating the original URL string.
I just noticed that there are small changes to the parse function in #251, but the merge shouldn't be tricky.

Result of Integration 1
Duration: 5 minutes and 39 seconds
Result: All 50 tests passed, but please fix 1 warning.

Very cool, thanks for fixing it properly and even adding tests! 💯 👍


Errors while building users-api within docker

Hi,
I am trying to run your project within Docker on Win10.
To get through I had to update todos-api\Dockerfile:line 8 - added ".js" in the path.
Then I bumped into an issue with users-api:

ERROR: Service 'users-api' failed to build: COPY failed: stat /mnt/sda1/var/lib/docker/tmp/docker-builder481249926/target/users-api-0.0.1-SNAPSHOT.jar: no such file or directory

I guess the build of the jar file is missing. There is info about building in microservice-app-example\users-api\README.md, but I have not been able to get it to work.
Could you please apply the necessary corrections?

@emwiechnik the dockerfile is expecting that you already compiled the target into a jar. You'll need to do that first

I have created a PR to fix this and do the build inside docker. Also, I have fixed some other issues like glide up to install go dependencies and fixed the missing .js extension
Fixing dependencies

#3 is merged as well as a couple of other fixes. Everything is getting compiled when docker images are being built.


Making rule engine execution order more deterministic

Clara currently uses clara.rules.platform/tuned-group-by in a number of places.  The grouping that this function does is unordered and often can vary from one JVM to the next.  In particular I've noticed this happening in the clara.rules.engine/flush-updates which was causing non-deterministic insertion order to happen across rules.  This makes it harder for us to track bad performance cases since they keep fluctuating as we run in different processes.
A side-effect of flush-updates not behaving deterministically was that insertion order across rules of the same salience group level were not deterministic and didn't necessarily respect rule load order as was implemented in https://github.com/rbrush/clara-rules/issues/192.  From my preliminary testing of this, I see a slight improvement in our execution times with this change in place.  This is likely due to the rule order now also being the insertion order used.  I've demonstrated this with a test that fails before this change and now passes.
My proposal is to (on the Java CLJ side) introduce a new clara.rules.platform/group-by-seq function that replaces the usages of platform/tuned-group-by.  Currently, I've left platform/tuned-group-by in case it ends up having some usages later.  It is also used in one CLJS place still, which I'm not wanting to change with this.  platform/group-by-seq returns a seq of tuples, which is the same as seq'ing over the map returned by platform/tuned-group-by before.  In all cases where we used platform/tuned-group-by we didn't use the map, but rather a seq over it.
I use a java.util.LinkedHashMap to maintain insertion order while building the groupings.  Then I convert this into an immutable seq structure.  If I tried to retain the usage of a map here, I'd have to return the java.util.LinkedHashMap or come up with some (expensive) way to port it into a Clojure sorted map.  This was not worthwhile to me, since we do not need a map anyways.  A java.util.LinkedHashMap would be harder to work with across the rest of the codebase since there are issues with the default Clojure seq implementations across mutable object iterators http://dev.clojure.org/jira/browse/CLJ-1738 that I see coming up when running this on JDK6 (oddly not on JDK8, which makes no sense and I want to investigate more for my own benefit).
Along with this, since I was messing with the clara.rules.compiler/create-get-alphas-fn already, I sorted the result of the ancestors-fn so that it would behave deterministically.  This would be more of an edge case, but since we cache this result and infrequently should hit this path, I figured it'd still be worthwhile.

Looks good to me after an early review. I'll merge later today unless there are unexpected problems.

I'll plan to look at this today.

+1 apart from my comments.

I've addressed all of the review comments made by @WilliamParker

+1

Merged. Thanks!


Can ComboLineColumnChartView have default column to show?

Here is the thing.
I have too much data (like 20 columna) , but I want to show 8 for one time and it can smooth to see the other columns.
Is there any method to do that?
Any help is appreciated.

I use the following code and get some similar effect but not the exact one.
private void setZoomXAxis() {
Viewport maxViewport = chart.getMaximumViewport();
//        由于我们的X轴用的1到n的整数,viewport 从(0,最大值)到(n,最小值)
//我们用n-7显示最新的7个
if(maxViewport.right<=7){
chart.setMaxZoom(1);//不需要缩放
}else {
float zoom = 10;
chart.setMaxZoom(zoom);
}
Viewport tempViewport = new Viewport(maxViewport.right>7?maxViewport.right-6:maxViewport.right, maxViewport.top, maxViewport.right, maxViewport.top>7?maxViewport.top-6:maxViewport.top);
chart.setCurrentViewport(tempViewport);
}
How can I get the zoom or maxViewport.bottom based on the data of Y Axis?

The whole chart is this. The current display will not show the 88% part which may bring some confusion to the user.

Viewport v = new Viewport(chart.getMaximumViewport());
v.left = 0;
v.right = 3;
chart.setCurrentViewport(v);


Added DateTime.

Added DateTime support. Saves a DateTime in the most efficient format, saving it to 7 characters versus 21 characters as a terminated string.

That's how I had it initially I think, I can do that though.


MessageSending subscribe fix and adding SubscriptionChangeEvent to AMQP 0.9.1

Changes Unknown when pulling f2c507aab27a2c8663d5ad816a8747f1c31bae25 on cd_subscription into ** on master**.


Evaluate use of Property parameter and Alias

-Property is an alias for -Entity in several commands to ease the transition from the Az module to AzBobbyTables, however it's also its own parameter in Get-AzDataTableEntity to select "columns" from the table.
One of these should probably be changed.

Removing the -Property alias is the way to go, even if it requires a major version bump because of a breaking change. While it does make it slightly more annoying to change from the AzTable module, I think it's the right move.


Provide fault-tolerant parsing

As a note, I'm volunteering to implement that, so, this issue is mostly to know if this is a suitable addition to cython.
My use case is that I'm using cython itself to create an AST for users in an IDE (so, it may be requested while the code is not complete).
Right now I'm using the code below to generate an AST from cython:
from Cython.Compiler.TreeFragment import parse_from_strings
mod = parse_from_strings(name, source)

My idea would be adding an additional parameter to parse_from_strings (something as fault_tolerant=True -- with a default of False) and when an error is found in a statement ignore errors until a next valid statement is found.
Errors would be attached to the returned AST.

I did some experiments here and just changing the fatal value in PyrexScanner.error (i.e.: https://github.com/cython/cython/blob/master/Cython/Compiler/Scanning.py#L480) to False (and storing the exceptions to be retrieved later) seems to be enough to make the parser fault-tolerant.
I was going to just monkey-patch this, but because those files are cython-compiled in the distribution, this is not possible (so, would it be ok if I did provide a pull request implementing this?)

Parser errors are usually fatal for the compiler, with only a few exceptions, but I wouldn't mind having a scanner-local setting that disables the raise in PyrexScanner.error(). Note that the attribute needs to be added to the Scanning.pxd as well. I think it's enough to add that option to the TreeFragment API for now, if that suits your use case. It can be added to other interfaces of the parser later, if necessary.
PR welcome.

Note that there are probably code sections in the parser code that implicitly expect an error() to raise and exit, instead of continuing with the rest of the code. If you run into such code, it should usually not be too hard to fix. PRs welcome for that, too.

Great, I'll take a look into that then.


Hallo doesn't display toolbar icons when using font-awesome 4.0.3

Hallo doesn't display  toolbar icons when using font-awesome 4.0.3 because the classes have changed:
http://fontawesome.io/icon/italic/
Font Awesome 4.0.3

Font Awesome 3.2.1

What files/lines would need to be changed in order to work with the newest font-awesome code?

Fixed by #213


feat: use at chops in pkam verb handler

- What I did

changed signing verification in pkam verb handler to at chops signing method
- How I did it
based on the pkam verb syntax, set the signing algorithm and hashing algorithm. ecc signing algo and sha256 hashing or rsa2048 signing algo and sha256 hashing(latter will be the default when pkam verb is sent the old way without any additional params)
construct the AtSigningVerificationInput and call at_chops.verify(..) method
changes done by @sitaram-kalluri and @purnimavenkatasubbu related to functional tests - loading pkam keys in server on startup
- How to verify it
run the functional test tests/at_functional_test/test/pkam_verb_test.dart

- Description for the changelog

Nice work everyone! A milestone PR.


With emacs --daemon, all-the-icons fonts are installed every time daemon is started

When using emacs --daemon, whenever the daemon is started, all-the-icons fonts are installed. This is caused by (font-family-list) returning nil in helm-icons--setup when there is no frame.

I will remove that call.


Parse multiple documents

This now has 2 tests as I created a new one before I spotted the one at the bottom.
I tested this locally and it happily works through my large Kubernetes manifests finding the correct rule violations.
fixes #746

Codecov Report
Attention: Patch coverage is 81.25000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 94.72%. Comparing base (de43af8) to head (c1aa751).
Report is 2 commits behind head on main.

Files
Patch %
Lines

internal/parser/parser.go
81.25%
2 Missing and 1 partial :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #894      +/-   ##
==========================================
- Coverage   94.75%   94.72%   -0.03%
==========================================
  Files          92       92
  Lines        8727     8738      +11
==========================================
+ Hits         8269     8277       +8
- Misses        315      317       +2
- Partials      143      144       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add kubefed validation webhook

NOTE: This PR is not reviewable until #857 #867 is merged.
This adds the scaffolding to enable kubefed webhook validation for core API types we plan to move to beta. Particularly the 2 APIs added for validation in this PR:

FederatedTypeConfig: currently only verifies target kind is set.
FederatedCluster: no validation at the moment.

This PR can be merged (pending #857 #867 merging) as follow-on work/discussion is needed to implement the actual validation pieces.
@pmorie did all the work here. I've just mainly added scaffolding for FederatedCluster while cleaning up the history, fixing few chart and deployment issues, etc.
TODO in follow-up PRs:

[ ] Implement validation pieces for core API types moving to beta
[ ] Add support for using k8s CA instead of self-generated one
[ ] Tests

Before WIP is removed, I'd like to see the reviewable changes squashed down to as few orthogonal commits as possible and highlighted for potential reviewers.  The amount of unreviewable content in this PR makes reviewing via 'Files Changed' all but impossible, and having to review changes commit by commit only works if the commits are few in number and easily identifiable as orthogonal.

@marun Is this more manageable now that deps have been merged separately?

I've rebased and this is now ready for review.

@font, is there any particular reason this PR not using the scaffolding provided by kubebuilder for webhooks? https://book.kubebuilder.io/beyond_basics/webhook_installer_generator.html

+1 with @shashidharatd. With kubebuilder, we do not need to maintain an independent webhook binary. It seems simpler.

@font, is there any particular reason this PR not using the scaffolding provided by kubebuilder for webhooks? https://book.kubebuilder.io/beyond_basics/webhook_installer_generator.html

From what I was able to tell, the kubebuilder support for webhooks is very minimal and depends on you using a particular controller framework which we are not using. We can refactor to it in the future if we want to but ironically using the kubebuilder facility would require significantly more intrusive changes than the approach this PR takes.

we do not need to maintain an independent webhook binary. It seems simpler

We're not adding a new binary; the webhook server is integrated with hyperfed so there will not be a new binary present in the image, just another symlink to hyperfed.

@font @pmorie I am not sure how is this tested. Is there a plan to introduce testing for these validations at some point?

@font @pmorie I am not sure how is this tested. Is there a plan to introduce testing for these validations at some point?

Generally validations are tested in unit tests for the validation methods; from an e2e standpoint they're exercised every time an object is created so they are naturally exercised by e2e scenarios. I think it would be fine to add a validation test for the initial validation in this PR in a follow, since the travis job has passed.
WDYT @irfanurrehman

@marun pending CI passing this should be ready to re-review.

LGTM

/lgtm


hotReloadCapable:true doesn't work  with java-springboot

/kind bug
/priority high

odo init --name java-springboot-starter --devfile java-springboot  --starter springbootproject
edit devfile and add hotReloadCapable: true to run exec command
run odo dev
edit source code
odo should just sync files and application should automatically pick up changes, but this is not happening.  When running locally with mvn spring-boot:run (the same command devfile uses) it works correctly.

When trying similar steps with nodejs, it works as expected.

odo init --name test --devfile nodejs --starter nodejs-starter
edit devfile and add hotReloadCapable: true to run exec command
install nodemon to project npm install --save-dev nodemon
modify package.json and use nodemon server.js as   start script
run odo dev
edit source code
right after file sync is done you can see changes in the application

Because it works in nodejs but doesn't with springboot it looks like this is specific to springboot container.

After some tests, I can see that files are correctly synced to the container, which are detected, and the rebuild is executed.
tools: [INFO] Scanning for projects...
tools: [INFO]
tools: [INFO] --------------------------< com.example:demo >--------------------------
tools: [INFO] Building demo 0.0.1-SNAPSHOT
tools: [INFO] --------------------------------[ jar ]---------------------------------
tools: [INFO]

tools: [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ demo ---
tools: [INFO] Deleting /projects/target
tools: [INFO]
tools: [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ demo ---
tools: [INFO] Using 'UTF-8' encoding to copy filtered resources.
tools: [INFO] Copying 1 resource
tools: [INFO] Copying 0 resource
tools: [INFO]
tools: [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ demo ---
tools: [INFO] Changes detected - recompiling the module!
tools: [INFO] Compiling 1 source file to /projects/target/classes
tools: [INFO]
tools: [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ demo ---
tools: [INFO] Not copying test resources
tools: [INFO]
tools: [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ demo ---
tools: [INFO] Not compiling test sources
tools: [INFO]
tools: [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ demo ---
tools: [INFO] Tests are skipped.
tools: [INFO]
tools: [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ demo ---
tools: [INFO] Building jar: /projects/target/demo-0.0.1-SNAPSHOT.jar
tools: [INFO]
tools: [INFO] --- spring-boot-maven-plugin:2.3.5.RELEASE:repackage (repackage) @ demo ---
tools: [INFO] Replacing main artifact with repackaged archive
tools: [INFO] ------------------------------------------------------------------------
tools: [INFO] BUILD SUCCESS
tools: [INFO] ------------------------------------------------------------------------
tools: [INFO] Total time:  1.773 s
tools: [INFO] Finished at: 2023-03-22T07:49:49Z
tools: [INFO] ------------------------------------------------------------------------

Adding this dependency to the project fixes the issue:
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <optional>true</optional>
    </dependency>

Separate issue: make the run command hotReloadCapable in the Devfile stack

I've tried adding spring-boot-devtools and it doesn't help

I had tested only on podman (odo dev --platform podman).
I can reproduce it on podman, but not on cluster.
After some research, I can see that the difference between both platforms is that, when you mark the Run command as hotReloadCapable, the Build command is executed after sources changes in Podman, but not on the cluster.
@kadel could you please double-check that you can reproduce on Podman?

As I understand:
springboot reload is done in 2 stages:

compiling sources into classes
reloading classes

The first stage is done by the Build command, after the sources are synced, and the second stage is done by mvn spring-boot:run (with spring-boot-devtools) when a class changes.
On the other hand, node reload works only with the command npm start; the Build command does not need to be ran again after each source change.

I've created a related issue regarding running the build command when Run is hotReloadCapable: https://github.com/redhat-developer/odo/issues/6695

After some tests, I can see that files are correctly synced to the container, which are detected, and the rebuild is executed.
tools: [INFO] Scanning for projects...
tools: [INFO]
tools: [INFO] --------------------------< com.example:demo >--------------------------
tools: [INFO] Building demo 0.0.1-SNAPSHOT
tools: [INFO] --------------------------------[ jar ]---------------------------------
tools: [INFO]

tools: [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ demo ---
tools: [INFO] Deleting /projects/target
tools: [INFO]
tools: [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ demo ---
tools: [INFO] Using 'UTF-8' encoding to copy filtered resources.
tools: [INFO] Copying 1 resource
tools: [INFO] Copying 0 resource
tools: [INFO]
tools: [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ demo ---
tools: [INFO] Changes detected - recompiling the module!
tools: [INFO] Compiling 1 source file to /projects/target/classes
tools: [INFO]
tools: [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ demo ---
tools: [INFO] Not copying test resources
tools: [INFO]
tools: [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ demo ---
tools: [INFO] Not compiling test sources
tools: [INFO]
tools: [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ demo ---
tools: [INFO] Tests are skipped.
tools: [INFO]
tools: [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ demo ---
tools: [INFO] Building jar: /projects/target/demo-0.0.1-SNAPSHOT.jar
tools: [INFO]
tools: [INFO] --- spring-boot-maven-plugin:2.3.5.RELEASE:repackage (repackage) @ demo ---
tools: [INFO] Replacing main artifact with repackaged archive
tools: [INFO] ------------------------------------------------------------------------
tools: [INFO] BUILD SUCCESS
tools: [INFO] ------------------------------------------------------------------------
tools: [INFO] Total time:  1.773 s
tools: [INFO] Finished at: 2023-03-22T07:49:49Z
tools: [INFO] ------------------------------------------------------------------------

Adding this dependency to the project fixes the issue:
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <optional>true</optional>
    </dependency>

Fixed when adding this dependency with latest odo version
/close


screenshots, window management

@jonsterling please, review

Sure, I'll look at it now. thanks!

This looks good :+1:

Thanks!


Fluid Storage

Will this include a complete fluid system with inserters/extractors and support for crafting as well?

Yes there will be inserters and extractors
Crafting I'm not sure if that is even possible?

I think that he was thinking along the lines of alloying. If that is the
case then it would be way too op.
On 20 June 2016 at 01:58, Raoul<EMAIL_ADDRESS>wrote:

Yes there will be inserters and extractors
Crafting I'm not sure if that is even possible?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/55#issuecomment-227018564,
or mute the thread
https://github.com/notifications/unsubscribe/ARh9c_6ZhnNSTAUTBqbGDP8jwfYteTBrks5qNaZ9gaJpZM4Ihl-E
.

How is it op to automate things that you can do in your crafting grid?
Cause that's what this mod already does.

How exactly can you alloy fluids in your crafting table? The only methods
that come to mind for me are Tinker's smelting and casting, and something
like an alloy smelter. Am I missing something?
On Mon 20 Jun, 2016 4:42 pm 4rz0<EMAIL_ADDRESS>wrote:

How is it op to automate things that you can do in your crafting grid?
Cause that's what this mod already does.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/55#issuecomment-227114489,
or mute the thread
https://github.com/notifications/unsubscribe/ARh9cyXM5bAAV9yHpXn9o3MlQoK3d2Kwks5qNnWbgaJpZM4Ihl-E
.

The crafter could output fluids, but then it wouldn't be able to output items on the same side.

I'm sure there are other machines that can take both; IC2 Canning Machine for example.

About the crafting, one feature that would be wonderful is the possibility to autofill buckets from the fluids storage, when crafting recipes that require such liquid. Like some foods from Pam for example, which requires water bucket. The system could get a available bucket and fill it with water (if there is water in the system) automatically.

Jup, that's pretty much what I meant.

http://mods.curse.com/mc-mods/minecraft/229218-extracells2

Fluid Disk Drive
Fluid Storage Disk
Fluid Constructor
Fluid Destructor
Fluid Importer
Fluid Exporter
Fluid Interface

All the features in the post above are implemented!

Awesome, thanks!


ARTEMIS-1476 HdrHistogram support on verbose SyncCalculation

It includes:

improved documentation of SyncCalculation::syncTest
HdrHistogram "write" latencies support using verbose

One example of the possible usage of the results using http://hdrhistogram.github.io/HdrHistogram/plotFiles.html is:

You told me you would just add verbose output here.. not change semantics of the test for no reason.
There was a lot of testing around this... changing the semantics like this out the blank without the proper testing won't help much.

This adds complexity that we don't need.. you're even requiring GCs to perform now.. the previous version wouldn't generate any garbage during the measurements (just simple buffer.write all the time.. always the same buffer).

@clebertsuconic
Good points! I've addressed all these things (with few other fixes), but I need feedbacks on the result.
I see 2 options here:
1) preallocate and cache the IO callbacks (but could be a large number too) in order to be garbage free again
2) let the IO callbacks to size the (young) GC regions in the first tests but uses a better way (ie more reliable) to be sure GC won't affect the measurements
I've pushed the second solution right now, using a method to perform reliable GCs (+ await them to happen) used in the official microbenchmark tool of the OpenJDK JMH too (http://hg.openjdk.java.net/code-tools/jmh/file/e96cad1fc480/jmh-core/src/main/java/org/openjdk/jmh/runner/BaseRunner.java#l309).
Feel free to give me feedbacks on it and if we need something similar in other parts (a more reliable way to perform/await GCs to happen) I can put it in our utlity classes too.
@michaelandrepearce @lulf I know you're pretty skilled in benchmarking too so please share your thoughts/opinions: are super wellcome!!!!

@clebertsuconic As a note: now without --verbose it is garbage free as in the original version :+1:

The issue is on the performWrite.  You are creating buffers and callbacks for each write.

In the last version I've pushed I'm reusing the same buffer

Here: https://github.com/franz1981/activemq-artemis/blob/8138e534d7b68d611e2425a0e7f65b20902c0fe0/artemis-cli/src/main/java/org/apache/activemq/artemis/cli/commands/util/SyncCalculation.java#L279

The license says public domain. And if you need a license BSD.

@clebertsuconic I'm not a lot familar with these, but:

For users of this code who wish to consume it under the "BSD" license
rather than under the public domain or CC0 contribution text mentioned
above, the code found under this directory is also provided under the
following license (commonly referred to as the BSD 2-Clause License)

what kind of licence I've to report on the pom.xml?

Lets use it under free public domain.  As they allow it.

@clebertsuconic ok!! I'm updating the PR with it!

@clebertsuconic Done!

@franz1981 i really like the ability to get a histogram of the sync test so you can see the percentiles of the disk performance from activemq's view. I do share some concerns with @clebertsuconic that it is quite a re-write in the logic, this is quite a core tool to get the journal settings right, i know @clebertsuconic spent a lot of time with us previously getting it to work and fix issues.
Have we an ability to validate the result is the same before and after this change? e.g. some sort of test?

@michaelandrepearce

To mitigate this concern, have we an ability to validate the result is the same before and after this change? e.g. some sort of test?

Right now the calculation of the timeout isn't changed at all while the test methodology (and the output of course) is changed only if --verbose is specified: it is always possible to "validate" the output (the output percentiles) turning  off --verbose to see if the timeout estimation is changed/different.
AFAIK the initial journal tuning is performed without --verbose hence I'm expecting that nothing will change from the original version.
The other improvements (without affecting the  no --verbose logic) are made to help the JIT to avoid OSR in order to get more reliable results when benchmarking vs temporary filesystem/very high speed devices/no fsync involved.

@franz1981, this makes sense and good to know +1, ill leave some comments in place in code in a sec, where looks like some bits still leaked into existing logic. and let you comment.

For instance.. I told you to use the same buffer, and the same Completion over and over.. like it was before. There's not guarantees about the ordering won't be changed... trusting that just makes it more fragile.
If you just added the Histogram like you said you would.. it would be one thing.. but this logic is completely changed.. and I had spent a lot of time to make sure about these calculations, using different hardwares (disk types).

@clebertsuconic

This is supposed to be a dumb piece of code.. making IO writes at garbage free.

And in the not verbose case it is garbage free as it was originally :) Please use the latest commit to see it.

For instance.. I told you to use the same buffer, and the same Completion over and over.. like it was before.

In the not verbose case I've reused the buffer and the IOCompletion while in the verbose case only the buffer, because I haven't found a way to maintain for each write operation its own start time: if you see a better way to do that please help me to understand it :+1:

If you just added the Histogram like you said you would.. it would be one thing.. but this logic is completely changed.. and I had spent a lot of time to make sure about these calculations, using different hardwares (disk types).

And it is the reason why I'm providing a new commit that split the 2 logic in a clean way in order to maintain the original 1:1 as it is (tiny bugs included).
Let me know if it is ok @michaelandrepearce too in order to have 6 eyes on it :+1:

I am using it...
I just don't understand why you would only use the last callback.. why so much complexity added?
if this complexity was added around say.. for instance... JournalImpl::show statistics.. I would go for it... but here at the sync calculation? I think this is being a lot of complexity for something that is supposed to be simple.

for instance.. you could have chosen to add the Histogram without making any other changes.. I don't understand why you wanted to do that? just to understand the code? (not that this would be a good reason).

@clebertsuconic

if this complexity was added around say.. for instance... JournalImpl::show statistics.. I would go for it... but here at the sync calculation? I think this is being a lot of complexity for something that is supposed to be simple.

+100 This is exactly the point mate :)
So do you think that is better to provide an ad hoc command (if not exists) for that (ie single write latencies measurements)?
This time I have trying to avoid having 2 tools similar but different, but I trust you if you see a different usage for it...

for instance.. you could have chosen to add the Histogram without making any other changes
This is sadly seems not possible (to me at least): each write request (with ASYNCIO and not syncWrites) need its own start time in order to compute the latency of the operation (and record it on the histogram).
How can you provide for each write its own start time with ASYNCIO and not syncWrites?

@clebertsuconic @michaelandrepearce Consider the last commit code the base for a "journal's show statistics" command: i will open a different JIRA with it if it seems ok and I will drop the timeout calculation too, the histograms are enough for it.

@franz1981 The Sync calculation is not part of the life cycle of the server running at all...
Think about it.. we first had nothings.. users had to figure out themselves based on their hardware specifications how many syncs their system supported, and then they would place it there.
At a later point, when we added create.. I added this sync tool to make an approximated auto-tune for the system. anything beyond that is out of context.
It doesn't really matter to add histograms at the sync-calculation, or to perform any writes...
However.. during the process (aka production)... it would matter...
Look at TimedBuffer::logRatesTimer...
there a Histogram would make a lot more sense... and actually that logRatesTimer doesn't belong into TimedBuffer maybe.

It doesn't really matter to add histograms at the sync-calculation, or to perform any writes...

That's the part I'm trying to understand: IMHO it matters to have a more detailed view of the sync test using standard de facto tools too as @michaelandrepearce  has stated above in other comments.
While having it in production in real time  (as another PR I've already pushed has provided) is a different but equally (maybe more) important purpose.
With IO, treating the latency just as 1/throughput risks to hide important informations: providing a test that shows the distribution of the percentiles could give to the expert user the chance to tune the buffer timeout based on the 99 percentile (taken from the real hw) instead of an average in order to be sure to met his SLA.

wouldn't be easier to keep all the original code and just add the Histogram.. without changing how callbacks are happening for example.. (The reason why you needed GCs).
just use the original code... make simple changes... add the histogram (when --verbose).. and it's a lot easier to understand the impact of the changes.

if you are actually measuring 99%.. do you really need to take out GCs from the calculation?
just keep it simple.. this is not simple as I see this.

@clebertsuconic

wouldn't be easier to keep all the original code and just add the Histogram.. without changing how callbacks are happening for example.. (The reason why you needed GCs).

There are good reasons I can't do it and one is written below:

//it is necessary to allocate new callbacks because each async write has its own elapsed time

      final boolean asyncWrites = journalType == JournalType.ASYNCIO && !syncWrites;
      final Supplier<? extends SyncIOCompletion> ioCallbackFactory;
      if (asyncWrites) {
         //it is necessary to allocate new callbacks because each async write has its own elapsed time
         ioCallbackFactory = () -> new LatencyRecorderIOCallback(writeLatencies);
      } else {
         //i can reuse the same callback each time
         final ReusableIOCallback ioCallback = new ReusableIOCallback(writeLatencies);
         //uses will perform a countUp and reset start
         ioCallbackFactory = () -> {
            ioCallback.beforeWrite();
            return ioCallback;
         };
      }

The others was related to compilation of methods by the JIT compiler if you have > 10000 writes and very fast storages/no fsync: you need to add -XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining to the command line to find it.

if you are actually measuring 99%.. do you really need to take out GCs from the calculation?

I'm providing all the percentiles offered by the HDR histogram and the GC is already taken out of the async measurements (if you have enough heap) by using the awaitGC method: it is already battle tested and defined in the official OpenJDK Java Microbenchmark Harness (http://openjdk.java.net/projects/code-tools/jmh/) here.
IMO could be a good idea to provide this feature it is to drop at all the timeout calculation if a user want the latencies percentile and let the original calculation being used otherwise. I simply don't know how in what form expose such tool, @clebertsuconic @michaelandrepearce any idea?make sense?

@clebertsuconic @michaelandrepearce

IMO could be a good idea to provide this feature it is to drop at all the timeout calculation if a user just want the latencies percentiles and let the original calculation being used otherwise. I simply don't know how/in what form expose such tool, @clebertsuconic @michaelandrepearce any idea?make sense?

I've provided an attempt to do that here: https://github.com/franz1981/activemq-artemis/tree/stat-journal-histogram
Today I'm doing other stuff so I will probably do something on that in the weekend...

@franz1981 this is much cleaner +1, also its good to see the existing logic remains pretty much the same, just now with HDR being captured.