In-place type definition/restriction

There are different cases, when it is desirable to indicate the type or meta-type of a symbol (especially, variable) in a concrete expression.

Some examples come from dependent type usage, e.g. Idris definition Refl : (x : a) -> Equal x x supposes that we have a variable of type a, and the output type is defined by this variable. Refl 42 is of type Equal 42 42. In MeTTa, we specify only types. For (: Refl (-> a (Equal a a))) , (Refl 42) will be of type (Equal Int Int). In type definitions, we are lacking the possibility to use the argument value passed to the constructor. There is a possible workaround, which partly works, if we define: (: (Refl $x) (=== $x $x)). !(get-type (Refl 42)) returns [(%Undefined% Number), (=== 42 42)]. The problem is also that the type of Refl itself is %Undefined%. Whether we should allow (and interpret in a special way) in-place definitions like (: $x $a) or should support definitions like (: (Refl $x) (=== $x $x)) more cleanly, is the question to be discussed.

In equalities, we are lacking the possibility to specify the type of the argument (or bind it to a variable).  For example, one may want to return the type of the argument, e.g. (= (type (: $x $t)) $t). It should be noted that we may need to specify a type (e.g., (: $x $t) instead of just $x), which is normally missing, while in type definitions we may need to specify the argument (e.g., (: $x $a) instead of just $a), which is normally missing. The example of using the type of the argument can be worked around with get-type grounded function: (= (type $x) (get-type $x)).

2b) We may want to use types of arguments as additional restrictions / conditions. For example, if we have a generic fmap type definition
(: fmap (-> (-> $a $b) ($F $a) ($F $b)))
it's tempting to define
(= (fmap $f $C0) $C0)
to handle (: Nothing (Maybe $t)) or (: Nil (List $t)). Unfortunately, the pattern (fmap $f $C0) will be matched against any expression to be bound to $C0. We may want to restrict $C0 meta-type to be Symbol. Something like
(= (fmap $f $C0)
   (if (== (get-meta-type $C0) Symbol)
       $C0
       (superpose ())))

may work, but looks cumbersome. It would be much more convenient to be able to write (= (fmap $f (: $C0 Symbol)) $C0).
The same is true for the type definition of fmap. We may want to indicate that $F should be of Functor type (the problem, however, is that if we write (: $F Functor) in type definition, it might mean that $F is the argument value).

We may want to have type restrictions in pattern matching in general (to make it more efficient). For example, we may want to retrieve all true relations, (match &self (= ($R $x $y) True) (related $x $y), but only about humans, e.g. (: $x Human), (: $y Human). We may imagine a composite query

(match &self (, (: $x Human)
                (: $y Human)
                (= ($R $x $y) True))
    (related $x $y))

but if the pattern matcher tries to satisfy conditions independently and then intersects them, then it will be very inefficient. Also, the query (: $x Human) may not work for derived types (only for explicit statements in the knowledge base?). Type restrictions might be more efficient to have inside one query (match &self (= ($R (: $x Human) (: $y Human)) True) ...). If match supports this form of restrictions, it might help to automatically solve (2) as well. More generally, such patterns as (match &self (= ($R (: $x $t) (: $y $t)) True) ...) are also desirable.
If we allow in-place type definitions, typing expressions will be treated specially. E.g., (match &self (: $x Human) $x) will search not for records like (: Plato Human), but for individual symbols (stored in Atomspace) with the corresponding type restriction. Then, the question will arise putting (: Plato Human) into Atomspace should result in putting Plato there together with its type, or one needs to put both (: Plato Human) and Plato into Atomspace in order for (match &self (: $x Human) $x) to retrieve the result. Thus, we either need to syntactically distinguish between in-place type restrictions and general type definitions (e.g., (match &self (:: $x Human) $x)) or to treat : uniformly on the level of implementation (e.g., it attaches a type to a symbol instance, and each symbol instance has its own set of types).
In the former case, it will be possible to distinguish between the desire to declare the type (which can be non-deterministic) and to check the type, e.g. with (: (Refl $x) (=== $x $x)), one can write (: (Refl MyTrue) (=== MyTrue MyFalse)), if MyTrue and MyFalse are both of MyBool type, because this is a (part of non-deterministic) type declaration. However, if one writes (:: (Refl MyTrue) (=== MyTrue MyFalse)) it will not work, because (Refl MyTrue) has (=== MyTrue MyTrue)) type.
In turn, ascribing individual types to symbol instances might be convenient in its own ways.
Thus, syntactic and semantic aspects should be analyzed together to support in-place type declarations and/or restrictions.

Similar question from Jonathan:

just wanted to bring up a few points about the representation of dependent products in MeTTa. For instance, in Idris, the following is valid:

F : Bool -> Type
F True = Nat
F False = Bool

f : (x : Bool) -> F x
f True = 2
f False = True

Here, the type (x : Bool) -> F x is equivalent to the expression Πx:Bool.(F x) in type theory, i.e. it represents a dependent product type. For a translation into MeTTa of the above, I came up with the following:

(: F (-> Bool Type))
(= (F True) Nat)
(= (F False) Bool)

(: DepProd (-> (-> $t Type) Type))
(: ($f $x) (match &self (: $f (DepProd $F)) ($F $x)))

(: f (DepProd F))
(= (f True) (S (S Z)))
(= (f False) True)

where I’ve introduced a custom type constructor, DepProd, to represent the dependent product. This appears to behave more-or-less as desired; for instance, the following assert statements are all valid:

!(assertEqualToResult
  (get-type f)
  ((DepProd F)))
!(assertEqualToResult
  (get-type (f True))
  (((DepProd F) Bool) ((DepProd F) Bool) Nat))
!(assertEqualToResult
  (get-type (f False))
  (((DepProd F) Bool) ((DepProd F) Bool) Bool))

The latter two contain the unreduced type expression ((DepProd F) Bool) in addition to the reduced type expressions Nat and Bool respectively; this isn’t necessarily a problem, since ((DepProd F) Bool) is effectively a name for the union-type Nat ∪ Bool, and hence can be seen as valid from a gradual typing perspective. I was wondering though if there is (or are plans to add) a more direct way of representing dependent products?  For instance, one possibility could be to directly mirror the Idris syntax, i.e. in the above to allow the syntax (: f (-> (: $x Bool) (F $x))), which is not currently possible in MeTTa

One more interesting example:
(= (random $a $b) $a)
(= (random $a $b) $b)
!(random Red (random Green Blue))

produces [Blue, Red, Green, Red] with two Red. One may want to define a type for random, e.g.
(: random (-> $t1 $t2 (Either $t1 $t2)))

but one may want to have only one Red in the result, which can be done via:
(: random (-> Atom Atom Atom))

Once again, we may want to have both normal types in the type definition and Atom annotation simultaneously.
As a side note: interaction of Atom with non-determinism is possibly an issue on its own.
Also, specifying
(: random (-> $t1 $t2 $t1))
(: random (-> $t1 $t2 $t2))

produces even more results (as per #235 )

@ngeiswei


collect /var/log/amazon/ directory

This directory can often contain /var/log/amazon/ssm and /var/log/amazon/efs
After this change these are both collected in the bundle directory:
$ ll amazon_logs/
total 8
drwxr-xr-x 2 root root 4096 Mar 10 21:27 efs
drwxr-xr-x 4 root root 4096 Mar 10 21:27 ssm

$ ll amazon_logs/ssm/
total 112
-rw-r--r-- 1 root root 79941 Mar 10 21:27 amazon-ssm-agent.log
-rw-r--r-- 1 root root 18802 Mar 10 21:27 AmazonSSMAgent-update.txt
drwxr-xr-x 2 root root  4096 Mar 10 21:27 audits
drwxr-xr-x 3 root root  4096 Mar 10 21:27 download
-rw-r--r-- 1 root root   565 Mar 10 21:27 errors.log

I've also updated the pull request template to only ask for testing on more modern and relevant OS variants.

Summary

Implementation details

Testing

[x] Works properly on Amazon Linux
[x] Works properly on Amazon Linux 2
[x] Works properly on Ubuntu 20.04
[x] Works properly on CentOS 8

Licensing

This contribution is under the terms of the Apache 2.0 License:

feel free to reopen if neeeded

Could we please reopen this?
We need this logs for EFS troubleshoot and it would be helpful to be collected from the tool.


Columns Test Added

Added Columns Test
Added environments bacomblog and blog to envs.js
Added workspace settings file to .gitignore for VSCode

These were the comments from the slack conversation in #milo-automation:
Santosh:
i am still thinking / bit debating on way of using -selector.js
/ const row = page.locator(${selectors[props.tag]} >> .row).first();
we should go with class object for selectors
Aaron:
So are you thinking that we shouldn’t use the idea of selectors.js and instead create a page object class?
I could make that change, but I think the reason we had the selectors.js was so we could tie it into the tags from the spec file.  Am I thinking incorrectly about that?
Chris M:
I dislike the idea of a class object abstraction for selectors. If you watch that video, even our current way of needing three files to get one test out the door seems a bit... much.
The proposal you had Santosh in your original PR required passing around objects, and there wasn't really any logic being done in that class file.
Santosh:
my thinking was ,  not to put too much code / logic in test files
looking from long run / maintenance perspective, how can we reduce changing at multiple scripts for a change in selector definition...
Chris M. :
Selectors are out of the test file. Adding a class isn't going to make that any better.
Santosh:
i would also think of adding re-usable function there or somewhere , rather in test scripts...
No class is okie,/  question is where to place common validation functions ?
should we have separate js?
So with that discussion I'm going to adjust the test so that the .row/.col class selectors are actually in the columns-selectors.js file.

@skumar09 , I fixed the github failures for the columns test.  You can see now that the playwright tests pass in GitHub below.  I also moved the selectors you mentioned into the columns.selectors.js file.

@amauch-adobe good point on the private slack convos. They need to be here. I'll get better at that.


Serialize cache value to JSON before trying to parse it as JSON

What kind of change does this PR introduce? (check at least one)

[x] Bugfix
[ ] Feature
[ ] Code style update
[ ] Refactor
[ ] Other, please describe:

[x] You consent that the copyright of your pull request source code belongs to Authlib's author.

Fixes #479.

Thanks


external_ref/id not handled correctly

When I execute the following AQL query SELECT e/ehr_status/subject/external_ref FROM EHR e I get the following result:
{
  "q": "SELECT e/ehr_status/subject/external_ref FROM EHR e",
  "columns": [
    {
      "path": "/ehr_status/subject/external_ref",
      "name": "#0"
    }
  ],
  "rows": [
    [
      {
        "id": {
          "_type": "GENERIC_ID",
          "value": "42",
          "scheme": "DEMOGRAPHIC"
        },
        "type": "PARTY",
        "_type": "PARTY_REF",
        "namespace": "test"
      }
    ]
  ]
}

However querying SELECT e/ehr_status/subject/external_ref/id FROM EHR e gives the exact same result rows. This causes querying all subsets like SELECT e/ehr_status/subject/external_ref/id/value FROM EHR e to be null.

Please check the external_ref is not null in the data you are querying, since by the RM it is not mandatory https://specifications.openehr.org/releases/RM/latest/common.html#_overview_3

Please check the external_ref is not null in the data you are querying, since by the RM it is not mandatory https://specifications.openehr.org/releases/RM/latest/common.html#_overview_3

Thank you for your reply! As you see in the given example, the value is obviously 42 so it's definitely not null. ;)

yes, but what is the complete EHR_STATUS? can you query that and paste the complete result? thanks

{
  "q": "SELECT e/ehr_status FROM EHR e",
  "columns": [
    {
      "path": "/ehr_status",
      "name": "#0"
    }
  ],
  "rows": [
    [
      {
        "name": {
          "value": "EHR Status"
        },
        "_type": "EHR_STATUS",
        "subject": {
          "_type": "PARTY_IDENTIFIED",
          "external_ref": {
            "id": {
              "_type": "GENERIC_ID",
              "value": "123",
              "scheme": "DEMOGRAPHIC"
            },
            "_type": "PARTY_REF",
            "namespace": "test"
          }
        },
        "is_queryable": true,
        "is_modifiable": true,
        "archetype_node_id": "openEHR-EHR-EHR_STATUS.generic.v1"
      }
    ]
  ]
}

thanks, it seems you have just one EHR, is that correct?
also, is GET /ehr/uid giving you a consistent result as that one from the AQL? (I mean the same data in the status)

Yes, just a single EHR. I started off a fresh database a few days ago.
GET/ehr/ehr_id gives me the following result:
{
  "compositions": null,
  "contributions": null,
  "system_id": {
    "_type": "HIER_OBJECT_ID",
    "value": "10f4b306-9431-465f-aaff-de5baabd7aba"
  },
  "ehr_id": {
    "_type": "HIER_OBJECT_ID",
    "value": "3fe034c6-635a-4a6a-b690-94cd219f0359"
  },
  "ehr_status": {
    "_type": "EHR_STATUS",
    "name": {
      "_type": "DV_TEXT",
      "value": "EHR Status"
    },
    "subject": {
      "_type": "PARTY_SELF",
      "external_ref": {
        "_type": "PARTY_REF",
        "namespace": "test",
        "id": {
          "_type": "HIER_OBJECT_ID",
          "value": "42"
        }
      }
    },
    "archetype_node_id": "openEHR-EHR-EHR_STATUS.generic.v1",
    "uid": {
      "_type": "HIER_OBJECT_ID",
      "value": "27e874f7-4371-4a35-8776-0c06f08cb196"
    },
    "is_modifiable": true,
    "is_queryable": true
  },
  "time_created": "2020-04-03T16:11:51.248370"
}

@chevalleyc This is related to your changes at 97d9324c59d5cfb884eff7ef9d532720f175f241 in service/src/main/java/org/ehrbase/aql/sql/queryImpl/attribute/partyref/PartyResolver.java
Reverting the changes in that file makes the query work again. But that's for sure not how it's meant to be...

I just checked this. It looks like there is no issue regarding null but the partial paths for e/ehr_status/subject/external_ref/id still is not correct. Can you confirm @chevalleyc @nr23730 ?

Yes, I can confirm this.
I made a PR some time ago (#229) that had the branch of #228 as target that partially addressed this issue. But looks like #228 was merged into master before #229 was merged into the branch of #228. I can create another PR (with master as target this time!) if you like...

This would be fantastic, thx!


sc-916 improve contact handling

This adds some utilities for dealing with the contacts. The main thing is a way to iterate through the contacts on a VASP more easily. Previously, we had to specify the contacts in a slice and then iterate through that slice every time a function wanted to iterate through the contacts. This PR adds two functions for iteration, IterContacts and IterVerifiedContacts, which both return anonymous functions which can be used to perform the iteration. This ensures that we are always iterating through the contacts in the same order and gets us access to the contact "kinds" if we need them ("technical", "administrative", "legal", and "billing").

Codecov Report

Merging #304 (c2f3bb6) into main (b8436e0) will decrease coverage by 0.02%.
The diff coverage is 49.17%.

@@            Coverage Diff             @@
##             main     #304      +/-   ##
==========================================
- Coverage   45.84%   45.82%   -0.03%
==========================================
  Files          58       59       +1
  Lines        7309     7298      -11
==========================================
- Hits         3351     3344       -7
+ Misses       3332     3324       -8
- Partials      626      630       +4

Impacted Files
Coverage Δ

pkg/gds/emails/client.go
8.12% <0.00%> (+0.78%)
:arrow_up:

pkg/gds/models/v1/models.go
70.27% <ø> (-1.68%)
:arrow_down:

pkg/gds/admin.go
54.63% <60.00%> (-0.76%)
:arrow_down:

pkg/gds/gds.go
66.77% <72.72%> (-1.00%)
:arrow_down:

pkg/gds/models/v1/contacts.go
72.72% <72.72%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8436e0...c2f3bb6. Read the comment docs.

@bbengfort I pushed some changes, let me know what you think of the new refactor. The goal was to make the caller code a lot simpler by using a Next() and Value() pattern for the iteration.

@bbengfort I've pushed your recursion changes and the new multierror handling for your review.


artifact extract tweaks

About the PR

make the button green since bozos still think extracting is bad because red
change every destroy reference to be Extract

Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog
no cl no fun

can you either increase or decrease the distance between the extract button and the other 3? The amount is like right in the middle and it bugs me.

can you either increase or decrease the distance between the extract button and the other 3? The amount is like right in the middle and it bugs me.

done

@EmoGarbage404 i died pls restart

@EmoGarbage404 merg


UI Crash on hovering over annotation conflicts which don't have UI representation

Actions before raising this issue

[X] I searched the existing issues and did not find anything similar.
[X] I read/searched the docs

Steps to Reproduce

Create a task from this backup task.zip
Go into the review section of the job
See the conflicts with the GT job
While hovering over the conflicts, suddenly, the UI crashes with the following error message

Expected Behavior
The conflicting annotation is to be highlighted if available
Possible Solution
I guess that this issue occurred when I tried to highlight conflicts which can't be displayed on UI like the mismatching groups. So we can just filter not to highlight such conflicts. https://github.com/cvat-ai/cvat/blob/develop/cvat-ui/src/components/annotation-page/canvas/views/canvas2d/canvas-wrapper.tsx#L483
Context
Error Message
TypeError
drawnState is undefined

deactivateShape@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:143205:11
deactivate@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:143249:10
highlight@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:143526:12
notify@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:142585:12
notify@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:147898:16
highlight@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:140847:10
highlight@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:140283:16
componentDidUpdate@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:179756:22
commitLifeCycles@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:442761:22
commitLayoutEffects@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:445729:23
callCallback@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:423114:14
invokeGuardedCallbackDev@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:423163:16
invokeGuardedCallback@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:423218:31
commitRootImpl@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:445467:30
unstable_runWithPriority@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:466190:12
runWithPriority$1@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:433965:10
commitRoot@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:445307:20
finishSyncRender@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:444733:13
performSyncWorkOnRoot@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:444719:23
../node_modules/react-dom/cjs/react-dom.development.js/flushSyncCallbackQueueImpl/<@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:434015:24
unstable_runWithPriority@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:466190:12
runWithPriority$1@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:433965:10
flushSyncCallbackQueueImpl@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:434010:24
flushSyncCallbackQueue@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:433998:3
batchedEventUpdates$1@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:444803:7
batchedEventUpdates@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:423721:12
dispatchEventForLegacyPluginEventSystem@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:426494:24
attemptToDispatchEvent@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:427193:44
dispatchEvent@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:427115:41
unstable_runWithPriority@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:466190:12
dispatchUserBlockingUpdate@http://localhost:3000/assets/cvat-ui.54562adfd37f714d4369.min.js:427098:18

The following was logged in the browser console,
Uncaught TypeError: drawnState is undefined
    deactivateShape canvasView.ts:2250
    deactivate canvasView.ts:2304
    highlight canvasView.ts:2614
    notify canvasView.ts:1557
    notify master.ts:27
    highlight canvasModel.ts:651
    highlight canvas.ts:113
    componentDidUpdate canvas-wrapper.tsx:492
    React 6
    unstable_runWithPriority scheduler.development.js:653
    React 5
    unstable_runWithPriority scheduler.development.js:653
    React 8
    unstable_runWithPriority scheduler.development.js:653
    dispatchUserBlockingUpdate React
canvasView.ts:2250:16

The above error occurred in the <CanvasWrapperComponent> component:
    in CanvasWrapperComponent (created by Connect(CanvasWrapperComponent))
    in Connect(CanvasWrapperComponent) (created by CanvasLayout)
    in div (created by GridItem)
    in Resizable (created by GridItem)
    in DraggableCore (created by GridItem)
    in GridItem (created by ReactGridLayout)
    in div (created by ReactGridLayout)
    in ReactGridLayout (created by WidthProvider)
    in WidthProvider (created by CanvasLayout)
    in main (created by Basic)
    in Basic (created by Content)
    in Content (created by CanvasLayout)
    in CanvasLayout
    in CanvasLayout (created by ReviewWorkspaceComponent)
    in section (created by BasicLayout)
    in BasicLayout (created by Layout)
    in Layout (created by ReviewWorkspaceComponent)
    in ReviewWorkspaceComponent (created by AnnotationPageComponent)
    in main (created by Basic)
    in Basic (created by Content)
    in Content (created by AnnotationPageComponent)
    in section (created by BasicLayout)
    in BasicLayout (created by Layout)
    in Layout (created by AnnotationPageComponent)
    in AnnotationPageComponent (created by Connect(AnnotationPageComponent))
    in Connect(AnnotationPageComponent) (created by Context.Consumer)
    in withRouter(Connect(AnnotationPageComponent)) (created by Context.Consumer)
    in Route (created by CVATApplication)
    in Switch (created by CVATApplication)
    in main (created by Basic)
    in Basic (created by Content)
    in Content (created by CVATApplication)
    in section (created by BasicLayout)
    in BasicLayout (created by Layout)
    in Layout (created by CVATApplication)
    in ShortcutsProvider (created by CVATApplication)
    in GlobalErrorBoundary (created by Connect(GlobalErrorBoundary))
    in Connect(GlobalErrorBoundary) (created by CVATApplication)
    in CVATApplication (created by Context.Consumer)
    in withRouter(CVATApplication) (created by Connect(withRouter(CVATApplication)))
    in Connect(withRouter(CVATApplication))
    in Router (created by BrowserRouter)
    in BrowserRouter
    in Provider

React will try to recreate this component tree from scratch using the error boundary you provided, GlobalErrorBoundary. react-dom.development.js:19527

On trying on multiple types of conflicts, I guess this behaviour only occurs for the mismatching groups type of conflict I have attached a screenshot of the console log detail of the action which led to this crash.

A similar UI crash was observed when the shape attribute was missing, like the drawnState attribute, as mentioned above.

While trying out the issue's scenario Ive noticed that conflicts are not shown properly on the 0 frame. If I change the the frame and go back, the conflicts are rendered properly and no crush occurs. I belive we need to adress the problem with why are those conflicts are not rendered.
Consider:

The crash was resolved in #7917. The issue with conflicts not showing appears to stem from inconsistent z-order behavior. As we navigate through frames, the z-order automatically sets to the maximum value, but when the job is first opened, it defaults to 0.


handle FETCH empty response

@wangsrGit119 Can you please review and test this

2024-10-09 15:57:00 INFO     Client lifecycle state is State.DISCONNECTED
2024-10-09 15:57:00 INFO     Openning connection...
2024-10-09 15:57:00 INFO     Connection to server is successfully established
2024-10-09 15:57:00 INFO     Client state changed from State.DISCONNECTED to State.NOT_IDENTIFIED
2024-10-09 15:57:00 INFO     Password is not required
2024-10-09 15:57:00 INFO     Client state changed from State.NOT_IDENTIFIED to State.IDENTIFIED
2024-10-09 15:57:00 INFO     Registered handler for jobtype: speaker-split
2024-10-09 15:57:00 INFO     Entering run loop..
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/conda/envs/speaker-split/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/conda/envs/speaker-split/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/client.py", line 275, in _heartbeat
    self._beat(rss_kb=self.rss_kb)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/util/decorators.py", line 27, in wrapper
    return fn(self, *args, **kw)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/util/decorators.py", line 40, in wrapper
    return fn(self, *args, **kw)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/client.py", line 510, in _beat
    self._raise_error(msg)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/client.py", line 264, in _raise_error
    if faktory_response[0] == "-":
IndexError: string index out of range
2024-10-10 08:17:00 ERROR    Shutting down due to an error: Traceback (most recent call last):
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/consumer.py", line 200, in run
    job = self.client._fetch(queues_tmp)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/util/decorators.py", line 27, in wrapper
    return fn(self, *args, **kw)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/util/decorators.py", line 40, in wrapper
    return fn(self, *args, **kw)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/client.py", line 448, in _fetch
    msg = self._send_and_receive(command)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/client.py", line 259, in _send_and_receive
    self._send(command)
  File "/opt/conda/envs/speaker-split/lib/python3.9/site-packages/pyfaktory/client.py", line 240, in _send
    self.sock.send(command.encode("utf-8"))
BrokenPipeError: [Errno 32] Broken pipe

2024-10-10 08:17:00 INFO     Run loop exited, state is State.IDENTIFIED
2024-10-10 08:17:00 INFO     Grace period of 25 seconds...
2024-10-10 08:17:00 INFO     Press Ctrl-C again to stop immediately
2024-10-10 08:17:00 INFO     End of the grace period. Stopping.
2024-10-10 08:17:00 ERROR    task error : [Errno 32] Broken pipe
---------------->fetch msg is None
---------------->fetch msg is None
---------------->fetch msg is None
2024-10-10 08:17:00 INFO     current server：tcp://<IP_ADDRESS>:7419
2024-10-10 08:17:00 INFO     Client lifecycle state is State.DISCONNECTED
2024-10-10 08:17:00 INFO     Openning connection...
2024-10-10 08:17:00 INFO     Connection to server is successfully established
2024-10-10 08:17:00 INFO     Client state changed from State.DISCONNECTED to State.NOT_IDENTIFIED
2024-10-10 08:17:00 INFO     Password is not required
2024-10-10 08:17:00 INFO     Client state changed from State.NOT_IDENTIFIED to State.IDENTIFIED
2024-10-10 08:17:00 INFO     Registered handler for jobtype: speaker-split
2024-10-10 08:17:00 INFO     Entering run loop..


location: Make directory before using it.

The previous test was wotking because D:\ exist, Installing in another directory failed with Error: The cwd: D:\M does not exist!

Thanks!


Ladybug starts a new report for each call from IbisLocalSender to JavaListener

❗ please do not add sensitive information in issues, you can provide extra information via email using issue number as reference ❗
Describe the issue
Since a few weeks, Ladybug starts a new report when an IbisLocalSender calls a JavaListener. The flow of the called adapter appears in the new report and is not present in the report of the calling adapter. Jaco suggested that something may be wrong with the correlationId. Initially I thought this was an issue of ibis-ladybug, see https://github.com/ibissource/ibis-ladybug/issues/193, but Jaco said to me that it is in the Frank!Framework.
Here is how this looks like in ZaakBrug:

Reporter
Martijn Dirkse

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Screenshots
If applicable, add screenshots to help explain your problem.
Configuration
Please provide the configuration of the Pipe or Receiver with the problem.

Input
Please provide an example of the input message.
Alternatively, provide a Ladybug report.

Environment
FF! 7.9-20230530.190019: ZaakBrug
Running on 420662a04158 using Apache Tomcat/8.5.85
Java Version: OpenJDK Runtime Environment (1.8.0_362-b09)
Heap size: 945M, total JVM memory: 1174M
Free disk space: 44GB, total disk space: 58GB
Up since: 2023-06-07 12:28:35 (2m)```

**Additional Environment**
 - DBMS: [e.g. Oracle, MSSQL, MariaDB, MySql, PostgreSQL]
 - Browser: [e.g. Chrome, Safari, Edge, Firefox]

**Additional Context**
Add any other context about the problem here. (f.e. ladybug report / test adapter with larva test)

Seems to be a duplicate of #4224 ?

Issue #4224 is quite old; it is from December 2022. We did not see this problem in ZaakBrug before May I think. Maybe issue #4224 was solved without the issue report being updated. Then the issue may have been reintroduced more recently.

Has synchronous been set to false or not?

Has synchronous been set to false or not?
synchronous has not been set to false.

Then it's not a duplicate! 😭

I checked the 7.8 branch and it looks like the issue will not appear there.


Alpine Linux build for Crossbar

Can you add an alpine Linux version for crossbar, image size on ubuntu is unnecessarily large
The ubuntu version weighs in at 366.9mb which takes roughly 30 seconds to download on a 100 Mbit/s connection.
I would love to see the image size down to <= 120mb which would bring the download time closer to 9 seconds.
20 seconds might not seem like it would make much of a difference but when your scaling up and down in response to traffic it makes a significant difference.
Note regarding this article it assumes that you are always deploying from the same machine; in most scaling scenarios you are adding and removing new machines/instances in response to traffic flow.
Adding new instances to support increased traffic will require downloading all of the layers used in a docker container, it is these download times that matter most when scaling up and down.

Yeah, that would be nice, agreed!
Problem is: this is more work, for the following reasons.
When you log into the Docker image
oberstet@thinkpad-t430s:~/scm/crossbario/crossbar-docker/crossbar/x86_64$ sudo docker run -it --entrypoint=/bin/bash crossbario/crossbar

this is what Crossbar.io will show as versions:
daemon@49e67969de28:/$ crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
     __  __  __  __  __  __      __     __
    /  `|__)/  \/__`/__`|__) /\ |__)  |/  \
    \__,|  \\__/.__/.__/|__)/~~\|  \. |\__/

 Crossbar.io        : 0.13.0
   Autobahn         : 0.13.0 (with JSON, MessagePack, CBOR)
   Twisted          : 16.0.0-EPollReactor
   LMDB             : 0.89/lmdb-0.9.18
   Python           : 2.7.10/PyPy-5.0.0
 OS                 : Linux-3.13.0-83-generic-x86_64-with-debian-jessie-sid
 Machine            : x86_64

Now, the Crossbar.io CLI is actually just a wrapper script loading the respective Python code
daemon@49e67969de28:/$ which crossbar
/opt/crossbar/bin/crossbar
daemon@49e67969de28:/$ cat `which crossbar`
#!/opt/crossbar/bin/pypy

# -*- coding: utf-8 -*-
import re
import sys

from crossbar.controller.cli import run

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(run())

The first lines shows that Crossbar.io is using its own bundled PyPy (/opt/crossbar/bin/pypy).
This PyPy again is built with most things bundled:
daemon@49e67969de28:/$ which pypy
/opt/crossbar/bin/pypy
daemon@49e67969de28:/$ ldd `which pypy`
    linux-vdso.so.1 =>  (0x00007ffcb51ac000)
    libpypy-c.so => /opt/crossbar/bin/libpypy-c.so (0x00007f93f3178000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f93f2f5a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f93f2b95000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f93f2992000)
    libz.so.1 => /opt/crossbar/lib/libz.so.1 (0x00007f93f2778000)
    libssl.so.1.0.0 => /opt/crossbar/lib/libssl.so.1.0.0 (0x00007f93f2506000)
    libcrypto.so.1.0.0 => /opt/crossbar/lib/libcrypto.so.1.0.0 (0x00007f93f20bb000)
    libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f93f1eab000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f93f1ca7000)
    libexpat.so.1 => /opt/crossbar/lib/libexpat.so.1 (0x00007f93f1a75000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f93f186d000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f93f1567000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f93f132e000)
    libffi.so.6 => /opt/crossbar/lib/libffi.so.6 (0x00007f93f1124000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f93f0efb000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f93f7b00000)

Sorting the dependent libraries, these come from:

/lib/x86_64-linux-gnu/  => Linux distro userland/system libraries
/opt/crossbar/lib => Crossbar.io bundled libraries

You can see that Crossbar.io bundles everything, including OpenSSL and CFFI!
What it does NOT, can cannot bundle is eg the C run-time library: /lib/x86_64-linux-gnu/libc.so.6
And Alpine Linux does not use GNU C lib (as above), but a different one!
This all means that the Debian package for Crossbar.io we provide would need to be built on Alpine: http://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package
And then it's unclear if PyPy is working with that alternative C run-time Alpine is using. Because ... there is no PyPy image on Alpine too https://pkgs.alpinelinux.org/packages?name=pypy&repo=all&arch=x86_64&maintainer=all
So, this is all breaking new land basically ..

Also, consider this:
oberstet@thinkpad-t430s:~/scm/crossbario/crossbarwww$ sudo docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
crossbario/autobahn-python   cpy3-minimal        43257bd83681        5 seconds ago       103 MB
crossbario/autobahn-python   cpy3-alpine         3d5f7049c4b9        12 seconds ago      279.5 MB
crossbario/autobahn-python   cpy2-alpine         58171b106e5f        3 minutes ago       274.8 MB
crossbario/autobahn-python   pypy2               85cafde22cec        6 minutes ago       766.4 MB
crossbario/autobahn-python   cpy3                faa30ce621dc        8 minutes ago       730.5 MB
crossbario/autobahn-python   cpy2                fc1b56ffeb12        11 minutes ago      726.3 MB
composetest_web              latest              63fc5d39e245        8 hours ago         683.8 MB
web                          latest              556b5fd02f20        8 hours ago         683.8 MB
crossbario/crossbar          latest              28702e991c54        10 hours ago        366.7 MB
crossbario/autobahn-cpp      latest              0fa36bef5bb9        18 hours ago        621.3 MB
<none>                       <none>              80378a3aa1e0        22 hours ago        188 MB
<none>                       <none>              475e80688483        30 hours ago        517.6 MB
<none>                       <none>              87d3cad05b0e        35 hours ago        366.9 MB
crossbario/autobahn-js       alpine-node-only    fe818df130ec        36 hours ago        24.55 MB
crossbario/autobahn-js       alpine              0ee0166ece64        36 hours ago        27.57 MB
crossbario/autobahn-js       latest              b712756d2374        36 hours ago        646.9 MB
<none>                       <none>              cabc9793b2a3        37 hours ago        279.5 MB
<none>                       <none>              888e5222470b        37 hours ago        726.3 MB
<none>                       <none>              316eab9effc6        37 hours ago        274.8 MB
<none>                       <none>              cd49fb36ab64        2 days ago          766.4 MB
<none>                       <none>              4e273e3d2fb5        2 days ago          730.4 MB
python                       3                   70c16d34e4c8        4 days ago          689.6 MB
python                       2                   e4a554df875e        4 days ago          676.8 MB
python                       2.7                 e4a554df875e        4 days ago          676.8 MB
pypy                         2                   d45ac503524a        4 days ago          725 MB
node                         latest              86cbce15c689        4 days ago          644.3 MB
python                       3-alpine            769e9328f383        10 days ago         89.74 MB
python                       2-alpine            dd22f748f304        10 days ago         72.59 MB
ubuntu                       latest              97434d46f197        10 days ago         188 MB
ubuntu                       trusty              97434d46f197        10 days ago         188 MB
redis                        latest              4f5f397d4b7c        3 weeks ago         177.6 MB
alpine                       latest              70c557e50ed6        3 weeks ago         4.798 MB
oberstet@thinkpad-t430s:~/scm/crossbario/crossbarwww$

There are a couple of things to note:

the size of the official Python 3 Alpine is already nearly 90MB
the size of Crossbar.io is roughly 400MB, which is "only" 4x that size
the size of Autobahn Alpine is 280MB in size - and this we can fix easily!

Because the latter includes all optional parts of Autobahn plus both Twisted and asyncio support. But we can have a "slim" variant with is CPython 3 on Alpine with Autobahn asyncio, and no optionals.
And this is: crossbario/autobahn-python:cpy3-minimal is 103MB!
Essentially, 100MB is the minimum size of running a Python container with a Autobahn client.
Now, we can compare that footprint to virtualenv. That is, we don't use Docker containers, but use virtualenvs for a different form (weaker) of isolation:
oberstet@thinkpad-t430s:~$ ./python351/bin/virtualenv ~/python351_4
Using base prefix '/home/oberstet/python351'
New python executable in /home/oberstet/python351_4/bin/python3.5
Also creating executable in /home/oberstet/python351_4/bin/python
Installing setuptools, pip, wheel...done.
oberstet@thinkpad-t430s:~$ source ~/python351_4/bin/activate
(python351_4) oberstet@thinkpad-t430s:~$ pip install autobahn[asyncio]
Collecting autobahn[asyncio]
Collecting txaio>=2.2.2 (from autobahn[asyncio])
Collecting six>=1.10.0 (from autobahn[asyncio])
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, txaio, autobahn
Successfully installed autobahn-0.13.0 six-1.10.0 txaio-2.2.2
You are using pip version 8.0.2, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(python351_4) oberstet@thinkpad-t430s:~$ du -hs ~/python351_4/
25M	/home/oberstet/python351_4/
(python351_4) oberstet@thinkpad-t430s:~$ pip install autobahn[twisted]
Requirement already satisfied (use --upgrade to upgrade): autobahn[twisted] in ./python351_4/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): txaio>=2.2.2 in ./python351_4/lib/python3.5/site-packages (from autobahn[twisted])
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in ./python351_4/lib/python3.5/site-packages (from autobahn[twisted])
Collecting Twisted>=15.5 (from autobahn[twisted])
Collecting zope.interface>=4.1.3 (from autobahn[twisted])
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./python351_4/lib/python3.5/site-packages (from zope.interface>=4.1.3->autobahn[twisted])
Installing collected packages: zope.interface, Twisted
Successfully installed Twisted-16.0.0 zope.interface-4.1.3
You are using pip version 8.0.2, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(python351_4) oberstet@thinkpad-t430s:~$ du -hs ~/python351_4/
42M	/home/oberstet/python351_4/
(python351_4) oberstet@thinkpad-t430s:~$ pip install autobahn[all]
Requirement already satisfied (use --upgrade to upgrade): autobahn[all] in ./python351_4/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in ./python351_4/lib/python3.5/site-packages (from autobahn[all])
Requirement already satisfied (use --upgrade to upgrade): txaio>=2.2.2 in ./python351_4/lib/python3.5/site-packages (from autobahn[all])
Collecting pynacl>=1.0 (from autobahn[all])
Collecting msgpack-python>=0.4.6 (from autobahn[all])
Collecting wsaccel>=0.6.2 (from autobahn[all])
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=4.1.3 in ./python351_4/lib/python3.5/site-packages (from autobahn[all])
Collecting pyqrcode>=1.1 (from autobahn[all])
Collecting ujson>=1.33 (from autobahn[all])
Collecting pytrie>=0.2 (from autobahn[all])
Requirement already satisfied (use --upgrade to upgrade): Twisted>=15.5 in ./python351_4/lib/python3.5/site-packages (from autobahn[all])
Collecting cbor>=0.1.24 (from autobahn[all])
Collecting cffi>=1.1.0 (from pynacl>=1.0->autobahn[all])
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./python351_4/lib/python3.5/site-packages (from zope.interface>=4.1.3->autobahn[all])
Collecting pycparser (from cffi>=1.1.0->pynacl>=1.0->autobahn[all])
Installing collected packages: pycparser, cffi, pynacl, msgpack-python, wsaccel, pyqrcode, ujson, pytrie, cbor
Successfully installed cbor-1.0.0 cffi-1.5.2 msgpack-python-0.4.7 pycparser-2.14 pynacl-1.0.1 pyqrcode-1.1.1 pytrie-0.2 ujson-1.35 wsaccel-0.6.2
You are using pip version 8.0.2, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(python351_4) oberstet@thinkpad-t430s:~$ du -hs ~/python351_4/
48M	/home/oberstet/python351_4/

As can be seen, the minimum is 25MB for Autobahn there. That is, microservices based on Autobahn and isolated via virtualenv can get down to 25MB footprint, while with OS containers, the minimum is 100MB. And Crossbar.io itself in a container is 400MB. I think this is a honest description of the current lower bounds.

Because of the above, we won't address this.


⚠ ECDC Shop has degraded performance

In ab50b7f, ECDC Shop (https://shop.memmingen-indians.de) experienced degraded performance:

HTTP code: 200
Response time: 6748 ms

Resolved: ECDC Shop performance has improved in f9b5299 after 22 minutes.


access to keys response

The consul docs suggest that the most recent ModifyIndex for a group of keys is returned as a header in the repsponse.
Is there a way to get access to the http request?

In the currently published version 0.40.0 the full response is available in the callback interface via the third parameter:
const consul = require('consul')();

consul.agent.self((err, body, res) => {
  console.log(res.headers);
});

It's not available in the promise interface.
The currently unreleased version is promise based and supports exposing the raw response, but the interface is not stable yet.

You can get access to this header in v1.0.0 using the following syntax:
const events = require('events');
const Consul = require('consul');

async function main() {
  const consul = new Consul();

  const ctx = new events.EventEmitter();
  ctx.includeResponse = true;

  const [res, data] = await consul.agent.self({ ctx });

  console.log(res.headers);
}

main();

Congratulations on 1.0.0
Can't wait to break it :D


feat(scripts): provide SLA for every existing versions

🧭 What and Why
🎟 JIRA Ticket: https://algolia.atlassian.net/browse/DI-2702
Changes included:
update the release PR script to generate the SLA of every existing versions that fit in the current 2 year range, which allows us to provide an exhaustive list to the customer and also one source of truth (this repo), that keeps track of legacy clients, and current ones


Baseline Core V1.0 links don't exist?

In this file in the section "[baseline-core-v1.0]" two provided links are broken. (404 not found)
Actually the link to the baseline project in the oasis website doesn't exist. Is this a placeholder? https://docs.oasis-open.org/baseline

@mehranshakeri - Good point - we should update this section to clearly indicate that it is a placeholder.

Thanks. Then I will close this issue.


Network Policy Manager merges rules incorrectly

Is this a request for help?:
No

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

Which release version?:
1.0.28

Which component (CNI/IPAM/CNM/CNS):
NPM

Which Operating System (Linux/Windows):
Linux

For Linux: Include Distro and kernel version using "uname -a"
Linux aks-agentpool-34239724-1 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

For windows: provide output of "$(Get-ItemProperty -Path "C:\windows\system32\hal.dll").VersionInfo.FileVersion"

Which Orchestrator and version (e.g. Kubernetes, Docker)
Kubernetes

What happened:
Network policy rules are merged in an unexpected way. Setting up two rules, e.g. egress to ip address <IP_ADDRESS> on port 80 and egress to <IP_ADDRESS> on port 443 results in a combination of the two rules being created. The rule applied will allow access to <IP_ADDRESS> on port 80 and 443 and similarly allow access to <IP_ADDRESS> on port 80 and 443

What you expected to happen:
The rules should operate independently. Allowing access to a particular port for one destination should not allow access to that port for all destinations.

How to reproduce it (as minimally and precisely as possible):
Create a network policy with two egress destinations, e.g.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: firewall-test
spec:
  policyTypes:
  - Egress
  podSelector:
    matchLabels:
      app: firewall-test
  egress:
  - to:
    - ipBlock:
        cidr: <IP_ADDRESS>/32
    ports:
      - protocol: TCP
        port: 80
  - to:
    - ipBlock:
        cidr: <IP_ADDRESS>/32
    ports:
      - port: 443
        protocol: TCP

(Or create two separate network policies - the behaviour will be the same.)
Run a pod with the label firewall-test that you can try out curl commands in. e.g.
kubectl run -l app=firewall-test -it --image=byrnedo/alpine-curl curl --command sh

From within the pod confirm that you can access both ip addresses specified in the rule on both ports.
curl -kv --resolve "xkcd.com:443:<IP_ADDRESS>" https://xkcd.com
curl -kv --resolve "xkcd.com:80:<IP_ADDRESS>" http://xkcd.com
curl -kv --resolve "xkcd.com:443:<IP_ADDRESS>" https://xkcd.com
curl -kv --resolve "xkcd.com:80:<IP_ADDRESS>" http://xkcd.com

The expected behaviour would be for two of those commands to fail but currently they all succeed.

Anything else we need to know:
I suspect that the problem is generic across both ingress and egress and that a similar problem exists for ingress rules, such that allowing access to a port from one source will allow access to that port from any other source specified.

fix here https://github.com/Azure/azure-container-networking/pull/456

that PR has merged and the changes will be in release v1.0.31


Add tag discovery support for AWS Lambda

Awesome thanks for all your work :) That makes my day!
Will merge, release this tomorrow


[WIP] feat(CST-2526): add loaded key check to health check

PR Description
Add loaded keys check to health check endpoint,
that way when loading 0 keys health check would fail
and would allow Kubernetes to restart the pod
Documentation

[x] I thought about documentation and added the doc-change-required label to this PR if updates are required.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Juan Dominguez seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Parallelize prepare_data.py

When I initially ran prepare_data.py on my folder of ~500 PDFs, it took 31:49, which is concerning when considering potentially much larger data sets.
Since each file is essentially being processed in isolation, this should be parallelizable with basically no problems. This PR splits up the filenames so that they are processed at the same time over n cores. Through empirical results, it seems as if prepare_data.py requires roughly ~1.5-2 cores of processing power for any given filename. That means that n cannot equal the number of logical cores and the overhead in python's interprocess communications (IPC) causes a significant slow down (I had gotten 4x slowdown on my machine).
Therefore, I'm calculating n as n = (# of cores - 2) / 2. Basically dedicate all cores in your machine to the task (as two cores correspond to one filename) and reserve two for the system / Python. Note that n can't go below 1. I haven't been able to get decent results by ever setting n to be more than this value.
Results
As mentioned before, when I ran the current prepare_data.py on ~500 PDFs, it took 31:49. When I ran the parallelized version in this PR on my computer (Intel i7 6700k--4 physical cores, 8 logical cores) and n=3, it finished in 9:57. Therefore, I had achieved a ~3x speed up on my machine. I'm sure that better results could be achieved with newer processors / more cores.

Brilliant work, thanks a lot for contributing! Just a small fix but LGTM otherwise.

Brilliant work, thanks a lot for contributing! Just a small fix but LGTM otherwise.


[Identity] Make breaking changes from API Review feedback

We have completed the API review for the Identity library across the various languages, notes are here: https://github.com/Azure/azure-sdk/issues/696#issuecomment-543224656
Relevant Notes
JavaScript:

Change underscored property names to camel case
Remove const AggregateAuthenticationErrorName
clientSecret issue - use constructor overloads
getToken needs to be updated to take a TokenRequestContext
Always use tenant_id, client_id and move to options bag when optional
For ManagedIdentityClient, make the constructor an overload so we can doc properly

Tasks

[X] (DECIDED: Do not rename) Rename DefaultAzureCredential to DefaultCredential
[x] Rename DeviceCodeDetails to DeviceCodeInfo #5662 @richardpark-msft
[x] Change ErrorResponse (errors.ts) fields to be camelCase, not snake_case #5662 @richardpark-msft
[ ] ~getToken should use TokenRequestContext interface as first parameter to take scopes and parent request ID #5642~ (This isn't needed for TypeScript library, we already have GetTokenOptions)
[x] Create constructor overloads for parameters which should be used together for a specific use case
[x] Move purely optional constructor parameters to the options type for the constructor @richardpark-msft

Non-breaking changes for later

[ ] Add DefaultAzureCredentialOptions to mirror what is done in .NET

Create our own interface type so users can control DefaultAzureCredential (just match a similar naming to .NET)

[ ] Move AggregateAuthenticationError logic up to DefaultAzureCredential, don't catch in ChainedTokenCredential (check .NET implementation)

Not sure if this makes in TS. Let's double-check what .NET does, we need to at least capture what happened. We could potentially pass in an option to ChainedTokenCredential (aka ThrowOnNoAuthentication, eg)

Moved the non breaking changes above to #5704
The constructor changes are done in https://github.com/Azure/azure-sdk-for-js/pull/5668


Broken characters when it's in spanish

Every character that has an accent on load are broken.
Example:
original: ECONOMÍA
on edition mode: ECONOMÃA.
í = Ã
á = Ã¡
ñ= Ã±
*Original text are in spanish

@igdamico I'm not able to repeat this issue using the editor, are you able to repeat it on the demo page: http://getcontenttools.com/demo?
To test I basically inserted the 'ECONOMÍA' content into the page (using the inspector) then clicked the edit button - all worked as expected. This looks like an encoding issue but without more detail or access to an example it's difficult to pinpoint - though at this point I don't think it's an issue related to CT (happy to be proved wrong though).

I've also just tested in a live site that I can edit content with the characters you've mentioned and I don't see the same issue.

Hi anthonyjb, thanks for the very fast answer.
in demo works but it's not the same because i just had text with accents in my "data-editable" div before using the editor.

this is my html that works perfect without the editor.
https://jsfiddle.net/0bac3nfb/

don't worry! i add in the header
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />

and now it's working!
thanks fo the advice!


Some question about the paper

Hi,
Thank you for sharing your excellent work! I have read the paper recently, and I have some questions about the position encoding. In your paper, you add position encoding to three places: self-attention, cross-attention, confidence matrix. And as you mention in the paper, position encoding in cross-attention and matching may mislead the registration at the beginning. So, I wonder if you have compared the performance at the situation you remove the position encoding at the cross-attention part and the confidence matrix part at the first stage but retain them at the second stage. Plus, I think the figures in your paper are very beautiful. So could you please tell me how you draw the point cloud and correspondences of figure 7 in your paper and the point cloud video in the readme? I will be very appreciative if you can give me some hints about the above question. Thank you very much!

We did not check what happens if remove them, but agree it worth trying. For the drawing, we transfer all points/lines to meshes and render them with blender's cycle engine.

ntion part and the confidence matrix part at the first stage but retain them at the second stage. Plus, I think the figures in your paper are very beautiful. So could you please tell me how you draw the point cloud and correspondences of figure 7 in your paper and the point cloud video in the readme? I will be very appreciative if you can give me some hints about the above qu

Sir, It is an excellent idea !  Did you do this experiment?  will the result increase or decrease?

ntion part and the confidence matrix part at the first stage but retain them at the second stage. Plus, I think the figures in your paper are very beautiful. So could you please tell me how you draw the point cloud and correspondences of figure 7 in your paper and the point cloud video in the readme? I will be very appreciative if you can give me some hints about the above qu

Sir, It is an excellent idea ! Did you do this experiment? will the result increase or decrease?

Sorry, I didn't. I am a little busy these days :(


Declarations (method/class/property/namespace/etc) must not contain embedded comments

I would like to propose a new diagnostic:
Category: ReadabilityRules
Rule: Declarations must not contain embedded comments
Description: When declaring a class/method/namespace/property, we shouldn't allow to put a comment between the declaration and the opening curly bracket
Rationale: In #42 we catch situations where a commment is put between a statement and the opening curly bracket. I think the same rule should apply to declarations. For example, code above should not be allowed:
public class IndexTests
//aa
{
    public int Prop
    //aa
    {
        get
        //bb
        {
            return 0;
        }

        set
        //cc
        {

        }
    }
  }

I am in favor of this, but I'm not sure how we're planning on assigning IDs to new diagnostics. For my proposal, I implemented it using the next available number in that category.
What should our policy/procedure be regarding additions?

I'd follow your pattern and choose the next number available. For this one, it'd be SA1127.

Choosing the next number sounds fine. Maybe we should mark these as SX category as well, since they are not original StyleCop?

@vweijsters Originally I was thinking of SX rules as conflicting in some way with the SA rules. It's easy to see that this rule wouldn't conflict with any. It's risky using an SAxxxx identifier, but it seems no new rules have been defined in at least 3 years now, so perhaps we are safe?
Another option is to initially implement it as an SX rule (or some one other designation indicating a preliminary or trial rule), and then promote it to an SA rule after it's been vetted/used in practice by a wider audience.

@sharwell I like the option of using SX for new rules, where the X means experimental stuff, including some form of promoting mechanism.
The alternatives proposal (#600) could be also be done by appending a letter code to the rule, for example: SA1100 for the original one, SA1100a for the first alternative, SA1100b for a second alternative, etc.

I am not an contributor here, I am big StyleCop fan and I used to
contribute lots of issues to original SC. So here are my thoughts:
I like SX for "alternative" rules. The opinionated nature of original SC
was its biggest weakness IMHO.
But I would not bother with new rules being introduced as SX and then
promoted to SA. First - original SC is effectively dead. I don't know what
happened to andyr and I wish him all the best. Unfortunately, it looks like
bus factor hit original SC project. As a result - there will be no
conflicts with your work and original SC. Second - changing identifier
later will be breaking change - e.g. suppressions with #pragma warning
disable SX1234 will stop working if you will change SX1234 to SA1234. Third

what will you do with new rule which comes with alternative flavour as
well? For these reasons, for new rules, I would just go with SA followed by
next available number in category. Of course, new rules should be as much
grouped as possible - e.g. possible new rules for ternary operator should
be added together and get adjacent numbers.

Keep up the good work guys.

@Przemyslaw-W Thanks for your feedback.

I am not an contributor here

All it takes to be a contributor here is providing feedback. :+1:

For these reasons, for new rules, I would just go with SA followed by next available number in category.

Everything you wrote makes perfect sense. Since the primary reason for a "trial" identifier and "promotion" would be making sure more people give feedback before finalizing a rule, how would you feel about keeping a trial period, but adding a statement that a rule will be promoted to its final form before it gets included in a stable release (e.g. 1.0.0 is stable but 1.0 Alpha 5 is a preview release). More specifically:
[Proposed Rule:] The identifier for a diagnostic cannot change after the diagnostic is enabled by default in a stable release.
Even if we assign SA rules from the start for new diagnostics, I think the above rule is a good idea to document (and of course follow).

I think trial identifiers have merit when new rule comes with two flavours.
So the community can help to decide which one goes with SA and which with
SX. If there is only one flavour then the only decision to make is whether
to enable the rule by default. And identifier is not important in such case

so go with SA by default.

I also would not bother with SA1234a, SA1234b, SA1234c. SC rules are
boolean by its nature. More complex styles can be build by
enabling/disabling proper set of yes/no rules. So SA and SX are enough IMHO.
And another thing - you guys start from scratch so you can "deviate" from
the original SC rules. E.g. there was common complaint for SA1513 - it was
reported for closing } in get and add accessor. Do you consider splitting
original rules into few? here I would see SA1513 splitted into three:

closing } for type or member declaration (class, struct, interface,
method, property, event)
closing } for accessor (get, set, add, remove)
closing } for code block (if, else, while, for, do, try, catch,
finally etc)

Similar approach for rule which triggered this thread:

comments inside type or member declaration
comments inside accessor declaration
comments inside block declaration

Basically, I think more granurality is needed. I don't thing separate rule
for method, property, etc. is necessary. But above 3 groups would be more
than welcome. Maybe here the "a", 'b", "c" flavours could be applied:

SA1513a - need blank line after closing } in type or member
declaration (or SA1513m - m like member)
SX1513a - no blank line after closing } in member declaration (or
SX1513m - m like member)
SA1513b - need blank line after closing } in accessor declaration (or
SA1513a - a like accessor)
SX1513b - no blank line after closing } in accessor declaration (or
SX1513a - a like accessor)
SA1513c - need blank line after closing } in code block (or SA1513b -
b like block)
SX1513c - no blank line after closing } in code block (or SX1513b - b
like block)

If such scheme would be introduced, then SA1513a rule would be suppressed
in code by both - "#pragma warning disable SA1513" and "#pragma warning
disable SA1513a"
What do you think?
2015-04-03 23:53 GMT+02:00 Sam Harwell<EMAIL_ADDRESS>
@Przemyslaw-W https://github.com/Przemyslaw-W Thanks for your feedback.
I am not an contributor here
All it takes to be a contributor here is providing feedback. [image: :+1:]
For these reasons, for new rules, I would just go with SA followed by next
available number in category.
Everything you wrote makes perfect sense. Since the primary reason for a
"trial" identifier and "promotion" would be making sure more people give
feedback before finalizing a rule, how would you feel about keeping a trial
period, but adding a statement that a rule will be promoted to its final
form before it gets included in a stable release (e.g. 1.0.0 is stable
but 1.0 Alpha 5 is a preview release). More specifically:
[Proposed Rule:] The identifier for a diagnostic cannot change after the
diagnostic is enabled by default in a stable release.
Even if we assign SA rules from the start for new diagnostics, I think the
above rule is a good idea to document (and of course follow).
—
Reply to this email directly or view it on GitHub
https://github.com/DotNetAnalyzers/StyleCopAnalyzers/issues/605#issuecomment-89431572
.

I like the proposal for the new comments rule


Running co-compiled models on one USB TPU

Hi,
maybe I am not seeing the obvious but nowhere in the doc for co-compiling it is stated on how to run those n models on the same TPU, or is it ?
https://coral.ai/docs/edgetpu/compiler/#co-compiling-multiple-models
https://coral.ai/docs/edgetpu/multiple-edgetpu/#using-the-tensorflow-lite-python-api
The thing is that I tried to specify the same device for both models like this:
interpreter_1 = Interpreter(model_1_path,
experimental_delegates=[load_delegate('libedgetpu.so.1', {"device": "usb:0"})])
interpreter_2 = Interpreter(model_2_path,
 experimental_delegates=[load_delegate('libedgetpu.so.1', {"device": "usb:0"})])
But it is failing. Should I leave out the whole device string part ?
Any help much appreciated.
Alexander

@alexanderfrey it really should works, does it fails with some error messages? Although if you;re just using the same tpu, there isn't a need to specify the {"device": "usb:0"} part. Here is how I'd do it on my dev board:
import tflite_runtime.interpreter as tflite
import numpy as np

model_1_path = 'mobilenet_v2_1.0_224_quant_edgetpu.tflite'
model_2_path = 'ssd_mobilenet_v2_face_quant_postprocess_edgetpu.tflite'

interpreter_1 = tflite.Interpreter(model_1_path,
  experimental_delegates=[tflite.load_delegate('libedgetpu.so.1', {"device": "pci:0"})])
interpreter_2 = tflite.Interpreter(model_2_path,
        experimental_delegates=[tflite.load_delegate('libedgetpu.so.1', {"device": "pci:0"})])

def set_random_inputs(interpreter):
  input_details = interpreter.get_input_details()
  image = np.random.random(input_details[0]['shape']).astype(input_details[0]['dtype'])
  interpreter.set_tensor(input_details[0]['index'], image)

interpreter_1.allocate_tensors()
interpreter_2.allocate_tensors()

for i in range(10):
    set_random_inputs(interpreter_1)
    interpreter_1.invoke()
    set_random_inputs(interpreter_2)
    interpreter_1.invoke()

Thanks a lot - yes leaving out the device part seems to do the trick.
Best,Alexander
ps Really looking forward to the new software release and any updates in general !!

Expect it next week or week after :)
There's going to be quite changes, particularly you'll get to use a newer version of tensorflow, which I feel would benefits many.


When creating an Independent EFModel Class library/Project, its missing settings in context & EF core migrations fail.

Been reading your docs a lot!! :)
I created an independent class lib in EF 6 (to keep this as clean as possible), and refer to the models from other projects.

this picture shows me, that I failed to add migrations. I assume its because there is a missing connectionstring in the DBcontext

Question 1: How do I get the EF designer 2k22 code gen to pick up the connection string?, so it can be scaffolded into the DBcontext. And, where does it pick it up from, I even tried to add a app.config file with a new connection string, but its not picking it up... should I be doing this differently?

Question 2: From the model generated, to create a new database -> am I correct in creating a new migration, using that I would create my Database?  or how do I create the DB from the EF Designer Models?

You can add the connection string as a property of the designer surface.

Click the design surface and these will appear in the property window. Note you can add either a connection string or a connection string name (with the connection string being stored in a config file somewhere else) and EFCore will be able to use it.

Generating the model gets you the entity classes and a DbContext -- from there you can create the migrations, use the context, etc. The EFCore docs talk about how the migrations can automatically create a database, how to update that structure, etc.

thanks


Make autocomplete available on macOS

The local autocomplete provider should be available on macOS as well.

This is available in 7.2.


Automated cherry pick of #1244: fix clean sched cache

Cherry pick of #1244 on release/2.10.0.
#1244: fix clean sched cache

/lgtm
/approve

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.10.0@989cc41). Click here to learn what that means.
The diff coverage is 0%.

@@               Coverage Diff                @@
##             release/2.10.0   #1245   +/-   ##
================================================
  Coverage                  ?   7.93%
================================================
  Files                     ?     367
  Lines                     ?   57354
  Branches                  ?       0
================================================
  Hits                      ?    4551
  Misses                    ?   52405
  Partials                  ?     398

Flag
Coverage Δ

#aFlag
7.93% <0%> (?)

Impacted Files
Coverage Δ

pkg/cloudcommon/db/sharablevirtual.go
0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 989cc41...f259507. Read the comment docs.


fix: add build script to intranet-header package

Preview environments are getting posted here, as soon as they are ready!


White space

Some animations gives a blank white space when fired.
eg. bounceIn, fadeInLeft

Same problem here on Safari, actually it's visible on my testingarea at studiotoia.testingarea.it.
It generates white space at the end after firing wow+animate.css

Fix your markup, you've got strayed tags.

I fixed all error caused by closing tags but problem still exist

Still, this is probably an animate.css issue. You'll have to isolate the problem to try to discover the real problem. Maybe it is a browser bug, but it doesn't look to be related to animate.css at all.

I tried to remove section by section but problem is not markup or else, it's an animate.css or wow.js problem, nothing else :)

Same issue. On tablets the page leaves a white space on the right. It only occurs if you have more than one slide, so it is related to the animation.

Same here, I'm pretty sure that is a animate.css problem.

Hope this helps [http://stackoverflow.com/questions/31512366/slider-with-whitespace-appearing-on-right-of-smaller-screen]


Fix bug: can't show the role of member

Coverage increased (+0.009%) to 53.978% when pulling 579a379e0884aefd5395527ad52f9d7bad4829ae on ywk253100:170622_project into b5cbba020694e39f437ceaf6988814076f382a28 on vmware:master.

Coverage increased (+0.01%) to 49.837% when pulling 419cf8dfc32a917db0680d8b32d6e5b714bbd354 on ywk253100:170622_project into 6e0f274c072d1b32e61f6dd0845172cb8d93774d on vmware:master.


Remove domain model in use by filter

Codecov Report

Merging #187 (2a2849a) into main (b3b3e9d) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

:exclamation: Current head 2a2849a differs from pull request most recent head 8c3a2ba. Consider uploading reports for the commit 8c3a2ba to get more accurate results

@@            Coverage Diff             @@
##             main     #187      +/-   ##
==========================================
- Coverage   80.15%   80.13%   -0.02%
==========================================
  Files          89       89
  Lines        5456     5453       -3
  Branches      981      980       -1
==========================================
- Hits         4373     4370       -3
  Misses        923      923
  Partials      160      160

Impacted Files
Coverage Δ

orchestrator/services/subscriptions.py
61.74% <0.00%> (-0.62%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Remove patch version check for plugin compatibility

Signed-off-by: Abhilasha Seth<EMAIL_ADDRESS>Description
As a first step towards supporting semVer range of compatible versions for plugins, we are starting with relaxing the patch version check (for default compatibility range).
Issues Resolved
https://github.com/opensearch-project/OpenSearch/issues/1707
Check List

[ ] New functionality includes testing.

[X] New tests pass

[ ] New functionality has been documented.

[X] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #6837 (2d82f13) into main (4d78bbf) will increase coverage by 0.03%.
The diff coverage is 50.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #6837      +/-   ##
============================================
+ Coverage     70.65%   70.68%   +0.03%
+ Complexity    59184    59163      -21
============================================
  Files          4810     4810
  Lines        283487   283488       +1
  Branches      40877    40877
============================================
+ Hits         200299   200397      +98
+ Misses        66761    66651     -110
- Partials      16427    16440      +13

Impacted Files
Coverage Δ

...in/java/org/opensearch/plugins/PluginsService.java
85.04% <50.00%> (-0.24%)
:arrow_down:

... and 492 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

We need to discuss whether relaxing plugin compatibility versions is a breaking change. @saratvemulapalli wdyt?

We need to discuss whether relaxing plugin compatibility versions is a breaking change. If it is, then we need this behavior to be configurable and default to false.
@saratvemulapalli wdyt?

Good question, looking at Semver[1] MINOR version when you add functionality in a backwards compatible manner.
This feature is backward compatible since we are relaxing patch version, our installation APIs will not break.
That said, for developers who are using this feature we really want to "caution" that its only for users "who know what they are doing". Because there are risks associated with relaxing patch versions and might up into runtime problems.
Gating it by a feature flag would help lower the risk.
@abseth-amzn we could add another parameter plugin-descriptor.properties[1] file and enable devs to configure this feature and set it false by default if it doesn't exist and open an issue in https://github.com/opensearch-project/documentation-website/ to document details of this feature.
[1] https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/src/main/resources/plugin-descriptor.properties

@dblock @saratvemulapalli I think we could not relax the versioning check in one directional way, the plugin should have a way to set the OpenSearch core baseline it need to works with. There are at least 2 examples why its needed:

bug that is present in one version and prevents plugin from functioning properly (example, 2.6.1 - OK,  2.6.2 - BUG, 2.6.3 - BUG FIXED), they way plugin developers could circumvent it right now - just skip 2.6.2 plugin release altogether
new plugin interface / method introduced (in a fully backward compatible fashion, non-breaking change in patch version, examples are CodecServiceProvider) - the plugin just won't with any previous version of core

bug that is present in one version and prevents plugin from functioning properly (example, 2.6.1 - OK, 2.6.2 - BUG, 2.6.3 - BUG FIXED), the way plugin developers could circumvent it right now - just skip 2.6.2 plugin release altogether

Thanks @reta, I hope this should answer your question: https://github.com/opensearch-project/OpenSearch/issues/1707#issuecomment-1459207057
Plugin able to define range of patch version compatibility i.e >2.5.1, <2.5.7, for some patch version upgrades on OpenSearch few plugins are not impacted and this feature enables them to completely ignore the check and not ship a new version.

new plugin interface / method introduced (in a fully backward compatible fashion, non-breaking change in patch version, examples are CodecServiceProvider) - the plugin just won't work with any previous version of core

Absolutely, if the plugin chooses to use the new interface it will bump its own patch version and will define it minimum compatibility version of OpenSearch when the new interface is added.
Would that solve the problem? Or am I missing something here.
Also this made me think, assuming the plugin shipped with 2.7.0 of OpenSearch defining it compatibility >2.7.
When a new patch version of OpenSearch releases 2.7.2, which doesnt work with plugin <IP_ADDRESS>. How does the plugin tell customers that it doesn't work with 2.7.2 since its already shipped and doesn't know about the future releases.

Would that solve the problem? Or am I missing something here.

Thanks @saratvemulapalli , the discussion are addressing the concerns, but I don't see that reflected in the pull request

How does the plugin tell customers that it doesn't work with 2.7.2 since its already shipped and doesn't know about the future releases.

This is somewhat tricky, but the suggestion from the comment you referred to (https://github.com/opensearch-project/OpenSearch/issues/1707#issuecomment-1459207057) gives the hint: the plugin developers could not modify the released plugins, but they do have a choice to keep lockstep releases (basically, just limiting to exact release, >= 2.6.1, < 2.6.2)

I think we're all in agreement of what we would like in this PR.

New feature: plugins have a way to explicitly define a semver version range.
Server will match the semver version range defined by the plugin when available.
No changes to existing plugins that don't want this feature.

DCO needs fixing please, and make sure gradle checks pass, will review when 🟢 again

Re-opening this PR. This is a needed change.

I'm opposed to this change as is (a boolean that says that a plugin is compatible with patch versions of OpenSearch).

@abseth-amzn Please address my comment.

Closing in favor of https://github.com/opensearch-project/OpenSearch/pull/11441


DoRA with fsdp model cannot work

System Info

peft 0.12.0
transformers version: 4.43.3
Platform: Linux-5.15.0-113-generic-x86_64-with-glibc2.35
Python version: 3.12.4
Huggingface_hub version: 0.24.2
Safetensors version: 0.4.3
Accelerate version: 0.33.0
Accelerate config: 	- compute_environment: LOCAL_MACHINE

distributed_type: MULTI_GPU
mixed_precision: fp16
use_cpu: False
debug: False
num_processes: 8
machine_rank: 0
num_machines: 1
rdzv_backend: static
same_network: False
main_training_function: main
enable_cpu_affinity: False
downcast_bf16: False
tpu_use_cluster: False
tpu_use_sudo: False

PyTorch version (GPU?): 2.4.0 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?:
Using GPU in script?:
GPU type: NVIDIA A800-SXM4-80GB

Who can help?
@BenjaminBossan @sayakpaul
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder
[ ] My own task or dataset (give details below)

Reproduction
hugginface trainer with fsdp enabled, with dora peft adapter
Expected behavior
work fine

I could successfully train DoRA with FSDP in the past, see #1806. Please provide the full error message and, ideally, a full reproducer, so that we can investigate.

I am also interested in using DoRA with Peft for inference.
DoRA with deepspeed at inference time has some issues where the adapter layers are not being properly initialized. Reinitializing the model seems to help but negates the benefits of using adapters

I also had an issue with bf16 weights + fsdp + dora
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 572, in forward
[rank0]:     result = result + self._apply_dora(x, lora_A, lora_B, scaling, active_adapter)
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 265, in _apply_dora
[rank0]:     lora_weight = lora_B.weight @ lora_A.weight
[rank0]:                   ^
[rank0]: RuntimeError: inconsistent tensor size, expected tensor [104858] and src [131072] to have the same number of elements, but got 104858 and 131072 elements respectively

I use stanford alpaca for training
torchrun --nproc_per_node=4 --master_port=<your_random_port> train.py \
    --model_name_or_path <your_path_to_hf_converted_llama_ckpt_and_tokenizer> \
    --data_path ./alpaca_data.json \
    --bf16 True \
    --output_dir <your_output_dir> \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2000 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True

with dora modification
        from peft import LoraConfig, get_peft_model

        loraconfig = LoraConfig(
            r=64,
            lora_alpha=16,
            lora_dropout=0.1,
            use_dora=True,
            bias="none",
            task_type="CAUSAL_LM",
            target_modules="q_proj,k_proj,v_proj,o_proj,down_proj,up_proj,gate_proj".split(","),
        )
        model = get_peft_model(model, loraconfig)
        model.print_trainable_parameters()

my error is similar with @winglian

BTW, anyone knows how to generate config parameter in AdaptionPromptModel method from peft llama-adapter? The doc seems to be very confusing.

Thanks for providing more details. I haven't tested bf16 yet. I'll take a closer look and report back.

Okay, so I managed to successfully train a DoRA model with FSDP and bfloat16. The training script is here:
https://github.com/BenjaminBossan/peft-experiments/blob/main/torchtune/llama3-lora-fsdp/train.py
Run it like so:
USE_DORA=1 torchrun --nproc_per_node <N> train.py
Note that this script uses some torchtune functionality, as I'm currently working on a comparison between torchtune and PEFT and wanted to match torchtune as closely as possible. However, the model itself is a transformers+PEFT model, not a torchtune model, so this should not make any difference. The dataset is a cleaned version of Alpaca.
You will have to adjust these lines to load the Llama3 tokenizer:
https://github.com/BenjaminBossan/peft-experiments/blob/3b006cb67231d04a8daaaabc340dce2d276c79d7/torchtune/llama3-lora-fsdp/train.py#L32-L33
Regarding the used versions, I have the latest PEFT, transformers, and torchtune (i.e. installed from main) and PyTorch 2.3.1.
LMK if you manage, or don't manage, to get the script to run.

I also tried this with qdora, and I'm getting the same error, my initial report was for full weights dora+fsdp.

my initial report was for full weights dora+fsdp

Note that the provided script is also full weights (non-quantized).

Are you able to reproduce the error on your end with accelerate + hf trainer + fsdp?

@winglian I hacked something together with Trainer (actually SFTTrainer) + FSDP + DoRA and it works. Note that this is a very rough script and needs improvement for proper training, but it shows that it can work:
# Using trl Trainer and accelerate (trl-0.9.6, accelerate-0.33.0)
# accelerate launch --config_file fsdp_config.yaml train-hf.py
#
# for DoRA:
# USE_DORA=1 accelerate launch --config_file fsdp_config.yaml train-hf.py

import os

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

def train():
    #########
    # SETUP #
    #########

    torch.cuda.manual_seed(0)
    # TODO: template not applied
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token

    ds = load_dataset(ds_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        torch_dtype=dtype,
    )
    lora_config = LoraConfig(
        r=rank,
        lora_alpha=alpha,
        target_modules=target_modules,
        lora_dropout=dropout,
        use_dora=use_dora,
    )
    training_args = SFTConfig(
        output_dir=output_dir,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size,
        gradient_accumulation_steps=gradient_accumulation_steps,
        num_train_epochs=total_epochs,
        save_strategy="no",
        bf16=True,
        learning_rate=lr,
        weight_decay=weight_decay,
        lr_scheduler_type="cosine",
        warmup_steps=num_warmup_steps,
        dataset_text_field="input",
        max_seq_length=max_seq_len,
    )

    trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        args=training_args,
        train_dataset=ds["train"],
        peft_config=lora_config,
        packing=False,
    )
    trainer.train()

if __name__ == "__main__":
    ##############
    # PARAMETERS #
    ##############

    model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
    dtype = torch.bfloat16
    weight_decay = 0.01
    lr = 0.0003
    gradient_accumulation_steps = 4
    num_warmup_steps = 100

    # data
    ds_name = "yahma/alpaca-cleaned"
    max_seq_len = 512
    total_epochs = 1
    batch_size = 2
    # max_steps_per_epoch = None
    output_dir = "/tmp/peft/sft"

    # LoRA
    rank = 8
    alpha = 16
    target_modules = ["q_proj", "v_proj"]
    dropout = 0.05
    use_dora = bool(int(os.environ.get("USE_DORA", "0")))
    if use_dora:
        print("USING DORA")

    train()

FSDP config:
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: false
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: false
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

I uploaded an adjusted training script for DoRA + FSDP here: https://github.com/BenjaminBossan/peft-experiments/tree/main/torchtune/llama3-lora-fsdp#using-trainer-and-accelerate
It uses Trainer and accelerate under the hood and works. I haven't done a long run yet to compare the training progress.

@BenjaminBossan I tried your script above on latest releases of bnb and transformers and as well as tried both the latest release 0.32.0 and main on accelerate, w torch==2.3.1 and I get:
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/fsdp/_init_utils.py", line 610, in _init_param_handle_from_params
[rank1]:     handle = FlatParamHandle(
[rank1]:              ^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/fsdp/_flat_param.py", line 582, in __init__
[rank1]:     self._init_flat_param_and_metadata(
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/fsdp/_flat_param.py", line 632, in _init_flat_param_and_metadata
[rank1]:     ) = self._validate_tensors_to_flatten(params)
[rank1]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/fsdp/_flat_param.py", line 779, in _validate_tensors_to_flatten
[rank1]:     raise ValueError(
[rank1]: ValueError: Must flatten tensors with uniform `requires_grad` when `use_orig_params=False`

looks like I needed to update peft. all looks good to me now!

I also had an issue with bf16 weights + fsdp + dora
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/peft/tuners/lora/dora.py", line 74, in forward
[rank0]:     x_eye = torch.eye(lora_A.weight.shape[1], device=lora_A.weight.device, dtype=x.dtype)
[rank0]:                       ~~~~~~~~~~~~~~~~~~~^^^
[rank0]: IndexError: tuple index out of range

I also encountered the same error (peft 0.12.0), did you solve it by upgrading peft? or using the new scripts provided by BenjaminBossan ?


Added TeaFiles and Emmy

Added TeaFiles and Emmy

Awesome to see Nim repos in the @unicredit org :)

Not the first one, either ;-)


JSON parsing error with Docker 1.7.0

I get the following error when using Clocker and trying to create a container using the Docker REST API with Docker 1.7.0 and Calico 0.4.8 installed. There's no extra configuration at this point beyond installing Docker and the calico-node container.
Caused by: org.jclouds.http.HttpResponseException:
command: POST http://<IP_ADDRESS>:2377/v1.16/containers/create?name=calico_05 HTTP/1.1 failed with response:
HTTP/1.1 500 Internal Server Error; content: [json: cannot unmarshal object into Go value of type []runconfig.DeviceMapping

I have worked around this by rolling back my Docker deployment to 1.6.2 for the moment, but it seems to be straightforwardly reproduceable in my environment.

Are you able to repro this without calico installed?
Could you create a diags bundle and send it to us? (either using calicoctl diags, or just zipping up /var/log/calico)

Hi Andrew,
Just a quick follow up - did you manage to gather diags as Tom mentioned above?
One thing to note, the most recent release of the calico-docker images uses libnetwork rather than powerstrip (this currently requires use of the experimental Docker release).  See our docs for details.  We've archived off the powerstrip branch.  Would it be worth retrying using the libnetwork release?

Closing as we can't fix without additional diags and this in on an old version of calico. Please reopen if you're still hitting this.


Please add TextFormField Text Alignment

intl_phone_number_input Version: 0.5.2+2
Flutter version:  1.23.0-18.1
Please add these two parameters from TextFormField:
textAlignVertical:
textAlign:

in current version, if I add surfix or prefix to inputDecoration, text and hintText possition will be moved up a little. This bug is also happens to TextFormField if these two params are not set.

#139


Issue: Layer not created for existing CSS

Hey Artf, as always, you've done an incredible job with GrapesJS and I was very pleasantly surprised to notice this plugin in the demo. There is but one issue I have found so far. The plugin won't create a background layer for existing background information within the CSS. There are three ways I currently know of to replicate this event.

I have noticed the background color property under Typography has been removed in the demo, however, this is a good place to test it. If you were to apply a background color through these properties, the plugin will not create a layer in its background property.

If a template (that was created before this plugin was added) is loaded in GrapesJS, any backgrounds that were set are unable to be edited as no background layer for them exists.

Finally, if a background layer is added and set, and you were to click the Import on the Import Template modal, the layer is also removed.

Likely, this is the result of this layer information not being present in the components or style objects created by GrapesJS, however, would it be possible to have this layer be created automatically if any of those 3 situations occur?
Thank you very much for all you have done.

+1 - any updates?

Not to my knowledge yet. I may have to add this to the GrapesJS github instead of it being isolated here

I published a new release of the plugin to support the new Style Manager API and from this demo you should be able to see how the editor loads initial HTML/CSS and all the background styles are properly shown


Allow optional preprocessing

This allows pre-processing to only be run when it is needed.  This eliminates a lot of extra work done by the workers, but also makes setup a lot more graceful, and errors cause fewer resources to be left around.
Copying some relevant bits from the docstrings for Evidence.preprocess(), see there for more info:
Tasks export a list of the required_states they have for the state of the
Evidence it can process in `TurbiniaTask.REQUIRED_STATES`[1].  Evidence also
exports a list of the possible states it can have after pre/post-processing
in `Evidence.POSSIBLE_STATES`.  The pre-processors should run selectively
based on the these requirements that come from the Task, and the
post-processors should run selectively based on the current state of the
Evidence.

If a Task requires a given state supported by the given Evidence class, but
it is not met after the preprocessing of the Evidence is run, then the Task
will abort early.  Note that for compound evidence types that have parent
Evidence objects (e.g. where `context_dependent` is True), we only inspect
the child Evidence type for its state as it is assumed that it would only be
able to run the appropriate pre/post-processors when the parent Evidence
processors have been successful.

[1] Note that the evidence states required by the Task are only required if
the Evidence also supports that state in `POSSSIBLE_STATES`.  This is so
that the Tasks are flexible enough to support multiple types of Evidence.
For example, `PlasoTask` allows both `CompressedDirectory` and
`GoogleCloudDisk` as Evidence input, and has states `ATTACHED` and
`DECOMPRESSED` listed in `PlasoTask.REQUIRED_STATES`.  Since `ATTACHED`
state is supported by `GoogleCloudDisk`, and `DECOMPRESSED` is supported by
`CompressedDirectory`, only those respective pre-processors will be run and
the state is confirmed after the preprocessing is complete.

Some other small fixes:

Fixed some format strings
Fixed input_evidence being attached to TurbiniaTaskResult

Note: Based on some feedback from the original PR, I've renamed 'status' to 'state', and 'capabilities' to 'possible_states'.

I'm requesting an initial review to cover the concept to get feedback on whether it is sound before completing the rest of the PR.  If it's agreed on in concept I'll add requirements for the rest of the Tasks and fix up the tests (including adding more tests to cover this specifically).

Actually, now that I'm thinking about this a bit more, I think it probably makes sense to check whether the status is required   by the task, and only attempt the corresponding preprocessing if that's needed rather than trying and allowing soft failures.  I'll adjust the PR to do this tomorrow.

@rgayon: Ok, I've updated this PR to only run the pre-processors when needed.  I also added an attribute to the Task for what 'capabilities' it has for preprocessing and evidence state.   More details in the description above.   Still a WIP, and if the general idea works I'll fix/add the tests and finish out the tasks.
@berggren : FYI, in case you had any thoughts on this.

naming
I think "state" is a fine word to use to describe it, instead of status. Status implies there is ongoing processing, which is not necessarily the case. So specifically:

capabilities -> possible_states
required_status -> required_states (also emphasising the fact that it is a list of acceptable states)
status -> current_state (perhaps clearer)

Preprocesing patterns
Thinking about this, two things come to mind:

A lot more examples of evidence processing come up that don't really make sense to be included as stand-alone tasks, as opposed to included in the evidence pre/post processing routines. This doesn't seem likely, as long as the statuses (or states :) ) are only used for evidence specific qualities, and not for globally useful attributes. For the same reason I can't think of many more if/else structures down the line. Let's see how future task definitions influence this.

Considering the fact that AFAIK  it is the role of TurbiniaTask's execute function to create the evidence objects for further processing if applicable, it makes sense to keep the logic in the evidence object, as the status check mechanism will ensure the task that will be doing further processing is able to safely repeat the status check mechanic, and the _pre/_post processing routines that fire will deliver a clear correlation between status tag and actual evidence features.

Ok, @berggren / @wajihyassine  PTAL.  I've updated the PR based on feedback so far, and added tests and the state requirements for each task.  I also a fixed a couple small things I ran into along the way.   I edited the original/first comment, so it should be up to date with the current state of things.

Thanks for the reviews!


force save option

For whenever you have all the time in the world, it would be nice to have an option to force saving of an expanded epub even if epubcheck reports errors.
During development phases, there are times when the errors reported are erroneous but I'm forced to manually package.

That would be good. Meanwhile, I have been using this:
zipEpub.java:
import com.adobe.epubcheck.util.Archive;
public final class zipEpub {
    public static void main(String[] args) throws Exception {
        Archive epub = new Archive(args[0], true);
        epub.createArchive();
    }
}

zipEpub.sh:
root=$(pwd)

EPUB_FOLDER="${root}/EPUB3/"

find "${EPUB_FOLDER}" -name ".DS_Store" -depth -exec rm {} \;

javac -classpath "${root}/tool/epubcheck/epubcheck.jar" "${root}/zipEpub.java" -d "${root}/tool/epub3-sliderizer/bin" -sourcepath "${root}" #-verbose

java -classpath "${root}/tool/epubcheck/epubcheck.jar:${root}/tool/epub3-sliderizer/bin" zipEpub ${EPUB_FOLDER}

Thanks, Daniel!
Another thing I discovered with some playing is that setting the --fatal flag will basically do what I need, as the errors I get are related to not-yet-implemented features, so don't register that high.
I suppose that's equivalent to a force option, since there's no point in packaging an epub with fatal errors (although I haven't looked to see what all is categorized as fatal). Maybe we just need some tips and tricks...