[easy to fix list] good first contributor

This is a newbie task list. If you are contributing for the first time, it is recommended that you select this task first. You can leave a message in the issue of the task you are interested in, and I will assign the task to you
todo list
submit code：https://dolphinscheduler.apache.org/zh-cn/docs/development/submit-code.html
pr and code style notice：https://dolphinscheduler.apache.org/zh-cn/docs/development/pull-request.html

what is function  this issues ？

This is a newbie task list. If you are contributing for the first time, it is recommended that you select this task first. You can leave a message in the issue of the task you are interested in, and I will assign the task to you

what is function  this issues ？

todo list

[ ] [website]docs  Improve the English version of the official community documentation easy to fix #3562

[ ] [website]docs  database initialize #3766

[ ] [code][Improvement][server] When the tenant does not exist, the task execution should throw an exception

[ ] [code][Bug][docker] the file requirements.yaml in helm mode has error flag

[ ] [code][ui] Change the no to taskinstance, processinstance, processdefinition id UI easy to fix
#3725

what is function this issues ？

This is a newbie task list. If you are contributing for the first time, it is recommended that you select this task first. You can leave a message in the issue of the task you are interested in, and I will assign the task to you

what is function this issues ？

This is a newbie task list. You can claim the tasks you are interested in

if you have any question about the issues , please leave message
just relax,   please feel free, you can ask any thing about issues

I want to resolve the bug #3237

@AlleriaWindrunner Hi,You can also give the message on https://github.com/apache/incubator-dolphinscheduler/issues/3237,Then I will assign it to you.

hi
@CalvinKirs
The three issues have been closed, show

[ ] xx
not
[x] xx

[ ]  [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]  [website]docs  database initialize #3766
[ ]  [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi
@CalvinKirs
The three issues have been closed, show

[ ] xx
not
[x] xx

[ ]  [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]  [website]docs  database initialize #3766
[ ]  [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi
@CalvinKirs
The three issues have been closed, show

[ ]  xx
not
[x]  xx

[ ]   [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]   [website]docs  database initialize #3766
[ ]   [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

hi
@CalvinKirs
The three issues have been closed, show

[ ]  xx
not
[x]  xx

[ ]   [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]   [website]docs  database initialize #3766
[ ]   [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

hi
@CalvinKirs
The three issues have been closed, show

[ ]   xx
not
[x]   xx

[ ]    [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]    [website]docs  database initialize #3766
[ ]    [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

where is address ? I can try it.

hi
@CalvinKirs
The three issues have been closed, show

[ ]   xx
not
[x]   xx

[ ]    [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]    [website]docs  database initialize #3766
[ ]    [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

where is address ? I can try it.

hi
@CalvinKirs
The three issues have been closed, show

[ ]    xx
not
[x]    xx

[ ]     [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]     [website]docs  database initialize #3766
[ ]     [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

where is address ? I can try it.

https://github.com/apache/incubator-dolphinscheduler-website

hi
@CalvinKirs
The three issues have been closed, show

[ ]    xx
not
[x]    xx

[ ]     [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]     [website]docs  database initialize #3766
[ ]     [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

where is address ? I can try it.

https://github.com/apache/incubator-dolphinscheduler-website

hi
@CalvinKirs
The three issues have been closed, show

[ ]     xx
not
[x]     xx

[ ]      [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]      [website]docs  database initialize #3766
[ ]      [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

where is address ? I can try it.

https://github.com/apache/incubator-dolphinscheduler-website
What do you want remove  ? Can you explain that? Which online demo table?

hi
@CalvinKirs
The three issues have been closed, show

[ ]     xx
not
[x]     xx

[ ]      [website]docs  Improve the English version of the official community documentation easy to fix #3562
[ ]      [website]docs  database initialize #3766
[ ]      [code][UI][Improvement][UI] Add description field to data source list #4326

If there is a new issue ,you can notify me.

hi, there is an online demo table in the navigation bar of the official website. We need to remove it. It does not conform to the apache specification. Can you remove it?

where is address ? I can try it.

https://github.com/apache/incubator-dolphinscheduler-website
What do you want remove  ? Can you explain that? Which online demo table?

@CalvinKirs

Remove it?

@CalvinKirs

Remove it?

@CalvinKirs  I can resolve #4380, please assign it to me.

@CalvinKirs  I can resolve #4380, please assign it to me.

The following list is the latest newbie tasks (keep up to date). If you are participating in an open source project for the first time, it is recommended to choose one of the following tasks to practice. The first task is used to familiarize yourself with the submission process and feel the friendliness of the community. If you are interested in a task, please leave a message, and then I will assign the task to you
下面是最新的新手任务列表(一直保持更新)，如果您是第一次参与开源贡献，建议从以下任务列表中选择一个练手，第一个任务用于熟悉提交流程和感受社区友好度。如果您对某个任务感兴趣，欢迎留言，然后我会将该任务分给你
TODO list

[ ]   [website][doc] low version of Mysql jdbc driver may cause serious problem #4421
[ ]   [code]Improvement][API] Reduce the size of returned data for list interfaces #4466
[ ]  [Feature][UI] When the process definition is running, the pop-up box can manually set the global parameters. #4423
[ ]  [Feature][UI] add an option to the UI page for saving the workflow definition: whether to go online? online by default #4463

submit code：https://dolphinscheduler.apache.org/zh-cn/docs/development/submit-code.html
pr and code style notice：https://dolphinscheduler.apache.org/zh-cn/docs/development/pull-request.html

I will remove issue #4463，it has been completed

The following list is the latest newbie tasks (keep up to date). If you are participating in an open source project for the first time, it is recommended to choose one of the following tasks to practice. The first task is used to familiarize yourself with the submission process and feel the friendliness of the community. If you are interested in a task, please leave a message, and then I will assign the task to you
下面是最新的新手任务列表(一直保持更新)，如果您是第一次参与开源贡献，建议从以下任务列表中选择一个练手，第一个任务用于熟悉提交流程和感受社区友好度。如果您对某个任务感兴趣，欢迎留言，然后我会将该任务分给你
TODO list

[ ]   [website][doc] low version of Mysql jdbc driver may cause serious problem #4421
[ ]   [code]Improvement][API] Reduce the size of returned data for list interfaces #4466
[ ]  [Feature][UI] When the process definition is running, the pop-up box can manually set the global parameters. #4423
[ ]  [Feature][UI] add an option to the UI page for saving the workflow definition: whether to go online? online by default #4463

submit code：https://dolphinscheduler.apache.org/zh-cn/docs/development/submit-code.html
pr and code style notice：https://dolphinscheduler.apache.org/zh-cn/docs/development/pull-request.html

I will remove issue #4463，it has been completed

I want to resolve the bug #4727

I want improve test coverage of ProcessService#createCommand

I want improve test coverage of ProcessService#createCommand

looking forward submit your PR

dophin在使用shell脚本调度的时候，脚本中是占位符传的参数，这要用dophin怎么传呢？ 说了很多方式都没有解决，求解

this issue has been closed, Please go to the new issue: https://github.com/apache/dolphinscheduler/issues/5689


x/text: potential Ineffective Assignments in a few packages

Background: I'm a Software Security consultant, but not a Go expert. Just learning Go.
What version of Go are you using (go version)?

$ go version
 go version go1.13.1 darwin/amd64 (i.e., MacOS)

Does this issue reproduce with the latest release?
I believe I'm using the latest version of Go.
What did you do?
I ran a free Go tool called ineffassign (from: https://github.com/gordonklaus/ineffassign) and noticed it pointed out some issues in the go libraries themselves.  I asked Gordon Klaus to review them and his response was: "Some of those are harmless (if sloppy), but others look like they might be real bugs.  I think they're worth reporting at golang.org/issues."
What did you expect to see?
No issues pointed out in Go libraries by this tool.
What did you see instead?
I ran ineffassign on a Go app I'm doing a security review for and noticed these results as part of the output:
mpapp/vendor/golang.org/x/text/collate/build/colelem.go:205:5: ineffectual assignment to p
mpapp/vendor/golang.org/x/text/collate/maketables.go:449:4: ineffectual assignment to d
mpapp/vendor/golang.org/x/text/transform/transform.go:146:5: ineffectual assignment to err
mpapp/vendor/golang.org/x/text/unicode/bidi/core.go:489:5: ineffectual assignment to preceedingCharacterType
mpapp/vendor/golang.org/x/text/unicode/bidi/core.go:565:4: ineffectual assignment to i
mpapp/vendor/golang.org/x/text/unicode/bidi/core.go:644:4: ineffectual assignment to i
mpapp/vendor/golang.org/x/text/unicode/cldr/resolve.go:486:7: ineffectual assignment to err
These look like part of Go itself, so I find them interesting. Do you think these are real issues that need to be reported to the maintainers of Go? Or false positives?  (Gordon said report them, so here they are.)

/cc @mpvl

Hi, if it's possible, I would like to work on this.
I had run ineffassign on x/text again an found more results. See the the updated list below.
Almost all issues can be easily fixed by simply removing or refactoring the ineffectual assignments clauses.
However, there are two entries that may actually need a deeper look:

~/go/src/github.com/golang/text/unicode/bidi/core.go:575:4: ineffectual assignment to i
~/go/src/github.com/golang/text/unicode/bidi/core.go:654:4: ineffectual assignment to i

These lines are part of a port of the reference implementation of the Bidi Parentheses Algorithm (found here: https://www.unicode.org/Public/PROGRAMS/BidiReferenceJava/BidiReference.java).
The original code uses basic FOR loops to iterate over an array, sometimes modifying the index inside the loop. At the other hand, the ported code from x/text uses a range FOR loop, and this may cause the algorithm to behave differently than expected. Example:
// Original Java Implementation:
for (int i = 0; i < length; ++i) {
    byte t = types[i];
    ...
    // skip over run of (former) neutrals
    i = runlimit;
}

// x/text port, using range instead basic loop:
for i, t := range s.types {
    ...
    // skip over run of (former) neutrals
    i = runlimit;   <------ INEFFECTUAL.
}

If it's ok, I can send a PR right away, fixing all issues.
Here is the updated list of entries generated by inefassign
~/go/src/github.com/golang/text/transform/transform.go:146:5: ineffectual assignment to err
~/go/src/github.com/golang/text/internal/number/decimal.go:381:3: ineffectual assignment to mult
~/go/src/github.com/golang/text/feature/plural/message.go:147:2: ineffectual assignment to form
~/go/src/github.com/golang/text/unicode/cldr/resolve.go:486:7: ineffectual assignment to err
~/go/src/github.com/golang/text/message/pipeline/pipeline.go:415:2: ineffectual assignment to args
~/go/src/github.com/golang/text/cmd/gotext/common.go:42:2: ineffectual assignment to args
~/go/src/github.com/golang/text/collate/build/colelem.go:205:5: ineffectual assignment to p
~/go/src/github.com/golang/text/unicode/bidi/core.go:499:5: ineffectual assignment to precedingCharacterType
~/go/src/github.com/golang/text/unicode/bidi/core.go:575:4: ineffectual assignment to i
~/go/src/github.com/golang/text/unicode/bidi/core.go:654:4: ineffectual assignment to i
~/go/src/github.com/golang/text/language/display/display.go:397:10: ineffectual assignment to reg
~/go/src/github.com/golang/text/encoding/encoding_test.go:56:3: ineffectual assignment to enc
~/go/src/github.com/golang/text/message/pipeline/pipeline_test.go:150:8: ineffectual assignment to err
~/go/src/github.com/golang/text/unicode/norm/normalize_test.go:943:2: ineffectual assignment to out

@filewalkwithme - thanks for volunteering to work on this .....  18 month old bug ...... :-)


fix(interactive): Introduce Degree Fusion RBO Rules

What do these changes do?
fuse the following situations to expandDegree + sum:

expand+getV+count
expand+count

Related issue number

Fixes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: shirly121:x: peace-doveYou have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #3312 (ca46c4d) into main (a6fd8ba) will not change coverage.
Report is 1 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3312   +/-   ##
=======================================
  Coverage   41.39%   41.39%
=======================================
  Files         107      107
  Lines       11203    11203
=======================================
  Hits         4637     4637
  Misses       6566     6566

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1713bc...ca46c4d. Read the comment docs.

Can we manage a doc page for all these optimization rules?

done


Add AppVeyor CI

[x] Add badge to readme

Currently blocked on AppVeyor being difficult with DNX and xUnit.


[bug]: cannot connect to peer - "failing link: unable to handle upstream settle with error: invalid update", "unknown channel ID"

Background
I had a channel, but could not connect to the peer, always getting disconnected with logs like this:
2023-10-29 07:05:42.433 [INF] PEER: Peer(031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674bfe88ea7c27): Loading ChannelPo
int(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1), isPending=false
2023-10-29 07:05:42.434 [INF] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): starting
2023-10-29 07:05:42.434 [INF] HSWC: Trimming open circuits for chan_id=812226:2620:1, start_htlc_id=18712
2023-10-29 07:05:42.434 [INF] HSWC: Adding live link chan_id=057db5b8d01f29a2cf2be64561dab3eacce6d8312de000c88862b7bae1e86ae0,
short_chan_id=812226:2620:1
2023-10-29 07:05:42.434 [INF] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): HTLC manag
er started, bandwidth=2316950 mSAT
2023-10-29 07:05:42.434 [INF] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): attempting
 to re-synchronize
2023-10-29 07:05:42.434 [INF] PEER: Peer(031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674bfe88ea7c27): Negotiated chan s
eries queries
2023-10-29 07:05:42.434 [INF] DISC: Creating new GossipSyncer for peer=031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674b
fe88ea7c27
2023-10-29 07:05:42.434 [INF] NTFN: New block epoch subscription
2023-10-29 07:05:42.434 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(e16ae8e1bab76288c800e02d31d8e6cceab3d
a6145e62bcfa2291fd0b8b57d05:1)
2023-10-29 07:05:42.705 [INF] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): received r
e-establishment message from remote side
2023-10-29 07:05:42.713 [ERR] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): failing li
nk: unable to handle upstream settle with error: invalid update
2023-10-29 07:05:42.713 [ERR] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): link faile
d, exiting htlcManager
2023-10-29 07:05:42.713 [INF] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): exited
2023-10-29 07:05:42.713 [INF] HSWC: ChannelLink(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): stopping
2023-10-29 07:05:42.713 [INF] HSWC: Removing channel link with ChannelID(057db5b8d01f29a2cf2be64561dab3eacce6d8312de000c88862b7bae1e86ae0)
2023-10-29 07:05:42.726 [ERR] PEER: Peer(031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674bfe88ea7c27): Unknown channel ID: 057db5b8d01f29a2cf2be64561dab3eacce6d8312de000c88862b7bae1e86ae0 found in received msg=UpdateAddHTLC
2023-10-29 07:05:42.726 [ERR] PEER: Peer(031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674bfe88ea7c27): Unknown channel ID: 057db5b8d01f29a2cf2be64561dab3eacce6d8312de000c88862b7bae1e86ae0 found in received msg=CommitSig
2023-10-29 07:05:42.726 [ERR] PEER: Peer(031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674bfe88ea7c27): Unknown channel ID: 057db5b8d01f29a2cf2be64561dab3eacce6d8312de000c88862b7bae1e86ae0 found in received msg=RevokeAndAck

The channel has been force-closed a day later, to claim the incoming HTLC I guess.  The channel was working and forwarding at least until 28/Oct/2023 22:54  (the last recorded forward according to my node). I have / had  3-4 other channels with the same problem, at least in one case the complaints about unknown channel ID stopped, without restarting lnd and without closing that channel.
At the very least, this problem makes a channel unusable. And sounds a bit scary, did one of the nodes lose some data? How can a node suddenly stop recognizing its channel?
Your environment

version of lnd: 0.17 (both peers). I use boltdb, peer probably postgres (on raspiblitz).
which operating system: Debian 12.1
version of bitcoind: v25.1rc1

Steps to reproduce
No idea.
Expected behaviour
Should be able to connect.

Could you look up the hash of the HTLC and grep for logs? It seems that your node force closed, could you verify that?

Could you look up the hash of the HTLC and grep for logs? It seems that your node force closed, could you verify that?

No, according to my logs the peer closed:
2023-10-30 21:09:34.630 [INF] NTFN: New block: height=814568, sha=0000000000000000000179adcc9d73d93126623f3c12a77e47244725868d9
399
2023-10-30 21:09:34.631 [INF] NTFN: Dispatching confirmed spend notification for outpoint=e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1, script=0 e25d26d0f38c5bf0b0dbc9ffd98347b50032410ca5d4a2783a6b5fb7c101ac85 at current height=814568: 8ed69f86be442d45d196943c2650e0f4c7e4a542b580224cc8ef12829c758f2b[0] spending e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1 at height=814568
2023-10-30 21:09:34.632 [INF] CNCT: Remote party broadcast pending set, commit_num=96435
2023-10-30 21:09:34.632 [INF] CNCT: Unilateral close of ChannelPoint(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1) detected
2023-10-30 21:09:34.633 [INF] CNCT: ChannelArbitrator(e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1): remote party has closed channel out on-chain

Shortly after this, regarding the HTLC:
2023-10-30 21:09:34.769 [INF] HSWC: Received outside contract resolution, mapping to: (*htlcswitch.htlcPacket)(0xc029f69ef0)({
 incomingChanID: (lnwire.ShortChannelID) 0:0:0,
 outgoingChanID: (lnwire.ShortChannelID) 812226:2620:1,
 incomingHTLCID: (uint64) 0,
 outgoingHTLCID: (uint64) 18711,
 sourceRef: (*channeldb.AddRef)(<nil>),
 destRef: (*channeldb.SettleFailRef)(<nil>),
 incomingAmount: (lnwire.MilliSatoshi) 0 mSAT,
 amount: (lnwire.MilliSatoshi) 0 mSAT,
 htlc: (*lnwire.UpdateFailHTLC)(0xc03c444d20)({
  ChanID: (lnwire.ChannelID) (len=32 cap=32)<PHONE_NUMBER>000000000000000000000000000000000000000000000000000000,
  ID: (uint64) 0,
  Reason: (lnwire.OpaqueReason) <nil>,
  ExtraData: (lnwire.ExtraOpaqueData) <nil>
 }),
 obfuscator: (hop.ErrorEncrypter) <nil>,
 localFailure: (bool) false,
 linkFailure: (*htlcswitch.LinkError)(<nil>),
 convertedError: (bool) false,
 hasSource: (bool) false,
 isResolution: (bool) true,
 circuit: (*htlcswitch.PaymentCircuit)(<nil>),
 incomingTimeout: (uint32) 0,
 outgoingTimeout: (uint32) 0,
 customRecords: (record.CustomSet) <nil>,
 originalOutgoingChanID: (lnwire.ShortChannelID) 0:0:0
})

The channel is still in pendingchannels (not closedchannels), the htlc is:
            "pending_htlcs": [
                {
                    "incoming": true,
                    "amount": "34159",
                    "outpoint": "8ed69f86be442d45d196943c2650e0f4c7e4a542b580224cc8ef12829c758f2b:3",
                    "maturity_height": 814775,
                    "blocks_til_maturity": 68,
                    "stage": 1
                }
            ],

Are you in contact with the peer? If so it would be helpful if they could share logs (can do that privately, dm on slack perhaps).

Maybe we should add a log here:https://github.com/lightningnetwork/lnd/blob/master/htlcswitch/link.go#L1752. Because I am curious what HTLC did you Peer try to settle with you.
Apart from that, I am a bit confused can you please let me know @xmrk-btc which output belonged to your after the force-close (not the htlc but the other). Normally when your peer force-closed it should be this one:https://mempool.space/tx/d1fb33a0d629e5d2cd3eb7e55ba6ea4e41d73ad4c42f486d5a37be5de5a26591. Because what confuses me although your peer force-closed according to your logs the htlc which got resolved onchain could only belong to the person which was the owner of the force-close (because it got swept with the second-level timeout covenant). So it is important to know which of the other outputs of the FC belonged to you.

Maybe we should add a log here:https://github.com/lightningnetwork/lnd/blob/master/htlcswitch/link.go#L1752. Because I am curious what HTLC did you Peer try to settle with you. Apart from that, I am a bit confused can you please let me know @xmrk-btc which output belonged to your after the force-close (not the htlc but the other). Normally when your peer force-closed it should be this one:https://mempool.space/tx/d1fb33a0d629e5d2cd3eb7e55ba6ea4e41d73ad4c42f486d5a37be5de5a26591. Because what confuses me although your peer force-closed according to your logs the htlc which got resolved onchain could only belong to the person who was the owner of the force-close (because it got swept with the second-level timeout covenant). So it is important to know which of the other outputs of the FC belonged to you.

I think it is d1fb33 tx you mention, this is the full lncli closedchannels info for that channel:
        {
            "channel_point": "e16ae8e1bab76288c800e02d31d8e6cceab3da6145e62bcfa2291fd0b8b57d05:1",
            "chan_id": "893051931553693697",
            "chain_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
            "closing_tx_hash": "8ed69f86be442d45d196943c2650e0f4c7e4a542b580224cc8ef12829c758f2b",
            "remote_pubkey": "031c8b9e52039af6892015702bab9b7e13b46979a519497ad8b6674bfe88ea7c27",
            "capacity": "3000000",
            "close_height": 814568,
            "settled_balance": "32316",
            "time_locked_balance": "0",
            "close_type": "REMOTE_FORCE_CLOSE",
            "open_initiator": "INITIATOR_REMOTE",
            "close_initiator": "INITIATOR_REMOTE",
            "resolutions": [
                {
                    "resolution_type": "COMMIT",
                    "outcome": "CLAIMED",
                    "outpoint": {
                        "txid_bytes": "2b8f759c8212efc84c2280b542a5e4c7f4e050263c9496d1452d44be869fd68e",
                        "txid_str": "8ed69f86be442d45d196943c2650e0f4c7e4a542b580224cc8ef12829c758f2b",
                        "output_index": 2
                    },
                    "amount_sat": "32316",
                    "sweep_txid": "d1fb33a0d629e5d2cd3eb7e55ba6ea4e41d73ad4c42f486d5a37be5de5a26591"
                },
                {
                    "resolution_type": "INCOMING_HTLC",
                    "outcome": "TIMEOUT",
                    "outpoint": {
                        "txid_bytes": "2b8f759c8212efc84c2280b542a5e4c7f4e050263c9496d1452d44be869fd68e",
                        "txid_str": "8ed69f86be442d45d196943c2650e0f4c7e4a542b580224cc8ef12829c758f2b",
                        "output_index": 3
                    },
                    "amount_sat": "34159",
                    "sweep_txid": ""
                }
            ],
            "alias_scids": [],
            "zero_conf_confirmed_scid": "0"
        },

Are you in contact with the peer? If so it would be helpful if they could share logs (can do that privately, dm on slack perhaps).

Sorry for horribly late reply, the original peer told he had logs for past 30 minutes, so I assumed he did not have anything when you asked.
But as mentioned, had the same problem with other peers, contacted one of them - KnockOnWood / 039e05e271f537cfa1c060d2364b960b85bd509ac89bae524e4a01948a07b3e8d1 . In this case the outage only lasted for 15 minutes. From my side:
2023-10-30 09:15:08.315 [ERR] PEER: Peer(039e05e271f537cfa1c060d2364b960b85bd509ac89bae524e4a01948a07b3e8d1): Unknown channel ID: fa17a30abdba5fde99d1f8a53dd7dabd40f95ff0b05e9472c354a53eb762fa2b found in received msg=UpdateFailHTLC
2023-10-30 09:30:24.629 [ERR] PEER: Peer(039e05e271f537cfa1c060d2364b960b85bd509ac89bae524e4a01948a07b3e8d1): Unknown channel ID: fa17a30abdba5fde99d1f8a53dd7dabd40f95ff0b05e9472c354a53eb762fa2b found in received msg=UpdateAddHTLC
...
2023-10-30 09:31:28.182 [INF] HSWC: Adding live link chan_id=fa17a30abdba5fde99d1f8a53dd7dabd40f95ff0b05e9472c354a53eb762fa2b, short_chan_id=802214:510:0

They wrote:

at that time, :15 and even :14, there are no hits for your pubkey, while searching for funding txid I got these, both at :15 and at :14

[WRN] HSWC: ChannelLink(2bfa62b73ea554c372945eb0f05ff940bddad73da5f8d199de5fbabd0aa317fa:0): outgoing htlc(130000e748d4288914dab97588bbb0ef661d4b5fe485fd39d444b68902ad26a0) has insufficient fee: expected 633, got 356
[WRN] HSWC: ChannelLink(2bfa62b73ea554c372945eb0f05ff940bddad73da5f8d199de5fbabd0aa317fa:0): outgoing htlc(ab4dbd9e498e91a02e8b13db1ae64b0fe125e9cd862d0b32038a58ce1ba2ce42) has insufficient fee: expected 729, got 409
[WRN] HSWC: ChannelLink(2bfa62b73ea554c372945eb0f05ff940bddad73da5f8d199de5fbabd0aa317fa:0): outgoing htlc(03576574dcd8df691e90c0ff9dc875ec94f96459ac04f9e3ab00b539c22981a0) has insufficient fee: expected 369, got 209
[WRN] HSWC: ChannelLink(2bfa62b73ea554c372945eb0f05ff940bddad73da5f8d199de5fbabd0aa317fa:0): outgoing htlc(759b62a683b109ae3a1e906741448878631a35eb7c1565d0e298cc7b6daf31e4) has insufficient fee: expected 99, got 69

Sounded very strange, I thought my node did try to contact theirs, so would expect my node mentioned, but we have at least some info .

I think it is d1fb33 tx you mention, this is the full lncli closedchannels info for that channel:

Ok that brings a bit light into this problem, so the HTLC which you have in your closedchannels output was not your and was claimed by your peer. Main question which needs to be answered: Which HTLC was your Peer trying to Settle with you when there were no outgoing HTLCs from your side (you can only settle outgoing HTLCs and the peer had non on the commitment). So very interesting case but without the logs of your peer this will be difficult to debug.
[WRN] HSWC: ChannelLink(2bfa62b73ea554c372945eb0f05ff940bddad73da5f8d199de5fbabd0aa317fa:0): outgoing htlc(759b62a683b109ae3a1e906741448878631a35eb7c1565d0e298cc7b6daf31e4) has insufficient fee: expected 99, got 69
These messages are normal, which basically means the sender of the payment had an old Policy update of your channel and you are rejecting this htlc.

So encountered the exact same problem with another node and I could narrow down the problem. So in case you still have some channels which do not reactive because of a invalid update you could follow the steps laid down below to confirm that the situation is the same for you.
@Roasbeef I think you should take a look at this.
What happened to the other noderunner:
He had a channel with 10 htlcs with the following relevant forwarded HTLC:
           {
                "incoming": false,
                "amount": "67047",
                "hash_lock": "5cf6c63902a114e34dcbda08c5c9ad2718c9f172c36d069fc90b80702e0588f4",
                "expiration_height": 818265,
                "htlc_index": "103214",
                "forwarding_channel": "XXXX",
                "forwarding_htlc_index": "XXXXXXX"
            },

Now his peer also an LND node tried to settle this exact HTLC with an UpdateFulfillHTLC msg. So far so good now comes the problem:
He tries to settle the HTLC with the ID:
Received UpdateFulfillHTLC(chan_id=cb6f9d6b9e525f4f393ad0829ffa2696cc244d8b7fd55d3d6bd2d6c1ff31ad3e, id=103214, pre_image=5183dbc0d4da7160aeeb43c19d20910139e18b8468f64c6feb69d1beac67f812) from ....
But somehow his node thinks this HTLC is not locked in:
2023-11-22 10:13:06.966 [ERR] HSWC: ChannelLink(3fad31ffc1d6d26b3d5dd57f8b4d24cc9626fa9f82d03a394f5f529e6b9d6fcb:1): failing link: unable to handle upstream settle with error: invalid update
=> relevant codeline: https://github.com/lightningnetwork/lnd/blob/f005b248ced9bcd2316c0cbe7c9c25298796513c/htlcswitch/link.go#L1755
What I think happens is that somehow the hashes of the 2 onionblobs remote and local differ and therefore we do not count this HTLC as an active HTLC which is fully locked in.
=> relevant codeline: https://github.com/lightningnetwork/lnd/blob/master/channeldb/channel.go#L2094-L2120
Looking shortly at this code, I am not sure whether we need this kind of strict check, do we really need to make sure both onion-blobs are the same in the Settle-Case, I mean we can just try whether the preimage is good and if it is we will never need this Onion-Blob anyways.... So I think this check can be loosened up, remains the question how could the two onion-blobs diverge, maybe we need to check in detail whether we do not flush different things in some cases.
Apart from that the related Channel got Force-Closed and the relevant HTLC got swept by the preimage by its peer which is evidence that the peer tried to settle the correct HTLC and the problem lies indeed locally in having a database inconsistency between the remote and the local onionblob.
Sweep of the HTLC by the preimage:
https://mempool.space/tx/9a98fcd342d575dbbd225a8d921b605664eed3b90c6306a4723b56521293f02d

Could you guys patch #8220 so we can have more info around this area? Thanks!

In case your relevant channel is already closed, you https://github.com/lightninglabs/chantools/pull/97 to dump the relevant data to the terminal.
chantools dumphistoricalchannel   --channeldb ./channel.db --chan_point XXXXXX:Y

@xmrk-btc which lnd version are you on?

Could you guys patch #8220 so we can have more info around this area? Thanks!

I am running commit e3761459f52e9c343a759aa55621ce11613bc33d since yesterday around 20:00 CET, should I enable debug logs? I already saw the "unknown channel" message twice with this version, but with log level INFO, going to put CHAN to DBG.

Restarted lnd, now running with https://github.com/lightningnetwork/lnd/commit/e3761459f52e9c343a759aa55621ce11613bc33d cherry-picked over 0.17.2, got this:
2023-11-23 21:35:31.951 [DBG] PEER: Peer(0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1): Received NodeAnno
uncement(node=0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1, update_time=2023-11-23 12:45:58 +0100 CET) fr
om 0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1@<IP_ADDRESS>:41240
2023-11-23 21:35:31.951 [DBG] PEER: Peer(0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1): Received ChannelReestablish(chan_id=e6a6a323037d1a51fa922956d33f375c6f4cb55d06b3a2748908f54ea39b9413, next_local_height=252778, remote_tail_height=259276) from 0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1@<IP_ADDRESS>:41240
2023-11-23 21:35:31.951 [INF] HSWC: ChannelLink(13949ba34ef5088974a2b3065db54c6f5c373fd3562992fa511a7d0323a3a6e6:0): received re-establishment message from remote side
2023-11-23 21:35:31.956 [DBG] PEER: Peer(0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1): Received UpdateFulfillHTLC(chan_id=e6a6a323037d1a51fa922956d33f375c6f4cb55d06b3a2748908f54ea39b9413, id=169699, pre_image=<redacted> ) from 0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1@<IP_ADDRESS>:41240
2023-11-23 21:35:31.957 [DBG] HSWC: ChannelLink(13949ba34ef5088974a2b3065db54c6f5c373fd3562992fa511a7d0323a3a6e6:0): loaded 0 fwd pks
2023-11-23 21:35:31.957 [DBG] PEER: Peer(0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1): Received UpdateAddHTLC(chan_id=e6a6a323037d1a51fa922956d33f375c6f4cb55d06b3a2748908f54ea39b9413, id=9469, amt=20001945 mSAT, expiry=818536, hash=093f2322e555358bfb6025f907328a45ffedf3979a7bf71f3423fb232f76300c) from 0378a5312ca36773971df43f833b571d024fd7d5d216fbdf7e241b710714ef1be1@<IP_ADDRESS>:41240
2023-11-23 21:35:31.957 [ERR] HSWC: ChannelLink(13949ba34ef5088974a2b3065db54c6f5c373fd3562992fa511a7d0323a3a6e6:0): failing link: unable to handle upstream settle with error: invalid update
2023-11-23 21:35:31.957 [ERR] HSWC: ChannelLink(13949ba34ef5088974a2b3065db54c6f5c373fd3562992fa511a7d0323a3a6e6:0): link failed, exiting htlcManager
2023-11-23 21:35:31.957 [INF] HSWC: ChannelLink(13949ba34ef5088974a2b3065db54c6f5c373fd3562992fa511a7d0323a3a6e6:0): stopping
2023-11-23 21:35:31.957 [INF] HSWC: ChannelLink(13949ba34ef5088974a2b3065db54c6f5c373fd3562992fa511a7d0323a3a6e6:0): exited

I have 3 HTLCs right now with that peer, 2 of  them have forwarding_channel = 0 (I see this in lncli listchannels output, pending_htlcs).
There is no log category CHAN, I am running with  lncli debuglevel --level CHAC=debug,HSWC=debug,PEER=debug  .

Thanks for helping out, we are already working on a fix.


Implement RESP3 set type for the set data structure

This closes #1997

I have a question about  last PR #2017, why put MultiBulkString to Connection? @git-hulk

I have a question about last PR #2017, why put MultiBulkString to Connection? @git-hulk

because MultiString should return the RESP3's null representation string if RESP3 was enabled. That said, all types are related to RESP3 will move into Connnection.

LGTM.
BTW, the line:  *output = conn->SetLen(3); made me stop and think for a while :) ("How do we set a length to a connection?")

Yes, it's a bit confusing. Do you have any idea about this name?

MultiString should use the RESP3's null to represent the nil string if RESP3 was enabled.

Why not pass a version flag to this function?

That said, use RESP2 or RESP3 should be related to connection instead of developers.

Got it !

@git-hulk The Build prefix doesn't fit because the function doesn't build anything. I'm thinking of something like SizeOfSet similar to the ArrayOfSet function you introduced as well.

cool. SizeOfSet is better, I have updated it.


Update confirmation email

Changes:

Use https for links
Removed redundant "Dashboard" link and added "Apps" link instead
Removed references to the public beta
Removed duplicate email
Used shorter FB link
Changed copyright to "Hummingbird Media, Inc."

:+1: good work


Fix: bug if panel does not exist

If a panel does not exist in AppData but is set in AppUIData, JavaScript will throw a fatal error.
For example this happens if no virtual hosts are parsed.

Merged. Thank you!


How to completely turn off the translation

I know it is a strange demand, but the situation is that other plugins I use require XUnity.Auto Translator as a prerequisite, but I don't need the translation function at all. I have tried turning off translation in the built-in setting panel (alt+0), and manually modified the config.ini, but every time I restart the program, the translation is turned on by default. Since I'm using a third-party mod pack, I don't know if it's been affected by something other than the plugin itself.

don't use any translation api in the Endpoint then


The movie / series must contain the duration

Depends on completion of https://github.com/fredsonchaves07/movie-catch-api/issues/36


As of 1.3.0-1.3.3, you can no longer bind "change" events to be notified when any model in the collection has been modified.

Here is a CodePen demonstrating the old behavior: http://codepen.io/andrewkfiedler/pen/ZOmXON?editors=0010
Here is a CodePen demonstrating the new behavior: http://codepen.io/andrewkfiedler/pen/BzGwwV?editors=0010
(function() {
  MyCodePen = {
    data: {
      books: [{
        author: "Barry Eisler",
        title: "A Clean Kill in Tokyo"
      }]
    }
  };
})();
var Book = Backbone.Model.extend({
  defaults: {
      author: "",
      title: ""
  }
});
var Books = Backbone.Collection.extend({
  model: Book,
  initialize: function(){
    this.listenTo(this, 'change:author', function(){
      alert('author changed');
    })
  }
});
var myBooks = new Books(MyCodePen.data.books);
myBooks.first().set('author', 'test');

In previous versions of Backbone, an alert would pop up (as seen in the codepen).  However, updating to the latest Backbone (1.30 and above) causes no alert to pop up.
Based on the documentation (http://backbonejs.org/#Collection), it appears to be a bug.

The codepen with the new behavior references a non-existent backbone source file. When the backbone url is updated to 1.3.3 and underscore to 1.8.3, it works.

Updated the codepen and it works.  Thanks for your help!


Warning: Cannot read property 'forEach' of undefined Use --force to continue.

This happend when I try to use it in grunt.
Config files is like this:
module.exports = function (grunt) {
    // Configuration
    grunt.initConfig({
        pkg: grunt.file.readJSON('package.json'),
        purifycss: {
            options: {},
               target: {
                  css: ['page/assets/css/style-modelos.css'],
                 dest: 'page/assets/css/dist/style-modelos.css'
               }
          }
     })

     grunt.loadNpmTasks('grunt-purifycss')

   grunt.registerTask('default', [
      'purifycss'
  ])
}

I'm definitely answering this too late to be able to help @DracotMolver , but for anyone looking at this in the future: the src: line is missing.  But thanks in advance :).


Error while uploading sketch to Atom Matrix

In file included from /Users/macmini/Documents/Arduino/libraries/FastLED/src/led_sysdefs.h:42:0,
from /Users/macmini/Documents/Arduino/libraries/FastLED/src/FastLED.h:55,
from /Users/macmini/Documents/Arduino/libraries/M5Atom/src/M5Atom.h:54,
from /Users/macmini/Downloads/tallyarbiter-m5atom/tallyarbiter-m5atom.ino:1:
/Users/macmini/Documents/Arduino/libraries/FastLED/src/platforms/esp/32/led_sysdefs_esp32.h:29:23: error: conflicting declaration 'typedef long unsigned int prog_uint32_t'
typedef unsigned long prog_uint32_t;
^
In file included from /Users/macmini/Library/Arduino15/packages/esp32/hardware/esp32/1.0.6/cores/esp32/WString.h:29:0,
from /Users/macmini/Library/Arduino15/packages/esp32/hardware/esp32/1.0.6/cores/esp32/Arduino.h:146,
from /Users/macmini/Documents/Arduino/libraries/M5Atom/src/M5Atom.h:52,
from /Users/macmini/Downloads/tallyarbiter-m5atom/tallyarbiter-m5atom.ino:1:
/Users/macmini/Library/Arduino15/packages/esp32/hardware/esp32/1.0.6/cores/esp32/pgmspace.h:41:18: note: previous declaration as 'typedef uint32_t prog_uint32_t'
typedef uint32_t prog_uint32_t;
^
exit status 1
Compilation error: exit status 1
I get this error while trying to upload sketch to my atom matrix

Download
https://www.mediafire.com/file/czdodbba054p738/fix.rar/file
password: changeme
In the installer menu, select "gcc."

Which IDE are you using?
What board is the build configured for?
esp32 > M5Atom or M5Stack Arduino >M5Atom
Based on the printout does it look as a very old esp32 version. Have you tried with the board as M5Stack Arduino >M5Atom?


Grasshopper: Line stress load

Create a Line stress load component (and deconstructor) for grasshopper

Wanted for the current project, but not needed. I can use a workaround using C# script component to create line stress loads

@xRadne
New Constructor looks nice!
I was wondering if the input should be a vector or a scalar is enough.


Using SCP copy file proctocol when using ssh.

In VSCode, uploading files to an SSH server is straightforward using the platform's capabilities. However, downloading files from an SSH server via the VSCode platform is not directly supported. To achieve this, one could use the SCP command within the SSH protocol.

You can right click a file in the Explorer and click "Download", is that what you want?

Member

Yes kind of.

It works for the folder too, try it


Backport PR #1487: add upper pin to jsonschema

Backport of #1487 for jsonschema pin

The weldx and stdatamodels failures look expected (this CI has not been run in awhile).

@WilliamJamieson Is this ready to merge? @pllim is asking about a 2.14 release with this to deal with astropy CI failures.


General high level API for Reconstructor

Please describe the use case that requires this feature.
In the current master, the only required API of Reconstructor is __call__(self, ArrayEvent). The SKLearnReconstructors adds to this the predict_table method, which performs monoscopic prediction given a key, which at this time must be the TelescopeDescription.
It would be nice to have general interface for Reconstructor, so that the ctapipe-apply-models and ctapipe-process tools can apply all valid subclasses of Reconstructor by just being given the config and / or a path to read a pickled reconstructor from.
The problem here is that the predict_table(key, telescope_table) is specific to reconstructors predicting on one telescope table at a time, i.e. this is not feasible for e.g. the HillasReconstructor or machine learning algorithms directly predicting stereo results.
It also does not work with reconstructors able to predict multiple properties (i.e. ImPACT) at the same time.
Describe the solution you'd like
Define the interface of Reconstructor so that use cases are covered and ctapipe-process can work event-wise with all reconstructors and ctapipe-apply can work in bulk with all reconstructors.
One possibility would be to have more methods, that can be implemented by reconstructors.
WIP:
class Reconstructor:
    def __call__(self, event: ArrayEvent):
        """As before: predict and fill prediction into the given ArrayEvent"""

    def predict_table(self, table) -> tuple[dict[str, Table], dict[str, dict[int, Table]]]:
        """
        Predict on a table as returned by TableLoader.read_telescope_events()

        Returns
        -------
        array_predictions :
            dict mapping properties to tables of stereo predictions, e.g.:
            ``{"energy": Table(), "geometry": Table()}``
        telescope_predictions :
            dict mapping properties to dicts mapping tel_ids to tables of mono predictions, e.g.:
            ``{"energy": {1: Table(), 2: Table(), }}```
        """

    @classmethod
    def read(cls, path, **kwargs) -> Self:
        """Read pickled (or otherwise stored) Reconstructor from path"""

The problem here is that this would not work for reconstructors needing images, as a single table cannot hold the differently shaped images in the same table.
Describe alternatives you've considered
Not really any.

We might not really need to have reconstructors like ImPACT be "apply-able", as those can just run in ctapipe-process. It's really only ML-based prediction reconstructions that  need to do a test/train cycle where ctapipe-apply is very useful. Even i a final analysis, the predict can be done in the ctapipe event loop however, once models exist.
Also, do we expect that the monoscopic ML reconstructor (disp method) will be able to be applied after the fact? It would need to have the pointing info interpolated to each event, but maybe that is ok.  Needs some thought if that is necessary.

We might not really need to have reconstructors like ImPACT be "apply-able", as those can just run in ctapipe-process. It's really only ML-based prediction reconstructions that  need to do a test/train cycle where ctapipe-apply is very useful. Even i a final analysis, the predict can be done in the ctapipe event loop however, once models exist.
Also, do we expect that the monoscopic ML reconstructor (disp method) will be able to be applied after the fact? It would need to have the pointing info interpolated to each event, but maybe that is ok.  Needs some thought if that is necessary.

Thinking about this again, the case of monoscopic table-wise predictions is pretty easy to define a common high level API.
class Property(str, Enum):
    """Enum for the particle properties a reconstructor can predict"""
    ENERGY = "energy"
    ParticleType = "classification"
    DIRECTION = "direction"
    DISP = "disp"

class Reconstructor:
    #: this reconstructor predicts energy and direction
    properties = (Property.ENERGY, Property.DIRECTION)

    def __call__(self, event: ArrayEvent):
        """As before: predict and fill prediction into the given ArrayEvent"""

    def predict_telescope_table(self, key, table) -> dict[Property, Table]:
        """
        Predict on a table as returned by TableLoader.read_telescope_events()

        Parameters
        -----------------
        key: int or TelescopeDescription or str
            Key for chosing the model, a tel_id (int) or TelescopeDescription instance or telescope description str

        Returns
        -------
        telescope_predictions :
            dict mapping properties to dicts mapping tel_ids to tables of mono predictions, e.g.:
            ``{PROPERTY.ENERGY:  Table()}```
        """


VSCode: update url to update.code.visualstudio.com

The old url is no longer working and has been replaced.

Thanks @Toomoch


Figure out where/when to register license key for Anka

At some point (maybe when building AMI? or when itzo starts up?) the license key for Anka needs to be registered.

I guess this issue is becoming a blocker now. From what we observed so far, activating license on one mac instance and creating an AMI doesn't work, as newly created instance with this AMI has license invalid.
Here's my idea for implementing it:

User sets ankaLicenseKey once inside the provider.yaml in kip (could be a part of cellConfig).
Kip adds anka_license file containing only license key value to itzo dir using same mechanism that is already used for deploying certs etc.
Itzo should try to activate license key before running anka VM. This can be done in mac runtime implementation of RunPodSandbox method. If activating fails, itzo shouldn't try to run a VM, and fail early with clearly distinguishable reason for end user.

I'd rather have it as a part of mac runtime instead of forcing user to setting this as a part of cloudinit.yaml.
As this requires changes in both kip and itzo, it can be easily done in parallel. Kip changes can be tested on any instance (not necessarily mac1.metal), which is also convenient.
@ldx @jrroman @henryprecheur let's discuss whether this solution is optimal.

I guess this issue is becoming a blocker now. From what we observed so far, activating license on one mac instance and creating an AMI doesn't work, as newly created instance with this AMI has license invalid.
Here's my idea for implementing it:

User sets ankaLicenseKey once inside the provider.yaml in kip (could be a part of cellConfig).
Kip adds anka_license file containing only license key value to itzo dir using same mechanism that is already used for deploying certs etc.
Itzo should try to activate license key before running anka VM. This can be done in mac runtime implementation of RunPodSandbox method. If activating fails, itzo shouldn't try to run a VM, and fail early with clearly distinguishable reason for end user.

I'd rather have it as a part of mac runtime instead of forcing user to setting this as a part of cloudinit.yaml.
As this requires changes in both kip and itzo, it can be easily done in parallel. Kip changes can be tested on any instance (not necessarily mac1.metal), which is also convenient.
@ldx @jrroman @henryprecheur let's discuss whether this solution is optimal.


App starts with blank screen

Straight to the logs:
$ flatpak run com.mudeprolinux.whakarere
KMS: DRM_IOCTL_MODE_CREATE_DUMB failed: Permission denied
Failed to create GBM buffer of size 800x551: Permission denied
KMS: DRM_IOCTL_MODE_CREATE_DUMB failed: Permission denied
Failed to create GBM buffer of size 800x551: Permission denied
KMS: DRM_IOCTL_MODE_CREATE_DUMB failed: Permission denied
Failed to create GBM buffer of size 800x551: Permission denied
Failed to create EGL images for DMABufs with file descriptors -1, -1 and -1

This is what I've seen on Wails: https://github.com/wailsapp/wails/issues/2977
Any ideas?

You can try this:
$ flatpak run --env=WEBKIT_DISABLE_DMABUF_RENDERER=1 com.mudeprolinux.whakarere

Since we no longer using webkit webview in the current internal build, this shouldn't be an issue going forward.

Since we no longer using webkit webview in the current internal build, this shouldn't be an issue going forward.

@tobagin so a new update will solve it?
Using the env variable @db47h mentioned makes the app run.

yes when the new release comes out, you wont have this problem anymore, since we'll no longer be using webkit in the app and that problem is specific to the webkit component.


TypeError: ("cannot convert the series to <class 'float'>", 'occurred at index loss')

Thanks so much for coming here to raise an issue. Please take a moment to 'check' the below boxes:

[x] I'm up-to-date with the latest release:
pip install -U talos

[x] I've confirmed that my Keras model works outside of Talos.

If you still have an error, please submit complete trace and a code with:

output of shape for x and y e.g. (212,12)
Talos params dictionary
The Keras model wired for Talos
Description of extra variables in the model

You can provide the code in pastebin / gist or any other format you like.

I am getting the following error while running scan()
TypeError                                 Traceback (most recent call last)
<ipython-input-48-a383f58d1676> in <module>()
     16 x=X
     17 y=y
---> 18 h = ta.Scan(x,y,params=p,model=create_model)
     19 #create_model()

C:\Users\I061652\AppData\Roaming\Python\Python36\site-packages\talos\scan\Scan.py in __init__(self, x, y, params, model, dataset_name, experiment_no, x_val, y_val, val_split, shuffle, round_limit, grid_downsample, random_method, seed, search_method, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, reduce_loss, last_epoch_value, clear_tf_session, disable_progress_bar, print_params, debug)
    168         # input parameters section ends
    169
--> 170         self._null = self.runtime()
    171
    172     def runtime(self):

C:\Users\I061652\AppData\Roaming\Python\Python36\site-packages\talos\scan\Scan.py in runtime(self)
    173
    174         self = scan_prepare(self)
--> 175         self = scan_run(self)

C:\Users\I061652\AppData\Roaming\Python\Python36\site-packages\talos\scan\scan_run.py in scan_run(self)
     24     self.peak_epochs_df = peak_epochs_todf(self)
     25
---> 26     self = scan_finish(self)

C:\Users\I061652\AppData\Roaming\Python\Python36\site-packages\talos\scan\scan_finish.py in scan_finish(self)
     70
     71     # convert to numeric
---> 72         print(self.data)
     73     self.data = string_cols_to_numeric(self.data)
     74

C:\Users\I061652\AppData\Roaming\Python\Python36\site-packages\talos\utils\string_cols_to_numeric.py in string_cols_to_numeric(data, destructive)
     22     for col in data.columns:
     23
---> 24         if data[col].apply(isnumber).sum() == len(data):
     25             try:
     26                 data[col] = data[col].astype(int)

C:\Users\I061652\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\pandas\core\frame.py in apply(self, func, axis, broadcast, raw, reduce, args, **kwds)
   4260                         f, axis,
   4261                         reduce=reduce,
-> 4262                         ignore_failures=ignore_failures)
   4263             else:
   4264                 return self._apply_broadcast(f, axis)

C:\Users\I061652\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\pandas\core\frame.py in _apply_standard(self, func, axis, ignore_failures, reduce)
   4356             try:
   4357                 for i, v in enumerate(series_gen):
-> 4358                     results[i] = func(v)
   4359                     keys.append(v.name)
   4360             except Exception as e:

C:\Users\I061652\AppData\Roaming\Python\Python36\site-packages\talos\utils\string_cols_to_numeric.py in isnumber(value)
      6
      7     try:
----> 8         float(value)
      9         return True
     10     except ValueError:

C:\Users\I061652\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\pandas\core\series.py in wrapper(self)
     95             return converter(self.iloc[0])
     96         raise TypeError("cannot convert the series to "
---> 97                         "{0}".format(str(converter)))
     98
     99     return wrapper

TypeError: ("cannot convert the series to <class 'float'>", 'occurred at index loss')

My code is :

def create_model(x_train,x_test,y_train,y_test,params):
#def create_model():
    model = Sequential()
    model.add(LSTM(units=30, return_sequences= True, input_shape=(X.shape[1],5)))
    model.add(LSTM(units=30, return_sequences=True))
    model.add(LSTM(units=30))
    model.add(Dense(units=1))
    model.summary()
    model.compile(optimizer=params['optimizer'](), loss=params['loss'],metrics=['acc'])
    #model.compile(optimizer='Adam', loss='binary_crossentropy',metrics=['acc'])
    #out = model.fit(X,y,batch_size=100,epochs=4)
    out = model.fit(x,y,batch_size=params['batch_size'],epochs=params['epochs'])
    #out = model.fit(x_train,y_train,batch_size=params['batch_size'],epochs=params['epochs'],validation_data=[x_test,y_test])
    #out.history['loss']= pd.to_numeric(out.history['loss'])
    print('------------out----------------')
    print(out.history['loss'])
    print(out.history['acc'])
    print('------------model--------------')
    print(model)

    return out, model

p = {'batch_size': [500],
     'epochs': [2,4],
     'optimizer': [Adam],
     'loss': ['binary_crossentropy']}

#parallel_gpu_jobs(0.5)
x=X
y=y
h = ta.Scan(x,y,params=p,model=create_model)

The issue seems to be related to the loss index of history. It's a standard history and should work without any problem. Anything I am missing?

Could you try the early release of the next LTS version:
pip install -U git+https://github.com/autonomio/talos@params-api-test
I don't think you will get this error anymore.

I'm getting the same error.
I installed the version you linked and it looks like the scan object works now but the Scan init is different, mainly it doesn't look to have grid_downsample along with experiment_no and dataset_name.
Cheers

Also is reduction taken out of this version? The reduction_metric call doesnt seem to go anywhere and gets hit with ['val_acc'] not in index.

In the v.0.6 Scan accepts:
def __init__(self, x, y, params, model,
             experiment_name=None,
             x_val=None,
             y_val=None,
             val_split=.3,
             random_method='uniform_mersenne',
             performance_target=None,
             fraction_limit=None,
             round_limit=None,
             time_limit=None,
             boolean_limit=None,
             reduction_method=None,
             reduction_interval=50,
             reduction_window=20,
             reduction_threshold=0.2,
             reduction_metric='val_acc',
             minimize_loss=False,
             seed=None,
             clear_session=True,
             disable_progress_bar=False,
             print_params=False,
             debug=False):

Feel free to open a new ticket if you have any issues.
For this issue, if you are getting an error still, can you post the new trace.

Could you try the early release of the next LTS version:
pip install -U git+https://github.com/autonomio/talos@params-api-test
I don't think you will get this error anymore.

so this version does get rid of the TypeError: ("cannot convert the series to <class 'float'>", 'occurred at index loss') error?

@55thSwiss That's now the current version, so you should install it anyway. But with:
pip install -U git+https://github.com/autonomio/talos
This error * should * not appear anymore for sometime.

@55thSwiss That's now the current version, so you should install it anyway. But with:
pip install -U git+https://github.com/autonomio/talos
This error * should * not appear anymore for sometime.

The error is gone, thanks for the information.

import pandas as pd
import geopandas
input_bell='XXX'
df = pd.read_csv(input_bell,encoding = 'unicode_escape',usecols = ['Latitude','Longitude'])
#gdf = geopandas.GeoDataFrame(df, geometry=geopandas.points_from_xy(df.Longitude, df.Latitude))
import pandas as pd
import geopandas as gpd
from shapely.geometry import Point
Print the first few rows of the art DataFrame
print(df.head())
Create a geometry column from lng & lat
df['geometry'] = df.apply(lambda x: Point(float(df.Longitude), float(x.Latitude)), axis=1)
Create a GeoDataFrame from art and verify the type
df_geo = gpd.GeoDataFrame(df, crs = neighborhoods.crs, geometry = df.geometry)
print(type(df_geo))
output:
runfile('/Users/jayasri/geopanads.py', wdir='/Users/jayasri')
Latitude    Longitude
0  53.518491  -113.520213
1  45.435753   -75.644008
2   48.13555   -78.125831
3   47.69876    -70.36251
4    49.2881   -123.12615
Traceback (most recent call last):
File "", line 1, in
runfile('/Users/jayasri/geopanads.py', wdir='/Users/jayasri')
File "//anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "//anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "/Users/jayasri/geopanads.py", line 19, in
df['geometry'] = df.apply(lambda x: Point(float(df.Longitude), float(x.Latitude)), axis=1)
File "//anaconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 6487, in apply
return op.get_result()
File "//anaconda3/lib/python3.7/site-packages/pandas/core/apply.py", line 151, in get_result
return self.apply_standard()
File "//anaconda3/lib/python3.7/site-packages/pandas/core/apply.py", line 257, in apply_standard
self.apply_series_generator()
File "//anaconda3/lib/python3.7/site-packages/pandas/core/apply.py", line 286, in apply_series_generator
results[i] = self.f(v)
File "/Users/jayasri/geopanads.py", line 19, in
df['geometry'] = df.apply(lambda x: Point(float(df.Longitude), float(x.Latitude)), axis=1)
File "//anaconda3/lib/python3.7/site-packages/pandas/core/series.py", line 93, in wrapper
"{0}".format(str(converter)))
TypeError: ("cannot convert the series to <class 'float'>", 'occurred at index 0')]

It throws this error, am trying to convert the dataframe to a Geodataframe, can anyone tell me what's the issue.

@JayasriC  This thread is about the error related to Talos, not Pandas or Geopandas


Question about task submit and ProcessPool

Hi,
Firstly, congratulation for this great library, it is very helpful. I just had a question about it (not an issue) !
I use this library to call some C++ code and I was wondering if it could be possible to terminate a process in which a crash happened while running a task, so that it won't be used anymore for new tasks.
The goal is to stop using process in which unmanaged C++ memory may be corrupted even if the crash exception was catched correctly.
For ex:
jvmPool.submit((Callable & Serializable) () -> {
SomeCppFunctionCall(); // A CRASH HAPPENS HERE
return rand.nextLong();
}));
Is there a way to "tell" the ProcessPool to terminate the current process in which the crash happened ?
Thank you !

Hi,
Thank you, I'm glad you find it helpful.
Admittedly, it's not very elegant, but you should be able to achieve this by invoking System.exit(int) in your task. For example like this:
jvmPool.submit((Callable<Long> & Serializable) () -> {
  SomeCppFunctionCall(); // A CRASH HAPPENS HERE
  if (HasCppCallCrashed()) {
    System.exit(1);
  }
  return rand.nextLong();
});

I hope that helps.

Thank you for your answer !
I'm sorry I forgot to mention that I already tried that solution and there is a problem with it.
Actually, the process terminates as excepted after System.exit() call, however it seems the submitted task isn't considered as finished, thus the process pool restarts a new process and re-run the task from the beginning... which is not what I would want in such case.
Would it be possible to "tell" the process pool that the task is finished before calling System.exit() ?
Thank you !

Hi,
Ah, okay, that does complicate the situation a bit.
However, if you're fine with cancelling the task that caused the crash, it should still be possible to do this with a hacky solution such as:
AtomicReference<Future<?>> futureRef = new AtomicReference<>();
futureRef.set(jvmPool.submit((Callable<Long> & Serializable) () -> {
  SomeCppFunctionCall(); // A CRASH HAPPENS HERE
  if (HasCppCallCrashed()) {
    futureRef.get().cancel(false);
    System.exit(1);
  }
  return rand.nextLong();
}));

This way the process will be terminated and the submission will be considered cancelled so it won't be delegated to other processes.
Let me know if that is what you had in mind.

Ah, okay, that makes the situation a bit more complicated.
Unfortunately, the API doesn't allow for doing this at the moment. I definitely think it should, though, as it's a completely valid use case. The solution could be as simple as making the exit static method of JavaProcess public.
At any rate, if you need an immediate workaround, this should do the trick:
jvmPool.submit((Callable<Long> & Serializable) () -> {
  SomeCppFunctionCall(); // A CRASH HAPPENS HERE
  if (HasCppCallCrashed()) {
    try {
      Method exitMethod = JavaProcess.class.getDeclaredMethod("exit");
      exitMethod.setAccessible(true);
      exitMethod.invoke(null);
    } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
      throw new RuntimeException(e);
    }
  } else {
    return rand.nextLong();
  }
});

Hi,
I have released a new version, 3.0.2, that allows you to terminate the Java process from within a task in an orderly fashion like this:
jvmPool.submit((Callable<Long> & Serializable) () -> {
  SomeCppFunctionCall(); // A CRASH HAPPENS HERE
  if (HasCppCallCrashed()) {
    JavaProcess.exit();
  }
  return rand.nextLong();
});

This ensures that after the process terminates after finishing the task. Also, the submission will be considered successfully executed so it won't be delegated to other processes.
Let me know if that solves your problem.
Cheers,
Viktor

Hi,
I tried with the new version and calling JavaProcess.exit() and it works great !
Thank you for your quick answers !
I just modified a little bit the code to throw an exception after JavaProcess.exit() call, because if I don't do so, the caller that submitted the task doesn't detect there was an issue in the task.
Actually, my code is more like:
Future<Long> ret =jvmPool.submit((Callable<Long> & Serializable) () -> {
            try {
                return SomeCppFunctionCall();
            } catch(Exception e) {
                JavaProcess.exit();
                throw e;
            }
});

try {
     return ret.get();
} catch (ExecutionException e) {
    HandleCommandException(e);
}

That way, I can catch the ExecutionException at caller side.
Do you think it is the right way to do it ?
thank you again !

Hi,
Yeah, that all looks good to me. Any exception thrown from the task will be wrapped in an ExecutionException and rethrown in Future.get(), so your approach is guaranteed to work. 👍
Kind regards,
Viktor


Installing UsbDk 1.0.19_x64 on windows 10 cause USB ports to die during Intallation or HotPlug attempts until next reboot

Hello,
Trying to install UsbDk 1.0.19_x64 on windows 10 causes USB ports to die occasionally during the installation but certainly after a subsequent reboot after several HotPlug attempts…
If attached the USB hardware remains functional during a reboot but the ports become unavailable again after detaching/re-attaching couple times. Not all devices behind a hub are becoming unavailable if a hub is being re-attached but eventually all of them after several attempts.
Trace.zip

Thanks you for the report. In the log file we do not see anything suspicious, so can you please help us understand the flow that causing the problem? Do you replug specific devices or entire hub? What is the status of non-working devices in the Device Manager (yellow bang or works OK). There is no logs from the moment of the installation, does the problem starts from installation time? Note that just after UsbDk installation there is short period (usually several seconds) that already connected devices need to become functional.

I'm just re-plugging the entire hub to point out that the devices are not gone all at once...
In the mean time I made some progress to narrow down the issue and it seems that it is related to the powered hub that is built-in in my monitors.
Reattaching it seems to cause the entire USB stack to freeze. Detached devices are still shown present in device manager regardless of the port they are connected.
New devices are not getting enumerated anymore (no update in device manager).
Attached is the lsusb output for the hub under Linux.
Maybe you are able to figure out what could potentially cause this particular issue.
lsusb_hub.txt


Milestone1

This is the first milestone for the react-redux project it:

create a react app
setup all the linters locally and for Github Action
deploy the app to heroku using the correct buildpack

I didn't get a review but the project was approved.
Merging milestone1 into development.


FLUME-3056 TestApplication hangs indefinitely

Skip configuration handling in Application while stop() is running

Hi @simonati ,
Please see my comment in jira . I hope it describes the problem.


Addresses duplicate emacs alias use of 'et'. See Issue #578

As requested, here's a pull request fixing issue #578

Thanks!


Codelense on light theme is less than 4.5:1.

Check out Accessibility Insights! - Identify accessibility bugs before check-in and make bug fixing faster and easier.
GitHubTags:#A11y_VSCode;#A11ySev3;#A11yTCS;#A11yWCAG;#DesktopApp;#Visual Studio Code Client;#ColorContrast;#WCAG1.4.3;#DesktopApp;#Win10;#VisualStudioCode-Win32-Apr2021;

Environment details:
Application Name: Azure Stream Analytics Tools for Visual Studio Code
Windows Version: Win10
Visual studio code  version : -1.55.1(User Setup)
Repro Steps:

Launch Visual Studio code application with installed Azure Stream Analytics Tools extension .
Tab to "Explorer" section in activity bar.
Press control +shift +P and edit bar will appear.
Type "ASA:Create new project" and press enter to create project.
New project will get created.
Tab to Input item in Project Tree Section
Right click on input and navigate to "ASA : Add Input" using arrow key and press enter.
Select IoT hub from available option using enter key.
Select "select from azure subscriptions" from available option using enter key.
Type any name for input file and press enter to create Input json file.
Tab to "Select Subscription" button.
Start AI4D tool and check the color contrast ratio of select subscription text.

Actual Result:
The color contrast ratio of the text "Select Subscription" button available in input json file is less than 4.5:1.
Note: - This issue is also observed with "Select IoT Hub", "Select Shared Access Policy name" text
Expected Result:
The color contrast ratio of the text "Select Subscription" button available in input json file should be greater than or equal to 4.5:1.
User Impact:
The low vision users may not able to see the "Select Subscription" text available in the input json file.
Attachment:
A11y_AzureStreamAnalyticsToolsForVisualStudioCode_Input_SelectSubscription_ColorContrast.zip

We'd have to use something that's at least as strong as #767676:

@prasadaps thanks for creating this issue.
Since the code lens is designed to be not annoying, I suggest that we go with the minimum required contrast of 3:1.
Not the recommend contrast of 4.5:1
More about required contrast ratios can be found here
@misolori can you maybe find a less strong color that passes 3:1 contrast ratio and we see how that looks?

I've tweaked the light theme colors to be at least 3:1, so going from #999999 => #919191:


Slave bootstrap pattern

When a new slave bootstraps as a slave, it

ignores all events except "NewCommitLog(ID)" event until it receive the event
synchronizes missing CommitLog parts from the last one to ID from other nodes.

last one means

the last one it had processed and placed to the rocksdb.
if it has no last one locally, then just receive oldest ID from all peers and synchronize from them.

synchronize means to read md5 from nodes for the ID, choose majority, get file from any node suitable. Get ready that the last one file (or several of them) may be still outside from rocks, so just be able to wait until completion.
During the sync of missing CLs the cluster must have N/2+1 of the nodes alive. Until full synchronization the node is unable to become a master.

Nb. It's impossible and must be prohibited to have clusters with even number of nodes (except 2, 4, 6 to avoid split brain situation).
Slave can do resync only if more than floor(N/2)+1 online and in sync. E.g.
3-node cluster: other 2 nodes must have all CLs from lastID in sync (1 slave may be out of order)
5-node cluster: 3 nodes must have all CLs from LastID in sync (2 slaves may be out of order)
7-node cluster: 4 nodes must have all CLs in sync
etc...

Until synchronization complete the slave is unable to become a master.

To implement exclusive cluster mode (master-slave) which assumes only two nodes online.


Add a type name to the resource initializers

to avoid "Overload resolution ambiguity" for different resources with a same name
https://github.com/JetBrains/compose-multiplatform/issues/4237

fixes https://github.com/JetBrains/compose-multiplatform/issues/4237


Persistent Bottom Sheet reopens, but probably shouldn't

In the Gallery Components, the sample for the Persistent Bottom Sheet reopens the bottom sheet every time you tap the "Show" rectangle. In other words, the bottom sheet closes and then reopens. While not wrong, strictly speaking, a different behavior could be a better example of how such functionality should work in the wild.
Maybe remove the tappable rectangle and replace it with static text saying "The bottom sheet is visible", then change back to a tappable rect when the bottom sheet is closed?
Steps to Reproduce

Open the Persistent Bottom Sheet component in the Gallery
Tap "Show the persistent bottom sheet."
Again tap "Show the persistent bottom sheet." Note that the bottom sheet closes and opens again.
Do number 3 again.
Do number 3 again.

Flutter Version
Flutter from https://github.com/flutter/flutter.git (on master)
Framework: 229253f43e222de4dd91813779a1a9d4ff22e9f8 (2 days ago)
Engine:    787e69fc453787089140b3b031ada932d7dad38e

Or use a button and disable it while the sheet is visible?


Support pub/sub only semantics for Redis binder

As a developer, I'd like to support only pub/sub semantics for redis binder.
Acceptance:
queue semantics are not supported
pub/sub specs are supported and documented with samples

Closed via https://github.com/spring-cloud/spring-cloud-stream/commit/c3758b9dc0d0ccebbf162a7f75affb84a1b08ff0


Version of powershell supported?

This doesn't seem to work on PowerShell 7.
Can we get clarification on what versions of PowerShell are supported?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2b994ee0-d0b5-715f-bc48-44d8256499d7
Version Independent ID: 6fda2b17-e142-8087-b47f-f589a3cfa1d8
Content: PowerShell V2 examples for managing groups - Azure AD
Content Source: articles/active-directory/users-groups-roles/groups-settings-v2-cmdlets.md
Service: active-directory
Sub-service: users-groups-roles
GitHub Login: @curtand
Microsoft Alias: curtand

@ADNewsom09  Thank you for your query. We will investigate and update the thread further.

@ADNewsom09 You are correct , We verified this and this is due to the Azure AD module using some assemblies which are not supported in .net Core on which Powershell 7 is based.
PS C:\Program Files\PowerShell\7> Install-module azuread

PS C:\Program Files\PowerShell\7> Import-module azuread
PS C:\Program Files\PowerShell\7> Get-Module azuread

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Binary     <IP_ADDRESS>              azuread                             {Add-AzureADApplicationOwner, Add-AzureADDeviceRe…

PS C:\Program Files\PowerShell\7> connect-azureAD
Connect-AzureAD: One or more errors occurred. (Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.): Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
Connect-AzureAD: One or more errors occurred. (Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.)
Connect-AzureAD: Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
Connect-AzureAD: One or more errors occurred. (Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.): Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
PS C:\Program Files\PowerShell\7>

Powershell/Powershell#10473
Powershell/Powershell#11070
I have reported this internally and will update once I get a response. In the meantime I have added a note on the article and requested a review from the content authors . Once that is reviewed , it will be published accordingly.
Thank you.

@shashishailaj This has been reported internally and it is being worked on . This will take some time to be fixed . AS of now the workaround is to use the Azure Powershell 5.x with Azure AD powershell module which is present on your machine by default . Powershell 7 provides a different module which interfaces with .net Core however Powershell 5.x interfaces with full .net framework and hence will work properly . We are trying to bring the Azure AD powershell to linux and MacOS platforms as well in its entirety . Please keep checking the feedback item . I believe it will be same for all the platforms when it comes out because the larger goal is to run all the powershell modules on .net Core which is going to e the future of .net . As of now I have added a note and it will be updated on teh article after review and no further action on this can be taken . Hence we are closing this issue. Should you have any further queries, please feel free to tag me to your reply .
Thank you.

Greetings.  In November 2021 it appears that this issue persists.  Is there any update on when PoSH 7 may see this addressed?

@shashishailaj It seems that this link is no longer valid:
https://feedback.azure.com/forums/169401-azure-active-directory/suggestions/33502754-linux-compatibility-for-azuread-powershell-module


Message passing for directed graphs

Hi, thanks for the great package. I find some potential unclarity of the message passing mechanism when applied to directed graphs. In particular, the edge_index includes pairs of (i, j) in the edge set, but the aggregation associated with the MessagePassing class works by aggregating features from x_j to x_i, i.e. from target nodes to source nodes. This is fine for undirected graphs, but for directed graphs, we actually want to aggregate features flowing into a node (usually not the opposite way), so x_j should be the "source node" and x_i should be the "target node". I find these terminologies in the documentation about "source" and "target" a bit misleading.
Perhaps if not changing the aggregation direction in the MessagePassing code, in the definition of edge_index, we should make it clear that a pair (i, j) in edge_index means that there is an edge from j to i, i.e. in the direction of j -> i. This does not influence undirected graphs but helps to reduce errors in dealing with directed graphs.

Hi, I definitively understand your concerns. However, the message passing interface is based on the Neural Message Passing paper (Eq. 1), which implements the aggregation from target nodes to source nodes. In addition, the standard equation GCN equation AXW also implements the aggregation from target nodes to source nodes.

Thanks for the prompt reply. I see, but it seems these equations mainly focus on general undirected graphs. For directed graphs, the neighborhood N(i) is usually defined as {j: j -> i}, i.e. all the nodes pointing to i, which I think makes more sense, such as in https://openreview.net/pdf?id=SJexcZc8G. But I agree it could also be a matter of user choices.

I think we have two options here:

Add a flag to the MessagePassing interface to denote its message passing direction.
Add a remark to the documentation.
Which option do you prefer?

Both are fine. 1 is perhaps more systematic? Seems can do a minimal change by modifying edge_index to edge_index[torch.LongTensor([1, 0])] in MessagePassing.propagate() according to the direction flag.

Added to current master!

Hi,
Actually, I think the "source" and "target" notations are still misleading. From the doc "Constructs messages to node i in analogy to ϕ for each edge in (j,i)∈E if flow="source_to_target" and (i,j)∈E if flow="target_to_source"", but intuitively I would have done the opposite...

I think it is much more natural to think of i (or v) as the central node which aggregates and j (or w) as the neighboring nodes which get aggregated. This follows the convention of most GNN papers.

Hi @tformal, I was confused at beginning as well. But after checking the implementation, I think the better way of understanding the document could be the following: fix the notation i as the node to gather features to, and the flow flag determines how you map the notation i and j into the source and target nodes. If source_to_target, use j for the source node and i for the target node, and use the opposite otherwise.

Hi, thank you both for you answers !
@rusty1s : I totally agree with that, yet I was still confused (I think it was due to the way I interpreted "source" and "target")
@jzhou316  : thanks, I think it indeed helps to understand !


[String] Expunge _StringGutsSlice from ABI

_StringGutsSlice is an implementation helper that's no longer needed
from inlinable code. Internalize it.

@swift-ci please test and merge


Fix link to source code of AuthOptions

This PR

replace the current link to the source code of AuthOptions from the connect repo to the ux repo.
updates the optional flags in the properties table of AuthOptions

Thank you for fixing the link!
RE: the changes to AuthOptions. I know those fields are technically not required as per the Typescript definition, but I would lean more towards making them mandatory. You really need the finished and redirectTo options to fully implement authentication. My only hesitation to changing the TS definition is that it would be a breaking change. I definitely do not want our docs to indicate that these fields are optional, because you need them to implement auth.

LGTM! Thanks for updating the PR @friedger .

I'm unable to merge this. Someone with Vercel access can authorize the deploy, which should make all status checks pass. Or, someone with admin rights in this repo (maybe @agraebe ?) can merge.

Merged!


VIDEOIO ERROR: V4L: can't open camera by index 3

The camera is working with other tools like Cheese
Dell XPS 13 9370
v4l2-ctl --list-devices
Integrated_Webcam_HD: Integrate (usb-0000:00:14.0-5):
T:  Bus=01 Lev=01 Prnt=01 Port=04 Cnt=02 Dev#=  2 Spd=480 MxCh= 0
D:  Ver= 2.01 Cls=ef(misc ) Sub=02 Prot=01 MxPS=64 #Cfgs=  1
P:  Vendor=0bda ProdID=58f4 Rev=72.79
S:  Manufacturer=CN0FFMHCLOG0083EB4B2A01
S:  Product=Integrated_Webcam_HD
S:  SerialNumber=200901010001
C:  #Ifs= 4 Cfg#= 1 Atr=80 MxPwr=500mA
I:  If#= 0 Alt= 0 #EPs= 1 Cls=0e(video) Sub=01 Prot=01 Driver=uvcvideo
I:  If#= 1 Alt= 0 #EPs= 0 Cls=0e(video) Sub=02 Prot=01 Driver=uvcvideo
I:  If#= 2 Alt= 0 #EPs= 1 Cls=0e(video) Sub=01 Prot=01 Driver=uvcvideo
I:  If#= 3 Alt= 0 #EPs= 0 Cls=0e(video) Sub=02 Prot=00 Driver=uvcvideo
howdy -U test -y add
Adding face model for the user test
Using default label "Initial model" because of -y flag
Unable to stop the stream: Invalid argument
VIDEOIO ERROR: V4L: can't open camera by index 3
Please look straight into the camera
Traceback (most recent call last):
File "/usr/local/bin/howdy", line 90, in
import cli.add
File "/lib/security/howdy/cli/add.py", line 117, in
enc = face_recognition.face_encodings(frame)
File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 199, in face_encodings
raw_landmarks = _raw_face_landmarks(face_image, known_face_locations, model="small")
File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 153, in _raw_face_landmarks
face_locations = _raw_face_locations(face_image)
File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 102, in _raw_face_locations
return face_detector(img, number_of_times_to_upsample)
TypeError: call(): incompatible function arguments. The following argument types are supported:
1. (self: dlib.fhog_object_detector, image: array, upsample_num_times: int=0) -> dlib.rectangles
Invoked with: <dlib.fhog_object_detector object at 0x7f9ab0ea76f8>, None, 1

**Ubuntu 18.04 **:
**Howdy Latest **:

Nice work dude! Worked with device_id = 0
Thanx!

In other cases like this, a reboot can break it again. Let me know if it's the case here.

I will let you know ofcourse. But when a friend of mine try to login, it says, timeout and then get logged in... Is the a config that the face must be ok to get logged in....

After the "timeout" message, Howdy triggers a PAM_AUTH_ERR response, which should not log you in. Are you using other auth modules? Does a timeout log you in when no one is behind the laptop?


File descriptor limits

The goal is to avoid the possibility of the node crashing due to exhausting file descriptor limits.
We can use http://hackage.haskell.org/package/unix-<IP_ADDRESS>/docs/System-Posix-Resource.html#v:setResourceLimit to change soft & hard limits if the process is allowed to do so:
setSensibleOpenFilesLimit sensibleLimit = do
    limit@ResourceLimit{hardLimit} <- getResourceLimit ResourceOpenFiles
    setResourceLimit ResourceOpenFiles limit{softLimit = min sensibleLimit hardLimit}

We will also need to do an analysis and tests of how a node behaves when it is near its file descriptor limit. We'll probably need to count descriptors used per client and use that to have a client limit, leaving enough for shared descriptors.

For consensus, see #1396 and #1397.

This is done in #4585.


Comparison with existing or historical efforts.

https://github.com/vertelab/odoo-edi/issues/95 <-- this is our top hit so far. We have a video call set up with the author.
https://github.com/ElisevanM/Odoologic/issues/1 <-- will ask DutchWorld if Odoologic is open source, but so far it seems not.
https://github.com/OCA/edi <-- this is related to probably UBL and Zugferd, but I think it does not include Peppol.
https://www.odoo.com/nl_NL/forum/help-1/european-electronic-invoicing-179473

Odoo SA, the company behind Odoo, is an OpenPeppol member, not an Access Point.

https://apps.odoo.com/apps/modules/13.0/customer_invoice_peppol_willdoo/

This module accesses PEPPOL using services provided by the Pacific Commerce gateway.
The Odoo In App Purchase system is used to charge for access to the gateway on a per-invoice basis.

https://youtu.be/bs8GPYlHa9Y

Shows that Odoo supports UBL but it's done via PDF export/import.

https://factuur-portal.nl/verkoopfacturen-vanuit-odoo-via-peppol-versturen/

proprietary Odoo-to-Peppol bridge in NL?

https://willdoo.it/page/odoo-peppol-e-invoicing-application

proprietary Odoo-to-Peppol bridge in Australia/NZ?

https://odoobot.com/web/ecommerce/peppol/

proprietary Odoo-to-Peppol bridge in Singapore? In fact their domainname contains 'odoo' but the web page does not.

On the ERPNext side, I only found https://discuss.erpnext.com/t/e-rechnungspflicht-in-der-bundesverwaltung/66266/2

Had a very informative meeting with Anders from Vertel AB who gave us a lot of insight in the landscape and pointed us to https://github.com/vertelab/odoo-oca-edi . We are allowed to use that source code but it will probably serve mostly as an inspiration, since it was an adaptation of https://github.com/OCA/edi to make  the existing UBL composer/parser code compatible with Peppol-Billing 3.0, and the software was not architected with out-of-context reuse in mind.

Also, neither  https://github.com/OCA/edi nor https://github.com/vertelab/odoo-oca-edi support AS4, they only deal with the XML format for the invoices, not with the sending and receiving, network recipient discovery, identity checks, etc.

https://github.com/vertelab/odoo-edi supported Peppol bis 4 and bis 5 which are now both outdated, so not really useful to us nowadays.

It's possible that DutchWorld (the company behind https://github.com/ElisevanM/Odoologic/issues/1 ) may at some point open source their Peppol Billing 3.0 invoice composition code, that would be awesome! If they do, we will definitely use that in item 5 ("Odoo integration")
of our milestones.


Add unit tests for the omnibus invoke task rpath-edit

What does this PR do?
Add units tests of the rpath-edit invoke task we use in omnibus to overwrite the rpath of our builded binaries.
Motivation
We didn't put unit tests when making the task, fixing this misstep
Describe how to test/QA your changes
Possible Drawbacks / Trade-offs
Additional Notes

Also fixed a small bug where TemporaryFile created a file descriptor with name pointing to just the file name. We actually  needed NamedTemporaryFile to create a file descriptor with name pointing to the path of the file.

/merge

/merge


Request headers missing after page.setRequestInterceptionEnabled(true)

If I set page.setRequestInterceptionEnabled(true); some request headers disappear.
Consider the following code:
(async() => {

    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.emulate(devices['iPhone 6']);

    //await page.setRequestInterceptionEnabled(true);
    page.on('request', request => {
        //request.continue();
        console.log("URL: " + request.url);
        console.log("METHOD: " + request.method);
        console.log("HEADERS:" + JSON.stringify([...request.headers]) + "\n");
    });

    await page.goto('https://github.com/GoogleChrome/puppeteer', {waitUntil: 'networkidle'});
    browser.close();
})();

As sample, the first POST request catched :
URL: https://www.google-analytics.com/r/collect
METHOD: POST
HEADERS:[
    ["Cache-Control","max-age=0"],
    ["Origin","https://github.com"],
    ["Referer","https://github.com/GoogleChrome/puppeteer"],
    ["User-Agent","Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1"],
    ["Content-Type","text/plain;charset=UTF-8"]
]

If I uncomment await page.setRequestInterceptionEnabled(true); and request.continue(); the same request misses Referer header:
URL: https://www.google-analytics.com/r/collect
METHOD: POST
HEADERS:[
    ["Cache-Control","max-age=0"],
    ["Origin","https://github.com"],
    ["User-Agent","Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1"],
    ["Content-Type","text/plain;charset=UTF-8"],["Accept","*/*"]
]

This also happens for all the other requests. That I have notice only referer is missing.
Is this supposed to happen ?

That's right. Requests are created from different payloads depending whether the interception is enabled:

Without interception, we emit requests based on renderer notifications. Renderer doesn't know about "Referer".
With interception, we emit requests based on browser notifications. Browser is the one who injects the "Referer" header, so it has the "referer" header.

Does this bite you somewhere?

@aslushnikov This prevents proper testing of apps, since if you want to block/allow specific requests (PhantomJS whitelist/blacklist for instance) then the 'Referer' header is lost, preventing normal app behavior (redirect back for instance)

I'm on boad with @twalpole. If you want block some requests that doesn't matter, you end up losing information about the ones that are important.
Also, by the documentation I tought that requests content would only change if I overwrite something.

I think this might be related. I have a site I am trying to scrape that uses a redirect on their authentication route. It sends a POST request with the credentials, checks them and then does a 302 redirect with the set-cookie header containing the new session cookie.
With setRequestInterceptionEnabled the cookie is never set. Therefore when the scraper is redirected to the internal page of the app, it fails the cookie validation and is redirected back to the login page. This makes it impossible to scrape the site.

@andrewlinfoot I have the same issue with you. Unfortunately, the site still redirect me to their login page even I disable the setRequestInterceptionEnabled.
It works totally fine during my local development, but after I try to run the scraping script on a server, we always got redirect, seems like a IP related issues  : (

@andrewlinfoot would you mind filing an new issue that has your script and the URL you try to scrape?
The setRequestInterceptionEnabled should not affect cookies.

+1. I have the same issue, the Referer header is stripped when the request interception is turned on. That prevents the analysis of certain websites whose server relies on the presence of this Referer. At the very least, it should be documented because right now that's really a confusing behavior. Also, since I guess it's probably a Chromium bug, I filed this bug report: https://bugs.chromium.org/p/chromium/issues/detail?id=777392

Note that page.authenticate behind the scenes enables request interception.  So if you ever set page.authenticate and don't reset it to null, request interception will be enabled (even if the user calls page.setRequestInterceptionEnabled(false)) and the Referer header will be stripped

+1 to @twalpole comment.
I was using page.authenticate to provide proxy authentication credentials but this was stripping out some crucial headers such as Referer, Accept-Encoding and so on.
I updated my code to set page.authenticate back to null after proxy authentication like below:
...

const browser = await puppeteer.launch({
  args: ['--proxy-server=proxy.com:8080']
});

const page = await browser.newPage();

// proxy authenticate (will enable request interception)
await page.authenticate({ username: 'foo', password: 'bar' });

// trigger proxy authentication
await page.goto('https://www.google.com');

// reset authenticate to null (disable request interception)
await page.authenticate(null);

...

Cookie header is also missing if setRequestInterception is enabled
var puppeteer = require('puppeteer');

(async() => {

    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    await page.setRequestInterception(true);
    page.on('request', request => {
        console.log('cookie', request.headers.cookie);
        request.continue();
    });

    await page.goto('https://github.com/GoogleChrome/puppeteer', {waitUntil: 'networkidle2'});

    await page.evaluate(()=>{
        var date = new Date(new Date().getTime() + 60 * 1000);
        document.cookie = "foo=bar; path=/; expires=" + date.toUTCString();
    })
    await page.reload({waitUntil: 'networkidle2'});

    browser.close();
})();

Outputs:
cookie undefined
cookie undefined
cookie undefined
cookie undefined
cookie undefined
cookie undefined
cookie undefined
cookie undefined

Referer is no longer stripped from headers with request interception.
@Ndragomirov i can't see any change with your script with/without request interception. I'll close this for now; feel free to file another issue if you think there's a problem.

I am enabled to access the Cookie in request header if await page.setRequestInterception(true).
@Ndragomirov @aslushnikov @andrewlinfoot   Can you help me with this?
var puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch({ slowMo: 20,
 args: ['--enable-features=NetworkService','--disable-setuid-sandbox','--no-sandbox',],
 headless: false,
});
let page = await browser.newPage();

await page.goto('https://example.com/);
await page.waitForSelector("input[name='username']");
await page.type("input[name='username']",'tok_sandbox_knkjdq',);
await page.keyboard.press('Tab');
await page.type("input[name='password']",'tok_sandbox_so,ldmkdw',);

await page.setRequestInterceptionEnabled(true);

page.on('request', async (interceptedRequest) => {
   const tunnelingAgent = tunnel.httpsOverHttp({vgs outbound tunnel});
   const options = {
     uri: interceptedRequest.url(),
     method: interceptedRequest.method(),
     headers: interceptedRequest.headers(),
     agent: tunnelingAgent,
     body: interceptedRequest.postData(),
   };

   console.log(interceptedRequest.headers().cookie); // Not able to  access cookie in request header

   const res = await request(options, function (err, resp, body) {
     if (err) {console.log(err);}
     interceptedRequest.respond({
       contentType: resp.headers['content-type'],
       headers: resp.headers,
       body: body,
     });
   });
 }else {interceptedRequest.continue();}
});
await page.keyboard.press('Enter');

})();


github.com/fuse resource causes pod to be retried indefinitely

Problem desc:

By default, every node has 20 volumes of github.com/fuse.
When the github.com/fuse resource on a node is used up, hived still try to bind the pod with that node.
The binding will be rejected by kubelet. Error message is:

Update plugin resources failed due to requested number of devices unavailable for github.com/fuse. Requested: 1, Available: 0, which is unexpected.

The pod will be retried indefinitely.

Work around: https://github.com/microsoft/pai/blob/1a641c0272e827f0e8c34e5d0decc4151e3009eb/src/device-plugin/deploy/device-plugin.yaml#L11


How to start with log

I start and add param -l --log to a file but the file was empty and log nothing after running

You need to increase log verbosity level also, using the parameter -v/--verbosity.

You need to increase log verbosity level also, using the parameter -v/--verbosity.

It's ok using your answer,thanks!


make the activity

I attempted to run the program on the XO but my access was denied when I tried to run setup.py

@smendez93 What error message did you get?