[SPARK-5036][Graphx]Better support sending partial messages in Pregel API

Better support sending partial messages in Pregel API
1. the reqirement
In many iterative graph algorithms, only a part of the vertexes (we call them ActiveVertexes) need to send messages to their neighbours in each iteration. In many cases, ActiveVertexes are the vertexes that their attributes do not change between the previous and current iteration. To implement this requirement, we can use Pregel API + a flag (e.g., bool isAttrChanged) in each vertex's attribute.
However, after aggregateMessage or mapReduceTriplets of each iteration, we need to reset this flag to the init value in every vertex, which needs a heavy joinVertices.
We find a more efficient way to meet this requirement and want to discuss it here.
Look at a simple example as follows:
In i-th iteartion, the previous attribute of each vertex is Attr and the newly computed attribute is NewAttr:

VID
Attr
NewAttr
Neighbours

1
4
5
2, 3

2
3
2
1, 4

3
2
2
1, 4

4
3
4
1, 2, 3

Our requirement is that:

Set each vertex's Attr to be NewAttr in i-th iteration
For each vertex whose Attr!=NewAttr, send message to its neighbours in the next iteration's aggregateMessage.

We found it is hard to implement this requirment using current Pregel API efficiently. The reason is that we not only need to perform pregel() to  compute the NewAttr  (2) but also need to perform outJoin() to satisfy (1).
A simple idea is to keep a isAttrChanged:Boolean (solution 1)  or flag:Int (solution 2) in each vertex's attribute.
2. two solution

2.1 solution 1: label and reset isAttrChanged:Boolean of Vertex Attr

init message by aggregateMessage
it return a messageRDD

innerJoin
compute the messages on the received vertex, return a new VertexRDD which have the computed value by customed logic function vprog, set isAttrChanged = true

outerJoinVertices
update the changed vertex to the whole graph. now the graph is new.

aggregateMessage. it return a messageRDD

joinVertices  reset erery isAttrChanged of Vertex attr to false
//	here reset the isAttrChanged to false
g = updateG.joinVertices(updateG.vertices) {
  	(vid, oriVertex, updateGVertex) => updateGVertex.reset()
	}

here need to reset the vertex attribute object's variable as false

if don't reset the isAttrChanged, it will send message next iteration directly.
result:

Edge: 890041895
Vertex: 181640208
Iterate: 150 times
Cost total: 8.4h
can't run until the 0 message

solution 2. color vertex

iterate process:

innerJoin
vprog using as a partial function, looks like vprog(curIter, _: VertexId, _: VD, _: A)
 i = i + 1; val curIter = i.
in vprog, user can fetch curIter and assign to falg.
outerJoinVertices
graph = graph.outerJoinVertices(changedVerts) { (vid, old, newOpt) => newOpt.getOrElse(old)}.cache()
aggregateMessages
sendMsg is partial function, looks like sendMsg(curIter, _: EdgeContext[VD, ED, A]
in sendMsg, compare curIter with flag, determine whether sending message

result
raw data       from

vertex: 181640208
edge: 890041895

iteration average cost
150 iteration cost
420 iteration cost

solution 1
188m
7.8h
cannot finish

solution 2
24
1.2h
3.1h

compare
7x
6.5x
finished in 3.1

the end
i think the second solution(Pregel + a flag) is better.
this can really support the iterative graph algorithms which only part of the vertexes send messages to their neighbours in each iteration.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

test this please

Test build #25029 has started for   PR 3866 at commit 2d2195b.

This patch merges cleanly.

Test build #25029 has finished for   PR 3866 at commit 2d2195b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25029/
Test FAILed.

test this please

ok to test

Test build #25034 has started for   PR 3866 at commit 4f3c95a.

This patch merges cleanly.

Test build #25034 has finished for   PR 3866 at commit 4f3c95a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25034/
Test FAILed.

Test build #25042 has started for   PR 3866 at commit a9668db.

This patch merges cleanly.

Test build #25042 has finished for   PR 3866 at commit a9668db.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25042/
Test FAILed.

Test build #25045 has started for   PR 3866 at commit 1cb8770.

This patch merges cleanly.

Test build #25045 has finished for   PR 3866 at commit 1cb8770.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25045/
Test FAILed.

@ankurdave @rxin

Test build #25098 has started for   PR 3866 at commit 6da1939.

This patch merges cleanly.

Test build #25098 has finished for   PR 3866 at commit 6da1939.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25098/
Test FAILed.

Test build #25099 has started for   PR 3866 at commit b584ac4.

This patch merges cleanly.

Test build #25099 has finished for   PR 3866 at commit b584ac4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25099/
Test PASSed.

I'm not sure I understand the problems you're trying to solve in this PR. Is this an accurate summary:

Activeness based on whether a vertex attribute changed from one superstep to the next (not implemented in the code yet?). An alternative to using an isAttrChanged flag could be to use a VertexRDD join such as VertexRDD#diff.
Arbitrary convergence tests based on iteration number and number of active messages (implemented using isTerminal). What is the use case for this?
Optional initial message (implemented by making initialMsg an Option).

@ankurdave thanks for reviewing :)

Activeness is from innerJoin's result by vprog the message of pre iteration.isAttrChanged is from solution1.  It can run, but the key problem is after 100 or more iteration, it's progressive increase one second every iteration. So, I consider the solution 2.   Solution 2 is surprised on both performance and code logic is more clear.
take k-core algorithm for example. The coreness from 4 to 3 need iterating 68 times, from 3 to 2 need hundreds.  But the sum coreness  of  4, 2, 1 in the final result is not important, or user can ignore the effect of lacking computing coreness 4, 3, 2,1. So they want't to finish the loop ahead of time.
isTerminal  provide a custom condition of terminal by current iterate id curIter and messageCount.
Default isTerminal is true.
making initialMsg an Option, as sometimes the initalMsg is empty. when  initalMsg is empty, aggregate message on every Vertex directly.

@ankurdave
Solution 1 is better. It have to add joinVertices  for reset isAttrChanged to default value.
It have two problem having no idea for solving:

Progressive increase one second every iteration.
We can't accept one second progressive increased, because after 600th iteration, the extra cost will become 600s. It like this:

iteration
increace
cost

100
1
150

101
1
151

102
1
152

103
1
153

104
1
154

..
1
..

700
1
850

701
1
851

703
1
852

704
1
853

2  joinVertices
joinVertices will own half cost of the whole Pregel-liked cost.
The more iteration the bigger of joinVertices. It's a linear increase.

@rxin @ankurdave have any other problem?

Can one of the admins verify this patch?

cc @andrewor14

Can one of the admins verify this patch?


Reduce typo count.

Thank you very much for finding these! Back then, I didn't use a spell checker on my docstrings 😅

Sorry about the build failures... I'll get that fixed.

@mgeisler See PR #560


doc: GraphQL API gen

What type of PR is this?
doc
What this PR does / why we need it:
Auto generate documentation from GraphQL API
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Please add more description of field in the code.


Add a delay time for blunderbuss to wait for sig label to be added.

Feature Request

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Is your feature request related to a Issue or PR(Optional)
None
Describe the feature you'd like:
Add a delay time for blunderbuss to wait for sig label to be added.
Do you have any ideas on how to implement it?(Optional)

/assign @Mini256
/milestone v0.16.5

I think it can be made a configuration item.

How long is the default delay appropriate?

How long is the default delay appropriate?

5 second.

Do we need to wait for the delay time, when the event triggered by the /auto-cc command?

Do we need to wait for the delay time, when the event triggered by the /auto-cc command?

No need. Because it is manually triggered.


Invalid URL

Try to get vk.com url and generates a invalid url inone machine and valid in other.
youtube-dl -g  https://vk.com/video-51189706_456244
634 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-g', u'https://vk.com/video-51189706_456244634', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.26
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-20-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: none
[debug] Proxy map: {}
[debug] Default format spec: bestvideo+bestaudio/best
https://cs6-8v4.vkuservideo.net/video/hls/p18/14f300a3c3c5/index-f2-v1-a1.m3u8?extra=RlT5yNqZ9KxoU-4nAtWVUGvD
WzzospqE-nVN_dXfjBq67JndnJKhMy9fxtyRgd0fBz8XcZKqMS1cmCTBkIWJFHCDPY-iJw46u44-F_W9JIY0UisNqfpvpc1fLlZqVSEC1P1we
s4RUZ2_0Q
I try the same command in other machine and works just fine.
youtube-dl -g  https://vk.com/video-51189706_456244634 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-g', u'https://vk.com/video-51189706_456244634', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.26
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-34-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: none
[debug] Proxy map: {}
[debug] Default format spec: bestvideo+bestaudio/best
https://cs6-8v4.vkuservideo.net/video/hls/p18/476d34e200fe/index-f2-v1-a1.m3u8?extra=vjNW9POhFcqoaSE_7NIWKL_V4zagIwd9-AtfMCLvLrfluxEpDJCo-haseCNQML_UljDYSWc9h2NnA1OuD0t7BE33QbQkwIXJrg11GafSuX6kBCJ26hSZuEDbIpfUAQWcKPx39VGqLalN
What is the problem?

What exactly do you mean by invalid URL?

the first url in my comments dosent works, but the second yes and is the same video for both.

Again: what exactly do you mean by "does not work". Post the output of youtube-dl https://vk.com/video-51189706_456244634 from this machine.

vk uses hotlink protection maybe because of that

https://github.com/rg3/youtube-dl#i-extracted-a-video-url-with--g-but-it-does-not-play-on-another-machine--in-my-web-browser

Case 1 (dosent works, you cannot download the video from the url)
youtube-dl -g  https://vk.com/video-51189706_456244634
https://cs6-8v4.vkuservideo.net/video/hls/p18/14f300a3c3c5/index-f2-v1-a1.m3u8?extra=RlT5yNqZ9KxoU-4nAtWVUGvD
WzzospqE-nVN_dXfjBq67JndnJKhPCddxtyRgd0fBz8XcZKqMS1cmCTBkIWJFHCDPY-iJw46u44-F_W9JIY0UisNqfpvpc1fLlZqVXoF0vdzK
50dXZzr1w
Case 2 (WORKS you can download the video from the url)
youtube-dl -g https://vk.com/video-51189706_456244634
https://cs6-8v4.vkuservideo.net/video/hls/p18/476d34e200fe/index-f2-v1-a1.m3u8?extra=vjNW9POhFcqoaSE_7NIWKL_V4zagIwd9-AtfMCLvLrfluxEpDJCo9R6ueCNQML_UljDYSWc9h2NnA1OuD0t7BE33QbQkwIXJrg11GafSuX6kBCJ26hSZuEDbIpfUBQCUeft1ogr8f61M

@lachuza You could not download the extracted url from another machine/ip because VK uses hotlink protection which allows to access from the same machine/ip


"No Stream Available"

Please follow the guide below

Put an "x" into all boxes [ ] relevant to your issue
Use Preview tab to see how your issue will actually look like

Kodi & Addon

Kodi verison: 7.1___
[ ] Filmy online - Mrknow
[ ] MrknowTV
[x ] Specto
[ ] FanFILM
Version / Wersja: _____

Descryption / Opis
Every title I select it say there is no stream available.  Not quite sure how to fix.  Please advise on this issue.  Thanks.
Donation / Darowizna
This addons are free, but you can always thank the author for ... / Te wtyczki są darmowe, ale zawsze możesz podziękować autorowi za jej stworzenie...

I have also noticed this where if I have autoplay enabled it says no stream available. If I have a list of links it plays a good link fine.


Problem compiling FaceBoxes

After trying a couple of times, I find impossible to correctly compile the file ./build_cpu_nms.sh. The error is:
cl : Command line error D8021 : **invalid numeric argument '/Wno-cpp'**
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.34.31933\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

I am not sure if this could be related to the operating system. I am running the code in Visual Studio Code on Windows 11.
Looking after the error I found this. So I changed in build.py inside FaceBoxes/utils the line of code 47:
from
extra_compile_args=["-Wno-cpp", "-Wno-unused-function"] to
extra_compile_args={'gcc': ['/Qstd=c99']}
After running this, I get the following warnings:
cl : Command line warning D9024 : unrecognized source file type 'gcc', object file assumed
cl : Command line warning D9027 : source file 'gcc' ignored

From where I assume that is basically not compiling the extra modules, but ignoring it. So, it does not show and error, but as expected, the code does not work. The error raises in FaceBoxes\utils\nms_wrapper.py :
  File "nms\cpu_nms.pyx", line 25, in nms.cpu_nms.cpu_nms
ValueError: Buffer dtype mismatch, expected 'int_t' but got 'long long'

I also tried commenting line 47 - in build.py inside FaceBoxes/utils - and uncommenting line 46, which raises the same error.
Any ideas how to solve this?
Many thanks.

Just the same for me

@RichestYoo I am pretty sure I have solved the problem adding this line of code:
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
Let me know

@pepeballesterostel
Where am I supposed to put that line in?
I put that in FaceBoxes/utils/build.py like the following failed:
ext_modules = [
Extension(
"nms.cpu_nms",
["nms/cpu_nms.pyx"],
extra_compile_args=["-Wno-cpp", "-Wno-unused-function"],
include_dirs=[numpy_include],
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
)
]
And "TRUE", instead of True seems to say that it's not a python source.

Put it in the main script that you are running,  for example singleImage.py. I added it after the imports.
The command basically tells python that is ok to have multiple copies of the library MKL loaded into memory, which seems to be the initial compiling error.

@RichestYoo
Put it in the main script that you are running, for example singleImage.py. I added it after the imports.
The command basically tells python that is ok to have multiple copies of the library MKL loaded into memory, which seems to be the initial compiling error.

@pepeballesterostel
Thank you for your help, but it didn't work out for me.
After first failure, I put that in the environment variable as "os.environ" is bringing/setting PC's environment variable,
and tried again, but same error message came up.

@RichestYoo
Try changing your code from  build.py to this code:
ext_modules = [
Extension(
"nms.cpu_nms",
["nms/cpu_nms.pyx"],
#extra_compile_args={'gcc': ["-Wno-cpp", "-Wno-unused-function"]},
extra_compile_args=['std=c99'],
include_dirs=[numpy_include]
)
]
Does this work for you?

@pepeballesterostel
With that, I passed the build failure you wrote at the beginning but now I'm at this "int_t" vs. 'long long' stuff...

@pepeballesterostel
Sure. Thank you for your sincere helps.

@RichestYoo
did you managed to get it working?
i have added extra_compile_args=['std=c99'], to FaceBoxes build.py and os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"  to singleImage.py butstuck at 'int_t' but got 'long long' problem.

Thank you for your help, but I'm transferred to another job and is not working on this any more.


cryptsetup (LUKS), LVM2, BTRFS, XFS, NBD, gptdisk,

Needed to switch back to GLIBC (uClibc gave 'segfault error 6 in libdevmapper' on cryptsetup open).
Using in production with v0.4.5 Rancher OS since about 5 months. Completely diskless statless iPXE booted, considering physical machine just like a pool of resources with just docker containers and volumes on encrypted BTRFS disk. Now I got necessity to have Rancher OS state partition on the same encrypted BTRFS as subvolume. And gonna to check at the end of rancher.bootstrap if rancher.state.dev specified but not mounted. And if so then to bring temporary shell via console and ssh (for possible remote unlocking of LUKS encrypted via cryptsetup or helper command). I am gonna to implement it within existing rancher.bootstrap.bootstrap service (within os-bootstrap image) or to add new service (with new image also os-base based) to same rancher.bootstrap service set. This service will complete/exit to continue normal boot sequence once special helper command (e.g. continue_boot) will be called there in shell (by admin or e.g. by remote script to simple unlock LUKS encrypted state partition or to push some remote binary and perform some more complex security checks, checksummig etc before unlocking - up to admin inventiveness). There could be implemented few another helper commands like unlock (which would automate process of unlocking state partition by checking whether it is specified by device path or by label and if by label then to scan and start mdadm and lvm devices first and then to check all block devices if any is LUKS encrypted and if so then to ask for passphrase. And another to initiate NBD device (in case of totally diskless machine, e.g. like scaleway.com real case I use btw, but without rancher for a while and bringing  rancher os to there would be a big win IMO BTW). Plus all flexibility of buildroot environment up to admin, so configuration and unlocking of any complexity can be performed by admin or automated remote script.
And actually any other arbitrary storage related checks/configs/unlocks can be pushed as extra service to rancher.bootstrap service set and based on same os-bootstrap (or our new service and image specially for it), via cloud-config specified in kernel arg or on OEM partition or so. Freedom and flexibility and at the same time simplicity are absolute incomparable with other distros
Please let me know in case of any advice or any other job done in this direction already.

We'd like not to switch to glibc - and we're not really sure of the side effects of adding this to the default distro. We need an idea of what i'll take to test this kind of thing for the future :/

We'd like not to switch to glibc
Reasons? Just a quick higlight please.
and we're not really sure of the side effects of adding this to the default distro
In't glibc much more mature and tested with much more kernels and software since used in majority of general purpose linux distros (where most of software is really used and tested)? So far I know uClibc is quite younger and used primarily in embeddted systems for its smaller size. Please correct me if I am wrong. As proof of my opinion - a simple fact that cryptsetup does not work with uClibc and no working solution I have found for a while. From the point of maximal reobustness, stability and eliminating of side effects - I would even switch os-base from buildroot to alpine which is also very tiny based on musl but where much more software and kernel modules/features are well tested/proven. At the same time it would give much more opportnnities for end users to customize management console and even initrd image keeping it tiny without switching to ubuntu/debian/centos console. In this case IMO, we would get best from both - your unique idea to use docker as actually first class sitizen on top of kernel and from already using of quite matures, secure, well tested and proven very compact minimalistic distro. If I will not hear from you weight reasons agains alpine in our case - I will find some time to make test build replacing buildroot with it and will see if it will pass all current tests or will create any issues. If none - I will try to switch to it in my production.

#31

Now cryptsetup/LUKS with SSH remote unlock for state partion is fully supported in my fork including LVM, BTRFS, XFS, NBD and works in my production

I'm presuming that to get it all working, we'd need to merge some version of your tetatetit/os master branch?
we're not ignoring your change, its just a large amount to digest and decide on :/

Just made 2 separate PRs to rancher/os to get all this consistent and fully working, just as #joshwget adviced to split my initial big PR. I hope it will simplify your decision making process :). Just stating that all of this except NBD working perfectly in my production, no issues so far. And I that tried to be extremely careful to make changes to do not affect anything else

Also I plan to make test branch using alpine instead os-base with included all above plus lvm thin, iSCSI and ZFS out of the box. This should make Rancher OS deployment and storage options even more advanced then in so mature distros like Ubuntu, Fedora, openSUSE, not even saying about CoreOS. Also once there features will be inlcuded in your upstream repo - I'll try to convince Scaleway.com to include Rancher OS as one of their default images which should open there possibility of all these feautres which are unavailable otherwise on their service even with custom image (due of only network boot and only storage as NBD). So Rancher OS the only way to enable all these features without chaning their architecture


Major refactor

I felt like writing some Rust this evening. This change should make things flow better and really separate the concerns between files.
The coolest changes are definitely in phidgets.rs:

pub trait PhidgetTraits
impl PhidgetTraits for PhidgetHandle

...since you can implement a Trait on an f64, nothing stops me from implementing a Trait on something that came from bindgen, the PhidgetHandle.
Since any of the more specific Phidget structs can be casted to a generic PhidgetHandle, once you cast, the Trait is available. You can see examples of this in the setup() functions.

I did end up going back to clap because it takes care of a lot of the boilerplate argument parsing I'm used to from Go's flag library.


Tests from_checksum_format() on filepaths with leading 's

Confirms from_checksum_format()'s handling of leading-backslash filepaths, like those in the last couple lines here:

Does this actually merit a changelog entry?


[SPARK-25922][K8S] Spark Driver/Executor "spark-app-selector" label mismatch (branch-2.4)

In K8S Cluster mode, the algorithm to generate spark-app-selector/spark.app.id of spark driver is different with spark executor.
This patch makes sure spark driver and executor to use the same spark-app-selector/spark.app.id if spark.app.id is set, otherwise it will use superclass applicationId.
In K8S Client mode, spark-app-selector/spark.app.id for executors will use superclass applicationId.
Manually run.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7914/
Test PASSed.

Test build #102316 has started for PR 23779 at commit bff23c8.

Test build #102316 has finished for PR 23779 at commit bff23c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/102316/
Test PASSed.

hmm, no k8s tests?

retest this please

Test build #102354 has started for PR 23779 at commit bff23c8.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7945/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7945/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7945/
Test PASSed.

Test build #102354 has finished for PR 23779 at commit bff23c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/102354/
Test PASSed.

All good now. Merging to 2.4.


feat: Update array, tuple, struct to have complex types

Update abi encoder to handle complex types and update tests

@camsjams Thanks! taking a look


Allow using shorthand types in other shorthands

For example:
exports.shorthands = {
  ref: {type: `integer`, onDelete: `cascade`},
  user: {type: `ref`, references: `users`},
}

This also currently doesn't work if the shorthand was defined in an earlier migration file.

Hmm, it can be to complicated and can lead to cyclic definition.
I can suggest to use
const ref = {type: `integer`, onDelete: `cascade`}
exports.shorthands = {
  ref,
  user: { ...ref, references: `users`},
}

If I submitted a PR that checked for a cyclic definition would you consider it?

 I'm used to tickets lasting for a little while.
Thanks!


Refactor mirror code & fix StartToMirror (#18904)

Backport #18904.

The potential break is when those instances have tasks in the mirror disk queue when upgrading.

I know ... just oint that up to admins on release info


Support installation-source for repository

We are not sure, if there is another way, to support installation-source for repositories.
Ideally, it should always be dist, so we added the form field for the repository to overwrite the value.
Since we had the problem, that source path was used,  even though we had defined preferred-install: dist.
If there is a better solution to always use the dist path, then this PR can also be discarded.

done ;-)


make error

/usr/local/include/g2o/core/base_fixed_sized_edge.hpp:179:14: error: ‘FixedArray {aka class ceres::internal::FixedArray<double, 3>}’ has no member named ‘fill’
add_vertex.fill(0.);

这个要检查一下g2o装好了没


Issue with specifying version for themes (Go modules) in themes.txt

If I add my theme with github.com/frjo/hugo-theme-zen (no version part) the preview does not include the Zen theme at all.
If I add my theme with github.com/frjo/hugo-theme-zen/v3 (with version part) the preview displays the Zen theme in order but with an incorrect path for the download button. It sets the download path to include the version part making the download link end in a 404.
There seems to be a bug with setting html_url for themes with a version. I suspect it works for some theme due to the correct html_url being cached.
Originally posted by @frjo in https://github.com/gohugoio/hugoThemesSiteBuilder/issues/453#issuecomment-2074554583

PR #435 is also related to this issue.

Yea, I see the problem, I will fix it.

Fixed now as far as I can see, thanks!
https://github.com/gohugoio/hugoThemesSiteBuilder/pull/455

Yea, I see the problem, I will fix it.

Thanks for the fix.


Content Updates

@mattroush @tskillern Do you want me to make these changes?

HOME PAGE
https://essential.cu.edu/impact-reports/onward
1. Under CU philanthropy:
Change dash to a colon: “Your generosity creates impact: bright futures for...”

2. Under Our Mission:
The word "donor" should be "donors"

FINANCIALS page
https://essential.cu.edu/impact-reports/onward/financials
1. In the intro:
Add a space on each side of both dashes.

2. In the 55,450 gifts paragraph:
Change the dash to a colon.

3. Under the COVID/DEI section:
a. Change both instances of "COVID" to "COVID-19"
b. Change "since March 1" to "from March 1 to June 30"
c. Change "since January 1" to "from January 1 to June 30"

4. In the You’re looking ahead paragraph:
Replace the dash with a space.

4. In $1.52 billion paragraph:
Change $73 million to $75 million:

Alt text on financials page
Also on the financials page, I think we should change the existing alt text for the graphs/charts that appear in the second half. One thing I didn’t anticipate was that because the graphs/charts are images, there is no literal text, so a page reader would not translate what’s in the image. Thus, we need to include the actual numbers. Here are the replacement alt text values for each:

$1.52 billion chart alt text:

“This is a bar chart that shows the value of the CU endowment by year. Here are the values: $1.09 billion for 2015, $1.06 billion for 2016, $1.22 billion for 2017, $1.36 billion for 2018, $1.45 billion for 2019 and $1.52 billion for 2020.”

How the endowment is invested alt text:

“This is a chart that shows how the LTIP is invested in asset classes. 46.3% for global public equities, $24.7% for global private capital, 13.1% for global hedge funds, 8.7% for real assets and 7.2% for fixed income and cash.”

Investment return vs. policy benchmark alt text:

“This is a dual bar chat that compares the LTIP and benchmark returns. Here are the percentage returns: 3 year for LTIP is 7.2%, 3 year for benchmark is 6.2%; 5 year for LTIP is 7.9%, 5 year for benchmark is 7.5%; 10 year for LTIP is 8.4%, 10 year for benchmark is 8.9%; 15 year for LTIP is 7.4%, 15 year for benchmark is 6.1%.”:

$209 million alt text:

“This is a waffle chart that shows where support from the CU Foundation to CU goes, by percentage. Academic Support 36%; Capital Projects 12%; Research 12%; Scholarships 12%; Advancement support 12%; Chairs, Professorships and Other Faculty Support 10%; Public Service, Administration, Library and Other Support 5%; and Athletics 1%”

@alexfinnarn yes, pls make these changes. Thanks.


Details page gets extremely slow with many and large relationships

Checklist

[X] The bug is reproducible against the latest release or master.
[X] There are no similar issues or pull requests to fix it yet.

Describe the bug
We have a model with quite a few (maybe 10) relationships (some one-to-many, some many-to-one). When showing the details for such an object, all those relationships are joinedload:ed:
https://github.com/aminalaee/sqladmin/blob/fadf4d6a91143f877c2a7053879202289b163015/sqladmin/models.py#L804-L805
This causes a combinatorial explosion when those relationships get bigger. Let's say one relation has 10 objects, another has 15 and a third has 37. This causes the resulting query to return 10*15*37 = 5550 rows! Doing that with three separate queries (selectinload or just default lazyload) would load the expected 1+10+15+37 = 63 rows.
This seems to have been introduced in https://github.com/aminalaee/sqladmin/pull/212.
Changing the joinedload on L805 to selectinload or simply removing L804-805 resolves the issue in my case. If the joinedload is really required to prevent regressions, maybe we could have a way to override what type of eager load is used?
Steps to reproduce the bug
No response
Expected behavior
No cross product between relations in detail view.
Actual behavior
No response
Debugging material
No response
Environment
SQLAdmin 0.16.0
PostgreSQL 14.7
Additional context
No response

I don't think this will cause a regression, but I'm thinking there will be some limitations with going to selectionload as joinedload is more suitable for Many-To-One relationships and there's a chance we are running in circles between the options because we have all possible cases of relationships.
I will create a PR to test this and see if people have any feedback on it.


Added the dependency librsvg to enable svg to pdf conversion plugin.

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

Jinja2 variable references are suggested to take a {{<one space><variable name><one space>}} form. See lines [8].

I'm sorry, not entirely sure why this build would be cancelled. Do you have any suggestions of what I'm missing?

In Travi CI, I also get: Format: "png" not recognized. Use one of:
Does anyone know why this is?

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

Hey!
Is there any chance this will be merged, or does the librsvg dependency fall out of the scope of the project?

@conda-forge-admin please rerender

Thanks for the work! This got handed recently with PR ( https://github.com/conda-forge/graphviz-feedstock/pull/59 ). Please give that a try and let us know how it goes 🙂


Remove {seedlotNumber} in upload endpoints

Waiting for @thiagosanchezenc to get back so we can confirm that this number will not be used in the backend.

Confirmed with @thiagosanchezenc, we can safely remove it.


Add Sepolia config

Motivation
Add Sepolia config for upcoming genesis in Sunday, June 19, 2022 2:00:00 PM +UTC

From https://github.com/eth-clients/merge-testnets/pull/14

Description
Closes https://github.com/ChainSafe/lodestar/issues/4160

REALLY YOU BEAT ME BY 32 SECONDS

REALLY YOU BEAT ME BY 32 SECONDS

We are really connected tho :heart:


Tradfri configuration failed in HASSIO 0.57.2 on RPi3

Version info:
Running HASSIO 0.57.2 on Raspberry PI3
Configuring TRADFRI lights after discovery fails. No visual feedback in UI. Only when checking home-assistant.log, I see the error reported below. This is a HASSIO installation without any configuration changed, apart from setting up SSH with keys, and Samba.
Expected:
HASSIO to accept security code and lights to show up.
(I'm aware of differences in i, l, 1 and o, O, 0. The entered security code is 100% correct. The result of an incorrect and a correct code differs. See testcases below).
Testcases

Entering incorrect code floods home-assistant.log. Again, with NO visual feedback in UI, with the result of filling up the SD card. (many many messages per second. )
Entering correct code results in single error below.

Additional info
Previously, I used HASSIO on RPi2, this worked flawlessly.
I also saw #10362. This concerned 0.57.1, and was closed. The solution in that issue was: "reset hub, and re-add all lights". This is an unworkable solution. I have many lights, and would not like to have to remove everything with every new release of hassio.
Steps to reproduce:

install hassio on RPi2, configure lights -> working flawlessly.
shutdown RPi2.
install hassio on Rpi3. configure lights -> errors. See below.

Logs:
When entering INCORRECT security code:
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.
2017-11-10 07:52:31 ERROR (MainThread) [coap] Fatal DTLS error: code 20
2017-11-10 07:52:31 ERROR (MainThread) [coap] Exception CancelledError() can not be represented as errno, setting -1.

When entering CORRECT security code:
2017-11-10 07:27:55 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/tasks.py", line 179, in _step
    result = coro.send(None)
  File "/usr/lib/python3.6/site-packages/homeassistant/components/tradfri.py", line 66, in configuration_callback
    psk = yield from api_factory.generate_psk(callback_data.get('key'))
  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 193, in generate_psk
    self._psk = yield from self.request(command)
  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 149, in request
    result = yield from self._execute(api_commands)
  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 141, in _execute
    api_command.result = _process_output(res, parse_json)
  File "/usr/lib/python3.6/site-packages/pytradfri/command.py", line 71, in result
    self._result = self._process_result(value)
  File "/usr/lib/python3.6/site-packages/pytradfri/gateway.py", line 28, in process_result
    return result[ATTR_PSK]
TypeError: 'NoneType' object is not subscriptable

I'm seeing the same error

Although I'm on a virtual env on the pi3, I experience the same. Maybe the logging helps to identify the problem:
File "/srv/hass3/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 193, in generate_psk
INFO:coap:Retransmission, Message ID: 18692.

I'm also on a pi3 and get the same error, been having the same issues for a couple of versions now:
2017-11-10 15:35:58 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last):   File "/usr/lib/python3.6/asyncio/tasks.py", line 179, in _step     result = coro.send(None)   File "/usr/lib/python3.6/site-packages/homeassistant/components/tradfri.py", line 66, in configuration_callback     psk = yield from api_factory.generate_psk(callback_data.get('key'))   File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 193, in generate_psk     self._psk = yield from self.request(command)   File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 149, in request     result = yield from self._execute(api_commands)   File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 141, in _execute     api_command.result = _process_output(res, parse_json)   File "/usr/lib/python3.6/site-packages/pytradfri/command.py", line 71, in result     self._result = self._process_result(value)   File "/usr/lib/python3.6/site-packages/pytradfri/gateway.py", line 28, in process_result     return result[ATTR_PSK] TypeError: 'NoneType' object is not subscriptable

this issue is probably caused by a reuse of the (currently hardcoded) ID.
this would be fixed by pull/10414
the current workaround is, as stated, reset and repair.
another fix would be to migrate the .tradfri_psk.conf to a new install.
hth

Ok, surrendered. Reset the hub, and re-added all the lights. Configured in HASSIO, and voila, up and running again. Hopefully a once in a lifetime! :)

Well..... this is getting annoying.... For some reason (don't know exactly how I did it, but it happened while changing the yaml files with previously backed up yaml file and restarting), I had the problem of the flooding home-assistant.log file again, resulting in an unusable homeassistant. So I re-installed the Pi3 with a fresh image and.... had to go through the very lenghty process of resetting the tradfri hub and re-adding all lights, because I had the initial error "TypeError: 'NoneType' object is not subscriptable" again.
Please give me and others a fix for this. If this keeps happening, I'll be forced to leave Home Assistant, and I don't want to!

Same issue here rpi3 - will try resetting g/w

I have the same errors in my log. I just tried to reset the tradfri gateway, but I can still not pair it with homeassistant. Any other way to bring it back up, since I do not have the old .tradfri_psk.conf file anymore.

I have the same issue - no feedback from the GUI and the log shows:
TypeError: 'NoneType' object is not subscriptable
When will this be fixed?

Same issue found at my setup. Even after resetting the gateway there's no way to add the security key. And log shows:
Error doing job: Task exception was never retrieved
etc.
Running Hassio on a newly installed RPi3.
Anyone know a fix?

#10630 should fix that. it has #10414 in it, which takes care of re-authentication with random IDs.
please try the new version, and report if that does/doesn't fix your issues.

A big thanks to everyone who got this fixed, working for me in 0.58.1.

It's still a problem for me, sadly.
I just reinstalled Raspbian and then provisioned it with my Ansible repo: https://github.com/atwright147/raspbian-homeassistant
I am still getting the following error below. Am I missing something?
2017-11-26 16:50:38 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/homeassistant/core.py", line 1031, in _event_to_service_call
    yield from service_handler.func(service_call)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/homeassistant/components/configurator.py", line 224, in async_handle_service_call
    call.data.get(ATTR_FIELDS, {}))
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/homeassistant/components/tradfri.py", line 74, in configuration_callback
    key = yield from api_factory.generate_psk(security_code)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 193, in generate_psk
    self._psk = yield from self.request(command)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 149, in request
    result = yield from self._execute(api_commands)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 141, in _execute
    api_command.result = _process_output(res, parse_json)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/pytradfri/command.py", line 71, in result
    self._result = self._process_result(value)
  File "/var/local/homeassistant/prod_venv/lib/python3.5/site-packages/pytradfri/gateway.py", line 28, in process_result
    return result[ATTR_PSK]
TypeError: 'NoneType' object is not subscriptable

@atwright147 I don't see any Tradfri config in your repo.
In case you are trying to configure the PSK in a config... don't! all you need is the gateway IP.

I only use discovery (https://github.com/atwright147/raspbian-homeassistant/blob/master/data/config/configuration.yaml#L39)
I used to have a specific Tradfri section but discovery seemed to be able to find it, so I removed the Tradfri section.
Could using discovery be my issue?

don't think so.
maybe remove the .tradfri_psk.conf (only 0.58 and later). if wrong values are on there setup will not work.

@flowolf I do not have a .tradfri_psk.conf file.
HASS Version:
cat .HA_VERSION
0.58.1

Here is my folder listing:
root@raspberrypi:/var/local/homeassistant/.homeassistant# ls -la
total 25224
drwxr-x--- 4 homeassistant homeassistant     4096 Nov 27 12:15 .
drwxr-xr-x 6 homeassistant homeassistant     4096 Nov 26 15:35 ..
-rw-r--r-- 1 homeassistant homeassistant        0 Nov 26 16:07 automations.yaml
-rw-r--r-- 1 homeassistant homeassistant     2952 Nov 26 15:36 configuration.yaml
-rw-r--r-- 1 homeassistant homeassistant        0 Nov 26 16:04 customize.yaml
drwxr-xr-x 2 homeassistant homeassistant     4096 Nov 26 16:10 deps
-rw-r--r-- 1 homeassistant homeassistant     1965 Nov 26 15:36 groups.yaml
-rw-r--r-- 1 homeassistant homeassistant        6 Nov 26 16:09 .HA_VERSION
-rw-r--r-- 1 homeassistant homeassistant     9549 Nov 27 12:09 home-assistant.log
-rw-r--r-- 1 homeassistant homeassistant 25632768 Nov 27 12:15 home-assistant_v2.db
-rw-r--r-- 1 homeassistant homeassistant     1122 Nov 26 16:11 options.xml
-rw-r--r-- 1 homeassistant homeassistant    96008 Nov 27 10:15 OZW_Log.txt
-rw-r--r-- 1 homeassistant homeassistant    32768 Nov 26 16:11 pyozw.sqlite
-rw-r--r-- 1 homeassistant homeassistant        0 Nov 26 16:09 scripts.yaml
-rw-r--r-- 1 homeassistant homeassistant      245 Nov 26 15:36 secrets.yaml
-rw-r--r-- 1 homeassistant homeassistant      190 Nov 26 15:36 security-zones.yaml
drwxr-xr-x 2 homeassistant homeassistant     4096 Nov 26 16:11 tts
-rw-r--r-- 1 homeassistant homeassistant       44 Nov 26 16:12 .uuid

Thank you for helping :)

I'd try adding the gateway explicitly with the IP.
also make sure all dependencies are up to date. (pytradfri, etc)

I told discovery to ignore ikea_tradfri and added the Tradfri host specifically, but it made no difference.

Don't know what the issue could be, maybe try asking in the discord chat.

Was this ever solved? I'm still seeing this with Hassio 0.60

For me, the new release solved the problems and allowed me to link to the
bridge without resetting it.
On 23 Dec 2017 9:46 pm, "Nick Schot"<EMAIL_ADDRESS>wrote:

Was this ever solved? I'm still seeing this with Hassio 0.60
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/home-assistant/issues/10499#issuecomment-353747214,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABzJ-f1wG9MmZiBUGKJzVW4ZyIV_UbyNks5tDWaOgaJpZM4QZM4i
.

I just wiped the SD card and switched to HASS.io and I am still seeing the same issue. I even have a new router now :(
In HASS.io how would I go about running pytradri manually?

Still receiving errors on HA 0.64.3:
Exception CancelledError() can not be represented as errno, setting -1.
13:36 core.py (ERROR)
Fatal DTLS error: code 20
13:36 core.py (ERROR)

2018-03-04 13:34:55 ERROR (MainThread) [homeassistant.core] Error executing service <ServiceCall configurator.configure: configure_id=1896452784-1, fields=security_code=NTgmT4QhmJgIhzZk>
Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/homeassistant/core.py", line 1010, in _event_to_service_call
    yield from service_handler.func(service_call)

  File "/usr/lib/python3.6/site-packages/homeassistant/components/configurator.py", line 224, in async_handle_service_call
    call.data.get(ATTR_FIELDS, {}))

  File "/usr/lib/python3.6/asyncio/futures.py", line 332, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.6/asyncio/tasks.py", line 250, in _wakeup
    future.result()

  File "/usr/lib/python3.6/asyncio/futures.py", line 245, in result
    raise self._exception

  File "/usr/lib/python3.6/asyncio/tasks.py", line 182, in _step
    result = coro.throw(exc)

  File "/usr/lib/python3.6/site-packages/homeassistant/components/tradfri.py", line 70, in configuration_callback
    key = yield from api_factory.generate_psk(security_code)

  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 196, in generate_psk
    self._psk = yield from self.request(command)

  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 152, in request
    result = yield from self._execute(api_commands)

  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 142, in _execute
    _, res = yield from self._get_response(msg)

  File "/usr/lib/python3.6/site-packages/pytradfri/api/aiocoap_api.py", line 92, in _get_response
    r = yield from pr.response

  File "/usr/lib/python3.6/asyncio/futures.py", line 332, in __iter__
    yield self  # This tells Task to wait for completion.

  File "/usr/lib/python3.6/asyncio/tasks.py", line 250, in _wakeup
    future.result()

  File "/usr/lib/python3.6/asyncio/futures.py", line 245, in result
    raise self._exception

  File "/usr/lib/python3.6/site-packages/aiocoap/protocol.py", line 816, in _run_outer
    yield from cls._run(app_request, response, weak_observation, protocol, log, exchange_monitor_factory)

  File "/usr/lib/python3.6/site-packages/aiocoap/protocol.py", line 865, in _run
    blockresponse = yield from blockrequest.response

  File "/usr/lib/python3.6/asyncio/futures.py", line 332, in __iter__
    yield self  # This tells Task to wait for completion.

  File "/usr/lib/python3.6/asyncio/tasks.py", line 250, in _wakeup
    future.result()

  File "/usr/lib/python3.6/asyncio/futures.py", line 245, in result
    raise self._exception

OSError: [Errno -1] No error information

I have this issue as well after I reset my gateway, highly annoying as most my lights are IKEA:


MCMC seed points outside of parameter ranges

@apace7 noted that the points we are using to seed the mcmc are not explicitly constrained to the allowed parameter range. This is a bug and should be fixed. @apace7, any chance you could submit a pull request with the fix you've been using?

Both of these updates sound good to me. We are happy to accept any hacks that you implement.

@apace7 Let's see if we can fix the bug (i.e. make sure to reject points outside of the parameter space) by the end of this week. We can push the uniform distribution enhancement to another issue.


Add note about "nightly" updates and time zone UTC

The article says

These Linux-based VMs use an Ubuntu image, with the OS configured to automatically check for updates every night.

It should be pointed out that this means always UTC time zone. So for a cluster running in Asia etc. this is very much not during the night.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: eadf3bc9-bea3-b51f-1477-8f76ccfce7da
Version Independent ID: 12d0be59-8947-e0a2-d73a-8834f2e5f8b2
Content: Handle Linux node reboots with kured - Azure Kubernetes Service
Content Source: articles/aks/node-updates-kured.md
Service: container-service
GitHub Login: @zr-msft
Microsoft Alias: zarhoads

Hi @zr-msft , can we add the specific timezone (that is UTC) in the document?


[stable/cluster-autoscaler] Update rbac apiversion from v1beta1 to v1

Signed-off-by: Mike Tougeron<EMAIL_ADDRESS>What this PR does / why we need it:
This updates the rbac apiVersion from v1beta to v1 (available since at least k8s 1.10)
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

cc @mgoodness / @yurrriq

/ok-to-test

/assign

/lgtm

Thanks!


Support Persistent Memory destroy method and fiberCacheManager.resetAll()

What changes were proposed in this pull request?
Due to memkind memory allocate issue. This patch will allow user to reinitialize persistent memory cache pool.
Added destoryKind() method in JNI to destory memkind persistent memory pool. Added a resetAll method in fiberCacheManager to reinitialize persistent memory. Added sql grammer "RESET FIBER CACHE" to call fiberCacheManager.resetAll to reset PM cache pool.
How was this patch tested?
ET

@LuciferYang Please take a look at this PR, see if this reset method could work for your scenario.


Add metadata for "Publish to BCR" app

@stefanbucur This is the last step I can do. Afterwards, you need to follow step 1 and 3 from https://github.com/bazel-contrib/publish-to-bcr. Let me know if you have any questions.

@stefanbucur Friendly ping

@stefanbucur Friendly ping

Whoops, thanks for the reminder! This slipped my attention. Trying to do this now.

@stefanbucur Friendly ping

Whoops, thanks for the reminder! This slipped my attention. Trying to do this now.

Step 1 done, moving to step 3 (cutting a new release).

@stefanbucur Friendly ping

Whoops, thanks for the reminder! This slipped my attention. Trying to do this now.

Step 1 done, moving to step 3 (cutting a new release).

@fmeum I think both steps are done. Should I see anything updated somewhere?

@stefanbucur This PR needs to be merged before creating the release, otherwise it won't trigger the app. After the merge, if you cut a release and something doesn't work, you should receive an email from the app.

@stefanbucur This PR needs to be merged before creating the release, otherwise it won't trigger the app. After the merge, if you cut a release and something doesn't work, you should receive an email from the app.

Argh! My apologies, I had somehow assumed I'm working on a Github issue, not PR. Let me merge this, then I'll cut v0.5.1.

@stefanbucur Thanks for the new attempt, unfortunately it didn't work.
According to the author of the BCR app, @kormide:

It looks like the app isn't installed to stefanbucur's BCR fork. It should have tried to send the error to the email here:

@stefanbucur Thanks for the new attempt, unfortunately it didn't work.
According to the author of the BCR app, @kormide:

It looks like the app isn't installed to stefanbucur's BCR fork. It should have tried to send the error to the email here:
<email in the metadata>

Per the BCR instructions at https://github.com/bazel-contrib/publish-to-bcr: "The fork can be in the same GitHub account as your ruleset or in the release author's personal account. If you use release automation and the release author is the github-actions bot, then the fork must be in ruleset's account unless you override the releaser."
The ruleset is under bazelbuild/ and the BCR repository is already there. Moreover, the configuration page didn't complain about anything, so I assumed this is the correct approach. I also didn't receive any notification email.
Why do I have to create a personal fork of the repository?

@stefanbucur Thanks for the new attempt, unfortunately it didn't work.
According to the author of the BCR app, @kormide:

It looks like the app isn't installed to stefanbucur's BCR fork. It should have tried to send the error to the email here:
<email in the metadata>

Per the BCR instructions at https://github.com/bazel-contrib/publish-to-bcr: "The fork can be in the same GitHub account as your ruleset or in the release author's personal account. If you use release automation and the release author is the github-actions bot, then the fork must be in ruleset's account unless you override the releaser."
The ruleset is under bazelbuild/ and the BCR repository is already there. Moreover, the configuration page didn't complain about anything, so I assumed this is the correct approach. I also didn't receive any notification email.
Why do I have to create a personal fork of the repository?

The app does not have write permissions to the BCR (by design) and cannot push an entry there, so rulesets under bazelbuild are an exception.
Perhaps you didn't receive an email because a no-reply email is listed here? https://github.com/bazelbuild/rules_fuzzing/blob/master/.bcr/metadata.template.json#L5


Unit Testing

Your code has no relationship with the given PR title. Please update this to make it a meaningful PR.


[Task]: Create Category token for switch component

Bug:

Incorrect background color in Not selected & disabled state. Expected is #cdd5df

Incorrect background color in Selected & disabled state. Expected is #4c5664


No CONTRIBUTE.md

I want to contribute actively and tremendously to the design of this project but I can't find a CONTRIBUTE.md file to guide me

Same here. I would like to. There are no issues :-(


Integrate DSCensor API into MultiQC Views

I can integrate the BUSCO stats into the MultiQC views using DSCensor as the data source, by re-writing the data parser to make a requests call.
The resulting HTML can be written into a flask view and served directly from the DSCensor domain.

MultiQC is currently feeding off of the DSCensor API and writing static HTML for testing.
Will port this into a flask view

MultiQC views are now available and have working table and figure synchronization.


Shed some light on segmented_cache design [DOC]

Hi guys and Happy New Year,
Can someone explain the segmented_cache internal design?

what's the purpose of using process group (pg)?
can we mix TTL and LRU for expiry?

An architecture design would be great.
Many thanks

Sorry for delay in response. I'm not sure if a separate document is really needed.
Here are the key points of design:

TTL implementation notes:

cache is stored in a multiple ETS tables.
there's only one table use writing cache (active/current table).
tables are rotated and data from the last table is dropped.
TTL is not 100% accurate, I would interpret it as a warranty that cache record live less than TTL but at least TTL - TTL/N where N is a number of ETS tables.
there's a gen_server process, which is responsible for tables rotation (see cache state notes for more details).

cache state notes:

persistent_term module is used to store cache state.
it's initialised just once and never changes after that.
there is a list of ETS tables - technically it's implemented as a tuple to improve access time.
there is an atomic ref that is used to track the index of the current (active) ETS table in the list (tuple).
amotic ref is constant, and it doesn't change
atomic value is changing, and what's important - changing in a lock free manner.

LRU implementation notes:

on is_member/2 and get_entry/2 calls, record is reinserted in the current (active) ETS table.

cache record deletion notes:

in distributed environment, cache is populated independently on every node.
however, we must ensure that on deletion the cache record is removed on all the nodes (and from all the ETS tables, see LRU implementation notes)
there's a gen_server process responsible for ETS tables rotation (see TTL implementation notes)
the same process is reused to implement asynchronous cache record deletion on other nodes in the cluster.
in order to simplify discovery of these gen_server processes on other nodes, they all are added into a dedicated pg group.

@NelsonVides, please add your comments if I missed anything, otherwise - close this issue.

@DenysGonchar thank you very much.


[bridge 13/n] wire up orchestrator and action executor

Description
Orchestrator submits action to ActionExecutor.
Test Plan
Unit tests

If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.
Type of Change (Check all that apply)

[ ] protocol change
[ ] user-visible impact
[ ] breaking change for a client SDKs
[ ] breaking change for FNs (FN binary must upgrade)
[ ] breaking change for validators or node operators (must upgrade binaries)
[ ] breaking change for on-chain data layout
[ ] necessitate either a data wipe or data migration

Release notes
wire up orchestrator and action executor

cc @Jayfromthe13th


Documentation/sample - loadTextContent() or loadFileAsBase64() for VMSS script extension

The new loadTextContent() and loadFileAsBase64() functions look like they're ideal for scenarios like sending a custom script into the VMSS custom script extension. However, in my testing, I wasn't able to find an elegant way to do actually use this. The way we ended up getting this to work was by using this:
{
  name: 'customScriptExtension'
  properties: {
    publisher: 'Microsoft.Compute'
    type: 'CustomScriptExtension'
    typeHandlerVersion: '1.10'
    protectedSettings: {
      commandToExecute:'powershell.exe -ExecutionPolicy Unrestricted -Command "& Invoke-Expression -Command ([System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String(\'${loadFileAsBase64('../scripts/vmss-configure-iis.ps1')}\')))"'
    }
  }
}

(source)
It'd be good to know if there's a better way to do this, and to have clear docs and a sample to help with this since it's likely to be a common scenario.

For context powershell does have a -encodedcommand param that takes base64 content but it uses utf16LE as the charset and it looks like bicep is utf8

@mumian FYI - something we will need to take care of after the we finish the migration to MS docs

I think the problem here is with the script extension itself not supporting new lines rather than bicep.  I had great problems using multi line strings with it, and ended up having to strip all newlines out.
I ended up refactoring my ps1 script to terminate statements with ;, then strip the new lines out in bicep:
  resource customScript 'extensions@2021-03-01' = if (!empty(vmCustomScript)) {
    name: 'CustomScript'
    properties: {
      publisher: 'Microsoft.Compute'
      type: 'CustomScriptExtension'
      typeHandlerVersion: '1.9'
      autoUpgradeMinorVersion: true
      protectedSettings: {
        commandToExecute: replace(replace(vmCustomScript, '\r', ' '), '\n', ' ')
      }
    }
  }

Only my 2 cents. I'm using the customData attribute to pass the script to the VM and run it via CustomScript extension (on Windows) and cloud-init on Linux. The new functions make it very easy and very elegant. Awesome work from the Team!
I'm also using sth. like format(loadTextContent(...), ...) which makes it even more awesome.  As it uses .NET String.format() i'm using server {{ ... hostname: {0} }} in my cloud-ionit.yml. I think we should add sth. like that to the docs.

This works for me locally:
iex "& { $([System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String('cGFyYW0oCiAgIFtzdHJpbmddICROYW1lCikKCldyaXRlLUhvc3QgIkhlbGxvICROYW1lISI='))) } -Name Anthony"

So something like the following should work (untested, may need some refinement!):
var command = '''
param(
   [string] $Name
)

Write-Host "Hello $Name!"
'''
var arguments = {
  Name: 'Anthony'
}
// builds a string of form '-ArgA ValA -ArgB ValB'
var argumentString = join(map(items(arguments), i => '-${i.key} ${i.value}'), ' ')

var commandToExecute = 'powershell.exe -ExecutionPolicy Unrestricted -Command "iex \\"& { $([System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String(\'${base64(command)}\'))) } ${argumentString}\\""'

@anthony-c-martin thanks for your help.
I was not able to make it work. Eventually got it working by using a runcommand.
For everyone struggling to automate the process of installing the integration runtime on an azure VM and register it with the data factory: this repository is a good starting point.
My runcommand script:
param location string
param name string
param shirId string
var scriptContent = loadTextContent('./test1.ps1', 'utf-8')

// https://learn.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines/runcommands?pivots=deployment-language-bicep
resource res_run_cmd 'Microsoft.Compute/virtualMachines/runCommands@2023-03-01' = {
  name: '${name}-vm-0-0/testtestinstall'
  location: location
  properties: {
    asyncExecution: false
    protectedParameters: [{
      name: 'gatewayKey'
      value: listAuthKeys(shirId, '2018-06-01').authKey1
    }]
    source: {
      script: scriptContent
    }
    timeoutInSeconds: 600
    treatFailureAsDeploymentFailure: true
  }
}

The shirId is the id of the integration runtime (child of data factory). test1.ps1 = test1.ps1


Putting files greater than ~1 GB fails.

Putting files greater than ~1 GB fail with the error below.
Files were sent to dirt.cyverse.org. Files copy fine using iCommands.
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 70, in send
    self.socket.sendall(string)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cotter/GitHub/irods_python_client/src/pyicmd/pyicmd/functional.py", line 21, in put
    dest_obj.write(f.read())
  File "/usr/local/lib/python3.6/dist-packages/irods/data_object.py", line 125, in write
    return self.conn.write_file(self.desc, b.tobytes())
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 470, in write_file
    self.send(message)
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 78, in send
    raise NetworkException("Unable to send message")
irods.exception.NetworkException: Unable to send message

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 70, in send
    self.socket.sendall(string)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/irods/data_object.py", line 114, in seek
    return self.conn.seek_file(self.desc, offset, whence)
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 487, in seek_file
    self.send(message)
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 78, in send
    raise NetworkException("Unable to send message")
irods.exception.NetworkException: Unable to send message

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyicmd", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/cotter/GitHub/irods_python_client/src/pyicmd/bin/pyicmd", line 4, in <module>
    pyicmd.main()
  File "/home/cotter/GitHub/irods_python_client/src/pyicmd/pyicmd/pyicmd.py", line 91, in main
    put(session,unknownargs)
  File "/home/cotter/GitHub/irods_python_client/src/pyicmd/pyicmd/pyicmd.py", line 40, in put
    F.put(session,args.files,args.loc, recursive = args.R)
  File "/home/cotter/GitHub/irods_python_client/src/pyicmd/pyicmd/functional.py", line 21, in put
    dest_obj.write(f.read())
  File "/usr/local/lib/python3.6/dist-packages/irods/data_object.py", line 108, in close
    self.conn.close_file(self.desc, **self.options)
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 505, in close_file
    self.send(message)
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 78, in send
    raise NetworkException("Unable to send message")
irods.exception.NetworkException: Unable to send message
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Unable to send message. Connection to remote host may have closed. Releasing connection from pool.
Exception ignored in: <bound method Connection.__del__ of <irods.connection.Connection object at 0x7fbebc677780>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 63, in __del__
    self.disconnect()
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 253, in disconnect
    self.send(disconnect_msg)
  File "/usr/local/lib/python3.6/dist-packages/irods/connection.py", line 78, in send
    raise NetworkException("Unable to send message")
irods.exception.NetworkException: Unable to send message

Error also happens using a local Docker based (https://github.com/mjstealey/irods-icommands) irods server


Model Subprocess / Call Activity for printing

https://digiwf-co-creation-demo.apps.capk.muenchen.de/modeler/#/cbaca5e6-e7b0-44cf-b99b-f5da97010234/latest


Symfony 4 compliant changes

Based on PR :
https://github.com/lightSAML/SymfonyBridgeBundle/pull/17
Adds the missing service public (needed for https://github.com/lightSAML/SpBundle)
@tmilos can you review these changes?

Coverage decreased (-0.7%) to 96.512% when pulling dca2cd5a25ce2fec4ed5a073b73528d679d22250 on rudott:master into 01ad843a0bafc5268bda65e49a9be3c0c57e2381 on lightSAML:master.

Would be great to see this merged soon!

Symfony 4 support is already merged in #17
Beside composer.json and DefinitionDecorator/ChildDefinition this PR also makes lots of services public. Is there any special need for that?

@tmilos this is because the Container is used with Container->get(....) in Symfony 4 this is only possible when the referenced service is public.


Django 2.2 Capability

django_rq_wrapper/management/commands/rqworkers.py
from django.utils.autoreload import reloader_thread
gives and import error
We are currently getting an error with the latest version of Django.
In Django 2.2 the autoreloader was rewritten so we need to figure out a function to hook into here
https://github.com/istrategylabs/django-rq-wrapper/blob/0e27133f63777461de7e9d2f9bfb8cb0c48afb1b/django_rq_wrapper/management/commands/rqworkers.py#L89

django_rq_wrapper/management/commands/rqworkers.py
from django.utils.autoreload import reloader_thread
gives and import error
We are currently getting an error with the latest version of Django.
In Django 2.2 the autoreloader was rewritten so we need to figure out a function to hook into here
https://github.com/istrategylabs/django-rq-wrapper/blob/0e27133f63777461de7e9d2f9bfb8cb0c48afb1b/django_rq_wrapper/management/commands/rqworkers.py#L89

Good luck :-P


I am done with Franz

I tried it, I liked it, id started not to work properly, I reinstalled it, it kept on not working, i deleted it. Nice try, let me know when it actually works.

If you cared about this working, you'd share what actually bothers you.

I did, got no answer. And I am sick of forums, where developers outsource the support. I would rather pay 20 or more dollars and have a working product and support if it doesn’t. No offense.

I totally agree with you. ATM the software is a dumpsterfire and the developer is not addressing any of the issues on Github. Yet he keeps sending E-Mails asking for donations.
Maybe people would donate if the software actually works. It takes 5 minutes of real world testing to see that this was never properly tested by anyone.
@henrebotha
https://github.com/meetfranz/franz/issues/784
https://github.com/meetfranz/franz/issues/790
https://github.com/meetfranz/franz/issues/786
And in addition to that the software sometimes freezes when I switch back to it from another workspace and has to be killed.
10/10 piece of software. Can't see why people would complain if it even cannot properly do the single thing it was designed to do.

ATM the software is a dumpsterfire and the developer is not addressing any of the issues on Github.

Every new update addresses issues I've experienced.

Maybe people would donate if the software actually works.

The software does actually work, which I know because I've been using it since pre-v5 for multiple services including work IM.

The software does actually work, which I know because I've been using it since pre-v5 for multiple services including work IM.

Out of 6 services I added to Franz a grand total of 2 work. I would not call that having a working product.

HipChat: Sign in not possible
Google calendar: Sign in not possible
Github: Sign in not possible
Google Keep: Sign in not possible

I just successfully signed in to Google Calendar, Google Keep, and Github. I don't have a HipChat account so not going to try it.
I'm sure you experience bugs. I'm saying that pretending this app is broken everywhere for everyone at all times is ridiculous.

I'm sure you experience bugs. I'm saying that pretending this app is broken everywhere for everyone at all times is ridiculous.

I never said it was broken for everyone, I said it was broken for me and some other people. If it works for you then congratz. For me it doesn't work.
The fact that OP created this issue is proof enough that it in fact doesn't work for some people.
What platform are you running on? I'm pretty sure that it'll also be broken for you if you're running it on Linux with i3wm.

@henrebotha I personally love Franz, I am happy with it and I use it everyday on multiple platforms. But that is only because I managed to work around all the issues I had! But all the complaints that @N0ps32 has are more than legitimate.
In general, I believe that this project has become much bigger to maintain than what two people can do in their time.

I use the app daily using pulse, messenger, gmail, ms teams, and recently added github. It works for everything I use it for pretty flawlessly. While I am not saying it is perfect but this application works  quite well for what its aim is and it is unjust to claim it is broken or a dumpster fire.
As a proof of this, I am using Franz to write and submit this comment :)

@alecive I can agree with that, but luckily it's open source and anyone can contribute.

I can totally understand that some people are frustrated with Franz as some parts might not always work as excepted on certain platforms or various service configurations.
We are constantly working on improving Franz. But please understand that we have very limited resources as we are still just two guys with an evergrowing list of tasks and bugs to tackle. We do what we can and we even get better every day – even though it doesn't seem like it for some users.
On top, we are also aware that there are some 3rd party services in Franz that are actively limiting the experience of our users with Franz. This is a major issue for us which we don't have solution for yet.

Please understand that I'm going to lock this issue in 24 hours.


Using atan2 instead of acos for rotation angles.

I have implemented the solution discussed in #543.

Thanks!

@simonpuchert As you mentioned last time, we should not use abs(w) but w to get the full range of angles between 0 and pi. In this PR you left the abs from previously. Wouldn't this cause issues for angles > pi/2 ?


SFS | Adjust label when linking credit transaction |0|

https://trello.com/c/duYdsNG0/1219-sfs-adjust-label-when-linking-credit-transaction-0
Task:

Current label: "Linking Credit Transfer"
Revised label: "Link Credit Transaction"

 SFS | Adjust label when linking credit transaction |1|


CodeCleanup caues extra whitespace in Groovy arguments

What version of OpenRewrite are you using?
org.openrewrite.rewrite.gradle.plugin:6.5.0
org.openrewrite.recipe:rewrite-recipe-bom:2.5.0
I believe it is a regression since org.openrewrite.rewrite:org.openrewrite.rewrite.gradle.plugin:6.4.3 org.openrewrite.recipe:rewrite-recipe-bom:2.4.1
As extra whitespace in Groovy files were not generated in the previous version.
How are you running OpenRewrite?
https://github.com/pgjdbc/pgjdbc/pull/2979
./gradlew rewriteRun
What is the smallest, simplest way to reproduce the problem?
public class Util {
    public static int findPort() {
        int port
        ServerSocket s = new ServerSocket(0)
        port = s.getLocalPort()
        s.close()
        return port
    }
    public static void appendToFile(String fileName, String text, truncate=false) {

        new File(fileName).with() { f ->
            if ( truncate ) {
                FileChannel outChannel = new FileOutputStream(f, true).getChannel()
                outChannel.truncate(0)
                outChannel.close()
            }
            f.append("$text\n ")
        }
    }
}

What did you see instead?
public class Util {
    public static int findPort() {
        int port
        ServerSocket s = new ServerSocket(0)
        port = s.getLocalPort()
        s.close()
        return port
    }
    public static void appendToFile(String fileName,          String text,          truncate=false) {

        new File(fileName).with(){ f          ->
            if (truncate) {
                FileChannel outChannel = new FileOutputStream(f, true).getChannel()
                outChannel.truncate(0)
                outChannel.close()
            }
            f.append("$text\n ")
        }
    }
    public static String readFile(String fileName) {
        new FileInputStream(fileName).text
    }
}

diff:
--- a/test-gss/src/main/groovy/Util.groovy
+++ b/test-gss/src/main/groovy/Util.groovy
@@ -9,10 +9,10 @@ public class Util {
         s.close()
         return port
     }
-    public static void appendToFile(String fileName, String text, truncate=false) {
+    public static void appendToFile(String fileName,          String text,          truncate=false) {

-        new File(fileName).with() { f ->
-            if ( truncate ) {
+        new File(fileName).with(){ f          ->
+            if (truncate) {

Oh wow, that's quite unfortunate. I'm adding your issue to the backlog. It's likely an issue with the Groovy handling of whitespace, and especially annoying that in increases with every run. Perhaps in the short term you can exclude groovy files from your recipe runs? I don't quite know when we'll get to resolving this issue; any help is welcome!


Android Webview should be added/tested

Android WebView in supported in React-Native 0.18.0-rc so we should upgrade to that and try out WebView with a scrolling htmlBook

Closing this with #17 just implementing WebView as the issue suggest. Also want to isolate the upgrade to RN 0.19.0 to this PR. Leaving html book to another issue.


Typescript errors in wonka 6.3.3

Building my project works correctly with wonka v6.3.2, but not with v6.3.3.  The issue is that typescript finds a issues in wonka.d.ts

node_modules/wonka/dist/wonka.d.ts:1323:13 - error TS2339: Property 'observable' does not exist on type 'SymbolConstructor'.
1323     [Symbol.observable]?(): Observable;
~~~~~~~~~~
node_modules/wonka/dist/wonka.d.ts:1352:13 - error TS2339: Property 'observable' does not exist on type 'SymbolConstructor'.
1352     Symbol.observable: Observable;
~~~~~~~~~~

This can be resolved by adding the following block to the wonka.d.ts file
declare global {
  interface SymbolConstructor {
    readonly observable: symbol;
  }
}

It appears that this was recently moved from the observable.ts file to the helpers.ts file, but I can't tell why it doesn't appear in the finished wonka.d.ts file, but it could be that index.ts doesn't contain a line like
export * from './helpers';
I will continue to look into this when I have more time.

Nice catch! I didn't actually notice that this went missing. I guess I didn't remember to check. I've moved it back and added a comment to prevent this from happening again. I think these issues always have a delay in being reported since skipLibCheck: true is a common option to have enabled in most scenarios


feat/photon: prevent node runner to set uatone as only gas fees

When running node on #57 ( feat/photon ), if the RPC is set with uatone as a gas fees, outputs could be weird to understand.
here is an example:
$ atomoned tx bank send my-wallet atone1xxxx 500000000uatone --yes --fees 5000uatone
[...]
raw_log: 'fee denom uatone not allowed: invalid fee token'

$ atomoned tx bank send my-wallet atone1xxxxxx 500000000uatone --yes --fees 5000uphoton
[...]
raw_log: 'insufficient fees; got: 5000uphoton required: 200uatone: insufficient fee'

A warning at start of the node or enforce a default minimum-gas-fees with format 0.001uatone,0.001uphoton would be a good idea i believe.

The first error is returned by the photon ante handler which only accepts PHOTON for the fee token.
The second error is returned by the cosmos-sdk TxFeeChecker which ensures that the fee tokens passed in the transaction are compliant with the minimum-gas-prices config of the node.
While this is a normal behavior, having them in a row like that is confusing I admit, even considering that the node operator has simply not followed the v2 upgrading procedure.
As you suggested, we could return an error if the minimum-gas-prices does not contain at least PHOTON, but that sets a kind of precedent, as this setting is usually left to the node operator. I am not sure if we should actually check it.
Wdyt @giunatale ?

agreed with @tbruyelle


Developers need better config-event testing

As we add more business logic into reporting and internal monitoring, we need a simple way to drive the logic through its paces.

Oh. I actually did this in #1693.


fetchOne returns always numeric-string instead of string|false

            $defaultVersionId = $conn->fetchOne('SELECT id FROM package_version WHERE package_id = :id ORDER BY defaultBranch DESC, releasedAt DESC LIMIT 1', ['id' => $id]);
            \PHPStan\dumpType($defaultVersionId);

Dumped type: numeric-string
From https://github.com/composer/packagist/blob/main/src/Command/PopulateDependentsSuggestersCommand.php#L71
fetchOne returns false if there is no result though.

Thanks


Feature/on layout change support

Describe what your pull request addresses

[ ] added onLayoutChange property for callbacks on layout changes
[ ] expanded layout property with absolute key that allows absolute positioning (using same values as returned by onLayoutChange)

Test Instructions

can test with https://deploy-preview-60--create-app.netlify.app/

[ ] move around and resize panels, then click refresh on browser.  Should see layout restored to last layout before refresh.

LGTM


Some minor updates to the script

I have modified the script in the following manner:

Renamed the file from a name containing spaces to underscores instead (for ease of use by many less-knowledgeable users)
Some linting and syntax cleaning
Obtaining from /etc/crypttab the name of devices and iterating over it - allows for using the TPM2 single key stored for all encrypted disks, and also - not hard-coded on /dev/sda3 anymore.

Assumptions:

This is a default Ubuntu installation, and thus - this is how crypttab looks
This installation has LVM spanning over two (or more) physical disks, meaning crypttab will reflect Ubuntu's configuration.
Therefore, I think this is safe enough, and if things are weird, the script will fail.

Those are reasonable enough expectations for 99% of cases, but because we are messing with low level stuff, I want to be extra cautious.  I will go ahead and merge this PR, but I should be done soon with the changes that should be safe in 99.9% of cases, and allows the user additional control over which devices are selected.

Listen - the purpose of the script is to save time and reduce chances of mistake to a newcomer to Linux or a person who has set mind about encrypting the disk. I think it should be rather simple - Assume "defaults", but if not - exit, and point the person to a real guide. The guide makes sense of the layers below, so there should be no problem there. My 2 cents. I am all into helping people, but it has to be reasonable. You cannot anticipate any and every scenario out there, and attempting to do so would make the whole process complex beyond easy maintenance. Again - my opinion.

I just created a v0.8.0-alpha.1 branch with my big changes.  I haven't merged it to main yet because I have only done preliminary testing so far.  I will try to get some additional testing done tomorrow, but it should be a LOT more friendly to a newcomer, and should keep people from having to edit the script itself and/or their /etc/crypttab.
If you have a chance to try it out and share your thoughts, I would love to hear it :)

FYI when I was testing my v0.8.0 update, I found that on my Ubuntu test VMs (using qemu/kvm and virt-manager) it defaulted to /dev/vd* for the virtual devices (e.g. /dev/vda) but dm_crypt-0 for the name in crypttab, so in that case assuming the crypttab name is _crypt would fail.  Granted, I doubt many people are trying to encrypt on a VM, but it is dangerous to assume crypttab will always use _crypt, especially when it isn't too hard to have cryptsetup just tell us what the device path is

I use a vm with snapshots (based on ZFS) to run the same process. In my case, because the next step is using grub-standalone, which has a bug with virtio and EFI (which almost no one encounters, so no fix for that), I am using "sata" disk controller, and my disks are /dev/sda and so on. I think that my fix will work just fine on vda devices. At the current state of the code, with a single disk - I see no reason for the script to fail.

Your fix would fail adding the key to the volume:
for disk in `cat /etc/crypttab | awk '{print $1}' | sed 's/_.*//'`
do
    echo
    echo Adding the new key to LUKS.  You will need to enter the current passphrase used to unlock the drive...
    echo
    sudo cryptsetup luksAddKey /dev/${disk} root.key
    if [ $? != 0 ]
    then
        echo Something went wrong adding the encryption key to /dev/${disk}. Check /etc/crypttab and/or lsblk to determine your encrypted volume, then update this script with the correct value
        exit
    fi
done

In my VM's case using the defaults, the code about would see dm_crypt-0 in /etc/crypttab and type to add a key to /dev/dm, which isn't valid because it should be /dev/vda1

How come? My line looks like this:
sda3_crypt UUID=4462a1ed-26c7-4bde-ae4a-e54466480c98 none luks,discard,keyscript=/usr/local/sbin/tpm2-getkey,keyscript=/usr/local/sbin/tpm2-getkey
There is no dm_crypt-0, and there should not be. The naming (as presented by Ubuntu. You can manually put whatever you like) is device-name_crypt -> very simple and predictable. Can you share your /etc/crypttab file?

The /etc/crypttab Ubuntu created during the clean install using all defaults was:
dm_crypt-0 UUID=b5b7f931-2335-4acf-8600-7ae8712163dd none luks

the associated device is /dev/vda1
I didn't manually put in anything, that is the default naming Ubuntu picked for the virtio drives that qemu/kvm used by default.

That is weird, and very unusual. Maybe the script needs to be modified to handle a case where the name is dm_*. It should not be the case, and it is most unusual. Or - it could be replaced by
blkid | grep crypto_LUKS | awk '{print $1}'
which seems to yield the same (correct) result

On my VM that command also grabs a trailing ":" character, but after correcting for that it would work.  In my update, I also wanted the name and device path associated with each other, so I got the device name with
sudo cryptsetup status ${TEMP_DEVICE_NAMES[$I]} | sed -n -E 's/device:\s+(.*)/\1/p'

where TEMP_DEVICE_NAMES is an array of all the device names it found in /etc/crypttab.  That works no matter what naming convention is used, even strange manual ones

by "my update" I mean the v0.8.0 branch


chore: port over infrastructure stuff

It's really just porting it over to trustify, and adapting it to the database backend.
Authentication/Authorization is present, but not used yet.

Just to be clear, I think there's room for improvement:

how the configuration works (clap arguments only at that point)
I still think it might make sense to give this it's own crate

@ctron I think this is definitely useful beyond trustification and deserves its own crate


About dataset bdd100k

Hello, your work is very impressive. Does the bdd100k dataset require some processing? I didn't find the path in the data set that you provided in the project.I downloaded the red line data set.

The file path you specified was not found:

Hello, the path should follow the structure of the BDD100K dataset outlined here: https://doc.bdd100k.com/download.html#semantic-segmentation
What do you mean by red line dataset? It seems there is a problem with the BDD100K host server, I cannot access https://bdd-data.berkeley.edu/ at the moment.

I also can't access the BDD100K website, I have tried for many days, could you please provide a BDD100k data set? My email is<EMAIL_ADDRESS>Thank you very much for your reply.

Unfortunately I can't share the data myself, sorry. My suggestion would be to try to contact the maintainers of the BDD100K dataset.


Action menu in grid. Not append new item

Laravel Version: 5.8.30
PHP Version: 7.3.3
Laravel-admin: 1.7

Description:
Can't add new item in dropdown row menu on action in grid list
  $this->actions(function (Grid\Displayers\Actions $actions) {
            $actions->disableDelete();
            $actions->append('show item 1');
            $actions->prepend('show item 2');
        });

"show item X" not showing in row menu.
but disabled working.
Please fix it. =)
Steps To Reproduce:

sorry ...  this section in  documentation is wrong. ...
$this->actions(function (Grid\Displayers\DropdownActions $actions) {             $actions->add(new CustomView())         });
where CustomView extends RowAction ....
will work...

$actions->add() is working, but it will get error "intelephense(1013)"


easy photo generate photo error,show error as below

Diagnostics-1715085561.log
emplate 1 error: Error info is 1 validation error for ControlNetUnit
control_mode
value is not a valid enumeration member; permitted: 'Balanced', 'My prompt is more important', 'ControlNet is more important' (type=type_error.enum; enum_values=[<ControlMode.BALANCED: 'Balanced'>, <ControlMode.PROMPT: 'My prompt is more important'>, <ControlMode.CONTROL: 'ControlNet is more important'>]).

I'm having the same issue. I wonder if it's somehow connected to recent updates in ControlNet, specifically, this part in their README.md

Log from container with traceback
supervisor-1  | 2024-05-09 11:27:45,942 - EasyPhoto -
supervisor-1  |             Start Generate template                 : 1;
supervisor-1  |             user_ids                                : ('abc123', 'none', 'none', 'none', 'none');
supervisor-1  |             sd_model_checkpoint                     : Chilloutmix-Ni-pruned-fp16-fix.safetensors;
supervisor-1  |             input_prompts                           : ['easyphoto_face, easyphoto, 1person, <lora:abc123:0.9>, <lora:FilmVelvia3:0.65>, masterpiece, beauty', 'none', 'none', 'none', 'none'];
supervisor-1  |             before_face_fusion_ratio                : 0.6;
supervisor-1  |             after_face_fusion_ratio                 : 0.6;
supervisor-1  |             first_diffusion_steps                   : 50;
supervisor-1  |             first_denoising_strength                : 0.45;
supervisor-1  |             second_diffusion_steps                  : 20;
supervisor-1  |             second_denoising_strength               : 0.3;
supervisor-1  |             seed                                    : 41366
supervisor-1  |             crop_face_preprocess                    : True
supervisor-1  |             apply_face_fusion_before                : True
supervisor-1  |             apply_face_fusion_after                 : True
supervisor-1  |             color_shift_middle                      : True
supervisor-1  |             color_shift_last                        : True
supervisor-1  |             super_resolution                        : True
supervisor-1  |             super_resolution_method                 : gpen
supervisor-1  |             display_score                           : True
supervisor-1  |             background_restore                      : False
supervisor-1  |             background_restore_denoising_strength   : 0.35
supervisor-1  |             makeup_transfer                         : False
supervisor-1  |             makeup_transfer_ratio                   : 0.5
supervisor-1  |             skin_retouching_bool                    : False
supervisor-1  |             face_shape_match                        : False
supervisor-1  |             id_control                              : True
supervisor-1  |             id_control_method                       : IP-Adapter Face
supervisor-1  |             ipa_weight                              : 0.3
supervisor-1  |             instantid_id_weight                     : 0.5
supervisor-1  |             instantid_ipa_weight                    : 0.5
supervisor-1  |             instantid_image_path                    : None
supervisor-1  |             ipa_image_path                          : None
supervisor-1  |             ref_mode_choose                         : Infer with User Lora
supervisor-1  |             no_user_lora_mode                       : IP-Adapter Face
supervisor-1  |             ipa_only_weight                         : 0.6
supervisor-1  |             ipa_only_image_path                     : None
supervisor-1  |             instantid_only_id_weight                : 0.5
supervisor-1  |             instantid_only_ipa_weight               : 0.5
supervisor-1  |             instantid_only_image_path               : None
supervisor-1  |
supervisor-1  | 2024-05-09 11:27:46,283 - EasyPhoto - Start Image resize to 512.0.
supervisor-1  | 2024-05-09 11:27:46,298 - EasyPhoto - Start face detect.
supervisor-1  | 2024-05-09 11:27:48,906 - EasyPhoto - Start First diffusion.
supervisor-1  | 2024-05-09 11:27:49,298 - ControlNet - WARNING - Deprecated alias 'input_image' detected. This field will be removed on 2024-06-01Please use 'image' instead.
supervisor-1  | Traceback (most recent call last):
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/easyphoto_infer.py", line 1522, in easyphoto_infer_forward
supervisor-1  |     first_diffusion_output_image = inpaint(
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/easyphoto_infer.py", line 388, in inpaint
supervisor-1  |     image = i2i_inpaint_call(
supervisor-1  |   File "/opt/micromamba/envs/webui/lib/python3.10/contextlib.py", line 79, in inner
supervisor-1  |     return func(*args, **kwds)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/sdwebui.py", line 721, in i2i_inpaint_call
supervisor-1  |     processed = processing.process_images(p_img2img)
supervisor-1  |   File "/workspace/stable-diffusion-webui/modules/processing.py", line 845, in process_images
supervisor-1  |     res = process_images_inner(p)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/batch_hijack.py", line 56, in processing_process_images_hijack
supervisor-1  |     cn_is_batch, batches, output_dir, input_file_names = get_cn_batches(p)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/batch_hijack.py", line 196, in get_cn_batches
supervisor-1  |     units = external_code.get_all_units_in_processing(p)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/internal_controlnet/external_code.py", line 133, in get_all_units_in_processing
supervisor-1  |     return get_all_units(p.scripts, p.script_args)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/internal_controlnet/external_code.py", line 146, in get_all_units
supervisor-1  |     return get_all_units_from(script_args[cn_script.args_from : cn_script.args_to])
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/internal_controlnet/external_code.py", line 174, in get_all_units_from
supervisor-1  |     all_units = [
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/internal_controlnet/external_code.py", line 175, in <listcomp>
supervisor-1  |     to_processing_unit(script_arg)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/internal_controlnet/external_code.py", line 228, in to_processing_unit
supervisor-1  |     return ControlNetUnit.from_dict(unit)
supervisor-1  |   File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/internal_controlnet/args.py", line 409, in from_dict
supervisor-1  |     return ControlNetUnit(**values)
supervisor-1  |   File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
supervisor-1  | pydantic.error_wrappers.ValidationError: 1 validation error for ControlNetUnit
supervisor-1  | control_mode
supervisor-1  |   value is not a valid enumeration member; permitted: 'Balanced', 'My prompt is more important', 'ControlNet is more important' (type=type_error.enum; enum_values=[<ControlMode.BALANCED: 'Balanced'>, <ControlMode.PROMPT: 'My prompt is more important'>, <ControlMode.CONTROL: 'ControlNet is more important'>])
supervisor-1  | 2024-05-09 11:27:49,375 - EasyPhoto - Template 1 error: Error info is 1 validation error for ControlNetUnit
supervisor-1  | control_mode
supervisor-1  |   value is not a valid enumeration member; permitted: 'Balanced', 'My prompt is more important', 'ControlNet is more important' (type=type_error.enum; enum_values=[<ControlMode.BALANCED: 'Balanced'>, <ControlMode.PROMPT: 'My prompt is more important'>, <ControlMode.CONTROL: 'ControlNet is more important'>]), skip it.

I wonder if there is a way to install previous version of extension in A1111 to check my hypothesis which is that ControlNet had a breaking change. Can someone please tell how I can do that?

I resolved the issue by downgrading ControlNet extension to commit 59d5998
This can be done by using following commands:
cd stable-diffusion-webui/extensions/sd-webui-controlnet
git checkout 59d5998

However, it messed my extensions tab a little

But it works

So downgrading works as a workaround, however it seems like either a bug was introduced in new version of ControlNet, or something is wrong with how it is being used by EasyPhoto after the update.

YES, I downgrade from newest version and it works. so it may be compatibility problem.

Me too

Closed via #424