Unable to run Spark jobs on an Azure Databricks cluster: java.lang.NullPointerException at org.apache.spark.util.LocalHadoopConfiguration.set(SparkHadoopConfiguration.scala:196)

Hi,
Since last Friday, 5th of November, we seem to be unable to use the osm4scala library on an Azure Databricks cluster with configuration: 8.3 (includes Apache Spark 3.1.1, Scala 2.12).
When we try to run this simple command in a notebook:
spark.read
     .format("osm.pbf")
     .load("dbfs:/FileStore/shared_uploads/andorra_latest_osm.pbf")
     .select("id","type")
     .show()

... we're getting following exception:
Caused by: java.lang.NullPointerException
	at org.apache.spark.util.LocalHadoopConfiguration.set(SparkHadoopConfiguration.scala:196)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115)
	at org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2877)
	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
	at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
	at org.apache.spark.SerializableWritable.$anonfun$readObject$1(SerializableWritable.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1609)
	at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:41)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$unBlockifyObject$4(TorrentBroadcast.scala:372)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:374)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$4(TorrentBroadcast.scala:279)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$2(TorrentBroadcast.scala:253)
	at org.apache.spark.util.KeyLock.withLock(KeyLock.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:248)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1609)
	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:248)
	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:118)
	at org.apache.spark.broadcast.Broadcast.$anonfun$value$1(Broadcast.scala:80)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:78)
	at com.acervera.osm4scala.spark.OsmPbfFormat.$anonfun$buildReader$1(OsmPbfFormat.scala:86)
	at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:155)
	at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:142)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:331)
	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:475)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$hasNext$1(FileScanRDD.scala:300)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:295)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:757)
	at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:80)
	at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$1(Collector.scala:178)
	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
	at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.Task.run(Task.scala:91)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:812)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:815)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

As we have been running osm4scala so many times and we didn't change anything on our side, we escaled this as a critical support case to Azure.
Unfortunately, this is the feedback we've received:

we no longer have the problem if we downgrade to 1.0.6
it is worth checking with the creators\support group of this library to see what changes are done post 1.0.6 version. As this is a third party library, we don’t have visibility to the backend configuration for these library.
(c) Azure Support

If we follow the stacktrace, you'll notice we'll end up at this part of your code:
com.acervera.osm4scala.spark.OsmPbfFormat:86

... which is this line:
val fs = path.getFileSystem(broadcastedHadoopConf.value.value)

Are there any insights what could be causing this very annoying behavior?
It is also very unclear from where the following classes are originating. I would assume those are Azure Databricks specific, but Azure support is not confirming this.
org.apache.spark.util.LocalHadoopConfiguration
org.apache.spark.util.SparkHadoopConfiguration

Aha, I was able to get it back working again:

go  to src/main/scala/com/acervera/osm4scala/spark/OsmPbfFormat.scala
revert this version to commit id: d4d706b
build the entire package: sbt assembly
deployed the snapshot version on Azure Databricks

Result is that it started to work again. So I can with 100% confidence we introduced a new bug in this commit: https://github.com/simplexspatial/osm4scala/commit/5f54f8d8ab6e627d43f95649a75af62967029329, but it got recently exposed (due to an underlying Azure Databricks upgrade).
Unfortunately, I cannot really tell you yet what exactly is wrong in OsmPbfFormat.scala

Hi @jochenhebbrecht
Which version of osm4scala are you using? Looks line Line 86 is a comment.
I'm going to try to reproduce the error using the last version v1.0.10

Yes, sorry, we are using 1.0.9 in production. We haven't upgraded to 1.0.10 yet because we still have to revert the workaround with the timestamp. So we're talking about this line. I've tested it with 1.0.10 as well, and the problem remains.
In the meantime, Azure support gave us a possibility to revert to the previous Databricks environment, but according to them, their underlying upgrade actually just exposes a bug on osm4scala and we would need to dive deeper on that one.

I was able to reproduce it in Azure.
Bug looks localized at com.acervera.osm4scala.spark.OsmPbfFormat.$anonfun$buildReader$1(OsmPbfFormat.scala:153) that is the equivalent to L86 in other version.
It is weird, because this line is getting the file system, so nothing special. Also, the NPE is triggered when in a no reader related class.

yes, it looks like the line val broadcastedHadoopConf = sparkSession.sparkContext.broadcast(new SerializableWritable(hadoopConf)) would need to be reverted to use SerializableConfiguration. (Similarly to the ParquetFileFormat)
I noticed that this change was done related to a bug, but it's not clear what the cause was: https://github.com/simplexspatial/osm4scala/pull/62

That fixed a back compatibility problem with Spark 2. So the problem could be a combination of the Spark version and the Hadoop version.
No time now to work on that. I hope to do it this evening.
Could you try to run in using Spark 2.4? Only to know if it is still working there.

When downgrading to 6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11) and installing the osm4scala-spark2-shaded_2.11-1.0.10.jar library, I can no longer reproduce the problem.

In AWS EMR with Hadoop 3.2.1 and Spark 3.1.2, version 1.0.10 is working without problems. Looks like there is something weird in Azure Databrick.

Thanks Ángel for already having a look at this. We still have the support case open and Azure are also looking further into this problem on their side. I'll provide this feedback as well. It is still perfectly possible something is broken on Azure Databricks side, and that your code is actually perfectly fine.
We are also still waiting for the source code on their side, to get smarter on where the NPE occurs

I will research more using Azure, but I will not have time until the weekend. Also, this type of error requires infrastructure that is not free. Do you know if Microsoft or Databricks would give me resources for testing? That would be great. Could ask them?
Thanks

That sounds like a very great idea, it's definitely something I would like to achieve. I will contact the support team of Azure and ask whether you could get some capacity on resources to further investigate this bug.

I cannot reproduce the problem on 8.3 (includes Apache Spark 3.1.1, Scala 2.12) and osm4scala:1.0.10 with cluster mode set to Single Node (instead of Standard). This would indeed learn us the problem is serializing the Hadoop configuration.

Feedback from Azure

Databricks team have already identified that the issue is caused by a maintenance release. They are currently investigating what change specifically caused the issue and a permanent fix to this issue will be planned for that.

So I would not worry too much on your codebase, let's give the support team some time and freedom to first investigate what they potentially broke.
With regards to asking resources to reproduce it on your end:

Thank you for letting us know about this. During the investigation if we get to a stage where help from third party library owner is required, we would definitely explore this option. Given that the investigation is underway, this is not immediately needed. But we appreciate your suggestion and willingness to collaborate.

I would suggest to close this ticket for now. We can always reopen it in case Azure Databricks would still come back to us indicating something's not valid in the codebase of osm4scala

Hi @jochenhebbrecht  Thanks for your update. I will close the ticket, but let's recap to keep it in mind. Maybe it is necessary to change something in the connector related to this, like find a way to avoid serializing the HadoopConfig object.
Tested in cloud providers with Spark 3.

AWS EMR multinode cluster worked fine.
Azure Databricks single node worked fine.
Azure Databricks multinode failed.

Looks like the error is related to Hadoop config serialization. It makes sense because in the single node execution serialization is not needed, so it does not fail.
FYI: I created ticket #103 to be able to test these types of problems quickly.

Just wanted to confirm that databricks fixed the issue on azure.


Re-divining doesn't clear notification

Observed behaviour with divine super combat pots: when the divine expires the notification is triggered as expected. When re-divining, the notification is not cleared.
Tested with melee boost threshold 0 & 1, same behaviour with both

I noticed some more weird behaviour, when I use salt in toa the notification is triggered


Two issues while running Spotify

Hi AVS team,
AVS SDK 1.8.1 ,  ARM based OpenWRT embedded Linux.
There are two issues while running Spotify.

While it set to DEBUG9, the streaming will get stutter, it seems the dump log issue. DEBUG5 is fine so far.
There is no display card supported while running spotify.

Thanks,

Hi @zeusshuang,
1- It is possible that the logs are affecting the audio quality. Usually, log statements are synchronized, so this could add an overhead to the media player thread. Do you see the same issue playing from a different source?
2- Do you mean in general or display cards related to spotify events?
Thanks,
Celina

Hi @celinval ,

so far I only see this case on Spotify, other Music playing, Prime and other radio streaming looks fine.
the display cards like playing Prime Music, it will indicate the current song name and playing information. From the Amazon APP, it will display Spotify current playing information, so I think it should be put on display card event ?

Thanks,

Hi @zeusshuang,
Thanks for the clarifications.

Please let us know if the same audio issue happens without logs and release build. As I mentioned, logs can cause performance degradation.
You are correct! Thanks for bringing this issue into our attention. I added your suggestion to our backlog.

Best,
Celina


Add gpgtools support for Yosemite by upgrading to 2014-12-b4

This is the current version being shown in project's homepage:
https://gpgtools.org
It says:

We encourage all of our users to install this release.
GPG Suite Beta 4 runs on OS X 10.6 and newer.

Problems are reported here:
http://support.gpgtools.org/discussions/problems/28092-mail-crashing-in-os-1010-yosemite-after-installing-gpg
Applying this locally works correctly as expected and I'm able to create emails and send them without having Mail.app crashing.

If you don't agree on having this one on main casks, I can make a PR against https://github.com/caskroom/homebrew-versions and provide a gpgtools-beta cask.

Yes, if the vendor recommends a development release, we can merge it here.
caskroom/homebrew-versions/pull/593 has been open for quite a long time, due in part to the complexity and unreliability of the cask, in part to unfortunate circumstances (neither @rolandwalker nor I can currently test).
I appreciate you verifying that Mail.app works normally. Before merging, could you confirm that there are no permission issues in ~/.gnupg, and that the Mail.app plugin works correctly?

@ndr-qef I confirmed that also because I saw a few reported issues on that and I was afraid it would mess with the permissions. I also did uninstall to check if those would be broken with it and then installed again and all good.
Basically, upgrading normally was good, uninstalling after the upgrade was good, and installing again was good :smile:

@alias-mac, thank you for your attention to these key tests. I "can" test it, and have been assigned for two weeks on a separate issue, but am behind in my responsibilities to this project.

All good thanks @rolandwalker!


Updating users ExternalIds

I need to update existing users externalIds. I follow the Wiki and see that I first need to create an object using Add-GSUserExternalId and then pass that object to the Update-GSUser command. However, when I try to create the first ID object the command always gives me an error saying: Add-GSUserExternalId : Cannot process argument transformation on parameter 'InputObject'. Cannot convert the "xxxxxxxxxx" value of type "System.Int32" to
type "Google.Apis.Admin.Directory.directory_v1.Data.UserExternalId[]".
My syntax looks like this:
$test = Add-GSUserExternalId -InputObject xxxxxxxxxx -Verbose
Update-GSUser -User<EMAIL_ADDRESS>-ExternalIds $test
Any idea what I might be doing wrong or is this a bug?

Hey @Chris4678 - The InputObject parameter set is there to allow piping existing ExternalId objects into it (i.e. for copying from another user). You'll want to use the other parameter set to specify the Type and Value of the external ID, i.e.:
$test= Add-GSUserExternalId -Type login_id -Value<PHONE_NUMBER>
Update-GSUser -User<EMAIL_ADDRESS>-ExternalIds $test

Well now the command runs without an error, however, I may still not be doing something correctly. What I want is for this ID to appear as the "EmployeeID" in Google. I dont see any direct command for updating employee ID and I thought this was it. Should I be using something else?

Hey I just figured it out. For anyone that finds this I had to change the -Type tag to "-Type Organization" Thanks for the help!

Nice!! Yeah, it's a bit odd with how it's worded but I'm glad you're sorted!!


@BeforeTest method in a super class will be called multiple time when alwayRun = true

If you have a base test class that contains a method with the annotation @BeforeTest (alwaysRun = true), it will be called by the amount of sub classes added to the testng.xml
I have a little code snippet to help you guys reproduce this issue, but I can't figure out how to attach it to this message...

You don't seem to understand the issue here.  @BeforeTest(alwaysRun = true)
has the same exact behaviour than @BeforeClass
Here is an example:

If I want to use @BeforeTest to run a configuration for "THE_TEST" only
once, since I have selected the group "sanity", I have no choice, but to
put wither @BeforeTest(groups = { "sanity"} ) or @BeforeTest(alwaysRun =
true).
The problem is:  @BeforeTest is supposed to run ONLY ONCE for "THE_TEST".
Because of the (alwaysRun = true) it runs 3 times (once before each
classes), as if I have used @BeforeClass(alwaysRun = true).   If I don't
have any groups, then I don't need the alwaysRun = true, and the behaviour
returns to normal (runs only once for "THE_TEST")
Please fix this.
Thanks!
On Thu, Sep 29, 2011 at 1:09 AM, lrivera <
<EMAIL_ADDRESS>
For before methods (beforeSuite, beforeTest, beforeTestClass and
beforeTestMethod, but not beforeGroups): If set to true, this configuration
method will be run regardless of what groups it belongs to.
For after methods (afterSuite, afterClass, ...): If set to true, this
configuration method will be run even if one or more methods invoked
previously failed or was skipped.
Always run is for before methods (beforeSuite, beforeTest, beforeTestClass
and beforeTestMethod, but not beforeGroups): If set to true, this
configuration method will be run regardless of what groups it belongs to.
For after methods (afterSuite, afterClass, ...): If set to true, this
configuration method will be run even if one or more methods invoked
previously failed or was skipped.
beforeSuite is only run once because it applies to the whole suite not
every class
http://testng.org/doc/documentation-main.html#annotations
--
Reply to this email directly or view it on GitHub:
https://github.com/cbeust/testng/issues/92#issuecomment-2234142

Ok, i got it, let me see what i can do

Ok, got it, let me see what i can do

I'm just waiting to the team accept my changes :)

That is great news.  Thank you.  My team really appreciate.
On Tue, Oct 25, 2011 at 1:41 AM, Julio Gutierrez <
<EMAIL_ADDRESS>
I'm just waiting to the team accept my changes :)
--
Reply to this email directly or view it on GitHub:
https://github.com/cbeust/testng/issues/92#issuecomment-2513280

Done, the fix is on the repo :D

@juherr - I think we can go ahead and close this issue. I tried using the same example that was shared by @mchiasson and I CANNOT recreate the problem in TestNG 6.10
Here's my output:
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
[XmlSuite] [WARN] 'parallel' value 'false' is deprecated, default value will be used instead: 'none'.
...
... TestNG 6.10 by Cédric Beust<EMAIL_ADDRESS>...

baseTestBeforeTest called
baseTestBeforeTestAlways called
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.314 sec - in TestSuite

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.588 s
[INFO] Finished at: 2016-12-09T23:05:04+05:30
[INFO] Final Memory: 22M/248M
[INFO] ------------------------------------------------------------------------

@krmahadevan Could you check if a test is covering the use case and propose a new one if not?

@juherr - The test test.bug92.Bug92Test is covering this.

`public class BaseTest {
@BeforeTest(alwaysRun=true)
public void BaseTestBeforeTest()
{
	System.out.println("baseTestBeforeTest called");
}

@BeforeTest(alwaysRun=true)
public void BaseTestBeforeTest1()
{
	System.out.println("baseTestBeforeTestAlways called");
}

} public class Test_1 extends BaseTest {
@Test
public void m1()
{
	//System.out.println("Test1");

}

@Test
public void m2()
{
	//System.out.println("Test1");

}

} public class Test_2 extends BaseTest{
@Test
public void m1()
{
	//System.out.println("Test2");

}

@Test
public void m2()
{
	//System.out.println("Test2");

}

} public class Test_3 extends BaseTest {
@Test
public void m1()
{
	//System.out.println("Test2");

}

@Test
public void m2()
{
	//System.out.println("Test2");

}

}

<test name="Test_3">
 <classes>
   <class name="testNGUseCases.Test_3"/>
 </classes>

`
output
baseTestBeforeTest called
baseTestBeforeTestAlways called
baseTestBeforeTest called
baseTestBeforeTestAlways called
baseTestBeforeTest called
baseTestBeforeTestAlways called
===============================================
temp Suite
Total tests run: 6, Failures: 0, Skips: 0
is this the correct output?


Linux install instructions verbose?

The Linux install instructions (GitHub, scala-sbt.org) are verbose,
such as the paragraph containing "but usually you don’t install your software from a local DEB file" could be replaced with a link to a more qualified external resource or simply removed.
When I installed the first few times, I would stop to read something I already knew as a Linux user :-)
The screenshot for "Software & Updates" is not entirely necessary either. There is an imbalance against the very concise Windows & Mac installation instructions.
Any comments and suggestions before I make a pull request for slimming down the instructions?
I think of covering the whole page.

These instructions were added over time (for example via https://github.com/sbt/website/pull/80) for reasons. If you want the spartan style, http://www.scala-sbt.org/0.13.2/docs/Getting-Started/Setup.html#installing-sbt is how it used to look for linux back then:

RPM package
DEB package

Even the developers who are familiar with Mac and Windows occasionally need to handle Linux machines for production or CI environments, and having more verbose info helps.

All right, thanks for the explanation!
I understand and won't be making a PR. Closing.


Invalid Diagnostic Position metrics for Record Symbol

Description:
Consider the Following Record definition

When checking the typeDefinition node, start line has become line 3 and end line has become line 1
This only happens when there are not fields within the record
Affected Versions:
v0.973.1-SNAPSHOT

This is not reproducible. I can see the correct line numbers for an empty record as below.

Issue can be observed when we look at the Diagnostics position of the typeNode

Already fixed in https://github.com/ballerina-platform/ballerina-lang/pull/9654.


(Python notebook) SVM Classification on DRR data

Hi Niloofar,
Thanks very much for committing！
I have reviewed the committed notebook, I may have a little bit of feedback as follows:

No number is shown in the confusion matrix.   Not sure if there is the GitHub render problem, please take a look at the rendered notebook ( here ).It may be good to see the numbers that we can know the performance.
It's always good to have some data description information, such as output the shape(total number) of considered data, how many for positive samples and how many for negative samples and output the top 5 or 10 rows for people to understand the data structure.  Could you also specify what dose the label 0 and 1 means for people do not familiar with the context?
Could you also specify the dataset in the GitHub link for the further reproducible?
Minor comment: besides the confusion matrix, will you also consider other evaluation metrics such as recall, accuracy, AUC to measure the performance in a number?

Thanks for the commit and please feel free to let me know if I misunderstood something :)


UX/UI Institutions - Editing program and offerings (when students are in it)

Describe the task
It was defined in a previous ticket 520 - all fields are editable (if NO students in it)
but we need to look at what happens if a student is in it.
What are the processes that need to happen?
Acceptance Criteria

[ ] Research workflow required process on ministry side
[ ] Research what happens with students (if anything)
[ ] Document Scenarios
[ ] What happens to the Draft Applications and Submitted Applications
[ ] Draft a workflow

Additional context

Use the new workflows that exist in place

https://app.zenhub.com/files/319429847/9a1dba75-bcd4-418b-bcf4-e70e2c8534db/download

Attached workflow showing what happens when institutions edit programs when students are enrolled.

Reopened to attach revised workflow with start showing institution must submit a request to change the program offering when students are enrolled.


fix: textmate service is not compatible with monaco 0.47

Types

[X] 🐛 Bug Fixes

Background or solution

Changelog
fix textmate service is not compatible with monaco 0.47

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.sarah.wangxj seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

/next


Add workflow_call to preview action to specify product version

Our infrastructure using the public preview images to pull from and test changes. This is useful for two reasons:

We can specify build versions that may occur between the CRON runs
We can specify builds outside the latest, e.g. release candidates, to pin our test infra to during regression

Oh wait, I think this is already handled here: https://github.com/rstudio/rstudio-docker-products/actions/workflows/build-preview-webhook.yaml

Yes! In fact, Workbench just started triggering specific builds in Jenkins here: https://github.com/rstudio/rstudio-pro/pull/3998


Test disabling transformers containers in docs CI

Related to https://github.com/huggingface/doc-builder/pull/487 and internal slack thread. There is now a custom_container option when building docs in CI. When set to "" (instead of "huggingface/transformers-doc-builder" by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487.
cc @mishig25

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Fix JDBC generated jar dependency

Purpose
$subject
Approach
Fix correct toml dependency
Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #23643 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #23643   +/-   ##
=======================================
  Coverage   14.59%   14.59%
=======================================
  Files          51       51
  Lines        1411     1411
  Branches      219      219
=======================================
  Hits          206      206
  Misses       1189     1189
  Partials       16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cdce86...3217f69. Read the comment docs.


Multi template.path

Hello everyone! I have a question on the creation of multiple Templates.path. I have my views patterns within my views folder, I created another folder called admin (containing what will be the admin panel).
I did my normal routes to the views that are outside the admin folder. Ok, so far so good.
But en route to the admin, I configured a new Templates.path to view the admin, this last while trying to access the address / admin an error appears on the screen, something like it was not possible to render the default layout of the view admin. The low leave the link to my config.php and admin route:
admin route: https://gist.github.com/raisiqueira/4902efdff256f75f84c2
config.php: https://gist.github.com/raisiqueira/b70d535bd16bf4197af4

Thats my solution ->
$app->get('/admin', function() use($app){
$adminView = $app->view();
$adminView->setTemplatesDirectory(ADMIN_VIEWS);
$app->render('layout.php',array('pagina'=>'login'));
});


Improve AST walking / mutating and pass manager

This is a follow up of #1607 / #1603

This adds some of the infrastructure required to easily add new AST walking and manipulating passes. By adding new passes instead of bolting more features into existing ones we can keep the code simpeler. E.g. we could have a pass that checks for statements that are always hold (e.g. if (1<2)) and either warn the user or simplify the generated code. To make this easy this adds:

A Visit() function to all visitors to do indirect dispatching. Instead of calling node->accept(this) we call Visit(). This makes it easier to write a generic visitor (like the node counter)
A Mutator() which visits the current tree and returns a new modified tree. To do this it implements its own virtual dispatch table.
A pass manager that makes it easier to run passes and do some diagnostics (e.g. print the tree) after each pass.

And some refactoring to make everything a bit easier to read.
#1603 has an example mutator pass that removes all PositionalParam nodes from the AST and replaces them with constants which simplifies the semantic analyser and generates efficient code. Its not fully up to date, once this is in I'll finish that one. (it should allow us to remove useless probereads as well). I want to use this for  #1539  too.
The diff is a bit ugly but each commit has details :)

Looks like clang doesn't like my code, should be a small fix

@danobi this is ready for review. I'll fix the clangformat issue when everyone agrees on this change. My local clangformat doesn't agree with the one we use here on some minor syntax things :/

@danobi this is ready for review. I'll fix the clangformat issue when everyone agrees on this change. My local clangformat doesn't agree with the one we use here on some minor syntax things :/

any objectiosn to landing this @danobi ? :)


Fix sesman signal processing

edit 2022-03-04: No longer draft.
This series of commits fixes #1729 and may also address reports of other session ending problems (e.g. #2021).
The "self-pipe trick" [Stevens 1992] is used to keep the processing performed in all signal handlers to a minimum. This is really simple to do in xrdp by using g_set_wait_obj() and related calls.
As a result, SIGPIPE and SIGHUP are now processed from the sesman main loop, like all other events.
Other problems with SIGHUP processing have been addressed:-

The log file was closed and re-opened. Since the log file is shared with the child processes, this will cause the parent and the children to have a different file table entries. Writes at the end of the file will not be synchronised.
The existing code does not allow for the listening port to be changed, which is a major reason why the user may have edited the config in the first place.

This series of 4 commits addresses these issues as follows:-

[tidy-up] Some of the global variables defined in sesman.c are declared in other c files as extern. This commit moves these declarations into sesman.h
[tidy-up] struct log_config contains a few members which are not always needed. For example, the per_logger_level member is only used for debug builds. This commit makes these members (and associated code) conditional.
The 3rd parameter of the log_start() call is change from a boolean dump_on_start to a more generic flags field. One of the new flags is LOG_START_DUMP_CONFIG, which provides the existing dump_on_start functionality. The other one, LOG_START_RESTART allows for logging to be restarted smoothly. Existing log files are left untouched by a restart, even if the name changes.
Finally, sesman is updated as follows:-

Signal handlers now just call g_set_wait_obj(). No significant processing is done in the signal handlers.
In the main loop, the wait objects are handled in the same way as file descriptors. All actions resulting from signals are synchronised with actions resulting from file activity.
The restart code now calls log_start() with LOG_START_RESTART, and allows the user to change the listening port dynamically.

As another note, when developing with sesman, it's now really easy to change the logging level, or add entries to [LoggingPerLogger]. After editing the config, a sudo pkill -HUP xrdp-sesman causes the file to be re-read without needing to stop and start sesman.


When creating an object in Salesforce the field ownerID should be optional

The API does not mandate it - when missing the authenticated user will be the owner

@ShkarupaNick could you please update the status here.


BITMAKER-2214: add minor fixes on documentation.

Description
As part of @Jgaldos onboarding ticket, some minor fixes on documentation were added.
Issue

BITMAKER-2214

Checklist before requesting a review

[x] I have performed a self-review of my code.
[x] My code follows the style guidelines of this project.
[x] I have made corresponding changes to the documentation.
[x] New and existing tests pass locally with my changes.
[x] If this change is a core feature, I have added thorough tests.
[x] If this change affects or depends on the behavior of other estela repositories, I have created pull requests with the relevant changes in the affected repositories. Please, refer to our official documentation.
[x] I understand that my pull request may be closed if it becomes obvious or I did not perform all of the steps above.

@Jgaldos  This PR is ready to merge

@raymond1242 merged


Mouse coords vs. SVG coords

There is a mix in some places between SVG coordinates and DOM element offsets. Make sure all mouse coordinates are always transformed to SVG coordinates for drawing operations. That will make sure drawing also works correctly on scaled/responsive images.
Bonus benefit: once that is implemented, we should be able to reuse this as-is for the OpenSeadragon implementation.

that would be a killer feature

This is now almost complete. There is one remaining issue, which is that the resize handles scale with the image. That's undesired - only the shape should scale, the resize handles should remain same size.


Simplify experience when using endpoint-based connection: leverage DefaultAzureCredential by default

The Connect(Uri, TokenCredential) API forces the consumer to always specify a token credential, which to me seems fairly unnecessary considering we have a very sane default called the DefaultAzureCredential class.
I propose one (or 2) of the following:

Create a new overload to IConfiguraitonBuilder.AddAzureAppConfiguration that takes just a Uri object, and uses new DefaultAzureCredential behind the scenes.
Modify the existing AzureAppConfigurationOptions.Connect(Uri, TokenCredential) method to make the tokenCredential argument optional: if it is not specified, default to new DefaultAzureCredential
[Alternative to 2 if default arguments are to be avoided] Create a new overload for AzureAppConfigurationOptions.Connect(Uri, TokenCredential) that takes only the Uri argument and defaults the credential internally to new DefaultAzureCredential

I believe this would significantly streamline configuration, especially when you connect to multiple instances like we do. This is currently our implementation for context:
builder.Configuration
    .AddAzureAppConfiguration(c =>
        c.Connect(
            endpoint: builder.Configuration.GetValue<Uri>("Azure:AppConfiguration:Common:Endpoint"),
            credential: new DefaultAzureCredential()))
    .AddAzureAppConfiguration(c =>
        c.Connect(
            endpoint: builder.Configuration.GetValue<Uri>("Azure:AppConfiguration:Endpoint"),
            credential: new DefaultAzureCredential()));

And this is how it would look like with option 1 above:
builder.Configuration
    .AddAzureAppConfiguration(builder.Configuration.GetValue<Uri>("Azure:AppConfiguration:Common:Endpoint")
    .AddAzureAppConfiguration(builder.Configuration.GetValue<Uri>("Azure:AppConfiguration:Endpoint");

PS: The reason we fetch the endpoint from configuration itself is because we have dedicated instances of Azure AppConfig for each of our environments.
I'm aware this would add a new dependency to Azure.Identity on the package (since every single implementation of TokenCredential comes from there) but to me that seems fully justifiable and I'd even make the argument that it would simplify this interaction even more to the consumer as he doesn't need to be aware of a separate package that needs to be introduced to even be able to call these endpoint-based overloads.
If the team is somehow concerned over this new dependency addition, then I'd propose this new overload exist as part of a separate Microsoft.Extensions.Configuration.AzureAppConfiguration.Identity package that references both Microsoft.Extensions.Configuration.AzureAppConfiguration and Azure.Identity. I personally think that would be overkill, however.
This expands:

https://github.com/Azure/AppConfiguration-DotnetProvider/issues/105

And apparently is somewhat of a "rollback" of this?

https://github.com/Azure/AppConfiguration-DotnetProvider/issues/88

Looks like at the time there were some nasty dependencies on NewtonsoftJson and System.Interactive.Async that don't exist anymore in Azure.Identity.

Hi @julealgon, the only thing I'm worried about here is the dependency on Azure.Identity, otherwise I think it kinda makes sense to use DefaultAzureCredential since that's what our examples will always show anyway. Recently I've noticed that Azure.Identity often has a new vulnerability that requires releasing a new version, and I don't know if we'd want to be consistently releasing new versions of this repo to update that dependency. I don't really know the nature of those vulnerabilities, but if it's marked as moderate severity it sounds like it'll require an update.
Introducing a separate package does seem like kind of a big deal as well if it's just for convenience. That might make releasing new versions easier, but we'd need to actively track the Azure.Identity package for vulnerabilities, and I'm not sure that's worth it. Let me know if you think differently.

Thanks @amerjusupovic .
I will say... we just hit another hurdle because of this due to key vault references... now we are also forced to add this (in my opinion incredibly unnecessary):
.ConfigureKeyVault(kv => kv.SetCredential(new DefaultAzureCredential());

I just don't see why things like this are not the default setting. Not only we have to specify it explicitly, but it also looks incredibly redundant considering we just specified that credential for the appconfig instance itself on the Connect method.
This API surface is not as user friendly as it could be ATM.

I kind of agree with @julealgon.
@zhenlan @drago-draganov what's your take on this?

Here are my thoughts.

It's often a design preference. Some prefer the "just works" magic, while others dislike it because it makes debugging difficult when things go wrong. Although the DefaultAzureCredential simplifies things to some extent, it doesn't tell the whole story. Users still need to understand which credential is used in each environment and ensure the proper role assignment. For more details, see my discussion on authentication with token credentials. Therefore, our design favors being explicit rather than relying on magic.

While the DefaultAzureCredential offers certain conveniences, it comes with costs such as debugging challenges, performance overhead, and unpredictable behavior. To learn more, see the Usage guidance for DefaultAzureCredential, which recommends using more specific token credentials.

The Azure.Identity library (where the DefaultAzureCredential resides) inherently carries higher security risks due to its nature. Depending on DefaultAzureCredential means we must explicitly depend on the library, which forces a specific version of Azure.Identity on our customers, even if they don't use Entra ID authentication at all.

@zhenlan

Here are my thoughts.

It's often a design preference. Some prefer the "just works" magic, while others dislike it because it makes debugging difficult when things go wrong. Although the DefaultAzureCredential simplifies things to some extent, it doesn't tell the whole story. Users still need to understand which credential is used in each environment and ensure the proper role assignment. For more details, see my discussion on authentication with token credentials. Therefore, our design favors being explicit rather than relying on magic.

If you think it should always be explicit like that, then I believe the class should be renamed to something else other than DefaultAzureCredential. Having Default in the name of a concrete implemetation of an interface, to me, signals it is the concrete class used by default, and that it can be replaced with something else.
I don't consider this "magic" behavior. It is just a sensible default. If it could not be overridden, then I'd agree with you.

While the DefaultAzureCredential offers certain conveniences, it comes with costs such as debugging challenges, performance overhead, and unpredictable behavior. To learn more, see the Usage guidance for DefaultAzureCredential, which recommends using more specific token credentials.

Some of these issues seem like problems on the SDK side? Debugging challenges and performance overhead should be able to minimized at the library side, surely? I also hope the perf overhead is only on the first access, as the information is cached. If that's not the case I'd be a lot more worried about it and would like to know.
Regarding "unpredictable behavior", to me, that's a super weird way of saying "being configurable using environment variables", which to us is an obviously beneficial feature. Without support for environment variables, I have no safe way to provide AZURE_CLIENT_SECRET to a bunch of legacy on-premise applications.  I strongly disagree with that perspective. If that was the case, anything "configuration-driven" would be just as problematic since configuration can be loaded from sources external to the application.
I'm a bit surprised such absurd argument (IMHO) is in the official docs.

The Azure.Identity library (where the DefaultAzureCredential resides) inherently carries higher security risks due to its nature. Depending on DefaultAzureCredential means we must explicitly depend on the library, which forces a specific version of Azure.Identity on our customers, even if they don't use Entra ID authentication at all.

You could easily create a separate package that contained the base AzureAppConfig library plus Azure.Identity and have it add extension methods there to configure the provider in a more sensible manner for those using Azure credentials though. For those who don't need that, they could keep just referencing the base package.
Many Microsoft libraries are built this way, so again it surprises me the team here sees that as some sort of big challenge.

In any case, I do hope the team changes its mind eventually. I'd like to avoid having to maintain custom extension methods/private nugets to streamline the API. I wish I didn't have to.


Add support for hostname keyword

Partial implementation of
https://github.com/magicmark/composerize/issues/359
Tests added

released - thanks!


[feature-request] Please create both Bitcoin and Bitcoin Cash wallets at first-run wizard

Bitcoin-com-wallet-3.6.0-win64 on Windows 7 x64. (Chrome v60)
Default wallet create is "Bitcoin", while I recommend to create BOTH Bitcoin and Bitcoin Cash wallets. (with different address spaces, to reduce user confusion)
-Technologov

We plan to add this in the next release.

I hope we are. Going to make. Some bitcoins


Add cumprod to the standard

This RFC requests to include a new API in the array API specification for the purpose of computing the cumulative product.
Overview
Based on array comparison data, the API is available in the majority of the libraries in the PyData ecosystem.
Prior art

NumPy: https://numpy.org/doc/stable/reference/generated/numpy.cumprod.html
PyTorch: https://pytorch.org/docs/stable/generated/torch.cumprod.html
MXNet: https://mxnet.apache.org/versions/master/api/python/docs/api/np/generated/mxnet.np.cumprod.html
TensorFlow: https://www.tensorflow.org/api_docs/python/tf/math/cumprod

Proposal:
def cumprod(x: array, /, *, axis: Optional[Union[int, Tuple[int, ...]]] = None, dtype: Optional[dtype] = None) -> array

dtype kwarg is for consistency with sum et al

cc @oleksandr-pavlyk

@steff456 for the axis kwarg, this should be limited to just an int and not a Tuple. Based on the various docs, libraries seem to only support an int.

One thing to note is that PyTorch, e.g., requires an axis argument and does not support NumPy's default behavior of computing the cumulative sum over the flattened array.


Taverns Vertical Slice Merge

PR Checklist

Please check if your PR fulfills the following requirements:
N/A
PR Type
What kind of change does this PR introduce?

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation update
[ ] Other... Please describe:

What is the current behaviour?

Currently doesn't have Common project updates with Missions updates
What is the new behaviour?

Updated common project with Mission updates
Other information

It looks like the BoardGameMoveMessage is being spam sent, have not seen this before but I believe it is a problem with HandlePlayerInput, I will have to put some condition on it but haven't been able to load any breakpoints yet though I also see the "Join Online Arena" so I believe my VS is messed up..


Support for reflection probes on ocean shader?

Foreword:
1st of all thanks for this amazing solution, which is by far the best I've seen! Crest was the reason I started to end up creating a triple-A tech test, where I put the best environment plugins into one common denominator, with the plan to create one day a game with it.
This is how I came so far....
https://vimeo.com/297553979
https://vimeo.com/297549976

ISSUE:
Would it be possible to add reflection probes to the shader? That would give an awesome look, when it comes to relfecting a whole island. I would create a specific very low reflection object for it.

Wow, that's nuts! <3 Very nice, thanks a lot for posting that, and please feel welcome to post your progress again in the future!
The system reports dynamic planar reflections (assign ocean planar reflections script to camera, and enable "planar reflections" on ocean material). Would this do the trick?
Otherwise, while I haven't tried it myself, I'm told one can simply feed the output of a dynamic reflection probe straight into the cubemap input - see this post in issue #4 from @Midda-C :
https://github.com/huwb/crest-oceanrender/issues/4#issuecomment-387584688

Wow, that's nuts! <3 Very nice, thanks a lot for posting that, and please feel welcome to post your progress again in the future!

With pleasure! =) ayntime!

Otherwise, while I haven't tried it myself, I'm told one can simply feed the output of a dynamic reflection probe straight into the cubemap input - see this post in issue #4 from @Midda-C :

Sadly, I am not a programmer, so I am not really able to follow the procedure xD :}
I also have troubles to get the buoyancy running! :/
As a non programmer I have to dive deeper, than I am able to xD
I have to try the 1st method you have mentioned! =) The second solution from Midda-C sounds much more interesting and performance wise. I think so...

I went again thru and did managed to get the reflection probe working, but you were right, the planar reflections do work great! =) and woooow, now it illuminates correctly! And it has its dreamy look!  Thank you!

Screenshot or it didn't happen :).
Seriously though glad it worked

Screenshot or it didn't happen :).

Ha-Haaa! I am uploading an ultra sharp video =) - on youtube :(
I did some tuning on the water and shader parameters - again - the whole time! It makes very addictive playing around with all the parameters of crest! And instead to make a simple screenshot, I start to play around with all the settings again xD - typically artist! :/ This tooked hours - till now shakeshead
video is up in a moment.....

Here it is.... https://www.youtube.com/watch?v=iRdANlplxL8&feature=youtu.be
I forgot how efficient planar reflections can be! =) Thanks again!

Thats awesome! Thanks for posting. Can i ask how you got specular on the water so nice and sparkly?

Looking neat indeed! I tried reflection too, unfortunately, I'm getting flickers on distant objects but only in reflection. Tried changing clipping plane on both ends but alas that doesn't work.
I'll try to demonstrate the bug once I restore my Unity installation, that darn Windows 10 update almost killed my machine  >:(

.... how you got specular on the water so nice and sparkly?
Yea, I know what you mean. It's really a tricky way to achieve the right effect, since there are five interdependent parameters, that react almost as same way, from the visual perspective. Which makes it somehow confusing, even if the naming of the parameters make totally sense. Turn more than two of the sliders in the wrong direction - and the whole effect gets lost, and it gets hard to find back.
I marked you the parameters on the screenshot: The arrows I added on the sliders show in which direction you should go rather to achieve a more smaller/detailed and spread sparkle effect. I hope that helps =)

Wow, thanks.. Next time I do a tweaking pass I'll give this a stab.


Devdojo Chatter Service Provider not found

I just want to remove this package. This package gives me a headache. I'm experiencing this error.
how to fix this error.
(1/1) FatalThrowableError
Class 'DevDojo\Chatter\ChatterServiceProvider' not found
in ProviderRepository.php (line 208)

Have you added the service provider to your app.php?

First try to remove the devdojo/chatter line your composer.json file and run composer update, if that doesn't work try composer dump-autoload

``

(1/1) FatalThrowableErrorClass 'DevDojo\Chatter\ChatterServiceProvider' not found
--
in ProviderRepository.php (line 208)
at ProviderRepository->createProvider('DevDojo\\Chatter\\ChatterServiceProvider')in ProviderRepository.php (line 144)

I just tested this with Laravel 5.4  and came across the same issue. ANy fixes? I am determining which versions of Laravel are supported


Bugfix: update color scheme references properly

CascadiaSettings::UpdateColorSchemeReferences had two bugs in it:

we would never check/update the base layer
we would explicitly set the color scheme on a profile referencing the
old name

This PR fixes both of those issues by checking/updating the base layer,
and ensuring that we check if a profile has an explicit reference before
updating it.
Since the affected code is in TSM, I also created an automated local
test.
Validation Steps Performed
Bug repro steps.
Specifically tested DHowett's scenario too.
Test added.
Closes #9094

@msftbot merge this in 5 minutes


OrbStack 1.6.3 is causing errors while building Symfony (PHP) Container

Describe the bug
After updating the OrbStack to 1.6.3 me and my colleague are having an issue with building the Symfony container.

When we fix (just comment out) the problematic line, it starts complaining about other. I see there are similar (but no the same) issues like #1295.
To Reproduce
Hard to say, but probably setting up a fresh instance of Symfony with OrbStack will cause issues.
Expected behavior
It'll work
Diagnostic report (REQUIRED)
No response
Screenshots and additional context (optional)
No response

I can confirm this is happening as well on my end after updating to 1.6.3.

Same issues here, upgrading breaks multiple installs (symfony, sulu, drupal)
Downgrading to 1.6.1 fixes it.

Same problem here. Searching files with symfony/finder returns 0 results. Looks like something is wrong with the filesystem. After a downgrade everything works as expected

I can confirm that this issue is global. Downgraded to 1.6.1 and everything works okay.

I can confirm I am having the same issue on Symfony 5.4

Same issue with Nette Framework as well. Downgrade to 1.6.2 works.

Same with Laravel

Kind of same here. For me RecursiveDirectoryIterator does not "see" any files anymore in directory anymore after Upgrading to 1.6.3

Happened to me as well You have requested a non-existent service. I wasted a lot of hours trying to figure out what happened until I switched back to docker desktop and it works again.

Fixed in a new hotfix release: v1.6.3 (17138). The old version was v1.6.3 (17136). Go to the OrbStack menu > Check for Updates or run orb update to get the fix.
Thanks for the reproducer! We'll use this to test future changes. This bug was caused by a failed attempt to fix #1287, but the reproducer reveals the real issue. Sorry for the inconvenience.

Thanks for the quick update.

Thank you so much for the quick fix

Thank you @kdrag0n !!


[GH-#369] Represent naming state using PMT

resolves #369

I think it would be better to have the state transform for persistence, in the persistence module, than in the chain worker.

I think that persistence module should do exactly what the name says - persist.

I assume the transformation is only done for persistence, so this has nothing to do with the chain either.


Update cache expiration time if refresh operation failed

When a refresh operation fails, we should also extend the cache expiration time for all refresh registered settings. This avoids making excessive requests to AppConfig service in case of failed refresh operations.

            }

If anything throws in this if block, do we still consider the refresh a success? In other words, should we retry upon failures?

Refers to: src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationProvider.cs:584 in e113102. [](commit_id = e11310251b5a7be45c46fa077ec98dc4b7477867, deletion_comment = False)

            }

If anything throws in this if block, do we still consider the refresh a success? In other words, should we retry upon failures?
Refers to: src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationProvider.cs:584 in e113102. [](commit_id = e113102, deletion_comment = False)

We have already updated the cache expiration time before SetData is called. The current refresh operation will fail if SetData throws, but the next refresh operation will be triggered after the cache expires.

using System;

Please include the license header.

Refers to: src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Extensions/TimeSpanExtensions.cs:1 in b531bdb. [](commit_id = b531bdb2dbaf3e5a7b9540f9e5b72ce9c571edd0, deletion_comment = False)

            throw new UnauthorizedAccessException("No key vault credential or secret resolver callback configured, and no matching secret client could be found.");

This exception will be swallowed if called during refresh, right? I guess we still need a way to extend the cache.

Refers to: src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureKeyVaultReference/AzureKeyVaultSecretProvider.cs:76 in a84a990. [](commit_id = a84a99047e60d10f8225d96697d22cbd4331c865, deletion_comment = False)

            throw new UnauthorizedAccessException("No key vault credential or secret resolver callback configured, and no matching secret client could be found.");

This exception will be swallowed if called during refresh, right? I guess we still need a way to extend the cache.
Refers to: src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureKeyVaultReference/AzureKeyVaultSecretProvider.cs:76 in a84a990. [](commit_id = a84a990, deletion_comment = False)

During refresh, this exception can only occur when a sentinel key update triggers RefreshAll, and we received a new KVR that this app has never resolved before. So now we are trying to resolve a KVR that doesn't have any registered SecretClient and there is no SecretResolver callback configured. This KVR can never be resolved no matter how many times we retry. So we never cache null value for this failure case and therefore, this failed KVR cannot affect the next refresh time of other secrets.

            throw new UnauthorizedAccessException("No key vault credential or secret resolver callback configured, and no matching secret client could be found.");

During refresh, this exception can only occur when a sentinel key update triggers RefreshAll, and we received a new KVR that this app has never resolved before. So now we are trying to resolve a KVR that doesn't have any registered SecretClient and there is no SecretResolver callback configured. This KVR can never be resolved no matter how many times we retry. So we never cache null value for this failure case and therefore, this failed KVR cannot affect the next refresh time of other secrets.

What if the current KVR is the key being monitored (or the sentinel key)?

In reply to: 960161400

Refers to: src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureKeyVaultReference/AzureKeyVaultSecretProvider.cs:76 in b531bdb. [](commit_id = b531bdb2dbaf3e5a7b9540f9e5b72ce9c571edd0, deletion_comment = False)


auto-configuration with postgresql

A Java JPA application connect to a "compose for postgresql" database using auto-configuration.
The databaseName of the datasource in server.xml is generated, e.g. databaseName="${cloud.services.postgresqlDB.connection.name}", but the variable is binding to "VCAP_SERVICES.compose-for-postgresql.creadientials.name" in runtime-vars.xml, which is "bmix_dal_yp_14f08adb_....". This is not the name of the database.
<variable name='cloud.services.postgresqlDB.connection.name' value='bmix_dal_yp_14f08adb_....
The JPA application can't connect to the postgressql database.
The error message is "java.sql.SQLException: FATAL: database "bmix_dal_yp_14f08adb_*** does not exist DSRA0010E: SQL State = 3D000, Error Code = 0."
P.S. it works fine if the databaseName is configured manually.

This has been addressed on Pull Request #333


refactor+fix trace server url initialization

Both TraceServerUrlProviderImpl and TraceViewerEnvironment are
riddled with concurrency issues. A lot of events happen asynchronously
but the logic doesn't take this into account, leaving the application
vulnerable to race conditions when trying to fetch the trace server url.
Refactor by removing TraceViewerEnvironment and moving its
functionality directly into TraceServerUrlProviderImpl, add logic to
prevent race conditions.
Unfortunately I was forced to introduce a LazyTspClient which is
implemented as a proxy to a TspClient and defers all async method
calls until baseUrl is asynchronously resolved. This only works because
all TspClient methods are asynchronous.
The proxy is a temporary measure which should buy you time until
you guys rework the frontend initialization sequence to make sure
everyone expects and handles the asynchronous steps.
Fixes https://github.com/theia-ide/theia-trace-extension/issues/493
Closes https://github.com/theia-ide/theia-trace-extension/pull/499

Some notes about the use of the LazyTspClient: classes seem to want to synchronously get a reference to a TspClient, but all work done through it is asynchronous. All places that uses the TspClient deal with asynchronous calls. This makes me think that it should be possible to instead store a promise to a reference of a TspClient and first await on this before using its async methods. We're dealing with async stuff anyway when it gets time to use it.

PR is good for re-review/merging again!

Ok I am ashamed of continuously removing your approvals but this wasn't just nitpicking.
The value returned by getTraceServerUrlPromise() used to be set once, and any subsequent call would only ever return the initial value. This is now fixed.
Thanks for bearing with me!

Thank you all for working on this! 🌳


Ready for release

Capstone Project
In this PR:

We set up an HTML document with the relevant placeholders for data to be fed in from JS.
We made a request to an API for the data.
We created a popup window that comes to life upon a clicked button.
We implemented counters for the data displayed on the homepage as well as the number of comments available.
WE implemented a like functionality that updates the number of likes when a button is clicked.
this is a link to the video presentation

The presentation video in the description has been updated

Please note that the PR has been updated with the new video as per request of the code reviewers

Presentation video updated in the Description


Explore supporting StreamExecute/OLAP workloads

Within some contexts, especially serverless, the memory limits are very low in process. So it can be desirable when performing aggregations in process, or if also streaming out to the end user, to be able to leverage StreamExecute from our API.
I will say, this is going to be rather non-trivial, but I'd like us to see what it'd take to support without fully adopting connect-web or protobufs.
Streaming requires adopting the Connect protocol, rather than simply application/json, which can be done, and is done within connect-web. https://connect.build/docs/protocol#streaming-response
To add to this, this feature is required to support OLAP queries.
cc @reconbot

This is a functioning POC https://gist.github.com/mattrobenolt/6ca8c37185578cac8970de5a82dcfd52
It definitely needs more work. I also talked with the buf folks, and they agree that we should vendor in their codec stuff like I'm doing here.

I need to use streaming api with Cloudflare Workers. Mysql drivers that support streaming are not compatible with workers. Is there any plan to support this ?


Crashes with Spells&Shields

I was hoping either this or Health Overlay would help the overlay to the spells&shields' extra mana bar (basically adds 1 or 2 HUD bars between breath and hunger but this mod crashes it.

The crash report
crash-2022-10-11_03.36.56-server.txt

It could be a neat and pretty compatibility to have, but I'm also content with just having it marked noncompatable or something.

Oh, completely forgot to add!

Ack, Pressed the wrong buttons
I meant to add that purple and green would be good colors IF you do actually want to make it function

Oops, it's probably my fault. I didn't update the mod to the latest forge version. If you could try downgrading a few versions until I get a fix ready in a few hours.

I would honestly just like Health Overlay to stay a colored health/absorption only mod, so probably the best to request this feature on Spells & Shields' issue tracker as it would be a slightly annoying workaround to make this work in an external mod.


"Anisotropy could not be calculated - ellipsoid fitting failed" when no. Directions > ~800

Ellipsoid fitting becomes unreliable at above around 800 directions. This compares rather unfavourably with BoneJ1 in which ellipsoids are fitted to 50,000 vectors by default. Notably, the default values on a fresh install of BoneJ2 is 2000 directions: a user's first experience of this plugin is likely to be "it didn't work".

This issue was most likely most likely fixed in PR #105. The problem was unrelated to the number of directions, rather we had incorrect criteria for checking whether a quadric equation could be an ellipsoid. The fix will be in the next release. If the problem persists in the next release, please reopen issue.


Cutting a W-spider with the magic wand generates a segmentation fault in Windows

I run ZXLive via Git Bash on Windows 10. I draw two-leg W-spider, then move to derivation mode and attempt to cut the W-spider using the magic wand (to split it into two W-spiders). This immediately causes my session of ZXLive to end, with Git Bash indicating a segmentation fault.

Do you get any error messages?

ZXLive immediately closes when I do with no error message. Git Bash's response is just "Segmentation fault" and moving on to a fresh prompt.
That said, while testing today I realized I'd missed that the fault only occurs if I try to cut the dot-marked wire of a W spider. If I cut one of the unmarked wires, the spider unfuses in the expected way. The former isn't a proper operation in the first place, but having that user error result in ZXLive closing doesn't seem ideal.

Hmm, probably it tries to reference a node that doesn't exist, due to some special case behaviour with the W-spider. Spider cutting with the W needs to have special case behaviour anyway


fix: e2e short

Closes: #XXX
What is the purpose of the change
When setting up the e2e parallel tests, I did not check the e2e-short command. The only two issues were:

E2E short was starting so fast that it didn't give the nodes time to spin up before starting the first test, causing an error
One of the E2E short tests were changed to use chainB, when chainB does not exist for E2E short tests, so this was changed back to chainA

It also appears that sometimes, the E2E long would also error running the first test without the sleep, so for now I am adding it there and will further diagnose the issue post CL

Can you file an issue as a reminder to ourselves?

Sure can


Signal RenderHighDensity fix and refactor

Purpose:
Fix not used maxRenderIndex in RenderHighDensity() mode. #448
Additionaly refactored RenderHighDensity().
For now it easy to switch between parallel and sequental implementation without code duplicates.
useParallel return back as Signal public field. No idea how better implement it in API, but let it stay configurable (imho). Parallel calculations not always better... #419
New functionality (code):
User can switch to sequental calculations by
var signal = plt.PlotSignal(....);
signal.useParallel = false;

New functionality (image):
Growing data demo correct render:

I have no idea which sort order Azure prefers. VS made the right sort order, as I see.

Fix not used maxRenderIndex in RenderHighDensity() mode

Thanks for identifying the problem and correcting it @StendProg!

Additional refactored RenderHighDensity(). useParallel is now public, so it is easy to switch between parallel and sequential implementation without code duplicates.

This is a wonderful contribution. The file is much easier to read now, and your use of LINQ is excellent! Thanks!

No idea how better implement it in API, but let it stay configurable (imho). Parallel calculations not always better...

You raise a good point in #419 that I did not consider: the user may have a good reason to not use parallel mode, so we should maintain both parallel and non-parallel code, and make it easy for the user to switch between them.
In this PR you made useParallel a public property, and this is a good idea.
I will reverse a change made by 6e6c86612ba15b02377cc2295588d8ef5a135bff and add bool useParallel = true as an optional argument in Plot.PlotSignal()


landing page footer

Footer section of landing page is quite messy after recent commit. Make it good.

assign me

@Astrodevil please look into PR #445 . I have added my project stopwatch application (issue #428). Along with that I also tried to sort out the bug in the footer section. Hopefully it will be sorted out. The mess was created due to my previous PR #410 which was merged earlier.


Intellisense not working

I've upgraded the vs code to the latest version ,and the Salsa was enabled.
Now, the code intellisense does not work without an jsconfig.json file. but version 0.10.6 worked well.

Same issues. I hope Salsa can works without a jsconfig.json. How about contain a default jsconfig.json without manual creating one?

In my case Intellisense (even with a jsconfig.json) only works for modules that have been required but not for modules that have been imported using ES6 import:

You can find my sample project here: https://github.com/nilshartmann/vsctest
My environment:

Visual Studio Code Version 0.10.8 with Salsa preview enabled
No extensions
MacOS
TypeScript installed locally (1.9.0-dev.20160218)
jsconfig.json (tried several more variants all without success):

{
    "compilerOptions": {
        "target": "ES6"
    },
    "exclude": [
        "node_modules"
    ]
}

I'm not sure if this is the same problem or to better open a new Issue?

This issue was moved to Microsoft/TypeScript#7166


Transfer workflows with history

Transfer doc related workflows from the workflows repo.

Part of issue #416 in the stock-synthesis repo.
Will update workflows by removing .yml and branches when actions are successful.

@kellijohnson-NOAA do these workflows look good to you now?

I noticed that the release html version needed to be updated with the recent changes to the navbar, otherwise it doesn't work right so I thought I would fix it with this PR.


[question] obs and new obs in DQN callback

Hi,
I noticed that the obs and new_obs entries of locals() call in the DQN callback done in deepq/dqn.py:188  are identical (apart from the first step after a reset), probably because of the obs = new_obs done at line deepq/dqn.py:214. Is this intentional for any reason?
Best,
Joao

Hello,
This is normal. We want to store transitions in the replay buffer. A transition is composed of:

the observation at time t
the action at time t
the received reward
the next/new observation, which is the observation at time t+1

Once stored, we need to keep track of the current observation for the next transition, hence the obs = new_obs.

It makes sense, thank you for your reply!


test: Functional Test Hello World!

:tada: This PR is included in version 0.4.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Jira Export unable to map the parent issue correctly

Describe the problem
A clear and concise description of what the problem/bug is.
I am trying to export the custom project from Jira to Azure DevOps. While creating the export the parent issue/epic is being mapped with the issue id instead of the issue reference. But, as the origin id contains the reference, the actual parent is not being mapped.
The following are details of the issue. ( P.S. modified the name of the project for security constraints)
When I export the JIRA issue to xml, the parent field is shown as follows:
<parent id="111210">MyProj-282</parent>
My config file has following link mapping:
{
"source": "Parent",
"target": "System.LinkTypes.Hierarchy-Reverse"
},
After export, the issue contains the following value:
"Links": [
{
"Change": 0,
"TargetOriginId": "111210",
"TargetWiId": 0,
"WiType": "System.LinkTypes.Hierarchy-Reverse"
}
],
Where as the files are created as MyProj-282.json
Hence, when performing the import, the import utility is unable to find "111210" file. So, it is not linking the parent epic.
I tried other ways by adding source-type. But, the link mapping does not support source-type.
Tool version

Version of the jira-azuredevops-migrator tool [e.g. 3.0.139]

Hi Team,
We have been encountering issues with Parent/Child links while migrating issues from JIRA to ADO for the Team managed project types. Migration Tool version: v3.0.139 used.
The epic link field and the link map in JSON file is provided like:
"epic-link-field": "Epic Link",
"link-map": {
"link": [
{
"source": "Epic",
"target": "System.LinkTypes.Hierarchy-Reverse"
},
{
"source": "Parent",
"target": "System.LinkTypes.Hierarchy-Reverse"
},
{
"source": "Child",
"target": "System.LinkTypes.Hierarchy-Forward"
},
{
"source": "Relates",
"target": "System.LinkTypes.Related"
},
{
"source": "Blocks",
"target": "System.LinkTypes.Related"
},
{
"source": "Cloners",
"target": "System.LinkTypes.Related"
},
{
"source": "is duplicated by",
"target": "System.LinkTypes.Related"
},
{
"source": "Duplicate",
"target": "System.LinkTypes.Duplicate-Forward"
}
]
}
If we check the JSON of the child item for the parent the link, it maps:
"Links": [
{
"Change": 0,
"TargetOriginId": "xxxxx",
"TargetWiId": 0,
"WiType": "System.LinkTypes.Hierarchy-Reverse"
}
The same migration if done with the 2.3.117 version, it works fine- but we end up encountering issues with broken inline images.
Could you help us to get this fixed or any alternate solutions available?

Maybe related to https://github.com/solidify/jira-azuredevops-migrator/issues/772?

Hi @Alexander-Hjelm,
Yes. Could you try reproducing this issue and help us out on getting them resolved?


scipy > 1.9.3 breaks QuarticFermionicSimulationGate decompose method.

Tracking this here. A bug was introduced into scipy for linalg.sqrtm for complex valued matrices which appears starting v1.10.0 (see #18250). This causes our tests to fail for QuarticFermionicSimulationGate due to loss of precision. I'll pin the scipy dependency to < 1.10.0 for the moment.

@fdmalone
Do you know if there is a fix for this?

It hasn't been fixed in scipy yet, but there does look to be an open PR. If there's no more activity there we may need to implement our own OR relax our test thresholds. The square root function is only used once in openfermion.

I'll look into it. Worst case we could probably just write a custom square root function.

Fixed with #864


Fix this bug introduced by upstream: thread waiting

Upstream declined my pull:
https://github.com/slime/slime/pull/770
can it be fixed on vlime's side?

It is fixed - at least on the JSON branch, see ac21427d85234b39ae301a0723bf9ab1126157ef.
I cherry-picked that as cfaac6ccb379f9f85bfb2ef17ca6369dfadf2be6 as well, is that enough?

I wasn't able to get the json branch working.
I didn't cherry pick the commit, I reverted it though? I just did a git bisect until I found it, then reverted and patched. I suppose it is better to actually fix the problem on the client end so I'll close it.


fix compile error in c++17

In c++17, the beta function is global.
Therefore, compiling with c++17 fails at quaternionFilters.cpp.

Thank you for your help!


Add a nopoll version of the POSIX TCP transport profile

This profile is similar to the POSIX UDP nopoll transport profile. It allows TCP to be used with LwIP, without FreeRTOS+TCP.

The implementation is basically a clone of src/c/profile/transport/ip/udp/udp_transport_posix_nopoll.c with "UDP" replaced by "TCP" and uxr_disconnect_tcp_platform() added.

Build status:

Linux
Windows

@Acuadros95 S&M if you are ok

@jseldent please rebase onto develop

@jseldent please rebase onto develop

Done

Build status:

Linux
Windows


e2e: try to parallelize VM installation in Ginkgo e2 tests

This could be good to either speed up the installation/test and a good way to test parallel node installation in Rancher Manager.
See this Ginkgo doc: https://onsi.github.io/ginkgo/#spec-parallelization

The Ginkgo parallelization is in fact a way to execute multiple "spec" files in parallel, not a way to start the same test in parallel.
After some tries goroutine seems the way to go, but a lot of changes are needed in bootstrap_test.go file.

With the latest changes I was able to provision 10 nodes in less than 1 hour compared to 5 nodes in the same time previously.
I did some quick tests with 20 nodes but I had issues. I need to go deeper to find why.


nsrails - intuitive integration with a rest api

I've been using nsrails to integrate my ios app with a mvc rest api and have been having an amazing time with it. Here's some code using it:
NSRRequest *req = [NSRRequest requestToFetchAllObjectsOfClass:[DevicePermission class]];
[req setQueryParameters:@{@"user":[[self user] pk]}];
[req sendAsynchronous:^(id json, NSError *error) {
    if(error) return;
    [[self permissions] addObjectsFromArray:[DevicePermission objectsWithRemoteDictionaries:json]];
}];

Its really nice because developers define objective-c classes that act exactly like orm models. Think its a good fit for this list? (I'm not a contributor to the library)

Looks good! Feel free to send a PR. :shipit:


Feature/minor additions

Some minor additions to have:

a MTLequalsTo: on Object with a default implementation
remove a useless copy
add a convenient printString implementation for MatcherResult

Pull Request Test Coverage Report for Build 834596186

1 of 11 (9.09%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.9%) to 83.799%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/PharoTransformationLanguage/PTLMatcherResult.class.st
0
10
0.0%

Totals

Change from base Build 832481716:
-0.9%

Covered Lines:
750

Relevant Lines:
895

💛  - Coveralls


Wrong asm doc for instructions with multiple documentation pages

There are multiple mov mnemonics in the Intel syntax, so looking only at the mnemonics is not enough

MOV within GPRs
MOV to/from CR registers,
MOV to/from DR registers

When adding suffixes the AT&T syntax is even more ambiguous since there are Intel instructions with q suffix in the mnemonic

MOVD/MOVQ: move between GPR and mm/xmm registers
MOVQ: move between mm/xmm registers or between mm/xmm register and memory

Here's the demo code. The AT&T asm doc always shows the vector version of MOVQ even though the instruction only acts on GPRs like movq %rsp, %rbp. On the Intel side CE can't recognize MOVs to/from control and debug registers
Other examples:

CMPSD and MOVSD for vector operations
CMPSD and MOVSD for string operations

For other cases like MOVSQ or CMPSQ we're lucky since dropping the suffix we still arrive at the same document page

As per the other ASM bugs, this is due to the way we parse Intel's documentation. Any help welcomed!


[#859][Improvement] Set MALLOC_ARENA_MAX in start-shuffle-server.sh

What changes were proposed in this pull request?
Reduce memory usage for some versions of glibc. Prevent oom.
Why are the changes needed?
Fix: https://github.com/apache/incubator-uniffle/issues/859
Does this PR introduce any user-facing change?
No.
How was this patch tested?
UT

Will it influence the performance of writing data? Have you test the patch in your local environment?

Will it influence the performance of writing data? Have you test the patch in your local environment?

I am a little doubtful about the effect of this pr. Because we don't have compress/decompress operation in the shuffle server.

Will it influence the performance of writing data? Have you test the patch in your local environment?

I am a little doubtful about the effect of this pr. Because we don't have compress/decompress operation in the shuffle server.

Does the Resident Set Size of the shuffle server process continuously increase in your environment？

Merged to master.

I never observe the resident set size. If I have time, I will observer it.

Have you used the Uniffle in your production environment?

Currently deployed in a test environment, running tpcds tests.

I'm still suffering from this issue even after this PR when setting rss.server.flush.localfile.threadPool.size to a very large value like 1000 in high concurreny scenarios.
The resident memory of the shuffle server can grow larger than Xmx + MaxDirectMemorySize, and as time goes by, the OS will kill the Java process.

I think we still need to find out the reason. @rhh777 @jerqi


Remove duplicate flag explanations

All subctl command explanations have their own flag descriptions on the subctl page. --kubeconfig and --context flags are common and default to all subctl commands. Remove these flag mentions as part of each command description and add a common section explaining that these flags apply to all subctl commands.

The common section already exists: https://submariner.io/operations/deployment/subctl/#common-options


Turbine Fluid Port

I'm not sure if this is a bug, or a function of the Turbines that I was unaware of. I wasn't able to find any information regarding it, and it's causing me quite bit of confusion.
Is there a limit to the number of Turbine Fluid ports that will draw steam at once? In my current set up, I have a large number of them but only five (The ones marked with red in the image) are actually pulling steam into them.
Each port is set up identically. A large drum is pushing steam using Extra Utilities Transfer Nodes into a smaller drum next to each port, which in turn has a liquid transfer node (each with a stack upgrade), connected to the port below it.
Yet only 5 of them actually pull steam (tested by removing the transfer pipe that is supplying the drums. Anyway. Is this a bug, or a feature?
http://i62.tinypic.com/2ecpxed.png

Turbines can only accept an amount of steam equivalent to their internal tank size each tick, which they will then empty. Once the steam tank is full, it won't matter how many inlet fluid ports you add.


Define reasonable viewports

Description
Although added to Storybook, the addon for viewports has not been configured and therefore only presents limited viewport options that are not the most illustrative of the features offered by Vocabulary.
Expected behavior
The addon should offer a richer selection of screen sizes and devices.
Additional context
The documentation for the addon describes the steps to define a new viewport: https://www.npmjs.com/package/@storybook/addon-viewport

Hey! I can take this up.
Thank you!
Dhruvi

@Dhruvi16 please go ahead.

Hey! I have a doubt about configuration. I am not sure if I need to configure the addon globally or per component and how does it make a difference?
Thank you
Dhruvi

The addon should be configured globally as all components need to satisfy our responsiveness criteria.


CC: 18.2 version and image updates

Product Version(s):

Issue:

Link to docs preview:

SME review:

[ ] An SME has approved this change.

DOCS review:

[ ] A member of the docs team has approved this change.

Additional information:

Merge checklist:

[ ] Deploy preview inspected wherever changes were made
[ ] Build completed successfully
[ ] Test have passed

LGTM


Adding EDPM based Zuul Job

It deploys the EDPM compute on single node.
It deploys the EDPM compute on single node.
This pull request includes following hooks:
- disable_os_marketplace.yml - Disabled CRC warning
- restart_nova_schedular.yml - nova schedular service is restarted

It improves following roles/playbooks:
- cluster_info.yml - collect data related to openshift cluster
- ci/playbooks/edpm/run.yml - Run edpm-deploy playbook for edpm
  deployment

scenarios/centos-9/edpm_ci.yml - contains the scenario files for
edpm CI deployment

It improves the edpm_deploy role to execute make edpm_deploy.
It updates the timeout to 5400.

This PR has:

[x] Appropriate testing (molecule, python unit tests)
[x] Appropriate documentation (README in the role, main README is up-to-date)

Recheck

recheck

/approve

recheck

recheck

recheck


Refactor KDA infrastructure as a module

Move Kinesis Data Analytics resources inside a module

If is possible delete the logs.tf and add the code to the main.tf. Adding a log.tf file for 10 lines of code doesn't make much sense for me.
Create a locals in the main and add variable prefix=flink-app- to reuse it  like a prefix  in the rest of the code.
Add these variables to vars.tf and leave them with a default so that if they change tomorrow you don't have to modify the submodules:

runtime_environment
python
jarfile

Creates a variable block with the following parameters something similar to this:
variable "rolesets_gcp" {
  description = "Variable to create roleset in gcp project."
  type = map(object({
    backend      = string
    roleset      = string
    project_name = string
    secret_type  = optional(string)
    token_scopes = optional(list(string))
    namespace    = optional(string)
    binding = list(object({
      resource = string
      roles    = list(string)
    }))
  }))
  default = {}
}

Add a file with outputs or similar.
Added a readme file to all submodules and modules with this content:
# Infra module
_description of the module_

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

and a precommit file .pre-commit-config-yaml whit this content and run pre-commit tu auto generate inputs and outputs:

repos:
  - repo: git://github.com/pre-commit/pre-commit-hooks
    rev: v3.3.0
    hooks:
      - id: trailing-whitespace
      - id: check-json
      - id: check-yaml
      - id: end-of-file-fixer
      - id: detect-aws-credentials
      - id: mixed-line-ending
  - repo: git://github.com/antonbabenko/pre-commit-terraform
    rev: v1.45.0
    hooks:
      - id: terraform_fmt
      - id: terraform_validate
      - id: terraform_docs


Implement KIP-005 (Message Signing)

Implement Kaspa-specific private key message signing (KIP-005)
https://github.com/kaspanet/kips/pull/8

Completed in https://github.com/kaspanet/rusty-kaspa/pull/278


fix(dialog,bottom-sheet): invert backdrop color on dark themes

Similarly to how we handle the sidenav backdrop and ripples, these changes invert the color for the dialog and bottom sheet backdrops in dark themes.
Note: this is a resubmit of #13065.

@crisbeto it looks like this is changing the backdrop color for light themes as well, which is causing a lot of screenshot diffs

AFAIK this isn't in the spec. Closing.


[Bug]: Disk I/O error on Google Drive

What happened?
I am storing my persistent ChromaDB in Google Drive.  Everything was going great until my database file hit around 16GB, now I can't upload any more documents, and I get an error: I/O Disk Error.
Versions
Chroma 0.4.15 in Colab
Relevant log output
OperationalError                          Traceback (most recent call last)

[<ipython-input-15-98b03a86c398>](https://localhost:8080/#) in <cell line: 40>()
     50         start_time = time.time()
     51
---> 52         collection.add(
     53             ids=[str(uuid.uuid4()) for i in range(0, batch_size)],  # IDs are just strings
     54             documents=[recordo.full_case_text for recordo in cases_batch_list],

4 frames

[/usr/local/lib/python3.10/dist-packages/chromadb/db/mixins/embeddings_queue.py](https://localhost:8080/#) in submit_embeddings(self, topic_name, embeddings)
    170             # the results. https://www.sqlite.org/lang_returning.html
    171             sql = f"{sql} RETURNING seq_id, id"  # Pypika doesn't support RETURNING
--> 172             results = cur.execute(sql, params).fetchall()
    173             # Reorder the results
    174             seq_ids = [cast(SeqId, None)] * len(

OperationalError: disk I/O error

@Tylersuard, that is interesting. Have you checked whether you've reached your disk usage quota?


lots of drupal errors on staging

URL: http://staging.beta.dosomething.org/campaigns/1286/confirmation

@aaronschachter this may be because i did clean slate on promoted images


update goss verify.yml to allow to pass host to run on

This PR allows for the goss verifier to run different tests on certain hosts.
Example: have a side-effect playbook, which makes keepalived move an IP to a different host in an active-active scenario. The verifier should verify that the IP actually moved to a different host, and that the side-effect playbook killed the supervised process.
Basically what you can do with testinfra by specifying the host in .get_hosts
testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE'])\
    .get_hosts('instance1')

You can do this for goss by prepending the test file with test_host_$hostname_as_given_in_molecule.yml.
PR Type

Feature Pull Request

Thanks for your patience @loomsen. Our CI is back in shape. It reported:
https://travis-ci.com/ansible/molecule/jobs/179293976
Warning, treated as error:
/home/travis/build/ansible/molecule/molecule/verifier/goss.py:docstring of molecule.verifier.goss.Goss:51:Error in "code-block" directive:
maximum 1 argument(s) allowed, 18 supplied.
.. code-block::
    test_default.yml (will run on all hosts)
    test_host_instance1.yml (will run only on instance1)
    test_host_instance2.yml (will run only on instance2)
ERROR: InvocationError for command '/home/travis/build/ansible/molecule/.tox/doc/bin/python setup.py build_sphinx -n -W --builder=html' (exited with code 1)

Just missing the --signoff DCO for the commit :cold_sweat:

Gah, every single time.

Woohoo, finally :)

Oh no .... we still didn't merge this ....
@loomsen, can you resolve the final merge conflicts? (sorry :pray:).
We're waiting on one final issue wrt. packaging and then we can merge this (for real).
:sweat_smile:

OK, so I pulled in the latest changes. Code style check fails, which shouldn't be related to this PR, and also a test randomly fails at destroy stage, maybe one of you could re-run just this build, if possible?
@decentral1se thanks for bearing with me :)

No, thank you!!!

It appears to be related to https://github.com/ansible/molecule/pull/1911!