DoSafely() doesn't actually do anything

When running a task with DoSafely, it never actually performs the task.
Here is an example main.go file:
package main

import (
	"io/ioutil"
	"log"

	"github.com/jasonlvhit/gocron"
)

func task() {
	log.Println("I am runnning task.")
	d1 := []byte("hello\ngo\n")
	ioutil.WriteFile("/tmp/dat1", d1, 0644)
}

func main() {
	s := gocron.NewScheduler()
	s.Every(3).Seconds().DoSafely(task)
	<-s.Start()
}

Test it out - if you swap in Do for DoSafely it works as expected, outputs to the terminal and writes the file. With DoSafely nothing happens.

@preslavrachev would you have the time to take a look into it? if not please let me know

@JohnRoesler @Streppel hey, guys, I'll have some free time by the end of the week. I promise to have a look.

@Streppel feel free to assign it to me.

If there is anything additional I can provide - let me know! Thanks for taking action so quickly.

I have had a brief look on the issue, in order to see, whether I can replicate it. I can. The funny thing is that the DoSafely function does not do anything more than set a deferred recovery, but otherwise, it should delegate straight to Do. As the underlying func I provide does not get executed at all, I am starting to believe that it has to do with some copying of value, where a pointer should have been in use.
I will keep you posted.

P.S. @Streppel one Q from my side. is there a particular need to require 1.13 in the go.mod file? I have not upgraded yet, so I simply commented it out for my purposes. I don't think that 1.13's adoption will become mainstream so quickly.

@preslavrachev no specific reason indeed. Which version are you currently using?

If I run a test just for the purpose of debugging, it generates a panic
func TestDoSafely(t *testing.T) {
	success := false
	Every(1).Second().DoSafely(mutatingTask, &success)

	<-Start()
}

=== RUN   TestDoSafely
panic: reflect: Call using []interface {} as type *bool

goroutine 21 [running]:
reflect.Value.call(0x133fae0, 0x13d4a70, 0x13, 0x13be056, 0x4, 0xc000144000, 0x1, 0x1, 0x5dbc7f46, 0x7ab5548, ...)
	/usr/local/Cellar/go/1.12.9/libexec/src/reflect/value.go:377 +0x1457
reflect.Value.Call(0x133fae0, 0x13d4a70, 0x13, 0xc000144000, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.12.9/libexec/src/reflect/value.go:308 +0xa4
github.com/jasonlvhit/gocron.(*Job).run(0xc0000cadc0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/john/go/src/github.com/jasonlvhit/gocron/gocron.go:129 +0x415
github.com/jasonlvhit/gocron.(*Scheduler).RunPending(0xc000118000)
	/Users/john/go/src/github.com/jasonlvhit/gocron/gocron.go:516 +0xa0
github.com/jasonlvhit/gocron.(*Scheduler).Start.func1(0xc0000cc410, 0xc000118000, 0xc00010e1c0)
	/Users/john/go/src/github.com/jasonlvhit/gocron/gocron.go:605 +0x42
created by github.com/jasonlvhit/gocron.(*Scheduler).Start
	/Users/john/go/src/github.com/jasonlvhit/gocron/gocron.go:601 +0x87

Process finished with exit code 1

@Streppel i have created a PR with the fix, and some minor improvements. Please, check it out and make sure to run the tests locally. Please, also keep in mind that three tests failed on my machine even when running them out of master:

TestDaily
TestWeekdayAferToday
TestWeekdayAt

Those should not have been caused by my changes. I suggest however that we open up a new issue, where we take care of fixing them, and set up some basic CI (GitHub Actions or Travis) to prevent further such cases.

@preslavrachev thank you! Yea, indeed the tests could use some revision... I'll take a look at this as soon as I get some time, thanks for the warning. I'll be creating the issues :)


Kublet watches Pods.

Added a kubelet config source for watching pods on apiserver.
The pods are converted to boundpods for merging with other
config sources.
Added code to watch services and inject env vars to containers.

This is a work in progress. Don't review.  I'm sharing it to coordinate with @pmorie on PR #3331

@pmorie I like your solution of writing your own cache.ListerWatcher rather than what I was trying to do (reuse pkg/client/cache).  It is cleaner for testing.

@erictune Pushed rebased (but not squashed) code with tests to my branch; I believe the issue of leaking service env vars from shadowed master services no longer exists.

This is based on pmorie's #3331.
The last commit could be reviewed now if anyone is interested.
The tests all pass except for a failure in pd.sh e2e which seems unrelated.

Looking at #3331 and this PR-- this PR seems to have an awful lot of commits not in #3331. Perhaps you can re-fetch @pmorie's change and rebase? Or just wait for it to get merged, I guess.

Okay, this has been rebased and is ready to review.
I still want to run e2e one more time in parallel with that.

Does this actually enable reading pods from apiserver instead of boundpods from etcd?

yes for GCE and other salt-based providers because the --api_servers flag is already set on kubelet, and its meaning is changed by this PR to mean do pods too.   Other cloud providers to have their flags set later.

the apiserver source is configured in pkg/standalone/standalone.go

OK I looked at the testing in kubelet/config/api_test.go and I'm a little nervous about this, especially considering that we just broke this over etcd. You say e2e still works? Unfortunately we would have needed a soak test to find the thing we broke over etcd.

What testing would you like to see?
On Wed, Jan 14, 2015 at 4:35 PM, Daniel Smith<EMAIL_ADDRESS>wrote:

OK I looked at the testing in kubelet/config/api_test.go and I'm a little
nervous about this, especially considering that we just broke this over
etcd. You say e2e still works? Unfortunately we would have needed a soak
test to find the thing we broke over etcd.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/3376#issuecomment-70020719
.

@lavalamp What specifically are you referring to with 'we just broke this over etcd' ?

I think lavalamp meant https://github.com/GoogleCloudPlatform/kubernetes/pull/3479

added the test that @lavalamp said he wanted.  it passed.

LGTM

Fixes #2483


Does User-space drivers, also available for ordinary Android phone?

Hi all,
Sorry if my question sound silly to you, but I really need to build an app that simulate a HID device to inject hardware event to be accessed by other apps. I found User-space drivers on https://developer.android.com/things/sdk/drivers and it looks like exactly what I needed. Except, it is listed under Android Things.
Does User-space drivers also available for commercial Android phone? Thanks.
Elton

No user-space drivers are only an api that can be used on Android Things devices.


Create an executable using ocra on Windows

I thought it would be easy to do, but it turns out it's not as easy as I thought.
I need to investigate how to make an executable using libui...

In order to build a exe with Ocra, it's important to include 3 DLLs from ruby_built_in_dlls folder:
libssp-0.dll, libgmp-10.dll and libffi-7.dll
ocra app.rb --add-all-core --gem-all=fiddle --dll ruby_builtin_dlls/libssp-0.dll --dll ruby_builtin_dlls/libgmp-10.dll --dll ruby_builtin_dlls/libffi-7.dll --chdir-first --icon assets\app.ico --windows --verbose --output out\app.exe
I've also included --gem-all=fiddle to ensure all of the Fiddle files are included.

Amazing!
I have confirmed that I can use this command to create an application that runs on Windows. Thank you very much!


[AIRFLOW-1294] Backfills can loose tasks to execute

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[X] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1294

Description

[X] Here are some details about my PR, including screenshots of any UI changes:

In the backfills we can loose tasks to execute due to a task
setting its own state to NONE if concurrency limits are reached,
this makes them fall outside of the scope the backfill is
managing hence they will not be executed.
Several bugs are the cause for this. Firstly, the state
reported by the executor was always reported as success, ie.
the return code of the task instance was not propagated.
Next to that, if the executor already has a task instance
in its queue it will silently ignore the task instance
being added. The backfills did not guard against this, thus
tasks could get lost here as well.
This patch introduces CONCURRENCY_REACHED as an executor
state, which will be set if the task exits with EBUSY (16).
This allows the backfill to properly handle these tasks
and reschedule them. Please note that the CeleryExecutor
does not report back on executor states.
Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Looking at an approach to updated tests.
Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@aoen @saguziel @mistercrunch

Codecov Report

Merging #2356 into master will decrease coverage by 0.23%.
The diff coverage is 53.19%.

@@            Coverage Diff             @@
##           master    #2356      +/-   ##
==========================================
- Coverage   69.47%   69.23%   -0.24%
==========================================
  Files         145      145
  Lines       11070    11105      +35
==========================================
- Hits         7691     7689       -2
- Misses       3379     3416      +37

Impacted Files
Coverage Δ

airflow/executors/celery_executor.py
0% <0%> (ø)
:arrow_up:

airflow/models.py
87.05% <100%> (-0.09%)
:arrow_down:

airflow/utils/state.py
87.09% <100%> (+0.43%)
:arrow_up:

airflow/executors/local_executor.py
64.81% <16.66%> (-3.19%)
:arrow_down:

airflow/jobs.py
74.22% <60%> (-0.93%)
:arrow_down:

airflow/executors/base_executor.py
93.1% <66.66%> (-1.54%)
:arrow_down:

airflow/executors/sequential_executor.py
92.3% <66.66%> (-7.7%)
:arrow_down:

airflow/bin/cli.py
50.22% <0%> (-1.84%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e23d28...9b73588. Read the comment docs.

I think it should be the responsibility the executor to reschedule tasks to run, with no "special" message passing between task instance run function and executors like sys.exist(EBUSY) (the task should just record it's state in the DB and the executor should use that to determine what to do with the task). Couldn't the job see that the state was None after running the task and reschedule appropriately?
I'm seeing the flow as something like this
Job -> Executor -> TI.run()
The executor will facilitate the running of the airflow command to run to completion without knowing anything about the task itself or communicating with the Airflow DB. It should be quite dumb. It will be the responsibility of the job to e.g. check TI state is none and re-send it to an executor.
Am I completely off here?

The current logic is Job -> Executor -> LocalTaskJob -> TI.Run.
We have Executor and LocalTaskJob to monitor the execution of tasks. You need this kind of logic to guard against tasks that are not able to set themselves to a certain state, e.g. out of memory, sigkill, segfault etc (which are set by the os). Thus here you cannot count on what the state of the TaskInstance is.
As we are executing the tasks as subprocesses and cannot just rely on what the task reports as its state, we will need to rely on how the OS passes messages (errors in this case) between the processes it runs.
In backfills we are relying on the monitoring (already in 1.7.x) to verify that what the task is reporting as state in congruent with what the executor is thinking the task can report at all. The need for CONCURRENCY_REACHED/EBUSY is to make sure the executor and the task stay in sync. Otherwise the executor would report either success or failed which would not make sense in case of State.NONE (or any other state that does not signal SUCCESS or FAILED). I didn't add CONCURRENCY_REACHED to the task states as that would requiring the Scheduler code to handle that as well. I tried to keep the changes to a minimum.

Forgot to mention that when I was not using this new executor state I ended up in all kinds Race conditions: the state of the task could already have changed to queued before the executor was able to report back.
This triggers me to attempt one more simplification

Closing this in favour of the simplified version


Update README example for multiple prefixes / suffixes

I was struggling with using multiple custom prefixes / suffixes.
I saw this comment in an issue, which showed a .babelrc that looked a bit different from the README, and that seems to work correctly - https://github.com/entwicklerstube/babel-plugin-root-import/issues/74#issuecomment-301325742
I figured I'd update the README and ask whether this is the correct way to use multiple prefixes / suffixes in one swoop 😄

Thanks! Both are supported, but there's no reason to use the 'paths' key currently, as there are no other top level options.


hdl_path_slice implementation

Hello once more,
I am trying to implement backdoor access to registers and facing an issue when trying to use hdl_path_slice field  property.
While playing in the actual  register RAL  class definition i noticed that the following  command does what i need :
this.REGNAME.add_hdl_path_slice("REGNAME", 29, 3);
Which eventually writes the relevant 3 msb bits to the regname with backdoor access.
Would you kindly provide a small example how to achieve such a class command using rdl  hdl_path/hdl_path_slice properties ?
I noticed  there is some  mentioning in Spec Errata regarding the hdl_path_slice property, but to be honest i did not understand whether this is a bug or not.
Thank you ,
Oleg Greenberg

Hello once more,
I am trying to implement backdoor access to registers and facing an issue when trying to use hdl_path_slice field property.
While playing in the actual register RAL class definition i noticed that the following command does what i need :
this.REGNAME.add_hdl_path_slice("REGNAME", 29, 3);
Which eventually writes the relevant 3 msb bits to the regname with backdoor access.
Would you kindly provide a small example how to achieve such a class command using rdl hdl_path/hdl_path_slice properties ?
I noticed there is some mentioning in Spec Errata regarding the hdl_path_slice property, but to be honest i did not understand whether this is a bug or not.
Thank you ,
Oleg Greenberg

Well dig a little inside uvm_reg  and found the following 👍
{%- for field in node.fields() %}
{%- if field.get_property('hdl_path_slice') is none -%}
{%- elif field.get_property('hdl_path_slice')|length == 1 %}
{{inst_ref}}.add_hdl_path_slice("{{field.get_property('hdl_path_slice')[0]}}", {{field.lsb}}, {{field.width}});
{%- elif field.get_property('hdl_path_slice')|length == field.width %}
{%- for slice in field.get_property('hdl_path_slice') %}
{%- if field.msb > field.lsb %}
{{inst_ref}}.add_hdl_path_slice("{{slice}}", {{field.msb - loop.index0}}, 1);
{%- else %}
{{inst_ref}}.add_hdl_path_slice("{{slice}}", {{field.msb + loop.index0}}, 1);
{%- endif %}
{%- endfor %}
{%- endif %}
{%- endfor -%}
Does this mean the add_hdl_path_slice will always start from position 1 ?
Thank you ,
Oleg


add_linear_inequality_constraint to dimod

This PR adds add_linear_inequality_constraint method for both DQM and BQM objects. For BQM object it adds Log2 (slack_range) number of binary slack variables to the constraint. For DQM Object there are two options: 1) log2 adds log2(slack_range) number of dqm variables each with two cases 2)  log10 method adds log10 dqm variables each with up to 10 cases.
Note that this method only handles integral coefficient correctly. When there are fractional coefficients for variables it is assumed that user will multiply both sides of equations by factor of 10s to remove fractional coefficients.

Codecov Report

Merging #837 (605098c) into main (c8aadbf) will decrease coverage by 0.21%.
The diff coverage is 72.00%.

@@            Coverage Diff             @@
##             main     #837      +/-   ##
==========================================
- Coverage   88.83%   88.61%   -0.22%
==========================================
  Files          69       69
  Lines        5695     5770      +75
==========================================
+ Hits         5059     5113      +54
- Misses        636      657      +21

Impacted Files
Coverage Δ

dimod/binary/binary_quadratic_model.py
92.01% <68.75%> (-0.96%)
:arrow_down:

dimod/discrete/discrete_quadratic_model.py
88.04% <74.41%> (-2.82%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8aadbf...605098c. Read the comment docs.

Also, one more thing, can you add the new methods to the release notes? See https://github.com/dwavesystems/dimod#release-notes. Let me know if you have trouble or it's not clear.
Something like
Add `BinaryQuadraticModel.add_linear_inequality_constraint` and `DiscreteQuadraticModel.add_linear_inequality_constraint` methods.


fix: export correct esm and cjs

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>4:
0.0%

Covered Lines:
23

Relevant Lines:
23

💛  - Coveralls


Align the kotlin version in the gradle plugin with gradle

Right now we are getting this warning when compiling the detekt-gradle-plugin:
w: Runtime JAR files in the classpath should have the same version. These files were found in the classpath:
    /Users/brais.gabin/.gradle/wrapper/dists/gradle-7.3-all/dfe0rb4hnplvqkibhoc45how8/gradle-7.3/lib/kotlin-stdlib-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/wrapper/dists/gradle-7.3-all/dfe0rb4hnplvqkibhoc45how8/gradle-7.3/lib/kotlin-stdlib-common-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/wrapper/dists/gradle-7.3-all/dfe0rb4hnplvqkibhoc45how8/gradle-7.3/lib/kotlin-stdlib-jdk7-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/wrapper/dists/gradle-7.3-all/dfe0rb4hnplvqkibhoc45how8/gradle-7.3/lib/kotlin-stdlib-jdk8-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/wrapper/dists/gradle-7.3-all/dfe0rb4hnplvqkibhoc45how8/gradle-7.3/lib/kotlin-reflect-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.5.31/ff5d99aecd328872494e8921b72bf6e3af97af3e/kotlin-stdlib-jdk8-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.5.31/77e0f2568912e45d26c31fd417a332458508acdf/kotlin-stdlib-jdk7-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.4.31/63db9d66c3d20f7b8f66196e7ba86969daae8b8a/kotlin-reflect-1.4.31.jar (version 1.4)
    /Users/brais.gabin/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.31/6628d61d0f5603568e72d2d5915d2c034b4f1c55/kotlin-stdlib-1.5.31.jar (version 1.5)
    /Users/brais.gabin/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.5.31/43331609c7de811fed085e0dfd150874b157c32/kotlin-stdlib-common-1.5.31.jar (version 1.5)
w: Consider providing an explicit dependency on kotlin-reflect 1.5 to prevent strange errors
w: Some runtime JAR files in the classpath have an incompatible version. Consider removing them from the classpath

Now that we use composite compilation we should be able to use kotlin 1.5 for our gradle-plugin and 1.4 for the rest of the project to ensure compatibility, right?

we should be able to use kotlin 1.5 for our gradle-plugin and 1.4 for the rest of the project to ensure compatibility, right?

The problem is that any other Gradle plugin that imports kotlin-sdtlib contributes to this problem. So even if we align the version, there might be another gradle plugin that is not aligned and triggering this warning.

But at least our users can't blame us for those errors.

But at least our users can't blame us for those errors.

Technically it would depend on the Kotlin versions that contributes to a customer's build, so technically they could blame us regardless of the Kotlin version we expose

Ok, so then there's nothing here we can do.


[Feature request] Read QR codes from storage.

I'm looking for a way to scan QR codes that are present in the gallery. This is a feature no QR scanning library offers and it's really essential for some applications like cryptocurrency wallets.
Do you know if this could be done with the underlying iOS & Android libraries?
Thanks!

Can someone please tell me why this issue was closed?

@Sheharyar566 Because we did not need this feature anymore. You should probably open a new issue.


Hand Tracking support (Oculus Quest)

[ ] Add Hand Tracking support for Oculus Quest

Ok, current state is we have basic support, but I want to refactor it / enhance it quit a bit. (not pushing current state)

This got very complex, is priority number one at the moment.
Also, now pushing everything to dev branch despite messy code, will be fixed of course.


Update go dependencies

Issue #, if available: N/A
Description of changes:
Ran go get -u ./... && go mod tidy
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Will merge this and add these files to the triggers for integ tests in a followup pr.


dita.xsl.xslfo is broken

Expected Behavior
ImportXSLAction resolves the path properly when applying dita.xsl.xslfo
Actual Behavior
the path is not resolved properly, it either gets copied as is or gets prepended wiht a series of ../../../
Steps to Reproduce

try to use dita.xsl.xslfo in a plugin

Copy of the error message, log file or stack trace
 [pipeline] Error on line 19 column 157 of topic2fo_shell_fop.xsl:
 [pipeline]   XTSE0165: I/O error reported by XML parser processing
 [pipeline]   file:/plugin:my.pdf:xsl/fo/topic2fo_shell.xsl:
 [pipeline]   /plugin:my.pdf:xsl/fo/topic2fo_shell.xsl (No such file or directory)

DITA-OT version:
Operating system and version:
Linux, Windows
How did you run DITA-OT?
"dita --install"

though this is probably just a symptom of something else :(

never mind, I missed I found the issue /facepalm
if you're trying to pass a file to an extension point, you have to tell it that it's a file otherwise it freaks out :D
anyone running into this, please refer to: https://www.dita-ot.org/dev/topics/plugin-configfile.html
though it's totally weird that it behaves differently on Windows and Linux. ref: https://dita-ot.slack.com/archives/C02DW51E3/p1599586930007800


VS2019 build tools are installed, rather than VS2017 as documented

This is probably an upstream (Microsoft) issue; I was bitten by the failure with GenerateProjectFiles.bat with VS 2019, despite believing I was using VS2017.
The build failure is because VS2019 changed the relative path to current MSBuild.exe, and the UE4 build scripts were falling back to the 4.0 version is part of the older Framework SDK, but cannot compile UE4's C# code. It's already been fixed in UE4 upstream. If you also had VS2017 installed, it would find that version of MSBuild and work correctly.
From what I can see, the Docker script is correctly pulling the VS2017 Build Tools installer (https://aka.ms/vs/15/release/vs_buildtools.exe for VS2017 contrasting with https://aka.ms/vs/16/release/vs_buildtools.exe for VS2019). So this is probably an issue with the upstream installer either having been accidentally updated to VS2019, or mis-set expectations of what the installer does.
It's possible the upstream installer is installing the VS2017 toolchain, but applies the VS2019 MSBuild.exe path change. That would be awful. >_<

Damn, it's even worse than that. I'm looking at the filesystem from a fresh build of the ue4-build-prerequisites image, and not only has it installed MSBuild in the new VS2019 location, it doesn't appear to have installed the Build Tools themselves at all, either the VS2017 or VS2019 version. I'll continue investigating and post any findings here.

Progress update:

The Visual Studio web installer is designed to automatically update itself at startup (which can't be avoided, since the Microsoft servers will refuse to serve the relevant data files to an outdated version of the installer.) Although I already knew that this update process happened within a given release (i.e. an old VS2017 installer will update to a newer VS2017 installer), what I didn't realise was that this now happens across releases too. As confirmed by the installer logs, the VS2017 installer is now downloading the VS2019 installer and running that as a child process.

Evidently the update process breaks the installer's --wait flag, and causes the parent process to end before the child process has completed. As a result, the container build ends before the VS2019 installer has finished, terminating it and resulting in an incomplete installation. This was confirmed by the lack of installed Build Tools files, empty output from vswhere, truncated installer logs that ended shortly after reporting the update step, and errors during the post-install cleanup step that indicated that a number of files were still in use by a running process.

Unfortunately, downloading the VS2017 release channel manifest and instructing the VS2019 installer to use that via the --channelUri and --installChannelUri flags does not appear to install the VS2017 Build Tools. Inspecting the completed installation via both vswhere and the filesystem shows that the VS2019 Build Tools are installed.

The flag --add Microsoft.VisualStudio.Component.VC.v141.x86.x64 can be used to install the v141 compiler toolchain (the VS2017 version) within a VS2019 Build Tools installation, in much the same manner as the Dockerfiles currently install the v140 (VS2015) toolchain when building images for Unreal Engine 4.19. By the same token, the flag --remove Microsoft.VisualStudio.Component.VC.Tools.x86.x64 can be used to avoid installing the v142 (VS2019) toolchain.

I am currently testing a build using these flags to determine if they produce the desired behaviour. Given that the version number currently listed for MSBuild on the Visual Studio workload and component IDs page is 16.0.28517.75, I strongly suspect that MSBuild will be placed in the new VS2019 location even when installed alongside the v141 toolchain. If this turns out to be the case, I will investigate copying or symlinking it into the old VS2017 location to ensure compatibility with existing versions of UE4.

As expected, MSBuild is installed to the new location, so I've simply copied it into the VS2017 location. (The only files that get overwritten are two DLLs in the Bin\Roslyn subdirectory, and the versions that overwrite them look to be the same as the files they're replacing.) I've pushed the patch up in commit  2ff06d1. Once I've tested it to make sure it works with 4.19 through to 4.22, I'll update the PyPI package and push the new build(s) of the ue4-build-prerequisites image to Docker Hub, addressing issue #25.

For the record, UE's bug tracker for the moved MSBuild fix is https://issues.unrealengine.com/issue/UE-72173
Overall, VS2019 with the VS2017 compiler toolchain installed seems reasonable, that's the part UBT actually uses, the IDE version is incidental.
The visualstudio.15.chman definitely lists
{"id":"Microsoft.VisualStudio.Product.BuildTools","version":"15.9.28307.586","type":"ChannelProduct",

so it's annoying and disappointing that it seems to be ignoring it.
Thank you for your fast turnaround.

Yeah, at the moment the Windows images use the VS2015 toolchain for UE4.19 and the VS2017 toolchain for everything else. Engine version detection is performed at runtime during the build process, since users can specify an arbitrary git repo and branch to clone for the ue4-source image, which prevents ue4-docker from making version assumptions at the start of the build process.
At some point in the future it'll likely become necessary to maintain multiple Windows base images (VS2015 toolchain, VS2017 toolchain, VS2019 toolchain, etc.) and have the user specify which base image to use when performing a build. I've been trying to keep things as automatic as possible so far, but the scalability issues inherent in the current approach have become more and more evident with each new release of UE4.

Update: unfortunately the new version of MSBuild doesn't appear to work correctly with Unreal Engine 4.21, even when copied into the correct location:
Running AutomationTool...
C:\BuildTools\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(1661,5): error MSB4036: The "GetReferenceNearestTargetFrameworkTask" task was not found. Check the following: 1.) The name of the task in the project file is the same as the name of the task class. 2.) The task class is "public" and implements the Microsoft.Build.Framework.ITask interface. 3.) The task is correctly declared with <UsingTask> in the project file, or in the *.tasks files located in the "C:\BuildTools\MSBuild\15.0\Bin" directory. [C:\UnrealEngine\Engine\Source\Programs\AutomationTool\AutomationTool.csproj]
RunUAT.bat ERROR: AutomationTool failed to compile.
BUILD FAILED

I'm going to investigate options for controlling the versions of components that get installed, in the hopes that the VS2017 version of MSBuild can be installed somehow.

So it turns out I was on the right track using the VS2017 channel manifest after all, I just needed to add the flag --channelId VisualStudio.15.Release to force the installer to use that channel. Inspecting an image built after commit 7940d85 confirms that the VS2017 Build Tools are now being installed once more:
C:\Program Files (x86)\Microsoft Visual Studio\Installer>vswhere -all -products *
Visual Studio Locator version 2.6.7+91f4c1d09e [query version 2.0.2250.60958]
Copyright (C) Microsoft Corporation. All rights reserved.

instanceId: d27b35da
installDate: 6/4/2019 12:27:09 AM
installationName: VisualStudio/15.9.11+28307.586
installationPath: C:\BuildTools
installationVersion: 15.9.28307.586
productId: Microsoft.VisualStudio.Product.BuildTools
productPath: C:\BuildTools\Common7\Tools\LaunchDevCmd.bat
state:<PHONE_NUMBER>
isComplete: 1
isLaunchable: 1
isPrerelease: 0
isRebootRequired: 0
displayName: Visual Studio Build Tools 2017
description: The Visual Studio Build Tools allows you to build native and managed MSBuild-based applications without requiring the Visual Studio IDE. There are options to install the Visual C++ compi
lers and libraries, MFC, ATL, and C++/CLI support.
channelId: VisualStudio.15.Release
channelUri: C:\Users\ContainerAdministrator\AppData\Local\Temp\visualstudio.15.release.chman
enginePath: C:\Program Files (x86)\Microsoft Visual Studio\Installer\resources\app\ServiceHub\Services\Microsoft.VisualStudio.Setup.Service
installChannelUri: C:\Users\ContainerAdministrator\AppData\Local\Temp\visualstudio.15.release.chman
releaseNotes: https://go.microsoft.com/fwlink/?LinkId=660692#15.9.11
thirdPartyNotices: https://go.microsoft.com/fwlink/?LinkId=660708
updateDate: 2019-04-05T14:07:09.0247689Z
catalog_buildBranch: d15.9
catalog_buildVersion: 15.9.28307.586
catalog_id: VisualStudio/15.9.11+28307.586
catalog_localBuild: build-lab
catalog_manifestName: VisualStudio
catalog_manifestType: installer
catalog_productDisplayVersion: 15.9.11
catalog_productLine: Dev15
catalog_productLineVersion: 2017
catalog_productMilestone: RTW
catalog_productMilestoneIsPreRelease: False
catalog_productName: Visual Studio
catalog_productPatchVersion: 11
catalog_productPreReleaseMilestoneSuffix: 1.0
catalog_productRelease: RTW
catalog_productSemanticVersion: 15.9.11+28307.586
catalog_requiredEngineVersion: 1.18.1049.33485
properties_campaignId:
properties_channelManifestId: VisualStudio.15.Release/15.9.11+28307.586
properties_nickname:
properties_setupEngineFilePath: C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installershell.exe

Frustratingly however, the VS2019 version of MSBuild is still being installed (there is no Bin subdirectory under the 15.0 folder, only the Current folder):
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin>MSBuild /?
Microsoft (R) Build Engine version 16.0.461+g6ff56ef63c for .NET Framework

Success! After 9907c58:
C:\BuildTools\MSBuild\15.0\Bin>MSBuild /?
Microsoft (R) Build Engine version 15.9.21+g9802d43bc3 for .NET Framework

It turns out the version of MSBuild under C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools was a red herring, since this executable is actually already present in the latest version of the microsoft/dotnet-framework:4.7.2-sdk base image and was not being created by the Build Tools installer at all.

The presence of the 2019 Build Tools MSBuild executable might prove a problem for UE4 4.22.1, as the hotfix they applied (per UE-72173) will prefer that version.

@TBBle fortunately this doesn't appear to be the case, as the VS2019 MSBuild executable is not reported by the vswhere query that UE4 uses to locate the toolchain.
I've now tested the latest updates with 4.19.2, 4.20.3, 4.21.2, and 4.22.0, and everything appears to be working correctly. The fixes are now live in ue4-docker version 0.0.27.


Error on cleanup after a --force

Running cleanup, then cleanup --force, then cleanup again to make sure it worked, I keep getting this error, but only on the run just after cleanup --force - also seems like it's not actually cleaning up since those packages are still around after the --force.  I'd expect heroku-node to stay because heroku demands it but the others should be gone or have similar warnings?
$ brew bundle cleanup
Would uninstall formulae:
libgpg-error                      libksba                           zlib                              heroku/brew/heroku-node
Would untap:
homebrew/cask

$ brew bundle cleanup --force
Error: Refusing to uninstall /usr/local/Cellar/heroku-node/12.13.0
because it is required by heroku, which is currently installed.
You can override this and force removal with:
  brew uninstall --ignore-dependencies libgpg-error libksba zlib heroku/brew/heroku-node
Uninstalled 4 formulae
Untapping homebrew/cask...
Untapped 1 command and 3494 casks (3,598 files, 3.9MB).

$ brew bundle cleanup
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask'...
warning: templates not found in /Users/katedonaldson/.git-templates
remote: Enumerating objects: 3606, done.
remote: Counting objects: 100% (3606/3606), done.
remote: Compressing objects: 100% (3598/3598), done.
remote: Total 3606 (delta 25), reused 541 (delta 6), pack-reused 0
Receiving objects: 100% (3606/3606), 1.21 MiB | 14.97 MiB/s, done.
Resolving deltas: 100% (25/25), done.
Error: 767: unexpected token at '==> Tapping homebrew/cask
Tapped 1 command and 3494 casks (3,598 files, 3.9MB).
[{"token":"docker","name":["Docker Desktop","Docker Community Edition","Docker CE"],"homepage":"https://www.docker.com/community-edition","url":"https://download.docker.com/mac/stable/40693/Docker.dmg","appcast":"https://download.docker.com/mac/stable/appcast.xml","version":"<IP_ADDRESS>,40693","sha256":"8f7098e72a672d3523ac3791096e14315a49c53da985ed8ba5156b2349ec1bda","artifacts":[{"delete":["/Library/PrivilegedHelperTools/com.docker.vmnetd","/private/var/tmp/com.docker.vmnetd.socket","$(brew --prefix)/bin/docker","$(brew --prefix)/bin/docker-compose","$(brew --prefix)/bin/docker-credential-desktop","$(brew --prefix)/bin/docker-credential-osxkeychain","$(brew --prefix)/bin/docker-machine","$(brew --prefix)/bin/hyperkit","$(brew --prefix)/bin/kubectl","$(brew --prefix)/bin/kubectl.docker","$(brew --prefix)/bin/notary","$(brew --prefix)/bin/vpnkit"],"launchctl":["com.docker.helper","com.docker.vmnetd"],"quit":"com.docker.docker","signal":{}},["Docker.app"],{"trash":["$(brew --prefix)/bin/docker-compose.backup","$(brew --prefix)/bin/docker-machine.backup","$(brew --prefix)/bin/docker.backup","~/Library/Application Scripts/com.docker.helper","~/Library/Caches/KSCrashReports/Docker","~/Library/Caches/com.docker.docker","~/Library/Caches/com.plausiblelabs.crashreporter.data/com.docker.docker","~/Library/Containers/com.docker.docker","~/Library/Containers/com.docker.helper","~/Library/Group Containers/group.com.docker","~/Library/Preferences/com.docker.docker.plist"],"rmdir":["~/Library/Caches/KSCrashReports","~/Library/Caches/com.plausiblelabs.crashreporter.data"],"signal":{}}],"caveats":null,"depends_on":{},"conflicts_with":null,"container":null,"auto_updates":true},{"token":"google-backup-and-sync","name":["Google Backup and Sync"],"homepage":"https://www.google.com/drive/download/","url":"https://dl.google.com/drive/InstallBackupAndSync.dmg","appcast":null,"version":"latest","sha256":"no_check","artifacts":[{"quit":"com.google.GoogleDrive","signal":{}},["Backup and Sync.app"],{"trash":["~/Library/Application Scripts/com.google.GoogleDrive.FinderSyncAPIExtension","~/Library/Application Support/Google/Drive","~/Library/Caches/com.google.GoogleDrive","~/Library/Containers/com.google.GoogleDrive.FinderSyncAPIExtension","~/Library/Cookies/com.google.GoogleDrive.binarycookies","~/Library/Group Containers/google_drive","~/Library/Preferences/com.google.GoogleDrive.plist"],"signal":{}}],"caveats":"Although google-backup-and-sync may be installed alongside google-drive-file-stream, you should not use the same account with both.\n\n  https://support.google.com/a/answer/7496409#allowboth\n","depends_on":{},"conflicts_with":{"cask":["google-photos-backup-and-sync"]},"container":null,"auto_updates":null},{"token":"google-chrome","name":["Google Chrome"],"homepage":"https://www.google.com/chrome/","url":"https://dl.google.com/chrome/mac/stable/GGRO/googlechrome.dmg","appcast":"https://omahaproxy.appspot.com/history?os=mac;channel=stable","version":"79.0.3945.88","sha256":"f0031ceb313f1f28b9b6f387cb2df4547b6c12fc70153c6a12a367747c2d0223","artifacts":[{"launchctl":["com.google.keystone.agent","com.google.keystone.daemon"],"signal":{}},["Google Chrome.app"],{"trash":["/Library/Caches/com.google.SoftwareUpdate.*","/Library/Google/Google Chrome Brand.plist","/Library/Google/GoogleSoftwareUpdate","~/Library/Application Support/Google/Chrome","~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.google.chrome.app.*.sfl*","~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.google.chrome.sfl*","~/Library/Caches/Google/Chrome","~/Library/Caches/com.google.Chrome","~/Library/Caches/com.google.Chrome.helper.*","~/Library/Caches/com.google.Keystone","~/Library/Caches/com.google.Keystone.Agent","~/Library/Caches/com.google.SoftwareUpdate","~/Library/Google/Google Chrome Brand.plist","~/Library/Google/GoogleSoftwareUpdate","~/Library/LaunchAgents/com.google.keystone.agent.plist","~/Library/Logs/GoogleSoftwareUpdateAgent.log","~/Library/Preferences/com.google.Chrome.plist","~/Library/Preferences/com.google.Keystone.Agent.plist","~/Library/Saved Application State/com.google.Chrome.app.*.savedState","~/Library/Saved Application State/com.google.Chrome.savedState","~/Library/WebKit/com.google.Chrome"],"rmdir":["/Library/Google","~/Library/Application Support/Google","~/Library/Caches/Google","~/Library/Google"],"signal":{}}],"caveats":null,"depends_on":{"macos":{">=":["10.10"]}},"conflicts_with":{"cask":["google-chrome-beta","google-chrome-dev"]},"container":null,"auto_updates":true},{"token":"iterm2","name":["iTerm2"],"homepage":"https://www.iterm2.com/","url":"https://iterm2.com/downloads/stable/iTerm2-3_3_7.zip","appcast":"https://iterm2.com/appcasts/final.xml","version":"3.3.7","sha256":"a993e66119b8ddf97e6b50a03a8aa6fbf3954f66a3f896a7503876c06702e724","artifacts":[["iTerm.app"],{"trash":["~/Library/Application Support/iTerm","~/Library/Application Support/iTerm2","~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.googlecode.iterm2.sfl*","~/Library/Caches/com.googlecode.iterm2","~/Library/Preferences/com.googlecode.iterm2.plist","~/Library/Saved Application State/com.googlecode.iterm2.savedState"],"signal":{}}],"caveats":null,"depends_on":{"macos":{">=":["10.12"]}},"conflicts_with":null,"container":null,"auto_updates":true},{"token":"java","name":["OpenJDK Java Development Kit"],"homepage":"https://openjdk.java.net/","url":"https://download.java.net/java/GA/jdk13.0.1/cec27d702aa74d5a8630c65ae61e4305/9/GPL/openjdk-13.0.1_osx-x64_bin.tar.gz","appcast":null,"version":"13.0.1,9:cec27d702aa74d5a8630c65ae61e4305","sha256":"593c5c9dc0978db21b06d6219dc8584b76a59c79d57e6ec1b28ad0d848a7713f","artifacts":[["jdk-13.0.1.jdk",{"target":"/Library/Java/JavaVirtualMachines/openjdk-13.0.1.jdk"}]],"caveats":null,"depends_on":{},"conflicts_with":null,"container":null,"auto_updates":null},{"token":"mysides","name":["mysides"],"homepage":"https://github.com/mosen/mysides","url":"https://github.com/mosen/mysides/releases/download/v1.0.1/mysides-1.0.1.pkg","appcast":"https://github.com/mosen/mysides/releases.atom","version":"1.0.1","sha256":"76946b8f7c5bf714125d75f1ada8140e034f05a9e288c73a7af445d76c2a5514","artifacts":[{"pkgutil":"com.github.mosen.mysides","signal":{}},"mysides-1.0.1.pkg (Pkg)"],"caveats":null,"depends_on":{},"conflicts_with":null,"container":null,"auto_updates":null},{"token":"postico","name":["Postico"],"homepage":"https://eggerapps.at/postico/","url":"https://eggerapps-downloads.s3.amazonaws.com/postico-1.5.10.zip","appcast":"https://eggerapps.at/postico/docs/?file=changelist.html","version":"1.5.10","sha256":"f0318ddfafa1d52438cc38e160507e8140f595576f829cd4c0b7ff1c4d1f6e1d","artifacts":[["Postico.app"],{"trash":["~/Library/Application Scripts/at.eggerapps.Postico","~/Library/Containers/at.eggerapps.Postico","~/Library/Preferences/at.eggerapps.Postico.plist","~/Library/Saved Application State/at.eggerapps.Postico.savedState"],"signal":{}}],"caveats":null,"depends_on":{},"conflicts_with":null,"container":null,"auto_updates":null},{"token":"slack","name":["Slack"],"homepage":"https://slack.com/","url":"https://downloads.slack-edge.com/mac_releases/Slack-4.2.0-macOS.zip","appcast":"https://macupdater.net/cgi-bin/check_urls/check_url_redirect.cgi?url=https://slack.com/ssb/download-osx","version":"4.2.0","sha256":"f531fac754cecb12c082e52e3fd5957eeeb0e804e8c44e43d81635aefe409016","artifacts":[{"quit":"com.tinyspeck.slackmacgap","signal":{}},["Slack.app"],{"trash":["~/Library/Application Scripts/com.tinyspeck.slackmacgap","~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.tinyspeck.slackmacgap.sfl*","~/Library/Application Support/Slack","~/Library/Caches/com.tinyspeck.slackmacgap","~/Library/Caches/com.tinyspeck.slackmacgap.ShipIt","~/Library/Containers/com.tinyspeck.slackmacgap","~/Library/Containers/com.tinyspeck.slackmacgap.SlackCallsService","~/Library/Cookies/com.tinyspeck.slackmacgap.binarycookies","~/Library/Group Containers/*.com.tinyspeck.slackmacgap","~/Library/Group Containers/*.slack","~/Library/Preferences/com.tinyspeck.slackmacgap.helper.plist","~/Library/Preferences/com.tinyspeck.slackmacgap.plist","~/Library/Saved Application State/com.tinyspeck.slackmacgap.savedState"],"signal":{}}],"caveats":null,"depends_on":{},"conflicts_with":null,"container":null,"auto_updates":true},{"token":"visual-studio-code","name":["Microsoft Visual Studio Code","VS Code"],"homepage":"https://code.visualstudio.com/","url":"https://update.code.visualstudio.com/1.41.1/darwin/stable","appcast":"https://vscode-update.azurewebsites.net/api/update/darwin/stable/VERSION","version":"1.41.1","sha256":"efb558da07adcdf7b5e69cabda2a8b9359855e9e6f7ae62b4e1db7eb975c98e4","artifacts":[["Visual Studio Code.app"],["/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code"],{"trash":["~/Library/Application Support/Code","~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.microsoft.vscode.sfl*","~/Library/Caches/com.microsoft.VSCode","~/Library/Caches/com.microsoft.VSCode.ShipIt","~/Library/Preferences/com.microsoft.VSCode.helper.plist","~/Library/Preferences/com.microsoft.VSCode.plist","~/Library/Saved Application State/com.microsoft.VSCode.savedState","~/.vscode"],"signal":{}}],"caveats":null,"depends_on":{},"conflicts_with":null,"container":null,"auto_updates":true}]
'
Please report this bug:
  https://github.com/Homebrew/homebrew-bundle/issues
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/json/common.rb:156:in `parse'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/json/common.rb:156:in `parse'
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-bundle/lib/bundle/cask_dumper.rb:30:in `formula_dependencies'
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-bundle/lib/bundle/commands/cleanup.rb:78:in `formulae_to_uninstall'
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-bundle/lib/bundle/commands/cleanup.rb:22:in `run'
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-bundle/cmd/brew-bundle.rb:83:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/utils.rb:82:in `require?'
/usr/local/Homebrew/Library/Homebrew/brew.rb:108:in `<main>'

$ brew bundle cleanup
Would uninstall formulae:
libgpg-error                      libksba                           zlib                              heroku/brew/heroku-node
Would untap:
homebrew/cask

$ brew bundle cleanup
Would uninstall formulae:
libgpg-error                      libksba                           zlib                              heroku/brew/heroku-node
Would untap:
homebrew/cask

What's in your Brewfile?


fix: Failed to create just in time channel due to disconnected peer

The change also includes error handling in case the payment could not be forwarded and sets the HTLC to failed.
resolves #330

bors r+


Barback asset conflict message could be more clear

 Issue by blois
Originally opened as dart-lang/sdk#16374

I've gotten stumped by failing builds twice because I had existing dart2js output in the folder when running pub build, would be great if the error message would be more clear.
Currently it just says "Transforms $transforms all emitted asset $id."
Would be great if this could say something about it conflicting with an existing file.
I'd be happy to update the string if someone has a decent suggestion for what it should be.

 Comment by nex3

Added Duplicate label.
Marked as being merged into dart-lang/barback#18.

 Comment by anders-sandholm

Removed Library-Barback label.
Added Pkg-Barback label.


Rust rewrite?

I'd much rather do it in Kotlin, but feel free to do it.
Here's the zeromq crate info: https://zeromq.org/languages/rust/
That's a cups wrapper: https://github.com/LegNeato/cups-sys
There's a QR-Generator: https://crates.io/crates/qrcode
And an image processing library, but opencv would work as well: https://crates.io/crates/image

This was supposed to be a joke :D

I thought so, but but it was too tempting not play along:D Tho it would not be too hard to do so.
But it would likely be a waste of time. nearly 90% of it is waiting either for a new command, or for a print to finish.


No indication of the nodes required in the external file

The page references an external file and shows how to link to it in the original file, but does not give an example of the root node and format required in the external file.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 9cf05d1a-3c6d-dc86-77ef-bd453f957a6c
Version Independent ID: e40bd8c3-89af-b89b-2d3d-6f015d8704b7
Content: <linkedConfiguration> element - .NET Framework
Content Source: docs/framework/configure-apps/file-schema/linkedconfiguration-element.md
Product: dotnet-framework
GitHub Login: @gewarren
Microsoft Alias: dotnetcontent

@sixeyes I believe it's the same format as an app.config file. However, note that only the <assemblyBinding> information will be read from the linked file:
"The settings in included configuration files only affect loader binding policy and are used only by the loader. The included configuration files can have settings other than binding policies, but those settings don't have any effect."


PANIC: called Option::unwrap() on a None value in query-engine/core/src/response_ir/internal.rs:364:77

Hi Prisma Team! My Prisma Client just crashed. This is the report:
Versions

Name
Version

Node
v17.4.0

OS
debian-openssl-1.1.x

Prisma Client
3.10.0

Query Engine
0.1.0

Database
sqlite

Logs
  prisma:tryLoadEnv Environment variables not found at null
  prisma:tryLoadEnv Environment variables not found at undefined
  prisma:tryLoadEnv No Environment variables loaded
  prisma:tryLoadEnv Environment variables not found at null
  prisma:tryLoadEnv Environment variables not found at undefined
  prisma:tryLoadEnv No Environment variables loaded
  prisma:client clientVersion: 3.10.0
  prisma:client clientEngineType: library
  prisma:client:libraryEngine internalSetup
  prisma:client:libraryEngine Searching for Query Engine Library in /home/arvind/cms/node_modules/.prisma/client
  prisma:client:libraryEngine loadEngine using /home/arvind/cms/node_modules/.prisma/client/libquery_engine-debian-openssl-1.1.x.so.node
  prisma:client:libraryEngine library starting
  prisma:client:libraryEngine library started
  prisma:client:libraryEngine sending request, this.libraryStarted: true  +9s
  prisma:client:libraryEngine sending request, this.libraryStarted: true  +6s
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true
  prisma:client:libraryEngine sending request, this.libraryStarted: true

Client Snippet
// PLEASE FILL YOUR CODE SNIPPET HERE

Schema
// This file is automatically generated by Keystone, do not modify it manually.
// Modify your Keystone config when you want to change this.

datasource sqlite {
  url      = env("DATABASE_URL")
  provider = "sqlite"
}

generator client {
  provider = "prisma-client-js"
  output   = "node_modules/.prisma/client"
}

model User {
  id       String @id @default(cuid())
  name     String @default("")
  email    String @unique @default("")
  password String
  posts    Post[] @relation("Post_author")
}

model NavBarItem {
  id           String       @id @default(cuid())
  label        String       @default("")
  link         String       @default("")
  children     NavBarItem[] @relation("NavBarItem_children")
  parent       NavBarItem[] @relation("NavBarItem_children")
  page         Page[]       @relation("NavBarItem_page")
  CallToAction String?      @default("menu")
}

model Page {
  id                String           @id @default(cuid())
  title             String           @default("")
  slug              String           @default("")
  description       String           @default("")
  sections          Section[]        @relation("Page_sections")
  FooterPartPosts   FooterPartPost[] @relation("FooterPartPost_page")
  NavBarItems       NavBarItem[]     @relation("NavBarItem_page")
  Posts             Post[]           @relation("Page_Posts")
  tags              Tag?             @relation("Page_tags", fields: [tagsId], references: [id])
  tagsId            String?          @map("tags")
  from_Article_Page Article[]        @relation("Article_Page")

  @@index([slug])
  @@index([tagsId])
}

model Section {
  id               String    @id @default(cuid())
  title            String    @default("")
  description      String    @default("")
  Content          String    @default("[{\"type\":\"paragraph\",\"children\":[{\"text\":\"\"}]}]")
  ButtonName       String    @default("")
  Url              String    @default("")
  FeactureImageUrl String    @default("")
  OtherImageUrl    String    @default("")
  order            Int?

Hello @janpio Sir,
Actually I have create CMS in keystonejs, When i am running dashboard, it is showing:  please see:
An error occurred handling a request for the Admin UI: Error: Prisma error
at Object.prismaError (/home/arvind/cms/node_modules/@keystone-6/core/dist/graphql-errors-4f230fdf.cjs.dev.js:27:12)
at runWithPrisma (/home/arvind/cms/node_modules/@keystone-6/core/dist/types-for-lists-c9e0cd07.cjs.dev.js:30:25)
at Object.count (/home/arvind/cms/node_modules/@keystone-6/core/dist/types-for-lists-c9e0cd07.cjs.dev.js:911:17)
at async Promise.all (index 0)
at Object.runGraphQL [as run] (/home/arvind/cms/node_modules/@keystone-6/core/dist/initConfig-2fd803cd.cjs.dev.js:2817:20)
at Object.count (/home/arvind/cms/node_modules/@keystone-6/core/dist/initConfig-2fd803cd.cjs.dev.js:2496:24)
at pageMiddleware (/home/arvind/cms/node_modules/@keystone-6/auth/dist/keystone-6-auth.cjs.dev.js:894:21)
at Object.pageMiddleware (/home/arvind/cms/node_modules/@keystone-6/auth/dist/keystone-6-auth.cjs.dev.js:1105:43)
at /home/arvind/cms/node_modules/@keystone-6/core/dist/createAdminUIMiddleware-abacc995.cjs.dev.js:178:30 {
locations: [ { line: 1, column: 35 } ],
path: [ 'count' ],
extensions: {
debug: {
message: '\n' +
'Invalid prisma.user.count() invocation:\n' +
'\n' +
'\n' +
'  PANIC: called Option::unwrap() on a None value in query-engine/core/src/response_ir/internal.rs:364:77\n' +
'\n' +
'This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.\n' +
'\n' +
'https://github.com/prisma/prisma/issues/new?body=Hi+Prisma+Team!+My+Prisma+Client+just+crashed.+This+is+the+report%3A
%23%23+Versions

|+Name++++++++++++|+Version++++++++++++|
|-----------------|--------------------|
|+Node++++++++++++|+v17.4.0++++++++++++|+
|+OS++++++++++++++|+debian-openssl-1.1.x|
|+Prisma+Client+++|+3.10.0+++++++++++++|
|+Query+Engine++++|+0.1.0++++++++++++++|
|+Database++++++++|+sqlite+++++++++++++|

%23%23+Logs
```
++prisma%3AtryLoadEnv+Environment+variables+not+found+at+null++
++prisma%3AtryLoadEnv+Environment+variables+not+found+at+undefined++
++prisma%3AtryLoadEnv+No+Environment+variables+loaded++
++prisma%3AtryLoadEnv+Environment+variables+not+found+at+null++
++prisma%3AtryLoadEnv+Environment+variables+not+found+at+undefined++
++prisma%3AtryLoadEnv+No+Environment+variables+loaded++
++prisma%3Aclient+clientVersion%3A+3.10.0++
++prisma%3Aclient+clientEngineType%3A+library++
++prisma%3Aclient%3AlibraryEngine+internalSetup++
++prisma%3Aclient%3AlibraryEngine+Searching+for+Query+Engine+Library+in+%2Fhome%2Farvind%2Fcms%2Fnode_modules%2F.prisma%2Fclient++
++prisma%3Aclient%3AlibraryEngine+loadEngine+using+%2Fhome%2Farvind%2Fcms%2Fnode_modules%2F.prisma%2Fclient%2Flibquery_engine-debian-openssl-1.1.x.so.node++
++prisma%3Aclient%3AlibraryEngine+library+starting++
++prisma%3Aclient%3AlibraryEngine+library+started++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B5s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B3s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B2s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B22s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B2s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B2s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++%2B12s
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
++prisma%3Aclient%3AlibraryEngine+sending+request%2C+this.libraryStarted%3A+true++
```

%23%23+Client+Snippet
```ts
%2F%2F+PLEASE+FILL+YOUR+CODE+SNIPPET+HERE
```

%23%23+Schema
```prisma
%2F%2F+PLEASE+ADD+YOUR+SCHEMA+HERE+IF+POSSIBLE
```

%23%23+Prisma+Engine+Query
```
{"X"%3A{}}
```
&title=PANIC%3A+called+`Option%3A%3Aunwrap()`+on+a+`None`+value+in+query-engine%2Fcore%2Fsrc%2Fresponse_ir%2Finternal.rs%3A364%3A77&template=bug_report.md\n' +
'\n' +
'If you want the Prisma team to look into it, please open the link above 🙏\n' +
'To increase the chance of success, please post your schema and a snippet of\n' +
'how you used Prisma Client in the issue. \n'
},
code: 'KS_PRISMA_ERROR'
}
}
Sir pls help me

Can you enable Prisma logging? That should log the query that is executed there exactly: https://pris.ly/d/logging + https://pris.ly/d/debugging

I am experiencing the same issue with 3.10.0, it is not present with 3.9.2 and with 3.11.0-dev.52

Same questions to you then @ivosabev - see above. We need more information to be able to do anything.

I have the same problem with 3.10.0, not present with 3.9.2
It's just a matter of querying one of our models - only this one, and it does not matter if I use findUnique or findFirst.  I tried commenting out all relations and then it worked, but when I tried to uncomment them one by one the behaviour wasn't consistent, I couldn't figure out if any specific relation was the problem.
Versions:

Name
Version

Node
v16.14.0

OS
darwin

Prisma Client
3.10.0

Query Engine
0.1.0

Database
mysql

Rust backtrace:
   0:        0x11d3bdc31 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h90c059b532d34bdd
   1:        0x11d3dffcb - core::fmt::write::hefb311138778afb5
   2:        0x11d3b6d5a - std::io::Write::write_fmt::h0e0ed87b18ae870c
   3:        0x11d3bf9b5 - std::panicking::default_hook::{{closure}}::h43f0e51e5825bd5b
   4:        0x11d3bf59f - std::panicking::default_hook::ha1c3473cbcf391f7
   5:        0x11c08c08d - query_engine::engine::set_panic_hook::{{closure}}::h5fd99e2c8cab0680
   6:        0x11d3c00c6 - std::panicking::rust_panic_with_hook::h0db3c4ba5fb4dc12
   7:        0x11d3bfb14 - std::panicking::begin_panic_handler::{{closure}}::hc6d8c92efeef5a19
   8:        0x11d3be0a7 - std::sys_common::backtrace::__rust_end_short_backtrace::h7ffb59e898e02a83
   9:        0x11d3bfaaa - _rust_begin_unwind
  10:        0x11d44862f - core::panicking::panic_fmt::h7d73f67464d78916
  11:        0x11d448587 - core::panicking::panic::h9721d5a8c5d33ad7
  12:        0x11c2a25d2 - query_core::response_ir::internal::serialize_objects::hcbb3e5a35bf1b163
  13:        0x11c29e881 - query_core::response_ir::internal::serialize_record_selection::hff326efe6929dc94
  14:        0x11c29b07b - query_core::response_ir::internal::serialize_internal::hd01e18f6cb103f47
  15:        0x11c330798 - query_core::response_ir::ir_serializer::IrSerializer::serialize::h63503a890a90ba56
  16:        0x11c04358d - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hfc544138efa64996
  17:        0x11c0340c5 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hbef1d4078c8baba1
  18:        0x11c01a9e5 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h3320e01c5126c526
  19:        0x11c03c463 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he6a8715be671cc2c
  20:        0x11c016c81 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h1f64163b60fc3f60
  21:        0x11c020d7f - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h5493be79bdc17417
  22:        0x11c0266b0 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h7a0ac9d980b9ee53
  23:        0x11c03e6ba - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hedcd61ded9f36990
  24:        0x11c027f7a - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h7f76a67c327697d5
  25:        0x11bfcb333 - tokio::runtime::task::harness::Harness<T,S>::poll::hfe010040e1d19aaa
  26:        0x11d14de4d - std::thread::local::LocalKey<T>::with::h21a4b8b337b67c27
  27:        0x11d148e8f - tokio::runtime::thread_pool::worker::Context::run_task::h59a149308d0766fd
  28:        0x11d148499 - tokio::runtime::thread_pool::worker::Context::run::hc868bc7b234ff7f5
  29:        0x11d135825 - tokio::macros::scoped_tls::ScopedKey<T>::set::h6e800fa1ea63250e
  30:        0x11d147964 - tokio::runtime::thread_pool::worker::run::h5e69089734701f92
  31:        0x11d14bca9 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h4ec0494860eecc5d
  32:        0x11d13fc13 - tokio::runtime::task::harness::Harness<T,S>::poll::h5bb4edf7863a87fc
  33:        0x11d12bac7 - tokio::runtime::blocking::pool::Inner::run::h6a23c4abad6b4fef
  34:        0x11d133cfe - std::sys_common::backtrace::__rust_begin_short_backtrace::ha216731a1b94a7c1
  35:        0x11d14e52c - core::ops::function::FnOnce::call_once{{vtable.shim}}::h3a881747160f83a4
  36:        0x11d3c5a77 - std::sys::unix::thread::Thread::new::thread_start::h13359fe05abc98e9
  37:     0x7ff814e404f4 - __pthread_start

Thanks for the report @erikbjoern. Can you share the schema of the model(s)? Does it depend on the data in the database table or does it also happen on an empty, newly created database of that schema?
Also: https://github.com/prisma/prisma/issues/12126#issuecomment-1057971305 If you can report the actual query (both Prisma and SQL) that triggers this would be most helpful.

@Jolg42 We are no longer experiencing the issue. I'm not sure which version resolved it, but at least since 3.12.0 it's been working. We did not perform any count() in the failing query - as I mentioned above it was only a single model that behaved this way, even if we didn't pass any params at all, just a simple prisma.model.findFirst().
I could post the schema but it is rather gigantic with many interconnected relations

@arvind-plaxonic @ivosabev Are you still experiencing this problem? If so, can you please provide some additional information that will help us reproduce this so we can actually fix it? Thanks. Otherwise we will have to close this issue.

This problem looks like a duplicate of https://github.com/prisma/prisma/issues/12105 which has a comment that it was fixed in 3.11.0 https://github.com/prisma/prisma-engines/pull/2732
@arvind-plaxonic @ivosabev Could you try 3.11.0 or a more recent version and confirm it was fixed?

Closing then until someone provides more information. Please comment or open a new issue. Thanks!


Migrating to ESBuild breaks SASS imports with "~" (tilde)

Command
build, test
Is this a regression?
Unknown
Description
When migrating to the new build system, "~" (tilde) is no longer supported, which is fine, I guess.
But when you remove the "~" the test build fails: https://github.com/fontsource/fontsource/issues/798#issuecomment-2166824082
Minimal Reproduction
s. OP in linked issue for reproduction
[EDIT] or simply set up an Angular 17 project and migrate it to 18 with the new build option selected. Then install npm i @fontsource/roboto and include the SASS mixin in your project.
Exception or Error
./src/styles.scss - Error: Module Error (from ./node_modules/postcss-loader/dist/cjs.js):
/home/pfeileon/git/my-app/node_modules/@fontsource/roboto/scss/mixins.scss:189:6: Can't resolve '../node_modules/<EMAIL_ADDRESS>in '/home/pfeileon/git/my-app/src'

Your Environment
Angular CLI: 18.0.4
Node: 20.12.2
Package Manager: npm 10.5.0
OS: linux x64

Angular: 18.0.3
... animations, cdk, common, compiler, compiler-cli, core, forms
... material, platform-browser, platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1800.4
@angular-devkit/build-angular   18.0.4
@angular-devkit/core            18.0.4
@angular-devkit/schematics      18.0.4
@angular/cli                    18.0.4
@schematics/angular             18.0.4
rxjs                            7.8.1
typescript                      5.4.5
zone.js                         0.14.6

Duplicate of https://github.com/angular/angular-cli/issues/26718


[web]: style is different when i use

i make a map with leaflet, is not same when i run it at chrome with Toggle device toolbar or not. There will be a shdow-root in front
`E:\app\map_info_collect>flutter doctor -v
[√] Flutter (Channel stable, 2.0.1, on Microsoft Windows [Version 10.0.18363.1379], locale zh-CN)
• Flutter version 2.0.1 at E:\app\flutter
• Framework revision c5a4b4029c (7 days ago), 2021-03-04 09:47:48 -0800
• Engine revision 40441def69
• Dart version 2.12.0
• Pub download mirror https://pub.flutter-io.cn
• Flutter download mirror https://storage.flutter-io.cn
[√] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
• Android SDK at C:\Users\tx\AppData\Local\Android\sdk
• Platform android-30, build-tools 30.0.2
• Java binary at: E:\wzc的文件夹\日常工具\androidStudio\jre\bin\java
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
• All Android licenses accepted.
[√] Chrome - develop for the web
• Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe
[√] Android Studio (version 4.1.0)
• Android Studio at E:\wzc的文件夹\日常工具\androidStudio
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
[√] IntelliJ IDEA Ultimate Edition (version 2020.3)
• IntelliJ at E:\app\IntelliJ IDEA 2020.3
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
[√] Connected device (1 available)
• Chrome (web) • chrome • web-javascript • Google Chrome 89.0.4389.82
• No issues found!
`

i make a map with leaflet,   is not same when i run it at chrome with Toggle device toolbar or not

i make a map with leaflet

sounds like a 3rd party plugin, maybe flutter_map or  syncfusion_flutter_maps
if that's the ase please open an issue in the dedicated github
or provide your flutter doctor -v, your flutter run -v
and a minimal reproducible code sample without 3rd party plugins

Hi @absc
Can you please provide what was asked above?
Thank you

run with html

run with cancaskit

Hi @absc
Can you please provide what was asked in https://github.com/flutter/flutter/issues/77878#issuecomment-796487249?
Thank you

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.


Add asStateFlow to the StateFlow variables in ViewModel class

Original RFC
Why
Currently we can emit values to our StateFlow variables by casting them to MutableStateFlow. We should update our StateFlow declarations to make them read-only
Solution

Update our StateFlow declarations as follows:private val _uiModels = MutableStateFlow<List<UiModel>>(emptyList())
val uiModels = _uiModels.asStateFlow()

Apply updates to template-compose and sample-compose ViewModels

Who Benefits?
Developers

@kaungkhantsoe please update your ticket status. Currently, it's in Icebox.


Use CodeMetric config 1.1.0

The most significant change in 1.1.0 is that @author tags are now flagged.

Refer to this link for build results (access rights to CI server needed):
http://jenkins.terasology.org/job/TerasologyPRs/340/
Hooray Jenkins reported success with all tests good!

Looks like it works as expected:
http://jenkins.terasology.org/job/TerasologyPRs/340/checkstyleResult/new/

Yep! Also get author tags highlighted in IntelliJ now as Checkstyle violations :-)


update mfem and conduit, build mfem with conduit support

Description
Resolves #3141
Updates build_visit to build mfem 3.4 and conduit 0.4.0.
Changes mfem build to enable conduit support.
Updates MFEM plugin to have conduit includes, etc.
Type of change

[ ] Bug fix
[x] New feature
[ ] New Documentation

How Has This Been Tested?
I have tested on my mac, we may need to rebaseline on surface.

@biagas windows will need to be updated as well

@cyrush Gotcha. Thanks for the heads up.


JDK9+ Jigsaw modules support

Hi,
I'm trying to add the maven dependency graphhopper-reader-osm as suggested in the docs to a JDK 11 Java project using the new Jigsaw module system (i.e. module-info.java exists) ... [, because the Jigsaw module system is needed by other dependencies].
Up to graphhopper version 0.10.3 it works peachy, but since version 0.11.0 up to the current 1.0 the project does not compile anymore with the error:
reads package com.graphhopper from both graphhopper.api and graphhopper.core
This is due to the fact, that since 0.11.0 some classes from graphhopper-core were split off into graphhopper-api, but the new Jigsaw module system needs the different modules/dependency-jars to be disjoint package-wise. So for example: com.graphhopper.util has some classes in maven dependency graphhopper-core and some in graphhopper-api, but this is not supported anymore when using the Jigsaw module system since JDK9+ because it needs to have com.graphhopper.util only be exported by one module/dependency.

Is there a plan to support the Jigsaw module system in the future with embedded module-info.java files?
Is it possible to make a third (optional) maven dependency/module "graphhopper-core-plus-api" which combines graphhopper-core and graphhopper-api back into one dependency again? (Which - I think - would solve the problem above)
Or am I just being dumb and doing something wrong... and the support for the Jigsaw module system is working already all along?

I might maybe be able to suggest a pull request with the second option "combined graphhopper-core-plus-api". But as an only intermediate maven user I would highly appreciate any push in the right direction, or maybe a "stop" if this is the wrong direction.
Thanks for the great work with Graphhopper in any case :-)
Greetings from Germany
Oli4

If jigsaw standard does not allow same package names in different modules I think we should not do it. Especially (core) com.graphhopper.util is quite a mess anyway (it has many unrelated classes) and should be easy to rename. com.graphhopper only has a few classes so they could be moved to some sub-package? Are these the only two packages that cause a problem?

Hi easbar,
thanks for taking this seriously :-).
As far as I see it, It's the packages:
com.graphhopper
com.graphhopper.col
com.graphhopper.routing.util
com.graphhopper.util
com.graphhopper is especially tricky, because e.g. Graphhopper and GraphhopperAPI are understandebly in core and api respecitvely but should be separated into different subpackages to adhere to the Jigsaw standard. My understanding is, that Jigsaw would like to have
com.graphhopper.core
com.graphhopper.api
com.graphhopper.web
etc.
As for the asked for reference, the stackflow answer https://stackoverflow.com/questions/51828014/how-split-packages-are-avoided-in-java-9 better explains the official jdk9 spec description http://openjdk.java.net/projects/jigsaw/spec/sotms/#readability.
If you would like to move the classes, my suggestion would be to move the split packages
com.graphhopper.col
com.graphhopper.routing.util
com.graphhopper.util
from the core module --> into the api module, so that the packages are only in one module again. Since the core module already depends on the api module, nothing should break...
Then it would suffice to put the classes under com.graphhopper in the module api into a subpackage com.graphhopper.api or com.graphhopper.xyz as you suggested and everything should be sorted and separated again for Jigsaw.
Would you be more ok with that solution?

What I'm just realising:
It could also be possible to just move the Graphhopper.java from core into api, then the com.graphhopper package would not be split anymore (so no need for com.graphhopper.api then)

OK, I tried the "moving" idea... unfortenately due to massive cross-references in the classes/packages, I ended up having to move everything from core module into api module... letting the core module to have no classes anymore.
I doubt that this is a solution you would like to have, is it ;-)?
I was very careful, not to move anything unnecessary, but unfortenately with every move of a class there were new cross-references that resulted in moving more and more classes.
If you would like to check the commit, in case this is an acceptable solution: https://github.com/oli0044/graphhopper/tree/issue_2053b
... but I really doubt it. I'm even too ashamed to transform it into a pull request ;-).
With the idea of introducing a package combining core and api called coreapi or core-plus-api, the separation of core and api would not be lost and for the time necessary to separate the modules properly, the JDK9+ project developers would have a chance to use the versions higher than 0.10.3.

I thought maybe there is no need to move any code between the modules, but we can just use different package names as for example com.graphhopper.util in core is not necessarily related to the package with the same name in api. Using a prefix like .api or .core could be an option. But lets see what others have to say.

I thought maybe there is no need to move any code between the modules, but we can just use different package names as for example com.graphhopper.util in core is not necessarily related to the package with the same name in api (and package privacy isn't used very often in GH either).

yes, that would be the "good" solution intended by the jigsaw standard, to introduce com.graphhopper.api.util and com.graphhopper.core.util.

Using a prefix like .api or .core for the package names could be an option. But lets see what others have to say.

Ok

FYI, I tried your suggestion and moved the classes in the api module into a subpackage com.graphhopper.api. Seems to work, so if this will be accepted by the others, I could provide a pull request.

I fully support the package-renaming solution.

Just to be completely transparent:
After I moved the content of the api module into a package com.graphhopper.api and fixed all the imports (using Netbeans 11.3) there was one Test that failed in the core module:
com.graphhopper.util.DistanceCalcEuclideanTest
This was due to the function calcShrinkFactor() in com.graphhopper.api.util.DistanceCalcEarth which had no private/protected/public declaration and therefore was only visible to same-package classes in com.graphhopper.api.util. However com.graphhopper.util.DistanceCalcEuclidean now was a child class, but not in the same package anymore which led to the wrong function to be called during runtime. It could be fixed easily by just declaring both funtions calcShrinkFactor() as protected, so that visibility was also ensured to the child classes.
However that made me think, that there might be some more occurences of methods that are only visible in the same package and therefore not called properly anymore. It turned out, that there were only 3 other occurences of such functions that had no public/protected/public declaration and all 3 functions were only used in the same package.
So I think no worries in that context.
However I just wanted to be careful, complete and transparent... maybe this sparks an idea for anyone, which should be considered...
(I'll attach my documentation for the search of all the occurences of non-public/private/procted-methods)
graphhopper.functions.txt

For which branch should I propose the pull request?
1.0 or master?

(And I also think this not only solves the immediate problem, but is the totally correct thing to do. Thanks for taking this on!)

np, that's one of the beauties of open source. Thx for the great tool and work :-).
ok, I proposed the pull request. At the moment I purposefully only added package com.graphhopper.api, not to change tooo much and also not knowing what future structure you would prefer.
Would you like me to introduce another subpackage already or is it ok this way for the beginning?

Can we do all the changes needed prior to actually renaming the package first / in a separate PR? The goal should be that we can update downstream dependencies or other branches by a single 'rename package' refactoring using the IDE. Otherwise it will be a pain to update code that is not in master yet or just uses the code in this repository.

Can we do all the changes needed prior to actually renaming the package first / in a separate PR? The goal should be that we can update downstream dependencies or other branches by straight forward 'rename package' or 'move files' refactorings using the IDE. Otherwise it will be a pain to update code that is not in master yet or just uses the code in this repository.

I think, I understand what you mean and provided a PR with the changes before moving into the submodule.
Refactoring with the IDE (I used Netbeans) already took care of 90% of the job when moving the files into the subpackage. The remaining 10% was for classes that used wildcard-imports like com.graphhopper.util.*. For those Netbeans (at least) does not know what to do when refactoring and I had to fix the imports "manually" (by opening the classes and pressing Ctrl+Shift+I).
And in one class there was the issue, that protected fields were used, instead of pre-existing public getter-methods. What's your take on this? Should the protected fields made public in the new package com.graphhopper.api, so that no depending code is broken? Or should the developers be "pushed" to use the already existing public getters and setters?
All in all I think what you would like to achieve is already done in the first PR. 90% is done automatically by the IDE when renaming the package. The 10% manual rework, I do not see, how to minimize it further.

Ok, but you only considered this repository right? For example there is also GH map-matching and navigation. If there are methods that have the same name and signature and do something different we should rename them before putting them into the same package or make them visible outside of some package.

Indeed, I did not know about the other repositories, but a quick look showed, that they are already in separate sub-packages com.graphhopper.navigation and com.graphhopper.matching. So there is no issue with them.
The only problem arises if a repository/downstream dependency etc. is in the same package as one of the moved subpackages inside the new com.graphhopper.api (e.g. com.graphhopper.util which now becomes partly com.graphhopper.api.util). So I believe only the core module might be in fact affected and thus only repositories and dependencies in combination with the core module.

Cool thanks for this!

The remaining 10% was for classes that used wildcard-imports like com.graphhopper.util.*

I find these rather annoying anyway. I think the default IntelliJ settings we use make use of star imports as soon as there are five or more imports from the same package. But this also means that sometimes the import statements go back and forth when the number of imports changes. Maybe we should set this limit to zero (never use star import) and apply this to the whole code base before renaming the package(s).

And in one class there was the issue, that protected fields were used, instead of pre-existing public getter-methods. What's your take on this? Should the protected fields made public in the new package com.graphhopper.api, so that no depending code is broken? Or should the developers be "pushed" to use the already existing public getters and setters?

Hm hard to say in general I did not look at these cases yet.

Indeed, I did not know about the other repositories, but a quick look showed, that they are already in separate sub-packages com.graphhopper.navigation and com.graphhopper.matching. So there is no issue with them.

I was more concerned about other repositories or branches making use of the classes we are moving to another package, because this requires updating all the import statements.

And would it not be much easier (to achieve the goal of JDK9+ compatibility) to move the classes of the other (not graphhopper-api which is used most frequently) modules first?
Also before doing this maybe we should have another try of applying the auto-formatting to the whole code-base, because this could also become messy with other branches otherwise, see this discussion: https://github.com/graphhopper/graphhopper/pull/1638#issuecomment-574628308

I was more concerned about other repositories or branches making use of the classes we are moving to another package, because this requires updating all the import statements

Yes, unfortenately this seems to have to be the case... but at least I believe we think alike: "Do not change anything more than you absolutely have to... ;-)"

Hm, ok, I see core/com.graphhopper.util is also used very often...

Yes exactely. The JDK9+ Jigsaw compatibility issues seem to mainly be only between api and core (since they share the split packages)... and core should even be used more often in dependencies... I'd imagine.

Is there a verdict on how to proceed? Can you foresee a (non-binding) approximate timeline for when this could be included in a release?
Depending on this, I'd be able to include version 1.0 in our project instead of 0.10.3
Greetings  from Germany.

The (preliminary) verdict seems to be that we first go forward with trying to use fewer classes in the api module. And then package-rename the rest.
Might take a while, it's a bit hairy, there's a reason why we haven't done it yet.

Hi michaz,
thanks for the quick feedback. Yes I understand, that will take a while for 402 classes ;-).
Could you then please re-consider my proposal of introducing a further maven-package graphhopper-coreapi as suggested in PR #2055 (https://github.com/graphhopper/graphhopper/pull/2055)?
With this proposal no change to the code would be necessary, but the JDK-9+ developers would not be shut out bit by bit (The problem is, that as soon as a maven project has another dependency which introduces a modules-info.java file for the dependency, the problem with the split classes in graphhopper-core and -api will prevent the project from compiling. So my fear is, that as the other maven dependencies slowly but surely will be updated to have a modules-info.java, graphhopper will become less and less compatible to projects).
Thanks for your consideration.

I don't think I want to do this. I'm not an expert for the new module system, so I don't really want to publish a hack for which I don't fully understand its implications.

I will go forward with orthogonalizing the packages though.

Well, for using it for our project, I would have to publish the maven dependency to a public maven repository. I think I'd call it graphhopper-coreapi in my own namespace.
Would I get your permission to do this? I wouldn't change anything in the code, just combine both modules into one and compile it... and publish it like described here: http://tutorials.jenkov.com/maven/publish-to-central-maven-repository.html
I'd keep the apache 2.0 license as used by yourself as well as the other parameters in the current pom-file, just using a sperate namespace.
So then maybe even other JDK9+ developers could use this workaround.
Would this be ok for you? Would you give me the permission to do this?

Hi,
any news here?
Would you allow for me to compile the preexisting source code of 1.0 with identical license and author recognisation into a new module graphhopper-coreapi and publish it onto a public maven repository?
Would be great, because that would allow us to use graphhopper 1.0 as well as maybe some jdk9+ developers who would be able to use this as a workaround, until the packages are orthogonalized.
Greetings from Germany

@oli0044 Or can you use gh-coreapi for simplicity :) ?

@oli0044 I just removed the api module in #2372. It's contents are now in web-api. Maybe this also helps for this issue?
There are still some packages split over multiple modules:

core and web-api both use com.graphhopper
core and web-api both use com.graphhopper.util
web and web-bundle both use com.graphhopper.http

And can we close #2066 and #2064? I think they are no longer up to date, especially because of #2372.

Hello @easbar
The changes in the current version 4.0 are definitely going in the right direction concerning this #2053.
For my project to work with the newest Graphhopper version again it would suffice to move the classes in com.graphhopper from the web-api module into the core module as well as the classes in com.graphhopper.util from web-api into core.
I believe these classes also belong more into the core module than into the web-api module or am I wrong?
I'll try to suggest a renewed pull request that would do that later this evening.
#2066 and #2064 can be closed then, because they are not up to date anymore.
Greetings

I believe these classes also belong more into the core module than into the web-api module or am I wrong?

I think they are in web-api, because they are needed for the client-hc module, our hand-crafted(=hc) Java client. The idea was probably that it should be as lightweight as possible, i.e. not depend on core. For Jigsaw compatibility wouldn't it be sufficient if all modules used their own package names (instead of spreading packages like com.graphhopper over multiple modules?

yes indeed, that would be sufficient and imho the most durable solution.
However giving each module its own subpackage - which would be the cleanest and most durable solution - would of course lead into massive refactoring of all import lines (also in dependent projects).
A not so intrusive solution might be to move com.graphhopper inside web-api into com.graphhopper.api and com.graphhopper.util inside web-api into com.graphhopper.api.util.
Might that be a compromise? Then the imports won't be changed aaaas massive as if each module gets its own subpackage.
How do you want to solve it? I can offer to create a pull request with the chosen solution, because I am very interested in using the new versions of graphhopper again.

However giving each module its own subpackage - which would be the cleanest and most durable solution - would of course lead into massive refactoring of all import lines (also in dependent projects).

Yes, I'd rather change the package names only for the classes of packages that are currently distributed over multiple modules, i.e. minimal changes to fulfill the Jigsaw compatibility requirement for now. GraphHopper's package structure needs an overhaul anyway, so if we change the package names we should actually think about what packages we want first, but that would be mostly for another issue.
Let's see what we got:
com.graphhopper:
core: GraphHopper,GraphHopperConfig
web-api: GHRequest,GHResponse,Trip,ResponsePath
com.graphhopper.util:
core: AngleCalc,ArrayUtil,BitUtil/Big/Little,Breadth/Depth/XFirstSearch,DistanceCalc/3D/Earth/Euclidean/Projection,Constants,DouglasPeucker,Downloader,EdgeExplorer/Iterator/State,FetchMode,GHUtility,GitInfo,Instructions,MiniPerfTest,NotThreadSafe,PathMerger,PathSimplification,ProgressListener,SimpleIntDeque,StopWatch,TranslationMap,Unzipper (wow this one is crazy!)
web-api: Custommodel,FinishInstruction,Helper,Instruction,InstructionList,JsonFeature,JsonFeatureCollection,NumHelper,Parameters,PMap,PointAccess,PointList,RoundaboutInstruction,ShallowImmutablePointList,Translation,ViaInstruction
com.graphhopper.http:
web: CORSFilter, GraphHopperApplication, GraphHopperServerConfiguration
web-bundle: DurationParam,FeedConfiguration,GHJerseyViolationExceptionMapper,GHLocationParam,GHPointParam,GraphHopper/RealtimeBundle,GraphHopper/RealtimeBundleConfiguration,GraphHopperManaged,IllegalArgument/MultiExceptionMapper,JsonErrorEntity,MultiExceptionGPXMessageBodyWriter,PtRedirectFilter,RealtimeConfiguration,RealtimeFeedLoadingCache,TypeGPXFilter
I would assume that the higher-level classes are used in downstream code the most. It is probably least invasive to leave the web-api module untouched. So we need to move GraphHopper/Config in core to another package (I think that's ok, and definitely better than changing the package name for GHRequest/Response for example).
Most of the classes in core's com.graphhopper.util are rather low level and it should be ok to move them to another package. Especially because 'utildoes not seem to be a create package for them to be in anyway. I wonder if some of them should go to theweb-api` module. @karussell I can't find it anymore, but wasn't there an issue that was about moving distance calc stuff (or instructions code(?)) into client-hc?
com.graphhopper.http should be easy to fix, because these classes are probably hardly ever used explicitly in downstream code. Maybe we can move the three classes that are in web into web-bundle?

I can offer to create a pull request with the chosen solution, because I am very interested in using the new versions of graphhopper again.

Thanks, that is very nice, I am not sure if this really will speed up the process, also because if we do this refactoring I will probably do it for our own downstream dependencies as well, so I can just as well do this. First we need to decide where the different classes should go though. @karussell, @michaz any thoughts?

And of course the easiest thing we can do is deleting classes, instead of moving them. NotThreadSafe and ProgressListener are very high on my list. The first one we don't really use consistently (only in RoutingAlgorithm), and the second one is just a LongConsumer. Instructions is only used in tests, so maybe we can remove it as well?

@karussell I can't find it anymore, but wasn't there an issue that was about moving distance calc stuff (or instructions code(?)) into client-hc?

Maybe you meant this? https://github.com/graphhopper/graphhopper/issues/2250

Maybe you meant this? #2250

Ah, yes exactly :) So we should fix this one first.

Hi,
long time no read ;-).
I saw that there were a lot of steps into the right direction to orthogonalize the modules.
However there are still the issues of the classes in
com.graphhopper:
core: GraphHopper,GraphHopperConfig
web-api: GHRequest,GHResponse,Trip,ResponsePath
com.graphhopper.util:
core:
AngleCalc,ArrayUtil,BitUtil/Big/Little,Breadth/Depth/XFirstSearch,DistanceCalc/3D/Earth/Euclidean/Projection,Constants,Downloader,EdgeExplorer/Iterator/State,FetchMode,GHUtility,GitInfo,Instructions,MiniPerfTest,PathMerger,PathSimplification,RamerDouglasPeucker,SimpleIntDeque,StopWatch,TranslationMap,Unzipper
web-api: CustomModel,FinishInstruction,Helper,Instruction,InstructionList,JsonFeature,JsonFeatureCollection,NumHelper,Parameters,PMap,PointAccess,PointList,RoundaboutInstruction,ShallowImmutablePointList,Translation,ViaInstruction
com.graphhopper.util.details:
core:
AbstractPathDetailsBuilder,EdgeIdDetails,KVStringDetails,TimeDetails,AverageSpeedDetails,EdgeKeyDetails,PathDetailsBuilderFactory,WeightDetails,BooleanDetails,EnumDetails,PathDetailsBuilder,DecimalDetails,IntDetails,PathDetailsFromEdges,DistanceDetails,IntersectionDetails,StringDetails
web-api:
PathDetail
com.graphhopper.util.shapes:
core:
BBox,Circle,Polygon,Shape
web-api:
GHPoint,GHPoint3D
Is there still some tidying up to do with the classes or is it possible to rename the packages? I have made a pr that put

all the classes in com.graphhopper in the core module into com.graphhopper.core as well as
all the classes in com.graphhopper.util? in the coremodule intocom.graphhopper.core.util`.

That would fix the issue of split packages with the new Jigsaw module system and the next graphhopper release could be used with all maven projects with Java 9+.
Is it possible to finish this? It is now over 2,5 year ;).
Greetings from Germany
oli0044

Heyhey :),
any tendencies here?
In order to compare, I created a second pull request, showing what would be necessary to not change anything in core module but to do the changes in web-api module:

all the classes in `com.graphhopper` in the core module into `com.graphhopper.core` as well as

all the classes in `com.graphhopper.util` in the core module into `com.graphhopper.core.util`.

In summary: The amount of changes is similar. But it seems more logical and a bit less intrusive to move/rename the packages in core module.
What are your thoughts?
Greetings
Oli4

If we can choose between:

GraphHopper cannot be used with JDK9+ as soon as there is a module-info file somewhere in the project
All package names must be prefixed by the Maven module name (or some other, unique prefix)

I'd say we should go for 2) and we should do the package renaming before the 7.0 release. But maybe wait for a few other things to be merged to not create too much of a mess.
Do you have an example project that shows what exactly is currently not possible?

Hi easbar,
I created a 1 file project in https://github.com/oli0044/graphhopper-java11-example (collaborator invite should also have been sent).
It is not compiling because with the module-info.java, but compiling without it.


added dockerfile

#10

Closing because this is so out of date. Will use the related work as cherry-picked commits in a new PR coming up.


Add missing log mount to ironic-api

ironic-api-log is not currently logging anything because /var/log/ironic is not mounted into the ironic-api container. This change fixes that.
(cherry picked from commit 4597a402a7f2b1dbe76560d9a3def63fd7b4ca9a)
Jira: OSPRH-10238

/retest


[💡FEATURE REQUEST]: Enhance CORS support

Current CORS headers settings doesn't covers my use cases.
First of all I want a support of multiple allowed origins (or even any origin) with credentials enabled. Currently http.headers.cors.allowed_origin = * and http.headers.cors.allow_credentials = true is not working together.
Would be nice if allowed_origin can be configured as single origin or array of origins, and option to reflect Origin from request header (maybe combination of allowed_origin = * and allow_credentials = true or some reserved allowed_origin value). CORS middleware example in Go, that can acts like described: https://github.com/rs/cors
Second little problem is deprecation message from browser like that, when allowed_headers = *:
"Authorization" will not be covered by the wildcard symbol (*)in CORS "Access-Control-Allow-Headers" handling.

I found out that it can be reflected from Access-Control-Request-Headers request header if it specified by client.

Multiply allowed_origins now supported. Fixed by the: https://github.com/roadrunner-server/headers/pull/48

RoadRunner can allow any origin reflected from request header

This is not an expected behavior (and non-standard), origins should be set explicitly. Not planned to implement.


Use execFile instead of exec

To prevent security risk & improve performance (do not rely on a shell)

Hi @Congelli501, thanks a lot for your contribution! The change indeed looks very promising. I'm merging the MR and if tests pass I'll publish a new version on npm.


[CSL-2313] Bump report server to 0.4.7

Builds and works for me locally.
Please reference [CSL-2313] when merging.


So is this project still supported?

I've not seen that much activity, since a while ago.

You can use shelljs by Google

You can use shelljs by Google

ShellJS is not a Google project. It's a project I personally maintain (I coincidentally work for Google).

I've not seen that much activity, since a while ago.

Correct, we haven't done much on this project in quite a while. We had plans to rewrite the core engine (vorpal) to follow proper bash syntax and semantics, and made good progress (this project), but work has stalled a bit.

My mistake regarding project ownership! I'll check out the project and see if i can help out


css has benn updated

css for the ratings has been updated and committed

@Sivasaiklu Please note that this is not in line with our code of conduct. To ensure clarity and proper assignment tracking, please attach the issue number to your PR description. This helps us confirm that the issue is assigned to you. Kindly update your PR with the issue number and resubmit.
Example:
Fixes #123 - Brief description of the fix.