Trouble when using a sysimage with PythonCall

I'm having trouble when building a sysimage with PythonCall-- or rather, I can built it OK, but then when I use it, like
julia --project=myproject --sysimage=mysysimage.so -e "using Pkg; Pkg.instantiate()"

I get
#29 92.79 fatal: error thrown and no exception handler available.
#29 92.79 InitError(mod=:C, error=ErrorException("no environment in the LOAD_PATH depends on CondaPkg"))
#29 92.84 error at ./error.jl:33
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 _resolve_top_env at /root/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:16
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 #resolve#17 at /root/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:222
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 resolve at /root/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:209
#29 92.84 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.84 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.84 envdir at /root/.julia/packages/CondaPkg/GoPlj/src/env.jl:66
#29 92.85 init_context at /root/.julia/packages/PythonCall/Z6DIG/src/cpython/context.jl:56
#29 92.86 __init__ at /root/.julia/packages/PythonCall/Z6DIG/src/cpython/CPython.jl:21
#29 92.87 jfptr___init___72494 at /deps.so (unknown line)
#29 92.87 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
#29 92.87 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#29 92.87 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
#29 92.87 jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:73
#29 92.87 _finish_julia_init at /buildworker/worker/package_linux64/build/src/init.c:796
#29 92.87 julia_init at /buildworker/worker/package_linux64/build/src/init.c:730
#29 92.87 jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:695
#29 92.87 main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
#29 92.88 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
#29 92.88 _start at /usr/local/julia/bin/julia (unknown line)
#29 93.11 ERROR: Failed to precompile MyPackage [...redacted...]
#29 94.21 Stacktrace:
#29 94.21  [1] error(s::String)
#29 94.60    @ Base ./error.jl:33
#29 94.77  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
#29 94.77    @ Base ./loading.jl:1466
#29 94.77  [3] compilecache(pkg::Base.PkgId, path::String)
#29 94.78    @ Base ./loading.jl:1410
#29 94.78  [4] _require(pkg::Base.PkgId)
#29 94.78    @ Base ./loading.jl:1120
#29 94.78  [5] require(uuidkey::Base.PkgId)
#29 94.78    @ Base ./loading.jl:1013
#29 94.78  [6] require(into::Module, mod::Symbol)
#29 94.78    @ Base ./loading.jl:997

(please excuse the #29 92.79 stuff, I'm in a dockerfile here).
I believe the issue is that PythonCall's __init__ tries to resolve the CondaPkg environment, but when a module is in a sysimage, that __init__ occurs at startup time (i.e. when julia itself is starting up), and at that point in the process the package environment isn't setup yet... or something like that.

You're saying __init__ is being called before LOAD_PATH is set? Surely that can't be right, it's a documented variable in Base, and Base must have already initialised? Can you inject some code somewhere to print out the LOAD_PATH to check if this is the case?

I tried adding println(Base.load_path()) to the first line of PythonCall's __init__, but for some reason that didn't print:
julia> using PackageCompiler # in an environment with my dev'd copy of PythonCall and PackageCompiler

julia> create_sysimage(["PythonCall"]; sysimage_path="pythoncall2.so")
✔ [01m:45s] PackageCompiler: compiling incremental system image
ld: warning: could not create compact unwind for _julia_Dict_15870: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for _julia_Dict_15862: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for _julia_YY.236_57076: stack subq instruction is too different from dwarf stack size

julia> exit()

❯ julia --sysimage=pythoncall2.so
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=ErrorException("no environment in the LOAD_PATH depends on CondaPkg"))
error at /Users/eph/pythoncall2.so (unknown line)
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
_resolve_top_env at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:16
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
#resolve#17 at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:222
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
resolve at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/resolve.jl:209
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
envdir at /Users/eph/.julia/packages/CondaPkg/GoPlj/src/env.jl:66
init_context at /Users/eph/pythoncall2.so (unknown line)
__init__ at /Users/eph/pythoncall2.so (unknown line)
jfptr___init___48878 at /Users/eph/pythoncall2.so (unknown line)
jl_apply_generic at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
jl_module_run_initializer at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
_finish_julia_init at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
julia_init at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)
jl_repl_entrypoint at /Users/eph/.julia/juliaup/julia-1.7.2+0~x64/lib/julia/libjulia-internal.1.7.dylib (unknown line)

Works for me:
❯ julia --project --sysimage="pythoncall.dylib"
    CondaPkg Found dependencies: /Users/kristoffercarlsson/.julia/packages/PythonCall/Z6DIG/CondaPkg.toml
    CondaPkg Removing environment
...

Due to the way these packages seem structured make sure that you are using the same project as that you created the sysimage with.

Due to the way these packages seem structured make sure that you are using the same project as that you created the sysimage with.

Ah ok, I was not doing that-- on purpose, because I want to build a sysimage w/ all dependencies but not my package code, so I don't need to rebuild the sysimage every time I change the code I'm working on. (Basically this strategy: https://github.com/beacon-biosignals/julia_pod/blob/94a91696ed68e97d7749ccb5659ed22c0d27b048/add_me_to_your_PATH/Dockerfile.template#L4-L16).

Did you ever resolve this? I guess like the error says you need to ensure that CondaPkg is actually a dependency of something in your LOAD_PATH.

The same error occurs when I'm trying to run app.exe (created with PackageCompiler.create_app)

Did you ever resolve this?

No, I just removed PythonCall from my sysimage as a workaround for now.

I guess like the error says you need to ensure that CondaPkg is actually a dependency of something in your LOAD_PATH.

Hm, I thought I had that, but maybe not.

Logging for posterity as I'm hitting the issue from the other direction: building a sysimage in julia then trying to execute it via juliacall.
MWE - inside a virtual env
$ julia --project=/path/tovenv/julia_env

julia> using PackageCompiler; Pkg

julia> project = Pkg.project()

julia> create_sysimage(collect(keys(project.dependencies)); sysimage_path="./venv/julia_env/vmr-image.so")

and then trying to load from a python session:
$ python -X juliacall-sysimage=/path/to/venv/julia_env/vmr-image.so -X juliapkg-project=/path/to/venv/julia_env
>>> import juliapkg
>>> juliapkg.project()
'/path/to/venv/julia_env'
>>> import juliacall
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=ErrorException("no environment in the LOAD_PATH (["/home/ubuntu/.julia/environments/v1.9/Project.toml", "/home/ubuntu/.julia-downloads/julia-1.9.1/share/julia/stdlib/v1.9"]) depends on CondaPkg"))
error at ./error.jl:35
_resolve_top_env at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/resolve.jl:16
#resolve#38 at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/resolve.jl:404
resolve at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/resolve.jl:384
envdir at /home/ubuntu/.julia/packages/CondaPkg/osUdN/src/env.jl:70 [inlined]
init_context at /home/ubuntu/.julia/packages/PythonCall/FCrXm/src/cpython/context.jl:63
__init__ at /home/ubuntu/.julia/packages/PythonCall/FCrXm/src/cpython/CPython.jl:21
jfptr___init___103567 at /home/ubuntu/projects/Maestro.jl/experiments/07_vanilla_map_reduce_sidecar2/venv/julia_env/vmr-image.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_module_run_initializer at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:75
_finish_julia_init at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/init.c:855
julia_init at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/init.c:804
ijl_init_with_image at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/jlapi.c:66 [inlined]
ijl_init_with_image at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/jlapi.c:55
unknown function (ip: 0x7f4d56ce1ff4)
unknown function (ip: 0x7f4d56ce1409)
_ctypes_callproc at /usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so (unknown line)
unknown function (ip: 0x7f4d56cfaae6)
_PyObject_MakeTpCall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
PyEval_EvalCode at python (unknown line)
unknown function (ip: 0x601173)
unknown function (ip: 0x5c52ef)
PyVectorcall_Call at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyFunction_Vectorcall at python (unknown line)
unknown function (ip: 0x5f3c40)
_PyObject_CallMethodIdObjArgs at python (unknown line)
PyImport_ImportModuleLevelObject at python (unknown line)
_PyEval_EvalFrameDefault at python (unknown line)
_PyEval_EvalCodeWithName at python (unknown line)
PyEval_EvalCode at python (unknown line)
unknown function (ip: 0x680000)
unknown function (ip: 0x68007e)
unknown function (ip: 0x4a44a3)
PyRun_InteractiveLoopFlags at python (unknown line)
PyRun_AnyFileExFlags at python (unknown line)
unknown function (ip: 0x4eeecd)
Py_BytesMain at python (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at python (unknown line)

I believe this is not stale (and autoclose bots are annoying).
Specifically, I think this is the current state:

If I build the sysimage w/ the whole top-level environment and pythoncall, then it works OK
If I build the sysimage without pythoncall, then add it dynamically after, then it also works
but if I try to save time by adding pythoncall to the sysimage (so it is baked in at that level) but add the final environment later, that's where I run into this issue

That could be considered an acceptable state of affairs (esp. if pkgimages mean sysimages are less relevant) but it can be very confusing to run into and I think unless it's clearly documented somewhere (or fixed) the issue should stay open. (in my opinion).

I'm not quite sure what you mean by "final environment"?
TBH I'm really (really really) not an expert in sysimages so I can't help much. The error (as far as CondaPkg sees it) is that your LOAD_PATH (at the time the package is loaded) does not have any entries depending on CondaPkg - which should not happen because you're loading CondaPkg from somewhere in your environment. So presumably some sysimage thing is messing that up - at which point my expertise stops and you'll have to figure out why.

(I agree autoclose bots are annoying, but it's also annoying and stressful for me to have more open issues than I can deal with sat there with no activity. The bot is simply a way to spread the load a bit - either the person who wrote the issue can help out a bit to fix the problem, or if they don't care enough then the issue may as well be closed. I've updated the text the bot uses to be a little less cold.)

If you think the issue is alleviated by better documentation, feel free to open a PR on docs/src/faq.md.

that your LOAD_PATH (at the time the package is loaded)

I think this might be basically the issue.
I am also not a sysimage expert, but I think it could be that when you add a package into a sysimage, you're essentially loading the package as normal, and then later serializing out the entire state of your julia session, loaded packages and all. So it could be that __init__ time is when the package is being added to the sysimage, which isn't really when we actually want to use the package; we want to use it later when we start our julia session (with the sysimage).
So I think the code "fix" would be to not do this work at init time, but rather some other time (e.g. on the first call to any function that needs to do something w/ python).
I think the docs "fix" would be a FAQ question like you said, and maybe that's preferable.

Actually I'm not sure my explanation is accurate, re-reading the original issue again... Since the error occurs later, once we start to use the sysimage. So I am lost as to the exact issue. But maybe I can still try to add a FAQ entry


Allow interpolated variable to contain =

Use case - Interpolate public key -v public_key=ssh-rsa ...G4/+VHa1aw==

Story: https://www.pivotaltracker.com/story/show/131323057


make variable, block raise error

    raise InvalidFlag(flag)
,printy.exceptions.InvalidFlag: ''' is not a valid flag

            if flag not in available_flags and is_raise_error:
                raise InvalidFlag(flag)

Hi, can you elaborate better what the issue is please? are you trying to use a single quote as a flag? or what are you trying to achieve? thanks!

im trying show message with quote as a flag

Not quite understanding what the issue is, can you add an example or a code snippet showing what you trying to acomplish, please?
Not sure why you're trying to add a single quote as a flag, flags define the cors you want to use, and the quotes are not one of them.


Miscellaneous EXT_structural_metadata changes

This PR is based on #716, so merge that first.
This PR includes a batch of changes related to EXT_structural_metadata support:

Log warning about EXT_feature_metadata when glTFs are loaded.
Add ability to access name, description, semantic properties from PropertyView
Add support for sentinel values when upgrading batch table. If possible, null can be converted into a sentinel value that is put as the noData value of the class property.
Add some missing checks to TestPntsToGltfConverter. The lines were originally written for EXT_feature_metadata and were not properly replaced.
Add additional typedefs for PropertyTexturePropertyView and PropertyAttributePropertyView variants. Even though they could probably fit into one variant typedef each, I wanted them to have symmetry with PropertyTablePropertyView, which has too many types to fit in one variant.

Looks great, happy to merge this one too, after the others.

While testing this out in Cesium for Unreal, I triggered an assert with New York 3D Buildings. This was due to an attempt to handle an instance of the "Year" property that contained all null values. I just pushed a commit with some code to catch this and ignore the property if this is the case.


Create forking code page

[x] Create page
[x] Add description of how to fork code from current documentation.
[x] Populate with schematron messages
[x] Provide explanations for how to fix schematron errors and warnings
[x] Review house style document to ensure all requirements are covered by GitBook and schematron
[x] Review schematron messages related to this topic to ensure they are user friendly
[x] Becky approved page
[x] Melissa approved page
[x] Fred approved page
[x] James approved page
[x] Naushin approved page
[x] Share page with Exeter and invite feedback

Definition of done

[x] Page completed
[x] Team reviewed and approved page?
[x] Assign Fred to update the Schematron

@FAtherden-eLife I think we can update the schematron messages to link to the forking code page - any revisions I need to make won't affect that section!


Clarify that maui-check must be ran as an elevated user

Maui-check must be ran in an elevated terminal. It doesn't do anything otherwise and the docs don't make this clear.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7607155a-9a56-d752-c5ac-c3aeb6b128bc
Version Independent ID: 9a62d421-d009-30b3-843e-a92bfc21c061
Content: .NET MAUI installation - .NET MAUI
Content Source: docs/get-started/installation.md
Product: dotnet-mobile
Technology: dotnet-maui
GitHub Login: @davidbritch
Microsoft Alias: dabritch

Hello @joehal9000  I'm prompted for elevation, are you not?


Implement partial_fit support in DaskBaseSearchCV

It looks like this will require passing a keyword argument to method.py#fit.
I think there also needs to be some specification of the random state to make sure the train/test split is preserved on repeated calls.

Migrate this to Dask-ML?

Done


Fix CJK symbols and punctuation unicode

fix #1127
Reference:
https://en.wikipedia.org/wiki/CJK_Symbols_and_Punctuation

I will add a test for this. Assuming it works with the test logic in place, I will go ahead and merge this.

It looks good @NatsukiFukazawa . Thanks for making the PR and reporting the issue. I will go ahead and merge this after the CI runs.


update go version to 1.19

Codecov Report

Merging #104 (9def2fe) into master (ddcfeb7) will not change coverage.
The diff coverage is n/a.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@           Coverage Diff           @@
##           master     #104   +/-   ##
=======================================
  Coverage   31.32%   31.32%
=======================================
  Files          19       19
  Lines        2503     2503
=======================================
  Hits          784      784
  Misses       1674     1674
  Partials       45       45

Flag
Coverage Δ

unittests
31.32% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


how can i get the logic inside static fetch data

Hello, I have a problem with redux-wait-for-action, I need to add the noindex meta tag to the head, if the array of elements from the backend is empty. When I use inspect, it works, but when I open the source code view page, the meta tag is not added . Can I use redux-wait-for-action to solve a problem?
I tried this
static fetchData({dispatch}){     dispatch({       type: HELMET_NOINDEX,       isShow: true     })   }
action
export const helmetNoindex = isShow => ({type: HELMET_NOINDEX, isShow})
this works when the isShow property takes a static value like true, but when I tried to use a dynamic variable like this
static fetchData({dispatch, isAddMeta}){     dispatch({       type: HELMET_NOINDEX,       isShow: isAddMeta     })   }
and call in componentDidMount
ListingsAndMapContainer.fetchData({dispatch: this.props.dispatch, isAddMeta: false})
isAddMeta is undefined on server. I used react-redux and express on server, please tell me how can I solve this problem or use own props inside static fetchData ?

If I understand correctly, you will need to add that meta tag on the server side.  If you see it correctly in the inspector, your front-end code is probably working correctly.

Yeah but I need to transfer properties to the server, can I do it with redux-wait-for-action? Cause when items from backend empty, store on the server not changing

@SaiRO97
You can put the data in the URL arguments so that you can read it from the server.
However, If the data you want to send to the server is generated by user interaction, there's nothing to do with this module because it only works with first-screen server-side rendering.


Issue on page /advanced/eeg_introduction.html

I noticed that some of the citations included in text are not properly rendered (see example screenshots below).
Otherwise I've been enjoying the eeg bit a lot!! Really helpful, especially since I'm getting into it more and more :) Nice work!

Thanks @JNPauli. @felixkoerber and/or @M-earnest who's currently maintaining the resources?

Thanks @JNPauli. I'll have a look over the next weekend. Glad you enjoy the content!
To answer you question @PeerHerholz,- I'm officially the maintainer of the DilER-Digitell position, although I hardly find the time to call myself such. As we're just entering a new funding stage this hopefully will change soon and we will update and redo some of the content.
@PeerHerholz & @JNPauli please feel free to suggest content you would like to see added or updated!


Please make a release

For packaging system, it's nice to have a proper release. Can you please make one? Thank you.

Done :)

Thank you very much, it is now packaged for pkgsrc!


Better error handling in our Build

This is a pretty big update to the build including fixes to all JavaScript files for linting errors. The following changes will add:

Gulp Plumber This is mostly for better general error handling. I replaced all the .on() event listeners on all pipes.
jsHint Integration This will stop any JavaScript errors and most stylistic errors.
Gulp Less Hinting This should fix stylistic issues with Less Files.
Actual Failing Builds If there are any errors, the build will actually fail now and will not pass.

approved -- looks great!

I like what you got! Nice job! Approved.


project replicate failed

1,Docker version 17.05.0-ce, build 89658be;docker-compose version 1.14.0, build c7bdf9e;
2  setting:i set the variables following this:
harbor.cfg
hostname = <IP_ADDRESS>
auth_mode = ldap_auth
dap_url = ldap://xxxx
ldap_basedn = ou=xxx,dc=xxx,dc=com
config/registry/config.yml
proxy:
remoteurl: https://registry-1.docker.io
username: username
password: passwoord
/etc/docker/daemon.json
"registry-mirrors":["my.registry.com"]
for example:
1,harbor A and harbor B,I replicate A's project to B,
2,project C(This project was pushed up from my local  docker command)，  project D（i pulled form registry.docker-cn.com）
when I replicate the project C,that is succeed,  but when i replicate the project D,it is failed,  i don't know why?can you help me?and i have seen the log,that is normal.

@reasonerjt

@chenliping972385283 let me clarify.
When you say "it is failed" do you mean the job is in Error State?

Close due to inactivity.


Do not remove H atoms mentioned in Substance Groups

This addresses #2716.
Atoms mentioned in any of the details of a Substance Group will be ignored by removeHs(), as removing any of them will cause the SGroups to be cleared from the molecule (as our policy is still to wipe them if the molecule is edited). Maybe in the future, when we implement proper edition of Substance Groups we can relax this, and allow removal of H atoms that are part of the SGroup but don't play any particular role (i.e. they are only mentioned in the atoms list, but are not part of any notable bonds are neither a parent atom).
A small note on the sgroupAtoms map: intuitively, it seems sane that H atoms will only belong to a single SGroup, but, even in the case that any of them would be in more than one, I don't think it is worth to store a vector of SGroup indexes, since we only use them for logging the reason for which H atoms have not been removed.

I think we're going to eventually want to add a new form of removeHs() that allows the caller to specify which exceptions to apply. But that's clearly not part of this PR.

I just realized (remembered?) that this has some more implications than I thought of when I noticed the SGroups disappearing: removing atoms from a mol that has SGroups might change the indexes of atoms and bonds. And, as SGroups store references to atoms and bonds as indexes, this will invalidate them. Which is why we cleared the SGroups when the molecule was edited...

Yeah, it will definitely require some thought and care.

@ricrogz : this just needs that one doc change and a C++ test and it's ready to merge

@ricrogz : this just needs that one doc change and a C++ test and it's ready to merge

Actually not, that is what I realized and commented about: if any atoms are removed from the molecule, the indexes of the other atoms that are included or referenced from the SGroups will change, and the SGroup will be in an invalid state.
Unless we add some code to update these indexes, we shouldn't push this, and just use removeHs=false instead.

These changes should be covered by #3169
Closing this, but please re-open if I missed something


Searching Delta

Is there a way to search only deltas after the repo has been scanned once?

@sohailnajar This has been moved to the new fork as this project is no longer maintained.


Feature nrtest

Addresses Issue #97. It wasn't pretty, but nrtest is now running from Travis CI.
Note that the tests are failing, however, nrtest is running properly. Testing identifies differences in water quality results between the EPANET v 2.0.12 reference and the current build.

@michaeltryby, Awesome work!

Suggest doing a squash and merge on this one!

Review of pull requests is a great way to learn and improve the quality of our work. If @samhatchett is too busy perhaps another member of the dev committee could review this. Sound good?

@michaeltryby, do you think there might be value in adding more tests?

@bemcdonnell Yeah I agree that more tests would be good. I asked @LRossman if he would contribute his bench marking library. Waiting to hear back from him.

@michaeltryby as you know, i've been on vacation all last week. Thus the delay.
Of course anyone else is welcome and encouraged to add their review as well, but I intend to look this over in the next 4-5 days.

Thanks @samhatchett!


Fix Demo project

For #129

Coverage remained the same at 89.266% when pulling c6cbdb1d3614167c42ee71a43d628fdc2f3a510f on embbnux:demo_v4 into 9e24982e2c396984abc116b46994174dc93689b1 on ringcentral:master.

Closed by fixed in master


In-memory engine: adjust name from range_cache_* to region_cache_*

What is changed and how it works?

Issue Number: Ref #16141

What's Changed:
adjust name from range_cache_* to region_cache_*

Manual test for checking metrics work as expected:

Related changes

[ ] PR to update pingcap/docs/pingcap/docs-cn:
[ ] Need to cherry-pick to the release branch

Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Release note

None

/test pull-unit-test

/test pull-unit-test

/test pull-unit-test

/test pull-unit-test

/retest

/retests

/cc PTAL @zhangjinpeng87 thanks.


fix labels #24

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 38.98%. Comparing base (13785e5) to head (aab57fe).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   39.54%   38.98%   -0.56%
==========================================
  Files          10       10
  Lines        1970     1970
==========================================
- Hits          779      768      -11
- Misses       1191     1202      +11

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Adapt to new grabl

What is the goal of this PR?
Grabl recently changed the automation.yml syntax to have image instead of machine
What are the changes implemented in this PR?

Replace all machine: usages with image:
Only depend on build versions of other repos

Merging despite the fact test-behaviour failed - the failure might have been introduced in a prior PR.


Add basic deadlock detection for container start/remove

We can easily tell if we're going to deadlock by comparing lock IDs before actually taking the lock. Add a few checks for this in common places where deadlocks might occur.
This does not yet cover pod operations, where detection is more difficult (and costly) due to the number of locks being involved being higher than 2.
Also, add some error wrapping on the Podman side, so we can tell people to use system renumber when it occurs.

LGTM, although it would have been better if tool just automatically renumbered rather then making the user do it.

Renumber should have all containers stopped in order to work properly, so that probably won't work

LGTM assuming happy tests

They don't seem to have fired. I'll repush.

/retest

This is ready

/lgtm


might be a bug, might not be

When running hcxdumptool for a long time (several hours) I sometimes get some "weird" values in the probe request that although I guess could be correct look a little weird. I've also had my screen go all weird (sorry no screenshot of that) where the text on the screen (ssh session via tmux) goes all weird with line drawing chartacters & graphic chars & unreadable and I have to disconnect and reconnect to get back to normal alphabetic characters

When running hcxdumptool for a long time (several hours) I sometimes get some "weird" values in the probe request that although I guess could be correct look a little weird.
-> Correct, caused by bit errors in the PLCP header
In monitor mode the adapter does not check to see if the cyclic redundancy check (CRC
values are correct for packets captured. The device is able to detect the Physical Layer
Convergence Procedure (PLCP) preamble and is able to synchronize to it, but if there is
a bit error in the payload. This can lead to unexpected results.
I've also had my screen go all weird
-> That is an issue. Thanks for reporting it. I'm going to fix that. Please stay tuned...

Should be fixed by this commit:
https://github.com/ZerBea/hcxdumptool/commit/1731c1fccca53740c0a98abcff3899e5f6307f41
Now we replace characters < 0x20 and characters > 0x7e with ".". Additional we handle ESCAPE sequences within ESSIDs. Terminal should no longer be destroyed.
Please test and close issue, if it works for you.
BTW:
Unfortunately we can't handle bit errors. To detect them, run hcxpcapngtool your_dump_file.pcapng.
hcxpcapngtool will give you an overview about this errors.
What WiFi adapter do you use? Regarding the the bit errors, I assume an ATHEROS chipset.
Cheers
Mike

Using a Tenda  W311U+ -- 148f:3070
i'll start a long run and keep an eye on it after coffee

4 hours and counting and so far all is looking good. i'll leave it running until tea time and see if anything happens

I have several Tenda W311U+. We can expect a few PLCP errors, running this devices. But that is ok.
BTW:
Please use latest git head. I fixed several other issues.

i had it running from 10:33 to 19:00 and it all seemed good. no corrupted screen so i'll close this for now as it's looking ok (so far)

Great, Thanks for your feedback.

ah it went weird again. latest commit.

btw the blank lines are because I was in the wrong window and pressed return key a few times. but it's not thise that cause it

Maybe tmux isn't configured correct:
https://hackernoon.com/customizing-tmux-b3d2a5050207
There are also several issues related to tmux:
https://github.com/tmux/tmux/issues/322
https://askubuntu.com/questions/410048/utf-8-character-not-showing-properly-in-tmux
https://superuser.com/questions/417027/why-are-random-characters-inserted-into-my-tmux-session
https://gitlab.com/gnachman/iterm2/issues/1737
https://stackoverflow.com/questions/19479810/zsh-tmux-oh-my-zsh-autocomplete-produces-remnant-characters
By the latest fix, we replace all non printable characters with a dot:
for(c = 0; c < essidlen; c++)
{
if((essid[c] < 0x20) && (essid[c] > 0x7e)) essidstring[p++] = '.';
else if(essid[c] == 0x5c)
{
essidstring[p++] = 0x5c;
essidstring[p++] = 0x5c;
}
else essidstring[p++] = essid[c];
}
Also we ignore escape sequences.
I suggest not to use tmux. Instead use hcxdumtpools build in server/client function.

Please take a look at this here:
https://superuser.com/questions/531537/tmux-visual-artifacts-strange-character-map
Looking similar to your output. So we can assume a tmux misconfiguration or a tmux issue.
Just do a google search: "tmux artifacts" and you'll get several hits.

yes that looks like it. sorry for the false issue report

No problem. Now we know that is was a good decision to add server/client function to hcxdumptool.


TOOLS-2211 Migrate from Evergreen v2 toolchain to v3 toolchain

TOOLS-2211

I accidentally did a straight merge instead of a squash + merge, so I did dirty the commit history a bit. Apologies.


[Bug] Wrong injection location for inject script on linux

Describe the Bug

After building and injecting BetterDiscord using npm run build and npm run inject Discord crashes with the following message:
A JavaScript error occurred in the main process
Uncaught Exception:
Error: Session can only be received when app is ready
    at Object.get [as defaultSession] (electron/js2c/browser_init.js:117:366)
    at Function.remove (webpack-internal:///./src/modules/csp.js:11:64)
    at eval (webpack-internal:///./src/index.js:34:60)
    at Object../src/index.js (/home/mm2pl/git/BetterDiscordApp/dist/injector.js:29:1)
    at __webpack_require__ (/home/mm2pl/git/BetterDiscordApp/dist/injector.js:142:41)
    at /home/mm2pl/git/BetterDiscordApp/dist/injector.js:194:37
    at Object.<anonymous> (/home/mm2pl/git/BetterDiscordApp/dist/injector.js:196:12)
    at Module._compile (internal/modules/cjs/loader.js:967:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1004:10)
    at Module.load (internal/modules/cjs/loader.js:815:32)

discordPath was overridden to be the true path for my system (~/Discord/resources). BD was cloned to ~/git/BetterDiscordApp.
To Reproduce

It just doesn't work, I have no idea if this bug is reproducible.
Expected Behavior

Discord actually working.
Screenshots

N/A
Discord Version

Stable
Additional Context

I'm using Ubuntu 20.04.1 LTS (Focal Fossa)

Contents of `~/git/BetterDiscordApp/dist`:
drwxrwxr-x  2 mm2pl mm2pl 4,0K maj 19 09:11 .
drwxrwxr-x 13 mm2pl mm2pl 4,0K maj 19 09:32 ..
-rw-rw-r--  1 mm2pl mm2pl  55K maj 19 09:11 injector.js
-rw-rw-r--  1 mm2pl mm2pl   45 maj 19 09:11 package.json
-rw-rw-r--  1 mm2pl mm2pl 1,1K maj 19 09:11 preload.js
-rw-rw-r--  1 mm2pl mm2pl 1,9M maj 19 09:32 renderer.js

injector.js
package.json
preload.js
renderer.js

Update: I managed to get the installer working but not `npm run ...`

The inject script is still using the incorrect injection style for linux.

Could you release some quick patch we could use in the meanwhile in a separate branch until the proper one is released which will support snap installation and other forms of discord installs?

No there is no quick patch for this, the fix is just the fix. This will not include support for snap installs however. Those are not supported by the injector, installer, or renderer at this time.

No there is no quick patch for this, the fix is just the fix. This will not include support for snap installs however. Those are not supported by the injector, installer, or renderer at this time.

Oh, I thought #828 had a fix, just incomplete because it didn't support some special ways of installations, so I figured it would just be a matter of keeping that branch and mentioning it in README, guess I misunderstood it, alright then.


fix(plugins): display correct tooltip (fixes #3342)

SUMMARY
Fixes #3342
Display the correct tooltip information for chord charts (source and target values were switched)
BEFORE

AFTER

TESTING INSTRUCTIONS
Example SQL Dataset:
SELECT 'Alice' AS source, 'Bob' AS target, 5 AS interactions UNION ALL
SELECT 'Alice', 'Charlie', 3 UNION ALL
SELECT 'Bob', 'Charlie', 2 UNION ALL
SELECT 'Charlie', 'David', 1  UNION ALL
SELECT 'Bob', 'David', 4

ADDITIONAL INFORMATION

[x] Has associated issue: #3342
[ ] Required feature flags:
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Agreed that it was wrong and that this fixes the display of the chart, but I'm so confused by the meaning of the diff. I mean, yes - swapping it swaps it, and that works. But now the code (modified by the diff) is
${nodes[d.target.index]} → ${nodes[d.source.index]} which displays the source column then → then the target column. How is this now displaying correctly but with the items named incorrectly? 🤯

Thanks for merging. When do you expect to release this change?
Will it be part of 4.1.0?
I would really appreciate it because our users were constantly confused.


Add react to rails API

What does this PR do?
This PR is raised to implement the following features: 👍 :

[x] Create rails API
[x] Create react front-end

Description of the task to be completed ↓↓

Build a rails and react app in one repo.

Thanks for reviewing my PR 😊

Thank you @IngDavidTM  for reviewing my PR👍🏽 cheers and happy coding 😎


Disable reserve node btn if user has insufficient balance

Description

Disable reserve node btn if user has insufficient balance
Added a hint clarifies that the user has to have a minimum balance to be able to reserve a node.

Related Issues

https://github.com/threefoldtech/tfgrid-sdk-ts/issues/2369

Documentation PR
For UI changes, Plaese provide the Documetation PR on info_grid
Checklist

[ ] Tests included
[x] Build pass
[ ] Documentation
[ ] Code format and docstrings
[x] Screenshots/Video attached (needed for UI changes)

the dedicated machines page will be removed so please update the node finder with this change.


Uncatchable DocumentClientException (429 "Request rate is large") in DocumentDB library for .net

When hitting the emulator rate limit the DocumentDB library throws and uncatchable DocumentClientException for http status code 429.
Message:
Microsoft.Azure.Documents.DocumentClientException: 'Message: {"Errors":["Request rate is large"]}
ActivityId: a0653757-d05e-49c5-b4ba-1811db878535, Request URI: /apps/DocDbApp/services/DocDbServer17/partitions/a4cb495d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: , SDK: Microsoft.Azure.Documents.Common/<IP_ADDRESS>'

StackTrace:
   at Microsoft.Azure.Documents.Client.ClientExtensions.<ParseResponseAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<>c__DisplayClass10.<<InvokeAsync>b__f>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass2.<<ExecuteAsync>b__0>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<InvokeAsync>d__1f.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<ProcessMessageAsync>d__2.MoveNext()

using nuget Microsoft.Azure.DocumentDB version 1.19.1
The code I'm using:
Uri CosmosAddress = new Uri("https://localhost:8081");
string CosmosAuthenticationKey = "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==";
string CosmosDatabaseName = "DATABASE_NAME";
string CosmosCollectionName = "COLLECTION_NAME";

DocumentClient client = new DocumentClient(
     CosmosAddress,
     CosmosAuthenticationKey,
     new ConnectionPolicy
     {
         EnableEndpointDiscovery = false,
         RetryOptions = new RetryOptions
         {
             MaxRetryAttemptsOnThrottledRequests = 1, // force throttle
             MaxRetryWaitTimeInSeconds = 1
         },
     }
 );

MyClass documenToInsert = new MyClass();
documenToInsert.MyField = "some very long string to force throttle";

for (int i = 0; i <= 9999; i++)
{
    Document inserted;
    Task<ResourceResponse<Document>> insertTask;
    try
    {
        insertTask = client.CreateDocumentAsync
        (
            UriFactory.CreateDocumentCollectionUri
            (
                CosmosDatabaseName,
                CosmosCollectionName
            ),
            traceItem
        );
        inserted = await insertTask;
        if (insertTask.Exception != null)
        {
            // never hits, Exception is always null
        }
    }
    catch (DocumentClientException dce)
    //when (dce.Error.Code == HttpRequestTooLargeErrorCode)
    {
        // never gets hit
    }
}

When running the code in visual studio the DocumentClientException does blow up in my face but
I can't catch the Exception to force a retry on the insertion. Using the built in retry I can decrease the odds of this happening but I can't avoid it.

@khdang please look into this.


bug: in case of no results because of the date filter: show only one error message (related to the date range)

Show the top message only

@lilimelgar @jblom
Issue for labo-components project done in branch:
issue/#335-show-only-one-error-msg

@avlisodraude @lilimelgar I fixed this in the master, I think :-P

@jblom I tested in production, unluckily I still see the two messages:

@lilimelgar ok I put this back into the backlog


Endpoints /pause, /resume and /restart not disable

I'll try to disable /pause, /resume and /restart endpoint but doesn't work.
endpoints.pause.enabled=false
endpoints.restart.enabled=false
endpoints.resume.enabled=false

pom.xml:
<parent>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-parent</artifactId>
	<version>1.3.5.RELEASE</version>
</parent>

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-dependencies</artifactId>
			<version>Brixton.SR1</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-actuator</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.cloud</groupId>
		<artifactId>spring-cloud-starter-consul-discovery</artifactId>
	</dependency>
</dependencies>

The endpoints still show in the mapping, but this is the response I get.
$ http POST :8080/pause
HTTP/1.1 404 Not Found
Content-Type: application/json;charset=UTF-8
Date: Mon, 20 Jun 2016 16:38:46 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: testeurekabrixtonclient

{
    "message": "This endpoint is disabled"
}

What happens if you try and post to /pause?

Hi @spencergibb,
Im thinking that im not explain really well.
If you disable this endpoints, is necessary load this endpoints(Objects) at memory?
It's only for improve the performance at startup.
Thank you for your time.

Right, not they could not be created. Will you open a new issue here https://github.com/spring-cloud/spring-cloud-commons/issues since that is where the code lives.


Having maraxsis active deletes constructron and constructron service station

I wanted to go build my trusty constructions only to find out they were missing
Checked spidertron research, nothing
Checked factorio ingame wiki, nothing
Then i remembered you joined the construction wiki the other day so decided to get rid of this mod to see if that was causing the issue and yup they came back after this mod was disabled.
I am guessing the submarine construction version is overwriting the default one or something just spitballing here tbh(at 6am)
But please can you have a look and restore them back.

You are correct, but there's no way I can restore them. Sorry. Construction continued only supports exactly 1 prototype with a hardcoded name. So either there is construction submarines or construction spidertrons. Not both.


Placeholder content for test-repo backend

Is your feature request related to a problem? Please describe.
I've been using the test-repo backend name value in the CMS config when running the CMS locally for developing the configuration file. However any collections referencing folders don't appear as obviously test-repo doesn't know where the content is.
Maybe I'm missing something, but the test-repo option doesn't seem to add much value.
Describe the solution you'd like
It would be nice for the CMS config to be fully fleshed out so you can test all the configurations. Each widget could have a placeholder value.
Describe alternatives you've considered
An alternative at the moment is working from deployment links using git-gateway (etc.) however it would be nice to do it all locally.
Additional context
How it looks in localhost

How it looks on a deployment link

Ideally localhost using test-repo would replicate how it looks on the deployment link, with the exception of the actual content value. These of course would need to be placeholder values.

You're ultimately looking for local development support - it won't look the way you're describing, but it should meet your general requirements. You can track it here (thumb up the issue to vote for it as a priority): https://github.com/netlify/netlify-cms/issues/2335


:host selector names are concatenated in shadow dom mode when applying mixins, resulting in errors

Description
When an element defines a mixin on :host some-selector and also applies a mixin(any mixin) to any selector, a nonsensical selector is created somewhere in Polymer's bowels. This results in the script breaking when the resulting selector is invalid. For example, when the original selector is :host.some-class [some-attribute], the resulting selector will be element-name.some-class [some-attribute]element-name, resulting in an error:

polymer.html:3637Uncaught DOMException: Failed to execute 'matches' on 'Element': 'element-name.some-class [some-attribute]element-name' is not a valid selector.

This also happens when the element does not apply a mixin but does contain another element that does apply a mixin.
Only shadow DOM is affected by this.
Live Demo
One element using and applying the mixin:
http://plnkr.co/edit/zsSAV8Kuboq8UJZpz5yM?p=preview
Element using a mixin, contained element applying another mixin:
http://plnkr.co/edit/J2Dp7tE7kbxpamd5uoBr?p=preview
Reproduce

Go to one of the above Plunkers, with Chrome;
Find errors in the console.

Expected Results
No errors, mixins are applied.
Actual Results
Errors appear in the console.
Browsers Affected

[ X ] Chrome
[ ] Firefox
[ ] Edge
[ ] Safari 9
[ ] Safari 8
[ ] IE 11

Versions

Polymer: v1.7.0
webcomponents: v0.7.23

This is probably the same problem as #3846.

The original Plunkers do not show an error on the console. Are you still able to reproduce this issue @whenlit ?

Unfortunately I don't have access to the original code anymore, but I see the error doesn't appear now. I did try the jsfiddle in the issue I linked to (#3846) with the most recent version of polymer, and that appears to work now too. It seems reasonable to assume the issue has gone away. Maybe @mgiuffrida knows more.

@whenlit Great to hear! Can confirm that I can't reproduce #3846 any longer as well. Cheers :smile:


Playing webm video, crashes the game

Godot version
3.3.4
System information
Windows 10
Issue description
Playing webm video, crashes the game
Steps to reproduce
Play a video using webm format
Minimal reproduction project
webm_video.zip

Related to #50862.
Consider converting to VP8 encoded or other format.


Remove scheduled workflows

Closes #1727

Coverage remained the same at 99.323% when pulling ad69290fded4066d08735910e0cef870ccdf698a on thetic:cronicles into a74cf2cc176fd34e67e0aee814db3ced74f2322a on cpputest:master.

This one is in conflict state now.


Add token_to_cstr

I've refactored token_to_str to use token_to_cstr internally.
token_to_str actually returns a String, which may be confusing. I'd suggest changing the function to return a &str to match expectations. This also allows the user to decide whether to allocate or not.

token_to_str actually returns a String, which may be confusing. I'd suggest changing the function to return a &str to match expectations. This also allows the user to decide whether to allocate or not.

Oop, I never noticed that. I'll fix that, thanks for pointing it out.


is it really random, code says yes, results says may not much so...

I totally got it that code has no training at all in starting example of enc-dec example around pages ch9 p227-p233 where two corners were fed into enc-dec model and final corner of output of two corner is displayed.
I see example input is starting from lower left going CCW. Final corner in the example is obviously [1,-1]. But the predicted final output is I am also seeing
tensor([[[ 0.3105, -0.5263]]], grad_fn=)
while it is not close to 1,-1 but it at least got the coordinate panes right (+, -), wondering how did it so, by pure luck??

Alright, printing full last two corners got following, 3rd corner also +, - signifying random nature. WIll be interesting to read the output once trained. Can close this issue or open a while is fine just to bring up interesting points.
['hidden_seq'] :  <class 'torch.Tensor'>
torch.Size([1, 2, 2])
tensor([[[ 0.0832, -0.0356],
[ 0.3105, -0.5263]]], grad_fn=)


Can't figure out how to have Babel applied to some of the node_modules?

I'm pulling my hair out trying to get Babel to transform some of my node_modules, which use async functions and so need to be transformed as they are packed.
I modified a copy of serverless-nodejs-starter to show what I'm trying to do. It just moves the "hello world" response generation out into a separate package which is included:
https://github.com/thenickdude/serverless-nodejs-starter
This is the package which is included:
https://github.com/thenickdude/serverless-nodejs-starter-dep
I modified my webpack configuration like so to remove the "excludes" bit and add a new "include" for the dependency that needs transformed:
  module: {
    rules: [{
      test: /\.js$/,
      loader: 'babel-loader',
      include: [__dirname, path.join(__dirname, 'node_modules', 'serverless-node-starter-dep')]
    }]
  }

Running sls package and examining the result, it's clear that the serverless-node-starter-dep package isn't being transformed by Babel. Trying to invoke it locally fails, complaining about the async keyword in that untransformed dependency.
I'm a Webpack newbie, so I think I'm missing something easy here.

Figured it out! I needed to add the package to the nodeExternals whitelist and avoid excluding it:
externals: [nodeExternals({whitelist: ["serverless-nodejs-starter-dep"]})],
  // Run babel on all .js files and skip those in node_modules
  module: {
    rules: [{
      test: /\.js$/,
      loader: 'babel-loader',
      include: __dirname,
      exclude: /node_modules\/(?!serverless-nodejs-starter-dep)/
    }]
  }

@thenickdude Yeah the whole include/exclude bit is weirdly confusing.


Better re-frame subscriptions - opinions/ideas requested

Better re-frame subscriptions
I'd like to pour some macro sugar on re-frame.core/register-sub. I'd like your feedback and ideas.
Naming
First question: what should we call this macro?   defsub, def-sub, defs?  Other?  Below, I assume defsub.
The Goals
Solve these current problems:

The existing method requires a bit too much knowledge about reaction and, in more complex cases, chaining reactions, and how one subscription can depend on the output of another subscription, etc.  All doable, and knowable, of course, but I'd love to make it simpler.
Currently, subscription handlers are not pure functions. They take a ratom app-db as a parameter, not a value. They also return a reaction rather than a value.  There's a technique to create pure subscription functions but it involves a bit too much boilerplate. I'd like pure functions without the boilerplate (defsub should look after the boilerplate for me)

Improve by adding:
3.  Subscription de-duplication has been added to re-frame (yet to be released) and I'd like to encourage the creation of "signal graphs" where one subscription uses anothers as input. For a start, it feels a very natural thing to do, and second there are efficiency gains to be had. de-duplication means that root branches of the signal graph can quickly prune value propagation when there's been no changes.
4.  Each subscription should optionally provide a cljs.spec for its output, and this should be checked. Maybe, this checking will be be disabled in production.
5.  To assist in debugging:
- provide a mechanism to trace subscriptions activity to js/console.  Trace when they are created, and rerun.
- provide a mechanism to measure subscriptions.  How many times has a subscription been rerun?  Accumulated time.
Other goals:
6. needs to be backwards compatible. Ie. re-frame.core/register-sub should still just work.
7. Macros are DSLs which bring their own cognitive overhead. Ideally the macro's design should use concepts, names etc already familiar to a clojure programmer.  A proficient programmer should almost be able to guess what's going on.
8. Being a lazy sod, I look at these things through the prism of "how easy would this be to teach".  How may pages of docs will I have to write :-) ?  And how many slack chats will there be explaining to people what they did wrong. Oh, yeah, and of course, how powerful.
And, what's going on is:
A. There's a pure function
B. it performs computation using the values from:
- one or more "input signals"
- data from a query vector
- zero or more dynamic query parameters
Question: are there other considerations/goals I've missed?
Backgrounder
This ticket is about subscription definition, but let's remind ourselves about use:
   (subscribe [:query-id 1 2 3] [r1 r2])

The 1st  parameter is the query vector, which starts with the query-id (generally a namespaced keyword).  The 2nd parameter is a vector of input signals (ratoms/reactions).  When they change, the subscription should be rerun.
Most subscriptions only involve one parameter.  Many re-framers don';t even know about the 2nd.
What defsub use might look like
A super simple example:
(defsub :items
  (fn [db v]
    (:items db))

Notice:

The first parameter to the macro is the query id (:items), just as it would be with re-frame.core/register-sub. So no change there.
The second parameter is a handler fn of the kind normally given to re-frame.core/register-sub except:
- db is a value, not a reagent.core/atom
- it returns a value and not a reagent.ratom/reaction
- so a pure function

Point 2 is 50% of the battle.
Question:  does this variation work better for you?
(defsub :items
 :fn [db _]          ;; using an `:fn` keyword
     (:items db))

Or maybe pare it right back to this:
(defsub :items
  [db _]            ;; no mention of fn
  (:items db))

This approach is lovely for simple cases. But, it could be problematic as we get into more complex cases. Read on.
Return Spec Checking
defsub can also take a registered cljs.spec which describes the structure of the value returned by the handler fn.
(defsub :items
 :ret-spec  :info-model/items
 (fn [db _]
     (:items db)))

Signal Graphs
re-frame automatically de-duplicates subscriptions.  As a result, an architecture in which subscriptions are built using inputs from other subscriptions will result in an efficient signal graph.
So we want to support that process.
Here's a complex registration from the existing re-frame README:
(re-frame.core/register-sub
 :sorted-items             ;; the query id
 (fn [db [_]]
   (let [items      (reaction (get-in @db [:some :path :to :items]))  ;; reaction #1
         sort-attr  (reaction (get-in @db [:sort-by]))]                ;; reaction #2
       (reaction (sort-by @sort-attr @items)))))                       ;; reaction #3

This formulation has problems. Firstly, it requires a bit of knowledge arranging all those reactions correctly.  Hard for a newbie. And easy enough for someone experienced to get wrong.  Secondly, this formulation offers no chance for de-duplication of the input reactions. If the same reactions are used elsewhere they will run as duplicates.
So, current best practice is to rewrite it like this:
(re-frame.core/register-sub
  :sorted-items
  (fn [db [_]]
    (let [items      (subscribe [:items])       ;; was a reaction, now a subscription
          sort-attr  (subscribe [:sort-by]))]     ;; was a reaction, now a subscription
      (reaction (sort-by @sort-attr @items)))))    ;; reaction #3 (hasn't changed)

Better!! But still a bit hard for a newbie to get right.  For example, someone can easily wrap the entire let in a reaction which is wrong.
And, even if that wasn't a problem, we don't want reaction used explicitly any more.
So, here's how it might look with defsub:
(defsub :sorted-items
 :ret-spec  :info-model/items
 :subscribe [:items]       ;; assumes we've already created this subscription
 :subscribe [:sort-by]     ;; ditto
 (fn [items sort-attr query-v]
     (sort-by sort-attr items)))

Note:

input signals are defined via zero or more :subscribe clauses. In the absence of a :subscribe a default will be provided :subscribe [:/]  which is a pre-defined subscription which gives you the root of app-db
The values from these N subscriptions are provided to the fn positionally as the first N parameters.
The fn is still pure. Values in, values out. No side effects.

Or perhaps this variation is better:
(defsub :sorted-items
 :ret-spec  :info-model/items
 :subscribe [:items]
 :subscribe [:sort-by]
 :fn [[items sort-attr] query-v]   ;; using :fn and a 1st parameter vector
     (sort-by sort-attr items)))

OR maybe we do awayaway with the fn formulation (args etc):
(defsub  :sorted-items
 :ret-spec  :info-model/items
 :subscribe items [:items]        ;; :subscribe, symbol and then query vector
 :subscribe sort-attr [:sort-by]
 :query [_ a b]                     ;; destructure the query vector, if those values are used
 (sort-by sort-attr items))        ;; don't even bother with whole `fn` args thing, just provide the fn body.

but this latest one makes the simple case a bit hard:
(defsub :items
  (:items db))        ;;  without fn "args" how to I name "db"

maybe I have to do this:
(defsub :items
  :subscribe db [:/]
  (:items db))        ;;  now db is explicitly named

Hmm.  I want my DSL to be immediately familiar to a clojure programmer. I'm not sure that loosing the fn part is good. We are writing a function. It is just that we want it to be reactively re-run.
OR maybe we prefer
(defsub :sorted-items
 :ret-spec  :info-model/items
 :fn   sorted-items
 :args  items      :from  :subscribe [:items]
        sort-attr  :from  :subscribe [:sort-by]
        [_ a b c]  :from  :query-v
 :body (sort-by sort-attr items))

So which of these appeals?  Other alternatives?
The Generated code
Let's switch gears and, for the moment, forget about DSL aesthetics.
What code will this macro generate?  It will still need to call  re-frame.core/register-sub.
;; create the pure function
(defn sorted-items             ;; named from the keyword?  Can name be programmer supplied?
  [items sort-attr query-v]    ;; if needed query-v might be [_ a b c]
  (sort-by sort-attr items)))

;; register the event with the pure function
(re-frame.core/register-sub
  :sorted-items
  (fn [app-db query-v dyn-v]
    ;; some checks
    (assert (vector? query-v)
    (assert (or (nil? dyn-v) (vector? dyn-v))

    ;; some tracing
    (when ^boolean goog.DEBUG (debug "re-frame: creating subscription: "  query-v))

    ;; gather the input signals
    (let [items     (subscript [:items])
          sort-attr (subscribe [:sort-by])]

      ;; now make the rerunnable reaction
      (make-reaction  (fn []
                        (when ^boolean goog.DEBUG (debug "re-frame: rerun subscription: " query-v))
                        (sorted-items @items @sorted-attr query-v (mapv deref dyn-v)))))    ;; perhaps also some monitoring here

Further Thoughts
Provide a spec for the query-v ?
Perhaps just fspec the handler?
Remember a subscription can use a value in the query-v
Remember a subscription can use the value from a dynamic parameter   :subscription [:some-q 42] [r]

About the fn discussion. I think it'd be better if we could pass a function to defsub, so we could do stuff like
(defsub
  :subscribe [:anything]
  :query (comp fn-1 fn-2 fn-3))

Which is better than
(defsub
  :subscribe [:anything]
  :query [x] ((comp fn-1 fn-2 fn-3) x))

Removing fn semantics from defsub can ruin composability (and with no relevant gains, from what i saw)

Personally I like that we follow a classic macro approach, keeping (fn [....) as returned value of subscribe, but...
...another thing I noticed from the above description (thanks Mike, as usual awesome writing) is that a data-only pattern is emerging, where you don't need macros and you define params in a map.
I am talking about this in particular:
defsub :sorted-items
 :ret-spec  :info-model/items
 :fn   sorted-items
 :args  items      :from  :subscribe [:items]
        sort-attr  :from  :subscribe [:sort-by]
        [_ a b c]  :from  :query-v
 :body (sort-by sort-attr items))

This is an accepted trade in the Clojure community as well (see Onyx), greatly improve composability and allows user to define their own dsl.
However, I am ignoring backward compatibility here and maybe going a bit off topic :)

I would prefer something like this
(defsub :sorted-items
 :ret-spec  :info-model/items
 [db query-v]
 [items [:items]
  sort-by [:sort-by]]
 (sort-by sort-attr items))

where
[items [:items]
 sort-by [:sort-by]]

is like "extracted" from let form

Hey @mike-thompson-day8 ! Thanks again for your work.
My 2c in the order that you listed items.

(defsub) :+1:
No mention of (fn) or :fn: IMO. Cleaner and doesn't give people the implication that they can do fancy function things ((partial) etc)... unless you want them to be able to do those things?
:+1: absolutely yes spec support in the (defsub) macro!
4: RE composing multiple subs; the first example imo. I do not like :/ and don't think it'd be familiar to clojure devs
5: RE further thoughts: yes spec, spec, spec support all the things!

In my humble opinon, I am not on-board with defsub. As an alternative I propose having pure functions with one arg per subscription and a function that adds metadata to the functions var (and possibly a macro to add sugar to the function). Then subscribe takes a var and looks up metadata.
On the plus side this is very unmagical, but it may not be backwards compatible.

Simplifying subscriptions is a good idea.
In my opinion, defsub should use a pure function (defined inline or otherwise) to do the extraction logic. One reason is that the extraction functions can be defined in cljc, and introspected, tested, and extended via clojure.  This also allows mocking http endpoints and testing the behavior of the frontend system from the jvm.
I'd vote:
(defsub :items
 :ret-spec  :info-model/items
 (fn [db _]
     (:items db)))

hmm... defsub is looking a bit like applicative-do aka alet syntax for a reaction-applicative http://funcool.github.io/cats/latest/#alet
(defsub :sorted-items
  :ret-spec :info-model/items
  [db [_ a b c]]
  (alet [items (subscribe [:items])
         sort-attr (subscribe [:sort-by])]
    (sort-by sort-attr items)))

i need to think on this some more

Good point @mccraigmccraig, it also reminds me in some ways of manifold's let-flow macro.

right @danielcompton let-flow is very similar to alet for Deferreds

one difficulty with alet would be (looking at cats' implementation the reaction functions would likely be anonymous (with an applicative instance which i'm guessing (really) wildly would look something like this ) which wouldn't be great for tracing... though maybe useful names for tracing could be passed as metadata on the reactions themselves (which would seem to require Reaction to implement IWithMeta too)

Using the current regsub proposal to implement todomvc subscriptions:
https://gist.github.com/mike-thompson-day8/077e5bd41ff6cd167f96170604380026
Compare to current:
https://github.com/Day8/re-frame/blob/4b931fe67208c10ae960c683fd837b03c38f88b0/examples/todomvc/src/todomvc/subs.cljs

Suggestion from @escherize via slack:
(regsub :footer-stats
  :inputs {:todos  (subscribe [:todos])
           :completed  (subscribe [:completed-count])}
  (fn [{:keys [todos completed-count]}  _]
    [(- (count todos) completed) completed]))

@mike-thompson-day8, :<- looks a bit weird, and I don't understand how this syntax handles Remember a subscription can use a value in the query-v. Actually, any syntax where input subscriptions are defined before query descructuring doesn't seem too clear for me...

After some reflection, I think that this version is my favorite fwiw.
(regsub :footer-stats
  :<- {:todos     (subscribe [:todos])
       :completed (subscribe [:completed-count])}
  (fn [{:keys [todos completed]} _]
    [(- (count todos) completed) completed]))

It looks like clojure, and the idea that todos and completed are inputs is very clear.
As long as the query-v for the subscriptions right after :<- are static, it works just fine there.
If dynamic subscriptions are nessicary... then maybe this makes sense? (not sure..)

(regsub :footer-stats
  :<- {:todos     (subscribe [:todos "apple"])
       :completed (subscribe [:completed-count @(subscribe [:order-by])])}
  (fn [{:keys [todos completed]} _]
    [(- (count todos) completed) completed]))

I'm not sure if I've addressed your objections though, @rasom.

@escherize, for example
(register-sub :something
  (fn [db [_ param]]
    (let [something-other (subscribe [:something-other param])]
      (reaction (some-magic (get-in @db [:something param]) @something-other)))))

I mean... Where we will get that param from when we will define input subscriptions inside :<-?

Okay, after a week of on-again off-again struggle,  a good solution turns out the be embarrassingly easy. How did I not see this before?
https://gist.github.com/mike-thompson-day8/9481b281584659d066cacc25a955ac62
So, I'm still not 100% sure about the whole :<- sugar thing (I've left it in for further comment), but that has now taken 2nd place behind a much, much more solid concept.

Using a function to setup the input signals to the subscription looks good, @mike-thompson-day8.
I was chatting with @gadfly361, and he had an interesting idea for defsub. Basically, his approach is to use the same approach as reagent.core/create-class. Meaning use a map to label the inputs, subscription function, and whatever else is important. There are a few good reasons to use this: it's semantically intuitive, easier to add to and augment later, and doesn't rely on ordering artifacts that actually have names.  It boils down to explicit is better than implicit.

@mike-thompson-day8  Thank you for bringing this up for discussion! The proposed changes
(https://gist.github.com/mike-thompson-day8/9481b281584659d066cacc25a955ac62) look great!
e.g.
(regsub :footer-stats
   :<- [:todos]
   :<- [:completed-count]
   (fn [[todos completed] _]
     [(- (count todos) completed) completed]))

I think the macro syntax does in fact facilitate seeing subscriptions more as derivations of other nodes/signals in a graph, rather than mostly (reactive) queries on the db. I think this concept is clear in the ‘first function’ approach a well:
(regsub :footer-stats
  (fn [query-v _]
    [(subscribe [:todos])
     (subscribe [:completed-count])])
  (fn [[todos completed] _]
    [(- (count todos) completed) completed]))

But it’s more obscured in the syntax here. Specifically the relation of the returned vector of signals in the first function, to the destructured first arg in the second fn is more difficult to recognize and easier to get wrong. The user also needs to remember that query-v is the first arg in the first fn and second arg in the second fn – not intuitive. I think in comparison the macro syntax will be significantly easier to learn and to remember and will be easier to read and to edit/maintain.
As we are already employing a macro to make subscriptions more user friendly, I was thinking if we should consider to even go a few steps further:
Say I want to subscribe to footer-stats passing a param (which will slightly exaggerate my stats):
(subscribe [:footer-stats 1.2])

(regsub :footer-stats
  :<- [:todos]
  :<- [:completed-count]
  (fn [[todos completed] [_ param]]
    [(- (count todos) completed) (* completed param)]))

Problems:

I have to remember how I receive the param: [_ param] in the second fn arg. I'm often mixing this up with event handlers which allow using the trim middleware to get rid of the mostly obsolete first param.
I have learned this syntax: (subscribe [:todos]), yet when I saw :<- [:todos] was wondering
why the subscription ID is written in a vector here. It would be easier to guess/recognize that this is the
subscription vector if there were params in the vector. Still I'd say in 60% of the cases subscriptions don't have params, and in these cases the vector in the macro syntax looks a bit confusing/redundant.
Related to the subscription vector and the destructuring we see 10 angle brackets in 3 lines (lines 2 to 4), still accumulating some visual noise.
With this syntax it doesn't seem plausible to 'pass on' subscription parameters to
'sub subscriptions', e.g. do :<- [:todos param] with the param passed to :footer-stats.

Here is a slightly modified syntax that attempts to address these issues:
(regsub :footer-stats [param]
  :< :todos
  :< :completed-count
  (fn [todos completed param]
    [(- (count todos) completed) (* completed param)]))

Arguments to the suscription are listed in an argument vector, right after the suscriptions ID, similar to clojure function definitions
The arguments to the computation function are [sub-1, .. sub-n, param-1, .. param-n]. No destructuring.

This would also allow to 'pass on' subscription arguments to the 'sub subscriptions' like this:
(regsub :visible-todos [limit]
  :< [:todos limit]
  :< :showing
  (fn [todos showing _]
    (let [filter-fn (case showing
                      :done   :done
                      :all    identity)]
      (filter filter-fn todos))))

Instead of writing
(regsub :visible-todos [limit]
  :< :todos
  :< :showing
  (fn [todos showing limit]
    (let [filter-fn (case showing
                      :done   :done
                      :all    identity)]
      (->> (filter filter-fn todos)
           (take limit)))))

which would (theoretically) be less efficient.
By omitting the vector around the sub-subscription ids, omitting the additional dash and omitting the destructuring, I believe that the syntax gets easier to scan/read and comprehend:
(regsub :visible-todos [limit]
  :< :todos
  :< :showing
  (fn [todos showing limit]
    ..))

vs. the original proposal:
(regsub :visible-todos
  :<- [:todos]
  :<- [:showing]
  (fn [[todos showing] [_ limit]]
    ..))

Interestingly mixing the vector form (when params are passed) with
the non vector forms appears intuitive to me (interested of cause what others think!):
  :< [:todos limit]
  :< :showing

Small thing: I did not recognize this :<- as an arrow pointing left at first, rather saw
it as the frightened monsters from pacman. :< also indicates that something is pulled in and then passed into the function, while it's saving one character/a bit of noise. (can still be seen as a sad monster however)
Lastly, I'd suggest to go all-in with the new regsub syntax and make the input signal declaration mandatory:
(regsub :showing []
  :< :db
  (fn [db]
    (:showing db)))

I think especially beginners will appreciate the consistency in how regsub appears in code: The input signals are always defined and are always matching the first args in the function signature. (the :db signal is available by default.)
In the current abbreviated (one function) form, the user has to know that in this case db is provided in the first arg:
(regsub :showing []
  (fn [db]
    (:showing db)))

It's visually quite different/inconsistent from the explicit signal listing form and may therefore be a slightly confusing.

@andreasthoelke  thanks for that very detailed write up.  I've read through it a few times fairly carefully.
I have already wrestled with virtually all the proposals you make.  For example, I've been backwards and forwards a few times over how to present signals to the computation function. You suggest positional arguments, rather than a data structure (vector?) as the first parameter.
But I've gone with a non positional approach because some may prefer to do this:
(regsub :footer-stats
  (fn [query-v _]
    {:todos (subscribe [:todos])
     :completed (subscribe [:completed-count])})
  (fn [{:keys [todos completed]} _]
    [(- (count todos) completed) completed]))

I've also agonized over the whole destructing of query params in a way that allows their access in subscriptions :< [:todos limit].  But every "solution" seemed to added lots of if then else to explanations about what was happening, which is never good.
I must admit, I'm pretty happy now with the current approach.
In fact, I'm in half a mind to not even allow for the sugar.  There's something very consistent and right about the signal fn.  The extra sugar, if it is to be allowed, will just be made available for the simply cases, which you estimate is 60% of the time.
I do like stuff minimal, so I'm thinking seriously about switching to :<

The extra sugar, if it is to be allowed, will just be made available for the simple cases, which you estimate is 60% of the time.

Looking through our codebase, I'd estimate that 99.9%* of our subscriptions don't use subscription parameters, so the syntactic sugar would be able to be used almost everywhere (I think?).

not really an estimate

I wonder about the need for multiple :< or :<- keywords. Why not just a vector-of-vectors:
(regsub :visible-todos
  :<- [[:todos] [:showing]]
  (fn [[todos showing] [_ limit]]
    ..))

This code is now in develop.  The todomvc example has been changed across to use regsub.

@mike-thompson-day8 I am thinking of putting the new reg-sub in a project, maybe we can cut an alpha release?
I saw the todo app and it looks very neat (there is typo, the sub fn is called register-pure-sub instead of reg-sub` in the comments), so I might give it a go. The code base is very little at the moment so it is the right time to do it.

Sorry, yes, I've been progressing slowly on this.  Can you give me two more days please and I'll cut an alpha?   In the meantime, you can of course,  grab develop and lein install it into your local maven,  if you absolutely have to have it sooner.

Here is a newbie prospective who has been with clojurescript and re-frame for just a couple of days.
I am continuously tempted to abandon re-frame patterns for two simple reasons. First is the annoying repetition that (register-sub ...) and (subscribe ...) bring along. Most of my subs are basic extractions like this
(register-sub :user-ids
  (fn [db]
    (reaction (get @db :user-ids []))))

The simple alternative of  using (reaction (:user-ids @db)) directly in views.clj is magnitudes simpler and very tempting indeed. Second reason is that the whole paradigm uses non-standard dispatch mechanisms - (subscribe [:id]) and (dispatch [:id ...]) instead of straightforward function calls.
So here is the proposal. Why not make defsub (and defhandler) look and behave like standard function definitions that would create both the actual subscription and the subcription invocation? Something along the following lines:
(defsub sorted-items []
   (sort-by @(sort-attr) @(items))))

The above would register the subscription function like this
(re-frame.core/register-sub
  :sorted-items
  (fn [db [_]]
    (let [items     (subscribe [:items])       ;; was a reaction, now a subscription
          sort-attr  (subscribe [:sort-attr]))]     ;; was a reaction, now a subscription
      (reaction (sort-by @sort-attr @items)))))    ;; reaction #3 (hasn't changed)

It also would define the actual subscription invocation function sorted-items
(defn sorted-items []
   (subscribe [:sorted-items]))

So no more awkward (subscribe [:sorted-items]) but direct call to (sorted-items) that ordinary people could understand. You can call it in views.clj or in other defsubs just as you would do with normal functions.
The above code assumes that defsub can identify which calls are subscription invocations (@(sort-attr) and @(items)) and which are standard functions. That could be done either by attaching metadata to the subscription invocation function (sorted-items) or looking into the database of already registered subscriptions. This also would address the redundancy issue. The only issue is that you would need to define subscription in order of their usage, just as with normal clojure functions.
In any case, it looks to me that, explicit declaration of subscriptions in defsub as in all proposed DSLs could be entirely avoided or at least be made optional.

@vspinu Both dispatch and subscribe are more than function calls, each in a different way.  For a start dispatch implements an asynchronous function call, with middleware.  That's critical.  Second subscribe implements a reactive function call, with deduplication (coming in v0.8.0).
If you application is simple, then sure abandon re-frame. If your application is a bit more complex, you'll be glad you didn't.

Both dispatch and subscribe are more than function calls, each in a different way.

Sure, I was not suggesting that. Both are not pure functions but I would still prefer to pass #(store-user-ids %) instead of #(dispatch [:store-user-ids %]), and get reactions with (user-ids param) instead of (subscribe [:user-ids param]). And if this can lead to getting rid of the explicit declaration of the signals in defsub and defhandler DSLs, even better.

@arichiardi I've cut a version 0.8.0-alpha1.  Will be in clojars shortly.
I'm cautious because this really is alpha quality. But you did ask.
There are breaking changes. See:
https://github.com/Day8/re-frame/blob/master/CHANGES.md#080--201607xxxx

That's awesome, I think I can handle he changes no problem, the new features are super cool ;)

Not related with re-frame but b I just saw what puts me off for now. Maybe it is not relevant for an alpha, but it might be good to track: http://dev.clojure.org/jira/browse/CLJS-1701

@arichiardi  CLJS-1701  shouldn't be an issue for re-frame. I've held off involving spec.  .

How sure are we that query de-dup is going to be a thing in re-frame's future?
I know API breakage is on the cards, as are bugs (given the alpha), but is it confirmed that the de-duplication concept is here to stay?
I ask because right now I'm struggling with incorporating de-duplication as mentioned in Subscribing-To-A-Database . Lots of using counter atoms to prevent firing off multiple remote queries just because a query reaction is de-refd multiple times, etc.

@rtomsick, it's very unlikely query dedup is going away, barring major issues. I wasn't clear from your post, are you having issues with re-frames query dedup in the alpha, or implementing your own dedup?

@vspinu not necessarily a permanent solution, but for repetitious simple db gets, I created a single :raw-query subscription that takes a vector param which is passed to (get-in) on the app-db.  This has been great for quick development.  Searching the codebase for the paths allows you to identify any re-used :raw-query's that deserve being converted into explicit subscriptions. You could make it more 'secure' by applying a filter to the paths in the :raw-query subscriptions itself to restrict access to parts of the db.

but for repetitious simple db gets, I created a single :raw-query subscription that takes a vector param which is passed to (get-in) on the app-db.

Yes. I did exactly that. It might seem natural, but it wasn't so obvious to me when I started with re-frame. The following lines essentially removed  80% of my sub and dispatch querying code:

(re-frame.core/register-sub :db-get-in
  (fn [db [_ & ks]]
    (reagent.ratom/reaction (get-in @db ks))))

(defn db-sub [ks]
  (re-frame.core/subscribe (into [:db-get-in] ks)))

(re-frame.core/register-handler :db-assoc-in
  (fn [db [_ ks val]]
    (assoc-in db ks val)))

(defn db-assoc-in [ks val]
  (re-frame.core/dispatch [:db-assoc-in ks val]))

Those handlers and subscriptions will reduce the number of subscriptions and handlers you have to write, at the cost of tightly coupling your application to the structure of app-db. The idea of subscriptions and handlers is to separate the what from the where. If you need to change the structure of app-db for some reason, then you will need to go through all of your view code, subscriptions, and handlers to makes sure you've updated those places that depend on it.
If you make more specific handlers and subscriptions, then you can refactor app-db without a lot of code churn. A middle ground we've sometime used is to create handlers to update parts of an app-db, e.g. :messages-update which has an assoc-in starting from the :messages key. That way, you have some degree of isolation.

What does the subscription in https://github.com/Day8/re-frame/wiki/Subscribing-To-A-Database look like with reg-sub? I've been playing with that for an hour or so now and haven't worked out the right magics.

@newhook I haven't had time to think that through.  So I'd just continue to use the same pattern described in that Wiki page, via reg-sub-raw

@danielcompton   I'm not having any issues with the alpha, but I'm trying to figure out how long-lived my own de-dup code will be. :smile: If it looks like de-dup is here to stay then I won't worry about implementation cleanliness (since I'll probably see a new release of re-frame before I ship).
Thanks!

STATUS:
We're getting close to a release of v0.8.0.
The various alpha releases (in clojars) fully implement the new reg-sub. We've been using the new subscriptions feature in production, and it is working out nicely, so I can see no further changes likely.  It is almost to the point I could close this issue - i think we are there.
But I won't because good docs are still pending. Currently the best explanation is:
https://github.com/Day8/re-frame/blob/develop/examples/todomvc/src/todomvc/subs.cljs

Released as part of v0.8.0. Although docs are still lacking.
At this point, todomvc tutotial continues to be the best soruce of information:
https://github.com/Day8/re-frame/blob/develop/examples/todomvc/src/todomvc/subs.cljs


ARROW-1893: [Python] Convert memoryview to bytes when loading from pickle in Python 2.7

It seems somewhere in the 2.7.x series, Python 2.7 acquired the ability to load from memoryview. To be on the safe side, we'll always convert memoryview to bytes. Here's a related workaround from IPython:
https://github.com/ipython/ipython_genutils/blob/master/ipython_genutils/py3compat.py#L153

+1


Define scope and objectives for document text extraction

e.g.

section and title extraction
paragraph blocks (incl. different number of columns), extracted in reading order
excluding text that's not part of the reading order e.g. headers, footers, sidebars, figure captions
contents pages, appendices, indexes
tables
figures
anything else in the PDF layout specification

Also look at datasets. Could we produce any datasets where we have both the PDF document and HTML (or some markup) representation: maybe eurlex or legislation.gov.uk.
Write a Notion page which summarises this.

I have put together a notion page describing the high level scope, objectives and requirements for the research and development of an improved approach to text extraction from policy documents. This will be shared with the team for feedback.


Make sure that tedious Request calls callback function on errors.

If a query fails because of a connection error, it only emits en error event for streams. For callbacks, it simply ignores the error, so the callback is never called.
This change makes sure that the callback is called with a proper error object, as would be expected.
Also, the errorHandler on connections is only set up to listen for the "errorMessage" event. But in some cases, Tedious emits an "error" event instead. So this change also sets up a listener for "error" on connections.
Lastly, an error handler is set up on the Tedious Request itself.
This change has been developed with Tedious 1.13.1 in mind, so pull request #230 should be applied, too.

Found a few bugs in my fix. Closing this PR and creating a new.


Install fails - Password authentication failed for user "lemmy" connecting to postgres://lemmy[etc]

Describe the bug

What happened? Post any relevant log snippets.

 Or an update to an existing one?
I am getting all kinds of problems on 0.18 on my own server, and now that 0.18 is out, the official Lemmy issue tracker is getting dogpiled with issues. They really shouldn't have rushed this release.
If this was a brand new deployment, can you try ./deploy.sh -u 0.17.4 -f and seeing if that works?
If you have the same issue on 0.17.4, I can try diagnosing further.

Also, I can see from your redacted strings that nvNHu and pvkKEoZ4 do not match.
Is the password in postgres.env different from the one in lemmy.hjson? That shouldn't happen (I'm actually not sure how it could happen), but it could be the cause.

Thanks I'll try rolling back to the 0.17
I had a previous installation on there that I had installed through docker-compose (what a pain that was). I've since wiped it and was attempting to install through the script for .18.
The passwords do match for postgres.env and lemmy.hjson
There's nothing important on this particular server, so I'll wipe the server and try again. Thank you for the help, I'll update if it doesn't work after the wipe.

Ah, I think I know what happened then. And I may need to find a clever way to warn users about it.
The variable may be called POSTGRES_PASSWORD, but the postgres container only uses it to initialize the database. And since deleting the ./live folder doesn't delete the volumes made by Docker Compose, you likely had an old postgres volume that had the old password set in it. If you are REALLY SUPER SURE you want to delete your old data, you can clear your old volume like this:
docker volume rm lemmy-easy-deploy_postgres_data

You can also list other volumes used by this project and optionally delete them just like above:
docker volume ls

I am like 99% sure that's what happened, so I will close this, but please feel free to come back and ping me if I need to look into the issue further!


Generate RPM for cx-papyrus that can be installed with no network connectivity

Issue and tracking information
Developer's time Estimated effort to fix (hours):
4
Developer's Actual time spent on fix (hours)
4
Issue reporter to provide a detailed description of the issue in the space below
I did a little investigation into packaging CX as an RPM now that you’re using Papyrus instead of RSA. I’ve collected some notes/action items below that I believe need to be addressed before having an RPM that works:

The spec file will need its version updated to 2.0.0 for Papyrus-based CX:  https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L8
All /opt/IBM/SDP calls in https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L48-L114 will need to be changed to /opt/Papyrus
Two of your installation units seem to have changed names: https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L92-L93
a.	com.zeligsoft.base.feature.group --> com.zeligsoft.base_feature.feature.group
b.	com.zeligsoft.cx.feature.group --> com.zeligsoft.cx_feature.feature.group
And lastly, the installIUs seem to be requiring dependencies that are not provided by Papyrus (downloaded from https://www.eclipse.org/downloads/download.php?file=/modeling/mdt/papyrus/rcp/2019-06/4.4.0/papyrus-2019-06-4.4.0-linux64.tar.gz) automatically, meaning that they attempt to reach out to the network to get them (cx_install_error.log). Previous CX installations could be installed with no network connectivity at time of install. This needs to be the case in order to install this type of software via RPM.

Sorry about the closing and opening of this ticket! I had accidentally clicked submit before giving a description so I closed the ticket then realized I could edit my original (empty) description, so I reopened it after editing.

@J17359 : Commit coming that definitely fixes the RPM version, papyrus location and feature references. I've updated the bundled dependencies so that they install successfully in the following scenario:

'Eclipse Modeling Tools, 2019-06 as the base installation
Papyrus for UML, 2019-05 (4.4.0) installed into that (from the 2019-06 update site
Install CX from there. (Check with no internet connection.)

@elder4p you need to update the executable name in /opt/Papyrus to "papyrus" instead of "eclipse":
https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L53
https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L68
https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L87
https://github.com/ZeligsoftDev/CX4CBDDS/blob/papyrus/releng/com.zeligsoft.dds4ccm.update.axcioma/rpm_build/zeligsoftCX_axcioma.spec#L100

cxInstallError.log
"Hacked" my way through the install (symlinked /opt/Papyrus/eclipse to /opt/Papyrus/papyrus) and found something else. See the above error log. Looks like your RPM install needs to also contain the xtend dependencies included in your zip?

It might be helpful to have your automated tests run installs of the RPMs generated by the rpmbuild job. That way we can hopefully catch some of these spec file issues earlier.
FYI, our Eclipse environment looks like the following:
Papyrus from the link you sent back in January: https://www.eclipse.org/downloads/download.php?file=/modeling/mdt/papyrus/rcp/2019-06/4.4.0/papyrus-2019-06-4.4.0-linux64.tar.gz
Is that Papyrus tarball equivalent to what you mentioned below?

'Eclipse Modeling Tools, 2019-06 as the base installation
Papyrus for UML, 2019-05 (4.4.0) installed into that (from the 2019-06 update site
Install CX from there. (Check with no internet connection.)

@J17359 : Thanks. I will correct this on Monday. For the record:

missing bundle is: org.eclipse.xtext.xbase

I will test with Papyrus RCP

@J17359 : merged a fix that:

includes org.eclipse.xtext.xbase feature in the update sites.
refers to the install papyrus as /opt/Papyrus/papyrus

I have tested, but my Linux test machine is not a native RPM platform, so I distrust it. Can you confirm this works for you.

I have tested, but my Linux test machine is not a native RPM platform, so I distrust it. Can you confirm this works for you.

Building your software now. Should have feedback in an hour.

That looks to have fixed it! I installed Papyrus RCP (2019-06) from tarball I linked here. Then I built and installed your CX RPM successfully.
I have just one small request left (and you may decide this is something we need to configure on our end). The axcioma spec file  today only installs the five Zeligsoft feature groups. Papyrus RCP doesn't come with any Eclipse CDT support so we've been installing the full CDT feature from your p2 site. Your Zeligsoft plugins all install successfully pulling in some of the CDT plugins already (org.eclipse.cdt.<codan.core, core, core.native, make.core, and managedbuilder_core>). Our plugins we build against your software requires a bit more from CDT and by installing the full CDT suite from your p2 site (manually), we can build our software successfully.
Would you be willing to install all of the CDT installation units along with your Zeligsoft installation units via the spec file/RPM? Maybe there could be a blurb in the %description field of the spec file stating that the RPM should only be installed into Papyrus RCP. If there are use cases in the future installing into different Eclipse environments, we could work together to set that up with spec files as well.

@J17359 : installing the CDT as part of the RPM seems reasonable to me. We produce it to meet your needs anyhow. I'll try to get to it tomorrow.

Pull request #78 resolves the last remaining issue. Closing.


Restarting File Sync Agent

Working with a customer who received error 0x80C83060, we found that restarting the file sync agent restored the communication between their server endpoint and Azure. It was not one of the troubleshooting steps listed for this error, and it may be that could be added to more errors if helpful as well.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7235bd07-1172-ac6a-3dab-0200fd73bd87
Version Independent ID: 9620b286-fe9e-0c56-c01f-ddf7c3c36ac0
Content: Troubleshoot Azure File Sync
Content Source: articles/storage/file-sync/file-sync-troubleshoot.md
Service: storage
Sub-service: files
GitHub Login: @jeffpatt24
Microsoft Alias: jeffpatt

Thanks for reporting this. I have assigned the issue to the content author to investigate further and update the document as appropriate.

I added the following note in the Troubleshooting guide for error code 0x80C83060:
Once network connectivity to the Azure File Sync service is restored, sync may not resume immediately. By default, Azure File Sync will initiate a sync session every 30 minutes if no changes are detected within the server endpoint location. To force a sync session, restart the Storage Sync Agent (FileSyncSvc) service or make a change to a file or directory within the server endpoint location._

#please-close


No license file

Hello,
There should be a global license file to make the licensing situation more clear. There is a note but not on all files.
This may make contacting with contributors necessary in case They modified some files without the headers.

@scrooloose
:+1:  It would be very helpful if you add a license file, whatever kind of license. If not specified, by default the code is "All Rights Reserved" as explained here


Refine Web UI Homepage (normal user and admin) + fix no available node

Refine homepage UI

home page - normal user
Update the margin and padding
Add My favorite job

home page - admin user
Update the margin and padding
Add My recent job

UI of no available GPU nodes
Show a sentence: No available GPU nodes now at the middle of the chart

Update the color of Utilization
Usage color: 0.0 - 0.9 Green, above 0.9 Red

Fix and keep the icons of Job state align with Job list.

we seem to have a new constraint: the name of vc has to be low-cased. @yqwang-ms

yep, VC name now is constrained by https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names


ballot: remove ATX ID

Description
given the fact that spacemesh is employing public key extraction in its signature scheme, the field ATX ID in ballot is redundant.
once public key is extracted from the signature, the node can look up the smesher's ATX for the same epoch.
note: the codebase currently also doesn't validate that the referenced ATX is indeed created by the same smesher :(

obsoleted by https://github.com/spacemeshos/go-spacemesh/issues/4139


Conan2

Adds conan2 packages for most dependencies. I haven't been able to get the whole thing to link yet, but I think we might as well get these in and iterate from there.

This looks ok to merge as all changes are in deps for now -- getting conan2 to work first and then move everything gradually to nix seems reasonable