add any_value function

Issue: Add a new ANY_VALUE function
ANY_VALUE will fail at analyze phase when FULL_GROUP_BY is set to true (default).
Besides, ANY_VALUE will choose the first value as the result for performance consideration.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi @Seaven @stdpain , thanks for your comments. I checked the usage for both two update_batch_single_state function just now and found that one is used for window function and another is used for aggregate function. Since any_value is only used as an aggregate function, my question is whether it is enough to implement virtual void update_batch_single_state(FunctionContext* ctx, size_t batch_size, const Column** columns, AggDataPtr __restrict state) const = 0; only?

Hi @Seaven @stdpain , thanks for your comments. I checked the usage for both two update_batch_single_state function just now and found that one is used for window function and another is used for aggregate function. Since any_value is only used as an aggregate function, my question is whether it is enough to implement virtual void update_batch_single_state(FunctionContext* ctx, size_t batch_size, const Column** columns, AggDataPtr __restrict state) const = 0; only?
Btw, it seems like all uts for be were passed but jenkins script ended with a failure. Something wrong with be uts? http://<IP_ADDRESS>:8090/job/starrocks_be_unittest/11175/console

No, should implement update method, you can find other method has call udpate in aggregate.h.

run starrocks_be_unittest

Hi @Seaven @stdpain , thanks for your comments. I checked the usage for both two update_batch_single_state function just now and found that one is used for window function and another is used for aggregate function. Since any_value is only used as an aggregate function, my question is whether it is enough to implement virtual void update_batch_single_state(FunctionContext* ctx, size_t batch_size, const Column** columns, AggDataPtr __restrict state) const = 0; only?
Btw, it seems like all uts for be were passed but jenkins script ended with a failure. Something wrong with be uts? http://<IP_ADDRESS>:8090/job/starrocks_be_unittest/11175/console

No, should implement update method, you can find other method has call udpate in aggregate.h.

@Seaven Yep, I think update is necessary in this function.
I mean virtual void update_batch_single_state(FunctionContext* ctx, size_t batch_size, const Column** columns, AggDataPtr __restrict state) const = 0; should be implemented while virtual void update_batch_single_state(FunctionContext* ctx, AggDataPtr __restrict state, const Column** columns, int64_t peer_group_start, int64_t peer_group_end, int64_t frame_start, int64_t frame_end) is unnecessary because any_value is only used for aggregation.
Btw, BE uts failed again.
Should we talk more about this PR in Slack channel? My Slack name is Gabriel Lee and feel free to contact me

@Gabriel39 you could rebase main and run be UT

@Gabriel39 you could rebase main and run be UT

@stdpain It works! Thanks


add tag for bumped angular js version

Could you add a tag like 0.8.0 for the bumped angularjs version?

We're busy right now with the next release 1.0.0. Can you just delete and download again the repo? It should work.

+1
I have installed videoangular with bower install videogular#master, which gives the correct version of AngularJS. Unfortunately the plugins (e.g. videogular-controls) reference the latest version of videogular, which is 0.7.2.

I've been updated all repos and bower configs.
It should work now.


[refactor][relay pass] Separate analysis and transform passes

As we are incrementally adding more and more passes to Relay. It makes sense to separate these passes into analysis and transforms namespaces. This makes the code cleaner and easy to find the pass that users want. In addition, commercial compilers like LLVM also organize the passes similarly.
This PR mainly separates the C++ side and creates analysis and transforms under passes. A followup PR will refactor the Python side.
Some discussion about this is here: https://discuss.tvm.ai/t/discuss-annotation-defined-subgraphs/5934/4
CC @tqchen @jroesch @icemelon9 @mbaret @comaniac

Let us remove the pass folder and just do relay/transforms and relay/analysis

@tqchen updated. PTAL. Thanks.

@zhiics please rebase against the master

Thanks @zhiics @tqchen @comaniac. this is now merged.


GattRequestRejectedException in common code

Hello everyone.
The GattRequestRejectedException is declared in Android module. Thus it is not possible catch it in common code. Is it possible declared it in common module of kable library?
Thank you.

The current exception hierarchy is as follows:

Exception

BluetoothException

BluetoothDisabledException
GattRequestRejectedException (Android-only)

IOException

NotConnectedException

NotReadyException
ConnectionLostException

ConnectionRejectedException
GattStatusException

Would it suffice for you to catch BluetoothException?
In general, the exception organization in Kable could use some re-structuring (e.g. #177), so I'm open to changing things up a bit, but I'm reluctant to move GattRequestRejectedException to common code as it is a state specific to Android.
Is there a reason BluetoothException won't suffice in common code?


spack-stack installations do not have prod_util

Describe the bug
The global-workflow is updating the ufs-weather-model hash to dd41cc6.  We rely on the model repository to provide the modules used in the compilation and execution of the model.
In addition, the workflow needs a utilities from a module called prod_util.
It seems on Orion (at least), that after the ufs-weather-model modules are loaded, prod_util is not an available module in the software stack.
To Reproduce
On Orion (at least)
501 ~ ❯❯❯ module purge
502 ~ ❯❯❯ module list
No modules loaded
503 ~ ❯❯❯ module use /work2/noaa/stmp/GFS_CI_ROOT/PR/1862/global-workflow/sorc/ufs_model.fd/modulefiles
504 ~ ❯❯❯ module avail

--------------------------------- /work2/noaa/stmp/GFS_CI_ROOT/PR/1862/global-workflow/sorc/ufs_model.fd/modulefiles ---------------------------------
   ufs_acorn.intel       ufs_common           ufs_hera.gnu      ufs_noaacloud.intel    ufs_s4.intel
   ufs_cheyenne.gnu      ufs_expanse.intel    ufs_hera.intel    ufs_odin               ufs_stampede.intel
   ufs_cheyenne.intel    ufs_gaea.intel       ufs_jet.intel     ufs_orion.intel        ufs_wcoss2.intel

--------------------------------------------------------------- /apps/modulefiles/core ---------------------------------------------------------------
   advisor/2019.5         fftw/3.3.8                  impi/2021.2                  motif/2.3.4                  qchem/5.3.0
...
...

505 ~ ❯❯❯ module load ufs_orion.intel
506 ~ ❯❯❯ module list

Currently Loaded Modules:
  1) intel/2022.1.2                    9) jasper/2.0.32      17) netcdf-fortran/4.6.0    25) crtm/2.4.0         33) ecbuild/3.7.2
  2) stack-intel/2022.0.2             10) zlib/1.2.13        18) parallel-netcdf/1.12.2  26) g2/3.4.5           34) yafyaml/0.5.1
  3) impi/2022.1.2                    11) libpng/1.6.37      19) parallelio/2.5.10       27) g2tmpl/1.10.2      35) mapl/2.35.2-esmf-8.4.2
  4) stack-intel-oneapi-mpi/2021.5.1  12) pkg-config/0.27.1  20) esmf/8.4.2              28) ip/3.3.3           36) scotch/7.0.3
  5) miniconda/3.9.7                  13) hdf5/1.14.0        21) fms/2023.01             29) sp/2.3.3           37) ufs_common
  6) stack-python/3.9.7               14) curl/8.0.1         22) bacio/2.4.1             30) w3emc/2.9.2        38) ufs_orion.intel
  7) cmake/3.23.1                     15) zstd/1.5.2         23) crtm-fix/2.4.0_emc      31) gftl/1.8.3
  8) libjpeg/2.1.0                    16) netcdf-c/4.9.2     24) git-lfs/2.12.0          32) gftl-shared/1.5.0

507 ~ ❯❯❯ module load prod_util
Lmod has detected the following error:  The following module(s) are unknown: "prod_util"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "prod_util"

Also make sure that all modulefiles written in TCL start with the string #%Module

Steps to reproduce the behavior:
See above.
Expected behavior
prod_util should be loaded.
System:
Seen on Orion.  Possibly others.
Additional context
prod_util is an integral module used in NCEP applications and should be included in every stack deployment.

On my macOS the module is called
prod-util/1.2.2
in spack-stack 1.5.0 rc1, spack-stack 1.4.1 and 1.4.0

On Sep 15, 2023, at 1:38 PM, Rahul Mahajan @.***> wrote:
Describe the bug
The global-workflow is updating the ufs-weather-model hash to dd41cc6 https://github.com/ufs-community/ufs-weather-model/commit/dd41cc61e5e52e2cfeb00af8930e7496a5fb91fd. We rely on the model repository to provide the modules used in the compilation and execution of the model.
In addition, the workflow needs a utilities from a module called prod_util.
It seems on Orion (at least), that after the ufs-weather-model modules are loaded, prod_util is not an available module in the software stack.
To Reproduce
On Orion (at least)
501 ~ ❯❯❯ module purge
502 ~ ❯❯❯ module list
No modules loaded
503 ~ ❯❯❯ module use /work2/noaa/stmp/GFS_CI_ROOT/PR/1862/global-workflow/sorc/ufs_model.fd/modulefiles
504 ~ ❯❯❯ module avail
--------------------------------- /work2/noaa/stmp/GFS_CI_ROOT/PR/1862/global-workflow/sorc/ufs_model.fd/modulefiles ---------------------------------
ufs_acorn.intel       ufs_common           ufs_hera.gnu      ufs_noaacloud.intel    ufs_s4.intel
ufs_cheyenne.gnu      ufs_expanse.intel    ufs_hera.intel    ufs_odin               ufs_stampede.intel
ufs_cheyenne.intel    ufs_gaea.intel       ufs_jet.intel     ufs_orion.intel        ufs_wcoss2.intel
--------------------------------------------------------------- /apps/modulefiles/core ---------------------------------------------------------------
advisor/2019.5         fftw/3.3.8                  impi/2021.2                  motif/2.3.4                  qchem/5.3.0
...
...
505 ~ ❯❯❯ module load ufs_orion.intel
506 ~ ❯❯❯ module list
Currently Loaded Modules:

intel/2022.1.2                    9) jasper/2.0.32      17) netcdf-fortran/4.6.0    25) crtm/2.4.0         33) ecbuild/3.7.2
stack-intel/2022.0.2             10) zlib/1.2.13        18) parallel-netcdf/1.12.2  26) g2/3.4.5           34) yafyaml/0.5.1
impi/2022.1.2                    11) libpng/1.6.37      19) parallelio/2.5.10       27) g2tmpl/1.10.2      35) mapl/2.35.2-esmf-8.4.2
stack-intel-oneapi-mpi/2021.5.1  12) pkg-config/0.27.1  20) esmf/8.4.2              28) ip/3.3.3           36) scotch/7.0.3
miniconda/3.9.7                  13) hdf5/1.14.0        21) fms/2023.01             29) sp/2.3.3           37) ufs_common
stack-python/3.9.7               14) curl/8.0.1         22) bacio/2.4.1             30) w3emc/2.9.2        38) ufs_orion.intel
cmake/3.23.1                     15) zstd/1.5.2         23) crtm-fix/2.4.0_emc      31) gftl/1.8.3
libjpeg/2.1.0                    16) netcdf-c/4.9.2     24) git-lfs/2.12.0          32) gftl-shared/1.5.0

507 ~ ❯❯❯ module load prod_util
Lmod has detected the following error:  The following module(s) are unknown: "prod_util"
Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
$ module --ignore-cache load "prod_util"
Also make sure that all modulefiles written in TCL start with the string #%Module
Steps to reproduce the behavior:
See above.
Expected behavior
prod_util should be loaded.
System:
Seen on Orion. Possibly others.
Additional context
prod_util is an integral module used in NCEP applications and should be included in every stack deployment.
—
Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/780, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2ROSHOQ7RBNMRUDFN4DX2SVDPANCNFSM6AAAAAA42IAP7U.
You are receiving this because you are subscribed to this thread.

I tried that as well.  There is no module named prod-util either to be found on Orion.

Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Ah I see what happened, it's because when we added the chained environment with pio 2.5.10, we only spec'd ufs-weather-model-env, so prod-util isn't in there. I think the solution would be to add global-workflow-env, preferably in a new chained environment since at this point a fair number of people are using ufs-pio-2.5.10.

Given that spack-stack-1.5.0 is available on Orion, why not use that?
https://spack-stack.readthedocs.io/en/release-1.5.0/PreConfiguredSites.html#msu-orion
Everyone is going to transition to that in the next several weeks.
Note I am out until Tuesday morning starting in 2 minutes ;-)

I see what happened, it's because when we added the chained environment with pio 2.5.10, we only spec'd ufs-weather-model-env, so prod-util isn't in there. If 1.5.0 doesn't work then we can always create a copy of the ufs-pio-2.5.10 env that also includes global-workflow-env.

In order to use 1.5.0, we would have to ensure that ufs-weather-model is updated to use 1.5.0.  Then we would have to use this updated hash of the ufs-weather-model for 1.5.0 in the global-workflow.  This will involve updating the global-workflow for any updates in the model (this is done once per 2 months or so or when the modeling team has a need to use the updates in the workflow application).
This might take some time.
Is it very difficult to install the missing modules in the ufs-pio-2.5.10 stack?

Would it make a difference whether we added it to ufs-pio-2.5.10 vs. creating a new environment? Specifically, just to clarify, is it partly about wanting to use the same spack environment as UFS? We might be able to get prod-util into the existing environment just by copying over the module file, in which case I'd say it's a very low-risk maneuver (but I'd need to test that before saying anything for sure).

@AlexanderRichert-NOAA
The desire and requirement here is to use the same spack environment as the UFS.
Adding prod_util into the existing environment by copying the modulefile is fine so long as we don't have to do anything special like adding MODULEPATHS etc.

We expect to move the UFS to spack-stack-1.5.0 in the next couple of weeks, is that good enough?

I'll test copying the modulefile on the Acorn installation and if that works we can make the same change elsewhere. @climbfuji I think the idea is he wants to test against current UFS without having to also update a bunch of package versions.

Copying /install/modulefiles///prod-util into the ufs-pio-2.5.10 modules on Acorn seems to work fine, the module loads and behaves as expected. @aerorahul would it be acceptable to make this change just on Orion to hold us over until the transition to spack-stack 1.5.0, or is there somewhere else you would also need this change sooner?

Hera and Orion.

@ulmononian can you take care of this on Orion? In spack-stack-1.4.1, under the ufs-pio-2.5.10 environment we created, we want to copy in the prod-util modulefiles from unified-env. @climbfuji can you do Hera?

@ulmononian can you take care of this on Orion? In spack-stack-1.4.1, under the ufs-pio-2.5.10 environment we created, we want to copy in the prod-util modulefiles from unified-env. @climbfuji can you do Hera?

done on orion.

Probably so, that's odd I didn't run into that on Acorn. In any case, please check the permissions, as I can't access the new prod-util modules.

Probably so, that's odd I didn't run into that on Acorn. In any case, please check the permissions, as I can't access the new prod-util modules.

i changed all the permissions, so please go ahead and try again.

Ok orion looks good for both compilers.

Thanks @ulmononian for taking care of both systems. @aerorahul can you test and confirm that it works on Orion and Hera?

Thanks @AlexanderRichert-NOAA and @ulmononian
I'll test on Orion.

I haven't had a chance to test on orion yet, but I did run into this same issue on hera yesterday and it does not look like there is a prod_util on hera yet for the pack-stack-1.4.1/envs/ufs-pio-2.5.10  (or I'm looking for the wrong module, but I don't think it's there) @aerorahul did your tests work on orion? Should I move my testing there?

I haven't had a chance to test on orion yet, but I did run into this same issue on hera yesterday and it does not look like there is a prod_util on hera yet for the pack-stack-1.4.1/envs/ufs-pio-2.5.10 (or I'm looking for the wrong module, but I don't think it's there) @aerorahul did your tests work on orion? Should I move my testing there?

@JessicaMeixner-NOAA
I have not been able to test the updates from @AlexanderRichert-NOAA on Orion yet (Orion was down yesterday).
I will do so today/tomorrow when I can find time.

I tested this on orion - and after loading the ufs-weather-model modules with spack-stack 1.4.1 I could not load prod_util
On hera, I tried to move to the ufs-weathe-rmodel with 1.5 since that PR is no longer a draft, and also there was no prod_util.
I could find a prod-util, but I don't know what that is or if that's the same thing and on WCOSS2 I know we have prod_util.  Thoughts @AlexanderRichert-NOAA or others?     I can move my global-workflow PR to use the PR of ufs-weather-model with 1.5 since we'll hopefully soon be there anyways, but I still have the no prod_util issue.  Note this is a blocker for us being able to effectively test some GEFS things and prepare for HR3.

The module is called prod-util, same as the package name, if I remember correctly.

On Sep 28, 2023, at 1:40 PM, Jessica Meixner @.***> wrote:
I tested this on orion - and after loading the ufs-weather-model modules with spack-stack 1.4.1 I could not load prod_util
On hera, I tried to move to the ufs-weathe-rmodel with 1.5 since that PR is no longer a draft, and also there was no prod_util.
I could find a prod-util, but I don't know what that is or if that's the same thing and on WCOSS2 I know we have prod_util. Thoughts @AlexanderRichert-NOAA https://github.com/AlexanderRichert-NOAA or others? I can move my global-workflow PR to use the PR of ufs-weather-model with 1.5 since we'll hopefully soon be there anyways, but I still have the no prod_util issue. Note this is a blocker for us being able to effectively test some GEFS things and prepare for HR3.
—
Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/780#issuecomment-1739902315, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RMX6IL5KCJMDZXPND3X4XHCLANCNFSM6AAAAAA42IAP7U.
You are receiving this because you were mentioned.

Okay - I've now seen prod-util on hera and orion with spack 1.4.  I did re-confirm the package is with an underscore on WCOSS2 though.

Yeah, I think it got changed under Spack because of their naming conventions...

Any chance to change it back to be consistent with WCOSS2? That seems like that could cause some issues.

One solution is to put in an if statement for wcoss2, then take it out once NCO moves to spack/spack-stack. How does that sound? I think the other solution is to copy the 'prod-util' directory to 'prod_util' in our existing deployments, then update our module file config to rename it for future deployments. It'd be nice to avoid that, partly because I don't want developers to start using the hyphenated version then wonder where it went in the next release, but it's one possibility.

It's possible that spack doesn't allow an underscore in the package name, but in this case it's no problem at all to just name the module file differently so that there will be no impact on downstream repositories.

in configs/common/modules.yaml:
      projections:
        prod-util: 'prod_util/{version}'

I'm good with that. For the existing releases, we can start a checklist of systems in the issue description and copy prod-util to prod_util on each one.

@AlexanderRichert-NOAA We could try something different, locally first on a sandbox: go to the existing environment, add the above projection, rerun spack module [lmod|tcl] refresh and spack stack setup-meta-modules. That should do.

Though I'll test it on Acorn

Don’t forget the virtual packages that depend on the module for prod_util. You need to refresh these as well.

On Sep 28, 2023, at 2:14 PM, Alex Richert @.***> wrote:
I would lean toward just refreshing prod-util (spack module [lmod|tcl] refresh prod-util), but, I don't think I have write access to the relevant installations anyway, so, the details are up to you and/or @ulmononian https://github.com/ulmononian :)
—
Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/780#issuecomment-1739945484, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RJAFCDZWPCLDYVVYG3X4XLCPANCNFSM6AAAAAA42IAP7U.
You are receiving this because you were mentioned.

Ah right, forgot about those

I’ll test it on my mac in a bit, have the full unified-env set up already

On Sep 28, 2023, at 2:16 PM, Alex Richert @.***> wrote:
Ah right, forgot about those
—
Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/780#issuecomment-1739947684, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RPY5HKBJ5DT6GU5U4DX4XLKBANCNFSM6AAAAAA42IAP7U.
You are receiving this because you were mentioned.

Okay, worked fine on Acorn. It's only used by global-workflow-env, so if you want to save 30 seconds you can just rebuild those, then delete the existing hyphenated one (:

@AlexanderRichert-NOAA This works as expected. With:
--- configs/common/modules.yaml	2023-09-26 09:03:49.000000000 -0600
+++ envs/unified-env/common/modules.yaml	2023-09-28 14:24:00.000000000 -0600
@@ -16,6 +16,7 @@
         nlohmann-json: '{compiler.name}/{compiler.version}/json/{version}'
         nlohmann-json-schema-validator: '{compiler.name}/{compiler.version}/json-schema-validator/{version}'
         libjpeg-turbo: '{compiler.name}/{compiler.version}/libjpeg/{version}'
+        prod-util: '{compiler.name}/{compiler.version}/prod_util/{version}'
       exclude:
       # List of packages for which we don't need modules
       - apple-libunwind
@@ -271,6 +272,7 @@
         nlohmann-json: 'json/{version}'
         nlohmann-json-schema-validator: 'json-schema-validator/{version}'
         libjpeg-turbo: 'libjpeg/{version}'
+        prod-util: 'prod_util/{version}'
       exclude:
       # List of packages for which we don't need modules
       - apple-libunwind

do:
source setup.sh
tar -cvzf unified_env_modulefiles_backup.tar.gz envs/unified-env/install/modulefiles/
spack env activate -p envs/unified-env
rm -fr envs/unified-env/install/modulefiles
spack module lmod refresh
spack stack setup-meta-modules
spack clean -a && spack env deactivate

Cool, good call with the backup. I'll put a PR in develop with the config update.

I suggest the following:

create bugfix branch release/1.5.1 from release/1.5.0 for spack-stack (hold off on spack submodule unless we need changes later)
Commit the above change to both develop and the bugfix branch (after https://github.com/JCSDA/spack/pull/328 and https://github.com/JCSDA/spack-stack/pull/790 are merged)
on each system

check out the bug fix branch, copy the modified file into the env and follow the above procedure

I cannot do this this week - but next week is ok.
If you agree, can you start the bugfix branch etc please?

Do you mean check out the 1.5.1 branch in the existing 1.5.0 deployments?

Do you mean check out the 1.5.1 branch in the existing 1.5.0 deployments?

Yeah ... I know, not the cleanest approaches of all, but because it's such a tiny change that doesn't involve any real code changes ... what do you think?
I expect a full bug fix release later in this quarter anyway with all the input we get from UFS and JEDI.

I think I'd rather either make the change to the 1.5.0 branch, or just update the existing envs manually and then incorporate the Proper Fix into 1.5.1 which as I recall we're doing anyway (+develop of course).

Ok, fine with me. Since the 1.5.0 tag is still warm (just created a few hours ago), I might as well retag.

@AlexanderRichert-NOAA I created the 1.5.1 release branches. This includes the tiny doc bugfix from @ncrossette that I just merged into develop. Please add this small bug fix to 1.5.0, 1.5.1 and develop (create once, cherry-pick twice, get three times the github credits!)

@AlexanderRichert-NOAA Can you create a checklist with all the hosts/envs somewhere please that we can use next week to fix the module name as per instructions above? Thank you!

@climbfuji done

There was a request/issue filed by @chan-hoo requesting to update version of prod_util/2.0.14:
https://github.com/ufs-community/ufs-srweather-app/issues/839
A PR is currently open in NCEPLIBS-prod_util , https://github.com/NOAA-EMC/NCEPLIBS-prod_util/pull/58,
after which this version should become available from NOAA-EMC/NCEPLIBS-prod_util repository.

There was a request/issue filed by @chan-hoo requesting to update version of prod_util/2.0.14: ufs-community/ufs-srweather-app#839
A PR is currently open in NCEPLIBS-prod_util , NOAA-EMC/NCEPLIBS-prod_util#58, after which this version should become available from NOAA-EMC/NCEPLIBS-prod_util repository.

This should become its own issue in spack-stack and, if the timeline fits, make it into spack-stack 1.5.1

It does not look spack-stack 1.5.1 has the prod_util package, as appears from
https://github.com/JCSDA/spack-stack/blob/release/1.5.1/configs/common/packages.yaml
Does it need to be added?..

https://github.com/JCSDA/spack-stack/blob/d7097144dee99896e88e426c1d74c5acf9e7e9d2/configs/common/packages.yaml#L205

Oh, does it need to be changed to prod_util, or the prod-util was an established name in the spack-stack environment?

No, please see the lengthy conversation above

On Oct 11, 2023, at 2:57 PM, Natalie Perlin @.***> wrote:
Oh, does it need to be changed to prod_util, or the prod-util was an established name in the spack-stack environment?
—
Reply to this email directly, view it on GitHub https://github.com/JCSDA/spack-stack/issues/780#issuecomment-1758528031, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RPAC73K4KWJUPZXO63X64B5FANCNFSM6AAAAAA42IAP7U.
You are receiving this because you were mentioned.

As far as I can see, only Gaea C4/C5 and ParallelWorks (EPIC installations) missing

As far as I can see, only Gaea C4/C5 and ParallelWorks (EPIC installations) missing

@natalie-perlin

@

As far as I can see, only Gaea C4/C5 and ParallelWorks (EPIC installations) missing

@natalie-perlin

Will take care of it today

Gaea and Gaea-c5 are ready.
However, no spack-stack-1.5.0 installed on ParallelWorks cloud clusters AWS, Azure or GCP (in /contrib/spack-stack), the latest is 1.4.1. Installing 1.5.0 on NOAA cloud.

Gaea and Gaea-c5 are ready.
However, no spack-stack-1.5.0 installed on ParallelWorks cloud clusters AWS, Azure or GCP (in /contrib/spack-stack), the latest is 1.4.1. Installing 1.5.0 on NOAA cloud platforms now.

I've shared the 1.5.0 binary cache with @mark-a-potts and @ulmononian a month ago ...

@ulmononian Any update on the missing platforms (Gaea C4 and C5) in the list? I know that @natalie-perlin is currently installing on the EPIC Parallelworks platforms, since those installs are from the latest release/1.5.0 branch / 1.5.0 tag, they will have the correct prod_util name, so we might as well consider them as done.

@climbfuji: c4/c5 are done.

@climbfuji only took me a year ;)


[Webpubsub] Add types for input/output/trigger

Add types for input/output/trigger

@vrdmr  Could you help to review or find someone to review the PR?

/azp run


Dashboard – Flow Runs – "Show all" button doesn't go away when showing all

On the dashboard Flow Runs card, when clicking the "Show more" button to view more runs for a flow, the button persists even when all flow runs have been retrieved. The expectation is that this button will stop showing once there are no more to show.

the issue with the "Show more" button on the dashboard Flow Runs card persists even after retrieving all flow runs. The expected behavior is that the button should disappear when there are no more flow runs to show. To address this, one should understand the code logic, inspect API responses, and check the front-end JavaScript code handling the button. Debugging and testing different scenarios, such as having no flow runs or a large number of runs, can help identify the problem. Once the issue is identified, propose and implement a fix, and ensure it is thoroughly tested and reviewed before deploying it to the production environment. Proper documentation of the investigation, resolution steps, and the fix will aid future troubleshooting. Communication with the team and stakeholders about the progress and resolution of the issue is also essential.


How to run SciWing on GPU?

This package is amazing, so thank you so much! I'm trying to parse a bunch of documents and I noticed that despite a GPU being available and torch recognizing it, the sciwing package won't run its models on the GPU. For example, if I run this code:
    from sciwing.models.neural_parscit import NeuralParscit
    neural_parscit = NeuralParscit()
    device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
    neural_parscit.to(device)

I get the following error suggesting that the model won't put the data on to the GPU.
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/models/neural_parscit.py", line 164, in predict_for_text
    predictions = self._predict(line=text)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/models/neural_parscit.py", line 111, in _predict
    predictions = self.infer.on_user_input(line=line)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/infer/seq_label_inference/seq_label_inference.py", line 285, in on_user_input
    return self.infer_batch(lines=[line])
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/infer/seq_label_inference/seq_label_inference.py", line 298, in infer_batch
    model_output_dict = self.model_forward_on_lines(lines=lines_)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/infer/seq_label_inference/seq_label_inference.py", line 140, in model_forward_on_lines
    model_output_dict = self.model(
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/models/rnn_seq_crf_tagger.py", line 115, in forward
    encoding = self.rnn2seqencoder(lines=lines)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/modules/lstm2seqencoder.py", line 121, in forward
    embeddings = self.embedder(lines=lines)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/modules/embedders/concat_embedders.py", line 46, in forward
    embedding = embedder(lines)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/sciwing/modules/embedders/trainable_word_embedder.py", line 75, in forward
    embedding = self.embedding(numericalized_tokens)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 156, in forward
    return F.embedding(
  File "/opt/anaconda/envs/sciwing/lib/python3.8/site-packages/torch/nn/functional.py", line 1916, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Input, output and indices must be on the current device

Is there a standard way of having SciWing run on a GPU? I couldn't find anything in the documentation or poking around. Thanks!

Hello @davidjurgens  :) I am Abhinav. Thanks for trying the package out! I had overlooked inference using GPUs.. I have now fixed it in the new release of SciWING which you can install using pip (pip install sciwing). You can use it as follows
 neural_parscit = NeuralParscit(device=0)
device=0 is the GPU number that is visible. You can pass device=-1 for cpu.
Please let me know if you need any help :)
Thanks
Abhinav

Closing due to inactivity.

Addressed the upstream repo.


[BUG] Updated version of azure-spring-cloud-stream-binder-servicebus-topic required which uses Spring Cloud Function 3.1.7 or 3.2.3 to address CVE-2022-22963

Describe the bug
CVE-2022-22963: Remote code execution in Spring Cloud Function by malicious Spring Expression
Link to vulnerability: https://tanzu.vmware.com/security/cve-2022-22963
To Reproduce
Link to vulnerability: https://tanzu.vmware.com/security/cve-2022-22963
Code Snippet
Maven version of artifact: azure-spring-cloud-stream-binder-servicebus-topic:2.14.0 still relies on Spring Cloud Function version 3.0.10.
Expected behavior
As per https://spring.io/blog/2022/03/29/cve-report-published-for-spring-cloud-function, Spring Cloud Function version 3.1.7+ or 3.2.3+ is required to address this vulnerability.
Setup (please complete the following information):

OS: MacOS
IDE: IntelliJ
Library/Libraries: com.azure.spring:azure-spring-cloud-stream-binder-servicebus-topic:2.14.0
Java version: [e.g. 8]
Frameworks: Spring Boot

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

In the future please refer to https://github.com/Azure/azure-sdk-for-java/blob/main/SECURITY.md#reporting-security-issues

@dubeyaru thanks for reporting this, we will look into it soon.

@dubeyaru thanks for reporting this, we will look into it soon.

Add dependency spring-cloud-function-context:3.2.3 in this PR to resolve

Close for the issue has been resolved.


Where on navigation property throws InvalidOperationException

I'm currently on beta7, I have the following model:
 public class Post
    {
        [Key]
        [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
        public long Id { set; get; }

        public virtual PublishingOptions PublishingOptions { set; get; }
}

public class PublishingOptions
    {
        public bool CommentsAllowed { set; get; }

        public bool VotesAllowed { set; get; }

        public bool SharingAllowed { set; get; }

        public DateTime? LastPublished { set; get; }

        [Key]
        public long Id { set; get; }

        public virtual Post Post { set; get; }

}

DbContext:
 public class DbContext : IdentityDbContext<BlogUser>
 {
        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            base.OnModelCreating(modelBuilder);
            var postEntity = modelBuilder.Entity<Post>();
            postEntity.Reference(p => p.PublishingOptions).InverseReference(po => po.Post);
            postEntity.ToTable("Posts");

            var pOpt = modelBuilder.Entity<PublishingOptions>().ToTable("PublishingOptions");
            pOpt.Reference(po => po.Post).InverseReference(post => post.PublishingOptions);

          }

        public DbSet<Post> Posts { get; set; }

        public DbSet<PublishingOptions> PublishingOptions { get; set; }
}

As you can see in PublishingOptions, LastPublished is nullable. Now, I try to perform query like this:
//Database is instance of DbContext
var query = from p in Database.Posts.Include(p=>p.PublishingOptions)
                where p.PublishingOptions.LastPublished == null
                select p;

or
var query = from p in Database.Posts.Include(p=>p.PublishingOptions)
                where p.PublishingOptions.LastPublished != null
                select p;

or even
var query = from p in Database.Posts.Include(p=>p.PublishingOptions).UseRelationalNullSemantics()
                where p.PublishingOptions.LastPublished == null
                select p;

What's even worse, this one does not work too (so it's probably not caused by nullable property):
IQueryable<Post> query =
                Database.Posts.Include(p => p.PublishingOptions).Where(p => p.PublishingOptions.CommentsAllowed == true);

After every query in my methods, I've called return query.ToList();
The exception I got from every similiar query is:
InvalidOperationException: The EF.Property<T> method may only be used within LINQ queries.
w Microsoft.Data.Entity.EF.Property[TProperty](Object entity, String propertyName)

//top of stacktrace:

w Microsoft.Data.Entity.EF.Property[TProperty](Object entity, String propertyName)
w lambda_method(Closure , QueryResultScope )
w System.Linq.Enumerable.<JoinIterator>d__1`4.MoveNext()
w System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
w Microsoft.Data.Entity.Query.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
w System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
w System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

I'm getting this too...

Is this under investigation? Could anyone from the team comment on this?

@marrrcin Is this the full repro? @gabrielgreen too, are you sure that you are not using the EF.Property() method anywhere in your code?
This seems bizarre, because we are only throwing that exception from within the body of the method (at least currently).
Also, would it be possible for you to try more recent bits? Some parts of navigation property translation (which your queries exercise) was only added after beta7.

Moving to rc1, as there are two reports and this seems like something we need to get to the bottom of.

I was able to figure out why we would ever see that exception: In NavigationRewritingExpressionVisitor we happen to create expressions that invoke EF.Property<T>(). However those are supposed to be either translated in the store query or transformed once more into access to the value buffer if evaluated in memory.
In theory if we don't do the latter correctly you could end up getting that exception. I haven't been able to repro using either beta7 or our current bits so I am going to assume that we only had a temporary bug that is already fixed. Feel free to reopen the issue if you can still repro this in current or later bits.

Here is the repro code I used with beta8 bits in case it helps:
using System;
using System.Linq;
using System.Data.SqlClient;
using Microsoft.Data.Entity;

namespace Repro3119
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            using (var context = new MyDbContext())
            {
                context.Database.EnsureDeleted();
                context.Database.EnsureCreated();

                (from p in context.Posts.Include(p => p.PublishingOptions)
                    where p.PublishingOptions.LastPublished == null
                    select p).Load();

                (from p in context.Posts.Include(p => p.PublishingOptions)
                    where p.PublishingOptions.LastPublished != null
                    select p).Load();

                (from p in context.Posts.Include(p => p.PublishingOptions)
                    where p.PublishingOptions.LastPublished == null
                    select p).Load();

                context.Posts.Include(p => p.PublishingOptions).Where(p => p.PublishingOptions.CommentsAllowed == true).Load();
            }
        }
    }

    public class Post
    {
        public long Id { set; get; }
        public virtual PublishingOptions PublishingOptions { set; get; }
    }

    public class PublishingOptions
    {
        public long Id { set; get; }
        public bool CommentsAllowed { set; get; }
        public bool VotesAllowed { set; get; }
        public bool SharingAllowed { set; get; }
        public DateTime? LastPublished { set; get; }
        public virtual Post Post { set; get; }

    }

    public class MyDbContext : DbContext
    {

        public DbSet<Post> Posts { get; set; }

        public DbSet<PublishingOptions> PublishingOptions { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer(
                new SqlConnectionStringBuilder
                {
                    DataSource = "(localdb)\\mssqllocaldb",
                    InitialCatalog = "Repro3119"
                }.ToString());
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<Post>(
                entity =>
                {
                    entity.HasOne(p => p.PublishingOptions).WithOne(po => po.Post);
                    entity.ToTable("Posts");
                });

            modelBuilder.Entity<PublishingOptions>(
                entity =>
                {
                    entity.ToTable("PublishingOptions");
                    entity.HasOne(po => po.Post).WithOne(post => post.PublishingOptions);
                });
        }
    }
}

I didn't deal with this bug on Beta7, but I'm afraid I'm facing it now in Beta8:
The following query is not working:
        var query = DbContext.Employees
            .Include(p => p.Worksheets)
            .Where(p => p.IsActive);
        if (name != null)
            query = query.Where(p => p.Name.Contains(name));
        if (erpNumber != null)
            query = query.Where(p => p.ERPNumber == erpNumber.Value);

        if (requestedPage.RequestMode == EmployeeWithWorksheetRequestMode.DriversWithMissingWorksheet)
            query = query.Where(p => !p.Worksheets.Any(q =>
                    q.EmployeeId == p.Id && q.Year == requestedPage.Year && q.Month == requestedPage.Month));
        return query;

The exception is: InvalidOperationException: The EF.Property method may only be used within LINQ queries.
The workaround I found (still didn't validate if it is efficient) was:
....
query = query.Where(p => DbContext.Worksheets.FirstOrDefault(q =>
q.EmployeeId == p.Id && q.Year == requestedPage.Year && q.Month == requestedPage.Month) == null);
return query;
Now it works, but I think it should be supposed to do the query that way.
Cheers!

FYI, we are using #3029 (which is still open) to track the issue now.


Ignore drushrc.php writing when settings disabled

The Problem/Issue/Bug:
public/sites/default/drushrc.php being written when settings disabled.
How this PR Solves The Problem:
Skips writing the file when settings are disabled.
Manual Testing Instructions:
On a D7 site
 ddev config \
  --docroot=public \
  --disable-settings-management

Automated Testing Overview:

Related Issue Link(s):
Release/Deployment notes:

@joelpittet I remember you mentioning this in Slack. Question though - Was it a Drupal 7 site? Because this isn't to be written at all in Drupal 8+ I don't think.

Yes this is for Drupal 7, sorry this got lost in my emails

@rfay I posted a new patch, I simplified the conditional to one level so the nesting is easier to read and just exit early, I hope you don't mind, I can revert this change if you think it's overstepping scope?

You didn't post a new PR right? Just added a commit to this one? Thank you ! I'll be catching up on all of these before too long.

Yeah tried not to create queue noise and just reused this fork/PR hope that is a good approach?
Thanks for merging!

Yeah tried not to create queue noise and just reused this fork/PR hope that is a good approach?
Thanks for merging!


test(core): fix factories

closes #228

FYI, Once this is merged, I'll raise another MR with tests for api app.


Field class needs ColumnFamily attribute

Field class needs ColumnFamily attribute
This is required in preparation for a HBase source and sink and possibly Cassandra.

Hbase is schema less and is capable of storing any binary values.
It requires a row key similar to a primary key on Kudu
Column Qualifiers are analogous to columns and belong to a Column Family, e.g. cf1:col1, cf1:col2
The only thing that's required when you create a HBase table are the column families.
Hive maps HBase coordinates to columns via a table property on the DDL, e.g. :key, cf1:col1, cf1:col2 - :key is the row key and always the first field cf1:col1 is the column col1 in the column family cf1 - checkout HBaseStorageHandler for setting up a Hive table - note would be nice to Hive sources and sinks to support this dialect.

We'll add a ColumnFamilyMapper that the HBaseSink accepts, and uses to map from Field to ColumnFamily.


Add retry logic for I/O operations under Windows

🐛 Bug
When running under Windows, I get intermittent I/O errors such as:
Traceback (most recent call last):
  File "C:\Users\Gili\Documents\myproject\aggregator\src\main\python\ai\predict_outdoor_temperature.py", line 820, in <module>
    main()
  File "C:\Users\Gili\Documents\myproject\aggregator\src\main\python\ai\predict_outdoor_temperature.py", line 799, in main
    tune_hyperparameters(graph_queue)
  File "C:\Users\Gili\Documents\myproject\aggregator\src\main\python\ai\predict_outdoor_temperature.py", line 705, in tune_hyperparameters
    study.optimize(lambda trial: optimize_train(trial, graph),
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\study.py", line 400, in optimize
    _optimize(
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 66, in _optimize
    _optimize_sequential(
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 163, in _optimize_sequential
    trial = _run_trial(study, func, catch)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 264, in _run_trial
    raise func_err
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "C:\Users\Gili\Documents\myproject\aggregator\src\main\python\ai\predict_outdoor_temperature.py", line 705, in <lambda>
    study.optimize(lambda trial: optimize_train(trial, graph),
  File "C:\Users\Gili\Documents\myproject\aggregator\src\main\python\ai\predict_outdoor_temperature.py", line 671, in optimize_train
    return train(dataset, learning_rate, max_epochs, seq2seq_type, seq2seq_layers, linear_layers,
  File "C:\Users\Gili\Documents\myproject\aggregator\src\main\python\ai\predict_outdoor_temperature.py", line 445, in train
    trainer.fit(model)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 552, in fit
    self._run(model)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 917, in _run
    self._dispatch()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 985, in _dispatch
    self.accelerator.start_training(self)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 995, in run_stage
    return self._run_train()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1044, in _run_train
    self.fit_loop.run()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\base.py", line 112, in run
    self.on_advance_end()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 177, in on_advance_end
    self._run_validation()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 257, in _run_validation
    self.val_loop.run()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\base.py", line 118, in run
    output = self.on_run_end()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\dataloader\evaluation_loop.py", line 141, in on_run_end
    self.on_evaluation_end()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\dataloader\evaluation_loop.py", line 204, in on_evaluation_end
    self.trainer.call_hook("on_validation_end", *args, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1219, in call_hook
    trainer_hook(*args, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\callback_hook.py", line 229, in on_validation_end
    callback.on_validation_end(self, self.lightning_module)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 322, in on_validation_end
    self.save_checkpoint(trainer)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 386, in save_checkpoint
    self._save_none_monitor_checkpoint(trainer, monitor_candidates)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 750, in _save_none_monitor_checkpoint
    self._del_model(trainer, self.best_model_path)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 528, in _del_model
    self._fs.rm(filepath)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\fsspec\implementations\local.py", line 149, in rm
    os.remove(p)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/Gili/Documents/myproject/aggregator/src/main/python/ai/lightning_logs/version_163/checkpoints/epoch=149-step=149.ckpt'

As far as I can tell, there are no workarounds for this. I expect the library to retry for upwards of 30 seconds in this case because files are routinely locked by antivirus scanners and so on. In an ideal world, you would add this retry mechanism for all I/O operations across the entire library...

It's probably better that you raise this issue in https://github.com/intake/filesystem_spec


Testing data set

I recently had the idea of creating a plain text archive format, searched the web and found this project, which is precisely what I was looking for. I needed a Java implementation though and decided I would implement an I/O library for Java for hrx instead of rolling my own custom archive format. I just found the time to do it yesterday and here are the results: https://github.com/topobyte/hrx-java
I implemented a test case for each of the example files, however I'm not 100% sure my implementation behaves exactly as specified / the same as the ruby implementation concerning trailing newlines. I think it would be very useful to have a set of testing files as a reference for current and future implementing developers. As a start, how about creating an extracted version of all the example files, i.e. we could have a directory extracted-examples containing a directory X for each X.hrx from examples. I'm not too fluent in setting up ruby and gems, so I would appreciate if you could create such a reference data set by unpacking all examples into corresponding directories and pushing it here. Thanks!

Great idea! I'll send out a PR momentarily.

Thanks, that helped getting the newline handling right


fix(tactic/split_ifs): do not process the same condition twice

This prevents non-termination if the ite/dite cannot be simplified.
TO CONTRIBUTORS:
Make sure you have:

[x] reviewed and applied the coding style: coding, naming
[x] for tactics:

[ ] added or adapted documentation in tactics.md
[ ] write an example of use of the new feature in tactics.lean

[ ] make sure definitions and lemmas are put in the right files
[ ] make sure definitions and lemmas are not redundant

For reviewers: code review check list

Does this build need to be restarted?

Ok, restarted.


Move example sketch to appropriately named folder

The Arduino IDE requires the sketch folder name to match the filename of the primary sketch file. This change causes the example sketch to be accessible via the Arduino IDE's File > Examples > LIBRARYNAME menu after the library is installed.

I have resolved the cause of the previous CI build failure via a force push and the Travis CI build is now successful.

Thanks :)


BREAKING: Port to OpenTelemetry

I've spent a bit of time skimming over the OpenTelemetry docs, wondering if now might be the time to switch over from LogManager and Prometheus.
I've always wondered how tracing actually works.  In the past, the presentations were somewhat unclear about how trace data was actually exported in common situations and new presentations are still somewhat unclear (at least to me).
For the Kubernetes world, it seems like our services would need to configure a trace exporter that communicates with Tempo by sending trace information.  This means that NeonService or something would need to know the internal service name and port for Tempo and be able to discover this somehow automatically or have this info passed into the app container as environment variables so we can configure the exporter.
Extending this to apps like neon-desktop or the Loopie phone apps implies that we'd need two exporters running in the app: one for exporting traces to the cluster and the other exporting logs.  Presumably, we'd need headend services that authenticate the app/user and then forward this data to Tempo and Loki.
It looks like we'd need to do the same thing for metrics, where we'd probably need to write a custom metrics exporter that we'd include in the app to forward metrics to a headend API which forwards to Mimir.
For extra credit, we might make these headend services configurable so they could forward data to other places like Grafana Cloud, etc.
This seems like it could be a fairly deep rabbit hole that I'm not entirely sure we want to go down right now.  On the other hand, we're effectively talking about writing a LogManager exporter for neon-desktop, so rather than doing that we could just write an OpenTelemetry exporter, perhaps getting some help from existing opensource exporters.
Regardless, we'd also need to figure out something for trace exporting.  We might be able to ditch this temporarily and rely solely on the trace IDs in logs but we'd lose trace span baggage which means that we'd have to add a bunch of log labels like you did for Loopie which is messy and will lead to code/log bloat.  I don't think we really want to standardize on that approach.
I'm a bit torn about this:

It would be nice to hop on this bandwagon and ditch the ancient LogManager code (I probably coded the original version way back in 2005).

This would probably need to be a breaking change since this is used all over the Neon libraries.  We'd probably want to fork a v2.x branch for the older version of the libraries and bump to v3.x for the next gen libs.

This might also be a good time to split the neonkube and neonsdk repos although I don't think that would be strictly necessary.
I could see us implementing the log, metrics, and trace exporters for client apps in neonkube and perhaps the cluster side receivers as neonkube services that our users could use as well, so our headend services would just deploy these services, probably behind and application gateway that handles authentication and throttling.

This will be a substantial change for the neon libraries but probably not too terrible for our or Loopie's service apps, since we're loosly based on MSFT's logging extensions.

It would also be nice to switch to OpenTelemetry before we author a bunch of new services for the headend as well as services we might add to neonkube.  Now that neonkube 1.0 is getting close, we're poised to begin writing probably a substantial number of new apps soon.

I can see writing "exporters" for our existing logger taking a few days and we'd also need to implement the headend side as well.  We'd need to do something similar for tracing, although I hope we could take advantage of existing exporters for tracing.

Finally, I'm not entirely comfortable with doing YouTube explainers using our existing LogManager and NeonService implementations since those are somewhat non-standard now and if we deferred this change for a few months, early explainers will be out-of-date and we might end up leading adopters astray.

I'm thinking about having a go at this now, first by splitting the neonkube/neonsdk repos and then forking a main-v2 branch in neonsdk to support people using the older code (mostly Loopie, the new RaspberryDebugger guy, and neonkube until the port is completed) and then convert to OpenTelemetry.
We should discuss this.

This is going to be a breaking change for neonSDK v3.0+.

[x] Remove ServiceContainer and replace it with the standard Microsoft ServiceCollection class?
So our ServiceContainer class is an enhancement over MSFT's ServiceCollection.  The benefit is that ServiceContainer allows for service lookup at anytime whereas ServiceCollection allows lookup only after IServiceCollection.BuildServiceProvider() is called.
After thinking about it, I think that the reason MSFT went with this approach is so that the ServiceCollection will effectively be read-only after the build call and won't require any locking thereafter.  We need to use a lock in our implementation which could easily result in a performance hit for things like high traffic websites where thinks may need to be injected into ASP.NET controller constructors.
I think I can tweak ServiceContainer to get make the think read-only after the BuildServiceProvider() call and not use a lock thereafter.
JEFF UPDATE: I'm keeping ServiceContainer but I did optimize this for performance after BuildServiceProvider() is called.

[x] Can we somehow unify the global IServiceCollection for both ASP.NET and Console apps so that NeonService can work the same for both environments?
So we're currently unifying this by having the user call the AddNeon(IMvcBuilder, IServiceContainer) method which copies services from a ServiceContainer to the ASP.NET service collection.  This actually seems pretty reasonable.

[ ] Neon.Web: I think we should remove the NeonController and NeonControllerBase classes.
These mainly add logger convenience methods which we're transitioning away from to encourage standard logging behavior that everyone else does.

[ ] I'd like to somehow retain the INeonLogger extensions that support lambda functions to fetch the event message for better performance.  We'll need to be able to obtain the current log level though for this to work.

[ ] Logging should be able to support labels:

[ ] As an optional IEnumerable<IKeyValuePair<string, string>> parameter
[ ] ...or an optional Func<IEnumerable<IKeyValuePair<string, string>>> parameter

[ ] Integrate OpenTelemetry:

[ ] Logging
[ ] Metrics
[ ] Tracing


Chimera does not attack enemies you have hit or are being attacked by

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Place block '...'
Right click to open menu.
Click Button '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
System (please complete the following information):

Occultism Version:
OS: [e.g. Windows]
Minecraft Version:
Modpack Link and Version, or list of mods:

Additional context
Add any other context about the problem here.

Accidental Submission


CSUB-84: DEBUG CI failures

Something pulled in ring-0.16.20 which is the root cause of these failures. The last successful commit on dev is a36903a19063b8b45eac0a0e73d2ba3c4e8fba2b (17 hrs ago) -> https://github.com/gluwa/creditcoin/runs/5665719433?check_suite_focus=true and this library isn't referenced anywhere in the logs.
libp2p-core has an unversioned dependency on ring introduced in da88e4aa4fb273894460fe1499717ffb0b02960b
- "ring 0.16.11 (registry+https://github.com/rust-lang/crates.io-index)",
+ "ring",

Independently of what caused the change above the actual failures we're seeing are coming from upstream (reported just 9 hours ago):
https://github.com/briansmith/ring/issues/1469

Trying to pin to the older version doesn't seem to be possible:
$ cargo update -p ring --precise 0.16.19
    Updating crates.io index
error: failed to select a version for the requirement `ring = "^0.16.20"`
candidate versions found which didn't match: 0.16.19
location searched: crates.io index
required by package `sct v0.6.1`
    ... which satisfies dependency `sct = ">=0.6.0, <0.7.0"` (locked to 0.6.1) of package `ct-logs v0.8.0`
    ... which satisfies dependency `ct-logs = "^0.8"` (locked to 0.8.0) of package `hyper-rustls v0.22.1`
    ... which satisfies dependency `hyper-rustls = "^0.22.1"` (locked to 0.22.1) of package `sc-offchain v4.0.0-dev (https://github.com/gluwa/substrate.git?rev=57cab2f30bcc5343771c8bc763dd6c9c9e10f0a8#57cab2f3)`
    ... which satisfies git dependency `sc-offchain` (locked to 4.0.0-dev) of package `creditcoin-node v2.0.0-beta.5 (/home/ubuntu/creditcoin-substrate/node)`

The version of sc-offchain has been updated in 75dd97a7 as part of switching to the https://github.com/gluwa/substrate fork.
@nathanwhit do you have any other suggestions ?

Superseded by #109


Player crashes on Hisense 4K ATV4 when launched from API

This is being reported on behalf of another user who observed that the player fails to launch from API on Hisense 4K ATV4.
I have attached a video showing the issue, and a screenshot of the device information. I can't provide media information but the user advised that this occurs on all media, not anything specific.
It seems that the crash is caused by the initial buffering but it's strange that the player fails completely
https://github.com/moneytoo/Player/assets/2313757/e7cbfe0a-a8da-4abf-a5e0-aae7a2c94166

Maybe it's ok already 🤞, otherwise I don't see much to do without a bugreport.


Can't create custom JSON encode/decode for java.sql.Timestamp

We have some existing model classes with java.sql.Timestamp fields on them that we would like to use without modification with Resty-GWT. This requires a custom encoder/decoder.
I was able to write one and have it be used by setting the org.fusesource.restygwt.restyjsonserializergenerator configuration parameter. However, the generated class gets created in package java.sql. which Java then refuses to load (in DevMode at least) throwing a SecurityException.
Would it be possible to detect that the generated code will end up in a protected package, and generate it in a different package?

Workaround added with #176


jicofo: 1.0-934 -> 1.0-940

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for jicofo is: "A server side focus component used in Jitsi Meet conferences"
meta.homepage for jicofo is: "https://github.com/jitsi/jicofo"
meta.changelog for jicofo is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.0-940 in filename of file in /nix/store/sv06rlzxj9pzhk4lps6vfjc4yqzq6000-jicofo-1.0-940

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
jicofo

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/sv06rlzxj9pzhk4lps6vfjc4yqzq6000-jicofo-1.0-940 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A jicofo https://github.com/r-ryantm/nixpkgs/archive/210cb46c85041d97e516f0d1667d3eedd00f40cb.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/sv06rlzxj9pzhk4lps6vfjc4yqzq6000-jicofo-1.0-940
ls -la /nix/store/sv06rlzxj9pzhk4lps6vfjc4yqzq6000-jicofo-1.0-940/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

jicofo

Maintainer pings
cc @cleeyv @ryantm @yu-re-ka for testing.

Update completed by https://github.com/NixOS/nixpkgs/pull/198303


Opensearch security configuration error

Describe the bug
opensearch query returned 503  Log printing  Not yet initialized (you may need to run securityadmin)
Related component
Cluster Manager
To Reproduce
none
Expected behavior
The opensearch cluster is working properly, and the system index is being created normally
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Triage]
Hi @CCTV-six ,
Can you please add more information on what distribution are you using and the exact steps to reproduce along with error messages seen?
Please see the installation guide for appropriate steps: https://opensearch.org/docs/latest/install-and-configure/
Also see forum for possibly same issues faced by other errors https://forum.opensearch.org/
Thanks!


textDensity prop is not reactive

Is your feature request related to a problem? Please describe.
No.
Describe the solution you'd like
Some of the props of the chrono component like textDensity is not reactive. Meaning if changed the Chrono does not change the mode. Can we make it reactive

@NehalDamania textDensity is controlled via the dropdown menu Change Density. even though its exposed as a prop, currently its not reactive. may i know the use case of yours, so i understand the requirement clearly.

Hi @prabhuignoto, we would like to use our own buttons and components to control some of the features of react-chrono, like text density and may be some other props. That will allow us to name and style the controls as per the application's style.


Http propagation support for multiple formats

OpenCensus is committed to support multiple HTTP propagation formats:

Zipkin (b3 format)
Stackdriver (x-cloud-trace-context)
etc.

In order to scale to the number of formats that OpenCensus will eventually support, it is easier to have an independent contrib-http-util package with all the supported formats.
The initial API can look like this:
public final class HttpPropagationUtil {
  /**
   * Returns the SD format.
   */
  public static TextFormat cloudTraceContextFormat();

  /**
   * Returns the b3-format.
   */
  public static TextFormat b3Format();

  // In the future we can add "multiFormat(List<TextFormat> extractors, TextFormat injector);"
  // which is able to read from more formats and writes into one format.
  // This can be used when users want to make the transition from an old format to the new
  // TraceContext format.
}

The core API will only have support for the new TraceContext format.

/cc @adriancole

Made comment on the other issue  but removing b3 and leaving an unstable
trace-context format as only choice is cutting off nose to spite the face.
B3 is supported by a large order of libraries and sites that are not
zipkin. It is already configured to pass through intermediaries and many
only use logging but use B3 format. It is the only well defined stable
format out there, and has many different companies who care about it.
Making this hard to reach will not improve maintenance as it doesnt change,
but it will reduce the ability to use it easily in 3rd party tools.
So, yeah up to you but I think doing this with B3 is much different
decision than doing this with cloud trace. I think it is a mistake to
equate the two.
On the api design, some formats cannot be instantiated statically also, as
they need other context such as host identification, or have configuration.
I wouldnt expect the propagation format to be able to work as a static
function class forever, though would for these.

On api design probably best to have interface and registry such as java
service loader. The interface can expose a key for the format. This could
be an option relatively easy for pluggability.
Ex
B3Format extends SomeGenericInterface {
String type() {
return "b3";
}
...
}
Then a format could even be internal class (if this is the concern) but
still visible and available for use via dynamic lookup.
I have also heard folks ask about a "detecting format" in jaeger for
example who use both B3 and x-uber.
Sometimes you cant affect your ingress (ex current version of envoy, cloud
foundry etc.. even when they change, sites take a long time to update). If
census decides to remove B3 it becomes less viable as an alternative to
other tools out there today.

/cc @adriancole
My initial thought is to use a name-based reflection and service loader to dynamically load the class, like the way Hadoop initialize different FileSystem:
if user want to access a file through path http://abc.txt

user call FileSystem.get("http://abc.txt"); to acquire the FileSystem implementation for scheme http
Hadoop will first lookup fs.http.impl in the conf for the impl class name; if not defined, it will use service loader to scan who claims to provide http support.(link)
user access file through fileSystem.open(), etc.

So what about we do it like this?
B3Format extends TextFormat {
  String type() {
     return "b3";
  }
}

public final class HttpPropagationUtil {
  /**
   * Returns the {@link TextFormat} which supports the specified {@param textFormatName}.
  public static TextFormat getTextFormat(String typeName) {
     // 1. try find class name in system property
     // 2. if null, try service loader with the typeName.
     // 3. use reflect to create class.
  }
}

to use in instrumented code:
TextFormat b3TextFormat = HttpPropagationUtil.getTextFormat("b3");
TextFormat cloudTraceTextFormat = HttpPropagationUtil.getTextFormat("cloudtrace");

Also I think we could leave B3 and whatever we have now in impl_core for default support. New text format could be put in contrib-http-util so they won't have to upgrade census if they want text format only.
With this we could have

Default support for B3 without config.
Extra support by putting jar in class path. (Again, no config).
Use -Dopencensus.http.propagation.Xxx in system property to workaround/hotfix a defective TextFormat implementation.

I would be careful about reflection especially with java 9, as a lot of
things don't work nice (though this might).
Iterating across all service loader instances isn't terrible.. has worked
in the past, though obv folks should be able to explicitly configure
something https://github.com/Netflix/denominator#third-party-providers
Anyway, something like this is good because at least in B3 the propagation
format is much wider than the exporter (should be separate for this
reason). At the same time formats that are coupled, such as cloud trace,
can be packaged with an exporter

Default support for B3 without config.
Extra support by putting jar in class path. (Again, no config).
Use -Dopencensus.http.propagation.Xxx in system property to
workaround/hotfix a defective TextFormat implementation.

I'd caveat "http" but all else sounds good. For example, routinely people
use same format for http and other RPC (until something interferes with
this like JMS constraints). That said could be fair to say something is
using "http" format even if it isn't http.

There's a choice to make about how to access the B3 format.. whatever that
is, I'd mark the api experimental as we will eventually have switching
concerns which might make things fragile.

@HailongWen we should simply add in the PropagationComponent an API:
abstract TextFormat getB3Format();

In the PropagationComponentImpl you return the real implementation. In the NoopPropagationComponent return a NoopTextFormat.
The PropagationComponent is already loaded with reflection via the TraceComponent so you don't have to implement anything about loading. When we want to change we will change how the TraceComponent is loaded.

@bogdandrutu so only B3 format will be accessed from core through PropagationComponent, and other format will be put in HttpPropagationUtil, right?

Yes and HttpPropagationUtil will be in that separate package


Crash in intmatcher.cpp - hitting MAX_PROTO_INDEX assertion

Crash at:
tesseract/src/classify/intmatcher.cpp:1155: void ScratchEvidence::UpdateSumOfProtoEvidences(INT_CLASS, BIT_VECTOR): Assertion `ClassTemplate->ProtoLengths[ActualProtoNum] < MAX_PROTO_INDEX' failed.

Environment

Tesseract Version: 4.1.0
Commit Number: 5280bbcade4e2dec5eef439a6e189504c2eadcd9
Platform: Debian GNU/Linux 9.11 (stretch)

Current Behavior:
Use case: running tesseract legacy ocr engine on multicast video streams and continuously extracting text from DVB bitmap subtitles.
Tesseract library built from source - you can find the build log here
Tessdata used: https://github.com/tesseract-ocr/tessdata/releases/tag/4.0.0
So far only tested with french language.
The way API is used:

Initialising TessBaseAPI

tessAPI->Init(trainedDataPath.c_str(), lang.c_str(), tesseract::OEM_TESSERACT_ONLY);

for each in memory dvb bitmap image extracted from the media stream:

char *res = tessAPI->TesseractRect(image.data(), 3, width * 3, 0, 0, width, height);
// some business logic using res
delete [] res;

where image is vector<uint8_t> containing RGB24 raw image data (dimensions given by width and  height.
The frequency of the crash is intermittent - sometimes it happens after 3-4 days of constant processing, sometimes just after a couple of minutes. It isn't super easy reproducible.
I thought it might be input related - so I added a code to store the image on disk before processing it - to be able to see which image caused the crash. Once the crash happened - I tried to load the same image in the same way - but it worked fine - it didn't crash again.
I have extended that code to store last couple of hundred images and now waiting on crash to happen again - so hopefully replaying them could trigger the crash and I can provide more data / reproduction scenario. Will update the issue with new findings.
Full stack trace:
/lib/x86_64-linux-gnu/libc.so.6(gsignal0x7fb1a4ba4fff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fb1a4ba642a]
/lib/x86_64-linux-gnu/libc.so.6(+0x2be67)[0x7fb1a4b9de67]
/lib/x86_64-linux-gnu/libc.so.6(+0x2bf12)[0x7fb1a4b9df12]
/usr/bin/zattoo-recorder(_ZN15ScratchEvidence25UpdateSumOfProtoEvidencesEP16INT_CLASS_STRUCTPj+0xb9)[0x55bed9b1c7e5
/usr/bin/zattoo-recorder(_ZN14IntegerMatcher5MatchEP16INT_CLASS_STRUCTPjS2_sPK18INT_FEATURE_STRUCTPN9tesseract13UnicharRatingEiib+0x1b9)[0x55bed9b1b025]
/usr/bin/zattoo-recorder(_ZN9tesseract8Classify11AdaptToCharEP5TBLOBiifP22ADAPT_TEMPLATES_STRUCT+0x2f5)[0x55bed9b0ee41]
/usr/bin/zattoo-recorder(_ZN9tesseract8Classify11LearnPiecesEPKciifNS_20CharSegmentationTypeES2_P8WERD_RES+0x4a0)[0x55bed9b0d820]
/usr/bin/zattoo-recorder(_ZN9tesseract8Classify9LearnWordEPKcP8WERD_RES+0x42a)[0x55bed9b0cf6e]
/usr/bin/zattoo-recorder(_ZN9tesseract9Tesseract19classify_word_pass1ERKNS_8WordDataEPP8WERD_RESPNS_13PointerVectorIS4_EE+0x2ce)[0x55bed9a6d986]
/usr/bin/zattoo-recorder(_ZN9tesseract9Tesseract17RetryWithLanguageERKNS_8WordDataEMS0_FvS3_PP8WERD_RESPNS_13PointerVectorIS4_EEEbS6_S9_+0xbf)[0x55bed9a6b141]
/usr/bin/zattoo-recorder(_ZN9tesseract9Tesseract26classify_word_and_languageEiP11PAGE_RES_ITPNS_8WordDataE+0x260)[0x55bed9a6d2fa]
/usr/bin/zattoo-recorder(_ZN9tesseract9Tesseract18RecogAllWordsPassNEiP10ETEXT_DESCP11PAGE_RES_ITP13GenericVectorINS_8WordDataEE+0x3c9)[0x55bed9a68315]
/usr/bin/zattoo-recorder(_ZN9tesseract9Tesseract15recog_all_wordsEP8PAGE_RESP10ETEXT_DESCPK4TBOXPKci+0x320)[0x55bed9a6875e]
/usr/bin/zattoo-recorder(_ZN9tesseract11TessBaseAPI9RecognizeEP10ETEXT_DESC+0x5a7)[0x55bed9a5523d]
/usr/bin/zattoo-recorder(_ZN9tesseract11TessBaseAPI11GetUTF8TextEv+0x3a)[0x55bed9a567bc]

I have checked similar issues and it seems that I am hitting the following one:
https://github.com/tesseract-ocr/tesseract/issues/2716
Also checked:
https://github.com/tesseract-ocr/tesseract/issues/444
(tried the patch suggested there - but that didn't help)
Expected Behavior:
Crash free.
Suggested Fix:

I was finally able to reproduce the crash again after a couple of days - this time with more debug data - which allows me to reproduce the issue at will on the affect platform.
I stored last 100 images before the crash - and I am able to reproduce the issue by replaying them - the crash happens on the same image every time (the 53rd).
This is the simple program that's reading stored raw images (together with their width and height):
https://drive.google.com/open?id=1c2VhtNP6ypxHHBYe3SpT_BH3con-1v2Z
And this is the zip archive containing these images:
https://drive.google.com/open?id=1DD0YVrGdPGALDunuNzsZD6AJ62gfcD3h
Instruction to for reproducing it:

Build the test program, name it e.g crasher
Unizp the archive containing raw images.
Run the program:

./crasher {path_to_folder_containing_images} {path_to_trained_tessdata}

Let me know if you need more info.

Did anyone have a chance to take a look into this one?

Hm the ProtoLength is 29 for some reason. I am not familiar with the codebase so I don't know how this can happen. But anyways, the assert should only happen when compiling in debug mode.

Any update on this?

I can only report that we are still suffering from it. Let me know if more info is needed aside from what I already provided.

I was actually asking on your part as i suffer from this as well 😞 . sorry for the false hope

the assert should only happen when compiling in debug mode.

@milos-pesic-zattoo, did you use a debug mode?
Anyway, you can try to disable the adaptive learning phase, maybe it causing the issue.

@amitdo Thanks a lot for checking the issue - we don't specify build type - and based on the build log I don't see debug cxx flags.
How could I disable the adaptive learning phase and could it cause the loss of accuracy?

actually i see that cmke_build_type is empty in my case:
-- General configuration for Tesseract 4.1.0
-- --------------------------------------------------------
-- **Build type**:
-- Compiler: GNU
-- Used standard: C++14

How could I disable the adaptive learning phase

set classify_enable_learning to false.

could it cause the loss of accuracy?

In principle, yes it could harm accuracy, but it depends on your input. You should check this yourself on sample of representative images.

@milos-pesic-zattoo, the assertion can be reproduced by just processing image last_session_dvb_sub_image.raw53 twice.
The assertion was removed in the latest code, so you won't get it anymore.
Nevertheless there remains the question why first and second processing differ.

@stweil,
Was this solved with #3035?

I am not sure. The assertion should be fixed, but it is still unclear why MAX_PROTO_INDEX was exceeded. But let's close the issue. If necessary, we can reopen it.


test Core_MulSpectrum fails only under release mode on 64bit ARM

Please state the information for your system

OpenCV version: master / 3.1
Host OS: Linux (Ubuntu 16.04 ARM 64bit ODROID-C2)
Compiler & CMake: GCC 5.3.1 & CMake 3.5.1

In which part of the OpenCV library you got the issue?

in test of core module, Core_MulSpectrums.accuracy fails with "Bard accuracy"
only on release mode, not in debug mode

Expected behaviour

The test passes on debug mode
Following is the output of the test on debug mode

odroid@odroid64:~/opencv-fork/testMulSpectrumDebug$ ./bin/opencv_test_core --gtest_filter=Core_MulSpec*
CTEST_FULL_OUTPUT
OpenCV version: 3.1.0-dev
OpenCV VCS version: 3.1.0-424-g9ebe29a-dirty
Build type: debug
Parallel framework: pthreads
CPU features: neon
OpenCL is disabled
Note: Google Test filter = Core_MulSpec*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Core_MulSpectrums
[ RUN      ] Core_MulSpectrums.accuracy
[       OK ] Core_MulSpectrums.accuracy (2168 ms)
[----------] 1 test from Core_MulSpectrums (2169 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (2175 ms total)
[  PASSED  ] 1 test.

Actual behaviour

Fails on test
Following is the result from test on release mode

odroid@odroid64:~/opencv-fork/testMulSpectrumRelease$ ./bin/opencv_test_core --gtest_filter=Core_MulSpec*
CTEST_FULL_OUTPUT
OpenCV version: 3.1.0-dev
OpenCV VCS version: 3.1.0-424-g9ebe29a-dirty
Build type: release
Parallel framework: pthreads
CPU features: neon
OpenCL is disabled
Note: Google Test filter = Core_MulSpec*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Core_MulSpectrums
[ RUN      ] Core_MulSpectrums.accuracy
/home/odroid/opencv-fork/modules/ts/src/ts.cpp:518: Failure
Failed

        failure reason: Bad accuracy
        test case #65
        seed: a7e10895f4ac9826
-----------------------------------
        LOG:
output: Too big difference (=1.26238e-12) at (34, 112)
input array 0 type=64fC1, size=(186, 57)
input array 1 type=64fC1, size=(186, 57)
ref output array 0 type=64fC2, size=(186, 57)

-----------------------------------
        CONSOLE: ........
-----------------------------------

[  FAILED  ] Core_MulSpectrums.accuracy (43 ms)
[----------] 1 test from Core_MulSpectrums (43 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (45 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] Core_MulSpectrums.accuracy

 1 FAILED TEST

Additional description

The test passes successfully on other platforms, x86_64 and 32bit ARM
I'm trying to trace the reason why, but the point that it only happens under release mode is making things difficult
Folliwings are some information from cmake output

  Platform:
    Host:                        Linux 3.14.29-56 aarch64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               debug

  C/C++:
    Built as dynamic libs?:
It seems that comparison is OK, and multiply is OK, but the input it already different between Debug and Release mode.
I dumped the src1 and src2 in test_dxt.cpp as bmp.
They already differ.
odroid@odroid64:~/opencv-fork$ md5sum testMulSpectrumDebug/src1.bmp  testMulSpectrumRelease/src1.bmp  testMulSpectrumDebug/src2.bmp testMulSpectrumRelease/src2.bmp
aa99dd4ce33dae0f1d0ec35b7549d982  testMulSpectrumDebug/src1.bmp
e324953bf714e4ae7c0ecd068171a363  testMulSpectrumRelease/src1.bmp
2b3cf84726244e09e486704c8149203b  testMulSpectrumDebug/src2.bmp
9f592656d242e778c7d32e61ed72c542  testMulSpectrumRelease/src2.bmp

I need to dig in more.

abstract
Now I found the cause of difference between src1 and src2, but still the test fails.
detail
The difference of the src1 and src2 in test_dxt.cpp comes from the optimization of following line in ts_func.cpp
template<typename _Tp> static void
randUniFlt_(RNG& rng, _Tp* data, size_t total, int cn, const Scalar& scale, const Scalar& delta)
{
    for( size_t i = 0; i < total; i += cn )
        for( int k = 0; k < cn; k++ )
        {
            double val = randReal(rng)*scale[k] + delta[k]; /* this line */
            data[i + k] = saturate_cast<_Tp>(val);
        }
}

Now, this multiplication of scale and addition of delta happens separately under debug mode,
fmul d1, d1, d0
fadd d0, d1, d2

Under the release mode, optimization packs this two operation in to fmadd (fused multiply and add)
fmadd   d8, d8, d10, d9

Now, this makes the result slightly different.
Here's some experiment result of fmul+fadd and fmadd
debug mode (w/ -g  -O0)
randomNumber       asm(FMADD)         asm(FMUL+FADD)
0x4080b94a0b6532aa 0x4080b94a0b6532a9 0x4080b94a0b6532aa
0xc0780b37385ca5b2 0xc0780b37385ca5b3 0xc0780b37385ca5b2
0xc06f718e8ca69298 0xc06f718e8ca69299 0xc06f718e8ca69298
0xc070ed13f8dbb0ea 0xc070ed13f8dbb0e9 0xc070ed13f8dbb0ea
0x408113091f06da4e 0x408113091f06da4d 0x408113091f06da4e
0xc05c12b0fbcdd770 0xc05c12b0fbcdd770 0xc05c12b0fbcdd770
0x406fe5798351d860 0x406fe5798351d861 0x406fe5798351d860
0x4081cbb8843165ca 0x4081cbb8843165cb 0x4081cbb8843165ca
0x408aef6b1604b714 0x408aef6b1604b713 0x408aef6b1604b714
0xc08db5386386d81e 0xc08db5386386d81e 0xc08db5386386d81e

release mode (w/ -O3)
randomNumber       asm(FMADD)         asm(FMUL+FADD)
0x4080b94a0b6532a9 0x4080b94a0b6532a9 0x4080b94a0b6532aa
0xc0780b37385ca5b3 0xc0780b37385ca5b3 0xc0780b37385ca5b2
0xc06f718e8ca69299 0xc06f718e8ca69299 0xc06f718e8ca69298
0xc070ed13f8dbb0e9 0xc070ed13f8dbb0e9 0xc070ed13f8dbb0ea
0x408113091f06da4d 0x408113091f06da4d 0x408113091f06da4e
0xc05c12b0fbcdd770 0xc05c12b0fbcdd770 0xc05c12b0fbcdd770
0x406fe5798351d861 0x406fe5798351d861 0x406fe5798351d860
0x4081cbb8843165cb 0x4081cbb8843165cb 0x4081cbb8843165ca
0x408aef6b1604b713 0x408aef6b1604b713 0x408aef6b1604b714
0xc08db5386386d81e 0xc08db5386386d81e 0xc08db5386386d81e

From the left column result, each hex number shows the hex express of double of

operation in C
operation in asm using fmadd
operation in asm using fmul+fadd

The result from debug mode shows same result as FMUL+FADD, and release mode shows same result as FMADD.
I confirmed in the .s file that fmadd was only used in release mode.
So, the cause of the different input was because of fmadd.
I modified just this part using asm and confirmed that src1 and src2 becomes identical.
Still, this test ends up in error in another test case.
CTEST_FULL_OUTPUT
OpenCV version: 3.1.0-dev
OpenCV VCS version: 3.1.0-424-g9ebe29a-dirty
Build type: release
Parallel framework: pthreads
CPU features: neon
OpenCL is disabled
Note: Google Test filter = Core_MulSpec*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Core_MulSpectrums
[ RUN      ] Core_MulSpectrums.accuracy
/home/odroid/opencv-fork/modules/ts/src/ts.cpp:518: Failure
Failed

    failure reason: Bad accuracy
    test case #161
    seed: 587aab954e525c23
-----------------------------------
    LOG:
output: Too big difference (=1.86391e-12) at (20, 96)
input array 0 type=64fC2, size=(94, 69)
input array 1 type=64fC2, size=(94, 69)
ref output array 0 type=64fC2, size=(94, 69)

-----------------------------------
    CONSOLE: ...................
-----------------------------------

[  FAILED  ] Core_MulSpectrums.accuracy (158 ms)
[----------] 1 test from Core_MulSpectrums (158 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (160 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] Core_MulSpectrums.accuracy

I need to dig in cvtest::mulComplex and cv::muilSpectrum.
It seemed OK, but I guess it's different in asm level.

Summary
I think I found the cause of this problem, but I need to ask a help before fixing
source code of expected and actual
In cv::mulSpectrum, the most core part is following.
double re = dataA[j]*dataB[j] - dataA[j+1]*dataB[j+1]; // this line is the cause
double im = dataA[j+1]*dataB[j] + dataA[j]*dataB[j+1];
dataC[j] = re; dataC[j+1] = im;

Also, in test_dxt.cpp, cvtest::mulComplex is called and inside, it's also doing
double re = a[j]*b[j] - a[j+1]*b[j+1]; // this line is the cause
double im = a[j+1]*b[j] + a[j]*b[j+1];

c[j] = re;
c[j+1] = im;

The line which is doing the subtraction is the cause of the difference.
Both of the lines are doing ''double * double - double * double''.
debug mode
It seems that in debug mode, these operations are done separately.
Two fmul instructions and one fsub instruction.
release mode
In only in dxt.cpp, the operations is performed by fmsub instruction.
(Fused Multiply and Subtract)
corner case
This usually doesn't matter.
There will be a slight difference, but relatively small.
The worst case comes when both product of the double becomes very similar value.
Then the difference of fsub and fmsub becomes larger, due to the rounding error.
I just wrote a simple code to reproduce the corner case.

double a = 3300.4555;
double b = 3200.3456;
double c = 3300.4554;
double d = 3200.3421;
double product;
double diffFSUB;
double diffFMSUB;
asm(
    "fmov  d1, %2     \n"
    "fmov  d2, %3     \n"
    "fmul  d1, d1, d2 \n"
    "fmov  %1, d1     \n"
    "fmov  d2, %4     \n"
    "fmov  d3, %5     \n"
    "fmul  d2, d2, d3 \n"
    "fsub  d1, d1, d2 \n"
    "fmov  %0, d1     \n"
    :"=r"(diffFSUB), "=r"(product)
    :"r"(a),"r"(b),"r"(c),"r"(d)
    :"d1", "d2", "d3"
    );
asm(
    "fmov  d1, %2         \n"
    "fmov  d2, %3         \n"
    "fmul  d3, d1, d2     \n"
    "fmov  %1, d3         \n"
    "fmov  d1, %4         \n"
    "fmov  d2, %5         \n"
    "fmsub d1, d1, d2, d3 \n"
    "fmov  %0, d1         \n"
    :"=r"(diffFMSUB), "=r"(product)
    :"r"(a),"r"(b),"r"(c),"r"(d)
    :"d1", "d2", "d3"
    );
std::cout << diffFSUB  << std::hex << "  0x" << *(long long int*)&diffFSUB  << std::dec << std::endl;
std::cout << diffFMSUB << std::hex << "  0x" << *(long long int*)&diffFMSUB << std::dec << std::endl;
std::cout << product   << std::hex << "  0x" << *(long long int*)&product   << std::dec << std::endl;

result
11.8716  0x4027be4615f00000
11.8716  0x4027be4615f6476c
1.05626e+07  0x41642584c798f382

You can see that the result of diffFSUB has almost 20bits of rounding error compared to the diffFMSUB.
So, the cause of the error is that

in Release mode,

fmsub is used for dxt.cpp
fsub is used for test_dxt.cpp

in Debug mode,

both of the code uses fsub
and this only happens few pixel per test

question
Now I have an idea about 3 types of solutions.

force to use fmsub at this place
use volatile for the inter medium value, (i.e. force ''NOT'' to use fmsub at this place)
loosen the error threshold

But I'm not sure which way to go, since they all have pros/cons

force to use fmsub at this place

pros:

fmsub is good for accuracy

cons:

we can't rely on optimization of the compiler, so we need to use either asm or intrinsic, which will conflict with other platforms, so we need to mess the code with #ifdef, here

use volatile for the  inter medium value, (i.e. force ''NOT'' to use fmsub at this place)

pros:

volatile will prevent the use of fmsub, so result accuracy doesn't rely on optimization of the compiler

cons:

performance might slow down, due to the memory access from volatile

loosen the error threshold

pros:

Keep the code consistent with other platforms

cons:

This will affect whole test_core module, so I don't think it's a good idea

I can try to fix this but before that, I'd like to hear the opinion from everyone.

Nice investigation!
My suggestion is to increase threshold for "ARM + release" mode for this test/tests (proposal 3) or "fix" input test data (generate/use another set of values).

Thank you for the advice, @alalek
I pushed some commits, and hope it'll help OpenCV.


Please provide a more detailed tutorial.

I went through the tutorial but the content is too brief.
Can you provide a more detailed tutorial like “tensorflowquickstart for experts” ？
because I need control every step of the training process(like customize a MultiBoxLoss class).

Thank you for your attention.
About the problem of the tutorial being too brief, yes, this is the first version of the tutorial. We are working on it, and more content will be launched in later versions.
You can also raise issues to tell us if you are interested in something.


Add requestor to confirmation for images

Added in https://github.com/makusu2/makubot/commit/8be9ece3261cdb3a6e91d9cf1e149dff9d1f5d2d


Encoding issue

Some character is encoding into some unrecognizable char. For example the text "Code will be approximately between Latitude 5°N and 6°S" is getting converted to  "Code will be approximately between Latitude 5�N and 6�S"

Were you able to fix this? I have the same issue with multiple specials chars. when I view the output in HTML. Changing HTML encoding doesn't resolve it.. Thanks.


Cannot use Xcode to put on device.

When I try to run it In Xcode I get loads of critical non warning errors. and the build fails.
Iphone 7+ (9,4) 10.3.2

It doesn't work atm. (Due to v0rtex issues)

Should be fixed!


feat: Adding new CLI entrypoints and better handling of scenarios

This PR clean-up the CLI and adds the following options:

r2x run - new entry point for doing a translation
r2x init - new entry point to copy a template file

New options for r2x run

-v enables debug verbosity
-vv enables trace verbosty
--pdb enables pdb (got tired of running python -m pdb -m r2x

Codecov Report
Attention: Patch coverage is 23.33333% with 46 lines in your changes missing coverage. Please review.

Project coverage is 73.90%. Comparing base (3caade5) to head (a68af3a).
Report is 1 commits behind head on main.

Files with missing lines
Patch %
Lines

src/r2x/utils.py
8.69%
21 Missing :warning:

src/r2x/config.py
16.66%
15 Missing :warning:

src/r2x/runner.py
44.44%
10 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #94      +/-   ##
==========================================
- Coverage   73.98%   73.90%   -0.09%
==========================================
  Files          37       37
  Lines        3729     3740      +11
==========================================
+ Hits         2759     2764       +5
- Misses        970      976       +6

Files with missing lines
Coverage Δ

src/r2x/core.py
100.00% <100.00%> (ø)

src/r2x/runner.py
52.00% <44.44%> (-1.85%)
:arrow_down:

src/r2x/config.py
76.02% <16.66%> (+8.75%)
:arrow_up:

src/r2x/utils.py
55.02% <8.69%> (-2.97%)
:arrow_down:


process_sensor_caldata.py: skip accel or gyro if temperature is NAN

FYI @NicolasM0

If thermal compensation is enabled but a sensor id is not found in TC_* parameters, an error appear:
 ERROR [temperature_compensation] gyro init: failed to find device ID 4325386 for                          instance 2

If thermal compensation is enabled but a sensor id is not found in TC_* parameters, an error appear:
 ERROR [temperature_compensation] gyro init: failed to find device ID 4325386 for instance 2

I was considering that to be a feature. Would you like to change it?

I was thinking that this file should be changed to add the sensor id in parameters even if temperature is nan and let other TC_* par for this sensor to 0 to avoid this error.

I was thinking that this file should be changed to add the sensor id in parameters even if temperature is nan and let other TC_* par for this sensor to 0 to avoid this error.

That could work, although I have a slight preference for not carrying the dead weight. If anything I was more interested in the case where the sensor is calibrated against a different temperature source (perhaps configurable per instance), however it doesn't seem to matter here.


Package demonstration on www.cheminfo.org

We have currently a demonstration of the tool on:
http://www.cheminfo.org/ML/Regression/Simple_Linear_Regression/index.html?viewURL=https%3A%2F%2Fcouch.cheminfo.org%2Fcheminfo-public%2F420cf9177e43f89292d5ca966213ab06%2Fview.json%3Frev%3D1-6a4f3bb721b7976d9ff3e346a2fe2a07
But it is quite old and should be updated to show all the possibilities of the regression package.
I would suggest that we have a list of function and we create a set of 100 points with a small error on the Y axis and the corresponding algorithm is applied
So all the examples would be based on the new slick grid module that allows to save the data in the view JSON.

I updated the view
https://my.cheminfo.org/?viewURL=https%3A%2F%2Fmydb.cheminfo.org%2Fdb%2Fvisualizer%2Fentry%2Fd53b0c83f9dc783b0b5bc32cf9f5b2c1%2Fview.json
But there are still couple of bugs. We should check this together

There should be bugs in the view because we included breaking changes on the last version to have the same API on the repositories of ML.

This was already fixed. Just need to think what we could do extra

If you can add some more information in the post-it. It is now indexed by google so it is important it is there.


[WIP] Use middleware on feature endpoints to check cluster and organization

Q
A

Bug fix?
yes

New feature?
no

API breaks?
no

Deprecations?
no

Related tickets
fixes #2195

License
Apache 2.0

What's in this PR?
Use middleware on feature endpoints to check cluster and organization. In case of wrong cluster/organization pair the new response:
{
    "type": "about:blank",
    "title": "Not Found",
    "status": 404,
    "detail": "could not get cluster from database: could not find cluster by ID: cluster not found"
}

Additional context

Checklist

[x] Implementation tested (with at least one cloud provider)
[x] Error handling code meets the guideline
[x] Logging code meets the guideline (TODO)

thx @sagikazarmark i made the requested changes


how to only dependency a part of swiper module like swiper/core,import all swiper too big

i see the package.json below:
"peerDependencies": {
"swiper": "^5.2.0",
"vue": "2.x"
},
looks like need import all swiper modules, which too large

https://github.com/surmon-china/vue-awesome-swiper/issues/680#issuecomment-666364812