BUG: stats.zscore returns all NaN if given masked array with first element masked

Describe your issue.
passing a masked array to scipy.stats.zscore yields results that are all nan when the first element of the array is masked.
Reproducing Code Example
import numpy as np
from scipy.stats import zscore

rng = np.random.default_rng(seed=8675309)  # tried with 3 different seeds
x = rng.standard_normal(10)
mask = np.zeros_like(x)
y = np.ma.masked_array(x, mask)
y.mask[1] = True
print(zscore(y))
# [-0.6327236439032247 -- -0.6005585026611974 0.03365730448338995
#  -0.9428312016417735 1.775001047420241 1.189629067553608
#  -0.5510199319620768 -1.2407231008627846 0.9695689615738183]
y.mask[1] = False
y.mask[0] = True  # still happens if additional elements are masked besides element [0]
print(zscore(y))
# [nan nan nan nan nan nan nan nan nan nan]

Error message
n/a

SciPy/NumPy/Python version and system information
1.11.1 1.25.2 sys.version_info(major=3, minor=11, micro=4, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /opt/mambaforge/envs/mwe/include
    lib directory: /opt/mambaforge/envs/mwe/lib
    name: blas
    openblas configuration: unknown
    pc file directory: /opt/mambaforge/envs/mwe/lib/pkgconfig
    version: 3.9.0
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /opt/mambaforge/envs/mwe/include
    lib directory: /opt/mambaforge/envs/mwe/lib
    name: lapack
    openblas configuration: unknown
    pc file directory: /opt/mambaforge/envs/mwe/lib/pkgconfig
    version: 3.9.0
  pybind11:
    detection method: pkgconfig
    include directory: /opt/mambaforge/envs/mwe/include
    name: pybind11
    version: 2.10.4
Compilers:
  c:
    commands: /home/conda/feedstock_root/build_artifacts/scipy-split_1687995550978/_build_env/bin/x86_64-conda-linux-gnu-cc
    linker: ld.bfd
    name: gcc
    version: 12.3.0
  c++:
    commands: /home/conda/feedstock_root/build_artifacts/scipy-split_1687995550978/_build_env/bin/x86_64-conda-linux-gnu-c++
    linker: ld.bfd
    name: gcc
    version: 12.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.35
  fortran:
    commands: /home/conda/feedstock_root/build_artifacts/scipy-split_1687995550978/_build_env/bin/x86_64-conda-linux-gnu-gfortran
    linker: ld.bfd
    name: gcc
    version: 12.3.0
  pythran:
    include directory: ../../../_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.11/site-packages/pythran
    version: 0.13.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /opt/mambaforge/envs/mwe/bin/python
  version: '3.11'

Thanks for reporting the issue, @drammock.
Here's another example, using a freshly built scipy (version '1.12.0.dev0+1454.d98afea'):
In [23]: from scipy.stats import zscore

In [24]: y = np.ma.masked_array([[1.5, 2, 3], [2, 1.5, 3]], mask=[[0, 1, 0], [1, 0, 0]])

In [25]: y
Out[25]:
masked_array(
  data=[[1.5, --, 3.0],
        [--, 1.5, 3.0]],
  mask=[[False,  True, False],
        [ True, False, False]],
  fill_value=1e+20)

In [26]: zscore(y, axis=1)
Out[26]:
masked_array(
  data=[[-1.0, --, 1.0],
        [nan, nan, nan]],
  mask=[[False,  True, False],
        [False, False, False]],
  fill_value=1e+20)

The second row should not be all nan.

This may be related to gh-8669 and gh-17755. If there is a quick fix we can consider patching zscore individually, since it appears to be one of the most used functions of its type. If not, these issues are part of the theme of gh-14651, and would be addressed as part of that.

IIUC it looks like the problem is with the use of _first() to determine whether the input was of constant value. When the first element is masked, the input is wrongly determined to be constant.
https://github.com/scipy/scipy/blob/d98afeae0fcb2302d0f25ba561e7984d5fadf65c/scipy/stats/_stats_py.py#L3153

@drammock if _first can be fixed, PR welcome with that plus a tiny test that gives nan on main but correct values on this PR! From a very quick look it seems separate from #17755

I'll take a look tomorrow and see if I can come up with a reasonable patch.

I didn't have much time to look today. Interim conclusion is that for finding the first non-masked value along an axis np.ma.nonmasked_edges seems like it will help but it probably won't work (it simply omits indices when there are no unmasked elements, which means the results don't broadcast). In a chat with @larsoner he suggested an approach of reshaping input array to always be 2D and/or operating on the mask itself to find the relevant indices; I'll have a go at that early next week (unless someone beats me to it, which, feel free).

@drammock please let me know if fh-19093 looks alright to you. Since any non-masked element will do, I used min.

@drammock please let me know if fh-19093 looks alright to you. Since we're just trying to detect whether all elements are the same, any non-masked element will do. I used min since it is simple, just as fast in fast cases, and fast enough compared to std in slow cases.

That looks fine, I was focused on fixing _first and should have just changed zmap to not call _first (as you did). Sorry this fell off my plate, it's been a very busy week


build(config): Update config variables to conform to mastarm build pr…

…ocess
BREAKING CHANGE: Config variables/files have drastically changed.

Codecov Report

:exclamation: No coverage uploaded for pull request base (dev@1511327). Click here to learn what that means.
The diff coverage is 0%.

@@          Coverage Diff          @@
##             dev     #47   +/-   ##
=====================================
  Coverage       ?   0.03%
=====================================
  Files          ?     283
  Lines          ?   10719
  Branches       ?    3262
=====================================
  Hits           ?       4
  Misses         ?   10715
  Partials       ?       0

Impacted Files
Coverage Δ

lib/scenario-editor/utils/valhalla.js
0% <ø> (ø)

lib/manager/reducers/user.js
0% <ø> (ø)

lib/manager/actions/user.js
0% <ø> (ø)

lib/public/containers/ActiveUserAccount.js
0% <0%> (ø)

lib/scenario-editor/utils/reverse.js
0% <0%> (ø)

lib/gtfs/components/GtfsMap.js
0% <0%> (ø)

lib/admin/components/UserSettings.js
0% <0%> (ø)

lib/common/util/user.js
0% <0%> (ø)

lib/public/components/UserAccount.js
0% <0%> (ø)

lib/common/user/UserSubscriptions.js
0% <0%> (ø)

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1511327...b43f47c. Read the comment docs.


Typescript Error

Error:
error TS2605: JSX element type 'StripeCheckout' is not a constructor function for JSX elements.
  Types of property 'setState' are incompatible.
    Type '{ <K extends never>(f: (prevState: void, props: StripeCheckoutProps) => Pick<void, K>, callback?:...' is not assignable to type '{ <K extends never>(f: (prevState: {}, props: any) => Pick<{}, K>, callback?: () => any): void; <...'.
      Types of parameters 'f' and 'f' are incompatible.
        Type '(prevState: {}, props: any) => Pick<{}, any>' is not assignable to type '(prevState: void, props: StripeCheckoutProps) => Pick<void, any>'.
          Types of parameters 'prevState' and 'prevState' are incompatible.
            Type 'void' is not assignable to type '{}'.

Use:
<StripeCheckout
    name="name"
    description="description"
    image="/picture.jpg"
    ComponentClass="div"
    stripeKey="pk_test_..."
    token={token => console.log("Token received!")}
>
    <RaisedButton label="Pay for Somethin'" primary />
</StripeCheckout>

Solution:
In index.d.ts, change
export default class StripeCheckout extends React.Component<StripeCheckoutProps, void> {}

to
export default class StripeCheckout extends React.Component<StripeCheckoutProps, {}> {}

or
export default class StripeCheckout extends React.Component<StripeCheckoutProps, Object> {}

I've fixed the same problem in material-ui-number-input and will be submitting a PR momentarily.

The problem is saying that your Typescript repo is now strict mode enabled which forces you specify the typing. If you wish to keep it silent, you can simply switch it to false in tsconfig.json:
{
  "compilerOptions": {
    "strict": false,
    // ...
  }
}

Change your function handleToken to be a variable with has the same type of your component prop as following:
const handleToken: StripeCheckout['props']['token'] = (token) => {
  // You can receive hint from token type here
}


Release xlsx file

Added an option to select release file type
Refactored unreleased page (replacing useState with useRef) to reduce unnecessary page re- rendering

merged to devel


Discussions about coding styles

I'm submitting a proposal regarding...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report
[ ] Feature request
[ ] Question
[x] Refactoring
[x] Coding styles and conventions

Current conventions
I would like to request a change in coding styles and refactorings

Block body for methods, constructors and properties
public class ClassA {
    private int _a;
    public int A
    {
         get {  return _a;  }
         set {  _a = value; }
    }
    public ClassA(int a) { _a = a; }
    public int square() { return _a*_a; }
}

Explicit usage of variable declaration
string a = "";

Importing namespaces in a clumsy way
using SimplCommerce.Module.Cms.Models;
using SimplCommerce.Module.Core.Models;
using SimplCommerce.Infrastructure.Web;
namepsace SimpleCommerce.Module.Cms
{
     class x {...}
}

Longer line
A.abcde.defg.hi(a.b.c, abc(asd));

Expected conventions

Expression body for methods, constructors and properties
public class ClassA {
    private int _a;
    public int A
    {
         get => _a;
         set =>  _a = value;
    }
    public ClassA(int a) => _a = a;
    public int square() => _a*_a;
}

Implicit usage of variable declaration
var a = "";

Importing namespaces in a cleaner way
namepsace SimpleCommerce.Module.Cms
{
     using Models;
     using Core.Models;
     using Infrastructure.Web;
     // static classes
     using static Infrastructure.Web.ViewComponentExtensions;
     //Specific classes
     using User = Core.Models.User;
     class x {...}
}

Break longer lines into shorter ones
query
     .Where(a=>
     {
        ...
       return ...;
     })
    . Select(a=> a.i==d);

Another suggestion
As I have been encountering many issues regarding indentation, an .editorconfig file in the solution root should be there to avoid collision between the settings developer PC's IDE and this project's conventions and best practices. @thiennn  If you agree with this, I can  post a PR for it
Other information
It'd increase code quality, readability as well as understandability.

I suggest that you should structure like this
Item 1
Current:
Suggestion:
Item 2
Current:
Suggestion:

I prefer block body for methods, constructors and properties. Setting default value for properties is expression
I prefer use var
I prefer the current importing style:

using System...
using Microsoft ...
using SimplCommerce ...

namespace ....
{

}

long lines. It depends, I not very strictly on this. Sometime long lines are easier to read than a lot of short line.

Btw, I have added the default .editorconfig. Please take a look and comments on this.
Thanks

@thiennn

I don't understand the confusing statement

Setting default value for properties is expression

Since you

prefer block body for methods, constructors and properties

Since you prefer var would you accept PR's about this

Simplified namespaces should be preferable as I mentioned because

unnecessary imports might degrade performanceusing SimplCommerce.Infrastructure.Web;

imports all the things included whileusing static SimplCommerce.Infrastucture.Web.ViewComponentExtensions;

would import only the class I need for the current class I'm working on
using simplified version should be preferableusing System;
namespace SimplCommerce.Module.Cms.Services
{
      using Models;
      using Core.Models;
      using ...
}

this code is much more cleaner and more readable

With due respect, I would say

In case of word wrap is enabled, long lines become nearly unreadable
Research says short lines are more readable to most of the developers.

I'm talking about these convention changes regardless of personal preference/opinion rather most of the developers around the world prefers these the most and these are introduced in the new version releases of c# on the basis of extensive researches on the developer community conducted by .NET team and some of them are discussed in best-practice docs from Microsoft.

I think we better track one by one and implement one by one. For example create an issue about using var, discuss on it, implement for it. Then another issue for example long lines

sounds good

As discussed https://github.com/simplcommerce/SimplCommerce/issues/588#issuecomment-424139614, feel free to open other issues (one by one)


Either add rust support to build-tools, build-tools-proxy, or make a build-tools-rust

what's your os ? I'm failed to build with following output
cargo build
   Compiling serde v1.0.145
   Compiling boring-sys v2.1.0
   Compiling h2 v0.3.14
   Compiling tower v0.4.13
   Compiling ztunnel v0.0.0 (/root/go/src/istio.io/ztunnel)
error: failed to run custom build command for `boring-sys v2.1.0`

the boring ssl issues depend on libclang-dev currently, looking into how to
drop that dependency
On Wed, Oct 19, 2022, 6:48 PM zirain @.***> wrote:

what's your os ? I'm failed to build with following output
cargo build
Compiling serde v1.0.145
Compiling boring-sys v2.1.0
Compiling h2 v0.3.14
Compiling tower v0.4.13
Compiling ztunnel v0.0.0 (/root/go/src/istio.io/ztunnel)
error: failed to run custom build command for boring-sys v2.1.0
—
Reply to this email directly, view it on GitHub
https://github.com/istio/ztunnel/issues/4#issuecomment-1284795391, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXIRIJAJF3OKKOL7LOTWECQGRANCNFSM6AAAAAARJQIGNQ
.
You are receiving this because you were assigned.Message ID:
@.***>

maybe my ubuntu(Ubuntu 18.04.6 LTS for envoy dev) is too old, still not working
Setting up libclang-6.0-dev (1:6.0-1ubuntu2) ...
Setting up libclang-dev (1:6.0-41~exp5~ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
root@ecs-zirain:~/go/src/istio.io/ztunnel# make build
cargo build
   Compiling serde_json v1.0.86
   Compiling hyper v0.14.20
   Compiling boring-sys v2.1.0
   Compiling tower-http v0.3.4
   Compiling ztunnel v0.0.0 (/root/go/src/istio.io/ztunnel)
error: failed to run custom build command for `boring-sys v2.1.0`

fixed by upgrade CMake to 3.20.0


No such file or directory while reproducing Multiview results

With python -m cosypose.scripts.run_cosypose_eval --config tless-vivo --nviews=4
I get the error:
(cosypose) amann@ipf-photolab-05:~/cosypose$ python -m cosypose.scripts.run_bop_inference_JA --config tless-pbr
Setting OMP and MKL num threads to 1.
pybullet build time: Dec  1 2020 13:20:08
0:00:01.382220 - Save dir: /home/amann/cosypose/local_data/results/tless-pbr--280187
0:00:01.382380 - --------------------------------------------------------------------------------
0:00:01.382415 - n_workers: 8
0:00:01.382439 - pred_bsz: 1
0:00:01.382460 - n_frames: None
0:00:01.382480 - n_groups: None
0:00:01.382499 - skip_evaluation: False
0:00:01.382518 - external_predictions: True
0:00:01.382536 - n_coarse_iterations: 1
0:00:01.382554 - n_refiner_iterations: 4
0:00:01.382574 - icp: False
0:00:01.382592 - debug: False
0:00:01.382612 - n_views: 1
0:00:01.382630 - ds_name: hb.bop19
0:00:01.382652 - save_dir: /home/amann/cosypose/local_data/results/tless-pbr--280187/dataset=hb
0:00:01.382671 - detector_run_id: detector-bop-tless-pbr--873074
0:00:01.382689 - coarse_run_id: coarse-bop-tless-pbr--506801
0:00:01.382708 - refiner_run_id: refiner-bop-tless-pbr--233420
0:00:01.382726 - --------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/amann/cosypose/cosypose/scripts/run_bop_inference_JA.py", line 270, in
main()
File "/home/amann/cosypose/cosypose/scripts/run_bop_inference_JA.py", line 266, in main
run_inference(this_cfg)
File "/home/amann/cosypose/cosypose/scripts/run_bop_inference_JA.py", line 105, in run_inference
scene_ds = make_scene_dataset(args.ds_name, n_frames=args.n_frames)
File "/home/amann/cosypose/cosypose/datasets/datasets_cfg.py", line 74, in make_scene_dataset
ds = BOPDataset(ds_dir, split='test_primesense')
File "/home/amann/cosypose/cosypose/datasets/bop.py", line 62, in init
assert ds_dir.exists(), 'Dataset does not exists.'
AssertionError: Dataset does not exists.
(cosypose) amann@ipf-photolab-05:~/cosypose$ python -m cosypose.scripts.run_cosypose_eval --config tless-vivo --nviews=4
Setting OMP and MKL num threads to 1.
pybullet build time: Dec  1 2020 13:20:08
0:00:00.000692 - Starting ...
0:00:01.383520 - SAVE DIR: /home/amann/cosypose/local_data/results/tless-vivo-n_views=4--1851779338
0:00:01.383580 - Coarse: tless-coarse--10219
0:00:01.383623 - Refiner: tless-refiner--585928
0:00:01.715244 - Building index and loading annotations...
[Memory]2.0s, 0.0min    : Loading build_index...
Traceback (most recent call last):
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/amann/cosypose/cosypose/scripts/run_cosypose_eval.py", line 491, in
main()
File "/home/amann/cosypose/cosypose/scripts/run_cosypose_eval.py", line 332, in main
scene_ds = make_scene_dataset(ds_name)
File "/home/amann/cosypose/cosypose/datasets/datasets_cfg.py", line 41, in make_scene_dataset
ds = _make_tless_dataset('test_primesense')
File "/home/amann/cosypose/cosypose/datasets/datasets_cfg.py", line 18, in _make_tless_dataset
ds = BOPDataset(ds_dir, split=split)
File "/home/amann/cosypose/cosypose/datasets/bop.py", line 74, in init
self.frame_index = pd.read_feather(save_file_index).reset_index(drop=True)
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/site-packages/pandas/util/_decorators.py", line 208, in wrapper
return func(*args, **kwargs)
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/site-packages/pandas/io/feather_format.py", line 119, in read_feather
return feather.read_feather(path, columns=columns, use_threads=bool(use_threads))
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/site-packages/pyarrow/feather.py", line 214, in read_feather
reader = FeatherReader(source)
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/site-packages/pyarrow/feather.py", line 40, in init
self.open(source)
File "pyarrow/error.pxi", line 80, in pyarrow.lib.check_status
File "pyarrow/io.pxi", line 1406, in pyarrow.lib.get_reader
File "pyarrow/io.pxi", line 1395, in pyarrow.lib._get_native_file
File "pyarrow/io.pxi", line 788, in pyarrow.lib.memory_map
File "pyarrow/io.pxi", line 751, in pyarrow.lib.MemoryMappedFile._open
File "pyarrow/error.pxi", line 80, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Failed to open local file '/home/amann/cosypose/local_data/bop_datasets/tless/index_test_primesense.feather', error: No such file or directory
I guess there is a problem with the test_primesense folder, I already deleted it and downloaded all available tless files again, but it didn't help.
Trying to run the inference I get an Error to.
Traceback (most recent call last):
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/amann/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/amann/cosypose/cosypose/scripts/run_bop_inference_JA.py", line 270, in
main()
File "/home/amann/cosypose/cosypose/scripts/run_bop_inference_JA.py", line 266, in main
run_inference(this_cfg)
File "/home/amann/cosypose/cosypose/scripts/run_bop_inference_JA.py", line 105, in run_inference
scene_ds = make_scene_dataset(args.ds_name, n_frames=args.n_frames)
File "/home/amann/cosypose/cosypose/datasets/datasets_cfg.py", line 74, in make_scene_dataset
ds = BOPDataset(ds_dir, split='test_primesense')
File "/home/amann/cosypose/cosypose/datasets/bop.py", line 62, in init
assert ds_dir.exists(), 'Dataset does not exists.'
AssertionError: Dataset does not exists.
I guess both Errors point to the test_primesense combined with the bop.py File.

I deleted Cosypose and set it up again, now it is working.


WCF service and DataContract/DataMember in client entities

Hi Toni,
first, thanks for a very useful extension!
I'm observing a strange behavior during probing the WCF scenario with different service/client entities.
My problem is that the receiving site do not see the data (IEnumerable jobs = queueService.GetJobs().Result) until I add DataContract and /DataMember attributes to the Job entity.
If I've understood correctly the attributes are not needed anymore since ModelBase is replaced by the EntityBase.
The namespaces on the client and service site are the same.
Thanks in advance,
Andrej

Hello @linqueue,
Glad you like the extension!  The DataContractSerializer is able to handle entities without [DataContract] and [DataMember] if your classes are public and your properties are read-write and public.  However, that won't work if your entities have cyclical references (for example, Product has Category and Category has Products).  In that case, you'll want to use a helper package I built, which configures the DCS to handle cycles without requiring the [DataContract] and [DataMember] attributes.
The NuGet package is called WcfSerializationHelper, and you add it to the project containing your ServiceContract.  Then place the following attribute on your service contracts: [DataContractSerializerPreserveReferences].
For an example, see the WCF sample for VS 2013: https://github.com/tonysneed/trackable-entities/tree/develop/Samples/VS2013/WcfSample/WcfSample.Service.Core.
Cheers!
Tony

Hi @tonysneed,
thanks - this didn't help directly but gave me a hint: I've created my solution by VS2015. Repeating the steps under VS2013 gave me a working solution. I guess the VS2015 extension is too old (v2.2.0). Copying few fragments from VS2013 solution into my VS2015 solution (containing several other projects) fixed the issue. There are plans to refresh the VS2015 VSIX?
Regards,
Andrej

@linqueue: Yes, in fact I am planning to get the VS2015 extension out this month.  There's been a delay because I needed to change the way in which NuGet packages are included in the VSIX: https://github.com/NuGet/Home/issues/598.  So keep an eye out this month for the VS2015 release of the extension for Trackable Entities.


content: adds plugin blog

@ncphillips - do you want to add more to this post?

oh we need to add the dependency files.


Rebuild fails on Sitecore 7.2.0 with strange message

12876 08:59:22 ERROR Attempted to load invalid xml.
Exception: System.Xml.XmlException
Message: Data at the root level is invalid. Line 1, position 1.
Source: System.Xml
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Sitecore.Xml.XmlUtil.LoadXml(String xml)
12876 08:59:22 ERROR First 200 characters: extranet\Anonymous
12876 08:59:22 ERROR Call stack:    at Sitecore.MainUtil.GetCallStack()
at Sitecore.Xml.XmlUtil.LoadXml(String xml)
at Sitecore.Data.Locking.ItemLocking.GetLockField()
at Sitecore.Data.Locking.ItemLocking.HasLock()
at Sitecore.Search.Crawlers.DatabaseCrawler.AddSpecialFields(Document document, Item item)
at Sitecore.Search.Crawlers.DatabaseCrawler.IndexVersion(Item item, Item latestVersion, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddItem(Item item, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Crawlers.DatabaseCrawler.AddTree(Item root, IndexUpdateContext context)
at Sitecore.Search.Index.Rebuild()
at Sitecore.Azure.Startup.SearchIndexRebuilder.<>c__DisplayClass5.b__4()
at System.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

The reason of the error is because the index-rebuilding operation has been run under the extranet\Anonymous user, which does not have access right to read the Lock field value.
The interesting part that it happened for Sitecore legacy search indexes based on the Sitecore.Search API. Most likely you are using the new Sitecore.ContentSearch API. If yes, it makes sense to rebuild only new indexes and completely disable the legacy one, which is going to be removed in Sitecore 8.X.
If you confirm you are not using the legacy indexes, I will disable the code which rebuilds them and release an update for the NuGet package.

I supposed previously that in Sitecore 7.2 legacy indexes is already disabled by default. And, as far as I see - we do not use legacy indexes

Unfortunately, the legacy indexes are on and running even in Sitecore XP 8.1.
Let me disable rebuilding of the legacy index and release a new package. For the meantime, you can perform the following steps to disable them in your Sitecore solution. I presume it should fix the above error.

In the Web.config file, set the Indexing.UpdateInterval setting to the 00:00:00 value.
In the Web.config file, remove the system index definition with entire <indexes> section under the \configuration\sitecore\search\configuration element:

<search>
...
  <configuration type="Sitecore.Search.SearchConfiguration, Sitecore.Kernel" singleInstance="true">
    <indexes hint="list:AddIndex">
      <index id="system" type="Sitecore.Search.Index, Sitecore.Kernel">
       ...
      </index>
    </indexes>
  </configuration>
</search>

In the file system, removed the \Data\Indexing\__system directory that belongs to legacy Sitecore Search API.


Bundle sources

This introduces a new parameter --bundle-sources that embeds the content of the source files into the sourcemap.
I figured adding a JSON encoder dependency is not desired, so I implemented a custom one. Apart from getting some inspiration from serde_json, I also made sure we follow the specification.
I was looking into how to fix the Rust sources, here are my preliminary findings:

The trim-paths RFC would solve our problems without us having to do anything.
There are a couple of workarounds to fix this on the Rust side:

Compile with -Zbuild-std.
Run a tool that fixes this beforehand (I'm not aware of one).
Custom toolchain with disabled virtualized paths.

The Rust compiler has some code we can look at to learn how to devirtualize these paths: source.
There is no perfectly reliable way to do this:

We can't differentiate between virtualized and real paths, except when they don't exist locally.
The hashes used in the virtualized path aren't available locally, unless we read them from the locally available debug information. (probably not a terrible idea)
Unlike the rust sources, which can be installed via rustup component add rust-src, related crates, like compiler_builtins or dlmalloc, aren't available locally at all unless the user specifically pulls them from crates.io.

Based on #9.
Fixes #3.

I figured adding a JSON encoder dependency is not desired, so I implemented a custom one. Apart from getting some inspiration from serde_json, I also made sure we follow the specification.

I'm not against using a JSON encoder dependency if it adds more value/performance than bloat/compilation slowdown. So if there is one you'd recommend, I'm happy to take a look. Serde seems like an overkill for such a flat JSON format as Sourcemap v3.

I was looking into how to fix the Rust sources, here are my preliminary findings:
* The [`trim-paths` RFC](https://rust-lang.github.io/rfcs/3127-trim-paths.html) would solve our problems without us having to do anything.

* There are a couple of workarounds to fix this on the Rust side:

  * Compile with `-Zbuild-std`.
  * Run a tool that fixes this beforehand (I'm not aware of one).
  * Custom toolchain with disabled virtualized paths.

* The Rust compiler has some code we can look at to learn how to devirtualize these paths: [source](https://github.com/rust-lang/rust/blob/0c2c243342ec2a2427f0624fac5ac59f0ee6fbcd/compiler/rustc_metadata/src/rmeta/decoder.rs#L1384-L1454).

* There is no perfectly reliable way to do this:

  * We can't differentiate between virtualized and real paths, except when they don't exist locally.
  * The hashes used in the virtualized path aren't available locally, unless we read them from the locally available debug information. (probably not a terrible idea)
  * Unlike the rust sources, which can be installed via `rustup component add rust-src`, related crates, like `compiler_builtins` or `dlmalloc`, aren't available locally at all unless the user specifically pulls them from crates.io.

I think we should make this a three-parter and bundle the project source code only at first, then investigate bundling dependencies, then the std. Focusing on project sources in this PR is a quick win and the rest can be undertaken as the right solution is found for each.
Let's continue the conversation for the third in #3

I noticed a couple of things that might be better to solve before adding a high-performance JSON encoder. We keep re/allocating large strings over and over just to add quotation marks and so on. I think writing the whole file should be done with a Writer instead of doing it with format!.

Yes, that's true. I planned to get rid of Strings in favor of a stack-based storage alternative as well in #1. That will likely be a huge speedup in itself. Since this is a bigger investigation, I think this should be part of a different PR.

Also, taking a quick look at crates.io as well, I honestly don't know of any good and reliably JSON encoder apart from serde_json.

Ok, let's attempt this with your inspired implementation. I'm pretty sure it's way faster than leveraging serde.

I think we should make this a three-parter and bundle the project source code only at first, then investigate bundling dependencies, then the std. Focusing on project sources in this PR is a quick win and the rest can be undertaken as the right solution is found for each.

Sounds good! FYI, dependencies are already working with this PR. It's only std which requires additional effort.

Ok, then just mark it ready for review if you feel comfortable with it and I'll review it the next couple of days. Thanks!

[..], then just mark it ready for review if you feel comfortable with it [..]

I only left it as a draft because it's based on #9, which isn't merged yet.
It's ready otherwise!

Rebased.
This is ready now.

Added a test as well.

Extended the test suite with a test for the json encode, so code coverage remains green, plus I renamed some of the test cases to have more expressive names.


Error trying to download maven

I created a new maven project and added dependency and repository like this:

4.0.0
com.mycompany
pandomium-test
1.0
jar

UTF-8
1.8
1.8

<repositories>
    <repository>
        <id>panda-repository</id>
        <url>https://repo.panda-lang.org/</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.panda-lang</groupId>
        <artifactId>pandomium</artifactId>
        <version>67.0.2</version>
    </dependency>
</dependencies>

And I got this error
Failed to execute goal on project pandomium-test: Could not resolve dependencies for project com.mycompany:pandomium-test:jar:1.0: Failed to collect dependencies for [org.panda-lang:pandomium:jar:67.0.2 (compile)]: Failed to read artifact descriptor for org.panda-lang:pandomium:jar:67.0.2: Failure to find org.panda-lang:pandomium-parent:pom:67.0.2 in https://repo.panda-lang.org/ was cached in the local repository, resolution will not be reattempted until the update interval of panda-repository has elapsed or updates are forced -> [Help 1]
Is anything I'm doing wrong?

Try using 1.0.7 Pandomium Version.

It worked, thanks.
The PandomiumTest.java is not working as well

java.lang.IllegalStateException: Couldn't create parent directory for /home/eduardobonfa/NetBeansProjects/pandomium-test/native/locales/te.pak.

67.0.2 is available: https://repo.panda-lang.org/org/panda-lang/pandomium/67.0.2/pandomium-67.0.2.jar
Remove org/panda-lang directory from your local repository (~/.m2/ by default).


Undo/Redo function

It would be very useful to have an undo/redo function in the editor.
To implement it one could use command pattern.

Undo and redo are available since very beginning.

Oh i was blind...short cut didn't work because of my keyboard layout. Sorry


enable live migration with dedicated cpus and isolated emulator thread

Blocking live migration with isolated emulator thread was introduced due to a misunderstanding: there is nothing that prevents a VM, with isolated emulator thread, to be live migrated (since the VM configuration, including emulator thread isolation, happens at VM initialization time).
This was introduced in PR
https://github.com/kubevirt/kubevirt/pull/8233

What this PR does / why we need it:
Users would like to migrate VMs with isolated emulator threads.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

/cc @xpivarc

/retest

@vladikr @rmohr ping

/retest

/retest pull-kubevirt-e2e-k8s-1.25-sig-compute-migrations

/test pull-kubevirt-e2e-k8s-1.25-sig-compute-migrations

the VM configuration, including emulator thread isolation, happens at VM initialization time

Currently virt-handler would reconcile to try to move processes to the new housekeeping cgroup. This means that there's a chance that right after the migration finishes the virt-handler would still be in the process of performing this task.
As I wrote in the original PR, the consequence of migrating a VM with dedicated CPUs is that possibly for a short period of time after the migration finishes the VM would, de-facto, run without real dedicated CPUs, since the "housekeeping" processes would also run on these CPUs.

In my opinion, we should allow migrations for such VMs.
While live-migrations are great, there's still a significant cost. It's a heavy operation that clearly has an effect on the VM's performance. It's obvious that a performant, real-time VM with dedicated CPUs pays the heaviest price for migrating and is better not to do so too often.
Saying that, it's also true that a performance price is payed only during the migration (and maybe slightly afterwards), and that live-migrations are absolutely crucial for certain use-cases.
As an example, let's say that a user runs a high-priority real-time VM and that the node has to be drained. It's absolutely reasonable for the admin to pay the performance price for a bounded amount of time in order to avoid shutting down the VM. Another example is cluster upgrades.
Since the performance cost is pretty obvious, time-bounded, and crucial for certain use-cases, I definitely think this should be allowed.

/lgtm from my side.
@matosatti, can you please add a link to the previous PR in the PR description for context?

from my side:
/lgtm
@matosatti, can you please add a link to the previous PR in the PR description for context?

/retest-required

/approve

/retest

/retest-required

/cherrypick release-0.58

/cherrypick release-0.59

/cherrypick release-1.0


Make menu into sidebar

Describe the solution you'd like
Change top left menu into a sidebar that launches from the left using semantic UI component
Split changelog and about section
About section should have credits and call-to-action for playstore and microsoft store

I'd like to help this issue and am on it.

I'd like to help with this issue and am on it.

Sure thing! Make sure to check out the contributing.md and don't hesitate to message me for any details.


Problem with Trigger Agent's "field>value" rule

Hey all,
I have a Website agent that passes the event field "price" to the trigger agent below:
{
  "expected_receive_period_in_days": "5",
  "keep_event": "false",
  "rules": [
    {
      "type": "field<value",
      "value": "500",
      "path": "{{ price | replace:'$','' | ceil | times:1 }}"
    }
  ],
  "message": {
    "title": "Availability {{price}}",
    "message": "Check out: https://[url]",
    "priority": "0"
  }
}

Price is a string, e.g. "$949.99," and the rule is true when the price is below 500. However, {{ price | replace:'$','' | ceil | times:1 }} does not return a number, so the "field<value" does not evaluate correctly. Is there any way of making the rule work?
Thank you in advance!

The path value is a dotted path through a hash in JSONPaths syntax.

It doesn't support Liquid templating, you'll have to pass it through an Event Formatting Agent first.

Bummer. I was trying to avoid creating yet another agent. Thank you for the quick and helpful reply!

Bummer. I was trying to avoid creating yet another agent. Thank you for the quick and helpful reply!

Agents are cheap.  Make as many as you need.  Don't try to be overly clever with them (ask me how I know...)


Add more information to (Multi)ESDTTransfers to fight scamming due to not actually knowing whats being sent

I've no idea, if this is possible, but let me try to explain, whats my problem with the current app:
Every time I transfer tokens (for example to stake them), I have no idea as a normal user, what I'm actually signing. This is because everything in the data field is encoded into a hex string and represented the same way, which isn't readable for a human.
You would have to write down the whole data-message from the ledger and decode the whole data and try to interpret it every time you transfer tokens with (Multi)ESDTTransfers. The non-technical user won't be able to do this.
In my opinion, this results in a big security flaw, because a harmful site (in whatever form) could target you by sending a transaction to the ledger where you think you only send a specific token to an (actually harmful) address, but in reality the Ledger gets a MultiESDTTransfer, in which all your tokens are being transferred to someone else, would could have you loose your whole ESDT-portfolio in a single transaction.
In my opinion, a more readable form would be better, e.g. before the gas and network strings another message "Send 1337420 MEX-A15E to ...".
I'm not sure though, if the ledger application has the capabilities to decode such data fields.
Important: Should be self explanatory, but the decoding and the interpretation of the decoded data MUST be done on the ledger itself, otherwise the attacker would be able to just fake the message, if the "Send 1337420 MEX-A15E to ..." can just be passed from outside.

@bogdan-rosianu
What exactly do you mean by "predicting"?
In my opinion, the only thing that is important to show is the tokens that will be OUTGOING. I don't see any reason to predict the INCOMING tokens. The idea is to prevent scammers showing a nice little UI that says "10 LKMEX for 1 EGLD" and then let you sign the tx data "100,000,000 LKMEX for 1 EGLD", maybe even sending all your other tokens aswell via MultiESDTNFT.... I would love to see "SEND 100,000,000 LKMEX-ABCDE, 100 RIDE-ABCDE, 200 ITHEUM-ABCDE" on the the ledger in those scenarios.
Do you have an example for scenarios, where it's hard to predict the tokens that are outgoing?
Or is the problem actually the decimal places?

Hello! Sorry for the late reply. Yes, one of the issues would be the decimal places, which is information that is not included in the transaction.
However, your suggestions are really good and we might implement something in the future. There are also other actions that could have been better explained (delegate, stake, unstake and so on)

@bogdan-rosianu
Its been over a year, this is still a problem and allows attackers to scam people by having people sign transfers of multiple esdt tokens in a pretty cryptic manner. This needs to change.
We should at least give a summary of WHAT tokens are being sent, even if we cant identify, how many (due to decimal precision).
The potential victim would at least see, that the token identifiers match with what he planned to sent, it would be a step into the right direction.


[CP] prevent tool crash when IntelliJValidatorOnMac encounters an installation with a missing CFBundleIdentifier

Issue Link
https://github.com/flutter/flutter/issues/138029
Commit Hash
94550c6
Target
beta
PR Link
https://github.com/flutter/flutter/pull/138176
Changelog Description
Prevents a crash in flutter doctor for macOS users who have an IntelliJ or Android Studio installation with a missing CFBundleIdentifier in its plist.
Impacted Users
Flutter devs developing on macOS
Impact Description
fluter doctor crashes for macOS users who have an IntelliJ or Android Studio installation with a missing CFBundleIdentifier (which is a required field) in its plist.
Workaround
Reinstall IntelliJ/Android Studio, or edit the installation's plist file to have an appropriate CFBundleIdentifier value.
Risk
low
Test Coverage
yes
Validation Steps
No response

LGTM

Released in 3.16.1


spatialdata_io.visium Drops Non-Tissue Spots

Our lab has a method that uses data from non-tissue spots. When I read a visium experiment with this method these spots are always dropped. I appreciate this is a reasonable default but it would be great if there was an option to read them into the resulting anndata structure.
Thanks,
Jeff

Hi @jeffquinn-msk, thanks for the feedback. We can definitely add this option as an extra argument to the visium() function, which would default to the current behavior.
Regarding your second message, would the file raw_feature_bc_matrix.h5, often available in data from SpaceRanger, work for your use case? It contains non-filtered reads.

@jeffquinn-msk I have implemented this in https://github.com/scverse/spatialdata-io/pull/107.


Prototype pollution (fixed)

Hi,
This package is vulnerable to prototype pollution.
POC
var gsonQuery = require("gson-query")
var obj = {}
console.log("Before : " + obj.polluted);
gsonQuery.set(obj,'__proto__/polluted','Yes! Its Polluted');
var obj1 ={}
console.log("After : " + obj1.polluted);

Added fix for this prototype pollution,
Thank you.

Nice. Thank you


[Security] Prevent Replay Attacks on MSA Signed Actions

Mutli PR

[ ] RPC and state

[ ] Refactor the storage for get_msa_keys and add a struct with the nonce and the keys (that is MSA_Id -> { nonce, keys })
[ ] Add a get_msa_nonce RPC call

[ ] Secure add_provider_to_msa (proof)
[ ] Secure add_key_to_msa (Proof)
[ ] Secure revoke_msa_key (Call)
[ ] Secure revoke_msa_delegation_by_delegator (Call)

Goal
Prevent replay attacks against MSA "signed" (signed by a key on the MSA) actions that are not Token transactions

add_provider_to_msa (proof)
add_key_to_msa (Proof)
Delegate Permissions Update (Future) (Proof)
revoke_msa_key (Call)
revoke_msa_delegation_by_delegator (Call)

Does NOT cover create_sponsored_account_with_delegation that is https://github.com/LibertyDSNP/frequency/issues/349
Current Working Solution

Add a piece of state that is MSA_Id -> nonce

Refactor the storage for get_msa_keys and add a struct with the nonce and the keys

Require that the nonce match for the above actions
Increment the nonce on any of the above actions
Add msa_nonce to the signed payload (for proofs) or the extrinsic (for calls) of each of the above items
Add a get_msa_nonce RPC call
Open Question:

Should we add a Signed Extension that checks this nonce? No

General Rule:

This nonce should be used for MSA signatures when that signature is not connected to tokens or capacity.

Discussion Notes
Questions

Do we need a nonce?
(assuming yes) Where does the nonce live?
Is it the same nonce for each of our three signed proofs?

How can we make sure that this solution assists with solving future
via a generic or standard structure?

Problems:

When you sign a payload, that payload should not be able to be used again.

If I add someone as a service provider with a signature, that
signature should not be able to be used to re-add them again.
If I sign to allow a key to be added to an MSA, that exact signature should
not be able to be used to re-added to the MSA

When you sign a payload, the service provider should have to use it within
a given time frame.
A revoked key that is compromised key should not be able to be added back
to the MSA
Situations:
Action is taken with a key that the chain knows about (KeyInfo)
Action is taken with a key that the chain does NOT know about (add/remove key)

Potential Solutions

Do nothing except the expireBlock
Add a nonce to the MSA

Con: Harder to add several providers in a row
Con: Requires always fetching the data from the chain first

Add (keep) a nonce on each key (KeyInfo)

ALSO do something around add_key replay

Have a nonce on the delegation

Do something else around add_key replay

Add a total key counter on the MSA (effectively a nonce just for key additions)

Do something different around delegations

Particular Situations
create_sponsored_account_with_delegation replay with an expire block

create_sponsored_account_with_delegation
add_key_to_msa
delete_key_from_msa (key from #1)
The Provider from #1 can create a NEW MSA with (key from #1)
Transaction 1-4 all executed before #1 payload expireBlock

add_key_to_msa replay with an expire block

add_key_to_msa
revoke_msa_key (key from #1)
Anyone can use the payload from #1
Transaction 1-3 all executed before #1 expireBlock

add_provider_to_msa replay with an expire block

add_provider_to_msa
revoke_msa_delegation_by_delegator
The Provider from #1 can add themselves back
Transaction 1-3 all executed before #1 payload expireBlock

I have been thinking about why we decided to add those payloads in the first place. We did this so that we shift the cost of the transaction. What if we rethink our cost shifting strategy.


Paid Resources updated

Videoblocks.com

I don't list some of the top paid resources, I don't feel that this is a good fit for the list. Thanks for your effort though.


yeah, I'm newbie on GitHub :P

I think if we add custom fonts feature and custom skin feature this tool become more cooler
so I suggest add this feature (and of course, i'll work more harder)
....so like this?

haha yeah, but the title should be a good short description of what you're telling in this issue :p


[Internals] Leverage SurvivalBase.jl

Once https://github.com/JuliaRegistries/General/pull/108545 is merged (registration of SurvivalBase.jl), the whole strata and surv stuff can be removed from here and leveraged directly from survivalbase.

Done in #68


Android ScanSettings

Per this discussion, this PR enables users to create a Scanner with Android-specific ScanSettings if the current default ones don't suffice. It simply adds a non-actual var scanSettings to the builder that's available when calling from an Android source set only. It defaults to the existing default. This means that users who don't care about customizing this can still invoke the DSL from a common source set, but those who want more control can invoke from an Android source set and get the control they need.

@mtrewartha looks great! Thanks so much!
Some minor tweaks I plan to make before merging:

Documentation formatting changes
Add @ObsoleteKableApi annotation and mark the scanSettings property with the annotation

Long term, I hope to provide all the ScanSettings options as native Scanner DSL builder options rather than having to use Android's directly. Although it is very low priority and will probably be a while before that changes, so this implementation is a great solution in the mean time. The annotation will simply warn consumers that the API is planned to change.

@twyatt sounds great! I definitely agree on the Scanner DSL front. I'm arguably less familiar with iOS, macOS, and JS BLE scanning capabilities. I poked around to see if those platforms had anything that made sense to include in some multiplatform "settings" notion that could be added, I just didn't see anything and that's why I opted for the Android-specific one for now. That annotation sounds good in the interim 👍🏻

@sdonn3 when you have a chance to review, please see updated changes via https://github.com/JuulLabs/kable/compare/main...twyatt/android-scan-settings.

Merged manually. Thanks for the contribution @mtrewartha!
This will be available in the next release. I had planned to get some additional changes into the next release, but let me know if you have a more immediate need for this and I can cut a release sooner.

Great, thanks @twyatt! We do have an immediate need for this change, but we can just use my fork for now, then swap back to the new release whenever it's ready.


Postgresql: ERROR: procedure merge_knownexploited(character varying, ...) does not exist

Describe the bug
When using version 8.0.1 and a postgres DB we get the exception stated in the subject. The DB is initialized with the latest script supplied for postgres with no errors showing when running the script.
Version of dependency-check used
The problem occurs using version 8.0.1 of the maven-plugin
Log file
An exception occurred while downloading updates. Please see the log file for more details. java.sql.BatchUpdateException: Batch entry 0 CALL merge_knownexploited('CVE-2021-27104','Accellion','FTA','Accellion FTA OS Command Injection Vulnerability','2021-11-03','Accellion FTA 9_12_370 and earlier is affected by OS command execution via a crafted POST request to various admin endpoints.','Apply updates per vendor instructions.','2021-11-17',NULL) was aborted: ERROR: procedure merge_knownexploited(character varying, character varying, character varying, character varying, character varying, character varying, character varying, character varying, character varying) does not exist
[ERROR]   Hint: No procedure matches the given name and argument types. You might need to add explicit type casts.
To Reproduce
Steps to reproduce the behavior:

Init postgres DB with script supplied for 8.0.1
Run version 8.0.1 with the updateOnly-goal

Any help with this is would be much appreciated!

Can you double check that you are using the latest DB initialization script for postgres? The procedure was introduced with 8.0.1.
https://github.com/jeremylong/DependencyCheck/blob/496ffc689ae588eea6e96a60c2ce02c9ca0fe518/core/src/main/resources/data/initialize_postgres.sql#L108

You pointed me in the right direction there. That script gave me error now when it wants to drop the procedure if it already exists. Turned out our DB also contained a function with that name but with different arguments so I must have used a script somewhere in the repo that added that one.
So when I manually dropped both of these functions and then ran the init-script again everything started working as expected.
Thank you for your quick assistance !


test: add another missing dependency

Signed-off-by: David Scott<EMAIL_ADDRESS>

I don't think the CI is running the tests of the PINS :( (otherwise this would have been caught before).

Perhaps we need to generalise PACKAGE into PACKAGES to cope with our multi-package repos?

Subsumed by #68


Apply inference in method arguments when using a discriminator

🔎 Search Terms
github: interface infer
github: interface discriminator
github: interface discriminator method
google: typescript method gets the result of another optional method
google: typescript one property sets the type of another property
google: typescript one property sets the type of the next property
🕗 Version & Regression Information
Happens from 3.3.3 (Removing NoInfer) up to the nightly version.
⏯ Playground Link
https://www.typescriptlang.org/play/?#code/JYOwLgpgTgZghgYwgAjgHgEoFEDKBGAPmQG8BYAKGSuRgFcQE8AKASgC5t8BuC6m+hACYA-GxAQAbtB6VqdBgGYmAEzhg4bAHIB7AJIgY0TLkLsJ24MpkBfCqEixEKAEbH8AGmSdBRMrKryjKwcJjJ8gYIqaho6+oZQbqYhOIJhcgJKqupaegZG3gRmFlYUtuRgAJ4ADigAtsAAHomeBcgAvKiJRAA+yK6ceC24PjIUgWDA2iA0zV7DBFHZ9U0DQymFJBSlY0x+4QLMLJv+fMhQEGC0UNN4ggq81NbuDwECkVlwR3unVOeX18gAES3BSAl7IJ7gwKZaJfZAAenhyAACnAoHBahdoMgAOQfHHIYC1KoAG2ACGAYBJFWQAAs4ABnVDTHFwEAVAmVGoAOi2JzK1hYXCAA
💻 Code
interface a<RES1> {
    func1():RES1;
    func2?:never;
    func3(data:NoInfer<RES1>):void;
}
interface b<RES1, RES2> {
    func1():RES1;
    func2(data:NoInfer<RES1>):RES2;
    func3(data:NoInfer<RES2>):void;
}
type mix<RES1, RES2> = a<RES1> | b<RES1, RES2>;

function f<RES1, RES2>(data:mix<RES1, RES2>) {

}
f({
    func1() {
        return 123
    },
    func2(data) {
        return "123"
    },
    func3(data) { // Parameter 'data' implicitly has an 'any' type.

    }
});

🙁 Actual behavior
It happens that func3 argument is not properly inferred as is trying to infer the a interface, although it doesn't apply because the requirement for that interface is to explicitly not have a func2 property.
If I change both func3 arguments to the same type (RES1 for example) it works, so it is actually getting both interfaces.
🙂 Expected behavior
I guess that it should first get the applicable interfaces, and then try to infer the values depending on the applicable interfaces only.
Additional information about the issue
Still, I'm not sure about this approach. What I want is to have a func3 method that receives the result of func1 or func2 depending if func2 exists or not, like a transformer, but everything inside the same interface.

Kind of looks like another one for the #47599 pile.

Yeah, this would require an unbounded number of passes (or maybe one pass per type argument, but it's unclear which order you'd use)


Mock requireNativeComponent instead of individual components

Summary

I believe this PR makes the native component mocks clearer in test output, and I wanted to start a dialog about it. I also believe this is a better approach because it should work for third-party native components out of the box as well, with no effort for the component maintainers.
The problem I believe it solves is that current state, snapshot output can be a little confusing. For instance, if I render a TouchableOpacity, my snapshot will be a View. That, to me, is confusing because:

I don't see my TouchableOpacity
I definitely didn't render a View

If, however, what I see is an RCTView with native props I maybe don't recognize, that, at least to me, more clearly communicates that what's in the snapshot is a native element--a primitive. It's similar to how react-dom developers wouldn't be confused if their Link component showed up in a shapshot as an a tag. It's implied that the snapshot output is the browser primitive that React rendered.
I think this makes it more clear that what shows up in a snapshot is a native primitive. I believe that ultimately, this will help users more clearly understand what's going on when components bridge from React to Native.
As an added benefit, once it was refactored this way, I just didn't need individual component mocks. I really believe this angle will make the mocks more robust and resilient, even if it takes some iteration to get to the finish line.
Changelog

[General] [Changed] - Changed requireNativeComponent mock
[General] [Removed] - Removed component mocks
Test Plan

I know this creates a big snapshot diff, but I think it's great because they now (to me at least) more clearly convey what's being sent across the bridge. I think this makes the snapshots more clear and reflective of what's going on at the native level, which I will ultimately help developers debug and test their apps.
The tests didn't really need changed, all I did was update the snapshot files now that the actual native components are being mocked. No warnings or console logs were introduced to the logs either, so I'm assuming we're peachy there 👍

I just wanted to clarify, I don't think this should be merged yet. I wanted to get some feedback on it before I spend too much more time on it. In order for this to be ready to merge, I'll need to go through and update the wording on some tests to accurately describe what they do now.

I don't think this is quite the approach we'd like to take.
I'd like to avoid having the implementations of the components built into React Native leak into product's snapshot tests. For example, if we get to a semver world, if we change the nesting inside of TextInput or something, that shouldn't be a breaking change. This is why we mock the components exported by React Native, so you end up with <TextInput ... /> in your snapshot test.
I think that requireNativeComponent should definitely be mocked as well in case someone doesn't add a custom mock for that component, but it should be the fallback.

This explanation makes sense to me, but let me give you another example of why I think this is a good approach. I have a set of existing tests that I ran with these new mocks, and it caught multiple instances of props that were incorrect or invalid since the component being used in the test was the “real” component. These invalid things were missed with the current mocks because they’re, well, mocks.
I could maybe see an argument that it’s leaking implementation details, but I think it might be the opposite; I think it’s communicating exactly what’s actually happening and not leaking implementation, which I believe is more beneficial to people learning the framework at a more intermediate to advanced level. If this were mocking things like Button or Touchable, I would agree because those are definitely implementations, but this is only mocking the actual native components and communicating things in a more realistic way.
To piggy back on your example of semver, I think that argument actually works in favor of this approach too — users might be relying on the core components working a certain way and their snapshots breaking could help them identify where issues lie. I think this would also help ease upgrades because it could point out where things may have broken at the native level between versions.
I understand if you guys don’t want the jest preset bundled with the library to be this way, but I really do think it’s beneficial. It makes the test environment more like the actual native environment, which will ultimately give users more confidence in their apps (especially in cases like the example I gave in the first paragraph).

Thanks for the additional info.

I have a set of existing tests that I ran with these new mocks, and it caught multiple instances of props that were incorrect or invalid since the component being used in the test was the “real” component.

Do you have some concrete code examples of this? I’m not exactly sure what this means so it would be helpful to see some examples.
I think something we are missing today that we are kinda talking around is the need for these unlocked snapshot tests in core. We don’t currently have those which means our downstream users don’t have confidence that we didn’t break things. Ideally that confidence would exist and mocks for our exports would be reasonable. I’d prefer we strive towards that goal.
However, people will always want to mock at different levels based on their needs and trade offs. I think this conversation is mostly just revisiting where the defaults should be. I think I’d prefer to stick with the current defaults and see if we can sufficiently improve them rather than expose a much deeper level.

When I get home I’ll send some examples ☺
I guess for me, you can remove a react developer from the DOM, but you can’t remove the DOM from the react developer. So what are those primitives and how do you very closely imitate them in tests? The core wrappers around requireNativeComponent or the native component itself?
Regardless of which answer is right, I think 1) some significant work needs done to the bundled mocks 2) teams who aren’t on the latest version of RN should be able to take advantage of them. I’d love to help with that, whatever it ends up looking like 👍
I took this approach because I think it’s easier, just as good as building better individual mocks, and almost exactly resembles the way the app works at a native level. But I digress, count me in either way 💯

I had a big thing written but I think I convinced myself I'm wrong and I hate the way these snapshots are as well. Like I promised though, here's a good example to answer your question from earlier:
<Image source="/finding-nemo.png" />

In the current mocks, this was no problem. Once I changed it to the way it is in this PR, I got a propTypes warning because this isn't a valid value for an Image source. That was one of the things that had me super stoked about this change. So, this part is still better I think.
BUT. The way this PR makes the snapshots is ugly. Using my own argument against myself, if I render this in the React: <input onChange={() => console.log('sup')} /> I'm going to get <input /> in a DOM snapshot. This is much better because who cares about event listeners in a snapshot? Also they aren't going to literally be in the DOM. It's also not misleading and shows me something that makes sense and I would have expected. Let me give you an outrageous snapshot example:
Given that I render this:
<View>
  <TouchableOpacity onPress={this.fetch}>
    <Text>Fetch</Text>
  </TouchableOpacity>
  {data ? <Text>{data.greeting}</Text> : null}
</View>

this was my snapshot output (🤢):
<View>
  <View
    accessible={true}
    isTVSelectable={true}
    onResponderGrant={[Function]}
    onResponderMove={[Function]}
    onResponderRelease={[Function]}
    onResponderTerminate={[Function]}
    onResponderTerminationRequest={[Function]}
    onStartShouldSetResponder={[Function]}
    style={
      Object {
        "opacity": 1,
      }
    }
  >
    <Text>
      Fetch
    </Text>
  </View>
  <Text>
    hello there
  </Text>
</View>

This is sure to be an "uhhh wow okay, so where's my onPress?" moment. And this PR would make that even worse? Yeah, that'd be a no from me too 👎 You said some things that convinced me that the correct snapshot output for the above example would be:
<View>
  <TouchableOpacity>
    Fetch
  </TouchableOpacity>
  <Text>
    hello there
  </Text>
</View>

What are your thoughts on working toward something like this? It might be hard to get to a place where this output is good enough to give confidence and not be confusing, but does this seem right to you?

Yay, thanks for providing some examples. Yeah, I agree those definitely don’t look pretty. For some other particularly nasty cases you might try a TextInput or ScrollView. Those both conditionally use 1 of something like 4 native components internally, and we hope with Fabric that will change to just one without breaking any functionality.
Another situation might be a test with FlatList that ends up rendering some complex cells and scrollviews.
I think there is definitely room for each component in core to have some unmocked snapshot tests like the ones you touched for ActivityIndicator in this PR. If there were PRs for those tests for more components we’d merge them right away.
For the example you gave, I think I agree, but isn’t that also how it works today? I guess I’d expect to see the onPress there. For example, if you conditionally attach a function to a prop, I’d expect to see it in a snapshot test and for it to fail if it is passed incorrectly.
You make a good point about validation though, there might be some things we want to validate that we are validating when not using the mocks. I’m not sure propTypes is a good example though as we’ve been removing those from the codebase in favor of flow/typescript. Maybe a better example is something like how we sided to error when a View was nestled inside a Text. I wonder if there are more of those we want to be able to validate at test time.

@TheSavior I just force pushed an update based on our conversation, let me know what you think. I think this may be a better starting place for iterating.
The difference in approach here is that rather than focusing on mocking requireNativeComponent I just made sure that all components that call requireNativeComponent are mocked appropriately. To me this makes sense--rather than requireNativeComponent being the primitive, this makes any core component that calls it the primitive.

Sorry just saw this.
1 - Yep I can definitely help with that!
2 - I pushed a few commits that I think actually get us closer to where we wanna be with this step than anything I've done so far. I think I basically finished step 2 before I knew it would be step 2.
3- With every component mocked, the biggest changes are to Lists, which I expected, they're the most intense. The coolest difference is that what shows up in snapshots will be what developers passed in (no leaky implementations 😉)
4 - The ScrollView/List snaps changed a lot with this PR, but it had more to do with other things being mocked differently than the lists themselves. The ScrollView is still mocked and the RefreshControl is mocked as well, just following the format of the others. latestRef doesn't show up in the repo as far as I could tell, so maybe you could tell me more about what that is/if it still matters?

Awesome, so I think this PR can turn into #2, but I'd want to merge all of #1 first because this might change the snapshots in the test for #1.
For #4, It looks like at Facebook we have one callsite that uses latestRef, doing this:
RefreshControl.latestRef.props.onRefresh();

It doesn't seem like this is the right way to test that, so we'll need to update that test if we are going to change it.

@TheSavior you know, as I'm working on task 1, I'm realizing that it's going to make the diff for this really really big once it's completed.
I want to have a plan of attack because task 1 is important for the core team since it'll help you make sure updates aren't breaking components, but task 2 is important for the community because it'll make snapshots look like what they're expecting in their app tests.
This is a huge topic and this might not be the right place for it, but should the community have the same Jest config as the core team? 🤔 As I'm thinking about it, I actually could see (from the perspective of the core team) my very first PR being the most useful because it'll show you the native render result which would give you more confidence you aren't accidentally breaking the components.
But for the community, they don't care about that. We could give them a Jest preset that shows clean "high-level" output they're expecting to see with things like onPress for touchables, and repurpose this preset to be one just meant for super low-level snapshots like native components.
More confidence for the core team, more confidence for the community. Thoughts?

Fun fact, the jest config in this repo is the same jest config that is used for products using RN at Facebook. My thinking for step one is that it includes snapshots for components the way products using RN would see them, as well as the unmocked version that we want to test the core. We don't want to unintentionally break either of those situations and thus would want tests for both.
I do think we want the snapshot tests first, because that will give us confidence in the changes that step 2 is causing. Otherwise we are making changes to how people's snapshot tests will appear without knowing what those changes are and that we are getting them right. As we mentioned on one of your other PRs, we should feel free to break up that work so that it isn't one large impossible to review PR. :)
Does that answer the questions?

Yep, I'm on the same page with all of that stuff, I think this was more of a future facing question. I'll continue down the same path with the other PRs to add current-state snapshots, just wanted to think about the implications it would have if we were to do something like that. No matter how much its broken up, it'll still be a big change.
I think for those building apps, it's beneficial to see:
<TouchableOpacity onPress={[Function]}>

because it reflects what they rendered. But for the internal library snapshots, my question is does it make more sense to see:
<RCTView>

(with all of the onGesture events that I don't wanna type, lol). Because that would ensure the core components are rendering the native components correctly without them being "mocked".

But for the internal library snapshots, my question is does it make more sense to see...

Aren't the snapshot tests in the other PR showing us that? But also showing us what product developers will see? At least, that's the point of having a test with them mocked and not mocked.

Hey @bcarroll22 what's the status of this PR?

@cpojer lets close it for now, until I get some other stuff done per my conversations with @TheSavior this can't really move forward until some other things are wrapped up.


Setting visibleGameSize for CameraComponent.withFixedResolution raises Assertion failed exception

Current bug behavior

The game can't run because of this exception:

Expected behavior

It used to work fine on Flame 1.6.0.
If it's a breaking change, it should be documented somewhere to know how to set visibleGameSize correctly.
Steps to reproduce

Minimal code to reproduce: https://zapp.run/edit/flame-zweq06qwwer0
Flutter doctor output

Output of: flutter doctor -v

More environment information

Flame version: main
Platform affected: desktop

Log information

Enter log information in this code block

More information

Thank you for posting a good MRE.
It looks like the problem is caused by the fact that camera.viewport.size is zero at the point when set visibleGameSize is called. Not sure what caused this bug, if it used to work in 1.6.0

@st-pasha I did a git bisect and #2317 is where the issue is introduced.

I converted it to a test for easier debugging:
    testWithFlameGame('can change visibleGameSize directly', (game) async {
      final world = World()..addToParent(game);
      final cameraComponent = CameraComponent(world: world)..addToParent(game);
      expect(
        () => cameraComponent.viewfinder.visibleGameSize = Vector2(100, 100),
        returnsNormally,
      );
    });


Adding live examples and code samples in documentation

Please add sample code and live example links  for each type of chart in documentation.. So that newbie would easily get idea about this js library.. And many more enhancements needed for current documentation.. It's not clear enough for newbies.

@shivarajnaidu are there any specific things about the documentation that are unclear?
There are already a number of sample files in the samples folder but we can post them online. #2018 discusses some of the enhancements to the sample files

Why you can't post them .. However if you can't post them.. At least give a Link for samples on official website documentation.. Without samples its hard to use library for newbie

Will be done as soon as new site is posted

Codepen supports embedding pens so that could be a viable option here.

Samples will be available online starting version 2.6, see #4256


adding vendor device IDs for powerbood trackpad and logitech M305

I found the trackpad ID for my macbook was not in the product def list. Additionally, I did not find the vendor product ID for my Logitech M305 mouse. I have added both. Attached is a screen shot of the event viewer showing the product IDs.
Macbook details: "Model Identifier: MacBookPro12,1", retina 13", early 2015

Thanks!


veeam B&R : ERROR while removing instance from backup offering

ISSUE TYPE

Bug Report

COMPONENT NAME

Logging

CLOUDSTACK VERSION

4.20-SNAPSHOT

CONFIGURATION

Veeam B&R provider configured with Veeam Backup and replication 12.2
SUMMARY

On removing the backup offering on an instance, although it gets removed there is an error in the management server log. Async job result in success.

2024-11-06 07:38:54,643 DEBUG [o.a.c.b.v.VeeamClient] (API-Job-Executor-22:[ctx-f25b54d6, job-723, ctx-fd699c6a]) (logid:c2072895) Response received in GET request is: [HttpResponseProxy{HTTP/1.1 400 BadRequest [Cache-Control: private, Content-Length: 264, Content-Type: application/xml, Server: Microsoft-HTTPAPI/2.0, X-Frame-Options: SAMEORIGIN, X-XSS-Protection: 1; mode=block, Strict-Transport-Security: max-age=31536000, X-Content-Type-Options: nosniff, Date: Wed, 06 Nov 2024 07:38:54 GMT] ResponseEntityProxy{[Content-Type: application/xml,Content-Length: 264,Chunked: false]}}] for URL: [https://<IP_ADDRESS>:9398/api//jobs/null/includes].
2024-11-06 07:38:54,643 DEBUG [o.a.c.b.v.VeeamClient] (API-Job-Executor-22:[ctx-f25b54d6, job-723, ctx-fd699c6a]) (logid:c2072895) HTTP request failed, status code is [400], response is: [HttpResponseProxy{HTTP/1.1 400 BadRequest [Cache-Control: private, Content-Length: 264, Content-Type: application/xml, Server: Microsoft-HTTPAPI/2.0, X-Frame-Options: SAMEORIGIN, X-XSS-Protection: 1; mode=block, Strict-Transport-Security: max-age=31536000, X-Content-Type-Options: nosniff, Date: Wed, 06 Nov 2024 07:38:54 GMT] ResponseEntityProxy{[Content-Type: application/xml,Content-Length: 264,Chunked: false]}}].
2024-11-06 07:38:54,643 DEBUG [o.a.c.b.VeeamBackupProvider] (API-Job-Executor-22:[ctx-f25b54d6, job-723, ctx-fd699c6a]) (logid:c2072895) VM was removed from the job so could not remove again, trying to delete the veeam job now. org.apache.cloudstack.api.ServerApiException: Got invalid API status code returned by the Veeam server
        at org.apache.cloudstack.backup.veeam.VeeamClient.checkResponseOK(VeeamClient.java:215)
        at org.apache.cloudstack.backup.veeam.VeeamClient.removeVMFromVeeamJob(VeeamClient.java:573)
        at org.apache.cloudstack.backup.VeeamBackupProvider.removeVMFromBackupOffering(VeeamBackupProvider.java:201)

STEPS TO REPRODUCE

1. Remove the backup offering from instance.
2. Watch the logs.

EXPECTED RESULTS

There shouldn't be any error in the management server log while removing an instance from the backup offering.

ACTUAL RESULTS

There is an unexpected error in the management server log while removing an instance from the backup offering.

@rajujith
just to confirm, irrespect to the error log, the operation still succeeded, right ?

@rajujith just to confirm, irrespect to the error log, the operation still succeeded, right ?

Yes.


Clipboard dependency unmaintained

This crate depends on https://github.com/aweinstock314/rust-clipboard which is unmaintained. Which also depends on https://github.com/quininer/x11-clipboard/issues/19 that unmaintained to. Which also depends on xkb which is unsound and also unmaintained :rofl:
Can we switch to a different crate for clipboard management?

Hi! I'm glad to replace it with something else, I just haven't found any suitable alternative. Any ideas for the replacement? Or maybe we'll end up implementing the functionality ourselves :)

@mvlabat I looked for libraries and found only three options:

https://github.com/ArturKovacs/arboard. It's already much better then clipboard, but the author is looking for a maintainer. We could switch to it in a short term, the author is active, pretty sure the maintainer will appear.
https://github.com/hecrj/window_clipboard. Experimental, but active :)
There is a fork of clipboard crate that we use: https://github.com/mcobzarenco/crossclip. It's maintained, but it still uses x11-clipboard that I mentioned in the first message. So the solution could be switch to this fork and send a PR with a replacement for x11-clipboard. I found only this crate: https://crates.io/crates/clipboard_x11 (it's from the repo I linked in 2.)

@Shatur thank you a lot for the research! I've just pushed a PR switching to arboard (#72), and it looks pretty good. Feel free to review and leave any comments there, I'll also be glad to have people testing it on different platforms.
I'm planning to release it with the next Egui upgrade, including it into the next minor release.


fix #93: return instance of prefsWindow

Return the preferencesWindow from preferences.show()

@pvrobays this was a quick solution, I think having a getter method would probably be a better route.

@lacymorrow It's OK for now, I'm thinking of creating a preference for the always on top.
Do you need these changes to be released ASAP?

This is blocking my next release for https://github.com/lacymorrow/crossover,
I don't mind using a fork in the meantime so I'm not in any rush but I would like to release somewhat soon :)


Significant difference in RSS memory usage between TF1 and TF2

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
TF 2.13.1
Custom code
Yes
OS platform and distribution
Redhat Enterprise Linux 8.9
Mobile device
No response
Python version
3.11.4
Bazel version
5.4.0
GCC/compiler version
10.4
CUDA/cuDNN version
CUDA 12.2, cuDNN 8.9.5
GPU model and memory
A100 80GB
Current behavior?
When running the same keras workload on TF1 vs. TF2, I'm seeing a significant increase in memory utilization per epoch. This happens when using both CPUs/GPUs. The utilization for an epoch climbs up significantly after every epoch. See below for TF1 vs. TF2:
# For TF2

Memory usage after epoch 0 [mem_usage =  3.41 GB]
Memory usage after epoch 1 [mem_usage =  3.88 GB]
Memory usage after epoch 2 [mem_usage =  3.88 GB]
Memory usage after epoch 3 [mem_usage =  4.32 GB]
Memory usage after epoch 4 [mem_usage =  4.81 GB]
Memory usage after epoch 5 [mem_usage =  5.26 GB]
Memory usage after epoch 6 [mem_usage =  5.70 GB]
Memory usage after epoch 7 [mem_usage =  6.14 GB]
Memory usage after epoch 8 [mem_usage =  6.70 GB]
Memory usage after epoch 9 [mem_usage =  7.15 GB]
Memory usage after epoch 10 [mem_usage =  7.36 GB]
Memory usage after epoch 11 [mem_usage =  7.36 GB]
Memory usage after epoch 12 [mem_usage =  7.36 GB]
Memory usage after epoch 13 [mem_usage =  7.37 GB]
Memory usage after epoch 14 [mem_usage =  7.37 GB]
Memory usage after epoch 15 [mem_usage =  7.37 GB]
Memory usage after epoch 16 [mem_usage =  7.37 GB]
Memory usage after epoch 17 [mem_usage =  7.59 GB]
Memory usage after epoch 18 [mem_usage =  7.81 GB]
Memory usage after epoch 19 [mem_usage =  7.81 GB]

# For TF1

Memory usage after epoch 0 [mem_usage =  5.13 GB]
Memory usage after epoch 1 [mem_usage =  5.14 GB]
Memory usage after epoch 2 [mem_usage =  5.14 GB]
Memory usage after epoch 3 [mem_usage =  5.15 GB]
Memory usage after epoch 4 [mem_usage =  5.15 GB]
Memory usage after epoch 5 [mem_usage =  5.15 GB]
Memory usage after epoch 6 [mem_usage =  5.15 GB]
Memory usage after epoch 7 [mem_usage =  5.15 GB]
Memory usage after epoch 8 [mem_usage =  5.15 GB]
Memory usage after epoch 9 [mem_usage =  5.15 GB]
Memory usage after epoch 10 [mem_usage =  5.15 GB]
Memory usage after epoch 11 [mem_usage =  5.15 GB]
Memory usage after epoch 12 [mem_usage =  5.15 GB]
Memory usage after epoch 13 [mem_usage =  5.15 GB]
Memory usage after epoch 14 [mem_usage =  5.15 GB]
Memory usage after epoch 15 [mem_usage =  5.15 GB]
Memory usage after epoch 16 [mem_usage =  5.15 GB]
Memory usage after epoch 17 [mem_usage =  5.15 GB]
Memory usage after epoch 18 [mem_usage =  5.15 GB]
Memory usage after epoch 19 [mem_usage =  5.15 GB]

Standalone code to reproduce the issue
import tensorflow as tf
import psutil
import time
import os

def mem_usage_str():
    process = psutil.Process(os.getpid())
    gb =  process.memory_info().rss / (1024.**3)
    return ' [mem_usage = {:5.2f} GB]'.format(gb)

if int(tf.__version__.split('.')[0]) < 2:
    """
    Patch to fix TF/numpy1.20 compatibility issue
    """
    from   tensorflow.math          import reduce_prod
    from   tensorflow.python.ops    import array_ops

    def _constant_if_small(value, shape, dtype, name):
        try:
            if reduce_prod(shape) < 1000:  # monkey patch
                return array_ops.constant(value, shape=shape, dtype=dtype,
                                          name=name)
        except TypeError:
            # Happens when shape is a Tensor, list with Tensor elements, etc.
            pass
        return None

    array_ops._constant_if_small = _constant_if_small
    """
    End of patch
    """

def build_model():
    inputs = [tf.keras.layers.Input(shape=(300, 6), name='input_layer')]
    current_layer = inputs[0]

    current_layer = tf.keras.layers.LSTM(
        50,
        dropout=0.1,
        recurrent_dropout=0.1,
        return_sequences=False,
        name='lstm',
    )(current_layer)

    current_layer = tf.keras.layers.Dense(1)(current_layer)
    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

    model = tf.keras.models.Model(inputs=inputs, outputs=current_layer)
    model.compile(loss='mse', optimizer=optimizer)

    return model

def run(model, X, y, n_epochs):
    tot_time = 0.

    print(f'Memory usage before training' + mem_usage_str())
    for i in range(n_epochs):
        start = time.time()
        model.fit(X, y, epochs=1, batch_size=4096, verbose=0)
        tot_time += time.time() - start
        print(f'Memory usage after epoch {i}' + mem_usage_str())

    print(f'Avg. time = {tot_time / n_epochs} seconds')

def run_example(p, n_epochs):
    import numpy as np

    model = build_model()
    X = np.random.randn(2 ** p, 300, 6)
    y = np.random.randn(2 ** p)

    run(model, X, y, n_epochs)

def main():
    run_example(
        16, # 2 ** 16 samples
        20, # 10 epochs
    )

# ------------------------------------------------------------------------------

if __name__ == "__main__":
    main()

Relevant log output
No response

@SuryanarayanaY, please take a look at this ASAP.

If it helps, running it through memray and generating a flamegraph with --leaks, we see the above as having ~18 GB memory allocated from 27k allocations. So, likely that something is up with those.

Hi @bergentruckung ,
You need to call tf.keras.backend.clear_session() in the loop to remove the global states generated by keras. Could you please add this at the beginning of epoch and let us know.
Thanks!

Hi @bergentruckung ,
I did replicated the behaviour with Keras3-nightly also.Attached gist for reference. Not sure whether this needs to be addressed in Keras repo or here itself. Could you please open an issue at Keras repo also.

Hi @SuryanarayanaY,
I think your suggested workaround may work for newer TF that has changes from https://github.com/tensorflow/tensorflow/pull/62154 pulled in. This wasn't cherry-picked onto 2.13 branch.
Could you backport that PR to 2.13?

@bergentruckung Could you please upgrade to the latest if possible as backported changes in 2.13 may introduce new bugs or regressions. It's unlikely for older versions to receive any bug fixes except when we have security patches.
Thank you!

We're building from source. We were able to build 2.14, but that has cuBLAS open issue that's problematic for us (the fix was to go back to 2.13 and that's what we did). We couldn't find a proper closure of dependencies for 2.15, and that's why we were trying previous versions.

@bergentruckung Could you please let us know if you have created another issue in Keras repo as suggested here?
Thank you!


Bug 1541335 - fix auth config parsing

Added a test but it requires some directory to be installed on your desktop. I marked it as skip for now.

TEST
See plan from other PR
https://github.com/openshift/ansible-service-broker/pull/729#issuecomment-363230398


iOS : SSL Pinning using custom certificate from Native  code

Hi,
We want to implement SSL pinning from Native iOS side so we require to override the RNCWebView.m class  didReceiveAuthenticationChallenge method. We are not able to override the below method because it’s not exposed to use outside of the class.
Is it possible to make it didReceiveAuthenticationChallenge method as a class method and expose to use in AppDelegate class then we can override the and use a custom certificate for SSL pinning?
- (void)webView:(WKWebView *)webView
  didReceiveAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge
                  completionHandler:(void (^)(NSURLSessionAuthChallengeDisposition disposition, NSURLCredential * _Nullable))completionHandler

Trying to achieve: When load URL on Webview we want to use a Custom certificate for SSL pinning.
Please let me know any other possibility to use custom SSL pinning from iOS native side.

Hi,
is there any update about it?


Move NAMB info to about us

move NAMB info to about us

So, do we want to move this line:
We are partnered with the NAMB, the UISBC, and the Salt Lake Baptist Association.
from the bottom of the What We Believe page to the bottom of the About Us page?


GLES2 Depth Texture not reaching shader material

Godot version:
3.1 beta 11
OS/device including version:
Arch Linux / Intel Graphics
Issue description:
Seems that the depth texture is not passing to the shader in GLES2 backend and even seems like using DEPTH_TEXTURE is interfering with z order.

Steps to reproduce:

Create a plane with a cube behind
Create a ShaderMaterial for plane
Output depth texture to Albedo.

Minimal reproduction project:
depth_texture_test.zip

Why close?

Why close?

Sorry I figured that depth texture is reaching the shader, but still there is a problem with the z order or something because part of the cube(above the plane) isn't rendering at all.

This is caused only when used in custom shaders, Depth works fine with standard spatial shader (used in Proximity Fade).
Proximity Fade disabled

Proximity Fade enabled

@akien-mga @faculezcano  I was looking at material.cpp and I found a solution ( or I would say the bug). and this is the work around:
shader_type spatial;

uniform float proximity_fade_distance = 5.0;

void fragment () {
	ALBEDO = vec3(0, 1, 0);
//	float depth_tex = textureLod(DEPTH_TEXTURE,SCREEN_UV,0.0).r;
	float depth_tex = texture(DEPTH_TEXTURE, SCREEN_UV).r;
	vec4 world_pos = INV_PROJECTION_MATRIX * vec4(SCREEN_UV*2.0-1.0,depth_tex*2.0-1.0,1.0);
	world_pos.xyz/=world_pos.w;

	// This is the magical part, I have used a value of 0.001 so it's a tiny clipping margin.
	float depth_clip = clamp(1.0-smoothstep(world_pos.z+0.001,world_pos.z,VERTEX.z),0.0,1.0);
	ALPHA *= depth_clip;

	// Do what ever you want with your depth (like proximity fade in this case)
	float depth = clamp(1.0-smoothstep(world_pos.z+proximity_fade_distance,world_pos.z,VERTEX.z),0.0,1.0);
	ALBEDO = vec3(depth);
}

I hope this help fixing it.

Currently when reading from DEPTH_TEXTURE depth testing is disabled. This is because, for GLES2 compliant hardware, it is not possible to do both and will result in undefined behaviour at best, and crash the graphics drivers at worst.
So unfortunately, in order to preserve compatibility with older hardware this cannot be implemented. @OBKF's workaround is your best option for fixing this problem.

@OBKF did you run into any problem where the direction of the camera re creates this clipping issue? I seem to be running into this issue everytime the camera lookat is at the -z . Can you help me with this issue? It only occurs when multiple depth texture reading shaders overlap.
This is the ideal shading.

This is the issue. Can anyone help with this issue?

I think it might be due to the camera matrix changing the projection depth but I'm not sure about it. a little code snippet on getting the world space depth independent of the camera transform might help. I can't seem to find any reference on that topic in godot.


Long paths conceal GDScript warnings

Godot version:

3.2 branch at commit 0b5a9bf7167e393e0b42a18383cccddd8b008d1f
OS/device including version:

Arch Linux
Issue description:

On a template I'm working on, I moved a deep folder hierarchy to the root of the project to make it easier to access. All of a sudden I'm now seeing warnings never shown previously.
Steps to reproduce:

Open the attached project with the Editor
Run the project (F5) and stop it (F8). Look at the Debugger -- no warnings.
Using the File System tab, move the autoload folder at res://addons/pragmatic-godot/autoload to res://
Run the project (F5) and stop it (F8). Look at the Debugger -- 27 warnings.

Minimal reproduction project:

concealed_warnings_bug.zip

Scripts under addons/ have warnings excluded by default. You can disable this behavior in the Project Settings.

I see. Thanks.


JsonParser in combination with Flowable does not return Parsing exception in error case

Hello,
we use the JSONParser in combination with Flowable in the request flow.
The user sends JSON documents and it is processed by the JSONParser.
If the user sends corrupt JSON, the process usually got an exception via onError and could sends a 400 error to client. After updating to Vertx3.8.1, the onError never triggered and the request goes into a timeout.
We investigated the issue and could recreate it in a simple example (see attachment)
The package contains a Unit Testcase that loads a JsonFile as ReadStream and process it with JSONParser.
Can you please check out and let us know if we need to change the usage the vertx code need to change?
Steps to recreate:

Download and extract project
configure dependency vertx 3.5.4 or vertx 3.8.1
Excecute parserDemo.ParserTest

Expected Result: Test throws parsing exception
Actual Result (3.8.1): Test run in timout
Non-Affected version (tried): 3.5.4
Affected version (tried): 3.8.1, 3.8.2
recreateFile.zip
We analyzed the issue further and found that the usual error processing is happen in the JSONParser and is pushed to the RX stream exception handler FlowableReadStream. The exception handler processes the release method and null the handler of the JSONParser. Afterwards the release method triggers the fetch again and when the JSONParser continue to process, the exception handler is null and so the JSONParser throw the exception directly. Our assumption is that the nulling of the Parser and trigger the processing crashs the complete flow and prevent to trigger the OnError event. An idea is may to instead of null the handler, just set a NOOPHandler as exception handler in Flowable scenario.
Let me know if you need any further information. Thanks. @vietj

thanks for the explanation too

I think we need to look at two things:

the stream resume failure should be caught in vertx-rx
in JsonParser since we are already parsing, the checkPending() should not trigger parsing

I will do a fix for the vertx-rx and hopefully it will solve the solution, the second point might take more time to study.
Also likley the handle method should not throw an exception and the error should always be done on the exception handler whatsoever.

see https://github.com/vert-x3/vertx-rx/issues/212

see also https://github.com/eclipse-vertx/vert.x/issues/3149

thanks for the reproducer and thorough analysis.


Not working with IE11

It seems to me that this package is not working in the old IE11 browser. Can you confirm that?

May be due to function declaration in main file, should be fixed in 1.1.2


[Bug] cannot reach redis...

Since I tried to update from v2.3.0 to newer versions v2.3.1, v2.3.2 and v2.3.3 I'm getting 'cannot reach redis' error.
Going back to v2.3.0 it's working again.
Here is container registry of the error with 'RUST_BACKTRACE=1': cryptgeon.csv
I'm deploying the container with this docker CLI command: docker run -d --name=cryptgeon -e MEMCACHE=”memcached:11211” -e SIZE_LIMIT=4MiB -p 8000:8000 cupcakearmy/cryptgeon; if you have any idea to resolve this issue, I'll be pleased to test it...
Thank you!!!

wait... version 2 moved from memcached to redis, memcached should not work xD

I can confirm that it works, because I have it working with v2.3.0...
I'll change the stack to use redis with latest version ASAP...

Thats... interesting to say the least xD
Maybe they have a similar enough api? it really should not be working with memcached xD


vscode.FileDecoration color overides gitDecoration color on folders, even when set to null

Hi,
I have been using this extension to add colors, badges and tooltips to the folders and files in my repo. The extension uses vscode.FileDecoration to add a color, badge and tooltip to a given path. It works well except that it overrides gitDecoration colors (for added and modified files) even if I dont provide a color argument. The docs I linked indicate that the color argument is nullable so the behavior I would expect is that if no color is provided the badge and tooltip will be applied but the color will be unaffected. This however is not the case, it seems a default color is used if none is provided.
This is frustrating because it makes it difficult to see what files have been added or modified, especially if they are within a collapsed folder. Essentially I want to be able to use the badge and tooltip features of vscode.FileDecoration without affecting the color. It would be great if this functionality could be added or if an alternative solution/workaround is available let me know. Thanks!
Jacob

It's not really defined, today extension decorations get sorted with the same weight and the first (which is arbitrary) defines the color - even if that's falsy.
I am not 100% sure but I believe we can relax that rule and take the first defined color.

Cool, it sounds logical to me that falsy values should be ignored and the first explicitly defined color takes precedence. Then I can use the FileDecoration class without providing a color argument and it shouldn't override the git colors :)

Alternatively, could there be a way to set the weight/priority of a decoration? It would be nice to use color decorations but have the git colors override my personal colors rather than the other way around.

could there be a way to set the weight/priority of a decoration

We generally dislike explicit weights/prios because an unfair player can use high values and even a fair player wouldn't know what values to use because it isn't know what values are used by other extensions. I believe "falsy yields the floor" is the better model here

Okay, I'm good with the "falsy yields the floor" solution. Just to clarify though I was thinking the weights/prios would be set on an individual level, not by the extension creators. Like somewhere in my settings.json I could have an array that ranks my extension priorities. Everyone could have their own unique set of weights.

Hi, I have the same problem with an extension I'm creating. If I just add tooltips to the extension, it still overwrites all the Git colors.
Before

After

The only colors visible in the images are for problems or warnings, but the rest is all white. The expectation would be that if the extensions haven't set a color, the existing ones shouldn't be replaced.

@pablotecheraa I don't know why your comment got marked as spam but I appreciate you demonstrating that this is a common issue and the current behaviour is not what's expected. I am not alone! 😁👍


Include line numbers

Is it possible to include automatic line numbers in the rendering? I'm unable to find information on how to do that.

You can to that by calling
\lstset{numbers=left}

(after having loaded pythonhighlight).
There are more options available with \lstset, see the listings documentation.


Explicit error about wrong instance password

Examine available prerequisite check should report wrong instance password usage explicitly (with the precise message).

not sure if needed. GAP is currently suggesting it in a common message about instance unavailability. In check runner GAP is time outing after 1 minute.


management/osimage: fix field ordering and set omitempty for optional fields

with the old ordering, the azure api would reject POST requests from
AddOSImage.

Hi @mischief, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;

forgot to mention i've added ShowInGui which is not listed as optional in the docs for the api call.

Closing as Private API. https://github.com/Azure/azure-sdk-for-go/pull/391#issuecomment-243882801