Build PipelineDB from source (v1.0.0) of PostgreSQL`s extension

Hello, developers, It's a pleasure to use this extension right away.
Here are such problems in installation this extension.
when execute command "make USE_PGXS=1" in pipelinedb extension (v1.0.0)
such error：
{
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -O2 -fPIC -I./include -I/home/postgresql/include -I. -I./
-I/home/postgresql/include/server -I/home/postgresql/include/internal -D_GNU_SOURCE
 -c -o src/aggfuncs.o src/aggfuncs.c
In file included from src/aggfuncs.c:12:0:
./include/analyzer.h:115:1: 错误：未知的类型名‘RawStmt’
 RawStmt *GetContWorkerSelectStmt(ContQuery* view, SelectStmt** viewptr);
 ^
./include/analyzer.h:116:1: 错误：未知的类型名‘RawStmt’
 RawStmt *GetContViewOverlayStmt(ContQuery *view);
 ^
src/aggfuncs.c:20:30: 致命错误：utils/fmgrprotos.h：没有那个文件或目录
 #include "utils/fmgrprotos.h"
                              ^
}

when execute command “make install”，such error：
{
src/aggfuncs.c:10:22: 致命错误：postgres.h：没有那个文件或目录
}

I can not find  fmgrprotos.h and postgres.h  in include directory.

@ercengsha do you have the PostgreSQL development binaries installed? You need those to build the extension.

@ercengsha do you have the PostgreSQL development binaries installed? You need those to build the extension.

OK. So, I will use ‘binaries installation’ way to installation PG rather than from 'Source code'.
And I will try again, Thanks for your timely reply @derekjn .

OK. So, I will use ‘binaries installation’ way to installation PG rather than from 'Source code'.
And I will try again,

@ercengsha you should still be able to build pipelinedb against a PostgreSQL built from source, as long as you've installed the PostgreSQL header files in the right location (make install when building PostgreSQL should do this).
Can you share the output of this?
pg_config

That should show you where the PostgreSQL headers are installed. Based on the output you shared, you should find postgres.h here:
find . /home/postgresql/include -name "postgres.h"

Is that header there?

@derekjn  I have seen the pipelinedb as an extension requiring PG 10.3 and up, but I did not see this restriction in the release(v1.0.0), so I use the test under the existing version 9.6.
PG_HOME=/home/postgresql/

pg_config
BINDIR = /home/postgresql/bin
DOCDIR = /home/postgresql/share/doc
HTMLDIR = /home/postgresql/share/doc
INCLUDEDIR = /home/postgresql/include
PKGINCLUDEDIR = /home/postgresql/include
INCLUDEDIR-SERVER = /home/postgresql/include/server
LIBDIR = /home/postgresql/lib
PKGLIBDIR = /home/postgresql/lib
LOCALEDIR = /home/postgresql/share/locale
MANDIR = /home/postgresql/share/man
SHAREDIR = /home/postgresql/share
SYSCONFDIR = /home/postgresql/etc
PGXS = /home/postgresql/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--prefix=/home/postgresql' '--with-openssl'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2
CFLAGS_SL = -fPIC
LDFLAGS = -Wl,--as-needed -Wl,-rpath,'/home/postgresql/lib',--enable-new-dtags
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lssl -lcrypto -lz -lreadline -lrt -lcrypt -ldl -lm
VERSION = PostgreSQL 9.6.10

PGXS = /home/postgresql/lib/pgxs/src/makefiles/pgxs.mk

but I find 'postgresh' in the path of ‘home/postgresql/include/server/postgres.h’  rather than ‘$PG_HOME/include/’

@ercengsha ah ok, are you able to build against 10.3?

@derekjn
Same problem here on Centos 7.
'postgresh' in the path of '/usr/pgsql-11/include/server/postgres.h' rather than '/usr/pgsql-11/include'
I installed PG11 using the yum repo from the official site.
[root@localhost pipelinedb-1.0.0]# /usr/pgsql-11/bin/pg_config
BINDIR = /usr/pgsql-11/bin
DOCDIR = /usr/pgsql-11/doc
HTMLDIR = /usr/pgsql-11/doc/html
INCLUDEDIR = /usr/pgsql-11/include
PKGINCLUDEDIR = /usr/pgsql-11/include
INCLUDEDIR-SERVER = /usr/pgsql-11/include/server
LIBDIR = /usr/pgsql-11/lib
PKGLIBDIR = /usr/pgsql-11/lib
LOCALEDIR = /usr/pgsql-11/share/locale
MANDIR = /usr/pgsql-11/share/man
SHAREDIR = /usr/pgsql-11/share
SYSCONFDIR = /etc/sysconfig/pgsql
PGXS = /usr/pgsql-11/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--enable-rpath' '--prefix=/usr/pgsql-11' '--includedir=/usr/pgsql-11/include' '--mandir=/usr/pgsql-11/share/man' '--datadir=/usr/pgsql-11/share' '--with-icu' 'CLANG=/opt/rh/llvm-toolset-7/root/usr/bin/clang' 'LLVM_CONFIG=/usr/lib64/llvm5.0/bin/llvm-config' '--with-llvm' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/usr/lib64' '--with-openssl' '--with-pam' '--with-gssapi' '--with-includes=/usr/include' '--with-libraries=/usr/lib64' '--enable-nls' '--enable-dtrace' '--with-uuid=e2fs' '--with-libxml' '--with-libxslt' '--with-ldap' '--with-selinux' '--with-systemd' '--with-system-tzdata=/usr/share/zoneinfo' '--sysconfdir=/etc/sysconfig/pgsql' '--docdir=/usr/pgsql-11/doc' '--htmldir=/usr/pgsql-11/doc/html' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,--as-needed' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
CFLAGS_SL = -fPIC
LDFLAGS = -Wl,--as-needed -L/usr/lib64/llvm5.0/lib -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/usr/pgsql-11/lib',--enable-new-dtags
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lpthread -lselinux -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lrt -lcrypt -ldl -lm
VERSION = PostgreSQL 11.0

And I rpm install the rpm file and added "shared_preload_libraries = 'pipelinedb'" into postgresql.conf, then I could not restart PG11 with error:
-- Unit postgresql-11.service has begun starting up.
Oct 25 11:06:33 localhost.localdomain postmaster[30742]: 2018-10-25 11:06:33.591 CST [30742] FATAL:  could not access file "pipelinedb": No such file or directory
Oct 25 11:06:33 localhost.localdomain postmaster[30742]: 2018-10-25 11:06:33.591 CST [30742] LOG:  database system is shut down
Oct 25 11:06:33 localhost.localdomain systemd[1]: postgresql-11.service: main process exited, code=exited, status=1/FAILURE
Oct 25 11:06:33 localhost.localdomain systemd[1]: Failed to start PostgreSQL 11 database server.

@vnnw it looks like you're using PostgreSQL 11. The specific versions PipelineDB currently supports are 10.1, 10.2, 10.3, 10.4 and 10.5. Building against any of those should resolve the build issues.


Improve build/test and submission instructions for v3

I am happy to work on this, but I figured it would be sensible to open an issue to submit against, and for opinions before doing so.
The build instructions have changed from v2 to v3, and some simple install/build/deploy instructions, or the beginnings of, could be useful for anyone looking to help.

Sure, will do.

Resolved via b93af2639cc322f64ad0f771e9d195ed01bdb3ff.


DOT Drop not identifying previously minted NFT

What happened?
go to -> https://canary.kodadot.xyz/ahp/drops/dot-drop
last one minted was me:

but when i enter the page, it lets me mint me again:

it is supposed to be like that for the DOT drop?

it is supposed to be like that for the DOT drop?

I'm not sure if it's allowed to mint items multiple times for dot dorp. Do you have any idea? @vikiival

Honestly, i dont know
But anything you can buy is possible to buy. Free drops are allowed as one per address

Honestly, i dont know But anything you can buy is possible to buy. Free drops are allowed as one per address

Thank you for your answer. I guess we could close this issue then.


Introduce Transformations Conditions

Why are these changes needed?
Introduces the option to set a specific condition to be checked in order for a transformation to be applied.
Example: a user may want to apply a MessageHistoryLimiter transformation only after the token limit of the model has been reached.
Related issue number
Closes #2306.
Checks

[ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[x] I've made sure all auto checks have passed.

Codecov Report
Attention: Patch coverage is 0% with 19 lines in your changes are missing coverage. Please review.

Project coverage is 17.30%. Comparing base (4a44093) to head (3936ae1).
Report is 19 commits behind head on main.

Files
Patch %
Lines

...togen/agentchat/contrib/capabilities/transforms.py
0.00%
9 Missing :warning:

...chat/contrib/capabilities/transforms_conditions.py
0.00%
7 Missing :warning:

...entchat/contrib/capabilities/transform_messages.py
0.00%
3 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2382       +/-   ##
===========================================
- Coverage   38.14%   17.30%   -20.85%
===========================================
  Files          78       82        +4
  Lines        7865     8114      +249
  Branches     1683     1728       +45
===========================================
- Hits         3000     1404     -1596
- Misses       4615     6656     +2041
+ Partials      250       54      -196

Flag
Coverage Δ

unittests
17.30% <0.00%> (-20.83%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@WaelKarkoub what do you think of this structure?

@giorgossideris I believe I understood the intent behind this PR.
Something I learned very recently is that abstraction can seem appealing, but it often introduces complexity and coupling that can escalate quickly (I have PRs to prove that.) I was working on audio support for autogen by integrating popular TTS and speech-to-text APIs, here are the two implementations: https://github.com/microsoft/autogen/pull/2220 and https://github.com/microsoft/autogen/pull/2098. Which implementation do you think is more enjoyable to use and, more importantly, extend?
The second thing is the concept behind message transformations https://github.com/microsoft/autogen/pull/1923 aligns with something called linear transformations. This approach involves using distinct matrices, each have a specific, single function like rotation or scaling—and combining them effectively. Our protocol implementations should act as specialized matrices, good at singular tasks such as managing token limits or compressing data.
In the case of this PR, simply adding an argument to the protocol implementors' constructors is completely fine. It might look like repeated code, and it is, but it's perfectly fine. Consider a scenario if we had to implement a new transformation that requires a new way of checking a condition. This going to force us to change the protocol (a breaking change) and refactor every single transformation, not just the ones supported by autogen but also the ones made by the community. What I liked about your first PR is that every single transformation is expected to make a change, and every single user expects those changes to be logged somewhere, so changing the protocol made perfect sense.
I hope this clarifies my perspective. Also I'm on discord with the same handle if you would like to discuss it more

Closing this PR, because it causes breaking changes to the contract. The intended functionality can be achieved by creating new MessageTransforms.


How to determine pathname without using window.location?

I've looked and looked, and I'm unable to find a way to determine the active pathname from the main
App Route component without using window.loation.pathname.  I will need it in the browser and for server side rendering, so I really don't want to use window.

hi @iamrandys would you mind sharing this work around here ? I'm also facing same issue and would like to see how you manage to solve it.
Thanks


Overlapping drawdowns

The get_top_drawdowns function in timeseries.py sometimes returns overlapping drawdown periods, which should not happen. This code reproduces the issue:
import pyfolio as pf
stock_rets = pf.utils.get_symbol_rets('FB')
pf.timeseries.gen_drawdown_table(stock_rets)

which produces the following output:

Drawdown with index 6 starts on 2014-10-24, finishes on 2015-01-15
Drawdown with index 5 starts on 2014-10-28, which cannot be a peak as then it would have meant the end of the previous drawdown.
Problem occurs on timeseries.py lines 770-772:
            underwater = pd.concat(
                [underwater.loc[:peak].iloc[:-1],
                 underwater.loc[recovery:].iloc[1:]])

the actual peak is getting removed from the series, allowing a subsequent drawdown to engulf the one that just got cropped. The cropping should start on the index immediately after peak. Assuming that indexes are always day-based replacing line 771 by
[underwater.loc[:peak + pd.to_timedelta("1d")].iloc[:-1]

seems to fix the bug.

Partially related to this, lines 770-772 seem like a rather inefficient way of removing data from the series, an overall more efficient way would be to directly drop the unneeded lines rather than chop and concat:
underwater.drop(underwater[peak: recovery].index[1:-1], inplace=True)

I can write a PR if this is a bug and not the intended behaviour.

@jlopezpena Many thanks for reporting. A PR with a fix and your suggested improvement would be greatly appreciated.


shop=* may be vertex in line without being tagging error

https://www.openstreetmap.org/edit?editor=id&way=688919381#map=23/38.19946/-85.74310

at first glance, this looks like a mapping mistake to me: shouldn't the way/688919381 (currently named Ticket Office) get the shop=ticket tag and the individual counters / ticket windows be just mapped as indoor=* features?

https://wiki.openstreetmap.org/w/index.php?title=Talk:Tag:entrance%3Dshop&curid=285297&diff=2411037&oldid=2338660 mentions another case, with highway=corridor connecting POI node and entrance node

If this is acceptable, then it would be not just shops but also offices and various amenities too, right? Is there a list somewhere of the things inside a building that can be connected to a corridor?
I think the intent of this validation rule was to dissuade people from mapping the shop as a vertex of a building or landuse area.

If this is acceptable, then it would be not just shops but also offices and various amenities too, right?

Maybe? So far I have seen it only for shops. And fairly rare, in cases where multiple levels, inconsistent level codes and overall complexity makes hard to guess where given shop has an entrance.

I think the intent of this validation rule was to dissuade people from mapping the shop as a vertex of a building or landuse area.

That makes sense and I guess that more custom one allowing only being vertex on pedestrian ways in theory can be created.

https://wiki.openstreetmap.org/w/index.php?title=Talk:Tag:entrance%3Dshop&curid=285297&diff=2411037&oldid=2338660 mentions another case, with highway=corridor connecting POI node and entrance node

Relevant section of my diary entry that I wrote four years ago:
https://www.openstreetmap.org/user/Anton Khorev/diary/45247#poi-1
It's in Russian, but you can start by looking at this image:
https://antonkhorev.github.io/osm/diary/6/i/poi-levels.png
There's no way you'd be able to tell which entrance corresponds to which shop without connecting them.

This "point" vs "vertex" distinction can create other problems. Here's a video of someone trying to draw a path to an area by first adding a gate it passes through: https://youtu.be/DC9t99fOcrI?t=240 . It's impossible to do this because a "gate" is a "vertex", not a "point".


Update sbt-scoverage to 2.2.1

About this PR
📦 Updates org.scoverage:sbt-scoverage from 2.0.12 to 2.2.1
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scoverage", artifactId = "sbt-scoverage" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.scoverage", artifactId = "sbt-scoverage" }
}]

labels: sbt-plugin-update, early-semver-minor, semver-spec-minor, commit-count:1

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

0 of 0   changed or added relevant lines in 0 files are covered.
39 unchanged lines in 11 files lost coverage.
Overall coverage decreased (-22.9%) to 62.567%

Files with Coverage Reduction
New Missed Lines
%

sequentially/src/main/scala/com/evolutiongaming/concurrent/sequentially/AsyncMap.scala
1
92.86%

sequentially/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentialMap.scala
1
95.24%

sequentially/src/main/scala/com/evolutiongaming/concurrent/sequentially/AsyncHandlerMap.scala
1
66.67%

sequentially/src/main/scala/com/evolutiongaming/concurrent/sequentially/Sequentially.scala
2
82.76%

sequentially/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentiallyHandler.scala
2
92.31%

benchmark/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentiallyHandlerBenchmark.scala
3
0.0%

benchmark/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentiallyBenchmark.scala
3
0.0%

benchmark/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentiallyAsyncBenchmark.scala
3
0.0%

benchmark/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentiallyStreamBenchmark.scala
3
0.0%

sequentially/src/main/scala/com/evolutiongaming/concurrent/sequentially/SequentiallyAsync.scala
4
66.67%

Totals

Change from base Build<PHONE_NUMBER>6:
-22.9%

Covered Lines:
117

Relevant Lines:
187

💛  - Coveralls

Superseded by #176.


Update system dependencies and available Dockerfile's in CK instructions

@guschmue As a bonus, I've fixed a couple of typos :)
But I have a question:

--dataset use the specified dataset. Currently we only support imagenet.
Is this true? How does object detection work then?

Yes,  you could pass --dataset imagenet. The once we understand are define here:
https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/main.py#L36
There is the --profile option that I use to default the command line, like --profile ssd-mobilenet-tf
that will set all args I need for say ssd-mobilenet-tf ... that is where I really set it (https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/main.py#L109).

Thanks @guschmue. I hope that if I want to run non-quantized MobileNet on a subset of ImageNet, I can install the model and dataset via CK, and define the environment variables as follows:
$ ck install package --tags=model,tf,mobilenet,non-quantized
$ ck install package --tags=dataset,imagenet,val,min --no_tags=resized

$ export MODEL_DIR=`ck locate env --tags=model,tf,mobilenet,non-quantized`
$ export DATA_DIR=`ck locate env --tags=dataset,imagenet,val,min --no_tags=resized`

Let me try this with a Docker image.

With the latest master:
$ git log -1
commit 3b18672063fb7695ec35e2356b10c469faa2bbd9 (HEAD -> master, origin/master, origin/HEAD)
Merge: d81366a 7efb902
Author: jveejay<EMAIL_ADDRESS>Date:   Fri Jun 7 06:54:05 2019 -0700

    Merge pull request #172 from dividiti/update-ck

    Update system dependencies and available Dockerfile's in CK instructions

I get the following:
$ ./run_and_time.sh tf mobilenet cpu
...
INFO:imagenet:reduced image list, 49500 images not found
INFO:imagenet:loaded 500 images, cache=0, took=3.5sec
...
[TestScenario.SingleStream-1.0 qps=62.57, mean=0.015941, time=16.36, acc=0.00, queries=1024, tiles=50.0:0.0157,80.0:0.0163,90.0:0.0167,95.0:0.0171,99.0:0.0182,99.9:0.0205
/mlperf/run_helper.sh: line 15:    10 Segmentation fault      (core dumped) python python/main.py $opts --output /output/results.json
ENDING RUN AT 2019-06-11 04:08:29 PM

$ ./run_and_time.sh tflite mobilenet cpu
...
STARTING RUN AT 2019-06-11 04:10:19 PM
INFO:main:Namespace(accuracy=False, backend='tflite', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min', inputs=['input:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224.tflite', output='/output/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=None)
Traceback (most recent call last):
  File "python/main.py", line 532, in <module>
    main()
  File "python/main.py", line 398, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 240, in get_backend
    from backend_tflite import BackendTflite
  File "/mlperf/python/backend_tflite.py", line 10, in <module>
    from tensorflow.contrib.lite.python import interpreter as interpreter_wrapper
ImportError: cannot import name 'interpreter' from 'tensorflow.contrib.lite.python' (/opt/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/lite/python/__init__.py)
ENDING RUN AT 2019-06-11 04:10:22 PM

$ ./run_and_time.sh pytorch mobilenet cpu
...
STARTING RUN AT 2019-06-11 04:11:07 PM
INFO:main:Namespace(accuracy=False, backend='pytorch', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min', inputs=None, max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224.onnx', output='/output/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-onnxruntime', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=None)
INFO:root:Failed to import cuda module: No module named 'caffe2.python.caffe2_pybind11_state_gpu'
INFO:root:Failed to import AMD hip module: No module named 'caffe2.python.caffe2_pybind11_state_hip'
WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
INFO:imagenet:reduced image list, 49500 images not found
INFO:imagenet:loaded 500 images, cache=0, took=5.4sec
Traceback (most recent call last):
  File "python/main.py", line 532, in <module>
    main()
  File "python/main.py", line 413, in main
    model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
  File "/mlperf/python/backend_pytorch.py", line 34, in load
    self.model = onnx.load(model_path)
  File "/opt/anaconda3/lib/python3.7/site-packages/onnx/__init__.py", line 113, in load_model
    s = _load_bytes(f)
  File "/opt/anaconda3/lib/python3.7/site-packages/onnx/__init__.py", line 29, in _load_bytes
    with open(cast(Text, f), 'rb') as readable:
FileNotFoundError: [Errno 2] No such file or directory: '/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224.onnx'
ENDING RUN AT 2019-06-11 04:11:14 PM

@guschmue Shall I open a ticket for TF and TFLite?

The core dump with tf - works for me but I think I had some issue that we install loadgen once into the image and if it updates on github it will not rebuild. I can find a fix for this to build/install as needed,
The tflite - in tf-1.13 tensorflow.contrib.lite.python moved to tensorflow.lite.python I think, can change that.

For TF, this is the first time I tried to build and use this image, so it must have pulled the latest code.
For TFLite, you are right. Have you thought about fixing the versions of tools (like we do with  K)?

I can fix the tflite thing today.
For the crash with tf: looks the benchmark itself was done (since it printed the message) and I guess it is going out of context and closing the tf session). I remember that Christine saw occasionally a crash there that we could not explain.
Can do some testing today.

I tried to fix the version of TensorFlow to 1.12.0 in Dockerfile.cpu as follows:
$ git diff
diff --git a/cloud/image_classification/Dockerfile.cpu b/cloud/image_classification/Dockerfile.cpu
index 8fce34e..e639458 100755
--- a/cloud/image_classification/Dockerfile.cpu
+++ b/cloud/image_classification/Dockerfile.cpu
@@ -32,7 +32,7 @@ RUN cd /opt && \
     conda config --set always_yes yes --set changeps1 no

 RUN conda install pytorch-cpu torchvision-cpu -c pytorch
-RUN pip install future pillow onnx tensorflow onnxruntime
+RUN pip install future pillow onnx tensorflow==1.12.0 onnxruntime
 RUN pip install Cython && pip install pycocotools
 RUN cd /tmp && \
     git clone https://github.com/mlperf/inference && \

but got an error:
Collecting tensorflow==1.12.0
  Could not find a version that satisfies the requirement tensorflow==1.12.0 (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.14.0rc0, 1.14.0rc1, 2.0.0a0, 2.0.0b0)
No matching distribution found for tensorflow==1.12.0

I'm surprised, as I can use pip install tensorflow==1.12.0 from outside Docker.

I still get a segfault with the HEAD:
$ git log -1
commit e82f38e0273906812844b0469e194113a6fd1b81 (HEAD -> master, origin/master, origin/HEAD)
Merge: 2142c70 4b7ce84
Author: christ1ne<EMAIL_ADDRESS>Date:   Wed Jun 12 10:07:59 2019 -0700

    Merge pull request #173 from briandersn/load_gen

    LoadGen: Update README.md; Fix offline+accuracy.

with TensorFlow:
 $ ./run_and_time.sh tf mobilenet cpu

STARTING RUN AT 2019-06-13 02:02:25 PM
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min', inputs=['input:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224_frozen.pb', output='/output/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=None)
INFO:imagenet:reduced image list, 49500 images not found
INFO:imagenet:loaded 500 images, cache=0, took=0.6sec
2019-06-13 14:02:27.240138: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-06-13 14:02:27.269087: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-06-13 14:02:27.270550: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5626208f0f90 executing computations on platform Host. Devices:
2019-06-13 14:02:27.270612: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
INFO:main:starting TestScenario.SingleStream, latency=1.0

]
[TestScenario.SingleStream-1.0 qps=62.66, mean=0.015907, time=16.34, acc=0.00, queries=1024, tiles=50.0:0.0157,80.0:0.0163,90.0:0.0165,95.0:0.0169,99.0:0.0185,99.9:0.0206
/mlperf/run_helper.sh: line 15:    10 Segmentation fault      (core dumped) python python/main.py $opts --output /output/results.json
ENDING RUN AT 2019-06-13 02:02:44 PM

$ ck install package --tags=model,tf,mlperf,resnet
$ export MODEL_DIR=`ck locate env --tags=model,tf,mlperf,resnet`
$ ./run_and_time.sh tf resnet cpu
...
STARTING RUN AT 2019-06-13 02:07:50 PM
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min', inputs=['input_tensor:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/anton/CK_TOOLS/model-tf-mlperf-resnet-downloaded/resnet50_v1.pb', output='/output/results.json', outputs=['ArgMax:0'], profile='resnet50-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=None)
INFO:imagenet:reduced image list, 49500 images not found
INFO:imagenet:loaded 500 images, cache=0, took=0.5sec
2019-06-13 14:07:52.652435: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-06-13 14:07:52.680658: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-06-13 14:07:52.681551: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55ab68c076b0 executing computations on platform Host. Devices:
2019-06-13 14:07:52.681576: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
INFO:main:starting TestScenario.SingleStream, latency=1.0

]
[TestScenario.SingleStream-1.0 qps=14.99, mean=0.066648, time=68.30, acc=0.00, queries=1024, tiles=50.0:0.0663,80.0:0.0671,90.0:0.0679,95.0:0.0690,99.0:0.0725,99.9:0.0789
/mlperf/run_helper.sh: line 15:    10 Segmentation fault      (core dumped) python python/main.py $opts --output /output/results.json
ENDING RUN AT 2019-06-13 02:09:03 PM

$ ck install package --tags=object-detection,model,tf,mlperf,ssd-mobilenet,non-quantized`
$ export MODEL_DIR=`ck locate env --tags=object-detection,model,tf,mlperf,ssd-mobilenet,non-quantized`

$ ck install package --tags=object-detection,dataset,coco.2017,original
$ export DATA_DIR=`ck locate env --tags=object-detection,dataset,coco.2017,original`

$ ./run_and_time.sh tf ssd-mobilenet cpu
...

let me try from scratch on my box the the master.

Same here:
[TestScenario.SingleStream-1.0 qps=65.15, mean=0.015260, time=10.07, acc=0.00, queries=656, tiles=50.0:0.0150,80.0:0.0151,90.0:0.0157,95.0:0.0160,99.0:0.0214,99.9:0.0345
./run_local.sh: line 14: 27773 Segmentation fault      (core dumped)

This must have come in recently - my build from a week ago did not have it.
Very possible that the PR Brian's just sent will fix it:
https://github.com/mlperf/inference/pull/175

$ ck install package --tags=object-detection,model,onnx,mlperf,ssd-mobilenet
$ export MODEL_DIR=`ck locate env --tags=object-detection,model,onnx,mlperf,ssd-mobilenet`

$ ck install package --tags=object-detection,dataset,coco.2017,original
$ export DATA_DIR=`ck locate env --tags=object-detection,dataset,coco.2017,original`

$ ./run_and_time.sh onnx ssd-mobilenet cpu

Executing SSD-MobileNet using the ONNX runtime ended in:
2019-06-14 14:38:09.861556405 [E:onnxruntime:, sequential_executor.cc:125 Execute] Non-zero status code returned while running Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Gather_90/Gather_3 Status Message: indices element out of data bounds, idx=0 data_dim=0
2019-06-14 14:38:09.862392091 [E:onnxruntime:, sequential_executor.cc:125 Execute] Non-zero status code returned while running Node: generic_loop_Loop__69 Status Message: Non-zero status code returned while running Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Gather_90/Gather_3 Status Message: indices element out of data bounds, idx=0 data_dim=0
ERROR:main:thread: failed on contentid=['/datasets/dataset-coco-2017-val/val2017/000000246522.jpg'], Method run failed due to: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Node: generic_loop_Loop__69 Status Message: Non-zero status code returned while running Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Gather_90/Gather_3 Status Message: indices element out of data bounds, idx=0 data_dim=0
[TestScenario.SingleStream-1.0 qps=4.93, mean=0.202886, time=207.81, acc=0.00, queries=1024, tiles=50.0:0.1731,80.0:0.1931,90.0:0.2078,95.0:0.2288,99.0:0.2719,99.9:7.9377
ENDING RUN AT 2019-06-14 02:38:10 PM

which does not look healthy despite some statistics in the end.

$ ck install package --tags=image-classification,model,onnx,mlperf,mobilenet
$ export MODEL_DIR=`ck locate env --tags=image-classification,model,onnx,mlperf,mobilenet`

$ ck install package --tags=dataset,imagenet,val,min --no_tags=resized
$ ck install package --tags=dataset,imagenet,aux
$ cp `ck locate env --tags=aux`/val.txt `ck locate env --tags=val,min`/val_map.txt
$ export DATA_DIR=`ck locate env --tags=dataset,imagenet,val,min --no_tags=resized`

$ ./run_and_time.sh onnxruntime mobilenet cpu

STARTING RUN AT 2019-06-14 03:08:26 PM
INFO:main:Namespace(accuracy=False, backend='onnxruntime', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min', inputs=None, max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/anton/CK_TOOLS/model-onnx-mobilenet-mlperf-downloaded/mobilenet_v1_1.0_224.onnx', output='/output/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-onnxruntime', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=20, time=None)
INFO:imagenet:reduced image list, 49500 images not found
INFO:imagenet:loaded 500 images, cache=0, took=8.5sec
INFO:main:starting TestScenario.SingleStream, latency=1.0

]
[TestScenario.SingleStream-1.0 qps=30.81, mean=0.032391, time=33.23, acc=0.00, queries=1024, tiles=50.0:0.0315,80.0:0.0316,90.0:0.0319,95.0:0.0354,99.0:0.0503,99.9:0.0889
ENDING RUN AT 2019-06-14 03:09:11 PM

yes, we know about this - some issue in the onnxruntime. It runs into some issue on some images and is being worked on.

merged https://github.com/mlperf/inference/pull/175 to master.
The crash seems to be fixed with the change.

Confirm it now works for me too:
$ ck install package --tags=object-detection,model,tf,mlperf,ssd-mobilenet,non-quantized`
$ export MODEL_DIR=`ck locate env --tags=object-detection,model,tf,mlperf,ssd-mobilenet,non-quantized`
$ cp ${MODEL_DIR}/{frozen_inference_graph.pb,ssd_mobilenet_v1_coco_2018_01_28.pb}

$ ck install package --tags=object-detection,dataset,coco.2017,original
$ export DATA_DIR=`ck locate env --tags=object-detection,dataset,coco.2017,original`

$ ./run_and_time.sh tf ssd-mobilenet cpu
...
STARTING RUN AT 2019-06-14 03:57:53 PM
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/datasets/dataset-coco-2017-val', inputs=['image_tensor:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json',
outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=20, time=None)
INFO:coco:loaded 5000 images, cache=0, took=0.9sec
2019-06-14 15:58:02.564012: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-06-14 15:58:02.684421: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-06-14 15:58:02.686238: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x56108c5b9df0 executing computations on platform Host. Devices:
2019-06-14 15:58:02.686272: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
INFO:main:starting TestScenario.SingleStream, latency=1.0

]
[TestScenario.SingleStream-1.0 qps=39.56, mean=0.025196, time=25.89, acc=0.00, queries=1024, tiles=50.0:0.0251,80.0:0.0253,90.0:0.0255,95.0:0.0257,99.0:0.0274,99.9:0.0293
ENDING RUN AT 2019-06-14 03:58:33 PM

fyi @briandersn

@guschmue Do you think you could add instructions on how to resize COCO for SSD-ResNet and run the ONNX model in the accuracy mode? (We are getting mAP of 16.5% rather than 20%.)

Hi @guschmue, I am getting the same warnings when running as ./run_and_time.sh tf ssd-resnet34 cpu:
2019-06-26 19:03:35.031933498 [W:onnxruntime:Default, execution_frame.cc:291 AllocateMLValueTensorSelfOwnBufferHelper] For mlvalue with index: 679, block in memory pattern size is: 2304 but the actually si  ze is: 1536, fall back to default allocation behavior

This is an issue in onnxruntime.
Most of it is fixed in the onnxruntime master (definitely fixed for ssd-mobilenet).


Error in the zh-TW translation

[Enter feedback here]
In the section "SignalR", the Traditional Chinese (zh-TW) translation of "For most applications, we recommend SignalR over raw WebSockets." is "針對大部分的應用程式，我們建議您不要透過 SignalR 原始 websocket。", which is telling me not to use SignalR for WebSockets.
I recommend translating to "針對大部分的應用程式，我們建議使用 SignalR，而不是原始的 WebSocket。", which is the Simplified Chinese (zh-CN) translation of this page.
Sometimes there are some translation errors in Microsoft Docs, but they're still readable. However, this one is the opposite of the original meaning, so I write this feedback.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a3b0cac1-adc7-434c-d867-6e28e39bdae7
Version Independent ID: 762efeed-010e-f422-03d0-22d6681b2851
Content: WebSockets support in ASP.NET Core
Content Source: aspnetcore/fundamentals/websockets.md
Product: aspnet-core
Technology: aspnetcore-fundamentals
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Hello @kent010341 ... I'll ping our localization manager to take a look ...
Ping @khoiph1

I don't think Khoi is receiving our pings. I've emailed him to take a look.

Logged FEEDBACK 562501


composer require zendframework/zend-captcha can't install php 7.1.16

composer require zendframework/zend-captcha
Using version ^2.8 for zendframework/zend-captcha
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- The requested package zendframework/zend-form (locked at 2.10.2, required as ^2.12) is satisfiable by zendframework/zend-form[2.10.2] but these conflict with your requirements or minimum-stability.
Problem 2
- Conclusion: don't install zendframework/zend-form 2.12.0
- Conclusion: don't install zendframework/zend-form 2.13.x-dev
- Conclusion: remove zendframework/zend-inputfilter 2.7.6
- Installation request for zendframework/zend-form ^2.12 -> satisfiable by zendframework/zend-form[2.12.0, 2.13.x-dev, 2.12.x-dev].
- Conclusion: don't install zendframework/zend-inputfilter 2.7.6
- zendframework/zend-form 2.12.x-dev requires zendframework/zend-inputfilter ^2.8 -> satisfiable by zendframework/zend-inputfilter[2.8.0, 2.8.1, 2.8.2, 2.9.x-dev, 2.8.x-dev].
- Can only install one of: zendframework/zend-inputfilter[2.8.0, 2.7.6].
- Can only install one of: zendframework/zend-inputfilter[2.8.1, 2.7.6].
- Can only install one of: zendframework/zend-inputfilter[2.8.2, 2.7.6].
- Can only install one of: zendframework/zend-inputfilter[2.9.x-dev, 2.7.6].
- Can only install one of: zendframework/zend-inputfilter[2.8.x-dev, 2.7.6].
- Installation request for zendframework/zend-inputfilter (locked at 2.7.6) -> satisfiable by zendframework/zend-inputfilter[2.7.6].
Installation failed, reverting ./composer.json to its original content.
help me install packge

Read the exact message: some of your dependencies are locked, so you need to relax them before you upgrade any component.

Thank my your forword


Drop SyncSets check

Which issue this PR addresses:

Fixes: https://issues.redhat.com/browse/ARO-7787
What this PR does / why we need it:
This PR drops the sync sets check from IsClusterDeploymentReady.
Monitoring of SyncSets will come as part of it's own monitoring in the future and since we have no SyncSets at present there is no temporary gap.
Test plan for issue:
Tested locally using the dev Hive

/azp run ci

/azp run ci


Move JMX statistics to an ehcache service

This is the first step of a series of refactoring. It moves the statistics into ehcache.

I don't know what is our politic right now. But we still have a lot of author tags everywhere. Depending on the project, people do or do not use them.


Update default Scala version to 2.11 in perform-release.sh

Build triggered. sha1 is merged.

Refer to this link for build results (access rights to CI server needed):
/job/datavec_PR/65/

Build finished. No test results found.


Add github linguist build backend

The goal of this PR is to get a programming language breakdown similar to what is achieved by an existing script. The difference is that by integrating Linguist we can work with repositories that are not on github (eg. gitlab, etc) and not be limited by rate limits or anything else related to hitting an external API. With this breakdown, RTD.org will better understand the projects it is hosting which comes with a variety of advantages.
Specifically, this PR:

Adds a linguist build backend which saves programming language breakdown
Only runs the linguist build on "latest" builds - we don't care about the breakdown on older builds
Does not break the build even when Linguist fails (eg. isn't installed, etc.)

For running on readthedocs.org, this will require https://github.com/rtfd/readthedocs-docker-images/pull/55.

I made updates based on the feedback.

the linguist command is the same everywhere (it's what you get when you gem install it
copying to the web servers is removed; that simplified a few things

I started with a quick glance here, but before nitpicking on review, a few thoughts on implementation:
I'm hesitant to make this part of the default implementation as this could make getting set up for development more confusing. For instance, we don't require our docker images for development, so this would require manual linguist configuration on the host for development. It also isn't really important to the core functionality of RTD, just to our production data collection, so it doesn't seem like a warranted dependency.
There are a few technical options here. I'll avoid getting deep into a technical decision immediately, so hopefully I'm make some sense with this :)

This code should live in -ext and should be optionally pluggable through the use of standard django application loading or additional settings
The builder lookup could possibly be made pluggable, so that external code can manipulate which builders (and the order) are excuted for sphinx or mkdocs builders
We could also just implement this on signal from the project.tasks task (also probably easiest (also also, probably a good idea to just have more signals everywhere anyways))
doc_builder.loaders probably needs to be pluggable in some fashion as well?

With these changes, this builder could optionally be brought in through production settings (we can also avoid it on the .com this way) and no development instances require linguist.
Thoughts?

This is definitely fair and perhaps putting this in the private extensions is a better idea. Just to clarify a few points:

Linguist is not required for builds. It will run on every "latest" build but if Linguist is missing, it will not fail the build. However, your point about this not being core functionality is quite true.
When you say "builder lookup could possibly be made pluggable", are you thinking additions to settings.py?
There already is a build_complete signal but we might need at least one additional one.

What's the status of this PR? Should we review it and merge or should be close it?

What's the status of this PR?

Right now, it is not a priority.
After gaining insight over the months I've worked on RTD, it's possible that this is better suited for a periodically run ./manage.py command rather than a backend that runs on every build.


"Got unnsupported command"

Got this error trying to run the app.
Atmosphere 0.13.0
Base OS - 10.0.4
Edit: The app opens then crashes Atmosphere

It would be so nice if the devs could actually make this app work again. The Switch's wifi is so unrelaible that FTP isn't an option sometimes, and removing the SD card gets annoying with having to reboot the Switch.

I get an endless loop of "Got unnsupported command UNKNOWN" known. On 11.0.1 Tried both clean atmosphere both DeepSea. It did work the first few times but no such luck any more.

Same issue here, I might look into it if when I get the time.

I get that same issue too and nothing shows on my File Explorer menu when I plug my Switch into my computer.

same issue

Same


Points of focus for the review is arcgis: it builds, but i haven't tested it locally to see that it does what it's supposed to. Do note the last paragraph on the cache class:

Caching is optional in the send ops; an instance of this should be injected only in applications where we know we can rely on change tracking!

If we do not trust change detection in arcgis (ie, difficult re object selection, layers, etc.), it can be skipped entirely!
https://spockle.atlassian.net/browse/DUI3-29

My main question on having caching nullable, how it can be null?

Basically I am not sure if all connectors will have this cache, so hence it's nullable (optinal) everywhere. Send should work without it - for example, if i comment out in rhino this line,
    // register send conversion cache
    builder.AddSingleton<ISendConversionCache, SendConversionCache>();

in the DUI3-DX/Connectors/Rhino/Speckle.Connectors.Rhino7/DependencyInjection/RhinoConnectorModule.cs, it will send as always, but just not use the cache.
Does this make sense @oguzhankoral?

As a general comment, the app is running and changedObjectIds are detected as before.
Re change detection: ATM, we track it on layer level, and it works well. More granularity (and more complexity) can be added for report purpose, but for Selecting/Updating selection looks fine to me

My main question on having caching nullable, how it can be null?

Basically I am not sure if all connectors will have this cache, so hence it's nullable (optinal) everywhere. Send should work without it - for example, if i comment out in rhino this line,
    // register send conversion cache
    builder.AddSingleton<ISendConversionCache, SendConversionCache>();

in the DUI3-DX/Connectors/Rhino/Speckle.Connectors.Rhino7/DependencyInjection/RhinoConnectorModule.cs, it will send as always, but just not use the cache.
Does this make sense @oguzhankoral?
I think I get it now (after i replied here #3491 (comment)) : pretty much the only place it actually needs to be nullable is in the shared RootObjectSender class, which is shared by any connector, including ones that will potentially not have this cache.
That okay ?

As I mentioned above we should probably have a NoCacheSendConversionCache() I am unsure if the Singleton() is the right scope also, maybe, probably it should be as the cache will persist across sends.

Oki doki, ready for review again:

made the cache non-optional anywhere, and added a do nothing one for future use in cnx's that can't support it (cc @BovineOx @oguzhankoral )
removed stale class and comments (cc @JR-Morgan @oguzhankoral)

Also thx @KatKatKateryna for testing arcgis!


Try to reduce anti-virus false flags

Some recommended steps to reduce risk of being flagged as harmful software, e.g. building Pyinstaller from source instead of prebuilt. May or may not work.

Tried building Pyinstaller bootloader from source, but VirusTotal reports it still flags on the same 4 antivirus programs (including Avast), but vast majority including Microsoft Defender don't complain.


Could not read input file ../../TEXT/100006.txt

100006.txt  which is used in the example notebook is not included in the repo. Could you please add that to the repo? Thanks!
--
kvgc

The file is now available. Thanks.


use “paypal-web-payments”  way,the first time it can call back to our app, the second time  it can't

PayPal Android SDK Version
com.paypal.android:paypal-web-payments:1.3.0
Environment
Sandbox
Android Version & Device
android 12
PayPal dependencies
com.paypal.android:paypal-web-payments:1.3.0
Describe the bug
use “paypal-web-payments”  way,the first time it can call back to our app, the second time  it can't
To reproduce
payPalWebCheckoutClient.listener = object : PayPalWebCheckoutListener {
override fun onPayPalWebSuccess(result: PayPalWebCheckoutResult) {
// order was approved and is ready to be captured/authorized (see step 7)
        }

        override fun onPayPalWebFailure(error: PayPalSDKError) {
            Log.e(TAG, "onPayPalWebFailure error: $error")

        }

        override fun onPayPalWebCanceled() {
            Log.e(TAG, "onPayPalWebCanceled")

        }
    }

can not listener
Expected behavior
expected result

fail result

Screenshots
No response

It seems the issue is partly related to the browser that selected to launch the checkout.
Using Chrome and Samsung browser are fine, but some other seems not. Would that possible to pre-select using Chrome to launch the Checkout?

@HoanChen we think this PR may resolve the issues you're having with the follow up transactions

Greetings, I've updated to 1.4.0 and also to 1.4.1-SNAPSHOT and still confront the same issue.  The listener, reverts back to the first transaction and the Capture API rejects the transaction because it had already registered it.  Going over the notes on the release 1.4.0, it stated that steps were taken to remove observers.  This is what PR-251 says is the root cause.
I am working on the Paypal Sandbox environment with a physical Samsung Galaxy S-22 as the testing device.


ip-whitelist seems not to work

Hello,
I did several test with ip-address whitelist but seems not to work. Here is my config:
[
{
"id": "mein_webhook",
"execute-command": "/usr/local/custom/webhook/pytest.py",
"command-working-directory": "/usr/local/custom/webhook/",
"pass-arguments-to-command":
[
{
"source": "entire-payload"
}
],
"trigger-rule":
{ "match": { "type": "ip-whitelist", "ip-range": "<IP_ADDRESS>/24" } }
}
]
same with { "match": { "type": "ip-whitelist", "ip-range": "<IP_ADDRESS>/32" } }  => Match one exact ip address.
In both cases the trigger not match regarding the "-verbose" output.
Whitout trigger-rule all works fine.
Source of the hook request is <IP_ADDRESS>
Regards
Stephan

.92

.397

@StephanHarrer, what version of webhook are you running?  Can you enable and share debug output?

Hello @moorereason ,
sorry for the inconvenience. I was so naive to think that I get the latest version with the official package of Ubuntu.
The Ubuntu package has the version 2.5.0-2, with this the error occurs. Now I have the pre build package of version 2.8.0 in use and now everything works!!!
Thanks for pushing me in the right direction.
Regards
Stephan


Allow wildcard expansion to be skipped for some arguments.

On UNIX, it is possible to avoid wildcard expansion by either quoting or escaping the wildcards.  For example find . -name "*.cxx" or find . -name \*.cxx.  This works fine if the wildcard expansion is done by the shell, but mainmacro does the expansion within the executable, and the quotes and escapes are not passed through to the executable.
One way around this is to provide mainmacro with a set of args which indicate that the following arg should not be expanded.  So, if -name is encountered, then *.cxx will be passed through unexpanded.

Fixed in commit 627fcecd (release v1.2).


Async Code Transformation

Summary
Support async code transformation as mentioned in #9504
Test plan

Test cases added

Still very early stage, but already noticing processAsync() and getCacheKeyAsync() would require multiple layers of function to be async, many of which only one or two lines of difference from the sync version. Will try to abstract the common flows into sharable functions

Super exciting!

require multiple layers of function to be async, many of which only one or two lines of difference from the sync version

Yeah, I expected that. Almost all the logic will be identical, just cache lookups and the transforms themselves.
One thing is that multiple processes will probably want to transform the same files. Not a big deal in sync code, but for async code we'll quickly end up transpiling the same file a bunch of time, wasting a lot of CPU. DOes not have to be solved in this PR, but having some sort of mutex or lock per file name might make sense?

One thing is that multiple processes will probably want to transform the same files. Not a big deal in sync code, but for async code we'll quickly end up transpiling the same file a bunch of time, wasting a lot of CPU. DOes not have to be solved in this PR, but having some sort of mutex or lock per file name might make sense?

Yes that makes sense. A way to implement per file name lock might be shared module map that tracks file name, whether being transpiled, and cached transpile result?

Yeah, something like that 🙂 Again, not something we need in a first iteration, but something to keep in mind

Considering the following case: a file is associated with a processAsync-only transformer in the cache, but a synchronous transform on it is attempted. We can only throw an error in this case?
If code cache is shared between processes in the future, might be able to reduce this kind of error? (When the file was transformed asynchronously and cached before the sync call)

We can only throw an error in this case?

Yes.

If code cache is shared between processes in the future, might be able to reduce this kind of error?

I don't think we should do that, it would make tests passing dependent on execution order (or warm cache), which we don't want

Hi @SimenB, can you take a look and see it the changes are on the right direction? Also, a bit curious why tsc complaints about require('../package.json'); which I hadn't touched 🤔

Try a rebase, I've had weird warnings like that before that a rebase took care of

Regarding tsc package.json is not under 'rootDir' complaint, it happens when import() is also used in the file. Haven't found much discussion about this. Will play around with it more and maybe reach out to TypeScript community.

Codecov Report

Merging #9889 into master will decrease coverage by 0.37%.
The diff coverage is 27.35%.

@@            Coverage Diff             @@
##           master    #9889      +/-   ##
==========================================
- Coverage   64.00%   63.62%   -0.38%
==========================================
  Files         297      297
  Lines       12528    12617      +89
  Branches     3093     3122      +29
==========================================
+ Hits         8018     8028      +10
- Misses       3860     3936      +76
- Partials      650      653       +3

Impacted Files
Coverage Δ

packages/jest-transform/src/ScriptTransformer.ts
51.29% <27.35%> (-16.14%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e0eeed...05e632d. Read the comment docs.

Hi @SimenB:
What was the motivation to use rootDir over include in tsconfig.json? I think there is a bug with tsc and has reported it. By using include, we won't bump into that issue. The following packages also require package.json so might encounter similar situation somewhere down the road:

jest-core
jest-haste-map
jest-runtime
Does it make sense to switch to include instead?

No particular reason. If it works I'm happy to merge a PR migrating

I was wrong. When we use include instead of rootDir, tsc does not complain about "outside of rootDir" but does not generate proper output for the package either :(

TypeScript team has resolved the issue with import(); continuing the work now.
@SimenB, a question: is there anything I should be aware when mocking a module that will be loaded by import()? when I try to mock an async preprocessor for ScriptTransformer which uses import() to load the module, it doesn't seem to recognise the mock.

@SimenB, a question: is there anything I should be aware when mocking a module that will be loaded by import()?

Native import() is not supported, but any import() you use here should be transpiled to Promise.resolve(require()) so normal mocking should work just fine

Thanks, @SimenB!
While trying to test ScriptTransformer, Jest complaints about import(). I made a simplified example here. Which files should I trace in order to better address the issue? Should I try to find the transformed code from Jest's cache?

Thanks, @SimenB!
While trying to test ScriptTransformer, Jest complaints about import(). I made a simplified example here.

That's complaining about missing SyntheticModule - you need to run with --experimental-vm-modules. See https://jestjs.io/docs/en/ecmascript-modules. That said, the error should be better...
I actually landed support for dynamic imports in CJS a few weeks ago (#10620), but that said, I think it should be transpiled? So as long as babel is set up (I see you esclude @babel/plugin-proposal-dynamic-import in your setup, which it why it fails) it should be fine.
Mocking native dynamic import is not really implemented yet (it's one of the things on the todo list in #9430)

Hi @SimenB
Thank you for your guidance! I realised that I should have asked this in the first place: do we need to support custom transformers as ESM? In the custom transformers section, the example is in CommonJS.
If transformers are written in cjs, we would not need to exclude @babel/plugin-proposal-dynamic-import for Transformer.ts and then things become a lot more straightforward 🙂

Oh, you were talking about the transformers themselves! No, that's out of scope for now 🙂 Can be tracked in https://github.com/facebook/jest/issues/8810, but that's totally separate from this PR 👍

Hi @SimenB
When testing ScriptTransformer.transform() asynchronously, later implementation mocks would replace previous one before the test case finishes(because it's async) .
A thought was to mock preprocessors as multiple modules (ex. 'preprocessor-banana', 'preprocessor-kiwi' etc) for each correct and incorrect return values but it does not appear to be a good idea, since jest.mock() would be hoisted and we cannot do the following in it():
[[0, 'apple.js'], ['str', 'banana.js']].forEach(([val, path)=>{
  jest.mock(`preprocessor-${path.replace('.', '-')}`,
  () => ({
    processAsync: jest.fn().mockResolvedValue(val)
  }),
  {
    virtual: true
  }
});

In short, I think it comes down to:

We are mocking a function and want to test what will happen when it returns different values
We define a collection of return values and test each case
The tested programme is async in nature, so mockReturnValue() or mockResolvedValue() simply overwrite before all the cases being tested.

What would be a better way to structure tests in this scenario? Thank you in advance!

jest.mock should only be hoisted to the top of its scope (e.g. function or top-level), so sticking it in a loop should work?

Bah, 2b748f67c25615a111330017a2bffc0baf51d558 is probably somewhat painful to rebase against, sorry about that...

Thanks for the headsup, let me rebase again and then reply to your comments 👍

@ychi just FYI, it looks like this PR will need to be rebased

Thanks for rebasing @ychi !  @SimenB do you have any more thoughts on this PR for async code transformations?

Oops, a mistake while rebasing, I have reset ychi:async-transform to 5f41c83; but this PR still regarded as no change 🤔

Thanks, @SimenB!

@ychi I'm really excited about your new changes! just FYI, it looks like lint is failing

Thanks, @benmccann. The lint error looks strange, there is no empty line on or around the complained spot. Similar situations had been reported.
@SimenB: For those existing synchronous transform test cases, I added their counterparts: synchronous preprocessor but called from transformAsync() and asynchronous proprocessor called from transformAsync()
Was wondering how cache should work when multiple places are calling transformAsync() concurrently, maybe end up updating the cache multiple times?

Was wondering how cache should work when multiple places are calling transformAsync() concurrently, maybe end up updating the cache multiple times?

probably, yeah. Updating the cache multiple times probably isn't an issue from a correctness standpoint (the result should be the same), but it's overhead both in CPU time spent transforming twice instead of reusing and in extra fs calls. We could add some sort of lock system so we don't transform the same file twice. Something acquires a lock for a certain file and releases it when it's present in the cache. That way concurrent transform calls to the same file should just wait for the process currently transforming it to complete

@ychi I noticed this PR was marked as a draft. Was there anything else you wanted to accomplish before marking it ready for review?

Would love to assist on this, this would unlock using toolchains like vite/snowpack et al to use their build pipelines as transforms.

Thanks @threepointone! I need to circle back to this and figure out what the status is and what potentially is missing. Been super busy this year, but (🤞) it seems I'll have some time this month.
From memory, this PR "only" adds support in the transform itself, Jest wouldn't know how to use it. Might need to do some PoC-ing to see how it fits together

No rush! I see, I see.

This would also allow Svelte to use Jest, which is why I'm very interested in this. Svelte has a concept of preprocessors to allow usage of things like TypeScript, PostCSS, etc. in templates. Right now you can't use preprocessors with Jest because they're async which prevents using most real-world Svelte projects with Jest
@SimenB is right that there's still more work needed after this, but this is a step in the direction that will help make it clearer how the feature might ultimately be implemented and accomplish some of the beginning pieces, so I'm very excited about this still

@ychi FYI I've opened up #11163 which will conflict horribly with this PR

Let me try to think through the mutex part this weekend and see if I can get ahead of the race😂

@SimenB: feedback addressed and added changelog.
MacOS test run on Azure somehow failed 😬

OK, landed the other one. Sorry about the conflicts 🙈 I think the changes over there makes the changes in this PR a bit cleaner though, as it centralizes "load transformer", and for async transformation we only need to deal with async cache key and async transformation

@SimenB  Some help would be wonderful😆 I am relocating in the coming week, available time might be fragmented. If you (or anyone) wants to chime in, please feel free. Otherwise I can come back to this in mid- to late-March. 🙂

Squashed, removed the mutex, and rebased except for the very latest commit. That one is a bit more hairy since we have a bunch of tests which assert we throw error about missing process during transform while that has been moved to load now. So I just pushed what I'm certain about now. Will do the rest as well

@ychi I think this came out correctly. The tests were a bit hairy, but I think I got it 😅

I've played with using this in the runtime, and I think we should be good to go 👍
Need to add some docs, but I can do that now


query_status_response_type should default to code if using PKCE flow

Raising this here in the hope it saves other folks some time :)
If you're using PKCE, you might find issues with silent refresh and session check methods on UserManager:

Silent requests to IdentityServer fail with errors like "Invalid grant type for client"
You may notice Iframe timeouts or CSP violations in the console.

To fix the issue, you need to change the value of the query_status_response_type UserManager setting – this will be id_token by default, and if you're using PKCE you've probably disallowed all the other grant types.
As a developer experience improvement, could we default query_status_response_type to code if the UserManager is configured using PKCE, or drop a debug warning in the console to remind the developer to check the value?
cheers!
Josh

Thanks @JoshBarr ;-),
Rather hilarious but this saved me and a team I'm helping quite some time today ;-), they were having exact the same issue.
You're the man!
Thanks,
Dave

Yea, this is on my backlog to do some sort of fix.

ok, i have updated the code to infer the query_status_response_type to the right value (if it's not provided) based on response_type. HTH

<EMAIL_ADDRESS>released; please have a look and see if it satisfies this issue.

Any chance to test on the beta?


elinks 0.16.0

Checklist

[x] Checked the nixpkgs master branch

[x] Checked the nixpkgs pull requests

Project name
nix search name: elinks

current version: 0.15.1
desired version: 0.16.0
Notify maintainers

maintainers: @iblech @gebner
Note for maintainers
https://github.com/rkd77/elinks/releases/tag/v0.16.0rc1
https://github.com/rkd77/elinks/releases/tag/v0.16.0

Thank you for bringing this to my attention :-) I just submitted a corresponding pull request.


remote guardrails model support

Description
Add a new type of guardrails, model type.
It supports a remote model as guardrails to validate user input or LLM output.
Issues Resolved
Closes https://github.com/opensearch-project/documentation-website/issues/7353
Version
2.15
Checklist

[ ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Thanks, @jngz-es ! Is this ready for review by a writer? Also, please address the Vale issues, if possible, and let us know if you have any questions or disagree with the suggestions. Thanks!


Cannot downgrade kernel from mainline to 4.4 on bionic-minimal-rock64-0.6.31-209-arm64

When I attempt to return to the 4.4 kernel after running the mainline kernel using dpkg -i, I run into a problem regarding old or unknown versions. I do not recall this issue being present while attempting to downgrade the kernel on build 0.5.15.
root@rock64:~# uname -r
4.16.0-rockchip-ayufan-209-g027e722bb
root@rock64:~# dpkg -i linux-image-4.4.120-rockchip-ayufan-209_0.6.31_arm64.deb
(Reading database ... 50689 files and directories currently installed.)
Preparing to unpack linux-image-4.4.120-rockchip-ayufan-209_0.6.31_arm64.deb ...
Unpacking linux-image-4.4.120-rockchip-ayufan-209 (0.6.31) over (0.6.31) ...
Setting up linux-image-4.4.120-rockchip-ayufan-209 (0.6.31) ...
update-initramfs: Generating /boot/initrd.img-4.4.120-rockchip-ayufan-209
Warning: root device  does not exist
Using DTB: rockchip/rk3328-rock64.dtb
Installing /usr/lib/linux-image-4.4.120-rockchip-ayufan-209/rockchip/rk3328-rock64.dtb into /boot/dtbs/4.4.120-rockchip-ayufan-209/rockchip/rk3328-rock64.dtb
Taking backup of rk3328-rock64.dtb.
Installing new rk3328-rock64.dtb.
Ignoring old or unknown version 4.4.120-rockchip-ayufan-209 (latest is 4.16.0-rockchip-ayufan-209-g027e722bb)
Use --force if you want version 4.4.120-rockchip-ayufan-209.
Installing /usr/lib/linux-image-4.16.0-rockchip-ayufan-209-g027e722bb/rockchip/rk3328-rock64.dtb into /boot/dtbs/4.16.0-rockchip-ayufan-209-g027e722bb/rockchip/rk3328-rock64.dtb
Taking backup of rk3328-rock64.dtb.
Installing new rk3328-rock64.dtb.
flash-kernel: deferring update (trigger activated)
Using DTB: rockchip/rk3328-rock64.dtb
Installing /usr/lib/linux-image-4.4.120-rockchip-ayufan-209/rockchip/rk3328-rock64.dtb into /boot/dtbs/4.4.120-rockchip-ayufan-209/rockchip/rk3328-rock64.dtb
Taking backup of rk3328-rock64.dtb.
Installing new rk3328-rock64.dtb.
Ignoring old or unknown version 4.4.120-rockchip-ayufan-209 (latest is 4.16.0-rockchip-ayufan-209-g027e722bb)
Processing triggers for flash-kernel (3.92) ...
Using DTB: rockchip/rk3328-rock64.dtb
Installing /usr/lib/linux-image-4.16.0-rockchip-ayufan-209-g027e722bb/rockchip/rk3328-rock64.dtb into /boot/dtbs/4.16.0-rockchip-ayufan-209-g027e722bb/rockchip/rk3328-rock64.dtb
Taking backup of rk3328-rock64.dtb.
Installing new rk3328-rock64.dtb.
flash-kernel: installing version 4.16.0-rockchip-ayufan-209-g027e722bb
Taking backup of Image.
Installing new Image.
Taking backup of initrd.img.
Installing new initrd.img.
Taking backup of dtb.
Installing new dtb.

I see that there is an option to use a --force option, but I am unsure as to how to pass this option to flash-kernel when using dpkg.
Any hints on how to downgrade would be appreciated since I seem to be lose access the server daily due to #158 and #157.

The easiest is to remove mainline.
apt-get remove linux-{image,headers}-4.16.0-rockchip-ayufan-209-g027e722bb

Thanks ayufan, that worked perfectly!
Feel free to close if this is the expected behavior.


gdspy dependency is failing in python3.7 and python3.8 venv environments

Describe the bug
There is an installation error for gdspy package. This is how it looks like when you install it -

To Reproduce
I created a venv for both python3.7 and 3.8. Let me know you are able to reproduce the error using this method.
Expected behavior
Clean installation
Suggested fix
No idea

Not sure if you have any idea on this fix. I don't see any development from past 2 months on the gdspy github repo.

@joamatab Not sure if you have any idea on gdspy codebase. I don't see any development (code) on the gdspy github repo.

Hi Sai,
to reproduce the error you should write all the commands that you used, a screenshot is not enough :)
my guess is that you did not follow the install instructions for gdsfactory
How did you install gdsfactory?
Did you follow the gdsfactory instructions and install it with mamba?
I also recommend that gdspy install issues should be addressed in the gdspy repo
https://github.com/heitzmann/gdspy

I added the screenshot under "Describe the bug" section cuz that is the only message I got.
I installed gdsfactory via pip which is one of the dependencies in OpenFASoC. Whenever I start the installation, I get error in the gdspy installation section. I explicitly tried to install gdspy to confirm the error.
These are the commands I used to insta gdspy -
python3.7 -m venv "my_env" && source ~/my_env/bin/activate && pip3 install gdspy
Sure, I will open there as well.

@saicharan0112 please try installing pip<22. I believe this is an error with the latest pip

also are you on windows or linux? make sure to always install with conda if on windows

Sure, let me give it a try. I am using Linux CentOS7 BTW.

Thanks alot, @tvt173 . That downgrading version trick removed this issue. But I got a few other error messages. Can you please take a look at this log and help me understand how to handle this?
gdsfactory_error.log

@tvt173 I think I fixed this. It is something to do with the python parallel installation. I didn't had required packages for the python3.7 to install properly. After that error, I followed the steps shown here (https://stackoverflow.com/questions/27022373/python3-importerror-no-module-named-ctypes-when-using-value-from-module-mul) and I got a clean installation of the gdsfactory (and gdspy) packages.
I don't think this is something to do with the pip version cuz I am able to install it with pip 22.2.2 version.
But thanks for your help @tvt173 . If I didn't get that error shown in the log file, I would have overseen this library dependency issues.


Add a benchmark app for Blazor

@pranavkm Sorry I missed this before, but this PR doesn't include the third-party notices for content in this app: https://github.com/dotnet-presentations/blazor-workshop/blob/master/THIRD-PARTY-NOTICES.md
Would you also be able to add the relevant notices from that file into a new THIRD-PARTY-NOTICES.md inside src/Components/benchmarkapps/?

Oh wait, you didn't include the image files (e.g., meaty.jpg). What's the idea with this - that they're not needed for benchmarking so we exclude them? It might be confusing that they still get referenced though.

Oh wait, you didn't include the image files (e.g., meaty.jpg). What's the idea with this - that they're not needed for benchmarking so we exclude them? It might be confusing that they still get referenced though.

We could add it though, wouldn't the app look bad if it doesn't have it when you F5?

Oh wait, you didn't include the image files (e.g., meaty.jpg). What's the idea with this - that they're not needed for benchmarking so we exclude them? It might be confusing that they still get referenced though.

Ignitor cares about the content, not the assets. I wanted to avoid checking in a bunch of files which wouldn't necessarily be used. I also edited the app to not reference these files, so you don't see 404s when you run it locally.
@javiercn it doesn't look entirely terrible 🍕:


Input field not working in action slots

When I use input field in afterList or beforeList slots I cannot type in it or select it
Here is a JSFiddle: https://jsfiddle.net/shentao/s0ugwmjp/

Hi there. The code link provided didn't use any slots, but I've play around and can get the before and after list slots working - check it out here: https://jsfiddle.net/mattelen/n7vm0uy8/41/
Key thing to note is that in the template element, to provide an attribute called slot, and provide the name in there.
Hope this helps. I'm going to close this issue as there is no further work to be done on this

Same problem here. Want to know how to fix it. Anybody help please.

@StepanovMax did you find something? I have same issue and still can't solve it :(


Add int8 data loader

@reminisce @zheng-da @szha @KellenSunderland @pengzhao-intel @TaoLv
PR major change:

introduce int8 data loader. This can be used for quantized network with rgb_mean != (0, 0, 0).
Add new parameter use_quantized_data_layer in quantize_model API to indicate quantized model is going to accept int8 data directly.

Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@ZhennanQin Thanks for your contribution!
@mxnet-label-bot [pr-awaiting-review]

@ZhennanQin Thanks for your contribution!
@mxnet-label-bot add [pr-awaiting-review]

@anirudh2290 Thanks for reviewing. That idea was in my head, but I was a bit worried about letting quantize_v2 accept int8 dtypes looks a bit weird, and would not be accepted by community.  I have to say, that way is good for user experience as user can use single int8 model for both fp32 and int8 data loader. If you think it's doable, then I will refactor towards it.

Yes I would prefer that and I think it would be doable. Would need to take care of inplace optimization if input and output is int8.

@anirudh2290 I tried to add FInplaceOption and FInplaceIdentity to quantize_v2 to optimize the case that input and ouput is int8. But failed with this check in plan memory pass:
https://github.com/dmlc/tvm/blob/master/nnvm/src/pass/plan_memory.cc#L227
sid_in >= 0 &&

export MXNET_MEM_PLAN_VERBOSE_LOGGING=1 provides below logging:
[14:02:23] src/executor/../common/exec_utils.h:300: node 0 var
[14:02:23] src/executor/../common/exec_utils.h:302: node 1 _contrib_quantize_v2
[14:02:23] src/executor/../common/exec_utils.h:306: 		input 0: [1,3,224,224] (147 KB) -> var (-1)
[14:02:23] src/executor/../common/exec_utils.h:312: 		output 1: [1,3,224,224] (147 KB) -> group 0
[14:02:23] src/executor/../common/exec_utils.h:312: 		output 2: [1] (0 KB) -> group 2
[14:02:23] src/executor/../common/exec_utils.h:312: 		output 3: [1] (0 KB) -> group 1

It seems that, data is an variable of this network, so its sid_in is -1, making it can't be inplaced with  quantize_v2 output. Is there any reason we can't inplace var with op output?

apologies for the delay,  I made an attempt on my end to make quantize use inplace optimization when in dtype is also int8. I couldnt find a way to save state in InferType to reuse it within function registered for Inplace optimization. Here is my attempt: https://github.com/anirudh2290/mxnet/commit/cc6bf9e96d982a31bb6bf953d76b2d6ce3525f22 . Also, I am not sure why input var cant be inplaced and would like to reproduce this issue at my end. Would you be able to share your current code for the inplace optimization for quantization op. If you are short on time, I think we can proceed with the quantization implementation without inplace optimization.

@anirudh2290 No need to apologies, you're one of the most active reviewers I've seen in MXNet project:)
The patch letting quantize_v2 accept int8 is ready at https://github.com/ZhennanQin/incubator-mxnet/pull/5. You can apply it based on this PR. Alternately,  you can pick whole patch with int8 data loader at https://github.com/ZhennanQin/incubator-mxnet/pull/6.
I couldnt find a way to save state in InferType to reuse it within function registered for Inplace optimization. Yes, that's the major problem when apply inplace optimization. But luckily, there's a strict dtype check in memory planning pass, that allows memory inplace happen only when input and output has same dtype. So registering FInplaceOption and FInplaceIdentity to quantize_v2 is enough.

Thanks @ZhennanQin ! Good to know that PlanMemory pass checks for the dtypes. will try out your patch later today.

I guess there are good reasons for not allowing inplace operations for inputs. One reason could be that the data allocation for inputs is controlled by the user instead of engine and it should not be mutated. We can argue that identity operations should still be supported, but this means that the memory cannot be reused if identity is followed by an inplace operation. Also, the reference count for reused input memory will never drop to 0. This means that it cannot be recycled and reused in the computation graph.
According to me, even if we cannot do inplace optimization we should still add support in the operator for int8 to make the user experience easier.

@anirudh2290 Thanks for having a look on this. I agree with your point that user experience is important. Then I will add operator support without inplace optimization. In future, we can implement another pass to delete such redundant op to get it optimized.

PR is updated to address all comments received. Please review again. Thanks. @anirudh2290 @zhreshold @reminisce

Looks like other comments have been addressed. Merging now.


Allow assigning values to a subset of a dataset

Both, positional and label-based (with .loc) indexing using
a dictionary as key are implemented.
All variables in the dataset are updated one by one with the given
value at the given location.
If the given value is also a dataset, corresponding variables
in the given value and in the dataset to be changed are selected.

Tests for all cases are added.

[x] Closes #3015
[x] Tests added
[x] Passes pre-commit run --all-files
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ]  Documentation still needs to be updated

I haven't updated the documentation yet, where it says that this feature is not supported yet. Do you think we need example code for this feature in the documentation?

@matzegoebel forgive the very long delay on the review. We're planning to find a better system to ensure these don't drop through.
I would be up for adding this, for consistency. I don't think I've ever needed the functionality, but it also doesn't make the interface more complicated given it's mirroring __getitem__.
We probably need to think through whether there are any corner cases here; I can't think of any atm.
Any other thoughts?

Strong +1 from me on narrowing scope whenever possible. Adding features
incrementally is much easier than doing things all at once :)
On Sun, Apr 18, 2021 at 6:15 PM Maximilian Roos @.***>
wrote:

@.**** commented on this pull request.
In xarray/core/dataset.py
https://github.com/pydata/xarray/pull/5045#discussion_r615490870:

       # loop over dataset variables

       for var, da in self.items():

           val = value

           if type(value) == xr.core.dataset.Dataset:

               val = value[var]

           # only set value if all dimensions are present

           if all([k in da.dims for k in key.keys()]):

               da[key] = val

V good point. Thanks.
Either a) raising on missing dimensions or b) "entirely set to zero" in
this case, would be reasonable imo.
To the extent you're less confident that (b) is correct, I'd suggest we
move forward with (a) and evaluating whether we should do (b) separately.
(though @shoyer https://github.com/shoyer if you're up for it, let's
discuss the general pace of reviews next week and whether you think this
sort of "narrowing" of PR scope is a reasonable tactic)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/5045#discussion_r615490870, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJJFVQR6Q6VI44EEY2SF43TJN7ZTANCNFSM4ZKTDBUA
.

ok I tried to include your suggestions. Concerning @shoyer's point 3: Since I guess there could be a lot of different errors appearing, I created a copy of the data to be changed to check if the setitem fails before doing the actual update. That's of course suboptimal concerning the performance. What do you think, should we include checks for all conceivable errors or keep this test update?

I don't understand what's the issue of the failing test. Do you?

that's a flaky test which randomly fails (see #4539). You can safely ignore it, the CI should pass on the next run.

Great, this is shaping up.
I think we can find a way of failing early on bad indexes without attempting the whole operation on a copy.
At the very least, we could call __getitem__ with the indexes and see whether that passes. There may be better ways yet.
I also think that because the currently proposed code uses a shallow copy, it may be mutating the original when bad indexes are passed — it's worth adding a test to confirm.

Calling  getitem is not enough to detect all possible errors, I guess. Another possibility would be to do a deep copy before the assignments, and if anything goes wrong, restore the original data from the copy. In this way, the assignments do not have to be done twice unless an error appears.

ok, I deleted the copy stuff and included a few checks to catch possible errors before setting the values. Did I miss anything? How do we check for "type errors that don't coerce", as you mentioned?
The setitem method of the LocIndexer now calls the setitem method of the Dataset class, so that we don't have redundant code.

ok, I deleted the copy stuff and included a few checks to catch possible errors before setting the values. Did I miss anything? How do we check for "type errors that don't coerce", as you mentioned?

Excellent. Re the checks — I mostly meant that it was going to be very rare for something to get through — I don't think it's necessary to check for something like "type errors that don't coerce".

@shoyer thanks for your suggestions! I included them as best as I could.

I revised the pre-assignment checks. In my opinion xr.align is not so helpful when checking that the dimension sizes and coordinates are consistent, because it doesn't fail when the dimension size of the two Datasets is different, but the coordinate of the second Dataset is a subset of the first one. Therefore, I reimplemented the check that I had previously in a similar way. I also added a check for the wrong order of the dimensions, that you mentioned @shoyer.
If, despite the checks, an error occurs during the assignment, e.g. due to a type error, and the dataset has been updated already partially, the user is informed about this.

I revised the pre-assignment checks. In my opinion xr.align is not so helpful when checking that the dimension sizes and coordinates are consistent, because it doesn't fail when the dimension size of the two Datasets is different, but the coordinate of the second Dataset is a subset of the first one.

Could you kindly elaborate on this issue, maybe with a specific example?

If, despite the checks, an error occurs during the assignment, e.g. due to a type error, and the dataset has been updated already partially, the user is informed about this.

np.can_cast with casting='unsafe can check this. It sounds like this would probably be something good to add to our checks :)

Could you kindly elaborate on this issue, maybe with a specific example?
I think I somehow forgot the join="exact" when testing the functionality of xr.align. So nevermind, I'll reimplement again. :P

np.can_cast with casting='unsafe can check this. It sounds like this would probably be something good to add to our checks :)
Ok good point. I'll give it a try.

np.can_cast with casting='unsafe can check this. It sounds like this would probably be something good to add to our checks :)

I'm not sure how to use this, because np.can_cast with unsafe casting mostly returns True, e.g. np.can_cast(str, float, casting="unsafe")==True. The error that is implemented in the tests would not be caught then. But I could explicitly try to cast the new values to the datatype of the original values with .astype. Would that be an option?

I'm not sure how to use this, because np.can_cast with unsafe casting mostly returns True, e.g. np.can_cast(str, float, casting="unsafe")==True. The error that is implemented in the tests would not be caught then.

Oh, good point, thanks for checking.

But I could explicitly try to cast the new values to the datatype of the original values with .astype. Would that be an option?

Yes, I like this idea!

ok done

could you also resolve the merge conflicts? It seems we can't "approve and run" the CI with conflicts.

could you also resolve the merge conflicts? It seems we can't "approve and run" the CI with conflicts.
Ok I resolved them

I've fixed the location of the whats-new note. Is this ready to go in?

I've fixed the location of the whats-new note. Is this ready to go in?

Yes. I have one typing question but we can merge regardless if needed

Thanks a lot @matzegoebel !

Thanks for your help @max-sixty and @shoyer!

I guess we should also add this feature to the documentation, right?

Docs would be great! Particularly if the current docs are out of date now. Thanks @matzegoebel


[Deployment] 2021.11.18

Gallery
Changes

~https://github.com/NuGet/NuGetGallery/pull/8869~
https://github.com/NuGet/NuGetGallery/pull/8870
~https://github.com/NuGet/NuGetGallery/pull/8867~

Progress

[x] Merge PR: https://github.com/NuGet/NuGetGallery/pull/8875, #8888
[x] Build: https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=5474859&view=results
[ ] Release: https://dev.azure.com/devdiv/DevDiv/_releaseProgress?_a=release-pipeline-progress&releaseId=1214009
[ ] DEV
[ ] INT
[ ] PROD

Email publisher job
Changes

Certificate authentication.

Progress

[ ] Release: https://dev.azure.com/devdiv/DevDiv/_releaseProgress?_a=release-pipeline-progress&releaseId=1214114
[x] DEV
[ ] INT
[ ] PROD

I've reassigned this to @dannyjdev as there is a critical change only advisory in effect from November 22 08:00 UTC to November 30th 08:00 UTC. This deployment can be resumed next week when the advisory is lifted.


Feat: form notifications

TIS21-5159

Remember: purge https://eu-west-2.console.aws.amazon.com/sqs/v2/home?region=eu-west-2#/queues/https%3A%2F%2Fsqs.eu-west-2.amazonaws.com%2F430723991443%2Ftis-trainee-notifications-prod-form-updated.fifo before pushing this to prod, otherwise a lot of 'stale' notifications will be sent.


fn.now takes optionally a precision argument.

fn.now can optionally take a precision argument, so reflect that in the TypeScript definition.

Released in 0.20.10.


ghc-mod version

I get
syntastic: error: checker haskell/ghc_mod: can't parse version string (abnormal termination?)

Running ghc-mod doesn't give me the version number
:echo system('ghc-mod')
ghc-mod: No command given (try --help)

Problem might lie with the fact that I have a git version of ghc-mod, since the current hackage version doesn't compile with ghc7.10.
~ ❯❯❯ ghc-mod --version
ghc-mod version 0 compiled by GHC 7.10.1

You could try writing a wrapper like the following and pointing g:syntastic_haskell_ghc_mod_exec to it:
#! /bin/sh
if [ $# -eq 1 -a x"$1" = x--version ]; then
    echo ghc-mod version <IP_ADDRESS> compiled by GHC 7.6.3
else
    exec ghc-mod "$@"
fi

That said however, unreleased versions of the checkers are explicitly not supported by syntastic.  I have enough fun making syntastic keep pace with the released ones.

I tried putting in the script, but it didn't help with the error.
On the other hand, exec system() works as expected.
:echo system('ghc-mod --version')

ghc-mod version <IP_ADDRESS> compiled by GHC 7.6.3

I tried putting in the script, but it didn't help with the error.

What did you do exactly?

~ ❯❯❯ echo $PATH
/home/atondwal/bin:/home/atondwal/0-research/weimer/liquid/rolph-haskell/.cabal-sandbox/bin:/opt/ghc/7.8.4/bin:/home/atondwal/.cabal/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
~ ❯❯❯ cat bin/ghc-mod
#! /bin/sh
if [ $# -eq 1 -a x"$1" = x--version ]; then
    echo ghc-mod version <IP_ADDRESS> compiled by GHC 7.6.3
else
    exec ~/.cabal/bin/ghc-mod "$@"
fi
~ ❯❯❯

Don't rely on Vim seeing the $PATH you see in a terminal.  Achieving that is possible, but on most environments is far from trivial. :)
What I was suggesting was something like this:

put the wrapper script in, say, $HOME/bin/ghc-mod.sh
let g:syntastic_haskell_ghc_mod_exec = $HOME . '/bin/ghc-mod.sh'

syntastic does execute the right ghc-mod script, but the wrapper doesn't fix version error. I think that syntastic is maybe doing something more interesting than calling ghc-mod --version to get the version?

Please try it my way before going any further.

  The incantation is ghc-mod version, rather than ghc-mod --version.

Yes, this is after doing it your way.

changed the script to use version instead of version and everything works great!


TODO: Improve breaking upgrade repair issue

Today we create a repair issue when we detect an incorrect server version: https://github.com/home-assistant/core/blob/dev/homeassistant/components/zwave_js/init.py#L161-L169
We should improve the mechanism to point users down the right path when we can

The main reason why we require breaking changes is when new properties are added or new commands are added in the server that we want to add support for in the integration. If we wanted to, there are two options that would eliminate the breaking changes, neither of which I like, but for full transparency:

Rely on the require_schema parameter in each model method in the lib. The end result would be that, if no other changes in HA were made, some functions would throw an error because the require_schema would be too high.
Add code to HA that selectively adds functionality based on the schema version of the server.

In my opinion, 1 results in a poor user experience and 2 adds unnecessary complexity. But they would allow us to stop bumping the min version as often as we could be selective about when we applied these strategies and when we decided a minor version bump was needed