[SIP-80] Traversing Hierarchical Data ("Drill Down")

[SIP-77] Proposal for Traversing Hierarchical Data ("Drill Down")
Motivation
Many data sets have a natural hierarchy.  When viewing these data sets, a common user request is to apply filters for each tier of the hierarchy iteratively.  While superset provides accommodations to define these filters, the available methods are far from intuitive.  A more appropriate user interaction would be:

Display only values for the highest tier in the hierarchy
User clicks one of the displayed values
Superset filters based on the selected value
Return to 1, showing only values for the next tier

Proposed Change
The proposed feature will provide an "enable drill down" check box on the control panels for all compatible chart types.  Examples of possible compatible charts include pie, bar, tree maps, etc.  When selected, the feature will use the column names in the "group by" field, in the order that they appear, as the data hierarchy.  The iteratively selected filters will, as usual, be applied across all charts in the current filter scope (tab, dashboard page, etc.).  The current state of the "drill down" will be tracked in the chart's OwnState object, as necessary.
For a demonstration of the proposed change on a pie chart, see the videos on https://github.com/apache/superset/pull/14688
New or Changed Public Interfaces
The only change to the interfaces, as envisioned, is the addition of the "enable drill down" checkbox to appropriate charts' control panels.
New dependencies
n/a
Migration Plan and Compatibility
n/a
Rejected Alternatives
n/a

I would also love seeing drill down/through functionality added to Superset! I think there are roughly two ways forward here:

adding drill down in the chart layer, much like has been done in #14688
adding support for hierarchical dimensions to the semantic layer.

Both probably have their pros/cons:

Adding drill down to a single chart is fairly straight forward and wouldn't require a lot of changes to the rest of the application. On the downside it's slightly hacky and might introduce some design debt.
Adding semantics for defining hierarchical dimensions would make it possible to introduce drill downs into any chart (including native filters), but on the downside it's a fair bit of work and could take long to implement.

As we're getting ready to kick off work for Superset 2.0, I'd personally vote for the second option, as I feel having full drill down support would be a very valuable (and cool!) feature.
@betodealmeida , has any consideration been given to adding hierarchical dimension semantics to the new dataset models?

First of all, thanks @EvilDrW and @zhaoyongjie for moving this forward. I believe this functionality is a must have and would love to see it in Superset.
However, I also feel we should go for the most scalable solution rather than implementing the drill down on a chart by chart basis.

Adding semantics for defining hierarchical dimensions would make it possible to introduce drill downs into any chart (including native filters), but on the downside it's a fair bit of work and could take long to implement.

As we're getting ready to kick off work for Superset 2.0, I'd personally vote for the second option, as I feel having full drill down support would be a very valuable (and cool!) feature.

I agree with what @villebro is proposing.

@geido @villebro Thank you for your feedback.
Is there an estimate of when this would do with the 2nd approach? Is the target release date for Superset 2.0?
Keith is an engineer here at Stylo. As a company that actively uses superset this functionality is crucial for us and I would love if we could get this in as an MVP, while the team is working on the most scalable/long-term solution.

@betodealmeida , has any consideration been given to adding hierarchical dimension semantics to the new dataset models?

@villebro We have not, but we definitely should. It could be as simple as defining the parent column in the dataset, eg, in city we'd set state as the parent dimension, in state we'd set country, and so on.
@EvilDrW I have questions about additive metrics. When you're at the country level, how do you compute the metrics, do you run a query with only country in the group by or do you run the full group by (country, state, city) and add the values? If the latter we should think how we're going to handle non-additive metrics like COUNT DISTINCT — this is another area where the semantic layer can help.

When does superset plan to support drill down？
thanks~

@villebro @geido @betodealmeida: thanks for the feedback.
I'm sure that what you're requesting is the right way forward for the project, but it is quite far beyond both my interest and skill level.  I'll be interested to see how this progresses, but it will have to be someone other than me who implements it.

Hey there, I renamed this to SIP-80 since there's already a SIP-77: https://github.com/apache/superset/issues/17296

When is "Drill Down" feature planned to implemented in Superset Roadmap ?

When is "Drill Down" feature planned to implemented in Superset Roadmap ?

+1
Waiting for drilldown feature!

+1 waiting for drilldown and drill through feature!!

Some initial drilling features are being developed in the normal course of feature work. This SIP may require substantive updates to keep current with that effort if you wish to keep it open. It will also need to be followed through with on the dev@ mailing list to close out the discussion and bring it to a vote. Thanks!

initial drilling features-- > u mean sunburst chart?  any more choice? thanks

initial drilling features-- > u mean sunburst chart? any more choice? thanks

By this I mean that recently there has been the addition of Drill to Details, which many of the newer charts support. This is currently behind a feature flag, but allows you to see the underlying data that a chart represents.
The next planned feature is Drill By, where you can right-click a chart, and quickly add a filter and select an additional column to Group By. This will essentially allow drill-down (and more), but you need to know the columns/schema/hierarchy of the data.
Following that, adding hierarchical relationships between columns at the semantic layer should allow the Drill By feature to be extended into Drill Down/Up actions more easily.

@rusackas Thanks for bringing these news to us. This sounds like great news or am I missing something?
Am I getting it right, that some of the ideas proposed in this issue are already implemented or is on the roadmap to be implemented soon? If that is the case, I wonder if the work done/being done is inspired by the description in this issue, and if not, how come (this question is coming from someone who lack an understanding of the project org)?
Also, where would I be able to read more about the feature flag and the planned features? Where do these discussions and design ideas/developments live?
Thanks

A lot of folks have wanted this feature for years, so there's been input/inspiration from numerous sources. A lot of conversation has been happening in various open fora, including the #drilling-actions channel on Slack, and in other small working groups that have spun out of Town Hall discussions. Thus far, Preset has done the bulk of the work, but other individuals and orgs are more than welcome to join these discussions, especially if there's an intent to contribute to the implementation effort.
The hierarchical mapping at the semantic layer is an un-tackled design problem thus far, which will be given more attention after those other precursor steps are production-ready. Drill to Detail is already available as mentioned, and we're fairly early in Drill By, but I'll ping @kasiazjc @lauderbaugh in case we have any Drill By design/docs we're ready to share out in this thread.

So I'm wondering when will this feature be released?

With whichever release is the next cut of master. That might be 2.1 or 3.0 depending on which way the wind blows. Hopefully by early spring... though it's largely a volunteer effort, so take that with a grain of salt ;)

Hey all... I'm closing this feature for a few reasons:
• It was never brought up for discussion or a vote
• DRILL TO DETAIL has shipped, and is on by default as of Superset 3.0
• DRILL BY has shipped, and is on by default as of Superset 3.0
If ayone wants to continue discussing/proposing additional drilling features, I propose that we regroup on a new SIP and rekindle the process.

who can share how to configure drill-by using "header table" and go down to "detail table" clicking on a dimension.
thanks


problem in build

I have this problem when building a Java app
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 203, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/metrics/decorator.py", line 18, in wrapper
    result = fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 368, in build
    self.project.build(
  File "/usr/lib/python3/dist-packages/compose/project.py", line 519, in build
    build_service(service)
  File "/usr/lib/python3/dist-packages/compose/project.py", line 501, in build_service
    service.build(no_cache, pull, force_rm, memory, build_args, gzip, rm, silent, cli, progress)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 1111, in build
    return builder.build(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 1948, in build
    with open(iidfile) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmphcevs75f'

We have removed the python version of compose in latest/candidate channel (replaced by the go version), could you try that?

We have removed the python version of compose in latest/candidate channel (replaced by the go version), could you try that?

How can I have that version?
I installed it by command snap install docker

How can I have that version? I installed it by command snap install docker

You can try with snap refresh docker --candidate

Did that work for you? Can I close this issue?

Did that work for you? Can I close this issue?

yes, thanks


Add environment support

The client app should be configurable on a per-environment basis, such that local development and staging environments can be used.

Switched to top priority because this issue blocks #9.

Environment support has been added for dev environments. Skipping stage for now.


⚠ Bofrost.com has degraded performance

In 4d8e8a7, Bofrost.com (https://www.bofrost.com) experienced degraded performance:

HTTP code: 200
Response time: 586 ms

Resolved: Bofrost.com performance has improved in dad687a after 6 minutes.


fix-120578 don't offer callstack step-in/over/out when not stopped

This PR fixes #120578
After this change, when running:

When stopped:

/assign @isidorn

Improved so the step-buttons get disabled rather than hidden. This means that the pause/continue button remains in the same position.

Love this fix ❤ . Thanks a lot 👏


Is there any pp-tsm feature extractor available?

Dear author, I want to use the table tennis temporal location project for my job, however, the input feature is extracted by PP-TSM from an original video. Could you please provide a PP-TSM feature extractor if it is convenient, thank you very much.

Hi there,
You can check https://github.com/PaddlePaddle/PaddleVideo/blob/develop/applications/FootballAction/extractor/extract_feat.py to modify based on your demands. It is a script for football but with a similar feature extraction process in table tennis.


Add ratios to the meh and woot feature

Closing this because this is server-sided. Might reopen later


tf.load_op_library doesn't load .so file and throws file not found error although the file exists

**System information**
== check python ===================================================
python version: 3.7.7
python branch:
python build version: ('default', 'Jan 13 2021 07:47:31')
python compiler version: GCC 7.5.0
python implementation: CPython

== check os platform ===============================================
os: Linux
os kernel version: 69-18.04.1-Ubuntu SMP Wed Feb 9 15:36:54 UTC 2022
os release version: 5.4.0-1066-aws
os platform: Linux-5.4.0-1066-aws-x86_64-with-debian-buster-sid
linux distribution: ('debian', 'buster/sid', '')
linux os distribution: ('debian', 'buster/sid', '')
architecture: ('64bit', '')
machine: x86_64

== are we in docker =============================================
No

== compiler =====================================================
c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

== check pips ===================================================
numpy                            1.21.5
protobuf                         3.14.0
tensorflow                       2.8.0
tensorflow-estimator             1.14.0
tensorflow-hub                   0.11.0
tensorflow-io-gcs-filesystem     0.24.0
wsproto                          1.0.0

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.version.VERSION = 2.8.0
tf.version.GIT_VERSION = v2.8.0-rc1-32-g3f878cff5b6
tf.version.COMPILER_VERSION = 7.3.1 20180303
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- TensorFlow installed from (source or binary):binary

== env ==========================================================
LD_LIBRARY_PATH /usr/lib64/openmpi/lib/:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib:/opt/amazon/efa/lib:/usr/local/mpi/lib:/opt/amazon/openmpi/lib:/usr/lib64/openmpi/lib/:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib:/opt/amazon/efa/lib:/usr/local/mpi/lib:/opt/amazon/openmpi/lib:/usr/lib64/openmpi/lib/:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

== cuda libs  ===================================================
/usr/local/cuda-9.0/lib64/libcudart_static.a
/usr/local/cuda-9.0/lib64/libcudart.so.9.0.176
/usr/local/cuda-9.0/doc/man/man7/libcudart.7
/usr/local/cuda-9.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
/usr/local/cuda-10.2/doc/man/man7/libcudart.7
/usr/local/cuda-10.2/doc/man/man7/libcudart.so.7
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so.10.1.243
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-10.1/doc/man/man7/libcudart.7
/usr/local/cuda-10.1/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.2/lib64/libcudart_static.a
/usr/local/cuda-9.2/lib64/libcudart.so.9.2.148
/usr/local/cuda-9.2/doc/man/man7/libcudart.7
/usr/local/cuda-9.2/doc/man/man7/libcudart.so.7
/usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
/usr/local/cuda-10.0/lib64/libcudart_static.a
/usr/local/cuda-10.0/doc/man/man7/libcudart.7
/usr/local/cuda-10.0/doc/man/man7/libcudart.so.7

== tensorflow installed from info ==================
Name: tensorflow
Version: 2.8.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author-email<EMAIL_ADDRESS>License: Apache 2.0
Location: /home/ubuntu/venvpipe1v1/lib/python3.7/site-packages
Required-by: spacy-universal-sentence-encoder

== python version  ==============================================
(major, minor, micro, releaselevel, serial)
(3, 7, 7, 'final', 0)

== bazel version  ===============================================

Describe the current behavior
When the code below is executed
print(config.COREF_KERNELS_SO_FILE_PATH.is_file())
coref_op_library = tf.load_op_library(str(config.COREF_KERNELS_SO_FILE_PATH))

The first line prints True which means that the file exists. But the second line throws the following error.
 File "/home/ubuntu/venvpipe1v1/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library     lib_handle = py_tf.TF_LoadLibrary(library_filename) tensorflow.python.framework.errors_impl.NotFoundError: libtensorflow_framework.so.1: cannot open shared object file: No such file or directory
Describe the expected behavior
The .so file should load

Do you want to contribute a PR? (yes/no): no

Standalone code to reproduce the issue
import tensorflow as tf
coref_op_library = tf.load_op_librarypath to .so file')

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
2022-03-24 17:39:30.761737: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-03-24 17:39:30.761779: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

@nirnayr ,
Can you please take a look at this issue1, 2 and also TensorFlow release is compatible with a certain version, for more information please take a look at the tested build configurations.

Tried linking libtensorflow_framework.so and libtensorflow_framework.so.2 using ln as suggested by https://stackoverflow.com/questions/56888781/tensorflow-notfounderror-libtensorflow-framework-so-cannot-open-shared-file-or?noredirect=1&lq=1
I have tried appending LD_LIBRARY_PATH with the libtensorflow_framework.so file as suggested by this thread https://github.com/tensorflow/tensorflow/issues/30488
I have tried downgrading to tensorflow 1.14, with Python 3.7. As that was the latest compatible according to https://www.tensorflow.org/install/source#gpu

Tried linking libtensorflow_framework.so and libtensorflow_framework.so.2 using ln as suggested by https://stackoverflow.com/questions/56888781/tensorflow-notfounderror-libtensorflow-framework-so-cannot-open-shared-file-or?noredirect=1&lq=1
I have tried appending LD_LIBRARY_PATH with the libtensorflow_framework.so file as suggested by this thread https://github.com/tensorflow/tensorflow/issues/30488
I have tried downgrading to tensorflow 1.14, with Python 3.7. As that was the latest compatible according to https://www.tensorflow.org/install/source#gpu

@nirnayr,
tf.sysconfig.get_link_flags(). returns ['-L/usr/local/lib/python3.7/dist-packages/tensorflow',  '-l:libtensorflow_framework.so.2']
Here, library directory is  -L and library is -l. Now replace the -L and -l  with library_dirs and libraries arguments to your Extension object in setup.py.
Finally, set your LD_LIBRARY_PATH environment variable to point to the directory with the libtensorflow_framework.so library is located.


Getting errors while running

Getting errors while running the script .For converting CSV to PPT (.\GenerateWAFReport.ps1)

Getting errors while running the script .For converting CSV to PPT (.\GenerateWAFReport.ps1)


Automated cherry pick of #758: fix go vet report bugs

Cherry pick of #758 on release/2.9.0.
#758: fix go vet report bugs

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.9.0@3abce70). Click here to learn what that means.
The diff coverage is 0%.

@@               Coverage Diff               @@
##             release/2.9.0    #760   +/-   ##
===============================================
  Coverage                 ?   7.33%
===============================================
  Files                    ?     347
  Lines                    ?   55722
  Branches                 ?       0
===============================================
  Hits                     ?    4085
  Misses                   ?   51259
  Partials                 ?     378

Flag
Coverage Δ

#aFlag
7.33% <0%> (?)

Impacted Files
Coverage Δ

pkg/mcclient/options/loadbalancerbackendgroups.go
0% <ø> (ø)

pkg/util/esxi/host.go
2.34% <ø> (ø)

pkg/mcclient/options/loadbalancercertificates.go
0% <ø> (ø)

pkg/mcclient/options/loadbalanceracls.go
0% <ø> (ø)

pkg/mcclient/modules/notify/mod_notification.go
66.66% <ø> (ø)

pkg/cloudcommon/db/namevalidator.go
0% <ø> (ø)

pkg/util/aws/imageowners.go
0% <ø> (ø)

pkg/mcclient/options/servers.go
0% <ø> (ø)

pkg/mcclient/options/loadbalanceragents.go
0% <ø> (ø)

pkg/compute/models/loadbalancerbackendgroups.go
2.02% <0%> (ø)

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3abce70...011e808. Read the comment docs.

/approve
/lgtm


Update: outreach committee mailing list w/marketing POCs

This is complete, as this is mostly contractual. However, the CDF routinely contacts our members to make sure they update their POCs. Let me know if you have any questions.

post GB meeting we will want to send out an email blast to the OC and make sure the marketing contact is accurate for member orgs. Suggest closing this and creating one that reflects and maintenance workflow


Figure out appropriate JVM version

Perhaps related to #154
We're installing both default-jdk and openjdk-8. Default now depends on the openjdk-11 package, which identifies as Java 10.0.1. We probably should only have one JVM version installed.

I don't see a great harm in having both installed; I feel like if we pick one default-jdk might be better. jGRASP seems to work well enough; I haven't seen any DrJava issues; Eclipse Photo should be fine on 10 and 11. We might be able to choose a default using the Alternatives module.
Hopefully we'll see a final release of jGRASP 2.0.5 which has Java 10-11 support before we ship this fall, but if we don't, maybe we can just set the alternative to Java 8.

jGRASP 2.0.5 is RTM now. When skipping openjdk-8-jdk installation, DrJava fails to launch with this error: https://sourceforge.net/p/drjava/support-requests/344/
openjdk-8 removal patch is available on my remove-jdk8 branch, but I'm skipping the PR for now until I understand this DrJava failure.

Since the last commit to DrJava was in 2014, we probably shouldn't hold our breath on that being fixed. We'll probably just need to ship openjdk-8-jdk and openjdk-11-jdk in the fall.
We should use this as a reason to get in touch with our CS 149 professors to see if they even use DrJava anymore or if they're all recommending jGRASP so we can take care of this for Spring 2019.
Assuming this works for you, we can either close this or retarget it to Spring 2019.

I'm going to retarget this for Spring 2019. Keeping JDK8/DrJava at the moment is the least risky option for getting this image shipped.

Based on some discussions with professors, it sounds like DrJava is slowly being phased out in favor of jGRASP. We'll see what happens in the spring and fall, but my understanding is that for the most part, the department has transitioned to using Java 10 and therefore using Java 10-compatible IDEs.
Maybe in that week after Thanksgiving, we can start surveying our professors on which IDE they plan to use for CS 149.


[Fix #11037] Fix a false positive for Style/CollectionCompact

Fixes #11037.
This PR fixes a false positive for Style/CollectionCompact when using to_enum.reject or lazy.reject methods with Ruby 3.0 or lower.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Thanks!


Don't allow releases in the evening / on fridays

We just came up with the idea to block releases after a certain time in the afternoon and maybe even on fridays in general.
Of course there should be an option to override this barrier by giving a special parameter to the function (or something like that).

We could call the flag to force a release --yes-mom-i-know-this-is-a-bad-idea.

IMO this is not the responsibility of this repo at all. The scs-commander is a tool/library for interacting with the Shopware Community Store, not for babysitting developers.


[] == ![], they are not being both converted to numbers

The spec says, in the ninth clause of https://www.ecma-international.org/ecma-262/#sec-abstract-equality-comparison, that if the right side is boolean, then the right side will become a number. That is;
[] == ![]
[] == false
[] == 0
Now, the spec (in the eleventh clause of the aforementioned section) says that an object on the left side (this is, being x) compared to a number will reduce to ToPrimitive(x) == y. Calling ToPrimitive on an array A is the same as calling A.join(), and [].join() === "". That is;
[] == 0
"" == 0 //true
Thus, it is not true (as we are saying in our first example) that both operands get converted to 0 (moreover, they are not being both converted to numbers). Can I please send a PR if this reasoning is valid?

Feel free to open a PR. Thanks!

Fixed by #150

@denysdovhan I don't think this one is fixed by #150 . May I please still open a PR with the explaination above? (I've been busy, sorry).


[NEW ICON] Pandas

Skill Name
PANDAS
Why?
pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.
Reference Image

Icon added on https://github.com/lelouchfr/skill-icons


Testing & Refactoring by Remco Tjeerdsma & Mart Oude Weernink

This pullrequest contains the added JUnit test cases as well as the refactorings in the texture package.

Sorry, not squashed yet.

Squashing is not working for us. Sorry!

Replaced by PullRequest #23, which contains one single squashed commit.


Changes to Sitemap for static views section

Changes to Sitemap for static views section of sitemaps.txt docs.
These are minor changes notices while working with Django.

class StaticViewSitemap(Sitemap):

Current:
class StaticViewSitemap(sitemaps.Sitemap):
Proposed:
class StaticViewSitemap(Sitemap):
Since in the import from django.contrib.sitemaps import Sitemap sitemaps is already imported so its not required in class.
Same section # urls.py:
Current:
sitemaps = {
'static': StaticViewSitemap,
}
Proposed:
sitemap = {
'static': StaticViewSitemap,
}
Since in the import is sitemap
from django.contrib.sitemaps.views import sitemap
we should write sitemap in urls.py not sitemaps.

The code looks correct as it is.


Raw Json

how would you get just the raw json that is returned from a query and not a typed object

@aday12345, can you explain the use case here? Do you want the raw query response as json?
public class QueryResult<T>
    {
        [JsonProperty(PropertyName = "nextRecordsUrl")]
        public string NextRecordsUrl { get; set; }

        [JsonProperty(PropertyName = "totalSize")]
        public int TotalSize { get; set; }

        [JsonProperty(PropertyName = "done")]
        public bool Done { get; set; }

        [JsonProperty(PropertyName = "records")]
        public List<T> Records { get; set; }
    }

Or do you just want the raw <T> as a json string?


Add StateReader with callbacks to Python.

This change is

Codecov Report

Merging #827 (b76fd5a) into main (9314c67) will decrease coverage by 1.38%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #827      +/-   ##
==========================================
- Coverage   66.78%   65.40%   -1.38%
==========================================
  Files          41       43       +2
  Lines        5504     5620     +116
  Branches     5504     5620     +116
==========================================
  Hits         3676     3676
- Misses       1481     1597     +116
  Partials      347      347

Files Changed
Coverage Δ

crates/native_blockifier/src/errors.rs
0.00% <0.00%> (ø)

crates/native_blockifier/src/lib.rs
0.00% <ø> (ø)

crates/native_blockifier/src/py_contract_class.rs
0.00% <0.00%> (ø)

crates/native_blockifier/src/py_state_reader.rs
0.00% <0.00%> (ø)

crates/native_blockifier/src/py_utils.rs
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Make main script runnable as a module

This PR features a really simple edit that remove the need for path manipulation: this way, travis_cargo can be run as a python module with the -m flag:
python -m travis_cargo test ...

as ~/.local/lib/python-*.*/site-packages will always be in the Travis-CI $PYTHONPATH.

Thanks for the pull request, and welcome! You should hear from @huonw (or someone else) soon.


pythonPackages.bcrypt: add missing libffi dependency

Motivation for this change
Broke HA without the libffi dependency
Cc: @fridh @dotlambda
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

If I am correct we use bcrypt with cffi instead, which as far as I know works correctly.

I'll close this again. We can revisit it later if needed.


camel-telegram: small improvements

Added fallback converter for generic types.
Added configuration at component level.

Merged.
(1st commit as Apache Committer :D)

YAY!

🎉


Error while training the model on any data

hello all,
im trying the get the model to train on the same data provided by the authors, I was able to recreate the results using the pre-trained weights and everything worked pretty much fine. I am using Windows and running tf on GPU.
The output I am getting is as follows-
Exception in thread background:
Traceback (most recent call last):
File "C:\Users\admin\Documents\temp\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\admin\Documents\temp\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\admin\Desktop\Lip2Wav-master\Lip2Wav-master\synthesizer\feeder.py", line 139, in _enqueue_next_train_group
examples = [self._get_next_example() for i in range(n * _batches_per_group)]
File "C:\Users\admin\Desktop\Lip2Wav-master\Lip2Wav-master\synthesizer\feeder.py", line 139, in
examples = [self._get_next_example() for i in range(n * _batches_per_group)]
File "C:\Users\admin\Desktop\Lip2Wav-master\Lip2Wav-master\synthesizer\feeder.py", line 194, in _get_next_example
input_data, mel_target = self.getitem()
File "C:\Users\admin\Desktop\Lip2Wav-master\Lip2Wav-master\synthesizer\feeder.py", line 172, in getitem
mel = np.load(os.path.join(os.path.dirname(img_name), 'mels.npz'))['spec'].T
File "C:\Users\admin\Documents\temp\lib\site-packages\numpy\lib\npyio.py", line 416, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'Dataset/chem//preprocessed/46\mels.npz'
The name of the files are numbers, so the "46" is pointing tawrds the directory where all the images are with the original .wav file.
PLEASE HELP! I have been stuck on this issue for a week now:(

Dataset/chem//preprocessed/46\mels.npz

The slashes are in the wrong direction for Windows.

Dataset/chem//preprocessed/46\mels.npz

The slashes are in the wrong direction for Windows.

Hi! thank you so much for your response!
unfortunatley, that was not the probkem. I changed the slashes and could make it to work. Furthermore, upuntil this point, I did not run into any issue with the slashes while infering the network.
WHat I seem to be missing is the point in the code the CREATES the .npz files - they are not being created at all from merely running the train.py commend, so the issue should be somewhere in the creating the .npz and later on the .npy, right?
Thanks for helping me out!

Hi! thank you so much for your response!
unfortunatley, that was not the probkem. I changed the slashes and could make it to work. Furthermore, upuntil this point, I did not run into any issue with the slashes while infering the network.
WHat I seem to be missing is the point in the code the CREATES the .npz files - they are not being created at all from merely running the train.py commend, so the issue should be somewhere in the creating the .npz and later on the .npy, right?
Thanks for helping me out!

Hi! thank you so much for your response!
Unfortunatley, that was not the problem. I changed the slashes and could not make it to work. Furthermore, upuntil this point, I did not run into any issue with the slashes while infering the network.
What I seem to be missing is the point in the code that CREATES the .npz files - they are not being created at all from merely running the train.py commend, so the issue should be somewhere in the creating the .npz and later on the .npy, right?
Thanks for helping me out!

Hi! thank you so much for your response!
Unfortunatley, that was not the problem. I changed the slashes and could not make it to work. Furthermore, upuntil this point, I did not run into any issue with the slashes while infering the network.
What I seem to be missing is the point in the code that CREATES the .npz files - they are not being created at all from merely running the train.py commend, so the issue should be somewhere in the creating the .npz and later on the .npy, right?
Thanks for helping me out!

Have you preprocessed your data? https://github.com/Rudrabha/Lip2Wav#preprocessing-the-dataset

Have you preprocessed your data? https://github.com/Rudrabha/Lip2Wav#preprocessing-the-dataset

sure, I get one directory named "preprocessed" with all the images in JPG and one .wav file made "audio", all ordered in directories for 30 seconds segments of the original video. If I run infering for that, there is no problem, I believed that this means that I did the preprocessing stage correctly. Am I suppose to get an .npz or .npy from preprocess.py?

sure, I get one directory named "preprocessed" with all the images in JPG and one .wav file made "audio", all ordered in directories for 30 seconds segments of the original video. If I run infering for that, there is no problem, I believed that this means that I did the preprocessing stage correctly. Am I suppose to get an .npz or .npy from preprocess.py?

Am I suppose to get an .npz or .npy from preprocess.py?

Yes.

Am I suppose to get an .npz or .npy from preprocess.py?

Yes.

Got it! works perfectlly, thanks!

Got it! works perfectlly, thanks!


Database Update Failure Modes & Retry Duration

Hi there!
I've been working through integrating this library into Boulder to replace our usage of the legacy v3 API for checking domain names prior to issuing a certificate. I thought it would be valuable to provide some feedback of things we've noticed so far. There are some rough edges that perhaps you have solutions for!
In the process of integration testing with a mock server I've noticed the same behaviour as @smoya in https://github.com/google/safebrowsing/issues/37 - If the initial /v4/threatListUpdates:fetch call done after constructing a safebrowsing client results in error the library is unable to do any lookups until the situation is rectified (with the DefaultUpdatePeriod this is no sooner than 30 minutes later). This could also be caused by a failed update on refresh requests.
It seems like there should be a separate "knob" to twist for a retry period when an error occurs. If we change DefaultUpdatePeriod to attempt to recover more quickly we will also produce unnecessarily frequent lookups after the error is resolved.
It's also not ideal that the library isn't able to communicate when it is ready to provide lookups with a healthy database. We can work around this by performing our own polling on the client's Status function looking for a nil return value for the error return parameter but ideally there would be a way
for the integrating system to block on, or be notified of, when the database is healthy and the client is capable of doing lookups.
Thoughts? Suggestions?
Thanks for all your work on this project!

@colonelxc Are you the right person to bump about this?

Hi @cpu, thanks for your message, and apologies for the latency in responding. I'm supportive of your suggestions :)
We actually have a back-off spec in our public docs that this client doesn't currently abide by. Since backend errors are rare, implementing this policy should help in getting the client back to a healthy state in a more timely fashion.
+1 for a blocking subscription channel for notifying once the client is ready.
I'm not sure how quickly we'd be able to commit these changes, but we'd be happy to review any code if you'd like to get these in sooner.
Thanks!
Alex

Hi @cpu, thanks for your message, and apologies for the latency in responding. I'm supportive of your suggestions :)

Not a problem! Thanks for replying :-)

We actually have a back-off spec in our public docs that this client doesn't currently abide by. Since backend errors are rare, implementing this policy should help in getting the client back to a healthy state in a more timely fashion.

This seems sensible and would definitely be nicer than waiting for the next scheduled database refresh.

I'm not sure how quickly we'd be able to commit these changes, but we'd be happy to review any code if you'd like to get these in sooner.

Understood. I think for the short term we're going to stick with our existing V3 implementation and wait for the improvements we need in the V4 client library. It's unlikely I'll be able to find time in the next few sprints for a PR from our side but if we decide adopting V4 is a priority I'll revisit our engagement too.
Thanks!

It would be nice to have this implemented asap.
Thanks for your message @alexwoz

One half of the issues raised (the backoff behaviour) were fixed with #49. The other half will be addressed by #52 and I believe this issue should be closed after it is merged.

I believe this issue is fully resolved now. Thanks for being patient.


Code appears corrupted to guest in live share session

Describe what happened:
A live share session was made in VS Code by a host user and shared with a guest user, but the session didn't work at first (guest had to reconnect to the session a couple of times) as the file explorer view didn't load.
After it eventually loaded, the host made some changes to one of the files and now the guest wasn't able to correctly view those changes, as the code on the guest's end appeared corrupted.
Also, when the guest tried to make changes to the code, those changes wouldn't get reflected on the host's view, both when trying to make changes to the corrupted file, or to a different file, which seems to indicate after that point anything done on the guest end would not be reflected in the host end. The only things that seemed to still be reflected where text selections in the code view, but changes weren't reflecting anymore.
What was your system configuration?
Product and Version: VS Code 1.43.2
OS Version: Windows
Live Share Extension Version: 1.0.1891
Target Platform or Language: React/TypeScript
Screenshots
Host view:

Guest view (corrupted):

Hey @AbrahamLopez10 , is this consistently happening for you?

Only happened once, but was weird enough that I thought was worth reporting.

Of course! Thanks for reporting. This should never happen. I was just curious because if this was consistently happening for you, it'd actually be easier to figure out why.
Anyway, we're looking into it.

Apologies for the delay here. As is, this issue doesn’t have enough information to be reproduced and we've made improvements in this area. Please reopen if the issue persists with any additional details. Thank you.


Fix cannot reopen after pushing commits to a closed PR (#23189)

Backport: #23189
Close: #22784

On GH, we can reopen a PR which was closed before after pushing commits. After reopening PR, we can see the commits that were pushed after closing PR in the time line. So the case of
issue is a bug which needs to be fixed.

After closing a PR and pushing commits, headBranchSha is not equal to sha(which is the last commit ID string of reference). If the judgement exists, the button of reopen will not display. So, skip the judgement if the status of PR is closed.

Even if PR is already close, we should still insert comment record into DB when we push commits.
So we should still call  function CreatePushPullComment().

https://github.com/go-gitea/gitea/blob/067b0c2664d127c552ccdfd264257caca4907a77/services/pull/pull.go#L260-L282 So, I add a switch(includeClosed) to the
GetUnmergedPullRequestsByHeadInfo func to control whether the status of PR must be open. In this case, by setting includeClosed to true, we can query the closed PR.

In the loop of comments, I use thelatestCloseCommentID variable to record the last occurrence of the close comment.
In the go template, if the status of PR is closed, the comments whose type is CommentTypePullRequestPush(29) after latestCloseCommentID won't be rendered.

 e.g.
1). The initial status of the PR is opened.
 2). Then I click the button of Close.  PR is closed now.
 3). I try to push a commit to this PR, even though its current status is closed.
 But in comments list, this commit do not display.This is as expected ：）
 4). Click the Reopen button, the commit which is pushed after closing PR display now.

🎺 🤖


Lowercase Title :settling in MB

Describe the issue and expected behaviours. If this is a bug, describe the steps to reproduce the problem, what you observe, and what you expected to observe.
Acceptance Criteria

[ ] In the Recommended for you section: settling in MB should be Settling in MB

Steps for QA:

See if the title starts with uppercase

Dupe of https://github.com/pg-irc/pathways-frontend/issues/1565


Mod checker should always check for other TM:PE

Currently, if user turns off mod checker, it won't check for other TM:PE
That can cause the save/load issues we are all familiar with.
I'll update it to always check for TM:PE, but only check other mods (incompatible_mods.txt) if the user has enabled checks on startup.

I suggest that TM:PE should shut down if other TM:PEs are detected, and either the most stable would remain running, or all of the shut down.

If there are more than one TM:PE, regardless of whether enabled or not, it will still cause problems. See #211, particularly Dymanoid's comments.


Estimator as a trait

Very much a janky WIP, and some of the objects I introduced could use better naming though I avoided doing so in the interest of retaining compatibility with existing code. Haven't tested this against older rustc, but as of my current rustc version (latest stable) there are no lints from either regular check or clippy. But let me know if this seems like an ok direction and I'll revise it further.
Somewhen I'll modify that example to show what a more bumpy file transfer looks like, and the advantage of averaging it by the method I used. done

Note I moved the estimator to it's own module. It had to be public for this to work, and I'm not sure whether you guys would want to make the state module public, and it didn't really seem to fit anywhere else.

The more I think about this, the more I wonder what other use cases somebody might have for needing this level of access to the API, and I might be over-engineering it as a result. This might be better done with allowing the user to specify whether estimations are done per tick, as the default estimator does, or over n seconds, milliseconds, etc. I think my example merely demonstrates a case where the default estimator simply doesn't work at all (basically the numbers are completely meaningless to the user.)
I think the efficiency improvements I've made in my alternative estimator might even make it more efficient than the existing one, but at the same time it probably shouldn't outright replace the existing one as I could see many people preferring that per-tick estimation that is currently used, and per-tick should probably remain the default as it is much more precise.
Let me know what your thoughts are, or you can outright tell me that I'm totally bonkers and I'll accept that too :)

The last push I did gets everything working the way I need it to. Please see the rate_smoothing example for the overall effect of what it is I'm trying to do, with example real world use cases mentioned in the comments.
I'm also tempted to implement a lock-free incrementing function, but I need to evaluate whether that will offer much benefit in what I'm working on.

That's odd, I compiled against rustc 1.53 and there were no lints, and all tests passed. I even added some very aggressive clippy lints and they were all clear. I also ran the cargo format check exactly as those checks do. All of that was done before being pushed. I'll look at it again to see if something slipped by somehow.

It's very possible that clippy has gotten more strict since 1.53, especially if you enable all those extra strict stuff. 1.53 is pretty old at this point, maybe upgrade to current stable?

Oh I thought those tests were specifically against 1.53? Sure, I'll use the latest stable, I used rustup to override to 1.53 so that I could guarantee compatibility with 6 months prior as you had asked in my other PR.

We run the tests against 1.53 to ensure the library works well, but all the linking happens on stable.

For this to be mergable, I'd like to:

Have the map be a std::collections::HashMap, not a DashMap
Move the map back into ProgressStyle
Keep the Estimator as is for now
If the Default parameter doesn't work, we can add a default() method to the trait that yields a trait object

As I've tried to explain a few times, this should be a minimal change.

Not sure what you mean move it 'back'? The format_map is still in progressstate. That particular map I made into a dashmap had to be that way because it originally did live in progressstate, because it provides internal mutability (it was Arc, hence why those variables were named 'adm'. It doesn't need to be dashmap if it's in progressstate though.
While I can put some sort of default in there, I'm not sure where you are wanting it to be called from.
The only other thing I can do is require that statefulformatter is immutable with internal mutability, which complicates things a lot from the user side as it will probably need atomics even if they're not really appropriate for the use case, or it forces them into using locks. I also don't really know how to meet my own use case while falling within those constraints. I'm looking it over to see what can be done, but right now I don't know.

Not sure what you mean move it 'back'? The format_map is still in progressstate.

Sorry, I meant we should not add a map to BarState and only replace the format_map with something more capable.

While I can put some sort of default in there, I'm not sure where you are wanting it to be called from.

From the impl for ProgressStyle::clone().

The only other thing I can do is require that statefulformatter is immutable with internal mutability

Why? BarState::tick() has mutable access to the ProgressStyle, right?

Why? BarState::tick() has mutable access to the ProgressStyle, right?

Right but in order to live inside of progressstyle it has to be Clone. I don't know how to do that while also making it thread safe by adding Arc, which removes mutability. Unless progressstyle isn't meant to be cloned across threads?

You can remove the derived Clone impl in favor of manual Clone impl, that gives you the flexibility to do something different for the format_map values.

Ah I see. I'll chew on this for a bit.

I'm still not really understanding the use case for cloning ProgressStyle. When ProgressStyle is cloned, how do we resume the state from the existing instance? How do we keep the new one syncronized?

Well whatever the use case is, it doesn't seem to impact mine at all. Still a draft, there are a few things I need to clean up, but overall let me know if this works.

I think I'm mostly satisfied with what I have there. I'd really like the ability to remove or at the very least suppress the built-in estimator as it will be running (I think? unless the compiler sees it as dead code somehow and removes it anyways) even though my use case doesn't need it at all. Also I still don't understand the need for cloning ProgressStyle, but it doesn't interfere with anything I'm doing in any way so no problem there.

The hook is entirely for my own convenience, it will go once we've got everything else sorted out.
To a lesser extent, the example is as well. It mainly works to ensure that my use case remains valid without needing me to screw with my other repos in the meantime, and ultimately will provide a working model after.
Just let me know once we're down to these two and I'll take them out last

You could just keep the example in your local workdir uncommitted, similar for the rusty-hook config (though that should really just be a Git pre-push hook probably). Putting stuff in your PR that is not slated to be merged makes reviewing harder.

Git always complains when I do that, there's probably more to its usage that I need to figure out

@1Dragoon do you still want to finish this? Would be nice; I think we're going to push out another rc soon and hopefully release 0.17.0 soon after.

I've been out on vacation, I'll get back to it some time next week
On Wed, May 11, 2022, 2:46 AM Dirkjan Ochtman @.***>
wrote:

@1Dragoon https://github.com/1Dragoon do you still want to finish this?
Would be nice; I think we're going to push out another rc soon and
hopefully release 0.17.0 soon after.
—
Reply to this email directly, view it on GitHub
https://github.com/console-rs/indicatif/pull/420#issuecomment-1123714811,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABCBKZVI5NUCBO4M3KVQSBTVJOT2BANCNFSM5RTZ2HCA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Great, thanks!

One thing I'm having difficulty making time for is that making ProgressTracker a single type rather than having the variant type is that I need to refactor all of the example code, but other than that what I have locally works

I'm a bit stuck. I put a TODO comment in style.rs showing where the problem is. Basically in order to fall within your requirement of having clone and having both stateful and stateless trackers as a single trait.
It is nice being able to toss either a simple closure or a more complex tracker type as the same parameter, but I'm not certain how to pull that off given the need for clone to be implemented. Little help anybody?

Sorry, did you mean on line 727? I didn't see an actual TODO anywhere. I think it's fine to ditch the closure option for now and require an instantce of a ProgressTracker-implementing type.

Oh sorry I accidentally committed before adding the todo comment. It's line 25 of style.rs. What I put there is basically just a filler. I just pushed the corrected one with the comment added.

Hmm, I think an Fn implementation must be stateless anyway, so we can just reuse the Self value? Otherwise I would just keep out the impl<F: Fn(..) -> ..> for now, let's tackle one thing at a time.

Hmm, I think an Fn implementation must be stateless anyway, so we can just reuse the Self value? Otherwise I would just keep out the impl<F: Fn(..) -> ..> for now, let's tackle one thing at a time.

I'm not sure what you mean. Trait definitions can't return Self, or am I misunderstanding?
Anyways I just pushed a build that just relies on a single trait definition

Got it!
Can you review all of the code again? I can't recall what else I needed to do.

Merged in the latest upstream commits and resolved the conflict.
One thing though -- I don't really like the default() method name because it's a bit misleading, and it may conflict if the user had a struct that they actually wanted to implement the real default trait for. Not sure what else to call it though, maybe respawn()?

@djc Have you had a chance to review my comments yet?

Uh, I think I already left a round of comments here? I was waiting for you.

Hmm is there some github functionality I'm missing? I just noticed that my comments only appear if I'm logged in:

Oh I figured it out, have to go to the file changes page and then click the review button. They really ought to add some kind of UI hint that you need to do that before anybody can even see your comments, otherwise there's nothing at all to indicate that you have to do so.

BTW I'd still like to be able to tell the built in estimator to noop if we're not using it.

@djc any further concerns with this?

@1Dragoon want me to take a crack at fixing the conflicts? (since they were caused by my most recent commits). I think I can make the changes right in this PR.

Sure, sorry I haven't gotten to this yet because I've been sick

I'm feeling a bit better this afternoon so I went ahead and squashed my repo and started to resolve the merge confilcts but this one has me a bit confused:
                    if let Some(formatter) = self.format_map.get(key.as_str()) {
                        formatter.push_str(&formatter(state).replace('\t', &" ".repeat(self.tab_width)));
                    }

I think this might need to be moved to the default impl, still looking at it

I'm not sure that I like what I had to do with line 246, but it works though

I'm not sure that I like what I had to do with line 246, but it works though
@djc please review when you can

It looks OK to me.

@djc See the latest commit. I think that's what you're looking for?
Also should probably mark this to resolve #391 and #394, not sure how that works though.

Great work, thanks for sticking with it!

Sure, no problem.
Also not a huge deal, but will it ever show me as a contributor here? Reason I ask is because I am thinking that at some point I might show potential future employers what kind of work I've done, and I'm not sure how I'll be able to do that once I've deleted my forked copy (for cleanliness sake.)

Yes, you should show up as a contributor for this project (or you can send the URL to this PR).


[SPARK-34055][SQL][TESTS][FOLLOWUP] Check partition adding to cached Hive table

What changes were proposed in this pull request?
Replace USING parquet by $defaultUsing which is USING parquet for v1 In-Memory catalog and USING hive for v1 Hive external catalog.
Why are the changes needed?
The PR https://github.com/apache/spark/pull/31101 added UT test but it checks only v1 In-Memory catalog. This PR runs this test for Hive external catalog as well to improve test coverage.
Does this PR introduce any user-facing change?
No
How was this patch tested?
By running the affected test suites:
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableAddPartitionSuite"

Test build #133899 has started for PR 31117 at commit 84a851a.

Test build #133899 has started for PR 31117 at commit 84a851a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38488/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38488/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38488/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38488/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38488/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38488/

Test build #133899 has finished for PR 31117 at commit 84a851a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133899 has finished for PR 31117 at commit 84a851a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133899/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133899/

@HyukjinKwon @cloud-fan Please, take a look at this.

@HyukjinKwon @cloud-fan Please, take a look at this.

thanks, merging to master!

thanks, merging to master!


Error updating eth1 chain cache

This is the general error displayed when running the project
JSON RPC error: query returned more than 10000 results
On git cloning the lodestar project and installing the necessary dependencies. Running the project produced the following error,
Steps to reproduce
Download the project and install the necessary dependencies.
run the project

Jun-10 17:26:13.970 [ETH1]            error: Error updating eth1 chain cache message=JSON RPC error: query returned more than 10000 results, method=eth_getLogs, params=[{"fromBlock":"0x391f63","toBlock":"0x39234a","address":"0x8c5fecdc472e27bc447696f431e425d02dd46a8c","topics":["0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"]}]
Error: JSON RPC error: query returned more than 10000 results, method=eth_getLogs, params=[{"fromBlock":"0x391f63","toBlock":"0x39234a","address":"0x8c5fecdc472e27bc447696f431e425d02dd46a8c","topics":["0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"]}]    at parseRpcResponse (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/jsonRpcHttpClient.ts:60:9)
    at JsonRpcHttpClient.fetch (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/jsonRpcHttpClient.ts:39:12)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Eth1Provider.getLogs (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/eth1JsonRpcClient.ts:104:21)
    at async Promise.all (index 0)
    at retry (/home/cayman/Code/lodestar/packages/lodestar/src/util/retry.ts:27:14)
    at Eth1Provider.getDepositEvents (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/eth1Provider.ts:27:24)
    at Eth1ForBlockProduction.updateDepositCache (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/eth1ForBlockProduction.ts:148:27)
    at Eth1ForBlockProduction.update (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/eth1ForBlockProduction.ts:134:31)
    at Eth1ForBlockProduction.runAutoUpdate (/home/cayman/Code/lodestar/packages/lodestar/src/eth1/eth1ForBlockProduction.ts:116:28)

Here is also the original screenshot to support @wemeetagain findings.

This seems to have been fixed in the latest release.

I'm still getting this with v0.28.0 on Pyrmont testnet.
Aug-12 11:46:41.280 []                 info: Lodestar version=0.28.0 master 2c010b25, network=pyrmont
Aug-12 11:46:41.396 [DB]               info: Connected to LevelDB database name=/srv/chain/lodestar/chain-db
Aug-12 11:46:45.294 []                 info: Initializing beacon state from anchor state slot=10240, epoch=320, stateRoot=0x2d5fc63377925d1f1a2378dc5277ebef4533cd8ae0b6a2e144fb008de9b59c9b
Aug-12 11:46:54.271 [METRICS]          info: Starting metrics HTTP server port=8008
(node:1301099) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
Aug-12 11:46:56.916 [NETWORK]          info: PeerId 16Uiu2HAmGWtTaoLRALWFUr5dFApu3mCiKUGdcGobF7VZseHZBtqA, Multiaddrs /ip4/<IP_ADDRESS>/tcp/9000,/ip4/<IP_ADDRESS>/tcp/9000
Aug-12 11:46:56.917 []                 warn: Low peer count peers=0
Aug-12 11:46:56.918 []                 info: Searching for peers - peers: 0 - finalized: 318 0xf918…f52c - head: 10239 0x198f…7f39 - clockSlot: 1921734
Aug-12 11:47:01.001 []                 info: Searching for peers - peers: 0 - finalized: 318 0xf918…f52c - head: 10239 0x198f…7f39 - clockSlot: 1921734
Aug-12 11:47:05.241 [ETH1]            error: Error updating eth1 chain cache message=JSON RPC error: query returned more than 10000 results, method=eth_getLogs, params=[{"fromBlock":"0x39234b","toBlock":"0x392444","address":"0x8c5fecdc472e27bc447696f431e425d02dd46a8c","topics":["0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"]}]
Error: JSON RPC error: query returned more than 10000 results, method=eth_getLogs, params=[{"fromBlock":"0x39234b","toBlock":"0x392444","address":"0x8c5fecdc472e27bc447696f431e425d02dd46a8c","topics":["0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"]}]
    at parseRpcResponse (/home/user/lodestar/packages/lodestar/src/eth1/provider/jsonRpcHttpClient.ts:139:9)
    at JsonRpcHttpClient.fetch (/home/user/lodestar/packages/lodestar/src/eth1/provider/jsonRpcHttpClient.ts:53:12)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at Eth1Provider.getLogs (/home/user/lodestar/packages/lodestar/src/eth1/provider/eth1Provider.ts:158:21)
    at async Promise.all (index 0)
    at retry (/home/user/lodestar/packages/lodestar/src/util/retry.ts:33:14)
    at Eth1Provider.getDepositEvents (/home/user/lodestar/packages/lodestar/src/eth1/provider/eth1Provider.ts:67:24)
    at Eth1ForBlockProduction.updateDepositCache (/home/user/lodestar/packages/lodestar/src/eth1/eth1ForBlockProduction.ts:164:27)
    at Eth1ForBlockProduction.update (/home/user/lodestar/packages/lodestar/src/eth1/eth1ForBlockProduction.ts:150:31)
Aug-12 11:47:13.001 []                 info: Searching for peers - peers: 0 - finalized: 318 0xf918…f52c - head: 10239 0x198f…7f39 - clockSlot: 1921735
Aug-12 11:47:17.169 [ETH1]            error: Error updating eth1 chain cache message=JSON RPC error: query returned more than 10000 results, method=eth_getLogs, params=[{"fromBlock":"0x39234b","toBlock":"0x392444","address":"0x8c5fecdc472e27bc447696f431e425d02dd46a8c","topics":["0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"]}]
Error: JSON RPC error: query returned more than 10000 results, method=eth_getLogs, params=[{"fromBlock":"0x39234b","toBlock":"0x392444","address":"0x8c5fecdc472e27bc447696f431e425d02dd46a8c","topics":["0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"]}]
    at parseRpcResponse (/home/user/lodestar/packages/lodestar/src/eth1/provider/jsonRpcHttpClient.ts:139:9)
    at JsonRpcHttpClient.fetch (/home/user/lodestar/packages/lodestar/src/eth1/provider/jsonRpcHttpClient.ts:53:12)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at Eth1Provider.getLogs (/home/user/lodestar/packages/lodestar/src/eth1/provider/eth1Provider.ts:158:21)
    at async Promise.all (index 0)
    at retry (/home/user/lodestar/packages/lodestar/src/util/retry.ts:33:14)
    at Eth1Provider.getDepositEvents (/home/user/lodestar/packages/lodestar/src/eth1/provider/eth1Provider.ts:67:24)
    at Eth1ForBlockProduction.updateDepositCache (/home/user/lodestar/packages/lodestar/src/eth1/eth1ForBlockProduction.ts:164:27)


Modify rendering to utilize HelperResults

Our HTML helpers will always return HtmlString with the variety that take a HelperResults will allow us to write directly to streams and never write to strings. and thus avoid LOH allocations (and consume less memory)
Conversation discussing the topic: https://github.com/aspnet/WebFx/pull/155#discussion_r11055448

https://github.com/aspnet/Mvc/commit/b6a109e2a3f6fac94a3cbec0d16ea1adf526e1d8


check for existence of err object before calling stuff on it

Fixes https://github.com/TestArmada/magellan/issues/84

Awesome, thanks.


How do you keep flags in sync across environments?

In other words, how do you define flags that should appear in your test and prod environments without needing to manually add them to both.
I can think of at least two ways to do this, but neither seem ideal. I'm curious what solutions others have used to manage flags at deploy time in different environments.
For context, the way we use flags is to hide frontend and backend code for trunk based development as well as traditional feature flipping for certain users. In practice, each code deploy w/ a new feature has a flag. Since code is coupled to flags, I hope there is an optimal way to ensure flags exist across environments as code moves through them.
What is the optimal way you've seen this managed in a Django project?

Write a data migration that creates the flag.


Codesandbox and bundler improvements for examples

Fixes #13865.
And use vite instead of parcel.

@ahocevar  hi sorry for this example  https://deploy-preview-13879--ol-site.netlify.app/examples/d3.html  is still not work (and with some errors this time  )?
Error
Could not fetch dependencies, please try again in a couple seconds:

@jipexu You need to set the OpenLayers version to one that is actually published. But note that this pull request does not address the problems with the d3 example, as the issue it solves is not about the s3 example.

@ahocevar ah ok thanks


riscv/m2gl025: timer tests broken

Describe the bug
It seems that 669730f030ae36a428b696e2210f6d521747ad6c from #16782 has introduced some regressions in timer-related tests for the m2gl025 board.
In order to reproduce locally with sanitycheck, the renode emulator must be installed on your system. Then just $ sanitycheck -p m2gl025_miv and behold the failures the below.  Backing out to f288d1e4a73c75938c702736bab73b19b96a4a97 will eliminate almost all of these (except kernel/timer/timer_api, I guess that's another bug report).
m2gl025_miv               samples/portability/cmsis_rtos_v2/timer_synchronization/sample.portability.cmsis_rtos_v2.timer_synchronization FAILED: timeout
        see: sanity-out/m2gl025_miv/samples/portability/cmsis_rtos_v2/timer_synchronization/sample.portability.cmsis_rtos_v2.timer_synchronization/handler.log

total complete:   31/ 126  24%  failed:    1

m2gl025_miv               tests/cmsis_rtos_v2/portability.cmsis_rtos_v2      FAILED: timeout
        see: sanity-out/m2gl025_miv/tests/cmsis_rtos_v2/portability.cmsis_rtos_v2/handler.log

total complete:   46/ 126  36%  failed:    2

m2gl025_miv               tests/kernel/context/kernel.common                 FAILED: failed
        see: sanity-out/m2gl025_miv/tests/kernel/context/kernel.common/handler.log

total complete:   50/ 126  39%  failed:    3

m2gl025_miv               tests/kernel/fatal/kernel.common.stack_sentinel    FAILED: timeout
        see: sanity-out/m2gl025_miv/tests/kernel/fatal/kernel.common.stack_sentinel/handler.log

total complete:   52/ 126  41%  failed:    4

m2gl025_miv               tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout FAILED: timeout
        see: sanity-out/m2gl025_miv/tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout/handler.log

total complete:   53/ 126  42%  failed:    5

m2gl025_miv               tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout.poll FAILED: timeout
        see: sanity-out/m2gl025_miv/tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout.poll/handler.log

total complete:   90/ 126  71%  failed:    6

m2gl025_miv               tests/kernel/sleep/kernel.common.timing            FAILED: timeout
        see: sanity-out/m2gl025_miv/tests/kernel/sleep/kernel.common.timing/handler.log

total complete:   95/ 126  75%  failed:    7

m2gl025_miv               tests/kernel/timer/timer_api/kernel.timer          FAILED: failed
        see: sanity-out/m2gl025_miv/tests/kernel/timer/timer_api/kernel.timer/handler.log

total complete:   97/ 126  76%  failed:    8

m2gl025_miv               tests/kernel/workq/work_queue/kernel.workqueue     FAILED: failed
        see: sanity-out/m2gl025_miv/tests/kernel/workq/work_queue/kernel.workqueue/handler.log

total complete:   99/ 126  78%  failed:    9

m2gl025_miv               tests/kernel/workq/work_queue/kernel.workqueue.poll FAILED: failed
        see: sanity-out/m2gl025_miv/tests/kernel/workq/work_queue/kernel.workqueue.poll/handler.log

total complete:  126/ 126  100%  failed:   10
116 of 126 tests passed with 0 warnings in 706 seconds

To reproduce, need to install 'renode'. I did this on my Ubuntu 18.04 system:

Add mono repository at https://www.mono-project.com/download/stable/#download-lin
Then install the .deb at https://renode.io/#downloads

Once renode is installed and in PATH, sanitycheck automatically starts using it.

Seems to have been broken after this commit https://github.com/zephyrproject-rtos/zephyr/commit/669730f030ae36a428b696e2210f6d521747ad6c

@kgugala @andyross pinging you guys as owners of RISCV and kernel timing, respectively

Looks like the root cause is a SoC bug.  From soc/riscv32/riscv-privilege/miv/Kconfig.defconfig.series:
config SYS_CLOCK_HW_CYCLES_PER_SEC
        int
        default 660000

Needless to say a 660 kHz processor is going to have a really rough time reliably delivering a default tick rate precision of 10 kHz.
There is an (AFAICT otherwise unused) "clock-frequency" DTS setting for this board of 66 MHz, which looks much more reasonable.
We could surely also fix this by setting the default tick rate back to 100 Hz for this SoC, but let's try fixing the base clock first.  The same driver works just fine with qemu_riscv32 at an emulated 10 MHz.

Oh, and I should also mention: I think renode is a cycle simulator and doesn't use host realtime like qemu does.  But if that's wrong not then 10 kHz will never work and we need to set this to 100, as is done on qemu targets.  I still no basically nothing about renode.

I think renode is a cycle simulator and doesn't use host realtime like qemu does.

If that's the case, maybe we should use it instead of qemu and save ourselves future grief.
.... only half-joking here.

@nategraff-sifive who's the right person to work on this? This is blocking a lot of people's PRs since this is broken in master

Sorry, I didn't realize this was actually breaking CI.  FWIW: if the earlier rate was working at 660 kHz, then it's probably enough to add something like this (completely untested) as a temporary fix:
--- a/boards/riscv32/m2gl025_miv/m2gl025_miv_defconfig
+++ b/boards/riscv32/m2gl025_miv/m2gl025_miv_defconfig
@@ -13,3 +13,6 @@ CONFIG_PLIC=y
 CONFIG_RISCV_MACHINE_TIMER=y
 CONFIG_GPIO=n
 CONFIG_XIP=y
+
+# Workaround for incorect SYS_CLOCK_HW_CYCLES_PER_SEC
+SYS_CLOCK_TICKS_PER_SEC=100

@nategraff-sifive who's the right person to work on this? This is blocking a lot of people's PRs since this is broken in master

@andrewboie My first thought would be @mgielda and the folks at Antmicro.


Fix sample app to send custom event

When someone clones the repo and sends the IoT traffic, it will send only network and error data and the custom event is inside a conditional loop and does not execute by default.

Merged.


Example in installation docs doesn't work

This is part of the JOSS review openjournals/joss-reviews#5618.
The example in the installation documentation doesn't work. After pip install matbench-genmetrics into a new Python 3.11 virtual environment, I have the following:
In [1]: from mp_time_split.utils.gen import DummyGenerator
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 1
----> 1 from mp_time_split.utils.gen import DummyGenerator

ModuleNotFoundError: No module named 'mp_time_split'

which is presumably a missing module required for the example.
In any case, the next line also fails to work.
In [2]: from matbench_genmetrics.core import MPTSMetrics10, MPTSMetrics100, MPTSMetrics1000, MPTSMetrics10000
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[2], line 1
----> 1 from matbench_genmetrics.core import MPTSMetrics10, MPTSMetrics100, MPTSMetrics1000, MPTSMetrics10000

ImportError: cannot import name 'MPTSMetrics10' from 'matbench_genmetrics.core' (/home/jrh/programs/JOSS/matbench-genmetrics/lib/python3.11/site-packages/matbench_genmetrics/core/__init__.py)

Looks to be easily fixed by replacing the text with contents of scripts/core/matbench_genmetrics_basic.py

Thanks for the fix.


feature: upgrade to libopenapi with orderedmap support

TODO:

[x] Merge https://github.com/pb33f/libopenapi/pull/205
[x] Merge https://github.com/pb33f/libopenapi-validator/pull/38
[x] Replace go.mod replace statements

🚢 🇮🇹


If label is not str worker fails without traceback

If a label with a type different from str is added anywhere in the worker task, the task will fail.
@broker.task
async def generation_task(params: RequestParams, context: Annotated[Context, TaskiqDepends()]):
    result = do_some_work(params)
    context.message.labels['non_str_field'] = 42
    return result

I suspect the issue lies in the construction of the TaskiqResult object at /taskiq/receiver/receiver.py:294.
While it's acceptable for the task to fail on unknown types, the problem is that it doesn't display anything in the log. This lack of information made it necessary for me to spend extra time figuring out what went wrong.
Perhaps, to make this workaround clearer, we could consider adding some logging?

Totally agree on that one.
That's weird that receiver fails after non-string label. Also we have this PR waiting to be merged that might solve your issue as well.
https://github.com/taskiq-python/taskiq/pull/243


fix case where nom-shell is in not in PATH

https://github.com/Mic92/nixpkgs-review/pull/389 was merged


Browse documents: Filter by indicator

Story
I can easily find all documents used to track the performance of a particular indicator.
Acceptance criteria

[x] 1. Next to the Filter by program menu is a Filter by indicator menu

[x] 1.1. The default text is "Filter by indicator".

[x] 2. The Filter by indicator menu is inactive when there is no program selected
[x] 3. If a program is selected, the Filter by indicator menu is active and only shows indicators associated with the selected program.
[x] 4. I can type an indicator number or portions of the indicator name and get type-ahead results.
[x] 5. When I select an indicator from the menu, I see all documents associated with the indicator.
[x] 6. If I filter on an indicator and then change programs, the following happens:

[x] 6.1. The list of documents reloads to show all documents associated with the program.
[x] 6.2. The Filter by indicator menu clears.
[x] 6.3. When I reopen the Filter by indicator menu, it shows indicators associated with the program I just switched to.

Testing:

[ ] IF results provide enough to paginate, ensure Pagination functions properly.
[ ] Type-ahead function provides results as text is input into search field
[ ] Results properly reflect information being searched
[ ] Multiple searches need to be completed to confirm functional implementation based on acceptance criteria.

Estimated Test Time w/ Acceptance:
2  hours
Housekeeping

[ ] Unit test/s written wherever applicable
[x] i18n tags included wherever applicable
[ ] Functionality is accessible to people using screen readers

Deployment Verification

[x] Confirmed “Ready for Deploy” status
[x] Media Verifying:


Update npm command in readme

I just tried npm install --save humanize and got a different library :(
This PR updates the README to the correct command.

Thx for the update!


demonstrate the utility of Apropos

The current examples produce mainly self-tests, and non-self-tests are easier to write with QuickCheck than with Apropos.
(Self-tests are testing the so-called Apropos morphisms which is not user code; it is written to help the testing of user code.)
I need one example which demonstrates that Apropos is better than QuickCheck.

Define data types A and B, a function f : A -> B and a list of QuickCheck property tests [t1, t2, ..., tN] for f.
Make sure that [t1, t2, ..., tN] is nearly optimal, i.e. you cannot write much shorter equivalent idiomatic QuickCheck code. (I volunteer to help in this step.)
Write Apropos code which is equivalent to [t1, t2, ..., tN].
Show that the Apropos code is faster/shorter than the definition of [t1, t2, ..., tN].

I understand that Apropos is still a proof-of-concept implementation of an idea, but being so it needs at least one convincing example.

Aren't self-tests a very big reason?

Aren't self-tests a very big reason?

Self tests are testing generators.
Probably it is easier to understand this with a QuickCheck example.
In QuickCheck there are predefined generators (= random distributions) for common data structures.
If you add a new generator, you may want to test it because good generators are essential for testing user code.
But testing generators alone is not enough; the ultimate goal is to test the user code.

@divipp The self tests for a HasParameterisedGenerator are representative of the property tests that would run against your user code. One of the main selling points of apropos is that you can specify your behavior before writing any user code. This is test driven development.
Building generators is what apropos does. It allows you to build generators that are parameterised by a logical model apropos a type. From this you can enumerate the generators for each property tests.
The quantity of tests generated is dependent on the complexity of the logical model. Due to the nature of boolean satisfiability it is trivial to write a model that generates an enormous amount of tests - to see this you can extend the IntPair model to IntTriple, IntQuadruple, etc... and watch as the number of tests grows exponentially.
RE: improving the examples with a comparison to quickcheck. It is worth noting that we are interested in testing a specific kind of code - smart contracts. These are functions with express constraints on the contract inputs/outputs and context - i.e. A -> Bool. apropos is perfectly suited to testing these functions since your logical model can partition the space of A into those that should validate and those that should fail. You can specify this partition in terms of a logical formula in the language of your properties.
Other kinds of functions may be less well suited to testing with apropos. - for example [f :: A -> B would require a model of the relation A X B, associativity may require a model of an AST, etc...]. Apropos can test these things but it may be the wrong tool for some of them - for example there is no point in testing associativity of + with apropos since a single property test is all you need.

@delicious-lemon

Other kinds of functions may be less well suited to testing with apropos.

It is fine to chose Bool for B.
If so, the challenge looks like this:

Define data type A, a function f : A -> Bool and a list of QuickCheck property tests [t1, t2, ..., tN] for f.
...

The self tests for a HasParameterisedGenerator are representative of the property tests that would run against your user code.

This fits the challenge too.
I interpret this as [t1, t2, ..., tN] is chosen to be the self-tests.
Maybe it works and you win the challenge.
It may not work because:

either the self-tests has no connection with f (so they doesn't test f, they test something else)
or the self-tests are trivial in the sense that they can be replaced by much shorter equivalent QuickCheck tests

@divipp I think this issue can be moved to the apropos-tx repo. We are planning on putting some examples for real world contracts (perhaps an auction) in there as well as library models for common things. The real world contract tests should be convincing.

@delicious-lemon I think it would be more convincing to have a synthetic self-contained example. Once you have a good candidate example in apropos-tx I can help to make it a bit more abstract so that it can be placed in this repo.

I'd like to be convinced as well.
My main concern is that it looks like the tests are no simpler than the code, which means tests aren't necessarily going to work at all.

My main concern is that it looks like the tests are no simpler than the code, which means tests aren't necessarily going to work at all.

I don't understand what you mean by this. In my experience it is often the case that tests are more complex than the code.
The complexity here is required to explain to a solver how to build your test suite - there is an inflection point in terms of complexity where you will start to profit from the power of the solver (below that it might seem that writing tests by hand is simpler).

@divipp Does #61 help?

@AriFordsham It doesn't. This doesn't show that apropos works better than manually writting validity-based tests.

@NorfairKing I acknowledge apropos is under-documented as well as under-exampled and on that basis this issue is correct.
I just wanted to open a dialogue by answering your question here, where it can be recorded.
As I understand it, validity is essentially a replacement for the Arbitrary typeclass that is lawful and supports invariants that are not intrinsic to a datatype's layout. There is a 1-to-1 relationship between types with invariants and their generators, hence the typeclass.
This, as an extension of property testing in general, works by testing the assertion:

Laws should hold for all (valid) examples of input data

Test data are generated 'randomly'. From what distribution is this drawn?
This is another question QuickCheck leaves open, is not easy to answer, and as far as I can see, validity does not attempt to answer either.
The simple answer (and also the straightforward definition of 'random') is 'a uniform distribution.' This is not necessarily helpful, because some values are more 'different' than others. It's certainly not what QuickCheck does.
QuickCheck establishes a loose convention by which generators generate values based on some undefined property called 'size' in order that both 'small' and 'large' values are tested. There is some vague notion that if value causes a test to fail, this failure may be rooted in a 'smaller' precursor to itself. 'Shrinking' is provided to generate this precursor.
The problem here is that values often have properties that determine their propensity to trigger bugs. Examples include 'big'- and 'small'-ness, as implied by QuickCheck, but these are not always relevant, nor are they the only ones. For example, for numbers, zero, positive and negative values are important, and for finite number types bounds matter.
In certain circumstances, for example when inter-relations between substructures become interesting, this problem scales faster than ad-hoc solutions can.
The solution Apropos explores is to partition a value into its interesting properties or subtypes, and to test properties (laws) against all (or a selection) of these.
Like this, for each 'law', if I can subdivide the input datatype n interesting ways, I get up to n^2 property tests for free for each property written.
The design for Apropos is certainly still evolving. I would welcome any collaboration you would be interested in.


Route53: Record overwrites

Changes introduced in #5725 were not tested properly because of wrong Terraform test-case names.
This PR corrects the test case names.
Source: https://github.com/hashicorp/terraform-provider-aws/blob/163210e073650650adba645b636128fbc19244f4/internal/service/route53/record_test.go#L1322
This needs to be accompanied with the fix. @bblommers would you be able to take this over?

No worries @viren-nadkarni - I'll have a look. Thanks for letting me know.

Codecov Report

Merging #5747 (fa0357d) into master (d551a4d) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #5747   +/-   ##
=======================================
  Coverage   96.19%   96.19%
=======================================
  Files         751      751
  Lines       73613    73615    +2
=======================================
+ Hits        70814    70816    +2
  Misses       2799     2799

Flag
Coverage Δ

servertests
36.93% <42.85%> (-0.01%)
:arrow_down:

unittests
96.13% <100.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

moto/route53/exceptions.py
96.73% <100.00%> (ø)

moto/route53/models.py
96.42% <100.00%> (+0.02%)
:arrow_up:

moto/s3/models.py
96.88% <0.00%> (-0.01%)
:arrow_down:

moto/core/responses.py
95.98% <0.00%> (ø)

moto/cognitoidp/models.py
95.52% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


install packages needed for QE tests and cleanup after

fixed typo
added root privileges for repo removal

removing the SLES-SDK repo's from deployer after installing packages needed for QE so there is no chance those repo's can interfere with future updates


_pywrap_file_io.BufferedInputStream raises UnicodeDecodeError

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock
example script provided in TensorFlow): No
OS Platform and Distribution (e.g.,
Linux Ubuntu 16.04): Windows10 Home Edition
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if
the issue happens on mobile device: No
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.2.0rc1
Python version: 3.7.7
Bazel version (if compiling from source): None
GCC/Compiler version (if compiling from source):  None
CUDA/cuDNN version: 10.1/7.6.5
GPU model and memory: GeForce GTX 980Ti 38845MB

Describe the current behavior
In jupyter notebook, I have tried TF Models tutorial(research/object_detection/object_detection_tutorial.ipynb) and got UnicodeDecodeError at following section.
List of the strings that is used to add correct label for each box.
PATH_TO_LABELS = 'models/research/object_detection/data/mscoco_label_map.pbtxt'
category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)
error log:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8e in position 90: invalid start byte
Describe the expected behavior
UnicodeDecodeError don't happen.
Standalone code to reproduce the issue
from tensorflow.python import _pywrap_file_io
_read_buf=_pywrap_file_io.BufferedInputStream('test', 1024*512)
Other info / logs
My system language is Japanese and this may affect encode behavior.

@Syaryoh
please refer to this issue
link1 link2  with similar error

Thanks to reply.
I'm sorry to my misunderstanding.
This problem occurs when the target file do not exist.
In my reproduce code, I have made empty file 'test' and the error has disappeared.
It would be appreciated if you could add error message for us to recognize the file not exist.

@Syaryoh
Can you please share simple sample code for us to replicate the issue faced by you,

In this python code:
from tensorflow.python import _pywrap_file_io
_read_buf=_pywrap_file_io.BufferedInputStream('test', 1024*512)
If a file named 'test' do not exist, it raises UnicodeDecodeError.
or this reproduces too:
Open the tutorial of TF Models(research/object_detection/object_detection_tutorial.ipynb) with Jupyter Notebook
and change value of PATH_TO_LABELS to 'test'(not exist file). Then it reproduce.

@Syaryoh
can you please share that test file, so we can replicate the issue

@Syaryoh
can you please update as per above comment

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

In this python code: from tensorflow.python import _pywrap_file_io _read_buf=_pywrap_file_io.BufferedInputStream('test', 1024*512)
If a file named 'test' do not exist, it raises UnicodeDecodeError.
or this reproduces too: Open the tutorial of TF Models(research/object_detection/object_detection_tutorial.ipynb) with Jupyter Notebook and change value of PATH_TO_LABELS to 'test'(not exist file). Then it reproduce.

thank you for the solution. I was trying to reach a directory full of photos but instead of photos directory I gave main directory path. e.g ; write ...--image_dir=D:\\AnacondaProjects\\tez\\Tensorflow\workspace\\training_demo\\images\\test  instead of ...--image_dir=D:\\AnacondaProjects\\tez\\Tensorflow\workspace\\training_demo\\images


CouchDB 3 Migration: Remove all_or_nothing flag

Status
not-tested
Description
As part of the CouchDB Migration to 3.x ( #8853 ), the all_or_nothing flag has to be removed because
it's no longer present (see https://github.com/apache/couchdb/issues/1235 for a discussion).
Is it backward compatible (if not, which system it affects?)
NO
Related PRs
External dependencies / deployment changes
It depends on CouchDB 3.x

test this please

test this please

test this please

test this please

Thank you for providing this patch, Max.
This PR seems to resolve 6-8 unit tests, bringing failures down to 80:
https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-WMAgentCouch31-TestAll/lastCompletedBuild/testReport/
However, a deep investigation of the possible impacts with the removal of this all_or_nothing flag is required. I will have to follow this up in the coming weeks.

This has been superseeded by https://github.com/dmwm/WMCore/pull/11011, so I am closing this PR.
Thank you very much, Max!


little problem on get name of a .png file

line  32, if the file name is 'company.project.png', you will get incomplete information.
you may consider use rfind('.') or rindex('.') instead. (_)

which file？

@chunlins

in image2webp.py
or you can use functions in os.path module.
>>> filename = 'microsoft.collections.dll'
# splitext divides the path on extension sperator
>>> os.path.splitext(filename)
# return a tuple, first item is the name, second item is file's extension
('microsoft.collections', '.dll')

it's easy to retrieve name, just name, ext = os.path.splitext(filename)

3Q, you can fix it and new a pull request, it will be fine. @chunlins


Add Zapier Natural Language Actions API generic utility

This PR adds basic support for Zapier NLA to LangChain! Key benefits:

Plug in the 5k+ apps, 20k+ search and create actions on Zapier's platform into chains (eg. Slack, GitHub, Trello, Salesforce, HubSpot, Google Sheets, MS Teams...)
NLA handles all the complexity around underlying API auth, custom fields, long arg lists, long return vals, etc.
Works in both server-side only (api key) and user-facing (oauth) setups
NLA is free for devs

Here's a basic example:
1. Use NLA to find an email in Gmail
2. Use `LLMChain` to generate a draft reply to (1)
3. Use NLA to send the draft reply (2) to someone in Slack via direct message

Best way to check it out is probably the docs/modules/utils/examples/zapier.ipynb this PR includes.

Going to merge this into a branch in the main repo so I can work off of it


Make assert_object_equals more accurate and give better error messages

As an alternative to https://github.com/w3c/testharness.js/pull/143, this pull request proposes to fix assert_object_equals.
Consider the following tests:
test(function() {
    assert_object_equals(false, true);
}, "1");
test(function() {
    assert_object_equals(1, true);
}, "2");
test(function() {
    assert_object_equals(false, {});
}, "3");
test(function() {
    assert_object_equals({}, false);
}, "4");
test(function() {
    assert_object_equals("abc", ["a", "b", "c"]);
}, "5");
test(function() {
    assert_object_equals(["a", "b", "c"], "abc");
}, "6");
test(function() {
    assert_object_equals({a:{}}, {a:false});
}, "7");
test(function() {
    assert_object_equals({b:{}}, {b:null});
}, "8");
test(function() {
    assert_object_equals({c:null}, {c:{}});
}, "9");
test(function() {
    assert_object_equals({d:false}, {d:{}});
}, "10");
test(function() {
    assert_object_equals({x:{y:{z:false}}}, {x:{y:{z:true}}});
}, "11");

Previously, the first 8 would incorrectly pass, and the last 3 would
give confusing error messages:
FAIL 9 assert_object_equals: property "c" expected object "[object Object]" got object "[object Object]"
FAIL 10 assert_object_equals: property "d" expected object "[object Object]" got object "[object Object]"
FAIL 11 assert_object_equals: property "z" expected object "[object Object]" got object "[object Object]"

After this patch, all the cases fail correctly and have clear error messages:
FAIL 1 assert_object_equals: expected true but got false
FAIL 2 assert_object_equals: expected (boolean) true but got (number) 1
FAIL 3 assert_object_equals: expected (object) object "[object Object]" but got (boolean) false
FAIL 4 assert_object_equals: expected (boolean) false but got (object) object "[object Object]"
FAIL 5 assert_object_equals: expected (object) ["a", "b", "c"] but got (string) "abc"
FAIL 6 assert_object_equals: expected (string) "abc" but got (object) ["a", "b", "c"]
FAIL 7 assert_object_equals: property ["a"]: expected (boolean) false but got (object) object "[object Object]"
FAIL 8 assert_object_equals: property ["b"]: expected null but got object "[object Object]"
FAIL 9 assert_object_equals: property ["c"]: expected object "[object Object]" but got null
FAIL 10 assert_object_equals: property ["d"]: expected (object) object "[object Object]" but got (boolean) false
FAIL 11 assert_object_equals: property ["x"]["y"]["z"]: expected true but got false

Thank you for the patch, but I really really just want this to die. It's too complex to be good for testing.

Thank you for the patch, but I really really just want this to die. It's too complex to be good for testing.


Cleanup.Spinner

Simplifies attribute naming conventions (removes data- prefix) as mentioned in issue #202

@maxatdetroit, I just wanted to clarify that #202 was the only issue that pertained to this component

Yes, good clarification. That was correct when you asked, but I did notice while reviewing this PR that the storybook stories are setting missing controls to undefined in the code examples, so I added this request to #201:

Also, unless the attribute is a boolean attribute, the component should treat attribute="undefined" the same as omitting the attribute entirely. In Storybook stories, we should prefer omitting the attribute entirely if the storybook control is not set.

Would you please update the Storybook story for the spinner to omit attributes if they're not set in the story (e.g. size and display-type should be omitted from the component in the earlier screenshot)? Then, let's make sure the spinner still functions as expected when those attributes are omitted.

Also, all PRs for 201, 202, and 205 should be based on the 2.0.0-alpha1 branch. You'll need to:
git rebase 2.0.0-alpha1
This is a case where a rebase is required instead of using merge. After you've done that and pushed the changes to this PR. We can update the base branch on the PR as well (using the edit button near the PR title):


ADF linked service for Magento - direct db/SSH support?

Good day!
We're happy to see the ADF linked service for Magento (preview). I suspect it is using Magento's REST API - pls confirm.
In general we've found limitations and unreliable operation of the Magento API. Our solution currently is w/SSIS and the Devart connector making a direct connection to the MySQL/MariaDB using an SSH tunnel. Are there any plans to support direct Magento database connection w/SSH key pair tunnel auth?
There are a few 3rd-party "no-code" ETL cloud services that are doing this now for a Magento->Snowflake data flow which is our target.
While ADF Linked Services supports MariaDB, the SSH "tunnel" is apparently NOT natively supported (like Devart's SSIS mySQL connector), although SFTP is supported (which naturally supports key-based SSH). While other developers have used a Bitvise SSH client running on Azure VM which port forwards to MGT MariaDB, this is somewhat cumbersome at best. While ADF Magento Link Service store is in PREVIEW, I suspect it's calling the infamous Magento API, NOT a direct db connection.
We need an ADF DIRECT connection to our Magento MariaDB (on AWS) database via an SSH tunnel, secured by public/private keys stored in Azure Key Vault - much like the ADF LS for SFTP.
Thanks! :-)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c6ed9515-bcc1-4609-ed55-68dcffc7e88e
Version Independent ID: 78e40e7b-57cc-8871-1da1-ac651a92dce8
Content: Copy data from Magento (Preview) - Azure Data Factory & Azure Synapse
Content Source: articles/data-factory/connector-magento.md
Service: data-factory
Sub-service: data-movement
GitHub Login: @jianleishen
Microsoft Alias: jianleishen

@RayGabePMI  Thanks for the question! We are investigating and will update you shortly.

Hi @RayGabePMI Thank you for sharing this feedback. Its product related feedback and not on document related feedback. I encourage you to please submit feedback item in below link and up-vote same. Product team closely monitor feedbacks there and consider them for future implementations.
https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c