[SPARK-34589][CORE] Make BypassMergeSortShuffleWriter close partitionWriters in case of success

What changes were proposed in this pull request?
This PR aims to close partitionWriters when BypassMergeSortShuffleWriter.stop with success.
Why are the changes needed?
In case of failure, we call writer.revertPartialWritesAndClose and delete the files.
Likewise, we had better close the resources at this level.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the existing UTs.

Hi, @mridulm and @Ngone51 .
How do you think about this?

Test build #135634 has started for PR 31704 at commit 79e6c5d.

Oops. Sorry guys. I realized that writer.write has try (DiskBlockObjectWriter writer = partitionWriters[I]). I'm closing this.
      writer.write(
        rdd.iterator(partition, context).asInstanceOf[Iterator[_ <: Product2[Any, Any]]])
      val mapStatus = writer.stop(success = true)

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40213/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40213/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40213/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135634/


Fn to Expand Control Strip stops working

After updating to macOS 10.12.6, the function stops working when Karabiner-Elements is on. Nothing happens to the control strip when Fn is pressed down. It works again after quitting the app.
I've tried to remove all the modifications but the problem remains.
Macbook Pro 2016 with touchbar 13.3"
macOS 10.12.6
Karabiner-Elements 0.91.9

Same issue (same MacBook and Karabiner)
Disabling modifications for "Apple Internal Keyboard / ..." makes it work, but it makes Karabiner useless.

+1


Fix/handle editor play mode state change

Reality Collective - Reality Toolkit Pull Request
Overview

Fixes a VERY annoying issue whereby on entering Play Mode, Unity will run "Validate" and attempt to initialise the Service Manager. Where this is normally required for normal Unity Editing, this instance is then destroyed before "Awake" is called.
This fix "finally" allows the Service Manager to "skip" initialisation if the Unity Editor enters play mode and prevents double initialization.
Does not affect normal "Play" mode as this is an editor only behaviour.
Also fixed a minor grievance where the Service Manager for whatever reason is not initialised in the Editor and causes the inspector to throw a null reference exception (rare issue)
Changes

Implemented a "EditorPlayModeStateChange" editor flag, which is set on entering play mode in the editor and reset when it leaves.
Fixed an edge case where Unity does NOT initialise the Service Framework in the editor that caused a null reference exception in the inspector.

Breaking Changes

None

Unbelievable, I think I found another edge case in the editor where it attempts to run both Validate and Awake at the same time.  Still need to test more, but on opening a scene when the editor loads.
Doesn't seem to happen if you just open a scene, just if your previous scene you had open when Unity closed, if unity actually reopens on the same scene, this issue can occur.


Need more detail in oden responses

right now we dont read the body when i get an error message

make the responses more specific for the requests aswell


Please add LICENSE to MANIFEST.in

The text of the MIT license requires that the license be shipped with the software.  Currently, python-highcharts is not including this license in the package.  This can be resolved by creating a MANIFEST.in file and adding LICENSE to it.  Let me know if you'd like me to submit a PR to fix this.

plz feel free to do it.
Hank
On Tue, Sep 13, 2016 at 8:26 AM, Eric Dill<EMAIL_ADDRESS>wrote:

The text of the MIT license requires that the license be shipped with the
software. Currently, python-highcharts is not including this license in the
package. This can be resolved by creating a MANIFEST.in file and adding
LICENSE to it. Let me know if you'd like me to submit a PR to fix this.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kyper-data/python-highcharts/issues/14, or mute the
thread
https://github.com/notifications/unsubscribe-auth/ALWsZOgK7NJ1Ckh6V84kbAA7igL-jyFZks5qpsCcgaJpZM4J7zlk
.

This has been addressed in the latest release (v0.3.1) -- LICENSE is included with the pypi package now. Thanks!


Get "Triggered" status from the Alarm

Hi.
I've been using the old integration with securitas, and had some automations based on the state change of the alarm. If i'm not work, the TRIGGERED status isn't one that appears to be in the variables. In the "old" api where this was forked, in the alarm_control_panel.py was the "STATE_ALARM_TRIGGERED" and isn't in this one.
Can the TRIGGERED stated be used? It was a good one to send alerts and turn on lights when the alarm reported it.

I also miss the TRIGERED state, I used it in case of alarm to close blinds and turn on a siren on another floor.

That will require to call the API more often and I'm not sure about it. We don't want to upset Securitas and their API.

But it isn't possible to recover that status even it now be from the moment, so it can be from any moment since the last scan. It could be good to have it when you're out even you (HA) know it with delay. I could then set an script to save out the cameras footage and be sure that isn't deleted with other recording, check status of other sensors, etc.


ConfigurationProfilePresetCodes.isKnownPreset is not working

Describe the bug
static {
registerProperties(String.class);
}
should be
static {
registerProperties(ConfigurationProfilePresetCodes.class);
}
so that the public static Strings get added to KNOWN_PRESETS
Expected Behavior
known presets should be validated correctly
What plugins are used? What other connection properties were set?
n/a
Current Behavior
isKnownPreset method returns false for any value, as KNOWN_PRESETS is empty
Reproduction Steps
ConfigurationProfilePresetCodes.isKnownPreset("D0") returns false
Possible Solution
static {
registerProperties(ConfigurationProfilePresetCodes.class);
}
Additional Information/Context
No response
The AWS Advanced JDBC Driver version used
2.3.6
JDK version used
21.0.1
Operating System and version
unix

https://github.com/aws/aws-advanced-jdbc-wrapper/blob/ed0082f31f65c7e8be560202e95bce937cd464a7/wrapper/src/main/java/software/amazon/jdbc/profile/ConfigurationProfilePresetCodes.java#L68

Hello @ivankhoosty
Thanks for reaching out and raising this issue.
Fix https://github.com/aws/aws-advanced-jdbc-wrapper/pull/1004 is available on our latest snapshot build.
Could you check it out and let us know if the issue still persists?
Thank you!

thanks


config.yml defaults do not match documentation

According to the Warehouse configuration documentation, "The default configuration for Warehouse is inside of the warehouse/config.yml file."
configuration.rst specifies defaults that do not match config.yml
For example:
Debug is False by default, but the documentation states debug should be True.
paths.documentation is None, but the default value in config.yaml is data/packagedocs.
There are also inconsistencies with the descriptions.  database.url requires postgresSQL 9.3+, but the contributing document specifies 9.2+.  There are others.
Should this document be updated to match config.yml?

I'm going to close this, I've started porting Warehouse over to Pyramid to ease the concerns of the custom framework-ish thing I've got going on here. That work is on the pyramid branch inside this repository. Since that includes all new configuration and configuration files, this issue does not really apply anymore.


bad counter increase query result

What did you do?
Ran a query to determine the increase in a counter for the last hour.
What did you expect to see?
The result to equal the counter (now) - counter (offset 1hr)
What did you see instead? Under which circumstances?
A value greater than the counter's current value.
Environment

System information:
Linux 3.13.0-93-generic x86_64

Prometheus version:
prometheus, version 1.1.1 (branch: master, revision: 24db241bd556fd45854dd310692ac291de7d24e1)
build user:       root@90d3f69e2d67
build date:       20160907-09:42:10
go version:       go1.6.3

Prometheus configuration file:

global:
  scrape_interval:     1m
  evaluation_interval: 1m

scrape_configs:
  - job_name: 'service'
    consul_sd_configs:
    - server: <IP_ADDRESS>:8500
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: ',((api|data)-service),'
        action: keep
      - source_labels: [__meta_consul_service_address, __meta_consul_service_port]
        separator: ':'
        target_label: __address__
      - source_labels: [__meta_consul_service]
        target_label: name
      - source_labels: [__meta_consul_tags]
        target_label: tags

I suspect you have had some counter resets due to restarts. This is only of the many edge cases that increase() and friends have to deal with.

@brian-brazil thanks Brian. To be clear, we're talking about restarts on the application exposing the counter and not Prometheus itself, correct?

Yes

@brian-brazil does prometheus expose any info to help debug? Maybe a metric for resets?

There's the resets() function, though in general for this sort of issue it's best to look at the raw data.


release v0.16.0 missing static assets

Missing static linux/x86_64 assets for v0.16.0 on the release section (only contains the source packages). Auto-install scripts are failing which rely on the static release binary. Static release bins were available for all previous versions. Please add ❤

@bradlarsen
Unfortunately, the released binary is still dynamic:
--> https://github.com/praetorian-inc/noseyparker/releases/download/v0.16.0/noseyparker-v0.16.0-x86_64-unknown-linux-gnu.tar.gz
file noseyparker
noseyparker: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=fe547c8ef316f1385e0628159c7ad0e49ad45a48, for GNU/Linux 3.2.0, with debug_info, not stripped

To build static:
!# cross
  git clone --filter "blob:none" "https://github.com/praetorian-inc/noseyparker" && cd "./noseyparker"
  cargo install -f cross
  export TARGET="x86_64-unknown-linux-musl" ; rustup target add "$TARGET" ; export RUSTFLAGS="-C target-feature=+crt-static"
  sed '/^\[profile\.release\]/,/^$/d' -i "./Cargo.toml" ; echo -e '\n[profile.release]\nstrip = true\nopt-level = "z"\nlto = true' >> "./Cargo.toml"
  cross build --target "$TARGET" --release

!# cargo
  git clone --filter "blob:none" "https://github.com/praetorian-inc/noseyparker" && cd "./noseyparker"
  export TARGET="x86_64-unknown-linux-gnu" ; rustup target add "$TARGET" ; export RUSTFLAGS="-C target-feature=+crt-static"
  sed '/^\[profile\.release\]/,/^$/d' -i "./Cargo.toml" ; echo -e '\n[profile.release]\nstrip = true\nopt-level = "z"\nlto = true' >> "./Cargo.toml"
  cargo build --target "$TARGET" --release

---
  file "./target/$TARGET/release/noseyparker-cli"
  ./target/x86_64-unknown-linux-musl/release/noseyparker-cli: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped

@Azathothas @SkyperTHC
Perhaps I read this initial issue too quickly — I thought you were asking simply for the prebuilt releases. Indeed, the prebuilt Nosey Parker releases are dynamically linked.
The releases always have been dynamically linked, actually. However, prior to v0.14.0, the releases were simply single-file zipped binaries. Starting in v0.14.0, the releases are tarballs with some added directory structure and additional assets (such as shell completion scripts).
I've wanted to provide statically-linked binaries from the start (#58). However, I never figured out the proper incantation to cargo to make that work; the build would always fail somewhere when building native-code dependencies with complicated CMake scripts that interrogate the environment (vectorscan, zlib, ...).
However, the invocations you share above do seem to result in a static binary! It seems like the RUSTFLAGS="-C target-features=+crt-static" does the trick. Thank you! I will experiment with this more, and may include statically-linked binaries for future releases. (It would help with the use case of running on older LInux systems, like in #58.)
Note, however, that the Nosey Parker releases will continue to be distributed as tarballs in the future (and not as zipped single binaries). The open-source noseyparker does run correctly at present when pulled out of the release directory tree, but I am not promising to preserve that deployment scenario.

P.S. @SkyperTHC Is it this project that's using Nosey Parker releases, or something else?
https://github.com/hackerschoice/segfault
I'm just curious to hear about how Nosey Parker is getting used 😎


Contradictory typing.TypedDict >=3.11 or >=3.12 support

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
Hi,
The following piece of code is contradictory with respect to the python version which supports typing.TypedDict:
def _typed_dict_schema(self, typed_dict_cls: Any, origin: Any) -> core_schema.CoreSchema:
        """Generate schema for a TypedDict.

        It is not possible to track required/optional keys in TypedDict without __required_keys__
        since TypedDict.__new__ erases the base classes (it replaces them with just `dict`)
        and thus we can track usage of total=True/False
        __required_keys__ was added in Python 3.9
        (https://github.com/miss-islington/cpython/blob/1e9939657dd1f8eb9f596f77c1084d2d351172fc/Doc/library/typing.rst?plain=1#L1546-L1548)
        however it is buggy
        (https://github.com/python/typing_extensions/blob/ac52ac5f2cb0e00e7988bae1e2a1b8257ac88d6d/src/typing_extensions.py#L657-L666).
        Hence to avoid creating validators that do not do what users expect we only
        support typing.TypedDict on Python >= 3.11 or typing_extension.TypedDict on all versions
        """
        with self.defs.get_schema_or_ref(typed_dict_cls) as (typed_dict_ref, maybe_schema):
            if maybe_schema is not None:
                return maybe_schema

            typevars_map = get_standard_typevars_map(typed_dict_cls)
            if origin is not None:
                typed_dict_cls = origin

            if not _SUPPORTS_TYPEDDICT and type(typed_dict_cls).__module__ == 'typing':
                raise PydanticUserError(
                    'Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.',
                    code='typed-dict-version',
                )

The code comes from /pydantic/_internal/_generate_schema.py, line ~780
Curious which of the 2 it is, support from >=3.11 or >=3.12? The actual check is performed for 3.12, but the docstring states that 3.11 is sufficient. In case support for 3.11 should be available, the following check should be adjusted:
_SUPPORTS_TYPEDDICT = sys.version_info >= (3, 12)

Line 70 in the same file.
Example Code
import pydantic
from typing import TypedDict

# breaks on python 3.11, while the docstring above mentions it should actually be supported

Python, Pydantic & OS Version
pydantic version: 2.0.2
        pydantic-core version: 2.1.2 release build profile
                 install path: C:\Users\vantongeren\Documents\code\server-devops-engine\api\venv\Lib\site-packages\pydantic
               python version: 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]
                     platform: Windows-10-10.0.19045-SP0
     optional deps. installed: ['typing-extensions']

Selected Assignee: @samuelcolvin

@adriangb can you look at this, I thought 3.11 was enough, but I might be wrong.

Can I ask why? The Required/NotRequired annotations were added to the standard library in Python 3.11, and the comment in the typing_extensions link specifically says that The standard library TypedDict below Python 3.11 does not store runtime information about optional and required keys when using Required or NotRequired., so shouldn't 3.11 be supported too?

https://github.com/python/cpython/pull/103698

I see, thank you for the quick response

pydantic.errors.PydanticUserError: Please use typing_extensions.TypedDict instead of typing.TypedDict on Python < 3.12.


add HTP

add HTP

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Literature: Definition of Contributor Levels

How can we categorize a repository's contributors?
e.g.
Core, Active, Peripheral.
Find support in the literature for operationalizations of contributor categorization either by statistical or network strategies.
The goal is to be able to identify contributors by their respective roles of contribution. This will help us operationalize health metrics that involve intra-project communication, e.g. "do the core contributors do a good job of keeping track of up-and-coming contributors or are they very insulated?"
Two ways of doing this are:
(1) Social network centrality measurements.
(2) Statistical strategies, e.g. "the smallest number of contributors who make up 80% of the work are the cores."
This task involves researching different ways to identify contributor category and planning how to implement the categorization on a live network of Ansible contributors. For now, we'll only look at a single repo at a time.

Understanding core vs peripheral developer definitions
Core developers - play an essential role in developing the system architecture and forming the general leadership structure, and they have substantial, long-term involvement.
Peripheral developers - typically involved in bug fixes/small enhancements, and they have irregular or short-term involvement.
Core dev categorizing Intuition:
Count-based:

80th percentile (Zipf distribution - typically the top 20% of contributors are responsible for 80% of the contributions)
higher commit count, more frequent contributions
higher lines of code counts

Network-based:
(better reflection of developer perception)

degree centrality - core developers should have a larger degree than peripheral developers (more connections to more contributors, more connections to other core members)
eigenvector centrality - core developers should occupy globally central positions in the developer network
hierarchy - core developers should have a high degree and low clustering coefficient, placing them in the upper region of the hierarchy (vice versa for peripheral)
role stability - core developers' stability in the developer network should be be higher than for peripheral developers
block model - expected pattern: P(core-core) > P(core-peripheral) > P(peripheral-peripheral)

Note - developer roles can change over time, account for this by using sliding window!
Source

Validating assigned contributor roles:
Social network centrality - difficult to identify threshold (degree threshold for core developer node)
Instead: implement statistical strategies first, compile a list of cntrb_ids classified as 'core developers' for each strategy (core = smallest # contributors who make up 80% work/commits/prs etc).
→  Then visualize network, identify nodes with highest degrees and compare with results from statistical strats. (Use this to
identify an optimal degree threshold)

Finding Core contributors via count-based operations:
https://colab.research.google.com/drive/1LWaNAHcIYl_aRCdqGDhklJ39aMUaaByZ#scrollTo=PfSUreCvg4qA

Awesome work @mariashev! Can we add this notebook to the graph analysis folder on the repo as well? So we can collaborate better

@JamesKunstle @mariashev Can this issue be closed?

@mariashev @cdolfi this issue is useful because it has our thoughts on the Active v. Peripheral v. Core question but I think it can be closed because we're not working on it anymore, and we've made some downstream decisions.


Supporting Lemmy API

Any interest in supporting the Lemmy API as an alternative to Reddit?
https://users.rust-lang.org/t/building-a-better-r-rust-together/95194

I have a little interest but I don't know how can I help.

Looks like others are looking into the same thing: https://github.com/burhan-syed/troddit/issues/167
https://github.com/derivator/tafkars

@erlend-sh I don't know what your plan is. Find a new place that users could post and comment? Or it must support ActivityPub? Or it has all old posts from r/rust?

we are setting up a Lemmy instance as an alternative to /r/rust
It supports AP, which means it’s connected to the fediverse of 10 million users.
We probably don’t need all old posts, just new ones.

On that last point, that’s where tafkars by @derivator comes in. If existing Reddit clients can learn to talk with Lemmy as well, then people’s browsing experience could continue with minimal friction.


before all hook sometimes times out in PRBMath Solidity test

I have seen this fail at least twice (972388, 970845) when running tests:
  298 passing (3s)
  1 failing

  1) PRBMath Solidity
       "before all" hook in "PRBMath Solidity":
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/tmp/ext-test-PRBMath-Z8KoNI/ext/test/contracts/index.test.ts)

Either something's going wrong there or the test just needs a longer timeout. I've seen a similar problem recently in Euler and there it's just a question of tests taking too long (https://github.com/euler-xyz/euler-contracts/issues/97).

Hmm, I think this is just Hardhat having changed the default maximum running time for tests. I bumped into the same issue in one of my other projects and I solved it by bumping the timeout to 30 seconds.
Can you pull from main and see it if it works now?

I haven't seen it fail again so far and it's always running on a fresh checkout from main so looks like it's fixed. In that case I'm closing this and I'll reopen if I bump into it again. Thanks!


table schema

Documentation doesn't mention what table schema should be, for this to work.
As I can tell it requires global commit timestamp column to exist?

The only requirement is that each table that is monitored contains a column with allow_commit_timestamp set to true, and that the column is filled with the latest commit timestamp after each successful commit. The name of the column may be freely chosen. It is also not a requirement that all tables contain such a column, only the column that you want to monitor need it.
The spanner-change-watcher library will also automatically create a table named LAST_SEEN_COMMIT_TIMESTAMPS in your database to keep track of the last commit timestamp for each table that is being monitored. If you don't want this, you could configure a different table to be used. Have a look at the samples for more information and examples on this.

@olavloite what if there are multiple columns with allow_commit_timestamp?
Also I tried to make sharded queries on global timestamp column, but it seems optimizer really doesn't like it: https://stackoverflow.com/questions/61999247/is-it-possible-to-have-efficient-timestamp-ordered-queries-in-spanner
I have seen that you are using same type of query and it seems it's vastly inefficient (always does full table scan). It works if timestamp index is not sharded, but then spanner folks recommend against it (due to hotspots).

@kanekv

what if there are multiple columns with allow_commit_timestamp?

That should not be possible. Only one column in a table can have the allow_commit_timestamp option set to true. If you already have a column with that option set, and then alter the table to set the option on a different column, the option will automatically be unset on the first column.

I have seen that you are using same type of query and it seems it's vastly inefficient (always does full table scan). It works if timestamp index is not sharded, but then spanner folks recommend against it (due to hotspots).

Yes, I know that is a potential problem at the moment, depending on the size of the table that you want to monitor. I'll look into possible solutions for that as well, but preferably without putting any strict requirements on exactly how your data model needs to be. E.g. if you just want to monitor one or more relatively small tables, then you should be fine without any additional indexes, and in other cases the library should be able to auto-detect which indexes are present and use those.

That should not be possible.

This doesn't seem to be correct, I am able to create tables with multiple commit timestamps allowed and it works (and sets them properly with commit timestamps).

This doesn't seem to be correct, I am able to create tables with multiple commit timestamps allowed and it works (and sets them properly with commit timestamps).

You seem to be right about that. I'm not sure where I got my (wrong) assumption from. The effect will in that case most probably be that it will use the first commit timestamp column it encounters, but I'll look into it.

I think it needs explicit config: I am using some timestamp columns in mixed mode, sometimes setting to commit timestamp and sometimes overriding it. It will be just more flexible if it's configurable.

I'll look into possible solutions for that as well, but preferably without putting any strict requirements on exactly how your data model needs to be.

I have created an issue about optimizer bugs: https://issuetracker.google.com/issues/157388602. Do you know of a better avenue to access spanner devs?

I am particularly interested in this, because I have manually built exactly what you are doing here, watching a table and processing/pushing it to pubsub =)

I'll look into possible solutions for that as well, but preferably without putting any strict requirements on exactly how your data model needs to be.

I have created an issue about optimizer bugs: https://issuetracker.google.com/issues/157388602. Do you know of a better avenue to access spanner devs?

For these kinds of questions, filing an issue/question there is the best way. As far as I can see, there is also already an answer to the specific question.

I am particularly interested in this, because I have manually built exactly what you are doing here, watching a table and processing/pushing it to pubsub =)

Always nice to hear :-)
You're free to try this out, and while there might be some changes coming up that could break the current API a little bit, it should be quite limited. One thing to take into account though, is that this solution currently relies on the async API for the Java Spanner client library that has not yet been released.
That means that you would need to add a couple of custom build steps if you want to integrate into your existing application.


Add some explanatory comments to the wordcount registration

This should help clarify wordcount in our starter examples. I didn't add this in all the examples since I think these are where people tend to start, but that wouldn't be a bad idea in the future.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @pcoet


I would suggest adding some screen shot of your editor in the markdown file

People will know how your editor looks like if you supply some screen shot, thanks.

Thank you for the suggestion! I embedded an image into the README.md . Sorry for the late response. I did not get any sort of notification that someone opened an issue.


Add option to not destroy headers

Currently, we always destroy the PE headers before execution. There should be an option to skip this in case the payload has some logic that requires the existence of those headers at runtime.

-k,--headers: <level>                 Preserve PE headers. 1=Overwrite (default), 2=Keep all

Resolved in dev, will be merged with next version.


Password Requested on wallet without a password

Hi there,
System information
Semux version: e.g., v1.5.0-rc.1
OS & Version: e.g., Windows 10
Expected behaviour
Description
Start client without a password still requests a password.
If a wallet has no password set (nevermind the security problem this imposses), the client still requests a password.  So, a user might waste time and worry about having lost a password that was never set.
Actual behaviour
Client requests password.
Steps to reproduce the behaviour

Start the client using a wallet without a password set.

#108 fixes this


DependancyCheck | Downloading 403 Forbidden

Describe the bug
When running the Azure DevOps Plugin "Dependancy Check" when it goes to https://dl.bintray.com/jeremy-long/owasp/dependency-check-5.3.2-release.zip I am getting a 403 Forbidden error message.
Version of dependency-check used
5.3.2
Log file
Starting Dependency Check...
Creating dependency check test results directory at d:\AgentData\Agent-Azure01\37\TestResults\dependency-check
Directory: D:\AgentData\Agent-Azure01\37\TestResults

Mode                LastWriteTime         Length Name

d-----        7/31/2020   3:18 PM                dependency-check
Downloading Dependency Check installer...
##[error]System.Net.WebException: The remote server returned an error: (403) Forbidden.
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
Ending Dependency Check...
To Reproduce
Steps to reproduce the behavior:

Go to any Windows Server or Powershell Core Terminal (Administrator Mode)
Execute => [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 => Ensures outbound connections for invoke-webrequest are going over TLS1.2
Execute => Invoke-WebRequest "https://dl.bintray.com/jeremy-long/owasp/dependency-check-5.3.2-release.zip" -OutFile "dependency-check-5.3.2-release.zip"
See error => 403 Forbidden.

You can also just go to that URL in your browser and return a 403.
Expected behavior
Download file and output to location
Additional context
Add any other context about the problem here.

Duplicate of #2740

@jeremylong -- not sure if intended, but you marked this ticket as a duplicate of itself.  ;)
I'm wondering if there will be a fix any time soon as this is actually blocking time-sensitive work.  just need to reset expectations if would be a while.

Sorry - duplicate of #2739. This should be resolved.


Wrong Federated identity credentials support scope

Hello there,
the Section Design Considerations on the Page "Overview of federated identity credentials in Microsoft Entra ID" states following:

Federated identity credentials are supported on applications only. [...]

Which is... not true? We are using Federated Credentials together with User Assigned Managed Identity for quite some time.
cheers
flash ⚡

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 6dd2c231-5d5c-f5c2-ce5b-9e66b890745b
Version Independent ID: 1eb57d56-5c8a-0984-ef01-972576dee1d2
Content: Overview of federated identity credentials in Microsoft Entra ID - Microsoft Graph v1.0
Content Source: api-reference/v1.0/resources/federatedidentitycredentials-overview.md
Product: applications
Technology: microsoft-graph
GitHub Login: @nickludwig
Microsoft Alias: MSGraphDocsVteam

Thank you @flash-me for raising this issue. We'll investigate the proper endpoint for adding these credentials to managed identities and update the documentation.

Hey @flash-me - you're totally right... not sure how this went undetected. Thanks for opening the issue! I'll go ahead and submit a PR to remove that note.


Flutter doesn't have a way to handle bottom view insets/notches on Android devices

Mirror issue for https://b.corp.google.com/issues/136122352. Customer related comments should go there, please.

does SafeArea bit work?

Looks like a legit bug:
https://github.com/flutter/engine/blob/3f4f606105a8f6217cfbe08e193aa9a54b19dbef/shell/platform/android/io/flutter/view/FlutterView.java#L537

cc @jason-simmons, with whom I discussed this last week. Jason, is this something you can look at this week?

The Android embedding is mapping the system window bottom inset to window.viewInsets.bottom on the framework side.  This is intended to force a relayout excluding the keyboard region when the keyboard is shown.
@Hixie is the framework expecting different behavior here?  Should the Android embedding also set window.viewPadding.bottom to the same value as window.viewInsets.bottom?

A note from customer: fun(g3):

Hey, just an update. I tested locally and I found out that Scaffold automatically avoids the bottom
insets. To see the behavior we're seeing, either don't use a scaffold, or set
resizeToAvoidBottomInset to false on the scaffold. There are some widgets that should extend
their layouts under the padding, but right now the bottom padding is mistaken as a view/keyboard
inset (which isn't expected to have content laid out under).

@Piinks was working on this recently.

Here's a few points of reference to some of the recent work here:
https://github.com/flutter/engine/pull/8848
https://github.com/flutter/flutter/pull/33999
https://github.com/flutter/flutter/pull/34298
cc/ @dnfield he may know the answer to your question @jason-simmons. Do we have a runnable reproduction of this? I'm not sure I understand what is happening here versus the expectation.

We introduced a distinction between Window insets and Window padding to maintain some compatibility with how it used to work, which turned out to have some problems on iPhone X: https://github.com/flutter/engine/blob/master/lib/ui/window.dart#L531 and https://flutter.github.io/assets-for-api-docs/assets/widgets/window_padding.mp4
It looks like that Java class should be setting the viewPadding now.

@csells Could you mark this with "customer: solaris"?

This is fixed on https://github.com/flutter/engine/pull/20921 for Android 11 but worth fixing for lower versions too. Thanks for your PR @lazylazyllama

This should now be fixed since flutter/engine#21730 landed.


FreeRTOS+TCP using same uncrustify rules as AWS and FreeRTOS kernel

Description
Until now, the formatting of the FreeRTOS+TCP source code was a bit different from other AWS libraries.
From now on the same uncrustify config file will be used, e.g. uncrustify.cfg.
I applied uncrustify with mentioned config file and pushed the changed source files.
In AWS/FreeRTOS, I submitted the same kind of PR (#2262).
For those addicted to using tabs ( like me ), you can make the following change in your uncrustify.cfg:
-indent_with_tabs                = 0        # unsigned number
+indent_with_tabs                = 2        # unsigned number

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/bot run checks

We would also want to update the pre_commit.py script to include the FreeRTOS+TCP files in the CI style checks. The change needed would be removal of this line: https://github.com/FreeRTOS/FreeRTOS/blob/lts-development/tools/git/hooks/src/pre_commit.py#L123

Since I submitted this PR, many things have changed in the repo, and it will take a lot og rebasing/merging to get it OK.
One of the good things that will changes is that for CBMC, the patch files won't be used anymore for FreeRTOS+TCP.  These were used to remove certain keywords like "static".
Once these changes are ready, I will create a new PR that applies the uncrustify rules.
Now closing this PR.


Vendor contributor not allowed to open internet

As a vendor contributorer

Go to a sandbox
Go to a vm
See that open internet is disabled

Confirmed in the network tab

As of now, request open internet is not a feature, but will be available in the future. As of now, the vendor contributor must call or send a mail to relevant personell


Enable WriteCore feature for hybridaks

We need to enable WriteCore feature to hybridaks. Therefore add use-write-core: true and do a regen.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.HybridContainerService


Included currentNonce in estimation endpoint respnse

Closes #711

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

81 of 81 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.08%) to 87.413%

Totals

Change from base Build<PHONE_NUMBER>:
0.08%

Covered Lines:
10160

Relevant Lines:
11623

💛  - Coveralls


Unable to install Py3.11 Linux wheel

Hi! Thanks for all the work on this fork! This is a follow-up to this thread.
OS: Ubuntu Linux 20.04. Python: 3.11.
Steps;

Download pycoral-2.13.0-cp310-cp310-linux_x86_64.whl from [pycoral-2.13.0-cp310-cp310-linux_x86_64.whl](The releases page)
python3.11 -m pip install pycoral-2.13.0-cp310-cp310-linux_x86_64.whl

Error:
ERROR: could not find a version that satisfies the requirement tflite-runtime==2.13.0 (from pycoral) (from versions: 2.14.0).
Error: No matching distribution found for tflite-runtime==2.13.0

Of note, the official tflite-runtime package on Pypi doesn't include this version. So, I'm not sure if the fix is only support 2.14+ on PyCoral for Python3.11, or if you think there's another way. I think the former is fine, but given you were able to build these wheels, I'm curious how. (https://pypi.org/project/tflite-runtime/2.13.0/#files)
I already ran into a similar error when attempting to build wheels from the source code in this repo for both Windows/Py3.11, and Ubuntu/Py3.11.

As stated you need to download and install first: https://github.com/oberluz/pycoral/releases/download/2.13.0/tflite_runtime-2.13.0-cp310-cp310-linux_x86_64.whl

$ python --version
Python 3.10.12

$ python3 -m pip install ~/Downloads/tflite_runtime-2.13.0-cp310-cp310-linux_x86_64.whl
Defaulting to user installation because normal site-packages is not writeable
Processing /home/eyeot-demo/Downloads/tflite_runtime-2.13.0-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.10/dist-packages (from tflite-runtime==2.13.0) (1.23.3)
Installing collected packages: tflite-runtime
Successfully installed tflite-runtime-2.13.0

$ python3 -m pip install ~/Downloads/pycoral-2.13.0-cp310-cp310-linux_x86_64.whl
Defaulting to user installation because normal site-packages is not writeable
Processing /home/eyeot-demo/Downloads/pycoral-2.13.0-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: Pillow>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pycoral==2.13.0) (10.1.0)
Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.10/dist-packages (from pycoral==2.13.0) (1.23.3)
Requirement already satisfied: tflite-runtime==2.13.0 in /home/eyeot-demo/.local/lib/python3.10/site-packages (from pycoral==2.13.0) (2.13.0)
Installing collected packages: pycoral
Successfully installed pycoral-2.13.0

Thank you.
python3.11 -m pip install pycoral-2.13.0-cp310-linux_x86_64.whl
ERROR: pycoral-2.13.0-cp310-linux_x86_64.whl is not a supported wheel on this platform.

Yes, 3.11 is also available, but in your description above you mentioned pycoral-2.13.0-cp310-cp310-linux_x86_64.whl

Oops, sorry; typo.

So, where you able to install 3.10 or 3.11?

No. I don't have Python 3.10, and 3.11 gives the error described.

Where is that error? seems you deleted it

somthing about unsupported architecture

I injected too much confusion here due to the typo in the first post. Let's start fresh, please ignore everything prior to this. I'm on my windows machine now, so was manually transcribing from the ubuntu laptop.
Hi! Thanks for all the work on this fork! This is a follow-up to https://github.com/google-coral/pycoral/issues/85#issuecomment-1793768842.
OS: Ubuntu Linux 20.04. Python: 3.11.
Steps;
Download pycoral-2.13.0-cp311-cp311-linux_x86_64.whl from [pycoral-2.13.0-cp311-cp311-linux_x86_64.whl](The releases page)
python3.11 -m pip install pycoral-2.13.0-cp310-cp311-linux_x86_64.whl

Error:
ERROR: could not find a version that satisfies the requirement tflite-runtime==2.13.0 (from pycoral) (from versions: 2.14.0).
Error: No matching distribution found for tflite-runtime==2.13.0
Of note, the official tflite-runtime package on Pypi doesn't include this version. So, I'm not sure if the fix is only support 2.14+ on PyCoral for Python3.11, or if you think there's another way. I think the former is fine, but given you were able to build these wheels, I'm curious how. (https://pypi.org/project/tflite-runtime/2.13.0/#files)
I already ran into a similar error when attempting to build wheels from the source code in this repo for both Windows/Py3.11, and Ubuntu/Py3.11.

edit: Oh, wow, you solved it. I missed the manual download the tflite runtime, under show all assets. Carry on! Sorry about that.

Yes you always need to install tflite_runtime first and then pycoral, using the same version for both


Mobile design

Add comments to organize CSS and Sass variables (for initial setup)

This also includes the desktop design, due to the fact that not any CSS had to be changed/written to match the desktop design.


Hashtags Updated

Description
Please include a summary of the changes and the related issue. Please also include relevant context. List any dependencies that are required for this change.
Hashtags u[dated in the sample messages in all IG and bootcamps
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New Feature (non-breaking change which adds functionality)
[ ] HotFix (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings

These changes are approved, but one thing is that in a file you have changed the placement of hashtag attribute of a component, it won't break the code but that won't work.

Here changing the location of the attribute hashtags do not make any change, instead you should track down the file/component named ExampleMessage and inside file update the code to get the required output.
Otherthan that its Perfectly Fine


STORM-3073: Uncap pendingEmits for bolt executors, and prevent LoadSpout from overflowing pendingEmits in spout executors

https://issues.apache.org/jira/browse/STORM-3073
The first commit contains the changes I made to ExclamationTopology to provoke the error. I'll remove it after review, it is just included so it's easier to understand the error.
There are two changes in this PR. The first is to uncap the pendingEmits queue for bolt executors. It's currently capped at 1024 elements, which makes it dangerous for bolts to emit more than 1024 tuples in an execute invocation. If the bolt executor is experiencing backpressure and tries to add the tuples to pendingEmits, the queue size will be exceeded and the worker will crash.
The second change is to make LoadSpout emit failed tuples from nextTuple instead of doing it from fail. Since the spout executor is also limited to 1024 tuples in the pending queue, it is likely that the spout executor will exceed the queue limit and crash if a bunch of tuples fail at the same time (e.g. due to timeout) while the spout is adding tuples to pendingEmits. Since the spout won't call nextTuple if there are tuples in pendingEmits, we can just move the retries to that method to prevent the spout from exceeding the pendingEmits limit.

@HeartSaVioR You're right, uncapped the queue in both types of executor, and removed the test code.

Thanks for the quick review


Is it possible that tcl/tk will build on a mac that is configured just to build portable-python?

If I use a Mac that has might have nothing else installed, so nothing in /usr/local; and that might have an updated version of tcl/tk installed, that portable-python will build with support for the included tcl/tk? Re the updated version of tcl/tk, if I am building a portable version of 3.10.4, I could install the regular install of this version of Python first.
I have a Arm iMac that I can easily reset to factory default. Would it be useful for me to try some experiments? If so, what would you suggest? Is the main issue to get the right settings on compiler flags?
I realize that this will not solve the more general problem of having a version of pp that will run on most user machines. But for me, it would still be helpful to create a build of pp that I could sign and then move to other machines I am using.

Do you mean you want to build a python binary that has references to shared libs? In your case tcl/tk?
That in theory is well supported, there is just a "default assumption" that you want a clean portable python that does NOT refer to any shared libs a user may not have.
You can configure things such as shared libs are OK, with allowed-system-libs: .* (it's a regex you can fine-tune).
That allowed-system-libs thing is just for the validation step after the build.

The general philosophy is to try and use the upstream ./configure scripts as-is, as much as possible.
For the main use-case (get a real "portable" python build), some isolation is attempted, but even that can be fine-tuned/configured further, to allow also for "yup, I'm OK with it not being 'purely portable'". That part may need a bit of work, but I think it's very doable :)
I did try to account for that case, where it is OK to have it refer to some shared libs.
The inspect command tries to make evident those shared libs, and a "validate" step checks against the config what is OK and what is not (so that you don't get a build out that uses some shared libs that you didn't want)

Yes, portable-python is intended to work that way: build on one machine, then install that on N machines (and when the produced tarball is truly portable, then the installation can be as simple as "unpack in some folder - any folder, and it'll just work from there). That's how I use it :)
One could also I guess "build on-the-fly" on each machine, but then if one does so then the "portability" is moot... since it was built on the target machine where it'll run, the fact that it is "portable" is over-kill in a way... (ie: you don't really need portable-python then one could argue...)
So yes, portable-python is intended to work like that (build once, install many times).

Regarding tcl/tk, I did try originally to allow for it to be included just like all the other components are: openssl etc.
However, it turned out that tcl/tk is hard to build, especially on linux :(
So I moved the tcl/tk code to here: https://github.com/codrsquad/portable-python/tree/main/retired
Awaiting to get some time again to try and get to the bottom of it (how to make it build properly, by delving into it doc).
We can try and make a branch where we try to resurrect it.

I'll try and resurrect the tcl/tk code, it should be OK to include it, provided it doesn't get "auto-selected" (and requires explicit opt-in basically, via config or command line args).
Once I do that, you can play with it and try to make the C compilation work :) (just gave it a quick try, and it fails miserably even on mac now)

I've re-activated tcl/tk: https://github.com/codrsquad/portable-python/commit/c1e223417159e764313dc3b853fd3d7c968ec116
I've also released portable-python 1.4.3, you can now try for example: portable-python build 3.10.5 -mtkinter, and then look at build/cpython-3.10.5/logs/ to see why it didn't pick it up :)
You'll see that apparently even with the documented --with-tcltk-includes= etc passed to cpython's ./configure, it still seems to ignore that, and eventually ends up using /Library/.../MacOSX.sdk/System/Library/Frameworks/Tcl.framework/Headers etc

I tried with 8.6.12, but that fails to C-compile (C compilation errors).
8.6.10 C-compiles at least, but ends up being ignore anyway.

I've added this file in case it helps, on how to run portable-python in the debugger, the tests etc: https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md


avoid data race read volume.IsEmpty

What problem are we solving?

phantom read: query empty state in different v.dataFileAccessLock scope may use outdated isEmpty when v.Destroy()
race read v.DataBackend.GetStat(): v.IsEmpty() use v.DataBackend.GetStat() out of v.dataFileAccessLock scope

How are we solving the problem?

query empty state & apply effect within one scope of v.dataFileAccessLock to avoid phantom read
let v.IsEmpty() use v.DataBackend.GetStat() in v.dataFileAccessLock scope

How is the PR tested?
Checks

[x] I have added unit tests if possible.
[ ] I will add related wiki document changes and link to this PR after merging.

It seems that you should not be smart with v.Destroy () and leave it as it is
https://github.com/seaweedfs/seaweedfs/pull/4576de

It seems that you should not be smart with v.Destroy() and leave it as it is #4576

The intention of modifying v.Destroy() is to avoid the phantom read issue which is showed by this demo:
https://github.com/bai-jie/seaweedfs/compare/onlyEmpty...onlyEmpty-demo
The output of the demo for both cleaner() implementations is
Query empty state & apply effect within one scope of v.dataFileAccessLock:
╭─    ~/remote/infrastructure/code/seaweedfs  on   onlyEmpty ⇣4⇡6 +4 ?6 ····················································································································································································· ✔  took 53s  ─╮
╰─ make tests                                                                                ─╯
cd weed; go test -tags "elastic gocdk sqlite ydb tikv" -v -count=1 -run TestPhantomRead github.com/seaweedfs/seaweedfs/weed/storage
=== RUN   TestPhantomRead
10000 at 2023-06-14 23:37:56.905628 +0800 CST m=+6.401290843
20000 at 2023-06-14 23:38:02.610577 +0800 CST m=+12.106146980
30000 at 2023-06-14 23:38:08.279773 +0800 CST m=+17.775250006
40000 at 2023-06-14 23:38:14.051018 +0800 CST m=+23.546401183
50000 at 2023-06-14 23:38:19.67352 +0800 CST m=+29.168811063
60000 at 2023-06-14 23:38:25.938255 +0800 CST m=+35.433444290
70000 at 2023-06-14 23:38:31.672151 +0800 CST m=+41.167246146
wrongCount: 0
--- PASS: TestPhantomRead (41.13s)
PASS
ok      github.com/seaweedfs/seaweedfs/weed/storage     41.252s

Query empty state & apply effect with different scopes of v.dataFileAccessLock:
╭─    ~/remote/infrastructure/code/seaweedfs  on   onlyEmpty ⇣4⇡6 +4 !1 ?6 ·················································································································································································· ✔  took 57s  ─╮
╰─ make tests                                                                                ─╯
cd weed; go test -tags "elastic gocdk sqlite ydb tikv" -v -count=1 -run TestPhantomRead github.com/seaweedfs/seaweedfs/weed/storage
=== RUN   TestPhantomRead
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/4864.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/6387.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/9217.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/9480.dat: file not exists after write
10000 at 2023-06-14 23:44:50.592717 +0800 CST m=+7.213177231
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/13561.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/15853.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/16128.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/19185.dat: file not exists after write
20000 at 2023-06-14 23:44:57.474897 +0800 CST m=+14.095244535
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/20740.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/22301.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/22723.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/23626.dat: file not exists after write
30000 at 2023-06-14 23:45:05.016005 +0800 CST m=+21.636229153
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/32496.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/32566.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/33121.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/33970.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/34241.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/38649.dat: file not exists after write
40000 at 2023-06-14 23:45:12.556986 +0800 CST m=+29.177086933
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/40782.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/44042.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/45537.dat: file not exists after write
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/49728.dat: file not exists after write
50000 at 2023-06-14 23:45:18.740834 +0800 CST m=+35.360833897
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/56453.dat: file not exists after write
60000 at 2023-06-14 23:45:24.629761 +0800 CST m=+41.249665187
writer /Volumes/RAMDisk/seaweedtest/1686757483415094000/64246.dat: file not exists after write
70000 at 2023-06-14 23:45:30.621242 +0800 CST m=+47.241047545
wrongCount: 24
--- PASS: TestPhantomRead (47.21s)
PASS
ok      github.com/seaweedfs/seaweedfs/weed/storage     47.357s
╭─    ~/remote/infrastructure/code/seaweedfs  on   onlyEmpty ⇣4⇡6 +4 !1 ?6 ·················································································································································································· ✔  took 57s  ─╮
╰─ ls /Volumes/RAMDisk/seaweedtest/1686757483415094000/64246.*                                                                                ─╯
ls: /Volumes/RAMDisk/seaweedtest/1686757483415094000/64246.*: No such file or directory
╭─    ~/remote/infrastructure/code/seaweedfs  on   onlyEmpty ⇣4⇡6 +4 !1 ?6 ····························································································································································································· 1 ✘ ─╮
╰─ ls /Volumes/RAMDisk/seaweedtest/1686757483415094000/64245.*                                                                                ─╯
/Volumes/RAMDisk/seaweedtest/1686757483415094000/64245.dat /Volumes/RAMDisk/seaweedtest/1686757483415094000/64245.idx /Volumes/RAMDisk/seaweedtest/1686757483415094000/64245.vif
╭─    ~/remote/infrastructure/code/seaweedfs  on   onlyEmpty ⇣4⇡6 +4 !1 ?6 ······························································································································································································· ✔ ─╮
╰─ ls /Volumes/RAMDisk/seaweedtest/1686757483415094000/64247.*                                                                                ─╯
/Volumes/RAMDisk/seaweedtest/1686757483415094000/64247.dat /Volumes/RAMDisk/seaweedtest/1686757483415094000/64247.idx /Volumes/RAMDisk/seaweedtest/1686757483415094000/64247.vif


Calendly integration for dashboard publish

closes #1733

Regarding the telemetry changes, I'll let @magorlick do the thorough review, but one idea is to add a schema version to each event type.

Thanks @ericpgreen2 . Good point about the schema. Feel like it fits the other PR https://github.com/rilldata/rill-developer/pull/1760 more than this.

Oh I hadn't seen 1760 – agreed.
I'll approve from the code standpoint, but I imagine @magorlick wants to review UX.

@magorlick Have made the changes you suggested. I am not an expert in css, so lemme know if it looks good.
The telemetry is firing for me. It fires when you open the calendly modal and after you have successfully scheduled a meeting. Lemme know if this works.

@AdityaHegde , I can see the start and success telemetry now and the felt experience is better. Very close!
The calendly link is no longer correct. Please update to the one I am DMing you, also in our docs. :)

@magorlick Updated the link.


Fix bug on request->has()

Inversion of terms in ternary operator within method has()
Method was failing to match multiple keys passed by means of an array. However, it would appear to be working properly if argument were a string or an array with a single key.
Reason: the ternary operator has its terms inverted. func_get_args() should be the result of the condition is_array($keys)

Please disregard.
For some reason, I was convinced it's purpose was to check the request items against keys passed as an array.
My apologies.


Fix ReachableStates::append_state results in a panic at units::Probability::from with a probability of 1.0000001 #27

#27

The bug was the check in ReachableStates::append_state at line 547:
debug_assert!(
            !(Probability::from(0.9999999) < self.probability_sum()
                && self.probability_sum() < Probability::from(1.0000001))
        );

While this check was designed to reduce bugs, the creation of the over-the-limit probability Probability::from(1.0000001). This was turned into assert_eq!(self.probability_sum(), Probability::from(1.)), as well as a check in Probability.
pub fn check_in_bound(&self) -> Result<(), ErrorKind> {
        if !(0. ..=1.).contains(&self.0) {
            return Err(ErrorKind::ProbabilityOutOfRange(OutOfRangeError::new(
                *self,
                Probability(0.),
                Probability(1.),
            )));
        }
        Ok(())
    }


chip_data_model.gni: add option for custom-implemented clusters

Add zap_clusters_with_custom_implementation option to chip_data_model() template, which allows to list clusters that need to be available in a device, but don't use the standard implementation in src/app/clusters/<clustername>/<clustername>.cpp.
Usage
To exclude a particular cluster's standard implementation in favor of an application level implementation:

in the BUILD.gn file for the application specific .zap file
in the instantiation of the chip_data_model template
add a variable zap_clusters_with_custom_implementation
assign it a list of strings with cluster names, corresponding to directories in the src/app/clusters directory.
this causes the cluster implementation file(s) not to be included in the build, allowing re-implementation at the application level.

Example, excluding standard implementation of level-control:
chip_data_model("zap") {
  zap_file = "myproject.zap"
  zap_clusters_with_custom_implementation = [ "level-control" ]
  zap_pregenerated_dir = "//zap-generated"
}

Background
Some clusters, for example Level Control, are implemented with the assumption of direct low-level access to the device hardware. For bridged devices, the actual interface might be much more high level, as in my case where I have lights which are fully capable of performing parametrized smooth transitions but are NOT capable of receiving output changes in millisecond intervals. In such cases, a cluster might need application-specific re-implementation.
This changeset allows, from the application's ZAP BUILD.gn, to exclude the standard cluster implementation file, allowing re-implementation of the needed ember callbacks in a application specific file outside the connectedhomeip repo.
Testing

tested to not affect normal builds not using this feature
tested with an application which re-implements level-control in a separate .cpp file. Without the feature, linking has duplicate symbol errors (from both custom and standard implementations) - using the feature as shown in the example above, the app links and runs ok.

I think this is an important feature. Its further review was postponed to post-1.0, which would be now ;-)

I still think this is an important feature. Its further review was postponed to post-1.0, which would be now ;-)

The PR is marked as draft - should this not be draft anymore?
It probably also suffers from being very far down the open PR list ... less likely to get reviews.

@plan44 - please remove this from draft if that is appropriate - that should get the bot start asking for reviews.

@plan44 - please remove this from draft if that is appropriate - that should get the bot start asking for reviews.

I will, as soon as I find time to rebase and re-test on master. Thanks for having had a look already :-)

@andy31415 I've rebased and retested a few days ago, and removed the draft status. Checks look fine except for pullaprove/license/cla, which I do not understand, as I did agree to the CLA a while ago and some PRs of mine are merged already. I have no idea how to re-agree to the CLA, if that's what the status is asking for.

@plan44, @andy31415 is on vacation right now, as is @woody-apple, and they're the obvious people to be able to override the CLA bits....
In practice you probably want to ping people after Jan 3.

@bzbarsky-apple thanks for the info - no problem waiting until next year. Or rather than waiting, maybe finding time to prepare the libev and gaps-in-endpointId PRs ;-)


Adapt Multi-Modal Use-Case Functionality for a range of databases

[ ] PostgreSQL
[ ] Clickhouse
[x] Snowflake
[x] MySQL
[x] SQLite
[x] DuckDB
[x] BigQuery

@fnikolai to re-test to check #1496.

PostgreSQL depends on #1558 so closing for now.


Fix containers tree view titles

In containers tree view the titles are mixed up.
The tab labeled "Relationships" does not show relationships, it show the container list.
The tab labeled "All containers" does not shoe All containers, it show subsections filtered from the containers list.
before:

after:

@miq-bot add_label providers/containers, ui
@moolitayer @zeari @dkorn @simon3z please review

@miq-bot add_label bug

LGTM :+1: @simon3z can we darga/yes this?

LGTM :+1:

LGTM :+1:

Nice! Thanks @yaacov @moolitayer! :+1:
@dclarizio @h-kataria objections?
@miq-bot add_label darga/yes
@miq-bot assign chessbyte

looks good.


Parallel processing in loop for more than 200 iteration causing zombie processes

I'm using amphp/parallel to download a number of image files parallel, so that it could take less time as compare to normal processing. It is seen that, it is working fine if there is request <= 200, all files are being downloaded successfully within few seconds.
The issue is that, if request > 200, it gets stuck and it is seen that there are 200 zombie processes are created in process table list. It can be seen using Linux top command. However, in this case 200 files are able to download successfully.
/**
 * $items An array of files to download
 */
Loop::run(function () use ($items) {
            try {
                $processArray = array();

                for ($counter = 0; $counter <= count($items); $counter++) {
                    // Create a new child process that does some blocking stuff.
                    $context = yield Process::run(__DIR__ . "/Blocking.php");

                    \assert($context instanceof Process);

                    array_push($processArray, $context);

                    // Pipe any data written to the STDOUT in the child process to STDOUT of this process.
                    rethrow(ByteStream\pipe($context->getStdout(), ByteStream\getStdout()));

                    yield $context->send($items[$counter]);
                }

                foreach ($processArray as $p) {
                    yield $p->receive(); // message from child;
                    yield $p->join(); // Terminate the child process
                }
            } finally {

            }
        });

Blocking.php
return function (Channel $channel): \Generator {
    $data = yield $channel->receive();
    file_put_contents($data['path'], file_get_contents($data['url']));
    yield $channel->send("");
    return true;
};

Due to zombie processes still in process table, there is no response further for new request. It needs to restart the Apache server and it clears the zombie processes.
Proof: https://drive.google.com/file/d/1Z4WMGKA_56QaBMdZ6wXkyx7ilBPYYSkw/view
I do not know why zombie processes are creating. Is there implementation issue of parallel processing in my code? Are processes are not exiting correctly? Is their any Apache limitation of process handling per request?

It appears xdebug was enabled. Please disable xdebug and let me know if the problem persists.

I have disabled the xdebug, but there is no success. Got the same issue again.

I have disabled the xdebug, but there is no success. Got the same issue again.

I have disabled the xdebug, but there is no success. Got the same issue again.

I have disabled the xdebug, but there is no success. Got the same issue again.

If you're just downloading files, I strongly recommend using https://github.com/amphp/http-client instead, which doesn't require multiple processes. It'll be much more efficient and performant.
Zombie processes might be due to open file limits kicking in and might only be a follow-up error.

If you're just downloading files, I strongly recommend using https://github.com/amphp/http-client instead, which doesn't require multiple processes. It'll be much more efficient and performant.
Zombie processes might be due to open file limits kicking in and might only be a follow-up error.

As per your suggestion, I'm modifying the code be like this
Loop::run(static function () use ($items): \Generator {

        // Instantiate the HTTP client
        $client = HttpClientBuilder::buildDefault();

        $requestHandler = static function (string $uri) use ($client): \Generator {
            /** @var Response $response */
            $response = yield $client->request(new Request($uri));
            return yield $response->getBody()->buffer();
        };

        try {
            $promises = [];

            foreach ($items as $item) {
                $promises[$item['url']] = Amp\call($requestHandler, $item['url']);
            }

            $bodies = yield $promises;

            foreach ($bodies as $item => $body) {
                print $uri . " - " . \strlen($body) . " bytes" . PHP_EOL;
            }
        } catch (HttpException $error) {
            // If something goes wrong Amp will throw the exception where the promise was yielded.
            // The HttpClient::request() method itself will never throw directly, but returns a promise.
            echo $error;
        }
});

But I'm unable to understand that, where to add the code for saving downloaded file to a specific path as I was doing in the blocking.php earlier.

As per your suggestion, I'm modifying the code be like this
Loop::run(static function () use ($items): \Generator {

        // Instantiate the HTTP client
        $client = HttpClientBuilder::buildDefault();

        $requestHandler = static function (string $uri) use ($client): \Generator {
            /** @var Response $response */
            $response = yield $client->request(new Request($uri));
            return yield $response->getBody()->buffer();
        };

        try {
            $promises = [];

            foreach ($items as $item) {
                $promises[$item['url']] = Amp\call($requestHandler, $item['url']);
            }

            $bodies = yield $promises;

            foreach ($bodies as $item => $body) {
                print $uri . " - " . \strlen($body) . " bytes" . PHP_EOL;
            }
        } catch (HttpException $error) {
            // If something goes wrong Amp will throw the exception where the promise was yielded.
            // The HttpClient::request() method itself will never throw directly, but returns a promise.
            echo $error;
        }
});

But I'm unable to understand that, where to add the code for saving downloaded file to a specific path as I was doing in the blocking.php earlier.

I'll work similar than before. You can also directly stream the response body to disk without buffering it.
Loop::run(static function () use ($items): \Generator {
        $client = HttpClientBuilder::buildDefault();

        $requestHandler = static function (string $job) use ($client): \Generator {
            /** @var Response $response */
            $response = yield $client->request(new Request($job['url']));
            yield Amp\ByteStream\pipe($response->getBody(), yield Amp\File\open($job['path'], 'w'));
        };

        try {
            $promises = [];

            foreach ($items as $item) {
                $promises[$item['url']] = Amp\call($requestHandler, $item);
            }

            yield $promises;
        } catch (HttpException $error) {
            echo $error;
        }
});

I'll work similar than before. You can also directly stream the response body to disk without buffering it.
Loop::run(static function () use ($items): \Generator {
        $client = HttpClientBuilder::buildDefault();

        $requestHandler = static function (string $job) use ($client): \Generator {
            /** @var Response $response */
            $response = yield $client->request(new Request($job['url']));
            yield Amp\ByteStream\pipe($response->getBody(), yield Amp\File\open($job['path'], 'w'));
        };

        try {
            $promises = [];

            foreach ($items as $item) {
                $promises[$item['url']] = Amp\call($requestHandler, $item);
            }

            yield $promises;
        } catch (HttpException $error) {
            echo $error;
        }
});

Wow, I'm able to download 284 files in a moment. There are some changes in the code as per the issues found.
Loop::run(static function () use ($items): \Generator {
        // Instantiate the HTTP client
        $client = HttpClientBuilder::buildDefault();

        $requestHandler = static function (Array $job) use ($client): \Generator {
            /** @var Response $response */
            $part_file = $job['part'];
            $response = yield $client->request(new Request($job['url']));
            yield ByteStream\pipe($response->getBody(), yield  File\openFile($job['path'], 'w'));
            if (file_exists($part_file)) {
                unlink($part_file);
            }
        };
        try {
            $promises = [];
            foreach ($items as $item) {
                $promises[$item['url']] = call($requestHandler, $item);
            }
            yield $promises;
        } catch (HttpException $error) {
            echo $error;
        }
});

But at the end there is an error thrown which is as follows.
Amp\Http\Client\TimeoutException: Allowed transfer timeout exceeded, took longer than 10000 ms in /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/http-client/src/Connection/Http1Connection.php:443
Stack trace:
#0 [internal function]: Amp\Http\Client\Connection\Http1Connection->Amp\Http\Client\Connection\{closure}()
#1 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Coroutine.php(118): Generator->send(NULL)
#2 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Internal/Placeholder.php(46): Amp\Coroutine->Amp\{closure}(NULL, NULL)
#3 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Internal/PrivatePromise.php(23): class@anonymous->onResolve(Object(Closure))
#4 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Internal/Placeholder.php(143): Amp\Internal\PrivatePromise->onResolve(Object(Closure))
#5 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Deferred.php(52): class@anonymous->resolve(Object(Amp\Internal\PrivatePromise))
#6 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/functions.php(282): Amp\Deferred->resolve(Object(Amp\Internal\PrivatePromise))
#7 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Internal/Placeholder.php(149): Amp\Promise\{closure}(NULL, NULL)
#8 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Deferred.php(52): class@anonymous->resolve(NULL)
#9 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/byte-stream/lib/ResourceInputStream.php(198): Amp\Deferred->resolve()
#10 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/byte-stream/lib/ResourceInputStream.php(182): Amp\ByteStream\ResourceInputStream->free()
#11 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/socket/src/ResourceSocket.php(166): Amp\ByteStream\ResourceInputStream->close()
#12 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/http-client/src/Connection/Http1Connection.php(122): Amp\Socket\ResourceSocket->close()
#13 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/functions.php(90): Amp\Http\Client\Connection\Http1Connection->close(Object(Amp\CancelledException))
#14 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/functions.php(121): Amp\call(Array, Object(Amp\CancelledException))
#15 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/CombinedCancellationToken.php(29): Amp\asyncCall(Array, Object(Amp\CancelledException))
#16 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/functions.php(90): Amp\CombinedCancellationToken->Amp\{closure}(Object(Amp\CancelledException))
#17 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/functions.php(121): Amp\call(Object(Closure), Object(Amp\CancelledException))
#18 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/CombinedCancellationToken.php(29): Amp\asyncCall(Object(Closure), Object(Amp\CancelledException))
#19 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/CancellationTokenSource.php(92): Amp\CombinedCancellationToken->Amp\{closure}(Object(Amp\CancelledException))
#20 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/CancellationTokenSource.php(77): class@anonymous->invokeCallback(Object(Closure))
#21 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/CancellationTokenSource.php(161): class@anonymous->Amp\{closure}(Object(Amp\CancelledException))
#22 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/TimeoutCancellationToken.php(30): Amp\CancellationTokenSource->cancel(Object(Amp\TimeoutException))
#23 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Loop/NativeDriver.php(111): Amp\TimeoutCancellationToken::Amp\{closure}('bnq', NULL)
#24 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Loop/Driver.php(138): Amp\Loop\NativeDriver->dispatch(true)
#25 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Loop/Driver.php(72): Amp\Loop\Driver->tick()
#26 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/vendor/amphp/amp/lib/Loop.php(95): Amp\Loop\Driver->run()
#27 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/src/Async/AsyncFactory.php(93): Amp\Loop::run(Object(Closure))
#28 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/src/Async/AsyncFactory.php(97): Blended\hostlib\Async\AsyncFactory->process(Array)
#29 /var/www/html/wp-async/wp-content/themes/blended_fw/hostlib/src/Backend.php(666): Blended\hostlib\Async\AsyncFactory->collect(Array)

I believe zombie process issues were fixed by https://github.com/amphp/process/commit/8c769ff2b3ed7f507640f0df25409ef97a3563a8 and https://github.com/amphp/process/commit/76e9495fd6818b43a20167cb11d8a67f7744ee0f some time ago.


ens3: bad gso: type: 1, size: 1432

After upgrading to Gardenlinux 1592.0 we see a lot of those error message in dmesg. Network performance is degraded.
[  140.932766] ens3: bad gso: type: 1, size: 1432
[  141.036511] ens3: bad gso: type: 1, size: 1432
[  141.079216] ens3: bad gso: type: 1, size: 1432

Probably this is related to https://bugzilla.kernel.org/show_bug.cgi?id=219129.
We run on top of OpenStack (2023.2) with virtio enabled.

Hi @berendt
thanks for reporting. If you are right that this is the mentioned issue, this should be fixed in linux 6.6.46. I don't know if that's actually the case, but this kernel should be part of our next patch release. I can't tell you when that will be available.

Hi @berendt
thanks for reporting. If you are right that this is the mentioned issue, this should be fixed in linux 6.6.46. I don't know if that's actually the case, but this kernel should be part of our next patch release. I can't tell you when that will be available.

I don't know your policy here. But from my point of view, a very timely patch release would make sense here as probably all deployments involving OpenStack clusters for shoots are actually affected. It would be nice to be able to solve the problem promptly without having to go back to 1443.10.

Yes, I understand. ping @pnpavlov

Have you tested/investigated if 6.6.46 does mitigate the issue?
We have binary packages available here https://github.com/gardenlinux/package-linux/releases/tag/6.6.46-0gardenlinux0, but no disk images (yet).

hey @berendt , nice to "see" you again, sorry for the background for the "meeting"!
Yes, we will update the kernel with #2273 . We are collecting the changes required and will work on patch next week.

Hi @berendt,
We won't be able to make a patch release today, but since the new kernel version is already in our nightly, and since the nightly version (1599 today) is still quite close to the release (1592), it would be super helpful for us if you could verify that the new kernel fixes the issue.
Here is the link to the github artefact for openstack-gardener_prod-amd64-1599.0-db367054.

The artefacts are also uploaded to S3, these should be publicly accessible:

s3://gardenlinux-github-releases/objects/openstack-gardener_prod-amd64-1599.0-db367054/
s3://gardenlinux-github-releases/objects/openstackbaremetal-gardener_prod-amd64-1599.0-db367054/
s3://gardenlinux-github-releases/objects/openstackbaremetal-gardener_prod-arm64-1599.0-db367054/

@berendt , texted you in slack k8s space, can you please double check if the nightly run that @Vincinator  shared above does solve the problem and also if the arm64 image is good? if so, we will go forward with those two changes in the patch1 next week.

We are currently testing. It looks as if the messages in the kernel log no longer occur with the test image.

@berendt Thanks for checking 👍

We are still running a few tests. But the network performance now looks normal again.

Thank you @berendt & team, we will go ahead and push that change in https://github.com/gardenlinux/gardenlinux/issues/2273. Should be published in next couple days together with couple other things.

Hi @berendt
We released 1592.1 yesterday which includes kernel 6.6.47 with the fix you described.
Thanks for the heads up and please let us know if you find any other issue.
Best,
Florian


DRAFT [LLC] Implement RequestOptions.AddPolicy

Provide AddPolicy method on RequestOptions (RequestContext for now) with the same API as AddPolicy on ClientOptions.
To include the policies added by RequestOptions, we create a new ReadOnlyMemory to hold all policies (client and request) when a protocol method with request policies is called. We use ArrayPool to pool the memory so we avoid allocations.
Fixes: https://github.com/azure/azure-sdk-for-net/issues/24907

API changes have been detected in Azure.Core. You can review API changes here
API changes
+         public void AddPolicy(HttpPipelinePolicy policy, HttpPipelinePosition position);
+         public HttpMessage CreateMessage(RequestContext context);

API changes have been detected in Azure.Core. You can review API changes here
API changes
+         public void AddPolicy(HttpPipelinePolicy policy, HttpPipelinePosition position);
+         public HttpMessage CreateMessage(RequestContext context);

Can you add a line to changelog please?


Normalize composer.json

Check composer.json normalization in CI workflow: see failing test here https://github.com/osiset/laravel-shopify/pull/561/checks?check_run_id=1040588204#step:6:5

Add support section and extra config to composer.json file and normalize the composer.json file (by running composer normalize)

Was a small conflict due to previous PR to add Laravel 8 support.
Pulled in your changes, resolved the conflict, re-ran the normalize and seems to be fine now (I hope).
Thanks!

@lucasmichot Any idea why Actions seems odd?

The diff doesn't make sense. It appears to be seeing dev requirements inside the standard requirements.
Running composer normalize --dry-run locally produces a positive result. Caching issue somewhere on Github's side maybe?

This seems like a caching issue.
But nothing to worry about.
I guess this will be resolved with a fresh PR on top of master 👍

Confirmed. I've merged in your latest PRs and fixed one minor issue, the action seems to pass now.


Dockerize Pebblo

Create a Dockerfile for pebblo to quickly run pebblo in a containerized environment.
Tasks:

[ ] Add Pebblo Dockerfile
[ ] CI: Add docker image push on release/tag

@srics :  I will  own and  start working on this Pebblo Open source contribution from my side. Kindly let me know if you want to assign this to me  ?

@akramIOT This change recently went in https://github.com/daxa-ai/pebblo/pull/277 (this issue should have been tagged). Let us identify a few other roadmap items that you can grab. Thanks for your interest in contributing!

@siddheshwar-more will update CI pipeline to push with :latest tag


Meta tracker for future work

Here's a few ideas on ways this could be extended. I probably won't work on these, as this was mostly just a fun day project, but if anyone would like to contribute here's some ideas!

[ ] Support loading from .flatpakrefs without installing (would probably download the flatpak as normal and run it straight from the repo
[ ] Support desktop integration (see: desktop-entry-daemon

Hi, interesting project. Please define what "run ... in a portable manner" actually means. Can this be used to turn a Flatpak'd application into something that can run on systems on which Flatpak is not installed?

Hi, interesting project. Please define what "run ... in a portable manner" actually means. Can this be used to turn a Flatpak'd application into something that can run on systems on which Flatpak is not installed?

Hi, thanks for the interest!
The project does require flatpak to be installed, and the runtime is not bundled with the application in this scenario. I think the idea of a fully portable application based on flatpak would be really cool, albeit with some caveats:
From what I understand, the runtimes for flatpak applications can get very large (the ones I know of are the freedesktop sdk and the GNOME and KDE sdks)
Bundling the runtimes into an image would be interesting, and I toyed with the idea for a bit, but even that would still require the flatpak application, unless we do some reverse engineering on what flatpak run does, then include it in some bootstrapping code, which is certainly possible. It'd be good to calculate the drawbacks though, the two big ones I can think of are startup times and application size.
If I had unlimited time to work on stuff like this, I'd be quite the happy programmer. Desktop integration stuff is of a lot of interest to me lately.

Thanks @ryanabx.
Hence, I'd say that the minimal set of files needed from the runtime(s) to run a particular application should be bundled with the application.
For example, the entirety of Qt is quite large (think hundreds of MB), but most applications only require a tiny subset of it (think under 20 MB).
This is the approach tools like https://github.com/probonopd/go-appimage/tree/master/src/appimagetool take. Maybe it could be used to convert Flatpaks into AppDirs.

I'm not sure what AppDirs are, but if they act similarly to ostree repositories then a very similar approach can be managed here.
A very raw but guaranteed to work approach would be to have an exclude list (like I see in the repo you shared) that users who want to package apps can use to ignore certain files that aren't needed. Basically it'd be cool to have a manifest that's like:
FROM org.inkscape.Inkscape

* include all files

---

FROM org.freedesktop.Sdk (or whatever they call it)

* exclude:
** insert a bunch of library files that aren't necessary, glob patterns can be included for an easier time **

Then the archiving program would just download the required stuff from flathub (or another remote if specified) and put it all together in one ostree repo (or two if we want to follow the flatpak way of separating runtime from app), then zip it together.
From there, a complementary program could be made to unpack the archive and run it, either using flatpak (since they have the code to mount /app and /usr already baked in) or if one would like to not use flatpak, the bootstrapping code could be embedded in the file like how appimages work.

@probonopd I'd be interested to hear about how your appimage format works sometime, particularly how you wrote your bootstrapping code to run whatever app a user executes

The principle is really simple: It's an AppDir inside a squashfs image, which is prepended by a tiny binary that loop-mounts the squashfs image (using FUSE) and then executes the payload inside the AppDir.
An AppDir is just a directory that contains everything an application needs to run (that cannot safely be assumed to be there on all target systems) plus a file (script, symlink) called AppRun that executes the main payload application.
AppImages also support --appimage-extract if you just want to unpack (like a zip) rather than mount and run the AppImage.
The rest are details. Ask me anything you're interested in.

@probonopd that's remarkably simple! I can see that being done for flatpaks without requiring flatpak to run it, but afaik the runtimes would still be necessary to an extent.
Basically flatpak has ostree repositories for runtimes and applications, ostree repos being fancy git-like repos for filesystems, they can be turned into static directories though, and then you could make an AppDir out of that, just merging the runtime and app into a single appdir, or by making an appdir with two separate folders for each.
The resulting file size might be massive though, and that's where we either cull unnecessary runtime libs or separate the runtime. I'd be fine with either approach but separating the runtime does mean that a lot of common libs are accounted for across executables. I know that might partially go against the ethos of appimages though.
I see a lot of promise in converting flatpaks to single executables, I just don't have a big-picture architecture laid out in my head. If anyone were similarly interested, I'd be down to figure something out with a group. Flatpak seems to have won over a lot of distributions and app developers, so I think it makes sense to build something off of what they have built, that accomplishes the portability goals that they won't seek out.

Yes: one would need to extract the OStree for the application and for the runtime(s) it needs. Then copy all of the application OStree into an AppDir (plain directory), and let a tool like go-appimage's appimagetool -s deploy do the rest - deploy the parts (only) of the runtime(s) that are actually needed. The resulting AppDir could then be zipped or be put into an AppImage.

Related discussion, possibly useful:

https://github.com/flatpak/flatpak/issues/408

Related discussion, possibly useful:

https://github.com/flatpak/flatpak/issues/408

Haha that was me at the end there!


RuntimeError: Broken pipe

Hello,
Thanks for your code, I really benefit a lot from it.
I have a question, I want to solve the error of "RuntimeError: Broken pipe".
But I can't find where defined the "num_workers", would you please tell me?
Thank you very much!

Hey @chenhuawei2019 ! What command leads to this error?

@okhat
Thanks for your reply!
【CUDA_VISIBLE_DEVICES="0,1,2,3"
python -m torch.distributed.launch --nproc_per_node=4 -m
colbert.train --amp --doc_maxlen 180 --mask-punctuation --bsize 32 --accum 1
--triples /path/to/MSMARCO/triples.train.small.tsv
--root /root/to/experiments/ --experiment MSMARCO-psg --similarity l2 --run msmarco.psg.l2】
I find the solution on the internet about "RuntimeError: Broken pipe". Some ways tell me to modify the "num_workers", but
I don't find where is it.

Ah okay, how many GPUs do you have? How much (CPU) memory do you have?

Ah okay, how many GPUs do you have? How much (CPU) memory do you have?

@okhat I have 4 GPUs(15g) and my CPU memory is 128g, video memory is 16g.

Hey @chenhuawei2019 did you intend to close this issue?

Feel free to re-open the issue. I am unable to see the cause of your error. Is there a longer/full error message you can share?

@okhat Thanks for your patient! This issue was caused by some other easy issues, but I didn't deal with them first so that I put this question, now  I have solved this well. Thanks for your patient again!


Broadcast proof with where clause needs explicit where to trigger

The following example doesn't verify unless you uncomment the where clause. It seems like this shouldn't be needed, since it's obvious that u32 implements PartialEq.
pub spec fn set_contains_element_opaque<T>(s: Set<T>, k: T) -> bool
    where
        T: PartialEq;

pub broadcast proof fn axiom_set_contains_element_opaque<T>(s: Set<T>, k: T)
    where
        T: PartialEq,
    ensures
        #[trigger] set_contains_element_opaque::<T>(s, k) <==> s.contains(k)
{
    admit();
}

fn test_axiom(s: Set<u32>, x: u32)
//    where u32: PartialEq,  // <== doesn't verify unless this clause is present
    requires
        set_contains_element_opaque(s, x)
    ensures
        s.contains(x)
{
    broadcast use axiom_set_contains_element_opaque;
}

Can you try it now?

Yep, it works now. Thanks!


ENTER

No idea what this command does

According to this document, the ENTER command collects input and allows you to specify the amount of time the user has to enter their answer. This explains how the game can limit how long you have to shoot.
The format is:
ENTER <#numeric variable> <time allowed (s)> <return variable> <read variable>
The numeric variable is set to the port number of the user but that's never used by the game so I'm going to ignore it.
The return variable is set to the time taken by the user.
The read variable is the string the user inputted.


Preparing v24.11

@JeromeMartinez Can the case study from BFI be included in the release?

Sorry I can try and complete that next week for you!

There should be a v24.12 with https://github.com/MediaArea/RAWcooked/pull/442 (it is not yet stable enough, still some crashes on one of the machines of the user who pushes his machine and his NAS to the limits).


Fixed a no attribute error of IkaUi.py

Ported functions of input/VideoInput to ui/VideoCapture.

Thanks!


Radio control Value property

Does Radio control have Value property? I'm not able to access it. Only Selected.Value works.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: aff9e431-4f75-a61c-bacc-fd904bf5bdde
Version Independent ID: 3f88278f-a036-882f-f520-853ebca1bbff
Content: Radio control: reference - PowerApps
Content Source: powerapps-docs/maker/canvas-apps/controls/control-radio.md
Service: powerapps
GitHub Login: @fikaradz
Microsoft Alias: fikaradz

That is correct, you can only get the value of the selected item via Selected.Value

@AlexZhidkov Thanks for your feedback. If we haven't answered your question, please reopen this issue, and specify more details about what you're looking for. Cheers!
cc: @fikaradz


[BUG] Valve Index Knuckles hand tracking is busted

to navigate menus, you have to point at a button with your headset and click with one of the triggers.
items in your hand render at the top center of your viewport, and you can't move them with the Knuckles controllers.
managed to get this cute little screenshot of the bug in action.

what... vr system are you using? According to a quick google, valve index? But I wanna confirm that before I say anything
As for the UIs, they're attached to your arms though, so they're there, it's just they're kinda inside of your face

steamvr

i'm running SteamVR, and yes, indeed using the valve index

progress seems to have been made! hand tracking... exists now!
the tracked hand is stuck outside the chaperone range given by steamvr tho
and locked in one place
looks like a debug thing for now, but it's good to see something happening


Custom Row/Footer

Question:
Is it possible to add a custom footer to a datatable? At the bottom of a datatable I need to add a row with "Totals" that needs to be fixed (always be visible on every page even when pagination is on), but not be filterable/sortable. I've tried creating a separate datatable component below the main table (and styling it to look as part of the main table), but there is a scrolling issue in responsive: when the main table is scrolled horizontally, last row is not scrolled with it - so not a good UI experience. Any suggestions on how it can be achieved?
This is what I need:

Thanks!

Why not create a special row added at the end of data in which you will add some computation, maybe add some style to it also

@jkeruzec, that was my first solution and until a user starts interacting with the table, it looks as per the mockup. However, with this solution, when a user wants to filter or sort the table, it fails. Ex: a user wants to filter by "Site" -> in the dropdown of filters by site there will now be 3 options: "Red Apple", "The Bargain Shop" and... "Total", which makes no sense. Sorting columns presents similar issues. So I need to either disable  sorting and filtering in this table or make my totals a separate table all together (in this case, as I outlined above, I run into scrolling issue - two tables scroll separately on mobile).
Thank you for the suggestion though!

I have a PR that should be merged shortly after some further testing that will allow you to override the current table footer (pagination) with a custom one.  This will obviously overwrite the pagination but should allow you to do anything.

That's awesome! I'm looking forward to using this enhancement in my project. Thank you!

Ok published 2.0.0-beta-18 give it a shot

@usotnyk DId it work out for you?

@gregnb, I did not have a chance to try it yet. We found a temporary work around for the existing datatable I had (disabled filtering and sorting) for the time being. I will however, use your enhancements for a few other tables I will start building next.

Ok no worries. I will close the issue for now. Thanks

@gregnb is it possible to access the table data from the footer? Because it is declared in the options,and it seems I dont have the rowData/tableMeta yet, i can't show/hide footer cells in correlation with their column indices, or take advantage of the responsive styles. My needs are exactly the same as usotnyk, essentially a fixed, non sortable, non filteralbe rowset, but that has the context of the current table state.

I'm looking to do just this -- the footer row holding all the totals
The jquery datatable has this option (and the big thing is that it scrolls with the table if it overflows X) and it seems like this custom footer option might require a lot of custom coding in order to achieve the same

Hi! I started to fork this repo to get this feature. First idea was to get only the totals, but I decided to develop it further by adding d3.js to get more detailed stats. Also stats to select are optional for every column. There is still som todo's e.g checking if some stats are valid for string columns.
You can find preview video from this link.
https://www.youtube.com/watch?v=3XefVyYv784
I will PR this feature as finished. I had to modify some core features to be able to pass data and column width references to stats footer.

@jukakoski That's incredible, just what I need. Do you have code you are willing to share?