Upgrade mbedtls to 3.4.0

Overview
PR #213 introduces some breaking changes:

Update vendor mbedtls code from version 2.28.3 to a 3.4.0

Please checkout C mbedtls migration guide for 3.X here 3.0-migration-guide.md to

Update the build code for mbedtls-sys-auto crate to sync up with vendor code change:

Changes in C DEFINE's for configuration
Changes in header files
Add binding code generation for static inline code in C side
Removing deprecated cargo features or dependencies
Add a cargo feature for TLS 1.3

Update rust wrapper code in ./mbedtls to sync up the changes in  mbedtls-sys-auto

Removing deprecated cargo features or dependencies
Update rust code to sync up API changes
Add types and functions for TLS 1.3
Add a cargo feature for TLS 1.3
Update integration tests for new API and TLS 1.3
Update dependencies

Breaking Changes:
Note: entries with :boom: are ensured that they will break the downstream.
Changes in mbedtls-sys-auto
Vendor code changes
Upstream changes:

:boom: Upgrade vendor mbeldtls code to 3.4.0

Changes on our side:

Cherry picked previous changes in old versions:

commit: vendor change: Adding mpi_force_c_code feature

New changes

vendor change: fix time call in tls13 client&server : This has been merged into upstream, see: https://github.com/Mbed-TLS/mbedtls/pull/7639 .

rust code changes
Features:

:boom: zlib is removed: support for TLS record-level compression is removed in mbedtls 3.X

Related C DEFINE MBEDTLS_ZLIB_SUPPORT is also removed

:boom: legacy_protocols is removed: all protocols early than TLS 1.2 is removed in mbedtls 3.X

Related C DEFINE's are also removed: MBEDTLS_SSL_PROTO_SSL3, MBEDTLS_SSL_PROTO_TLS1, MBEDTLS_SSL_PROTO_TLS1_1, MBEDTLS_SSL_CBC_RECORD_SPLITTING

:boom: pkcs11 is removed: wrapper for libpkcs11-helper is removed in mbedtls 3.X, see 3.0-migration-guide.md
:boom: pkcs12 is removed: because https://github.com/fortanix/rust-mbedtls/pull/269 pkcs12 still defined since it is needed for reading some pkcs8 format keys
Put TLS 1.3 behind a feature tls13 : because the dependency of TLS 1.3 in mbedtls 3.X are using a global state RNG which breaks the requirements for FIPS, so this feature enables use to avoid these code from compilation.
:boom: Deprecated features  custom_threading , custom_time , custom_gmtime_r , pthread  are removed

Dependencies:

:boom: Bump mbedtls-sys version to 3.4.0
:boom: Deprecated dependencies are removed

libz-sys : support for TLS record-level compression is removed in mbedtls 3.X
libc: libc is not needed in sgx

Build code changes:
Following changes are made according to 3.0-migration-guide.md.

Remove MBEDTLS_CONFIG_H in mbedtls_config.h
Remove #include <mbedtls/check_config.h>
mbedtls-sys/build/bindgen.rs

Allow bindgen to generate bindings for functions, types and variables start with psa_, and put them in a sub mod psa because they are needed by TLS 1.3
Use  bindgen experiment feature to generate C function wrapper for C static inline functions

mbedtls-sys/build/headers.rs : Update header files
mbedtls-sys/build/config.rs : Remove/add C defines

:boom: Added MBEDTLS_PSA_CRYPTO_EXTERNAL_RNG : this is added because TLS 1.3 need to use PSA library which need this when feature std is off, user need to provide their implementation. In rus-mbedtls, one implementation is proveded, see: mbedtls/src/rng/mod.rs.

Changes in mbedtls-platform-support

Added once_cell for initializing PSA only one times: see fn psa_crypto_init() in mbedtls-platform-support/src/lib.rs, this is needed because:

MBEDTLS_USE_PSA_CRYPTO means that X.509 and TLS will use PSA Crypto as much as possible (that is, everywhere except for features that are not supported by PSA Crypto, see "Internal Changes" below for a complete list of exceptions). When it is enabled, you need to call psa_crypto_init() before calling any function from PK, X.509 or TLS;

Ref: https://github.com/Mbed-TLS/mbedtls/blob/0b3de6fcec4aa4b23a9ee1e076714cbc796f3ac4/docs/use-psa-crypto.md#general-considerations

Add function pointer mbedtls_psa_external_get_random  which is needed when C DEFINE option MBEDTLS_PSA_CRYPTO_EXTERNAL_RNG is turned on. This will be used in case when system default RNG or entropy is no available (for example in SGX)

Using function pointer here ensure there is no link time conflict in future when mutiple rust-mbedtls is using this crate.
User need to call function set_psa_external_rng_callback before using any PSA functions or TLS 1.3

Add rust implementation of explicit_bzero, which is needed in SGX. Because in SGX, our rs-libc does not support this function.
Update self tests:

following are removed:

arc4_self_test
md2_self_test
md4_self_test
x509_self_test
xtea_self_test

following are added:

sha384_self_test
chacha20_self_test
chachapoly_self_test
poly1305_self_test
sha224_self_test

Changes in mbedtls
Features:

:boom: zlib, legacy_protocols removed to sync with changes in mbedtls-sys
:boom: pkcs12 and pkcs12_rc2 are removed: see https://github.com/fortanix/rust-mbedtls/pull/269
Put TLS 1.3 logic behind a feature tls13: check reason above

Dependencies:

:boom: Bump rust-mbedtls version to 0.11.0
:boom: Bump dependency mbedtls-sys-auto version to 3.4.0
Added rstest rstest_reuse lazy_static env_logger: used for improving code of tests

Code changes

Function EcPoint::mul now need to pass in a RNG to ensure blinding.
Add prefix private_ to some fields of mbedtls types under mbedtls/src/cipher/raw
:boom: Removed and added some options in Error, CipherType, CipherMode and CipherSuite  to sync with changes in mbedtls 3.X
:boom: mbedtls/src/pk/ec.rs : User need to provide a RNG callback for function EcPoint::mul, this originally is not a hard requirement, but in C mbedtls 3.X this become a hard requirement for blinding to defend side channel attack.
:boom: mbedtls/hash :

Md2 and Md4 are removed since they are no longer supported in mbedtls 3.X
fn pbkdf_pkcs12 is removed since pkcs12 is removed

:boom: mbedtls/pk/mod.rs :

Remove CustomPk
User need to provide a RNG callback for Pk::from_private_key, this originally is not a hard requirement, but in C mbedtls 3.X this become a hard requirement for blinding to defend side channel attack.

:boom: mbedtls/src/ssl/ciphersuites.rs: Rename TlsCipherSuite to Tls12CipherSuite, and add enum: Tls13CipherSuite, IanaTlsNamedGroup, TLS13SignatureAlgorithms: these are introduced by TLS 1.3
mbedtls/src/ssl/ssl_states.rs: Add SslStates to represent the state of SSL handshake
Update tests accordingly

Special code need to notice:
In impl of std::io::Read under mbedtls/src/ssl/io.rs and tokio::io::AsyncRead under mbedtls/src/ssl/async_io.rs, there are some code to handle the special case when using mbedtls as client to connect to a server whose session ticket extension is enabled.
This case is found when connecting to goolge.com, where Google's server send the session ticket after the completion of handshake, which cause C-mbedtls throw errors when client is try to read msg data.
CI changes

Use cargo-nextest to run tests

Reduce time to run tests
Have ability to run some tests in serial

tests under hyper.rs need to access to google.com which has QPS limit
some tests function use some system resource, see https://github.com/fortanix/rust-mbedtls/blob/yx/upgrade-mbedtls/mbedtls/tests/support/net.rs

With some workflows it can be inconvenient when a non-empty source file does not finish with a newline character. For example, an editor might add the newline character and cause a spurious diff. So it would be nice to add a final newline character to these files:
.config/nextest.toml
3.X-migration-guide.md
Cargo.toml
ct_locally.sh
mbedtls-platform-support/src/psa_external_rng.rs
mbedtls/src/error.rs
mbedtls/src/rng/mod.rs
mbedtls/src/ssl/mod.rs
mbedtls/tests/support/debug.rs

This symbolic link was presumably added by accident:
mbedtls-sys/vendor/build/library/ssl_debug_helpers_generated.c

note to self: reviewed platform and sys changes

About comments about newly add RNG arguments for mul from_private_key private_from_ec_components , I think:
in mbedtls , the changes are fine because they just represent the API change from upstream and current PR will be a major version bump, so it's OK to have such change.
Some better comments about these public API are added to explain this clearly.

bors cancel

bors retry

bors try

bors cancel

bors cancel

bors try-

bors try-

bors cancel

bors try

Added back pkcs12 C define because this is needed for importing RSA 2048 Pkcs8 encrypted key with pbeWithSHAAnd3-KeyTripleDES-CBC, see test:
https://github.com/fortanix/rust-mbedtls/blob/8538c469d7679071840fc460aa56eb5bb5759d72/mbedtls/src/pk/mod.rs#L1197-L1231

bors r+

bors cancel

bors merge


"use strict"; -- document how to convince biomejs that a file is not a module?

Documentation URL
https://biomejs.dev/linter/rules/no-redundant-use-strict/
Description
the noRedundantUseStrict shows removing redundant strict -- and the biome playground allows for both "Module" and "Script" forms -- but I can't find how to configure that from the cli and so I am always getting a diagnostic for removal of 'use strict';
Expectations
it may be nice to show how to teach biome that these are scripts and not modules -- perhaps in the noRedundantUseStrict docs as well since that's where it is relevant to me!
Code of Conduct

[X] I agree to follow Biome's Code of Conduct

Can you send a PR @asottile ?

but I can't find how to configure that from the cli

I can if I first learn how to do it

At the time, the only way I know is using .cjs as fiile extension.

@asottile We have extensive documents:

https://github.com/biomejs/biome/blob/main/CONTRIBUTING.md
https://github.com/biomejs/biome/blob/main/crates/biome_analyze/CONTRIBUTING.md#document-the-rule


Details of the standard objects (especially tables) in Visual Studio Code

When you jump into a standard object with F12 in Visual Studio code, it has little detail. If you use the Customer table, for example, the fields do not display whether the fields are FlowField or FlowFilter. This can be a bit problematic because it's so easy to forget to call the CALCFIELDS function. Would it be possible to display the standard objects in more detail when navigating with F12? That would simplify the work considerably.

Makes sense. I will add it to the backlog.

Isn't this resolved now? F12 should load a .dal file with all page properties/code present.

True, that is to be implemented with the latest version already. Can therefore be closed from my point of view, which is what I do with this.


ci: merge staging to master

This is an automatic PR generated by the pipeline CI/CD. This will be automatically fast-forward merged if successful.

Pipeline Attempt on<PHONE_NUMBER> for 1f2a2cabcf0a9ec45d776deee70a968c65b8da50
https://gitlab.com/MatrixAI/open-source/js-mdns/-/pipelines/1045627383

Pipeline Succeeded on<PHONE_NUMBER> for 1f2a2cabcf0a9ec45d776deee70a968c65b8da50
https://gitlab.com/MatrixAI/open-source/js-mdns/-/pipelines/1045627383


Content assist proposes names of indexes etc instead of tables on PostgreSQL

Content assist works in a strange way when trying to complete a regular select * from on Postgresql.
When I write a query and hit ctrl+space before typing the name of the table - it works correctly:

but when I begin to type the name of the table and hit ctrl+space I get a list of indexes, not even from my active schema:

I would expect to get a list of tables that start with "adas" or just get the whole table name filled out automatically if there is only one such table.
I'm working with PostgreSQL 9.0 and DBeaver 3.5.9 but I've seen this behaviour in earlier versions too.

Could you try DBeaver 3.6.0. We did a major update of PostgreSQL driver there.

on it.

I see that you've added a feature that makes this silent assumption that a postgres schema is created per user like in oracle.
I am unable to make a schema of my choice active and now the content assist gives me all tables that are present in the existing schemas on the database server.
Could you please reconsider allowing a user to activate a single schema? I know that there is a filter mechanism for schemas but In my case I often move from schema to schema - changing filters each time would be extremely painful.
As for the content assist. When I give no name for the table the content assist gives me a list of all schemas on the server so that I choose one first:

and if I start typing the name of the table it will list all tables with that name in all schemas:

Notice that although I've filtered out all but one schema - the content assist doesn't pick up on that.
So in a way of speaking it's a bit worse than it used to be :-(

I'm with you @norbitheeviljester, once you have a lot of databases, content assist will bring all tables sorted by schema name. I think that would be nice with when I have one schema selected, content assist show me only content of selected one.

Unfortunately PostgreSQL doesn't has such thing as "default schema". But it has "search path" session variable. I'll try to use it to narrow content assistant search.
Thanks for report.

Hi @serge-rider , it seems on dbeaver 3.6.7 this has been reverted. When using content assist I see tables from all schemas, not just the active one.

Is this a bug or have you decided to drop this functionality?

also content assist shows the list of schemas instead of tables:

This is on Postgresql 9.5

Yep. Side effect of another feature request.

It'll be fixed in 3.6.8, thanks for report.
Please next time create a separate issue. Its hard to track different problems in a single issue.

I'll make sure to do that, thanks.


Question: metrics exporter for monitoring and visibility for users

As a user and/or cluster administrator and/or front-end UI developer, I would like a mechanism to easily extract metrics for answering questions such as the following

Where is my job in the queue(s)? Is my job being properly prioritized?
What is the ordering of all the jobs in all queues?
What is MCAD's breakdown of the current resource availability?
How many jobs have been dispatched to date? What is the total job throughput?

The MCAD logs are helpful, but exporting relevant metrics that e.g. a Dashboard could leverage would be better. Is this already done?

I believe this important issue and certainly could be developed as a separate monitoring component for MCAD. requirement #3 should be doable. we need a new public API that exposes the point-in-time stats. The bigger question is where should we expose such metrics, please feel free to share a design document.

some discussion from here @asm582
https://ibm-research.slack.com/archives/C03M52HMHKQ/p1667964123130559
@sseelam
I know we were having this discussion around if there is an easy way to figure out what jobs were run, how many gpus, how long they took. how many active/pending/running. and what we were discussing is that, one way to go about this would be to ask k8s MCAD controller using the k8s cli, how many mcad appwrapper jobs are there and in what state etc.
@asmalvan
@alaa.youssef
can correct me on what might be the right commands to extract this information and if all these can be extratcted, but here is the list that I think we can:
how many jobs have run, are running, or are waiting in different states (broadly something like k8s get appwrappers -o wide)
number of gpus for each job that has run, running or waiting in queue. I think same as above but with some more jq . k8s get apprwrappers -o yaml | jq
how long the jobs have been running or spend time in the queue. I think again a combination of get appwrappers + some jq
I think some of this information is already exposed and some might need to be surfaced over to the k8s status object, but that might be the easiest way to get all this information.
we can technically put all these commands on a guidebook and easily surface them via codeflare cli as well.

broadly, at a very very core level metrics that we can surface as a part of events/status I think we should do that.
additionally, I think the MCAD operator can expose the /metrics endpoint for some of these metrics for prometheus to consume.
I will leave it up to you to decide which path is easier/fastest and which metrics gets captured where . I do believe that some of the things that are being asked for are possibly already being exposed via status/events

metrics exporter > prometheus > Dashboard (which one?)  sounds like the way to a richer dashboard.
In the same time, as much as we can surface regarding the user jobs' status using oc commands wrapped in codeflare sdk will be very useful from within notebook/cli

There is a world view and then there is a user view. I think a fast path to metrics is user view, we can use commands like :

kubectl get appwrapper  -n
Kubectl describe appwrapper  -n  and see status block. In status block, all conditions and states are reported. pending and running pods details appear as the scheduler creates pods and binds them

@MEllis-github closing this issue since we have parallel efforts on the dashboard and logging front that will expose metrics to different users. please feel free to re-open.


waitSafely does not work as expected

So, I have a small DeferredAsyncTask
public DeferredAsyncTask<Void, Integer, WebCall> getRequestTask(WebCall webcall) {
        return new DeferredAsyncTask<Void, Integer, WebCall>() {
            @Override
            protected WebCall doInBackgroundSafe(Void... voids) throws Exception {
                 //do stuff
                return webcall;
            }
        };
    }

that is called in a small flow
try {
    DeferredAsyncTask<Void, Integer, WebCall> requestTask = http.getRequestTask(webCall);
    new AndroidDeferredManager().when(requestTask).done(new DoneCallback<WebCall>() {
        @Override
        public void onDone(WebCall result) {
            value.set(result.getResponsePayload());
            Log.i(Constants.LOG_TAG, "webcall - result " + result.getResponsePayload());
        }
    }).fail(new FailCallback<Throwable>() {
        @Override
        public void onFail(final Throwable tr) {
            Log.i(Constants.LOG_TAG, "webcall - " + tr.getMessage());
        }
    }).waitSafely();
} catch (Exception e) {
    e.printStackTrace();
}

When using waitSafely() it just hangs. Nothing is written to the logcat, program execution is halted. Implementation seem to be fine according to the documentation. Any ideas?

@mplogas which sdk version are you using?

Android: SDK 21, build tools 21.1.2
Java SDK: 1.8u25

You're deadlocking the application by waiting on the UI thread, and this waiting prohibits the done or fail methods to get called on that thread. I just wonder why no ANR shows up.

eak yeah.. sorry it was there mostly for testing purposes and when the deferred tasks were executing on a different thread. waitSafely should probably be removed from AndroidDeferredManager, or warn that it shouldn't be used.


Added catsu.txt

Catanduanes State University
WEBSITE: https://catsu.edu.ph
IT COURSES: https://catsu.edu.ph/col_CICT.php

@isaacdarcilla First stage of verification passed, official domain is facebook.com as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://www.facebook.com/CatanduanesStateUniversity2012/posts/catsu-hands-over-ppes-to-cmsthe-catanduanes-state-university-catsu-through-suc-p/2158533770951261/

@isaacdarcilla Regrettably, this request has signs of fraud. I have to close it without merging.

Lol


Terraform demo project deployment issue for the release pipeline.

Received this error message when deploying the Terraform project into my DevOps environment:
"Error while creating release definition: Tasks with versions 'Replace tokens in terraform file:5.*' are not valid for deploy job 'Agent job' in stage Dev."
The release pipeline definition was not created and no additional information was provided.

Please install the replace tokens extension from the marketplace.
If you are running Demo Generator on a brand new organization, please enable the Third-party application access via OAuth under  Organization Setting >> Policies


Added  tag to redirect singup/login pages and updated css

Fixes #43
Hi,
I added  tag to redirect sign up and login pages and assumed the first icon is signup, the second one is login. Also, I applied nav-links css class which is more clear vs user css class.
Let me know if there's something that needs to be changed. Thanks!
p.s Prettier is automatically applied to the index.html file

Can you please review the PR? Thanks, and let me know if there's something to be fixed.

@seog-jun still it has some conflict. Please look into it.

@samratghosh291 Fixed again


fix: frontend build

Add a console.log for successful API keys in /auth route
Let the ApolloWrapper use the OSO_API_KEY if it finds it

Closing this without merge. Didn't fix this bug
https://github.com/opensource-observer/oso/issues/1285


azure-core.CurlTransportOptions.httpsDefault test failing with a TransportException with description "Timeout waiting for socket to read." followed by a segfault

From unrelated PR: https://github.com/Azure/azure-sdk-for-cpp/pull/5474 on Validate Ubuntu22_included_release
https://dev.azure.com/azure-sdk/public/_build/results?buildId=3665652&view=logs&j=ba3dad2b-ce90-52d5-0c84-4172d8b9d063&t=6c6e0346-5c6c-5b96-a9d2-deaa2096ecd7&l=22207
2024-04-03T23:37:12.1528622Z test 11
2024-04-03T23:37:12.1529266Z         Start  11: azure-core.CurlTransportOptions.httpsDefault
2024-04-03T23:37:12.1529536Z
2024-04-03T23:37:12.1530132Z 11: Test command: /mnt/vss/_work/1/s/build/sdk/core/azure-core/test/ut/azure-core-test "--gtest_filter=CurlTransportOptions.httpsDefault" "--gtest_also_run_disabled_tests"
2024-04-03T23:37:12.1530985Z 11: Working Directory: /mnt/vss/_work/1/s/build/sdk/core/azure-core/test/ut
2024-04-03T23:37:12.1531417Z 11: Test timeout computed to be: 10000000
2024-04-03T23:37:12.1572293Z 11: Note: Google Test filter = CurlTransportOptions.httpsDefault
2024-04-03T23:37:12.1572837Z 11: [==========] Running 1 test from 1 test suite.
2024-04-03T23:37:12.1573381Z 11: [----------] Global test environment set-up.
2024-04-03T23:37:12.1573877Z 11: [----------] 1 test from CurlTransportOptions
2024-04-03T23:37:12.1574202Z 11: [ RUN      ] CurlTransportOptions.httpsDefault
2024-04-03T23:37:12.1574765Z 11: [2024-04-03T23:37:12.1570284Z T:<PHONE_NUMBER>02528] DEBUG : [CURL Transport Adapter]: Creating a new session.
2024-04-03T23:37:12.1576608Z 11: [2024-04-03T23:37:12.1570428Z T:<PHONE_NUMBER>02528] DEBUG : [CURL Transport Adapter]: Spawn new connection.
2024-04-03T23:37:12.3681406Z 11: [2024-04-03T23:37:12.3678685Z T:<PHONE_NUMBER>02528] DEBUG : [CURL Transport Adapter]: No Host in request headers. Adding it
2024-04-03T23:37:12.3682030Z 11: [2024-04-03T23:37:12.3678897Z T:<PHONE_NUMBER>02528] DEBUG : [CURL Transport Adapter]: Send request without payload
2024-04-03T23:37:12.3682434Z 11: [2024-04-03T23:37:12.3679720Z T:<PHONE_NUMBER>02528] DEBUG : [CURL Transport Adapter]: Parse server response
2024-04-03T23:38:12.4373255Z 11: /mnt/vss/_work/1/s/sdk/core/azure-core/test/ut/curl_options_test.cpp:316: Failure
2024-04-03T23:38:12.4374211Z 11: Expected: response = pipeline.Send(request, Azure::Core::Context::ApplicationContext) doesn't throw an exception.
2024-04-03T23:38:12.4374813Z 11:   Actual: it throws Azure::Core::Http::TransportException with description "Timeout waiting for socket to read.".
2024-04-03T23:38:12.7329984Z  11/533 Test  #11: azure-core.CurlTransportOptions.httpsDefault ......................................................***Exception: SegFault 60.58 sec

cc @LarryOsterman, @antkmsft - this issue seems new

It appears to be a timeout on a network read. It's kinda a thing which happens when talking to live resources.

this is a timeout, most likely sone infra issue, since it passes everywhere else , the request in the test is nothing magical or fancy. since this did not fail before or since, i suggest we close for now and reopen if it starts to do it on a more frequent basis.


istio/tools: cpu and memory statistics for max and average should be collected properly from prometheus

Istio Perf tests should collect cpu and memory per proxy (client and server) for their max and average usage.

also need to investigate why recent perf test csv files are missing cpu/mem data.


button not losing active state

https://github.com/user-attachments/assets/ebe7f226-e857-4190-956e-879d6750462a
I have changed nothing in the component but the active affect will not go away

@ImamJanjua Can you provide a minimal reproduction repo?
I am unable to reproduce your issue.
https://github.com/user-attachments/assets/a3a201fa-5363-4c0e-b6cd-e7d4d9e946ca

@mrzachnugent nevermind  just needed to clean the cache. But could u maybe response to the issue related to toast example

@mrzachnugent Now I am very sure the issue is related to the component. Check out the clip. So as u can see here I started the app with cleaned cache. So button works fine, until I added just to call the func on press where i get many error and this press affect
https://github.com/user-attachments/assets/a300f943-893a-48ee-ae46-cff1cc8b5141


RMSE/AUE error calculation

There are different ways how the error on the RMSE, AUE is calculated in this freeenergyframework and pmx. The freeenergyframework method
draws bootstrap samples from the original sample, which does not account
for errors of the sample. The pmx method takes the datapoints from the original sample, and draws
the bootstrap sample's data point by drawing from a normal distribution
with center and standard deviation of the original sample's data point. Which is the correct way?
See the example below:
import numpy as np
from sklearn import metrics
import scipy

Example data
Two sets of delta G values with associated errors
# fake data
DG1  = np.random.random(100) * 10.0 - 5.0
dDG1 = np.random.normal(0, 1, size = 100)
DG2  = DG1 + np.random.normal(0, 2, size = 100)
dDG2 = np.random.normal(0, 1, size = 100)

from matplotlib import pyplot as plt
plt.figure(figsize=(6,6))
plt.errorbar(DG1, DG2, xerr=dDG1, yerr=dDG2, ls='', marker='o')
plt.xlabel('DG1')
plt.ylabel('DG2')
plt.xlim(-7,7)
plt.ylim(-7,7)

(-7, 7)

freeenergyframework method
Take bootstrap samples from the actual sample, which does not account for errors of the sample
nbootstrap = 1000
ci = 0.95
assert len(DG1) == len(DG2)
sample_size = len(DG1)
s_n = np.zeros([nbootstrap], np.float64) # s_n[n] is the statistic computed for bootstrap sample n
for replicate in range(nbootstrap):
    indices = np.random.choice(np.arange(sample_size), size=[sample_size], replace=True)
    DG1_sample, DG2_sample = DG1[indices], DG2[indices]
    s_n[replicate] = np.sqrt(metrics.mean_squared_error(DG1_sample, DG2_sample))

rmse_stats = dict()
rmse_stats['mle'] = np.sqrt(metrics.mean_squared_error(DG1, DG2))
rmse_stats['stderr'] = np.std(s_n)
rmse_stats['mean'] = np.mean(s_n)
# TODO: Is there a canned method to do this?
s_n = np.sort(s_n)
low_frac = (1.0-ci)/2.0
high_frac = 1.0 - low_frac
rmse_stats['low'] = s_n[int(np.floor(nbootstrap*low_frac))]
rmse_stats['high'] = s_n[int(np.ceil(nbootstrap*high_frac))]
print(rmse_stats)

{'mle': 1.852492378187966, 'stderr': 0.12139385458708932, 'mean': 1.8450353834007485, 'low': 1.601269239221521, 'high': 2.0890130880257103}

The pmx method:
Take the original sample and draw each bootstrap sample's data point from a normal distribution with the center and standard devivation of the original sample's data point.
nbootstrap = 1000
ci = 0.95
assert len(DG1) == len(DG2)
sample_size = len(DG1)
s_n = np.zeros([nbootstrap], np.float64) # s_n[n] is the statistic computed for bootstrap sample n
for replicate in range(nbootstrap):
    DG1_sample = np.zeros_like(DG1)
    DG2_sample = np.zeros_like(DG2)
    for i in range(sample_size):
        DG1_sample[i] = np.random.normal(loc=DG1[i], scale=np.fabs(dDG1[i]), size=1)
        DG2_sample[i] = np.random.normal(loc=DG2[i], scale=np.fabs(dDG2[i]), size=1)
    s_n[replicate] = np.sqrt(metrics.mean_squared_error(DG1_sample, DG2_sample))

rmse_stats = dict()
rmse_stats['mle'] = np.sqrt(metrics.mean_squared_error(DG1, DG2))
rmse_stats['stderr'] = np.std(s_n)
rmse_stats['mean'] = np.mean(s_n)
# TODO: Is there a canned method to do this?
s_n = np.sort(s_n)
low_frac = (1.0-ci)/2.0
high_frac = 1.0 - low_frac
rmse_stats['low'] = s_n[int(np.floor(nbootstrap*low_frac))]
rmse_stats['high'] = s_n[int(np.ceil(nbootstrap*high_frac))]
print(rmse_stats)

{'mle': 1.852492378187966, 'stderr': 0.13816081347307912, 'mean': 2.3424216885058007, 'low': 2.091178479342661, 'high': 2.6185886859019063}

The estimate ('mle') is obviously the same for both methods and the standard
deviation ('stderr') is quite similar. However, the confidence interval
differs a lot. For the pmx method, the confidence interval has a
similar range, but the mean, lower and upper bounds are much higher. The
estimate ('mle') lies not even in the confidence interval.
Do both
Bootstrap on datapoints AND errors
nbootstrap = 1000
ci = 0.95
assert len(DG1) == len(DG2)
sample_size = len(DG1)
s_n = np.zeros([nbootstrap], np.float64) # s_n[n] is the statistic computed for bootstrap sample n
for replicate in range(nbootstrap):
    DG1_sample = np.zeros_like(DG1)
    DG2_sample = np.zeros_like(DG2)
    for i in np.random.choice(np.arange(sample_size), size=[sample_size], replace=True):
        DG1_sample[i] = np.random.normal(loc=DG1[i], scale=np.fabs(dDG1[i]), size=1)
        DG2_sample[i] = np.random.normal(loc=DG2[i], scale=np.fabs(dDG2[i]), size=1)
    s_n[replicate] = np.sqrt(metrics.mean_squared_error(DG1_sample, DG2_sample))

rmse_stats = dict()
rmse_stats['mle'] = np.sqrt(metrics.mean_squared_error(DG1, DG2))
rmse_stats['stderr'] = np.std(s_n)
rmse_stats['mean'] = np.mean(s_n)
# TODO: Is there a canned method to do this?
s_n = np.sort(s_n)
low_frac = (1.0-ci)/2.0
high_frac = 1.0 - low_frac
rmse_stats['low'] = s_n[int(np.floor(nbootstrap*low_frac))]
rmse_stats['high'] = s_n[int(np.ceil(nbootstrap*high_frac))]
print(rmse_stats)

{'mle': 1.852492378187966, 'stderr': 0.1628450105340105, 'mean': 1.8693378181772506, 'low': 1.5683291549599805, 'high': 2.2080338926207124}

We should do both!


Install opts notest

HWY_DLLEXPORT doesn't do what we thought it does, it is only defined if we are building the package.


Clean up 1.4 telemetry v2 file

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[ ] Does not have any changes that may affect Istio users.

@bianpengyuan: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
a8bdaef2a97d4dfc4fb9375865e66120b73f537c
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


chore: minor typo for function's description

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@next-major dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Huge performance drop in MCM v3

I tried to port my mod from MBOptionScreen v2.0.10 to MCM.Integrated v3.1.0, and the performance drops badly. Both are using attributes api v2.
I ran a performance profiler of CPU in Visual Studio comparing both versions:
MBOptionScreen is taking no more than 0.05% of CPU time;
MCM is using more than 30%. DefaultDropdown::ctor is using more than 20% alone (and there is only a single instance of DefaultDropdown is my settings class). GlobalSettings::get_instance is using 7%, and AttributeGlobalSettings::ctor is using 2.46%.
It looks like AttributeGlobalSettings.Instance is not working correctly, that everytime it is called it creates and returns a new instance, otherwise the ctors should only be called once and not eating so much CPU time.

The reason why the constructor is called every time .Instance is called is because it need the Id property.
I could cache the instance so it won't be created every time.
The 'workaround' will be available in 3.1.2

I find that keeping a static reference to the instance can overcome this issue.
Instead of calling Settings.Instance every time, one could assign the instance to a static member once, and use that every time.
It is up to you whether the 'workaround' is really necessary, or just clarify Settings.Instance is expensive and one should keep his own reference to the instance.


Security and protection

How can i control - prevent - secure the mail server of sending blast mails ?

Is there any way to check for "attacks" to mail server ? For example multiple tries for login or DoS

Try fail2ban for secure up login

Yes, i have fail2ban enabled. It's ok with that.

You should setup your ptr, dkim and spf correctly.
https://github.com/tomav/docker-mailserver/wiki/Configure-DKIM
https://github.com/tomav/docker-mailserver/wiki/Configure-SPF


gawk: 4.2.1 -> 5.0.1

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/gawk/versions.
meta.description for gawk is: '"GNU implementation of the Awk programming language"'.

Checks done (click to expand)

built on NixOS
/bin/sh passed the binary check.
1 of 1 passed binary check by having a zero exit code.
0 of 1 passed binary check by having the new version present in output.
found 5.0.0 with grep in /nix/store/bhrdx0n2afrmr0xf194am9qbg57sw1jh-gawk-5.0.0
directory tree listing: https://gist.github.com/ee9ae1533fa01bbc0755ec61b956c68e
du listing: https://gist.github.com/d83d21aeab87bfd8c84cca39b72169d9

Rebuild report (if merged into master) (click to expand)

94296 total rebuild path(s)
27138 package rebuild(s)
26983 x86_64-linux rebuild(s)
24809 i686-linux rebuild(s)
18582 x86_64-darwin rebuild(s)
23922 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
AMB-plugins
AgdaSheaves
AgdaStdlib
CoinMP
DisnixWebService
EBTKS
EmptyEpsilon
FIL-plugins
Fabric
LASzip
Literate
MMA
NSPlist
OVMF
OVMF-CSM
PlistCpp
QuadProgpp
R
SDL
SDL2
SDL2_gfx
SDL2_image
SDL2_mixer
SDL2_net
SDL2_ttf
SDL_gfx
SDL_image
SDL_mixer
SDL_net
SDL_sixel
SDL_sound
SDL_stretch
SDL_ttf
SkypeExport
TotalParserCombinators
Xaw3d
_0x0
_1password
_2048-in-terminal
_20kly
_2bwm
_389-ds-base
_90secondportraits
_915resolution
_9pfs
a2jmidid
a2ps
a52dec
aacgain
aacskeys

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/bhrdx0n2afrmr0xf194am9qbg57sw1jh-gawk-5.0.0 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A gawk https://github.com/r-ryantm/nixpkgs/archive/569249f0848acccffe92d396a948e7978db0b9c3.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/bhrdx0n2afrmr0xf194am9qbg57sw1jh-gawk-5.0.0
ls -la /nix/store/bhrdx0n2afrmr0xf194am9qbg57sw1jh-gawk-5.0.0/bin

NEWS: https://git.savannah.gnu.org/cgit/gawk.git/tree/NEWS
Changes from 4.2.1 to 5.0.0

Support for the POSIX standard %a and %A printf formats has been added.

The test infrastructure has been greatly improved, simplifying the
contents of test/Makefile.am and making it possible to generate
pc/Makefile.tst from test/Makefile.in.

The regex routines have been replaced with those from GNULIB, allowing
me to stop carrying forward decades of changes against the original
ones from GLIBC.

Infrastructure upgrades: Bison 3.3, Automake 1.16.1, Gettext <IP_ADDRESS>,
makeinfo 6.5.

The undocumented configure option and code that enabled the use of
non-English "letters" in identifiers is now gone.

The `--with-whiny-user-strftime' configuration option is now gone.

The code now makes some stronger assumptions about a C99 environment.

PROCINFO["platform"] yields a string indicating the platform for
which gawk was compiled.

Writing to elements of SYMTAB that are not variable names now
causes a fatal error. THIS CHANGES BEHAVIOR.

Comment handling in the pretty-printer has been reworked almost completely
from scratch. As a result, comments in many corner cases that were
previously lost are now included in the formatted output.

Namespaces have been implemented! See the manual.  One consequence of this
is that files included with -i, read with -f, and command line program
segments must all be self-contained syntactic units. E.g., you can no
longer do something like this:
gawk -e 'BEGIN {' -e 'print "hello" }'

Gawk now uses the locale settings for ignoring case in single byte
locales, instead of hardwiring in Latin-1.

A number of bugs, some of them quite significant, have been fixed.
See the ChangeLog for details.

This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/12

@GrahamcOfBorg 	test installer.simple

@GrahamcOfBorg build libgpg-error
This seems to fail

@GrahamcOfBorg build libgpgerror

ping @Fuuzetsu @vrthra about the libgpgerror failure.

@Fuuzetsu @vrthra (libgpgerror maintainers) any chance you'd be able to look at this?

@GrahamcOfBorg build libgpgerror

cleaned up commit history, merging into staging.


Implement remote signing for Tendermint validator

NOTE: This is a progress tracking epic; new issues will be added as work continues.
Work proposal:

[ ] https://github.com/eris-ltd/eris-db/issues/523 implement a private validator in eris-db/consensus that signs locally (ie not a remote signer yet):

provide the basis for a fall-back mechanism;
move signing from tendermint into eris-db

[ ] provide eris-db/crypto
[ ] [...]
[ ] provide benchmarking for tendermint local signing; eris-db local signing and remote signing

Just make this "move keys into eris-db"


arduino-ide: Init at 2.2.1

Description of changes
This is the new Arduino IDE based on Theia IDE.
The old Arduino IDE is already packaged as arduino in nixpkgs, but upstream is unmaintained since two years.
I'm not really sure what the state between the two IDEs are, but it seems like both are still a thing, so I think we should keep both around for a while too. Therefore I took the upstream name arduino-ide as the package name for the new package.
I packaged it from the AppImage off GitHub as building it from source just looks very painful.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

By the way, I tried packaging this from source, and actually got it to compile
It's very hacky but it compiles.
https://github.com/TomaSajt/nixpkgs/blob/7381e9d4187039192ed1be56ee5dfd14f836d9da/pkgs/by-name/ar/arduino-ide/package.nix
I have not done work on it for a while now, and it's pretty messy.
The sad thing is that there are undefined symbol errors being thrown when starting the app.
The issue is with the packaged-together binaries, not being able to call those some v8 functions, like GlobalizeReference from here
probably has to do something with electron/node version mismatches. maybe somehow related to electron rebuild or npm rebuild, just guessing


Have better errors around "no nodes with  are available on the network"

Is this a duplicate of https://github.com/bacalhau-project/bacalhau/issues/2371?
Also, slight nit: In future issues provide the text in the issue rather than a screenshot, it makes it easier to search for the error in question e.g.
15:15:25.764 ERR pkg/requester/scheduler_state.go:103 > [transitionJobState] failed to find enough nodes to retry error-"not enough nodes to run job, requested: 1, avail
able: 0" [NodeID:QmZgkr18]
15:15:25.764 | ERR pg/requester/scheduler.go: 518 > error completing job d115889-d188-4087-92c8-656fb48eea41 error-"node QmfKYS1RL3G1yGnTVTQ1gLtQbUwSAKbWRQK2P7 JkgvLLza fa
iled due to: compute.AskforBidrequest: failed to decode response from peer OmfKyS1RL3G1yGnTVTO1gLtObUwSAKWROK2P7JkavLLza: stream reset" [NodeID:QmZgkr18] (stack: [ " func" :
"(*Basescheduler) .stopJob", "line": "518", "gource": "scheduler.go"), ("fune":" (*Basescheduler) checkForFailedExecutions. funci", "line": "96", "gource": "scheduler_state.go"), (" fun c":" (*Basescheduler) .checkForFailedExecutions", "line": "105", "source": "scheduler state.go"), ("func":" ("Basescheduler) .transitionJobstateLockFree", "line": "12"
"source": "sche
duler_state.go"), ("fune":" (*Basescheduler) transitionJobstate", "line": "21", "source": "scheduler_state.go"), ("fune": " (*Basescheduler) handleExecutionFailure"
"line": "510" "gource":"scheduler.go"), ("fune":" (*BaseScheduler) . doNotifyAskForBid", "line": "177", "source": "scheduler.go"), ("func": "goexit", "line": "1598", "source": "asm and64.s")]]
15:15:25.764 | INF pkg/eventhandler/chained handlers.go: 73 > Handled event [ClientID:] (EventName:Error] [HandleDuration:0.104946] [JobID:d115889f-d188-4087-92c8-656fb48ee a41] [NodeID:Om2gkr18] |SourceNodeID:OmZakr18hzcSam56ta7DZNtGJSiOz5BDo41XbrUPxKS4Vo] [Status: "node OmfRyS1RI3GlyGnTVT01qLtobUwSAkbWROK2P7JkavLLza failed due to: compute.As kForBidRequest: failed to decode response from peer OmfKy51RI3G1 yGnTVT01gLtobUwSAkbWROK2P7JkgvLLza: stream reset" [TargetNodeID:]
15:15:26.266 | IN pkg/publicapi/handlerwrapper/log_handler.go:32 [NodeID:QmZgkr18] [Request: ("ClientID": "dOe91377168bfceb07186f3da0044 fb61fccee62326829d0dfe41b10010bd
"Duration": 0, "Ipaddr": "<IP_ADDRESS>", "JobID": "a115889-d188-4087-92c8-656fb48eea41", "Method": "PoST", "NodeID": "Qm2gkr18h2C5qm56ta7DZNtGJSiQz5BDo41XbrUPKS4Vo", "Size": 1 007, "StatusCode": 200, "URI":"/requester/events", "User^gent": "Go-http-client/1.1"y

If I submit a job like:
Job:
  APIVersion: V1beta1
  Spec:
    Deal:
      Concurrency: 1
    Docker:
      Entrypoint:
      - sleep
      - "300"
      Image: ubuntu
    Engine: Docker
    Language:
      JobContext: {}
    Network:
      Type: None
    Publisher: Estuary
    PublisherSpec:
      Type: Estuary
    Resources:
      GPU: ""
    Timeout: 1800
    Verifier: Noop
    Wasm:
      EntryModule: {}
    Inputs:
    - Name: inputs
      StorageSource: LocalDirectory
    outputs:
    - Name: outputs
      StorageSource: IPFS
      path: /outputs

I get:
% go run . create job-local.yaml
Error executing job: failed to submit job: not enough nodes to run job. requested: 1, available: 0
exit status 1

So it's not clear why I have got the error. In #2487 we fixed the issue by printing the errors reported by individual nodes, we now need to apply the same sort of logic here (however, note that this error has come from the requester, i.e. we didn't reach out to any nodes).
This may involve updating the "ranking strategy" interface to include a reason for discarding nodes, and then printing a summary of why nodes have been discarded.


.properties files

I dont remember if developers are used to editing .properties in Eclipse?
File formats will keep changing forever, so I do not partially have a favorite. However, I like to stick to as few formats as possible
Can JSON be used instead of .properties? that way there is just one type of metadata format
Also, YAML is popular, TOML etc.

The file format for text-elements is .properties as is the corresponding ABAP part.


Setting context of called function

I think it would be a good thing to call the async function with the same context as the $async wrapper so if we for example have function line
this.$async(async function(){})  and we want to set this of this function we dont have to do this
this.$async(async function(){}.bind(this))()
but we could run it like
this.$async(async function(){}).call(this);
This way we could easily set the scope of the function to the one we want.

I see you merged my pull request, but it seems that you did not tag it hence I can not pull it from npm. Could you fix it ? :)


JoltPhysics apply_torque() uses different transforms than GodotPhysics?

I'm not quite sure in which coordinate system does the angular_velocity member of a RigidBody3D stores its value but after trying to understand this for a couple of days now i thought i might open an issue and see if someone could enlighten me on this.
Basically, when applying torque on one singular axis in global coordinates on a RigidBody3D in regular godot physics the angular velocity changes accordingly on that singular axis. In Jolt however, if the body is rotated then the angular_velocity will increase on multiple axises for some reason. I dont think this is right unless Jolt calculates stuff some other way.
I put the zip file for a minimal recreation project thing where u can test this for yourself. Rotate the object on an axis by inputting a direction and after some time reset the velocity by pressing SPACE. So far so good, Jolt shows correct velocity as far as u dont touch another axis. However, if you start rotating on another axis now the angular velocity changes on several other axises too. Change back and forth between Godot Physics and Jolt Physics to see the difference.  (I couldnt include the Jolt addon along with the project cus the file size got too big to upload here)
Angular Velocity Test.zip

I'm not quite sure in which coordinate system does the angular_velocity member of a RigidBody3D stores its value

It's in global-space.

However, if you start rotating on another axis now the angular velocity changes on several other axises too.

I'm not quite sure I can see the problem. Since angular_velocity is in global-space, neither of the two transformations you're doing in the logging (multiplying by basis and global_basis) are applicable, so if I just focus on the logging of the actual value of angular_velocity it all seems to make sense to me with both Godot Physics and Jolt.

EDIT: If you're looking for the local-space angular velocity, global_basis.inverse() * angular_velocity should do the trick, or global_basis.transposed() * angular_velocity for a faster version if you know you have a global_scale of (1, 1, 1).

ye this worked altough theres tiny precision errors when the angular velocity gets really high but i guess thats normal(?)

here, torque is applied only on the local Y axis and the logged vector is the global_basis.inverse() * angular_velocity

I'm not quite sure I can see the problem. Since angular_velocity is in global-space, neither of the two transformations you're doing in the logging (multiplying by basis and global_basis) are applicable, so if I just focus on the logging of the actual value of angular_velocity it all seems to make sense to me with both Godot Physics and Jolt.

yes, well i logged two additional transformations of the angular_velocity cus for some reason when i tried only using global space for torque and logging just the angular_velocity they didnt really match up (meaning that even tho i applied torque on one axis other axises were affected too in the angular_velocity logged)
It seems it works tho, may I have one last question tho. Does both angular_velocity and apply_torque() use Euler angles? And if I understand correctly, both use the global space.

theres tiny precision errors when the angular velocity gets really high but i guess thats normal(?)

Yes and yes. :)

Does both angular_velocity and apply_torque() use Euler angles? And if I understand correctly, both use the global space.

The angular_velocity is the amount of radians per second (clockwise) around that particular axis. apply_torque is specified in Newton-metre.
They're both in global-space, yes.

alright thanks a lot i was just really confused cus the docs dont really specify these stuff


Latest commit is causing failure when running non-interactive: /dev/tty: No such device or address

The commit below is breaking all helm install when running non-interactive.
Error:
/home/***/.helm/plugins/helm-tiller/scripts/tiller.sh: line 79: /dev/tty: No such device or address
Commit:
tiller installation should not break the silence
Commit Snip:
16: REDIRECT=/dev/tty
17: if [[ "${HELM_TILLER_SILENT}" != "false" ]]; then
18:   REDIRECT=/dev/null
19: fi
. . .
79: echo "Installed Helm version $INSTALLED_HELM" &> $REDIRECT

System Info:
-bash-4.2$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.6 (Maipo)
Release:        7.6
Codename:       Maipo

-bash-4.2$ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

ok, I'm going to revert back the last release and keep that way from now, as I see it is very difficult to make work for everybody.

Oh, I'm sorry about that. I didn't thought about non interactive. I think that my solution would work for both cases if we replace /dev/tty with /dev/fd/1 would you mind trying that?

for now I reverted back code to v0.9.1 in v0.9.3

but your fix was to revert the changes, we can keep the changes and use /dev/fd/1 instead of /dev/tty

please give https://github.com/rimusz/helm-tiller/pull/69 a try

yes, for now I reverted changes back to the code which works.
@luisdavim please test #69 on Mac and Linux, also in linux containers, as I do not have time to do such testing myself.

Sure, I'll run some tests.


fix import cycle not allowed

Just remove some code to avoid it, but we need to revisit code structure.

link to #16

no file change, so close PR.


[CLI] wandb.init() fails with HTTPError entity  not found during upsertBucket

Description
wandb==0.11.1
python==3.8.10
OS==Linux/Ubuntu 20.04.2 LTS
Running wandb.init(settings=wandb.Settings(start_method="fork")) in the beginning of a notebook or script works fine, but fails when run after setup steps with the following debug.log (I replaced my username with <username> everywhere below):
wandb: Currently logged in as: <username> (use `wandb login --relogin` to force relogin)
wandb: ERROR Error while calling W&B API: entity <username> not found during upsertBucket (<Response [404]>)
Thread SenderThread:
Traceback (most recent call last):
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 127, in execute
    six.reraise(*sys.exc_info())
  File "/project/path/env/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 121, in execute
    return self.client.execute(*args, **kwargs)
  File "/project/path/env/lib/python3.8/site-packages/wandb/apis/client.py", line 91, in execute
    raise requests.HTTPError(message, response=response)
requests.exceptions.HTTPError: entity <username> not found during upsertBucket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/path/env/lib/python3.8/site-packages/wandb/apis/normalize.py", line 22, in wrapper
    return func(*args, **kwargs)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 916, in upsert_run
    response = self.gql(mutation, variable_values=variable_values, **kwargs)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 118, in __call__
    if not check_retry_fn(e):
  File "/project/path/env/lib/python3.8/site-packages/wandb/util.py", line 726, in no_retry_auth
    raise CommError("Permission denied, ask the project owner to grant you access")
wandb.errors.CommError: Permission denied, ask the project owner to grant you access

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 55, in run
    self._run()
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 105, in _run
    self._process(record)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 292, in _process
    self._sm.send(record)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 181, in send
    send_handler(record)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 604, in send_run
    self._init_run(run, config_value_dict)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 626, in _init_run
    server_run, inserted = self._api.upsert_run(
  File "/project/path/env/lib/python3.8/site-packages/wandb/apis/normalize.py", line 36, in wrapper
    six.reraise(CommError, CommError(message, err), sys.exc_info()[2])
  File "/project/path/env/lib/python3.8/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/project/path/env/lib/python3.8/site-packages/wandb/apis/normalize.py", line 22, in wrapper
    return func(*args, **kwargs)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 916, in upsert_run
    response = self.gql(mutation, variable_values=variable_values, **kwargs)
  File "/project/path/env/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 118, in __call__
    if not check_retry_fn(e):
  File "/project/path/env/lib/python3.8/site-packages/wandb/util.py", line 726, in no_retry_auth
    raise CommError("Permission denied, ask the project owner to grant you access")
wandb.errors.CommError: Permission denied, ask the project owner to grant you access

A few things I tried:

Omitting settings=wandb.Settings(start_method="fork") or changing to "thread" does not help (same error for "thread", Error communicating with wandb process when I omit this setting).
Logged in from CLI just fine. API key correctly set as WANDB_API_KEY environment variable, and WANDB_ENTITY set to <username>.
Again, when I run wandb.init in the beginning of the pipeline, everything works fine.
Perhaps related to #1409, but the workaround of retrying doesn't solve the problem -- I get the same error on each retry.

Definitely a bug.  Can you try adding entity= to your wandb.init call?

I tried adding entity=<username> as a wandb.init argument -- no effect: same error as above.

Very strange.  It's almost like we're losing your credentials for the second call to wandb.init.  Are you able to share your code or psuedo code so we can reproduce.  Can you also try running pip install --force wandb==0.11.0 and see if the error still persists?

My bad: the username was misspelled in my .env. Doh!
When I just ran wandb.init() at the beginning of a script, it ran prior to loading the .env into the environment -- hence everything was fine. Running it after produced the above error due to the misspelling.


SSL reconnect fail at Linux environments

Fix SSLSocket so that it cancels the timer appropriately. The bug was to fail the cancellation of the timer if connect failed which caused the timer method to try access an invalid memory location of an already destroyed error_code object.
Also, changed the cluster test so that it runs with both TcpSocket and SSLSocket.

Test PASSed.


fix deprecation date

Can one of the admins verify this patch?

Thanks for the change. Can you revert the markdown changes, those are autogenerated from the source file.
@gunan do we want to merge this into r0.12? if so do we also need to get it separately into master?

Thanks for bringing this to my attention.
I agree with all your comments @danmane
Markdown changes should be automated, so should be rolled back,
Yes, we want this in 0.12
And yes, we want this also separately merged into master. My preference is to merge it to master before 0.12, but I am not picky about the order.

Jenkins, test this please.

Given that this is such a trivial change, I feel OK just merging it in.


[PE-D][Tester C] Filter Does Not Work After Adding New Person While Viewing Wedding Event

After adding a new contact while viewing a specific wedding event, the contact list returns to the full contact list instead of staying within the current wedding event's contact list. When using a filter, such as t/ designer, the system responds with "cannot find" even though the contacts should be visible in the list. This seems to be caused by a mismatch between the actual list being shown in the GUI and the list used in the background, where the backend is still under the current wedding context.

what it show, designer is there!

filter is not working...

Labels: type.FunctionalityBug severity.Medium
original: iamdiluxedbutcooler/ped#9

Duplicate


[bug] cannot create webview tauri 2

Describe the bug
I'm trying to create a webview inside my current WebviewWindow using javascript.
It fails with the following error:
Failed to load resource: the server responded with a status of 400 (Bad Request)
Reproduction
pnpm create tauri-app --rc
Update permissions:
{
  "$schema": "../gen/schemas/desktop-schema.json",
  "identifier": "default",
  "description": "Capability for the main window",
  "windows": ["main"],
  "permissions": [
    "shell:allow-open",
    "core:webview:allow-create-webview-window",
    "core:webview:allow-create-webview",
    "core:webview:allow-reparent",
    "core:webview:default",
    "core:webview:allow-set-webview-position",
    "core:webview:allow-set-webview-size",
    "core:window:allow-create",
    "core:window:default",
    "core:app:default"
  ]
}

Run the following javascript:
import { Webview, getCurrentWebview } from "@tauri-apps/api/webview";
import { Window } from "@tauri-apps/api/window";

  const window = Window.getCurrent();
  const webview = new Webview(window, 'main2', {
    x: 0,
    y: 0,
    width: 200,
    height: 200,
  });

  webview.once("tauri://created", () => {
    console.log("webview created");
  });

Expected behavior
I would expect a new Webview to be embedded in my window
Full tauri info output
[✔] Environment
    - OS: Mac OS 14.5.0 X64
    ✔ Xcode Command Line Tools: installed
    ✔ rustc: 1.80.0 (051478957 2024-07-21)
    ✔ cargo: 1.80.0 (376290515 2024-07-16)
    ✔ rustup: 1.27.1 (54dd3d00f 2024-04-24)
    ✔ Rust toolchain: stable-aarch64-apple-darwin (default)
    - node: 20.13.1
    - pnpm: 9.7.0
    - npm: 10.5.2
    - bun: 1.1.18

[-] Packages
    - tauri [RUST]: 2.0.0-rc.0
    - tauri-build [RUST]: 2.0.0-rc.0
    - wry [RUST]: 0.41.0
    - tao [RUST]: 0.28.1
    - @tauri-apps/api [NPM]: 2.0.0-rc.0
    - @tauri-apps/cli [NPM]: 2.0.0-rc.1

[-] App
    - build-type: bundle
    - CSP: unset
    - frontendDist: ../build
    - devUrl: http://localhost:1420/
    - framework: Svelte
    - bundler: Vite

Stack trace
No response
Additional context
No response

Did you enable the unstable feature flag on the tauri dep in Cargo.toml? By default multiple webview support is not enabled.

Yes I did

It works using the rust api but not with the javascript :(

Hi @LeoMartinDev  - it looks like you are trying to inject a Webview onto the current window.  I'm doing the following, which is to create a new window, and then waiting for that window to be created before putting a Webview on it.  This is working for me (other than positioning), maybe it will give you an idea on pursuing what you are doing...
  let lbl = 'xxx' // generate unique label

  const new_window = new Window(lbl, {
      parent: Window.getCurrent(),
      width: 600,
      height: 600,
      center: true,
      visible: false,
      skipTaskbar: true,
  })

  new_window.once('tauri://created', (e) => {
      const webview = new Webview(new_window, lbl, {
          url: info.url,
          x: 0,
          y: 0,
          width: 600,
          height: 600,
          incognito: true,
      })

      webview.once('tauri://created', function () {
          new_window.show()
      })

      webview.once('tauri://error', function (e) {
          feedback.toast(`${e.payload ? e.payload : e}`, ToastSeverity.Error)
      })
  })


--error-format=json support

Available in Rust 1.12

My bad, check already passes the arguments it doesn't understand...


Persistent Message map for preserve threading

Is your feature request related to a problem? Please describe.
Current options for preserve threading allows for platforms that support replies/threads to represent that properly
However when the bot is restarted that information is lost and therefore threads/replies are lost
Describe the solution you'd like
Using a persistent data struct store message mapping IDs that may be referenced back to for long term thread preservation.
TediCross utilizes a sqlite DB to store the data and read from long term.
For higher performance rocksdb can be considered, a fork of TediCross has implemented this, however I have not confirmed if it works or not.
https://github.com/TediCross/TediCross/commit/36d016fe6937d11158b30640dcbcc1db51d1735f
I took a peek at the source and a fallback can be added to FindCanonicalMsgID()
Describe alternatives you've considered
An alternative is to allow for a fallback for a text in-line reply when a message isnt found in the in-memory message map.
However, this is a much worse solution.
(Currently a reply/quote is not shown at all when threading cannot be resolved)
Additional context
After trying many chat bridges this is by far the best one. I really really appreciate and admire the effort taken in this project!

Realized this is a duplicate of #541.
Closing.


adding a for into listenAndPublishMessage func so it continuously lis…

…tens for updates.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Harter, Nicholas (Contractor) seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #104 into master will increase coverage by 0.06%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #104      +/-   ##
==========================================
+ Coverage   84.85%   84.91%   +0.06%
==========================================
  Files          79       79
  Lines        3559     3560       +1
==========================================
+ Hits         3020     3023       +3
+ Misses        448      446       -2
  Partials       91       91

Impacted Files
Coverage Δ

webhook/aws/sns_handler.go
75.22% <88.88%> (+2.08%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65deaee...f4a9732. Read the comment docs.


k8s: fix infinite loop when waiting for container

Coverage decreased (-0.09%) to 13.001% when pulling 4a096e04dc738488dc6a3be4c53af5253d0d4044 on cryptobioz:fix-infinite-loop into 18550297f759344d4b3cffcd17ded4df0d0b2d07 on camptocamp:master.

This PR is supposed to fix, among other, the situation where a worker is evicted.
On hold until we find a way to test it.


CM-275 Generate Random Double Number in Range

Hey @sethdtwigg, in the future it would be better to create a separate branch for each article and then to create a PR from that branch to our main. That way we can get a clean commit history.


GraphCast: Add the option to use transformer_engine.LayerNorm

Note from Max: Can we add the option to use transformer_engine.LayerNorm? In AIFS benchmarks, we just could get a 1.3x end-to-end improvement from doing so since the PyTorch implementation is rather bad for the sizes we encounter in these workloads.

https://github.com/NVIDIA/modulus/pull/510


StylePreprocessor Async request to compile Sass/Scss in the browser

What problem does this feature solve?
solve
Can compile Vue and Sass in the browser.（CDN）

Library to compile Vue

URL:https://github.com/FranckFreiburger/vue3-sfc-loader
CDN:https://cdn.jsdelivr.net/npm/vue3-sfc-loade

Library to compile Sass

URL:https://github.com/medialize/sass.js/blob/master/docs/getting-started.md
<EMAIL_ADDRESS>reason
the package vue-compile-sfc does not support async function, in brower sass.js doesnt support sync.
https://github.com/FranckFreiburger/vue3-sfc-loader/issues/125
What does the proposed API look like?
const result = nodeSass.renderSync(finalOptions)

↓
const result = await nodeSass.renderSync(finalOptions)

https://github.com/vuejs/core/blob/4a3237ad9300693e465f82a6be3552565a1c4be3/packages/compiler-sfc/src/stylePreprocessors.ts#L36

Can you explain more than one sentence:

What the problem is
Why it needs to be fixed at our end.

Closing due to lack of response

#5231


Mcuboot fails to compile when using single image and usb dfu

Describe the bug
I am not sure if this is a bug or a feature request, because I can't find much information on my requirement.
I am writing an application which needs to be bootloadable over USB, and does not have physical access to a boot button. Because of this, I would prefer that the bootload process can be initiated from within the application context. My current idea is to use mcumgr with the mcuboot bootloader. I have done this before with success.
However, in this case my application is slightly too large and so I cannot fit the image into the image-0 location if there is an equally sized image-1. Instead, I have removed image-1 and the scratch partition and built mcuboot with CONFIG_SINGLE_IMAGE_DFU=y.
This compiles perfectly fine and I am able to flash it to my device with no problems. As far as I understand this is supported functionality in the zephyr fork of mcuboot.
Similarly, I can build the application perfectly fine with CONFIG_BOOTLOADER_MCUBOOT=y. However if I try to include an image manager with CONFIG_MCUMGR=y things no longer work because it is looking for the image-1 partition, even though it is not required in my case.
To Reproduce

Remove image-1 partition from board dts
Build mcuboot with CONFIG_SINGLE_IMAGE_DFU=y to confirm this is supported
Build an application with CONFIG_MCUMGR=y and CONFIG_BOOTLOADER_MCUBOOT=y to find errors regarding image-1

Expected behavior
mcumgr and dfu util in general should be aware of CONFIG_SINGLE_IMAGE_DFU=y
Impact
Very difficult to proceed on my application if I must have two image slots
Environment (please complete the following information):

OS: Windows 10
Toolchain: gnuarmemb
latest master

Additional context
I am also unable to build mcuboot with CONFIG_SINGLE_IMAGE_DFU=y and CONFIG_BOOT_WAIT_FOR_USB_DFU=y as this also causes errors looking for image-1.
Edit:
I have realized using mcumgr doesn't really make sense for single image applications since it has to overwrite the application. However, mcuboot with usb dfu should work and it doesn't. I've edited the title to reflect this.

@bpbradley, if you only have one partition why are you trying to use mcumgr. The application can never upgrade itself, so mcumgr does not really do anything (except maybe capture a reboot request).

The main things I wanted with mcumgr were:

Shell transport layer for the actual dfu, although this isn't a big deal.
I had thoughts of doing bluetooth transport at a later stage, but also not a big deal.
Mainly I need a way to initiate the bootload process from the application, and couldn't figure out how else to do that because trying to build mcuboot directly with CONFIG_BOOT_WAIT_FOR_USB_DFU also didn't work, and trying to do it with CDC_ACM and using BOOT_SERIAL doesn't work because mcuboot expects a boot pin.

If there is a way to do usb dfu without a boot pin and without a second image slot, then I am happy to do it that way.

@bpbradley, maybe you can trick mcuboot into believing there is a image-1 by making image-1 equal to image-0 in the dts.

That does indeed work to trick it. However there is another issue where both subsys/dfu/boot/mcuboot.c has a second definition of multiple functions defined within bootloader/mcuboot/boot/zephyr/flash_map_extended.c and bootloader/mcuboot/boot/bootutil/src/bootutil_misc.c. Namely flash_area_erased_val flash_area_read_is_empty boot_magic_compatible_check and boot_read_swap_state_by_id

I don't think you should trick it, but instead just remove image-1. MCUBoot should be able to deal with this, but it could be that the Zephyr DFU is not 100% up to date with this option at the moment. Might be a bit somewhat related to: https://github.com/zephyrproject-rtos/zephyr/issues/27673
/cc @de-nordic @nvlsianpu

I don't think you should trick it, but instead just remove image-1.

It should be enough to remove image-1 as usually the dts would be altered so that image-1 would be removed and image-0 would reuse the space after image-1.

It is enough to define CONFIG_BOOT_WAIT_FOR_USB_DFU=y to catch most of the errors. So the CONFIG_SINGLE_IMAGE_DFU is not culprit here. I need to check it closer but I think that when the USB is selected the "non-single image" code is build, since the problem with missing image-1 is reported.

1. Remove image-1 partition from board dts

USB-DFU depends on it and is not designed to work with CONFIG_SINGLE_IMAGE_DFU=y

1. Remove image-1 partition from board dts

USB-DFU depends on it and is not designed to work with CONFIG_SINGLE_IMAGE_DFU=y

I am not acquainted with the particular implementation, but USB-DFU is a USB class, no? As far as I understand MCUBoot can use USB-CDC for it's BOOT_SERIAL operation (I don't think I've ever tried it myself though!). So I assume when you say USB-DFU, this does not involve USB-CDC; sorry if I am misinterpreting the comment.

1. Remove image-1 partition from board dts

USB-DFU depends on it and is not designed to work with CONFIG_SINGLE_IMAGE_DFU=y

I am not acquainted with the particular implementation, but USB-DFU is a USB class, no? As far as I understand MCUBoot can use USB-CDC for it's BOOT_SERIAL operation (I don't think I've ever tried it myself though!). So I assume when you say USB-DFU, this does not involve USB-CDC; sorry if I am misinterpreting the comment.

This was my original plan. Unfortunately when you configure BOOT_SERIAL it appears to expect CONFIG_BOOT_SERIAL_DETECT_PORT and CONFIG_BOOT_SERIAL_DETECT_PIN to be defined as well, and checks that this pin is at an expected value in order to enter serial recovery mode. But in my application there is no access to a physical button or switch which can be used to trigger this. I am not sure a way forward here other than finding a way to trim my app down far enough to fit it into two images.

1. Remove image-1 partition from board dts

USB-DFU depends on it and is not designed to work with CONFIG_SINGLE_IMAGE_DFU=y

Is that a software configuration issue or implementation issue?  It is very simple thing that is just linked in place of full loader. It is basically called when update path has not been taken.
The heavy lifting regarding serials, USB, DFU, etc is really done by other mcuboot code.
@utzig  Shouldn't we be able to weasel out of the problem by proper #ifdefing and configuring the code?

Hi @bpbradley
Opion 1:
MCUBoot serial recovery using USB CDC ACM virtual port should works for you - just need to modify mcuboot to wait some time for command instead of checking GPIO. Am I right?
This is something which can be added to mcuboot pretty quick if you can open such a PR to the jullabs/mcuboot.
Option 2:
CONFIG_BOOT_WAIT_FOR_USB_DFU=y
MCUBoot with USB DFU class can't work now as it uses zephyr library which assumes loading to the secondary slot (and has issue described above). This is something which might be improved as well.
What is your choice?
Also be aware that name of Konfig option CONFIG_SINGLE_IMAGE_DFU was changed to CONFIG_SINGLE_APPLICATION_SLOT.

1. Remove image-1 partition from board dts

USB-DFU depends on it and is not designed to work with CONFIG_SINGLE_IMAGE_DFU=y

Is that a software configuration issue or implementation issue?  It is very simple thing that is just linked in place of full loader. It is basically called when update path has not been taken.

In USB DFU implementation, image_0 is read only area, image_1 can be read and written. The option CONFIG_SINGLE_IMAGE_DFU was unknown when the code was last revised.

^^ that's right.

What is your choice?
Also be aware that name of Konfig option CONFIG_SINGLE_IMAGE_DFU was changed to CONFIG_SINGLE_APPLICATION_SLOT.

I believe you missed upstreaming from JuulLabs-OSS to have this already available for Zephyr, no?

^^ No, zephyr is in code-freeze for 2.4 release. Only bug-fixes can go in.

Also registered https://github.com/zephyrproject-rtos/zephyr/issues/28504

Hi @bpbradley
Opion 1:
MCUBoot serial recovery using USB CDC ACM virtual port should works for you - just need to modify mcuboot to wait some time for command instead of checking GPIO. Am I right?
This is something which can be added to mcuboot pretty quick if you can open such a PR to the jullabs/mcuboot.
Option 2:
CONFIG_BOOT_WAIT_FOR_USB_DFU=y
MCUBoot with USB DFU class can't work now as it uses zephyr library which assumes loading to the secondary slot (and has issue described above). This is something which might be improved as well.
What is your choice?
Also be aware that name of Konfig option CONFIG_SINGLE_IMAGE_DFU was changed to CONFIG_SINGLE_APPLICATION_SLOT.

Thanks for the info. Either of these options is acceptable to me. I think I might move forward with option 1 and creating a PR for now since it seems like a lighter lift.
Appreciate the clarifications.

#28506 resolves this issue. Closing.


Error when parsing positions and line cues that contain text

Library version: 0.1.7
Environment: WSL2 running under Windows 10
Current behaviour: Code panics when parsing a vtt file with a specific formatting
From the specs, the WebVTT format allows to define position and line cues with an optional text parameter, like this:
line:90%,end position:50%,center

The current code for parsing both the line and position cues will split the cue at the colon, remove the percentage and try to parse the rest as a number. However, having the optional string parameter will lead to a panic, because the code will try to parse 90,end and 50,center.
I've tried modifing the code in the vtt.rs file at line 421 from
spos.pos = py.replace('%', "").parse::<i32>().expect("number");

to
spos.pos = py.replace('%', "")
    .split(",")
    .collect::<Vec<&str>>()
    .first()
    .unwrap_or(&"")
    .parse::<i32>().expect("number");

and it works.
Of course, this is not the best way to handle the situation (the second value is simply discarded, which is probably an unwanted behaviour).
I'm a rust noob (the one I'm doing is literally the first project I write in this language) and that's why I didn't fix it and put it as a PR, but I still hope that I've been somewhat helpful 🙂

I see this example mentioned in the example above this bit here https://developer.mozilla.org/en-US/docs/Web/API/WebVTT_API#webvtt_cues but then lower down the line when they go into details I see no more mentions of this specific style of aligning the text.
I'll try to get a bit of time to look into it as well, but so far I haven't seen any subtitles/captions making use of a bunch of functionality in the standard...mostly because the players don't properly render it 🙄

Yes, indeed it seems that they didn't port that possibility inside the MDN...
Thank you a lot for the work you're putting into this!


build: redo releases & client testing

The original goal here was just to improve our publishing process, but this escalated into general de-GitHub-actions-ification.
Not fully finished yet, here's where it is going:
Overall philosophy

Avoid building a distributed system if we can. Zig can cross-compile, so we have a fighting chance of keeping everything on a single machine. Maaaybe things like glibc ABI will push us against cross compiling literally everything, but we should try!

Use Zig, both for positive reasons (build up a re-usable ecosystem for everything zig) and negative (avoid GHA lock-in).

Use CI only for distributing work across machines with different configurations (as a consequence of the first two points).

Embrace essential variability. We are not building software to run on our tightly controlled hardware. We are building software for other people to run elsewhere. So, don't try to precisely control the environment, and instead try to ask as little of it as possible.

Build process for releases
Three steps:

zig build dist creates a ./dist folder with a "distribution" of TigerBeetle. Distribution includes TigreBeetle binary cross-compiled for all supported targets, as well as package archive for all clients (.nuget for dotnet, .jar for java, .tar.gz for node and just a folder with sources and pre-built binaries for go).
zig build dist runs on a single computer, but, because the result is just a directory with files, we should be able to distribute this, if needed (but the plan is to try to avoid that, if at all possible).

zig build dist --publish publishes relevant parts of the distribution to the relevant registries.

zig build client_ci --verify-release runs after publishing and sanity checks that the stuff in remote registries makes sense.

There's intentionally no testing during release process. All the testing we currently do should be done before the code gets to a releasable state. Nonetheless, the dist/publish separation will allow us to run extra QA on the actual binaries we release.

Release process

Releases process is triggered once a new commit is pushed to the release branch.
Releases are manual and weekly. Every Monday, someone pushes current main to release
Weekly release is accompanied by a changelog. Changelogs are stored in changelog/ folder using changelog/$(date --iso).md naming scheme (with weekly changelogs, using a single file will not last us many years).
There are no nightly releases --- weekly cadence should be fast enough.
Because releases are weekly, CI's run number is used for the patch version. Version doesn't appear in the source code, as weekly releases would require too many changes.

Details
src/scripts/dist.zig is a script which manages building a distribution and publishing. This is a stand-alone script, rather than a build.zig task, because it orchestrates different build systems.
src/clients/ci.zig is a script which orchestrates CI for clients. It delegates to src/clients/$LANG/ci.zig.
Client testing is now run as a part of core pipeline. If we want releases, we must test our code properly. I am fairly certain we can make this non-flaky and fast, especially given that tests now don't try to do things, unrelated to testing.
Client testing includes running "unit tests" withing the project itself, as well as running all our samples with a real tigerbeetle.
I am not sure what matrix we should go for! Right now, I am using GitHub and self-hosted runners, but I think it might make sense to use upstream langauge-specific docker containers, as in
https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container for the purpose of embracing essential variability
Publishing docker images is TODO

Weekly release is accompanied by a changelog. Changelogs are stored in changelog/ folder using changelog/$(date --iso).md naming scheme (with weekly changelogs, using a single file will not last us many years).

Apologies for bikeshedding, but can you elaborate on this? I would expect that "one big changelog file" would scale better to frequent releases than "one small changelog file per release".
It's also more convinient (at least imo) if you are an operator trying to see what has changed since your last version. Release X includes all changes < X, so you just keep reading down until you hit the version you're upgrading from (or vice-versa, depending whether the file is ordered chronologically or reverse-chronologically).

That’s not bikeshedding, that’s important, thanks for raising the point.
rust lang uses a single for the changelog, and, with release every six weeks, GitHub no longer can always display this file:
https://internals.rust-lang.org/t/rust-1-72-1-pre-release-testing/19566/4?u=matklad
With our aim to have more frequent releases, I think we’ll hit this problem sooner.
My other bit of thinking here is that we might want to pull reach separate changelog to a page on the website, so that it can be tweeted and such, a-la https://rust-analyzer.github.io/thisweek. This also feels easier to do with separate files.
no super strong feelings here, can go with traditional single file!

:+1: I don't feel strongly about it either.

My other bit of thinking here is that we might want to pull reach separate changelog to a page on the website, so that it can be tweeted and such, a-la https://rust-analyzer.github.io/thisweek. This also feels easier to do with separate files.

Very good point, this tipped me in favor of multiple files.
:thinking: I wonder if the changelog belongs under docs/ -- and if we are going to be linking to the changelog externally, it could be built into docs.tigerbeetle.com too.
Aside: https://github.com/rust-lang/rust/blob/master/RELEASES.md is only 13k lines, which is not that much more than replica.zig!

One more thing to consider is how the changelog gets updated:

Is it updated as part of the same PR that implements the change?
Or do we go and update it later, all at once, just before each release?

Yeah, this PR adds docs/changelog, and, yeah, the idea is to expose that to tb.com!

One more thing to consider is how the changelog gets updated:

Unclear! There’s value in both “everyone adds stuff to changelog” and “changelog editor compiles the weeks worth of PRs”.
id maybe go with the second one by default: that’s a lot more work, but higher quality: if someone else writers about your PR in the changelog, this acts as a forcing function to create externally relevant&understandable descriptions

There’s value in both “everyone adds stuff to changelog” and “changelog editor compiles the weeks worth of PRs”.

Let's give it a try and do both!

In the spirit of "everything on main is good to go", PRs also add to changelog or else put Changelog: Skipped in the commit, and we lint that.

In the spirit of "editing and quality", before a release there can be a commit to edit the changelog if need be.

each separate changelog to a page on the website, so that it can be tweeted and such, a-la https://rust-analyzer.github.io/thisweek.

To solve this, and still have a single changelog file, we could have the latest release on the one pager changelog have a "latest" anchor.
If it helps, I'd prefer a single changelog file (one less directory, and less files)—this also increases the probability that someone carries on reading into the next one. If a single changelog doesn't work, we could then split it out later if need be.

Yeah, this PR adds docs/changelog, and, yeah, the idea is to expose that to tb.com!

Yes, that is excellent, and to have it at docs/changelog.

There's intentionally no testing during release process. All the testing we currently do should be done before the code gets to a releasable state. Nonetheless, the dist/publish separation will allow us to run extra QA on the actual binaries we release.

and

Releases are triggered manually, on Monday, by pushing the the current tip of origin/main to origin/releases. This triggers the release.yml workflow. The workflow doesn't run unit-tests, not rocket science rule is assumed.

Do you have something in mind for how we'll handle the testing-before-releasing code? I'm thinking of the long running stuff specifically when we have a time budget for testing (bounded things like unit tests should be every commit and simple enough).

Do you have something in mind for how we'll handle the testing-before-releasing code? I'm thinking of the long running stuff specifically when we have a time budget for testing (bounded things like unit tests should be every commit and simple enough).

Yup! The "push to release" workflow runs dist --build, but not dist --publish. The resulting dist is then uploaded as GHA artifact. Any further testing works with this uploaded artifact, to enforce the invariant that we release exactly the thing we've tested. After that extra round of testing (which might be arbitrary long), another action runs dist --publish (but not dist --build) to upload the tested artifacts.


Doesn't render images in iOS when content-type is image/jpeg; charset=utf-8

If the URI responds with a content-type of image/jpeg it's fine.  Responding with the charset included (image/jpeg; charset=utf-8) breaks rendering in iOS.
I'm unsure if this is an issue related to SDWebImage.

It would be related to SDWebImage I think. I can look into it. Do you get any error message in Xcode?


Edit

👷 Deploy Preview for adoring-leavitt-e31a11 processing.

Name
Link

🔨 Latest commit
2d5ca4b56abaf45d75d701dd124a580dd9e8513d

🔍 Latest deploy log
https://app.netlify.com/sites/adoring-leavitt-e31a11/deploys/629e637f5e4c520008b35ddb