image: "mysql:5.6"

Hi,
is there a way how to make this working with image: "mysql:5.6" ?
Currently I get:
logs test-mysql-0 -c mysql
2021-01-31 16:10:33+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.51-1debian9 started.
mkdir: cannot create directory '/usr/data/': Permission denied

and
logs test-mysql-0 -c init
Create Google Drive service-account.json file.
Create rclone.conf file.
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "MY_SERVER_ID_OFFSET"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "INIT_BUCKET_URI"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "BACKUP_COMPRESS_COMMAND"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "BACKUP_DECOMPRESS_COMMAND"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "RCLONE_EXTRA_ARGS"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "XBSTREAM_EXTRA_ARGS"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "XTRABACKUP_EXTRA_ARGS"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "XTRABACKUP_PREPARE_EXTRA_ARGS"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "XTRABACKUP_TARGET_DIR"}
2021-01-31T16:09:41.806Z        INFO    sidecar environment is not set  {"key": "INITFILE_EXTRA_SQL"}
2021-01-31T16:09:41.806Z        INFO    sidecar cloning command {"host": "test-mysql-0"}
2021-01-31T16:09:42.821Z        INFO    sidecar service was not available       {"service": "test-mysql-replicas", "error": "Get http://test-mysql-replicas:8080/health: dial tcp <IP_ADDRESS>:8080: connect: connection refused"}
2021-01-31T16:09:43.845Z        INFO    sidecar service was not available       {"service": "test-mysql-master", "error": "Get http://test-mysql-master:8080/health: dial tcp <IP_ADDRESS>:8080: connect: connection refused"}
2021-01-31T16:09:43.845Z        INFO    sidecar nothing to clone from: empty cluster initializing
2021-01-31T16:09:43.845Z        INFO    sidecar configuring server      {"host": "test-mysql-0"}
2021-01-31T16:09:43.845Z        INFO    sidecar error while reading PURGE GTID from xtrabackup_binlog_info      {"error": "open /var/lib/mysql/xtrabackup_binlog_info: no such file or directory"}

my config:
apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlCluster
metadata:
  name: test
spec:
  replicas: 1
  secretName: test
  image: "mysql:5.6"

thank you
M

It seems problem is generated init script /etc/mysql/conf.d/operator-init.sql

Hi @MilanDasek,
Can you point out wherein the generated init script is the problem? The above logs are not helping.
Thank you!

Hello @AMecea
ERROR: 1064  You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IF EXISTS sys_operator@'%';' at line 1 2021-02-02 06:59:35 1 [Note] Execution of init_file '/etc/mysql/conf.d/operator-init.sql' ended.
DROP USER IF EXISTS
percona - which image you recommend ?
image: percona:5.6.51
Please, is there anything else I need to set in terms of 5.6 version? Like f.e. operator's image, sidecarImage etc. ?
thank you
M

I've checked the MySQL 5.6 user manual and it seems that some statements that we are using are not supported:

DROP USER ~IF EXISTS~ used here
ALTER USER ... ~WITH MAX_USER_CONNECTIONS~ used here, this can be bypassed.

Is it complicate to upgrade your application to use MySQL 5.7?

Unfortunately not, that 3rd party app is legacy, but very important and supports only 5.6

If you have any idea of how it should be rewritten, the create user logic from here, in order to work with 5.6 I'm glad to add support for it. Please open a PR and I will review it and merge it.

I run a percona 5.7 image also have this error


Example

Hi!
Cool component. Would you be willing to link to an example REPL showing how to use this in a Svelte application? Thanks!!

Hi, I don't have an example right now of it being used in a larger svelte application, but it should work the same as any other component. For a plain example of the component, take a look at [index.html](https://github.com/scottbedard/svelte-heatmap/blob/master/index.html, the live demo hosted here https://svelte-heatmap.netlify.app/
Best of luck!

Hi, I don't have an example right now of it being used in a larger svelte application, but it should work the same as any other component. For a plain example of the component, take a look at [index.html](https://github.com/scottbedard/svelte-heatmap/blob/master/index.html, the live demo hosted here https://svelte-heatmap.netlify.app/
Best of luck!


implemented GoogleTest with CMake, closes #40

After this pull request is merged you WILL have to update CMake to get the game to build again. This is currently not possible in the VM.
CMake version 3.23.1 is available on the official website.
https://cmake.org/download/

Since I had to do this from my desktop I had to use my other GitHub account but it is still me that did the commits.


Metrics: Re-expose pre-aggregation APIs in 2.7-Beta1 release.

This PR re-exposes the new metric APIs in the 2.7 Beta-1 release.
Prepared as discussed with @MS-TimothyMothra .
Background info on the current release plan: https://github.com/Microsoft/ApplicationInsights-dotnet/issues/785

@cijothomas , I am working on an issue that @MS-TimothyMothra  and I discovered relating to a bunch of merges that happened over the last 2 weeks of so. I hope to resolve it today. I will resolve the current merge conflict right after.


Exception on ContentProvider query

If I generate ContentProvider with DBFlow, and query model with CursorLoader like this:
return new CursorLoader(this, CustomModel.CONTENT_URI, null, null, null, CustomModel_Table._id + " DESC");
I get exception
java.lang.NullPointerException: Attempt to get length of null array                                                              at com.raizlabs.android.dbflow.runtime.BaseContentProvider.toProperties(BaseContentProvider.java:57)                                                              at ch.dotpay.db.DotpayDatabase_Provider.query(DotpayDatabase_Provider.java:136)
The exception is caused by reading length property of null projection field. But in Android SDK it is valid to specify null projection - it only indicates that CursorLoader should query all fields.

I have the problom

By the way, the "selection and selectionargs" seems no work in cursorLoader.

@agrosner, I think you should make a special wiki page about how DBFlow can and should be used with CursorLoaders. And how Model instances could be restored from Cursor objects.

in develop it now directly just calls down to native methods ...no conversions to worry about.


Chapter1/homework/8

The code is pretty complex, but I am not sure if you can do anything better without advanced C# features like LINQ or generics.


[Impeller] Flutter 3.13.0 regression of box shadow from container with partial border radius drawn over scrollable

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce

Create a Container with box shadow and bottom left border radius (actually any partial border radius reproduces the bug) at the top. Place a ListView below it with some simple text items. See sample code for details.

Run on iOS with impeller enabled and disabled and compare results.

flutter run --enable-impeller
flutter run --no-enable-impeller

Scroll the list and observe the extra white background on the text 'item x' of the list when Impeller is used.

This bug does not reproduce on Flutter 3.10.6 on the same device. Also it does not reproduce on Android with Flutter 3.13.0 with the flag --enable-impeller. The border radius on the box shadowed container must be partial to reproduce the bug.
Expected results
No extra white box should be drawn over the box shadow while scrolling the list.
Actual results
Extra white box is drawn over the box shadow after the list is scrolled. If scrolled up for the first time, the extra white blinks.
Code sample
Code sample
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Box shadow with impeller bug',
      home: Scaffold(
        appBar: PreferredSize(preferredSize: const Size.fromHeight(60), child: Container(
          decoration: const BoxDecoration(
            color: Colors.white, // Background color can be removed and this bug still reproduces.
            borderRadius: BorderRadius.only(bottomLeft: Radius.circular(30)), // BorderRadius.circular(30) does not reproduce the bug.
            boxShadow: [BoxShadow(color: Colors.black,blurRadius: 120)],
          ),
        )),
        body: ListView.builder(
          physics: const BouncingScrollPhysics(),
          itemBuilder: (context, index) => index < 100 ? Padding(padding: const EdgeInsets.all(16), child: Center(child: Text('item $index'))) : null
        ),
      ),
    );
  }
}

Screenshots or Video

Screenshots / Video demonstration

https://github.com/flutter/flutter/assets/4703033/7e63e979-aef3-4a9f-ac06-f98e419545e3

Flutter Doctor output
Doctor output
[✓] Flutter (Channel stable, 3.13.0, on macOS 12.6.1 21G217 darwin-x64, locale en-US)
    • Flutter version 3.13.0 on channel stable at /Users/jipeng/Desktop/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision efbf63d9c6 (5 days ago), 2023-08-15 21:05:06 -0500
    • Engine revision 1ac611c64e
    • Dart version 3.1.0
    • DevTools version 2.25.0
    • Pub download mirror https://pub.flutter-io.cn
    • Flutter download mirror https://storage.flutter-io.cn

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/xxx/Library/Android/sdk/
    • Platform android-32, build-tools 32.0.0
    • ANDROID_HOME = /Users/xxx/Library/Android/sdk/
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14C18
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] Connected device (3 available)
    • xxx (mobile) • xxx-xxx • ios            • iOS 15.6.1 19G82
    • macOS (desktop)     • macos                     • darwin-x64     • macOS 12.6.1 21G217 darwin-x64
    • Chrome (web)        • chrome                    • web-javascript • Google Chrome 116.0.5845.96

[✓] Network resources
    • All expected network resources are available.

• No issues found!

cc @jonahwilliams

I did a bisect between $(git merge-base master 3.13.0) and $(git merge-base master 3.10.0) ( e749db6f915c341d15b9cb81dc19bdc9e791c72f..bff6b93683de8be01d53a39b6183f230518541ac ), and reproduced the bug starting from e2ae260b429b91b496a1ae56432fa1801bcaa541.

The bounds computation in AIKS canvas is correct, and there is no incorrect culling before that point:
[VERBOSE-2:canvas.cc(167)] Draw Path Bounds: ((-1227, -1098), (2610, 1761))
So possibly something is going wrong with the coverage hint computation.

It looks like we clamp the X,Y of the coverage hint, and the resulting transform puts the RRect blur offscreen:
2:canvas.cc(167)] Draw Path Bounds: ((-1227, -1080), (2610, 1761))
[VERBOSE-2:gaussian_blur_filter_contents.cc(94)] Coverage hint: ((0, 0), (156, 2083))
[VERBOSE-2:gaussian_blur_filter_contents.cc(121)] ((0, 0), (156, 2083))

Seems like this isn't the issue after all, we're not actually clipping out the filter where I thought we were.

I think this is actually a clipping issue!

This works correctly when the border radius is uniform and/or there is no border radius as we're able to use the analytical solution and that is positioned correctly.

Okay, the actual problem seems to be that we compute the coverage incorrectly, and as a result either cull or render incorrectly:
[VERBOSE-2:gaussian_blur_filter_contents.cc(363)] CHECK FILTER COVERAGE: ((-507, -409), (1170, 321))
[VERBOSE-2:gaussian_blur_filter_contents.cc(372)] Computed result: ((-867, -409), (1890, 321))
In this case we seem to either create a height that evaluates to 0 or is otherwise very small.

Here is an impeller playground that demontrates the issue:
TEST_P(AiksTest, BlurFilterWithNonZeroOriginRendersCorrectly) {
  Canvas canvas;
  //canvas.Translate({0, -400, 0});
  Paint paint;
  paint.mask_blur_descriptor = Paint::MaskBlurDescriptor{
      .style = FilterContents::BlurStyle::kNormal,
      .sigma = Radius{120},
  };
  paint.color = Color::Red();
  PathBuilder builder{};
  builder.AddRect(Rect::MakeLTRB(0, 0, 800, 800));
  canvas.DrawPath(builder.TakePath(), paint);

  ASSERT_TRUE(OpenPlaygroundHere(canvas.EndRecordingAsPicture()));
}

I have a few occurrences of this bug. It is nothing to do with uniformity for me. Any border radius will reproduce.
https://github.com/flutter/flutter/assets/26356162/9c401ff5-3c8e-4006-8012-ca763bd70f07
There is the obvious top nav bar shadow. Then also take note of the scroll view card items, as they get near the top, the bottom shadow disappears.
Then I have another page where the top nav bar doesn't have shadow on top of a google map:

Only happens on Flutter 3.13, on iOS.

We discussed this in the Impeller sync today and decided to elevate priority. @bdero said he would take a closer look later today.

Will have a patch out for this today. There turned out to be a few overlapping problems here:

The mask blur can get double-applied in some circumstances.
One of the new coverage optimizations is interacting poorly with dirty regions. In the Gaussian Blur this optimization wasn't incorporating the full transform of the blur radius, causing the included blur region to wind up being too small on high DPI devices. I believe this is also the cause of the other issues we've seen come in. Going to cross reference.
There's a texture coordinate mapping issue.

This is how it looks with 2 out of the 3 problems fixed:

Once problem 3 (the texture coordinate mapping issue) is fixed, there will be no visible banding. We can easily prevent issues like this from reoccurring by tracking a couple of new mask blur goldens.

This issue can be affect v3.10.7 too?


adding high

Description
Please include a summary of the change with relevant motivation and context
Linked Issues

resolves

GitHub Issue Linking Docs
Checklist

[ ] I have added one of the patch, minor, major or no-release labels
[ ] I have linked an issue from this repository using the Development option
[ ] I have performed a self-review of my own code
[ ] I have checked for redundant or commented out code
[ ] I have commented my code where I can't make it self-documenting
[ ] I have made corresponding changes to the documentation
[ ] I have added any appropriate tests

Codecov Report
Patch and project coverage have no change.

Comparison is base (806de3b) 65.21% compared to head (b5797fb) 65.21%.

:exclamation: Current head b5797fb differs from pull request most recent head b59ca9f. Consider uploading reports for the commit b59ca9f to get more accurate results

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #25   +/-   ##
=======================================
  Coverage   65.21%   65.21%
=======================================
  Files           7        7
  Lines          46       46
=======================================
  Hits           30       30
  Misses         16       16

Impacted Files
Coverage Δ

pkg/parser/defaults.go
100.00% <ø> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


[Cross Platform]Refactor thread pool

Why
The current thread pool don't support other platform, refactor it to provide a cross-platform, work-steal thread pool.

Based on another PR, i will rebase main soon.


RouteGroup class don't respect RouteGroupInterface

When I used method "group"  about  "Slim\App" for my router same the documentation a have this error
PHP Fatal error:  Class Slim\RouteGroup contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (Slim\Interfaces\RouteGroupInterface::getInstance) in ~\MyProject\vendor\slim\slim\Slim\RouteGroup.php on line 19

I need add method "getInstance()" myself in the "vendor" directory for fixit this error.

What are you doing exactly?

Closing as old and no follow up


Refactoring - Change Smart-NIC to DPU

Data Processing Units are a new breed of Smart-NICs
A key difference is having a mulitcore processor able
to run commodity software as part of the SOC.
Some operations performed on DPUs are not supported
by standard Smart-NICs, properly rename relevant
methods to reflect that.
Note: As there has not been a release with the old
names, it is OK at this point to change naming of
user facing methods.
Signed-off-by: Adrian Chiris<EMAIL_ADDRESS>

@moshe010


File potentially missing

Traceback (most recent call last):
Wrote Shapes Dict
Traceback (most recent call last):
File "/tscc/nfs/home/ydoctor/datasets/UCE/eval_single_anndata.py", line 155, in
main(args, accelerator)
File "/tscc/nfs/home/ydoctor/datasets/UCE/eval_single_anndata.py", line 84, in main
processor.generate_idxs()
File "/tscc/nfs/home/ydoctor/datasets/UCE/evaluate.py", line 122, in generate_idxs
species_to_pe = get_species_to_pe(self.args.protein_embeddings_dir)
File "/tscc/nfs/home/ydoctor/datasets/UCE/data_proc/data_utils.py", line 263, in get_species_to_pe
species_to_pe = {
File "/tscc/nfs/home/ydoctor/datasets/UCE/data_proc/data_utils.py", line 264, in
species:torch.load(pe_dir) for species, pe_dir in embeddings_paths.items()
File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 771, in load
with _open_file_like(f, 'rb') as opened_file:
File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 270, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 251, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/dfs/project/cross-species/yanay/code/uce_code/UCE_public/model_files/protein_embeddings/Gallus_gallus.bGalGal1.mat.broiler.GRCg7b.pep.all.gene_symbol_to_embedding_ESM2.pt'

Also, here is the command I was running:
singularity exec --nv /tscc/nfs/home/ydoctor/containers/data_science_box.sif python /tscc/nfs/home/ydoctor/datasets/UCE/eval_single_anndata.py --adata_path /tscc/nfs/home/ydoctor/datasets/CM4AI/Perturb-seq_10_26_23_RNA_Only.h5ad --dir /tscc/nfs/home/ydoctor/datasets/CM4AI/UCE_Outputs/ --species human --model_loc /tscc/nfs/home/ydoctor/datasets/UCE/model_weights/UCE_params_33_layer.torch --batch_size 8 --nlayers 33
If you could point me towards what I might be able to do to fix this that would be great. Thanks so much!

I resolved it by commenting out these lines

I think what happened was latest merge had something to do with testing out incorporating new species but the ESM embeddings are saved locally so it breaks, would maybe consider just commenting these lines out

I think you can also just remove the line in the CSV file for protein embeddings. I've removed it on the main branch now. Thanks for pointing this out!


[FLINK-4717] Add CancelJobWithSavepoint

This builds on top of #2608.

Adds CancelJobWithSavepoint message, which triggers a savepoint before cancelling the respective job. The periodic checkpoint scheduler is stopped before doing this so that no checkpoints happen between the savepoint and cancellation. Note that users could manually trigger another savepoint while cancel-with-savepoint happens, but I think this is tolerable.

Adds -s [targetDirectory] option to CLI cancel command:

Regular cancelling:bin/flink cancel <jobID>
Cancel with savepoint to default dir: bin/flink cancel -s <jobID>
Cancel with savepoint to target dir: bin/flink cancel -s [targetDir] <jobID>

To help review, the main change is in JobManager and CliFrontend. Some questions there may be:

Should we keep the behaviour that the job is not cancelled if the savepoint fails? My reasoning for this was that the user might otherwise accidentally loose state when cancelling the job.
Does the way that the cancellation is specified on the CLI make sense? Should we keep the -s option or add another proper command like bin/flink cancel-with-savepoint?

Thanks for this review, too. Good catch with the broken stopCheckpointScheduler. I will add your proposed fix as a separate commit.

Addressed the issue and found another bug in #2608 that I've fixed in f769e8e. If Travis gives the green light, I will rebase on #2608 and merge this later today.

Local Travis build passed (https://travis-ci.org/uce/flink/builds/167108720). I'm going to rebase and merge this later.


README for Tracing library

The README for the tracing library needs to be filled out.

Closed in PR https://github.com/Azure/azure-sdk-for-java/pull/5632


net/relayd: relayd.conf template failed with http checks

Fix failed syntax check of the relayd.conf with check type HTTP.

Merged, thanks!


Additional baud rates or custom

Type: Feature Request
Please add standard baud rates up to 921600bps or allow for a custom baud rate to be entered.

Hi, @ecarter-anduril, thanks for the feedback! This is currently supported in the most recent release. There is a "Custom Baud Rates" Setting where you can input custom baud rates that you want to be used. Please let me know if you have any trouble using this feature!
Thanks

dup #1


Unset OMP_NUM_THREADS because it may break user parallelism expectations

#719  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @dongreenberg and the rest of your teammates on  Graphite


Fix and update examples (no format change)

Reopen of #14 that removes formatting changes

Thank you, please merge? took me some time to find this

Just chiming in to say that this PR is still relevant, the examples as they exist don't work currently. The fixes in this PR do work.

Is there a good reason not to merge this? The current example is outdated and does not work, so this is really helpful.

The code in examples/ was part of my explorations while designing this library.
I just redid the drag example to cover a case I need in practice from time to time: having two panels that you can resize by dragging the border between them. It now lives in https://github.com/elm/browser/tree/master/examples in place of the exploratory files.
Sorry for the confusion on this!


CSV import broken in v15.7.3, creates some duplicate edges

Preflight Checklist

[ x] I agree to follow the Code of Conduct that this project adheres to.
[x ] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Describe the bug
CSV import creates a duplicate edge between some vertices.  This does not happen for all edges, seems to happen only once for each source vertex, independent of the number of target vertices.  The CSV file only contains one link between each two vertices.
Remark it works fine up to version v15.4.0, which is the most recent one in the release page before the current v15.7.3
To Reproduce
Steps to reproduce the behavior:

Go to 'Arrange->Insert->Advanced->CSV...'
Copy&Paste content of attached  'import_issue.csv' file into the dialog
Click 'Import'
See screenshots

Expected behavior
Single edge created, as in v15.4.0.
Screenshots
Screenshots attached showing issue in v15.7.3 (double edge) and no issue in v15.4.0 (single edge)
draw.io version (In the Help->About menu of the draw.io editor):

draw.io version 15.7.3

Desktop (please complete the following information):

OS: Windows

Additional context
Add any other context about the problem here
import_issue.csv

.

Thanks for the report. This will be fixed in the next release.


Update prepare target in package.json to use pnpm

Summary
Our project use pnpm to manage dependencies and requires corepack installed and enabled:
corepack enable
corepack enable npm
Related Issue
https://github.com/web-infra-dev/rspress/issues/1496
Checklist

[x] Tests updated (or not required).
[x] Documentation updated (or not required).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


beginTimes are not on month boundaries

Need to round the timestamp before adding it to the DB, otherwise our graphs don't look so good.

Well, that's puzzling -- issuerReputation does truncate the month...


Add macOS 10.13 and 10.14 readme updates for week 51.

This includes the changes to the readme files for Mac OS 10.13 and 10.14 as deployed in Azure Pipelines for week 51.

This files have been removed. Abandoning.


Gradient Inconsistency

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 14.04.5
TensorFlow installed from (source or binary): pip install from binary
TensorFlow version (use command below): v1.4.0-19-ga52c8d9 1.4.1
Python version: Python3.5 (Anaconda)
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: CUDA-8.0
GPU model and memory: GeForce GTX 1070(8GB) & GeForce GTX 770(4GB)
Exact command to reproduce: N/A

Describe the problem
When use operation tf.transpose(), tf.gather(), tf.cholesky() all together in a row over instance of tf.Variable(), the backward gradient computation may seems inconsistent. By using 'inconsistent', I mean that after run the same script multiple times with fixed random seeds, the computed gradient of result of tf.cholesky() over input of tf.transpose() are not always identical.
Source code / logs
import tensorflow as tf
import numpy as np

np.random.seed(1024)
tf.set_random_seed(1024)

N = 10

# indices for `tf.gather()`
indices_for_gather = tf.constant(np.concatenate((np.zeros(N), np.ones(N))).reshape(-1,).astype(np.int32))

# build a 2-by-2 PSD matrix as `param` for 'tf.gather()'
W = tf.constant(np.random.rand(2, 1), dtype=tf.float32)
W = tf.Variable(W)

PSD2x2 = tf.matmul(W, W, transpose_b=True)

# do the `tf.gather()`
M_temp = tf.gather(PSD2x2, indices_for_gather)

# Then transpose the M_Temp and tf.gather() again (to ensure the result is a PSD for cholesky)
M_temp_T = tf.transpose(M_temp)

PSD = tf.gather(M_temp_T, indices_for_gather) + tf.eye(2*N, dtype=tf.float32) * 0.01

# cholesky
L = tf.cholesky(PSD)

# compute the gradient of `L` over `M_temp`
grad = tf.gradients(L, M_temp)

# build a session and compute the gradient
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
results = []
with tf.Session(config=config) as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(10):
        result = sess.run(grad)
        results.append(result[0])

# check the results are whether identical or not
for i in range(len(results) - 1):
    if isinstance(results[i], np.ndarray):
        print(np.all(np.equal(results[i], results[i+1])))
    else:
        print(np.all(np.equal(results[i].values, results[i+1].values)))

The output result is
False
False
False
False
False
False
False
False
False

If I comment the line W = tf.Variable(W) making the W a constant tensor, the results are ideally all True. And I've also tried to compute the gradient ofL over M_temp_T and PSD, both of them are all True. So I think the problem lies in the using of tf.transpose(), tf.gather(), tf.cholesky() all together over instance of tf.Variable().

I suspect that reinitializing the variables will reinitialize them inconsistently. Is it consistent if you tf.reset_deafult_graph() and rebuidl the graph every iteration of checking?

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 31 days with no activity and the awaiting response label was assigned. Is this still an issue?


service URLs in configmap not consistent with service names

Problem: if I install the helm-chart with a different Release.Name than "podtato-head", the arms and legs microservices are not found in the podtato-head-entry pod.
Error message:
Get "[http://podtato-head-left-arm:9003/images/left-arm.svg":](http://podtato-head-left-arm:9003/images/left-arm.svg%22:) dial tcp: lookup podtato-head-left-arm on <IP_ADDRESS>:53: no such host
Reason:
The service URLs defined in the configmap are hard coded to "http://podtato-head-" which is only correct when the Release.Name is "podtato-head".
The logic how the service name is built is here https://github.com/podtato-head/podtato-head/blob/main/delivery/chart/templates/service-left-arm.yaml#L8 and here https://github.com/podtato-head/podtato-head/blob/main/delivery/chart/templates/_helpers.tpl#L8-L24.
So maybe the configmap-discovery also needs this logic to define the correct names in the configmap?

Thanks @jkleinlercher - I submitted #187 to fix, PTAL!


You might be able to further whittle down leaf binaries

https://github.com/guardianproject/orbot/issues/625#issuecomment-1061639272
On this Orbot android leaf thread eycorsican says that using leaf/config-json instead of leaf/config-conf drops a bulky regex package and gives you an even smaller binary.
I still haven't gotten a chance to return to leaf on Android myself, but if this is something you can do for iOS I'm sure it'd go a long way!

It can be further reduced to 8.4MB by using leaf/config-json instead of leaf/config-conf because the conf format brings in a big dependency regex which is much bigger compares to a json parser. However the tun-fd option is missing in the json config as I never mind using this format for client applications.

I need the tun-fd option for iOS and the conf file format per se. Leaf is heavily underdocumented and the JSON config confuses me even more.


publish 2.5.5 docs

Note that this also updates the dropdown to replace 2.5.3 with 2.5.5 and update root to 2.5.5 (also instead of 2.5.3).


Experimenting with writing results in semsimian itself

This PR is an experimentation of writing the result as follows:

subject_id
object_id
jaccard_similarity
ancestor_information_content
phenodigm_score
cosine_similarity
ancestor_id

banana
apple
0.3333333333333333
1.3219280948873622
0.6638092333613531
-0.0811959702204112
banana

apple
apple
1
2.321928094887362
1.523787417879332
1
apple

Fixes #74

I believe that's correct - may be helpful to try it out with negative values too

If need be I think I see some spots where we maybe could speed this up in the future

Hey @justaddcoffee ! Could you point out those improvements? If these are straightforward enough, I could include them in this PR (?)

Hey @justaddcoffee ! Could you please point out those improvements? If these are straightforward enough, I could include them in this PR (?)

actually I see you are using par_iter here so I think this is as fast as what I had in mind - nevermind!

This was exactly what I was thinking. Another alternative is make the column_names an attribute (output_features) of the RustSemsimian class
impl RustSemsimian {
    pub fn new(spo: Vec<(TermID, Predicate, TermID)>, output_features:Vec<String>) -> RustSemsimian {
        RustSemsimian {
            spo,
            output_features,
            ic_map: HashMap::new(),
            closure_map: HashMap::new(),
            embeddings: Vec::new(),
        }
    }
}

This way the param list is not growing as we add features and RustSemsimian can have all needed information from oaklib` within itself as attributes.
Thoughts?

I like that - it makes for a more tidy main class and would allow for adding new args as needed.


Unclear on use of custom inventory files

Type of question

How to implement a specific feature

Question
Upon review of this merged commit, it doesn't look like there's been much documentation updated on its use.  I'm having trouble understanding how to determine how a custom ansible inventory is copied into the operator image and invoked.
My goal is to simply have the ansible-runner be aware of my inventory when processing a playbook.  The playbook is found and processed without issue but when the ANSIBLE_INVENTORY environment variable is added, controller fails to run due to it looking for an inventory file that doesn't exist.
What did you do?

My Dockerfile
FROM quay.io/operator-framework/ansible-operator:v0.19.0

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible

COPY watches.yaml ${HOME}/watches.yaml

COPY roles/ ${HOME}/roles/
COPY *-playbook.yml ${HOME}/
COPY inventory/ ${HOME}/inventory/
COPY ansible.cfg ${HOME}/ansible.cfg

The env section of the operator deployment
- name: ANSIBLE_INVENTORY
   value:  ${HOME}/inventory/inventory.yml

What did you expect to see?

I expected the pod to start and run properly, and reconciliation of my CR to recognize the inventory that I specified in the environment variable
What did you see instead? Under which circumstances?

Without the environment variable, the Ansible runner starts and stops the playbook without running it (due to no inventory hosts found).  With the environment variable, the pod returns error":"stat ${HOME}/inventory/inventory.yml: no such file or directory"
Environment
Operator type:

/language ansible

Kubernetes cluster type:

GKE
$ operator-sdk version
operator-sdk version: "v0.19.0", commit: "8e28aca60994c5cb1aec0251b85f0116cc4c9427", kubernetes version: "v1.18.2", go version: "go1.14.5 darwin/amd64"

$ go version (if language is Go)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.11-gke.5", GitCommit:"baccd25d44f1a0d06ad3190eb508784efbb990a5", GitTreeState:"clean", BuildDate:"2020-06-25T22:55:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

Additional context

Figured it out.   When entering the pod itself, I found that everything that was put into ${HOME} was put into /opt/ansible.  I changed the environment variable to be /opt/ansible/inventory/inventory.yml and it's working fine now.


Methods target and pos are documented in class Cursor, but that is an alias for Match

The problem
Cursor is an alias for Match, but the methods target and pos are documented just for class Cursor
Suggestions
Fold the documentation of Cursor into Match, and just document that Cursor is an alias for Match.

Some methods are part of NQPMatchRole, which is, well, NQP. And there are lots of methods there. Let's see...


build instability

need to track down why there is variability in the tests

062ac6b6497918656efc57384f777f2e7d35a745 solves at least part of this. closing for now hoping that this was the whole problem. will reopen if more problems are noticed


Salt (Next) release announcement

### Tasks
- [x] Create 1 word doc where we can consolidate
- [ ] 5 sections (below) in that word doc
- [ ] Move source content into one word doc
- [ ] Refine content [allocate/ together]
- [ ] Agree where content lives (internal/ external) [Together]
- [ ] Update Glossary: https://www.saltdesignsystem.com/salt/about/glossary
- [ ] Update Internal Site
- [ ] Publish Figma Library [BH or YY?]
- [ ] Publish pages [ZC]
- [x] What is the publish/release process in the future? Should Salt 1 & Salt Next be released together (in the same email?)
- [x] Release Banner [YY]
- [ ] Announcement News in Internal Site [YY]
- [ ] Announcement Communication (Email/Symphony/Teams)[YY]

### Announcement Communication
- [ ] An Email Newsletter (similar to "Pepper Pattern Library Goes Live!" in the internal site)
- [ ] Channels (Symphony/teams)
- [ ] News report (Like "Pepper Pattern Library Goes Live!" in the internal site)

[ZC] ‘Theming’- People interpret it as ‘make it look different’. What’s the better way to describe it? ‘Styling options of Salt theme’
[BH] ‘Endorsed JPM Brand foundations enable delivery teams to create optimised design languages using brand approved styling options for application users/ use cases.’

  Create 1 word doc where we can consolidate

  5 sections (below) in that word doc

  Move source content into one word doc

  Refine content [allocate/ together]

  Agree where content lives (internal/ external) [Together]

  Publish [ZC]

  Release note/ email [YY]

Thu 4 Jul - document reviewing WIP, discussed release banner and WIP

Added a task: to discuss - What is the release process in the future? Should Salt 1 & Salt Next be released together (in the same email etc.?) cc @bhoppers2008 @origami-z @pseys

12 Fri - Salt Next announcement communications:

A Newsletter (similar to "Pepper Pattern Library Goes Live!" in the internal site)
Channels (Symphony/teams)
News report Home page - (like "Pepper Pattern Library Goes Live!" in the internal site)

For the updates releases in the future, will be with the same release email that Salt 1 originally has

15 July - News page PR created for internal site - awaiting for links and review then can be published

Jul 15 - #3814 ready to be merged

News page in the Internal site updated. Announcement Communication (Email/Symphony/Teams) done! @origami-z @bhoppers2008


initial commands

It will be convenient for me to import a bunch of stuff in the ammonite REPL, but "initialCommand in console" is ignored inside the ammonite REPL

How are you running Ammonite?

In sbt, starting project/test:console
On Wed, Apr 22, 2015 at 9:19 AM Li Haoyi<EMAIL_ADDRESS>wrote:

How are you running Ammonite?
—
Reply to this email directly or view it on GitHub
https://github.com/lihaoyi/Ammonite/issues/58#issuecomment-95019133.

In the usual console, my imports work.
In my build.sbt I have:
initialCommands in console := """ammonite.repl.Repl.main(null); import
provingground._ ; import HoTT._"""
I also tried putting the ammonite.repl.Repl.main(null) command last.
On Wed, Apr 22, 2015 at 9:26 AM Siddhartha Gadgil <
<EMAIL_ADDRESS>wrote:

In sbt, starting project/test:console
On Wed, Apr 22, 2015 at 9:19 AM Li Haoyi<EMAIL_ADDRESS>wrote:

How are you running Ammonite?
—
Reply to this email directly or view it on GitHub
https://github.com/lihaoyi/Ammonite/issues/58#issuecomment-95019133.

Yeah that's not going to work =/ we don't grab things from the surrounding scope, even though I'd like to at some point. https://github.com/lihaoyi/Ammonite/pull/56 makes predefs slightly-more-but-not-quite possible, and with a bit of work should be sufficient

Fixed in d57ffac384f68c927b809c01b1f31c41d3ecb4c7, will push an updated version 0.3.0 some time

Thanks.
On Wed, Apr 22, 2015 at 11:27 AM Li Haoyi<EMAIL_ADDRESS>wrote:

Closed #58 https://github.com/lihaoyi/Ammonite/issues/58.
—
Reply to this email directly or view it on GitHub
https://github.com/lihaoyi/Ammonite/issues/58#event-286900740.


«where» clauses on contextually generic declarations

This proposal is about getting rid of the restriction on attaching where clauses to declarations that inherit the surrounding generic environment, but themselves do not introduce generic parameters. Only declarations that can already be generic fall under this enhancement.
class Foo<T> {
  func foo() where T: Equatable { ... }
}

Implementation: #23489

@DougGregor Would you like to manage this?

@tkremenek Is there an opportunity to stage this for review in the near future?

@tkremenek Is there any way I can reach someone from the core team for some feedback?


Adding travis for 3

Coverage increased (+0.09%) when pulling 8f9c292d604568f0e0bf2a024dfddc9dc0f3e5d7 on trustrachel:travis into b57555dc9113a5f5b60c914b02067335ff2a795d on linkedin:master.


Missing right click functionality

Is your feature request related to a problem? Please describe.
An important feature for desktop platforms is to be able to execute code if an element is clicked with the right button of the mouse.
This is, as far as I can tell, not possible currently.
Describe the solution you'd like
This can be easily implemented using the Listener class(from the library itself), implementing the onPointerDown method and checking if the device type is a mouse and the buttons field == kSecondaryMouseButton (aka 0x02).
The implementation is quite simple, but many widgets only support particular children types so you can't wrap them with a listener yourself(ex. DropDownButton which takes List)
Describe alternatives you've considered
An alternative is to make all widgets with a children field accept any widget instead of only particular types. This may be helpful even with this feature introduced as I explained.

Hello,
Maybe I am misunderstanding, but;

many widgets only support particular children types so you can't wrap them with a listener yourself

Why exactly would you want the Listener as a child of DropDownButton?
I was able to wrap a DropDownButton in a Listener and use the strategy you mentioned (device type is a mouse and the buttons field == kSecondaryMouseButton)

This produces a result like this:

If you are trying to make each dropdown item have a right click menu that can be accomplished as well like so:

@Auties00 what widgets are you talking about?

Hello,
Maybe I am misunderstanding, but;

many widgets only support particular children types so you can't wrap them with a listener yourself

Why exactly would you want the Listener as a child of DropDownButton?
I was able to wrap a DropDownButton in a Listener and use the strategy you mentioned (device type is a mouse and the buttons field == kSecondaryMouseButton)

This produces a result like this:
If you are trying to make each dropdown item have a right click menu that can be accomplished as well like so:

Oh yeah I guess that would work, I didn't think about that, but for example if you had a leading or a trailing you would have needed to duplicate the Listener which is kind of annoying. That's why i was proposing it as a first party feature, I could open a PR myself if it's something people want

@Auties00 for DropDownButton, specifically, you can create a new class that extends MenuFlyoutItem and use it on the dropdown button. Something like PaneItem.
Here's an example:
class CustomItem extends MenuFlyoutItem {

...

  Widget build(BuildContext context) {
    return Listener(
       child: super.build(context);
    );
  }

...

}

@Auties00 for DropDownButton, specifically, you can create a new class that extends MenuFlyoutItem and use it in the items. Something like PaneItem.
Here's an example:
class CustomItem extends MenuFlyoutItem {

...

  Widget build(BuildContext context) {
    return Listener(
       child: super.build(context);
    );
  }

...

}

yeah thanks. Sorry for the late response!


Suggest to add pre-commit for docstrings (numpy.doc)

I've noticed that a lot of good pull review comments are regarding the docstrings.
As we are using the numpydoc style: to make reviews easier, I suggest to add the numpy doc pre-commit hook (see https://numpydoc.readthedocs.io/en/latest/validation.html#).
I've checked the comments Tobias had in PR #885 , and the hook picked all of them up (and some others):
+----------------------------------------------------+----------------------------------------------------+---------+--------------------------------------------+
| file                                               | item                                               | check   | description                                |
+====================================================+====================================================+=========+============================================+
| simtools/applications/db_development_tools/add_mod | add_model_parameters_from_repository_to_db.add_val | PR01    | Parameters {'db_name'} not documented      |
| el_parameters_from_repository_to_db.py:75          | ues_from_json_to_db                                |         |                                            |
+----------------------------------------------------+----------------------------------------------------+---------+--------------------------------------------+
| simtools/applications/db_development_tools/add_mod | add_model_parameters_from_repository_to_db._add_me | PR01    | Parameters {'logger', 'db', 'db_name'} not |
| el_parameters_from_repository_to_db.py:110         | tadata_to_db                                       |         | documented                                 |
+----------------------------------------------------+----------------------------------------------------+---------+--------------------------------------------+
| simtools/applications/db_development_tools/add_mod | add_model_parameters_from_repository_to_db._add_me | PR02    | Unknown parameters {'db: DatabaseHandler'} |
| el_parameters_from_repository_to_db.py:110         | tadata_to_db                                       |         |                                            |
+----------------------------------------------------+----------------------------------------------------+---------+--------------------------------------------+

Note that depending on how many of the validation checks we allow, this might be a bit painful in the beginning. But I think it is worth to do it. Any objections?
(in general we don't spend enough time on the documentation, so this maybe forces us to do so)

If it helps you I have no objection. I think even private functions should be included (and documented). Unfortunately, there is no way to generate the documentation automatically ...

Ruff has pydocstyle integrated and allows and option lint.pydocstyle.convention = "numpy".
Looking through the documentation, it seems that this is enough and we do not need the additional numpydoc validation. I am therefore closing this issues (and open one on ruff documentation checks).


Sign the Eclipse ECA

Projects under the Eclipse Foundation Governance require that pull requests are coming from authors that have signed the Eclipse ECA https://www.eclipse.org/legal/ECA.php
This checks voids any contributions from dependabot as they are signed off, but the email address is not listed on the ECA database.
Project maintainers will in most cases, reject the PR and re-create a new one to ensure that the contract is valid.

Related: https://github.com/dependabot/dependabot-core/issues/3849

Closing as a duplicate of #3849.
Bots generally don't need to sign CLA/ECA's... the only question is how to guarantee the commit is coming from an approved bot (which is what #3849 tracks).


.Net: Get Method missing in Handlebars planner to parse JSON output

When using the Handlebars planner in semantic kernel, I noticed an issue when the planner was parsing the actual JSON output. For example: I have an API that calculates the price target for a given stock ticker. This API is documented in an OpenAPI spec. After loading the spec as a plugin using the ImportPluginFromOpenApiAsync method and asking the query "what is the price target of GOOGL" the planner produces this plan:
**Original plan:
{{!-- Step 1: Set the ticker symbol for which the price target is to be retrieved --}}
{{set "ticker" "GOOGL"}}
{{!-- Step 2: Use the custom helper to retrieve the price target for the specified ticker symbol --}}
{{set "priceTarget" (PriceTargetPlugin-getPriceTarget Ticker=ticker)}}
{{!-- Step 3: Output the result --}}
{{json (concat "The price target of " ticker " is: $" priceTarget)}}
Result:
The price target of GOOGL is: ${"value":180}**
The problem in the result is that the json helper is not getting the actual value, instead it is returning the actual json object.
Ideally it should look like: The price target of GOOGL is 180. I am using Semantic Kernel 1.0.1 in .NET 6.
Is there a setting allow getting the actual value, I have attempted to tweak the response section of my OpenAPI spec but it is still not retrieving the exact value.

Isn't this the same as this one: https://github.com/microsoft/semantic-kernel/issues/4596 ".Net: get helper needs to be added back to the handlebars planner #4596"?
(found this after being mentioned during yesterday's Office Hours)

This appears to be a different issue than #4596. I think the real issue is that we are not including the response schema for custom helpers in the handlebars prompt template, so the planner does not know how to parse data out of the response object. I am working on a fix for this.


Adding host gets stucked in "Provisioning with ubuntu(systemd)

Rancher versions:

rancher/server: 1.6.14
rancher/agent: 1.2.7

Infrastructure Stack versions:

healthcheck: v0.3.3
ipsec:v0.13.5
network-services:
rancher/metadata:v0.9.5
rancher/network-manager:v0.7.18
scheduler:v0.8.2
kubernetes (if applicable): rancher/k8s:v1.8.3

Docker version: (docker version,docker info preferred)
$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.03.2-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 23
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1049-aws
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.952 GiB
Name: ip-10-0-1-38
ID: 3NGX:SYKA:NDIT:MZDZ:7FDX:XN5J:4E5O:25CO:XHRI:WOEC:SATK:63ZT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false

WARNING: No swap limit support

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
$ uname -r
4.4.0-1043-aws

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
AWS - using t2.small instance for the Server.
The Rancher server is behind a classic ELB with SSL termination (on the ELB)
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)

1 Rancher Server
1 Rancher Node
External db (RDS instance)

Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:

Add a host to a Cattle environment
When adding hosts to the respective infrastructure, select the proposed (rancher-machine) security group for them

Results:
Process stucks in

Provisioning with ubuntu(systemd)...

Please report back so we can investigate.

I'm seeing the same issue that occurs only when trying to add a host with a private IP. Using ami-d15a75c7. Does not occur when using a public IP.

This issue

I'm seeing the same issue that occurs only when trying to add a host with a private IP. Using ami-d15a75c7. Does not occur when using a public IP.

Me too. Rancher and provisioning nodes are in same subnet but cluster nodes won't provision further than "Provisioning with ubuntu(systemd)" with only private IP addresses

Hey guys u have any idea... i am seeing the same issue.

@superseb please re-open this issue.
The main problem here as described by @rflume, @namehta and @shitalp is cluster creation stuck when using Private IP only using EC2 cluster.
I'm also facing the same issue and I really need to use only private IP for those instances. Please @superseb re-open so we can try to reach to a solution.

Hey @rflume, @namehta, and @shitalp again
reading more at the internet I found https://forums.rancher.com/t/add-clone-ec2-host-whit-no-public-ip/6657/2 where a Rancher Labs guy explain:

In my case this option b is the problem why my environment stucked (since I was using 3 subnets).
I hope this can help you guys !!

Same here. Stuck on Provisioning with ubuntu(systemd)...
Ubuntu 18.04


fix(KONFLUX-4292): upload a fixed gpg key id to pyxis

To fix KONFLUX-4175, we need to upload a gpg key id to pyxis.
The right way to do this is for the SBOM itself to contain the gpg key id. Our task here would simply copy it to pyxis.  However, to do that fully, we'll need to get a change into both syft and cachi2, which may take some time.
As a stopgap, this change inspects the "publisher" field of the sbom. If it is set to Red Hat, we'll publish the Red Hat release2 key id to pyxis.
You might ask, what if the rpms are not actually signed by Red Hat keys? We have a separate layer of protection against that in KONFLUX-3149, but the data about which rpms are signed by which keys will not be exposed by that work in the way that the release process can use here.

https://issues.redhat.com/browse/KONFLUX-4292
https://issues.redhat.com/browse/KONFLUX-4175

This change should be reverted as a part of completing KONFLUX-4175.

Yes, please. Merge when you're ready. I'll rebase now.


Compiler: allow assigning Proc(R) to Proc(Nil)

This PR:

reverts #7527
fixes #7698

Then it implements what #7527 was trying to accomplish in a different way.
What #7527 did was merging any two proc types with same argument types but different return types, where one return type was nil, to that return type with nil. Example:
a = ->{ 1 }
b = ->{ nil }  # Proc(Nil)
typeof(a)      # => Proc(Int32)
typeof(b)      # => Proc(Int32)
typeof(a || b) # => Proc(Nil)

The problem is, it's not good to discard the information returned by b.
What this PR does is just allowing the assignment of such procs to instance and class variables, and local variables with a type annotation. For example:
class Foo
  @block : -> Nil

  def initialize(block)
    @block = block
  end

  def block
    @block
  end
end

a = ->{ 1 }
b = ->{ nil }

Foo.new(a) # okay
Foo.new(b) # okay

c = a || b
typeof(c) # => Proc(Int32) | Proc(Nil)
Foo.new(c) # also okay!

Note that c preserves the real type. But when we assign it to the instance variable @block, which is already typed to be Proc(Nil), just then the information is lost. And this is fine because, since the return type is Nil there's no other thing to do than losing that information.
The compiler had no such logic for this before. The only thing the compiler did, all the time was:

if you assign to a variable, you make the union of the variable's existing type with the new type, then you assign that to the variable

So, this introduces new logic to apply just when assigning a type to a variable whose type is already determined by the user. This is the new assign_type method. You don't need to know about this. I'm just mentioning it here because this might feel like a cheap implementation. However, this is the only case need so far. If we need more special cases like this, we can refactor the code to make it a bit more generic.
Also, because the compiler will now try to do this when you specify a type, this now behaves like this:
def foo : Proc(Nil)
  ->{ 1 }
end

typeof(foo) # => Proc(Nil)

Which I think is good. If you explicitly declare the return type of the Proc to be Nil, the only reason you might want to do that (other than verify the type matches) is to return such type, not a type with more information in the return type. So this would be analogous to explicitly using the Nil return type.
Finally, I think this is important to have in the language because the case of wanting to store callbacks is very common. In many cases callbacks don't return anything, and forcing the users to type that extra nil inside their procs is not nice. Also, the compiler error of that is probably not very intuitive.

Does this work for container types too, like Array(-> Nil) being able to take a -> Int32? In effect, does this casting still work in type restrictions too?

Does this work for container types too, like Array(-> Nil) being able to take a -> Int32? In effect, does this casting still work in type restrictions too?

Apparently yes. There's also #8969 which should be merged too, but without that it still works for Arrays for some reason... I think it's because the restriction is T, then a different code path is taken and that codepath is already covered.


Xbox Series wireless controller detected as two devices

Godot version
3.5 RC4
System information
Windows 10, GLES3, Intel HD 620
Issue description
Xbox Wireless Controller gets detected as two controllers.
I detect connected controllers with Input, like so:
var joypads = Input.get_connected_joypads()
If I connect an Xbox Wireless controller i get two entries. One for an XInput Gamepad, and one for an Xbox Series Controller.
They both give input actions from the physical controller.
In a local multiplayer game this causes problems because one controller gets interpreted as two players, and thus the same physical controller controls two characters.
I have not hade this problem with my wired Xbox clone controller, my wireless 8Bitdo or my wired Xbox 360 clone controller.
Steps to reproduce
Connect an Xbox Wireless Controller (or maybe a wired, though I have not tested this).
Get a list of connected controllers using Input.get_connected_joypads()
See that the list contains two entries, instead of the expected one entry.
Minimal reproduction project
Xbox Controller detected twice.zip

I tested the same controller on the Gamepad Tester website, and it does not register as two devices there, but instead as a single "Xbox 360 Controller (XInput STANDARD GAMEPAD)"

Just tested on 4.0 beta 6 and the problem is the same.

I can confirm I'm having this issue in 4.0 beta 11

If I connect an Xbox Wireless controller i get two entries.

Are you using an USB adapter or Bluetooth?

The controller was connected to the computer via USB cable but was sending inputs via bluetooth. I just found that switching the usb port fixed the issue for me but I'm still not sure of the cause.

The controller was connected to the computer via USB cable but was sending inputs via bluetooth. I just found that switching the usb port fixed the issue for me but I'm still not sure of the cause.

This was likely the culprit, and I'm not sure if we can do anything about it.
@FarbrorMartin Can you check whether you were charging the controller while using it via Bluetooth?

I’m sorry but I returned the controller, and I don’t have another one like
it. (It had some  connection issues where it constantly lost BT-connection)
It was a Xbox wireless, so I assume I connected it via Bluetooth.
If I remember correctly it worked as it should when I tested it on that
js-based web page. Can’t remember the name right now. It’s for testing
controllers. I think that one is based off the same libraries as Godot’s
controller code.
SDL?
Maybe you could test that and see if it works?
On Fri, 13 Jan 2023 at 23:24, Hugo Locurcio @.***>
wrote:

The controller was connected to the computer via USB cable but was sending
inputs via bluetooth. I just found that switching the usb port fixed the
issue for me but I'm still not sure of the cause.
This was likely the culprit, and I'm not sure if we can do anything about
it.
@FarbrorMartin https://github.com/FarbrorMartin Can you check whether
you were had the controller connected via USB while using it via Bluetooth?
—
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/69257#issuecomment-1382460539,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIZHL64UN3RYTPPQOTKZKPLWSHI3HANCNFSM6AAAAAASMRHWSA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I tested the controller with both usb ports on gampad-tester.com and both times it detected the controller as xinput with index 0.

I have the same issue, but only when the controller is in Bluetooth mode. Hooking up the controller with a USB-C cable fixes the issue for me.

Note: This seems like a duplicate of #41684.

Duplicate of https://github.com/godotengine/godot/issues/41684.


Add shadows to thumbnails

What feature or improvement do you think would benefit Files?
Right now Files app does not display any shadows on thumbnails, and this causing white images to almost blend with background:

Windows Explorer shows shadows on thumbnails, preventing thumbnail blend with background, and also looks nice:

Requirements

Shadows are added to thumbnails.

Files Version
<IP_ADDRESS>
Windows Version
10.0.22631.3447
Comments
No response

Thanks for the feedback, I've added this to the project board.

It's my own opinion but I feel these shadows don't work well with fluent design.

I think this should be an option (so I can disable it, since I think it's not look good on fluent design).

Instead of a drop shadow like Windows File Explorer a thin all sided border could probably be used.


[Suggestion/Enhancement] Ability to get more SoulBlight Mushrooms

Currently, Soulblight mushrooms are non-renewable and are only obtainable through the naturally generated mushrooms in the biome.
Perhaps the Blightwart should have a small chance of dropping more mushrooms...
or alternatively, have the player use bonemeal on Soul Soil to get both the mushrooms and the foliage found in the Soulblight Forest

Fixed in 2.0.0. Bonemealing Soulblight Sprouts will now result in a Soulblight Fungus. Soulblight sprouts are renewable by growing giant soulblight fungi.


Fix issue with iOS7 and status bar

@samvermette @calebd @rocket0423 @getaaron
Another fix for the issue with iOS7 and the status bar. This worked for me

@borisoks, you are a genius. You solved my problem!!!! Thank you :)


Peering Request

Full Name
Enki Multimedia
ASN
209823
PeeringDB Integration

[X] My PeeringDB Entries are current
[X] Peer at ALL EXCHANGES (and skip the Peer Address(es) field below)

Peer Address(es)
No response
Multihop

[ ] Multihop Peering Request

Hi Nina,
Thanks for the answer. Yes It's OK for me. I will setup the connection as well on my side
Benoit

Hi Benoit
I have configured our side as well.  I have nexthop_set failed messages in my log.  Do you see anything in your log?  I notice a 90+ ms RTT when I ping your IP address.

@nihb333 this should be OK now :
decix_nyc_routeviews_6447_ipv4 BGP        ---        up     23:28:21.379  Established
decix_nyc_routeviews_6447_ipv6 BGP        ---        up     23:28:22.282  Established
sorry for the delay, I got sidetracked.

perfect - thank you!


Error in browser: Uncaught ReferenceError: global is not defined

Lolex version : 2.7.1
Environment   : Chrome 68.0.3440.106, Windows
Other libraries you are using: no

What did you expect to happen?
expet to have window.lolex to be accesible
What actually happens
error at:
var defaultImplementation = withGlobal(global || window);
How to reproduce
to include tag into page's head
<script<EMAIL_ADDRESS>crossorigin></script>
even if to patch this
global = window
next error occures at
exports

Checking.

PR at https://github.com/sinonjs/lolex/pull/199 - sorry for this! Thanks for the report.

You need<EMAIL_ADDRESS>and not<EMAIL_ADDRESS>

thanks, it solves the problem!


[Bug]:

What happened?
When I deploy the Chroma vector service through an interface, there is too much vector data. I need to delete Chroma.sqlite3 and other files in the persist-directory directory. However, after deleting the files, when calling the interface again, the following error will occur: OperationalError: attempt to write a readonly database``
import json
import requests
import re
from log import LOGGER
from flask import request, Flask
import os
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.docstore.document import Document
from urllib.parse import urlparse

DB_PATH = "/opt/model/db"
DEVICE = 'cuda'
BERT_MODEL = "/opt/model/models--infgrad--stella-base-zh-v3-1792d"
EMB_FUNC = HuggingFaceEmbeddings(model_name=BERT_MODEL,
                                 model_kwargs={'device': DEVICE},
                                 encode_kwargs={'normalize_embeddings': True}
                                 )

def add_files_to_db(raw_docs, collection_name):
    documents = [Document(page_content=raw_docs, metadata={'source': collection_name})]
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=50)
    docs = text_splitter.split_documents(documents)
    db = Chroma.from_documents(docs, EMB_FUNC, persist_directory=DB_PATH, collection_name=collection_name)
    db.persist()

def query_db(db: Chroma, query: str = ""):
    query_emb = EMB_FUNC.embed_query(query)
    docs = db.similarity_search_by_vector(query_emb, k = 15)
    docs_dict = [doc.page_content for doc in docs]
    return docs_dict

app = Flask(__name__)
@app.route('/retrieval', methods=['POST'])
def coder_write():
    try:
        param = request.get_json()
    except:
        param = request.form
    query = param['query']
    parse_doc = param['doc']
    flag = param['flag']
    LOGGER.info(f"flag值：{flag}")
    session_id = param['session_id']
    collection_name = session_id
    if flag == '1':
        add_files_to_db(parse_doc, collection_name)
        db = Chroma(persist_directory=DB_PATH, embedding_function=EMB_FUNC, collection_name=collection_name)
        docs = query_db(db, query)
        LOGGER.info(f"相似性度量后的docs：{docs}")
    else:
        db = Chroma(persist_directory=DB_PATH, embedding_function=EMB_FUNC, collection_name=collection_name)
        docs = query_db(db, query)
        LOGGER.info(f"第二轮问答相似性度量后的docs：{docs}")

    search = [(i, j) for i, j in enumerate(docs)]
    r = ''
    for i, j in search:
        r += f"""\n（{i+1}）{j}"""
    result = f"{r}"
    LOGGER.info(f"retrieval最终return的结果：{result}")
    return {"status": 1, "result": result}

if __name__ == "__main__":
    IP = '<IP_ADDRESS>'
    app.run(port='35010', host = IP, threaded = False)`

Versions
chroma0.5.23
Relevant log output
File /opt/miniconda/lib/python3.10/site-packages/chromadb/db/mixins/embeddings_queue.py:243, in SqlEmbeddingsQueue.submit_embeddings(self, collection_id, embeddings)
    240 # The returning clause does not guarantee order, so we need to do reorder
    241 # the results. https://www.sqlite.org/lang_returning.html
    242 sql = f"{sql} RETURNING seq_id, id"  # Pypika doesn't support RETURNING
--> 243 results = cur.execute(sql, params).fetchall()
    244 # Reorder the results
    245 seq_ids = [cast(SeqId, None)] * len(
    246     results
    247 )  # Lie to mypy: https://stackoverflow.com/questions/76694215/python-type-casting-when-preallocating-list

OperationalError: attempt to write a readonly database

@4AM-GodVZz, how do you run the above flask app? If you use multiple workers, it is possible that two concurrent calls run in those workers. Chroma is not process safe and the way that workers operate in uvicorn or gunicorn is by spawning a separate process. Therefore concurrent calls will likely end up in the error above.
The root cause is sqlite3 not being process safe and one process holding exclusive lock on the DB.
If on the other hand you are not running with multiple workers, is it possible that another process (outside of the flask app) is accessing Chroma's dir - DB_PATH?

Hey! I see what's happening here. The issue occurs because of how Chroma handles database access and file permissions. Let me help you resolve this.
First, there are a few important things to note:

Chroma is thread-safe but not process-safe
After deleting the SQLite files, proper cleanup and initialization is crucial

Here's how to fix this:

First, ensure proper cleanup when deleting the database:

import shutil
import os

def clean_chroma_db():
    if os.path.exists(DB_PATH):
        # Close any existing connections
        try:
            db = Chroma(persist_directory=DB_PATH, embedding_function=EMB_FUNC)
            db.persist()
            del db
        except:
            pass
        # Remove the directory
        shutil.rmtree(DB_PATH)
        # Ensure the base directory exists
        os.makedirs(DB_PATH, exist_ok=True)

Modify your Flask application to use a single client instance:

# At the module level
_db_client = None

def get_db_client():
    global _db_client
    if _db_client is None:
        _db_client = Chroma(persist_directory=DB_PATH,
                           embedding_function=EMB_FUNC)
    return _db_client

@app.route('/retrieval', methods=['POST'])
def coder_write():
    # ... your existing code ...
    if flag == '1':
        add_files_to_db(parse_doc, collection_name)
        db = get_db_client()  # Use the singleton client
        docs = query_db(db, query)
    else:
        db = get_db_client()  # Use the singleton client
        docs = query_db(db, query)

Important considerations for deployment:

If you're using multiple workers (gunicorn/uvicorn), set workers=1
For scaling, consider using a connection pool or a separate Chroma server instance

The key changes here:

Using a singleton pattern for the Chroma client
Proper cleanup when deleting the database
Ensuring proper file permissions after recreation

Based on similar issues (see chromadb#1441), this approach should resolve the readonly database error.
If you're still seeing issues, could you let me know:

Are you running multiple workers in your Flask deployment?
What are the file permissions on your DB_PATH directory after deletion?

References:

Issue #1441: Process safety and client management
Issue #1209: Multiple client access patterns
Chroma documentation on client usage: https://docs.trychroma.com/usage-guide

Let me know if this helps or if you need any clarification!


HSEARCH-3849 Create-or-validate schema management by default

https://hibernate.atlassian.net/browse/HSEARCH-3849
Based on #2230, which should be merged first.

Merged, thanks!


Use NotNan<f64> to implement Eq, Hash, PartialOrd and Ord for Number, Map and Value.

Float values in JSON can never be NaN, and thus can always be compared and hashed. I exploited this property with the ordered-float crate to implement Eq, Hash, PartialOrd and Ord for the Number, Map and Value types.

Oh, something went wrong...

I see, my bad, I haven't been very thorough with my testing.

The issue is that IndexMap is a hash map implementation, contrarily to BTreeSet, so it cannot implement Ord nor Hash (well it could implement Hash but I think they are waiting to copy the standard HashSet Hash impl which has not yet been determined).
There is no easy solution for this, so for now I feature gated my contribution so that Value only implement Ord and Hash when the preserve_ord feature is disabled (which is kind of ironic for Ord). I still think it is better than nothing.

Right, I see how that could be problematic. Then the only solution would be some kind of BTreeIndexMap as discussed in https://github.com/bluss/indexmap/issues/199.


Introducing classes to track issues

When working on a project using Pylasu I noticed the need for a few supporting classes, that I introduced in that project but which I think it would be useful to have in Pylasu.
I am adding that here, but I a Pylasu newbie, so a review would be welcome.
I am asking a review to @alessiostalla as I got "inspiration" from what he was doing in a project

Looks good to me, @loradd is the maintainer of Pylasu so I'd let him review and merge

Thank you @alessiostalla !

I should learn some Python :)
I made another attempt, by making WithIssues a dataclass. When doing so, if I defined the value (as field(default_factory=list) for example) then the subclasses complain as it was a field with a default value preceeding fields with a non-default value, so I defined the value of issues in subclasses. Does this make sense?
I thought that if WithIssues was not marked as a dataclass then issues would not have been considered a dataclass field in subclasses, but I could well be wrong

I should learn some Python :) I made another attempt, by making WithIssues a dataclass. When doing so, if I defined the value (as field(default_factory=list) for example) then the subclasses complain as it was a field with a default value preceeding fields with a non-default value, so I defined the value of issues in subclasses. Does this make sense? I thought that if WithIssues was not marked as a dataclass then issues would not have been considered a dataclass field in subclasses, but I could well be wrong

IMHO this kinda defeats the point of WithIssues, because it forces you to redeclare the field in subclasses.
To avoid the problem you could mark the issues field as init: False:
    issues: List[Issue] = field(default_factory=list, init=False)

This won't include the field in the generated constructor, and it will make it possible for subclasses to have mandatory fields. If for a certain subclass one wants to include the issues list in its constructor for whatever reason, they can redeclare the field only for that specific subclass.

Thank you @alessiostalla for the suggestion!

Thank you!
Would it be possible to do a release, so I can use these changes on the PDF Association project?


test showing alias matching is case-sensitive

this is wrong according to JPQL

Ugh, this is because I wasn't in JPA compliance mode, and I was mislead by the old docs.


Access time/Modification time check breaks file watching with high latency file uploads

Describe the bug
We use Rsync to copy files from our local machines to a remote host.  Using Rsync -apz includes copying local access time and modification time up to the remote host.  As of the change in 2.1.1 the Atime and Mtime when Rsyncing are high enough to skip the this._emit('change', file, newStats);
Versions (please complete the following information):

Chokidar version 2.1.1
Node version 8.x.x and 11.x.x tested
OS version: Ubuntu 16.04

To Reproduce
We are currently running webpack --watch on an EC2 instance and using rsync -apz to copy files up.  Changing to rsync -rlopgDz stops syncing the time stamps and the problem gets resolved.
Expected behavior
I would expect that these files would be treated as changed as they were in prior releases.

@markablov we need to resolve the issue or have the change reverted

@paulmillr i am going to track previous stats object (prevStats = stats) and add additional check if prevStats.mtime != newStats.mtime so it should handle such situations. Sounds ok to you?

yes

@paulmillr https://github.com/paulmillr/chokidar/pull/800
@jeremycondon issue is fixed, but i have noticed another caveat with rsync approach:

Modify file and make rsync
chokidar fires change event
Again modify file and perform another rsync
chokidar is not receiving FS notification (fs.watch is not calling callback).

May be it's because i was using local directories as source and destination for my tests, but still want to warn you about this ;)

Much appreciated!  Nice PR on that.  I'll let my team know to keep an eye out for issues like that, I haven't heard any issues on my side yet.


Fix installation docu in README.md

Fix a small glitch in the installation instruction
What this PR does / why we need it:
There is a glitch in the installation instruction.
Which issue(s) this PR fixes:
Fixes the glitch.
Special notes for your reviewer:
Release note:
NONE

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thank you @welfwalter for your contribution. Before I can start building your PR, a member of the organization must set the required label(s) {'reviewed/ok-to-test'}. Once started, you can check the build status in the PR checks section below.


visibleTextRange made public

Can you remind me why you need this? I'm slightly worried about conflicts, as this library really isn't supposed to be messing with NSTextView.

I have implemented my own highlight handler that conforms to TextSystemInterface. Need access to visibleRect

Right right!


Fixed flags position

@sergak01 thank you for the PR! What did it add to the component? I see that you've closed it, but if you think that's useful please let's discuss about it 😺

@P3trur0 I tried to correct the position of the flag in the text, but it did not work. My code corrected only the wrong height of the block (when the “size” property is “small” or “normal” - the flag received an implicit height of 64px)


Cannot find Suggest Topics button

Is it in the new Microsoft Copilot? Clicking on Topics & Plugins tab, I only see Topics, Plugin actions (preview), System

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c78f1747-04bf-3bb7-c1e1-9025710e022f
Version Independent ID: 1da1f8d8-f0e5-64e1-7503-39973b0f38e2
Content: Get topic suggestions from online content (contains video) - Microsoft Copilot Studio
Content Source: power-virtual-agents/advanced-create-topics-from-web.md
Service: copilot-studio
GitHub Login: @iaanw
Microsoft Alias: iawilt

Thanks for your comment, @jxu. This issue has been assigned to the doc author for review.

I know, you told me that for the last two issues and they haven't been reviewed yet.


Invalid client id

Describe the bug
Can't download anymore in archive mode even in download mode cause Invalid client id
To Reproduce
Steps / command to reproduce the behavior:
:: An error occurred: Invalid client id (https://cr-play-service.prd.crunchyrollsvc.com/v1/G9DUE5Q41/console/switch/play)
Errore nel download del file: Command 'C:\crunchyrust\crunchy.exe archive --include-fonts --include-chapters --merge audio -s en-US -o "[CR] {series_name} S{season_number}E{episode_number} {title}.mkv" https://www.crunchyroll.com/series/GKEH2G0P9/stardust-telepath' returned non-zero exit status 1.

Client (please complete the following information):

OS: Windows
Version [v3.4.3]

Same here.

Same here. I tried the v3.4.3 and v3.4.2 version

Maybe DRM-FREE endpoint is dead

I can also confirm with the current version.
Running in PowerShell, error occurs in both archive and download.
PS C:\Users\brent> crunchy-cli archive -a ja-JP -a en-US -o "{series_name} [S{season_number}E{episode_number}].mkv" https://www.crunchyroll.com/series/G69PZ5PDY/overlord
:: — Logging in                                                                                :: \ Logging in                                                                                :: | Logging in                                                                                :: / Logging in                                                                                :: — Logging in                                                                                :: \ Logging in                                                                                :: | Logging in                                                                                :: √ Logging in                                                                                :: — Parsing url 1                                                                                :: \ Parsing url 1                                                                                :: | Parsing url 1                                                                                :: / Parsing url 1                                                                                :: — Parsing url 1                                                                                :: \ Parsing url 1                                                                                :: | Parsing url 1                                                                                :: / Parsing url 1                                                                                :: — Parsing url 1                                                                                :: \ Parsing url 1                                                                                :: | Parsing url 1                                                                                :: √ Parsed url 1                                                                                :: — Fetching series details                                                                                :: \ Fetching series details                                                                                :: → Series Overlord is not available with ar-ME, ar-SA, ca-ES, de-DE, en-IN, es-ES, es-LA, fr-FR, hi-IN, id-ID, it-IT, ja-JP, ko-KR, ms-MY, pl-PL, pt-PT, ta-IN, te-IN, th-TH, tr-TR, vi-VN, zh-CN, zh-CN, zh-TW subtitles
:: \ Fetching series details                                                                                :: | Fetching series details                                                                                :: / Fetching series details                                                                                :: — Fetching series details                                                                                :: \ Fetching series details                                                                                :: | Fetching series details                                                                                :: / Fetching series details                                                                                :: — Fetching series details                                                                                :: \ Fetching series details                                                                                :: | Fetching series details                                                                                :: / Fetching series details                                                                                :: — Fetching series details                                                                                :: \ Fetching series details                                                                                :: | Fetching series details                                                                                :: / Fetching series details                                                                                :: — Fetching series details                                                                                :: \ Fetching series details                                                                                :: √ Loaded series information for url 1                                                                                :: Overlord Season 2 (Overlord II)
::      1. The dawn of Despair » S02E01
::      2. Departure » S02E02
::      3. Lizard men, gathering » S02E03
::      4. Army of Death » S02E04
::      5. The freezing god » S02E05
::      6. Those who pick up, those who are picked up » S02E06
::      7. Blue roses » S02E07
::      8. A boy's feeling » S02E08
::      9. Soaring sparks of fire » S02E09
::      10. Disturbance begins in the royal capital » S02E10
::      11. Jaldabaoth » S02E11
::      12. The final Battle of the disturbance » S02E12
::      13. The ultimate trump card » S02E13
:: Overlord Season 3 (Overlord III)
::      1. A Ruler's Melancholy » S03E01
::      2. Carne Village Once More » S03E02
::      3. Enri’s Upheaval and Hectic Days » S03E03
::      4. Giant of the East, Demon Snake of the West » S03E04
::      5. Two Leaders » S03E05
::      6. Invitation to Death » S03E06
::      7. Butterfly Entangled in a Spider’s Web » S03E07
::      8. A Handful of Hope » S03E08
::      9. War of Words » S03E09
::      10. Preparation for War » S03E10
::      11. Another Battle » S03E11
::      12. Massacre » S03E12
::      13. Player vs Player » S03E13
:: Overlord Season 4 (Overlord IV)
::      1. Sorcerer Kingdom Ains Ooal Gown: Ains Ooal Gown Nation of Leading Darkness » S04E01
::      2. Re-Estize Kingdom » S04E02
::      3. Baharuth Empire » S04E03
::      4. The Ruler of Conspiracy » S04E04
::      5. In Pursuit of the Land of Dwarves » S04E05
::      6. The Impending Crisis » S04E06
::      7. Frost Dragon Lord » S04E07
::      8. An Unexpected Move » S04E08
::      9. Countdown to Extinction » S04E09
::      10. The last king » S04E10
::      11. Well-prepared Traps » S04E11
::      12. Invasion of the royal capital » S04E12
::      13. The Witch of the Falling Kingdom » S04E13
:: An error occurred: Invalid client id (https://cr-play-service.prd.crunchyrollsvc.com/v1/G6K503J9Y/console/switch/play)
PS C:\Users\brent> crunchy-cli archive -a ja-JP -a en-US -o "{series_name} [S{season_number}E{episode_number}].mkv" https://www.crunchyroll.com/watch/GN7UDGW07/the-dawn-of-despair
:: — Logging in                                                                                :: \ Logging in                                                                                :: | Logging in                                                                                :: / Logging in                                                                                :: — Logging in                                                                                :: \ Logging in                                                                                :: | Logging in                                                                                :: √ Logging in                                                                                :: — Parsing url 1                                                                                :: \ Parsing url 1                                                                                :: | Parsing url 1                                                                                :: / Parsing url 1                                                                                :: — Parsing url 1                                                                                :: \ Parsing url 1                                                                                :: √ Parsed url 1                                                                                :: — Fetching series details                                                                                :: \ Fetching series details                                                                                :: | Fetching series details                                                                                :: → Episode 1 is not available with ar-ME, ar-SA, ca-ES, de-DE, en-IN, es-ES, es-LA, fr-FR, hi-IN, id-ID, it-IT, ja-JP, ko-KR, ms-MY, pl-PL, pt-PT, ta-IN, te-IN, th-TH, tr-TR, vi-VN, zh-CN, zh-CN, zh-TW subtitles
:: | Fetching series details                                                                                :: √ Loaded series information for url 1                                                                                :: Overlord Season 2 (Overlord II)
::      1. The dawn of Despair » S02E01
:: An error occurred: Invalid client id (https://cr-play-service.prd.crunchyrollsvc.com/v1/G6K503J9Y/console/switch/play)

Same here

This error differs from the DRM problems we already had. Luckily I have a unpatched switch that is able to run CFW, I'll try if I can intercept the switchs' traffic and see what the requirements for a correct client id are.

Same here

The issue got fixed in v3.5.0.
If you've used a package manager to install crunchy-cli, it might take some time until the update is available for it.

@bytedream @frootaside we could use some help over at https://github.com/yt-dlp/yt-dlp/issues/9745 with a similar issue if either of y'all have the bandwidth 🙏🏻


Add [dm|fs] verity support to puzzlefs

We need some research into https://github.com/containers/composefs first and then reason how to go about this.

From: https://lwn.net/Articles/917416/

As I said above, I don't think fs-verity actually works if underlayfs doesn't support fs-verity, which also includes one of composefs example -- FUSE.

https://www.spinics.net/lists/linux-fscrypt/msg06900.html

Although FUSE lacks the support of "unrestricted" ioctl, which makes it impossible for the filesystem to receive the fs-verity ioctls. Same to statx.  I think that's where we'd need a change in FUSE protocol.

Background
From https://www.kernel.org/doc/html/next/filesystems/fuse.html
fuseblk
The filesystem is block device based. The first argument of the mount system call is interpreted as the name of the device.

From https://manpages.ubuntu.com/manpages/xenial/man8/mount.fuse.8.html
blkdev Mount a filesystem backed by a block device.  This  is  a  privileged  option.  The
              device must be specified with the fsname=NAME option.

Setup
$ dd if=/dev/zero of=backing-fs bs=4K count=4000
$ sudo losetup --find --show backing-fs
/dev/loop1
$ sudo mkfs -t ext4 -F -b4096 /dev/loop1

puzzlefs mount (see notes):
sudo target/debug/puzzlefs mount -f -o blkdev,fsname=/dev/loop1,allow_other /tmp/oci-simple first_try /tmp/puzzle

Conclusions
Using fuseblk requires:

privileged operation
an existing block device

So it doesn't seem we could use dm-verity with puzzlefs for now.
Other options:
buse? (it's experimental and not production-ready)
Notes
Patch to fuser
fuser running as root will bypass the fusermount call, this patch will prevent it from doing it
diff --git a/src/mnt/fuse_pure.rs b/src/mnt/fuse_pure.rs
index ea366df..31037c6 100644
--- a/src/mnt/fuse_pure.rs
+++ b/src/mnt/fuse_pure.rs
@@ -84,13 +84,13 @@ fn fuse_mount_pure(
         return fuse_mount_fusermount(mountpoint, options);
     }

-    let res = fuse_mount_sys(mountpoint, options)?;
-    if let Some(file) = res {
-        Ok((file, None))
-    } else {
+    // let res = fuse_mount_sys(mountpoint, options)?;
+    // if let Some(file) = res {
+    //     Ok((file, None))
+    // } else {
         // Retry
         fuse_mount_fusermount(mountpoint, options)
-    }
+    // }
 }

 fn fuse_unmount_pure(mountpoint: &CStr) {

dmverity on block devices:
#!/bin/bash
set -e
set -x

backing_file=fake-fs
devname=ariel-device

dd if=/dev/zero of="$backing_file" bs=4K count=4000
loopback_dev=$(sudo losetup --find --show "$backing_file")
sudo mkfs -t ext4 -F -b4096 "$loopback_dev"
sudo mount "$loopback_dev" /mnt
echo 'ana are mere' | sudo tee -a /mnt/foo
echo 'dar nu are pere' | sudo tee -a /mnt/bar
sudo umount /mnt
root_hash=$(sudo veritysetup format "$loopback_dev" mydev.hash | grep "Root hash:" | awk '{print $NF}')
sudo veritysetup open "$loopback_dev" "$devname" mydev.hash "$root_hash"
sudo mount /dev/mapper/"$devname" /mnt

Proposal for fs-verity:

puzzlefs build generates an array of [blob1-shasum, fs-verity-root-hash] and appends it to the puzzlefs image manifest (the file which index.json points to and which has media_type = application/vnd.puzzlefs.image.rootfs.v1).
puzzlefs enable-verity    will be a new subcommand which compares the measured fs-verity root hash of the image manifest with the one passed on the command line, enables fs-verity on the image manifest, reads the array of [blob1-shasum, fs-verity-root-hash] and enables fs-verity for each file in the array.
puzzlefs mount will take another argument which will dictate how to use fs-verity (enabled or disabled); if it's enabled, then it will also take an additional fs-verity-root-hash argument which contains the root hash of the puzzlefs image manifest, issue an ioctl requesting the fs-verity root hash of the image manifest, compare the two and exit if there's a mismatch; puzzlefs mount will also issue an ioctl requesting the fs-verity root hash of each blob that it opens and it will compare it to the fs-verity-root-hash stored in the image manifest; if it doesn't match, then it will return an error (TBD: if we want to exit the process instead or make this behavior configurable).

fsverity configuration
Check fsverity feature in the kernel:
$ rg -z CONFIG_FS_VERITY /proc/config.gz
9987:CONFIG_FS_VERITY=y
9988:# CONFIG_FS_VERITY_DEBUG is not set
9989:CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y

Enable fs-verity on ext4 filesystem:
ariel@selected-chipmunk:~$ mount | grep ext4
/dev/sda1 on / type ext4 (rw,relatime,discard,errors=remount-ro)
ariel@selected-chipmunk:~$ sudo tune2fs -O verity /dev/sda1
tune2fs 1.46.5 (30-Dec-2021)
ariel@selected-chipmunk:~$ sudo tune2fs -l /dev/sda1 | grep -i verity
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum verity

Enable fsverity on a file:
echo lorem ipsum > immutable
# get the hash without enabling fsverity
ariel@selected-chipmunk:~$ fsverity digest immutable
sha256:fda0e2d973b5c81e4a8ccc43a0bb13cbc9b5da8d83c4ff4416f6860ea20df0d6 immutable
# enable fsverity
ariel@selected-chipmunk:~$ fsverity enable immutable
# get the root hash
ariel@selected-chipmunk:~$ fsverity measure immutable
sha256:fda0e2d973b5c81e4a8ccc43a0bb13cbc9b5da8d83c4ff4416f6860ea20df0d6 immutable

Test fsverity:
ariel@selected-chipmunk:~$ echo dolor sit amet >> immutable
-bash: immutable: Operation not permitted

Overwrite the file by modifying the blocks directly:
ariel@selected-chipmunk:~$ sudo debugfs -R "stat /home/ariel/immutable" /dev/sda1
Inode: 260139   Type: regular    Mode:  0664   Flags: 0x180000
Generation:<PHONE_NUMBER>    Version: 0x00000000:00000001
User:  1001   Group:  1001   Project:     0   Size: 12
File ACL: 0
Links: 1   Blockcount: 16
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x63ff2716:dbec86d8 -- Wed Mar  1 10:21:10 2023
 atime: 0x63ff26f0:10cff610 -- Wed Mar  1 10:20:32 2023
 mtime: 0x63ff2716:dbec86d8 -- Wed Mar  1 10:21:10 2023
crtime: 0x63ff26d6:e1b44e64 -- Wed Mar  1 10:20:06 2023
Size of extra inode fields: 32
Inode checksum: 0x2d3e414c
EXTENTS:
(0):394785, (16):25551

Get block size:
ariel@selected-chipmunk:~$ sudo dumpe2fs -h /dev/sda1 | grep "Block size"
dumpe2fs 1.46.5 (30-Dec-2021)
Block size:               4096

Check that we can read the contents of the file by reading directly from the disk:
ariel@selected-chipmunk:~$ sudo dd if=/dev/sda1 of=success.txt bs=1 count=12 skip=$((4096*394785))
10+0 records in
10+0 records out
10 bytes copied, 0.000388477 s, 25.7 kB/s
ariel@selected-chipmunk:~$ cat success.txt
lorem ipsum
ariel@selected-chipmunk:~$

Now modify the file by writing directly to the disk:
ariel@selected-chipmunk:~$ sudo dd of=/dev/sda1 if=/dev/zero bs=1 count=1 seek=$((4096*394785))
1+0 records in
1+0 records out
1 byte copied, 0.00350661 s, 0.3 kB/s

And check that the first byte is 0:
ariel@selected-chipmunk:~$ sudo dd if=/dev/sda1 of=success.txt bs=1 count=12 skip=$((4096*394785))
12+0 records in
12+0 records out
12 bytes copied, 0.000639262 s, 18.8 kB/s
ariel@selected-chipmunk:~$ hexdump -C success.txt
00000000  00 6f 72 65 6d 20 69 70  73 75 6d 0a              |.orem ipsum.|
0000000c

Now check that we can't open immutable:
ariel@selected-chipmunk:~$ cat immutable
lorem ipsum

Huh? Oh, we need to clear the page cache:
sync; echo 1 | sudo tee /proc/sys/vm/drop_caches

Now we shouldn't be able to open the file:
ariel@selected-chipmunk:~$ cat immutable
cat: immutable: Input/output error

It fails with EIO as mentioned here