feat(todoist): Add task from project + task actions for detail

Description

Added the ability to add a new task from a project and a project's empty state
Shared the task actions between the task detail and the task list item

Screencast
Add new task from project
https://user-images.githubusercontent.com/16003285/168386178-3f4e66c1-4aee-4e42-8e9d-fd14162a7360.mp4
Actions for task detail

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

It seems there's a bug in the PR. If I update a task from the task detail view, it's not updated afterward but the list item is:
https://user-images.githubusercontent.com/16003285/168387631-105af508-a594-47ae-ac8a-9830623eaa73.mp4
It may be a bug on Raycast's end. It seems that props passed in a target prop through an Action.Push are not updated correctly.
What do you think? Am I doing something wrong?

Hi @thomaslombart 👋
Thanks for this update, we'll take a look at the possible bug and get back! I'll wait with the merge until we have some information about it

So it's not a bug per se, but I can see how it can be misconceived. Basically when the Action.Push is triggered, it takes the component that is the target at that point and push it to the navigation stack.
It means that the target isn't a child of the List and so it doesn't get re-rendered when the list is updated.
I guess using the useNavigation and push method directly makes it more obvious.
So maybe, to solve your specific issue, you could create the Detail from just an id (that's not mutated) so that when the data is updated, the detail is updated as well?

Thanks for the answer. It makes total sense!
I'll look into that then.

A small comment to let you know that I haven't had the time to make the modifications. I still have that in mind, though. Hopefully, I'll get it done this week 🙂

@pernielsentikaer It's ready for review again 🙂
The only downside in fixing the above is that there's a slight loading for each task detail. I intend to fix that by adding persistent cache next week 🚀

@pernielsentikaer Under the hood, SWR is used to fetch the different pieces of data including the task list which shares the same key (Tasks) across the different commands. So when you change projects, SWR will use the key to display what it had in cache (the previous task list or nothing for the first load), then revalidate and show you the right list. This is why you have this "flickering" list.
This is indeed a bug but it has been there since the beginning. I'm happy to fix it with the persistent cache update as it'll be closely tied to how SWR is used in the extension. Should I do it in this PR? 🙂


can't find the right one using falconn v 1.3.1

I built a hash table with about 20,000 data, feature dimension is 256, but when using find_k_nearest_neighbors ,it can't find the right one;
and when using [get_unique_candidates] , it may return 2000~4000 candidates, including the right one ;
still i search all the data one by one ,
the final result is same as get_unique_candidates.
For example
When using find_k_nearest_neighbors (in our case k =9) can not find the shortest distance FuMingming, and may return 9 candidates with distance 0.66~0.7;  so the right one is missed;
while using get_unique_candidates or get_duplicate_candidates it can find the right one ;

Internally, find_k_nearest_neighbors relies on the methods for finding candidates. So in principle what you described shouldn't happen.
Just to clarify: what distance function are you using when you find the nearest neighbor yourself among the candidates returned by get_unique_candidates?

I use Euclidean distance，for two normalized vector v1 v2，||v1-v2|| is calculated .

FALCONN can certainly miss the nearest neighbor. But if the nearest neighbor appears in the candidate set, FALCONN should find it.
What distance function are you using when you set up the FALCONN data structure?

I used parameter as follows:
params_hp.dimension = vec_len_;
params_hp.lsh_family = LSHFamily::CrossPolytope;
params_hp.distance_function = DistanceFunction::EuclideanSquared;
params_hp.storage_hash_table = StorageHashTable::BitPackedFlatHashTable;
params_hp.k =2;
params_hp.l = 10;
params_hp.feature_hashing_dimension = 8;
params_hp.last_cp_dimension = 8;
params_hp.num_rotations = 3;
params_hp.num_setup_threads = 0;
params_hp.seed = seed ^ 833840234;
and set_num_probes(2464);


App

New Regression Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Regression Information

Breaking version: [e.g. 2.x.x]
Last working version: [e.g. 2.x.x]

Regression Description

Complete output when running fastlane, including the stack trace and command used

[INSERT OUTPUT HERE]

Environment

[INSERT OUTPUT HERE]

@engynzss Hi, I'm closing this issue as description is empty. Please open a new one with description of an issue if needed.


Cannot import scanpy: "PackageNotFoundError: umap-learn"

Hey!
I just updated to latest master branch and I can no longer load scanpy.
import scanpy as sc gives me the error:
PackageNotFoundError                      Traceback (most recent call last)
<ipython-input-1-0074c9bc0b31> in <module>
----> 1 import scanpy as sc

~/new_scanpy/scanpy/scanpy/__init__.py in <module>
     25 __version__ = get_versions()['version']
     26
---> 27 check_versions()
     28 del get_versions, check_versions
     29

~/new_scanpy/scanpy/scanpy/utils.py in check_versions()
     38
     39     anndata_version = version("anndata")
---> 40     umap_version = version("umap-learn")
     41
     42     if anndata_version < LooseVersion('0.6.10'):

~/anaconda3/envs/sc-tutorial/lib/python3.6/site-packages/importlib_metadata/api.py in version(package)
    103         "Version" metadata key.
    104     """
--> 105     return distribution(package).version
    106
    107

~/anaconda3/envs/sc-tutorial/lib/python3.6/site-packages/importlib_metadata/api.py in distribution(package)
     84     :return: A ``Distribution`` instance (or subclass thereof).
     85     """
---> 86     return Distribution.from_name(package)
     87
     88

~/anaconda3/envs/sc-tutorial/lib/python3.6/site-packages/importlib_metadata/api.py in from_name(cls, name)
     50                 return resolved
     51         else:
---> 52             raise PackageNotFoundError(name)
     53
     54     @staticmethod

PackageNotFoundError: umap-learn

I have umap-learn 0.3.9 installed.
Scanpy version: 1.4.3+115.g1aecabf
Anndata version:  0.6.22rc1
It seems to work with umap-learn 0.3.8, scanpy 1.4.3+105.gc748b35. and anndata 0.6.22.post1+1.g8dcc3cd

OK, so you’re using Python < 3.8 and importlib_metadata. The line umap_version = version("umap-learn") throws an error. It works for me with the same setup:
$ python -c 'from importlib_metadata import version; print(version("umap-learn"))'
0.3.0

Yes, I am using python 3.6.7. And I do have the umap_learn-0.3.9.dist-info folder in my conda env. Reverting to the commit before the last one in #704 works. I hope I don't have to git bisect for the moment...
A couple more observations:
$ python -c 'from importlib_metadata import version; print(version("umap-learn"))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/icb/malte.luecken/anaconda3/envs/sc-tutorial/lib/python3.6/site-packages/importlib_metadata/api.py", line 105, in version
    return distribution(package).version
  File "/home/icb/malte.luecken/anaconda3/envs/sc-tutorial/lib/python3.6/site-packages/importlib_metadata/api.py", line 86, in distribution
    return Distribution.from_name(package)
  File "/home/icb/malte.luecken/anaconda3/envs/sc-tutorial/lib/python3.6/site-packages/importlib_metadata/api.py", line 52, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.api.PackageNotFoundError: umap-learn

But, the last couple lines you posted (from imporlib_metadata import Distribution.... raise...) work.

I've played around a bit with version() from importlib_metadata, and it tells me the version of scanpy and anndata, but never finds umap or umap-learn. I can however do
In [3]: import umap
In [4]: umap.__version__
Out[4]: '0.3.9'

But, version("umap") or version("umap-learn") doesn't work. Any ideas? I've not installed anything manually, but everything via conda's pip. Does this have to do with package names and import names being different?

Figured it out: it works with version("umap_learn") but not with version("umap-learn"). Maybe this should be changed in the commit? I wonder if this is a python version issue.

Seems like a bug in your version of importlib_metadata, it works fine on my system:
$ python3 -c 'from importlib_metadata import version; print(version("importlib_metadata"))'
0.18
$ ls -d1 ~/.local/lib/python3.6/site-packages/umap*
~/.local/lib/python3.6/site-packages/umap
~/.local/lib/python3.6/site-packages/umap_learn-0.3.9-py3.6.egg-info
$ python3 -c 'from importlib_metadata import version; print(version("umap-learn"))'
0.3.9

I have importlib_metadata version 0.6. I would put that into the requirements then.

haha... you were a bit quicker than me... I had to rebase and update first ;)

:grin:

By the way: I suggested right at the beginning that it’s possible…

You have an older version of importlib_metadata with a bug or so

In the future, please try updating a package that might have a problem, then we save a lot of time.

ah, but then we would not have found the version where this is an issue ;). But yes... will update all relevant packages next time.

True. Setting the requirement to 0.18 wouldn’t have hurt either though!


Update to Elasticsearch 6.8.22

Update all packages to ES 6.8.22

elasticsearch-odfe:

Use the newer opensearch-project/security repository
Change the security plugin ES version to 6.8.22
Use HEAD of the opendistro-0.10 branch for any patches
Ensure that hab can run the tools/securityadmin.sh script
Changed the package version to arbitrary <IP_ADDRESS> to represent 6.8.22

kibana-odfe:

Use the newer opensearch-project/security-dashboards-plugin
Use HEAD of the opendistro-0.10 branch for any patches

$ curl  https://<IP_ADDRESS>:9200 --insecure -u admin:admin
{
  "name" : "gABUx5m",
  "cluster_name" : "chef-insights",
  "cluster_uuid" : "sBGR_rP4Q0aPZBt9TL_t-A",
  "version" : {
    "number" : "6.8.22",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "71fcb50",
    "build_date" : "2021-12-19T01:10:56.497443Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

elasticsearch-odfe.default(O): [2021-12-22T19:19:57,175][INFO ][o.e.c.r.a.AllocationService] [gABUx5m] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...]).
kibana-odfe.default(O): {"type":"log","@timestamp":"2021-12-23T00:19:57Z","tags":["listening","info"],"pid":2564557,"message":"Server running at https://<IP_ADDRESS>:5601"}

Note for Kibana I had to add elasticsearch.ssl.verificationMode: none for this to work locally in this configuration due to:
kibana-odfe.default(O): {"type":"log","@timestamp":"2021-12-23T00:15:54Z","tags":["error","elasticsearch","admin"],"pid":2564140,"message":"Request error, retrying\nHEAD https://localhost:9200/ => Hostname/IP does not match certificate's altnames: Host: localhost. is not cert's CN: node"}

I did this by hacking the default.toml locally after install.
These messages may be worth looking into as a followup:
kibana-odfe.default(O): {"type":"log","@timestamp":"2021-12-23T00:15:52Z","tags":["warning","elasticsearch","config","deprecation"],"pid":2564140,"message":"Config key \"url\" is deprecated. It has been replaced with \"hosts\""}
kibana-odfe.default(O): {"type":"log","@timestamp":"2021-12-23T00:15:53Z","tags":["status","plugin:opendistro_security@6.8.22","info"],"pid":2564140,"state":"yellow","message":"Status changed from yellow to yellow - 'opendistro_security.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'","prevState":"yellow","prevMsg":"Default cookie password detected, please set a password in kibana.yml by setting 'opendistro_security.cookie.password' (min. 32 characters)."}

Note that since this is ES 6.8.22 it contains only log4j 2.17.0: 👍
$ find /hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/ -name '*log4j*'
/hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/plugins/repository-s3/config/log4j2.properties
/hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/plugins/opendistro_security/log4j-slf4j-impl-2.11.1.jar
/hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/config/log4j2.properties
/hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/lib/log4j-core-2.17.0.jar
/hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/lib/log4j-api-2.17.0.jar
/hab/pkgs/btm/elasticsearch-odfe/<IP_ADDRESS>/20211223000603/lib/log4j-1.2-api-2.17.0.jar

@btm  - i had to do this to get journalbeat to build with 6.8.22 when I was messing with this earlier this week -- https://github.com/chef/opendistro-for-elasticsearch/pull/18/files#diff-310c2596448b875980c6aa386714cb12df327eb293a5c61b52fabc1ec63bd97c
the beats/journalbeat path doesn't exist in the main version of that repo anymore so had to pull the repo down and checkout the tag first before trying to use it.

@collinmcneese thanks! I had noticed that folder was removed on master, but I hadn't built the journalbeat (or metricbeat) package yet so I hadn't seen a failure. Added that.

log4j-slf4j-impl-2.11.1.jar

Someone should look into how/if this is affected by recent log4j vulnerabilities, but otherwise this should be good to merge, build, and start testing upgrades for our deployments that use these packages.

Apparently ES 6.8.23 will ship soon (Estimated Jan 13) with Log4J 2.17.1 to fix CVE-2021-44832


Got an error reading communication packets with MariaDB > 10.4

Bug description
Using any MariaDB Version > 10.4 leads to random connection errors with the prisma query engine.
Tested with different Prisma versions, on different OS/Architectures. Native mysql connectors doesn't face this problem.
mysql error log
Aborted connection 5 to db: '****' user: '****' host: '*****' (Got an error reading communication packets)
This not only causes connections to fail sometimes but also spams the db server with tons of errors in logs&statistics
Nasty Workaround
Increase your wait_timeout or keep the connection alive within your wait_timeout range SELECT 1 etc.
How to reproduce
Fresh setup of MariaDB > 10.4
Any Prisma version
Simple script with connect, select 1, disconnect.
8/10 it will connect but produce these errors in the mysql logs
Expected behavior
Fixating 10.4 at the docs for MariaDB.
Prisma information

generator client {
   provider = "prisma-client-js"
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")

}

model test {
  id    Int    @id @default(autoincrement())
  name  String
}

import { PrismaClient } from '@prisma/client'

const prisma = new PrismaClient()

async function main() {
  let res= await prisma.$queryRaw`SELECT 1`
  console.log(res)
}

main()
  .then(async () => {
    await prisma.$disconnect()
  })

Environment & setup

OS: Windows 11 AMD64 / Ubuntu 22.04 ARM
Database: MariaDB > 10.4
Node.js version: v20.10.0

Prisma Version
<EMAIL_ADDRESS>

To clarify: Does this actually lead to Prisma and its connection not to work, or is this "only" a cosmetic problem (that might be indicating a performance problem of frequent retries) in the logs?

To clarify: Does this actually lead to Prisma and its connection not to work, or is this "only" a cosmetic problem (that might be indicating a performance problem of frequent retries) in the logs?

Like mentioned, this leads to prisma randomly not being able to establish a connection. In a long running process, across 24h I had like over 90 of these entries. Even if establishing a connection, your database statistics will get clogged with errored/aborted connections, if you care about db maintenance this would go beyond just some cosmetic problem.
The nasty workaround I've mentioned cut this down significant but depending on your type of project having a permanent connection may not be so favourable, so it seems like it really only affects the connect procedure.

this leads to prisma randomly not being able to establish a connection.

What error do you get in these cases from Prisma?


backport: ceph-rgw-loadbalancer: Modify keepalived master selection

Currently the keepalived template only works when system hostnames exactly match the Ansible inventory name. If these are different, all generated templates become BACKUP without a MASTER assigned. Using the inventory_hostname in the template file resolves this issue.
Signed-off-by: Stanley Lam<EMAIL_ADDRESS>(cherry picked from commit 2ca33641091366a9f69150a331b0fdf364833bf4)

jenkins test centos-container-all_daemons

jenkins test centos-container-collocation

jenkins test centos-container-podman

jenkins test centos-container-all_daemons

jenkins test centos-non_container-update

jenkins test  centos-container-all_daemons

jenkins test centos-container-all_daemons

bump


.profile no such file or directory

Hi, Thanks for sharing this set up. Just did a clean install with Sierra because Im trying to go super streamlined with a dev only machine. I've been testing out dotfile configurations that I can reproduce easily next time i want to set up my machine. I guess Im shooting for a totally portable set up(on Mac and Ubuntu). Anyway, ran your bot. No real issues so far. When I open Iterm i get this error. What is supposed to be in .profile? Did I do something wrong?  Thanks.
/Users/devworx/.profile:source:9: no such file or directory: /Users/devworx/.iterm2_shell_integration.zsh
The old complaint that mass culture is designed for eleven-year-olds
is of course a shameful canard.  The key age has traditionally been
more like fourteen.
		-- Robert Christgau, "Esquire"

Ah, great catch. In the iTerm menu, select "install shell integration". That will add the needed ".iterm2_shell_integration.zsh" file. I'll make sure that file gets added to the project too :)


Add warning that browser setting "Pop-ups and redirects" should be enabled for a new Payout proposals

If the browser setting "Pop-ups and redirects" is disabled - the new proposal is not showing up in a DAO.

fixed in dev


Set Focused property after internal logic

Focused on TextInputBase is currently set on click or left mouse pressed. However, this happens before internal logic to place the cursor. This makes it difficult to customize the behaviour of TextInputBase on focus such as selecting all text on focus.
This change moves setting of the Focused property to after internal logic has run to place the cursor.
Tested locally. With this change it's now possible to set the selection range directly in the handler for the InputFocusChanged event.

Seems like a reasonable change.  👍
Thank you!


Add filtering for browse page derived topics

While migrating Mainstream Browse pages to topics we need to prevent
the migrated clones from being visible to the user. Here we're adding
filtering to prevent items being show if the relevant field doesn't
exist or is true.
https://trello.com/c/m13ffUVc/292-update-publishing-apps-that-know-of-specialist-topics-to-hide-new-synced-ones
⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

I assume this is no longer needed. Can always reopen later if this changes.


Mistake in Russian language model

Hello. Russian language alphabet not have this letter: "і", while Ukrainian or Belarusian have. This entries with it are mistake:
https://github.com/pemistahl/lingua/blob/master/src/main/resources/language-models/ru/bigrams.json
"кі","725/3944471"
"фі","5311/210102"

And I see a lot of issues like this in Russian folder. As you refer to Wortschatz corpora offered by Leipzig University, Germany - their data quality seems like a big question. At least alphabet easy to check I guess... I looked in unigrams too and there is:
"ћ","7836216/97387225"
This not Russian or even not Greek/Latin char (it could be something in math/science documents). Just curious what language will detect your library on this page article https://www.unian.ua/politics/10699929-tramp-doruchiv-u-travni-skasuvati-zaplanovaniy-vizit-na-inavguraciyu-zelenskogo-rozsekrechena-skarga.html (This is Ukrainian)

Hi @pavlomorozov, thank you for letting me know about this.
The reason behind it is simple: For those languages that use the Cyrillic alphabet, I've taken all Cyrillic characters into account to build the language models. Or, to be more precise, I've used all characters that match the character class \p{IsCyrillic}. Of course, this might lead to the inclusion of characters that are not officially part of the respective language.  However, this shouldn't be a big deal because the probabilities for those "unsupported" characters will be very low and will thus not affect the result much.
Are you satisfied so far with the detection of Russian or are you of the opinion that detection could be better here?

Hello.
I not tried your library. My goal is to find a good solution to detect language and few more natural language processing features, so your lib is just an option for one of the tasks I have. I looking for libraries to have as much as possible work done on our servers to not rely to third party services. I found thread on Reddit and get here to take a look. As soon as I understand few Cyrillic languages I get into model and you left you a not about what I found. Your assumption about probabilities could work sometimes. For example to distinct Russian and Tatar (Kazan). The Tatar language is mostly Cyrillic but it is very different (close to Turkish and Russians not understand it mostly) and here probabilities could help a lot. But it could be harder to distinct Kazakh text written in Cyrillic and Tatar (Kazan) (both are Turkish family). One more problem Kazakhstan seems to want to use Latin alphabet and reform should be ended in 2025. Ukrainian is quite similar to Belorussian - it is also could be not so easy to distinct. This all a bit dynamic but should not be a big problem once you will be sure about what language used to write the text you take model from. Russian language is the most popular of using Cyrillic alphabet, but there are few others and they are distinct a lot.
Have a good luck!

Thank you @pavlomorozov for giving me insight into these language families. I will definitely add support for more languages that use the Cyrillic alphabet and thereby pay attention to what you said. I'm going to close this issue now as there is nothing to do immediately.
Good luck for your language detection tasks!


tests/framework: additions to framework to query from Prometheus/Grafana

This commit adds few high level helpers to allow querying for various resources
on both Prometheus and Grafana.
The current set of helpers added are sufficient query and track
CPU and Memory trends, as well as graphing the overall control plane namespace
CPU and memory consumption through Grafana remote rendering, all at any point
during a test.
This is part for an effort to better instrument our tests and potentially unlock minimal scale
testing using the very same tools we've written our tests with.
Relevant APIs:
common_metrics.go:
  struct Prometheus {}
  (*Prometheus) VectorQuery(string) (float, err)                                     // raw query call, since it's none-range, expects only one sample in vector form
  (*Prometheus) GetNumEnvoysInMesh() (float, err)                                    // returns number of current envoy in mesh as seen by Prom, ie. '20'
  (*Prometheus) GetMemRSSforContainer(ns, pod, container) (float, err)               // returns RSS footprint of container in bytes, ie. '52562'
  (*Prometheus) GetCPULoadAvgforContainer(ns, pod, container, avgTime) (float, err)  // returns CPU load avg for container given a timerange, ie. '0.62'
  (*Prometheus) GetCPULoadAvgforContainer(ns,pod,container) (float, float, float, err) // returns 1m, 5m and 15m CPU Load averages for a container

  struct Grafana {}
  (*Grafana) PanelPNGSnapshot(dashboard, panelId, timeMinutes, saveFilepath)        // Triggers a dashboard/panel render and stores it in 'saveFilepath'

common_apps.go:
  GetPrometheusPodHandle(ns, pod, port) (*Prometheus, err)  // port-forwards
  GetGrafanaPodHandle(ns, pod, port) (*Grafana, err)        // port-forwards

Metrics                [X]
Tests                  [X]
Performance            [X]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?
No

Codecov Report

Merging #2138 (0a0c83e) into main (d12f43b) will decrease coverage by 0.81%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2138      +/-   ##
==========================================
- Coverage   58.67%   57.86%   -0.82%
==========================================
  Files         144      144
  Lines        5936     5995      +59
==========================================
- Hits         3483     3469      -14
- Misses       2450     2523      +73
  Partials        3        3

Impacted Files
Coverage Δ

cmd/cli/dashboard.go
0.00% <0.00%> (ø)

cmd/cli/proxy_configdump.go
4.54% <0.00%> (ø)

pkg/kubernetes/portforward.go
0.00% <ø> (ø)

pkg/endpoint/providers/kube/fake.go
0.00% <0.00%> (-70.84%)
:arrow_down:

pkg/catalog/service.go
91.66% <0.00%> (-8.34%)
:arrow_down:

pkg/endpoint/providers/kube/client.go
82.85% <0.00%> (-1.83%)
:arrow_down:

pkg/catalog/routes.go
81.86% <0.00%> (-0.68%)
:arrow_down:

pkg/endpoint/types.go
0.00% <0.00%> (ø)

pkg/catalog/mock_catalog.go
0.00% <0.00%> (ø)

pkg/trafficpolicy/trafficpolicy.go

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc87610...0a0c83e. Read the comment docs.


Added section about composite entities to the docs

Proposed changes:
Add documentation about composite entities.
Status (please check what you already did):

[ ] added some tests for the functionality
[x] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

CRFEntityExtractor is not deprecated.


Undefined payload

I am using the ssh2 package to create my own sftp server.
It got to a state where quite all sftp functions are supported and working - at least on Filezilla.
I just tried Cyberduck and I do get this error sometimes, especially when uploading large folders:
/Users/pascal/code/ips-hosting-daemon/node_modules/ssh2-streams/lib/ssh.js:5292
  pktLen = payload.length + 9;
                   ^

TypeError: Cannot read property 'length' of null
    at send_ (/Users/pascal/code/ips-hosting-daemon/node_modules/ssh2-streams/lib/ssh.js:5292:20)
    at /Users/pascal/code/ips-hosting-daemon/node_modules/ssh2-streams/lib/ssh.js:5272:7
    at afterWrite (_stream_writable.js:455:3)
    at onwrite (_stream_writable.js:446:7)
    at Deflate.afterTransform (_stream_transform.js:89:3)
    at Zlib.processCallback (zlib.js:572:8)

Any idea what could be the reason and how I can prevent it?

Others have encountered issues with Cyberduck and compression before, so in that case you should explicitly disable compression support with algorithms: { compress: ['none'] } in your ssh2 server configuration.

Okay, thank you for the hint!


Rename function to match spec

With the recent change to rename a function to match spec in #1987, I noticed that several ops also rely on this meta function that can also benefit from name change.

Naming is tricky :) The other synonym I could think of is is_distinct, but I guess this has the same source of confusion (distinct to what?). I agree there can be source of confusion, but I still like the idea of prefixing with the canonical is_ for boolean functions (other similar meta functions in the spec also prefix with is_ e.g. is_nan or is_sorted). Thoughts?

Naming is tricky :)

Agreed!

I still like the idea of prefixing with the canonical is_ for boolean functions (other similar meta functions in the spec also prefix with is_ e.g. is_nan or is_sorted). Thoughts?

I think for predicates both is_ and has_ are quite common as prefixes. IMO in this case is_unique is not the clearest name because it is a property of the elements of the container, not of the container itself, yet we are applying it to the container. has_no_duplicates is a property of the container, which IMO aligns with our intent more clearly.
I don't have strong feelings here though, I'm happy to approve the change. Anecdotally it seems the name is_unique is used in pandas: https://pandas.pydata.org/docs/reference/api/pandas.Series.is_unique.html

Great to see we're not coining a new term for it. If we're not the first ones, it's probably okay to keep it then. Thanks for the reference!


File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3069, in main     raise AttributeError(f"2bit reference genome file '{two_bit_reference_file}' does not exist!")

Hi,
I am trying to run GCparagon with singularity and I get the following error:
Traceback (most recent call last):
File "/opt/conda/envs/GCparagon/bin/gcparagon", line 8, in
sys.exit(main())
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3069, in main
raise AttributeError(f"2bit reference genome file '{two_bit_reference_file}' does not exist!")
AttributeError: 2bit reference genome file 'resources/hg38/gcpropagon' does not exist!
My call:
singularity run gcparagon.sif -b healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam  -rtb  resources/hg38/gcpropagon/hg38_analysisSet.2bit -c resources/hg38/gcpropagon/hg38_minimalExclusionListOverlap_1Mbp_intervals_33pcOverlapLimited.FGCD.bed    -rgcd resources/hg38/gcpropagon/hg38_reference_GC_content_distribution.tsv -rgb hg38
I have the file in the specified location for sure:
user@lm4-f001 ~]$ ls /globalscratch/ulb/bctr/astanciu/resources/hg38/gcpropagon/hg38_analysisSet.2bit
/globalscratch/ulb/bctr/astanciu/resources/hg38/gcpropagon/hg38_analysisSet.2bit
I have created the singularity image using the def script you provided.
Can you help me with this issue?
Thanks,
Alexandra

Hi Alexandra,
I am sorry that you experience problems with the GCparagon singularity image. I will try to test it locally and understand it better.
Do you get the same error (file not found) if you don't specify the paths to the reference files (i.e., none of what you provided above except for -b healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam since -rgb hg38 is default)?
BR Benjamin

The error message is different, but the problem is the same. It seems that it still cannot find the file, but it looks for it in the default location.
See below:
$ singularity run gcparagon.sif -b /globalscratch/ulb/bctr/astanciu/healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam
cannot proceed - no two-bit reference file defined and default expected file not present under {'hg38': PosixPath('/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/2bit_reference/hg38.analysisSet.2bit'), 'hg19': PosixPath('/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/2bit_reference/hg19.2bit')}. Terminating ..

yes, I built the image myself from the current repo.
If I try the pre-built image I get the following error:
singularity pull --arch amd64 library://bgspiegl/gcparagon/gcparagon-ubuntu-22_04-container:latest
FATAL:   Unable to get library client configuration: remote has no library client (see https://apptainer.org/docs/user/latest/endpoint.html#no-default-remote)

I am not sure how apptainer plays into this but if you would be using apptainer to pull the container, you should be able to fix the problem by running the commands that are provided under the URL of your last message:
apptainer remote add --no-login SylabsCloud cloud.sycloud.io and
apptainer remote use SylabsCloud and
apptainer remote list (which should list ''SylabsCloud' among your list of available remotes)
The command singularity remote list should also list SylabsCloud.
What does it show for you?
After that the minimal pull command singularity pull bgspiegl/gcparagon/gcparagon-ubuntu-22_04-container:latest should work.
I need more information to give you any reasonable support here.
I tested the most simple pull command singularity pull bgspiegl/gcparagon/gcparagon-ubuntu-22_04-container:latest on another machine and it worked fine.
Which OS are you on?
What is the version of your singularity? (run singularity --version from within the GCparagon conda env; the version of my singularity from the GCparagon conda is 3.8.6)

Concerning the file not foundproblem - not every location is accessible for the singularity image.
You would have to use paths under, e.g., $HOME Apptainer/SingularityCE to find the files (I am also a friend of absolute paths here).
If you want to use another path, you can always mount it as described in the Apptainer docs:
https://apptainer.org/docs/user/latest/quick_start.html#working-with-files
Example of the 2bit reference files:
For now, the hg38 2bit reference genome file is downloaded to
/opt/github/GCparagon/src/GCparagon/2bit_reference/hg38.analysisSet.2bit
but the program expects to find it under  /opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/2bit_reference/hg38.analysisSet.2bit.
You should be able to get the anticipated behaviour by passing the absolute paths under /opt/github/GCparagon/src/GCparagon/ where the files are actually located. I will look into how to get a consistent behaviour, irrespective of whether the user used the pip install` command (also in singularity image) or not.

Hi,
After running the apptainer commands you suggested I get the following error:
[astanciu@lm4-f001 ~]$ singularity pull --arch amd64 library://bgspiegl/gcparagon/gcparagon-ubuntu-22_04-container:latest
FATAL:   While pulling library image: error fetching image: error making request to server:
Get "https://library.sylabs.io/v1/images/bgspiegl/gcparagon/gcparagon-ubuntu-22_04-container:latest?arch=amd64": dial tcp: lookup library.sylabs.io on <IP_ADDRESS>:53: no such host
And this is the list I get:
[astanciu@lm4-f001 ~]$ singularity remote list
NAME           URI                  DEFAULT?  GLOBAL?  EXCLUSIVE?  SECURE?
DefaultRemote  cloud.apptainer.org            ✓                    ✓
SylabsCloud    cloud.sycloud.io     ✓                              ✓
I am working on a Linux OS cluster and my singularity version is: apptainer version 1.3.0-1.el8

Hi,
It eventually worked to download the image, but I get the same issue:
singularity run  gcparagon-ubuntu-22_04-container_latest.sif -b /healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam
cannot proceed - no two-bit reference file defined and default expected file not present under {'hg38': PosixPath('/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/2bit_reference/hg38.analysisSet.2bit'), 'hg19': PosixPath('/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/2bit_reference/hg19.2bit')}. Terminating ..

Does it find the required hg38 files if you specify them like this: --two-bit-reference-genome /opt/github/GCparagon/src/GCparagon/2bit_reference/hg38.analysisSet.2bit --exclude-intervals /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_minimalExclusionListOverlap_1Mbp_intervals_33pcOverlapLimited.FGCD.bed --reference-gc-content-distribution-table /opt/github/GCparagon/src/GCparagon/accessory_files/accessory_files/hg38_reference_GC_content_distribution.tsv

I am sorry, I made yet another mistake (it is --intervals-bed, not --exclude-intervals). Please try these fixed parameters:
--two-bit-reference-genome /opt/github/GCparagon/src/GCparagon/2bit_reference/hg38.analysisSet.2bit --intervals-bed /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_minimalExclusionListOverlap_1Mbp_intervals_33pcOverlapLimited.FGCD.bed --reference-gc-content-distribution-table /opt/github/GCparagon/src/GCparagon/accessory_files/accessory_files/hg38_reference_GC_content_distribution.tsv

I ran this:
singularity run gcparagon-ubuntu-22_04-container_latest.sif -b /globalscratch/ulb/bctr/astanciu/healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam  --two-bit-reference-genome /opt/github/GCparagon/src/GCparagon/2bit_reference/hg38.analysisSet.2bit --intervals-bed /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_minimalExclusionListOverlap_1Mbp_intervals_33pcOverlapLimited.FGCD.bed --reference-gc-content-distribution-table /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_reference_GC_content_distribution.tsv
Error:
Traceback (most recent call last):
File "/opt/conda/envs/GCparagon/bin/gcparagon", line 8, in
sys.exit(main())
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3060, in main
raise FileNotFoundError(f"the reference GC content distribution file could not be found/accessed!")
FileNotFoundError: the reference GC content distribution file could not be found/accessed!

Made another mistake. This time in the assumed correct default path. Please try:
--two-bit-reference-genome /opt/github/GCparagon/src/GCparagon/2bit_reference/hg38.analysisSet.2bit --intervals-bed /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_minimalExclusionListOverlap_1Mbp_intervals_33pcOverlapLimited.FGCD.bed --reference-gc-content-distribution-table /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_reference_GC_content_distribution.tsv
I am curerntly also testing this locally. Sorry for the series of mistakes. I am planning to change the singularity.def file, create another image and make it available soon.

I have the same error:
singularity run gcparagon-ubuntu-22_04-container_latest.sif -b /globalscratch/ulb/bctr/astanciu/healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam --two-bit-reference-genome /opt/github/GCparagon/src/GCparagon/2bit_reference/hg38.analysisSet.2bit --intervals-bed /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_minimalExclusionListOverlap_1Mbp_intervals_33pcOverlapLimited.FGCD.bed --reference-gc-content-distribution-table /opt/github/GCparagon/src/GCparagon/accessory_files/hg38_reference_GC_content_distribution.tsv
Traceback (most recent call last):
File "/opt/conda/envs/GCparagon/bin/gcparagon", line 8, in
sys.exit(main())
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3060, in main
raise FileNotFoundError(f"the reference GC content distribution file could not be found/accessed!")
FileNotFoundError: the reference GC content distribution file could not be found/accessed!

This is very odd. I am trying to create a new image now from the current code (tested successfully in script form). I am sorry for this persisting problem.
An alternative would be to create the conda environment from the YAML file and run GCparagon simply as a python3 script.
This, of course, can also fail if the conda environment can't be created for som dubious reason.
I will get back to you when I know more.

I would have tried with conda, but I am working on a cluster and I don't have this option unfortunately.
Thank you for your help!

Hi Maria.
Thank you once again for bringing up this issue and helping me making GCparagon a more convenient tool.
I think commit 93aa29c solved your issue.
Please also have a look at release v0.6.8 and the updated README.md.
The new singularity_definition_file/build_and_test_singularity_image.sh shows an example for running GCparagon from the container and binding/mounting a directory that is inaccessible to any singularity container by default (the -B parameter).
Please get the new container:
singularity pull library://bgspiegl/gcparagon/gcparagon_0.6.8-ubuntu-22_04-container:latest
Does the --reference-genome-build hg38 parameter work now for you with this new container?
The problem of the default file paths breaking persists though for local installations of GCparagon (running pip install .). I will fix this in another release.
BR Benjamin

Hi,
I have downloaded the image you suggested  and I ran it with the -B parameter:
singularity run -B /globalscratch/ulb/bctr/astanciu/ gcparagon_0.6.8-ubuntu-22_04-container_latest.sif  --bam /globalscratch/ulb/bctr/astanciu/healthy_sWGS/bams/Genome-IJB-HP-10-xx_S81.markDup.bam
And it worked.
Thank you.
Best regards,
Alexandra


Call onReaderClosed when reader is shut down

Codecov Report
Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (45472d5) 40.16% compared to head (fdaf9a6) 40.16%.

Files
Patch %
Lines

...java/tech/ydb/topic/read/impl/AsyncReaderImpl.java
0.00%
5 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             develop     #214   +/-   ##
==========================================
  Coverage      40.16%   40.16%
  Complexity      1186     1186
==========================================
  Files            246      246
  Lines           9716     9714    -2
  Branches         927      928    +1
==========================================
  Hits            3902     3902
+ Misses          5546     5544    -2
  Partials         268      268

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Support kebab-case

Hi,
Docs say you support polished but certain helpers in polished produce standard css names (kebab-case) which throw warnings in the console.
See my use of polished.padding here https://codesandbox.io/s/nJ6nE73R
Notice in the console there are warnings Warning: Unsupported style property padding-left. Did you mean paddingLeft?
Thanks!

Hi @mattcolman!
Hmmm... I was under the impression polished was supposed to always produce camelCase properties. cc @nikgraf @bhough
In any case, this wouldn't be resolved by glamorous, this would have to be done in glamor. glamorous simply forwards these properties to glamor. With that in mind I'll close this issue, do you mind filing an issue there and suggesting a solution?
Thanks! And good luck!
P.S. if this isn't something that will be resolved, it may be good to add a note to the polished integration example on the new website...

@kentcdodds @mattcolman this has been merged into our V2 branch which I'm aiming to get out by the end of the month, so that should resolve this issue.

Wonderful! :tada: thanks so much!

✨ polished 1.3 moves to camel-casing and is out now: https://github.com/styled-components/polished/releases/tag/v1.3.0

Thanks!!!


Add OCS tolerations to pods

Add tolerations to token-exchange-agent pod and odfmo-controller-manager.
Signed-off-by: Divyansh Kamboj<EMAIL_ADDRESS>

/test all

/test integration-test


Update LaunchRuleTransformer.kt

@GeorgCantor - Thank you for the pull request. Can you please switch the base branch for this PR to dev and use the pull request template to update the description.


Owned Shadow Properties: design considerations

I have tried to model a workaroud of not yet implemented feature many-to-many relationship #1368 trough owned shadow properties via fluent api and got an error message that at first sight seems conflict with definition of shadow properties given in documentation.
From documentation about shadow properties

Shadow properties are properties that are not defined in your .NET entity class but are defined for that entity type in the EF Core model.

but the message i've got tells me that shadow property dont have correspoding CLR property.
This leads me to a question:
Is there anything wrong with trying to make a shadow owned properties or im misunderstool the thing i have read in documentation?
Im 100% understand my use case is silly, im just seeing owned properties as a way to do a table splitting and shadow properties as an instrument of keeping underlying EF Core mechanics away from POCO that represents business data and trying to glue then together
More details:
Exception message:
The navigation property 'PropertyRelations' cannot be added to the entity type 'Entity' because there is no corresponding CLR property on the underlying type and navigations properties cannot be added to shadow
Stack trace:
Microsoft.EntityFrameworkCore.Metadata.Internal.InternalRelationshipBuilder.Navigations(Nullable`1 navigationToPrincipal, Nullable`1 navigationToDependent, EntityType principalEntityType, EntityType dependentEntityType, Nullable`1 configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalRelationshipBuilder.Navigations(Nullable`1 navigationToPrincipal, Nullable`1 navigationToDependent, Nullable`1 configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalRelationshipBuilder.PrincipalToDependent(String name, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.Relationship(InternalEntityTypeBuilder targetEntityTypeBuilder, Nullable`1 navigationToTarget, Nullable`1 inverseNavigation, Boolean setTargetAsPrincipal, ConfigurationSource configurationSource, Nullable`1 required)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.Owns(TypeIdentity& targetEntityType, PropertyIdentity navigation, Nullable`1 inverse, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.Owns(Type targetEntityType, String navigationName, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Builders.EntityTypeBuilder.OwnsManyBuilder(TypeIdentity& ownedType, String navigationName)
   at Microsoft.EntityFrameworkCore.Metadata.Builders.EntityTypeBuilder.OwnsMany(Type ownedType, String navigationName, Action`1 buildAction)
   at DataAccessLayer.SQL.AppDbContext.OnModelCreating(ModelBuilder modelBuilder)

Steps to reproduce
First i defined shadow property then i tried to use .OwnsMany to model many to many relationship trough helper object. Im using reflection since i have a lot of model that have many to relationship and i personally dont like copy paste approach
var model = modelBuilder.Entity(modelType);

var propType = classProp.PropertyType;
var propName = classProp.Name;

model.Ignore(propName);

var newShadowProp = $"{propName}Relations";
var newShadowPropType = typeof(ICollection<>)
    .MakeGenericType(new[] { typeof(ManyToManyRelation) });

model.Property(newShadowPropType , newShadowProp);

model.OwnsMany(typeof(ManyToManyRelation), newShadowProp, a =>
{
       var tableName = allModelTypes[modelType];
       a.HasForeignKey(nameof(ManyToManyRelation.SourceId));
       a.HasForeignKey(nameof(ManyToManyRelation.TargetId));
       a.HasKey(nameof(ManyToManyRelation.SourceId), nameof(ManyToManyRelation.TargetId));
       a.ToTable($"{tableName}_{classProp.Name}");
}).HasOne(propType)
       .WithMany()
       .HasForeignKey(nameof(ManyToManyRelation.TargetId));

model.HasMany(typeof(ManyToManyRelation), newShadowProp)
      .WithOne()
      .HasForeignKey(nameof(ManyToManyRelation.SourceId));

Further technical details
EF Core version: 2.2.0
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Windows Server 2012 R2
IDE: (e.g. Visual Studio 2017 15.8.9)

What you are trying to add is not shadow property but shadow navigation. Shadow navigations are not supported. See #3864
Also your model configuration code has many mistakes.
Leaving this issue open to discuss if we want to update the exception message or documentation to be clearer.

@smitpatel thanks to pointing to ticket, a change in error message to pinpoint the fact that shadow property and shadow navigation is different kind of beast help understand situation better
Proposing change of error message from(feel free to change it English language is not my primary):

The navigation property 'PropertyName' cannot be added to the entity type 'Entity' because there is no corresponding CLR property on the underlying type and navigations properties cannot be added to shadow
To:
The navigation property 'PropertyName' cannot be added to the entity type 'Entity' because there is no corresponding CLR property on the underlying type and shadow navigation(shadow pointing to another model) currently not supported.

Also im going to create issue in relevant docs.microsoft.com repo for change in documentation to reflect  the present limitation and difference between shadow property vs shadow navigation because currently there are clear statement what kind of shadow not supported


Envoy egress bpf_metadata filter looking up conntrack global map with <IP_ADDRESS>

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Could be a similar bug to this one #29406, but different error message printed. When I setup a dedicated ingress with NodePort as the service type, and try to send HTTP request to that node port, I saw a 503 error after the connection hangs for a few seconds:
curl -v http://localhost:30109/ -v
*   Trying [::1]:30109...
* connect to ::1 port 30109 failed: Connection refused
*   Trying <IP_ADDRESS>:30109...
* Connected to localhost (<IP_ADDRESS>) port 30109
> GET / HTTP/1.1
> Host: localhost:30109
> User-Agent: curl/8.3.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Fri, 15 Dec 2023 20:35:46 GMT
< server: envoy
<
* Connection #0 to host localhost left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

then I can see Envoy proxy prints out error message like this immediately:
[info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory

After looking into it deeper with debug log level, I realized that somehow Cilium envoy filter is looking up record in conntrack global map with <IP_ADDRESS> as the source and dest IP addresses:
[2023-12-15 20:37:52.946][17][debug][misc] [cilium/bpf_metadata.cc:215] EGRESS POD IP: <IP_ADDRESS>, destination IP: <IP_ADDRESS>
[2023-12-15 20:37:52.946][17][debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
[2023-12-15 20:37:52.946][17][trace][filter] [cilium/conntrack.cc:197] cilium.bpf_metadata: Looking up key: 7f000001, 7f000001, 288d, d51c, 6, 0
[2023-12-15 20:37:52.946][17][debug][filter] [cilium/conntrack.cc:128] cilium.bpf_metadata: Opened IPv4 conntrack map global
[2023-12-15 20:37:52.946][17][info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory
[2023-12-15 20:37:52.946][17][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[2023-12-15 20:37:52.946][17][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: <IP_ADDRESS> has ID 2
[2023-12-15 20:37:52.946][17][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[2023-12-15 20:37:52.946][17][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: <IP_ADDRESS> has ID 2
[2023-12-15 20:37:52.946][17][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: a1401c6, prefixlen: 32
[2023-12-15 20:37:52.946][17][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: <IP_ADDRESS> has ID 8
[2023-12-15 20:37:52.946][17][debug][filter] [./cilium/socket_option.h:184] Cilium SocketOption(): source_identity: 8, ingress: false, port: 10381, pod_ip: <IP_ADDRESS>, source_addresses: /<IP_ADDRESS>:0/, mark: 80b00 (magic mark: b00, cluster: 0, ID: 8)

I think it doesn't make too much sense as I anticipate the conntrack look up should be using the actual curl client's source IP and the dest pod's IP.
Cilium Version
1.14.5
Kernel Version
Linux um790-00 6.1.55 #1-NixOS SMP PREEMPT_DYNAMIC Sat Sep 23 09:11:13 UTC 2023 x86_64 GNU/Linux
Kubernetes Version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hm, I also see the cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory error, but the rest looks different for me. Not sure I'm trying to do the same thing, though. I'm very new to Cilium and not sure how to debug my issue, so maybe I'm looking at a config error on my part - but I found this issue when googling for the error above :)
$ curl <URL of ingress> -I
HTTP/1.1 400 Bad Request
content-length: 11
content-type: text/plain
date: Wed, 03 Jan 2024 10:28:42 GMT
server: envoy
connection: close

cilium-envoy-dxx4r cilium-envoy [2024-01-03 10:23:40.613][17][debug][misc] [cilium/bpf_metadata.cc:215] EGRESS POD IP: <IP_ADDRESS>, destination IP: <IP_ADDRESS>
cilium-envoy-dxx4r cilium-envoy [2024-01-03 10:23:40.613][17][debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
cilium-envoy-dxx4r cilium-envoy [2024-01-03 10:23:40.613][17][trace][filter] [cilium/conntrack.cc:197] cilium.bpf_metadata: Looking up key: c0a800da, c0a800b6, 77a6, f3bd, 6, 0
cilium-envoy-dxx4r cilium-envoy [2024-01-03 10:23:40.616][17][debug][filter] [cilium/conntrack.cc:128] cilium.bpf_metadata: Opened IPv4 conntrack map global
cilium-envoy-dxx4r cilium-envoy [2024-01-03 10:23:40.616][17][info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory

nvm, my issue was actually that I had proxy-protocol enabled on the LoadBalancer Service but not on the envoy - well, support for enabling proxy protocol on the cilium-envoy was only added post-1.14

@fangpenlin thanks for the report. Could you supply your ingress configuration exactly?
Separately, making requests to localhost can be fragile. What happens if you make a request to the NodePort externally?

We're also encountering a similar issue with Envoy via the GatewayAPI.
We can narrow it down to when Envoy is on the name node as the destination pod, we hit these 503 errors.
To reproduce this issue:

Deploy an HTTP application on K8s with a standard Service definition. To simplify repro, just have a single replica.
Deploy Cilium with Envoy either embedded or daemonset
SSH onto the node where the application is running and try to curl the NodePort
Or you can Curl the NodeIP with NodePort (external)
Or you can Curl the LoadBalancer configured to forward to the NodePort. Depending on the Cloud infra in question, it helps to narrow/reduce the targets to the node where the app replica is running. (For example, I've labeled the node with cilium-debug=true and then set the annotation to service.beta.kubernetes.io/aws-load-balancer-target-node-labels: cilium-debug=true).
Any of those 3 scenarios results in the same output (example below, curling LoadBalancer)

curl -v http://<redacted>.com/ip
*   Trying 10.x.x.x:80...
* Connected to <redacted>.com (10.x.x.x) port 80 (#0)
> GET /ip HTTP/1.1
> Host: <redacted>.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Tue, 30 Jan 2024 16:58:44 GMT
< server: envoy
<
* Connection #0 to host <redacted>.com left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

Snippet from envoy debug log:
[2024-01-30 16:56:23.786][12][trace][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:86] tls inspector: new connection accepted
[2024-01-30 16:56:23.787][12][trace][filter] [external/envoy/source/common/network/listener_filter_buffer_impl.cc:95] onFileEvent: 5
[2024-01-30 16:56:23.787][12][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:250] item added to deferred deletion list (size=1)
[2024-01-30 16:56:23.787][12][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:125] clearing deferred deletion list (size=1)
[2024-01-30 16:56:23.938][13][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:793] [C280] connect timeout
[2024-01-30 16:56:23.938][13][debug][connection] [external/envoy/source/common/network/connection_impl.cc:139] [C280] closing data_to_write=0 type=1
[2024-01-30 16:56:23.938][13][debug][connection] [external/envoy/source/common/network/connection_impl.cc:250] [C280] closing socket: 1
[2024-01-30 16:56:23.939][13][trace][connection] [external/envoy/source/common/network/connection_impl.cc:423] [C280] raising connection event 1
[2024-01-30 16:56:23.939][13][debug][client] [external/envoy/source/common/http/codec_client.cc:107] [C280] disconnect. resetting 0 pending requests
[2024-01-30 16:56:23.939][13][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:484] [C280] client disconnected, failure reason:
[2024-01-30 16:56:23.939][13][debug][router] [external/envoy/source/common/router/router.cc:1290] [C279][S16061982579707376886] upstream reset: reset reason: connection failure, transport failure reason:
[2024-01-30 16:56:23.939][13][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:250] item added to deferred deletion list (size=1)
[2024-01-30 16:56:23.939][13][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:250] item added to deferred deletion list (size=2)
[2024-01-30 16:56:23.939][13][debug][http] [external/envoy/source/common/http/filter_manager.cc:996] [C279][S16061982579707376886] Sending local reply with details upstream_reset_before_response_started{connection_failure}
[2024-01-30 16:56:23.939][13][trace][misc] [external/envoy/source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x55ebf7ff340 for 300000ms, min is 300000ms
[2024-01-30 16:56:23.939][13][trace][http] [external/envoy/source/common/http/filter_manager.cc:1158] [C279][S16061982579707376886] encode headers called: filter=cilium.l7policy status=0
[2024-01-30 16:56:23.940][13][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1773] [C279][S16061982579707376886] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '91'
'content-type', 'text/plain'
'date', 'Tue, 30 Jan 2024 16:56:23 GMT'
'server', 'envoy'

Note: the routing seems to work fine when the Envoy pod is on a different node from the app replica.

Some additional observations, providing Hubble traces:

Envoy proxy is on a different node than the app replica. Note that all traffic is directed to the pod replica

Feb  3 06:18:07.531: 100.x.x.x:38579 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-proxy FORWARDED (TCP Flags: SYN)
Feb  3 06:18:07.532: 100.x.x.x:38579 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-proxy FORWARDED (TCP Flags: ACK)
Feb  3 06:18:07.532: 10.x.x.x:53011 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) http-request FORWARDED (HTTP/1.1 GET http://10.x.x.x:31666/)
Feb  3 06:18:07.532: 100.x.x.x:38579 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-proxy FORWARDED (TCP Flags: ACK, PSH)
Feb  3 06:18:07.539: 10.x.x.x:45644 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-endpoint FORWARDED (TCP Flags: SYN)
Feb  3 06:18:07.539: 10.x.x.x:45644 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-endpoint FORWARDED (TCP Flags: ACK)
Feb  3 06:18:07.542: 100.x.x.x:38579 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) http-request FORWARDED (HTTP/1.1 GET http://10.x.x.x:31666/)
Feb  3 06:18:07.545: 10.x.x.x:45644 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb  3 06:18:07.553: 100.x.x.x:38579 (ingress) <- test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) http-response FORWARDED (HTTP/1.1 200 17ms (GET http://10.x.x.x:31666/))
Feb  3 06:18:07.557: 10.x.x.x:53011 (ingress) <- test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) http-response FORWARDED (HTTP/1.1 200 26ms (GET http://10.x.x.x:31666/))
Feb  3 06:18:09.550: 10.x.x.x:45644 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Envoy proxy on the same node as the app replica. Note that it tries to use the LoadBalancer service, not the underlying pod. Incidentally, if one tries to curl the LoadBalancer service from inside the cluster, the request times out.

Feb  3 06:16:24.808: 100.x.x.x:37889 (ingress) -> test/httpbin-alb-7c54bb4487-hhch2:8080 (ID:2611) to-proxy FORWARDED (TCP Flags: SYN)
Feb  3 06:16:29.802: 10.x.x.x:53007 (ingress) -> test/cilium-gateway-gateway-nlb-test:31666 (world) http-request FORWARDED (HTTP/1.1 GET http://10.x.x.x:31666/)
Feb  3 06:16:29.803: 10.x.x.x:53007 (ingress) <- test/cilium-gateway-gateway-nlb-test:31666 (world) http-response FORWARDED (HTTP/1.1 503 4997ms (GET http://10.x.x.x:31666/))

Service:
> kubectl get svc cilium-gateway-gateway-nlb-test
NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP                             PORT(S)                      AGE
cilium-gateway-gateway-nlb-test   LoadBalancer   172.x.x.x   k8s-test-ciliumga-<redacted>.amazonaws.com   80:31666/TCP                 46h

After hours of research into tickets related to upstream connect error or disconnect/reset before headers. reset reason: connection timeout errors.  This issue appears to be the closest to the cause.  I can confirm @jschwartzy's findings.  There is always one Cilium daemonset pod that is unable to deliver gateway traffic to workloads on the same node.
I have also worked directly with some other community members to enable gateway on their cluster and they experienced the same issue.
Example where 'bad' cilium pod cannot deliver to workload on the same node.
hubble observe -f --identity 8
Feb  5 21:17:44.348: <IP_ADDRESS>:38265 (ingress) -> test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-endpoint FORWARDED (TCP Flags: SYN)
Feb  5 21:17:44.348: <IP_ADDRESS>:38265 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 21:17:49.349: <IP_ADDRESS>:65017 (ingress) -> cilium-gateway/cilium-gateway-internal:443 (world) http-request FORWARDED (HTTP/1.1 GET https://httpbin.mydomain.tld/)
Feb  5 21:17:49.350: <IP_ADDRESS>:65017 (ingress) <- cilium-gateway/cilium-gateway-internal:443 (world) http-response FORWARDED (HTTP/1.1 503 5001ms (GET https://httpbin.mydomain.tld/))
Feb  5 21:17:49.390: <IP_ADDRESS>:34395 (ingress) -> test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-endpoint FORWARDED (TCP Flags: SYN)
Feb  5 21:17:49.390: <IP_ADDRESS>:34395 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 21:17:51.642: <IP_ADDRESS>:38265 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 21:17:56.546: <IP_ADDRESS>:34395 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 21:17:59.745: <IP_ADDRESS>:38265 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 21:18:04.652: <IP_ADDRESS>:34395 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 21:18:16.385: <IP_ADDRESS>:38265 (ingress) <- test-zone/httpbin-5668f47445-mk7r2:80 (ID:96395) to-network FORWARDED (TCP Flags: SYN, ACK)

Functional on every other node
hubble observe -f --identity 8
Feb  5 17:28:25.419: <IP_ADDRESS>:39087 (ingress) -> test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-endpoint FORWARDED (TCP Flags: SYN)
Feb  5 17:28:25.419: <IP_ADDRESS>:39087 (ingress) <- test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-network FORWARDED (TCP Flags: SYN, ACK)
Feb  5 17:28:25.419: <IP_ADDRESS>:39087 (ingress) -> test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-endpoint FORWARDED (TCP Flags: ACK)
Feb  5 17:28:25.420: <IP_ADDRESS>:39087 (ingress) -> test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb  5 17:28:25.423: <IP_ADDRESS>:39087 (ingress) <> test-zone/echo-679b5c479f-ct8rr (ID:90104) pre-xlate-rev TRACED (TCP)
Feb  5 17:28:25.427: <IP_ADDRESS>:39087 (ingress) <- test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-network FORWARDED (TCP Flags: ACK, PSH)
Feb  5 17:28:30.433: <IP_ADDRESS>:39087 (ingress) <- test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-network FORWARDED (TCP Flags: ACK, FIN)
Feb  5 17:28:30.434: <IP_ADDRESS>:39087 (ingress) -> test-zone/echo-679b5c479f-ct8rr:80 (ID:90104) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

I have tested with Cilium 1.15.0 on Kubernetes 1.29.0-k3s1.  Nodes using linux kernel 5.10.160-rockchip
autoDirectNodeRoutes: true
bandwidthManager:
  enabled: true
  bbr: false
bpf:
  masquerade: true
cluster:
  name: home-cluster
  id: 1
containerRuntime:
  integration: containerd
  socketPath: /var/run/k3s/containerd/containerd.sock
endpointRoutes:
  enabled: true
hubble:
  enabled: true
  metrics:
    enabled:
      - dns:query
      - drop
      - tcp
      - flow
      - port-distribution
      - icmp
      - http
  relay:
    enabled: true
    rollOutPods: true
  ui:
    enabled: true
    rollOutPods: true
    ingress:
      enabled: false
ipam:
  mode: kubernetes
ipv4NativeRoutingCIDR: "<IP_ADDRESS>/16"
k8sServiceHost: "<IP_ADDRESS>"
k8sServicePort: 6443
kubeProxyReplacement: true
kubeProxyReplacementHealthzBindAddr: <IP_ADDRESS>:10256
l2announcements:
  enabled: true
loadBalancer:
  algorithm: maglev
  mode: dsr
localRedirectPolicy: true
operator:
  replicas: 1
  rollOutPods: true
rollOutCiliumPods: true
securityContext:
  privileged: true
routingMode: native
gatewayAPI:
  enabled: true
  secretsNamespace:
    create: false
    name: cilium-secrets
ingressController:
  enabled: false
  loadbalancerMode: shared
  enforceHttps: false
  secretsNamespace:
    create: false
    name: cilium-secrets
  service:
    allocateLoadBalancerNodePorts: false

and with some of the most recent images from the main branch ci https://github.com/cilium/cilium/actions/runs/7788397285/job/21237711996 using the following values
image:
  override: "quay.io/cilium/cilium-ci:840cc579b7b5aac24ba00c4d8c8f1d10334882fa"
operator:
  image:
    override: "quay.io/cilium/operator-generic-ci:840cc579b7b5aac24ba00c4d8c8f1d10334882fa"

So far the only workaround I have found is keep restarting cilium daemonset pods until the 'bad' pod is on a Kubernetes control-plane that hosts no website application pods, mitigating the impact.

Any updates?

I was hoping that https://github.com/cilium/cilium/pull/32367 released in v1.15.6 would solve this. Didn't. So what we do to currently solve it is also to use hostLegacyRouting. This is annoying as that is less performant. But, we can live with it on the clusters whereon we use the Gateway API through Cilium.


Update README.md

Changelog:

Added "Table of Contents" index
Formated all the features, with purpose and usage sections

@m-sameer I have suggested some changes; please implement them, and I will acept this PR.

Doing. :smile:

Good, and thanks for the PR.
On Feb 5, 2018, 18:06, at 18:06, Mohammad Sameer<EMAIL_ADDRESS>wrote:

Doing. :smile:
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/FadedCoder/Monika-Bot/pull/1#issuecomment-363072337

Accepted.

Thanks for the merge. :)


xWebApplication SSLFlags - Results from Get do not match Set/Test parameters

When running Get-TargetResource the SslFlags parameter returns an array which may contain up to 4 unique values:
Ssl, SslNegotiateCert, SslRequireCert, Ssl128
However, the Set/Test methods for SslFlags will only allow the following due to 'ValidateSet' restrictions:
Ssl, SslNegotiateCert, SslRequireCert
This creates an error when trying to use the output of Get for the input of Set/Test. It also prevent users from enforcing Ssl128 via DSC

Code has been merged. resolving issue.


Bring code coverage to 100%

current status:
npm run test:coverage

...
=============================== Coverage summary ===============================
Statements   : 97.09% ( 400/412 )
Branches     : 91.2% ( 114/125 )
Functions    : 92.86% ( 65/70 )
Lines        : 97.09% ( 400/412 )
================================================================================

[x] claim this issue (@chrifpa)
[x] setup the repo locally and make sure that all tests pass when you run npm test
[x] get a coverage report by running npm run test:coverage. That also creates a coverage folder. Open coverage/lcov-report/index.html for a nice UI that shows you exactly what lines / branches etc are not yet covered
[x] add a test to fix one of the not covered lines / branches. Commit with WIP #66 coverage
[x] start a pull request #78
[x] write tests until 100% coverage is reached, just keep commiting all changes, don’t worry the amount of commits, we will clean them up before merging. Comment below when you are ready for review :)

Ping us in the Hoodie Chat or on Twitter if you have _any_ questions

Starting to work on it now :)

wo00ooohoo000oo :clap: done via #78
That was a hell of a pull request, thanks a lot! And welcome to Hoodie :wave: :dog: I’ve invited you to the Hoodie Contributors team, you should see an invitation on https://github.com/hoodiehq.
Mind us giving you a shout out on Twitter? Would you like to keep contributing to Hoodie?

Thanks for the invite!

Mind us giving you a shout out on Twitter?
Not at all :)

Would you like to keep contributing to Hoodie?
Absolutely! Anything specific that needs doing or should I just browse the issues?

Absolutely! Anything specific that needs doing or should I just browse the issues?

I don’t have anything specific right now, browse around and see if you can find something :) Ping us anytime if you have any questions


xs:all nodes ignored by DomGen.exe

Hi there,
Not quite sure how to create a pull request so just creating this as an issue.
The DomGem.exe tool currently doesn't actually do anything xs:all nodes in a schema, it just ignores them. I added a change to XmlSchemaTypeLoader.cs to fix this.
XmlSchemaTypeLoader.cs, line 797, insert:
// Check for <xs:all> node and continue processing it:
XmlSchemaAll allNode = particle as XmlSchemaAll;
if (allNode != null)
{
    foreach (XmlSchemaElement elementInAll in allNode.Items)
        WalkParticle(elementInAll, nodeType);
}
else
{
... the old parsing the XmlSchemaSequence etcetera...
}

That should fix it. :-)
Best,
Jelle

Thanks for finding out the issue and posting a fix.
I make the suggested change in the next update.
Alan

Fixed
Thanks for reporting the bug.
Thanks
Alan


Compile failed

[ERROR] Failed to execute goal com.mycila:license-maven-plugin:3.0:check (default) on project driver-pulsar: Some files do not have the expected license header -> [Help 1]

mvn license:format

Thank you for tracking this issue. I think it's worth PR into Readme.


Adjust the title level of "Validation rules" on custom-resource-definitions.md

Adjusted the title level of "Validation rules" from h2 to h3 to align with the "Advanced topics" section it belongs to.

/lgtm
/approve


always store the abs path to files

If a topology was loaded like: PothosGUI relative/path/to/topology, it should be stored absolute so it can be opened from any working directory next time the PothosGUI is opened.

https://github.com/pothosware/pothos-gui/commit/b576843776ab175017bda35df47e61a2dceb2589


Make CMakeLists.txt ready to release

Current CMakeLists.txt is not ready to be used. Need some work to prepare it.
TODO:

[x] setup install settings
[x] support add_subdirectory usage (test if is the main project, disable tests and examples if not)
[x] make sure all variables and options have prefix UREACT_

make sure all variables and options have prefix UREACT_
To make sure about the name of the options:
cmake -LA . | awk '{if(f)print} /-- Cache values/{f=1}' | grep -v '^CMAKE_' | grep -v '^ProcessorCount_'
Sample output:
build_examples:BOOL=ON
build_tests:BOOL=y

Need to add such a check in the Makefile as part of the repository sanity check.

Install would be added with a separate issue

Install target is added


fix: filter out duplicate EIP-6963 connectors on mount (server-side)

Description

During on mount if ssr is true in createConfig the EIP-6963 connectors might be introduced multiple times. This is happening in hydrate.ts. By the time we add them they were already added in createConfig.ts
Created a simple function that would filter out duplicated connectors here

Additional Information
Before submitting this issue, please make sure you do the following.

[ ] Read the contributing guide
[ ] Added documentation related to the changes made.
[ ] Added or updated tests (and snapshots) related to the changes made.

Closing in favor of #3715


Ensure all spo contenttype commands have all available list options

Noticed some sort of inconsistency regarding list options. To reference a list, you could use 3 ways: title, id or URL.
Below is a summary of all commands with their missing option(s).

Command
Options to add

spo contenttype add
--listId, --listUrl

spo contenttype get
--listId, --listUrl

spo contenttype field remove
--listId, --listUrl

@milanholemans I could work on this!

Great! All yours @MathijsVerbeeck!


Add smptest target for AMPI tests

megampi currently fails when running with +p4 ++ppn 2 +vp4 with the following error: Recv'ed bad value for large message send/recv
I'm not sure if this is an issue with the test, the test configuration, or a bug in AMPI, but I figured I'd put the PR up for discussion.

@evan-charmworks @SamTWhite This is another AMPI related PR that should probably be discussed by the AMPI subgroup as to its relevance. Not sure if something else ended up fixing the issues we were seeing here or not.


feat(node): poc of a node/typescript integration

Proposed changes
A proof-of-concept of an integration that works with NodeJS and Typescript. It currently downloads and installs the Ockam Command on-demand using an install script which (should) be executed as part of npm install once this is packaged and published to the npm registry. All methods are ultimately running a child process with Command rather than integrating with the Rust bindings directly in any way.

This is awesome,
Backed up your branch here: https://github.com/build-trust/ockam/tree/backup/glenngillen/node-spike
I refactored this PR:

to be a package in implementations/typescript, so we can benefit from shared infra and publish this as @ockam/command on npm registry.
Remove the child_process package, which seems unmaintained and held as a place holder by the npm security team. Replaced it with node's child_process.
Added some infra for testing with jest and formatting with prettier.

Instructions to try with pnpm:
https://github.com/build-trust/ockam/blob/975f0f3e6323d4681e5355b04100d8b354905c50/implementations/typescript/DEVELOP.md

Merged, will land the other commands from https://github.com/build-trust/ockam/tree/backup/glenngillen/node-spike in a follow up PR.


copy files in webpack stream: Issue #15

Checking if the file we are trying to copy is one of the current assets
in webpack. If this is the case then we set the source of file to copy
to the one already in the assets. Otherwise we proceed as before.

We currently ignore files that are already in compilation.assets unless the force option is passed, so I'm not quite sure what you're proposing


OPT: Support list query parameters

Before this change it would URL encode the python representation of the list.
With this change it will pass the query parameter as ?multivalue=value1&multivalue=value2
I was doubting whether to use isinstance(.., list) vs something more generic like isinstance(.., Collection), but i believe list is the most appropriate typing. It has order, and it clearly defines what would be expected when passing a list as a query parameter.

Also, please don't care fixing the 3.8 checks – it's end-of-life, I'll remove the support completely


Please support OME-NGFF Zarr

@kgabor I was wondering what the status of that is right now? Does it work already? Also tagging @tpietzsch @krokicki


Update Emoji Picker Search to use Trie Search

cc @roryabraham since you were heavily involved in the initial creation of the trie search
Details
This PR changes the emoji picker search to use the Trie Search which is much more efficient.
Unfortunately, the results are not in the same order as the initial list, so they don't show up how you would expect. To fix this, I gave each emoji an index so that regardless of which order the search returns them in, we can put them back in the right order.
Fixed Issues
$ https://github.com/Expensify/Expensify/issues/239099
PROPOSAL: N/A
Tests
Web and Desktop Only:

Open the emoji picker
Search for an emoji like "smile"
Make sure you see the emojis that you expect, in the order that you expect.

WRONG

RIGHT

All Platforms:

Type a colon code emoji into a chat e.g. :smile:
Make sure that the emoji is shown correctly

[x] Verify that no errors appear in the JS console

Offline tests
This is not reliant on any API requests, so offline tests are not needed
QA Steps
Web and Desktop Only:

Open the emoji picker
Search for an emoji like "smile"
Make sure you see the emojis that you expect, in the order that you expect.

WRONG

RIGHT

All Platforms:

Type a colon code emoji into a chat e.g. :smile:
Make sure that the emoji is shown correctly

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/42391420/211682458-4aba09e6-01f3-4870-8d0b-42ace2bfcdd1.mp4

Mobile Web - Chrome
https://user-images.githubusercontent.com/42391420/211681569-061cb8c0-0b68-4bea-a22b-147892c375d0.mp4

Mobile Web - Safari
https://user-images.githubusercontent.com/42391420/211679817-ae4cd08b-1038-4e8d-b203-5749419e911f.mp4

Desktop
https://user-images.githubusercontent.com/42391420/211682468-db2139cd-aa11-4f20-8df9-7801264eadec.mp4

iOS
https://user-images.githubusercontent.com/42391420/211680060-c69c1023-875f-4017-94e6-6efd652b0ecb.mp4

Android
https://user-images.githubusercontent.com/42391420/211681518-6eb33d0b-a0ca-4e6f-93cc-ebd2a0cd528b.mp4

Starting to review this.

@stitesExpensify How is the search supposed to work?
We have this emoji:
{
        name: 'stuck_out_tongue_closed_eyes',
        code: '😝',
        keywords: [
            'prank',
            'stuck_out_tongue_closed_eyes',
            'eye',
            'face',
            'horrible',
            'taste',
            'tongue',
        ],
        index: 27,
    }

and if I search by the name, it doesn't show up in the result.

Hm playing with this even more, I think our trie might not be building correctly....
Prod (with expected results)
https://user-images.githubusercontent.com/42391420/211886320-5997966a-73ca-45d3-aad1-e7e160e65ba9.mp4
This branch (st doesn't bring up stuck_out_tongue but stu does)
https://user-images.githubusercontent.com/42391420/211886377-178d54fb-9c00-4a60-bd6b-ba916a5aa8ad.mp4
CC @roryabraham @Karim-30

@stitesExpensify Ops, this regex missing _,+,- and digits, we should use this instead :[a-zA-Z0-9_+-]{1,40}$
https://user-images.githubusercontent.com/108357004/212131073-ab5421d6-551b-458b-b181-adac7856f738.mov
I'm working on the order issue now.

@Karim-30 I made those changes and did not get the expected results.
The regex change allowed searching stuck_ to find the correct emojis, but that did not fix the fact that searching st does not show 😛 as an option.
Additionally, using push did not properly order the emojis unfortunately.

@stitesExpensify getAllMatchingWords missing the limit arg.
const nodes = emojisTrie.getAllMatchingWords(emojiData[0].toLowerCase().slice(1), limit);

https://user-images.githubusercontent.com/108357004/212261384-373cbbb4-6967-46e8-b05d-6b9f981d7a19.mp4

Additionally, using push did not properly order the emojis unfortunately.

It work without this sorting
https://user-images.githubusercontent.com/108357004/212262769-7be853af-0ff9-4f92-b6a4-677acdf21539.mp4

@stitesExpensify Is it ready for the next round of review and testing?

@mananjadhav not quite.
@Karim-30 I just pushed my changes, am I missing something? I'm still getting the wrong order.

In my understanding of the data structure, we won't always find emojis in the same order as they appear in the main list right? That would only happen if they were alphabetically listed?

Swapping only one to push, and not the other fixed it for :smile: specifically. I chatted with @roryabraham  about this and we decided it doesn't really matter if the search results come up in the normal order. Slack, for instance, sorts by most used first which is something we probably want to do in the future, but we can discuss that when we create the design doc for the potential replacement of trie search.

Updated and ready for review!

@stitesExpensify The Trie searching order is FIFO for nodes.
If you insert Robert then Rock then Robertson into the Trie and search for Ro, because the b node in (Robert) inserted before the c node in (Rock), the b node and all of its children will appear before the c node and all of its children, the result will be :

Robert.
Robertson
Rock.

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/3069065/212556483-77debda7-a2cc-4935-b499-875bca7ad5b2.mov

Mobile Web - Chrome
https://user-images.githubusercontent.com/3069065/212556711-e974e63b-5e07-4a85-b402-d8b6eeeac9db.mov

Mobile Web - Safari
https://user-images.githubusercontent.com/3069065/212556793-4f60e248-59af-4b5b-8f21-a2d2751ba837.mov

Desktop
https://user-images.githubusercontent.com/3069065/212556844-2a3a52d6-17f6-4665-9231-4c00994aa448.mov

iOS
https://user-images.githubusercontent.com/3069065/212556871-d341c239-95a4-4e38-a552-e02e165d8fc8.mov

Android
https://user-images.githubusercontent.com/3069065/212556875-7325a88c-3402-4ad1-9246-a57d51117a44.mov

Thanks @stitesExpensify and @Karim-30 for the changes here. These test well now and faster too. I've completed the checklist and attached the screencasts for all platforms (while for search only Web and Desktop are relevant).
@Luke9389 @roryabraham All yours to merge.

🚀 Deployed to staging by @roryabraham in version: 1.2.56-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/AndrewGable in version: 1.2.56-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Fix mitmproxy

Motivation for this change
#35499
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using (none)
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Thanks!


[Feature] Be able to update FileAttestationTrustStore during execution

Feature description
Hello,
We are using Matter in a Linux board (as commissioner).
We load the PAA certificates from a directory and we have a script that update this directory based on the DCL when we need (for exemple every 24h).
Currently it seems that we can't update the PAA/CD certificates during the "runtime" (after the initialisation of matter stack), so if we update the certificates we need to restart the software that run matter stack.
It should be good if something is implemented that permits to update the PAA certificates during the matter-stack execution.
This is the reasons why we think it's currently not possible:

We instanciate FileAttestationTrustStore on startup, so the certificates from the specified folder are loaded at instanciation.
We get the DAC verifier like this: chip::Credentials::GetDefaultDACVerifier(trustStore)
Then we use it.

The function chip::Credentials::GetDefaultDACVerifier(trustStore) say's:
/**
 * @brief Get a singleton implementation of a sample DAC verifier to validate device
 *        attestation procedure.
 *
 * @param[in] paaRootStore Pointer to the AttestationTrustStore instance to be used by implementation
 *                         of default DeviceAttestationVerifier. Caller must ensure storage is
 *                         always available while the DeviceAttestationVerifier could be used.
 *
 * @returns a singleton DeviceAttestationVerifier that satisfies basic device attestation procedure requirements.
 *          This has process lifetime, so the paaRootStore must also have
 *          process lifetime.  In particular, after the first call it's not
 *          possible to change which AttestationTrustStore is used by this verifier.
 */

It say's:
This has process lifetime, so the paaRootStore must also have process lifetime.
In particular, after the first call it's not possible to change which AttestationTrustStore is used by this verifier.

What is unclear:

Is it possible to pass a static truststore object that can be updated after calling this function ?
If yes, one possible solution (for us) is to re-implement  FileAttestationTrustStore and add a function to update mPAADerCerts ? (what do you think?) Did we have to do something else to take effect ?

Also some questions:

Can you confirm that currently (AS-IS) It's not possible to update the PAA certificates when the matter stack is initialized ?
If yes, is it wanted ? what is the reason ? Is it going to change ?

Platform
all
Platform Version(s)
No response
Anything else?
No response

The trust store will be asked any time a trust check needs to be made, right?  So if you change its internal state, that affects checks after that point.


User - readme: names of all members

a user should be able to read the names of all members of the team in the readme, so they know who worked on this project

Everyone is added!


Fix value card layout

Summary

value card multi-value layout fixes
use carbon colors in dashboard story for charts
default to 16 col grid layout

:tada: This PR is included in version 1.63.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.0.0-beta.9 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.63.3 :tada:
The release is available on npm package (@release-1.x dist-tag)
Your semantic-release bot :package::rocket:


Specify how mini_batch_size works differently based on datastore

I have a tabular dataset backed by SQL, and it seems to me that mini_batch_size had no effect then (similarly to how it can not split a large parquet file). This should be mentioned somewhere.
It would also make more sence is the mini_batch_size specified nr of rows;-)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fd8b44e8-87c3-66be-5e41-3a1bda9cb586
Version Independent ID: fa97464c-1e45-cbc2-8126-02ee2f6dea1f
Content: Troubleshooting the ParallelRunStep - Azure Machine Learning
Content Source: articles/machine-learning/how-to-debug-parallel-run-step.md
Service: machine-learning
Sub-service: core
GitHub Login: @tmccrmck
Microsoft Alias: trmccorm

Thanks for the feedback! I have assigned the issue to the doc author to update appropriately.

Hi @epa095, thanks for your report! Great point, I've added a note warning future users that SQL-backed Tabular datasets cannot be partitioned. You should see the changes live in a few hours. I'll proceed to #please-close this issue.
I've also shared your feedback regarding mini_batch_size with the product team.
If you feel that your documentation issues hasn't been resolved, feel free to open a new issue with additional information.