Warning "unable to get a valid CUDA device id from the renderer. Defaulting to GPU0."

Describe the bug
A cuda toolkit version setting might changed while verifing installation by python -m omnigibson.examples.robots.robot_control_example --quickstart.

And it may have caused the subsequent infinite loop later:
2024-12-06 08:41:50 [8,581ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-12-06 08:41:50 [8,582ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-12-06 08:41:50 [8,583ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.

To Reproduce
Steps to reproduce the behavior:

Run python -m omnigibson.examples.robots.robot_control_example --quickstart

My environment variable: (OMNIGIBSON_HEADLESS=1)
export OMNIGIBSON_ASSET_PATH="/root/autodl-tmp/digital-cousins/omnigibson_data/assets"
export OMNIGIBSON_DATASET_PATH="/root/autodl-tmp/digital-cousins/omnigibson_data/og_dataset"
export HF_ENDPOINT=https://hf-mirror.com
export SAM2_REPO_ROOT=/root/autodl-tmp/digital-cousins/deps/segment-anything-2/sam2
export PYTHONPATH="${SAM2_REPO_ROOT}:${PYTHONPATH}"
export OMNI_KIT_ALLOW_ROOT=1
export OMNI_KIT_ACCEPT_EULA=YES
export OMNIGIBSON_HEADLESS=1
export XDG_CACHE_HOME=/root/autodl-tmp/.cache
export LD_LIBRARY_PATH=/root/miniconda3/envs/acdc/lib/python3.10/site-packages/isaacsim/extscache/omni.usd.libs/bin:$LD_LIBRARY_PATH
export OMNIGIBSON_GPU_ID=0

My cuda environment:
(acdc) root@autodl-container-07904e86c4-e8dc2739:~# nvidia-smi
Fri Dec  6 17:04:26 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        On  |   00000000:1F:00.0 Off |                  Off |
| 30%   29C    P8             20W /  450W |       1MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
(acdc) root@autodl-container-07904e86c4-e8dc2739:~# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Expected behavior
Don't change the toolkit version.
Screenshots

Desktop (please complete the following information):

OS: [e.g. Ubuntu 22.04]
Isaac Sim Version [e.g. 4.1.0]
OmniGibson Version [e.g. 1.1.1]

Additional context
Some other errors but might be caused by no moniter display: (I had set OMNIGIBSON_HEADLESS=1)

Full logs

Logs
logs.txt

APP Ready↑

cremebrule/digital-cousins/issues/16
I have tried running it for 7 hours, but it still prints OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0. I suspected that it is still compiling shaders. 😭😭😭

Hi @Snowflake-Pink ,
Hmmm I think this is the fundamental issue:

2024-12-06 08:41:43 [1,227ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-12-06 08:41:43 [1,227ms] [Error] [carb.graphics-vulkan.plugin] vkEnumeratePhysicalDevices failed. No physical device is found.
2024-12-06 08:41:43 [1,227ms] [Error] [carb.graphics-vulkan.plugin] No physical device is found.
2024-12-06 08:41:43 [1,228ms] [Error] [gpu.foundation.plugin] carb::graphics::createInstance failed.

Seems like the GPU wasn't found for some reason. Can you verify that CUDA can be used in the python process, e.g.: by importing torch and running torch.cuda.is_available()?

Hi @cremebrule ,
I am excited to announce that the solution to this issue!! Thanks a lot to the engineer from AutoDL!! 🥳🥳
This is probably that libegl1 or other dependencies about Vulkan are not installed (especially on cloud server environment).
Run Command:
apt update && apt install -y libegl1

OR

apt update && apt install -y vulkan-tools libvulkan1 libsm6 libglu1-mesa-dev libegl1

To solve. 🥳🥳
Thanks again to the engineers from AutoDL.

This issue will be closed in a few days if there are not other comments.

@cremebrule I greatly appreciate your kind help. This issue is closed.


Improve ol

Still have space for improving

still have space to optimize since converting large BitVector to DitStr is time consuming. However, good enough for this pr


commit url breaks for private packages

Clicking on the commit id of a private project yields a 404 because of insufficient access.
Is it possible to open the URL in the default OS browser instead of an atom webview?

This used to be possible in an older version of Atom, but the api for opening anchors with target="_blank" was removed at one point. They may have exposed a new API in recent versions. i'll check

Also @chrishiestand you should be able to login to the private service from inside the atom webview as a stop gap. Is that possible?

Yes, logging in is a workaround. I'd just prefer to not have atom authorized for my account.

@chrishiestand I've created a pull request that opens the url in your default browser.
https://github.com/alexcorre/git-blame/pull/136


Add Queryable interface

I came across https://github.com/jmoiron/sqlx/issues/344 and thought it may be useful to open a PR proposing this change since it looks like a few folks could find it useful.

When writing applications it's useful to pass a sqlx.DB and a sqlx.Tx
interchangeably so that you can compose functions that are runnable in
isolation or as part of a transaction.
This introduces the Queryable interface, which includes the common
exportable methods shared between sqlx.DB and sqlx.Tx so users of the
package don't have to implement it themselves.
This also adds tests that validate any new shared methods are
added to the interface.

This is actually very helpful, even tho  it hasn't being merged yet, but it does help me to implement Unit of Work

I think it would make more sense to split the giant interface into a bunch of smaller ones. I do completely agree that having the common methods to sqlx.Tx and sqlx.DB is really useful. But, I also think that a giant interface is hard to mock and deal with.
The Queryable interface carries a lot more functionality than is usually needed(in the use cases that I've seen). Instead, having interfaces like: Getter, NamedExecer, Selector, and their Context alternatives, makes more sense for small CRUD operations, where you also need to mock test them.
I think that composing Queryable out of smaller interfaces would also tie in better with the current style. There already are some interfaces like that.
https://github.com/jmoiron/sqlx/pull/809/files#diff-e6a1c006245175d26ca4650b0bf4fc8484a43de15056a1634e83e825193e9018R244-R248

I haven't had a chance to discuss this with @ardan-bkennedy; that's why the label is set to needs testing.
I think we don't need this interface in the package at all. This should be an application concern to define/use. That way, the application establishes the functionality it expects. We should ensure that both sqlx.Tx and sqlx.DB always have identical signatures for the shared methods to make it possible to define such interfaces. In this case, I suggest creating this interface but not exporting it.

This interface already exists and I use it today to substitute a DB for a Tx when needed.
sqlx.ExtContext
I haven't looked too closely to this issue, but why won't this interface work for you?


Bugs in Password Reset Flow

There are 3 major bugs in password reset flow. Detected manually while updating the README.

Bug I Any one can reset password for anyone if they have the token. This is incorrect. We should reset the password only if the token matches the email authorizer. To reproduce, register two users, request password reset link for both, look into the DB and call password reset.

Bug II Right now you can use the same password reset token to reset user password multiple times. This is incorrect. A token should only be used and "claimed" once. To reproduce, call the same password reset API multiple times

Bug III User can generate multiple password reset tokens. This is fine. However, once a particular token is chosen, we should invalidate all active tokens and not rely on password token expiry. To reproduce, create multiple password reset tokens for a user, use one to reset password. After a while use the other token to reset password again.

Fixed by #56


discord.me/dks

My friend has 2 days in discord.me/dks and today he received INSULT from iDevastate:
apple has try hestitate my friend, he didn't comment if he can send dm or not. SO my brother write it directly in general and iDevastate directly insult my brother.

Today at 11:27 AM
this is inhuman. when my brother try extreme blocker from here : http://ps4exploits.darksoftware.xyz/5.05/Update-Blocker-Extreme/ it comment incompatible firmware 5.07
apple i have 3 private message from you, stop hestitate me. First you need to ask me in oublic  if i want to accept direct message.

iDevastate - Today at 11:31 AM
lol
whatever stuff u smoke must be that good that u lost of what u is talking, fool
LM, What actions you are gonna take now. This is not the first time that discord.me/dks has insulted peoples who are users and they don't have developers knowledge. This don't mean iDevastate can go and insult everybody. LM first i'm human and second and we are not shits, to treat peoples lie shits.
I don't have twitter, but my friend do his magic and he find for one week discord.me/dks has been reported more than 3 times, users like radziu has insulted other person too, adzx has send NUDE PICTURES, jamhouse insulted again other persons this happend for 1 week.
I'm from Australia and here we don't like RUDE peoples.    In the moment we can't remember the word, but it is near ABUSE, include this is not first time that discord.me/dks is sending PORNOGRAPHICS PHOTOS.


Magic-js prevents the use of Polygon nodes provided by Alchemy.com for security reasons

Our support team can most likely get this sorted for you. Please follow the steps provided in our documentation.
Also, as a general open-source tip, it helps to add as much context to your GitHub issue as possible. Thankfully, I have a good feeling that I know what you are asking for here, but just as easily someone else might not understand the underlying problem.


yo whats ur user and tag?

ok

For discord

yo this is my main so ama add you my name is x btw so when i add u accept it

yo


backport: Merge bitcoin#21750

bitcoin backports

Closing for now; feel free to reopen if it comes to a reviewable state


Where is 0.10 ?

I am using 0.10.0.-c6e0eba0-en but I can't find no tags, release or even the commit with this hash...
In the tags I see the new, few days old, release and then the previous one is 0.9.2.
?

Mia logunda, it looks like my clean up was a little too aggressive. I've added the latest 0.10 tag back now:
https://github.com/onetimesecret/onetimesecret/releases/tag/v0.10.1

Haha! No problemo! :D


Phys Spec - full text search

Use a proper full text search index against physical specifications
Relates to  #1996

Woohoo Issue 2000!   ;)

I don't know if this is best suited for full text.  The SQL server full text requires completed words, whereas the current implementation works with 'contains'.  Because spec names are often one word with - or _ separators.  Will using full text make it less searchable?

Agreed, closing - thanks


Add new filters to reservations

🛠 Changelog

See ticket for filters.
Also changes CustomerTypeChoice, ReservationStateChoice, and ReservationTypeChoice enum to uppercase. Refactor some mutations and tests to support this.

🧪 Test plan

Automated tests

🚧 Dependencies

Requires frontend to implement new enums. Should be pretty simple since the are now GraphQL enums instead of strings.

🎫 Tickets

TILA-3400

Forgot to add most tests (I was busy fixing old ones). Added one for is_recurring. Should add quick ones for the rest.

Added tests for the new filters

Why change some and not all State -> ReservationStateChoice?

It's probably a library issue: Enum name is constructed differently depending on where it is used (unless I do some extra work), and it's not visible to the backend that the name is different. Just point out where the name is different and I can make the change.

The needed enum changes are in the ticket. These should be fixed before mering

Enums should now be correct in the GraphQL schema.

/ff


Derivation does not work with value case classes

Magnolia version: 0.13.1
Scala version: 2.13.1
Code snippet:
final case class ServiceName1(value: String) extends AnyVal

test("derive for a value class") {
  val showServiceName: Show[String, ServiceName1] = Show.gen[ServiceName1]
  showServiceName.show(ServiceName1("service"))
}.assert(_ == "service")

Compilation error:
too many arguments (2) for constructor ServiceName1: (value: String)magnolia.tests.ServiceName1
[error]       val showServiceName: Show[String, ServiceName1] = Show.gen[ServiceName1]

Seems the functionality was broken in the scope of #215 MR: https://github.com/propensive/magnolia/pull/218

Highly likely the condition is incorrect for the value case classes https://github.com/propensive/magnolia/blob/master/core/shared/src/main/scala/magnolia.scala#L286, but I'm unable to find out what is exactly wrong.

Thanks, unfortunate that we didn't have a test case for this 😞


Bugfix: response header not reflecting correct version of http #1689

link to the issue: #1689

This is a good start and I think only one change is needed, which is to move Response.setProtocol() to Response.Builder.protocol() so that Response remains immutable.

Change committed!

Thanks!


Force 'records' argument type to list

Latest terraform versions are forced to check input types and faults with error:
Error: module.implementation.module.route53-record.aws_route53_record.this: records: should be a list

@kyxap1 Thank you. I merged MR and marked with tag v0.4


test: fixing a race in cx_limit_integration_test

Fixing a race where the TCP session closed the sockets before the accept counter was updated, and the next connect failed.
Risk Level: n/a (test only)
Testing: yes
Docs Changes: no
Release Notes: no
Hopefully fixes #11841

/backport review
This has been source of stable release CI flakes cc @lambdai

Failed due to a different flake. Will force merge.


Price feed for taker and maker

In order to be able to price the CFD as the maker we will need the current price feed. Given, that the maker might want to open counterpositions on Bitmex we should potentially use the Bitmex price feed.
This will enable: https://github.com/comit-network/hermes/issues/33

See also: https://github.com/comit-network/hermes/discussions/77
Bitmex price feed sounds good to me.
Oracle and daemons should likely have independent price feeds at runtime (i.e. don't get the feed from the oracle).


python310Packages.mat2: 0.13.1 -> 0.13.2

Description of changes
Diff: https://0xacab.org/jvoisin/mat2/-/compare/0.13.1...0.13.2
Changelog: https://0xacab.org/jvoisin/mat2/-/blob/0.13.2/CHANGELOG.md

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@ofborg build python310Packages.mat2 mat2 metadata-cleaner python311Packages.mat2


Fix dropdown example in docs (Fix #20410)

Fix #20410

This is a dupe of #20411

Indeed, I should have search more.
Although I also removed the part of the doc that recommend to use data-target for links as it seems it's irrelevant in Bootstrap v4 (The JS find the parent .dropdown element without requiring a data-target or an href).


Fixed bad URL escaping of queries and labels

If you happened to provide a query or label that included certain
syntactically meaningful URL characters like '#' or '&', the resulting URL
would be broken as those characters would appear in the raw and not be
escaped.
This is because Uri.encodeFull is a terrible function that should be
abolished. It lures people into thinking that you can put arbitrary strings
into URL segments & parameters and have Uri.encodeFull automagically escape
them for you while keeping the rest of the URL syntax intact. Alas, this is
not the case as it won't -- and can't -- escape characters that are
syntactically meaningful in URLs like '/', '#', '?', etc. The result is that
some characters get escaped but not others and any query parameter that
happens to include them will break the URL.
A more effective solution is to create a proper Uri from the components and
then stringify it using toString().

Wow I did not see this. Sorry for being so late, I'll merge it in, thank you!


Implement GUIDs as user identifiers

All users will have at least one user identifier, generated during registration
They are exposed to the user, either on the profile page or in the settings page
When sharing information with an organization (through the authorization code grant flow) a GUID is generated for that user-org pair
-One is always shared. Therefore we should not need to implement a scope for this.
Allow them to be used as user identifier in api calls (required for at least the user info call, especially when we eventually drop username support)
Also allow them to log in???
Find a way to transition existing sharing of username and replace this by sharing GUID as primary identifier

Duplicated by #581


Error due to NG ivy incompatibility

ERROR in node_modules/nativescript-accordion/angular/index.d.ts:107:22 - error NG6002: Appears in the NgModule.imports of HelpModule, but could not be resolved to an NgModule class.
This likely means that the library (nativescript-accordion/angular) which declares AccordionModule has not been processed correctly by ngcc, or is not compatible with Angular Ivy. Check if a newer version of the library is available, and update if so. Also consider checking with the library's authors to see if the library is expected to be compatible with Ivy.
If the plugin won't be updated anytime soon is there perhaps a way I can handle this in ngcc.config file? Any help will be greatly appreciated.

@Meyer-dev a rewrite is scheduled for this just needs some time

Ok great, thanks for letting me know. Iooking forward to it. Is there a way
to avoid this issue for now by some config in ngcc or something like that?
On Thu, 13 Aug 2020, 1:29 am Osei Fortune<EMAIL_ADDRESS>wrote:

@Meyer-dev https://github.com/Meyer-dev a rewrite is scheduled for this
just needs some time
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/triniwiz/nativescript-accordion/issues/84#issuecomment-673159014,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHNDDIGEMY6FT7E25JX2473SAMQUVANCNFSM4P4TVJTQ
.

Hi, is there an update for this issue?

@FerhatKoca I started the update but will be for 7+


[RHO-4290] Consolidate logging interface to require channel to be specified

Description
Consolidated the log methods such that logging to default without specifying a channel is no longer an option.
Now it looks like:

Testing plan
Case by case basis, we will need to rollout one-by-one and case by case basis
JIRA + Optional Slack context
https://series-ai.atlassian.net/browse/RHO-4290
https://seriesai.slack.com/archives/C04QCDML1AR/p1726674375358839

New PR: https://github.com/series-ai/logger-lib/compare/main...jpersons/RHO-4290_require-log-channel?expand=1


#492 Parse duration cap implemented for PDFs

Initially explored a way to interrupt the indexer thread, but the Tika AutoParser doesn't acknowledge interrupts.  I ended up creating a 'capped' parse handler for the Tika parser so it won't run indefinitely.
New com.tle.core.freetext > optional.properties:
# Tuning config for attachment indexing.  Currently implemented for PDFs.
# The attachment indexing duration will go over this amount (in milliseconds),
# but will be checked each time the parser returns data to the controller,
# so it will be close
textExtracter.parseDurationCap=30000

# Tuning config for attachment indexing.  Specifies how many times to ignore
# checking that the duration has not gone over it's cap before checking.  Slight
# performance boost for higher values
textExtracter.durationCheckFrequency=15

@doolse  - not sure why codeclimate is complaining since I changed the interface TextExtracterExtension.   Unless you know of another place codeclimate is keying off of for the method signatures.
Any concerns with merging this pull request?  It'd be nice to get it in before 6.6 is branched.

@doolse - thanks for the review.  I've removed the duration check frequency parameter and hard-coded it to 200.


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/102016

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/109246


python312Packages.lancedb: init at 0.9.0

Description of changes
Developer-friendly, serverless vector database for AI applications. Easily add long-term memory to your LLM apps!
https://github.com/lancedb/lancedb
Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
https://github.com/lancedb/lance
closes https://github.com/NixOS/nixpkgs/pull/266845

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 325164 at 1fdd41e8 run on aarch64-darwin 1

4 packages built successfully:

python311Packages.lancedb
python311Packages.pylance
python312Packages.lancedb
python312Packages.pylance


BUGFIX: fix threshold

Taking a bit of a guess here...
From the code here:
https://github.com/ava-labs/ortelius/blob/0a18147c1974a22f11c9f8ac6db42eb89216bd3c/services/indexes/avm/db_writers.go#L237
A holder output is created.  And the threshhold is set to 0.  But if we see the output later here:
https://github.com/ava-labs/ortelius/blob/0a18147c1974a22f11c9f8ac6db42eb89216bd3c/services/indexes/avm/db_writers.go#L299
The row is considered a duplicate..  So it won't update the threshold.

merged with: https://github.com/ava-labs/ortelius/pull/63


Add npm support

It would be nice to be able to install this via npm

Fixed. See https://www.npmjs.com/package/hearthstone-card-images
npm install --save hearthstone-card-images
const images = require('hearthstone-card-images');

for (const card of images) {
  console.log(`${card.id} -> ${card.url}`);
}

The NPM package will be updated when cards are added and art is updated.


formfactor: use rpi instead of raspberrypi to apply to all rpi machines

Fixes #266

Now I've looked at the contents of machconfig, I think we can just drop recipes-bsp/formfactor entirely. Those settings aren't valid for all Raspberry Pi users.
What do you think?

I also think that recipe is "highly variable" and should be something configured in an end-user's layer and not here in the BSP layer. Tese variables are very dependent on the end-user, for example if she attached a keyboard or a touch screen.
However, oddly enough, this bbappend in meta-raspberrypi is used as an example in the official Yocto Documentation: https://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#using-bbappend-files
So maybe is less confusing to just leave it with the current very conservative defaults, or else the Yocto docs should be updated to use another example.

Ok. Agreed it's less confusing to leave as is for now. To my knowledge these settings haven't caused issues in the ~ 6 years they're been in the layer so no rush to rip them out. In which case this PR is good.


Depending on random seed, binary input data causes omega_hat to become imaginary in GMLVQ

Depending on the random seed, omega_hat might be real or complex:
import numpy as np
import pandas as pd

from sklvq import GMLVQ

#create fake data
N = 19
n = 1000
p = 0.1
np.random.seed(1)
X = pd.DataFrame(np.random.randint(0, 2, size=(n, N)), columns=['feat%s' % i for i in range(N)])
y = np.random.choice([0, 1], p=[1-p, p], size=n).astype(float)

#creates complex omega_hat
model = GMLVQ(solver_type='steepest-gradient-descent', activation_type='swish', activation_params={'beta': 2}, random_state=1)
model.fit(X, y)
print(model.omega_hat_)
print(model.omega_hat_.dtype)

#creates real omega_hat
model = GMLVQ(solver_type='steepest-gradient-descent', activation_type='swish', activation_params={'beta': 2}, random_state=2)
model.fit(X, y)
print(model.omega_hat_)
print(model.omega_hat_.dtype)

Note that the size of data has influence on the behavior and might result in twice real results in the above (e.g., N=3 does; n=100 does), but p does not seem to have influence.

Proposed solution: change  np.linalg.eig to np.linalg.eigh for the computation of the eigenvalues and omega_hat_ in the after_fit() function. Which will need to be done for both GMLVQ and LGMLVQ.
Reasoning: The relevance matrix (lambda_), which the eigenvalue decomposition is performed on, is a real and symmetric matrix. The eigenvalue decomposition of a real and symmetric matrix always results in real eigenvalues, see Matrix cookbook, section 5.2.4, equation 283.
The reason why np.linalg.eig produces non-real eigenvalues (and consequently eigenvectors?) has likely to do with the finite precision of the calculation. See stackoverflow1 and stackoverflow2.
The issue of np.linalg.eig is not shared with np.linalg.eigh as np.linalg.eigh is defined for real symmetric (and complex Hermitian) matrices for which the resulting eigenvalues are always real. Additionally, np.linalg.eigh uses the fact that the matrices are symmetric and therefore computes (much) faster.

Thanks for looking into this. Looks like a good solution to me.


coral-web: show loading text when tool events are still streaming in

Thank you for contributing to the Cohere Toolkit!
https://github.com/cohere-ai/cohere-toolkit/assets/140425731/b48210b3-c52a-4fc5-9c08-36d52ce5a09d

[ ] PR title: "area: description"

Where "area" is whichever of interface, frontend, model, tools, backend, etc. is being modified. Use "docs: ..."  for CI changes.
Example: "deployment: add Azure model option"

[ ] PR message: Delete this entire checklist and replace with

Description: a description of the change
Issue: the issue # it fixes, if applicable
Dependencies: any dependencies required for this change

[ ] Add tests and docs: Please include testing and documentation for your changes

[ ] Lint and test: Run make lint and make run-tests

AI Description

This pull request introduces the isStreamingToolEvents state and prop to various components and hooks in the coral_web interface. This new feature allows the rendering of a "Working on it" message with an ellipsis animation while tool events are being streamed.

The Props interface in MessagingContainer.tsx is updated with the isStreamingToolEvents property.
The Messages component in MessagingContainer.tsx now includes isStreamingToolEvents in its prop types and rendering logic.
The MessageRow component in MessageRow.tsx has been modified to include isStreamingToolEvents in its prop types and rendering logic.
The ToolEvents component in ToolEvents.tsx now includes isStreaming in its prop types and renders the "Working on it" message with an ellipsis animation when isStreaming is true.
The useChat hook in chat.ts has been updated to include the isStreamingToolEvents state and logic to update it based on different stream events.
The Conversation component in index.tsx now includes isStreamingToolEvents in its prop types.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 86.30%. Comparing base (9125660) to head (52c086e).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #356      +/-   ##
==========================================
- Coverage   86.46%   86.30%   -0.17%
==========================================
  Files         178      178
  Lines        6873     6908      +35
==========================================
+ Hits         5943     5962      +19
- Misses        930      946      +16

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Empty screen

Fresh Laravel app.
View:
<!DOCTYPE html>
getLocale()) }}">

        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">

    <x-maps-leaflet></x-maps-leaflet>

Gives the following html:
<!DOCTYPE html>

        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">

    <link rel="stylesheet"<EMAIL_ADDRESS>      integrity="sha512-xodZBNTC5n17Xt2atTPuE1HxjVMSvLVW9ocqUKLsCC5CXdbqCmblAshOMAS6/keqq/sMZMZ19scR4PsZChSR7A=="
      crossorigin=""/>

But renders as this:

Hi,
Thanks for creating an issue when you encountered a bug! ( You are actually the first real issue so congrats 🎉).
Let me look in to it and ill be right back.

I released a new version 0.2 that should be downloadable soon( packagist has to update first ). This seems to work for me with your code.  @Wulfheart  please let me know if it also works on your side :)

Awesome. Works on my end!


Adding TaPaCo Dataset with README.md

We want to keep the repo as light as possible so that it doesn't take ages to clone, that's why we ask for small dummy data files (especially when there are many of them). Let me know if you have questions or if we can help you on this

Hello @lhoestq , made the changes as you suggested and pushed, please review. By default, the dummy data was generated the way it was by the dummy data auto generate command. Thank you.


Naming of models in the new @binding API

@essenciary
The new API is really cool stuff. Here's my idea to make it more robust and performant (still not breaking 🤣 )
We introduce an abstract subtype of ReactiveModels. All model-specific definitions like js_watch would always be defined for the abstract type.
It could be along the following lines:

function Stipple.root(::M)::String where {M<:ReactiveModel}
  root(M)
end

# remove extra chars from the `gensym()` call and convert "." to ","
# if we define the real model name to be the abstract model name prefixed with `'#'`
# the `vm()` will be identical for real and abstract type
function Stipple.root(::Type{M})::String where {M<:ReactiveModel}
  replace(replace(string(M), "." => "_", r"var\"#*([^#]+).*\"" => s"\1"), r"[^0-9a-zA-Z_-]+" => "")
end

# ------------------------------------------------------------
abstract type HHModel <: ReactiveModel end

# all js definitions go with the abstract model type
js_watch(::HHModel) = "console.log('Hi')"

# the real model type is defined as subtype of the abstract type
@reactive! mutable struct var"#HHModel" <: HHModel
    a::R{Int} = 1
    b::R{Float64} = 2
end

model = init(var"#HHModel")

vm(HHModel)
# "HHModel"

vm(var"#HHModel")
# "HHModel"

vm(model)
# "HHModel"

js_watch(model)
# "console.log('Hi')"

@reactive! mutable struct var"#HHModel" <: HHModel
    a::R{Int} = 1
    b::R{Float64} = 2
end

# after redefinition we need to make sure deps and routes are replaced
# for this to work we need to modify `stipple_deps()` to register the route under Symbol(vm(m)) instead of Symbol(m)
delete!(Stipple.DEPS, filter(x -> x isa Type && x <: HHModel, keys(Stipple.DEPS)))
Genie.Router.delete!(Symbol(vm(HHModel)))

# So the `@binding` macro should do something like

@reactive! mutable struct var"#HHModel" <: HHModel
    a::R{Int} = 1
    b::R{Float64} = 2
    c::R{Float64} = 3
end

delete!(Stipple.DEPS, filter(x -> x isa Type && x <: HHModel, keys(Stipple.DEPS)))
Genie.Router.delete!(Symbol(vm(HHModel)))

# -----------------------  testing  --------------------------

Stipple.DEPS[:test] = () -> ["Just for testing that we don't delete too much..."]

route("/") do
    page(model, "Hello")
end

up()

Genie.Router.routes()
Stipple.DEPS

Any js functions would be preserved without redefinition and inference. That should be much more performant.
What do you think?

Well, if two models have the same vm name, they share the same route.
It's highly improbable though, if we keep the module names in the vm-string. It only happens, if there are some ugly characters in the type name.

I have addressed the "same model name" issue already by including the name of the module that contains the model in the model name. This combination has to be unique because it's enforced by the compiler.

I put all definitions in a macro. It's quite compact
using Stipple
import Stipple: js_watch

Stipple.root(::M) where M<:ReactiveModel = Stipple.root(M)
Stipple.root(::Type{M}) where M<:ReactiveModel = replace(replace(string(M), "." => "_", r"var\"#*([^#]+).*\"" => s"\1"), r"[^0-9a-zA-Z_]+" => "")

initmodel(m; kwargs...) = init(isabstracttype(m) ? eval(Symbol(m, '!')) : m; kwargs...)

macro newmodel(modelname, expr)
    modelconst = Symbol(modelname, '!')
    realname = gensym(modelname)

    esc(quote
        abstract type $modelname <: ReactiveModel end

        $modelconst = @reactive mutable struct $realname <: $modelname
            $(expr.args...)
        end

        delete!.(Ref(Stipple.DEPS), filter(x -> x isa Type && x <: $modelname, keys(Stipple.DEPS)))
        Genie.Router.delete!(Symbol(vm($modelname)))

        $modelconst
    end)
end

#  -----------  testing  -------------
@newmodel HHModel begin
    a::Int = 1
    b::Float64 = 2
end

hhmodel = initmodel(HHModel)

vm(hhmodel)
# HHModel

vm(HHModel)
# HHModel

js_watch(::M) where M <: HHModel = "HH"
js_watch(hhmodel)

# HH

For the REACTIVE_STORAGE, I'd follow the same concept but rather store expressions for the model types and not for the modules. That way it is easy to have more than one model in one module.
using OrderedCollections
REACTIVE_STORAGE = LittleDict{Type{<:ReactiveModel}, LittleDict{Symbol, Expr}}()

function Base.push!(rs::LittleDict, m::Union{ReactiveModel, Type{<:ReactiveModel}}, expr::Expr)
    mkey = modeltype(m)
    f = expr
    while ! isa(f, Symbol)
        f = f.args[1]
    end
    haskey(rs, mkey) ? push!(rs[mkey], f => expr) : push!(rs, mkey => LittleDict(f => expr))
end

expr = :(a::R{Integer} = 1 + 3)
push!(REACTIVE_STORAGE, HHModel, :(a::R{Integer} = 1 + 3))
push!(REACTIVE_STORAGE, HHModel, :(b::R{Integer} = 2))
push!(REACTIVE_STORAGE, HHModel, :(c::R{Float64} = 3))

REACTIVE_STORAGE[HHModel]

macro model_from_storage(modelname)
    output = Core.eval(__module__, :(
        values(REACTIVE_STORAGE[$modelname])
    ))

    modelstring = string(modelname, "_")
    modelconst = Symbol(modelstring)
    realname = gensym(modelname)

    esc(quote

        abstract type $modelname <: ReactiveModel end

        $modelconst = @reactive mutable struct $realname <: $modelname
            $(output...)
        end

        for p in Stipple.DEPS
            p[1] isa Type && p[1] <: HHModel && delete!(Stipple.DEPS, p[1])
        end

        # oldname != "" && Genie.Router.delete!(oldname)
        for r in Genie.Router.routes()
            endswith(string(r.name), r"var\"##HHModel#\d+\"") && Genie.Router.delete!(r.name)
        end

        $modelconst
    end)
end

@model_from_storage HHModel

It shouldn't be difficult to define a default model for the module.
That way we would have the best of all worlds:

hidden models for users who don't want to deal with models
overwritable models
heritage of js_methods without redefinition
automatic cleaning of routes and DEPS

js_methods could be also defined by a macro
macro js_methods(modelname, methods)
    quote
        Stipple.js_methods(::M) where M <: $modelname = $methods
    end
end

@js_methods HHModel """
    increaseme: function() {this.a += 1}
    """

js_methods(model)
# "increaseme: function() {this.a += 1}\n"

I think we need to keep the storage by module (so it could be by module and type). Because the macros are executed in the context of the module, we can easily check and retrieve what is defined in the context of that module. Also, this guarantees that there are no type name conflicts between different types defined in different modules.
However, multiple models per module looks like premature optimisation at this point. I would prefer to roll out the implicit model API ASAP, with one model per module, and get feedback from users. If multiple implicit models will be a common requirement (though I haven't seen this use case with explicit models so far) we can add it later. Otherwise less code and complexity is better -- there's less stuff to debug, document, and maintain.
Re subtypes and js_methods that sounds great. I'll leave it up to you as I don't have a close knowledge of that codebase nor do I use it on a regular basis. Just please don't break the implicit models API that I'm rolling out (so again I'd prefer to leave REACTIVE_STORAGE for now as that's breaking and I want to release the API ASAP).

@essenciary I want to revive this topic as it is really annoying that all modeltype-based definitions get lost after redefining the model. I'v put some work into it and it just works. It could be easily implemented in your ReactiveTools as well.
Below is the code that I ended up with.
Stipple.root(::M) where M<:ReactiveModel = Stipple.root(M)
Stipple.root(::Type{M}) where M<:ReactiveModel = string(rootname(M))

rootname(x) = Symbol(replace(replace(string(x), "." => "_", r"var\"#*([^#]+).*\"" => s"\1"), r"[^0-9a-zA-Z_]+" => ""))

function Stipple.stipple_deps(m::Type{M}, vue_app_name, debounce, core_theme, endpoint, transport)::Function where {M<:ReactiveModel}
    @info "here"
    () -> begin
      if ! Genie.Assets.external_assets(Stipple.assets_config)
        if ! Genie.Router.isroute(rootname(M))
          Genie.Router.route(Genie.Assets.asset_route(Stipple.assets_config, :js, file = endpoint), named = rootname(M)) do
            Stipple.Elements.vue_integration(m; vue_app_name, debounce, core_theme, transport) |> Genie.Renderer.Js.js
          end
        end
      end

      [
        if ! Genie.Assets.external_assets(Stipple.assets_config)
          Genie.Renderer.Html.script(src = Genie.Assets.asset_path(Stipple.assets_config, :js, file = vue_app_name), defer = true)
        else
          Genie.Renderer.Html.script([
            (Stipple.Elements.vue_integration(m; vue_app_name, core_theme, debounce) |> Genie.Renderer.Js.js).body |> String
          ])
        end
      ]
    end
  end

function initmodel(m::Type{<:ReactiveModel}; kwargs...)
    model = isabstracttype(m) ? Core.eval(Base.parentmodule(m), Symbol(Base.nameof(m), "!")) : m
    Base.parentmodule(m), Symbol(Base.nameof(m), "!")
    Base.invokelatest(init, model; kwargs...)
end

macro model(modelname, expr)
    modelconst = Symbol(modelname, '!')

    esc(quote
        abstract type $modelname <: ReactiveModel end

        @reactive! mutable struct $modelconst <: $modelname
            $(expr.args...)
        end

        delete!.(Ref(Stipple.DEPS), filter(x -> x isa Type && x <: $modelname, keys(Stipple.DEPS)))
        Genie.Router.delete!(rootname($modelname))

        $modelconst
    end)
end

@model HHModel begin
    a::Int = 1
    b::Float64 = 2
end

Should I prepare a PR?

Note, I defined rootname() to strip the number from the gensym name in order to have always the same name of the js file.
And I just added a correction above that fixes backwards compatibility.

Just edited the code above...

@reactive! mutual struct HHModel <: ReactiveModel
    a::R{Int} = 1
    b::R{Float64} = 2
end

is replaced with
@model HHModel begin
    a::R{Int} = 1
    b::R{Float64} = 2
end

Thought a  bit about naming. @model creates a model instance, therefore I renamed my macro to @modeltype

Reading through ReactivityTools I think a better naming would be @type instead of @modeltype

@essenciary
I have pushed a working version now and it's ready for testing.
After refactoring the ReactiveTools I still propose the following changes:

@reactive! works as is.

@type begin ... end will be the recommended replacement as discussed above.

@type accepts a new syntax which is closer to the one you used in ReactiveTools, e.g.
 @type begin
     a = "Hello"
     b::AbstractString = "World"
     n = 10
     m::Real = 11, READONLY
     readonly = "hi", READONLY, NON_REACTIVE
     private = "you don't see me", PRIVATE, NON_REACTIVE
 end

BTW, we currently don't support listeners for non_reactive fields, but they could be triggered manually. Should we change that?

Cool! so the @in @out etc still work as previous? This is quite important for my use case as I want users to be able to tag variables where they appear in their code, vs explicitly declaring types. With the ReactiveTools API the whole idea of a type/model is hidden.
Thinking about this, there is really no trace of types in the new API, so a @type macro would leave people wondering. Maybe @vars or something makes more sense?
Re the @type example that's practically the @reactive declaration, minus a few words. The benefit or using it is that users don't chose the name of the type and the name of the model does not change when the fields change?

BTW, we currently don't support listeners for non_reactive fields, but they could be triggered manually. Should we change that?

I don't understand the implications of this. If the users don't want to make the fields reactive, why have listeners?

The only possible application I see is parameters that are typically changed by the front end and not by the backend.
But I think we should be clear by design. We allow that the value is changed from the front end, so if someone executes the correct js code, the value will be changed. If we don't want that we should exclude that.
If we don't install listeners but want to keep the possiibility of updates, we should provide a js function that does trigger such change, e.g. push(model, fieldname, value) in analogy to push!(model, fieldname => value). I do use the latter one to change client_data.
I just tested installing listeners, that's easy and works. So now we have the choice ...

I thought about it, I tend to have no listeners for non_reactives. But I would offer a js-push method.
It makes the setup more understandable:

Reactive: with listeners
Non-Reactive: without listeners
PUBLIC: pushing from both sides allowed
READONLY: no pushing from fron-end
PRIVATE: not shared to frontend

We could abandon NO_WATCHER in that case. I don't see a use case where we have no backend watcher. Switching off frontend watcher is done by READONLY.

We discussed this quite a while ago, but we said we leave it as is. But during all that time I never had a use case

Cool! so the @in @out etc still work as previous?

Yip! Even better, it's also possible to just import Stipple and ReactiveTools and it will work as well.

Maybe @vars or something makes more sense?
Sounds good for the ReactiveTools. I'd still favour @type for people who want to declare their own models.

Re the @type example that's practically the @reactive declaration, minus a few words.

Exactly.

The benefit or using it is that users don't chose the name of the type and the name of the model does not change when the fields change?

The benefits are

the model doesn't change its name
all type-specific function declarations will keep their validity without redefining
old syntax is supported
new syntax brings

no type declarations needed if the type is well-defined by the initial value
reactive type declaration without surrounding R{}. This is nice for people that use import rather than using because the parse_expression will check this and prefix with Stipple.otherwise.
non_reactive types are declared in a very similar way and types are also automatically attached

But this still needs to be written ...
BTW, I added a js push function to do model.push(field) which works for both, reactive and non-reactive types as long as they are PUBLIC

Amazing features! Let me know how I can help to release them.

Amazing features! Let me know how I can help to release them.

Test it 😄
There is @vars and @add_vars both supporting old and new syntax.  Old Syntax is chosen as soon as an explicit Reactive Type is detected, otherwise new Syntax is used.
@vars begin
    s1 = "Hello"
    s2 = "World", NON_REACTIVE, PRIVATE
    n::Int = 12, JSFUNCTION
end

@add_vars begin
    s1 = "Hi"
    s2 = "there", NON_REACTIVE, READONLY
    m::Int = 13, PRIVATE
end

Later definitions overwrite (as in merge())

Solved by #151


Release 0.1.0

Initial release

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@a012a13). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master      #1   +/-   ##
========================================
  Coverage          ?   0.00%
========================================
  Files             ?      22
  Lines             ?     218
  Branches          ?       0
========================================
  Hits              ?       0
  Misses            ?     218
  Partials          ?       0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a012a13...76a0db8. Read the comment docs.


Add support for notes?

There's a notes API, though I don't use notes so I'm not super-motivated here.
I'm not sure if notes also show up as posts. The docs imply maybe they do? If so some de-duplication may be needed.

Thank you for this tool, I've just exported my bookmarks and it worked like a charm!
The notes don't show up as posts. I have a grand total of six notes, never really got into that habit.


Dev igenomesbase

Added simple documentation for how to set path to local install of igenomes using igenomes_base.
 #637

Thanks for the comments @ggabernet. I have committed your edits :)

Ahhh linting, my archnemesis...

The Check Markdown links / markdown-link-check (pull_request) is still spinning away. I'm unfamiliar with the step it's stuck on and was going to wait until the check has finished before merging. Does anyone know how long this step normally takes?


Change: Adds last used to the ssh keys table

Adds the last used timeframe to the SSH Keys table, and updates both created & last used to follow a similar format to Github.

Closes https://github.com/uselagoon/lagoon-ui/issues/298

I wonder if its worth having a tooltip that can show the actual date too? Then created could be the same? Thoughts?

I wonder if its worth having a tooltip that can show the actual date too? Then created could be the same? Thoughts?

Is something like this what you were thinking for the combined created/last used?

Yeah! Probably don't need the divider between them though?

Would it be worth left aligning these?

Looks good now


Template Tests Should Have Compiled Code Seperated

Right now they don't.
I think the advice nowadays is for everything to for best compatibility with source control.

Is JKI VI Tester supported on RT targets? Last I remember compiled code separated can't be used on VIs that are being used with LabVIEW RT.


sqls has been deprecated in the neovim/nvim-lspconfig

do you mind if I try to reactivating sqls in the neovim/nvim-lspconfig?
https://github.com/neovim/nvim-lspconfig/issues/2530

already undeprecated. 🙇♀🙇♀🙇♀🙇♀
https://github.com/neovim/nvim-lspconfig/pull/2950


Incorrect trigger ready

It seems on method updateCounter we need change from:
this.countersLeft--;
if (this.countersLeft === 0) {
    this.appear();
    this.ready();
}

to:
if (this.countersLeft === 0) {
    this.appear();
    this.ready();
}
this.countersLeft--;

What issue are you trying to solve?

I try to get all social likes number
    $('.social-likes').on('ready.social-likes', function(event, number) {
       console.log('ready')
        if (number != parseInt($(this).attr('data-likes-count'))) {
            id = $('.social-likes').attr('data-id');
            $.get('/ajax/likes/?id=' + id + '&count=' + number);
        }
        $('.social-likes__counter_empty').each(function(index, value) {
            $(value).prev().css('background-image', 'none');
        });
    });

Screen without fix
![]
(http://pic.oldmin.org/do_not_del/1.jpg)
Result from console
![]
(http://pic.oldmin.org/do_not_del/2.jpg)
Screen with fix
![]
(http://pic.oldmin.org/do_not_del/3.jpg)
Result from console
![]
(http://pic.oldmin.org/do_not_del/4.jpg)

Could you please make a pull request?

Done


Make this crate no_std

As nothing actually depends on std, there is no need to keep a dependency on it.  This allows users such as the jid crate to actually be no_std as well.

Ah oops, that works in nightly but doesn’t yet in stable, due to core::error::Error not being exported yet… I could add a std feature that would be default and impl this trait, just like in jid, or just keep that PR open until that export gets stabilised, which option do you prefer?

I think a std feature seems reasonable. If it becomes a no-op in the future once Error-in-core stabilizes that's fine.

There, I have added an optional but enabled by default std feature.

Can you also update the CI logic to also test with the std feature disabled?
LGTM otherwise

I think it'd be safer to have an enabled-by-default std feature to avoid being blocked on adding future APIs that do require std. That'd also allow us today to continue to support older Rust versions by importing Error from std when enabled.


Getting items by ID is unnecessary

Category

[x] Content fix

Related issues:

fixes https://github.com/SharePoint/sp-dev-docs/issues/4531

What's in this Pull Request?
Instead of using code:
employeeList.getItemById(itemArray[i].get_id()).deleteObject();
we can use only this:
itemArray[i].deleteObject();
because we've already got these items by the previous clientContext.executeQueryAsync(deleteCompletedItems, onGetCompletedItemsFail), and that's enough.
Guidance
Updated code sample as Getting items by ID is unnecessary.

thx @nanddeepn, appreciated.


WIP: Add classic mode constructor

I needed a way to create files in a classic mode to replace this part of the opendap hyrax functionality.
There's probably a better way to expose this API, I am open to any suggestions

add_variable_with_type(...) seems to not work on a "classic model" file. Will investigate further

Classic netCDF is a subset of hdf5 which emulates the older CDF standards. This means groups and custom types are not supported. netCDF is correct in emitting this error.
I think it would be a good idea to expose all of the file options, to avoid having a dozen methods to open a file. I'll create a PR for this soon.

Thanks for the explanation! I'll close this one then


Align TestMode import across test classes

Small cleanup to align with other test classes in this package.
🧹 Whilst an ad-hoc cleanup when working on something else, I'm not sure it can be generalised (and likely one you have already considered so I'm interested to know your opinions!).
What factors come into play when deciding between an import and a static import for accessing static members of a class? Ones I consider that cannot be generalised (trivially):

Trade-off between the class qualification providing extra descriptive value, and adding to verbosity. Requires understanding the descriptive value of a the qualifying class name in the context it is (statically or not) imported.
Frequency as a multiplier. Within one class I assume a check would be trivial, across many classes of the same structure not trivial with EP. In this case, it is common to all(?) #replacement tests and therefore you could argue we could drop the qualifier. This leads onto another question: should we not provide some form of test abstraction (harness) for BugChecker tests? Might also be nice to look at a cleaner test data setup, or perhaps there is a reason to providing varargs String's in CompilationTestHelper#addSourceLines over reading from a .java source file?

Suggested commit message (not much to say about this change so probably too verbose/redundant):
Qualify non-static `TestMode` imports across `BugChecker` test classes (#630)

Prefer non-static imports and qualification of
`BugCheckerRefactoringTestHelper#TestMode` members across bug checker test
classes.

Heyy @philleonard , thanks for opening the PR! :rocket:
First some context related to this specific issue and the question about generalization:

We have a ticket that is related to this specific issue: https://github.com/PicnicSupermarket/error-prone-support/issues/389.
@cernat-catalin opened a PR for a related check that is the exact opposite of the StaticImport check.

Nice thoughts, and yes you are right, it is not easy to generalize this and auto-detect candidates for static importing. For that reason we have a curated set of candidates for statically importing things.
We have the rule that we should not "lose" any context if we statically import something. This usually holds for imports that have
words (or parts) that are (partially) duplicate with the method. Additionally it should not be ambiguous. For example, if there are multiple places from which you can import the constant UTC, it is not directly clear from where you imported something, so we don't want to do it.
I have to go now so will get back to your question about the test abstraction later 😄.

where we'll move to text blocks. That in turn unlocks IntelliJ IDEA language injection support :)

Nice! Having played around with #198 this is already a much nicer experience for writing these test classes (which I have been doing in a scratch file for now). Copilot suggestions and IntelliJ autocomplete also "work" within language injections  so that is nice. After a quick scan, GitHub and the IntelliLang both have somewhat "incomplete" syntax highlighting, but already much better than what we currently have.


feat: use tolerations from values in ebs-csi-node daemonset

Is this a bug fix or adding new feature?
This is a new feature.
What is this PR about? / Why do we need it?
For the driver to work on tainted nodes, the ebs-csi-node daemonset needs to put pods on these nodes. The tolerations value is already present in the chart values.yaml, this pull request actually use it in the template file manifest.yaml.
What testing is done?
This has been tested in a live development k8s cluster.

Please use OSC-MIGRATION branch and i think this issue is out-of-date


feat: Keap - read with custom fields

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1151
#1150  👈 (View in Graphite)
#1125
#1124 : 1 other dependent PR (#1126 )
#1134 : 2 other dependent PRs (#1097 , #1135 )
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


[Task]: Add Daily Delegation Statistic

Task description
Add Daily Delegation Statistic
For now we only have 1 data point but we will have more

Total

Partialy done and needs just query adjustment


Not pushing the changes to bitbucket

Hi I tried to add, commit and push to my bitbucket repo. It should ask for my password but nothing happened it was showing pushing changes.... and hiding the popup.
And I found its not commited. To commit it I needed a pull request again and again a push manually in terminal.

I believe you need to use ssh to push. I ran into this problem too using https for my remote as https requires a password. Even if you store the password in the keychain it won't work. Switch your remote to ssh and it should work great. Keep in mind that you will need to create an ssh key for that to work: https://help.github.com/articles/generating-ssh-keys/

switching the remote from http to ssh did it or me - it was crated as https...needs to be SSH for git-plus
command looks like this:
git remote set-url origin git@github.com:path/to/my/repo.git

refer to #336


Svelte child components

Code changes
This PR has been organized into logical commit history so that it is more easily digestable. Also, I'll describe below what changes happened so that there is more context.

svelteWebConfig is where the magic happens. We compile svelte components first, then the web components denoted with .wc.svelte and build out the compiled css - due to limitations with svelte's web component support, we need to manually attach the compiled css to the shadow DOM of the parent web component.

named the build css to nylas-component.css so that devs will know what this resource is for when inspecting the Network Tab

Apply this config to Composer only. Fortunately, each component has their own rollup.config so we can add this new support easily.
Update Composer child elements to svelte components instead of web components. This is applied for obvious reasons to:

AlertBar.svelte
Attachment.svelte
ContactsSearch.svelte
DatepickerModal.svelte
HTMLEditor.svelte

There are shared child components between the parent Web components such as NError.svelte that we can no longer share in order to support child svelte components in Composer and not everywhere else. So we need to remove this shared import and add it in each component.
Because NError.svelte was made to be a web component we need to duplicate it as DefaultError.svelte to be a svelte child component to support both cases.
The rest of the changes are code cleanup.
The resulting build looks like this:

Screenshots
Composer locally with yarn start

Composer in react land

License
I confirm that this contribution is made under the terms of the MIT license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

@philrenaud I did get some build failures from vercel when I initially had the svelteWebConfig logic in svelte.config.js. My guess is that it's using svelte.config.js. I'll take a look.

@philrenaud issue should be fixed


Garbage collection for Archetypes

Hello! I stumbled upon this library looking for a good array-of-structs ECS system and I've been working with the for the past month or so in my game. I'm building a renderer using React, but I am not using miniplex-react (frankly, it doesn't seem to provide anything I would use that isn't very easy to implement myself).
I want to be able to render a list of entities matching a criteria, and I would like to be able to use an Archetype for this, however this causes a memory leak. I have a Location component which has a section field, and my React component should render all entities which have a given section. That's easy:
export function useEntities<T>(bucket: ArchetypeBucket<T>): T[] {
  const [_, forceUpdate] = React.useState({});

  React.useEffect(() => {
    const rerender = () => forceUpdate({});
    bucket.onEntityAdded.add(rerender);
    bucket.onEntityRemoved.add(rerender);

    return () => {
      bucket.onEntityAdded.remove(rerender);
      bucket.onEntityRemoved.remove(rerender);
    };
  }, [bucket]);

  return [...bucket];
}

function MyList({ section }) {
  const entities = useEntities(React.useMemo(
    () => world.with("section").where(e => e.location.section === section),
    [section]
  ));
  return (
...
);
}

This has a memory leak, since it creates a new Archetype every time the section prop changes, but Archetypes are never removed. The feature request is simple: once the last listener for a Bucket is removed, the Bucket should remove its own listeners. If future listeners for the bucket are added, the Bucket should reattach its own listeners and recalculate its entities.
I know I can create a global cache of every section and look up in that cache when rendering my component... but a cache without expiration is still just a memory leak.
Obviously this isn't critical, but since you're changing the API for 2.0 I figured now was an appropriate time to raise this. Thanks for continuing work on this library!

Hi, and thanks for submitting this!
First of all, which version of Miniplex are you using exactly? Your code looks like you're using one of the 2.0 betas — which one?
To be entirely fair, I'm not sure if there really is a memory leak, or if there is one, that it's because of the archetype creation. A call to world.with("section") will be idempotent, only creating the actual archetype on its first call; on subsequent call, the memoized archetype will be returned. archetype.where(predicate) won't even create any new archetype instances, but just a temporary iterator that allows you to iterate through archetype, with the predicate function applied.
If for some reason multiple instances of the same archetype are being created, that would be a bug. May I ask what steps you took to identify this memory leak? Thanks!

Addendum: I'm honestly surprised that the code example you've posted here works at all, considering you're creating an iterator using where and passing that to your version of useEntities(), which will try to call .onEntityAdded et al on it, which should raise an error because those events don't exist on the iterator object where returns.
I'm assuming you're using 2.0b3 — please let me know if this is wrong.
The implementation of where is here, for reference.
Heads-up: because the nature of how where works has proven a bit iffy to explain to people, it'll be removed in the next beta. If things go as planned, the next beta will also come with a significant reduction in API surface and implementation complexity (but it'll be a breaking change, so be ready.) See #275 for details.

Sorry, I should have mentioned. I am using ^2.0.0-beta.3, which is apparently<EMAIL_ADDRESS>Also, the code snippet in my original is what I'd like to do, rather than what I am presently doing (which is using a normal archetype and filtering manually). 🫢 Given that you're saying it wouldn't work, well, it wouldn't be a leak either, I guess!
What I really want is to have a component that only rerenders when the entities selected by the predicate change. My actual code presently caches the world.with("section") globally, which is advised in the current README, and then as I mentioned I filter out other entities in my render method. This causes unnecessarily component renders of the list which I was hoping to avoid.
If you're moving towards the API you describe in #275, it would be awesome to have something like this (again, this is just pseudocode):
function MyList({ section }) {
  const entities = useQuery(world, React.useCallback(
    q => q.with("section").where(e => e.location.section === section),
    [section]
  ));
  return (
...
);
}

If it's possible to create such a useQuery hook, it would greatly reduce my number of rerenders.


Fix Sheobill logo in README.md

Bring https://raw.githubusercontent.com/pruten/pruten.github.io/master/web/stork_tiny_head3.jpg into repo
Link to image in README.md

Fixed


Add the ability to redirect based on a radio button chosen.

Based on #227 with an added example from @joelanman's #338 (also covers #362 too, I think).
The main difference with this PR from previous ones is that now we have data being stored in the session it's possible to address the comments from #338 and include the option to save data at the same time as redirecting.
The example page now contains two examples. The first shows the method of redirecting via radio button values and the second shows a more complex example involving aliens.

Thanks for continuing to work on this @tsmorgan.
I feel wary of using value for this. It feels somewhat unorthodox and might be hard to explain to people. What do others think?
Spitballing - what about putting a data-redirect="url" on each input, and using JS to append a hidden field that gets submitted to the server with the redirect url? That way values stay 'normal', and we're reusing data-thing to do interactive stuff - which people will be familiar with if they're doing progressive disclosure.

Not sure I agree. The value is where you'd place the data that you'd do more complex branching on so it makes sense to me that you use the same thing to trigger this redirect.

I think any approach is going to be non standard HTML, but I agree with Tom's reasoning. If we went with data-redirect we'd have two attributes to teach people. I think I favour this:
<input type="radio" name="work-status" value="self-employed">
Then talk about how to do branching in routes. Then introduce that we have a shortcut for radios:
<input type="radio" name="work-status" value="self-employed, redirect(/self-employed)">
I think it's good practice to say there's always a value, and you can optionally add our redirect function.
As I'm writing this, is it odd that the redirect isn't quoted? It doesn't need to be I don't think.

Yeah, I don't think it should be quoted. It's within with quotes of the value attribute anyway and it's within brackets. Having more quotes could confuse things and we might get into the realms of having to tell people to pick single or double quotes depending on what they'd used in their HTML.
Just realise I've got the data being passed after the redirect (eg. value="redirect(<url>), <data>") which wasn't what you guys arrived at in your discussion on #338. Also thinking about "it's good practice to say there's always a value", it does make more sense to think of this as an add-on to the normal passing of data. So if you'll bare with me I'll update the regex to expect <data> first and redirect(<url) after a comma (eg. value="<data>, redirect(<url>)").

OK that's done now. Would appreciate you guys checking it through a bit as the regex is new. It's definitely better this way around though, the explanation on the example page feels much simpler now.

Thanks @tsmorgan, so sorry for the long delay on this. We know it's a need teams have, so we've raised an issue for it: https://github.com/alphagov/govuk-prototype-kit/issues/576
Unfortunately this particular approach is now out of date (our fault), so I'm closing this pull request.


several issues

I want to share several issues I found. Maybe it can be useful to someone.
When I was looking for Web Sockets library in ObjC, this was referenced as the best one. So I used it in my application. But then I got a few crash reports. I didn't have any since release in App Store. I haven't reviewed this code good enough before adding it. When I did, I found several issues in it. I found that it is "mostly working", but for production code, it is not good enough. Generally speaking, code just needs refactoring. Still, I want to thank the authors, it was a good start point for me anyway.

crash in [JFRWebSocket dealloc]: #70

[JFRWebSocket writeError:] may be called many times, and it always sends closing-handshake packet, even if it was already sent. This creates redundant traffic and violates the protocol.

[JFRWebSocket disconnect] does not ensure socket closure. It relies on server to close it first, and usually it does so. But there should be a timeout, after which socket is forced to close.

Calling [outputStream write: maxLength:] in [JFRWebSocket dequeueWrite: withCode:] is not safe. Must be called from run loop queue, in which outputStream was created.

Should wait for NSStreamEventHasSpaceAvailable event instead of trying to call [outputStream write: maxLength:] forever in loop in [JFRWebSocket dequeueWrite: withCode:].

*((uint16_t *)(buffer + offset)) and *((uint64_t *)(buffer + offset)) may crash because of misalignment.

The [JFRWebSocket processRawMessage] method is the most confusing. It seems unmaintainable. I'm almost sure it is buggy. I found it easier to rewrite it completely than try to correct it.

There are also few race condition issues, for example when using isCreated and didDisconnect properties.

Note that I don't use JFRSecurity class in my project (I removed it), so I don't tell anything about its issues if there are any.

Maybe try SocketRocket?


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_bctlxynqbl/clone/recipe/meta.yaml:14: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_bctlxynqbl/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
WARNING: /tmp/abs_bctlxynqbl/clone/recipe/meta.yaml:46: missing_description: The recipe is missing a description
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_a454cmm9px/clone/recipe/meta.yaml:14: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_a454cmm9px/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
WARNING: /tmp/abs_a454cmm9px/clone/recipe/meta.yaml:46: missing_description: The recipe is missing a description
Errors were found


[CP][beta channel] Fix for Function.== in DDC

commit(s) to merge: https://github.com/dart-lang/sdk/commit/8e2539d24c93ebb238c8681250ab67c4bf19c4e9
merge instructions: clean merge
What is the issue: DDC has two dispatch mechanisms: a direct member call or an indirect call using a symbol in the prototype of the target. We use the former for most methods implemented in classes when we know the static type of the call, and the latter for various reasons (dynamic calls and calls to potentially native APIs like html elements from the browser).
DDC is generating the wrong one for Function.== under null safety. This causes issues whenever users compare f1 == f2 when the type of f1 is a non-nulllable Function.
What is the fix: Force DDC to use the proper dispatch mechanism on Function.
Why cherrypick: User visible regression, noticed by external users and in the flutter build.
Risk: low, fix is very contained and small, the change uses the conservative dispatch mechanism that is always supported.
Link to original  issue(s): https://github.com/dart-lang/sdk/issues/45601 https://github.com/dart-lang/sdk/issues/45762
/cc @kevmoo @mit-mit @whesse @athomas @vsmenon @franklinyow @nshahan

Has this rolled internally yet - that'd be a good sanity check.  Otherwise, lgtm.

It did, all green there too.

lgtm

Approved in SCRUM

Merged to beta in 5cae07b3980f4684d3d477457c307a67235adf99 (2.13.0-211.13.beta).


Unsuscribe link in Email broken for dockerized cachet

I am deploying several instances of cachet (2.3.14) in our docker swarm environment. I have written a python service that queries kibana and opens incidents when needed in cachet.
Everything is working as expected.
I have then added configuration to the swarmfile to send emails through mailgun; also this part seems to work flawlessly. I did get the confirmation email, and after confirmation I started receiving emails for each incident event.
Now, revising all components before releasing the solution to prod, I have noticed that the "unsubscribe" and "manage subscription" links are broken.
Instead of the full domain name of the cachet instance, the URL is built using the container name + port:
http://preprod_cachet_cachet_customer:8000/subscribe/manage/a8C7frCxRR5cwQcNwGaODh3NyifmNwiSENg29m8yxy
The only place I am using that notation is in my python service. The service does not have access to a public DNS, so I am pointing it to cachet using the stack + service_name (docker embedded dns service for internal networks). This is also somehow "safer" and faster, because the request is local to the LAN.
My guess is that the unsuscribe email link is generated using the header of the request that generated the notification in the first place. This is not a sane behavior, because it forces api calls to be sent through the internet.
This is a blocker for our usage of cachet in production. Can this behavior be altered though some configuration parameter?

Did you set the app url on setup?


Fix appTP onboarding release test

Task/Issue URL: https://app.asana.com/0/414730916066338/1205936253216099/f
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

Current dependencies on/for this PR:

develop

PR #3852   👈

This stack of pull requests is managed by Graphite.


setuptools ==69.3.0 builds my-package CLI as my_package (69.2.0 would build my-package CLI as my-package

setuptools version
setuptools==69.3.0
Python version
Python 3.11
OS
Linux, Mac
Additional environment information
No response
Description
We have a CLI that is built with setuptools. We didn't have our version pinned. Right after setuptools==69.3.0 was released to PyPi, our automated jobs started failing that use this package.
Let's call the package my-package.
We install the CLI with:
setup-env:
	pip3 install pipenv
	pipenv install
build: setup-env clean
	python3 -m pip install setuptools==69.2.0 wheel==0.43.0
	python3 -m setup -q sdist bdist_wheel

I noticed that setuptools came out with a release less than an hour ago (at the time of this writing) so I tried pinning the setuptools version to the prior version (69.2.0). It worked.
Changing make build to this fixed the error:
build: setup-env clean
	python3 -m pip install setuptools==69.2.0 wheel==0.43.0
	python3 -m setup -q sdist bdist_wheel

The setup.py script is like this:

"""Setup script"""
import setuptools
import os

setuptools.setup(
    name="my-package",
    include_package_data=True,
    version="420.69",
    author="Kinnaird McQuade",
<EMAIL_ADDRESS>    description="My package",
    url="https://github.com/my-org/my-package",
    packages=setuptools.find_packages(exclude=["test*"]),
    install_requires=[
        "requests"
    ],
    classifiers=[
        "Programming Language :: Python :: 3",
        "Operating System :: OS Independent",
    ],
    entry_points={"console_scripts": "my-package=my_package.bin.cli:main"},
    zip_safe=True,
    python_requires=">=3.11",
)

Before the downgrade, we were getting messages like this in GitHub Actions when we would try to install the package
python3 -m pip install -q ./dist/my-package*.tar.gz
WARNING: Requirement './dist/my-package*.tar.gz' looks like a filename, but the file does not exist
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/home/runner/work/e2e-tests/e2e-tests/my-package/dist/my-package*.tar.gz'

And I saw that if you checked out the contents of the dist/ folder, the file that was there was titled my_package-420.69.tar.gz, not my-package-420.69.tar.gz as it was before 69.3.0.
Hope this helps. Our current workaround of pinning the version works, but wanted to flag as soon as possible. Thanks.
Expected behavior
See the description
How to Reproduce
See the description
Output

See the description

I think it was due to this?
https://github.com/pypa/setuptools/pull/4286/files#diff-b9c5224191f52b3ea80acfdc52fce5ea9e840a6a9237d479918bb7e974642f0bR266
It's unfortunate that this affects any CLI tool that wants to be my-package versus my_package and builds with setup.py.
from setuptools import setup

setup(
    # ...,
    entry_points={
        'console_scripts': [
            'hello-world = timmins:hello_world',
        ]
    }
)

Yup, this broke our CI as well. I would argue this is probably not a patch release.

Please note that the specific name of the built file is subject to change according to the changes Packaging PEPs (that is what happened in this case). Depending on the exact name, without considering that it might be cannonicalised to a different but corresponding form, will likely introduce points of failure in your CI.
So this is not a bug, but an intentional change that was required to align to the latest standardisation efforts.
@kmcquade, @mikealfare, vould you please try to use patterns/regexes in your CI scripts to make them robust to name normalisation?

Yeah, we're making those updates currently.

It's unfortunate that this affects any CLI tool that wants to be my-package versus my_package and builds with setup.py.

I only expect the name of the package as reflected by the package metadata to be affected. It should not affect the package's true name nor its import name nor the names of any console scripts. It's unfortunate that applications were depending on this naming convention, but since it was merely an implementation detail and had no backing tests or specifications, the transition to something that is specified and standardized means you can now start to rely on the naming.

I only expect the name of the package as reflected by the package metadata to be affected.

The issue was that the names of the artifacts also changed. So if you want to do anything with them locally (e.g. verify them, log the generated files that fit a regex, etc.) prior to publishing them, then you rely on the names to some extent.

the transition to something that is specified and standardized means you can now start to rely on the naming.

I agree that we should move towards standardization. However, here are the changes we wound up making as a result of this change:

broaden our regex (effectively don't rely on the name)
hard pin setuptools to the minor in our CI to avoid being broken in the future, and rely on things like dependabot changes to push patch updates

Is this update is only intended to canonicalize sdist files? Or is it also supposed to apply to wheel files?
The binary-distribution-format specification seems to suggest a similar naming standardization (i.e., no periods or hyphens). However, for our packages, while sdist files are now being canonically-named, bdist_wheel is still producing the original file names. E.g.:
/Users/runner/work/1/s/dist/rcsb_utils_config-0.39.tar.gz
/Users/runner/work/1/s/dist/rcsb.utils.config-0.39-py2.py3-none-any.whl

The binary-distribution-format specification seems to suggest a similar naming standardization (i.e., no periods or hyphens). However, for our packages, while sdist files are now being canonically-named, bdist_wheel is still producing the original file names. E.g.:

Likely to be related to the update in the standard:

February 2021: The rules on escaping in wheel filenames were revised, to bring them into line with what popular tools actually do.

This update is not implemented yet, only PEP 625 is in the process of being adopted by setuptools.
In the future it is likely to change.


ACTION REQUIRED: Changes to pulling Chainguard Images

Hey there Chainguard here.
We noticed that you are using Chainguard Images, thank you! We wanted to make you aware of an upcoming change that will impact your project.
Starting August 16, 2023 public users will no longer be able to pull images from our registry (cgr.dev/chainguard) by tags other than latest or latest-dev. Please see the announcement for more information.
You are currently using the following.
In https://github.com/open-sauced/pizza/blob/21e704dca76c47d31121cbaffe6c31c2946ff94f/Dockerfile:

cgr.dev/chainguard/go:1.20

Our goal is to prevent your project from experiencing any disruptions. Please see the migration guide for options.
If there's more we can do to help please reply to this issue or email us at<EMAIL_ADDRESS>Thank you!

Closed by #10 (which uses the alpine-go and scratch images)


Updated screenshots in manual for device simulator

Updated installation manual screenshots with new logo for device simulator.
@Sander, can you take a look?

All is well.

Looks good!


[SPARK-8848] [SQL] Refactors Parquet write path to follow parquet-format

This PR refactors Parquet write path to follow parquet-format spec.  It's a successor of PR #7679, but with less non-essential changes.
Major changes include:

Replaces RowWriteSupport and MutableRowWriteSupport with CatalystWriteSupport

Writes Parquet data using standard layout defined in parquet-format
Specifically, we are now writing ...

... arrays and maps in standard 3-level structure with proper annotations and field names
... decimals as INT32 and INT64 whenever possible, and taking FIXED_LEN_BYTE_ARRAY as the final fallback

Supports legacy mode which is compatible with Parquet format used by Spark 1.5 and prior versions
The legacy mode is by default off, and can be turned on by flipping SQL option spark.sql.parquet.writeLegacyFormat to true.

Eliminates per value data type dispatching costs via prebuilt composed writer functions

Cleans up the last pieces of old Parquet support code

As pointed out by @rxin previously, we probably want to rename all those Catalyst* Parquet classes to Parquet* for clarity.  But I'd like to do this in a follow-up PR to minimize code review noises in this one.

Merged build triggered.

Merged build started.

Test build #43258 has started for   PR 8988 at commit d1583f8.

Test build #43258 has finished for   PR 8988 at commit d1583f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43258/
Test PASSed.

Merged build triggered.

Merged build started.

Test build #43272 has started for   PR 8988 at commit 6fd20f7.

Test build #43272 has finished for   PR 8988 at commit 6fd20f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43272/
Test PASSed.

Merged build triggered.

Merged build started.

Test build #43284 has started for   PR 8988 at commit d395bfd.

Fixed a bug related to UDT: an exception is thrown when reading Parquet files containing UDT values under standard mode. Regression tests are added in [UserDefinedTypeSuite] 1.
In 1.5 and earlier versions, when reading Parquet files containing UDT values, we pass schema containing UDT to CatalystRowConverter and translate UDTs into corresponding Parquet types there to create value converters. The problem is that, CatalystRowConverter isn't aware of standard/legacy mode, and always generate a Parquet type in legacy format. The last commit fixes this issue by expanding UDTs early in CatalystRowMaterializer, so that CatalystRowConverter doesn't need to worry about UDTs anymore.

Test build #43284 has finished for   PR 8988 at commit d395bfd.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43284/
Test FAILed.

The last Jenkins build failure was caused by artifact download failure.

retest this please

Merged build triggered.

Merged build started.

Test build #43288 has started for   PR 8988 at commit d395bfd.

Merged build triggered.

Merged build started.

Test build #43291 has started for   PR 8988 at commit a680f09.

Test build #43288 has finished for   PR 8988 at commit d395bfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43288/
Test PASSed.

Test build #43291 has finished for   PR 8988 at commit a680f09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43291/
Test PASSed.

Merged build triggered.

Merged build started.

Test build #43355 has started for   PR 8988 at commit 5b08a20.

@liancheng As we discussed offline, we should turn the legacy mode off by default, which is compatible for 1.4 and prior versions.

Test build #43355 has finished for   PR 8988 at commit 5b08a20.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43355/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #43360 has started for   PR 8988 at commit f03ef93.

Merged build triggered.

Merged build started.

@davies Thanks for the review. Turned legacy mode off by default, and made it a public option. Other offline comments are also addressed.

Test build #43361 has started for   PR 8988 at commit 2bc5ebc.

The last build failure was caused by a flaky artifact downloading failure.

Merged build triggered.

Merged build started.

Test build #43365 has started for   PR 8988 at commit c542ae9.

Merged build triggered.

Merged build started.

Test build #43372 has started for   PR 8988 at commit af50f9c.

Test build #43361 has finished for   PR 8988 at commit 2bc5ebc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43361/
Test FAILed.

Merged build finished. Test FAILed.

Test build #43360 has finished for   PR 8988 at commit f03ef93.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43360/
Test PASSed.

Test build #43365 has finished for   PR 8988 at commit c542ae9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43365/
Test FAILed.

Test build #43372 has finished for   PR 8988 at commit af50f9c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43372/
Test FAILed.

@davies The last build failure was because Hive only recognizes decimals written as FIXED_LEN_BYTE_ARRAY. This might be a good reason for turning legacy mode on by default?

Fixed failed test Hive test cases by enabling legacy mode explicitly.

Merged build triggered.

Merged build started.

Test build #43387 has started for   PR 8988 at commit e67d0b1.

Test build #43387 has finished for   PR 8988 at commit e67d0b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43387/
Test PASSed.

LGTM, with except some minor comments, could you also update the PR description? (1.5 -> 1.4)

@davies All comments addressed. Thanks!

Merged build triggered.

Merged build started.

Test build #43417 has started for   PR 8988 at commit fb6ee9f.

Test build #43417 has finished for   PR 8988 at commit fb6ee9f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43417/
Test FAILed.

The last build failure was caused by #8983, which broke master and has just been reverted.

retest this please

Merged build triggered.

Merged build started.

Test build #43427 has started for   PR 8988 at commit fb6ee9f.

A note about interoperability:
Hive 1.2.1 can read Parquet arrays and maps written in standard format. However, it still doesn't recognize Parquet decimals stored as INT32 or INT64 ([HIVE-12069] 1).  There are two options to workaround this issue:

Always turn on legacy mode if the written Parquet files are supposed to be used together with Hive.
Legacy mode is turned off by default in this PR.

Add a separate SQL option spark.sql.parquet.writeCompactDecimal to indicate whether decimals can be written as INT32 and INT64.
This PR hasn't implemented this option yet.  If we prefer this approach, I can do it in another PR.  We probably want this option to be false by default.

I'd vote for 2.
@davies @marmbrus @rxin Thoughts?

Since we already have an option for being compatible with Hive (the legacy mode), then we should not worry that (do not need to change anything in this PR).
Hive 1.2 and Spark 1.4 will exists for a long time, If we plan to be compatible with them out of box (without any configurations), then we can't move forward.
Parquet format 2 will have the same issue (compatibility).

@davies Although Hive doesn't write using standard Parquet format, it can read standard LIST and MAP. It just doesn't recognize compact decimals. So even if we turn off legacy mode, we can still interoperate with Hive as long as no compact decimals are written (either by disabling it explicit using extra SQL option, or by writing decimals with precision larger than 18). The benefit of adding an extra option is that we can still let Spark write standard Parquet files by default.

Test build #43427 has finished for   PR 8988 at commit fb6ee9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43427/
Test PASSed.

Merged build finished. Test PASSed.

Merged to master, thanks @davies for the detailed review!
Finally fixed all the Parquet compatibility issues after 6 months!


Allow custom awc Client for fetching OIDC configuration

Currently  the OIDC configuration is fetched with a default awc Client. Although this is convenient, it does not support fetching OpenID configuration from providers that use self-signed certificates, e.g. in local test environments.
[2023-04-06T09:26:21Z WARN  rustls::conn] Sending fatal alert BadCertificate
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ConnectivityError(Connect(Io(Custom { kind: InvalidData, error: InvalidCertificateData("invalid peer certificate: UnknownIssuer") })))', apps/api-actix/src/main.rs:21:87
stack backtrace:

See: let client = awc::Client::default();.  in https://github.com/spectare/actix-4-jwt-auth/blob/main/src/oidc.rs#L106
It would be great if actix-4-jwt-auth would allow users to provide their own awc Client that they could configure appropriately.

Perhaps we could also use a "default" client provided to Actix itself, such that all requests use the same http client.

My current workaround is fetching the JWKSKeySet with a custom awc client with openssl and than create the OidcConfig from the JWKS:
...
    // workaround for fix default awc client used by OidcConfig::Issuer
    let oidc = fetch_oidc_config(&config).await;
    //let oidc = Oidc::new(OidcConfig::Issuer(config.oidc_issuer.clone().into())).await.unwrap();
...

async fn fetch_oidc_config(config: &Config) -> Oidc {

    // fetch jwks, and construct OidcConfig from JWKS set

    let client_tls_config = SslConnector::builder(SslMethod::tls()).unwrap() //
        .build();

    let client = awc::Client::builder() //
        .add_default_header((header::USER_AGENT, "api-actix/0.1.0")) //
        .connector(Connector::new().openssl(client_tls_config)) //
        .finish();

    let jwks_key_set = client.get(config.oidc_issuer.as_str().to_string() + "/protocol/openid-connect/certs").send().await //
        .unwrap().json::<JWKSet<Empty>>().await.unwrap();

    Oidc::new(OidcConfig::Jwks(jwks_key_set)).await.unwrap()
}

Hi @thomasdarimont,
Do you have a sample oidc config that's using self signed certificates ?
Using Oidc::new(OidcConfig::Jwks(your_jwks_key_set) is indeed the way to setup when you like to use your own keys.


Shuffle Auto playlist

I would like to request a feature in the options to shuffle the auto playlist when music bot starts

The songs played from the autoplaylist are picked at random, so therefore are already 'shuffled'.


Simplify nix-shell expressions for IceCap

The build system for Veracruz on IceCap uses nix-shell to manage the dependencies and environment of Veracruz rust components. This PR moves some complexity out of icecap/nix/env/*.nix and into new environment-specific makefiles.
This change is part of a wider effort to move complexity out of Nix and into more familiar planes of the build system.

@nspin is this ready for review or are you still working on it?

Yes, it is ready now.

LGTM.  +1 reached, merging.


⚠ ClamAV has degraded performance

In 2f228e8, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 75 ms

Resolved: ClamAV performance has improved in 673263a after 539 days, 14 hours, 46 minutes.


Receiving endpoint

Reopening this PR to finish this endpoint.
I was wondering if it would be okay to remove some tests to speed up the test suite. In particular, there are two tests for each endpoint where one has optional query params and one doesn't. Would it be okay to just add the params for each one? Less requests made that way.

@alexeyzab that change sounds fine, go ahead and push it into this PR.

@bitemyapp Great, will do. There are a few more things to do with this endpoint (namely, more functions like canceling a receiving and so on) I'll be working on that now.

@bitemyapp I think that covers all of the /receivings endpoint functions. If it looks good, we can merge it in and I'll move on to /products.

@alexeyzab :+1: :smile_cat:


Update podspec

When everything is passed a resource the plist get's added twice
because Xcode adds it anyway. This ends in a compiler error

Sorry, I'm no longer maintaining it. Feel free to fork it.


Specialize EntityDecoder for strict entities

I thought about moving this method into Entity. The thing that put me off was deciding what to name it. chunkF?

Or, we could have something like Stream#compile. So entity.compile.toChunk etc.

Here's a a related/different idea: what if we have in Entity:
def toStrict[F[_]: Concurrent]: F[Entity.Strict[Byte]]

Then, you can grab the chunk directly.

The thing that slightly concerns me about that is that different Entity types have different semantics with respect to http (in ways I don't really understand). I think converting to a different kind of Entity just to collect its bytes risks muddying the concepts.

I might like entity.compile.toChunk, but what other compile ops are there to justify the compile?  There's toByteVector.

Superseded.