DeltaTable - Files Used List

Is there a way to find the list of files actively used by a delta table? The delta table states that it is using 20k files but we find almost 13 million files in the storage account. To check whats going on and seperate the files that arent used by delta anymore, do we have a command?
The vacuum command is already running with the defailt 7days. What is the best way to clean the unused files?

use the generate method of the DeltaTable class this will generate a manifest that contain all parquet files path used. May be we should try to delete manualy all other parquet not listed by the manifest.

use the generate method of the DeltaTable class this will generate a manifest that contain all parquet files path used. May be we should try to delete manualy all other parquet not listed by the manifest.

@Flexron , thanks for the reply. Apologies that I haven't mentioned I am on Azure Databricks. I have tried that one but it isn't supported in Azure databricks. Error: "Generation of manifests for Delta table is currently only supported on AWS S3." So I think I cannot use this option.

I'm using  Azure Databricks too. You Can disable the fileSystem check by set in your sparkSession conf spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled to false.

I'm using Azure Databricks too. You Can disable the fileSystem check by set in your sparkSession conf spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled to false.

Thanks @Flexron , after changing this setting, am able to generate the manifest. But when I compare the count of files from the manifest and the one displayed in the interface, they are not matching. I tried to run an optimize, vacuum with 0 hours retention and then generate and compare the counts. Still the counts don't match. I have a feeling the manifest generation is also not giving the latest set of files being used. Kindly let me know if I am missing someting.
The following is the code used to generate and get the file list.
`# Set the configurataion for AzureDatabricks
spark.conf.set("spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled", False)
Generate symlink_format_manifest
from delta import *
delta_table = DeltaTable.forPath(spark, "/mnt/datalake/pathofthedeltalocation")
delta_table.generate("symlink_format_manifest")
generatedManifestPath = "/mnt/datalake/pathofthedeltalocation/_symlink_format_manifest/"
tempManifestFilesOnlyPath = "/mnt/datalake/temp/"
#List the manifest Files
def get_dir_content(ls_path):
dir_paths = dbutils.fs.ls(ls_path)
subdir_paths = [get_dir_content(p.path) for p in dir_paths if p.isDir() and p.path != ls_path]
flat_subdir_paths = [p for subdir in subdir_paths for p in subdir]
return list(map(lambda p: p.path, dir_paths)) + flat_subdir_paths
#Exclude empty folders and copy the manifest files to temp path
paths = get_dir_content(generatedManifestPath)
i=1
for path in paths:
if "/manifest" in  path:
dbutils.fs.cp(path, tempManifestFilesOnlyPath + str(i) + ".csv")
i = i + 1
#Read the Manifest Files and give the count
df = spark.read.csv(tempManifestFilesOnlyPath)
df.count()`
I get the count as 4700 while the interface shows 935. Thanks in advance

Since this is Azure Databricks issue, please contact Databricks customer support so that they can take a look at this. The GH issues are generally focused on Delta OSS issues. If you are seeing same set of issues using OSS generate manifest as well.. then please report that.

@tdas , sure, trying that way already. Will close this issue. Thanks for the support!


Update How-to-release

Coverage remained the same at 19.191% when pulling 1d2b75b225b8c36c7242dedca44c7f2cb6d0d25b on wu-sheng-patch-1 into 4589f4e46c8e2c73f6499182ff557d5ecd1f6c36 on master.


update Tomcat sample to use EventSources

Sample demonstrating multiple Controllers and EventSource on Deployments.

@adam-sandor I'm not convinced we need to put this back, I would keep it in the separate repo.
@psycho-ir @metacosm  what do you think?

I would do the update here, because that's where all the samples are, then move it to where we agree to put them.

@adam-sandor but we already moved this to a separate repo. Don't understand why we want to put this back.

It was never removed so I'm not really moving it back. I would like to deprecate the tomcat-operator repo.

kk, then its fine.
However let's discuss what samples to keep here in main repo, since are more samples with no clear purpose. We should have clearly defined goal with each of them, since it takes time to maintain them.


login as sonarqube user to install sonarqube

sonarqube should not be installed as root user

@iam-veeramalla


[android] Restart scanners on refresh

This fixes a bug with refreshing mdns results, since mdns results
are typically only found once in a scan, and not frequently like ble
results.

@matthewsibigtroth, don't accept this just yet, but my suspicion is that this is the issue you were running into.

Alright @matthewsibigtroth, this is ready to go.

LGTM


I want to add Dark Mode to New UI Home Page

Is your feature request related to a problem? Please describe.

I want to add Dark Mode to New UI Home Page

Describe the solution you'd like

I already did dark mode in recent Homepage UI I want to implement in this also with more Amazing colors hue changes.

Do you want to work on this issue - [Yes]

@purnima143 If you think This is valid Please assign me as I did this in last Homepage UI

@vedant-jain03 Do u think it's needed for the home page? Can you make the dark mode from the signup sign-in page? Don't show the dark mode button on the homepage.

So I should make dark mode for Sign up and Sign in only?
I think that their is lot missing in Homepage as compared to modern Website like animations, so I think to look more attractive we should definitely go for Dark mode for Homepage.
My next issue will be of animation of section on Scrolling so that it will look more Professional
What do you think?
Correct me if I am wrong

@vedant-jain03 except homepage place dark mode everywhere...signup sign in, feed page, profile etc


Allow servo-macpro1 to help with the load

r? @edunham @aneeshusa

This change is

@bors-servo r+
Make sure to run a highstate on servo-macpro1, I'm pretty sure I deleted the Buildbot services to stop the logspam on the Buildbot master.

:pushpin: Commit 2427897 has been approved by aneeshusa

:hourglass: Testing commit 2427897 with merge 512f4cd...

Thanks, will do!

:sunny: Test successful - travis


Support reentrant/multithreaded use

Currently Slang has a few global variables that really need to be stored on the session. Moving these into per-session storage should enable a user to have one session per thread.
Ideally we'd be able to use the same session across multiple threads, and only require that a particular compile request be locked to a thread, but that is harder to achieve with the current design, because we end up with reference-counted pointers from per-request data to per-session data.
We could "fix" that by making the reference-counted pointers use atomics, but that would slow down the 99% case. The right long-term fix is to not use "smart" pointers for that stuff.
For now, it is easiest to support one session per thread.

Usage of global variables has been removed in #153, so basic usage of Slang from multiple threads should now be possible. Each thread still needs to use its own SlangSession, though, which could slow things down a lot if clients aren't careful.
I'm going to close this issue for now, while recognizing that the larger technical-debt issue remains.


Support for macOS (Intel)?

As the build compiled for iOS works with M1 Mac's as foundation and UIKit are available there.
Any ideas how to make it work on intel based Macs?

This isn't intended to work for M1 mac


Audio not downloaded as "transcript.mp3" in Chrome

In Chrome, I see the option to download the audio here:

This always downloads the audio as 60aa3761-59f1-4519-baca-f029e9a0d8bc.mp3 instead of transcript.mp3 like when I download through this:

Should the GUID be removed?

That's a feature that chrome is adding. We can't control it.


Fix getTeams() and getTeam()

This PR fixes a bug I found in the client. If you tried to execute getTeams() or getTeam(), it gave an error about .map(). I found that the client was trying to run map on the entire response data from the API (which is an object so has no map method).
For reference:
{
   "status": 200,
   "error": "",
   "teams": [
      {
         "_id": "5e2f142e714294ee19c1ad2a",
         "name": "TestTeam",
         "creator": "5e2efe22714294ee19c1ad29"
      }
   ]
}

I also improved the readability of the README when it is rendered.

LGTM, good spot 👍


luhn: update tests and version file (v1.0.0 -> v1.2.0)

Please update the version specified in the luhn version file from 1.0.0 to 1.2.0.
This update has been caused by the application of a new input policy, and a new test being added.
Please additionally update the test suite to match the canonical-data. New test strings with non-digits is invalid should be added.

Closed via #259


Fix websocket_client version

Currently last version(1.4+) of websocket_client cannot compile with rebar3.
This issue was reproduced in three different environments
** (Mix) Could not compile dependency :websocket_client, "/home/gabriel/.asdf/installs/elixir/1.5.2/.mix/rebar3 bare compile --paths "/home/gabriel/workspace/iexikos/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile websocket_client", update it with "mix deps.update websocket_client" or clean it with "mix deps.clean websocket_client"

Coverage remained the same at 99.716% when pulling 940bbd00a877ee953b918e1a8eedd122249dcb2d on paulobsa:master into 7c6186ba8a90d9406d2b79351081373a52baffdf on shinyscorpion:master.


fix bug when push a branch name with / & fix an integration test bug

fix #1685

Trusted LGTM

LGTM

make L-G-T-M work


CI: update Fedora 34 -> 36

Proposed Changes

Update Fedora 34 to 36.
Fedora 34 has reached EOL on 2022-06-07
https://docs.fedoraproject.org/en-US/releases/eol/
Types of Changes

CI
Verification

Observe the CI result
Testing

Observe the CI result
Linked Issues

🚫
User-Facing Change

🚫
Further Comments

🚫

This works fine for me locally, I'm not sure why it's failing in GH Actions.

@dereknola do you know of a good way to get the conformance test to output more information when it fails? It works fine locally for me, and the output that we do get isn't really enough to troubleshoot with.
    cgroup-unified: Waiting for deployment "coredns" rollout to finish: 0 of 1 updated replicas are available...
    cgroup-unified:
    cgroup-unified: real	8m20.007s
    cgroup-unified: user	0m3.903s
    cgroup-unified: sys	0m3.069s
The SSH command responded with a non-zero exit status.

Its all just bash scripts, you would need to modify https://github.com/AkihiroSuda/k3s/blob/fedora-36/tests/cgroup/unified/fedora-36/Vagrantfile#L85 to output more info on detecting an exit != 0

Locally testing this, I'm still not sure why fedora-36 is broken, it pass just fine on my machine. I did test with fedora-35 on GH and that passes locally and on CI. So for now we can just move to fedora-35 instead. Yes it EOL in November, but that gives us more time to fix fedora-36.

Locally testing this, I'm still not sure why fedora-36 is broken, it pass just fine on my machine. I did test with fedora-35 on GH and that passes locally and on CI. So for now we can just move to fedora-35 instead. Yes it EOL in November, but that gives us more time to fix fedora-36.

Sure, opened https://github.com/k3s-io/k3s/pull/5996

Closing this PR as we went with updating to 35 instead.


How to get money back from PhonePe if wrong transaction?

Proposed changes
Answer. If you mistakenly sent money to the wrong number9123--892--436 on PhonePe, it is generally not possible to reverse the transaction directly through the app. However, you can contact PhonePe customer support 09123-892436 as soon as possible to report the issue and request assistance in resolving the situation.

Subject Matter
India
Content Location
Answer. If you mistakenly sent money to the wrong number9123--892--436 on PhonePe, it is generally not possible to reverse the transaction directly through the app. However, you can contact PhonePe customer support 09123-892436 as soon as possible to report the issue and request assistance in resolving the situation.
Additional information
Answer. If you mistakenly sent money to the wrong number9123--892--436 on PhonePe, it is generally not possible to reverse the transaction directly through the app. However, you can contact PhonePe customer support 09123-892436 as soon as possible to report the issue and request assistance in resolving the situation.

Answer. If you mistakenly sent money to the wrong number9123--892--436 on PhonePe, it is generally not possible to reverse the transaction directly through the app. However, you can contact PhonePe customer support 09123-892436 as soon as possible to report the issue and request assistance in resolving the situation.

Answer. If you mistakenly sent money to the wrong number9123--892--436 on PhonePe, it is generally not possible to reverse the transaction directly through the app. However, you can contact PhonePe customer support 09123-892436 as soon as possible to report the issue and request assistance in resolving the situation.


Incorrect decimal places for some BitAssets in /markets

Specifically, the "total" field is 10x too large for BitGold shorts in version 0.6.1.
I suspect the problem is in setting of order.cost which occurs here.  This is web_wallet commit 3044b4fdd9c59960de81c18950011375b4f6143f

@theoreticalbts could tell me where do you see this issue? Featured Markets charts?

thx


It would be great if we could specify a timezone as well

e.g.
every :day, at: 10, :timezone: "UTC"

every :day, at: "10:00 UTC"

@Trevoke It's on the roadmap, I will close this for now.


Keep terminal running when reloading, optionally

My workflow often is:

Run an npm script do build and serve my node/javascript app with a watcher
Open another terminal in vs code to perform basic tasks
Make changes to the code
See the changes in the browser

This workflow is great, except when I have to reload the vs code instance due to a new extension update or some other change. When this happens, my terminal processes are lost too.
What I’d like is an option to have the terminal processes restart or keep running when the instance reloads.
My workaround today is to restart them manually or to use an external terminal.

/duplicate https://github.com/microsoft/vscode/issues/117408 you can also configure tasks to run when opening a folder


deno: fix darwin build

Description of changes

Add missing framework for darwin
resolves: #275375
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

It might also need this CoreGraphics framework too.
If someone can try out this PR that'd be helpful

and actually if this builds we can probably remove the postPatch because a recent change here broke the substitute so it would mean it's not necessary anymore
https://github.com/denoland/deno/commit/393abed3873d83019feb5bcebb10a6929133862a

@xanderio

Result of nixpkgs-review pr 275377 run on aarch64-darwin 1

9 packages failed to build:

deno
python310Packages.nbdev
python310Packages.nbdev.dist
python311Packages.nbdev
python311Packages.nbdev.dist
quarto
quartoMinimal
vimPlugins.denops-vim
vimPlugins.vim-pluto

@06kellyjac
       > error: linking with `/nix/store/vwh2qqal1q101wv4jy40giavp73q0msd-clang-wrapper-16.0.6/bin/cc` failed: exit status: 1
       >   |
       >   = note: env -u IPHONEOS_DEPLOYMENT_TARGET -u TVOS_DEPLOYMENT_TARGET LC_ALL="C" PATH="/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/bin:/nix/store/30m4ghksg1dlnwfy3qybfixjwnrhgrh9-cargo-1.74.0/bin:/nix/store/6a8n27y9r10pnr653344yybxr4qpd7g9-cargo-auditable-0.6.1/bin:/nix/store/9m0szzwibi1rwp3wvn36r7w478i753ll-cmake-3.27.8/bin:/nix/store/yc7vkwp4fcz2kwwzmbd118garra589j1-ps-adv_cmds-119/bin:/nix/store/ysijl023j19h9ls25djxq43dm897v094-protobuf-24.4/bin:/nix/store/rp517rlw5kbxcwm382ap5lg0jvn4nakr-auditable-cargo-1.74.0/bin:/nix/store/30m4ghksg1dlnwfy3qybfixjwnrhgrh9-cargo-1.74.0/bin:/nix/store/i2q9ff0fpgr20zza6kr6nlpki6mn43if-rustc-wrapper-1.74.0/bin:/nix/store/vwh2qqal1q101wv4jy40giavp73q0msd-clang-wrapper-16.0.6/bin:/nix/store/3vzbr8yb4c233lhi1qz78b8kaaxrjqcj-clang-16.0.6/bin:/nix/store/pzajj7gw0nazs3vzp6jld53q8ird5izr-coreutils-9.4/bin:/nix/store/0i7w6qsij8akg6h39qn67w9ig5hirv1a-cctools-binutils-darwin-wrapper-16.0.6-973.0.1/bin:/nix/store/i4sfjfz08vmnrj84asl2j5imakga12za-cctools-binutils-darwin-16.0.6-973.0.1/bin:/nix/store/pzajj7gw0nazs3vzp6jld53q8ird5izr-coreutils-9.4/bin:/nix/store/6x5inimwy8rlvf678yy080zccbw71alb-findutils-4.9.0/bin:/nix/store/01bzxsyn8dqz9fsdhp9r6ya796g5ydmf-diffutils-3.10/bin:/nix/store/9z9k8dwhhiryr7z2pz70zca0vwa15yqc-gnused-4.9/bin:/nix/store/zsm4xd527427rnjfnvfhvsld2n57b822-gnugrep-3.11/bin:/nix/store/hhc2x65zvp89lh15grqk21r6dcvifggy-gawk-5.2.2/bin:/nix/store/gclclfhq9blxp32lv8kxm9iv5060rx5f-gnutar-1.35/bin:/nix/store/crpi4clk07gy760ydbb710f74dy8q6c9-gzip-1.13/bin:/nix/store/qay0pa86lb712162d6m5lyw62iczb71d-bzip2-1.0.8-bin/bin:/nix/store/zc9qsb0rzlw0ckhy7097frcwb4pkfx3d-gnumake-4.4.1/bin:/nix/store/6nxav88iiz0g8m598xy643f8hhdz5kkx-bash-5.2-p21/bin:/nix/store/kizmwlnj40d11ixbap74c0syaqxn05wc-patch-2.7.6/bin:/nix/store/xcy0fr1dangwh9r9isay6fqbg91g5b3h-xz-5.4.5-bin/bin:/nix/store/x1bzwj6pz7f3ajg5y36x07b1328jzx7r-file-5.45/bin" VSLANG="1033" ZERO_AR_DATE="1" "/nix/store/vwh2qqal1q101wv4jy40giavp73q0msd-clang-wrapper-16.0.6/bin/cc" "-Wl,-exported_symbols_list,/private/tmp/nix-build-deno-1.39.0.drv-0/rustcaLrw0q/list" "-arch" "arm64" "/private/tmp/nix-build-deno-1.39.0.drv-0/rustcaLrw0q/symbols.o" "/private/tmp/nix-build-deno-1.39.0.drv-0/source/target/aarch64-apple-darwin/release/deps/slug-fc0809b284e651e9.slug.a92eeefe0ccb5586-cgu.0.rcgu.o" "/private/tmp/nix-build-deno-1.39.0.drv-0/source/target/aarch64-apple-darwin/release/deps/slug-fc0809b284e651e9.kfthj897sv3xwa7.rcgu.o" "-L" "/private/tmp/nix-build-deno-1.39.0.drv-0/source/target/aarch64-apple-darwin/release/deps" "-L" "/private/tmp/nix-build-deno-1.39.0.drv-0/source/target/release/deps" "-L" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib" "/private/tmp/nix-build-deno-1.39.0.drv-0/source/target/aarch64-apple-darwin/release/deps/libdeunicode-1017d4e5790a63c6.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libstd-03c53b54a0b1f97d.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libpanic_unwind-32c6908016812ada.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libobject-d8028f9d6e099c46.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libmemchr-b44bd6fb2b1b839e.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libaddr2line-c8dcf438cf6d13da.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libgimli-703d937e20906b80.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/librustc_demangle-b7ac46c30207e45c.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libstd_detect-4d8dae81211e4895.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libhashbrown-d204bf1a3582275b.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_alloc-e447863e017a3c6b.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libminiz_oxide-ac4d1414355d5dc4.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libadler-6cc53510bafd9383.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libunwind-cc9a97bec3aa68fb.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libcfg_if-1b9937edf2e43f1a.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/liblibc-b935d087394ceac5.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/liballoc-d062ef7598770b49.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-e068a539d3cd8c41.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libcore-c654b0e49d6357d6.rlib" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-738eb43ff73087cd.rlib" "-lSystem" "-lc" "-lm" "-L" "/nix/store/y24qqyfbnnxza5841hd9hnda5icmzymq-rustc-1.74.0/lib/rustlib/aarch64-apple-darwin/lib" "-o" "/private/tmp/nix-build-deno-1.39.0.drv-0/source/target/aarch64-apple-darwin/release/deps/libslug-fc0809b284e651e9.dylib" "-Wl,-dead_strip" "-dynamiclib" "-Wl,-dylib" "-nodefaultlibs" "-fuse-ld=lld" "-weak_framework" "Metal" "-weak_framework" "MetalPerformanceShaders" "-weak_framework" "QuartzCore" "-weak_framework" "CoreGraphics"
       >   = note: clang-16: error: invalid linker name in argument '-fuse-ld=lld'
       >
       >
       > error: could not compile `slug` (lib) due to previous error
       > warning: build failed, waiting for other jobs to finish...

ok so lld was an issue. I've tried providing lld but if that doesn't work I guess I'll just fix the substitute that removes it
lmk if that works ok now @xanderio  :)
I don't have a macos vm at the moment

ok thanks. I'll just put the substitute back and let you know when that's pushed

pushed @xanderio

Result of nixpkgs-review pr 275377 run on aarch64-darwin 1

9 packages failed to build:

deno
python310Packages.nbdev
python310Packages.nbdev.dist
python311Packages.nbdev
python311Packages.nbdev.dist
quarto
quartoMinimal
vimPlugins.denops-vim
vimPlugins.vim-pluto

I'm getting
substituteStream(): WARNING: pattern '"-C", "link-arg=-fuse-ld=lld"' doesn't match anything in file '.cargo/config.toml'

Since upstream introduced a newline:
https://github.com/denoland/deno/commit/393abed3873d83019feb5bcebb10a6929133862a

Minimal patch that fixes the build:
diff --git a/pkgs/development/web/deno/default.nix b/pkgs/development/web/deno/default.nix
index 5ab831adb2d9..abd85b8059fe 100644
--- a/pkgs/development/web/deno/default.nix
+++ b/pkgs/development/web/deno/default.nix
@@ -27,7 +27,7 @@ rustPlatform.buildRustPackage rec {
   postPatch = ''
     # upstream uses lld on aarch64-darwin for faster builds
     # within nix lld looks for CoreFoundation rather than CoreFoundation.tbd and fails
-    substituteInPlace .cargo/config.toml --replace '"-C", "link-arg=-fuse-ld=lld"' ""
+    substituteInPlace .cargo/config.toml --replace '"link-arg=-fuse-ld=lld"' ""
   '';

   # uses zlib-ng but can't dynamically link yet
@@ -39,9 +39,10 @@ rustPlatform.buildRustPackage rec {
     protobuf
     installShellFiles
   ];
+
   buildInputs = lib.optionals stdenv.isDarwin (
     [ libiconv darwin.libobjc ] ++
-    (with darwin.apple_sdk.frameworks; [ Security CoreServices Metal Foundation QuartzCore ])
+    (with darwin.apple_sdk.frameworks; [ Security CoreServices Metal MetalPerformanceShaders Foundation QuartzCore ])
   );

   # work around "error: unknown warning group '-Wunused-but-set-parameter'"

@aqrln you have extra double quotes and link-arg instead of link-args in there, that won't work.
substituteInPlace .cargo/config.toml --replace "link-args=-fuse-ld=lld " ""

if you want to use substituteInPlace.

Oh I guess I didnt actually push. I did change the substitute to work but its not showing up on GH

actually pushed it now

@nekowinston ah yeah you're right, I was under the impression that nix build .#deno succeeded but I was looking at the wrong terminal.

Result of nixpkgs-review pr 275377 run on aarch64-darwin 1

9 packages built:

deno
python310Packages.nbdev
python310Packages.nbdev.dist
python311Packages.nbdev
python311Packages.nbdev.dist
quarto
quartoMinimal
vimPlugins.denops-vim
vimPlugins.vim-pluto

Result of nixpkgs-review pr 275377 run on aarch64-darwin 1

9 packages built:

deno
python310Packages.nbdev
python310Packages.nbdev.dist
python311Packages.nbdev
python311Packages.nbdev.dist
quarto
quartoMinimal
vimPlugins.denops-vim
vimPlugins.vim-pluto

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1325

fixes https://hydra.nixos.org/build/244339583 https://hydra.nixos.org/build/244340479 https://hydra.nixos.org/build/244340144 https://hydra.nixos.org/build/244340026
Don't have a mac but LGTM, thanks!


[WIP] Reduce sources file size

Why are the changes needed?

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

The alternative is to delete the *.map and *es-bundle* files.
(kyuubi) ➜  swagger git:(master) ✗ ll
total 4096
-rw-r--r--  1 chengpan  staff   665B Jul 14 17:31 favicon-16x16.png
-rw-r--r--  1 chengpan  staff   628B Jul 14 17:31 favicon-32x32.png
-rw-r--r--  1 chengpan  staff   3.3K Jul 14 17:31 index.html
-rw-r--r--  1 chengpan  staff   3.3K Sep  7 20:14 oauth2-redirect.html
-rw-r--r--  1 chengpan  staff   1.0M Jul 14 17:31 swagger-ui-bundle.js
-rw-r--r--  1 chengpan  staff   317K Jul 14 17:31 swagger-ui-standalone-preset.js
-rw-r--r--  1 chengpan  staff   140K Jul 14 17:31 swagger-ui.css
-rw-r--r--  1 chengpan  staff   502K Jul 14 17:31 swagger-ui.js
(kyuubi) ➜  swagger git:(master) ✗ du -hd1
2.0M    .

cc @yaooqinn, which way are you preferred?

LGTM

Thanks, merging to master/1.6


feat(NODE-3625): add the addFields function to AggregationCursor

What changed?
The addFields function was added to AggregationCursor class for adding an addFields stage to the pipeline
Are there any files to ignore?
no

Hi @massivefermion, thank you for submitting this suggestion. I created a jira ticket to track it here: https://jira.mongodb.org/browse/NODE-3625.

@massivefermion Hi, just to give you an update: following a discussion of this issue, the team feels that it would be better to provide a general builder helper method for aggregation stages (something along the lines of addStage(<stageName>, <props>)) instead of implementing a special one for every stage. We believe that approach would be more flexible for future aggregation stages and, in general, more maintainable from a test coverage perspective. Currently, we intend to support explicit helpers only for the most frequently used stages. I am sorry that we are unable to accept the PR at this time, but we appreciate your suggestion and will consider the general stage helper feature (NODE-3639) for our next minor release. Thank you for understanding!


Start user.slice if not running before trying to use systemctl --user

On my highly stripped down hosts, starting the user systemd session seems to be required before the systemctl --user commands will work.

I hadn't seen this since issue since I run the typical distro package groups.  Thanks for the contribution.


Testing with -race breaks tests with Go 1.14

Apparently there is a pointer misuse

Fixed ages agi


[BugFix] make config enable_new_publish_mechanism static (backport #35338)

Why I'm doing:
Fe config enable_new_publish_mechanism can't support dynamic modify, because we need to rebuild transaction dependency before switch to new publish mechanism. We support rebuild transaction dependency when config change after version 3.2+, but now we don't plan to support it in 2.5/3.0/3.1.
What I'm doing:
Make enable_new_publish_mechanism become static config. So if we want to enable enable_new_publish_mechanism, we have to restart FE, and transaction dependency will rebuild when restart.
backport #35338
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

https://github.com/apps/mergify backport branch-3.1-cs


Groupby Shift produces incorrect result

What happened:
Data shifts to incorrect location when using groupby shift operation.
What you expected to happen:
Data should be shifted along the sorted index within each group.
Minimal Complete Verifiable Example:
df1 = pd.DataFrame(
    {
        "a": [1, 2, 3, 4],
        "b": [2, 1, 2, 1],
        "c": [None, None, 10, 20]
    }
)
ddf1 = dd.from_pandas(df1, npartitions=1)
df2 = pd.DataFrame(
    {
        "a": [5, 6],
        "b": [2, 1],
        "c": [30, 40]
    }
)
ddf2 = dd.from_pandas(df2, npartitions=1)
df3 = pd.concat([df1, df2])
ddf3 = dd.concat([ddf1, ddf2])

df3 = df3.set_index("a").sort_index()
ddf3 = ddf3.set_index("a")

with pytest.warns(UserWarning):
    assert_eq(
        df3.groupby("b")["c"].shift(1),
        ddf3.groupby("b")["c"].shift(1),
    )

Anything else we need to know?:
There is a comment in the source code that "If the grouper does not align with the index then this causes a full shuffle.  The order of rows within each group may not be preserved." Although this is fine for apply/transform, preserving order is vital to the correctness of the shift operation.
Potential solution:
Sort each partition after shuffling. From dask/dataframe/groupby.py shift method:
        if should_shuffle:
            sort_idx = df.index.name
            df2, by = self._shuffle(meta)
            df2 = df2.map_partitions(lambda x: x.sort_values(sort_idx))

Environment:

Dask version: 2022.2.0
Python version: 3.8.2
Operating System: Mac OS
Install method (conda, pip, source): pip

I ran into this problem while working on #8659 in https://github.com/dask/dask/pull/8659#issuecomment-1037266953
I'm not sure sorting each partition after a full shuffle would resolve this problem, as the partitions themselves may be different. A naive approach could be to create a monotonically increasing global index column and sort based on that.
import dask.dataframe as dd
import pandas as pd

df = pd.DataFrame({
    "a":[0,0,0,1,1,2,1,0,2,1,0],
    "b":[10,3,3,5,5,4,2,9,3,10,4]
})
ddf = dd.from_pandas(df, 2) #.reset_index(drop=True)

# pandas
df["lagged"] = df.groupby("a").rank()

# Dask
ddf["monotonic"] = 1
ddf["monotonic"] = ddf["monotonic"].cumsum()

def func(df):
    ranked = df.groupby("a")[["b"]].rank()
    df["lagged"] = ranked
    return df

ddf = ddf.shuffle(on="a").map_partitions(func).sort_values(["monotonic"])
ddf = ddf.drop("monotonic", axis=1)

pd.testing.assert_frame_equal(
    ddf.compute(),
    df
)

print(ddf.compute())
print(df)
    a   b  lagged
0   0  10     5.0
1   0   3     1.5
2   0   3     1.5
3   1   5     2.5
4   1   5     2.5
5   2   4     2.0
6   1   2     1.0
7   0   9     4.0
8   2   3     1.0
9   1  10     4.0
10  0   4     3.0
    a   b  lagged
0   0  10     5.0
1   0   3     1.5
2   0   3     1.5
3   1   5     2.5
4   1   5     2.5
5   2   4     2.0
6   1   2     1.0
7   0   9     4.0
8   2   3     1.0
9   1  10     4.0
10  0   4     3.0

cc @jsignell , interested in your thoughts. Perhaps further discussion should move to this issue rather than the groupby.rank PR

Hmmm for the original issue I am not totally convinced that the shift is doing the wrong thing. The c value for each index value matches that in pandas. It is just that the index is not in order.
expected = df3.groupby("b")["c"].shift(1)
actual = ddf3.groupby("b")["c"].shift(1).compute()

In [5]: actual
Out[5]:
a
2     NaN
4     NaN
6    20.0
1     NaN
3     NaN
5    10.0
Name: c, dtype: float64

In [6]: expected
Out[6]:
a
1     NaN
2     NaN
3     NaN
4     NaN
5    10.0
6    20.0
Name: c, dtype: float64

Hmm, interesting. I just reran it a few times and it seems to be non-deterministic. Sometimes I get the same result that you did (which I agree is the expected result). But other times I get the following result:
a
1     NaN
2     NaN
3     NaN
4     NaN
5    10.0
6    20.0
Name: c, dtype: float64

a
6     NaN
2    40.0
4     NaN
5     NaN
1    30.0
3     NaN
Name: c, dtype: float64

Ah ok. I admit it was a little bit wishful thinking. Thanks for indulging me.

I have opened a PR with the within-partition-sorting approach. I can also offer some help if you would like to go with the global index (or some other) approch to fix the issue discussed on #8659.

Thanks @kori73! So this is closed by #8782


InvalidSyntaxError while runnig spring boot app based on graphql-dgs-spring-boot-starter

Discussed in https://github.com/Netflix/dgs-framework/discussions/1479

Originally posted by kgromov April 14, 2023
Hi, I faced with the following issue:
InvalidSyntaxError while runnig spring boot app based on graphql-dgs-spring-boot-starter

Below are some details/expectations.
Expected behavior:
Application is up and running with successfully generated classes from defined schema

Actual behavior:
Application is failing on startup with not meaningful error message

Steps to reproduce:

I'm a  using graphql-dgs-platform-dependencies:6.0.1 with spring-boot 3.+ (3.05), java 17 and code generation plugin ('com.netflix.dgs.codegen' version "5.7.0") for home examples.
Code generation is passed successfully application is failed at startup with the following unclear error:

There are problems with the GraphQL Schema:
	 * InvalidSyntaxError{ message=Invalid Syntax : offending token '"name"' at line 2 column 3 ,offendingToken="name" ,locations=[SourceLocation{line=2, column=3}] ,sourcePreview={
  "name": "Untitled GraphQL Schema",
  "schemaPath": "schema.graphql",
  "extensions": {
    "endpoints": {
}

Absolutely same schema is working fine with spring-boot-starter-graphql based spring boot app v3.0.5 and java 17 (there is only difference in graphql-java version - 19.5 vs 19.3 in spring-boot-starter-graphql) with hand written classes corresponding to schema types.
To my mind if schema is incorrect code generation should fail, isn't it?
Here is schema:
type Query {
    category(id: String): Category
    categoryByName(name: String): Category
    categories: [Category]
    recipe(id: String): Recipe
    recipes: [Recipe]
}

type Mutation {
    addRecipe(description: String): Recipe
    addRecipeWithPayload(recipeDto: RecipeDto): Recipe
    addCategory(name: String): Category
}

type Category {
    id: ID
    name: String
    recipes: [Recipe]
}

type Recipe {
    id: ID
    description: String
    prepTime: Int
    cookTime: Int
    servings: Int
    source: String
    url: String
    notes: String
    difficulty: Difficulty
    category: Category
}

input RecipeDto {
    description: String
    prepTime: Int
    cookTime: Int
    servings: Int
    source: String
    url: String
    notes: String
    difficulty: Difficulty
    categoryId: String
}

enum Difficulty {
    EASY, MODERATE, KIND_OF_HARD, HARD
}

Full source code can be found here
Working solution without netflix platform/starters - here.
Thanks in advance

Hey @kgromov thank you for sharing this issue and your source code. I was able to determine that the issue seems to be that when there is a .graphqlconfig file in the schema/ path, it is being ingested as possible schema. On our end, we're going to dig deeper into why that is, however, in the meantime, my suggestion would be to move this file outside of schema/. If you can, I would move it to your application repository root folder.
After our investigation, we may update this thread with our findings. Please let us know whether the alternative suggestion above is helpful or not.

Hi, @coleturner, moving .graphqlconfig outside shema folder helped.
So I guess issue can be closed.


Persist the UI configs to ZAP

Fixes #321
The tools are now read from the ZAP configs via the API - if they are not set then they are defaulted to the values now held in serviceworker.js
I suspect this is slowing down start up a little bit but I dont have any measurements to prove it.
Injecting the default values into serviceworker.js would probably involve fewer changes and be quicker, but I thought I'd try the 'right' way first :)
I've seen a couple of cases where the tools are duplicated - this looks like its a client side issue as the values stored in the ZAP config file have been correct. I'm guessing its a timing issue when loading the tools but will need to check that.
General feedback appreciated - I'll look into the above bug asap.
It might also be worth waiting until #351 is merged before merging this one.

OK, this is now working nearly all of the time.
About 1 in 6 times (for me) the left frame fails to load.
If you put debug logging on utils.loadFrame and utils.saveFrame you'll see its because on startup the left panel load frame (called from panel.js via utils.loadPanelTools) happens before the save frame (called from utils.initializeHUD)
@dscrobonia (or anyone else) any suggestions?
I guess this could have happened without my changes, but the delay introduced by calling ZAP to read the configs just makes it a little bit more likely to happen.

The easiest way to reproduce the problem is to just keep launching Firefox from ZAP until the left frame fails to load.

After some playing around I've changed it so that the default tools are injected.
This seems to have fixed the problem and should speed up the initial load ever so slightly :)
I think its now ready :)

As @thc202 has pointed out, this is actually broken.
Dont bother reviewing it until I've fixed it :/

Have fixed most of the problems with this, but just spotted another one - once the Reset option is used all of the tools appear in the 'add tool' dialog - will look into that now...

Arg, not quite there yet - resetToDefaults is only working in the first tab :/

The problem is that with Firefox the webSocket is null for all except the first tab, as per #199 - it works fine in chrome.
I'll see if I can work out a solution...

I'm struggling with this one :/
@dscrobonia can you think of a way to communicate to the first service worker or will we just need to say that on Firefox you can only change the configs in the first tab until that Fx bug is fixed?

Just pushed a fix which prevents the tools re-registering for ZAP events, which could have significantly increased the number of messages received. Good spot @thc202 :)

Think I've just fixed the problem on Firefox :)
The issue was that I'd moved the websocket initialization into the onInstall handler to reduce problems with making the upgradedDomains API call too early.
The problem is that onInstall appears to only be called for the first tab.
Moving the key websocket initialization code back inline has fixed this.
Hoping this is good to go now!

Just tested it out and no bugs! Woohoo! I'm running low on sleeps tho so I'll do the code review tomorrow morn. :p Excited for this one!

@dscrobonia think I've addressed all of those comments with code comments or replies :)

Yes we should, but I think we should do that in another PR. I think there are various other places where we should be returning promises but are not :)

Raised #438 for that :)


support env for docker plugin set

The plugin vieux/sshfs was updated to support the evironment variable DEBUG to be set
	"env": [
	    {
		"name":"DEBUG",
		"settable":["value"],
		"value":"0"
	    }
	]

This PR adds support for docker plugin set vieux/sshfs DEBUG=1
ping @anusha-ragunathan @tiborvass
I'm not a big fan of the code I produced :/, feel free to comment! I added unit test.
The biggest ugliness is the code is how to update the env. It requires parsing it back from string array to key/value to detect if we should append to the env and update an existing variable.

@tiborvass @anusha-ragunathan PR was updated :)

@tiborvass PTAL

failure unrelated, ping @anusha-ragunathan

LGTM

Please take care of doc related changes in this PR:

Add/update CLI/API reference docs for plugin set and plugin install.
Provide examples for set, since the usage is a bit non-obvious
Update the manifest reference doc and example.

@vieux tests are failing;
19:21:41 # github.com/docker/docker/integration-cli
19:21:41 ./docker_cli_plugins_test.go:130: no new variables on left side of :=
19:21:41 ---> Making bundle: .integration-daemon-stop (in bundles/1.13.0-dev/test-integration-cli)

LGTM (granted all tests pass)

@vieux @anusha-ragunathan does this need squashing?

@thaJeztah ok I'll squash and merge once the tests are green


Analyzer for more than one gateway with the same host+port combination

Kiali implementation: https://www.kiali.io/documentation/validations/#_more_than_one_gateway_for_the_same_host_port_combination

Ok, so I think the analyzer as implemented by Kiali might be overly restrictive. The example given shows two gateways bound to the same host "*". Something like:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: webserver
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

While this looks like it might be an error, it's not problematic in practice. I can define virtual services and bind them to each gateway, and they continue to work. I believe this is because pilot merges gateways into a single representation before emitting routes (some hint of this is described here: https://istio.io/docs/ops/traffic-management/deploy-guidelines/#multiple-virtual-services-and-destination-rules-for-the-same-host).
I'm not an expert in pilot but I believe this code shows how the merging is done, and redundant or overlapping gateways shouldn't be a problem: https://github.com/istio/istio/blob/36314f4231d1dd69c6374e44be55b49916cf0b80/pilot/pkg/model/gateway.go#L70

@nrjpoddar Neeraj, any thoughts on this?

Closing for now, unless someone can confirm that my reasoning above is wrong.

@selmanj @sushicw I missed this notification so apologies for the delay. I think this analyzer is still needed and yes, Kiali’s implementation might be too restrictive. Pilot in master has fairly complicated logic for the merge but I still think there are scenarios in which it will randomly pick one of the two gateways.

So, maybe handling something like:
// If servers with different protocols attempt to listen on the same port, one of the protocols will be chosen at random.

It certainly is a bad situation, but I see that use case as a bit lower priority. Still, happy to leave it open (but anyone else is free to tackle it as I'm not working on it atm).


User not being immediately logged out on deleting themselves

The current build of Concorda is kicking you when you're logged in and delete the user you're currently logged in as, but only when you navigate to Overview, and then back to Users.
The user should be kicked once the delete themselves, to prevent them from deleting other users afterwards.

@mirceaalexandru @ckiss Is this fixed?

No, is not fixed.
I will take a look

fixed, on next action the user will be redirected to logout page.


Edit member role

Allow edit of role

Codecov Report

Merging #111 (3e96610) into master (b41ae14) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master     #111   +/-   ##
=========================================
  Coverage     61.79%   61.79%
  Complexity      464      464
=========================================
  Files            93       93
  Lines          1691     1691
  Branches        176      176
=========================================
  Hits           1045     1045
  Misses          594      594
  Partials         52       52

Impacted Files
Coverage Δ
Complexity Δ

.../address/logic/parser/EditMemberCommandParser.java
90.00% <ø> (ø)
6.00 <0.00> (ø)

...eedu/address/logic/commands/EditMemberCommand.java
95.45% <100.00%> (ø)
14.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b41ae14...3e96610. Read the comment docs.


Rename from mdast to remark

Hey! Just wanted ping and request a name change from mdast to remark!
You’ve probably already seen it but here’s a list of tips for updating.
Let me know if I can help! :smile:

P.S. as a side note, I suggest using toUpperCase() here. Currently, I think this plug-in would fail if someone specifies a definition identifier of __proto__. I haven’t tested it but something similar caused issues in remark.

Thanks!

I suggest using toUpperCase() here. Currently, I think this plug-in would fail if someone specifies a definition identifier of __proto__.

I have spotted no issues with such identifiers so far, perhaps because I use Object.create(null) both for identifiers and links while visiting them. Is there any other reason for toUpperCase here, based on your experience?

Oh, that’s probably fine too, the only reason was for prototypal properties!


Enhancement: Make dates more visible and usable in VCIO

As we step back a bit and look at our newest release of vulnerablecode.io  I think it is time to start thinking about how users can get the most out of the data.  I believe that it would be a very useful and valuable use case for a user to be able to search VCIO for Vulnerabilities that have been created or modified within the last day, 7 days, 30 days, or similar, and then to see the pertinent dates in the list of returned results and in the details.
I found these date fields in models.py:
date_published
date_collected
date_improved
and there may be others that I missed.
These dates are actually quite valuable information, but I don't see them in the UI (unless I'm missing something).
It also seems to me that the database should really have, on every table, the equivalent of the following:
date_created
date_modified
which would simply be the date of the create/modify in our DB.
This idea should be discussed.  I think we could start by making the already existing fields visible (or more visible) in the UI, and then we might look into searching by, filtering by, and ordering by selected dates.

I would like to consider this in the context of #1007 and also possibly using something like https://github.com/justquick/django-activity-stream to store events. The reason is that implementing dates implies quite a few updates and this more ambitious approach may not be much more work.

Note: we are trying to make the dates consistent across aboutcode projects.

Ater all discussion PR https://github.com/nexB/vulnerablecode/pull/1310 is ready for review now


Update tomcat to version 11

Change description
Update tomcat to version 11
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

Assume this is not needed anymore so closing


you realize that discord has its own official account manager

^^^

my english dies

yes, but with this tool, you can login to discord using the login-token, while discord official 'alt-manager', is your usual login system.


Add link to Godoc

woops; just noticed the godoc image link at the top!

Yup, exactly, there is already a godoc badge.


Failure when trying to perform pg_upgrade

Describe the bug
Fail of pg_upgrade
How are you accessing AGE (Command line, driver, etc.)?
Accessing AGE through command line.
What data setup do we need to do?
Apache AGE (master branch) with PostgreSQL (REL_15_STABLE).
What is the necessary configuration info needed?
./configure CFLAGS=" -Og" --enable-tap-tests --enable-debug --enable-cassert --prefix=/tmp/pgsql

What is the command that caused the error?
PGDATA=/tmp/pgsql/data

killall postgres
rm -rf $PGDATA

initdb -U postgres -k -D "$PGDATA"
pg_ctl -D "$PGDATA" -l `pwd`/pgsql.log start

createdb -U postgres test
psql -U postgres -dtest -c "CREATE EXTENSION age"

pg_ctl -D "$PGDATA" -l `pwd`/pgsql.log stop

pg_upgrade -U postgres -d /tmp/pgsql/data -D /tmp/pgsql/data -b /tmp/pgsql/bin/ -B /tmp/pgsql/bin

Result
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 fatal

Your installation contains one of the reg* data types in user tables.
These data types reference system OIDs that are not preserved by
pg_upgrade, so this cluster cannot currently be upgraded.  You can
drop the problem columns and restart the upgrade.
A list of the problem columns is in the file:
    /tmp/pgsql/data/pg_upgrade_output.d/20231127T141248.693/tables_using_reg.txt

Failure, exiting

Expected behavior
Successfull pg_upgrade
Best regards,
Egor Chindyaskin
Postgres Professional: http://postgrespro.com/

@saygoodbyye What are you trying to upgrade from?

@jrgemignani Hello! You can get the presented error when upgrading from any version to any, my message contains only the fastest reproduction of the problem

@jrgemignani
As documentation says "pg_upgrade allows data stored in PostgreSQL data files to be upgraded to a later PostgreSQL major version without the data dump/restore typically required for major version upgrades."
To demonstrate a typical situation when upgrading from one major edition of PostgreSQL to another, I wrote the following scripts.
The essence of the work is that we install PostgreSQL 14 along with Age (PG14 branch) and then run data.sql to add some AGE graphs.
start14.sh:
killall postgres

export PGDATA=/tmp/REL_14_STABLE/data
export PG=/tmp/REL_14_STABLE
export PATH=$PG/bin:$PATH

rm -rf $PG

cd postgres

make distclean >/dev/null
git clean -dfx >/dev/null
git checkout REL_14_STABLE

./configure --enable-cassert --enable-tap-tests --enable-debug --prefix=$PG --quiet
make -j$(nproc) -s install  && make -j$(nproc) -s -C contrib install

cd age

make distclean >/dev/null
git clean -dfx >/dev/null
git checkout PG14

make -j$(nproc) install

initdb -D $PGDATA
pg_ctl -l logfile start
createdb $USER
psql -d $USER -f data.sql
pg_ctl stop

data.sql:
CREATE EXTENSION age;
SET search_path TO ag_catalog;

SELECT * FROM create_graph('ag_graph_1');
SELECT * FROM create_graph('ag_graph_2');
SELECT * FROM create_graph('ag_graph_3');

SELECT create_vlabel('agload_test_graph1','Country1');
SELECT create_vlabel('agload_test_graph2','Country2');

SELECT * FROM create_complete_graph('gp1',5,'edges','vertices');
SELECT * FROM create_complete_graph('gp2',5,'edges');

The second script installs PostgreSQL 15 along with Age (PG15 branch).
start15.sh:
export PG_old=/tmp/REL_14_STABLE
export PG=/tmp/REL_15_STABLE
export PATH=$PG/bin:$PATH
export PGDATA_old=/tmp/REL_14_STABLE/data
export PGDATA=/tmp/REL_15_STABLE/data

rm -rf $PG

cd postgres

make distclean >/dev/null
git clean -dfx >/dev/null
git checkout REL_15_STABLE

./configure --enable-cassert --enable-tap-tests --enable-debug --prefix=$PG --quiet
make -j$(nproc) -s install  && make -j$(nproc) -s -C contrib install

cd age

make distclean >/dev/null
git clean -dfx >/dev/null
git checkout PG15

make -j$(nproc) install

initdb -D $PGDATA
pg_upgrade -d $PGDATA_old -D $PGDATA -b $PG_old/bin -B $PG/bin

Next, we run pg_upgrade from version 14 to version 15, but we get an error when upgrading. This is wrong behavior.
Upgrade results:
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 fatal

Your installation contains one of the reg* data types in user tables.
These data types reference system OIDs that are not preserved by
pg_upgrade, so this cluster cannot currently be upgraded.  You can
drop the problem columns and restart the upgrade.
A list of the problem columns is in the file:
    /tmp/REL_15_STABLE/data/pg_upgrade_output.d/20231207T124124.859/tables_using_reg.txt

Failure, exiting

Upgrade log:
In database: test
  ag_catalog.ag_graph_namespace_index.namespace
  ag_catalog.ag_graph.namespace

@saygoodbyye

As documentation says "pg_upgrade allows data stored in PostgreSQL data files to be upgraded to a later PostgreSQL major version without the data dump/restore typically required for major version upgrades."

pg_upgrade is only for data and it even states that not all data is preserved -

Your installation contains one of the reg* data types in user tables.
These data types reference system OIDs that are not preserved by
pg_upgrade, so this cluster cannot currently be upgraded.

AGE is tightly integrated with PG and a lot of the internal structures are based off of initial system settings and OIDs. These settings can change and be different from system to system. As you are using pg_upgrade, I would carefully review the documentation page for it, keeping this in mind.
Currently, we only support upgrades within a version of PG. However, we could add this as something to look into, resources permitting, in the future.

@jrgemignani
Thanks for the clarification, but it turns out that there is no way for people with a petabyte of graphs to upgrade to a major release?

@saygoodbyye

Currently, we only support upgrades within a version of PG. However, we could add this as something to look into, resources permitting, in the future.

Our priorities, atm, are to fully support all current versions of PostgreSQL (nearly done) and the majority of the openCypher specification (nearly done). Afterwards, we will have more resources to focus on additional features and possibly cross version upgrades.

This is important issue for us too. is pg dump and restore, the only way to upgrade  major version of postgres?
what is the status of this topic?

@sda399 Hi, The issue is still relevant as some customers with large databases may have a problem when upgrading to a major version.


Allow EditorTranslationParser to add POT comments

This adds an optional fourth item to the msgids_context_plural array in EditorTranslationParserPlugin::parse_file that can be used to add comments to the generated POT file.
This means that anyone using the custom POT generation system can provide extra help for translators.

Bugsquad edit, closes: https://github.com/godotengine/godot-proposals/issues/8917

Please open a proposal to track the support and details of this feature 🙂

Please open a proposal to track the support and details of this feature 🙂

Done 👍  https://github.com/godotengine/godot-proposals/issues/8917

See also #98099.

Superseded by #98099. Thanks for the contribution!


first crack at a pipeline to download flow data and compute some metrics

This pipeline downloads daily flow data, as well as peak data and some site info needed for EflowStats, runs all the HIT metrics for each site and then compiles it back into one file.  Theres a lot of comments in the code where future functions will probably be needed or just notes to myself.  Also - I wasn't quite sure what to save as a file and what to keep as an object target, so other than the initial daily streamflow download, they are all objects but I can change them as needed. I'm sure you will find a lot of other stuff in there, I was just pretty psyched that it ran!

You can merge after you've made those edits


Covid Tracking Project Aggregation

This is a tracking issue for identifying aggregations and writing aggregation routines for CTP race data.

We definitely want to join with FIPS codes. We also (probably) want to join with ACS population data. There are some states and territories which don't report any race data (reports_race or reports_ethnicity == 0). In those cases, we should not compare to population.
The only aggregation we want to do is to the national level, which should be done for both CRDT and population (don't use total US population from ACS, use aggregate from state/territory pops).

We will export 4 CRDT tables:
covid_tracking_project_cases_std
covid_tracking_project_deaths_std
covid_tracking_project_hosp_std
covid_tracking_project_tests_std

Joining CTP data with ACS is the only thing left here, moving to backlog


Interactive Learning ouput stories file shows unicode code not character string

Rasa version:
Rasa 1.0.1
Python version:
Python 3.6.8
Operating system (windows, osx, ...):
Windows 10 Pro 1903
Issue:
Interactive Learning output stories file shows unicode code but not character string,
like I have submitted in #2499.
This is a story by Interactive Learning ouput:
## Generated Story -4851626544216716596
* morning_report
    - action_get_user_info
    - slot{"family_members": [{"\u8001\u5a46": 614}]}
    - slot{"account_name": "\u63a5\u73ed\u4eba"}
    - action_identification_ask
* exit
    - utter_exit_record
    - action_slot_reset
    - reset_slots

I have try this code:
a = "\u5317\u4eac"
print(a)

with open("test.md", "w", encoding="utf8") as file:
    file.write("\u5317\u4eac")

and I get "北京" with the print function and in "test.md" file. I don't konw why this happend.
thanks a lot.

And maybe it will influence the visualization of the stories.  All these unicode code like "\u5317\u4eac" will shows �������� in visualization output file graph.html, but I am not very sure, becasue in ./rasa/core/training/visualization.html some code not use "utf8", so the file encode will depended by system. like line 288:
def persist_graph(graph, output_file):
    """Plots the graph and persists it into a html file."""
    import networkx as nx

    expg = nx.nx_pydot.to_pydot(graph)

    with open(visualization_html_path(), "r") as file:
        template = file.read()

    # customize content of template by replacing tags
    template = template.replace("// { is-client }", "isClient = true", 1)
    template = template.replace(
        "// { graph-content }", "graph = `{}`".format(expg.to_string()), 1
    )

    with open(output_file, "w") as file:
        file.write(template)

Thanks for raising this issue, @erohmensing  will get back to you about it soon.

Hi @HatuneMiku. I'm sorry this doesn't seem to have been resolved after a long time asking for it. I've checked into it, here's somehting interesteing:
## Generated Story<PHONE_NUMBER>665157564
* 北京{"\u5317\u4eac": "\u5317\u4eac"}
    - 北京

This only seems to happen if the characters are inside entities or slots. My instict is that this is happening because of some sort of json.loads or json.dumps that isn't being encoded, since the regular output is being encoded. I'll look into a fix for you.

Thanks @erohmensing , If we are using json to save characters are inside entities or slots, maybe this can help:
import json
strings = "北京"
print(json.dumps(strings))
print(json.dumps(strings, ensure_ascii=False))

the result are:
"\u5317\u4eac"
"北京"

Thanks, I actually found that already, I've fixed it locally. Glad we came to the same conclusion :) I'm working right now on the visualization. On my side, characters show up fine, are you still having issues with visualization?

It is better to use utf-8 everywhere just like the fix for #2539 in rasa_core, so maybe we should add encoding=utf-8 to the open function or IO function in visualization.


Add logo resources

Summary of changes
Adding logo assets as requested in issue 2227
https://github.com/pypa/setuptools/issues/2227
Closes 2227
Pull Request Checklist

[ ] Changes have tests
[ ] News fragment added in changelog.d. See documentation for details

Thanks again. I'm excited to present these assets in the project!

My pleasure to help a fellow open source software project. Best regards
from the Inkscape project! :)
-C
On Sat, 4 Jul 2020, 16:30 Jason R. Coombs<EMAIL_ADDRESS>wrote:

Thanks again. I'm excited to present these assets in the project!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pypa/setuptools/pull/2229#issuecomment-653779592, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ADB3HTZYQX5BA4RD4FQTB63RZ5DJJANCNFSM4OQB7GFA
.


Trim leading/trailing whitespace from attribution copy button result

Description

The text copied from Credit the creator section sometimes has newlines at the beginning (and some results will have more than others). Reported by @AetherUnbound.
Reproduction

Visit this image
Click the "copy" button for "Credit the creator" HTML or Plain text tab
Paste the content into a text editor and notice the leading whitespace

Screenshots

See video in Chrome browser.
Additional context

Confirmed is an issue in WordPress/openverse-frontend#1437.
Resolution

[ ] 🙋 I would be interested in resolving this bug.

I can try tackling this issue


feat(react-svg): add index.d.ts

@aleclarson since we're not getting anywhere with PRs on this repo, maybe it's time to fork it? I would be willing to take that on. I'd start be merging your PRs and updating some dependencies, but if you have any other ideas for cleanup/improvements I'd love to chat about that.


github.token path broken on MACOS

github.token path seems to be broken when executing script qt-release on MacOS :
When executing this command
sudo qt-release -v 0.4.0 -t 0.4.0 ./ressource/MBPMid2010-GPU-Fix-client_mac.ini

Getting this error :
loging in to GitHub...Traceback (most recent call last):
  File "/usr/bin/qt-release", line 230, in <module>
    release.run()
  File "/usr/bin/qt-release", line 223, in run
    self.loginToGitHub()
  File "/usr/bin/qt-release", line 150, in loginToGitHub
    with open(self.credentialsFile, 'r') as fd:
IOError: [Errno 2] No such file or directory: '/Users/Julian/.qt-release/github.token'

What can be done :

[ ] Add link to GitHub help to tell people how to generate token
[ ]  Import path to GitHub token from configuration file

My bad, closing this one, didn't see --authorize option


Added docs for knex pool config

Signed-off-by: sharadpattanshetti<EMAIL_ADDRESS>Hey, I just made a Pull Request!

:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[x] Added or updated documentation
[x] Tests for new functionality and regression tests for bug fixes
[x] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Hi @freben, created new PR with the changes to documentation on overriding the Knex Pool configuration.
Please review and approve.
FYI - @benjdlambert

Gonna link this to #10868 also, as I think really it's pretty unfortunate that users that run things in production have to have this config in the first place. I think that the root cause is that because of how the scaffolder works, it doesn't poll the DB and the queue, it waits for a ping and a promise resolution in process when a task is created. This method, although it works, probably should be moved over to ping the DB too like the catalog, but it's also brought to the surface an underlying issue that the when the connections get killed because they're idle, they don't reconnect unless there is a pool available. The pool config is a great suggestion and we should recommend it for production, but I think that we should also be sending some keepalive / heartbeat down the connections to make sure of two things. 1) that we are able to connect to the DB, otherwise we need to be alerting the user that either out network is down, or the DB is down. 2) that we keep the connection active and fresh so that it doesn't get to this timeout limit from the cloud providers and DB hosts.

Also just fixed up the prettier stuff that was missing to make this PR green ✅

Also just fixed up the prettier stuff that was missing to make this PR green ✅

Thanks @benjdlambert, sorry could not recognize extra spaces. Completely agree with above 2 points mentioned by you, along with that we also need validate the connection before passing it to catalog or scaffolder tasks.
The knex pool config that really worked for us was this :
database:
client: pg
connection:
database: backstage_plugin_catalog
host:
user:
port: 5432
password:
ssl:
rejectUnauthorized: false
ca:
knexConfig:
pool:
min: 0
max: 15
acquireTimeoutMillis: 60000
idleTimeoutMillis: 60000

Hey @sharadpattanshetti, congrats on your first merged PR 🎉 Nice work 👍 Please let me know if we can be helpful in any way.

Thank you @suuus, not much as of now. Thanks @benjdlambert , @freben and @Rugvip for the guidance.

Hi @sharadpattanshetti
Does statement_timeout setup work under database.connection in app-config.yaml file? I followed this https://node-postgres.com/api/client mentioned in docs/tutorials/switching-sqlite-postgres.md
p


Get code ready for additional protocols

Right now the sinkhole is built assuming HTTP only but people may want to add other protocols like SMTP, POP, IRC, etc.  The code should at least be ready for these additional protocols to be added without having to restructure too much code.

As part of the TLS additions this was taken care of (as of commit 39767c2b9eb25efee823f72ae33ee7778a058c6d).  Listening sockets now have some protocol information with them (currently only "http") that gets sent to the workers so the workers know what protocol to speak.  This mostly readies the code for additional protocols.

I'm reopening this because I realized that some protocols like SMTP expect the server to say something first before the client says anything.  The code we have currently expects the client to talk first.  We'll have to keep this in mind when we want to adapt the code to support another protocol like SMTP.

Splitting the workers into readers and workers made this a bit easier.  SMTP support was added with commit bebcb31573a61f153787965b6cec930586e50458


Maintainers: Add dyladan, remove danielkhan

As my responsibilities have changed and @dyladan is doing a great job in this SIG, I am nominating him to take over maintainership from me.
GitHub Username
@dyladan
Organization you are requesting membership in
opentelemetry-js
Requirements

[x] I have reviewed the community membership guidelines (https://github.com/open-telemetry/community/blob/master/community-membership.md)
[x] I have enabled 2FA on my GitHub account. See https://github.com/settings/security
[x] I have subscribed to the Gitter channel
[x] I am actively contributing to 1 or more OpenTelemetry subprojects
[x] I have two sponsors that meet the sponsor requirements listed in the community membership guidelines
[x] I have spoken to my sponsors ahead of this application, and they have agreed to sponsor my application

Responsibilities

Make and approve technical design decisions for the subproject.
Set technical direction and priorities for the subproject.
Define milestones and releases.
Mentor and guide approvers, reviewers, and contributors to the subproject.
Escalate reviewer and maintainer workflow concerns (i.e. responsiveness, availability, and general contributor community health) to the TC.
Ensure continued health of subproject:
Adequate test coverage to confidently release
Tests are passing reliably (i.e. not flaky) and are fixed when they fail
Ensure a healthy process for discussion and decision making is in place.
Work with other maintainers to maintain the project's overall health and success holistically.

Sponsors

@mayurkale22
@danielkhan

List of contributions to the OpenTelemetry project

PRs reviewed / authored
Authored
Reviewed

Issues responded to
Issue list

SIG projects I am involved with
opentelemetry-specification
opentelemetry-js

Codecov Report

Merging #607 into master will increase coverage by 1.39%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #607      +/-   ##
=========================================
+ Coverage    92.2%   93.6%   +1.39%
=========================================
  Files         183     178       -5
  Lines        9093    8892     -201
  Branches      798     785      -13
=========================================
- Hits         8384    8323      -61
+ Misses        709     569     -140

Impacted Files
Coverage Δ

...es/opentelemetry-node/src/instrumentation/utils.ts
90.47% <0%> (-9.53%)
:arrow_down:

packages/opentelemetry-plugin-mysql/src/utils.ts
90.9% <0%> (-4.55%)
:arrow_down:

...entelemetry-exporter-jaeger/test/transform.test.ts
100% <0%> (ø)
:arrow_up:

...opentelemetry-base/test/resources/resource.test.ts
100% <0%> (ø)
:arrow_up:

...ges/opentelemetry-exporter-zipkin/src/transform.ts
100% <0%> (ø)
:arrow_up:

...try-node/test/instrumentation/PluginLoader.test.ts
100% <0%> (ø)
:arrow_up:

.../opentelemetry-exporter-zipkin/test/zipkin.test.ts
100% <0%> (ø)
:arrow_up:

...pe-async-hooks/test/AsyncHooksScopeManager.test.ts
100% <0%> (ø)
:arrow_up:

.../opentelemetry-exporter-jaeger/test/jaeger.test.ts
100% <0%> (ø)
:arrow_up:

packages/opentelemetry-plugin-https/src/utils.ts
100% <0%> (ø)
:arrow_up:

... and 11 more


error on pod listing in JetBrains is not clear

When executing program with mirrord on JetBrains with invalid (kubernetes) context/config it shows "mirrord run cancelled" instead of more informative error.

pretty sure this was solved? @Razz4780 @eyalb181

This is what's shown when I use malformed config, I believe it was solved


Resume download from post number? Or other ways to get around error 22 using captions as filenames

I'm using captions as filename with the following command, useful when downloading paintings:
instaloader --filename-pattern="{caption:.255}" profile example
But this breaks when filenames uses disallowed characters for NTFS filenames I believe, and my workaround is to restart download a day or two before the creation date of the last image with --post-filter="date_utc <= datetime(2020, 6, 24)"
Is there a way to specify the post number to resume from? OR Is there a way to either ignore posts that throw error 22, or even replace the characters that break filename as was discussed here: https://www.gitmemory.com/issue/instaloader/instaloader/262/479178345

Apart of #262 you refered, you can download the posts with the default filename and use a separate script reading the caption from the txt-file, adjust the string to a valid filename and rename the post media.  Also, you can just combine instaloader as module with this step. The only thing you need to memorize is the last post you downloaded if you use the parameter --fast-update.

Thanks, not a bad idea using a separate script and could probably figure it out, only issue is that I have 0 experience with Python and very little with coding, do you have such a script perhaps?

No, I don't have such a script, it was just a theoretical idea to solve your problem.
If you google a bit, you can find the parts you need.
A function for cleaning the caption to use as filename f.e. is described here: https://gist.github.com/wassname/1393c4a57cfcbf03641dbc31886123b8
Reading caption txt file and renaming media can be done with core python functions.
Be careful, the file extension counts to the max length of the file, so the caption max length can be 251 chars. Another thing is the max length of path and file, I currently don't know the max value, afaik there was a change in recent Windows version.


Tidy custom resources

Why are these changes needed?

Background
Closes https://github.com/ray-project/kuberay/issues/167
Ray start has a resources option which allows users to specify custom resource capacities for a Ray node.
Custom resources serve as hints to the Ray scheduler and autoscaler: If a Ray node has custom resource {"BLAH": 10}, you can schedule 10 tasks decorated with @ray.remote(resources={"BLAH": 1}). Attempting to schedule an 11th such task will cause the autoscaler to bring up another Ray node to accommodate the request.
One way to use the resources field in the context of KubeRay is described in https://github.com/ray-project/kuberay/issues/167:

A use case for this might be to deploy a heterogenous cluster with multiple worker groups, where each worker group uses a different image packaged with different 3rd-party utilities. Some tasks that require specific utilities could then be marked as requiring such resource, and only execute on the workers that provide it.

Problem
The value of resources must be supplied as a json string, which makes it awkward to specify correctly in a yaml file:
rayStartParams:
   resources: "'{\"customRes\": 1, \"anotherOne\": 2}'"

Solution
To provide a more convenient interface, this PR introduces an optional map field rayCustomResources.
Before:
rayStartParams:
   resources: "'{\"customRes\": 1, \"anotherOne\": 2}'"

After:
rayCustomResources:
    customRes: 1
    anotherOne: 2

Properties of the change in this PR

Purely cosmetic. Simply allows specifying Ray custom resources as a map rather than as a carefully quoted/escaped JSON string.

We can consider deeper integration with K8s extended resources later.

Backwards-compatible -- specifying rayStartParams["resources"] will continue to work.

An upcoming Ray PR would update the autoscaler to read the newly introduced field when determining node resource capacities.
Note on CPU/GPU/memory
The resources option for ray start does not support the keys CPU, GPU, or memory. These fields are inferred from the container spec and can be overridden with the optional ray start parameters num-cpus, num-gpus, memory.
Attempting to supply keys CPU, GPU, or memory to resources will cause Ray start to fail with an error message telling the user not to do that.
I considered adding CRD validation to exclude these keys. However, Kubernetes does not support "patternProperties" in CRDs -- you can't use a regex to restrict the keys of a map/object. A potential workaround would be to use a list of key, value pairs and add validation for key. However, that would make the interface and the code more complicated.
We already do some validation of rayStartParams in the operator code.
TODO in a follow-up PR: Validate absence of CPU, GPU, memory keys in the same spot, after https://github.com/ray-project/kuberay/pull/334 is merged.
Autoscaler
A follow-up Ray PR would have the autoscaler consider the rayCustomResources field when determining resource capacities.
Related issue number

Closes https://github.com/ray-project/kuberay/issues/167
Checks

[ ] I've made sure the tests are passing.
Testing Strategy

[x] Unit tests
[ ] Manual tests
[ ] This PR is not tested :(

The change looks good to me. Please help rebase the master change

@DmitriGekhtman do we have a use-case where the RayCustomResource is not a physical HW on the node (like a TPU or NPU)?

@DmitriGekhtman do we have a use-case where the RayCustomResource is not a physical HW on the node (like a TPU or NPU)?

Yeah, this one https://github.com/ray-project/kuberay/issues/167.

Paraphrasing @pcmoritz from offline:
As we're a bit unsure where we're going with resource configuration, it may be best to just leave things as they are, and just document the format "'{\"customRes\": 1, \"anotherOne\": 2}'".
I think this probably makes sense.

@DmitriGekhtman do we have a use-case where the RayCustomResource is not a physical HW on the node (like a TPU or NPU)?

Basically, any time you have multiple worker groups and you want certain tasks to be scheduled on a specific worker group.
The reason for scheduling on a given node group may or may not be related to physical hardware.

Will close for now and consider reopening if there are further complaints about the format.

@akanso as a related item, we can consider automatically advertising K8s extended resources to Ray by inserting those into the relevant rayStartParam


Cross-msgs: Finalize design doc for cross-net messages

We are still missing the fine-grained design of cross-net messages. We decided to postpone this until we had at least the implementation of #14 and #15 so we have a better sense of the architecture and the interactions between the IPC Agent and eudico.

Closing for now, once we reach the next deadline we should considering documenting the current implementation, updating the design and planning next steps protocol design-wise.


Docs baby steps

closes #42
Not much to say here :) I wrote a bit of an explanation of how to use the CLI to access the 2 fundamental workflows with memote.

Codecov Report

Merging #126 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #126   +/-   ##
========================================
  Coverage    86.29%   86.29%
========================================
  Files            9        9
  Lines          416      416
  Branches        87       87
========================================
  Hits           359      359
  Misses          51       51
  Partials         6        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a767e3...036ab52. Read the comment docs.

Please add the Sphinx numpydoc extension to the conf.py since all our docstrings are in that format. And while you're at it maybe choose another Sphinx theme. The default one is getting quite old. Here's some inspiration: http://www.writethedocs.org/guide/tools/sphinx-themes/

Just to help visualise the changes, I went for the Sphinx Bootstrap Theme, using a modified 'Paper' theme from Bootswatch ontop of that.
The result looks like this:

It is worth noting that the autogenerated .rsts will probably need some more love in terms of formatting and streamlining. In addition, the actual pytest modules that make up the testing suite are not being automatically documented as of yet. This is primarily due to pytest conventions conflicting with sphinx implementation. In other words, sphinx cannot import pytest modules without init.py. This issue details the problems and discussions associated with that.


feat: bump to Python 3.12 and drop 3.8

As title says

Codecov Report

Merging #2052 (0f3f876) into main (e032eb9) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #2052      +/-   ##
==========================================
- Coverage   95.27%   95.23%   -0.04%
==========================================
  Files           3        3
  Lines         127      126       -1
==========================================
- Hits          121      120       -1
  Misses          6        6

Files
Coverage Δ

src/ansys/tools/repo_sync/__init__.py
100.00% <100.00%> (ø)


Remove expressionCode property from vectors, numerics and bools

make sure asVertex works fine

waiting for the new pipelines with this one ^^


Allow some boolean indexing operations to return correct shape

For some masking operations it's very easy to figure out the shape of the array:
import dask.array as da

a = da.arange(0, 10)

# shape=(10,):
m = da.ones(10, dtype=bool)
a[m]
Out[46]: dask.array<getitem, shape=(nan,), dtype=int32, chunksize=(nan,), chunktype=numpy.ndarray>

# shape=(10-1,):
m = da.ones(10, dtype=bool)
m[0] = False
a[m]
Out[48]: dask.array<getitem, shape=(nan,), dtype=int32, chunksize=(nan,), chunktype=numpy.ndarray>

# shape=(10-n,):
n = 2
m = da.ones(10, dtype=bool)
m[0:n] = False
a[m]
Out[48]: dask.array<getitem, shape=(nan,), dtype=int32, chunksize=(nan,), chunktype=numpy.ndarray>

Not having to do .compute_chunk_sizes() after these simple operations seems like a nice feature and will make duck typing with other 3rd party numpy-like packages easier.

I totally agree that if there is a simple way to deduce shape that is desirable. Especially if it doesn't add too much complexity to the codebase.

My intuition is that this would not be easy to do without adding complexity. In order to do it, you'd have to be able to inspect the graph of the argument to figure out that, semantically, the argument is a ones operation. Graphs, even HLGs, aren't currently designed for that sort of semantic introspection. You'd get into some gnarly stuff poking at the dsk attribute of the Blockwise object.
This is another good example case for https://github.com/dask/dask/issues/7933, which is all about making that sort of semantic introspection possible for optimizations much like this.

I think this (shortterm, high level expressions seems longlongterm) can be solved by under the hood by switching boolean arrays to integer indices.
Here's a proof of concept that fixes my examples, probably neeeds a little more thought on n-dimensions:
import dask.array as da

def simple_masker(size, x=None):
    if x is None:
        x = (0, 0)
    elif not isinstance(x, tuple):
        x = (x, x+1)

    i0 = da.arange(0, x[0], dtype=int)
    i1 = da.arange(x[1], size, dtype=int)
    return da.concatenate((i0, i1), axis=0)

a = da.arange(0, 10)

m = da.ones(10, dtype=bool)

# a[m].compute():
# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
a[simple_masker(a.size)]
# dask.array<slice_with_int_dask_array_aggregate, shape=(9,), dtype=int32, chunksize=(9,), chunktype=numpy.ndarray>
a[simple_masker(a.size)].compute()
# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

# m[0] = False
# a[m].compute()
# array([1, 2, 3, 4, 5, 6, 7, 8, 9])
a[simple_masker(a.size, 0)]
# dask.array<slice_with_int_dask_array_aggregate, shape=(9,), dtype=int32, chunksize=(9,), chunktype=numpy.ndarray>
a[simple_masker(a.size, 0)].compute()
# array([1, 2, 3, 4, 5, 6, 7, 8, 9])

n = 2
# m[0:n] = False
# a[m].compute()
# array([2, 3, 4, 5, 6, 7, 8, 9])
a[simple_masker(a.size, (0, n))]
# dask.array<slice_with_int_dask_array_aggregate, shape=(7,), dtype=int32, chunksize=(7,), chunktype=numpy.ndarray>
a[simple_masker(a.size, (0, n))].compute()
# array([2, 3, 4, 5, 6, 7, 8, 9])


virtualbox/eval-win10x64-enterprise and x86 timeout waiting for WinRM

Building these boxes is hanging for me because the Windows machine finishes installing but then waits at the login prompt for the vagrant user to logon so the post-install script (which install WinRM and enable the service) never run. From my reading of the docs (https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unattend/microsoft-windows-shell-setup-autologon-logoncount) LogonCount must be specified for AutoLogon to work in Autounattend.xml but that doesn't seem to be specified in any of the files. Adding a large count to eval-win10x64-enterprise/Autounattend.xml fix the issue and the vagrant user gets logged in after boot and the post scripts run. I'm not sure in which windows version this requirement was added but it would seem that at least for win10, this is required.

Actually the Autounattend.xml setting is not the issue. It seems that it is a bug with this build of Windows (1703.build-15063). There is a KB article about it here: https://support.microsoft.com/en-in/help/4022716/windows-10-update-kb4022716
And the bug is only coming up now because there are windows updates that have been released that require a reboot. So the first boot auto-login fine and gets to update.bat which installs some updates and reboots and then we hang because we never login again to run the zz script. The solution appears to be that before the update.bat script, we need to re-enable auto-logon via setting some registry keys.

Actually, I think this is a duplicate of #119
For reference, https://support.microsoft.com/en-us/help/324737/how-to-turn-on-automatic-logon-in-windows provides steps on how to enable auto-logon via registry settings so someone just needs to code that up and have it run before update.bat


Question: Projection layer with video

Hi @cabanier
Context
I am working on Pixel Streaming in Unreal Engine and we are rendering a left-right stereo backbuffer from UE, streaming it over WebRTC as a video, then displaying that video in a VR headset using WebXR layers. We also stream the HMD transform back to UE to drive the mock HMD on the UE side. Giving us a full VR experience rendered on a remote machine.
Anyway, currently we are displaying this video using the quad layer. We then orient the quad layer to match the user's HMD and take up their full field of view. However, it seems to me this is a more suitable job for a projection layer; however, they do not take video as input.
We can of course render that video into the WebGL texture and copy that texture into the projection layer; however, I am doubtful that we will enjoy the same performance/quality benefits offered by using the media layers?
Question
Do I have any better options than what I am currently doing?
Regards, Luke.

A projection layer is basically a quad layer that fills the entire FOV and has the viewer space.
Your current approach is totally fine :-)

Thanks Rik.
Out of interest, is this assumption of mine true:

We can of course render that video into the WebGL texture and copy that texture into the projection layer; however, I am doubtful that we will enjoy the same performance/quality benefits offered by using the media layers?

Thanks Rik.
Out of interest, is this assumption of mine true:

We can of course render that video into the WebGL texture and copy that texture into the projection layer; however, I am doubtful that we will enjoy the same performance/quality benefits offered by using the media layers?

Yes, we play a couple of tricks internally that you don't have access to through WebGL. Cross-origin video will also work with media layers.


In Auto mode, "Once Copied" marking in real time

emoacht, how are you?
I ask a very trivial and detailed question.
In Check&Copy mode...
When I delete the files already transferred and click Check&Copy,
Once Copied indication is displayed in real time.
But in Auto mode...
When I delete the files already transferred, Once Copied indication does not appear in real time.
"Once Copied" appears only when I manually press the Check (or Check&Copy) button after Stopping the Auto menu.
Even in Auto mode, can Once Copied marking be displayed in real time after deleting the transferred files?
Without pressing the Check&Copy(or Check) button...
See the video below
https://youtu.be/KkbE6V4O2tU

I have been good. Thank you.
As for your question, it is probably because there is no new photo taken by the camera during the period. In auto mode, Snowy checks whether any change in the FlashAir and if no, it will return to waiting time. It is a key function to suppress unnecessary access to FlashAir in auto mode.

Thank you for answer.
It is to prevent unnecessarily excessive access to card in auto mode.
Only when new files are found on the card, then Auto mode starts checking card and copy new files.
Did I understand well?

I know Toshiba discontinued FlashAir in early 2019.
https://www.mouser.com/PCN/Toshiba_Memory_America_Inc_EOL_Notification_FlashAir_Jan_15_2019.pdf
So I don't have to wait for the W-05 generation anymore.
Now I have a powerful app like Snowy Plus, but I can't get FlashAir anymore.
So too sad.

Still there are updates
https://personal.kioxia.com/en-emea/software/flashair-configuration.html
https://personal.kioxia.com/en-emea/news.1.html

@dentphoto

Only when new files are found on the card, then Auto mode starts checking card and copy new files.

Your understanding is correct except checking will be triggered only when any write action has been done in FlashAir card.
I copied FlashAir product availability matter to another issue.


Query to edit subclient properties using commvault python SDK

Hi,
It is not an issue but a query as I didn't find any other mode for my query to get resolved. So raised issue for the same.
My query is through commvault python SDK is it possible to edit  properties of storage policy or subcient?
Example: I don't want to use deduplication,encrytion and compression which by default is enabled while creating the subclient.
So is there any way from sdk to disable those or edit the properties of subclient,storage policies,disk lib etc.

Hi Anu,
You can reach out to us anytime using GitHub Issues or at our mailing address<EMAIL_ADDRESS>Coming to your query, we generally write getters and setters for updating the properties of a subclient.
Example:
Get Storage Policy of a subclient:
subclient.storage_policy
Set Storage Policy of a subclient:
subclient.storage_policy = "new_policy"
Please share a list of properties that you wish to update, and we'll see how best they can be added.
You can call the internal method _set_subclient_properties, if its an immediate requirement.

Hi Jatin,
I want to disable deduplication,software compression and encryption which are the properties of subclient.
Below is traverse path:
1.Disable the deduplication option which is under storage device the properties of already created subclient.
2.Disable Software compression under data transfer option of subclient.
3.Mark encryption to none which is under advance option of subclient.
I also checked the _set_subclient_properties method having two arguments attr_name, value.
Now that I want to disable deduplication what values should I set to attr_name and value.

Hi Anu,
I'll get back to you with the details shortly.

Hi Jatin,
Any update.

Thanks Jatin for adding getters and setters for the properties that I mentioned.

Hi Anu,
The code has been pushed to the dev branch.
Please let us know if you see any issues.

Hi Jatin,
I am able to change value for generateSignature but for disabling deduplication it is throwing an error.

subclient.deduplication_options
'ON_MEDIA_AGENT'

subclient._set_subclient_properties("_commonProperties['storageDevice']['deDuplicationOptions']['generateSignature']", 'ON_CLIENT')
subclient = backupset.subclients.get('nfssc1')
subclient.deduplication_options
'ON_CLIENT'

subclient._set_subclient_properties("_commonProperties['storageDevice']['deDuplicationOptions']['enableDeduplication']", 'FALSE')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.6/site-packages/cvpysdk-1.0.0-py3.6.egg/cvpysdk/subclient.py", line 655, in _set_subclient_properties
output = self._process_update_response(flag, response)
File "/usr/lib/python3.6/site-packages/cvpysdk-1.0.0-py3.6.egg/cvpysdk/subclient.py", line 740, in _process_update_response
raise SDKException('Response', '101', response_string)
cvpysdk.exception.SDKException: Response was not success
}errorMessage":"Service error","errorCode":102

Hi Anu,
Can you please try from the public getters and setters?
disable deduplication
subclient.deduplication_options = (False, None)
enable deduplication
subclient.deduplication_options = (True, {"ON_CLIENT" / "ON_MEDIA_AGENT"})
https://github.com/CommvaultEngg/cvpysdk/blob/dev/cvpysdk/subclient.py#L2041

Hi Jatin,
Below is the output

subclient.software_compression
'OFF'
subclient.software_compression = ('USE_STORAGE_POLICY_SETTINGS')
subclient = backupset.subclients.get('nfssc1')
subclient.software_compression
'USE_STORAGE_POLICY_SETTINGS'
subclient = backupset.subclients.get('nfssc1')
subclient.deduplication_options
'ON_CLIENT'
subclient.deduplication_options = (False, None)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.6/site-packages/cvpysdk-1.0.0-py3.6.egg/cvpysdk/subclient.py", line 1623, in deduplication_options
"enableDeduplication": enable_dedupe,
File "/usr/lib/python3.6/site-packages/cvpysdk-1.0.0-py3.6.egg/cvpysdk/subclient.py", line 655, in _set_subclient_properties
output = self._process_update_response(flag, response)
File "/usr/lib/python3.6/site-packages/cvpysdk-1.0.0-py3.6.egg/cvpysdk/subclient.py", line 740, in _process_update_response
raise SDKException('Response', '101', response_string)
cvpysdk.exception.SDKException: Response was not success
}errorMessage":"Service error","errorCode":102
subclient = backupset.subclients.get('nfssc1')
subclient.deduplication_options
'ON_CLIENT'

Hi Jatin,
It is working after uninstalling  and installing it again.

subclient.deduplication_options
'ON_CLIENT'
subclient.deduplication_options = (False, None)
subclient = backupset.subclients.get('nfssc1')
subclient.deduplication_options
False

Thanks for the help

Hi Anu,
There was a small issue that I missed in the older fix.
I've made the fix and pushed the updated subclient.py file to the dev branch.
Please download that file, and you can replace it directly under:
%PYTHON_FOLDER%\Lib\site-packages\cvpysdk-11.13-py3.6.egg\cvpysdk\
Please let us know if you see any more issues.
Closing the issue.


Basic examples fail to build

When I try to build the basic example, it gives me this error:
src/main.rs:19:26: 19:29 error: macro undefined: 'dsl!'
src/main.rs:19     let api = Api::build(dsl!(|api| {

It appears the dsl_macros crate needs to be included... Though after including it, I get:
error: unresolved import `rustc::plugin`. There is no `plugin` in `rustc` ... dsl_macros-0.1.2/src/lib.rs:8 use rustc::plugin;

I wasn't able to get other examples to work, either, using rustc 1.5.0, beta, or nightly.
Also note the example on the front page of rustless.com has extern crate rustless; twice. When building it I get error: box expression syntax is experimental; you can call Box::new instead. (see issue #27779), and after fixing:
error: The attribute `phase` is currently unknown to the compiler and may have meaning added to it in the future (see issue #29642)
src/main.rs:3 #[phase(plugin)]

This is what I have managed to do so that it compiles and starts, but I could touch any route with curl, they all give me 404
https://gist.github.com/aoprisan/1e3b40f18bc92c779698900bd5df7854
Cargo.toml looks like this:
[dependencies.rustless]
git = "https://github.com/rustless/rustless"

[dependencies]
rustc-serialize = "0.3.19"
hyper = "0.8.1"
iron = "0.3.0"
valico = "0.8.2"

Curls I have tried:
curl -H Accept:application/vnd.chat.v1+json http://localhost:4000/chats/1/users/2 -v
curl -H Accept:application/vnd.chat.v1+json http://localhost:4000/chats -v

Gist content:
extern crate rustless;
extern crate hyper;
extern crate iron;
extern crate rustc_serialize;
extern crate valico;

use hyper::status::StatusCode;
use iron::Iron;
use rustless::{
    Application, Api, Nesting, Versioning
};
use rustc_serialize::json::ToJson;
use valico::json_dsl;

fn main() {

    let api = Api::build(|api| {
        // Specify API version
        api.version("v1", Versioning::AcceptHeader("chat"));
        api.prefix("api");

        // Create API for chats
        api.mount(Api::build(|chats_api| {

            chats_api.after(|client, _params| {
                client.set_status(StatusCode::NotFound);
                Ok(())
            });

            // Add namespace
            chats_api.namespace("chats/:id", |chat_ns| {

                // Valico settings for this namespace
                chat_ns.params(|params| {
                    params.req_typed("id", json_dsl::u64())
                });

                // Create endpoint for POST /chats/:id/users/:user_id
                chat_ns.post("users/:user_id", |endpoint| {

                    // Add description
                    endpoint.desc("Update user");

                    // Valico settings for endpoint params
                    endpoint.params(|params| {
                        params.req_typed("user_id", json_dsl::u64());
                        params.req_typed("name", json_dsl::string())
                    });

                    endpoint.handle(|client, params| {
                        client.json(&params.to_json())
                    })
                });

            });
        }));
    });

    let app = Application::new(api);

    Iron::new(app).http("localhost:4000").unwrap();
    println!("On 4000");

    println!("Rustless server started!");
}

@aoprisan
I think you should add /api prefix to your curl because of api.prefix("api"); setting.

You mean this:
curl -H Accept:application/vnd.chat.v1+json http://localhost:4000/api/chats/1/users/2 -v
? I did that, to no avail.
Here is a minimal project: https://github.com/aoprisan/try-rust

@aoprisan ok, thanks, I'll take a look

@aoprisan everything works as expected in your example. The issue here is that you're trying to receive answer via GET request, but your routes are configured to use POST.
Correct request
curl -X POST -H Accept:application/vnd.chat.v1+json http://localhost:4000/api/chats/1/users/2\?name\=name

Answer:
{"id":1,"name":"name","user_id":2}

Silly me :), I can't believe I missed that. I didn't read the code properly, I was expecting it to be a GET. Thanks a lot for your effort. I saw that the example is updated as well :). I guess the issue can be closed.