feat: add restore cmd, improve backup

Changes:

feat: Add restore command to cli, as a counterpart of backup.
feat: add optional verbosity switch to backup output
feat: timing and download rate info in verbose backup
fix: truncated schema in backup
chore: comments added to functions in backup
chore: add some util tests

❯ ./tigris backup -d /tmp/benchmark -v
 [i] using timeout 3600
 [i] ensuring backup dir exists /tmp/benchmark
 [*] /tmp/benchmark/tigris_starter_ts1.orders.backup
 [>] 0.0 MB (0.0 MB/s in 0.1s)
 [*] /tmp/benchmark/tigris_starter_ts1.products.backup
 [>] 0.0 MB (0.0 MB/s in 0.1s)
 [*] /tmp/benchmark/tigris_starter_ts1.users.backup
 [>] 0.0 MB (0.0 MB/s in 0.1s)
 [*] /tmp/benchmark/eshop.users.backup
 [>] 0.0 MB (0.0 MB/s in 0.5s)
 [*] /tmp/benchmark/eshop.orders.backup
 [>] 0.0 MB (0.0 MB/s in 0.2s)
 [*] /tmp/benchmark/eshop.products.backup
 [>] 0.0 MB (0.0 MB/s in 0.5s)
 [*] /tmp/benchmark/meteors.landings.backup
 [>] 8.0 MB (1.4 MB/s in 5.9s)
 ...

Restore:
❯ ./tigris drop database db1_backup && make && ./tigris restore -d /tmp/benchmark -P backup -D db1
CGO_ENABLED=0 go build -tags=release -ldflags "-w -extldflags '-static' -X 'github.com/tigrisdata/tigris-cli/util.Version=v1.0.0-alpha.7-81-g4466c02'" -o tigris  .
[i] using timeout 3600
[*] db1
[.] db1 => ffr
[.] db1 => sorted_collection
[.] db1 => test_nested_objects_arrays
[.] db1 => u2
[.] db1 => u3
[.] db1 => user1

...

❯ ./tigris read db1_backup u2
{"strName":"121","id":1}
{"strName":"fr","id":2}

In the next iterations we would need to add backup validation.
File checksums and backup completeness mark.

@efirs how should I test this?

Can we just extend backup test?


Automated Splunk TA Update 27

This PR contains updates to Splunk TAs made by GitHub Actions workflow.

Looks good!


For This Project Long Time Existing, Please Use Patch Files Instead Sources From E M B.

For This Project Long Time Existing, Please Use Patch Files Instead Sources From E M B.
This is a great project, I don't want to see emb enforce close this.

It's was a mistake in the commit process, i will see how to do with patch file ! thanks


Allow admins to access bucket policies

This will help debug S3 access issues by confirming that the policy is formatted as expected.

@zaro0508: I broadened the admin permissions to s3:* so we don't need to add more permissions later. If you approve this, then feel free to merge and create a tag called v1.6.1.

@BrunoGrandePhD i have already approved.  You and @clarisse-lau are already maintainers of this repo so you have the power to do whatever you need.


Custom css var does not work

I want to use my custom css var like color: var(--test-var);, so I use color-$test-var and test in playground, it's works. but in my project I use it, it doesn't work.

But when I modified it to color="$test-var", it's working.
I'm not sure if it's a problem with the unocss configuration, here is my configuration:
// unocss.config.ts
import { defineConfig, presetAttributify, presetIcons, presetUno } from 'unocss'

export default defineConfig({
  presets: [
    presetUno(),
    presetAttributify(),
    presetIcons({
      extraProperties: {
        'display': 'inline-block',
        'vertical-align': 'middle',
      },
    }),
  ],
})

That is because color-$test-var is not a valid attribute name, so you better to void using this name in Attributify mode, use class="color-$test-var" instead.

This is the limitation of HTML spec. As mentioned, you need to workaround with it.


3.0.3 Compilation problems

crypto/bn/bn_dh.c:34:5: error: initializer element is not constant
crypto/bn/bn_dh.c:34:5: error: (near initialization for 'modp_1536_p[0]')
Error when compiling 3.0.3, 1.X version does not have this problem，The compiler used is the ARM cross compiler

ca2bf555cd64bc9624af1259ce3cd27f95a5763e is the culprit commit.

Except this is a clear bug of the compiler - the initializer is a constant expression.

I've seen a few (ancient) compilers didn't like constant expressions.
The C89 standard says it is allowed in 6.5.7 but doesn't admit it in foot note 54 in 6.4 which defines constant expressions.  It doesn't appear to mandate that constant expressions are evaluated at compiler time (they can be).
C99 mandates that they are evaluated at compile time.

I think the answer to the original question is your compiler is at fault.

I think the reason is that the compiler I'm using doesn't recognize this form of initialization
static const BN_ULONG modp_1536_p[] = {
BN_DEF(0xFFFFFFFF, 0xFFFFFFFF), BN_DEF(0xCA237327, 0xF1746C08),

Yes, and that's a serious deficiency of the compiler as it is just a preprocessor macro expanding to a simple constant expression (BN_ULONG)0xF1746C08 << 32 | 0xCA237327


Twitch integration

There are 2 goals for this:

[x] Twitch notifications when someone goes live.
[ ] VOD timestamps for score feed.

TODO: figure out how to handle orphan TwitchBroadcasters


Developing Pointdevice Mode for all TH danmaku games

Hi all, I am a CS student who is developing a fork of thprac that implements a Pointdevice Mode that supports all TH danmaku games (probably except th9 and th19). Some basic functionalities are already done like detecting loss of life and overwriting section in practice mode. But there are some obstacles.
Two biggest challenge: (1) Is there a function that can immediately reload the current stage? (2) How to get the current section number?
(Edit: the second is solved by Cheat Engine. For example, 4e73f0/4e73f8 is the stage/section in th15.)
Forgive my limited knowledge of reverse engineering...

Here is something I tried but doesn't work well:

When section or stage is changed, store it in global variables
After player chooses "Restart game" (the option that restarts from the first stage), invoke a modified version of THSectionPatch() (the function that warps to a section in practice mode), which sets the section and stage

What will happen:

If this is stage 0 in Story mode, it works fine most of the time
If this is not stage 0 in Story mode, the game may crash, which makes sense since "Restart game" loads the first stage while I tell the game to jump to a section in a later stage
It works fine in Practice mode regardless of the stage, which also makes sense because "Restart game" loads the current stage in Practice mode
Sometimes boss' AI can be messed up, stopping attacking player until the next section, which is a minor issue compared to previous ones

I also don't understand what are ECLJump and ECLSetChapter. I just invoke THPatch() with modified parameters, and sometimes it works sometimes it doesn't.

Any help is greatly appreciated! I hope this mode will bring joy to all of us soon.

First, I would recommend that you check out https://github.com/exphp-share/th-re-data. Now, to answer your questions.
Is there a function that can immediately reload the current stage?
Yes... sort of.
In Touhou there is a variable called the game mode. Well, it's three actually (the current gamemode, the next gamemode (which is the variable you actually want to modify if you want to change it), and the previous gamemode). Examples of gamemodes are:

4: main menu

7: in game

10: reload the game. This does the same thing as choosing retry in the pause menu

15: ending
But also:

14: reload the current stage

So setting the gamemode to 14 (or 0xE) does exactly what you want.
I just invoke THPatch() with modified parameters, and sometimes it works sometimes it doesn't + Sometimes boss' AI can be messed up
This is because reloading does not reload the ECL scripts. The game keeps using the scripts it had already loaded previously.
thprac also handles warping to a chapter within the stage vs warping to a boss with different functions. THStageWarp is used to warp to stage sections, not THPatch. The function THSectionPatch knows which one to call. The reason for that is that thprac didn't originally support stage section warping. As a result, the feature is very hacked on.
Of course, if you perform a reset in a stage that's not 1, the game will reload the script for stage 1. However, setting the gamemode to 14 will never reload the script.
ECLJump
This is a utility function that creates a jump instruction in the ECL script to skip over a few other instructions. This does turn the data immediately after the newly created jump into garbage, but the ECL parser will never try to run that since it will jump over that due to the jump instruction.
ECLSetChapter
Because making sure that the internal chapter number is set correctly matters a lot in th15, thprac will also do that. It wants to make sure though that it sets the chapter number at the correct time though. The code Because of that, instead of just doing it directly, it will activate a hook that will set the chapter number and then deactivate itself. The thXX_patch_main hook is at the end of GameThread::thread_start

Hi 32th-System,
Thank you so much for the detailed explanation! I have been checking out th-re-data (astonishing work!) and implementing PointDevice mode on TH15's EX stage.
By setting gamemode_next to 14 and invoking THPatch() and THWarpStage() with tweaked parameters that represents the current section of player, I am glad to say now it kind of works! I am indeed able to play through EX stage and reset to the checkpoint when about to lose life.
However, my current implementation has some bugs:

The BGM always replays, and is always the first BGM "A World of Nightmares Never Seen Before", even if I use Everlasting BGM. I think I can solve this one by myself with the help of the existing ElBgmTest() function
Seems that simply invoking THPatch() with tweaked parameters doesn't work well sometimes. Reseting during any Non Spellcard will actually reset the game to the previous Spellcard, and Doremi insists to come out and cast her third Spellcard if player resets at warp section 6~9, LOL

About the second problem, do you think it's because the ECL script manipulation instructions in THPatch() only works for jumping to a section with Practice mode, and doesn't work for cleaning the game's current script and jumping to a section with  gamemode_next = 14?
If so, it would be more tricky than I thought since I don't know how you did those fantastic job with ECL scripts. Is there a tutorial for this? Maybe I can force player to exit to practice mode and start practice with parameters that will jump to the current section, but this would be a bad design when I develop the PointDevice mode for other TH story modes.


[feat] Support for dragging files from Tauri window to filesystem

Describe the problem
I'm implementing a file manager application, but I'm not sure if Tauri supports dragging files from the Tauri window to outside the window, e.g. onto File Explorer or the desktop?
I know Tauri currently supports dropping files onto the Tauri window, but I'm looking for the opposite - dropping files from Tauri onto the user system.
Describe the solution you'd like
Implement dragging and dropping from the Tauri window onto the user system.
Alternatives considered
No response
Additional context
No response

Thanks for the request! This is currently tracked in https://github.com/tauri-apps/tauri/issues/2593 :)


Show a selected Bone Gizmo when Clicking on it keyframe

Tested versions
Godot v4.3.stable
System information
Ubuntu 24.04 LTS 24.04 - Wayland - GLES3 (Compatibility) - Mesa Intel(R) HD Graphics 2000 (SNB GT1) - Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (4 Threads)
Issue description
Screencast from 2024-08-23 16-28-05.webm
Animating inside godot is pretty annoying after the Skeleton3D lose focus:
1 - it disables edit mode
2 - Clicking on a specific keyframe belong to a Bone rotation ... will do nothing (even if the Skeleton3d is on edit mode)
Steps to reproduce

clicking on a keyframe should show a Gizmo for the selected bone
currently, it does nothing
Minimal reproduction project (MRP)
N/A

Thank you for your report
This is for reporting bugs, not proposing new features, by your description this isn't a bug but a missing feature, so please open a proposal here instead unless this used to work like this previously

I can take a look at adding this feature!

https://github.com/godotengine/godot/pull/99696

https://github.com/godotengine/godot/pull/99696


Implement a solution for separators

In the current implementation, buttons can be dangerously close together and you may accidentally run a task by accident if moving too quickly.
Add a solution to allow for additional spacing or have an object type that would inject a "|" at minimum if desired.

I found playing with the priority argument in createStatusBarItem() that you can re-order the items you add.
Given a low/high enough priority it ends up "between" other statusBarItems of other extensions.
I think whatever is done to enhance this it needs to be within the status bar item.
We could add a config option for a kind of "Prefix-Postfix" that is applied for every button.
But on the contrary, is it that much better than just adding it manually?


Optimize AudioServer::_driver_process()

Move expensive calculations outside inner hot loops.
Forward-ported from 3.6: https://github.com/godotengine/godot/pull/63430. This func has been unchanged since Godot 3.x.

@akien-mga: style is fixed

Thanks!


Fancy national pie animation

Wider fig below main fig that shows nations total water use as a number & put that into containers for each category. Illustrates the idea of water use distribution across categories. One simple transitions that morphs glass of water into pie slice that fits into over national pie chart. Scroll-based trigger. Help people understand the pie charts.

Can someone weigh in on whether we have a simple method for view-area-based function calls? My experience w/ these was in the older vizzies that used parallax, and that was pretty challenging. But, I know Walker implemented some GA events that were based on whether or not you looked at part of a chapter for longer than x seconds, so it seems there may be simpler options.

I have nothing on this. Maybe @wdwatkins or @ldecicco-USGS knows?

Me neither

assuming yes, it would be js?
Here is what I think walker did: https://github.com/USGS-VIZLAB/climate-fish-habitat/blob/master/target/js/app.js#L42


Support mapping cities -> counties

When a user searches for a popular city, we need to be able to show data for the county the city is in.
This is just an engineering task and isn't connected to any UI in particular, but just for illustration purposes:

Example way that this could be used: when the user searches for "Seattle", the King County suggestion should be shown.

amazing - though the screenshot doesn't show it off!

@szhu is their a PR/Issue for adding search?

@crwilcox Not as far as I know. Should I add it?
(I didn't add it yet because I don't know what's the correct threshold for adding feature issues here, and also I didn't want someone to start building it before I got confirmation that it was a good idea.)

Found a dataset we can use for city -> county mapping:
https://simplemaps.com/data/us-cities
Note: they require a linkback to this url.
I looked at the dataset and it has everything we need, including columns for:

City/County/State
Some cities (like Portland) are in multiple counties -- it has this data as well
Lat/Long (for showing on map + getting closest city to user)
Population (for sorting suggestions)

I was curious about the feature before city -> county mapping. This shows a search by county name. Has that been tracked/pr'd?


Possible (total) loss of crate ownership when revoking GitHub access rights.

I maintain a couple of crates and I almost lost the ownership of them. What I believe happened is that I revoked crates.io's priviledges in my Github settings by mistake. When I tried to login a few minutes ago, it created a new account, with new tokens and everything (I had to use cargo login again because I just re-installed my OS). There also are no crates listed in my dashboard. Anyway, I tried to publish with the new token and it failed. Luckily, I still had the token from my old account on another computer and I was able to publish using that one.
So my crates are kind of in a weird state now, because the crate's owner is a revoked account which I can still publish with, and the new account is not an owner of the crate (or at least can't publish to them).
If I try to add myself with using cargo owner --add SBSTP and the old token, I get an error saying that SBSTP is already an owner.

Did your github login change in the meantime? In theory we should update that row even if our token is revoked...

Oh yes! I also renamed from SBSTP to sbstp.

Aha! In that case I'm gonna close this in favor of https://github.com/rust-lang/crates.io/issues/136 as I think it's covered by that.
If you need any help getting access to your previous crates, though, please just let me know!

Should I keep using the old token?

You probably want to use your new token, so I can just transfer all the old crates to your new user instead if you like?

I managed to re-gain access. I'm there twice though, idk if this is a problem https://crates.io/crates/igd. Maybe just remove the old owner now? I have two crates, igd and loirc.

Ah yeah if you just remove the old owner it should be fine, but if you run into more problems please just let me know!

Everything is fixed now, thanks again.


Error during build

Hi,
I am somewhat of a noob with this and i was trying to run both react-calendar and react-angular calendar.
I was able to build react-angular-calendar but i get the following error when i load it on Sharepoint application library
Deployment failed.  Correlation ID: 860e449f-d0d6-b000-1c55-627e73942935
For react-calendar, i was unable to build the file and get the following errors:
**Build target: SHIP
[21:16:11] Using gulpfile ~/sp-dev-fx-webparts/samples/react-calendar/gulpfile.js
[21:16:11] Starting gulp
[21:16:11] Starting 'bundle'...
[21:16:11] Starting subtask 'configure-sp-build-rig'...
[21:16:11] Finished subtask 'configure-sp-build-rig' after 6.07 ms
[21:16:11] Starting subtask 'pre-copy'...
[21:16:12] Finished subtask 'pre-copy' after 22 ms
[21:16:12] Starting subtask 'stylelint'...
[stylelint]: By default style lint errors will not break your build. If you want to change this behaviour, modify failAfterError parameter in gulpfile.js.
[21:16:12]
src/webparts/reactCalendar/components/ReactCalendar.module.scss
13:5  ✖  Expected empty line before declaration   declaration-empty-line-before
[21:16:12] Finished subtask 'stylelint' after 658 ms
[21:16:12] Starting subtask 'copy-static-assets'...
[21:16:12] Starting subtask 'sass'...
[21:16:12] Warning - [sass] src/webparts/calendar/components/calendar.scss: filename should end with module.sass or module.scss
Browserslist: caniuse-lite is outdated. Please run next command npm update
[21:16:12] Finished subtask 'copy-static-assets' after 255 ms
[21:16:12] Finished subtask 'sass' after 314 ms
[21:16:12] Starting subtask 'tslint'...
[21:16:14] [tslint] tslint version: 5.12.1
[21:16:14] Starting subtask 'tsc'...
[21:16:14] [tsc] typescript version: 3.2.4
[21:16:22] Finished subtask 'tslint' after 9.26 s
[21:16:22] Finished subtask 'tsc' after 8.51 s
[21:16:22] Starting subtask 'post-copy'...
[21:16:22] Finished subtask 'post-copy' after 149 μs
[21:16:22] Starting subtask 'collectLocalizedResources'...
[21:16:22] Finished subtask 'collectLocalizedResources' after 4.24 ms
[21:16:22] Starting subtask 'configure-webpack'...
[21:16:22] Finished subtask 'configure-webpack' after 324 ms
[21:16:22] Starting subtask 'webpack'...
[21:17:22] Finished subtask 'webpack' after 59 s
[21:17:22] Starting subtask 'configure-webpack-external-bundling'...
[21:17:22] Finished subtask 'configure-webpack-external-bundling' after 501 μs
[21:17:22] Starting subtask 'copy-assets'...
[21:18:18] Finished subtask 'copy-assets' after 57 s
[21:18:18] Starting subtask 'write-manifests'...
[21:18:19] Finished subtask 'write-manifests' after 513 ms
[21:18:19] Finished 'bundle' after 2.12 min
[21:18:19] ==================[ Finished ]==================
Warning - [sass] src/webparts/calendar/components/calendar.scss: filename should end with module.sass or module.scss
[21:18:20] Project react-calendar version:1.0.1
[21:18:20] Build tools version:3.12.1
[21:18:20] Node version:v10.15.1
[21:18:20] Total duration:2.17 min
[21:18:20] Task warnings:1
The build failed because a task wrote output to stderr.
Exiting with exit code: 1**
I want to use both of them for my org and am wondering if anyone can help.

Hi @Ishan-seth  I assume you're talking about the react-calendar solution?
@joaojmendes I noticed that I got some warnings, which could be what we're experiencing here:
There are multiple modules with names that only differ in casing.
c:/temp/sp-dev-fx-webparts/samples/react-calendar/node_modules/fbjs/lib/setImmediate.js
There are multiple modules with names that only differ in casing.
This can lead to unexpected behavior when compiling on a filesystem with other case-semantic.
Use equal casing. Compare these module identifiers:
* c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\source-map-loader\index.js!c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\fbjs\lib\setImmediate.js
    Used by 1 module(s), i. e.
    c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\source-map-loader\index.js!c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\draft-js\lib\editOnBeforeInput.js
* c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\source-map-loader\index.js!c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\fbjs\lib\setimmediate.js
    Used by 2 module(s), i. e.
    c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\source-map-loader\index.js!c:\temp\sp-dev-fx-webparts\samples\react-calendar\node_modules\fbjs\lib\setImmediate.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/node_modules/fbjs/lib/setImmediate.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/node_modules/draft-js/lib/editOnBeforeInput.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/node_modules/draft-js/lib/DraftEditorEditHandler.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/node_modules/draft-js/lib/DraftEditor.react.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/node_modules/draft-js/lib/Draft.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/lib/controls/Event/event.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/lib/webparts/calendar/components/Calendar.js
 @ c:/temp/sp-dev-fx-webparts/samples/react-calendar/lib/webparts/calendar/CalendarWebPart.js

Hi @hugoabernier , yes that's correct, I was able to get the react-angular-calendar working, but I am stuck at the react-calendar, and haven't been able to compile it yet, as a matter of fact, i tried another sample a couple of days ago react-tree-org-orgchart and ran into similar issues.

@Ishan-seth would you be able to run spfx doctor and see if there are any issues with your environment/project settings and get back to us with the results?
Also, next time you open an issue, please take the time to fill the template we provided so that we can help you better.

@hugoabernier , thank you for helping, I just ran spfx doctor and this was the result:
✔ SharePoint Framework v1.10.0
✔ Node v10.15.1
✔ npm v6.14.5
✔ yo v2.0.5
✔ gulp v3.9.1
✔ bundled typescript used
Noted, i will keep that in mind for the next time.

@Ishan-seth I just upgraded the sample to SPFx 1.10 and took care of some build errors. Can you try and get the latest and run:
npm i
gulp build

From the samples\react-calendar\ folder and see what happens?

@hugoabernier it seems to be working right now, I am doing some additional testing and will let you know.


Incorrect mention of dataset....

URL(s) with the issue:
https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_lite/tflite_c05_exercise_rock_paper_scissors.ipynb

Description of issue (what needs changing):
I think this exercise doesn't make use of cats_vs_dogs dataset, right??
Clear description
In place of cats_vs_dogs dataset, rock_paper_scissors dataset should be mentioned.
Submit a pull request?
Yes, shortly

Closing this issue since the associated PR has merged.
Feel free to reopen if necessary. Thanks!


No volume control for LS49AG95

Before opening the issue, have you...?

[X] Searched for existing issues
[X] Looked through the wiki
[X] Updated MonitorControl to the latest version (if applicable)

Describe the bug
I've tried using monitor control and better display to control the volume of my LS49AG95 (apple silicon). Both display a slider but the volume doesn't change no matter the position of the slider. Brightness however works as expected.
Steps to reproduce
Change volume on application.
Expected behavior
Monitor volume should adjust.
Anything else?
No response
Environment Information (please complete the following information)
- macOS version: Ventura 13.2.1
- Mac model: MacBook Pro 14 inch 2021
- MonitorControl version: 4.1.0 Build 7034
- Monitor(s): LS49AG95
- Apple Silicon/M1 (yes or no): yes

You are either using HDMI which is not supported in MonitorControl (you can use BetterDisplay for that) or the display does not support DDC or if hardware brightness works then DDC volume.


Improve typed_endpoint validation for full RemoteServerNotificationPayload types

In #29428, we convert zilencer.py to use the new @typed_endpoint decorator, but we've still got some Dict[str, Any] types for fields of RemoteServerNotificationPayload that would ideally be validated more fully.
It might be a pain to do correctly, so I'm leaving this as a follow-up project.

@zulipbot claim

yes

@zulipbot abandoned

@zulipbot abandon

@zulipbot abandon

@zulipbot claim

@zulipbot claim

#30098 resolves this issue

#30102 pull request fix

@zulipbot claim


Drop 5 not available and Documentation issue.

I still get only version 0.2.1 but not version 0.3.0 because it is not a valid version.
The npm registry also shows the wrong version
In additon there is a bug on the drop 5 release note.
"@microsoft/sp-webpart-base": "~0.1.0",
As far as i've seen not part of previous versions so this entry needs to be newly added and the version number cannot be updated.

Unclear why we wound up at 2.1 rather than 3.  I believe it is the current version though (when I tried it on my home machine).
I've updated the drop notes accordingly.  Thanks for the heads up.

Great!

:(  What did I forget this time?

Sorry @patmill. :( I'm on version 0.2.1 but the package json doesn't contain @micrsoft/sp-webpart-base
{
  "name": "spfx-test-2",
  "version": "0.0.1",
  "private": true,
  "engines": {
    "node": ">=0.10.0"
  },
  "dependencies": {
    "@microsoft/sp-client-base": "~0.4.0",
    "@microsoft/sp-client-preview": "~0.5.0"
  },
  "devDependencies": {
    "@microsoft/sp-build-web": "~0.7.0",
    "@microsoft/sp-module-interfaces": "~0.4.0",
    "@microsoft/sp-webpart-workbench": "~0.5.0",
    "gulp": "~3.9.1"
  },
  "scripts": {
    "build": "gulp bundle",
    "clean": "gulp nuke",
    "test": "gulp test"
  }
}

Ah, yes.
So the new code that will be getting updated will be in sp-webpart-base.  There was some concern that because some of the properties had changed, we shouldn't default to using it in this drop, but we will switch in the next one.  I'll add some additional comments around this.  Thanks.

Nice thanks!

@patmill sorry but soon you gonna hate me ;)
Is "@microsoft/sp-webpart-base": "~0.1.0" a devDependencie or a dependencie?

Dependency

Hopefully it is much more clear now.  Sorry for the confusion, and thanks for the feedback.


[PERF] OnyxUtils keyChanged cached collection retrieval optimisation

Details

PR introduces an improvement to how getCachedCollection is called in keyChanged.
After investigating this trace, we've noticed that when looping through subscribers list in keyChanged, the getCachedCollection is called for every iteration with the same key.
To do:

[X] cache getCachedCollection results to reuse them between iterations (similar to what we have in keysChanged)
[x] check if cached collection should not be invalidated

I could not reproduce a case from the trace where getCachedCollection is called on each iteration to compare the performance gains. By comparing PR perf to main there was no regression seen and from static analysis we should help when the aforementioned edge case happens.
Related Issues

https://github.com/Expensify/App/issues/46863
Automated Tests

Manual Tests

Author Checklist

[X] I linked the correct issue in the ### Related Issues section above
[X] I wrote clear testing steps that cover the changes made in this PR

[X] I added steps for local testing in the Tests section
[X] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[X] I included screenshots or videos for tests on all platforms
[X] I ran the tests on all platforms & verified they passed on:

[X] Android / native
[X] Android / Chrome
[X] iOS / native
[X] iOS / Safari
[X] MacOS / Chrome / Safari
[X] MacOS / Desktop

[X] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[X] I followed proper code patterns (see Reviewing the code)

[X] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[X] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[X] I verified that comments were added to code that is not self explanatory
[X] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[X] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[X] I verified the JSDocs style guidelines (in STYLE.md) were followed

[X] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[X] I followed the guidelines as stated in the Review Guidelines
[X] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[X] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[X] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[X] I verified that if a function's arguments changed that all usages have also been updated correctly
[X] If a new component is created I verified that:

[X] A similar component doesn't exist in the codebase
[X] All props are defined accurately and each prop has a /** comment above it */
[X] The file is named correctly
[X] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[X] The only data being stored in the state is data necessary for rendering and nothing else
[X] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[X] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[X] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[X] All JSX used for rendering exists in the render method
[X] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[X] If any new file was added I verified that:

[X] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[X] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[X] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[X] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

Changes look good to me as well. Just wanted to check with y'all if you think we can solve this more holistically - discussion here

Can you create App issue for this with details and link it? Also update the cla and checklist, thank you!

Done ✅ !

Just wanted to check with y'all if you think we can solve this more holistically - discussion here

Thanks for your input @hannojg ❤
I think it's definitely worth looking into how the cache retrieval process works!
For now, let's go ahead with the current improvement. It should still have a positive impact, even when we consider the changes you've suggested. What do you think?

Continuation of Slack thread where Hanno wrote:

I think one solution could be to have an additional map directly in the OnyxCache that stores for each collection key the member keys that are in cache. This way we could easily retrieve a collection from cache.

Option A where we would just store the keys of the collection that we have cached

// OnyxCache.ts (pseudo code)
const collectionKeyMap = new Map<string, string>();

set(key: OnyxKey, value: OnyxValue<OnyxKey>): OnyxValue<OnyxKey> {

  if (isCollectionKey(key) {
    collectionKeyMap[key] = splitCollectionMemberKey(key)[1]
  }
}

getCachedCollection(key: CollectionKey) {
  const result = {}
  Object.values(collectionKeyMap[key]).forEach(...)
  return result
}

And maybe there could be an option B, where we have a collectionCacheMap that stores the collections directly :thinking_face:

in regards to the PR the Problem with solution A would be that we might still for the same key (lets say reports_) create the collection twice, so Option B might be better :thinking_face:

Options B seems more practical but I am not sure of what would be the memory impact of storing such collections once again just to serve as a middleman between in-memory cache and end-user. Let's think of how we can improve the idea even further.
Having that said @mountiny, what are the next steps if there is a positive response from reviewers?

For now, let's go ahead with the current improvement. It should still have a positive impact, even when we consider the changes you've suggested. What do you think?

Absolutely, lets do that 😊

Options B seems more practical but I am not sure of what would be the memory impact of storing such collections once again just to serve as a middleman between in-memory cache and end-user. Let's think of how we can improve the idea even further.

Maybe, instead of having an extra copy just for collections, we could store collections and regular keys separately in the cache (so the data is only there once)

Options B seems more practical but I am not sure of what would be the memory impact of storing such collections once again just to serve as a middleman between in-memory cache and end-user. Let's think of how we can improve the idea even further.

Maybe, instead of having an extra copy just for collections, we could store collections and regular keys separately in the cache (so the data is only there once)

@hannojg Though I agree with this but if we do this it would require us adding some if condition in set function which would check whether a key is collection or otherwise and then add to the right data structure.
I was thinking if we can do something like this:
diff --git a/lib/OnyxCache.ts b/lib/OnyxCache.ts
index 530f8e6..69da4f6 100644
--- a/lib/OnyxCache.ts
+++ b/lib/OnyxCache.ts
@@ -9,7 +9,7 @@ import type {OnyxKey, OnyxValue} from './types';
  */
 class OnyxCache {
     /** Cache of all the storage keys available in persistent storage */
-    private storageKeys: Set<OnyxKey>;
+    private storageKeys: Map<string, Set<OnyxKey>>;

     /** A list of keys where a nullish value has been fetched from storage before, but the key still exists in cache */
     private nullishStorageKeys: Set<OnyxKey>;
@@ -30,7 +30,7 @@ class OnyxCache {
     private maxRecentKeysSize = 0;

     constructor() {
-        this.storageKeys = new Set();
+        this.storageKeys = new Map();
         this.nullishStorageKeys = new Set();
         this.recentKeys = new Set();
         this.storageMap = {};
@@ -59,8 +59,8 @@ class OnyxCache {
     }

     /** Get all the storage keys */
-    getAllKeys(): Set<OnyxKey> {
-        return this.storageKeys;
+    getAllKeys(key: string): Set<OnyxKey> {
+        return this.storageKeys.get(key) || new Set();
     }

     /**
@@ -74,15 +74,15 @@ class OnyxCache {
      *
      * @param keys - an array of keys
      */
-    setAllKeys(keys: OnyxKey[]) {
-        this.storageKeys = new Set(keys);
+    setAllKeys(key: string, keys: OnyxKey[]) {
+        this.storageKeys.set(key, new Set(keys));
     }

     /** Saves a key in the storage keys list
      * Serves to keep the result of `getAllKeys` up to date
      */
     addKey(key: OnyxKey): void {
-        this.storageKeys.add(key);
+        this.storageKeys.set(key, this.storageKeys.get(key)?.add(key) || new Set([key]));
     }

     /** Used to set keys that are null/undefined in storage without adding null to the storage map */

I have read the CLA Document and I hereby sign the CLA

All yours @MariaHCD


Detox Maintenance: Update deps, fix import, fix failing tests -- iOS only

Summary

Updated deps to fix import issue
Fixed and stabilized failing tests
Added known issue for MM-50085 and MM-50206
Added test IDs
Removed bottomsheet backdrop tap and replaced with bottomsheet swipe down

Note: This PR is for iOS only. Android test failures will be addressed on a separate PR.
Ticket Link
NA
Screenshots
NA
Release Note
NONE

/e2e-test

Successfully triggered e2e mobile testing!
https://git.internal.mattermost.com/qa/mobile-e2e-testing/-/pipelines/296126

@josephbaylon It looks like ios-build-pr job has been skipped but I'm not sure why --> https://git.internal.mattermost.com/qa/mobile-e2e-testing/-/pipelines/296126

@metanerd It looks like ios-build-pr job has been skipped but I'm not sure why --> https://git.internal.mattermost.com/qa/mobile-e2e-testing/-/pipelines/296126

/e2e-test

Successfully triggered e2e mobile testing!
https://git.internal.mattermost.com/qa/mobile-e2e-testing/-/pipelines/296169

@metanerd Looks like it proceeded this time --> https://git.internal.mattermost.com/qa/mobile-e2e-testing/-/pipelines/296169
Not sure if there's a race condition that happened in the previous run for it not to proceed properly

/e2e-test

Successfully triggered e2e mobile testing!
https://git.internal.mattermost.com/qa/mobile-e2e-testing/-/pipelines/296246


Add upstream field.

Fixes #249.

LGTM

Thanks for the suggestions @luhring !

Without reading too deeply into this change it does seem like an interesting restriction to have available. Is anyone committing to producing OSV records which use this relation?

Without reading too deeply into this change it does seem like an interesting restriction to have available. Is anyone committing to producing OSV records which use this relation?

We have a lot of positive feedback on https://github.com/ossf/osv-schema/issues/249 from various distro feed owners. The change itself is also proposed by @luhring from Chainguard. While "committing" is a strong word, I think this would be a field that will be used from these feeds.

Gotcha. Well then seems like a reasonable reference type to me 👍

Ditto what @oliverchang said — and we plan on using it for the Chainguard feed.


Feature/retry delivery attempt

Added delivery attempt in push endpoint payload

Codecov Report

Merging #245 (28cc1ca) into feature/retry_with_delay (013c266) will decrease coverage by 2.38%.
The diff coverage is 0.00%.

@@                     Coverage Diff                      @@
##           feature/retry_with_delay     #245      +/-   ##
============================================================
- Coverage                     44.46%   42.08%   -2.39%
============================================================
  Files                           104      104
  Lines                          4797     4798       +1
============================================================
- Hits                           2133     2019     -114
- Misses                         2501     2632     +131
+ Partials                        163      147      -16

Flag
Coverage Δ

integration
?

unittests
42.08% <0.00%> (+0.30%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

internal/stream/stream.go
35.86% <0.00%> (+6.90%)
:arrow_up:

internal/subscriber/subscriber.go
0.00% <0.00%> (ø)

pkg/messagebroker/factory.go
33.33% <0.00%> (-33.34%)
:arrow_down:

pkg/messagebroker/kafka.go
10.46% <0.00%> (-29.93%)
:arrow_down:

internal/subscriber/model.go
20.83% <0.00%> (+2.08%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 013c266...28cc1ca. Read the comment docs.


Extension mechanism AWS testing

initial fixes to aws config validation and cloudflare dns autoprovisioning

Reference Issues or PRs

What does this implement/fix?
Put a x in the boxes that apply

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds a feature)
[ ] Breaking change (fix or feature that would cause existing features not to work as expected)
[ ] Documentation Update
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build related changes
[ ] Other (please describe):

Testing

[x] Did you test the pull request locally?
[ ] Did you add new tests?

Any other comments?
Test plan

[x] Log into keycloak as root

[x] Add a user

[x] Log into conda-store and create

[x] conda environment in a shared namespace
[x] ... in personal workspace

[x] Launch a dask gateway cluster, test autoscaler

[x] Validate dask lab extension is working

[x] Create basic CDS dashboard
[x] Open VS Code extension

[x] Add python extenion
[x] Create .py file and run it

6

I opened this PR to merge the extension-mechanism branch into this one.


Methods to retrieve and set the line data

For filtering and redacting lines.

I introduced LineBufferMut trait, that use &[u8] to access the log line data.


Mipro crash in mipro_optimizer.py

Notice in the watch window that there is nothing in the example_sets for i=3. Needs a check here?

Perhaps the should look like this?
                        if i in example_sets[id(predictor)]:
                            new_instruct = dspy.Predict(
                                BasicGenerateInstructionWithExamplesAndDataObservations,
                                n=1,
                                temperature=self.init_temperature,
                            )(
                                basic_instruction=basic_instruction,
                                observations=self.observations,
                                examples=example_sets[id(predictor)][i],
                            )

cc @klopsahlong

Why was this closed? The issue still persists..


Add description for different types of blocks

#3827
Added description of Volto Blocks and for every different type of block. Also added relevant screenshots.

Thank you so much for your thorough review and for taking the time to provide such thoughtful feedback. Your insights and suggestions are greatly appreciated and will be very valuable to me.

Provide sufficient detail of what each option is and does.

I thought of doing it but it made me nervous as it got too long 😖, I'll make sure nxt time.

Avoid repetitive statements.
Avoid marketing speak. Words such as "powerful", "easy", "simple" come off as hyping a feature, instead of plainly describing what it is and does.

I am wondering if it would be better to apply these changes now or wait until a later time?
Thanks, everyone for your help.

@MAX-786 the grid and the system ones are not official Plone/Volto blocks, although they are present in the demo.
I'd leave the grid one for now (it's planned for Plone 6.1) and remove the system one, since it's a demo site block only.
For the rest, great addition!

Thanks for the review, Can you please tell me if these changes should be done by me or if you have already done them, as I can still see the System Block in the last commit?

@MAX-786 please make the requested changes from @sneridagh. Thank you!

It's Done! Thanks.

OMG you're actually doing this! Writing a user manual for Volto. You two are awesome!

And as I have limited knowledge of cypress..... this 1 failing test is giving me horror, something's wrong in listingblock.js
Below is the error,  I'll try googling if i can find what's the problem ..... and.....please help (>_<)

Timed out retrying after 4050ms: cy.click() failed because this element is detached from the DOM.
<button class="ui basic icon button listing">...</button>
Cypress requires elements be attached in the DOM to interact with them.
The previous command that ran was:

cy.contains()

This DOM element likely became detached somewhere between the previous and current command.
Common situations why this happens:

Your JS framework re-rendered asynchronously
Your app code reacted to an event firing and removed the element
You typically need to re-query for the element or add 'guards' which delay Cypress from running new commands.

I think you can ignore the failing test as this PR involves documentation only, and nothing to do with core.
Nonetheless, how does one rerun a failed test in Cypress?

OMG you're actually doing this! Writing a user manual for Volto. You two are awesome!

It ain't gonna write itself. 😝

nm, I got lost in Cypress. I found it in GHA. Rerunning it now.

The failing test was a glitch, and we're all green. Merging! 🥳


Refactoring the C++ extension codes, introducing pointing flags with code optimization

This PR introduces the following breaking changes in the code base:

Refactoring the C++ extension codes

[x] ProcessTimeSamples
[x] SparseLO --> PointingLO
[x] BlockDiagonalPreconditionerLO

Refactoring the Python interface

[x] brahmap/utilities/process_time_samples.py
[ ] brahmap/interfaces/linearoperators.py

[x] Introducing the pointing flags
[ ] Optimization of the for-loops with nested conditionals

In the last commit, I:

introduced the pointing flags.

refactored brahmap/utilities/process_time_samples.py, its C++ extensions and relevant tests

Attempted to optimize the for-loops structures like this:
for (int idx=0; idx<nsamples; ++idx) {
  int pixel = pointings[idx];
  if (flag[idx]) { // boolean flag
    quantity[pixel] += set1[idx]*set2[idx];
  } // if
} // for

with this:
for (int idx=0; idx<nsamples; ++idx) {
  int pixel = pointings[idx];
  quantity[pixel] += (flag[idx)*set1[idx]*set2[idx];
} // for

Benchmarking is pending for this attempt.

In the _rmult_* methods of PointingLO, we had loops like this:
for i in range(Nrows):
    if pointings[i] == -1:
        continue
    vec_out[2 * pointings[i]] += v[i] * cos[i]
    vec_out[2 * pointings[i] + 1] += v[i] * sin[i]

and in _mult_*, we had loops like this:
for i in range(Nrows):
    if pointings[i] == -1:
        continue
    x[i] += v[2 * pointings[i]] * cos[i]
    +v[2 * pointings[i] + 1] * sin[i]

We can translate _mult_* above with pointings_flag as following:
for i in range(Nrows):
    pixel = pointings[i]
    x[i] += pointings_flag[i] * v[2*pixel] * cos[i]
            + v[2*pixel+1] * sin[i]

However, since PointingLO is now based on new_npix, v[2*pixel] can simply go out of range of v. In the original version of the code, the pointings for the flagged pixels were changed to -1, thus avoiding the out of range error. In our case, this issue can be solved by changing the in the pointings array the pathological pixels to 0 while flagging the sample sample. But changing the pointings array and pointings_flag seems little expensive. So we introduced in ProcessTimeSamples the pixel_flag given by
pixel_flag = np.zeros(npix, dtype=bool)
for pixel in pixel_mask:
    pixel_flag[pixel] = True

With this, we can now write the _mult_* as
for idx in range(Nrows):
    pixel = pointings[idx]
    pixflag = pixel_flag[pixel]
    pixel *= pixflag   # This marks pathological pixel to pixel index 0
    pointflag = pixflag && pointings_flag[idx]  # This flags the sample corresponding to pathological pixel to zero

    x[i] += pointings_flag[i] * v[2*pixel] * cos[i]
            + v[2*pixel+1] * sin[i]

Picking from the last comment, we had the for loop for _mult_* given by
for idx in range(Nrows):
    pixel = pointings[idx]
    pixflag = pixel_flag[pixel]
    pixel *= pixflag   # This marks pathological pixel to pixel index 0
    pointflag = pixflag && pointings_flag[idx]  # This flags the sample corresponding to pathological pixel to zero

    x[i] += pointings_flag[i] * v[2*pixel] * cos[i]
            + v[2*pixel+1] * sin[i]

While it successfully deals with pathological pixels stored in the pointings array, v[2*pixel] can still go out of range. Note that the pixels in the pointings array ranges from 0 to npix while the shape of PointingLO is now nrows = nsamp, ncols = new_npix x solver_type. So the pixel in this loop must range from 0 to new_npix x solver_type. We could not come up with a solution to this problem without relying on if-statements. In the end, we decided to update the pointings and pointings_flag when ProcessTimeSamples is called such that,

pointings[pathological_pixels] = 0: This is just to avoid out-of-range errors/warnings, when the pointing sample corresponding to a pathological pixel is accessed.
pointings[idx] = old2new_pixel[pointings[idx]]: Once we flagged the pathological pixels, we can renew the pixel index such that they vary from 0 to new_npix (where new_npix = npix - n_pathological_pixels). The array old2new_pixel simply maps the old pixel indices to new pixel indices: new_pix_idx = old2new_pixel[old_pix_idx]. In fact, this was already in place in the earlier versions of our code.
pointings_flag[pathological_pixels] = False: To ensure that pointing samples corresponding to pathological pixels are excluded from PointingLO operator and eventually the entire map-making pipeline. This makes sure that the pointing samples set to zero (see the first point) will not be accounted in the map-making.

With these changes, now loop becomes
for idx in range(Nrows):
    pixel = pointings[idx]
    pointflag = pointings_flag[idx]

    x[i] += pointings_flag[i] * (v[2*pixel] * cos[i]
            + v[2*pixel+1] * sin[i])

Shouldn’t the flag be applied to both operations ,e.g. x[i] += pointings_flag[i] * (v[2pixel] * cos[i]
+ v[2pixel+1] * sin[i])On Apr 4, 2024, at 6:16 PM, Avinash Anand @.**> wrote:
Picking from the last comment, we had the for loop for mult given by
for idx in range(Nrows):
pixel = pointings[idx]
pixflag = pixel_flag[pixel]
pixel *= pixflag   # This marks pathological pixel to pixel index 0
pointflag = pixflag && pointings_flag[idx]  # This flags the sample corresponding to pathological pixel to zero
x[i] += pointings_flag[i] * v[2*pixel] * cos[i]
        + v[2*pixel+1] * sin[i]

While it successfully deals with pathological pixels stored in the pointings array, v[2*pixel] can still go out of range. Note that the pixels in the pointings array ranges from 0 to npix while the shape of PointingLO is now nrows = nsamp, ncols = new_npix x solver_type. So the pixel in this loop must range from 0 to new_npix x solver_type. We could not come up with a solution to this problem without relying on if-statements. In the end, we decided to update the pointings and pointings_flag when ProcessTimeSamples is called such that,
pointings[pathological_pixels] = 0: This is just to avoid out-of-range errors/warnings, when the pointing sample corresponding to a pathological pixel is accessed.
pointings[idx] = old2new_pixel[pointings[idx]]: Once we flagged the pathological pixels, we can renew the pixel index such that they vary from 0 to new_npix (where new_npix = npix - n_pathological_pixels). The array old2new_pixel simply maps the old pixel indices to new pixel indices: new_pix_idx = old2new_pixel[old_pix_idx]. In fact, this was already in place in the earlier versions of our code.
pointings_flag[pathological_pixels] = False: To ensure that pointing samples corresponding to pathological pixels are excluded from PointingLO operator and eventually the entire map-making pipeline. This makes sure that the pointing samples set to zero (see the first point) will not be accounted in the map-making.
With these changes, now loop becomes
for idx in range(Nrows):
pixel = pointings[idx]
pointflag = pointings_flag[idx]
x[i] += pointings_flag[i] * (v[2*pixel] * cos[i]
        + v[2*pixel+1] * sin[i])

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

@giuspugl Yes, it has already been applied to both operators.

With respect to the refactoring done so far, the previous class/function interfaces are now incompatible with the new one. In light of this, I have now implemented the function brahmap.mapmakers.compute_GLS_maps() to provide a general interface for GLS map-maker. Another function is brahmap.mapmakers.LBSim_compute_GLS_maps() that provides the interface for GLS map-maker from litebird_sim.

For noise-less time-ordered data (TOD) from litebird_sim in the absence of half-wave plate (hwp), we expect the preconditioned conjugate gradient (PCG) solver to converge in a single iteration, as the preconditioner linear operator would be a diagonal operator in this case. However, we found that the PCG was never converging. Digging in the linear operators, we found that few 3x3 diagonal blocks in the preconditioner operator were zero, that was not what we expect. Turned out it was due to the test we were performing during multiplication between BlockDiagonalPreconditionerLO and a vector. The test was intended to identify and ignore the singular 3x3 matrix blocks.
To fix this issue, we revisited the function get_pixel_mask_pol(). In this function, we flagged all the pixels where the condition number of the following matrix was above certain threshold:
$$\begin{bmatrix}
\sum\cos^2\phi_i/w_i & \sum\sin\phi_i\cos\phi_i/w_i\
\sum\sin\phi_i\cos\phi_i/w_i & \sum\sin^2\phi_i/w_i
\end{bmatrix}$$
We decided to replace this flagging scheme with the one where we flag all the pixels that might result in singular matrix block in the BlockDiagonalPreconditionerLO. For QU solver, this block matrix will be
$$\begin{bmatrix}\sum\cos^2\phi_i/w_i & \sum\sin\phi_i\cos\phi_i/w_i\
\sum\sin\phi_i\cos\phi_i/w_i & \sum\sin^2\phi_i/w_i \end{bmatrix}$$
whereas for IQU solver, this matrix block will be
$$\begin{bmatrix}
\sum 1/w_i & \sum\cos\phi_i/w_i & \sum\sin\phi_i/w_i \
\sum\cos\phi_i/w_i & \sum\cos^2\phi_i/w_i & \sum\sin\phi_i\cos\phi_i/w_i\
\sum\sin\phi_i/w_i & \sum\sin\phi_i\cos\phi_i/w_i & \sum\sin^2\phi_i/w_i
\end{bmatrix}$$
Eventually in the new pixel flagging scheme, we are computing the determinant of the matrix blocks in get_pixel_mask_pol() and then flagging it if the determinant is below certain threshold. Since the determinant is now a member of ProcessTimeSamples class, now we won't compute it every time the matrix vector multiplication of BlockDiagonalPreconditionerLO is called.

Everything looks good, few observations :

i think even in presence of HWP signal only sims should converge with 1 iteration
please remember the factor of 2 in the trigonometric functions , e.g. $\sin 2 \phi $

i think even in presence of HWP signal only sims should converge with 1 iteration

Yes, it does converge with one iteration in the presence of HWP

please remember the factor of 2 in the trigonometric functions , e.g. $\sin2\phi$

Thanks, I have now updated the comment!


Support for "dashed" polylines

This PR adds support for screen space "dashed" polylines.  This attempts to get something similar to OpenGL line stippling to work with Cesium's neat wide polyline algorithm.  I tried various techniques using the texture coordinates of the line but none ended up looking as nice as this one.
There is a new PolylineDashMaterialProperty that controls the PolylineDash material.
I've modified the PolylineMaterialAppearanceVS.glsl shader to compute a varying, v_angle, that is fed to the fragment shader.  This angle is the screen space angle between the line and the x axis quantized to 45 degree increments.  I've quantized rather than using the actual value to try to avoid artifacts as the polyline expands.
The fragment shader takes the incoming pixel coordinate, rotates it by the incoming angle and then does a simple stippling along the x-axis on the rotated pixel coordinate.  The fragment location is rotated to create a striped pattern in the direction of the line so it actually appears like it's being stippled along the line.
I've updated the Polyline and Polyline CZML sandcastle examples to have a dashed line so you can see it in action.

This is awesome @jasonbeverage, thanks!  This has been a frequently requested feature and will make the Cesium community very happy :smile:

Fixes #2584

Thanks @jasonbeverage.  This is a pleasant surprise and will be a very popular feature!
I would like to review this before the final merge.  @bagnell should also take a look at this first since he wrote the original polyline shader, but he's busy for probably all of this week.  In the meantime, if anyone has cycles to review or even just test this, please jump in!
@jasonbeverage I'm not sure if you've already read it, but you might like this paper: Shader-Based Antialiased, Dashed, Stroked Polylines

Awesome work @jasonbeverage. You keep cranking out some great stuff. I'm not qualified to review the shaders here, but the Entity and CZML pieces look spot on (will look closer as soon as I have time if no one else beats me to it).
One of the things me and @bagnell have talked about for stippled lines is the ability to specify a custom pattern (perhaps through a string like -- - ---).  Would that be an easy extension to this implementation?  (I'm just asking out of curiosity).  I know @bagnell also had some ideas (such as using images as part of line styles but that's also outside the scope of this PR).

This is crazy awesome.
@mramato You probably know shaders can't directly take a string, but we could pass a float parameter to indicate a percentage of dash vs. gap, or I could even imagine passing a one-dimensional alpha map that supplies a dashing pattern to use.  That would be for a future version of course, this seems cool enough as-is that it should probably go in without a lot of feature creep.  Also I was thinking it could be useful to supply a second color for use in the gaps, instead of just discarding them (for example, use the same color with less alpha, or alternate the brightness, etc).  But again, that's feature creep, and this is plenty good enough already.
Dashed lines (as they are now) will be very useful in combination with the depthFail feature going on in #5160.  Underground lines are crying out for dashes.

Don't merge this just yet, I'm going to try to push a couple more things and see what you think.

@jasonbeverage great.  If you're making changes can you switch Color.BLUE to Color.CYAN to make the line more visible?  Likewise in the CZML crank the green channel to 255.  Thanks!

Sure thing, just pushed that change.

I got asked by @GatorScott to test with a more curved path, and I had an inclined GEO handy.  Here's a gist.  It works pretty well, even for this tough case.

I just pushed a few changes.
There is a new dashPattern property for the PolylineDashMaterial that defines a 16 bit stipple pattern.
I changed some of the defaults so that dashLength is 16 pixels so it lines up per pixel (approximately) with the dash pattern.  The default dash pattern is 255 (0000 0000 1111 1111).
I've also added a new Polyline Dash sandcastle example.
Also, just wanted to say thanks to @timoore for doing the initial work on this effort, including the dashPattern logic.  Our original approach just used the distance along the line to drop pixels, but that caused too many artifacts with your wide polyine algorithm (similar to what you'd see with the StipeMaterial with the direction set to vertical).  I came up with the screen space fragment shader based stippling just recently and the artifacts aren't nearly as bad.

I wonder if CZML (and maybe Entity) can take a 16-character string instead of numeric dashPattern.  For example:
var dashPattern = '        --------';

Could this be converted to 255 by testing each char position and setting a bit?  This would make the pattern more user-friendly.

Here's some code to turn an arbitrary-length string of spaces and dashes into a closest-approximation 16-bit field.  This just iterates the 16 bits, picks the closest char based on length of input string, and figures out if it will set that bit or not.
function dashStringToBits(dashPatternString) {
        var dashPatternLength = dashPatternString.length;
        var dashPatternNumber = 0;
        var currentBitValue = 65536;
        for (var i = 0; i < 16; ++i) {
            currentBitValue = currentBitValue >> 1;
            var charPos = Math.floor((i / 16) * dashPatternLength);
            var char = dashPatternString.charAt(charPos);
            if (char !== ' ') {
                dashPatternNumber |= currentBitValue;
            }
        }
        return dashPatternNumber;
    }

dashStringToBits(' -')
255
dashStringToBits('- ')
65280
dashStringToBits('    ----')
255
dashStringToBits('        --------')
255
dashStringToBits('  -  ---')
3135

Just pushed a fix for my fat finger typos :)

Would PolylineDashMaterialProperty.getValue just need to call that

Off the top of my head I'm not sure where it goes.  I'd have to research a bit.  Curious to hear @mramato's thoughts before we go too far on it.

I've just been mashing 0 and 1 into the Windows programmer calculator and copying the resulting decimal value :)
I also just tried your gist and it looks awesome!

Also, just wanted to say thanks to @timoore for doing the initial work on this effort, including the dashPattern logic...

Thanks again to both of you!

While I was an original proponent of the string technique, it's actually not needed.  There are two easy ways just use binary literals in JavaScript.

In "old" JS, you can use parseInt, for example: parseInt('1010101010101010', 2).
ES6 has binary literals with the 0b as a prefix.  For example: 0b0000111100001111.

We should probably update the example to use 1, since it will work everywhere, but the doc example might want to reference both.
I think these are easy enough to grok that the string approach is obsolete.
For CZML, you would need the number in the data instead of a string, but the writer could easily handle this rather than forcing the client to do it (plus sending a number down is way more efficient than putting a string in CZML).  What do you guys think?

+1 for @mramato's ideas.  This is also how we would represent it in 3D Tiles.

I like it.  I replaced the explicit dashPattern in the example with parseInt, much easier to read.

What do you think about adding a gapColor instead of discard?
float maskTest = floor(dashPattern / pow(2.0, maskIndex));
vec4 fragColor = (mod(maskTest, 2.0) < 1.0) ? gapColor : color;
if (fragColor.a < 0.005) {   // matches 0/255 and 1/255
    discard;
}

material.emission = fragColor.rgb;
material.alpha = fragColor.a;
return material;

+1 for gapColor!

I just pushed the new gapColor property and updated the Polyline Dash example to demonstrate it.

It's sooo tempting to merge this for 1.32.  Someone remind me why we shouldn't.

You could change the getPolylineWindowCoordinates function in PolylineCommon to have an out angle parameter to have a performance boost and avoid a little bit of duplicate work.  I won't pretend to understand exactly what is going on in there, but you can do something like this:
vec4 getPolylineWindowCoordinates(vec4 position, vec4 previous, vec4 next, float expandDirection, float width, bool usePrevious, out float angle) {
    vec4 endPointWC, p0, p1;
    bool culledByNearPlane, clipped;

    vec4 positionEC = czm_modelViewRelativeToEye * position;
    vec4 prevEC = czm_modelViewRelativeToEye * previous;
    vec4 nextEC = czm_modelViewRelativeToEye * next;

    vec4 positionWC = czm_eyeToWindowCoordinates(positionEC);
    vec4 pWC = czm_eyeToWindowCoordinates(prevEC);
    vec4 nWC = czm_eyeToWindowCoordinates(nextEC);

    clipLineSegmentToNearPlane(prevEC.xyz, positionEC.xyz, p0, clipped, culledByNearPlane);
    clipLineSegmentToNearPlane(nextEC.xyz, positionEC.xyz, p1, clipped, culledByNearPlane);
    clipLineSegmentToNearPlane(positionEC.xyz, usePrevious ? prevEC.xyz : nextEC.xyz, endPointWC, clipped, culledByNearPlane);

    vec2 prevWC = normalize(p0.xy - endPointWC.xy);
    vec2 nextWC = normalize(p1.xy - endPointWC.xy);

    // Determine the relative screen space direction of the line.
    vec2 lineDir;
    if (usePrevious) {
        lineDir = normalize(positionWC.xy - pWC.xy);
    }
    else {
        lineDir = normalize(nWC.xy - positionWC.xy);
    }
    angle = atan(lineDir.x, lineDir.y) - atan(1.0, 0.0);

    // Quantize the angle so it doesn't change rapidly between segments.
    angle = floor(angle / czm_piOverFour + 0.5) * czm_piOverFour;

    if (culledByNearPlane)
    {
        return vec4(0.0, 0.0, 0.0, 1.0);
    }

    float expandWidth = width * 0.5;
    vec2 direction;

    if (czm_equalsEpsilon(previous.xyz - position.xyz, vec3(0.0), czm_epsilon1) || czm_equalsEpsilon(prevWC, -nextWC, czm_epsilon1))
    {
        direction = vec2(-nextWC.y, nextWC.x);
    }
    else if (czm_equalsEpsilon(next.xyz - position.xyz, vec3(0.0), czm_epsilon1) || clipped)
    {
        direction = vec2(prevWC.y, -prevWC.x);
    }
    else
    {
        vec2 normal = vec2(-nextWC.y, nextWC.x);
        direction = normalize((nextWC + prevWC) * 0.5);
        if (dot(direction, normal) < 0.0)
        {
            direction = -direction;
        }

        // The sine of the angle between the two vectors is given by the formula
        //         |a x b| = |a||b|sin(theta)
        // which is
        //     float sinAngle = length(cross(vec3(direction, 0.0), vec3(nextWC, 0.0)));
        // Because the z components of both vectors are zero, the x and y coordinate will be zero.
        // Therefore, the sine of the angle is just the z component of the cross product.
        float sinAngle = abs(direction.x * nextWC.y - direction.y * nextWC.x);
        expandWidth = clamp(expandWidth / sinAngle, 0.0, width * 2.0);
    }

    vec2 offset = direction * expandDirection * expandWidth * czm_resolutionScale;
    return vec4(endPointWC.xy + offset, -endPointWC.z, 1.0);
}

I don't think you can use the "clipped" window coordinates in this situation.  Every time I've tried I end up with artifacts.

I went ahead and tried to consolidate the code in getPolylineWindowCoordinates.  This avoids a couple of duplicate operations and also cleans up the polyline shaders so they don't have duplicated code.  Give it a look and see what you think.

It's sooo tempting to merge this for 1.32. Someone remind me why we shouldn't.

I'd like @bagnell and me to look at this first.  We are on a few other projects right now so I don't know that we can realistically do this by Monday.

OK, I found a 168 meg CZML file full of polylines, that's quite a stress test bringing Cesium down to about 5fps on my Quadro K2000M. I tried this with the performance counter turned on, and it appears to hover around 170ms per frame with or without this branch. So, for my card at least, there's no performance concern.

The bottleneck will depend on lots of factors like screen size, number of vertices, view direction, etc.  We'd like this to be as efficient as possible, especially if we build 3D Tiles vector tiles on this.
We usually just #ifdef parts of the shader out when it is not being used.  For example, see

BillboardCollectionVS.glsl
BillboardCollection.js

No rush to get it merged, I just wanted it out of my fork so you all could
see it :)
I've also got another PR that is even cooler but need a bit to clean it up
;)
On Wed, Mar 29, 2017, 4:54 PM Patrick Cozzi<EMAIL_ADDRESS>wrote:

It's sooo tempting to merge this for 1.32. Someone remind me why we
shouldn't.
I'd like @bagnell https://github.com/bagnell and me to look at this
first. We are on a few other projects right now so I don't know that we can
realistically do this by Monday.
OK, I found a 168 meg CZML file full of polylines, that's quite a stress
test bringing Cesium down to about 5fps on my Quadro K2000M. I tried this
with the performance counter turned on, and it appears to hover around
170ms per frame with or without this branch. So, for my card at least,
there's no performance concern.
The bottleneck will depend on lots of factors like screen size, number of
vertices, view direction, etc. We'd like this to be as efficient as
possible, especially if we build 3D Tiles vector tiles on this.
We usually just #ifdef parts of the shader out when it is not being used.
For example, see

BillboardCollectionVS.glsl
https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Source/Shaders/BillboardCollectionVS.glsl
BillboardCollection.js
https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Source/Scene/BillboardCollection.js

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AnalyticalGraphicsInc/cesium/pull/5159#issuecomment-290222586,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAT74tWPXetJWfhk6Gb1aM-uo9oL6MMGks5rqsT1gaJpZM4MqyVr
.

I've also got another PR that is even cooler but need a bit to clean it up

Now you're just teasing us 😄

I only have that one comment. Looks good! Thanks @jasonbeverage!
I believe @lilleyse also wants to review this.

@jasonbeverage I spent some time playing with your branch and the PolylineDashMaterial shader today. I created a branch (dashedlines-2 that has your changes plus some of my own. It adds a length in meters from the first point on the line as a new vertex attribute and I tried to compute dashPosition from that. This way you won't see the abrupt changes from v_angle or much of a change in the dash pattern when panning from using gl_FragCoord. It looks pretty good when panning or rotating as long as the camera is mostly looking top-down, but as soon as you start rotating, certain camera views will cause the dashes to shrink or stretch. Any ideas you have about this would be greatly appreciated!
I still want to see this PR merged into master. Computing the dashes based on line length can sit in a branch until someone has time to work on it.

Hey @bagnell , we tried almost the exact same thing initially, and I think it would work fine if you were just drawing regular old lines.  But I couldn't figure out a way to get it to play nice with the wide polyline aglo.  I think the problem is that the length in meters actually changes along the line when you move the verts around, so you end up with funky artifacts depending on how the line rotates.  Your algo also does something along the near plane where it looks like vert is pushed really far towards the near plane so you end up stretching the distance in meters artificially resulting in really wide dashes.
I tried and tried but couldn't come up with something that worked so I ended up scrapping it in favor of this purely fragment based stippling.

@bagnell I tried wrapping the block in polylinecommon with the ifdef, but it looks like just adding #define POLYLINE_DASH to the material didn't work.  It behaves like it's not actually set.  Is there somewhere else the define would need to go?

@mramato for CZML, I'd think it'd be easier and more intuitive to write a bit mask instead of a 16bit number, especially for users more familiar with the data side of things.
IE:
{
dashLength:16,
dashPatthern: '0000000011111111'
}

is more human read/writeable than:
dashPattern: 255

@bagnell and @pjcozzi do you have any additional comments here?  I want to take another pass on the Entity updates (just because I haven't looked at this in a few weeks) but I don't expect any surprises.
Barring any issues, I would like to get this merged ASAP so that it has time to stew before 1.32 on May 1st.
Thanks.

@mramato review and merge when ready.

@bagnell I tried wrapping the block in polylinecommon with the ifdef, but it looks like just adding #define POLYLINE_DASH to the material didn't work. It behaves like it's not actually set. Is there somewhere else the define would need to go?

Sorry, it needs to be in the vertex shader. Can you modify the PolylineMaterialAppearance.vertexShaderSource getter to search the material shaderSource for varying float v_angle and then append the #define?

@mramato for CZML, I'd think it'd be easier and more intuitive to write a bit mask instead of a 16bit number, especially for users more familiar with the data side of things.

We can probably add this in as a CZML-only feature in a small PR after this gets merged.  We'll also need to update the spec and writer too. (Of course we need that for the new dash stuff as well).  CC @shunter

@jasonbeverage we'd like to merge this in the next few days so it has some time in master before it ships in Cesium 1.33 on May 1.  Can you make the change @bagnell suggested, https://github.com/AnalyticalGraphicsInc/cesium/pull/5159#issuecomment-294560604, or do you want us to?
Does anyone else want to test or review this?

If you can do it that would help me out, I'm knee deep in other stuff right
now and you'll be faster at it anyway :)
On Thu, Apr 20, 2017 at 10:55 AM Patrick Cozzi<EMAIL_ADDRESS>wrote:

@jasonbeverage https://github.com/jasonbeverage we'd like to merge this
in the next few days so it has some time in master before it ships in
Cesium 1.33 on May 1. Can you make the change @bagnell
https://github.com/bagnell suggested, #5159 (comment)
https://github.com/AnalyticalGraphicsInc/cesium/pull/5159#issuecomment-294560604,
or do you want us to?
Does anyone else want to test or review this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AnalyticalGraphicsInc/cesium/pull/5159#issuecomment-295766567,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAT74gT0RL6pFeyMJwLjfO6K7fdZy8K0ks5rx3HSgaJpZM4MqyVr
.

@mramato Can you merge when ready?

I threw some fairly complex linework at it and it performed admirably.

Thanks again @jasonbeverage!

Thanks for merging this!  Glad to be able to contribute.

Fantastic!  This will quickly become a popular feature.  Thanks again @jasonbeverage!

@jasonbeverage I had a thirst for dashed lines that you have now quenched, thankyou.

Hello, now there is a problem when the camera is moving, the dash is also moving, which is not nice, I've a made a first pull request to fix that :
https://github.com/CesiumGS/cesium/pull/12343
After this pull request validated, we can use the distance from the first point to calculate better the pattern :)
it's working on my computer


Check for script execution events requires data: URLs to be allowed in the CSP

The CSP 2 spec states that * as CSP does not include the data: scheme.
Though this scheme is used for testing the script execution events, which now incorrectly fails on Firefox due to bug 1086999.
To fix this either the CSP needs to be changed to include data: or the script source needs to be added via s.textContent="window"; instead of s.src="data:text/javascript;charset=utf-8,window".
Sebastian

Is this something that you could commit as a pull request?

Added data: to the CSP and created the PR.
Sebastian

Will be fixed for the upcoming release

Fixed, committed to the release-7.0 branch


Cleanup dead code after metadata refactoring

Some unused code could be removed becasue it's not needed anymore
after the metadata endpoint refactoring.

This was a quick test to determine if the old metadata logic could be removed. This isn't possible because this is used for legacy AM code.
https://www.pivotaltracker.com/story/show/168980881
TLDR This won't be fixed now because the legacy code support will be dropped may 2020.


Fix handling of backslashes in raw strings

A story in two acts.
Act 1: The scanner consumes too much
Inside raw strings, backslashes are passed through without interpretation. This is implemented in the scanner on lines 168-170:
https://github.com/tree-sitter/tree-sitter-python/blob/b14614e2144b8f9ee54deed5a24f3c6f51f9ffa8/src/scanner.cc#L167-L171
Basically, we call lexer->advance() to consume the backslash, and then continue with the scanning.
However, from this point the execution continues until line 227, where we also advance the lexer. The effect of this is that when you write r"\foo", the f is never seen by the scanner. For the most part this is unproblematic, but in cases like fr"\{0}" we get an error, since the { is consumed, and thus the interpolation is never started. Similarly, fr"\N{latin capital letter a}" would consume the N, and thus not interpret it as a named unicode escape, leading to the substring enclosed in curly braces being interpreted as an interpolation instead.
The fix is straightforward: we simply continue after advancing the lexer over the backslash.
Or... It would be straightforward, but there's a catch.
Act 2: The scanner (now) doesn't consume enough
When I said above that backslashes in raw strings are passed through without interpretation, that was not entirely true. Backslashes in raw strings do escape quotes. This means that the above fix breaks when parsing strings such as r"\"", because we now consume the backslash and then terminate the string too early at the quote character immediately after.
To fix this, after advancing the lexer we check if the lookahead is the same character as the ending character of the string. If so, we we also consume that character.

See https://github.com/tree-sitter/tree-sitter-python/pull/182/commits/285f84e5c810916f7718c5163b8585219ca718c8 for the current behaviour of the scanner/parser.

Fixes #111 and #179.


[vx.dev] implements #92

[vx.dev] This PR implements #92, created by vx.dev.

[vx.dev]: This Svelte code creates a music web app prototype with an orange theme. The app features a sidebar for navigation, a menubar for application options, and tabs for different content sections like "Music," "Podcasts," and "Live." The main content area displays curated music selections and personalized playlists with images sourced from Unsplash. The "Add music" button is styled with the primary orange color theme. The prototype uses Tailwind CSS for styling and includes custom properties for the orange theme colors.

$lib/components/ui/sidebar is an invalid import

[vx.dev]: This Svelte code represents a music web application with an orange theme. The sidebar and main content areas are styled with custom CSS variables to match the orange theme. The sidebar includes navigation items for different sections of the app, and the main content area features tabs for Music, Podcasts, and Live content, with placeholders for album artwork. A menu bar at the top provides additional navigation and settings options. The "Add music" button is styled to fit the theme and includes an icon. The images have been replaced with solid colored rectangles to serve as placeholders.


Update the event stream section on the RFC30.

Motivation and Context
Based on these comments
Description
updates the event stream section on the RFC30.
Testing
NA
Checklist
NA
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@jdisanti
Thank you for your suggestions!

Github says, At least 6 approving reviews are required by reviewers with write access..
I feel something isn't working right.

Gotcha. Thanks!


CASMCMS-8599: Add CLI support for BOS v2 session include_disabled option

Summary and Scope
In CSM 1.4, BOS added the include_disabled option when creating V2 sessions. This exposes that option for use in the CLI. This adds a single flag to a single BOS CLI command.
Issues and Related PRs

Resolves CASMCMS-8599
stable/1.5 backport PR
release/1.4 backport PR
metal-provision main PR
metal-provision lts/csm-1.5 PR

Testing
Ran CLI regression tests (including a new test added to cover this option specifically).
Risks and Mitigations
Low risk -- single new option to single BOS command.
Pull Request Checklist

[X] Version number(s) incremented, if applicable
[X] Copyrights updated
[X] License file intact
[X] Target branch correct
[X] Testing is appropriate and complete, if applicable

/backport stable/1.5


(OraklNode) Customizable aggregator frequency

Description
This PR implements customizable aggregator frequency.
On synchronizing from orakl-config, it'll try to read field aggregateHeartbeat from aggregator. If the field aggregateHeartbeat is not found, default interval 5000 (millisecond) will be set.
{
  "aggregatorHash": "0x9f09b00a7525c29679a8121f1a73f54dcfece55b3b8d9f9e80db8cd32d037b2d",
  "name": "ADA-USDT",
  "address": "0x70cDE6bE67486017C52215Ad5d6740ce8EaBC9b8",
  "aggregateHeartbeat": 5000,
  "heartbeat": 15000,
  "threshold": 0.05,
  "absoluteThreshold": 0.1,
  "adapterHash": "0xf15e595e641f1447b5ad1c128007574b329e00937e6db3857aff1487345ed5f6"
}

Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist before requesting a review

[x] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.

Deployment

[ ] Should publish npm package
[ ] Should publish Docker image

@coderabbitai review


youtube-dl DASH video: Show the ffmpeg command output

I am using youtube-dl. I wanted to know how DASH videos are downloded using ffmpeg. Can we see in the verbose the ffmpge command output also

Checklist

[ ] I'm asking a question
[ ] I've looked through the README and FAQ for similar questions
[ ] I've searched the bugtracker for similar questions including closed ones

Question

WRITE QUESTION HERE

youtube-dl does not use ffmpeg to download DASH videos.


ci(bindings/java): enable auto staging JARs on Apache Nexus repository

Waiting for approval - https://issues.apache.org/jira/browse/INFRA-24928
We need this action to set up protoc on all platforms instead of manually tweaks.

@Xuanwo we can proeprly staging the artifacts now!
But I noticed that the staged osx_aarch64 JAR contains x86_64's dylib. I'm digging into the details and will remove the debugging trick before calling a merge.

@Xuanwo @suyanhanx This is ready for review now!


China Kia account cannot log in due to unknown error

Here is the error log
此错误来自自定义集成。
Logger: custom_components.kia_uvo.config_flow
Source: custom_components/kia_uvo/config_flow.py:159
Integration: Hyundai / Kia Connect (documentation, issues)
First occurred: 01:17:48 (1 occurrences)
Last logged: 01:17:48
Unexpected exception
Traceback (most recent call last):
File "/config/custom_components/kia_uvo/config_flow.py", line 159, in async_step_user
await validate_input(self.hass, user_input)
File "/config/custom_components/kia_uvo/config_flow.py", line 63, in validate_input
token: Token = await hass.async_add_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/hyundai_kia_connect_api/KiaUvoApiCN.py", line 145, in login
device_id = self._get_device_id()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/hyundai_kia_connect_api/KiaUvoApiCN.py", line 1018, in _get_device_id
_check_response_for_errors(response)
File "/usr/local/lib/python3.11/site-packages/hyundai_kia_connect_api/KiaUvoApiCN.py", line 92, in _check_response_for_errors
raise APIError(f"Server returned: '{response['resMsg']}'")
hyundai_kia_connect_api.exceptions.APIError: Server returned: 'Invalid request body - Invalid parameter.'

Will need someone in China to debug this.  @toraidl could you take a look?

I hope if you have time, you can help troubleshoot the problem. Thank you very much!

Finally found someone who encountered the same problem. I hope the expert can help to solve it.Thank you very much!

The same problem.
EV5 with Kia connect China version.

This error message is from kia server? Maybe they have updated the api or banned the integration?

More likely api changed.  Someone needs to sniff the traffic and create a pull request with the fix.

More likely api changed. Someone needs to sniff the traffic and create a pull request with the fix.

Tried sniffing the request and app returned error. Mitm has been restricted when app visiting “prd.cn-ccapi.kia.com”. Maybe app coe should be modified before sniffing.

What do we need to do? I'm here to help.

Sniff the traffic from the native app or website.  Compare to our code in the api repo and make any corrections.

Sniff the traffic from the native app or website. Compare to our code in the api repo and make any corrections.
Tried but failed. Once MitM, kia connect prevented me from logging in and returned error.

Sniff the traffic from the native app or website. Compare to our code in the api repo and make any corrections.

Tried but failed. Once MitM on iOS, kia connect prevented me from logging in and returned error

Knox player to run android with Charles proxy is my usual approach but each apps different.
If a website exists in China for Kia Hyundai can also just use browsers developer mode.

I looked around and it looks like it was intercepted. 😟

图挂了

这个只是网络信息，数据包内容没有看到。

这个只是网络信息，数据包内容没有看到。

我不知道他是不是拦截代理方式打开kia Connect 抓包里面就是没有数据 还是我的抓包方式有问题

这个只是网络信息，数据包内容没有看到。

我不知道他是不是拦截代理方式打开kia Connect 抓包里面就是没有数据 还是我的抓包方式有问题

是的，我在iOS上面一样的，应该是对SSL证书进行了校验

no way to support China EV?

Yes can be done but someone in China needs to take this on.


We need a test to ensure that setup.py is capturing all the modules.

In #561 we ended up shipping a pypi version that didn't have a module deepforest.models. This is because it existed in the git repo and so passed all the tests, but wasn't properly passing. Is there a way to mock setup.py to create a tarball and then check if all the modules are in the tarball?
The key line is https://github.com/weecology/DeepForest/blob/450e051ef41f1812a64be72302d662c952d1d0fc/setup.py#L53

Just came across this recommendation for directory structure to avoid this sort of issue:
https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-structure.html#the-src-layout-and-testing


TrueLayerPaymentsSDKWrapper.configure ~crashes~ throws an error on Android when calles second time (but works fine on iOS)

Hey,
When I initialize library with different environment using TrueLayerPaymentsSDKWrapper.configure(Environment.Sandbox) (or .Production) - it works fine on iOS.
But ~crashes~ returns cryptic error on Android:
{
    "nativeStackAndroid": [],
    "userInfo": null,
    "message": "Error not specified.",
    "code": "EUNSPECIFIED"
}

Would you be able to support dynamic env switching as it's handly for testing? And if not - would be nice to get consistent behaviour & proper error message explaining reason.

Hey Oles, we currently don't support env switching on Android and it requires relaunching the app to take effect. Would definitely be handy to add in the future when we can get to it.

Hey Oles, we've looked into this crash and haven't been able to reproduce it. Can you share how you're using the configure method? For reference if you call configure twice on Android with different environments it should return an error saying that configuration has failed not crashing. This is how we recommend you use it.
    TrueLayerPaymentsSDKWrapper.configure(
          Environment.Sandbox,
          theme,
        ).then(
          () => {
            log('Configure success')
          },
          reason => {
            log(
              'Configure failed ' +
                JSON.stringify(reason.userInfo),
            )
          },
        )

Sorry, I mean't it throws an error (like on top), not crashes. console.  You can reproduce by calling .configure(...) 2 times but with different environment.

Im also getting this same issue.


Add TrackCountObservation

Add from data sheet, add NTP fields

add TrackTallyObservation with tally of all tracks if no details are given! let's shove the problem onto the analyst's table who then has to merge TTO with nestenc/nestobs.


Use globs for file list

Anyone knows why we were not just doing built/*.js and built/*.d.ts and instead listed files?

I don't recall any adverse reason.


App API Token support

https://docs.dapr.io/operations/security/app-api-token/
Hi, can we support App API Token in .NET SDK? :)
https://github.com/dapr/go-sdk/blob/40d8a4eabe2bc6244a0de6789f74d2fe6e302db9/service/http/service.go#L53
https://github.com/dapr/go-sdk/blob/40d8a4eabe2bc6244a0de6789f74d2fe6e302db9/service/http/invoke.go#L40-L46

Sdk contains authentication processing for requests from Dapr.
https://github.com/dapr/dotnet-sdk/blob/master/src/Dapr.AspNetCore/DaprAuthenticationHandler.cs
According to the documentation, DaprAuthenticationOptions.Token should read the value of the APP_API_TOKEN environment variable, but currently it reads the value of the DAPR_API_TOKEN environment variable.
Go sdk and php sdk both do this.
https://github.com/dapr/php-sdk/blob/f0d66c38fbe7ade9b7f51a432fa1f7ed1f015bf8/src/lib/Middleware/Defaults/TokenAuth.php#L20
https://github.com/dapr/go-sdk/blob/40d8a4eabe2bc6244a0de6789f74d2fe6e302db9/service/http/service.go#L53


Ability to point to php config by default

It would be great if there was a config option that could go in phinx.yml to tell it to load a different config file by default. This would just allow for simplifying the migration commands for those of us using a PHP -based config file.

Sounds like a duplicate of https://github.com/cakephp/phinx/issues/466


outdated link

This is a Bug Report

Problem:
I wrote a simple script to scan for outdated links, I got rid of a few https://github.com/kubernetes/website/pull/44571, but there are still 20 pending.
Fixing them would probably require changes to the original text, but I have no energy for that.
Proposed Solution:
cleanup
Page to Update:
https://kubernetes.io/...

1 https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#before-you-begin anchor not found in content/en/docs/concepts/workloads/pods/user-namespaces.md source: content/en/docs/tasks/configure-pod-container/user-namespaces.md:53
2 https://kubernetes.io/docs/concepts/security/pod-security-policy/#example anchor not found in content/en/docs/concepts/security/pod-security-policy.md source: content/en/docs/tasks/configure-pod-container/migrate-from-psp.md:330
3 https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#what-things-get-dns-names anchor not found in content/en/docs/concepts/services-networking/dns-pod-service.md source: content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md:32 content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md:323
4 https://kubernetes.io/docs/contribute/generate-ref-docs/quickstart/#before-you-begin anchor not found in content/en/docs/contribute/generate-ref-docs/quickstart.md source: content/en/docs/contribute/generate-ref-docs/kubernetes-components.md:16
5 https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#manually-constructing-apiserver-proxy-urls anchor not found in content/en/docs/tasks/access-application-cluster/access-cluster.md source: content/en/docs/concepts/configuration/overview.md:99
6 https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level anchor not found in content/en/docs/concepts/cluster-administration/logging.md source: content/en/docs/concepts/configuration/manage-resources-containers.md:255 content/en/docs/concepts/configuration/manage-resources-containers.md:290 content/en/docs/concepts/configuration/manage-resources-containers.md:310 content/en/docs/reference/command-line-tools-reference/feature-gates/cri-container-log-rotation.md:30
7 https://kubernetes.io/docs/concepts/configuration/secret/#service-accounts-automatically-create-and-attach-secrets-with-api-credentials anchor not found in content/en/docs/concepts/configuration/secret.md source: content/en/docs/concepts/security/controlling-access.md:182
8 https://kubernetes.io/docs/concepts/security/pod-security-policy/#users-and-groups anchor not found in content/en/docs/concepts/security/pod-security-policy.md source: content/en/docs/concepts/security/windows-security.md:33
9 https://kubernetes.io/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations anchor not found in content/en/docs/tasks/configure-pod-container/configure-gmsa.md source: content/en/docs/concepts/windows/intro.md:325
10 https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/#visit-your-new-wordpress-blog anchor not found in content/en/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume.md source: content/en/docs/tutorials/stateless-application/guestbook.md:420
11 https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune anchor not found in content/en/docs/tasks/manage-kubernetes-objects/declarative-config.md source: content/en/docs/reference/labels-annotations-taints/_index.md:152 content/en/docs/reference/labels-annotations-taints/_index.md:172 content/en/docs/reference/labels-annotations-taints/_index.md:193 content/en/docs/reference/labels-annotations-taints/_index.md:211 content/en/docs/reference/labels-annotations-taints/_index.md:227 content/en/docs/reference/labels-annotations-taints/_index.md:246
12 https://kubernetes.io/docs/concepts/configuration/overview/#container-images anchor not found in content/en/docs/concepts/configuration/overview.md source: content/en/docs/reference/kubectl/conventions.md:41
13 https://kubernetes.io/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions anchor not found in content/en/docs/concepts/overview/kubernetes-api.md source: content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md:1521 content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md:1534
14 https://kubernetes.io/docs/reference/access-authn-authz/authentication/#x509-client-certs anchor not found in content/en/docs/reference/access-authn-authz/authentication.md source: content/en/docs/concepts/security/hardening-guide/authentication-mechanisms.md:32
15 https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#sidecar-containers-and-restartpolicy anchor not found in content/en/docs/concepts/workloads/pods/init-containers.md source: content/en/docs/concepts/workloads/pods/_index.md:120
16 https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#example-states anchor not found in content/en/docs/concepts/workloads/pods/pod-lifecycle.md source: content/en/docs/concepts/workloads/controllers/job.md:320
17 https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#external-ca-mode anchor not found in content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md source: content/en/docs/reference/setup-tools/kubeadm/implementation-details.md:234
18 https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#expanded-dns-configuration anchor not found in content/en/docs/concepts/services-networking/dns-pod-service.md source: content/en/docs/reference/command-line-tools-reference/feature-gates/expanded-dns-config.md:24
19 https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#advanced-audit anchor not found in content/en/docs/tasks/debug/debug-cluster/audit.md source: content/en/docs/reference/command-line-tools-reference/feature-gates/advanced-auditing.md:24
20 https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/#disable-accelerator-metrics anchor not found in content/en/docs/concepts/cluster-administration/system-metrics.md source: content/en/docs/reference/command-line-tools-reference/feature-gates/disable-accelerator-usage-metrics.md:24

I can help with this.
/assign

/triage accepted

This is fixed in https://github.com/kubernetes/website/pull/44571
/close


Update ansible-install-configure.md

Add "sudo" for Ansible installation on Ubuntu 16.04.

@kyliel : Thanks for your contribution! The author, @iainfoulds, has been notified to review your proposed change.

Thanks, @kyliel. This looks good to merge.
#sign-off


bump 1.2.1-rc1

Signed-off-by: Dong Chen<EMAIL_ADDRESS>

LGTM
A little concerned about the integration test issue though, we should try to fix it soon.


String Encoding

How should I specify in a schema that my string field is in an unusual character encoding?

WONTFIX / INVALID. @davidmiller closing as still not quite sure what the issue is. Please feel free to re-open.


RKE1 - Retry Backoff

We want to give users more control over the RKE1 cluster provisioning process:
Retry Backoff
Currently, when RKE1 fails to provision, the error state is saved on the cluster, and Rancher continues retrying to provision the cluster. Unfortunately, there is no way for users to stop Rancher from retrying.
We want to introduce a backoff logic so Rancher can gracefully stop after the requested number of retries.

Separate issues will be made in the rancher/rke and rancher/dashboard to track relevant works.

Due to a shift in priorities, this is no longer in the v2.6.6 release plans, so moving it to Unscheduled for now.