Deltastreamer -Global bloom Index resulting Duplicates across partitions for Same record Key

Tips before filing an issue

Have you gone through our FAQs?
Yes

Join the mailing list to engage in conversations and get faster support at<EMAIL_ADDRESS>

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
We are using Deltastreamer to read from topic and  persist the data into HDFS location .
enabled the GLOBAL_BLOOM index on COPY_ON_WRITE Storage Type with UPSERT mode, as our requirement is not to have duplicates across partitions ,but we are seeing duplicates for same record key across partitions .
We tried using two of the latest hudi utilities jars mentioned below , but the same result.
hudi-utilities-bundle_2.11-0.6.0-SNAPSHOT.jar
hudi-utilities-bundle_2.11-0.5.2-SNAPSHOT.jar
To Reproduce
Steps to reproduce the behavior:

Expected behavior
No Duplicates across partitions for the same record key.
Environment Description

Hudi version :
0.5.2

Spark version :
2.2.1

Hive version :

Hadoop version :
Hadoop Version 2.7

Storage (HDFS/S3/GCS..) :
HDFS

Running on Docker? (yes/no) :
No

Additional context
Sample duplicate record :
`
scala> spark.sql("SELECT * FROM hudiTab where MBR_SYS_ID ='ABC:18:4587543:XZ:123456778'").show(5,false)
+-------------------+-------------------------+-------------------------------------------+----------------------+-----------------------------------------------------------------------+-------------------------------------------+-------+-------------+----------+-----------------------------------------------+
|_hoodie_commit_time|_hoodie_commit_seqno     |_hoodie_record_key                         |_hoodie_partition_path|_hoodie_file_name                                                      |MBR_SYS_ID                                 |INDV_ID|modifiedTs   |modifiedDt|indicators               |
|20200615211346     |20200615211346_24_1166263|ABC:18:4587543:XZ:123456778                |2020-06-15            |790b3252-c4f7-46ed-ad86-98877541b89b-0_24-22-717_20200615211346.parquet|ABC:18:4587543:XZ:123456778|                        |1592268506762|2020-06-15|[[2018-11-01,9999-12-31,N,Y,N,N,N,N,Y,1,1,Y,Y]]|
|20200616161512     |20200616161512_0_2586    |ABC:18:4587543:XZ:123456778                |2020-06-16            |3a47934b-ffb7-41f5-a9a4-6b4f40519e79-0_0-22-696_20200616161512.parquet |ABC:18:4587543:XZ:123456778|                        |1592339194660|2020-06-16|[[2018-11-01,9999-12-31,N,Y,N,N,N,N,Y,1,1,Y,Y]]|
`
Property file:
hoodie.datasource.write.recordkey.field=MBR_SYS_ID
hoodie.datasource.write.partitionpath.field=modifiedDt
hoodie.datasource.write.precombine.field=modifiedTs
hoodie.index.type=GLOBAL_BLOOM
hoodie.bloom.index.update.partition.path=true
hoodie.auto.commit=false
enable.auto.commit=false
hoodie.deltastreamer.kafka.source.maxEvents=10000000
#client.id=kaas.prod.elr.edzprod.mcm.hudi.new.cow
group.id=kaas.prod.elr.edzprod.mcm.hudi.cow.dedup.full.load
bootstrap.servers=xyyyyyyz:443
metadata.broker.list=xyyyyyyz:443
auto.offset.reset=earliest
Stacktrace
our job is running every one hour reading from kafka topic and persisting into hudi dataset in HDFS location using deltastreamer .
Total 117 Million data having 49K duplicate records on record key split across 5 partitions .

@nsivabalan: Hi Siva , we had worked with you  last time on the similar kind of issue for test data ,you suggested to use new version of jar(0.5.2).
we did use the new jar in production env , resulting duplicates for same record key across partitions , could you please have look and suggest if there anything missing from our end .
127 million total records have 49 K duplicated across 5 partitions in HDFS.

@nsivabalan : Can you look into this when you get a chance ?

@prashanthpdesai this is something we don’t typically hear about. Did you write to that table at any point using non global bloom and later switch?

Could you check the spark UI and confirm that you have HoodieGlobalBloomIndex show up instead of HoodieBloomIndex?

@nsivabalan i don’t recall fixing anything around this in 0.5.2. Was there a reason why you suggested that?

yes, this should never be feasible unless you created a table as regular and then switched to global later. We have a unit test for the same exact case when we added the "update partition path" config.
https://github.com/apache/hudi/blob/master/hudi-client/src/test/java/org/apache/hudi/index/bloom/TestHoodieGlobalBloomIndex.java Check for testTagLocationWhenShouldUpdatePartitionPath .
@vinothchandar : I don't think I suggested to try out 0.5.2 with the intention that we had some fixes. And guess last time the question was bit different.

Hi @vinothchandar @nsivabalan : Thank you for checking , No we didn't write table anytime using Bloom index , we wrote as Global Bloom from Beginning .
Please find the attached spark UI , which is using Global bloom.
 .
as mentioned in the above thread , the data is partitioned on modified_dt we expect to delete the record key present in the older partition (2020-06-15) .

@prashanthpdesai to help narrow this one down (given its tricky), are you convinced that this happens when the modified_dt actually changed? is it the same keys duplicated every time . is it possible to trim the input down to those keys and give us a reproducible test case?
Also checked that nothing much has changed around this.. https://github.com/apache/hudi/blame/master/hudi-client/src/main/java/org/apache/hudi/index/bloom/HoodieGlobalBloomIndex.java#L123
for the kicks of it, also please see if you can just using master branch?.. (I don't expect it to change; just to rule things out)

@vinothchandar : Yes Vinoth ,we are using current date as our modified_dt , just pulled the latest record from recent run , which we started on 19th , have 3 partitions 19th, 20th and 21st run.
FYI: Input is kafka topic with avro format in it.
scala> spark.sql("select *  from hudimcmfulldumpnew where _hoodie_record_key='cdb:19:YYY:CS:12345678").show(false)
|_hoodie_commit_time|_hoodie_commit_seqno   |_hoodie_record_key                         |_hoodie_partition_path|_hoodie_file_name                                                     |MBR_SYS_ID                                 |INDV_ID|modifiedTs   |modifiedDt|indicators                                         |
|20200619111141     |20200619111141_5_224791|cdb:19:YYY:CS:12345678|2020-06-19|8c72d601-48e9-46c7-bd16-8e49f15dd63b-0_5-22-569_20200619111141.parquet|cdb:19:YYY:CS:12345678|       |1592580775819|2020-06-19|[[2016-10-01,2019-03-31,Y,Y,N,N,N,N,N,Y,Y,Y,1,1,1]]|
|20200621030755     |20200621030755_0_26958 |cdb:19:YYY:CS:12345678|2020-06-21|ff35e455-6bb9-4882-9fce-8c8f3a8a1677-0_0-22-582_20200621030755.parquet|cdb:19:YYY:CS:12345678|       |1592726093365|2020-06-21|[[2016-10-01,2019-03-31,Y,Y,N,N,N,N,N,Y,Y,Y,1,1,1]]|
pdesai@edgenode:/mapr/datalake/xxx/yyy/prd/mcm/mcm_hudi_cow_dedup_full_load_fix
$ ls -ltr
total 2
drwxr-sr-x. 2 xxx yyy 11 Jun 21 04:07 2020-06-21
drwxr-sr-x. 2 xxx yyy 20 Jun 21 04:08 2020-06-19
drwxr-sr-x. 2 xxx yyy  6 Jun 21 04:08 2020-06-20
have attached the .hoodie commit time for<PHONE_NUMBER>0755.commit for reference if that helps .

@prashanthpdesai  - i think you need to use below property along with global bloom. Have you tried this?
bloomIndexUpdatePartitionPath(updatePartitionPath = false)Permalink
Property: hoodie.bloom.index.update.partition.path
Only applies if index type is GLOBAL_BLOOM.
When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition.

@rvd8345 thanks for chiming in.. I think there might be a bug where the deletion on old partition is not propagating correctly..
Will update once we have a better resolution.. It was tricky.. but we are making progress.. :)

now that we have the PR up.. closing this.

Can you link a pr for this issue? @vinothchandar

https://github.com/apache/hudi/pull/1793


[WEB] Canvaskit: PathOperation.difference doesn't work properly when one geometry contains the other one

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce
Run the example code.
You have to use --web-renderer canvaskit otherwise #44572 will throw an error.
You can observe when modify second rect from Rect.fromLTRB(90, 90, 110, 149) to  Rect.fromLTRB(90, 90, 110, 150) the PathOperation.difference operation works properly.
Expected results
It should look like similar to Impeller / Skia renderer

Actual results

Code sample
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Rect Difference Example',
      home: Scaffold(
        body: CustomPaint(
          painter: RectDifferencePainter(),
        ),
      ),
    );
  }
}

class RectDifferencePainter extends CustomPainter {
  @override
  void paint(Canvas canvas, Size size) {
    canvas.drawPath(
      Path.combine(
        PathOperation.difference,
        Path()..addRect(const Rect.fromLTRB(50, 50, 150, 150)),
        Path()..addRect(const Rect.fromLTRB(90, 90, 110, 149)),
      ),
      Paint()..color = Colors.blue,
    );
  }

  @override
  bool shouldRepaint(RectDifferencePainter oldDelegate) => false;
}

Screenshots or Video
No response
Logs
No response
Flutter Doctor output
Doctor output
[√] Flutter (Channel stable, 3.7.8, on Microsoft Windows [Version 10.0.19045.2728], locale hu-HU)
    • Flutter version 3.7.8 on channel stable at C:\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 90c64ed42b (3 weeks ago), 2023-03-21 11:27:08 -0500
    • Engine revision 9aa7816315
    • Dart version 2.19.5
    • DevTools version 2.20.1

[√] Windows Version (Installed version of Windows is version 10 or higher)

[√] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at C:\Users\maRci002\AppData\Local\Android\sdk
    • Platform android-33, build-tools 33.0.0
    • Java binary at: C:\Program Files\Android\Android Studio\jbr\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-9505619)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2022 17.4.5)
    • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community
    • Visual Studio Community 2022 version 17.4.33403.182
    • Windows 10 SDK version 10.0.20348.0

[√] Android Studio (version 2022.1)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-9505619)

[√] VS Code (version 1.77.1)
    • VS Code at C:\Users\maRci002\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.44.0

[√] Connected device (4 available)
    • sdk gphone64 x86 64 (mobile) • emulator-5554 • android-x64    • Android 13 (API 33) (emulator)
    • Windows (desktop)            • windows       • windows-x64    • Microsoft Windows [Version 10.0.19045.2728]
    • Chrome (web)                 • chrome        • web-javascript • Google Chrome 112.0.5615.50
    • Edge (web)                   • edge          • web-javascript • Microsoft Edge 112.0.1722.34

[√] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Thanks for the report. Using code sample provided and running with canvaskit, seeing same behavior as reported.

 stable, master flutter doctor -v
[!] Flutter (Channel stable, 3.7.10, on macOS 12.2.1 21D62 darwin-x64, locale
    en-GB)
    • Flutter version 3.7.10 on channel stable at
      /Users/dhs/documents/fluttersdk/flutter
    ! Warning: `flutter` on your path resolves to
      /Users/dhs/Documents/Fluttersdk/flutter/bin/flutter, which is not inside
      your current Flutter SDK checkout at
      /Users/dhs/documents/fluttersdk/flutter. Consider adding
      /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path.
    ! Warning: `dart` on your path resolves to
      /Users/dhs/Documents/Fluttersdk/flutter/bin/dart, which is not inside your
      current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter.
      Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front
      of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 4b12645012 (6 days ago), 2023-04-03 17:46:48 -0700
    • Engine revision ec975089ac
    • Dart version 2.19.6
    • DevTools version 2.20.1
    • If those were intentional, you can disregard the above warnings; however
      it is recommended to use "git" directly to perform update checks and
      upgrades.

[!] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (5 available)
    • SM G975F (mobile)       • RZ8M802WY0X • android-arm64   • Android 11 (API 30)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 98.0.4758.80

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

[!] Flutter (Channel master, 3.10.0-4.0.pre.28, on macOS 12.2.1 21D62
    darwin-x64, locale en-GB)
    • Flutter version 3.10.0-4.0.pre.28 on channel master at
      /Users/dhs/documents/fluttersdk/flutter
    ! Warning: `flutter` on your path resolves to
      /Users/dhs/Documents/Fluttersdk/flutter/bin/flutter, which is not inside
      your current Flutter SDK checkout at
      /Users/dhs/documents/fluttersdk/flutter. Consider adding
      /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path.
    ! Warning: `dart` on your path resolves to
      /Users/dhs/Documents/Fluttersdk/flutter/bin/dart, which is not inside your
      current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter.
      Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front
      of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision caa4b5bc02 (16 minutes ago), 2023-04-13 01:02:58 -0400
    • Engine revision 5e595ff568
    • Dart version 3.0.0 (build 3.0.0-437.0.dev)
    • DevTools version 2.23.1
    • If those were intentional, you can disregard the above warnings; however
      it is recommended to use "git" directly to perform update checks and
      upgrades.

[!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/dhs/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for
      more details.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 13C100
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] IntelliJ IDEA Ultimate Edition (version 2021.3.2)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 65.1.4
    • Dart plugin version 213.7228

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.29.0

[✓] Connected device (3 available)
    • Darshan's iphone (mobile) • 21150b119064aecc249dfcfe05e259197461ce23 • ios
      • iOS 15.3.1 19D52
    • macOS (desktop)           • macos                                    •
      darwin-x64     • macOS 12.2.1 21D62 darwin-x64
    • Chrome (web)              • chrome                                   •
      web-javascript • Google Chrome 109.0.5414.119

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

[!] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (5 available)
    • SM G975F (mobile)       • RZ8M802WY0X • android-arm64   • Android 11 (API 30)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 98.0.4758.80

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

AFAIK we don't customize this on the web engine side, so it must be how Skia works. I'm going to accept this to verify that we're consistent with mobile.

Workaround


Add X button on channel details window

Hi,
I think should have a X button on top right of channel details window.
Something like this:

Fixed in https://github.com/btcontract/lnwallet/commit/c81b668aa6b2eb05df5804e3f67ce693ab5cd41f


Enable Macenko Normalisation for P1000 images

Describe the bug
The P1000 images have an offset that needs to be taken into consideration. Otherwise the normalisation will occur with stain vectors that are largely based on white background.
To Reproduce
Run macenko with any P1000 images and visualise output.
Expected behavior
The slide_image needs to be offset to compute the stain vectors across the wsi area.
Environment
dlup version:
How installed:
Python version:
Operating System:
Additional context
Add any other context about the problem here.

Hi @sinberlin2 , thanks for raising the issue.
I'm trying to understand it. Do you mean to say that stain vectors are hard to calculate over the WSI for P1000 images since there is a lot of background and very little tissue region in these images?

Hello Ajey, I tried to add more details in the PR. You can also check the lines I have added in the code for further clarification. It is just 3 lines basically

Awesome, I'll check it and also test it. Thanks Shannon! :)


Android back button in hybrid closes the app

Hi,
of course the normal behavior is close the app on back button, so how to avoid this action and make it go to the previous page ?

Ah, you want Android Back button to navigate back in the Web View of the hybrid app?

Ah, you want Android Back button to navigate back in the Web View of the hybrid app?

Yes exactly

Got it, we'll have to take a look at this, I'm sure there's a way to do it.

Since it's a  Webview , you can use javascript window.history.back() and override default back button press in Mainactivity.cs
I hope it helps :)

@Eilon any update on this?
Thanks

@ozzioma no update at this time. For now this is considered a low priority item but if someone wants to investigate and propose a fix, we could certainly look at including the fix in the next update.

I know i'm kind of reviving an old issue, but i managed to handle the back button using Xamarin's OnBackPressed override in MainActivity.cs.
It catches the event, and i use the already referenced hybrid project with a "bridge-like" singleton class which contains a reference to a navigation manager wrapper, which handles the history.
It works properly, but it requires a loading screen which makes the injection of the service first, otherwise it throws "not initialized" error.
Another problem occurs when you minimize and resume the app, you need to save the history somehow, or it resets.
I'm going to try saving it through the OnSaveInstance override.


CTC-2504 Add WebSpotlight samples

Add samples for newly added Management API support of Web Spotlight

Thanks for the PR @arguit! The prepared code samples look fine. I'm going to merge this.


Only Webkit works on macOS with VO

Hi there,
first of all thank you for creating this tool. I've been trying it out for the last couple of weeks and it looks really promising.
While running and testing things I realized that I can only do it with webkit. It happens locally on my mac(m1) and also on Github. Whenever I run Chrome for example, it crashes and similar thing in Github actions.
Is this an issue or a known thing? Let me know if I can help debugging or in any other way. Thank you!
Github actions with chrome:
Error: Unable to activate application: Playwright
    Command failed: /usr/bin/osascript
    66:87: execution error: Can’t get application "Playwright". (-1728)

        at activate (/Users/runner/work/screen-reader-testing/screen-reader-testing/node_modules/@guidepup/guidepup/lib/macOS/activate.js:14:15)
        at Object.voiceOver [as fn] (/Users/runner/work/screen-reader-testing/screen-reader-testing/node_modules/@guidepup/playwright/lib/voTest.js:18:13)

Locally:
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Playwright [3491]
Path:                  /Users/USER/Library/Caches/*/Playwright.app/Contents/MacOS/Playwright
Identifier:            org.webkit.Playwright
Version:               1.0 (1)
Code Type:             ARM-64 (Native)
Parent Process:        launchd [1]
User ID:               502

Date/Time:             2023-01-20 15:53:54.1100 +0100
OS Version:            macOS 12.6.2 (21G320)
Report Version:        12
Anonymous UUID:        137CA2C3-0F82-B801-679A-CDDC5FE1E2FC

Time Awake Since Boot: 1600 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace DYLD, Code 4 Symbol missing
Symbol not found: (_OBJC_CLASS_$__WKBrowserInspector)
Referenced from: '/Users/USER/Library/Caches/*/Playwright.app/Contents/MacOS/Playwright'
Expected in: '/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit'
(terminated at launch; ignore backtrace)

Thread 0 Crashed:
0   dyld                          	       0x100a1fe98 __abort_with_payload + 8
1   dyld                          	       0x100a27024 abort_with_payload_wrapper_internal + 104
2   dyld                          	       0x100a27058 abort_with_payload + 16
3   dyld                          	       0x1009dea28 dyld4::halt(char const*) + 580
4   dyld                          	       0x1009d9fb8 dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3644
5   dyld                          	       0x1009d906c start + 488

Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000006   x1: 0x0000000000000004   x2: 0x000000016f837af8   x3: 0x00000000000000d9
    x4: 0x000000016f8376f8   x5: 0x0000000000000000   x6: 0x0000000000000000   x7: 0x000000016f837170
    x8: 0x0000000000000020   x9: 0x0000000000000009  x10: 0x000000016f8377b8  x11: 0x00000000000000fd
   x12: 0x0000000000000000  x13: 0x0000000000000033  x14: 0x000000021c2182dc  x15: 0x0000000000000000
   x16: 0x0000000000000209  x17: 0x00000001009d714c  x18: 0x0000000000000000  x19: 0x0000000000000000
   x20: 0x000000016f8376f8  x21: 0x00000000000000d9  x22: 0x000000016f837af8  x23: 0x0000000000000004
   x24: 0x0000000000000006  x25: 0x000000016f8376f8  x26: 0x0000000000000400  x27: 0x0000000000000400
   x28: 0x00000000000000c0   fp: 0x000000016f8376c0   lr: 0x0000000100a27024
    sp: 0x000000016f837680   pc: 0x0000000100a1fe98 cpsr: 0x00001000
   far: 0x00000001006ec000  esr: 0x56000080  Address size fault

Binary Images:
       0x1009d4000 -        0x100a33fff dyld (*) <f0564ec5-bf9c-3107-82b3-485f927b489f> /usr/lib/dyld

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=919.9M resident=0K(0%) swapped_out_or_unallocated=919.9M(100%)
Writable regions: Total=9216K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9216K(100%)

                                VIRTUAL   REGION
REGION TYPE                        SIZE    COUNT (non-coalesced)
===========                     =======  =======
STACK GUARD                       56.0M        1
Stack                             8176K        1
VM_ALLOCATE                         16K        1
__AUTH                            1903K      172
__AUTH_CONST                      14.6M      325
__CTF                               756        1
__DATA                            9359K      309
__DATA_CONST                      11.8M      327
__DATA_DIRTY                       865K      120
__FONT_DATA                          4K        1
__LINKEDIT                       577.6M        3
__OBJC_CONST                      1650K      147
__OBJC_RO                         83.0M        1
__OBJC_RW                         3168K        1
__TEXT                           342.3M      343
__UNICODE                          592K        1
dyld private memory               1024K        1
===========                     =======  =======
TOTAL                              1.1G     1755

-----------
Full Report
-----------

{"app_name":"Playwright","timestamp":"2023-01-20 15:53:55.00 +0100","app_version":"1.0","slice_uuid":"6b41f14e-eea4-31ce-bdb8-44b0126c27f0","build_version":"1","platform":1,"bundleID":"org.webkit.Playwright","share_with_app_devs":0,"is_first_party":0,"bug_type":"309","os_version":"macOS 12.6.2 (21G320)","incident_id":"17ECD77D-A6B6-43E7-BFA0-880BDC41AB51","name":"Playwright"}
{
  "uptime" : 1600,
  "procLaunch" : "2023-01-20 15:53:54.0686 +0100",
  "procRole" : "Background",
  "version" : 2,
  "userID" : 502,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro17,1",
  "procStartAbsTime" :<PHONE_NUMBER>9,
  "coalitionID" : 2091,
  "osVersion" : {
    "train" : "macOS 12.6.2",
    "build" : "21G320",
    "releaseType" : "User"
  },
  "captureTime" : "2023-01-20 15:53:54.1100 +0100",
  "incident" : "17ECD77D-A6B6-43E7-BFA0-880BDC41AB51",
  "bug_type" : "309",
  "pid" : 3491,
  "procExitAbsTime" :<PHONE_NUMBER>3,
  "translated" : false,
  "cpuType" : "ARM-64",
  "procName" : "Playwright",
  "procPath" : "\/Users\/USER\/Library\/Caches\/*\/Playwright.app\/Contents\/MacOS\/Playwright",
  "bundleInfo" : {"CFBundleShortVersionString":"1.0","CFBundleVersion":"1","CFBundleIdentifier":"org.webkit.Playwright"},
  "storeInfo" : {"deviceIdentifierForVendor":"F0B95E8B-F6BF-5E57-AC7C-F1EBAE12EBC8","thirdParty":true},
  "parentProc" : "launchd",
  "parentPid" : 1,
  "coalitionName" : "org.webkit.Playwright",
  "crashReporterKey" : "137CA2C3-0F82-B801-679A-CDDC5FE1E2FC",
  "sip" : "enabled",
  "isCorpse" : 1,
  "exception" : {"codes":"0x0000000000000000, 0x0000000000000000","rawCodes":[0,0],"type":"EXC_CRASH","signal":"SIGABRT"},
  "termination" : {"code":4,"flags":518,"namespace":"DYLD","indicator":"Symbol missing","details":["(terminated at launch; ignore backtrace)"],"reasons":["Symbol not found: (_OBJC_CLASS_$__WKBrowserInspector)","Referenced from: '\/Users\/USER\/Library\/Caches\/*\/Playwright.app\/Contents\/MacOS\/Playwright'","Expected in: '\/System\/Library\/Frameworks\/WebKit.framework\/Versions\/A\/WebKit'"]},
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":36134,"threadState":{"x":[{"value":6},{"value":4},{"value":6165854968},{"value":217},{"value":6165853944},{"value":0},{"value":0},{"value":6165852528},{"value":32},{"value":9},{"value":6165854136},{"value":253},{"value":0},{"value":51},{"value":9061892828},{"value":0},{"value":521},{"value":4305285452,"symbolLocation":392,"symbol":"__simple_bprintf"},{"value":0},{"value":0},{"value":6165853944},{"value":217},{"value":6165854968},{"value":4},{"value":6},{"value":6165853944},{"value":1024},{"value":1024},{"value":192}],"flavor":"ARM_THREAD_STATE64","lr":{"value":4305612836},"cpsr":{"value":4096},"fp":{"value":6165853888},"sp":{"value":6165853824},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":4305583768,"matchesCrashFrame":1},"far":{"value":4302225408}},"frames":[{"imageOffset":310936,"symbol":"__abort_with_payload","symbolLocation":8,"imageIndex":0},{"imageOffset":340004,"symbol":"abort_with_payload_wrapper_internal","symbolLocation":104,"imageIndex":0},{"imageOffset":340056,"symbol":"abort_with_payload","symbolLocation":16,"imageIndex":0},{"imageOffset":43560,"symbol":"dyld4::halt(char const*)","symbolLocation":580,"imageIndex":0},{"imageOffset":24504,"symbol":"dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*)","symbolLocation":3644,"imageIndex":0},{"imageOffset":20588,"symbol":"start","symbolLocation":488,"imageIndex":0}]}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" :<PHONE_NUMBER>,
    "size" : 393216,
    "uuid" : "f0564ec5-bf9c-3107-82b3-485f927b489f",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  }
],
  "sharedCache" : {
  "base" :<PHONE_NUMBER>,
  "size" :<PHONE_NUMBER>,
  "uuid" : "be7fe6ad-4560-3ae2-883e-432f78b45062"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=919.9M resident=0K(0%) swapped_out_or_unallocated=919.9M(100%)\nWritable regions: Total=9216K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9216K(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nSTACK GUARD                       56.0M        1 \nStack                             8176K        1 \nVM_ALLOCATE                         16K        1 \n__AUTH                            1903K      172 \n__AUTH_CONST                      14.6M      325 \n__CTF                               756        1 \n__DATA                            9359K      309 \n__DATA_CONST                      11.8M      327 \n__DATA_DIRTY                       865K      120 \n__FONT_DATA                          4K        1 \n__LINKEDIT                       577.6M        3 \n__OBJC_CONST                      1650K      147 \n__OBJC_RO                         83.0M        1 \n__OBJC_RW                         3168K        1 \n__TEXT                           342.3M      343 \n__UNICODE                          592K        1 \ndyld private memory               1024K        1 \n===========                     =======  ======= \nTOTAL                              1.1G     1755 \n",
  "legacyInfo" : {
  "threadTriggered" : {

  }
},
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "5fb4245a1bbfe8005e33a1e1",
      "factorPackIds" : {

      },
      "deploymentId" : 240000021
    },
    {
      "rolloutId" : "5ffde50ce2aacd000d47a95f",
      "factorPackIds" : {

      },
      "deploymentId" : 240000242
    }
  ],
  "experiments" : [

  ]
}
}

Model: MacBookPro17,1, BootROM 8419.60.44, proc 8:4:4 processors, 16 GB, SMC
Graphics: Apple M1, Apple M1, Built-In
Display: Color LCD, 2560 x 1600 Retina, Main, MirrorOff, Online
Display: C27FG70, 1920 x 1080 (1080p FHD - Full High Definition), MirrorOff, Online
Memory Module: LPDDR4
AirPort: Wi-Fi, wl0: Mar 23 2022 19:57:59 version <IP_ADDRESS>.7.8.129 FWID 01-570be953
Bluetooth: Version (null), 0 services, 0 devices, 0 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
USB Device: USB31Bus
USB Device: USB 2.0 BILLBOARD
USB Device: USB31Bus
USB Device: USB3.0 Hub
USB Device: USB3.0 Card Reader
USB Device: USB 10/100/1000 LAN
USB Device: USB2.0 Hub
USB Device: Keyboard Hub
USB Device: USB Receiver
USB Device: USB Receiver
USB Device: Apple Keyboard
USB Device: USB Billboard Device
Thunderbolt Bus: MacBook Pro, Apple Inc.
Thunderbolt Bus: MacBook Pro, Apple Inc.

@joan-domingo I've just transferred this issue across to this playwright repo - going by the stack trace I think it's an issue with this module.
Specifically I've naively assumed all applications run by Playwright are named "Playwright". This is the case for the WebKit browser, but not the Chrome ones it seems. See https://github.com/guidepup/guidepup-playwright/blob/main/src/voTest.ts#L17.
Browsers we need to support:
chromium, chrome, chrome-beta, msedge, msedge-beta, msedge-dev, firefox, webkit

Thanks for all the help @cmorten ! Feel free to close the issue when you are done or let me know if you want me to test it.


Update version to 0.0.10

Deploy seems to actually have failed: https://travis-ci.org/GoogleCloudPlatform/gcloud-java/jobs/85613263#L8834
Tests failed because of

com.google.gcloud.storage.testing.RemoteGcsHelper$GcsHelperException: Environment variable GCLOUD_TESTS_PROJECT_ID not set
My guess is that variables are not exported in after_success.sh but in verify.sh, and thus not visible. We either have to export them twice or add them to .travis.yml (I suppose).

Yes, we realized that.
Third option is to have integration_test_env.sh file with
# Export test env variables
export GCLOUD_TESTS_PROJECT_ID="gcloud-devel"
export GCLOUD_TESTS_KEY=$TRAVIS_BUILD_DIR/signing-tools/gcloud-devel-travis.json

and source it from both verify.sh and after_success.sh like so:
.   integration_test_env.sh

or
source ./integration_test_env.sh

Sure, either solution is fine. Putting them in travis.yml might be a more travis-style solution but fine with source :)


Cannot create a Symfony 4 Project

Running composer create-project symfony/skeleton my_project "4.0.*" -s dev produces the following:
Installing symfony/skeleton (4.0.x-dev 592f0dae3ce846fbe9a612a4cf86336f49687ab6)

Installing symfony/skeleton (4.0.x-dev 592f0da)
Cloning 592f0dae3ce846fbe9a612a4cf86336f49687ab6

Created project in my_project
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- symfony/flex v1.0.9 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.8 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.7 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.6 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.5 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.41 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.40 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.4 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.39 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.38 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.37 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.36 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.35 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.34 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.33 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.32 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.31 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.30 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.3 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.29 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.28 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.27 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.26 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.25 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.24 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.23 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.22 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.21 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.20 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.2 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.19 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.18 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.17 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.16 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.15 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.14 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.13 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.12 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.11 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.10 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.1 requires composer-plugin-api ^1.1 -> no matching package found.
- symfony/flex v1.0.0 requires composer-plugin-api ^1.1 -> no matching package found.
- Installation request for symfony/flex ^1.0 -> satisfiable by symfony/flex[v1.0.0, v1.0.1, v1.0.10, v1.0.11, v1.0.12, v1.0.13, v1.0.14, v1.0.15, v1.0.16, v1.0.17, v1.0.18, v1.0.19, v1.0.2, v1.0.20, v1.0.21, v1.0.22, v1.0.23, v1.0.24, v1.0.25, v1.0.26, v1.0.27, v1.0.28, v1.0.29, v1.0.3, v1.0.30, v1.0.31, v1.0.32, v1.0.33, v1.0.34, v1.0.35, v1.0.36, v1.0.37, v1.0.38, v1.0.39, v1.0.4, v1.0.40, v1.0.41, v1.0.5, v1.0.6, v1.0.7, v1.0.8, v1.0.9].
Potential causes:

A typo in the package name
The package is not available in a stable-enough version according to your minimum-stability setting
see https://getcomposer.org/doc/04-schema.md#minimum-stability for more details.

Read https://getcomposer.org/doc/articles/troubleshooting.md for further common problems.
Not sure if this should be here or in the Flex repository?
Thanks!

Try composer create-project -s RC symfony/skeleton my_project

That's because you're using an obsolete version of composer, try upgrading it from getcomposer.org

Thanks, upgrading composer sorted it.


Add sparse minimum eigen solver

What is the expected behavior?
Currently Qiskit supports only a NumPy eigensolver that works on dense arrays.  Typically this is used to solve for the ground state energy of some Hamiltonian model.  However, it is well known that Hamiltonians constructed from composite quantum systems are nominally very sparse.  As such, it is numerically much more efficient to use iterative sparse solvers to compute the desired eigenvalue(s).  Fortunately, SciPy includes such a solver, and should be relatively straight forward to implement.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.eigsh.html#scipy.sparse.linalg.eigsh

Has this already been resolved ?  From https://github.com/Qiskit/qiskit-terra/blob/b898359192946912bbe827bdf08bb0d1ffc9a4cf/qiskit/algorithms/eigen_solvers/numpy_eigen_solver.py#L135 it seem like yes?

Yes it seems this has been added in #6987 but this issue never closed. @nonhermitian feel free to re-open if we're missing something!


There is no explanation of "STDP" in our documentation.

There is an implementation of snntorch.functional.stdp_learner.py on the GitHub.
However, when I search for STDP in the snnTorch documentation, there are no results, and when I go to the functional page, STDP does not exist.

The file functional.stdp_learner.py is not even registered in the functional init. There are also no open issues related to STDP, is there anything we should be discussing in this regard?

Hello, I have the same problem about using STDP in snntorch, did you have any idea about how to use it?

I found a similar example here.
https://spikingjelly.readthedocs.io/zh-cn/latest/activation_based_en/stdp.html
Note that it uses spikingjelly instead of snntorch, but I think there are similarities between the two libraries.

Although it is a simple explanation, I created a document so that other people can easily provide further explanation. I'll add an explanation later if I have time.


Add markdown formatting to event message

This way the messages should look like this:
A pod in namespace staging has been deleted:
default/hello-world-aafdf56d8bc-r49fq

/lgtm  merging, thanks for the PR


[fix] 	gql ingestion tests need to be updated to ingest the nouns first before the verbs

With the introduction of batch ingestion with #999, the test data ingestion needs to be updated to ingest packages, sources, artifacts...etc before calling isDependency, IsOccurrence...etc. Currently, it will output:
{"level":"error","ts":1688125602.265793,"caller":"cmd/ingest.go:88","msg":"Error in ingesting: input: certifyScorecard CertifyScorecard ::  input: Source type \"git\" not found\n\n","stacktrace":"github.com/guacsec/guac/cmd/guacgql/cmd.ingestScorecards\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacgql/cmd/ingest.go:88\ngithub.com/guacsec/guac/cmd/guacgql/cmd.ingestData\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacgql/cmd/ingest.go:44"}
{"level":"error","ts":1688125602.266811,"caller":"cmd/ingest.go:135","msg":"Error in ingesting: input: ingestSLSA IngestSLSA :: Subject artifact not found\n\n","stacktrace":"github.com/guacsec/guac/cmd/guacgql/cmd.ingestSLSA\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacgql/cmd/ingest.go:135\ngithub.com/guacsec/guac/cmd/guacgql/cmd.ingestData\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacgql/cmd/ingest.go:45"}

The solution will be similar to how the unit tests in the inmemory backend are done.

Can pick this issue up if the approach aligns with what is expected. For instances, for this step, I just included the ingest source block, ran the ingestion locally and it gave no errors. Can you please confirm whether this is what is expected? Thanks.

Can pick this issue up if the approach aligns with what is expected. For instances, for this step, I just included the ingest source block above it, ran the ingestion locally and it gave no errors. Can you please confirm whether this is what is expected? Thanks.

Yes that is correct!

fixed with PR #1003


Notify/visualize if spending is over the monthly average

hi @gluckzhang
Thanks for adding gcp earlier.
I do have an additional idea. now that we can catalog all the cost for all gcp projects, is there a way we can alert/notify or have a alert within the plugin page instead of resorting to a slack integration/ or directly within the component such that if the associated gcp_project of that component has a spending average higher than last month, we can see it or be notified?
I think the plugin itself currently tackles multicloud costs data visualization very well already, but i believe the next step is to take the data of what we process, alert users that if they are spending over their last monthly average.
The example above is based on gcp_project_id since i know gcp environment best, but im assuming the same logic for determining if the cost was going up for a certain project-id/service-account/resource-id  logic is the same, it is just the cloud data category + resource type would be different.
Once again this is just a an idea that i think would be helpful and bring it to your radar, it doesnt even have to be a integration to slack or directly visisble in the component overview page, it could just all be shown in the plugin page after select a specific gcp-project-id. or in backstage's logic a certain group own services that are linked to their gcp-project-id.

Hi @nia-potato, thanks for opening such a great discussion! Yes, I very much agree with you about the plan: after cost aggregation and categorization, we need to better understand the data.
We are also having some discussions internally regarding savings, trends, alerts, etc. and we will take your feedback into consideration. Here is a question regarding your proposal and also another idea that we would like to hear your feedback.

When you say "monthly average", do you mean the average cost of the previous X months (based on the time range you select)? then for example, we display this monthly average as a baseline in the column chart so that you can better understand the current month's costs, and get alert if it goes above the average. Does this correctly describe what you need? Or maybe you are talking about the changes compared to the last month only and when it is larger than it, you would like to have an alert.
There are also cost optimization suggestions or insights provided by cloud providers, for example, AWS has cost Cost Optimization Hub and GCP has FinOps hub. I am not sure if such information can be fetched easily using their APIs though, it could be helpful to display some suggestions on InfraWallet so that users can get these insights from different cloud providers at a single place. What do you think of this direction?

The example above is based on gcp_project_id since i know gcp environment best, but im assuming the same logic for determining if the cost was going up for a certain project-id/service-account/resource-id logic is the same, it is just the cloud data category + resource type would be different.

Yes, this can be a generic (probably frontend) feature for any aggregations.

Once again this is just an idea that i think would be helpful and bring it to your radar

It is helpful. Thank you again for mentioning this :)

hi @gluckzhang
Thanks for the response!

by monthly average, this was exactly what i meant

do you mean the average cost of the previous X months (based on the time range you select)? then for example, we display this monthly average as a baseline in the column chart so that you can better understand the current month's costs, and get alert if it goes above the average.

I took a look at what we had, (GCP) and i dont know what your experience with your providers are, but the suggestions from FinOps hub is not that useful. What i was thinking originally is a way for us to  define what are the standards of cost optimizations are for each category we recommend the users to switch to.  For instance:

The current month's Compute Engine Cost was higher than the previous  monthly average (if we select monthly average from the above example) because each admin of each provider knows whats best to save costs based own their own infra optimizations instead of a cloud provider giving generic advice on discounts, i think a static mapping per category type which will trigger a alert only if a category was higher than average makes more sense than using FinOps hub, unless Azure's + AWSs cost optimization is really good (i may be wrong)
if we are doing this way it would just be the plugin can read a json/yaml hosted somewhere or it can even directly live within the app config, or be mounted as a seperate configmap.
If we take the output of the monthly average:

Compute Instance cost is higher than previous month
within our static list that admins define in some format, we have a entry of Compute Instance High our generic solution is to either Migrate them to Spot VMs or use GKE.

Of course what i proposed is just one way to do this.
Other thoughts:
UI:

The current overview for all providers makes sense to be all in one page, but im assuming if we were to add a alert page + average page, these would be in a separate tab.

Additional Entity Cards

Before stumbling on your plugin, i really wanted to build something similar to what you had, so i took inspiration on backstage's Cost Insight plugin however their additional configurations of defining engineer costs and additional settings didnt make sense to me. But what i thought was missing at the time was if we were able to see each group owned resource Cost insights.

If i take GCP as an example, if groupA in backstage owned these resources as type: gcp-project:

gcp-projectA
gcp-projectB
gcp-projectC

because Infrawallet already can fetch costs based on the projectIDs, it will be able to show costs in the group page as a separate entity card. Within that card, it could just utilize the current bar chart that is separate by product/Provider category but the aggregated cost data is dependent on what the group owns for each resource, which in this case would the cost data for gcp-projectA, gcp-projectB, gcp-projectC. and then a second tab for this GroupCard can be just the alerts + suggestions. This can also be extended to the component service level.
Sorry for the long response, I will also try to take on some PRs if there you think any of these are worth implementing.

Hi @nia-potato, thanks for the detailed thoughts! I understand them better now.
Yes, I also agree that many suggestions from cloud providers are sort of generic. It may be better to implement something besides them. Regarding your proposals, I am thinking of breaking them into small pieces:
For an aggregation view
The frontend can calculate the average cost of each row in a table so that it can be compared with the cost in the current period (no matter whether daily or monthly, we can always calculate an average of the selected periods, excluding the current period).
It can also be done by the backend in the future, when, for example, we would like to have a periodic task that checks and sends out alerts. For now, since the frontend gets all the needed data for any aggregations, I think we can calculate average costs in the frontend as well.
How to display alerts
It can be a separate tab, or a panel on the same page because we already support filters, it is possible to show alerts directly when some conditions are met (such as category equals Compute)
How to config alerts
I think we can rely on the plugin database so that such configurations can be easily changed at runtime, we also need to think about how to design these alerts (a simple way would be a set of conditions plus an alert message)
Additional entity cards
This is also something we are thinking of, it would be very nice to integrate InfraWallet with Catalog, this task is also relatively independent of the alert one
I will also discuss the ideas with other team members next week (they are currently on vacation). At the same time, feel free to explore the code and make a PR. Or have more discussions here to inspire us both to improve the plugin :)

@gluckzhang
Thanks for the breakdown, that makes way more sense. I will try to tinker with the codebase in the meantime.


KEY-57: Added option to rename the output file

Can one of the admins verify this patch?

ok to test

@jpoo90 / @jansensan
This line has the output value hardcoded
https://github.com/kenzanmedia/pipeline-compile-less/blob/master/test/index.spec.js#L23
Should we make sure the tests cover a custom name vs. default name?  Also, I think for default, would it make sense to use the name of the current project's package.json?

@jpoo90 / @obuckley-kenzan / @jansensan
Yes, we should have the default be the project name from the package.json.

@jpoo90 @obuckley-kenzan @csharon While I agree with the logic that it should be the project name from the package.json, I think it's not logical to set that in the context of this project. Let me explain:
We want the resulting CSS file to have the name of the project that includes and uses pipeline-compile-less, not to be named pipeline-compile-less.css.
So the logic would be to have a default value set internally, like there is currently, and then have the config be made available to override, should the dev want to set the name of the resulting CSS.
Unless you see this as possible to do from within the pipeline-compile-less project, I think it's best to eave that to be done as an override.

@jansensan
That was indeed our intent, to grab the value from the package.json of the downstream (consuming project)

I'm not sure I know right now how that could be done, but if that is possible, all the better. However, I think it may be better to have that as a separate task, just so we go one improvement at a time, rather than bundling too much together?

Agreed, let me make an issue over there, then we can merge this in.

Handyman already has a function to get the package name https://github.com/kenzanmedia/pipeline-handyman#getpackagename

@jansensan @obuckley-kenzan  I can update the tests here to validate that the generated file has the name the user provided in the config.

Oh great, always good when we've already made it.  :)

@obuckley-kenzan I'll use the master branch rather than my local one to see how this goes.

@obuckley-kenzan So as I understand, I could easily change the line outputFileName: 'concat.css' to outputFileName: handyman.getPackageName().
@jpoo90 When you say you cant change the test, do you mean I would resubmit my PR as is and you'll change the unit test or do you want me to change it beforehand?

@jansensan You can update the tests in your PR, that would be great. If you want me to help you doing that it isn't a big deal.
I would use outputFileName: handyman.getPackageName() + '.css' , the handyman pipeline will take care of that.

Changed and used outputFileName: handyman.getPackageName() + '.css'. Forced push branch, let's hope this PR gets updated properly.

Also updated unit test that was already in place.

I will merge and fix in my testing PR


update place holder tag /committee product-security

/committee security-response is not work now

Here is an example https://github.com/kubernetes/kubernetes/issues/104980

/assign @lukehinds @tallclair

/hold
We should get this fixed so that committee/security-response will work, rather than holding on to the old committee name forever.
I'm asking for help from contribex to learn how to do that.

This problem will be resolved when https://github.com/kubernetes/test-infra/pull/23747 merges and labels are reconciled.
Thanks for bringing this to our attention, @pacoxu

@tabbysable
Should I close this PR, or should we merge it?

Checking to make sure it worked:
/committee security-response

Now that it works correctly, let's close this.
Thanks again!
/close


[EventHubs] Mypy refactor

/azp run python - eventhubs - tests

Can one of the admins verify this patch?

/azp run python - eventhubs - tests


Add config options to ignore pathfinding when obstacle tokens have certain status effects, like invisibility.

Added in v0.10.11.