Analyzer response to LSP requests extremely slow w/ high latency reported

I'm intermittently having extremely slow dart analyzer performance with VS code around 10-15 times an hour.
A typical instance starts with me typing to fix an error or triggering autocomplete. When I do so, I get Loading... in the autocomplete pop-up, sometimes the VS Code status bar shows "Analyzing" with a spinner. While the analyzer is loading, error indicators in the editor window don't update (they'll report errors at locations that don't have errors) and autocomplete won't show anything. Sometimes, but not always, my editor will freeze completely until the analyzer unfreezes. The hangs usually last anywhere from 30 seconds to 3 minutes.
I suspect the issue might have something to do with how large the project is, or maybe there's something strange about our project configuration.
The analysis server diagnostics typically looks like this after freezing:

I understand that "latency" means the server is busy and can't start handling a request, is there any way to determine what it's busy doing?
Please let me know what other information I can provide or what debugging steps I can take.
Dart Info:

If providing this information as part of reporting a bug, please review the information
below to ensure it only contains things you're comfortable posting publicly.

#### General info

- Dart 3.4.4 (stable) (Wed Jun 12 15:54:31 2024 +0000) on "windows_x64"
- on windows / "Windows 10 Enterprise" 10.0 (Build 22631)
- locale is en-US

#### Project info

- sdk constraint: '>=3.3.4 <4.0.0'
- dependencies: accordion, app_links, auto_size_text, badges, barcode_widget, cached_network_image, camera, carousel_slider, chewie, collection, community_charts_flutter, confetti, configurable_expansion_tile_null_safety, connectivity_plus, data_table_2, datetime_picker_formfield_new, device_info_plus, easy_debounce, email_validator, exif, file_picker, firebase_core, firebase_core_platform_interface, firebase_messaging, flutter, flutter_barcode_scanner, flutter_cache_manager, flutter_date_pickers, flutter_expandable_table, flutter_file_dialog, flutter_keyboard_visibility, flutter_local_notifications, flutter_quill, flutter_slidable, flutter_speed_dial, flutter_spinkit, flutter_svg, flutter_typeahead, flutter_uploader, flutter_video_info, geocoding, geolocator, google_maps_flutter, googleapis, hive, horizontal_data_table, http, image, image_gallery_saver, image_network, image_picker, infinite_scroll_pagination, intl, jiffy, keyboard_actions, latlong2, liquid_pull_to_refresh, logger, material_design_icons_flutter, material_symbols_icons, mime, modal_progress_hud_nsn, multi_dropdown, neat_periodic_task, open_filex, package_info_plus, path, path_drawing, path_provider, pdf, percent_indicator, permission_handler, phone_form_field, photo_view, pointer_interceptor, printing, provider, screenshot, scroll_to_index, scrollable_positioned_list, share_plus, shared_preferences, sliding_up_panel, super_tooltip, table_calendar, timeago, toggle_switch, transparent_image, tuple, universal_html, universal_io, url_launcher, uuid, vector_math, version, video_player, video_thumbnail, visibility_detector, wakelock_plus
- dev_dependencies: flutter_launcher_icons, flutter_lints, flutter_test, integration_test, pubspec_dependency_sorter
- elided dependencies: 1

#### Process info

|  Memory | CPU | Elapsed time | Command line |
| ------: | --: | -----------: | ------------ |
|   12 MB |  -- |              | dart.exe     |
| 2828 MB |  -- |              | dart.exe     |
|   34 MB |  -- |              | dart.exe     |
|   15 MB |  -- |              | dart.exe     |
|   68 MB |  -- |              | dart.exe     |
|  167 MB |  -- |              | dart.exe     |
|   95 MB |  -- |              | dart.exe     |

Flutter Doctor:
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 3.22.3, on Microsoft Windows [Version 10.0.22631.3880], locale en-US)
[√] Windows Version (Installed version of Windows is version 10 or higher)
[√] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
[√] Chrome - develop for the web
[√] Visual Studio - develop Windows apps (Visual Studio Professional 2022 17.7.7)
[√] Android Studio (version 2024.1)
[√] VS Code (version unknown)
    X Unable to determine VS Code version.
[√] VS Code, 64-bit edition (version 1.91.1)
[√] VS Code (version 1.92.0-insider)
[√] Connected device (4 available)
[√] Network resources

@DanTup In case you can help in terms of getting enough information to be able to determine the reason for the long analysis times.

I understand that "latency" means the server is busy and can't start handling a request, is there any way to determine what it's busy doing?

Latency here is the time difference between use constructing a request to the analysis server (done in the VS Code extension) and the server starting to process it. It generally means the analysis server was busy (it's single threaded) although there is the possibility that the delay could be in the VS Code side (especially if it itself is hanging!) or affected by other VS Code extensions (since all extensions run single-threaded in a single Node process).
Temporarily enabling the analysis server instrumentation log might give some insight into what the server is doing during this period. The file can get very large and will contain contents of your files, so sharing these logs may be difficult, but if you can get a copy of one after this happens, we may be able to provide some idea of what to look for in it (in particular I wonder if something like a context rebuild is happening - something that can occur if a pubspec.yaml or package_config.json changes).

Sometimes, but not always, my editor will freeze completely until the analyzer unfreezes.

Is it only the editor that freezes when this happens, or the system in general? Have you ever been able to view Task Manager (or run code --status in a terminal) while this is happening? VS Code runs all extensions in another process to protect itself from any bad extensions ever hanging the editor (even if they're very slow or running infinite loops) so if VS Code is hanging, there might be something more.
The output of code --status might be useful when you have this project open (even when it's not hanging - although while hanging would be more useful). I notice you have 7 Dart processes in your list before - if you have multiple VS Code windows open, please do the same when running code --status so the stats of both are included.

Latency here is the time difference between use constructing a request to the analysis server (done in the VS Code extension) and the server starting to process it. It generally means the analysis server was busy (it's single threaded) although there is the possibility that the delay could be in the VS Code side (especially if it itself is hanging!) or affected by other VS Code extensions (since all extensions run single-threaded in a single Node process).

Thank you for the info. I also use VSCode for dotnet and Angular development and haven't encountered the issue there, but it's possible that it's a dart specific plugin. Think all I've got installed is the official Dart plugin and the Dart Data Class Generator.

Temporarily enabling the analysis server instrumentation log might give some insight into what the server is doing during this period.

I'll do that today and update this issue with some logs.

Is it only the editor that freezes when this happens, or the system in general? Have you ever been able to view Task Manager (or run code --status in a terminal) while this is happening? VS Code runs all extensions in another process to protect itself from any bad extensions ever hanging the editor (even if they're very slow or running infinite loops) so if VS Code is hanging, there might be something more.

The editor freezing is relatively uncommon, seems like it only happens when I save a file, status bar say something along the lines of "saving and applying code actions". I've never had the system itself freeze, and whenever I've looked at the task manager during one of these episodes, I haven't noticed anything strange. Some high-ish CPU usage, but not like it's pegged at 100% or anything.

The output of code --status might be useful when you have this project open (even when it's not hanging - although while hanging would be more useful). I notice you have 7 Dart processes in your list before - if you have multiple VS Code windows open, please do the same when running code --status so the stats of both are included.

Here's that output, not hanging currently but I'll grab it again as soon as it is:
Version:          Code 1.91.1 (f1e16e1e6214d7c44d078b1f0607b2388f29d729, 2024-07-09T22:06:49.809Z)
OS Version:       Windows_NT x64 10.0.22631
CPUs:             Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz (12 x 2592)
Memory (System):  31.76GB (12.17GB free)
VM:               0%
Screen Reader:    yes
Process Argv:     --crash-reporter-id 0ca3f427-aab5-4c5b-b043-89b5f1cceb7e
GPU Status:       2d_canvas:                              enabled
                  canvas_oop_rasterization:               enabled_on
                  direct_rendering_display_compositor:    disabled_off_ok
                  gpu_compositing:                        enabled
                  multiple_raster_threads:                enabled_on
                  opengl:                                 enabled_on
                  rasterization:                          enabled
                  raw_draw:                               disabled_off_ok
                  skia_graphite:                          disabled_off
                  video_decode:                           enabled
                  video_encode:                           enabled
                  vulkan:                                 disabled_off
                  webgl:                                  enabled
                  webgl2:                                 enabled
                  webgpu:                                 enabled

CPU %   Mem MB     PID  Process
    0      151   20420  code main
    0      110    1428  shared-process
    0      431   12092  extensionHost [1]
    0        4    3212       C:\WINDOWS\system32\cmd.exe /d /s /c ""c:\Users\morganj\.vscode\extensions\vscjava.vscode-gradle-3.15.0\lib\gradle-server.bat" 64097"
    0        6     632         C:\WINDOWS\system32\conhost.exe 0x4
    0       16   21588         "java.exe"  "-Dfile.encoding=UTF-8"    -classpath "c:\Users\morganj\.vscode\extensions\vscjava.vscode-gradle-3.15.0\lib\gradle-server.jar" com.github.badsyntax.gradle.GradleServer 64097
    0     2139   10052       C:\Users\morganj\fvm\versions\3.22.3\bin\cache\dart-sdk\bin\dart.exe language-server --protocol=lsp --client-id=VS-Code --client-version=3.92.0
    0        6   21648         C:\WINDOWS\system32\conhost.exe 0x4
    0       15   11664       java -Dfile.encoding=utf8 -jar c:\Users\morganj\.vscode\extensions\vscjava.vscode-gradle-3.15.0\lib\gradle-language-server.jar
    0        6    1492         C:\WINDOWS\system32\conhost.exe 0x4
    0       15   12484       C:\Users\morganj\fvm\versions\3.22.3\bin\cache\dart-sdk\bin\dart.exe devtools --machine --allow-embedding --dtd-uri ws://<IP_ADDRESS>:64047/9YLvBwARxGQndzcX
    0        6   20944         C:\WINDOWS\system32\conhost.exe 0x4
    0       79   12572       "C:\Program Files\Microsoft VS Code\Code.exe" "c:\Program Files\Microsoft VS Code\resources\app\extensions\json-language-features\server\dist\node\jsonServerMain" --node-ipc --clientProcessId=12092
    0        4   19860       electron-nodejs (config.js )
    0        6   14108         C:\WINDOWS\system32\conhost.exe 0x4
    0      112   26084         electron-nodejs (config.js )
    0       66   10180<EMAIL_ADDRESS>6b82d25fbc481d1d2ed7514c707d2d8834b1716bfa49e2165de7076c310ab38b /ControllerCooldownTimeout:30000 "/TelemetrySession:{\"TelemetryLevel\":\"all\",\"IsOptedIn\":false,\"HostName\":\"Default\",\"AppInsightsInstrumentationKey\":null,\"AsimovInstrumentationKey\":null,\"CollectorApiKey\":\"0c6ae279ed8443289764825290e4f9e2-1a736e7c-1324-4338-be46-fc2a58ae4d14-7255\",\"AppId\":1010,\"UserId\":\"9db3f0d7-e564-4a0d-a0d1-141ce41d6a3f\",\"Id\":\"c766d778-5823-44a2-aefb-ad2d612eed681721697226550\",\"ProcessStartTime\":133661708407598265,\"SkuName\":null,\"VSExeVersion\":null,\"BucketFiltersToEnableWatsonForFaults\":[{\"AdditionalProperties\":[{\"Key\":\"DumpType\",\"Value\":\"Heap\"}],\"Id\":\"0bf51120-427a-4d11-9fa3-8825e8a6b42b\",\"WatsonEventType\":\"VisualStudioNonFatalErrors2\",\"BucketParameterFilters\":[\"(?i)servicehub.identityhost.exe\",null,\"(?i)servicehub.host.serviceactivationfailure\",\"(?i)microsoft.servicehub.hostlib.proxyexception\",null,null,null,null,null,null]}],\"BucketFiltersToAddDumpsToFaults\":[{\"AdditionalProperties\":[{\"Key\":\"DumpType\",\"Value\":\"Heap\"}],\"Id\":\"0bf51120-427a-4d11-9fa3-8825e8a6b42b\",\"WatsonEventType\":\"VisualStudioNonFatalErrors2\",\"BucketParameterFilters\":[\"(?i)servicehub.identityhost.exe\",null,\"(?i)servicehub.host.serviceactivationfailure\",\"(?i)microsoft.servicehub.hostlib.proxyexception\",null,null,null,null,null,null]}]}"
    0       76    1932<EMAIL_ADDRESS>dotnet$C94B8CFE-E3FD-4BAF-A941-2866DBB566FE net.pipe://1018076C939D52D2717834D7A0F41DF38AD89 "/TelemetrySession:{\"TelemetryLevel\":\"all\",\"IsOptedIn\":false,\"HostName\":\"Default\",\"AppInsightsInstrumentationKey\":null,\"AsimovInstrumentationKey\":null,\"CollectorApiKey\":\"0c6ae279ed8443289764825290e4f9e2-1a736e7c-1324-4338-be46-fc2a58ae4d14-7255\",\"AppId\":1010,\"UserId\":\"9db3f0d7-e564-4a0d-a0d1-141ce41d6a3f\",\"Id\":\"c766d778-5823-44a2-aefb-ad2d612eed681721697226550\",\"ProcessStartTime\":133661708407598265,\"SkuName\":null,\"VSExeVersion\":null,\"BucketFiltersToEnableWatsonForFaults\":[{\"AdditionalProperties\":[{\"Key\":\"DumpType\",\"Value\":\"Heap\"}],\"Id\":\"0bf51120-427a-4d11-9fa3-8825e8a6b42b\",\"WatsonEventType\":\"VisualStudioNonFatalErrors2\",\"BucketParameterFilters\":[\"(?i)servicehub.identityhost.exe\",null,\"(?i)servicehub.host.serviceactivationfailure\",\"(?i)microsoft.servicehub.hostlib.proxyexception\",null,null,null,null,null,null]}],\"BucketFiltersToAddDumpsToFaults\":[{\"AdditionalProperties\":[{\"Key\":\"DumpType\",\"Value\":\"Heap\"}],\"Id\":\"0bf51120-427a-4d11-9fa3-8825e8a6b42b\",\"WatsonEventType\":\"VisualStudioNonFatalErrors2\",\"BucketParameterFilters\":[\"(?i)servicehub.identityhost.exe\",null,\"(?i)servicehub.host.serviceactivationfailure\",\"(?i)microsoft.servicehub.hostlib.proxyexception\",null,null,null,null,null,null]}]}"
    0        6    5212               C:\WINDOWS\system32\conhost.exe 0x4
    0      147   19852<EMAIL_ADDRESS>dotnet.projectSystem$C94B8CFE-E3FD-4BAF-A941-2866DBB566FE net.pipe://1018076C939D52D2717834D7A0F41DF38AD89 "/TelemetrySession:{\"TelemetryLevel\":\"all\",\"IsOptedIn\":false,\"HostName\":\"Default\",\"AppInsightsInstrumentationKey\":null,\"AsimovInstrumentationKey\":null,\"CollectorApiKey\":\"0c6ae279ed8443289764825290e4f9e2-1a736e7c-1324-4338-be46-fc2a58ae4d14-7255\",\"AppId\":1010,\"UserId\":\"9db3f0d7-e564-4a0d-a0d1-141ce41d6a3f\",\"Id\":\"c766d778-5823-44a2-aefb-ad2d612eed681721697226550\",\"ProcessStartTime\":133661708407598265,\"SkuName\":null,\"VSExeVersion\":null,\"BucketFiltersToEnableWatsonForFaults\":[{\"AdditionalProperties\":[{\"Key\":\"DumpType\",\"Value\":\"Heap\"}],\"Id\":\"0bf51120-427a-4d11-9fa3-8825e8a6b42b\",\"WatsonEventType\":\"VisualStudioNonFatalErrors2\",\"BucketParameterFilters\":[\"(?i)servicehub.identityhost.exe\",null,\"(?i)servicehub.host.serviceactivationfailure\",\"(?i)microsoft.servicehub.hostlib.proxyexception\",null,null,null,null,null,null]}],\"BucketFiltersToAddDumpsToFaults\":[{\"AdditionalProperties\":[{\"Key\":\"DumpType\",\"Value\":\"Heap\"}],\"Id\":\"0bf51120-427a-4d11-9fa3-8825e8a6b42b\",\"WatsonEventType\":\"VisualStudioNonFatalErrors2\",\"BucketParameterFilters\":[\"(?i)servicehub.identityhost.exe\",null,\"(?i)servicehub.host.serviceactivationfailure\",\"(?i)microsoft.servicehub.hostlib.proxyexception\",null,null,null,null,null,null]}]}"
    0       45     968               "C:\Program Files\dotnet\dotnet.exe"<EMAIL_ADDRESS>    0        6    6180                 C:\WINDOWS\system32\conhost.exe 0x4
    0        6   18720               C:\WINDOWS\system32\conhost.exe 0x4
    0       13   20148       C:\Users\morganj\fvm\versions\3.22.3\bin\cache\dart-sdk\bin\dart.exe tooling-daemon --machine
    0        6    5532         C:\WINDOWS\system32\conhost.exe 0x4
    0       77   23848       electron-nodejs (serverMain.js )
    0        4   24996       C:\WINDOWS\system32\cmd.exe /c C:\Users\morganj\fvm\versions\3.22.3\bin\flutter.bat debug_adapter
    0        6   26524         C:\WINDOWS\system32\conhost.exe 0x4
    0       37   26600         electron-nodejs (config.js )
    0        4   10688           C:\WINDOWS\system32\cmd.exe /c C:\Users\morganj\fvm\versions\3.22.3\bin\flutter.bat run --machine --start-paused -d emulator-5554 --devtools-server-address http://<IP_ADDRESS>:9100/ --target C:\Users\morganj\source\CAFDExGO\mobile_app\lib\main.dart
    0       62    3816             electron-nodejs (config.js )
    0        8    2416               C:\Users\morganj\AppData\Local\Android\sdk\platform-tools\adb.exe -s emulator-5554 shell -x logcat -v time -T "'07-23 10:35:50.197'"
    0        6   14060                 C:\WINDOWS\system32\conhost.exe 0x4
    0       49   13840               electron-nodejs (config.js )
    0        6   10092                 C:\WINDOWS\system32\conhost.exe 0x4
    0        6   19840             C:\WINDOWS\system32\conhost.exe 0x4
    0      181   25184       c:\Users\morganj\.vscode\extensions\ms-dotnettools.csharp-2.34.12-win32-x64\.roslyn\Microsoft.CodeAnalysis.LanguageServer.exe --logLevel Information --razorSourceGenerator c:\Users\morganj\.vscode\extensions\ms-dotnettools.csharp-2.34.12-win32-x64\.razor\Microsoft.CodeAnalysis.Razor.Compiler.dll --razorDesignTimePath c:\Users\morganj\.vscode\extensions\ms-dotnettools.csharp-2.34.12-win32-x64\.razor\Targets\Microsoft.NET.Sdk.Razor.DesignTime.targets --starredCompletionComponentPath<EMAIL_ADDRESS>--devKitDependencyPath c:\Users\morganj\.vscode\extensions\ms-dotnettools.csharp-2.34.12-win32-x64\.roslynDevKit\Microsoft.VisualStudio.LanguageServices.DevKit.dll --sessionId c766d778-5823-44a2-aefb-ad2d612eed681721697226550 --extension c:\Users\morganj\.vscode\extensions\ms-dotnettools.csharp-2.34.12-win32-x64\.xamlTools\Microsoft.VisualStudio.DesignTools.CodeAnalysis.dll --extension c:\Users\morganj\.vscode\extensions\ms-dotnettools.csharp-2.34.12-win32-x64\.xamlTools\Microsoft.VisualStudio.DesignTools.CodeAnalysis.Diagnostics.dll --telemetryLevel all --extensionLogDirectory c:\Users\morganj\AppData\Roaming\Code\logs\20240722T181345\window1\exthost\ms-dotnettools.csharp
    0        4   25404       C:\WINDOWS\system32\cmd.exe /d /s /c ""C:\Users\morganj\fvm\versions\3.22.3\bin\flutter.bat" "daemon""
    0        6    2820         C:\WINDOWS\system32\conhost.exe 0x4
    0       28   12672         electron-nodejs (config.js )
    0        6   22380           C:\Users\morganj\AppData\Local\Android\sdk\emulator\emulator.exe -avd Pixel_7_API_33 -no-snapshot-load
    0        6   11324             C:\WINDOWS\system32\conhost.exe 0x4
    0     2065   24372             C:\Users\morganj\AppData\Local\Android\sdk\emulator\qemu\windows-x86_64\qemu-system-x86_64.exe -avd Pixel_7_API_33 -no-snapshot-load
    0        6    5768               C:\Users\morganj\AppData\Local\Android\sdk\emulator\crashpad_handler.exe --no-rate-limit --database=C:\Users\morganj\AppData\Local\Temp\\AndroidEmulator\emu-crash.db --url=https://clients2.google.com/cr/report --annotation=prod=AndroidEmulator --annotation=ver=standalone-0 --initial-client-data=0x3e0,0x3e4,0x3e8,0x3bc,0xa8,0x7ff70cace3a8,0x7ff70cace3b8,0x7ff70cace378
    0       79   25880       electron-nodejs (tailwindServer.js )
    0       89   26116       electron-nodejs (languageserver.js )
    0       77   26312       electron-nodejs (eslintServer.js )
    0       74   12204  fileWatcher [1]
    0      177   19704     gpu-process
    0       42   20892     utility-network-service
    0      662   25144  window [1] (priorities_new.dart - mobile_app (Workspace) - Visual Studio Code)
    0       25   25964     crashpad-handler
    0       80   25980  ptyHost
    0       79     392       C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -command "try { . \"c:\Program Files\Microsoft VS Code\resources\app\out\vs\workbench\contrib\terminal\browser\media\shellIntegration.ps1\" } catch {}"
    0        5   30276         C:\WINDOWS\system32\cmd.exe /c ""C:\Program Files\Microsoft VS Code\bin\code.cmd" --status"
    0      101   24764           electron-nodejs (cli.js )
    0      129   26284             "C:\Program Files\Microsoft VS Code\Code.exe" --status
    0       90   14752               crashpad-handler
    0       92   19108               utility-network-service
    0      104   21316               gpu-process
    0        7   20596       conpty-agent
    0       29   24984       "C:\Program Files\dotnet\dotnet.exe" watch run C:\Users\morganj\source\CAFDExGO\APIServer-Test/CAFDExGOServer.csproj --property GenerateFullPaths=true
    0       34   26108         "C:\Program Files\dotnet\dotnet.exe" "C:\Program Files\dotnet\sdk\6.0.408\DotnetTools\dotnet-watch\6.0.408-servicing.23177.15\tools\net6.0\any\dotnet-watch.dll" run C:\Users\morganj\source\CAFDExGO\APIServer-Test/CAFDExGOServer.csproj --property GenerateFullPaths=true
    0       48    2812           "C:\Program Files\dotnet\dotnet.exe" run --no-restore C:\Users\morganj\source\CAFDExGO\APIServer-Test/CAFDExGOServer.csproj --property GenerateFullPaths=true
    0      179   25208             "C:\Users\morganj\source\CAFDExGO\APIServer-Test\bin\Debug\net6.0\CAFDExGOServer.exe"  C:\Users\morganj\source\CAFDExGO\APIServer-Test/CAFDExGOServer.csproj
    0        7   25512       conpty-agent

Workspace Stats:
|  Window (priorities_new.dart - mobile_app (Workspace) - Visual Studio Code)
|    Folder (Database): 2304 files
|      File types: svn-base(1332) sql(690) xlsx(3) db(1) db-journal(1) json(1)
|                  docx(1) js(1)
|      Conf files: tasks.json(1)
|    Folder (APIServer-Test): 5211 files
|      File types: svn-base(3446) cs(935) dll(153) pak(60) json(47) cache(10)
|                  exe(8) bin(8) vsidx(7) config(6)
|      Conf files: package.json(4) project.json(2) launch.json(1)
|                  settings.json(1) tasks.json(1) csproj(1) sln(1)
|      Launch Configs: coreclr(2)
|    Folder (mobile_app): more than 20293 files
|      File types: svn-base(2816) png(1773) dart(1737) h(1351) svg(1206)
|                  class(926) jar(919) json(417) xml(377) txt(255)
|      Conf files: cmake(25) sln(14) launch.json(1) settings.json(1)
|      Launch Configs: dart(11)

I also should have mentioned that my teammates have also complained about hangs/slowness in the same project, which is what makes me suspicious that there might be something about the project itself that is causing some issues.
In my initial efforts to debug, I went through analysis_options.yaml and denylisted everything in the workspace directory except for lib, but I still had the issue.

Some additional info:
I dug into the Analysis Server Diagnostics and saw that there are a handful of exceptions. The exceptions throw out an enormous wall of text that starts with: Exception Invalid argument(s): Missing library: package:com.infintech.cafdexgo/assessment/widgets/assessment_date_card.dart Libraries: [dart:async, dart:collection, dart:convert, dart:core, dart:_internal, dart:math, dart:typed_data, dart:isolate, dart:developer, dart:ffi, dart:_http, dart:io, ...
I verified that the indicated library exists at lib/assessment/widgets/assessment_date_card.dart
Stack trace from the diagnostics page looks like this:
#0      LinkedElementFactory.createLibraryElementForReading (package:analyzer/src/summary2/linked_element_factory.dart:153:7)
#1      LinkedElementFactory.elementOfReference (package:analyzer/src/summary2/linked_element_factory.dart:212:14)
#2      LinkedElementFactory.libraryOfUri (package:analyzer/src/summary2/linked_element_factory.dart:241:12)
#3      LinkedElementFactory.libraryOfUri2 (package:analyzer/src/summary2/linked_element_factory.dart:245:19)
#4      ResolutionReader.libraryOfUri (package:analyzer/src/summary2/bundle_reader.dart:1940:28)
#5      LibraryElementLinkedData._readLibraryOrAugmentation (package:analyzer/src/summary2/bundle_reader.dart:587:30)
#6      LibraryElementLinkedData._read (package:analyzer/src/summary2/bundle_reader.dart:536:5)
#7      ElementLinkedData.read (package:analyzer/src/summary2/bundle_reader.dart:256:5)
#8      LibraryElementLinkedData.read (package:analyzer/src/summary2/bundle_reader.dart:520:13)
#9      LibraryElementImpl.exportNamespace (package:analyzer/src/dart/element/element.dart:4176:17)
#10     LibraryElementExtensions.exportedExtensions (package:analysis_server/src/utilities/extensions/element.dart:61:12)
#11     _ImportLibraryContainingExtension.compute (package:analysis_server/src/services/correction/dart/import_library.dart:461:42)
#12     FixProcessor._addFromProducers.compute (package:analysis_server/src/services/correction/fix_processor.dart:123:24)
#13     FixProcessor._addFromProducers (package:analysis_server/src/services/correction/fix_processor.dart:161:17)
<asynchronous suspension>
#14     FixProcessor.compute (package:analysis_server/src/services/correction/fix_processor.dart:75:5)
<asynchronous suspension>
#15     computeFixes (package:analysis_server/src/services/correction/fix_internal.dart:1587:8)
<asynchronous suspension>
#16     DartCodeActionsProducer.getFixActions (package:analysis_server/src/lsp/handlers/code_actions/dart.dart:178:23)
<asynchronous suspension>
#17     OperationPerformanceImpl.runAsync (package:analyzer/src/util/performance/operation_performance.dart:172:14)
<asynchronous suspension>
#18     CodeActionHandler.handle (package:analysis_server/src/lsp/handlers/handler_code_actions.dart:227:16)
<asynchronous suspension>
#19     ServerStateMessageHandler.handleMessage (package:analysis_server/src/lsp/handlers/handlers.dart:396:11)
<asynchronous suspension>
#20     LspAnalysisServer._handleRequestMessage (package:analysis_server/src/lsp/lsp_analysis_server.dart:1010:20)
<asynchronous suspension>
#21     LspAnalysisServer.handleMessage.<anonymous closure>.<anonymous closure> (package:analysis_server/src/lsp/lsp_analysis_server.dart:461:15)
<asynchronous suspension>
#22     OperationPerformanceImpl.runAsync (package:analyzer/src/util/performance/operation_performance.dart:172:14)
<asynchronous suspension>
#23     LspAnalysisServer.handleMessage.<anonymous closure> (package:analysis_server/src/lsp/lsp_analysis_server.dart:444:11)
<asynchronous suspension>

There were 10 similar errors listed, it seems possible that they're connected to the hangs.

@scheglov

This is the same exception again, I still suspect that this is because we try to do more than one operation concurrently in DAS: analysis and handling requests. The solution is probably using a single scheduler.

I've captured some analysis-server.log output during a time period that I was experiencing a lot of hangs. The worst hangs were happening on save, with the status bar showing "applying quick-fixes," but I did have a period after a save hang was complete where the autocomplete suggestion box was just showing "loading" and inline diagnostics weren't updating.
excerpt.txt
The excerpt that I extracted only covers ~3 minutes but it's about 5600 lines long.
The project that I'm working in does have 442 warnings and >2k infos, could that be why the analyzer is getting stuck?

Thanks for the log! Running "Fix All" (particularly with a large number of diagnostics) can be an expensive operation (because we compute fixes, apply them to temporary overlays, re-analyze all affected files, and then check if there are more fixes and repeat). From the log it seems you're running this on-save and saved multiple files, which means this operation is occurring multiple times.
That said, I see a lot of diagnostics being sent while the operation is running, and this is unexpected. While we re-analyze during this operation (to look for another round of fixes), the diagnostics from analyzing should be suppressed (because they're based on temporary overlays that don't match what the client has). I'm not sure it accounts for a lot of the time (versus all of the analysis), but it doesn't seem right so I'll try to reproduce.
Something I overlooked in your original screenshot it that the first bar in the graph was blue. Can you confirm that this was a executeCommand request? (if so, this explains the latency on the subsequent requests, since the Fix All command locks the request queue and so any new requests while it is running will be stalled until it completes).

That said, I see a lot of diagnostics being sent while the operation is running, and this is unexpected.

Digging more, this is actually from an in-process analysis started before the Fix All command. The log shows:

Modification to file
Analysis starts
Fix All starts

I actually think the issue here might not be related to Fix All itself, but could be that the analysis took a long time after you modified _priorities_utils.dart and that Fix All locks the request queue and then waits for any in-progess analysis before it proceeds. Because you're running Fix All on-save, this means the saving is blocked in VS Code on the analysis.
Is it possible you can try capturing logs and modify the file priorities/_priorities_utils.dart (but do NOT save) and wait for "Analyzing..." to disappear from the status bar before stopping the log, then see what the time difference is between the begin and end analyzing statuses?:
{"jsonrpc":"2.0","method":"$/progress","params":{"token":"ANALYZING","value":{"kind":"begin","title":"Analyzing…"}}}
{"jsonrpc":"2.0","method":"$/progress","params":{"token":"ANALYZING","value":{"kind":"end"}}}

The difference in the previous log is > 10s and this could be a result of how much code needs to be re-analyzed when you modify that file (files that depend on it).

I set disabled Fix All on save in my vscode settings, which will hopefully remove that as a factor.

Is it possible you can try capturing logs and modify the file priorities/_priorities_utils.dart (but do NOT save) and wait for "Analyzing..." to disappear from the status bar before stopping the log, then see what the time difference is between the begin and end analyzing statuses?:

Been trying to do this for a couple of days but I don't actually have any changes to make to that file at the moment, and my contrived examples don't seem to be replicating the issue.
I am having the issue really badly today in another file. Basically, any time I try to trigger code completions to pop up, the analyzer is hanging for around a minute. If I wait for it to return with analysis and change what I'm typing, it immediately freezes again.
I've restarted the language server and captured the log demonstrating this, attaching it below. What I did was:

Restart the server
Pick a variable, type a few characters, trigger completion
Waited until the suggesting window popped-up
Once it popped up, I deleted the string that I had used to trigger the completion request and typed a new one
Waited again

example.log
I do see one pair of ANALYZING entries in the log that are separated by about 45 seconds, starting on line 2899 and with 225 lines between the begin and end.
Any ideas?

We've got a directory structure that I suspect might be non-standard, looks something like this
Project Root
t pubspec.yaml
t analysis-options.yaml (ignores api-client directory)
d  lib
--- t (App's dart files)
d  api-client (Auto-generated from our backend's openapi spec)
-- t  pubspec.yaml
-- t  analysix_options.yaml (blank)
-- d  lib
------ t api.dart (library ApplicationServer.api)
------ d api (part of ApplicationServer.api) in each file

It's set up with a 2nd project root inside of our app's project root, and then has this as a dependency in the pubspec.yaml:
dependencies:
  ApplicationServer:
    path: ./api-client

It definitely seems like the hangs are happening much more frequently when trying to get diagnostics from models inside of the ApplicationServer library, and I'm not familiar enough with flutter workspace norms to tell if having nested projects where the outside project depends on the inside project could cause that.

@josephemorgan I wouldn't expect nested projects or path dependencies to cause any additional problems. Changing code in one project may result in code in the other project being re-analyzed, but I don't think this would be different to if they were in the same project (@scheglov may be able to confirm).
Are you able to capture an instrumentation log (see https://github.com/dart-lang/sdk/issues/56307#issuecomment-2248234047) for this issue? That log is captured by the server rather than the client, and includes additional logging (such as file watcher events and analysis context rebuilds). There's some analysis in your log that it's not obvious what the trigger is, so I wonder if this log gives the whole picture.
Like the logs above, the log file can get quite large and it will contain parts from your source files (such as in didOpen and didChange) events, so pleasure ensure there's nothing sensitive in any logs you share. What might be useful is the logs between the largest analysing begin/end, as well as the handful of logs prior to that which might indicate what triggered the analysis.

@josephemorgan I wouldn't expect nested projects or path dependencies to cause any additional problems. Changing code in one project may result in code in the other project being re-analyzed, but I don't think this would be different to if they were in the same project (@scheglov may be able to confirm).

Yes, when DAS sees a file change, it notifies all AnalysisDrivers about it, if some driver does not care, it is just ignored. Otherwise, if the driver read this file from a sibling package, it will update its state. It does not matter how you slice the code - same packages, nested, siblings.

Sorry, know it's been two weeks here, haven't forgotten about this, just been swamped, will try to update with the instrumentation log tomorrow.

Here's that instrumentation log:
snippet.log
At this point, I'm pretty confident that the issue ONLY happens when the analysis is operating on files that our in our api library (which is a subfolder in the general project).
LSP responses are totally normal if they're triggered on code from a Flutter library or anything from the /ourapp/lib/ directory.
When the analyzer is triggered against anything from /ourapp/ourapp-server/lib/, it hangs for 60-90 seconds. I'll add that the files in /ourapp/ourapp-server/lib/ very rarely change.
Those files are exported through a file called /ourapp/ourapp-server/lib/api.dart, which looks someting like this:
library OurAppServer.api;

import 'package:OurAppServer/model_base.dart';

import 'dart:async';
import 'dart:convert';
import 'dart:io';

import 'package:http/http.dart';
import 'package:flutter/foundation.dart';
import 'package:intl/intl.dart';
import 'package:meta/meta.dart';

part 'api/some_autogenerated_client.dart' // Few dozen of these, for each controller in our asp.net backend
part 'model/some_autogenerated_model.dart' // close to a thousand of these

And then, of course each of those client or model files have a part of OurAppServer.api
I have no idea if any of this is useful but I can't shake the suspicion that something about the api library configuration is causing issues.

close to a thousand of these

That's quite likely the source of the problem.
When the analyzer needs to resolve a file, it ends up analyzing all of the files in the library cycle [1]. Even if you only have one library in the library cycle (which appears to be the case, as long as OurAppServer/model_base.dart doesn't import OurAppServer.api), that's close to a thousand files that all have to be analyzed together.
You might want to consider breaking this single monolithic library up into multiple smaller libraries, organized in such a way that, as much as possible, they are not all dependent on everything else. The smaller a cycle in the library graph is (in terms of number of files), the faster the analysis of that cycle will be.
[1] "cycle" in the sense of a cycle in a graph, where the vertices are the libraries and the edges are defined by the imports between those libraries.

I see, that makes sense to me. The code is all auto-generated using openapi-generator-cli from an OpenAPI spec that's auto-generated using swashbuckle, but I imagine that there has to be a way to configure the generators to do some sort of modularization.

From the last few comments above, it sounds like the poor performance might be caused by the structure of the package, resulting in a lot of analysis happening during typing. I don't know how much can be done about that, although it would be useful to try and get a way to repro this.
@josephemorgan I'm not familiar with openapi-generator-cli but given the comments above, do you think it would be difficult to construct a sample project laid out like yours that triggers this bad performance that could be shared?

I've made some progress into identifying exactly what's causing the issue with my project setup.
openapi-generator-cli has two generators available for dart, the dart generator (which is described as the default since it uses dart's built-in http library), and the dart-dio generator (which uses dio instead of http). Our project has been using the dart generator, which produces output as a single, massive multi-part file.
Imagine:
// api.dart
library someNamedLibrary; // Tool produces a named library, not configurable

part "models/modelA.dart";
part "models/modelB.dart";
part "api/clientA.dart";
part "api/clientB.dart";

The individual files look something like:
// modelA.dart
class ModelA extends Whatever {
  // What you'd expect
}

I've been experimenting with some solutions, I tried generating the our api library with the dart-dio generator, which produces a barrel file instead, something like:
// api.dart
export "models/modelA.dart";
export "api/clientA.dart";

I found that the analyzer performance issues totally disappear when the files are split into multiple libraries, and then exported in a barrel file (which aligns with what @bwilkerson indicated was the likely cause of the problem)
But of course dart-dio produces very different models than dart does, and I'm not sure that I'm going to be able to get buy-in from my team to do the rework of our project required to make it work. dart-dio produces classes using the build_value library, which requires a specific interface for creating and updating instances.
So to me, it seems like my team's problem is that the dart analyzer has a much harder time analyzing a single large library declared across multiple files using part/part of than it does analyzing a single large library containing a similar number of symbols, but declared as multiple libraries and then exposed as a single library using exports.
I'm not sure if that's the dart analyzer's fault, I wouldn't necessarily expect it to perform well with that kind of setup. And I don't know how many dart/flutter developers are using openapi-generator-cli with the dart generator - perusing google results show that most people who post online recommend dart-dio. The dart generator hasn't had a commit in 2 years, which might explain why it's still producing a named library.
Anyway, if it is a goal to have the analyzer perform well on large multipart libraries, I can spend some time trying to contrive a project that reproduces the issue.
In the meantime, I have this:

the only call where elapsedSelf is anywhere near the value of elapsed is
(name: getLibraryByUri, count: 46, elapsed: 0:00:00.099968, elapsedSelf: 0:00:00.099968) under notImportedContributor,
which seems to support the theory that the analyzer is having trouble parsing the symbols in the library.
I've tried getting rid of the explicit library name manually, since Effective Dart mentions that make it harder for the analyzer to find files, but that didn't seem to have an effect.
Also, weirdly, my only coworker who has access to an arm macbook claims that the analysis server doesn't lag for nearly as long when he's using it.


Incorporate the latest pre-built models for all skills and general

Is your feature request related to a problem? Please describe.
We have the latest pre-built models for all skills and general

Closed by #900


Access Jupyter Notebook when VM was setup with SSH access and managed identity

The documentation to access Jupyter states :

Access https://<Full Domain Name or IP Address of the DSVM>:8000
Sign in with the same password as your login for Data Science Virtual Machines.
I provisioned a DSVM with my public SSH key and a managed identity. What is the user and password to access the notebook?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a8170964-1a01-a38c-e29d-8e0727b1d509
Version Independent ID: f4278485-90c5-0c64-f615-8b37c1b0ec68
Content: Samples and walkthroughs for Data Science Virtual Machines - Azure
Content Source: articles/machine-learning/data-science-virtual-machine/dsvm-samples-and-walkthroughs.md
Service: machine-learning
GitHub Login: @gopitk
Microsoft Alias: gokuma

@serbrech
Thanks for the feedback! We are currently investigating and will update you shortly.

@gopitk Hi, can you please clarify the details in the document? Thanks a lot!

If this solves the problem, then #please-close. Thank you!

@serbrech
We will now proceed to close this thread. If there are further questions regarding this matter, please respond here and @YutongTie-MSFT and we will gladly continue the discussion.


Add ChefSpec Unit Tests

Hello again Chris,
Here are the ChefSpec unit tests.
This PR also includes the following minor enhancements:

Run travis-ci against Ruby 2.1.
Run travis-ci against Chef 11 and Chef 12.
Add integration group to the Gemfile to avoid installing these gems in travis-ci.

Excellent, thank you @zuazo.


Server certificate cannot be validated

Hello,
using the latest docker image as of today I receive an error message that the server certificate cannot be validated (copied verbatim below). The server is using Let's encrypt certificates which work fine when visiting the domain by browser (it's self-hosting sentry).
Any quick suggestions on what might be going? I saw there was a fix a while back dealing with missing SSL certs and think this should still be active. I haven't dug deeper into building the image myself and for whatever reasons cannot run a shell in the container to dig around in there myself (slightly puzzled by that actually).
Thanks!

sentry_tunnel  | ERROR - the server certificate could not be validated - Host = not-shown-here

Closing this as I am using a Cloudflare worker now instead.

I'll try to dive into that I think that the root certificates are hardcoded into the image which might create some issues.

I just ran into the same problem, I tried to integrate sentry_tunnel as a service within the docker-compose stack behine a traefik proxy. I got rid of the certificate errors by building the image instead of pulling it:
services:
  sentry_tunnel:
    build: https://github.com/gbip/sentry_tunnel.git
    environment:
      - TUNNEL_IP=<IP_ADDRESS>
      - TUNNEL_REMOTE_HOST=https://oXXXXXX.ingest.sentry.io
      - TUNNEL_PATH=/sentry
      - TUNNEL_PROJECT_IDS=XXXXXX
    labels:
      - traefik.enable=true
      - traefik.http.routers.sentry-http.rule=PathPrefix(`/sentry`)
      - traefik.http.services.sentry.loadbalancer.server.port=7878

however, all POST requests to http://localhost/sentry still fail with a 400 code and no additional information in the response...

I try to take a look tonight.

Looking at this right now.

I pushed a new image that should solve the problem at least for now.


Adrienne / Fix issue localstorage-sync iframe being rendered

Changes:

Localstorage-sync is still being rendered in production https://smarttrader.deriv.com/localstorage-sync, thus this is causing logout issues in Deriv.app

Type of change

[x] Bug fix
[ ] New feature
[ ] Update feature
[ ] Refactor code
[ ] Translation to code
[ ] Translation to crowdin
[ ] Script configuration
[ ] Improve performance
[ ] Style only
[ ] Dependency update
[ ] Documentation update
[ ] Release

[Q] what about the reference to the file, don't we need to remove those too?

I didn't remove them because there will more changes involved, since we are not sure this is the root cause yet I think we can try removing the reference to the file first, since the references will return null and I've also checked there are if guards to make sure the iframe exists first


upgrade actions/setup-node to v4

Summary
Node.js 16 actions are deprecated and we want to update them to use the latest actions

Updated actions/setup-node@v3 to actions/setup-node@v4

Pull Request checklist

~- [ ] Add/update test to cover these changes~
~- [ ] Update documentation~

[x] Update CHANGELOG file

Summary by CodeRabbit

Chores

Updated GitHub Actions to use actions/setup-node@v4 for improved Node.js compatibility and support.

Documentation

Updated CHANGELOG.md to reflect the changes in GitHub Actions, ensuring they use Node.js 20.0 for better compatibility.

@justin808 the failing specs seem to be an existing issue that happened in the previous commit too . Have opened a new issue for that https://github.com/shakacode/shakapacker/issues/498

@adriangohjw latest merge to master is failing. Can you please fix?

@justin808 i suspect this is a github runners issue (i face this in my work CI sometimes too) - possible to re-run the CI?


add MINIMUM kapr calculation

Moderate:
id:if has partial show partial, otherwise show masked.
sex: show full.
race: if different show full, otherwise show masked.
others: if same show masked, otherwise show full.
Minimum:
id: if has partial show partial, otherwise show masked.
sex: show full.
race: show masked.
others: if partial show partial, otherwise show masked.


Adjust condition to access submit button

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/48778
PROPOSAL: https://github.com/Expensify/App/issues/48778#issuecomment-2401202249
Tests

QA steps section

[x] Verify that no errors appear in the JS console

Offline tests

QA Steps

Create a workspace
Invite a submitter and 2 approvers (both NON-ADMINS)
Enable approvals setting Submitter -> submitsTo approver 1 -> forwardsTo approver 2
Enable Delay submissions
Create an expense as the submitter
As the approver 1, submit & approve the report
As the approver 2, navigate to the submitter's workspace chat
See that the second-level+ non-admin approver should NOT see the 'submit' button.

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

@cretadn22 ⚠ Looks like you're missing the content for the following checkbox:

[x] I included screenshots or videos for tests on all platforms

♻ I will be moving forward with the PR Reviewer Checklist but won't be able to 🟢 Approve until the checked checkboxes are actually completed.

@cretadn22 ⚠ The code looks clean, but I found the following issues while testing the functionality (see below).

Given the following workflow:

Once the submitter (John Krakauer) posts a new expense, this is how it looks on their side:

the first one I pressed the Submit button
the second one I did not Submit yet

This is how things look on the 1st Approver's side (K3vin Brian):

This is how things look on the 2nd Approver's side (kevin.bader96+2@gmail.com):

⚠ The issues are as follows
NOTE: The pictures above and issues mentioned below are were reproduced on an existing workspace already setup and used before.

Only the submitter (John Krakauer) is able to actually Submit the expense, approver 1 and 2 while they both see the Submit button, if pressed -> they get BE error.
Once the submitter pressed Submit and submitted the expenses, 1st approver doesn't see any buttons on the expense and 2nd approver will see the Approve button.
Once the 2nd approver pressed Approve, the 1st approver will see the Approve button which allows them to approve as well.

What happens if we test exactly the same flow with exactly the same workflow member setup ?

The approver roles are reversed, meaning that in this case the 1st approver will see the Approve button once the submitter pressed Submit after they posted the expense.
If the 1st approver will press Approve on the first expense which was submitted by the submitter, then the Approve button will show up on the 2nd approver's side.
For the second expense on which the submitter did not press Submit yet, both approvers see the Submit button which should not be the case for the 2nd (as per this fix).
If 2nd approver will press the Submit button, they will get BE error, not being able to submit.
If 1st approver will press the Submit button, they are able to submit and now they will have the possibility to Approve.
If 1st approver will press Approve -> 2nd approver will be able to approve as well.

Summary
Looks like regardless of being on an already setup and previously used WS , or on a freshly setup WS with exactly the same workflow setup -> in both cases there are issues with the flow as described above.

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/748cb6e7-9e4c-45e4-8f98-a1c45aaa716e

MacOS: Desktop

@cretadn22 🟢 I retested the fix again today after the latest sync w/ main and the tests passes now:

after step (6.) in which 1st approver performes Submit and Approve on the expense, the 2nd approver will only see Approve (step 8.)

https://github.com/user-attachments/assets/748cb6e7-9e4c-45e4-8f98-a1c45aaa716e

⚠ I noticed that you still did not add the screenshots / videos as asked in https://github.com/Expensify/App/pull/51012#issuecomment-2420560276, which is currently a blocker for this PR. Once that is done, I'll be able to 🟢 Approve and move forward with the PR 👍

Not emergency, all checks were passing :/

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.


Increasing time out for SageMaker HPO tests

HPO jobs are timing out recently. Bumping the timeout for hpo jobs.
Error:
tests/test_hpo.py:121: AssertionError
------------------------------ Captured log call -------------------------------
INFO     root:resource.py:311 Condition ACK.ResourceSynced has status False, continuing...
ERROR    root:__init__.py:109 Wait for status: Completed timed out. Actual status: InProgress'

@goswamig can you also decrease this to 1?
Thx

@goswamig can you also decrease this to 1?
Thx

why is that needed ? HPO job with one training job does not make sense to me.

num training jobs is set to 2 and max parallel jobs is set to 1 which means service is running training job serially and the test might be timing out when service side waits for instances to be available(this is my suspect because the training job itself runs for 3-5 mins and the job is not timing out). Reducing the number of jobs might help with this
Since the objective of this test is to check the APIs of hyper_parameter_training_job, we are checking that passing an int in this parameter is working. rest is service side detail

HPO job with one training job does not make any difference wrt one training job. HPO runs multiple jobs and selects best parameters. The timeout should fix it, assuming its sufficient.

HPO job with one training job does not make any difference wrt one training job

max num of jobs is bound by the resource limit specified by user
boto3 ref: MaxNumberOfTrainingJobs (integer) -- [REQUIRED]
The maximum number of training jobs that a hyperparameter tuning job can launch
Example below from one of the runs, there are 2 training jobs and the second one did not start till 20 mins after first job.

first job was waiting around for instances to be ready

the situation becomes worse if one of the job fails because the service will retry it and test will timeout. (these are less frequent compared to training job taking long time to start)

If the job takes 3-5 mins why it was passing for 1 mins only before ? Did we look into it before ?
I disagree that we only check for api, if that’s the case you should not even have a test waiting for completion right?
Since I disagree on above point I still don’t see a point how a single job test HPO feature .
The intention of change is to allow enough time for training job to finish. If it’s an infra issue on service side, service  should have alarm and we should not do anything to cope with it.

Check the definition of HPO jobs here
A hyperparameter tuning job finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify.

/ok-to-test

Check the definition of HPO jobs here
A hyperparameter tuning job finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify.

This is a service side detail and the service allows user to control the number of jobs to run given the parameter ranges. It will output the best parameters from the runs it was allowed to create. If the user sets resource limit to 1 it will run only one experiment and vice versa

Since I disagree on above point I still don’t see a point how a single job test HPO feature.

Let me ask the question in a different way. Is there any reason you think the behaviour of controller is different when maxNumberOfTrainingJobs is 1 vs n?
Note: the controller is not creating or tracking the child training jobs created by the parent HPO job. This is done on service side. The controller only looks at HyperParameterTuningJobStatus to decide if it needs to requeue.

If the job takes 3-5 mins why it was passing for 1 mins only before ? Did we look into it before ?

In general, the job takes ~10mins given the instance startup time, dataset download etc. and so we started with 15 mins timeout. Increased it to 30 when we saw longer start up times on the training jobs(maybe 5-10% of the time). But we still find it is not sufficient because there are 2 training job and not one. Hence I think it is reasonable to scale down the instances instead of waiting for 45-60 mins for this test to complete. FYI: Wait time for single training job is set to 30 because of similar reason.

I disagree that we only check for api, if that’s the case you should not even have a test waiting for completion right?

I was focussing on the role of using this parameter under discussion rather than whole controller. By using this parameter in the test, we are verifying that we are able to take this int64 from the CRD type and construct a valid input for create in aws-go-sdk. The actual number does not change the controller logic.

I don't think either of these is the right fix here -

We have already increased the timeout several times to no avail. Should it really be taking 90minutes for 2 instances to spin up and complete 2 short trainingjobs even if all these steps happen sequentially?
While I agree that testing 1 vs 2 trainingJobs for HPO does not impact the quality of our integ test in any way, IMO we should not be reducing the number of trainingJobs to fix this test since that feels more like a workaround than fix.

For now, as discussed with @surajkota offline, it might be better to increase parallelTrainingJobs before we reduce the maxNumberOfTrainingJobs to ensure reduced runtime.
Eventually, we should try and root cause why this is taking so long -

Do we know if each of these runs on the same instance or different one - trying to understand why the bootstrapping time is so high.
Can we make sure we have enough account limits in place
Can we sync up with the SageMaker team about the expected runtimes ?

/test

/test all

closed in favor of https://github.com/aws-controllers-k8s/sagemaker-controller/pull/146


Support function arguments

Sometimes the command to execute depends on state other than the filename. For example, you might want to execute different commands if the target file already exist or if it doesn't. The current template format does not allow for this kind of usage.
One simple solution to support this usage is that exec accept a function:
gulp.src(myfiles)
  .pipe(
    exec(file => accessAsync(file.path, F_OK)
      .then(() => 'command to execute if file exists')
      .catch(() => 'command to execute if file does not exist'))
  );

I can put together a PR to support this over the weekend.

Good point.  Arguably all of gulp is "legacy".  I typically don't use gulp directly anymore, leveraging Webpack instead.  For the rare case where I do need to pipe files through a process I use vinyl-fs and through2, then after the last thing, I turn the stream into a promise using stream-promise, stream-to-promise, or this simple fn and then await it.
function waitStreamDone(stream) {
  return new Promise((resolve, reject) => {
    stream
      .on('error', reject)
      .on('end', resolve)
      .on('finish', resolve);
  });
}

Well, gulp still has its place in 2020, doesn't it? It is a general task runner after all, you do not need to run streams on it necessarily. webpack provides an easy solution to bundle a website, but what about other applications, for example, something like electron? Or what if you do not want to use webpack, but are like me and want to build all parts of the application separately with rollup, postcss, minify images, manage gettext files (for which I want to use this plugin btw), etc.
What do you think about a major version of gulp-exec that accepts either a plain string command, or a function (file: VinylFile) => String? Adding promise support, now that I think about it, is not critical, because most fs functions have a sync version anyways. Although there is probably some use-case for them.


fix: only add DATAHUB_SYSTEM_CLIENT envs when metadata service auth enabled

The secret which is referensed by DATAHUB_SYSTEM_CLIENT_SECRET in charts/datahub/templates/datahub-upgrade/_upgrade.tpl can only be expected to be present when global.datahub.metadata_service_authentication.enabled = true.  Fixes bug introduced in #199.
Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[x] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)

Thank you for this contribution!!


FixedChannelPool#close() can return before idle channels are closed

Expected behavior
FixedChannelPool#close() should return only after all idle channels are closed. Such behavior is expected after https://github.com/netty/netty/pull/7927. Before this PR, method was asynchronous and provided no such guarantees.
Actual behavior
FixedChannelPool#close() offloads actual closing of idle channels to GlobalEventExecutor and can return before the submitted task completes.
Steps to reproduce

Create a FixedChannelPool
Acquire and release a channel, but keep a reference to it
Call #close() on the pool
Assert channel close future is done

Minimal yet complete reproducer code (or URL to code)
The following test can be added to the FixedChannelPoolTest class to reproduce the problem:
@Test
public void testClose() {
    LocalAddress addr = new LocalAddress(LOCAL_ADDR_ID);
    Bootstrap cb = new Bootstrap();
    cb.remoteAddress(addr);
    cb.group(group).channel(LocalChannel.class);

    ServerBootstrap sb = new ServerBootstrap();
    sb.group(group)
      .channel(LocalServerChannel.class)
      .childHandler(new ChannelInitializer<LocalChannel>() {
        @Override
        public void initChannel(LocalChannel ch) throws Exception {
            ch.pipeline().addLast(new ChannelInboundHandlerAdapter());
        }
    });

    // Start server
    Channel sc = sb.bind(addr).syncUninterruptibly().channel();

    FixedChannelPool pool = new FixedChannelPool(cb, new TestChannelPoolHandler(), 2);
    Channel channel = pool.acquire().syncUninterruptibly().getNow();
    pool.release(channel).syncUninterruptibly().getNow();

    pool.close();

    assertTrue(channel.closeFuture().isDone()); // <<-- failing assertion

    sc.close().syncUninterruptibly();
}

Netty version
4.1.26+
JVM version (e.g. java -version)
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

OS version (e.g. uname -a)
Darwin MacBook-Pro.local 17.7.0 Darwin Kernel Version 17.7.0: Fri Jul  6 19:54:51 PDT 2018; root:xnu-4570.71.3~2/RELEASE_X86_64 x86_64

More details
We use Netty in a client app which hangs extremely infrequently during shutdown when updated from 4.1.22 to 4.1.26+. Shutdown sequence looks roughly like this:
fixedChannelPool.close();
eventLoopGroup.shutdownGracefully();
eventLoopGroup.terminationFuture().awaitUninterruptibly(); // this line can block forever

Unfortunately, I'm unable to reproduce such hanging with a test. Was only able to catch it once in a debugger with the following code: https://gist.github.com/lutovich/dc1fe9414e728d12b29b65c15727df93.
Stacktraces of the two blocked threads look like:
"main@1" prio=5 tid=0x1 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:502)
	  at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:253)
	  at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:33)
	  at io.netty.channel.pool. NettyPoolCloseTest.poolClose(NettyIT.java:62)
          ...

"globalEventExecutor-1-1@2059" prio=5 tid=0x1a nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:502)
	  at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:253)
	  at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:137)
	  at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:30)
	  at io.netty.channel.pool.SimpleChannelPool.close(SimpleChannelPool.java:398)
	  at io.netty.channel.pool.FixedChannelPool.access$1301(FixedChannelPool.java:40)
	  at io.netty.channel.pool.FixedChannelPool$6.run(FixedChannelPool.java:480)
	  at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:248)
	  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	  at java.lang.Thread.run(Thread.java:748)

First one is the blocked executing clientGroup.terminationFuture().awaitUninterruptibly(). Second is blocked executing channel.close().awaitUninterruptibly() in SimpleChannelPool#close(). I also have a heap dump of a hanging test run and can upload it somewhere, if needed.
I think the problem happens because FixedChannelPool#close() might exit before all pooled channels are closed and closing of the event loop group will race with closing of pooled channels.
FixedChannelPool#close() offloads actual closing of idle channels to the GlobalEventExecutor, which executes SimpleChannelPool#close(). So FixedChannelPool#close() can return to the caller before idle channels are closed. This might be a scenario that causes hanging:

Given a FixedChannelPool with an idle connection
Code calls FixedChannelPool#close() which adds a task to close all idle channels to the GlobalEventExecutor and returns
Code calls NioEventLoopGroup#close() which now races with the GlobalEventExecutor who tries to close the channel
Somehow, GlobalEventExecutor manages to add a close task to the event executor while it is shutting down and a message "An event executor terminated with non-empty task queue (1)" is logged. Tbh, I can't understand how this step is possible by looking at Netty code :(
GlobalEventExecutor is blocked in SimpleChannelPool#close() waiting for the channel close future which will never be notified because executor has been terminated
Main thread is blocked on NioEventLoopGroup#terminationFuture(). Termination future in  NioEventLoopGroup relies on GlobalEventExecutor to get notifications but GlobalEventExecutor is blocked, as described in step 5.
GlobalEventExecutor is blocked and will never receive the notification about the channel being closed. It also contains a queued task to notify the event loop termination future, which will never get executed.

If FixedChannelPool#close() returns only after all idle channels are closed then there will be no race in step 3 and the problem will hopefully be solved.

Here is a tentative fix for the described problem: https://github.com/lutovich/netty/commit/bb6b371f0e974814ce1175283c0c06e660180202. I can make it into a PR if the change is acceptable.

Thanks, @normanmaurer. I created a PR.

Fixed in https://github.com/netty/netty/pull/9226.


Replace Databricks CLI subprocess calls with Databricks SDK for Python?

https://github.com/databricks/databricks-sdk-py/
The Databricks SDK for Python is beta but at least

it can be installed with pip (unlike the newest Databricks CLI)
and users would probably get better error messages when things go wrong (instead of cryptic "subprocess exited with status 1" kind of thing)
and maybe easier to maintain even?

Wdyt @JenspederM ?

AFAIK, the SDK doesn't support asset bundles. The bundle architecture is more than just deployment, it's also validation and environment separation, all things that would have to be implemented from scratch with the sdk.
Even validation would have to be implemented separately, as they chose to use dataclasses instead of a proper validation tool such as pydantic.
I can give it another look, but I feel as if I went pretty deep the first time around 😊

Indeed https://github.com/databricks/databricks-sdk-py/issues/628
Thanks for the quick response! Maybe we can wait a bit longer and see if the issue above is addressed

#75 introduces databricks-sdk as a dependency already

@astrojuanlu I know, but changing the other parts would make it incompatible with asset bundles. And I'm still not convinced that's a smart choice.
As it is now, there is very little lock in as a user can choose to go fully for asset bundles at any time.
Instead of rewriting the whole system, I'd rather work to improve error handles by forwarding information from the Databricks cli to a larger extend.

Yes yes, it was a cross reference without any attached meaning. The upstream lack of support for DAB is still unaddressed 👍🏼


Enable filtering by an arbitrary task string

See issue 16 https://github.com/ivan-lednev/obsidian-task-archiver/issues/16 for what this solves.
I have managed to enable a new preference called "Task Pattern" in the plugin.
Let's say you have four tasks in your file as follows:
## Test Cases
- [ ] Task 1
- [x] Task 2 is completed SHOULD NOT BE ARCHIVED
- [ ] #task #context/test Task 3 is real and not completed
- [x] #task Task 4 is real and  completed

If you set your Task pattern to be "#task" as in this picture:

Then when can use the "Archiver: Archive Tasks..." command, it would only archive Task 4 because it is completed and has the #task pattern in it.
I do not think I have done a good job with Javascript encapsulation or code format protocols, so sorry about that.  Im just pushing this to offer up a feature which I would find very useful.

Hello, @wealthychef1!
Thank you for your contribution, I appreciate it very much! I merged and released the code after massaging it a little bit. Check out 0.7.0.


Pimoroni TrackBall breakout description

I've raised this draft PR for the purpose of illustrating a real-world use of the yaml definition files. My intent is to - hopefully - illustrate some of the pain points with generating a definition and, subsequently, usable code.
A couple of points to discuss and perhaps raise into issues/PRs are:

It would be nice to have a bit property for functions for cases where bitStart == bitEnd
It could be useful to group registers into logical sets with an overarching description and/or defaults. IE: the LEDs in this definition - red, green, blue and white - all share behaviour in common.
Continuous integration could build a test definition into target languages and lint the result (flake8 for Python, etc)

Playing off the point above, a register group might appear thus:
registers:
        - LEDs:
            startAddress: 0x00
            length: 8
            title: TrackBall Red, Green, Blue and White LEDs
            description: Each LED is dimmable, with 256 brightness levels from 0-255 (off-full).
            - Red:
            - Green:
            - Blue:
            - White:

Admittedly this has more practical implications for the clarity and presentation of documentation than it does for code.

In your LED case, each LED is a separate register?
Our CI process currently does generate code based on the example peripherals and checks against our cached versions and runs lint on the output.

Yes- each LED is a separate register of width 8 bits, starting at offset 0x00. So Red would be 0x00,  Green 0x01, etc. Another downside of this idea is that it's less explicit about addresses, but this is also an upside if- for example- you take a whole group of registers out of one definition and insert it into another, even if its address offset is different. (This may happen across IC families, although in practise I haven't encountered it much)

If there was a good example of this, it'll be good to consider. I'm not sure if there's too much trouble in just keeping the four registers in close proximity to one another without having to logically group them.
It could be useful to create a separate top-level section for groups of registers, which in the code-gen may result in a function with multiple parameters.
set_leds(red = 0x90, green = 0x85, blue = 0x77, white = 0x20)

What if we defined registers in the current way:
registers:
  - Red:
      address: 0x00
      length: 8
      title: Red LED
      description: Red LED saturation
  - Blue: ...

And then created a separate top-level:
groups:
  LED:
    - '#/registers/Red'
    - '#/registers/Blue'
    - '#/registers/Green'
    - '#/registers/White'

@Fleker For inspiration, OpenAPI uses tags to create logical groupings.
https://swagger.io/docs/specification/grouping-operations-with-tags/
Following their implementation we could alternatively implement grouping and shared behavior the following way:
registers:
  - Red:
      address: 0x00
      length: 8
      title: Red LED
      description: Red LED saturation
      tags:
         - LED
  - Blue: ...

And then use $ref to reuse psuedoyaml behavior across different registers.


Add LLVM via choco

Description
This adds Clang, LLD, compiler-rt, various LLVM tools
Install time is ~ 3.5 minutes
I've manually tested windows 2016/2019 and 2022 images generated.
Related issue:
https://github.com/actions/virtual-environments/issues/2530
Check list

[ ] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

/azp run windows2016, windows2019, windows2022

@henti sorry for such a late response, but it seems we forget to mention that pester tests and software readme should also be updated. I suggest adding tests for all the main tools included in the llvm choco bundle (version output should be enough). And the same tools should be mentioned in the software report, somewhere in Language and Runtime section probably

@henti sorry for such a late response, but it seems we forget to mention that pester tests and software readme should also be updated. I suggest adding tests for all the main tools included in the llvm choco bundle (version output should be enough). And the same tools should be mentioned in the software report, somewhere in Language and Runtime section probably

Thanks @miketimofeev
I've added pester tests for the main clang/clang++ binaries and update the Readme.md files accordingly.

/azp run windows2016, windows2019, windows2022

@henti thank you for your contribution! The changes will be available next week.

FWIW somehow this seems to have broken our builds on x86 (but not x64), that were previously working fine with clang-cl from the MSVC toolchain, because CMake fails at "Detecting C compiler ABI info". Still need to dig into this further...


"Mewt"ing an object from another object can break immutability

Should probably create a clone.
const a = ['foo']
const b = mewt(a)
a.push('bar')
// 2
console.log(b)
// Proxy {0: "foo", 1: "bar", length: 2}

This could be a pretty difficult one to solve especially if the object is deeply nested and/or has circular references...

Why not just clone prior to creating the Proxy?
https://github.com/sdgluck/mewt/pull/29

That wouldn't be a complete solution with the current clone implementation I guess. This still would occur:
const foo = { bar: { x: 42 } }
const immutableFoo = mewt(foo)

foo.bar.x = 43 // We can still mutate the nested props of the mewted object
console.log(immutableFoo.bar.x) // :(

If you want to do deeply nested immutability, then that seems like different issue.

@sscotth  I know I don't need to separate the object creation and the wrapping, but essentially this is what you did in the original question just with an array. Shallow cloning the object absolutely did not solve this issue in my opinion.
This works ok now:
const a = ['foo']
const b = mewt(a)
a.push('hi') // doesn't affect b

But can we call the result of a mewt truly immutable if you can still do something like this:
const a = [{ foo: bar }]
const b = mewt(a)
a[0].bar = 'ooops!' // we still mutate b since the property is not accessed through the Proxy

Yes, if we want mewt to be truly immutable, this library must recursively mewt every Object inside every Object inside every Object...
Similar to the difference between Object.freeze and the deepFreeze example near the end of its MDN article
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze

Re-opening as I wonder if it would be possible to fix this issue within the 1kb limit. Maybe with some aggressive manual minification. I'm going to give it a go soon, would be interested to hear your ideas.

Had a go on #30, but need to think about how best to tackle nested calls to $set/$unset.

Closing as nested immutability is implemented.


Added Defaulting type to experimental

Background
In order to remove Chisel._, we need to somehow tackle the difference in semantics between chisel3.<> and Chisel.<>. Our approach has been to revisit connections from first principles, and arrived at the conclusion that the FixChisel3 operators from RocketChip are an excellent starting place, where we can likely describe the behaviors of chisel3.<> and Chisel.<> as different combinations of those operators with erroring in cases which were buggy or unintuitive behavior.
The PR https://github.com/chipsalliance/chisel3/pull/2713 adds these operators with the following changes:

Any field being assigned to must have a corresponding field to assign from
No support for CustomBulkAssignable
The datatype used for recursing (left-of-connect or right-of-connect) is clarified

The final blocking issue: CustomBulkAssignable
We believe we can handle Chisel.<> and chisel3.<> now, but the FixChisel3 operators have one additional behavior which we have not figured out how to support in chisel3: BundleMap's CustomBulkAssignable.
When at first glance this appears to be a catch-all loophole to support weird connection behavior, it is only ever used for the single use-case of assigning default values to unassigned BundleMapFields (which is always associated with the field definition).
Proposals
We considered two major proposals:
Proposal 1: Relies on last-connect-semantics to give the default value, removes the strictness of the previously referenced chisel3 PR, and relies on initialization errors to give back the strictness.
Proposal 2: Enable letting Chisel types define default values, and these are used rather than erroring for the bulk operators (similar to BundleMap behavior).
After some playing around, we realized that BundleMaps would often be nested within other type hierarchies, making last-connect-semantics more painful to rely on for the default behavior, as you would need to only assign defaults to the subfields of bundlemap-kind of things. In addition, the new support for opaque types provided an excellent mechanism to create a special defaulting wrapping type. Hence, we selected Proposal 2.
Specific Reviewer Requests

Whether mdoc should be written
Whether you have any ideas for the name besides Defaulting

Contributor Checklist

[x] Did you add Scaladoc to every public function/method?
[x] Did you add at least one test demonstrating the PR?
[x] Did you delete any extraneous printlns/debugging code?
[x] Did you specify the type of improvement?
[ ] Did you add appropriate documentation in docs/src?
[x] Did you state the API impact?
[x] Did you specify the code generation impact?
[x] Did you request a desired merge strategy?
[x] Did you add text to be included in the Release Notes for this change?

Type of Improvement

new feature/API

API Impact

Adds a new experimental datatype, Defaulting which enables associating a default value with a type. In a future PR, the new connection operators will assign this default value for missing fields of Defaulting type, rather than erroring.
Release Notes

Adds a new experimental datatype, Defaulting which enables associating a default value with a type.
Reviewer Checklist (only modified by reviewer)

[ ] Did you add the appropriate labels?
[ ] Did you mark the proper milestone (Bug fix: 3.4.x, [small] API extension: 3.5.x, API modification or big change: 3.6.0)?
[ ] Did you review?
[ ] Did you check whether all relevant Contributor checkboxes have been checked?
[ ] Did you do one of the following when ready to merge:

[ ] Squash: You/ the contributor Enable auto-merge (squash), clean up the commit message, and label with Please Merge.
[ ] Merge: Ensure that contributor has cleaned up their commit history, then merge with Create a merge commit.

Whether mdoc should be written

I think we can add the mdoc when we PR the :<>= with defaulting, as this type seems pointless without that context, yes?

Whether you have any ideas for the name besides Defaulting

I am wondering whether we want the name to make this clear that this is DefaultingInAssignment, as in Wire(new BundleWithADefaulting()) is not going to assign the default value, nor is it assigned to its default on IOs.

Does this compose w things that would be opaque for other reasons (should there be a defaulting trait...?)

This feature might be pretty useful for the AXI Bundle definitions in order to provide good defaults for rarely used fields: https://github.com/chipsalliance/chisel3/pull/2441
(Currently many developers will write a "initialization" function that needs to be called in order to assign default values)

Does this compose w things that would be opaque for other reasons (should there be a defaulting trait...?)

The composition module would be to wrap a Data multiple times, e.g. Defaulting[OtherOpaqueType[T]]. Inheritance should be discouraged, because if you have a known default value for an extensible type, that default value will not be valid for subtypes.

Closed in favor of adding it directly to #2713


[Question] How to update the properties of node and apply to Cpg

I want to add a new property to Cpg node or modify an existing property, such as modifying the isExternal of Method node, and apply the modified information to Cpg, and can be persisted (serialization).  I'm not familiar with overflowdb. if this is feasible and how to do it. Thanks!

Yes, see e.g., https://github.com/joernio/joern/blob/master/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/passes/ConstClosurePass.scala#L80

Thanks! it helps me


Fixes #1913 Installing packages sometimes causes a crash

Fixes #1913 Installing packages sometimes causes a crash
Ensures package search results are only updated on the UI thread.

👍


Return type from Feather.read

With the current native implementation, we're just returning a tuple like (header, data, ctable) from Feather.read, with data being a Vector{Arrow.AbstractColumn}. While the Arrow.jl package is cute, it currently doesn't provide much real functionality other than as a convenient format to convert feather files into. My plan is to work more on Arrow.jl long-term, but I think it might be a ways out from being a serious consideration for functionality.
I'd like to propose we re-structure the Feather package to be based on the DataStreams framework, providing appropriate Sink and Source types, while retaining the higher-level Feather.read/Feather.write methods. Feather.read would then return a Data.Table, which is a small wrapper around a Vector{NullableArray}. We would then provide easy conversion routines to/from DataFrames, similar to other DataStream packages (CSV, SQLite, ODBC).
The advantages are that it's a very easy change/update and it will provide a strong basis for providing interop methods with the other DataStreams packages (e.g. return the result of of an ODBC query directly as a feather file).

In the dmb/Cxx branch I had defined a minimal Reader structure as
type Reader
    path::AbstractString
    tpt::CTablePt
    buf::IO
    columns::Vector{Column}
end

where a column is
type Column{T}
    values::AbstractVector{T}
    name::String
end

The CTablePt is a Cxx.CppPtr object for accessing the FlatBuffers table.  There are methods to convert to a DataFrame, extract individual columns, etc.
This structure is somewhat fragile because you need to keep the Reader object in scope if you want the columns to retain their values.  As soon as the buf field in the Reader is garbage collected you lose the contents of the columns.  Obviously this is not a good long-term solution but it was enough for me to use it.
I am not that familiar with the DataStreams framework.  It seems that a Feather format file needs to be available in its entirety to be usable, because the metadata is at the end of the file.  Would treating it as a stream lose some of the advantage of memory-mapping?  In other words, would the generality of a DataStream come with a petrformance hit?
By the way, it is not a good time to try to benchmark the dmb/Cxx branch because the Cxx package uses pointer_to_string excessively resulting in slowdown from the deprecated warnings.

Alright, we now have a fully functioning native implementation that reads/writes DataFrames. Closed via 243d0dbef69f967c124fdff16f1b9389dd5548d1

Not sure that this comment belongs to here, so I apologize in advance...
I was curious whether it is possible to change the form of Feather.read output in a straightforward way. Currently one gets a DataFrame filled with NullableArrays. I want to define alternative way to read .feather files to julia that would give DataArrays instead of NullableArrays. So in the heart of code in Feather.jl I replace the call to constructor of NullableArray:
arr = Feather.unwrap(convert(Ptr{typ}, pointer(m) + values.offset + bitmask_bytes), rows)
column = NullableArray{eltype(arr),1}(arr, nulls, parent)

by call to DataArray constructor:
arr = Feather.unwrap(convert(Ptr{typ}, pointer(m) + values.offset + bitmask_bytes), rows)
column = DataArray(arr, nulls)

It seems that it should fine. But after reading file and trying to access columns of a data frame I get the following error:
signal (11): Segmentation fault: 11
alignment at show.jl:1014
print_matrix at show.jl:1114
print_matrix at show.jl:1081
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1691
anonymous at replutil.jl:23
with_output_limit at /Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
writemime at replutil.jl:23
jlcall_writemime_22231 at  (unknown line)
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1691
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia.h:1331
sprint at iostream.jl:206
julia_render_22230 at  (unknown line)
julia_render_22223 at  (unknown line)
....

Closer inspection shows that despite one gets nullable arrays without any problem, ordinary arrays or dataarrays read in a similar way cause same segmentation fault.
Any ideas what's going on?
Thanks!

As I wrote above, the original design had a lightweight Feather.Reader type that provided the description of the data frame and memory-mapped contents of the columns and, if present, the bitmap for the missing values.  Feather provides an excellent opportunity to sidestep the whole issue of arrays with missing values as NullableArrays or as DataArrays because it records the number of missing values in a column as part of the column's metadata.  If that number is zero, as is frequently the case, then you can return a regular Julia  Array directly.
As time allows I will populate the dmb/Reader branch to revert to this deign.

@dmbates, it was never clear to me how you actually could "use" a Feather.Reader though. How was I supposed to traverse the values in a column? How do I index a single cell value? All of those operations require dealing with something like NullableArrays/DataArrays, or at a minimum, a Nullable type. Otherwise, the type instability kills you.
That said, we could certainly return a regular Array in cases where the null_count is 0, but I just don't see any other way around getting a Feather file into some kind of Julia structure that you can actually use.
@Flomastruk, the issue there is that we need to keep references around to the memory-mapped file, otherwise the data is invalidated when you try ot use it. Right now, we pass parent to NullableArray, which keeps that reference and makes everything safe to use. If you want ot use a DataArray, you should be fine doing it the way you showed, you just need to keep a reference to the parent field as well.

There was a DataFrame constructor that takes a Feather.Reader.  The point is that you lose information when you return a DataFrame directly and there is no cost in retaining the Feather.Reader object.

I don't think that's any different then than the current Feather.Source type. It's basically the parsed feather metadata + memory map.