fix: azure setup csp now retries SPN role assignment until it succeed

Tested with
$ azure/biganimal-csp-setup -d valerio-test-spn-new -s <SUBSCRIPTION_ID>
...
Creating Azure AD Service Principal in subscription <SUBSCRIPTION_ID>...
Updating Azure AD Service Principal application <SPN_ID> and configuring its access to Azure resources in subscription <SUBSCRIPTION_ID>...
Waiting 15 seconds for Azure AD Service Principal to propagate...
...

and with
$ azure/biganimal-csp-setup -d valerio-test-spn-new -s <SUBSCRIPTION_ID> -i <SPN_ID> -t <TENANT_ID>
...
Checking if Azure AD Service Principal application <SPN_ID> already exists in subscription <SUBSCRIPTION_ID> and tenant <TENANT_ID>...
Updating Azure AD Service Principal application <SPN_ID> in subscription <SUBSCRIPTION_ID> and tenant <TENANT_ID>...
Waiting 15 seconds for Azure AD Service Principal to propagate...
...

@valeriodelsarto how can we automate this testing?

@valeriodelsarto how can we automate this testing?

I was hoping to automate some testing while moving this into a Go library to have a single source of truth between these onboarding scripts used by the biganimal CLI and the services that runs in our Control Plane. Do you agree?
Also, some errors like this are hard to catch because they depends on Azure API replication slowness that have never happened while running these scripts in our tenant...

@valeriodelsarto I changed the PR a bit and test with @sszhao1020 in the Support Azure subscription.

@valeriodelsarto how can we automate this testing?
@nstott For AWS and GCP, we will run the script every time when we create a vcluster.

In the terratest, https://github.com/EnterpriseDB/upm-terraform/blob/main/.github/workflows/terratest.yml, we will run the script for Azure, AWS, and GCP every time. cc @Simonzhaohui


Automatically increase Build version with BTDF

Hi,
At a customer we are rewriting custom MSBuilds-targets (which were executed by BTDF), to automatically increase the Build version of the MSI, the Build version in the Deployment project and in the check in comment. Therefore, someone else, wrote custom MSBuild targets, which checked out the Deployment project, increased the Build version in the Deployment project and checked in the Deployment project.
This worked perfectly fine with BizTalk 2006 and Deployment Framework 3. Each time a MSI was generated, the Deployment project and the checkin comment were all provided with the increased version number.
Nowadays we work wth BizTalk 2013 R2 and BTDF 6 and we want to use the same targets again. After we had built in those targets, it seemed that during the generation of the MSI, the Build version was increased in the Deployment project and the check in comment, but not in the MSI.
After checking the output of the generation of the MSI I assume that, during the generation of the MSI, a script (batch file) becomes generated, which contains the still valid Build version. But that Build version becomes increased by one of the custom MSBuild targets. As a result the Deployment project and the check in comment are provided with the increased Build version, but not the MSI.
Does anyone has experience with increasing the Build version by BTDF? Does someone recognize the mentioned behaviour of BizTalk 2013 R2/BTDF versus BizTalk 2006/BTDF 3?
Looking forward to hear from you!
Best regards,
Lex Hegt
This work item was migrated from CodePlex
CodePlex work item ID: '11018'
Vote count: '1'

[BizTalkers@10/14/2015]
Hi Lex,
I use the powershell script UpdateVersionInfo.ps1 to achieve this. It is described on the following link, look for the heading "Track Your Builds" [https://biztalkalm.codeplex.com/wikipage?title=Build and deploy multiple BizTalk applications&referringTitle=Build with dependent BizTalk applications](biztalk alm)
Regards,
Rob.


32 bit support

I haven't had the time to take a deep look at the ARChon project yet, but I noticed that the current build does not include 32 bit libraries.
So why does this matter?  While one might think this is a non-factor in the modern world of 64 bit CPUs, it's worth mentioning that millions of tablets and convertible laptops sold within the last year based on Intel's Bay Trail architecture were shipped with 32 bit Windows 8(.1), and a 32 bit UEFI.

i need help because it keeps going just standing on the loading screen but when i try to remove key from manifest there isnt "such thing"in the manifest.json


fixing display errors with file upload.

Fix display problem with sound library. Uploaded sound block now get added vertically instead of horizontally.

Can u also add keys when displaying the blocks? Getting warnings with that.

Can u also add keys when displaying the blocks? Getting warnings with that.

Good call. On it then I'll merge.


Allow end-user to create schema class or property without c11y validation

When a user creates a schema item it should be accepted even if the c11y does not contain one of the words used. I.e., all schema classes and properties should be accepted.

@bobvanluijt Should the above always be true for everyone, or should there be a config setting (with a default for one or the other?) Where if the flag is set, behavior is as is current, but if the flag is not set, the above becomes possible?

A config setting is of course great. But I would see that it would be true by default. It seems that people get confused otherwise.

Implemented as part of #1062


[bug]: Unable to burn asset and at the same time I lose all units of that asset

Background
I tried to burn a certain amount of asset units (always < 100%). After executing the burn command output, this asset completely disappeared from the tapcli assets list, but the asset record in tapcli assets balances and tapcli universe roots are still fully populated.
Your environment
tapcli getinfo
{
    "version":  "0.3.0-alpha.rc2 commit=",
    "lnd_version":  "0.16.3-beta",
    "network":  "testnet3"
}

uname -mrsv
Linux 5.4.0-164-generic

bitcoind --version || btcd --version
Bitcoin Core version v24.0.0

lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

Steps to reproduce
First I minted asset type normal.
tapcli assets mint --type normal --name fantasycoin --supply 100 --meta_bytes "fantastic money"
tapcli assets mint f
tapcli assets list:
assets_list_before_burn_output.txt
after mining several blocks is time for burn
tapcli assets burn --asset_id {asset_id} --amount 50
output:
Please confirm destructive action.
Asset ID: 23a7f87be15b96427286d2bf0c4960b7f3bff2c713792ad728818e5f93b82de7
Current available balance: 100
Amount to burn: 50
 Are you sure you want to irreversibly burn (destroy, remove from circulation) the specified amount of assets?
Please answer 'yes' or 'no' and press enter:

I entered yes
output:
burn_command_output.txt
Expected behavior
Maximum amount of the asset will be reduced, I will lose the same amount along with it, but I will have access to the rest
Actual behavior
log:
log_output.txt
tapcli universe roots | grep -A 4 -B 9 fantasycoin:
universe_roots_output.txt
tapcli assets transfers:
assets_transfers_output.txt
tapcli assets list:
assets_list_output.txt
tapcli assets list --show_spent:
assets_list_show_spent_output.txt
tapcli assets balance:
assets_balance_output.txt
note: these command outputs are when the burn transaction has at least 3 confirmations in the chain

same result. I deleted the .tapd and ran the latest version -> tapcli getinfo:
{
    "version": "0.3.0-alpha.rc3 commit=",
    "lnd_version": "0.16.3-beta",
    "network": "testnet3",
    "lnd_identity_pubkey": "032c6698ed356a4b64faebe989e162399a5450af58f73dca41a80be3ee05f57e5a",
    "node_alias": "MyLightningKul",
    "block_height": 2533901,
    "block_hash": "0000000000000037ef124df2f6de6c85cda18b529999a857ce83eac6c0c217fe",
    "sync_to_chain": true
}

the procedure I subsequently carried out was completely identical to the one I described above. I am attaching txt files with outputs, obtained in the same way as above and at the same time named the same for better clarity (daemon started with --debuglevel=trace)
assets_balance_output.txt
assets_list_before_burn_output.txt
assets_list_output.txt
assets_list_show_spent_output.txt
assets_transfers_output.txt
burn_command_output.txt
log_output.txt
universe_roots_output.txt

Thanks a lot for the detailed documentation! I was now able to reproduce and a fix is on its way: https://github.com/lightninglabs/taproot-assets/pull/600


[dart2wasm] crashing dart2wasm with a flame demo

Flutter 3.22.0-14.0.pre.55 • channel master • git@github.com:flutter/flutter.git
Framework • revision a5fa351c89 (20 hours ago) • 2024-04-20 01:33:06 +0200
Engine • revision 55670b71eb
Tools • Dart 3.5.0 (build 3.5.0-80.0.dev) • DevTools 2.35.0-dev.8

https://github.com/kevmoo/flame/tree/crashing_wasm
cd examples

flutter build web --wasm

┌─ New feature ────────────────────────────────────────────────────────────────────────────┐
│   WebAssembly compilation is new. Understand the details before deploying to production. │
│   See https://flutter.dev/wasm for more information.                                     │
└──────────────────────────────────────────────────────────────────────────────────────────┘

Warning: In index.html:28: Manual service worker registration deprecated. Use flutter.js service worker bootstrapping instead.
Target dart2wasm failed: ProcessException: Process exited abnormally with exit code 64:
*NOTE*: Compilation to WasmGC is experimental.
The support may change, or be removed, with no advance notice.

Unhandled exception:
Null check operator used on a null value
#0      _ClosureTrampolineGenerator.generate (package:dart2wasm/translator.dart:1128)
#1      Translator.translate (package:dart2wasm/translator.dart:397)
#2      compileToModule (package:dart2wasm/compile.dart:186)
<asynchronous suspension>
#3      generateWasm (package:dart2wasm/generate_wasm.dart:24)
<asynchronous suspension>
#4      main (file:///Volumes/Work/s/w/ir/x/w/sdk/pkg/dart2wasm/bin/dart2wasm.dart:10)
<asynchronous suspension>
  Command: /Users/kevmoo/github/flutter/bin/cache/dart-sdk/bin/dart compile wasm --packages=.dart_tool/package_config.json --extra-compiler-option=--dart-sdk=/Users/kevmoo/github/flutter/bin/cache/dart-sdk
  --extra-compiler-option=--platform=/Users/kevmoo/github/flutter/bin/cache/flutter_web_sdk/kernel/dart2wasm_platform.dill --extra-compiler-option=--delete-tostring-package-uri=dart:ui
  --extra-compiler-option=--delete-tostring-package-uri=package:flutter --extra-compiler-option=--import-shared-memory --extra-compiler-option=--shared-memory-max-pages=32768 -Ddart.vm.product=true
  -DFLUTTER_WEB_CANVASKIT_URL=https://www.gstatic.com/flutter-canvaskit/55670b71eb00fbe80601b7227f68c1df4cae827b/ -DFLUTTER_WEB_AUTO_DETECT=false -DFLUTTER_WEB_USE_SKIA=false -DFLUTTER_WEB_USE_SKWASM=true
  --extra-compiler-option=--depfile=/Users/kevmoo/github/flame/examples/.dart_tool/flutter_build/d82f58e382ac372528cd5826f4daf5a6/dart2wasm.d -O4 --no-name-section -o
  /Users/kevmoo/github/flame/examples/.dart_tool/flutter_build/d82f58e382ac372528cd5826f4daf5a6/main.dart.wasm /Users/kevmoo/github/flame/examples/.dart_tool/flutter_build/d82f58e382ac372528cd5826f4daf5a6/main.dart
#0      RunResult.throwException (package:flutter_tools/src/base/process.dart:125:5)
#1      _DefaultProcessUtils.run (package:flutter_tools/src/base/process.dart:321:19)
<asynchronous suspension>
#2      Dart2WasmTarget.build (package:flutter_tools/src/build_system/targets/web.dart:329:5)
<asynchronous suspension>
#3      _BuildInstance._invokeInternal (package:flutter_tools/src/build_system/build_system.dart:875:9)
<asynchronous suspension>
#4      Future.wait.<anonymous closure> (dart:async/future.dart:524:21)
<asynchronous suspension>
#5      _BuildInstance.invokeTarget (package:flutter_tools/src/build_system/build_system.dart:813:32)
<asynchronous suspension>
#6      Future.wait.<anonymous closure> (dart:async/future.dart:524:21)
<asynchronous suspension>
#7      _BuildInstance.invokeTarget (package:flutter_tools/src/build_system/build_system.dart:813:32)
<asynchronous suspension>
#8      FlutterBuildSystem.build (package:flutter_tools/src/build_system/build_system.dart:642:16)
<asynchronous suspension>
#9      WebBuilder.buildWeb (package:flutter_tools/src/web/compile.dart:92:34)
<asynchronous suspension>
#10     BuildWebCommand.runCommand (package:flutter_tools/src/commands/build_web.dart:230:5)
<asynchronous suspension>
#11     FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1394:27)
<asynchronous suspension>
#12     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:153:19)
<asynchronous suspension>
#13     CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#14     FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:372:9)
<asynchronous suspension>
#15     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:153:19)
<asynchronous suspension>
#16     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:308:5)
<asynchronous suspension>
#17     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:130:9)
<asynchronous suspension>
#18     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:153:19)
<asynchronous suspension>
#19     main (package:flutter_tools/executable.dart:93:3)
<asynchronous suspension>

Compiling lib/main.dart for the Web...                             15.5s
Error: Failed to compile application for the Web.

See https://github.com/flame-engine/flame/blob/main/CONTRIBUTING.md
flutter pub global activate melos
melos bootstrap

Smaller repro:
class Base {
  final Object? a;
  Object? b;

  Base({this.a, this.b});

  @override
  String toString() {
    return 'a: $a, b: $b';
  }
}

abstract class A extends Base {
  A({required super.a, super.b});
}

class B extends A {
  B({super.a, super.b}) {}
}

void main() {
  var f1 = Base.new;
  var f2 = B.new;

  print(f1());
  print(f2());
}

Output:
Unhandled exception:
Null check operator used on a null value
#0      _ClosureTrampolineGenerator.generate (package:dart2wasm/translator.dart:1131:37)
#1      Translator.translate (package:dart2wasm/translator.dart:397:30)
#2      compileToModule (package:dart2wasm/compile.dart:186:33)
<asynchronous suspension>
#3      generateWasm (package:dart2wasm/generate_wasm.dart:24:28)
<asynchronous suspension>
#4      main (file:///usr/local/google/home/omersa/dart/sdk/sdk/pkg/dart2wasm/bin/dart2wasm.dart:10:14)
<asynchronous suspension>

I think this is a CFE bug. The named parameter of B.new tear-off isn't required but also doesn't have a default value:
  class A extends core::Object {
    final field core::Object? a;
    constructor •({required core::Object? a}) → test::A
      : test::A::a = a, super core::Object::•()
      ;
    static method _#new#tearOff({required core::Object? a}) → test::A
      return new test::A::•(a: a);
  }

  class B extends test::A {
    constructor •({core::Object? a = #C1}) → test::B
      : super test::A::•(a: a) {}
    static method _#new#tearOff({core::Object? a}) → test::B // <-------------- HERE
      return new test::B::•(a: a);
  }

  static method main() → void {
    ({a: core::Object?}) → test::B f2 = #C2;
    core::print(f2(){({a: core::Object?}) → test::B});
  }

@johnniwinther @chloestefantsova is my diagnosis of the issue correct?
I remember debugging similar issues before. Would it make sense to update the kernel verification pass to check for this? If a VariableDeclaration is not required then it should have a non-null initializer.

Here's another example of a similar issue we've debugging before: https://github.com/dart-lang/sdk/issues/53239#issuecomment-1850062260

Interesting that this code builds fine with the VM and dart2js. 🤷


Show AirPlay server name in output

Before locating an Apple TV on the network, peerflix will show this:

After detecting an Apple TV on the network, it will show the name of the Apple TV inline:

0.37.0


Enabling Suspense hangs browser

I have tried to setup an example project using the SWR hook with GraphQL, as explained in the documentation. However, enabling suspense in the options of useSWR causes the browser to hang.
I have an example project on CodeSandbox which shows this. Simply go to either screens/movies.tsx or screens/movie-details.tsx and uncomment the line with the // { suspense: true }.

I have same problem but without GraphQL. Minimum app for reproduction is here => https://codesandbox.io/s/react-suspense-with-zeitswr-r5z2k

Me too. Just a bug with this feature I think
To reproduce, copy/paste the code below:

getMessages runs a loop each time and Suspense rests in fallback

Looks like a duplicate of #46

Closing this issue as it seems to be resolved in #49 — there were a "Close" keyword, but it wasn't separated for each issue, so it only closed the first issue regarding this.


Updated tests to work with nightly 1.0.0-beta (9854143cb 2015-04-02)

I noticed that lazy-static is running the user initialization code in an unsafe block, which seems wrong. Before I fix that, though, it seems important to be able to run the tests. The changes are minor--just some updates to slicing syntax.
Cheers,
Jesse

Okay, this branch now also includes the fix that moves the user code out of the unsafe block.
J

Thanks! Never even realized that the init code runs in a unsafe context :)

Great!
On Mon, Apr 6, 2015 at 8:30 AM Marvin Löbel<EMAIL_ADDRESS>wrote:

Thanks! Never even realized that the init code runs in a unsafe context :)
--
Reply to this email directly or view it on GitHub
https://github.com/Kimundi/lazy-static.rs/pull/18#issuecomment-90040298.


Dapr cli looking for components folder in the wrong path

Expected Behavior
The application successfully persists the state.
Actual Behavior
Getting the following logs :
== APP == Got a new order! Order ID: 41
== APP == Failed to persist state.
Additionally when the app is run seeing the following error logged as a warning
== DAPR == time="2020-06-05T16:05:34.860674-07:00" level=warning msg="failed to load components: open ./components: no such file or directory" app_id=nodeapp instance=MININT-9H9J6NC.europe.corp.microsoft.com scope=dapr.runtime type=log ver=0.8.0-rc.0
Steps to Reproduce the Problem
download the 0.8.0-rc.0 version of cli and runtime
Follow the samples/1.hello-world sample from master branch https://github.com/dapr/samples/tree/master/1.hello-world
The RC cli is looking for ./components by default when the default path has been modified to ~/.dapr/components

Closing this as it's tracked here: https://github.com/dapr/dapr/issues/1655


I added types to aki-api

Hey! I just wanted to say it's cool that you added types!
I didn't know this was requested until recently, but I added types to aki-api.
I'm new to typescript, so any feedback helps!

Note: Sorry for the late response, wasn't on GitHub as such yesterday.
Thanks for doing that, now I can archive this repository. I'd made this repository only to fulfil the need for some Typings in the Package, since I mostly work with TypeScript and packages without Typings cannot be used in TypeScript projects.


Handling of TPM limit errors for Azure (x-rate-limit-reset-tokens)

Confirm this is an issue with the Python library and not an underlying OpenAI API

[X] This is an issue with the Python library

Describe the bug
We've been noticing an increasing number of TPM limit errors when calling an Azure-hosted model via the library. We have a couple of retries configured but these do not help. The reason seems to be that recently the Azure API stopped returning the Retry-After header in case of limit errors and now return x-rate-limit-reset-tokens. The library currently only knows how to handle Retry-After .
To Reproduce

Force a token limit error on an Azure hosted model
Observe the response headers. Example

[2023-12-03 18:48:27.180] DEBUG worker_pool_8 [httpcore.http11.trace:45] receive_response_headers.complete return_value=(b'HTTP/1.1', 429, b'Too Many Requests', [(b'Content-Length', b'329'), (b'Content-Type', b'application/json'), (b'x-rate-limit-reset-tokens', b'55'), (b'apim-request-id', b'<uuid>'), (b'Strict-Transport-Security', b'max-age=31536000; includeSubDomains; preload'), (b'x-content-type-options', b'nosniff'), (b'policy-id', b'DeploymentRatelimit-Token'), (b'x-ms-region', b'West US'), (b'x-ratelimit-remaining-requests', b'52'), (b'Date', b'Sun, 03 Dec 2023 18:48:27 GMT')])

The retry-after header is no longer there and instead the x-rate-limit-reset-tokens is returned.
Code snippets
No response
OS
macOS
Python version
Python 3.12
Library version
openai 1.3.6

@kristapratico could you take a look at the azure api change of behavior here?

@plamzd thanks for bringing this to our attention. The Azure team is investigating the removal of the Retry-After header. Will respond asap with any updates.

@plamzd Retry-After should be restored. I just did a quick test to confirm and do see it present in the headers. Let us know if the issue persists for you.

Hi, guys, do I understand it correctly that you don't want to support the behavior described in the docs for OpenAI here?
Do you expect it to be supported by the users of the library instead?

Hooray, thank you @kristapratico !
@pkit this library does perform exponential backoff, as suggested in that document. It does not yet provide helpers for proactive client-side rate-limiting during batch scripts; we do hope to do that in the future. I'm not sure if we have an open issue for that, but you're welcome to file one if not.


New printing functions

In WebView2 1.0.1414, new APIs have been added to control the printing process.
I can't find those functions in the *.pas files.
Could it be possible to add them?
Reference: https://learn.microsoft.com/en-us/microsoft-edge/webview2/release-notes?tabs=dotnetcsharp#101414-prerelease

Microsoft recommends that we should avoid using the prerelease features in production apps. They are considered experimental and they are subject to change :
https://learn.microsoft.com/en-us/microsoft-edge/webview2/concepts/versioning#use-a-prerelease-version-of-the-sdk-along-with-a-preview-channel-of-microsoft-edge
WebView4Delphi will support those features as soon as they are considered stable and they are included in the "Release" API.

I just added a new Wiki page about this :
https://github.com/salvadordf/WebView4Delphi/wiki/Frequently-asked-questions


drivers: dma: intel-adsp-hda: change stop ordering

Clearing the GEN/FIFORDY bits with intel_adsp_hda_disable() disconnects the DMA from DSP memory. This can block pending transfers to DSP memory to complete, so change the order so that the check of GBUSY is done first, and intel_adsp_hda_disable() is only called in the end if the DMA is idle.
This also addresses the case where DMA remains busy and stop cannot be completed and -EBUSY is returned. Without this patch, further calls to intel_adsp_hda_dma_stop() fail with no action as the GEN/FIFORDY is already cleared and the runtime-PM ref is lost forever.
Suggested-by: Peter Ujfalusi<EMAIL_ADDRESS>

Closing, this is not a correct fix.


not all templates are showing up in dataset selection dropdown

When I try to use https://www.synapse.org/#!Wiki:syn20681266/ENTITY
I don't e.g. see scRNA-seq Level 4 in the dropdown. Just Level 1:

@inodb - thanks for spotting. I was just bumping into a similar issue myself.
In the latest codebase update from Fri, we have erroneously not pulled in the latest HTAN schema; it accidentally uses and old one...
Tagging @sujaypatil96 -  highest priority fix:

update DC app HTAN schema to use latest one (note this schema is on 'main' branch; need to update develop)
confirm the set of metadata templates in the DC app contains all the components supported by the latest HTAN schema

Thanks for catching this @inodb. This happened because we hadn't updated the HTAN data model to it's latest version. The file (config.json) that controls what items are in the template drop down wasn't updated either.
To fix this, we will update the default branches on both schematic and the Data Curator App's end to ensure that this doesn't come up again.
It has been fixed for now. You can see all expected templates in the drop down.

For reference, you can also use this link to access the Data Curator App @inodb: https://shinypro.synapse.org/users/spatil/HTAN-prod/

Awesome - thanks for the quick fix @milen-sage and @sujaypatil96 !

this issue was resolved based on issue #154.


chore: clean up in-code uk translations | FC-55

Clean up after #422
make pull_translations should be used

Thanks for the pull request, @OmarIthawi! Please note that it may take us up to several weeks or months to complete a review and merge your PR.
Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.
Please let us know once your PR is ready for our review and all tests are green.

@OmarIthawi, could you please fix the tests? You've deleted files, but something is still looking for them.
The following build commands failed:
	CopyStringsFile /Users/runner/Library/Developer/Xcode/DerivedData/OpenEdX-bsdxmseilanfjjcjomwdntdeffpi/Build/Products/DebugDev-iphonesimulator/Core.framework/uk.lproj/Localizable.strings /Users/runner/work/openedx-app-ios/openedx-app-ios/Core/Core/uk.lproj/Localizable.strings (in target 'Core' from project 'Core')

@OmarIthawi, could you please fix the tests? You've deleted files, but something is still looking for them.
The following build commands failed:
	CopyStringsFile /Users/runner/Library/Developer/Xcode/DerivedData/OpenEdX-bsdxmseilanfjjcjomwdntdeffpi/Build/Products/DebugDev-iphonesimulator/Core.framework/uk.lproj/Localizable.strings /Users/runner/work/openedx-app-ios/openedx-app-ios/Core/Core/uk.lproj/Localizable.strings (in target 'Core' from project 'Core')

I tried to fix the failing test but I'm unsure how to find the step CopyStringsFile. Could you please help?
Please try to checkout the branch, run Project -> Clean and it should fix the issue. I don't have XCode on my Linux machine which makes this not possible.

@OmarIthawi, could you please fix the tests? You've deleted files, but something is still looking for them.
The following build commands failed:
	CopyStringsFile /Users/runner/Library/Developer/Xcode/DerivedData/OpenEdX-bsdxmseilanfjjcjomwdntdeffpi/Build/Products/DebugDev-iphonesimulator/Core.framework/uk.lproj/Localizable.strings /Users/runner/work/openedx-app-ios/openedx-app-ios/Core/Core/uk.lproj/Localizable.strings (in target 'Core' from project 'Core')

I tried to fix the failing test but I'm unsure how to find the step CopyStringsFile. Could you please help?
@volodymyr-chekyrta Please try to checkout the branch, run Project -> Clean and it should fix the issue. I don't have XCode on my Linux machine which makes this not possible.

@OmarIthawi, could you please fix the tests? You've deleted files, but something is still looking for them.
The following build commands failed:
	CopyStringsFile /Users/runner/Library/Developer/Xcode/DerivedData/OpenEdX-bsdxmseilanfjjcjomwdntdeffpi/Build/Products/DebugDev-iphonesimulator/Core.framework/uk.lproj/Localizable.strings /Users/runner/work/openedx-app-ios/openedx-app-ios/Core/Core/uk.lproj/Localizable.strings (in target 'Core' from project 'Core')

I tried to fix the failing test but I'm unsure how to find the step CopyStringsFile. Could you please help?
@volodymyr-chekyrta Please try to checkout the branch, run Project -> Clean and it should fix the issue. I don't have XCode on my Linux machine which makes this not possible.

I tried this, but the project build fails.
The files are not deleted correctly because the project files are still referenced; you have to remove them from XCode.
Does anyone on your team have XCode? Or maybe I can help you with this.

@volodymyr-chekyrta please help, that would be really appreciated.
We only need to delete those files now they're backed to Transifex and I'm filling up gaps whenever I can.
Our iOS engineer is juggling a couple of tasks already and I need her focused on SAML SSO :)

@volodymyr-chekyrta please help, that would be really appreciated.
We only need to delete those files now they're backed to Transifex
Our iOS engineer is juggling a couple of tasks already and I need her focused on SAML SSO :) and I'm filling up gaps whenever I can.

Could you please give me write permissions to this branch in your repo?
I'll fix it and push changes

@volodymyr-chekyrta you now have write access to the branches. I think we need to revert the change and remove the files manually again via the XCode UI.

@OmarIthawi Okay, that approach won't work :(
If we remove links from the translation files and then pull them via make pull_translations, files will be physically on the disk, but the project will not know about them.

Okay, that approach won't work :(

@volodymyr-chekyrta That's an issue. How did we manage the tx pull in previous versions of the application such as:

https://github.com/openedx-unsupported/edx-app-ios/commits/master/Source/ar.lproj/Localizable.strings

I suppose people didn't just manually edit those files, did they?

@volodymyr-chekyrta I've checked about the issue more and I have some ideas:
Option 1: Use xcString files
I'm not an iOS expert, but if the xcstrings can fit all translations in a single file, I suppose that'll help.

Apple demonstration video: https://developer.apple.com/videos/play/wwdc2023/10155/
Example Localizable.xcstrings file: https://github.com/glushchenko/fsnotes/blob/814948cdd46091739b7c975bf86f129ba32c8ca5/FSNotes/Localizable.xcstrings#L6-L29

Which is as far as I understand, uses one JSON file for all languages.
Option 2: Automate adding the files
The other alternative it to automatically add the uk and other languages after pulling them via Transifex. This can be done via libraries like https://github.com/kronenthaler/mod-pbxproj

@volodymyr-chekyrta I've made some progress regarding the XCode project file edits, I'll pickup the work again in a day or two.

@OmarIthawi, seems the script adding files to the wrong location

It should be like this

That's a bumper, a pretty bad one. Thanks for checking @volodymyr-chekyrta.
I'll study the problem more and get back to you. I'm not ready to give up yet 😅

@brian-smith-tcril @volodymyr-chekyrta Hello again! I think this pull request is ready for merge.
Please take a look and let me know if you have any notes or questions.

@brian-smith-tcril @volodymyr-chekyrta Hello again! I think this pull request is ready for merge.
Please take a look and let me know if you have any notes or questions.

Hi @OmarIthawi, sorry for the delay, I'll check it now

@OmarIthawi, very last thing, and we are good to go.
Could you please remove as well this file:
https://github.com/Zeit-Labs/openedx-app-ios-contrib/blob/remove-uk/OpenEdX/uk.lproj/Localizable.strings

And then add it during the make pull_translations script.
This file is empty, but this is the way to tell the system that we support that language
https://github.com/user-attachments/assets/f54610ad-cc81-47cc-a5f3-57aee4bea7d5
Thank you 🙏

Thanks @volodymyr-chekyrta, I'll do that.

@OmarIthawi, could you please tell me if PR is ready for the final pass?
I still see the uk file at OpenEdX/uk.lproj/Localizable.strings

Not ready yet @volodymyr-chekyrta. Still working on the last bit.

@volodymyr-chekyrta, it's currently ready for testing.
I've used git diffs and it seemed okay. Please let me know if it looks good in XCode.
I see that we're still missing plurals and countries.json localization. So, let's see what can we improve and whether XCStrings would be used eventually.

Thanks @volodymyr-chekyrta for the review.
I don't have merge access, I appreciate a merge to avoid future git conflics.

@OmarIthawi 🎉 Your pull request was merged!  Please take a moment  to answer a two question survey so we can improve your experience in the future.


BROKEN LINK

On the readme file, the link for the superheroes link is broken.

Correct link is http://make.sc/superhero-team-dueler
cc @neptunius

fixed!


Deploy pangeo-forge bakery in USGS cloud

Dependencies:

gathering of evidence to support the decision to deploy this (#383)
gathering of code resources needed to deploy to build (#383)

🚀 💥
We successfully ran the pangeo-forge-flink runner integration test on the USGS aws account. There were several bugs along the way, but we did it. These are partially problems with pangeo-forge and its flink-runner Greg (at Development Seed) and Charles (at Columbia) are making strides on these and upcoming PRs should make this all easier and better documented. And part of the problem is getting it all working on USGS cloud. In hindsight the USGS piece was easy, but it's quickly overwhelming
I'll start formulated notes here, which can transition to a more formal document. To the extent practicle, these changes should be incorporated as PRs to pangeo-forge or NASA-IMPACT to minimize the amount of custom USGS documentation.
Deploying the flink runner

install the repo
and awscli v2, kubectl, saml2aws, and (optionally) k9s
create a s3 bucket for storing the terraform state
modify the terraform deployment to pass the correct permission boundary and aws profile (if used).
also modify the deployment to use an existing VPC (It will try to use default, which is disabled on USGS cloud). This VPC does not require an interconnection; however, because everything is orchestrated through kubectl and the AWS API.
terraform init, terraform apply
the eks cluster should deploy but will fail to deploy any of the helm modules
next run aws eks update-kubeconfig --name <cluster-name> --profile <aws profile> --region <aws-region>, which will populated ~/.kube/config with your cluster credentials.
at this point you should be able to interrogate the cluster via kubectl or k9s
add `config_path="~/.kube/config" to the Kubernetes section of the helm provider block
now run terraform apply again to install the helm modules.
after a couple rounds all the modules should install, but you will still get an error related to ingress.
investigating the logs should show that ingress is indeed running; for example, https://kubernetes.github.io/ingress-nginx/troubleshooting/

Installing pangeo-forge-runner
this process can be rough but should improve once some PRs get merged.

create a python environment
(currently) pip install beam==2.52.0
pip install pangeo-forge-recipes
pip install pangeo-forge-runner
to verify, run pangeo-forge-runner --help

Running a recipe

follow the steps in the integration test
modify config.py to point to a preexisting bucket. Also critical that the beam version matches that in our environment
modify the config.py to include aws secret, key, and token from your current session.
run the recipe via the pangeo-forge-runner CLI tool
we can monitor things by running watch -n 1 "kubectl get flinkdeployment,deploy,pod,svc"
or kubectl port-forward --pod-running-timeout=2m0s --address <IP_ADDRESS> svc/recipe-rest 0:8081 and the opening a browser to <IP_ADDRESS>:XXXX will drop us directly into flink.

Additional links

https://github.com/hytest-org/pangeo-forge-usgs/issues/1
https://pangeo-forge-runner.readthedocs.io/en/latest/tutorial/flink.html
https://github.com/NASA-IMPACT/veda-pforge-job-runner/blob/main/.github/workflows/job-runner.yaml

Next steps

to setup an action, we'll need to verify whether we can saml2aws login via a github action or must this occur via gitlab?

@thodson-usgs would you call this done based on what you accomplished last FY?

Yes, each of the next steps was completed


Finite element calculation of 3D surface

Dear MFEM Maintainer
I have a finite element model of 3D curved surface, and I wonder if MFEM can solve it.
I put the mesh model into THE EX2 of MFEM to try to solve it, but it seems that THE EX2 of MEFM only solves the problems of 2D  plane model and 3D solid model.I input my model into the EX2 solution, but the node displacement is only in the X and Y directions, which is obviously not what I want.
For this 3D surface model, if it is in ANSYS, a thickness parameter needs to be specified, I would like to ask if MFEM can do the same.
If possible, please let me know what I need to know, I would be very grateful!

Duplicate of #2547


ci: add shellcheck to lint workflow and fix issues

For now, this fixes findings with severity "warning" and also uses this severity level for the workflow.

Thanks for the review @yoheiueda

Does the shellcheck action detect this file as well, or do we need to change the script filename to have .sh?

This file should be checked. Beside file extension, the action looks for executable files with a shebang.

Great work @katexochen !
One question: is it possible to have a Makefile target for running shellcheck locally as part of check?  So people don't get used to run the CI just for the sake of running shellcheker as it waste resources.

is it possible to have a Makefile target for running shellcheck locally as part of check?

Why isn't golangci-lint part of make check?

Good question. When I introduced golangci-lint I forgot to add on make check :(

There seems to be some weird issue with golangci-lint and the --path-prefix flag. When not using the flag, there are findings in the volumes/csi-wrapper package. If I set the flag, these findings disappear (and the weren't found by the lint workflow before, which used the flag). I didn't manage to reproduce this behavior in another directory. The version without the flag is currently on this branch, you can check the one with this debug branch.

@katexochen The findings are intentionally excluded by the settings in .golangci.yaml. This YAML file is for all modules in this repository, so a path prefix is added to each package path. When --path-prefix is not specified, paths do not include the prefix, and this exclude rule does not match.
https://github.com/confidential-containers/cloud-api-adaptor/blob/f8a9de44e384fa42dd8e35a86ced526750ff34b8/.golangci.yaml#L299-L303

Why custom scripts (shellcheck.sh and golangci-lint.sh) are used, instead of actions?

@yoheiueda My intention was to execute the exact same checks in CI as in local dev environment. The shellcheck action is essentially a script around shellcheck, like the one I wrote. I think duplicating things like exclude and include list will let them diverge at some point. Also the call of make check was already there, I just deleted the steps that would now be duplicated.
But I agree that it would be nice to not terminate if a step fails to also get the results of the other linters.
I see the following two ways we could enable this:

Don't run the check target but the targets underneath, execute in different jobs.
Add a meta script that sequentially executes the different checks and returns the result afterwards.

Don't run the check target but the targets underneath, execute in different jobs.

I've implemented 1. for now.

Ping @snir911 @jtumber-ibm @wainersm

This looks good to be merged.
Thanks @katexochen


Set timeout for d3.xhr requests

Either it's not possible to set timeouts for xhr reqeusts, or it isn't documented at all. IMHO this is a very important feature, especially for projects with many concurrent connections. In case there's an issue with one or more web services your project depends on, sending xhr requests to services that simply don't respond will block a connection slot. As browsers have only a small number of connection slots per server (http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser), other xhr connection may not come through at all, rendering some or all features of your project unusable.

At the moment I'm taking care of aborting long-running requests myself using setTimeout, but having to do that in all places I'm using d3.xhr is quite cumbersome.

It’s possible using the beforesend event and then accessing the underlying XMLHttpRequest object:
d3.xhr(url)
    .on("beforesend", function(xhr) { xhr.timeout = 2000; xhr.ontimeout = xhr.onload; })
    .get(callback)

The unfortunate part is that you have to register an ontimeout listener if you want to be notified of the timeout. Here, I’m recycling the onload listener that’s registered by d3.xhr, such that when the timeout fires an error is passed to the callback.
That said I think it’s worth adding this to d3-request, so that you can just set the timeout and it will work automatically. I’ll do that now.

Added request.timeout in d3-request 0.2.5.


0.7.2: Possible for message sending interface to become confused about state of encryption

When typing a message in an encrypted room, sometimes my message gets deleted and the icon next to the text field changed from encrypted to unencrypted. It is just a UI bug though. If I type the message again it sends as an encrypted message even though the icon is unlocked.
Possibly related to using the Gboard keyboard but I haven't done a lot of testing.
I will add steps to reproduce as soon as I can find them.

I've been observing the same problem (typed in message content disappearing) recently, but not in rooms that have encryption, so it's probably related to something else.
It seems like it's related to the loading spinner.
It goes something like this:

I'd open the app, the loading spinner would be there
I'd start typing a message anyway
the spinner would disappear after a while, clearing the already typed in (and yet unsent) message and closing the keyboard

I'd be forced to enter my message once again after that.

Thank you for your feedback. A fix will be embedded in TF release 0.7.3(166).


feat: implement async sync with inheritance and unasync-cli

Run command unasync postgrest_py tests to generate synchronous code and tests version
Fix #27
This PR uses inheritance to avoid code duplication as much as possible.
Pros: smaller library size
Cons: code more "difficult" to understand
I did another PR without using inheritance so that you accept the alternative that seems best to you: #30

I think we should go with #30, let's discuss there.


[BUG] AppShell not working in 0.13.0a2

From what I understand, AppShell() should accept a Navbar() and a Header() in the respective arguments.
This doesn't work in the current version, however.
Wrapping the elements in a list (i.e., passing header=[dmc.Header("", height="50px")]) runs, but causes an error in the browser related to invalid props:
Objects are not valid as a React child (found: object with keys {props, type, namespace}). If you meant to render a collection of children, use an array instead.

MWE:
from dash import Dash
import dash_mantine_components as dmc

app = Dash(
    __name__,
)
app.layout = dmc.AppShell(
    header=dmc.Header("", height="50px"),
    footer=dmc.Footer("", height="50px"),
    children=dmc.Text("Some placeholder Text")
)

app.run(debug=True)

Yields
TypeError: The `dash_mantine_components.AppShell` component (version 0.13.0a2) detected a Component for a prop other than `children`
Prop footer has value Footer(children='', height='50px')
Did you forget to wrap multiple `children` in an array?
For example, it must be html.Div(["a", "b", "c"]) not html.Div("a", "b", "c")

The same code works in v0.12.1

0.13.0a2 was an alpha version which is no longer supported. Please migrate to 0.14 if possible.

The current version (0.14.4) has a similar problem, or am I missing something?
from dash import Dash
import dash_mantine_components as dmc

app = Dash(
    __name__,
)
app.layout = dmc.AppShell(
    header=dmc.AppShellHeader(""),
    footer=dmc.AppShellFooter(""),
    children=dmc.Text("Some placeholder Text")
)

app.run(debug=True)

TypeError: The `dash_mantine_components.AppShell` component (version 0.14.4) detected a Component for a prop other than `children`
Prop header has value AppShellHeader('')
Did you forget to wrap multiple `children` in an array?
For example, it must be html.Div(["a", "b", "c"]) not html.Div("a", "b", "c")


Accepts lazy evaluation for prefix opt

Codecov Report

Merging #102 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #102   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          14     14
  Lines         228    234    +6
  Branches       41     43    +2
=====================================
+ Hits          228    234    +6

Impacted Files
Coverage Δ

src/base.js
100% <100%> (ø)
:arrow_up:

@caiogondim I tested it and since whether logs are outputed is determined by prefix value, prefix getter will actually break existing functionality.
Rather than modyfing the actual prefix, can't we just have a way to prepend a string to the log calls?

That's something I didn't think of.
Will close this PR without merging.
Let's keep discussing about it on #48


Platform deinitialization function.

This PR addresses #1446

Codecov Report

Merging #1451 into master will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1451      +/-   ##
==========================================
- Coverage   70.98%   70.82%   -0.16%
==========================================
  Files         143      143
  Lines       19492    19492
  Branches     2385     2385
==========================================
- Hits        13836    13806      -30
- Misses       4754     4790      +36
+ Partials      902      896       -6

Impacted Files
Coverage Δ

src/core/thread/lowpan.hpp
86.84% <0%> (-13.16%)
:x:

src/ncp/ncp_base.cpp
36.88% <0%> (-2.32%)
:x:

src/core/api/thread_api.cpp
45.45% <0%> (-1.74%)
:x:

src/core/mac/mac.cpp
79.4% <0%> (-1.04%)
:x:

src/core/thread/network_data_leader_ftd.cpp
67.99% <0%> (-0.47%)
:x:

src/core/thread/mle.cpp
80.8% <0%> (-0.36%)
:x:

src/core/thread/mle_router.cpp
79.58% <0%> (-0.06%)
:x:

src/core/thread/network_data_leader.cpp
80.59% <0%> (+0.49%)
:white_check_mark:

src/core/thread/mesh_forwarder.cpp
74.94% <0%> (+0.55%)
:white_check_mark:

src/core/common/message.cpp
96.26% <0%> (+0.82%)
:white_check_mark:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9efbb99...dac10b3. Read the comment docs.


Feature request: Snow timer

A snow timer when using a snowglobe or raindeer hat, a timer for when to put the snow down and a timer for when the snow will be gone.

Used for f2p tick manipulation
#feature-request

(idk how to add feature request as a label)

also maybe a left click option for the snow to let it fall?

@Yiaro222 You can already use the shift-click configuration plugin to access that option more easily.

@Nightfirecat but thats holding shiftttt :/ ok ty

Ideally, when #1556 is merged, that can be resolved more nicely.


Work around erroneously inherited execute = false option

Closes #1125, closes #1161 (issues observed with @xstate/react@next which has execute: false).
This is an alternative for #1162 that introduces no breaking changes.
The childOptions option will not be documented, as it is a temporary fix that will be deprecated in version 5.

From what I understand - this is only to handle @xstate/react@next case that is being caused by scheduling all actions through the React's commit phase. We've agreed that this is not needed as XState itself already guarantees actions not being re-executed multiple times so shouldn't we just revert this scheduling behavior from @xstate/react instead?


Cannot list more than 50 roles in the permission panel of a registry collection in the carbon console.

Description:
When an user have more than 50 roles, it will not listed under the permission panel of a registry collection in carbon console. For an example, browse to following registry collection in the carbon console and open permission panel.
/_system/governance/apimgt/customsequences
If you have more than 50 roles, those will not listed under the role dropdown.
This will prevent users from giving permissions to different roles to access and modify registry resources, if they have more than 50 roles.
Suggested Labels:

Suggested Assignees:

Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

This is an invalid use case where APIM doesn't need to assign roles from the registry console to APIM resources.


Recompute Mean/Std for Satellite data

Detailed Description
Now that we are using the original projection of the satellite data, the mean and std are probably close, but still wrong for the new format. So we need to rerun the calculation for all the channels.
Context
We need it to have the correct normalization.
Possible Implementation

You probably know this already but, in principle, now that the satellite data is in Zarr, it should be possible to use Dask to compute the means and standard deviations across the entire Zarr dataset (and Dask will handle loading the data in chunks so as not to swamp the main memory).  But, in practice, I still find that dask doesn't always work perfectly out-of-the-box!


Makin quickstart section more look like terminal

Hello Mr Juan,
In this pull request I'm making your 'Quickstart` section more look like terminal/cmd
It will be awesome if you accept this pull request, thanks!

I think its more better to use $ 😁 like https://github.com/id-python/direktori

Yeah it looks nice

Thanks mr @JuanPotato , Hoping you remember me in telegram 😇

I immediately recognized your profile picture :smile:

Wergh, I was suggesting using # instead of - on that line, because it's a comment in a shell script

@nabijaczlewli oooh,


SDL 2.0.7 for OS X

@mellinoe
Can you please SDL to 2.0.7 for OS X too? As I can see - https://github.com/mellinoe/veldrid/tree/master/src/Veldrid.SDL2/native- right now is only Windows x64 dll is updated.

I've updated it: https://github.com/mellinoe/veldrid/commit/4428e3d3d8d7c9ddae8ab43608791bf132b5c41d
By the way, Veldrid will prefer the system library if it is installed over what is in the NuGet package. So if you install it from homebrew (the current version is 2.0.7), then that binary will be used.


Memory for PepQuery2

Not 100% sure where to place this issue, but I thought it might be interesting for all users of the shared database. Otherwise I can of course move it to EU.
I am currently debugging an error the pepquery2 tool. The job errored because the JVM run out of memory.
When I tried to run the job locally I had to stop at 14G because my laptop (16G) started to lag.
I noticed that:

it uses 8 cores (from the logs) even though it was allocated 1 core
the index creation set is much slower on the server than on my laptop (might be storage related)
there is no rule for TPV (and was no rule for sortinghat)

I would like to change that, but I am not sure which values I should consider. In their documentation I found a recommendation for 8 GB of memory and 4 CPUs which is too little for at least the job I am looking at. When I tried to use gxadmin query tool-memory-per-inputs I found:
    id    |                                tool_id                                 | input_count | total_input_size_mb | mean_input_size_mb | median_input_size_mb | memory_used_mb | memory_used_per_input_mb | memory_mean_input_ratio | memory_median_input_ratio
----------+------------------------------------------------------------------------+-------------+---------------------+--------------------+----------------------+----------------+--------------------------+-------------------------+---------------------------
 ######### | toolshed.g2.bx.psu.edu/repos/galaxyp/pepquery2/pepquery2/2.0.2+galaxy0 |           1 |                  36 |                 36 |                   36 |         283829 |            7948 |                    7948 |                      7948

While gxadmin report job-info returned the following:
## Destination Parameters

Key | Value
--- | ---
+Group | `""`
accounting_group_user | `#####`
description | `pepquery2`
docker_memory | `3.8G`
metadata_strategy | `extended`
request_cpus | `1`
request_memory | `3.8G`
requirements | `(GalaxyGroup  ==  "compute")`
submit_request_gpus | `0`

I am now trying to figure out how to implement a rule here and if we have to change something in the wrapper because of the CPU usage. Since I never used the tool myself I would be happy about any hints from people who have some experience with it.

Unfortunately the tool seems not to be satisfied and condor complains it tries to exceed the 16G:
007 (44662823.000.000) 07/12 09:25:00 Shadow exception!
        Error from<EMAIL_ADDRESS>Job has gone over memory limit of 16384 megabytes. Peak usage: 16331 megabytes.
        0  -  Run Bytes Sent By Job
        0  -  Run Bytes Received By Job
...
012 (44662823.000.000) 07/12 09:25:00 Job was held.
        Error from<EMAIL_ADDRESS>Job has gone over memory limit of 16384 megabytes. Peak usage: 16331 megabytes.
        Code 34 Subcode 0

I am not an Java expert, but I would assume, that there is some kind of overhead that adds to the 16GB that the JVM uses as heap

Looks like it supports a -cpu parameter: http://www.pepquery.org/document.html#saparameter
and the wrapper will indeed need to be modified.

f no one opposes, I would open a PR with that set accordingly in _JAVA_OPTIONS

Looks like it supports a -cpu parameter: http://www.pepquery.org/document.html#saparameter and the wrapper will indeed need to be modified.

yes by default it uses all cores available to it, but it would be cleaner to use it, I guess.

I got this from Galaxy, probably, because I run je job manually and it was still watched by the Galaxy Handlers:
Job Metrics
cgroup

CPU Time | 2 hours and 54 minutes
-- | --
Failed to allocate memory count | 0E-7
Memory limit on cgroup (MEM) | 48.0 GB
Max memory usage (MEM) | 17.5 GB
Memory limit on cgroup (MEM+SWP) | 8.0 EB
Max memory usage (MEM+SWP) | 17.5 GB
OOM Control enabled | No
Was OOM Killer active? | No
Memory softlimit on cgroup | 0 bytes

...

Destination Parameters

Runner | condor
-- | --
Runner Job ID | 44667975
Handler | handler_sn06_3
+Group | ""
accounting_group_user | 55103
description | pepquery2
docker_memory | 16G
metadata_strategy | extended
request_cpus | 1
request_memory | 16G

The job is stopped by condor for exceeding its memory:
007 (xxxxxxxxx.000.000) 07/14 11:35:09 Shadow exception!
        Error from slot1_8@vgcnbwc-worker-xxxxxxx: Job has gone over memory limit of 16384 megabytes. Peak usage: 16328 megabytes.
        0  -  Run Bytes Sent By Job
        0  -  Run Bytes Received By Job
...
012 (xxxxxxxxxxxx.000.000) 07/14 11:35:09 Job was held.
        Error from slot1_8@vgcnbwc-worker-xxxxxxxxl: Job has gone over memory limit of 16384 megabytes. Peak usage: 16328 megabytes.
        Code 34 Subcode 0
...
013 (xxxxxxxxxx.000.000) 07/14 11:45:02 Job was released.
        via condor_release (by user galaxy)

Here is a PR to increase it

https://github.com/galaxyproject/tpv-shared-database/pull/32
And here for the Wrapper:
https://github.com/galaxyproteomics/tools-galaxyp/pull/722


Usability: fill the parameters with default value when "resubmit failed"

When user  resubmit failed workflow, the UI won't pre-fill the parameters with default value and data.  For better usability, I suggest that argo fill the parameters with default value when "resubmit failed"

Resubmit Failed functionality has been temporary disabled. Looks like we forgot to hide Resubmit Failed from timeline page, this is fixed now.


refactor: rename ScalarMultiplication -> ScalarMul

Rename ScalarMultiplication() to ScalarMul(). Needs https://github.com/ConsenSys/gnark-crypto/pull/220 PR to be merged first.

In Groth16 the prover computes 3 ScalarMul() with random scalars r and s (2 G1, 1 G2). These call the internal mulGLV() which first split the random r and s into sub-scalars before hitting the for loop. I tried to remove the SplitScalar() from the mulGLV() and put it explicitly in ScalarMul(). This way one can sample directly random sub-scalars and use MulGLV() instead of ScalarMul(). However it's just a -1% after PR https://github.com/ConsenSys/gnark-crypto/pull/213 which speeds up the scalar decomposition.

closing this after https://github.com/ConsenSys/gnark-crypto/pull/224


Resource exhaustion

This issue is a placeholder epic for any issue that can be classed as a "using more resources than you should" issue

[ ] disk filling up
[ ] Queries into the system - rate limiting, sizes, etc
[ ] Holding open connections too long
[ ] Forcing too large a return
[x] https://github.com/filecoin-project/bacalhau/issues/840

@wdbaruni for productionization efforts? feel free to merge/close

closing in favour of production-readiness label


Use pytest-httpserver

I tested locally and didn't manage to get the hanging issue we were having (I could reproduce it before this PR).
I had to change the way we test fsspec_open_kwargs because basic authentication doesn't seem to be possible with pytest-httpserver (it seems to only provide digest authentication). But anyway the point of this test was rather to see if the kwargs were propagated to the fsspec opener, so it doesn't really matter what we propagate. The simplest thing I could think about is to force an HTTP request to time out.
Closes #66

Codecov Report

Merging #67 (6becd0d) into master (aa5f8a9) will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
- Coverage   91.27%   91.24%   -0.04%
==========================================
  Files           4        4
  Lines         275      274       -1
  Branches       22       22
==========================================
- Hits          251      250       -1
  Misses         21       21
  Partials        3        3

Impacted Files
Coverage Δ

pangeo_forge/utils.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa5f8a9...6becd0d. Read the comment docs.

Codecov Report

Merging #67 (6becd0d) into master (aa5f8a9) will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
- Coverage   91.27%   91.24%   -0.04%
==========================================
  Files           4        4
  Lines         275      274       -1
  Branches       22       22
==========================================
- Hits          251      250       -1
  Misses         21       21
  Partials        3        3

Impacted Files
Coverage Δ

pangeo_forge/utils.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa5f8a9...6becd0d. Read the comment docs.

I added back the authentication test and used the respond_with_handler to avoid loading all files in memory, thanks again @csernazs.
It doesn't seem to solve our issues with the hanging test though. But even though, I think it is better to rely on pytest-httpserver than setting up our own server.

Actually maybe pytest-httpserver can help debugging this issue, through the use of check_assertions(). I'm going to add one in the handler and see if we can catch an error. I'll trigger the CI until the issue appears.

I had the issue locally, and check_assertions() didn't help. Here is the trace after pressing CTRL-C (from where it enters fsspec):

Click to expand

???

h5py/h5fd.pyx:158:

self = <File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>, b = <MemoryView of 'array' at 0x7f3aba4f51e0>
def readinto(self, b):
    """mirrors builtin file's readinto method

    https://docs.python.org/3/library/io.html#io.RawIOBase.readinto
    """
    out = memoryview(b).cast("B")

  data = self.read(out.nbytes)

../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/site-packages/fsspec/spec.py:1442:

self = <File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>, length = 512
def read(self, length=-1):
    """Read bytes from file

    Parameters
    ----------
    length: int
        Read up to this many bytes. If negative, read all content to end of
        file. If the server has not supplied the filesize, attempting to
        read only part of the data will raise a ValueError.
    """
    if (
        (length < 0 and self.loc == 0)
        or (length > (self.size or length))  # explicit read all
        or (  # read more than there is
            self.size and self.size < self.blocksize
        )  # all fits in one block anyway
    ):

      self._fetch_all()

../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/site-packages/fsspec/implementations/http.py:436:

args = (<File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>,), kwargs = {}
self = <File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>
@functools.wraps(func)
def wrapper(*args, **kwargs):
    self = obj or args[0]

  return maybe_sync(func, self, *args, **kwargs)

../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/site-packages/fsspec/asyn.py:121:

func = <function HTTPFile.async_fetch_all at 0x7f3aba51b310>, self = <File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>
args = (<File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>,), kwargs = {}, loop = <_UnixSelectorEventLoop running=True closed=False debug=False>
def maybe_sync(func, self, *args, **kwargs):
    """Make function call into coroutine or maybe run

    If we are running async, run coroutine on current loop until done;
    otherwise runs it on the loop (if is a coroutine already) or directly. Will guess
    we are running async if either "self" has an attribute asynchronous which is True,
    or thread_state does (this gets set in ``sync()`` itself, to avoid nesting loops).
    """
    loop = self.loop
    # second condition below triggers if this is running in the thread of the
    # event loop *during* the call to sync(), i.e., while running
    # asynchronously
    if getattr(self, "asynchronous", False) or getattr(
        thread_state, "asynchronous", False
    ):
        if inspect.iscoroutinefunction(func):
            # run coroutine while pausing this one (because we are within async)
            return _run_until_done(func(*args, **kwargs))
        else:
            # make awaitable which then calls the blocking function
            return _run_as_coroutine(func, *args, **kwargs)
    else:
        if inspect.iscoroutinefunction(func):
            # run the awaitable on the loop

          return sync(loop, func, *args, **kwargs)

../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/site-packages/fsspec/asyn.py:100:

loop = <_UnixSelectorEventLoop running=True closed=False debug=False>, func = <function HTTPFile.async_fetch_all at 0x7f3aba51b310>, callback_timeout = None
args = (<File-like object HTTPFileSystem, http://<IP_ADDRESS>:46711/001.nc>,), kwargs = {}, f = <function sync..f at 0x7f3aba43c9d0>
def sync(loop, func, *args, callback_timeout=None, **kwargs):
    """
    Run coroutine in loop running in separate thread.
    """

    e = threading.Event()
    main_tid = threading.get_ident()
    result = [None]
    error = [False]

    async def f():
        try:
            if main_tid == threading.get_ident():
                raise RuntimeError("sync() called from thread of running loop")
            await asyncio.sleep(0)
            thread_state.asynchronous = True
            future = func(*args, **kwargs)
            if callback_timeout is not None:
                future = asyncio.wait_for(future, callback_timeout)
            result[0] = await future
        except Exception:
            error[0] = sys.exc_info()
        finally:
            thread_state.asynchronous = False
            e.set()

    asyncio.run_coroutine_threadsafe(f(), loop=loop)
    if callback_timeout is not None:
        if not e.wait(callback_timeout):
            raise TimeoutError("timed out after %s s." % (callback_timeout,))
    else:
        while not e.is_set():

          e.wait(10)

../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/site-packages/fsspec/asyn.py:68:

self = <threading.Event object at 0x7f3aba42b280>, timeout = 10
def wait(self, timeout=None):
    """Block until the internal flag is true.

    If the internal flag is true on entry, return immediately. Otherwise,
    block until another thread calls set() to set the flag to true, or until
    the optional timeout occurs.

    When the timeout argument is present and not None, it should be a
    floating point number specifying a timeout for the operation in seconds
    (or fractions thereof).

    This method returns the internal flag on exit, so it will always return
    True except if a timeout is given and the operation times out.

    """
    with self._cond:
        signaled = self._flag
        if not signaled:

          signaled = self._cond.wait(timeout)

../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/threading.py:574:

self = <Condition(<unlocked _thread.lock object at 0x7f3aba42bba0>, 0)>, timeout = 10
def wait(self, timeout=None):
    """Wait until notified or until a timeout occurs.

    If the calling thread has not acquired the lock when this method is
    called, a RuntimeError is raised.

    This method releases the underlying lock, and then blocks until it is
    awakened by a notify() or notify_all() call for the same condition
    variable in another thread, or until the optional timeout occurs. Once
    awakened or timed out, it re-acquires the lock and returns.

    When the timeout argument is present and not None, it should be a
    floating point number specifying a timeout for the operation in seconds
    (or fractions thereof).

    When the underlying lock is an RLock, it is not released using its
    release() method, since this may not actually unlock the lock when it
    was acquired multiple times recursively. Instead, an internal interface
    of the RLock class is used, which really unlocks it even when it has
    been recursively acquired several times. Another internal interface is
    then used to restore the recursion level when the lock is reacquired.

    """
    if not self._is_owned():
        raise RuntimeError("cannot wait on un-acquired lock")
    waiter = _allocate_lock()
    waiter.acquire()
    self._waiters.append(waiter)
    saved_state = self._release_save()
    gotit = False
    try:    # restore state no matter what (e.g., KeyboardInterrupt)
        if timeout is None:
            waiter.acquire()
            gotit = True
        else:
            if timeout > 0:

              gotit = waiter.acquire(True, timeout)

E                   KeyboardInterrupt
../../../mambaforge/envs/pangeo-forge-dev/lib/python3.9/threading.py:316: KeyboardInterrupt

@martindurant any chance you can see something wrong by looking at where we are in fsspec?

We are waiting on an event loop in another thread. Perhaps the server is waiting for the same event loop? I don't know how this server is implemented.

Interesting, we had the same issue before using pytest-httpserver, when the server was just SimpleHTTPServer running in another process. From what I saw, the issue always happens the second time we run the tests involving HTTP requests, never the first time.

If I understand correctly, fsspec is natively async, and when it is not running in an async environment, runs the async functions on a loop in another thread (at least for HTTP file systems)?

Yes, that's correct

@csernazs I can see in the documentation:
Multi-threading support
When your client runs in a thread, everything completes without waiting for the first response. To overcome this problem, you can wait until all the handlers have been served or there’s some error happened.

But I guess this is not the case here, since our tests pass (when they don't hang).
More generally, do you see an issue with our client running in another thread?

Nothing obvious, sorry

Thanks @martindurant, this was more a question for @csernazs (sorry I was not clear).
I guess we can't use the with httpserver.wait context manager, since our handlers are permanent.

You might just want to play with the scope of the fixture, though. Perhaps ensuring that it outlives the test function.

From the documentation on the server lifetime:

Http server is started when the first test uses the httpserver fixture, and it will be running for the rest of the session. The server is not stopped and started between the tests as it is an expensive operation, it takes up to 1 second to properly stop the server.
To avoid crosstalk (eg one test leaving its state behind), the server’s state is cleaned up between test runnings.

I wonder why it would take so long!

I'm not sure I fully understood your problem (I ran into this issue by accident and I wanted to help), but here's what I can add:
pytest-httpserver is based on werkzeug which uses the socketserver implementation of the standard library.
This is the code which causes this delay, see poll_interval's default value of 0.5 seconds:
https://github.com/python/cpython/blob/fc40b3020cf3c869833fd5d3720cf9768fe3bb46/Lib/socketserver.py#L232
The server is started in a thread by the pytest-httpserver's code, so you can run your client's code in the main thread if that's possible. As you said, using the wait() context manager is only possible with "expiring" handlers such as oneshot and ordered ones.
In pytest-httpserver, the plugin's code is just a glue code for the real server so you can implement your own version (eg if you want to start/stop the server for each test) if you want. All you need is to instantiate the HTTPServer object and call it's start and stop methods accordingly.
Replacing the socketserver with a better implementation is on my todo list and I think it could be possible without breaking the API but there's no ETA for this.

I'm not sure I fully understood your problem (I ran into this issue by accident and I wanted to help)

No worries, I was just wondering if there was something obviously wrong in our use case that you could spot .

The server is started in a thread by the pytest-httpserver's code, so you can run your client's code in the main thread if that's possible.

Here we don't have a choice, our client (fsspec) makes requests from another thread (not the main thread).

In pytest-httpserver, the plugin's code is just a glue code for the real server so you can implement your own version (eg if you want to start/stop the server for each test)

I don't think we want to start/stop the server for each test, as long as its state is cleared between tests.

Well, I think I cannot help more from the httpserver's point, but maybe it helps:
There's a log attribute of the httpserver which could be polled periodically to see which requests had been served and consider the test completed on some condition (eg amount of requests, or checking the last request served). Also you can set a threading event from the request handler on some condition which finish the test. There would be some condition which means the end of the test running I think.

Let's discuss this PR at our meeting on Monday.


Update actions to use google-github-actions/auth@v1 and fix input syntax

So it looks like the new version of google-github-actions/setup-gcloud@v1 action no longer uses service_account_key as input. See this failed deploy, under the gcloud setup header.
Warning: Unexpected input(s) 'service_account_key', valid inputs are ['skip_install', 'version', 'project_id', 'install_components']
...
Warning: No authentication found for gcloud, authenticate with `google-github-actions/auth`.

Which is leading to a fail in the deploy config toml step.
ERROR: (gcloud.storage.cp) You do not currently have an active account selected.
Please run:

  $ gcloud auth login

to obtain new credentials.

From the actions repo, it looks like they now require use of the google-github-actions/auth action to first auth with the service_account_key (now called credentials_json) before running the gcloud setup.
Also updates the .github/workflows/deploy_container.yaml to use the @v1 version of the gcloud setup as well.

Mea culpa, I should have noticed that when I reviewed #117 as we had already run into the same thing in the metamist repo.
I previously made a similar change to this one to fix the same accident in populationgenomics/metamist#550, but we took the opportunity to move to use a federated identity instead in populationgenomics/metamist#553. Possibly that's something we would want to do in this repo too?

Hey @EddieLF, @jmarshall beat me to the suggestion, it's unclear to me at this stage which permissions one needs to do this so I've done it and added a couple of commits to your branch to address this.

This quiet change in action parameters is exactly the sort of thing that actionlint should be able to detect. It knows about a bunch of common actions but so far not this one. However I've built a version locally that does, so hopefully we won't make this mistake again! And also proposed adding it upstream — see rhysd/actionlint#380.

I'm going to revoke the gh-images-deployer credentials, keep an eye out for anything weird.


EventEmitter memory leak detected. 11 exit listeners added

This is the full error message:
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
I run Node v8.1.2 with tip-of-tree Puppeteer
The code is written with Promise.all, but even if I run work() sequentially the same error appears.
const puppeteer = require('puppeteer')
const R = require('rambda')

const resolution = {
  x : 1920,
  y : 1080,
}

const args = [
  '--disable-gpu',
  `--window-size=${ resolution.x },${ resolution.y }`,
  '--no-sandbox',
]

const work = async () => {
  const browser = await puppeteer.launch({
    headless     : true,
    handleSIGINT : false,
    args         : args,
  })
  const page = await browser.newPage()

  const url = 'https://ilearnsmarter.com/learning-meme'

  await page.setViewport({
    width  : resolution.x,
    height : resolution.y,
  })

  await page.goto(url, { waitUntil : 'networkidle' })
  await browser.close()
}

const fn = async () => {
  const promised = R.range(0, 12).map(() => work())
  await Promise.all(promised)

  console.log('DONE')
}

fn()

One possible solution is using child process:
// workChild.js
const puppeteer = require('puppeteer')

const resolution = {
  x : 1920,
  y : 1080,
}

const args = [
  '--disable-gpu',
  `--window-size=${ resolution.x },${ resolution.y }`,
  '--no-sandbox',
]

const work = async () => {
  const browser = await puppeteer.launch({
    headless     : true,
    handleSIGINT : false,
    args         : args,
  })
  const page = await browser.newPage()

  const url = 'https://ilearnsmarter.com/learning-meme'

  await page.setViewport({
    width  : resolution.x,
    height : resolution.y,
  })

  await page.goto(url, { waitUntil : 'networkidle' })

  const result = await page.content()
  await browser.close()

  return result.length
}

work()
  .then(result => {
    process.send(result)
    process.exit()
  })

const { fork } = require("child_process")
const R = require('rambda')

const work = () => new Promise(resolve=>{
  const child = fork("workChild.js", [], {
    cwd : __dirname,
    env : process.env,
  })

  child.on("message", result => {
    return resolve(result)
  })
})

const fn = async () => {
  const promised = R.range(0, 12).map(() => work())
  const result = await Promise.all(promised)

  console.log(result, 'DONE')
}

fn()

I still wonder if this is the only current solution.

I've also hit this in node 7.10 but I'm getting 'finish' listener. My service renders a pdf one at a time, it runs launch and creates a new page for each pdf. But this seems to have occurred after the service was running for a while.
(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 finish listeners added. Use emitter.setMaxListeners() to increase limit

Lighthouse and chrome-launcher handled this in https://github.com/GoogleChrome/lighthouse/pull/2959.

@selfrefactor your script should be working fine now.
@domarmstrong there's no 'finish' event in the puppeteer - this looks like something from other modules.

@aslushnikov Thank you for the fast fix, but it still doesn't work for me. I tested it on local machine and Cloud9 instance and I make sure that the code reads your changes.

@selfrefactor ah, indeed, you launch chrome processes in parallel. Every chrome instance adds a listener to the process's "exit" event to cleanup properly. 12 chrome instances add 12 listeners, which yields the warning.
You can fix this by adding process.setMaxListeners(Infinity) in the very beginning of your script:
const puppeteer = require('puppeteer')
const R = require('rambda')

process.setMaxListeners(Infinity); // <== Important line

const resolution = {
  x : 1920,
  y : 1080,
}
// ...

Note: you don't need to launch browser just to create a page. Instead, you can open multiple pages in the same browser:
const puppeteer = require('puppeteer')
const R = require('rambda')

const resolution = {
  x : 1920,
  y : 1080,
}

const args = [
  '--disable-gpu',
  `--window-size=${ resolution.x },${ resolution.y }`,
  '--no-sandbox',
]

let browser;

const work = async () => {
  const page = await browser.newPage()

  const url = 'https://ilearnsmarter.com/learning-meme'

  await page.setViewport({
    width  : resolution.x,
    height : resolution.y,
  })

  await page.goto(url, { waitUntil : 'networkidle' })
  await page.close();
}

const fn = async () => {
  browser = await puppeteer.launch({
    headless     : true,
    handleSIGINT : false,
    args         : args,
  })
  const promised = R.range(0, 12).map(() => work())
  await Promise.all(promised)
  browser.close()

  console.log('DONE')
}

fn()

OK, the finish listener was a red herring. But I'm also getting these errors when not running in parallel.
const puppeteer = require('puppeteer');

// bad url with no protocol fails fast
const jobs = [...new Array(12)].map(() => 'google.com');

const work = async (url) => {
  try {
    const browser = await puppeteer.launch({ dumpio: true });
    const page = await browser.newPage();
    await page.goto(url);
    browser.close();
  } catch (err) {
    console.log(err.message);
  } finally {
    if (jobs.length) {
      work(jobs.pop());
    } else {
      process.exit();
    }
  }
};

work(jobs.pop());

The listeners don't seem to get cleaned up, and if I dont have the process.exit it doesn't ever exit the process.  These are only shown when using dumpio. Setting max event listeners is not an option for me as I'm using as part of a long running process so it will leak memory.

@domarmstrong the reason your script doesn't work with tip-of-tree puppeteer is because you don't close browsers if exception is thrown. This is a memory leak.
The following script works just fine:
const puppeteer = require('puppeteer');

// bad url with no protocol fails fast
const jobs = [...new Array(12)].map(() => 'google.com');

const work = async (url) => {
  const browser = await puppeteer.launch({dumpio: true});
  try {
    const page = await browser.newPage();
    await page.goto(url);
  } catch (err) {
    console.log(err.message);
  } finally {
    browser.close();
    if (jobs.length) {
      work(jobs.pop());
    } else {
      process.exit();
    }
  }
};

work(jobs.pop());

if you pass https://google.com instead of google.com, no exception will be thrown and your original script will work just fine.

OK, that's a very good point lol. But still you get the exit and SIGINT errors.
const puppeteer = require('puppeteer');

// bad url with no protocol fails fast
const jobs = [...new Array(12)].map(() => 'https://google.com');

const work = async (url) => {
  let browser;
  try {
    browser = await puppeteer.launch({ dumpio: true });
    const page = await browser.newPage();
    await page.goto(url);
  } catch (err) {
    console.log(err.message);
  } finally {
    if (browser) {
      browser.close();
    }

    if (jobs.length) {
      work(jobs.pop());
    } else {
      process.exit();
    }
  }
};

work(jobs.pop());

DevTools listening on ws://<IP_ADDRESS>:44485/devtools/browser/61c2fd13-086d-4546-8738-eb91ebe3eb6b

DevTools listening on ws://<IP_ADDRESS>:36193/devtools/browser/0f73b4b5-9ac1-48be-a137-e7bec712d949

DevTools listening on ws://<IP_ADDRESS>:43067/devtools/browser/7b19c8f0-f173-417d-a77d-b27c22d75805

DevTools listening on ws://<IP_ADDRESS>:45905/devtools/browser/39c2c0fb-ef8a-4f58-8d31-90b7347c0138

DevTools listening on ws://<IP_ADDRESS>:35571/devtools/browser/b6125682-90ed-4f6b-b456-f3c79078f9ee

DevTools listening on ws://<IP_ADDRESS>:40393/devtools/browser/dd7a2064-0053-40f6-b957-9e7e5fceba9b

DevTools listening on ws://<IP_ADDRESS>:40311/devtools/browser/3f944449-ac8d-4165-b239-48597520ee1c

DevTools listening on ws://<IP_ADDRESS>:46481/devtools/browser/ea98a4aa-11d6-4e3e-b614-ae9968ba4187

DevTools listening on ws://<IP_ADDRESS>:40055/devtools/browser/66e1626e-c038-470b-b585-3c9b0b051169

DevTools listening on ws://<IP_ADDRESS>:46827/devtools/browser/717005b0-7033-4b4a-aeeb-fdd756dbb5d6
(node:30356) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:30356) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit

DevTools listening on ws://<IP_ADDRESS>:35135/devtools/browser/e29d2f79-67e9-4128-956e-bfac92064682

DevTools listening on ws://<IP_ADDRESS>:44217/devtools/browser/c4cb9006-155f-49f5-8012-ff38be09ed3f

@domarmstrong What's your puppeteer version? Can you run the npm list --depth=0 from the root of your project? It looks like you're not using the tip-of-tree puppeteer, which is labeled as 0.10.2-alpha.

Yep your right. That's fixed then. :+1:  thanks.

@aslushnikov Actually I don't need it to run in parallel and the code really works in sequential context. Thank you for the efforts.

Had the same issue with gulp-critical. Solution is to use gulp-limit

Got a queue of 12 items to process at once. I was creating 12 browsers with a page for each and getting the above error. Then I tried 1 browser for all tabs and noticed the time to process all 12 of those items doubled and had issues when closing pages breaking the others.
Then tried 4x browser with 3 pages each and still got lots of issues with previously closed pages causing the others to break.
Not sure what happened but for me I'm going to just increase the limit to remove the error.

right before on your app.listen(port,() => {}) add process.setMaxListeners(0)` for infinity event listener chaining

For me, I would like to ask, whats the best way to capture more than 20 screenshots or different Urls?
I have tried the following code.
async function sCapture(url, site_name) {
 	const browser = await puppeteer.launch();
 	const page = await browser.newPage();
 	await page.setViewport({ width: 1280, height: 720 })
 	await page.goto(url);
 	await page.screenshot({
 	path:`statusImage/${site_name}.jpg`
 	});
 	await browser.close();
 }

 Am getting the Urls from my DB like this.

 db_connection.promise()
.execute("SELECT * FROM `urls`")
.then(([rows]) => {

    rows.forEach(user => {
        const url 	= user.link;
    	const name 	= user.link_name;
    	console.log(name);
    	sCapture(url, name)

    });
	db_connection.end();
}).catch(err => {
    console.log(err);
});

Because my DB Table contains more than 50 urls
Before, I was getting this error:
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
After I added the line below. Its just killing my server and I have to do a manual reboot for my site to work again.
require('events').EventEmitter.prototype._maxListeners = 100;
I will appreciate any help rendered.

Be aware that there are well known event listener leaks in several base Node modules.  For example, net socket has this issue.


Joints do not have a "controlled by vehicle" parameter

There's no way for this value to be set by the library, so there's no way to make joints controllable by vehicles, despite it being possible to do in the game's level editor and via LevelXML.

Solved in commit https://github.com/ultrako/LevelXML/commit/4f200d8ebd396d0d743e8aa4aaecd9189185085c


Remove unused classes kept for unused backward compatibility

@bot-gradle test rFR

OK, I've already triggered ReadyForRelease build for you.

Should be handled as part of #15585


added home screenshot in readme and added styling in home

Added the screenshot  of home in Readme and enhanced the CSS, page looks a bit visually appealing now, on all devices

Thanks @vp4140 !


Run one build on two CPUs

I don't know if it's currently possible but I want to use the full potential of my server by running a build on two CPUs. With my actual config I can run two builds simultaneously but when I have only one build running (and it is what happens most of the time) it can use only half of the power of my instance.
Can we configure Drone to launch docker container with the good configuration for that ? Maybe something like the docker config CpuSet ?
We have an heavy build that take 30 minutes to run, to divide it by 2 would be very great to improve the feedback loop of continuous integration.

hey there, agreed, we're tracking this in #902 which has a yaml file that better aligns with docker / docker compose and exposes more options

specifically, you can see how we'll model the yaml internally:
https://github.com/drone/drone/issues/902#issuecomment-77308459
which will expose most (eventually all) of the available configuration options:
http://godoc.org/github.com/samalba/dockerclient#ContainerConfig
I'm going to close this in favor of #902 which encapsulates this functionality. Thanks for the feedback!

Thanks for your answer. I will follow the progress of this issue !


"Type" for members of an environment should not include auth provider.

Racher server build from master
Setup - ping federator auth enabled setup
Log in as any admin user to a ping federator auth enabled setup.
Go to "Manage Environment"
Under "Access Control" , "Type" for members of an environment is listed as "shibboleth_user?".

With any auth provider , we expect the "type" to be shown as user/group without including auth provider name.

Version - 1.6.1-rc10
Now says "User"
Verified fixed


Frontend sometimes the websocket connecting drops randomly find why

Found it it was a key error that is caught as a connection error and ending the websocket.


flutter analyze is 10x pessimistic about its speed

String elapsed = (stopwatch.elapsedMilliseconds / 100.0).toStringAsFixed(1);

I know the analyzer's slow, but it's not that slow. That 100.0 should be 1000.0 to convert from milliseconds to seconds.

cc @eseidel


Flake: added IDTime

Coverage: 82.705% (+0.06%) from 82.648% when pulling fa3a52a98c7444fc4f942f4996e70c59ad5fbd74 on dissoupov into 2c9b4cbf37a7ee97d03d880e86b14121b30d6459 on main.


test: unexpected PG regress test results comparison of the last turned-off cases

Describe the bug
At #6410 , in the modified float4.sql, I made a workaround to make the regress test succeed by adding --@  to the last three lines, thus the last 4 lines are
--@ 	offset 0) s;
--@
--@ -- clean up, lest opr_sanity complain
--@ drop type xfloat4 cascade;

However, if we turn off the last case the right way, i.e., the last 4 lines being:
--@ 	offset 0) s;

-- clean up, lest opr_sanity complain
--@ drop type xfloat4 cascade;

Although the last test case is turned off, the expected output and the actual output are still compared, producing:
2022-11-16T16:16:24.625315Z ERROR risingwave_regress_test::schedule: query input:
-- clean up, lest opr_sanity complain
--@ drop type xfloat4 cascade;
2022-11-16T16:16:24.625587Z ERROR risingwave_regress_test::schedule: Diff:
-NOTICE:  drop cascades to 6 other objects
-DETAIL:  drop cascades to function xfloat4in(cstring)
-drop cascades to function xfloat4out(xfloat4)
-drop cascades to cast from xfloat4 to real
-drop cascades to cast from real to xfloat4
-drop cascades to cast from xfloat4 to integer
-drop cascades to cast from integer to xfloat4

To Reproduce
modify the last 4 lines of src/tests/regress/data/sql/float4.sql to be
--@ 	offset 0) s;

-- clean up, lest opr_sanity complain
--@ drop type xfloat4 cascade;

and run
rm -rf `pwd`/src/tests/regress/output

./risedev k
./risedev clean-data
./risedev d

RUST_BACKTRACE=1 target/debug/risingwave_regress_test -h <IP_ADDRESS> \
  -p 4566 \
  -u root \
  --input `pwd`/src/tests/regress/data \
  --output `pwd`/src/tests/regress/output \
  --schedule `pwd`/src/tests/regress/data/schedule \
  --mode risingwave

, we'll get
2022-11-16T16:16:24.625315Z ERROR risingwave_regress_test::schedule: query input:
-- clean up, lest opr_sanity complain
--@ drop type xfloat4 cascade;
2022-11-16T16:16:24.625587Z ERROR risingwave_regress_test::schedule: Diff:
-NOTICE:  drop cascades to 6 other objects
-DETAIL:  drop cascades to function xfloat4in(cstring)
-drop cascades to function xfloat4out(xfloat4)
-drop cascades to cast from xfloat4 to real
-drop cascades to cast from real to xfloat4
-drop cascades to cast from xfloat4 to integer
-drop cascades to cast from integer to xfloat4

Expected behavior
Succeed without error
Additional context
#6410 #5440

Could you please take a look? cc: @xiangjinwu


PT#138943351 remove designer state

https://www.pivotaltracker.com/story/show/138943351
No notable visual changes, BUT our code coverage goes up 🌮 💃

@AllenBW PLease avoid pushing branches to the main repo (as opposed to your fork).  Not sure how you were able to push in the first place 😕

@Fryguy apologies for this!!!! yeah this is a huge no no. (what i get for not reading on a friday afternoon)


[Task]: Release apache-beam-jupyterlab-sidepanel with notebook-managed cluster manager widget

What needs to happen?
Release the cluster manager UI widget (JupyterLab extension) for the interactive FlinkRunner on notebook-managed clusters feature.
The release is standalone from other Beam releases and happens to Beam's svn repo and NPM.
Issue Priority
Priority: 2
Issue Component
Component: runner-py-interactive

Documented here
Git repo tag: https://github.com/apache/beam/releases/tag/jupyterlab-sidepanel-v3.0.0
Git commit: https://github.com/apache/beam/commit/e679b49e755e28b15f4900b6e450177907c52fd1
Staged to Beam dist/dev repo: https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v3.0.0/

Voting in dev list...

Released to NPM: https://www.npmjs.com/package/apache-beam-jupyterlab-sidepanel.


Add VNet Integration for managed API in Static Web App

The need: reaching a VNet-bound resource from a managed API in Static Web App.
A VNet-bound resource could be a VM, or - more likely - a data source with a Private Endpoint (CosmosDB, SQL Database, Storage Account).
Today the data source must allow incoming public traffic, without even the ability of filtering by Service Tag. Most of the times this is forbidden by company policies.
If Static Web Apps could integrate into a VNet, that could solve the issue, i.e. one could block public access to the data source.

I have the same issue, any ideas?

Thanks for the feedback @PaoloCastAway and @ics-development, this is something that we've also heard others ask for. At the moment, due to Static Web Apps' distributed architecture, it can't be deployed to VNETs (and therefore cannot proxy requests to network isolated backends). We're definitely noting the feedback.

Hey @PaoloCastAway and @ics-development! I've been investigating this topic since I saw this thread. Regarding the managed API, it would not be possible to integrate it with VNET due to our distributed architecture like I mentioned, and also due to the fact that managed APIs are consumption Functions that have limited network integration options.
What you could do is create a standalone Function App with one of the plans that supports virtual network integration (https://learn.microsoft.com/en-us/azure/azure-functions/functions-networking-options?tabs=azure-cli#matrix-of-networking-features), enable virtual network integration on that Function App to allow outbound requests on that Function App to access your resources in the virtual network, and then link that resource to your Static Web App resource as a Bring Your Own Function.
With this method, the Function App would be publicly accessible so that Static Web Apps' distributed runtime can route calls to these. Note that we lock down the inbound traffic to your Function App to ensure it only comes from us (Static Web Apps) for the backend integration feature.
On a similar note, I've written up a blog post considering different approaches for integrating network-bound APIs with Static Web Apps which may be an interesting read as well.

Hey @PaoloCastAway and @ics-development! I've been investigating this topic since I saw this thread. Regarding the managed API, it would not be possible to integrate it with VNET due to our distributed architecture like I mentioned, and also due to the fact that managed APIs are consumption Functions that have limited network integration options.
What you could do is create a standalone Function App with one of the plans that supports virtual network integration (https://learn.microsoft.com/en-us/azure/azure-functions/functions-networking-options?tabs=azure-cli#matrix-of-networking-features), enable virtual network integration on that Function App to allow outbound requests on that Function App to access your resources in the virtual network, and then link that resource to your Static Web App resource as a Bring Your Own Function.
With this method, the Function App would be publicly accessible so that Static Web Apps' distributed runtime can route calls to these. Note that we lock down the inbound traffic to your Function App to ensure it only comes from us (Static Web Apps) for the backend integration feature.
On a similar note, I've written up a blog post considering different approaches for integrating network-bound APIs with Static Web Apps which may be an interesting read as well.
Hope this helps!

Azure frontdoor have private endpoint integration, they figured out how to do that with distributed nature of FrontDoor, so shall be possible with static web apps too from that point of view.

@artisticcheese Static Web Apps also has the ability to have a private endpoint to restrict any inbound traffic to a specific vnet. However, vnet integration to communicate to any downstream backend services is not possible with managed APIs of Static Web Apps. This is a use case we are looking into improving however, but at this time, the approaches I document above are recommended.

I did not mean private endpoints for incoming connection but Frontdoor allows connections to services behind private endpoints (not exposed externally) in FrontDoor premium (https://learn.microsoft.com/en-us/azure/frontdoor/private-link), so technically Azure already allowed similar setup for different resource type with similar characteristic to Static Web Apps.

@thomasgauvin I am trying to follow your linked guide, but I get a 401 when I try to access the API after linking it instead of having it managed. I am deploying the existing api folder to a function app (running in node and tested unauthorized e.g. /api/ works).. I am using a custom B2C provider.  Do I need to setup the B2C identity provider in conjunction with the automatically created "Azure Static Web Apps (Linked)" provider in the authorization setup on the function app? I am setting it up exclusively on the staging environment (before moving the setup to production). Could that be an issue? Are there any other issues I need to be aware of to get authorized into the API function app?

I would be very keen to see this feature added too. Due to policy we are not allowed to turn on public networking for functions, container apps etc. and I imagine that this cannot be uncommon in other organisations.
It would also be useful to have an error message on creation of a linked backend to a resource that doesn't have public networking enabled as this would really help with diagnosing problems.
I had been tearing my hair out trying to understand what could have been causing a "Backend call failure" when the linked backend was successfully created and the auth appeared to be setup properly. After much searching I found this thread and realised it must be that the Static Web App backend cannot see our Container App sat on the VNet.

When will this available?? It is a failure in the architecture design of SWA, since it has the option to be behind a private endpoint but no access to private apps service it has no sense at all.

+1


Vacuum docs should be updated to remove no-op comment

VACUUM now actually does something, see:

https://github.com/duckdb/duckdb/pull/7794
https://github.com/duckdb/duckdb/pull/7824
https://github.com/duckdb/duckdb/issues/109
https://github.com/duckdb/duckdb/issues/9263

It only vacuums indexes currently - not yet tables

The index vacuum functionality is not coupled to the VACUUM statement. It happens implicitly after each operation/query altering the index. For example, after inserting 1M new values, the index will trigger a vacuum operation if the potential freeable memory passes a threshold. This happens in the LocalStorage::Flush, without the need to call VACUUM explicitly. I've also added similar code to clean up the index at locations other than LocalStorage::Flush, e.g., when rolling back due to a constraint violation, deleting from an index, etc.
	// possibly vacuum any excess index data
	table.info->indexes.Scan([&](Index &index) {
		index.Vacuum();
		return false;
	});

As far as I can remember I have not touched VACUUM 🤔.

@szarnyasg: other than postgres, we never rebuild indexes. This PR removes the last code paths that do so when replaying the WAL. Once this is merged, we only build indexes when creating them and then only alter them in any subsequent queries.

Thanks. So the plain VACUUM is still doing a no-op. I'm pushing some minor adjustments to the docs and closing this.