Build file '.pub-cache/hosted/pub.dev/audioplayers_android-3.0.1/android/build.gradle' line: 31

I am getting this error when running flutter run
FAILURE: Build failed with an exception.

* Where:
Build file '/Users/one/.pub-cache/hosted/pub.dev/audioplayers_android-3.0.1/android/build.gradle' line: 31

* What went wrong:
A problem occurred evaluating project ':audioplayers_android'.
> No signature of method: build_a63qjm1kfe3oz0etex305ie55.android() is applicable for argument types: (build_a63qjm1kfe3oz0etex305ie55$_run_closure2) values: [build_a63qjm1kfe3oz0etex305ie55$_run_closure2@16440ebf]

The flutter run --verbose shows this output:
 > Configure project :audioplayers_android
[        ] Evaluating project ':audioplayers_android' using build file '/Users/one/.pub-cache/hosted/pub.dev/audioplayers_android-3.0.1/android/build.gradle'.
[        ] Creating configuration androidTestUtil
[        ] FAILURE: Build failed with an exception.
[   +1 ms] * Where:
[        ] Build file '/Users/one/.pub-cache/hosted/pub.dev/audioplayers_android-3.0.1/android/build.gradle' line: 31
[        ] * What went wrong:
[        ] A problem occurred evaluating project ':audioplayers_android'.
[        ] > No signature of method: build_a63qjm1kfe3oz0etex305ie55.android() is applicable for argument types: (build_a63qjm1kfe3oz0etex305ie55$_run_closure2) values: [build_a63qjm1kfe3oz0etex305ie55$_run_closure2@5a21456d]
[        ] * Try:
[        ] Run with --debug option to get more log output. Run with --scan to get full insights.
[        ] * Exception is:
[        ] org.gradle.api.GradleScriptException: A problem occurred evaluating project ':audioplayers_android'.
[        ]      at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:93)
[        ]      at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:133)
[        ]      at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:77)
[        ]      at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:136)
[        ]      at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
[        ]      at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
[        ]      at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
.
.
.
.

Please guide how to proceed.
My pubspec.yaml file is:
name: pavlov_alarm
description: A new Flutter application.

publish_to: 'none' # Remove this line if you wish to publish to pub.dev

version: 1.0.0+1

environment:
  sdk: ">=2.12.0 <3.0.0"

dependencies:
  flutter:
    sdk: flutter

  cupertino_icons: ^1.0.3
  intl: ^0.18.0
  # android_alarm_manager: ^2.0.2
  android_alarm_manager_plus: ^2.1.2
  shared_preferences: ^2.0.7
  url_launcher: ^6.0.10
  flutter_local_notifications: ^13.0.0
  flutter_local_notifications_platform_interface: ^6.0.0
  timezone: ^0.9.1
  rxdart: ^0.27.2
  flutter_native_timezone: ^2.0.0
  # flutter_datetime_picker: ^1.5.1
  # vibration: ^1.7.4-nullsafety.0
  vibration: ^1.7.6
  audioplayers: ^4.0.1
  # audioplayers: ^3.0.1
  # audioplayers: ^0.20.1
  future_progress_dialog: ^0.2.0
  flutter_credit_card: ^3.0.6
  flutter_stripe: ^9.0.0
  http: ^0.13.3
  hive: ^2.0.4
  hive_flutter: ^1.1.0
  path_provider: ^2.0.3
  flutter_picker: ^2.0.2
  hive_generator: ^2.0.0
  # hive_generator: ^1.1.1
  build_runner: ^2.1.2

dev_dependencies:
  flutter_test:
    sdk: flutter

flutter:
  uses-material-design: true
  fonts:
    - family: JosefinSans
      fonts:
        - asset: assets/fonts/JosefinSans-Regular.ttf
  assets:
    - assets/sound1.wav

Thanks

Plz use the issue template.
Also did you try flutter pub cache cleanfollowed by flutter pub get.
What OS? What flutter version?

I have the same problem
I've tried your suggestion with flutter version 3.7.11, and android 9 and 12 and it doesn't work
Thanks

I hava same problem:
problem:
FAILURE: Build failed with an exception.

Where:
Build file 'C:\Users\Administrator\AppData\Local\Pub\Cache\hosted\pub.dev\audioplayers_android-3.0.1\android\build.gradle' line: 44

What went wrong:
A problem occurred evaluating project ':audioplayers_android'.

Could not find method disable() for arguments [InvalidPackage] on task ':audioplayers_android:lint' of type com.android.build.gradle.tasks.LintGlobalTask.

Try:

Run with --stacktrace option to get the stack trace.
Run with --info or --debug option to get more log output.
Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 1s
Android 13
Flutter (Channel master, 3.10.0-5.0.pre.1, on Microsoft Windows [版本 10.0.22621.1555], locale zh-CN)
• Flutter version 3.10.0-5.0.pre.1 on channel master at D:\flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 954760829b (25 hours ago), 2023-04-12 22:42:12 -0400
• Engine revision 39ec48fd7e
• Dart version 3.0.0 (build 3.0.0-437.0.dev)
• DevTools version 2.23.1

Same problem here...

+1

I had audioplayers 1.0.1 integrated into my app. Then I updated to the latest (4.0.1) and got the same error.
When I rolled back to 1.0.1 the error was gone.
I've tried cleaning project and pub cache
Reopening VSCode
Upgrading Dart and Flutter to the latest versions
Nothing helped.
So, for now, keeping 1.0.1 and waiting for the resolution

**/.pub-cache/hosted/pub.flutter-io.cn/audioplayers_android-3.0.1/android/build.gradle
// apply plugin: 'com.android.library'

Can someone post their build.gradle file in android/app folder? Seems like you aren't allowed to use compileKotlin in newer versions :)
See: https://youtrack.jetbrains.com/issue/KT-45183
For a working template see: https://github.com/bluefireteam/audioplayers/blob/main/packages/audioplayers/example/android/build.gradle

@Gustl22 , please find it attached.
android/app/build.gradle

def localProperties = new Properties()
def localPropertiesFile = rootProject.file('local.properties')
if (localPropertiesFile.exists()) {
localPropertiesFile.withReader('UTF-8') { reader ->
localProperties.load(reader)
}
}
def flutterRoot = localProperties.getProperty('flutter.sdk')
if (flutterRoot == null) {
throw new GradleException("Flutter SDK not found. Define location with flutter.sdk in the local.properties file.")
}
def flutterVersionCode = localProperties.getProperty('flutter.versionCode')
if (flutterVersionCode == null) {
flutterVersionCode = '1'
}
def flutterVersionName = localProperties.getProperty('flutter.versionName')
if (flutterVersionName == null) {
flutterVersionName = '1.0'
}
apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply from: "$flutterRoot/packages/flutter_tools/gradle/flutter.gradle"
android {
compileSdkVersion flutter.compileSdkVersion
compileOptions {
    sourceCompatibility JavaVersion.VERSION_1_8
    targetCompatibility JavaVersion.VERSION_1_8
}

kotlinOptions {
    jvmTarget = '1.8'
}

sourceSets {
    main.java.srcDirs += 'src/main/kotlin'
}

defaultConfig {
    // TODO: Specify your own unique Application ID (https://developer.android.com/studio/build/application-id.html).
    applicationId "com.soven.brain_train"
    minSdkVersion flutter.minSdkVersion
    targetSdkVersion flutter.targetSdkVersion
    versionCode flutterVersionCode.toInteger()
    versionName flutterVersionName
}

buildTypes {
    release {
        // TODO: Add your own signing config for the release build.
        // Signing with the debug keys for now, so `flutter run --release` works.
        signingConfig signingConfigs.debug
    }
}

lintOptions {
    checkReleaseBuilds false
}

}
flutter {
source '../..'
}
dependencies {
implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
}

To get rid off this problem you have to update gradle in your project to min 7.3. I also had to change to:
distributionUrl=https://services.gradle.org/distributions/gradle-7.4-all.zip
in gradle-wrapper

I have just changed the gradle configuration as stated before:
classpath 'com.android.tools.build:gradle:7.3.1'
distributionUrl=https://services.gradle.org/distributions/gradle-7.4-all.zip

And I could generate my app for Android in a Windows environment.
I will check Linux and post the results here later! Thanks!

Has there been any progress on this issue? I am having the same problem

@luomo-pro plz try the solution right above your comment...

I have successfully run my software and modified gradle to 7.3.1 as described above.
However, after the modification, I also encountered a lot of problems, here to say, I hope it will help others:

after gradle to 7.3.1, the use of kotlin version must be 1.5.20 and above.
My software uses a lot of packages on pub, and the kotlin version of those packages are relatively low, I can only manually modify the build.gradle file of each package one by one.
If these packages are subsequently upgraded, but the kotlin version is not modified, then I have to modify them again, which is very troublesome.
there are some build.gradle configuration, in gradle7.0 or above is no longer, if you still include these configuration content, it will report an error, and the error is reported inexplicably, no detailed error message.
For example, the most critical, in 7.0 and above, has been removed
useProguard true
If you still use this, it will compile with errors.
In short, if your software is older and uses a lot of packages, then upgrading to gradle 7.3 will require a lot of modifications to work properly.

Such issue appears on projects with Android Gradle plugin < 4.2, but I would suggest to move to 7.x versions already, like the person above did.
Here is a discussion about an issue: https://github.com/flutter/flutter/issues/125181#issuecomment-1525894244

Even after setting
distributionUrl=https://services.gradle.org/distributions/gradle-7.4-all.zip
still getting the error

What went wrong:
A problem occurred evaluating project ':audioplayers_android'.

No signature of method: build_6kvx1qs7mu428qnu2nzhgyojp.android() is applicable for argument types: (build_6kvx1qs7mu428qnu2nzhgyojp$_run_closure2) values: [build_6kvx1qs7mu428qnu2nzhgyojp$_run_closure2@2b761be6]

-- Anyone has any suggestions how to fix the issue?

Even after setting
distributionUrl=https://services.gradle.org/distributions/gradle-7.4-all.zip

You changed the version of your Gradle wrapper, which is also something that you need to do.
Android Gradle Plugin version should be changed in build.gradle of Android folder of your project.

in my androig/build.gradle
putting
 ext.kotlin_version = '1.7.10'
 and dependencies to classpath 'com.android.tools.build:gradle:7.3.1'

Has got rid of audio player issue but now I ran into another lib giving me trouble :D
project ':flutter_config' -> org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.61

Had the same issue. But there is a solution on Stack Overflow to this, see

https://stackoverflow.com/questions/76067863/no-signature-of-method-in-flutter-project

The solution is to remove
lint {
        disable 'InvalidPackage'
    }

from the .pub-cache/hosted/pub.dev/audioplayers_android-3.0.1/android/build.gradle gradle file.

Had the same issue. But there is a solution on Stack Overflow to this, see

https://stackoverflow.com/questions/76067863/no-signature-of-method-in-flutter-project

The solution is to remove
lint {
        disable 'InvalidPackage'
    }

from the .pub-cache/hosted/pub.dev/audioplayers_android-3.0.1/android/build.gradle gradle file.

This is not a solution, don't mislead other users. Updating Android Gradle Plugin to >= 4.2.0 in android folder of your project is a solution that wouldn't require updating all packages.

FAILURE: Build failed with an exception.

What went wrong:
A problem occurred configuring project ':audioplayers_android'.

Could not load compiled classes for build file 'C:\Users\imroh\AppData\Local\Pub\Cache\hosted\pub.dev\audioplayers_android-3.0.2\android\build.gradle' from cache.
Failed to notify project evaluation listener.
Could not get unknown property 'android' for project ':audioplayers_android' of type org.gradle.api.Project.
Could not get unknown property 'android' for project ':audioplayers_android' of type org.gradle.api.Project.
pubsec.txt

What could be the possible issue?

So, what is the standard solution?


Add preview pane item support.

To make previewable pane items part of the atom core as discussed in #3683 and https://github.com/atom/tree-view/issues/3, I added this pull request to help speed up the process.
I added a previewItem property similar to the activePane property, which allows for simple implementation in the tree-view and tabs packages.
I know this is not a high priority for atom right now, but if this is helpful I can add specs and post my tree-view and tabs implementation as well?
Suggestions are very welcome! :smiley:

This has been superseded by https://github.com/atom/tabs/pull/145, so I'm going to close this now.


Issue with spectator mode noclip

Receiving information about going spectator gamemode allows players to clip through walls. Cancelling this on self would result in inability to go through walls.

If I am in spectator gamemode, and rejoin the server, I can't clip through walls. I think velocitab cause this. And this is the explain in plugin TAB

Hi there - I can't reproduce this (latest build as of time of writing).
https://gyazo.com/493995f8f872406996574b3e454bc6b3
Velocitab already has logic to prevent the issue described in TAB's documentation - the remove_spectator_effect option doesn't cancel spec mode for the player themselves.
# Remove gamemode spectator effect for other players in the TAB list.
remove_spectator_effect: true


Added playback hotkey in media preview to keyboard shortcuts documentation

Added hotkey for media playback in keyboard shortcuts documentation to reflect my recent pull request in main Files repository.

Awesome, thank you for adding this feature and updating the documentation @TomaszSteblik ! @yaichenbaum Should we merge this now or wait until the feature is also available to users?

Good question, let's wait until the next release before merging this.

Then let's wait, @yaichenbaum do you happen to know approximately when this could be?

Then let's wait, @yaichenbaum do you happen to know approximately when this could be?

@chingucoding between one to two months.

@TomaszSteblik can you resolve the conflicts?

@TomaszSteblik can you resolve the conflicts?

Of course


anyNumber in dsl generate String numbers in json

full contract, full stub and the generated test are uploaded.
web.zip

Could you please upload it as a github project? It will be much easier to collaborate. Thanks!

uploaded here!!

The example you've presented (BTW I couldn't run it cause you were missing base class for the generated test and production code...)
Having such code
@SpringBootApplication
public class AnynumberApplication {

	public static void main(String[] args) {
		SpringApplication.run(AnynumberApplication.class, args);
	}
}

@RestController
class Controller {
	@RequestMapping(value = "/simple/", produces = MediaType.APPLICATION_JSON_VALUE)
	String foo() {
		return "{ \"number\": 123 }";
	}
}

such a base class
public class BaseClass {

	@Before
	public void setup() {
		RestAssuredMockMvc.standaloneSetup(new Controller());
	}
}

such a contract

import org.springframework.cloud.contract.spec.Contract

Contract.make {
    request {
        method "GET"
        url  value(regex('/simple/[a-z]*'))
    }
    response { // (6)
        status 200 // (7)
        body(
                number: anyNumber()
        )
        headers { // (9)
            contentType('application/json')
        }
    }
}

such a generated test
public class ContractVerifierTest extends BaseClass {

	@Test
	public void validate_simple() throws Exception {
		// given:
			MockMvcRequestSpecification request = given();

		// when:
			ResponseOptions response = given().spec(request)
					.get("/simple/");

		// then:
			assertThat(response.statusCode()).isEqualTo(200);
			assertThat(response.header("Content-Type")).matches("application/json.*");
		// and:
			DocumentContext parsedJson = JsonPath.parse(response.getBody().asString());
			assertThatJson(parsedJson).field("['number']").matches("-?\\d*(\\.\\d+)?");
	}

}

Doesn't generate any String numbers. However, it doesn't assert the types ( I guess that's what you mean...).  So this test passes if the response json is
{ "number": 123 }

and
{ "number": "123" }

which is a bug. I'll try to address it in branch 2.0.x (I'll bump jsonassert and add a feature to check types)

Can you @hatzhang confirm that my understanding is correct?

I want the stub boot application to return Number type value when the value is defined in contract using anyNumber。
After run generateStubs, the following mapping was generated:
    {   "id" : "e89105e8-808f-40d9-9d92-32f2b8ba6457",   "request" : {     "urlPattern" : "/simple/[a-z]*",     "method" : "GET"   },   "response" : {     "status" : 200,     "body" : "{\"number\":\"-1207046272\"}",     "headers" : {       "Content-Type" : "application/json"     },     "transformers" : [ "response-template" ]   },   "uuid" : "e89105e8-808f-40d9-9d92-32f2b8ba6457" }
but i think the body part should be like this:
 "{\"number\":-1207046272}"

Oh so in the generated stub the problem's there too... Thanks for clearing this up for me!

i have some other concerns about this:

i think contract dsl should support all json native types
there may be need for String type number
wonder what happens when defining "anynumber"(or regx) in restdoc, then use that to generate dsl, then to mapping.

I have noticed a similar problem in 1.1.3 and 1.2.1
My expected response in groovy will be like this for one of the field. When I run the build it passes.
"id"           : $(anyNumber())
Now I will change the field name in the expected response. When I run the build it passes, which seems to be a bug for me. The bug might be in your anyNumber regular expression.
"ids"           : $(anyNumber())
Again I have changed the expected response as shown below and the build fails noting the field ids doesn't exist.
"ids"           : $(regex('^\\d+\\.?\\d*$'))
Just give a try in the project given by @hatzhang

@hatzhang @vkrams - thanks for the feedback. That's really valuable and will help to get around this!

Somewhat related:
        body([
            coordinate: [
                lat: 48.5,
                lng: 9.5
            ]
        ])

leads to
{"coordinate":{"lng":"9.5","lat":"48.5"}}

instead of
{"coordinate":{"lng":9.5,"lat":48.5}}

so instead of 9.5 we have "9.5"


added WhatsApp Bot #5 (RaspberryPi + ngrok + Twilio)

Added a new project link to Readme.md file.  Thanks!

Thanks @hujanais 🙏 😄


Update double.dart

80 columns comments fix.

This is my first contribution. ;-)

[x] I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.

Note that this repository uses Gerrit for code reviews. Your pull request will be automatically converted into a Gerrit CL and a link to the CL written into this PR. The review will happen on Gerrit but you can also push additional commits to this PR to update the code review.

Thanks for noticing the overlong line, and fixing it. This PR is trying to do a little too much, but that particular long line was too long, and reflowing it was the correct choice.

@Jonadabe would you like to follow up or should we close this PR?


Improve checkmark toggle logic

Continues from https://github.com/fregante/webext-domain-permission-toggle/pull/33
Fixes #33
Should fix #13

Completely untested at this point. Will review once I have more time

cc @aspiers

I think the description above is supposed to say "Fixes #29"?

But I like the look of this!  Maybe I will test shortly.

Tested and I can confirm that this fixes #29.

It seems to work in my tests too 🙌
Thanks for starting the discussion!


Simple Composition Container throws exception when there are assemblies with generic interfaceswhich inherit from ICompositionModule

Add class constraint

Solid area


IT-3421: Build just before pushing image

Previously the workflow (1) built the Docker image, (2) saved as a tar-blob, (3) scanned (using Trivy), (4) reload the tar-blob as an image and pushed the image to ghcr.io.  However it turns out that when a tar-blob is reloaded, crucial metadata about the image is lost.  One discussion on the topic is here..  The fix is to build the image just before pushing (with no save/load in between).  Building the image twice makes the workflow less efficient but it works.

If you want to try again with your original approach, I see that docker has docker save and docker load commands that might do what you want.
This PR is also fine the way it is.

A quick experiment shows that save/load works where export/import did not.  Thank you for the suggestion, @jesusaurus .


Documentation - Default Task Name Incorrect

In the GMS documentation, screenshot shows the GMA default task name however it is preceded by a reference to the 1.0 default.
on step 3, "Reload task of Metrics Library" should change to "Reload Governed Metrics Application" and the screen shot should align.

Resolved


[WIP] Tidy MathListIndex

Currently MathListIndex is supposed to have a SubIndex if an only if SubIndexType <> MathListSubIndexType.None. (Can you confirm that @Happypig375 ?)
This PR brings this constraint into the type structure, removing MathListSubIndexType.None and having:
public (MathListSubIndexType SubIndexType, MathListIndex SubIndex)? SubIndexInfo;

Yes.

Getting errors I'm having trouble diagnosing.
Inputs: 1, power, 2
Debug code gives:
InsertAndAdvance index before: [0, Superscript:0]
InsertAndAdvance index before: [0]
InsertAtAtomIndexAndAdvance atom: Number 2
InsertAtAtomIndexAndAdvance advance before: [0]
InsertAtAtomIndexAndAdvance advance after: [1]
InsertAndAdvance index at end: [1]
InsertAndAdvance index at end: [0, Superscript:0] // should be [0, Superscript:1]

I have the feeling it's to do with mutability and the refs. I didn't consider the refs when editing.
static void InsertAtAtomIndexAndAdvance(this MathList self, int atomIndex, MathAtom atom, ref MathListIndex advance

Independently of getting this PR working, it's good to remove the refs and make MathList immutable. This would involve more recursive code. But it could be good to get this PR working before tackling that I think.
Once this PR goes in and the immutable change we will have a clean base to work on https://github.com/verybadcat/CSharpMath/pull/117 .
Any thoughts @Happypig375 ?

Independently of getting this PR working, it's good to remove the refs

I don't think removing refs can be independent of this PR. Currently InsertAtAtomIndexAndAdvance advances the advance parameter and now that behaviour is gone, hence test failures.


LoRA finetune not working

So its not enough to edit the attentions.py to implement LoRA. Which makes sense, as I didnt really look at the VITS model structure. The MultiHeadAttention class is only applied for the TextEncoder part of the model. There are 3 other parts. So I need to do some more research into it.

If anyone has any ideas. Very interested to hear it

I need to read the LoRA paper in detail and see if i can add more lora layers to the VITS model

Removing LoRA for now.


Updated dashboard installation command to use v2.0.0

K8s dashboard release v2.0.0 is out: https://github.com/kubernetes/dashboard/releases
Time to update the docs.

/assign

/lgtm
/approve


LLVM error when running  scripts

:~/tvm$ sh ./tests/scripts/task_python_topi.sh
: CommandLine Error: Option 'debug-pass' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine option
Probably it's some mistake when building or LLVM related environment issues, could you guys give some clues? thanks.
ubuntu 14.0.4 & GNU 4.8.4 & LLVM 4.0

Try to use llvm's official binary repo. This does not seems to be a tvm issue. For quick fresh llvm installation.
You can try https://apt.llvm.org/ or http://releases.llvm.org/download.html to get prebuild binary

thanks, will try.


[kmac] fit: Message Masking

What does this commit?
This commit implements the message masking when a config bit is set.

Why is this beneficial?
By masking the message, the 1600bit (2share) storage becomes more
randomized by PRNG. It alleviates the distinct moment of the
prefix/key/message data shift moments. In the previous design, the
prefix, key are padded by 0. But as those blocks are XORed with PRNG
before fed into the SHA3 module, it is more randomized.

In contrast, with this config bit, the operation consumes more power as
LFSR runs when the message is fed into the hashing module.

Please review the last commit. first two commits are the PR #10461 contents.

Hey @timothytrippel could you please assign to proper DIF owner of KMAC? I added a new config field (msg_mask) in this PR.

@cindychip I updated DV to randomize msg_mask . Could you please check if it is OK? I've checked the waveform and sees that the msg is randomized.

Maybe... I should just turn on the LFSR when msg_valid && msg_ready not just msg_valid.


mem-ruby, gpu-compute: fix SQC/TCP requests to same line

Currently, the GPU SQC (L1I$) and TCP (L1D$) have a performance bug
where they do not behave correctly when multiple requests to the same
cache line overlap one another.  The intended behavior is that if the
first request that arrives at the Ruby code for the SQC/TCP misses, it
should send a request to the GPU TCC (L2$).  If any requests to the
same cache line occur while this first request is pending, they should
wait locally at the L1 in the MSHRs (TBEs) until the first request has
returned.  At that point they can be serviced, and assuming the line
has not been evicted, they should hit.
For example, in the following test (on 1 GPU thread, in 1 WG):
load Arr[0]
load Arr[1]
load Arr[2]
The expected behavior (confirmed via profiling on real GPUs) is that
we should get 1 miss (Arr[0]) and 2 hits (Arr[1], Arr[2]) for such a
program.
However, the current support in the VIPER SQC/TCP code does not model
this correctly.  Instead it lets all 3 concurrent requests go straight
through to the TCC instead of stopping the Arr[1] and Arr[2] requests
locally while Arr[0] is serviced.  This causes all 3 requests to be
classified as misses.
To resolve this, this patch adds support into the SQC/TCP code to
prevent subsequent, concurrent requests to a pending cache line from being
sent in parallel with the original one.  To do this, we add an
additional transient state (IV) to indicate that a load is pending to
this cache line.  If a subsequent request of any kind to the same cache
line occurs while this load is pending, the requests are put on the
local wait buffer and woken up when the first request returns to the
SQC/TCP.  Likewise, when the first load is returned to the SQC/TCP, it
transitions from IV --> V.
As part of this support, additional transitions were also added to
account for corner cases such as what happens when the line is evicted
by another request that maps to the same set index while the first load
is pending (the line is immediately given to the new request, and when
the load returns it completes, wakes up any pending requests to the same
line, but does not attempt to change the state of the line) and how GPU
bypassing loads and stores should interact with the pending requests
(they are forced to wait if they reach the L1 after the pending,
non-bypassing load; but if they reach the L1 before the non-bypassing
load then they make sure not to change the state of the line from IV if
they return before the non-bypassing load).
As part of this change, we also move the MSHR behavior from internally
in the GPUCoalescer for loads to the Ruby code (like all other
requests).  This is important to get correct hits and misses in stats
and other prints, since the GPUCoalescer MSHR behavior assumes all
requests serviced out of its MSHR also miss if the original request to
that line missed.
Although the SQC does not support stores, the TCP does.  Thus,
we could have applied a similar change to the GPU stores at the TCP.
However, since the TCP support assumes write-through caches and does not
attempt to allocate space in the TCP, we elected not to add this support
since it seems to run contrary to the intended behavior (i.e., the
intended behavior seems to be that writes just bypass the TCP and thus
should not need to wait for another write to the same cache line to
complete).
Additionally, making these changes introduced issues with deadlocks at
the TCC.  Specifically, some Pannotia applications have accesses to the
same cache line where some of the accesses are GLC (i.e., they bypass
the GPU L1 cache) and others are non-GLC (i.e., they want to be cached
in the GPU L1 cache). We have support already per CU in the above code.
However, the problem here is that these requests are coming from
different CUs and happening concurrently (seemingly because different
WGs are at different points in the kernel around the same time).
This causes a problem because our support at the TCC for the TBEs
overwrites the information about the GPU bypassing bits (SLC, GLC) every
time. The problem is when the second (non-GLC) load reaches the TCC, it
overwrites the SLC/GLC information for the first (GLC) load. Thus, when
the the first load returns from the directory/memory, it no longer has
the GLC bit set, which causes an assert failure at the TCP.
After talking with other developers, it was decided the best way handle
this and attempt to model real hardware more closely was to move the
point at which requests are put to sleep on the wakeup buffer from the
TCC to the directory. Accordingly, this patch includes support for that
-- now when multiple loads (bypassing or non-bypassing) from different
CUs reach the directory, all but the first one will be forced to wait
there until the first one completes, then will be woken up and
performed.  This required updating the WTRequestor information at the
TCC to pass the information about what CU performed the original request
for loads as well (otherwise since the TBE can be updated by multiple
pending loads, we can't tell where to send the final result to).  Thus,
I changed the field to be named CURequestor instead of WTRequestor since
it is now used for more than stores.  Moreover, I also updated the
directory to take this new field and the GLC information from incoming
TCC requests and then pass that information back to the TCC on the
response -- without doing this, because the TBE can be updated by
multiple pending, concurrent requests we cannot determine if this memory
request was a bypassing or non-bypassing request.  Finally, these
changes introduced a lot of additional contention and protocol stalls at
the directory, so this patch converted all directory uses of z_stall to
instead put requests on the wakeup buffer (and wake them up when the
current request completes) instead. Without this, protocol stalls cause
many applications to deadlock at the directory.
However, this exposed another issue at the TCC: other applications
(e.g., HACC) have a mix of atomics and non-atomics to the same cache
line in the same kernel.  Since the TCC transitions to the A state when
an atomic arrives. For example, after the first pending load returns to
the TCC from the directory, which causes the TCC state to become V, but
when there are still other pending loads at the TCC. This causes invalid
transition errors at the TCC when those pending loads return, because
the A state thinks they are atomics and decrements the pending atomic
count (plus the loads are never sent to the TCP as returning loads).
This patch fixes this by changing the TCC TBEs to model the number of
pending requests, and not allowing atomics to be issued from the TCC
until all prior, pending non-atomic requests have returned.
Change-Id: I37f8bda9f8277f2355bca5ef3610f6b63ce93563

Can you rebase this change to have only your changes?

Can you rebase this change to have only your changes? Let me or @Harshil2107 know if you have any problems or need any help.

This is done, although as I noted a couple minutes ago, when testing the weekly tests, some of them fail with a segfault when the line with a pending load is evicted by another memory request before the response is returned -- so I am still digging into that.  All per commit and daily GPU tests pass, so this is something exclusive to larger workloads with more contention.  I have attempted to model the support in the new ldmi action directly on how the GPU atomics and stores handle this same case in the TCP (the cache line potentially not being present in the cache on a return), but thus far running with gdb has been extremely difficult as the source of the segfault seems to keep changing when gdb and/or gem5.debug is used ...
Hoping @abmerop may spot something wrong with the ldmi action I added while I dig further.

@abmerop , @powerjg : I have a fix now for the Pannotia applications, but it presents a new challenge that requires discussion.
The issue is: in some of the kernels in the Pannotia applications have accesses to the same cache line where some of the accesses are GLC (i.e., they bypass the GPU L1 cache) and others are non-GLC (i.e., they want to be cached in the GPU L1 cache).  We have support already per CU in the above code.  However, the problem here is that these requests are coming from different CUs and happening concurrently (seemingly because different WGs are at different points in the kernel around the same time).
Here is the issue: our support at the L2 for the TBEs overwrites the information about the GPU bypassing bits (SLC, GLC) every time: https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/src/mem/ruby/protocol/GPU_VIPER-TCC.sm#738 --> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/src/mem/ruby/protocol/GPU_VIPER-TCC.sm#542.  The problem is when the second (non-GLC) load reaches the TCC, it overwrites the SLC/GLC information for the first (GLC) load.  Thus, when the the first load returns from the directory/memory, it no longer has the GLC bit set, which causes an assert failure at the TCP.
I see three possible fixes, but need feedback:

We change the pending requests at the TCC/L2 to not allow additional requests for a cache line to pass through while another request to the same line is pending.  Basically this: https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/src/mem/ruby/protocol/GPU_VIPER-TCC.sm#737, would do a st_stallAndWaitRequest instead.  I have implemented this locally and verified that the Pannotia apps work with this change.  However, it is a big change to the L2 as it now (correctly?) blocks multiple requests from the same cache line from going beyond the L2 in parallel.

We change the way messages flow through Ruby such that we send the GLC/SLC bits all the way to the final destination (e.g., Directory, Main Memory) such that we can pass them all the way back -- this way the information can never be overwritten by another concurrent request.  But it requires changes to components that are not solely used by the GPU.  Originally @v-ramadas had made this change: https://gem5-review.googlesource.com/c/public/gem5/+/66991/5/src/mem/ruby/protocol/RubySlicc_MemControl.sm#77, but @powerjg 's feedback was to use the TBE at the L2 instead, which @v-ramadas subsequently did: https://gem5-review.googlesource.com/c/public/gem5/+/67191.  Given the new access pattern we didn't envision at the time, we could revert to Vishnu's old version and resolve this problem without blocking requests at the L2 like above in option 1?

Have L2 TBEs track multiple pending requests to the same line, which would then require a) changes to the TBE to hold multiple pending requests and b) need a unique identifier to differentiate these requests once the request returns.  This is probably the most invasive change, but I suspect probably more realistic relative to real hardware?

Please let me know what you think.  I can push the updates to this accordingly.

@abmerop , @powerjg : I have a fix now for the Pannotia applications, but it presents a new challenge that requires discussion.
The issue is: in some of the kernels in the Pannotia applications have accesses to the same cache line where some of the accesses are GLC (i.e., they bypass the GPU L1 cache) and others are non-GLC (i.e., they want to be cached in the GPU L1 cache). We have support already per CU in the above code. However, the problem here is that these requests are coming from different CUs and happening concurrently (seemingly because different WGs are at different points in the kernel around the same time).
Here is the issue: our support at the L2 for the TBEs overwrites the information about the GPU bypassing bits (SLC, GLC) every time: https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/src/mem/ruby/protocol/GPU_VIPER-TCC.sm#738 --> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/src/mem/ruby/protocol/GPU_VIPER-TCC.sm#542. The problem is when the second (non-GLC) load reaches the TCC, it overwrites the SLC/GLC information for the first (GLC) load. Thus, when the the first load returns from the directory/memory, it no longer has the GLC bit set, which causes an assert failure at the TCP.
I see four possible fixes, but need feedback:
1. We change the pending requests at the TCC/L2 to not allow additional requests for a cache line to pass through while another request to the same line is pending.  Basically this: https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/src/mem/ruby/protocol/GPU_VIPER-TCC.sm#737, would do a st_stallAndWaitRequest instead.  I have implemented this locally and verified that the Pannotia apps work with this change.  However, it is a big change to the L2 as it now (correctly?) blocks multiple requests from the same cache line from going beyond the L2 in parallel.

2. We change the way messages flow through Ruby such that we send the GLC/SLC bits all the way to the final destination (e.g., Directory, Main Memory) such that we can pass them all the way back -- this way the information can never be overwritten by another concurrent request.  But it requires changes to components that are not solely used by the GPU.  Originally @v-ramadas had made this change: https://gem5-review.googlesource.com/c/public/gem5/+/66991/5/src/mem/ruby/protocol/RubySlicc_MemControl.sm#77, but @powerjg 's feedback was to use the TBE at the L2 instead, which @v-ramadas subsequently did: https://gem5-review.googlesource.com/c/public/gem5/+/67191.  Given the new access pattern we didn't envision at the time, we could revert to Vishnu's old version and resolve this problem without blocking requests at the L2 like above in option 1?

3. Have L2 TBEs track multiple pending requests to the same line, which would then require a) changes to the TBE to hold multiple pending requests and b) need a unique identifier to differentiate these requests once the request returns.  This is probably the most invasive change, but I suspect probably more realistic relative to real hardware?

4. Require bypassing accesses to create a TBE at the L1/TCP, which we can use to get the SLC/GLC information when it returns.  This would prevent bypassing and non-bypassing requests from proceeding in parallel from the same CU, ensurnig that the TBE information has the correct SLC/GLC information.  However, this would not allow even multiple non-bypassing requests from the same CU to proceed in parallel, which seems like the opposite of what hardware would do.

Please let me know what you think. I can push the updates to this accordingly.

After talking with Brad and @abmerop , we decided the best way to model this was option 5: have the directory be the point at which we do not allow additional requests for a cache line to pass through while another request to the same line is pending.  Brad felt this was more realistic to support.  Thus, in the version I just pushed, I added that support.
I have tested this on all nightly/daily and weekly tests, and it passes for them.

Seems OK to me. I'll let @abmerop or Brad approve.

Thanks @powerjg , I just updated the commit message as I realized I did not explain/justify one case.  Done now.
I don't know if Brad has a Github ID, so @abmerop will have to stand in for him :)

Rebased to head of develop

This looks fine to me. It seems to me that the TBE SLC/GLC bit is only used for local atomic now, is that right?
I'd like to test all of the atomic changes together but right now there are 3 different patches that all conflict on the TCC, so whoever submits first doesn't have to rebase I guess..

At some level I think the TBE GLC/SLC is not used directly anymore, even for local atomics -- right?
Which other patches?  @DanKouch merged in one, which I've integrated when I rebased develop.  My guess is you mean Anoop's and #120?

@abmerop : my previous commit had some sort of merge conflict for TCP, which I attempted to resolve and push again.  However I see see this above:
This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
src/mem/ruby/protocol/GPU_VIPER-TCP.sm
Not sure what more to do ...

@abmerop : my previous commit had some sort of merge conflict for TCP, which I attempted to resolve and push again. However I see see this above:
This branch has conflicts that must be resolved Use the web editor or the to resolve conflicts. Conflicting files src/mem/ruby/protocol/GPU_VIPER-TCP.sm
Not sure what more to do ...

I tried manually editing in Github instead this time and that seems to have worked ... so I guess this is ready for review again.
Let me know what tests you want me to run, or if you want me to pull their patches locally and test things.

The 3 patches I meant haven't been committed yet.  Was referring to this one, #120 and #546 all touch TCC in similar places.  Shouldn't be a difficult merge just a heads up..

The 3 patches I meant haven't been committed yet. Was referring to this one, #120 and #546 all touch TCC in similar places. Shouldn't be a difficult merge just a heads up..

Thanks.  Once my current tests are done, I will re-run will these two (and @DanKouch's #397) all together to see what happens and report back.

The 3 patches I meant haven't been committed yet. Was referring to this one, #120 and #546 all touch TCC in similar places. Shouldn't be a difficult merge just a heads up..

Thanks. Once my current tests are done, I will re-run will these two (and @DanKouch's #397) all together to see what happens and report back.

There is one compilation issue after merging #397 into my branch, but I'll hold off on pushing until the tests complete so to avoid multiple pushes here as much as possible.

The 3 patches I meant haven't been committed yet. Was referring to this one, #120 and #546 all touch TCC in similar places. Shouldn't be a difficult merge just a heads up..

Thanks. Once my current tests are done, I will re-run will these two (and @DanKouch's #397) all together to see what happens and report back.

There is one compilation issue after merging #397 into my branch, but I'll hold off on pushing until the tests complete so to avoid multiple pushes here as much as possible.

All tests in nightly and weekly except LULESH (which already had a bug so I didn't test it) and FW (which fails to compile ... looking into) pass with this patch and #397 together.
There are also compilation issues when integrating #120 and #546 into this patch + #397.  Will just push all compilation updates as one once done testing.

What's the LULESH bug this time? ...

What's the LULESH bug this time? ...

src/gpu-compute/gpu_dyn_inst.cc:787: fatal: flat access at addr 0x2000000000000800 has a memory violation

What's the LULESH bug this time? ...

src/gpu-compute/gpu_dyn_inst.cc:787: fatal: flat access at addr 0x2000000000000800 has a memory violation

OK, we should discuss that elsewhere.  For the record, I just tested LULESH on GPUFS with develop tip (PR 563) and it completes successfully with the 0.01 2 options and ~75 minutes and matches hardware output.

What's the LULESH bug this time? ...

src/gpu-compute/gpu_dyn_inst.cc:787: fatal: flat access at addr 0x2000000000000800 has a memory violation

OK, we should discuss that elsewhere. For the record, I just tested LULESH on GPUFS with develop tip (PR 563) and it completes successfully with the 0.01 2 options and ~75 minutes and matches hardware output.

Digging deeper, it appears the problem is that with the old weekly.sh script (which I'm using to run tests locally), when one specifies the VEGA_X86 build, it does not specify --gfx-version=gfx900 or --dgpu.  So it seems like it must be running with the gfx8 ISA on the VEGA architecture and this causes problems.  When I manually add those flags and run locally, it does not have the above failure.
Since we are going to be moving to the new testing system, I'm not sure if it's worth fixing this or not.

What's the LULESH bug this time? ...

src/gpu-compute/gpu_dyn_inst.cc:787: fatal: flat access at addr 0x2000000000000800 has a memory violation

OK, we should discuss that elsewhere. For the record, I just tested LULESH on GPUFS with develop tip (PR 563) and it completes successfully with the 0.01 2 options and ~75 minutes and matches hardware output.

Digging deeper, it appears the problem is that with the old weekly.sh script (which I'm using to run tests locally), when one specifies the VEGA_X86 build, it does not specify --gfx-version=gfx900 or --dgpu. So it seems like it must be running with the gfx8 ISA on the VEGA architecture and this causes problems. When I manually add those flags and run locally, it does not have the above failure (it is still running since I didn't set to 2 iterations, but it is past the prior failure point). Probably I should update locally for tests though ...
Since we are going to be moving to the new testing system, I'm not sure if it's worth fixing this or not.

Yea, probably not worth looking into a fix.  Running gfx803 (likely) on Vega build is well into undefined behavior territory.

What's the LULESH bug this time? ...

src/gpu-compute/gpu_dyn_inst.cc:787: fatal: flat access at addr 0x2000000000000800 has a memory violation

OK, we should discuss that elsewhere. For the record, I just tested LULESH on GPUFS with develop tip (PR 563) and it completes successfully with the 0.01 2 options and ~75 minutes and matches hardware output.

Digging deeper, it appears the problem is that with the old weekly.sh script (which I'm using to run tests locally), when one specifies the VEGA_X86 build, it does not specify --gfx-version=gfx900 or --dgpu. So it seems like it must be running with the gfx8 ISA on the VEGA architecture and this causes problems. When I manually add those flags and run locally, it does not have the above failure (it is still running since I didn't set to 2 iterations, but it is past the prior failure point). Probably I should update locally for tests though ...
Since we are going to be moving to the new testing system, I'm not sure if it's worth fixing this or not.

Yea, probably not worth looking into a fix. Running gfx803 (likely) on Vega build is well into undefined behavior territory.

So far with all of #120, #397, and #540 integrated into this patch, all daily/nightly tests and all weekly tests through PageRank have passed (except LULESH, which is running separately with the above change).  I suspect the others will pass too given so many other large applications are passing.  However, this is not exhaustive for GPU atomic tests.  @abemerop any chance you can run the internal atomic test you mentioned on #540 with this?  And @DanKouch can you please run the additional atomic microbenchmarks on this to validate?

I could.. but this patch needs rebasing.  I could also share the test

@abmerop : updated to head of develop, pushed commit that integrates all of minor changes to integrate all of #120, #397, and #540.  All GPU tests pass for daily/nightly and weekly (except FW, which doesn't compile and which I'm looking into) when tested for VEGA_X86.
@DanKouch please test this on our internal GPU atomic microbenchmarks, @abmerop please test on Aaron's GPU atomic microbenchmark AMD has internally.  I think those should be the last tests needed?

The atomic benchmark is passing for both gfx900, gfx90a in GPUFS.

The atomic benchmark is passing for both gfx900, gfx90a in GPUFS.

Great, I think we are just waiting on @DanKouch's checks then.

The atomic benchmark is passing for both gfx900, gfx90a in GPUFS.

Great, I think we are just waiting on @DanKouch's checks then.

@mattsinc All of our internal atomic benchmarks run to completion, and all of my correctness-checking test programs pass as well.

Thanks @DanKouch , @abmerop , @BujSet & co. for helping with this!

What's the LULESH bug this time? ...

src/gpu-compute/gpu_dyn_inst.cc:787: fatal: flat access at addr 0x2000000000000800 has a memory violation

OK, we should discuss that elsewhere. For the record, I just tested LULESH on GPUFS with develop tip (PR 563) and it completes successfully with the 0.01 2 options and ~75 minutes and matches hardware output.

Digging deeper, it appears the problem is that with the old weekly.sh script (which I'm using to run tests locally), when one specifies the VEGA_X86 build, it does not specify --gfx-version=gfx900 or --dgpu. So it seems like it must be running with the gfx8 ISA on the VEGA architecture and this causes problems. When I manually add those flags and run locally, it does not have the above failure (it is still running since I didn't set to 2 iterations, but it is past the prior failure point). Probably I should update locally for tests though ...
Since we are going to be moving to the new testing system, I'm not sure if it's worth fixing this or not.

Yea, probably not worth looking into a fix. Running gfx803 (likely) on Vega build is well into undefined behavior territory.

@Harshil2107 the error you are seeing is what was reported here.


Add option to set output file

Description
in #156, we changed how we trigger the retry by saving the output of Rspec run to detect failed tests. The output of Rspec run was saved in a temporary file that will be deleted at the end of operation.
This PR adds a new option called BUILDKITE_SPLITTER_RESULT_PATH that can be used to specify the path to save the output of Rspec run. If the configured path has *, it will be replaced with a UNIX timestamp, this is useful when the retry function is enabled, and we want to store the output for each run separately without overriding the previous run.
Testing
I've done integration testing manually

set BUILDKITE_SPLITTER_RESULT_PATH=./tmp/my-rspec-result-*.json
run ./test-splitter
check if a json file exists in $BUILDKITE_SPLITTER_RESULT_PATH configured above

What happens if the configured path doesn't have * and retries are enabled? Does that mean it won't work properly?

It still works properly, because we interpret the report before we execute the retry runs. It just means that the result from retries will override the result from previous run(s).

What happens if the configured path doesn't have * and retries are enabled? Does that mean it won't work properly?

@niceking  It still works properly, because we interpret the report before we execute the retry runs. It just means that the result from retries will override the result from previous run(s).


LastInsertId is not supported by Postgres

Bug description
Hi, I've tried to update to the v0.23 release and I received the following error:
{"level":"fatal","error":"LastInsertId is not supported by this driver","time":1708000343,"message":"Migration failed: LastInsertId is not supported by this driver"}

Just to make sure, I've updated Postgres to the latest 16.x version, but I'm still receiving this error message.
I'm not certain what LastInsertId should do, I'm guessing it should return the latest used sequential number on INSERT? Since it doesn't seem supported by Postgres, I would maybe suggest looking into RETURNING (see PostgreSQL - INSERT) if there is no simpler workaround.

The optional RETURNING clause causes INSERT to compute and return value(s) based on each row actually inserted [...]. This is primarily useful for obtaining values that were supplied by defaults, such as a serial sequence number.

Environment

Version of headscale used: headscale/headscale:0.23.0; Postgres 15.x and 16.x
[x] Headscale is behind a (reverse) proxy (envoyproxy/envoy:v1.29.1)
[x] Headscale runs in a container

To Reproduce

Have an existing (working) installation using PostgreSQL
Update the config to accommodate for the breaking changes (db_* -> database.postgres.*) and set the database to postgres again.
Update the command in docker from ´headscale servetoserve`
Deploy the new version
See error messages in logs

Logs and attachments
(Headscale doesn't start)
Host Headscale

Linux acme 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) x86_64 GNU/Linux
Docker version 25.0.1, build 29cf629

ID    NAME    CORES   CPU TYPE    ARCHITECTURE   MEMORY     DISK     STORAGE TYPE   TRAFFIC
1     cx11    1       shared      x86            2.0 GB     20 GB    local          20 TB

Host Postgres

Linux pg01 6.1.0-17-arm64 #1 SMP Debian 6.1.69-1 (2023-12-30) aarch64 GNU/Linux
Docker version 25.0.1, build 29cf629

ID    NAME    CORES   CPU TYPE    ARCHITECTURE   MEMORY     DISK     STORAGE TYPE   TRAFFIC
93    cax21   4       shared      arm            8.0 GB     80 GB    local          20 TB

This seem to be a regression in Gorm: https://github.com/go-gorm/gorm/issues/6812
Also tracking issue in the migration framework: https://github.com/go-gormigrate/gormigrate/issues/217
I'm rolling back the versions for now in a PR.


Handle error when page loads without query

This error shows up in logs "ActionController::RoutingError (No route matches [POST] "/undefinedannif/v1/projects/tfidf-en/suggest"):"

This error is no longer occurring.
The change probably responsible for this is that the sem_recs code does not executes its main functions if the sem recs div is not present on the page (which itsef is not included unless there's a query parameter without a URI parameter).
https://github.com/LD4P/blacklight-cornell/blob/dev/app/assets/javascripts/semsearch/sem_recs.js#L87 is probably what was causing the original error (a post to the ANNIF service when the div was not there and thus the base url was not accessible).


Fix proto converter

@Framba-Luca noted that the Problem conversion method from proto to up has a Problem as a parameter, and this does not make any sense.
I created this PR to fix the problem.
@arbimo does it make sense to you?

Yes, I agree that the problem as an argument serves no use in reading back the Problem message.


feat(k3d): refactor suggestion

Disclaimer:
I was playing around with kubefirst and k3d on GitHub and was checking out the code, since I'm currently traveling and looking for some OSS projects to contribute. The PR turned out to be way bigger than anticipated... I understand in case you guys don't want to invest the time in reviewing. I could also split up and open up multiple PRs containing the changes described next, in case some changes are of interest. I am also willing to expand these changes to the other supported cloud provider.
I want to suggest some alternative approaches when building CLI apps with cobra, I choose k3d as an example, since I am able to fully e2e test it locally.
The changes can be summarized into:
prechecks
the k3d create now has a precheck suite, where certain assertions can be made (env var exists, command is available, ...) and eventually error and thus inform the user early on about missing dependencies, before creating any resources. This can be easily tested and expanded for other prechecks.
		PreRunE: func(cmd *cobra.Command, args []string) error {
			// example implementation of a pre check suite where we could validate & verify various required settings (Github access, env vars, commands, ...)
			fmt.Println("[PRECHECKS] Running prechecks")

			// invalid git provider
			if !utilities.StringInSlice(opts.gitProvider, supportedGitProviders) {
				return fmt.Errorf("\"%s\" is not a supported git provider", opts.gitProvider)
			}

			// invalid git protocol
			if !utilities.StringInSlice(opts.gitProtocol, supportedGitProtocolOverride) {
				return fmt.Errorf("%s is not a support git protocol", opts.gitProtocol)
			}

			// github specific prechecks
			if strings.ToLower(opts.gitProvider) == "github" {
				// enforce for GITHUB_TOKEN
				if !prechecks.EnvVarExists("GITHUB_TOKEN") {
					return fmt.Errorf("GITHUB_TOKEN not set, but required when using GitHub")
				}

				// github.com is available
				if err := prechecks.URLIsAvailable("https://github.com"); err != nil {
					return fmt.Errorf("github.com is not available, try again")
				}

				// check for known hosts of github.com if git provider is github
				if err := prechecks.CheckKnownHosts("github.com"); err != nil && strings.ToLower(opts.gitProvider) == "github" {
					return err
				}
			}

			// gitlab specific prechecks
			if strings.ToLower(opts.gitProvider) == "gitlab" {
				// enforce GITLAB_TOKEN if provider is gitlab
				if !prechecks.EnvVarExists("GITLAB_TOKEN") {
					return fmt.Errorf("GITLAB_TOKEN not set, but required when using GitLab")
				}

				// gitlab.com is available
				if err := prechecks.URLIsAvailable("https://gitlab.com"); err != nil {
					return fmt.Errorf("gitlab.com is not available, try again")
				}

				// when gitlab, check for a specified group
				if opts.gitlabGroup == "" {
					return fmt.Errorf("a gitlab-group is required when using Gitlab")
				}

				// check known hosts for gitlab
				if err := prechecks.CheckKnownHosts("gitlab.com"); err != nil && strings.ToLower(opts.gitProvider) == "gitlab" {
					return err
				}
			}

			// enforce NGROK_AUTH_TOKEN
			if !prechecks.EnvVarExists("NGROK_AUTHTOKEN") {
				return fmt.Errorf("NGROK_AUTHTOKEN not set, but required")
			}

			// docker is installed
			if err := prechecks.CommandExists("docker"); err != nil {
				return fmt.Errorf("docker is not installed, but is required when using k3d")
			}

			// check docker is running
			if err := prechecks.CheckDockerIsRunning(); err != nil {
				return fmt.Errorf("docker is not running, but is required when using k3d")
			}

			// portforwarding ports are available
			if err := k8s.CheckForExistingPortForwards(portForwardingPorts...); err != nil {
				return fmt.Errorf("%s - this port is required to set up your kubefirst environment - please close any existing port forwards before continuing", err.Error())
			}

			// verify user has specified valid catalog apps to be installed
			isValid, apps, err := catalog.ValidateCatalogApps(opts.installCatalogApps)
			if !isValid || err != nil {
				return err
			}

			opts.catalogApps = apps

			// check available disk space
			if err := prechecks.CheckAvailableDiskSize(); err != nil {
				return fmt.Errorf("error checking available disk size: %s", err.Error())
			}

			fmt.Println("[PRECHECKS] all prechecks passed - continuing with k3d cluster bootstrapping")

			return nil
		},

decluttered the main function
I was able to declutter the main function to more idomatic way and moved the bubbletea and logging initialiaztion in cobra.OnInitialize(), as well as the bubbletea quitting in the PostRun():
package main

import (
	"fmt"
	"os"

	"github.com/kubefirst/kubefirst/cmd"
)

func main() {
	if err := cmd.Execute(); err != nil {
		fmt.Fprintf(os.Stderr, "[ERROR] %v.\n", err)
		fmt.Fprintf(os.Stderr, "\nIf a detailed error message was available, please make the necessary corrections before retrying.\nYou can re-run the last command to try the operation again.\n\n")

		os.Exit(1)
	}
}

split up the k3d create logic in functions
The biggest change might be the split up of the runK3dCreate function into functions. Where possible, I avoided if/else for more readability. To do so, I added a struct that contains all required options and sane defaults for creating the k3d cluster. I believe this way the code is more readable and most important testable, since I haven't found any tests so far.. I think some refactors in the underlying libs (kubefirst/runtime, kubefrist/kubefirst-api etc) might be worth it
type createOptions struct {
	ci bool

	cloudRegion string
	clusterName string
	clusterType string
	clusterId   string

	containerRegistryHost string

	githubUser string
	githubOrg  string

	gitlabGroup   string
	gitlabGroupId int

	gitProvider   string
	gitProtocol   string
	gitToken      string
	gitHost       string
	gitUser       string
	gitOwner      string
	gitDescriptor string

	gitopsTemplateURL    string
	gitopsTemplateBranch string
	gitopsRepoURL        string

	useTelemetry bool
	segClient    telemetry.TelemetryEvent

	httpClient *http.Client

	installCatalogApps string
	catalogApps        []types.GitopsCatalogApp
}

func defaultCreateOpts() *createOptions {
	return &createOptions{
		clusterName: "kubefirst",
		clusterType: "mgmt",

		gitProvider: "github",
		gitProtocol: "ssh",

		gitopsTemplateURL:    "https://github.com/kubefirst/gitops-template.git",
		gitopsTemplateBranch: "main",

		useTelemetry: true,

		httpClient: http.DefaultClient,

		segClient: telemetry.TelemetryEvent{},
	}
}

This allows us to eadily call functions based on the current gitprovider:
// github specific auth settings
if strings.ToLower(o.gitProvider) == "github" {
	if err := o.githubAuth(); err != nil {
		return fmt.Errorf("error during github auth: %s", err.Error())
	}
}

func (o *createOptions) githubAuth() error {
	o.gitHost = k3d.GithubHost
	o.containerRegistryHost = "ghcr.io"

	// Attempt to retrieve session-scoped token for GitHub user
	gitHubService := services.NewGitHubService(o.httpClient)
	gitHubHandler := handlers.NewGitHubHandler(gitHubService)

	ghToken := utilities.EnvOrDefault("GITHUB_TOKEN", viper.GetString("github.session_token"))
	gitHubAccessToken, err := wrappers.AuthenticateGitHubUserWrapper(ghToken, gitHubHandler)
	if err != nil {
		log.Warn().Msgf(err.Error())
	}

	o.gitToken = gitHubAccessToken

	if err := github.VerifyTokenPermissions(o.gitToken); err != nil {
		return err
	}
        ....
}

other smaller changes

check error when saving viper config
check error of telemetry events
use cobra flags constraints for marking flags required or mutually exclusive
correctly disable telemetry when specifying via flag
added TF_CLI_ARGS="-no-color" to tf apply runs for having no color encodings in the logfile
added some simple tests showing how to test the refactored kubefirst k3d command by passing custom flags and envs

@FalcoSuessgott the PR is still in draft, but do you want us to review and validate the refactoring idea now?

HI @fharper, yes its ready for review, I added some simple tests illustrating the purpose of some changes

@FalcoSuessgott: sorry for the delay, I asked the engineering team again to give it a closer look as soon as possible.

Merged already😳 I hoped someone tested this and was able to verify the changes :D Nevertheless Im available and can fix if something shows up

@FalcoSuessgott oh weird, my mistake, not sure what happened as I just wanted to merge main into your main branch to be sure people have the latest when reviewing it. I'll remove the merge on main as nobody from the engineering team had the time to check it yet, but I pinged them again this morning! Thanks for noticing...

Oh no, it's fine, nothing was merged in kubefirst main. GitHub wasn't clear on how it wrote that, but I merged kubefirst main in your fork main... You can see there are still 51 files changed when looking at this PR, and nothing about this is in kubefirst main branch logs.
As a side note, it's part of why I always suggest that folks create a branch in their fork before doing a PR, so the verbose and actions are clearer in the GitHub interface (will add this to the contributions guidelines) :)

@FalcoSuessgott I'm so sorry about the delay, we usually respond a lot faster to PR. @jarededwards told me he will review it as soon as possible.

Hi @jarededwards,
hm bubbletea seemed to have blocked the cobra execution. I've put them both in goroutines and listened for the first error that occurred and exited the program in that case. Maybe give it a try

I was able to go further with the latest fix. I got an error executing command: the StatefulSet was not created within the timeout period. for the configuring Vault step, but I don't think it's related to this PR.
@jarededwards can you give it another try on your side please?

Hey @FalcoSuessgott
[ERROR] error executing command: NGROK_AUTHTOKEN not set, but required (see https://docs.kubefirst.io/k3d/quick-start/install#local-atlantis-executions-optional).

Just want to call out that NGROK_AUTHTOKEN is not required that is an optional env var to enable local Atlantis executions

@FalcoSuessgott I'm so sorry we let you down with this PR. Since the acquisition, we were all swamped to get back to business as usual. The good news is that our team is growing, and kubefirst will be back on track really soon, so we'll better manage external pull requests like yours. With that said, there no excuses I can give that justify waiting two months to properly start to test and review your pull request. Thanks again for the help, and so sorry for how we handled things: we are still learning as a team.


[BUG] Clear scroll request fails with sigv4 signature mismatch for java client 2.8.0

What is the bug?
clearScroll request to opensearch client fails with sigv4 signing mismatch:
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'DELETE
/_search/scroll
... remaining request info...
How can one reproduce the bug?
How can one reproduce the bug?
Create an OpenSearchClient following instructions from here (https://opensearch.org/docs/2.7/clients/java/#connecting-to-amazon-opensearch-service) and then attempt to use the client's clearScroll API.
I believe this is because it is a DELETE request with a body. The search/scroll method works as intended. Sending the request results in a sigv4 error
What is the expected behavior?
The clearScroll API call does not have a signature mismatch
What is your host/environment?
Ubuntu 22.04
implementation 'org.opensearch.client:opensearch-java:2.8.0'
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.

Is this a dup of https://github.com/opensearch-project/opensearch-java/issues/521?

Replacing ApacheHttpClient by NettyNioAsyncHttpClient solves the issue.
cf https://github.com/opensearch-project/data-prepper/pull/3338

[Catch All Triage - 1, 2, 3, 4]


Missing functions in python API?

User manual contains python API examples of functions such as wbt.extract_raster_values_at_points() but these tools do not appear to actually be in the python API. Calling the function fails, and it does not show up in wbt.list_tools().
It looks like there are other missing functions as well, filter_raster_features_by_area, find_lowest_or_highest_points, etc.

I'm not sure why you say these tools are missing because as far as I know, they are not. For example,
filter_raster_features_by_area:

Python API function
Rust tool source code called by API
Help documentation

I'm not sure why you say these tools are missing because as far as I know, they are not. For example,
filter_raster_features_by_area:

Python API function
Rust tool source code called by API
Help documentation

This is the message I get back from a help call to the function:
help(wbt.extract_raster_values_at_points)
Traceback (most recent call last):

  File "<ipython-input-80-2a9eb75925ef>", line 1, in <module>
    help(wbt.extract_raster_values_at_points)

AttributeError: 'WhiteboxTools' object has no attribute 'extract_raster_values_at_points'

A list of the raster tools also has no indication of the extract_raster_values_at_points function:
print(wbt.list_tools(['raster']))
{'absolute_value': 'Calculates the absolute value of every cell in a raster.', 'add': 'Performs an addition operation on two rasters or a raster and a constant value.', 'aggregate_raster': 'Aggregates a raster to a lower resolution.', 'and': 'Performs a logical AND operator on two Boolean raster images.', 'anova': 'Performs an analysis of variance (ANOVA) test on a raster dataset.', 'arc_cos': 'Returns the inverse cosine (arccos) of each values in a raster.', 'arc_sin': 'Returns the inverse sine (arcsin) of each values in a raster.', 'arc_tan': 'Returns the inverse tangent (arctan) of each values in a raster.', 'aspect': 'Calculates an aspect raster from an input DEM.', 'average_overlay': 'Calculates the average for each grid cell from a group of raster images.', 'block_maximum': 'Creates a block-maximum raster from an input LAS file. When the input/output parameters are not specified, the tool grids all LAS files contained within the working directory.', 'block_minimum': 'Creates a block-minimum raster from an input LAS file. When the input/output parameters are not specified, the tool grids all LAS files contained within the working directory.', 'buffer_raster': 'Maps a distance-based buffer around each non-background (non-zero/non-nodata) grid cell in an input image.', 'ceil': 'Returns the smallest (closest to negative infinity) value that is greater than or equal to the values in a raster.', 'centroid': 'Calculates the centroid, or average location, of raster polygon objects.', 'clip_raster_to_polygon': 'Clips a raster to a vector polygon.', 'convert_nodata_to_zero': 'Converts nodata values in a raster to zero.', 'convert_raster_format': 'Converts raster data from one format to another.', 'cos': 'Returns the cosine (cos) of each values in a raster.', 'cosh': 'Returns the hyperbolic cosine (cosh) of each values in a raster.', 'count_if': 'Counts the number of occurrences of a specified value in a cell-stack of rasters.', 'create_plane': 'Creates a raster image based on the equation for a simple plane.', 'cumulative_distribution': 'Converts a raster image to its cumulative distribution function.', 'd8_flow_accumulation': 'Calculates a D8 flow accumulation raster from an input DEM.', 'd8_pointer': 'Calculates a D8 flow pointer raster from an input DEM.', 'd_inf_flow_accumulation': 'Calculates a D-infinity flow accumulation raster from an input DEM.', 'd_inf_pointer': 'Calculates a D-infinity flow pointer (flow direction) raster from an input DEM.', 'decrement': 'Decreases the values of each grid cell in an input raster by 1.0 (see also InPlaceSubtract).', 'diversity_filter': 'Assigns each cell in the output grid the number of different values in a moving window centred on each grid cell in the input raster.', 'divide': 'Performs a division operation on two rasters or a raster and a constant value.', 'edge_proportion': 'Calculate the proportion of cells in a raster polygon that are edge cells.', 'elev_percentile': 'Calculates the elevation percentile raster from a DEM.', 'equal_to': 'Performs a equal-to comparison operation on two rasters or a raster and a constant value.', 'erase_polygon_from_raster': 'Erases (cuts out) a vector polygon from a raster.', 'euclidean_allocation': 'Assigns grid cells in the output raster the value of the nearest target cell in the input image, measured by the Shih and Wu (2004) Euclidean distance transform.', 'exp': 'Returns the exponential (base e) of values in a raster.', 'exp2': 'Returns the exponential (base 2) of values in a raster.', 'extract_raster_statistics': 'Extracts descriptive statistics for a group of patches in a raster.', 'extract_streams': 'Extracts stream grid cells from a flow accumulation raster.', 'fd8_flow_accumulation': 'Calculates an FD8 flow accumulation raster from an input DEM.', 'fd8_pointer': 'Calculates an FD8 flow pointer raster from an input DEM.', 'find_parallel_flow': 'Finds areas of parallel flow in D8 flow direction rasters.', 'flatten_lakes': 'Flattens lake polygons in a raster DEM.', 'flightline_overlap': 'Reads a LiDAR (LAS) point file and outputs a raster containing the number of overlapping flight lines in each grid cell.', 'floor': 'Returns the largest (closest to positive infinity) value that is less than or equal to the values in a raster.', 'flow_accumulation_full_workflow': 'Resolves all of the depressions in a DEM, outputting a breached DEM, an aspect-aligned non-divergent flow pointer, a flow accumulation raster.', 'greater_than': 'Performs a greater-than comparison operation on two rasters or a raster and a constant value.', 'highest_position': 'Identifies the stack position of the maximum value within a raster stack on a cell-by-cell basis.', 'hillshade': 'Calculates a hillshade raster from an input DEM.', 'histogram_matching': 'Alters the statistical distribution of a raster image matching it to a specified PDF.', 'histogram_matching_two_images': 'This tool alters the cumulative distribution function of a raster image to that of another image.', 'increment': 'Increases the values of each grid cell in an input raster by 1.0. (see also InPlaceAdd)', 'integer_division': 'Performs an integer division operation on two rasters or a raster and a constant value.', 'ks_test_for_normality': 'Evaluates whether the values in a raster are normally distributed.', 'kappa_index': 'Performs a kappa index of agreement (KIA) analysis on two categorical raster files.', 'less_than': 'Performs a less-than comparison operation on two rasters or a raster and a constant value.', 'lidar_point_stats': 'Creates several rasters summarizing the distribution of LAS point data. When the input/output parameters are not specified, the tool works on all LAS files contained within the working directory.', 'line_thinning': 'Performs line thinning a on Boolean raster image; intended to be used with the RemoveSpurs tool.', 'ln': 'Returns the natural logarithm of values in a raster.', 'log10': 'Returns the base-10 logarithm of values in a raster.', 'log2': 'Returns the base-2 logarithm of values in a raster.', 'lowest_position': 'Identifies the stack position of the minimum value within a raster stack on a cell-by-cell basis.', 'majority_filter': 'Assigns each cell in the output grid the most frequently occurring value (mode) in a moving window centred on each grid cell in the input raster.', 'max': 'Performs a MAX operation on two rasters or a raster and a constant value.', 'max_absolute_overlay': 'Evaluates the maximum absolute value for each grid cell from a stack of input rasters.', 'max_overlay': 'Evaluates the maximum value for each grid cell from a stack of input rasters.', 'maximum_filter': 'Assigns each cell in the output grid the maximum value in a moving window centred on each grid cell in the input raster.', 'min': 'Performs a MIN operation on two rasters or a raster and a constant value.', 'min_absolute_overlay': 'Evaluates the minimum absolute value for each grid cell from a stack of input rasters.', 'min_overlay': 'Evaluates the minimum value for each grid cell from a stack of input rasters.', 'minimum_filter': 'Assigns each cell in the output grid the minimum value in a moving window centred on each grid cell in the input raster.', 'modulo': 'Performs a modulo operation on two rasters or a raster and a constant value.', 'multiply': 'Performs a multiplication operation on two rasters or a raster and a constant value.', 'multiscale_topographic_position_image': 'Creates a multiscale topographic position image from three DEVmax rasters of differing spatial scale ranges.', 'negate': 'Changes the sign of values in a raster or the 0-1 values of a Boolean raster.', 'new_raster_from_base': 'Creates a new raster using a base image.', 'not': 'Performs a logical NOT operator on two Boolean raster images.', 'not_equal_to': 'Performs a not-equal-to comparison operation on two rasters or a raster and a constant value.', 'or': 'Performs a logical OR operator on two Boolean raster images.', 'percent_equal_to': 'Calculates the percentage of a raster stack that have cell values equal to an input on a cell-by-cell basis.', 'percent_greater_than': 'Calculates the percentage of a raster stack that have cell values greather than an input on a cell-by-cell basis.', 'percent_less_than': 'Calculates the percentage of a raster stack that have cell values less than an input on a cell-by-cell basis.', 'pick_from_list': 'Outputs the value from a raster stack specified by a position raster.', 'plan_curvature': 'Calculates a plan (contour) curvature raster from an input DEM.', 'power': 'Raises the values in grid cells of one rasters, or a constant value, by values in another raster or constant value.', 'profile_curvature': 'Calculates a profile curvature raster from an input DEM.', 'quantiles': 'Transforms raster values into quantiles.', 'range_filter': 'Assigns each cell in the output grid the range of values in a moving window centred on each grid cell in the input raster.', 'raster_cell_assignment': 'Assign row or column number to cells.', 'raster_histogram': 'Creates a histogram from raster values.', 'raster_summary_stats': 'Measures a rasters average, standard deviation, num. non-nodata cells, and total.', 'rasterize_streams': 'Rasterizes vector streams based on Lindsay (2016) method.', 'reciprocal': 'Returns the reciprocal (i.e. 1 / z) of values in a raster.', 'reclass': 'Reclassifies the values in a raster image.', 'reclass_equal_interval': 'Reclassifies the values in a raster image based on equal-ranges.', 'reclass_from_file': 'Reclassifies the values in a raster image using reclass ranges in a text file.', 'remove_off_terrain_objects': 'Removes off-terrain objects from a raster digital elevation model (DEM).', 'rho8_pointer': 'Calculates a stochastic Rho8 flow pointer raster from an input DEM.', 'round': 'Rounds the values in an input raster to the nearest integer value.', 'sin': 'Returns the sine (sin) of each values in a raster.', 'sinh': 'Returns the hyperbolic sine (sinh) of each values in a raster.', 'slope': 'Calculates a slope raster from an input DEM.', 'square': 'Squares the values in a raster.', 'square_root': 'Returns the square root of the values in a raster.', 'standard_deviation_filter': 'Assigns each cell in the output grid the standard deviation of values in a moving window centred on each grid cell in the input raster.', 'subtract': 'Performs a differencing operation on two rasters or a raster and a constant value.', 'tan': 'Returns the tangent (tan) of each values in a raster.', 'tangential_curvature': 'Calculates a tangential curvature raster from an input DEM.', 'tanh': 'Returns the hyperbolic tangent (tanh) of each values in a raster.', 'thicken_raster_line': 'Thickens single-cell wide lines within a raster image.', 'to_degrees': 'Converts a raster from radians to degrees.', 'to_radians': 'Converts a raster from degrees to radians.', 'total_curvature': 'Calculates a total curvature raster from an input DEM.', 'trend_surface': 'Estimates the trend surface of an input raster file.', 'truncate': 'Truncates the values in a raster to the desired number of decimal places.', 'vector_lines_to_raster': 'Converts a vector containing polylines into a raster.', 'vector_points_to_raster': 'Converts a vector containing points into a raster.', 'vector_polygons_to_raster': 'Converts a vector containing polygons into a raster.', 'weighted_overlay': 'Performs a weighted sum on multiple input rasters after converting each image to a common scale. The tool performs a multi-criteria evaluation (MCE).', 'weighted_sum': 'Performs a weighted-sum overlay on multiple input raster images.', 'xor': 'Performs a logical XOR operator on two Boolean raster images.', 'z_scores': 'Standardizes the values in an input raster by converting to z-scores.'}

I'm not sure what I'm missing. Below is the conda version readout if that helps.
whitebox_tools            0.8.0dev                 py36_0    conda-forge

This is the message I get back from a help call to the function:
help(wbt.extract_raster_values_at_points)
Traceback (most recent call last):

  File "<ipython-input-80-2a9eb75925ef>", line 1, in <module>
    help(wbt.extract_raster_values_at_points)

AttributeError: 'WhiteboxTools' object has no attribute 'extract_raster_values_at_points'

A list of the raster tools also has no indication of the extract_raster_values_at_points function:
print(wbt.list_tools(['raster']))
{'absolute_value': 'Calculates the absolute value of every cell in a raster.', 'add': 'Performs an addition operation on two rasters or a raster and a constant value.', 'aggregate_raster': 'Aggregates a raster to a lower resolution.', 'and': 'Performs a logical AND operator on two Boolean raster images.', 'anova': 'Performs an analysis of variance (ANOVA) test on a raster dataset.', 'arc_cos': 'Returns the inverse cosine (arccos) of each values in a raster.', 'arc_sin': 'Returns the inverse sine (arcsin) of each values in a raster.', 'arc_tan': 'Returns the inverse tangent (arctan) of each values in a raster.', 'aspect': 'Calculates an aspect raster from an input DEM.', 'average_overlay': 'Calculates the average for each grid cell from a group of raster images.', 'block_maximum': 'Creates a block-maximum raster from an input LAS file. When the input/output parameters are not specified, the tool grids all LAS files contained within the working directory.', 'block_minimum': 'Creates a block-minimum raster from an input LAS file. When the input/output parameters are not specified, the tool grids all LAS files contained within the working directory.', 'buffer_raster': 'Maps a distance-based buffer around each non-background (non-zero/non-nodata) grid cell in an input image.', 'ceil': 'Returns the smallest (closest to negative infinity) value that is greater than or equal to the values in a raster.', 'centroid': 'Calculates the centroid, or average location, of raster polygon objects.', 'clip_raster_to_polygon': 'Clips a raster to a vector polygon.', 'convert_nodata_to_zero': 'Converts nodata values in a raster to zero.', 'convert_raster_format': 'Converts raster data from one format to another.', 'cos': 'Returns the cosine (cos) of each values in a raster.', 'cosh': 'Returns the hyperbolic cosine (cosh) of each values in a raster.', 'count_if': 'Counts the number of occurrences of a specified value in a cell-stack of rasters.', 'create_plane': 'Creates a raster image based on the equation for a simple plane.', 'cumulative_distribution': 'Converts a raster image to its cumulative distribution function.', 'd8_flow_accumulation': 'Calculates a D8 flow accumulation raster from an input DEM.', 'd8_pointer': 'Calculates a D8 flow pointer raster from an input DEM.', 'd_inf_flow_accumulation': 'Calculates a D-infinity flow accumulation raster from an input DEM.', 'd_inf_pointer': 'Calculates a D-infinity flow pointer (flow direction) raster from an input DEM.', 'decrement': 'Decreases the values of each grid cell in an input raster by 1.0 (see also InPlaceSubtract).', 'diversity_filter': 'Assigns each cell in the output grid the number of different values in a moving window centred on each grid cell in the input raster.', 'divide': 'Performs a division operation on two rasters or a raster and a constant value.', 'edge_proportion': 'Calculate the proportion of cells in a raster polygon that are edge cells.', 'elev_percentile': 'Calculates the elevation percentile raster from a DEM.', 'equal_to': 'Performs a equal-to comparison operation on two rasters or a raster and a constant value.', 'erase_polygon_from_raster': 'Erases (cuts out) a vector polygon from a raster.', 'euclidean_allocation': 'Assigns grid cells in the output raster the value of the nearest target cell in the input image, measured by the Shih and Wu (2004) Euclidean distance transform.', 'exp': 'Returns the exponential (base e) of values in a raster.', 'exp2': 'Returns the exponential (base 2) of values in a raster.', 'extract_raster_statistics': 'Extracts descriptive statistics for a group of patches in a raster.', 'extract_streams': 'Extracts stream grid cells from a flow accumulation raster.', 'fd8_flow_accumulation': 'Calculates an FD8 flow accumulation raster from an input DEM.', 'fd8_pointer': 'Calculates an FD8 flow pointer raster from an input DEM.', 'find_parallel_flow': 'Finds areas of parallel flow in D8 flow direction rasters.', 'flatten_lakes': 'Flattens lake polygons in a raster DEM.', 'flightline_overlap': 'Reads a LiDAR (LAS) point file and outputs a raster containing the number of overlapping flight lines in each grid cell.', 'floor': 'Returns the largest (closest to positive infinity) value that is less than or equal to the values in a raster.', 'flow_accumulation_full_workflow': 'Resolves all of the depressions in a DEM, outputting a breached DEM, an aspect-aligned non-divergent flow pointer, a flow accumulation raster.', 'greater_than': 'Performs a greater-than comparison operation on two rasters or a raster and a constant value.', 'highest_position': 'Identifies the stack position of the maximum value within a raster stack on a cell-by-cell basis.', 'hillshade': 'Calculates a hillshade raster from an input DEM.', 'histogram_matching': 'Alters the statistical distribution of a raster image matching it to a specified PDF.', 'histogram_matching_two_images': 'This tool alters the cumulative distribution function of a raster image to that of another image.', 'increment': 'Increases the values of each grid cell in an input raster by 1.0. (see also InPlaceAdd)', 'integer_division': 'Performs an integer division operation on two rasters or a raster and a constant value.', 'ks_test_for_normality': 'Evaluates whether the values in a raster are normally distributed.', 'kappa_index': 'Performs a kappa index of agreement (KIA) analysis on two categorical raster files.', 'less_than': 'Performs a less-than comparison operation on two rasters or a raster and a constant value.', 'lidar_point_stats': 'Creates several rasters summarizing the distribution of LAS point data. When the input/output parameters are not specified, the tool works on all LAS files contained within the working directory.', 'line_thinning': 'Performs line thinning a on Boolean raster image; intended to be used with the RemoveSpurs tool.', 'ln': 'Returns the natural logarithm of values in a raster.', 'log10': 'Returns the base-10 logarithm of values in a raster.', 'log2': 'Returns the base-2 logarithm of values in a raster.', 'lowest_position': 'Identifies the stack position of the minimum value within a raster stack on a cell-by-cell basis.', 'majority_filter': 'Assigns each cell in the output grid the most frequently occurring value (mode) in a moving window centred on each grid cell in the input raster.', 'max': 'Performs a MAX operation on two rasters or a raster and a constant value.', 'max_absolute_overlay': 'Evaluates the maximum absolute value for each grid cell from a stack of input rasters.', 'max_overlay': 'Evaluates the maximum value for each grid cell from a stack of input rasters.', 'maximum_filter': 'Assigns each cell in the output grid the maximum value in a moving window centred on each grid cell in the input raster.', 'min': 'Performs a MIN operation on two rasters or a raster and a constant value.', 'min_absolute_overlay': 'Evaluates the minimum absolute value for each grid cell from a stack of input rasters.', 'min_overlay': 'Evaluates the minimum value for each grid cell from a stack of input rasters.', 'minimum_filter': 'Assigns each cell in the output grid the minimum value in a moving window centred on each grid cell in the input raster.', 'modulo': 'Performs a modulo operation on two rasters or a raster and a constant value.', 'multiply': 'Performs a multiplication operation on two rasters or a raster and a constant value.', 'multiscale_topographic_position_image': 'Creates a multiscale topographic position image from three DEVmax rasters of differing spatial scale ranges.', 'negate': 'Changes the sign of values in a raster or the 0-1 values of a Boolean raster.', 'new_raster_from_base': 'Creates a new raster using a base image.', 'not': 'Performs a logical NOT operator on two Boolean raster images.', 'not_equal_to': 'Performs a not-equal-to comparison operation on two rasters or a raster and a constant value.', 'or': 'Performs a logical OR operator on two Boolean raster images.', 'percent_equal_to': 'Calculates the percentage of a raster stack that have cell values equal to an input on a cell-by-cell basis.', 'percent_greater_than': 'Calculates the percentage of a raster stack that have cell values greather than an input on a cell-by-cell basis.', 'percent_less_than': 'Calculates the percentage of a raster stack that have cell values less than an input on a cell-by-cell basis.', 'pick_from_list': 'Outputs the value from a raster stack specified by a position raster.', 'plan_curvature': 'Calculates a plan (contour) curvature raster from an input DEM.', 'power': 'Raises the values in grid cells of one rasters, or a constant value, by values in another raster or constant value.', 'profile_curvature': 'Calculates a profile curvature raster from an input DEM.', 'quantiles': 'Transforms raster values into quantiles.', 'range_filter': 'Assigns each cell in the output grid the range of values in a moving window centred on each grid cell in the input raster.', 'raster_cell_assignment': 'Assign row or column number to cells.', 'raster_histogram': 'Creates a histogram from raster values.', 'raster_summary_stats': 'Measures a rasters average, standard deviation, num. non-nodata cells, and total.', 'rasterize_streams': 'Rasterizes vector streams based on Lindsay (2016) method.', 'reciprocal': 'Returns the reciprocal (i.e. 1 / z) of values in a raster.', 'reclass': 'Reclassifies the values in a raster image.', 'reclass_equal_interval': 'Reclassifies the values in a raster image based on equal-ranges.', 'reclass_from_file': 'Reclassifies the values in a raster image using reclass ranges in a text file.', 'remove_off_terrain_objects': 'Removes off-terrain objects from a raster digital elevation model (DEM).', 'rho8_pointer': 'Calculates a stochastic Rho8 flow pointer raster from an input DEM.', 'round': 'Rounds the values in an input raster to the nearest integer value.', 'sin': 'Returns the sine (sin) of each values in a raster.', 'sinh': 'Returns the hyperbolic sine (sinh) of each values in a raster.', 'slope': 'Calculates a slope raster from an input DEM.', 'square': 'Squares the values in a raster.', 'square_root': 'Returns the square root of the values in a raster.', 'standard_deviation_filter': 'Assigns each cell in the output grid the standard deviation of values in a moving window centred on each grid cell in the input raster.', 'subtract': 'Performs a differencing operation on two rasters or a raster and a constant value.', 'tan': 'Returns the tangent (tan) of each values in a raster.', 'tangential_curvature': 'Calculates a tangential curvature raster from an input DEM.', 'tanh': 'Returns the hyperbolic tangent (tanh) of each values in a raster.', 'thicken_raster_line': 'Thickens single-cell wide lines within a raster image.', 'to_degrees': 'Converts a raster from radians to degrees.', 'to_radians': 'Converts a raster from degrees to radians.', 'total_curvature': 'Calculates a total curvature raster from an input DEM.', 'trend_surface': 'Estimates the trend surface of an input raster file.', 'truncate': 'Truncates the values in a raster to the desired number of decimal places.', 'vector_lines_to_raster': 'Converts a vector containing polylines into a raster.', 'vector_points_to_raster': 'Converts a vector containing points into a raster.', 'vector_polygons_to_raster': 'Converts a vector containing polygons into a raster.', 'weighted_overlay': 'Performs a weighted sum on multiple input rasters after converting each image to a common scale. The tool performs a multi-criteria evaluation (MCE).', 'weighted_sum': 'Performs a weighted-sum overlay on multiple input raster images.', 'xor': 'Performs a logical XOR operator on two Boolean raster images.', 'z_scores': 'Standardizes the values in an input raster by converting to z-scores.'}

I'm not sure what I'm missing. Below is the conda version readout if that helps.
whitebox_tools            0.8.0dev                 py36_0    conda-forge

I believe the format of the function you are trying to call is:
wbt.tool_help("extract_raster_values_at_points")

I believe the format of the function you are trying to call is:
wbt.tool_help("extract_raster_values_at_points")

This is the result of that input:
wbt.tool_help("extract_raster_values_at_points")
Out[82]: "thread 'main' panicked at 'Unrecognized tool name ExtractRasterValuesAtPoints.', src\\main.rs:49:21\nnote: Run with `RUST_BACKTRACE=1` for a backtrace.\n"

The typical help() call works great on other wbt functions so I expected it to work universally:
help(wbt.aggregate_raster)
Help on method aggregate_raster in module whitebox_tools:

aggregate_raster(i, output, agg_factor=2, type='mean', callback=<function default_callback at 0x000001E687ACAF28>) method of whitebox_tools.WhiteboxTools instance
    Aggregates a raster to a lower resolution.

    Keyword arguments:

    i -- Input raster file.
    output -- Output raster file.
    agg_factor -- Aggregation factor, in pixels.
    type -- Statistic used to fill output pixels.
    callback -- Custom functon for handling tool text outputs.

Help calls notwithstanding this is the output of an actual function call with data:

  File "<ipython-input-61-b9bccf65a85a>", line 2, in ExtractFromRaster
    results = wbt.extract_raster_values_at_points(raster, points, out_text=True)

AttributeError: 'WhiteboxTools' object has no attribute 'extract_raster_values_at_points'

This is the result of that input:
wbt.tool_help("extract_raster_values_at_points")
Out[82]: "thread 'main' panicked at 'Unrecognized tool name ExtractRasterValuesAtPoints.', src\\main.rs:49:21\nnote: Run with `RUST_BACKTRACE=1` for a backtrace.\n"

The typical help() call works great on other wbt functions so I expected it to work universally:
help(wbt.aggregate_raster)
Help on method aggregate_raster in module whitebox_tools:

aggregate_raster(i, output, agg_factor=2, type='mean', callback=<function default_callback at 0x000001E687ACAF28>) method of whitebox_tools.WhiteboxTools instance
    Aggregates a raster to a lower resolution.

    Keyword arguments:

    i -- Input raster file.
    output -- Output raster file.
    agg_factor -- Aggregation factor, in pixels.
    type -- Statistic used to fill output pixels.
    callback -- Custom functon for handling tool text outputs.

Help calls notwithstanding this is the output of an actual function call with data:

  File "<ipython-input-61-b9bccf65a85a>", line 2, in ExtractFromRaster
    results = wbt.extract_raster_values_at_points(raster, points, out_text=True)

AttributeError: 'WhiteboxTools' object has no attribute 'extract_raster_values_at_points'

The "Home" entry in conda-forge seems to suggest it's coming from this repository.
https://anaconda.org/conda-forge/whitebox_tools

The "Home" entry in conda-forge seems to suggest it's coming from this repository.
https://anaconda.org/conda-forge/whitebox_tools

It looks to me that the conda repo, which again is not something that I created or maintain, has not been updated for over 2 years. This repository is for the back-end only. The back-end does come with a simply Python API, in the whitebox_tools.py script, and I do support that. However, I cannot provide support for the various front-ends to WhiteboxTools that various users in the community have created.

It looks to me that the conda repo, which again is not something that I created or maintain, has not been updated for over 2 years. This repository is for the back-end only. The back-end does come with a simply Python API, in the whitebox_tools.py script, and I do support that. However, I cannot provide support for the various front-ends to WhiteboxTools that various users in the community have created.

@NO645 You can try the Python frontend https://github.com/giswqs/whitebox-python.
I just tested it and it worked fine.

@NO645 You can try the Python frontend https://github.com/giswqs/whitebox-python.
I just tested it and it worked fine.

Python frontend from giswqs recognizes the functions in question. Thank you!

Python frontend from giswqs recognizes the functions in question. Thank you!


Add support for distil-large-v3

Tested using the following example:
cd candle/candle-examples/examples/whisper
cargo run --example whisper --release --features symphonia -- --model distil-large-v3

Output:
No audio file submitted: Downloading https://huggingface.co/datasets/Narsil/candle_demo/blob/main/samples_jfk.wav
pcm data loaded 176000
loaded mel: [1, 128, 3000]
english: 0.9849657
portuguese: 0.0024733297
spanish: 0.0019878238
russian: 0.001226182
french: 0.0011861552
0.0s -- 30.0s:  And so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.

Nice, thanks, will merge as soon as the CI comes back.


mdsvex does not recognize code blocks marked with 'svelte'

code blocks marked with ```svelte are not processed correctly in mdsvex, instead they are just outputted as a single string, this will prevent us from getting fancy highlighting in the same way we could get here in github, this is a limitation of the processor mdsvex is using (prismjs)
a possible solution to this is to mark them as html instead
a more forward looking solution would be to keep them marked as html but somehow preprocess that to html instead ? (maybe this should be done by mdsvex itself)

a third solution would be to write our own highlighter

I think for now, just using html is probably the easiest. In the long run I can see value in doing that, maybe that's something that should actually be included in MDSveX, or as a plugin of some sort. Does @pngwgn (can I tag people like this? Hmm) maybe know what would be a good way forward here?

@pngwn

I don’t understand the problem. Can you please provide examples? I have svelte syntax highlighting on the mdsvex docs for example.

The problem is that prismjs does not support svelte, but it seems you made a package for exactly this: prism-svelte which you probably use in the mdsvex docs
Will create a PR adding this here as well
Apologies for the disturbance

That's great!

I half remember including this in mdsvex by default but I may have imagined it. I’ll see if there is a simple way to do that.

I imagined it. I’ll take a look at this.

Managed to fix it by adding:
import Prism from 'prismjs';
import 'prism-svelte';

to the rollup.config.js

Neat!