[SPARK-17689][SQL][STREAMING] added excludeFiles option for file source.

What changes were proposed in this pull request?
added excludeFiles specially( but not limited to) for excluding temporary files created during file copy(.*_COPYING) or creation(_temporary) in HDFS.
How was this patch tested?
Contains a unit test.
Tested it while copying file and creating files, by deploying against real hdfs cluster.

Test build #65961 has started for PR 15258 at commit c9d4bfe.

Test build #65965 has started for PR 15258 at commit 8e15103.

Test build #65966 has started for PR 15258 at commit 2cc3eac.

Test build #65969 has started for PR 15258 at commit 6671c79.

Test build #65961 has finished for PR 15258 at commit c9d4bfe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65961/
Test PASSed.

Merged build finished. Test PASSed.

Test build #65965 has finished for PR 15258 at commit 8e15103.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65965/
Test PASSed.

Merged build finished. Test PASSed.

Test build #65966 has finished for PR 15258 at commit 2cc3eac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65966/
Test PASSed.

Test build #65969 has finished for PR 15258 at commit 6671c79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65969/
Test PASSed.

This change allows FileInputStream to consume partial outputs of a system such as Hadoop or another copy of Spark, provided that the system adheres rigidly to the write policy of recent versions of Hadoop. That is: First, write to a temporary file. Then close and flush the temporary file. Then rename the temporary file, using one of the newer, atomic HDFS APIs for renaming files. I worry that users might write data in a subtly different way that does not follow this procedure 100%, which could result in Spark reading incorrect data every once in a while. I recommend documenting under exactly what conditions the "ignore temporary files" option guarantees correct behavior.
Also, it would be a good idea to include a mode in which FileInputStream will ignore a directory of files until the special file _SUCCESS appears, indicating that the directory is complete. Otherwise, Spark could end up consuming partial results from failed jobs.

@frreiss That is correct, I will look into it. Moving the PR to WIP, as at least I would like to document these situations.

As we suspected, _SUCCESS file does not appear on copying file in HDFS. So it can not be a trusted way to know, that input directory is not partial output of a failed job. Problem, that spark can process a truncated output of a failed job seems like a broader problem and might be okay to not address as part of this JIRA ?

Test build #66096 has started for PR 15258 at commit 9469bef.

Test build #66096 has finished for PR 15258 at commit 9469bef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66096/
Test PASSed.

Merged build finished. Test PASSed.

Test build #66374 has started for PR 15258 at commit 01cb666.

Test build #66374 has finished for PR 15258 at commit 01cb666.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

retest, this please.

Jenkins, retest this please

Test build #66384 has started for PR 15258 at commit 01cb666.

Test build #66384 has finished for PR 15258 at commit 01cb666.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66384/
Test PASSed.

@ScrapCodes Could you resolve the conflict? Thanks!

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

@gatorsmile Thanks for pinging this, I am closing it myself for now. And once it is ready, I will open it again. I am planning to update it soon.


Get fontawesome.css into pattern library scss

classes like <i class="fa fa-checkbox"></i> aren't working in Drupal cuz we're not including it.  I think we should add it to the bower component.

also remove glyphicon css from pattern lib


eventToHotkeyString outputs ? event as Shift+?

Since the updates in the latest release which include a logic change to address a different bug, eventToHotkeyString is includingShift unexpectedly.
? ==> Shift+?
! ==> Shift+!
Can we revert the logic to what it was before?
Related PRs:

https://github.com/github/hotkey/pull/40
https://github.com/github/hotkey/pull/41
https://github.com/github/hotkey/pull/43

Moving form  a convo from another repo:

Based off this eventToHotkeyString test, it doesn't look like an uppercase alpha character is necessarily expected

Hrm that test is simulating the events though. Maybe we need to add playwright or something to that project and do some real browser tests? I will definitely test this carefully when I work on it to figure out exactly what's actually happening in the browser.

For helpful background; in https://github.com/github/hotkey/pull/43 @keithamus states that "on macOS the "Meta+Shift" plane is always lowercase". I just checked and this does appear to be the case

This is also true of special characters

I think what this means is that:

? is a valid shortcut if we want to leave the Shift+ out of the string and is equivalent to Shift+?
But combined with the metaKey, that ? key could be Meta+/ on Mac. So, in order to specify that shiftKey is pressed,  Meta+Shift+/ would be the required shortcut.

This leads me to think, that we should continue to require the Shift key in the keyboard string. What do you think @khiga8

For the time being, I have taken this odd behaviour into account in our shortcuts https://github.com/github/github/commit/2c3902e14c647b14a664c832f2be5d2cb5691eee but this is pretty ugl

y

Notice above that when the Control+Meta+Shift are pressed together the result is an upper case event.key! 🤔

Use https://wicg.github.io/keyboard-map/#h-keyboard-getlayoutmap

Maybe polypill it?

we can ask the user for their layout if the API is not available and store it in local storage .

store key as event.code

display key localized for user's keyboard layout using layout map

Maybe have UI to capture key codes

Store as QWERTY keyCode

show QWERTY locale by default

provide UI for user to select their keyboard layout (ASERTY, DVORAK etc.)

Here is my proposed "logical" hotkey format:

ctrlKey
altKey
metaKey
shiftKey
event.code

Examples

To represent the key combination ctrl+alt+cmd+shift+/ we would write ctrlKey+altKey+metaKey+shiftKey+Slash.
To represent shift+cmd+k we would write shiftKey+metaKey+KeyK
To represent cmd+space we would write metaKey+Space
To represent the key sequence g i, we would write KeyG KeyI

Humanizing
We then would want to provide a formatting function that could show the above logical representations in a human readable form.
humanizeHotkey("shiftKey+metaKey+Slash") //=> "cmd+?"

This helper could take a keyboard layout (eg. DVORAK, AZERTY...) with QWERTY as default.
humanizeHotkey("shiftKey+metaKey+Slash", { layout: "DVORAK"}) //=> "cmd+Z"

event.code "Slash" maps to the "z" key on a DVORAK keyboard

Based on what you know, can we rely on keyboard map API at this point? Are there polyfills we could use or would we need to write our own? If we are unable to rely on the keyboard map API, have you found there other tools out there to help retrieve the humanized output given a keyboard layout?

What does support (if any) for non-latin keyboard layouts look like? Are we supporting only: qwerty, dvorak, and colemak?

Another thought: what if we support both event.key and event.code and let developers decide for themselves. We could use a prefix key:?, key:k, key:K etc. or code:Slash, code:KeyK etc. to let the developer decide between. semantic meaning and keyboard location.
This would put us back in the same place when key is combined with Shift/Meta such as Meta+Shift+key:k.

Note: I added a test and note in the README about the inconsistent treatment of shift.
When working on this issue, that test and README note should be updated to account for however we resolve this!

A soutien I am leaning towards is: when we encounter Shift, we should split the hotkey into multiple trie nodes:
For example, for the hotkey "Shift+a" (or "A"), we could produce the following trie nodes:

Shift+a
Shift+A
A

For the hotkey "Mod+Shift+a" (or "Mod+A") ...
... on Windows, we could produce the following trie nodes:

Control+Shift+a
Control+Shift+A
Control+A

... and on Mac we could produce the following nodes:

Meta+Shift+a
Meta+Shift+A
Meta+A

A separate thing I am considering is that we should support the square-bracket syntax from VSCode for event.code.


[ASCII-1434] Migrate config routes to providers

What does this PR do?
Previous work:

https://github.com/DataDog/datadog-agent/pull/24045
https://github.com/DataDog/datadog-agent/pull/24872
https://github.com/DataDog/datadog-agent/pull/25234
https://github.com/DataDog/datadog-agent/pull/25336
https://github.com/DataDog/datadog-agent/pull/25367

Continuation of the componentization work, this PR registers the /config setting routes with EndpointProviders on the setting component using fx ValueGroups

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

there are e2e tests for most of these routes

/merge


What about migrating to 3.x?

And another question - why did you use App.Metrics package but not the prometheus-net one?
Thank you.

Hi @ASalihov.
Currently this project use .NET Standard 2.0 so can be referenced from various platforms including .NET Core 3.x (I guess you referring to it). What would be the benefit from migration for this project ?
Regarding second question: feel free to implement prometheus-net package support or use Prometheus reporter for App.Metrics https://www.app-metrics.io/reporting/reporters/prometheus


Added greeting section to showcase view

Added greeting section to applicaton showcase view with short
information section regarding applications and how to add them to the
system.

Looks good. Is there a rationale for the greeting to be in page_primary_action block instead of pre_primary? It may be just a matter of taste, but since the greeting is not technically part of the "action" part of the page, it could semantically be better situated in pre_primary.

The pre_primary section uses the whole page width for the section. This causes category and platform lists on the left to shift quite low when the greeting message is long, so I opted to change where search and the greeting messages are located rather than changing pre_preprimary styling (as it will change other pages also).


upgrade Go to 1.23.1

coverage: 4.37%. remained the same
when pulling cc111a9912aaadb949e701b0d88a844a49112634 on upgrade-go
into 01ad0e9bf1e7e2977463af0897749fc9af0e2374 on main.


Add MTD for AT25 eeprom

Summary
Adds MTD driver for AT25 SPI EEPROM devices, based on character driver code by @slorquet and the existing at24xx.c block driver code
Impact
None.
Testing
Custom board, SAMA5D27C-D1G with 32kbyte SPI EEPROM on board.

As per usual ABSOLUTELY NO IDEA why this is failing CI. It passes checkpatch. No clues in CI output is NOT helpful.

I very definitely do run checkpatch.sh before subbitting a PR.
If I run:
./tools/checkpatch.sh -f drivers/mtd/at25ee.c
I get NO ERRORS!!!
It had found loads, which is why I run it of course....get a clean run (on all files) and I submit the PR.
If I look at line 235, I see nothing wrong either.
How are you running it?

Thanks for taking time to reply. I am finding some logical/programmatic errors at the moment so will leave it as a draft and once I am sure it is 100% working on my test bed I will go through the check/squash/push marlarkee again....and see what happens!!

according to the first failure reported by CI https://github.com/apache/nuttx/actions/runs/7265338667/job/19794901605?pr=11423#step:7:75, you can try compile this locally:
./tools/configure.sh sama5d4-ek/nxwm

this reports error and that's why CI fails:
warning: the default selection USE_AT25EE_EEPROM_BLOCK_SIZE (undefined) of <choice> (defined at drivers/mtd/Kconfig:446) is not contained in the choice
make: *** [tools/Unix.mk:688: olddefconfig] Error 1

CI doesn't report errors and warnings related to Kconfig at all, so usually if for some reason CI fails - the problem is in Kconfig. It sucks, but no one has been willing to fix it yet :)

Thank you! I was 99% sure it was Kconfig related but couldn't spot the error. This has bitten me before :(

I still can't see the error in the CI logs thought? May be I "can't see for looking"?
SOmeone else suggested I used python kkonfig-frontends to find these things but the method suggetsed to install that (Ubuntu) isn't behaving like it dd and and I can't reinstall (I removed it as I didn't really like the way it behaved, but will have to get used to it if if finds errors!!)

I still can't see the error in the CI logs thought? May be I "can't see for looking"?

Errors and warning reported by Kconfig are not printed in CI and this is the root of the problem

I still can't see the error in the CI logs thought? May be I "can't see for looking"?

Errors and warnings reported by Kconfig are not printed in CI and this is the root of the problem

Ah ok - see what you mean now. I've been trying to work out the correct way to manually call codechecker but it's fighting me...like everything NuttX related LOL

Pushed minor update with SPI_SETDELAY functionality, to speed up accesses


Fix #1760, check resource ID idx is less than max

Describe the contribution
Fixes #1760
Testing performed
Steps taken to test the contribution:

Build and run cfe functional tests

Expected behavior changes
No impact to behavior
System(s) tested on

Hardware: PC
OS: Ubuntu 20.04

Contributor Info - All information REQUIRED for consideration of pull request
Niall Mullane - GSFC 582 Intern

CCB:2021-08-11 APPROVED with CHANGES

Add call to "unregister"

CCB:2021-08-11 APPROVED


INVESTIGATE: Create a mechanism to check if a use case would result in a success without invoking the interactor

Solved in #71
Just create a second pipeline that doesn't invoke the interactor and instead invokes a success output port (or do nothing, whatever is preferable)


Add DataStax.com logo

Motivation
DataStax recently acquired Kesque.com, this patch replaces the references to Kesque.com with DataStax.
Modifications
Update the website

@Jennifer88huang please take a look

@Jennifer88huang I have set logo_write to true
please take a look

@eolivelli Good job~


Support Async Enumerables

Closes #56
Closes #58
Could be refactored so that
ValueTask<ulong> SaveAllAsync(NpgsqlConnection connection, IEnumerable<TEntity> entities, CancellationToken cancellationToken = default);
wraps the new method by calling entities.ToAsyncEnumerable() . However, more package references would be required.
TODO

[ ] Add Release Notes
[ ] Add Documentation for new public method

@Cooksauce I think for the most part this was on track. I took the liberty to plumb cancellation tokens and add tests. How does this look to you?

@say25 I like the refactor. Just added additional cancellation tests. Thoughts on refactoring ColumnDefinition.Write to:
public Func<NpgsqlBinaryImporter, TEntity, CancellationToken, Task> WriteAsync { get; set; }?

@say25 Went ahead and changed column definition - open to reverting if you guys are against it.

@Cooksauce Im fine with the change sets, I'm going to go ahead and rebase the commits to attribute them to you.
It would appear your git config it likely set for your global machine as a work account?
https://help.github.com/en/github/using-git/setting-your-username-in-git#setting-your-git-username-for-a-single-repository

@say25 Yup good catch there. Thanks :)

@Cooksauce made a couple small tweaks. You thinks its all good to go?

@say25 Were tests failing without the async/await on the Map method? Like this:
public PostgreSQLCopyHelper<TEntity> Map<TProperty>(string columnName, Func<TEntity, TProperty> propertyGetter, NpgsqlDbType type)
{
     return AddColumn(columnName, (writer, entity, cancellationToken) => writer.WriteAsync(propertyGetter(entity), type, cancellationToken));
}

My understanding is this async/await would cause the compiler to create an extra, unnecessary async state machine and task object. Instead of passing the task returned by writer.WriteAsync(...) back out to the caller.

@Cooksauce you are correct

@say25 All looks well to me


Disable proxying to api servers in Ensembl deployments

Description
Kamal has been asking us not to enable in kubernetes deployments request handlers that we would normally use when developing locally, to proxy requests to appropriate backends. Although ingress rules set in kubernetes mean that requests intended for other services will never reach our rendering server, it is safer to know that if something goes wrong and such a request does get to the rendering server, the response will be a 404 error rather than something that our staging server would respond with. Failing early helps with debugging.
This PR makes use of the fact that all our kubernetes deployments will have an environment variable called "ENVIRONMENT" set (we use it to distinguish between production, staging, internal, and review deployments).
Knowing this, we can assume that if this this environment variable is not set, then the code is running in locally, and will need to proxy api requests and requests for static assets to relevant backends.
Result
Before, the log from a kubernetes pod with the rendering service showed that we instantiated proxy handlers for /api and /static routes:

Now, the log from the pod shows that the rendering service does not initialize any proxies:

Related JIRA Issue(s)
https://www.ebi.ac.uk/panda/jira/browse/ENSWBSITES-1517
Deployment URL(s)
http://no-proxy-in-deployment.review.ensembl.org
Tested

local development mode (npm start) — ✅
local production build (npm run build, then node dist/server/server.js) — ✅

Looks good. Other way to get away with proxy middleware, setting up environment variable and getting as close to production as it can be is to use telepresence :).
With telepresence

Create branch review app
Intercept the FE service to route traffic
Do development as you would

The only thing this would need is telepresence installed on laptop, knowing few commands. Would be easy to integrate complex workflows as well. Most importantly, everyone needs to adopt it and involves bit learning curve.
Here is the link to my presentation Improving-Dev-Workflow.pptx
As it stands looks good to me


Use extensions for the sensing category

I skipped this file on a previous pass because I think some of the output types are wrong.  On this change I've preserved the existing output types, even if they are wrong, and switched to using blockly extensions for shape and colour.

So I'm not so familiar with the sensing category blocks and outputs. What blocks are you suspicious about?

"distance to" is returning a string.
"answer" is returning a number.
"video on menu one" and video on menu two" are returning strings.
"video toggle menu" is returning a string.
"username" is returning a number.


Changed compared variables of range id intervall in ruleRemoveById #2111

Changed compared variables of range id intervall in ruleRemoveById cl action. #2111

changed the variables in the comparison
added test case (@theMiddleBlue)
fixes #2111

Merged! Thanks.


segmantation fault issue

I installed matio on ubuntu following the instructions, namely:

1. # apt-get install libmatio2
2. # luarocks install matio

when I tried to run this little testfile.lua :

local matio = require 'matio'
tensor1 = matio.load('testdata.mat')

with torch (# th testfile.lua), I got the following error:

Segmentation fault (core dumped)

Did you find solution of this problem ?

no

Still no news on this?

Same problem here..

Any recent updates?

Any updates here? I am getting the same problem of "Segmentation fault"!

same problem

Same problem when I use Lua 5.2.
It's solved after I install LuaJIT.


Added check if the image changed sice last download

I've added control if the image have been changed since last download.

It would be better if you opened several PRs for different changes, but here are my comments:

Do we need a check if the image changed since last download? If you execute script in every 10 minutes, the image is guaranteed (well, *very* likely I mean) to be changed, since they updated it in every 10 minutes.

Wallpaper scaling is a great catch, thank you! Can you please open another PR and make sure to add your name to the AUTHORS list. :)

I really want to keep himawaripy as simple as possible and while offering a installation script is a great idea, it's also an another responsibility burden. If you think that installation instruction in README lacks something, feel free to add or change though.

Thanks a lot for your contribution!

Thanks for your feedback, I will do it as you described.
The installation script is for my purposes, I didn't want to include it to the pull request, sorry.


Component updating instructions

Readme.md is updated with instructions how to create a new component in Stencil.
I hope it doesn't have many grammatical mistakes. If there are some please fix them (as sending them back to me and then checking it again will take more time).
Status: Ready for review
Reviewers: @avelinet @jeffkamo @cole-sanderson @mlworks @kpeatt @ry5n

@nastiatikk I've applied some grammer tweaks.
Also, some of your steps were slightly complex. So instead, I broke them up into a few smaller simpler steps.

@nastiatikk I made a couple of minor changes. Otherwise looks good to me!

Changes look pretty good! :+1:

Yup looks good now, thanks for bearing with me. +1.


Finish Mobile Version Pull Request

Add Work section
Add About myself section
Add Images
Make necessary corrections

@MaganaAsiati done the changes please look at it and comment before i resubmit it for review

@GedeonTS Hey Ushindi...on the changes i used px and you have suggested to use %...to my knowledge percentages aren't supposed to exceed 100% so when you say i change all my width to percentage should i use a converter or just estimate? Please clarify


WIP/ENH: stats: Add support for nan_policy to stats.median_test

I merged @Inoryy's two commits from https://github.com/scipy/scipy/pull/5723, and added a couple tweaks in a new commit:

When 'propagating' a nan, it doesn't make sense for the fourth return value (which is usually the contingency table) to be nan, so I changed it to return an empty array with shape (2, 0).  (Alternatively, I suppose None would also be a reasonable value; i.e. "there was a nan in the data, so there is no contingency table".)

numpy.nanmedian was added to numpy in version 1.9.0, so until that is the minimum version supported by scipy, the grand median is calculated using
    grand_median = np.median(cdata[~np.isnan(cdata)])

Don't merge this yet.  I'm having second thoughts about the value of the fourth argument when nans are propagated.

In the nan propagation case, I changed the return value of the fourth argument from an empty array to None.

Thanks Warren, merging.


Allow the daily system to use the new repeat parameter on the daily trivia file

Description:

Duplicates the daily questions the x amount of time - 1, that is indicated in the daily question.

Other checks:

[X] I have tested all my changes thoroughly.

As discussed, this isnt a thing.


Add minimal travis config file

test rule will fails with:
  Error Message:
   System.IO.FileNotFoundException : Could not find file '/home/travis/build/malaterre/charls-native-dotnet/bin/Debug\DataFiles\t8c0e0.jls'.
  Stack Trace:
     at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)

dotnet test is currently failing on focal since libcharls2 was providing /usr/lib/x86_64-linux-gnu/libCharLS.so.2.0 (case sensitive issue).

Hi Mathieu,
Some feedback:

There is already an existing Azure CI pipeline configured. This CI pipeline can also build Linux targets. I think it is easier to only maintain 1 single CI pipeline.
The proposal install the latest packed CharLS version (libcharls2). During development it is however needed to use the latest version of the CharLS version from the sub-module. Otherwise it becomes impossible to add a new C API and a C# function that uses that new function. End-users should probably use indeed apt-get install libcharls2 in most cases.

I see it now at 819fa8b0cc71b93e37b86ed487593cb92e1fa8b5. Feel free to close then.


Draft: Proof of concept for append pack tag helper

Another stab at improving #39
Adds new helper to allow providing additional packs in addition that get appended to the main javascript_pack_tag call
javascript_pack_tag calls still raises when used multiple times but helper can streamline addition of packs in partials or conditionally.
Once we get list of all the tags to output, we deduplicate them and prioritise non-deferred ones - if particular bit is required by both deferred and non deferred call, the non-deferred one will be the one loaded.
Still needs testing and cleanup but just wanted to get initial version out to get thoughts and opinions
cc @justin808

Hi @tomdracz! We need some docs on this.
I like the new API!
  def append_javascript_pack_tag(*names, defer: true)

This would seem to well handle the cases where the:

javascript_pack_tag is only on the layout or a partial called from the layout.
The layout never calls append_javascript_pack_tag. In other words, when the main view (not layout) is rendered, it can call append_javascript_pack_tag (and so can any partials called by it as well). But don't try this with partials called by the main view, such as a footer partial.

The tough thing about this API is that there are a few edge cases that are subtle.
How about if we raise if append_javascript_pack_tag is called after javascript_pack_tag?

@tomdracz would be great if we had a full test cases app, as started in #67.

Thanks @justin808

How about if we raise if append_javascript_pack_tag is called after javascript_pack_tag?

Great point! Haven't thought about this but it makes perfect sense.
I'll get this changed, add some docs and see if I can pick up full test app

@tomdracz let me know when this is ready for final review!

@tomdracz I'm looking forward to this one! big time!

@justin808 Ready for review!

LGTM!
@tomdracz Go ahead and merge if you're confident! 🚀 🚀 🚀
We really need a sample rails app in the repo for proper tests, like
https://github.com/shakacode/react_on_rails/tree/master/spec/dummy

@tomdracz did we address this issue for stylesheets?


claim init failed. please reboot device  (AEGHB-186)

Hi
Received device. Is nice. Doing setup. Updated the firmware to 0.3. That went well. However, when using the app on iphone to try to setup, when I get to step to try to connect to wifi, i always get message "claim init failed. please reboot device". Any advice please?

the network I am trying to connect to does support 2.4ghz

I got the same error.
At first, I used iPhone X  +  iOS 16.1.1   + ESP Box V1.1.0.
I tried to clear the network info but failed again and again.
Then I used Android(Huawei Mate 30) and succeeded.

@ESP-Mars phone is latest iphone se. ios 16.3.1. ESP Box V1.1.0
Unfortunately, I dont have an android to work around the problem 🗡

After Provisioning started end.  Adding the following steps , it can temporarily fix this issue.
ble_gap_set_prefered_default_le_phy(BLE_HCI_LE_PHY_1M_PREF_MASK, BLE_HCI_LE_PHY_1M_PREF_MASK);

Same issue. iPhone 11, iOS 16.5, ESP Box v1.1.0

Same issue: ESP32-S3-BOX 01.01 with firmware v0.3.0 from ESP-BOX app on iPhone 11 IOS 16.5.  I also tried flashing the box with firmware v0.2.1 but same claim init issue.
Please could you explain about the text below:

After Provisioning started end. Adding the following steps , it can temporarily fix this issue.
ble_gap_set_prefered_default_le_phy(BLE_HCI_LE_PHY_1M_PREF_MASK, BLE_HCI_LE_PHY_1M_PREF_MASK);

Same issue. ESP32-S3-BOX was already on FW version 0.3 when I tried to set it up.
iPhone 11, iOS 16.5, ESP Box v1.1.0

can you try build the fw version of master. and try again?

Same issue: ESP32-S3-BOX 01.01 with firmware v0.3.0 from ESP-BOX app on iPhone 11 IOS 16.5. I also tried flashing the box with firmware v0.2.1 but same claim init issue.
Please could you explain about the text below:

After Provisioning started end. Adding the following steps , it can temporarily fix this issue.
ble_gap_set_prefered_default_le_phy(BLE_HCI_LE_PHY_1M_PREF_MASK, BLE_HCI_LE_PHY_1M_PREF_MASK);

Same issue: ESP32-S3-BOX 01.01 with firmware v0.3.0 from ESP-BOX app on iPhone 11 IOS 16.5. I also tried flashing the box with firmware v0.2.1 but same claim init issue.
Please could you explain about the text below:

After Provisioning started end. Adding the following steps , it can temporarily fix this issue.
ble_gap_set_prefered_default_le_phy(BLE_HCI_LE_PHY_1M_PREF_MASK, BLE_HCI_LE_PHY_1M_PREF_MASK);

I may have made a mistake.
I tried to reproduce this issue in the master version, and it appeared. So I modify it here, and it's alright.
Later, I couldn't reproduce this problem in the master version, even without the above changes, only appeared in 0.3.0.

I have now downloaded esp-idf and esp-box, done "idf.py build flash monitor" for the factory demo, and the device now connects with the ESP-BOX app on my iPhone 11 and is connected to WiFi. When I look at the "About us" screen on the device it says "Software Ver: v0.4.0" (NOTE: version four) and "ESP-IDF Ver: v5.2-dev-1010-g213504238"
But when I look in the ESP-BOX app under Device Settings it says "Firmware ver: v0.3.0-75-g7e5256f"
I then ran "python3 -m esptool --chip esp32s3 write_flash 0x0 Downloads/ESP-BOX_Demo_EN_V0.3.0.bin", removed the device from the ESP-BOX app and tried to add it again and got "Claim init failed..." In the "About us" screen the device reports "Software Ver: v0.3.0" (NOTE: version three) and "ESP-IDF Ver: v4.4-264-g2bdea81b2a"
I then did "idf.py flash monitor" for the factory demo and the ESP-BOX app connected with no problem. But it shows thre devices (light, switch, fan) whilst the device shows four (light, switch, fan, air).  The app shows the version as v0.3.0-75... whilst the device shows the version as v0.4.0. The app shows "IP address: --" and "Config Time: 2020-03-20"
Please may you update the ESP-BOX_Demo_EN_V0.3.0.bin with a more recent version of the binary. Please may you update the "factory demo" example code and ESP-BOX app so that they show the same number of devices, and the same version of the firmware, and the app displays the IP address and has the right Config Time?

Received my two ESP32-S3-BOX devices from Digi-Key today.  Experienced the failure to connect 2.4GHz wifi, same messages: claim init failed. please reboot device.  Happened twice, looking for updated firmware to flash.

Due to recent certificate adjustments, the IOS APP can't add devices.
We updated last week and are ready to release.

@keith721 @RobG-git @dustins @HelloFOFO Hey friends, we have solved the titled issue, but here is a packaging error during the process of uploading APP to Apple Store, we have to wait for the official solution from Apple Developer (https://developer.apple.com/forums/thread/727564). Until then, please use ESP-BOX for Android to experience the function. Thank you for your patience and understanding.

It appears a fix/solution/workaround was posted by user vadimwe on the first page:
Workaround is to update all the generated ...-frameworks.sh files to add the -f flag to the call to readlink. In other words, need to replace:

    source="$(readlink "${source}")"

with

    source="$(readlink -f "${source}")"

Seems this is CocoaPods issue...

It appears a fix/solution/workaround was posted by user vadimwe on the first page:
Workaround is to update all the generated ...-frameworks.sh files to add the -f flag to the call to readlink.
In other words, need to replace:

    source="$(readlink "${source}")"

with

    source="$(readlink -f "${source}")"

Seems this is CocoaPods issue...

The bug has been fixed, please update your ESP-BOX from App Store.

Thank you all for your testing. I will now close this issue. If you have any further questions or concerns, feel free to reopen it.

@ESP-Mars
Same error in ESP-BOX App in Android version 13.


Add dependencies from pipfile to requirements.txt

This PR adds the package dependencies specified in Pipfile to requirements.txt. Since Pipfile is not used by the downstream package in the installation, this ensures that a package specifying tinker-engine as dependency does not have to take into account tinker-engine's dependencies.

This PR adds the package dependencies specified in Pipfile to requirements.txt. Since Pipfile is not used by the downstream package in the installation, this ensures that a package specifying tinker-engine as dependency does not have to take into account tinker-engine's dependencies.

What's the error that you get when using tinker engine downstream?

import error associated with smqtk-core


Connection to server lost

Connection with server keeps getting lost everytime I open Syncplay and create a room
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Open Syncplay
Create and Join room

Expected behavior
Server connection should be established
Screenshots

Version and platform:

OS: Windows 10 64-bit
Syncplay version and build type: Syncplay v1.7.0
Media player and version:  VLC 3.0.18 32-bit]

Additional context
Add any other context about the problem here.

I can confirm that our public servers seem to be a bit unreliable at present. I have asked our server administrator to restart the public server. Hopefully when they do so this will resolve the issue. Private servers should be unaffected by this issue.

The public servers should be working again now :-)

Yep its working now!

Same problem again right now ((

Thanks for reporting this issue again. We've restarted our public servers and that should have fixed it.


Cleanup features

Description

https://trello.com/c/YWiBYi5C/13-styling-and-ui-cleanup

I created a toggle function in the recipes controller that is called by the javascript file toggle_favorite. In short this makes it so user can choose to add recipe to favorites from the recipe show page as opposed to using the checkbox on the form. This makes the UI look better and makes for better user experience.
I also changed the styling on the form page. Mostly tailwind classes.  Added some styling in the application css to manage tom-select styling to make the category input match the rest of the form fields.
I removed white space and ordered some files. Nothing crazy.

This change is to make the UI a little cleaner as well as removing is favorite from the form saving room and making layout more fluid.

Testing

I did test the labels, I had an issue with the label covering the data making it look blurry. I added a higher translate styling to make the label go higher to fix this issue.

Please merge master and run prettier to format javascript and css files


Memory leak on Raspberry Pi4 with Raspbian

Summary of Problem

What are you trying to do?
I try to use Serialport to connect a port on a Raspberry Pi4 with Raspbian installed. We have used the same setup before on a Raspberry Pi3 and it was working perfectly fine. Changing to Raspberry Pi4 results in the problem described below.

What happens?
The result is, that the memory usage rises 1-2MB per second until it is full and the application crashes.

What should have happened?
As in the other setups there should be no additional memory consumption so the Serialport library can be used in a stable setup.

This problem does not occur with the following setups:
Raspberry Pi3 + Raspbian
Raspberry Pi4 + Ubuntu Server
Raspberry Pi3 + Ubuntu Server
--> Any suggestions on what to change and/or is this a known issue that I am just not able to find in the recent issues?
--> In our real world setup, we list all Serialports with the function SerialPort.list() and try to connect to one port after another in the list. Apparently the error occurs and leads to the memory leak, when trying to establish a connection to a port and sending a command to a "faulty" port (like a port that has no device connected).
Code to Reproduce the Issue
--> For this code no device needs to be connected to the Pi.
const SerialPort = require('serialport')
const path = '/dev/ttyAMA0'  // or any other port, like '/dev/serial1'
const port = new SerialPort(path, { autoOpen: false })
port.open(function (err) {
        if (err) {
                console.log(err)
        } else {
                port.write('TESTDATA', (err) => {
                        console.log(err)
                })
        }
})

// Start reading from stdin so we don't exit.
console.log('--- Press key to continue ---')
process.stdin.resume()

Versions, Operating System and Hardware

<EMAIL_ADDRESS><-- same issue on<EMAIL_ADDRESS>Node.js v10.19.0 <-- same issue on Node.js v12.18.2 (and v14.5.0 @mawi-cs)
Linux Raspbian <-- not happening on Ubuntu Server
Raspberry Pi 4  <-- not happening on Raspberry Pi 3

I have just tested this with Node 14.5.0 and Serialport 9.0.0 on my RPi 4 with the newest Raspi OS as it is called now, and i am getting the same results.

When using Raspberry Pi OS on a Raspberry Pi 4, /dev/ttyAMA0 is the default Bluetooth UART. By default /dev/serial1 is a link to /dev/ttyAMA0 so it's the same device.
Running the following command from the command line does the same thing as the test code in the first post above:
echo TESTDATA > /dev/ttyAMA0

Running this command from the command line also results in issues. It doesn't result in memory leaks but it does result in the corresponding bash process blocking one of the 4 CPUs on the Pi 4 requiring 100% CPU.
Do you really want to use the Bluetooth UART and not the UART broken out to pins 8 and 10 on the Raspberry Pi GPIO header?
If the test program is modified to use /dev/ttyS0 (or /dev/serial0) then the test program will function as expected and the UART broken out to pins 8 and 10 on the Raspberry Pi GPIO header will transmit the string TESTDATA successfully.

Thank you very much for pointing this out. In fact deactivating Blutooth solves the issue for us. But with your input we might just check that we do not connect to /dev/ttyAMA0 or /dev/serial1.
Is there anything one can find out if a port is UART like a property that can be read after opening the port? Then I could just not test the port (e.g. not send data to it) but skip it. In this case the solution would be more generic when trying to avoid such a situation.

Is there anything one can find out if a port is UART like a property that can be read after opening the port?

To the best of my knowledge all serialports are UARTs. Bluetooth uses a UART and the serialport broken out to pins 8 and 10 on the GPIO header is a UART. I don't know if node-serialport can be used to determine if a serialport on a Raspberry Pi is for Bluetooth. Perhaps there is information related to the topic in the data returned by SerialPort.list().

I can verify that a Serialport stream on all serial ports on a Raspberry Pi 4 running the August 2020 Raspbian Buster will result in a  rapid memory leak if bluetooth is enabled at boot.

This problem still persists as of today. Disabling Bluetooth through the /boot/config.txt fixes the issue.
My RPi: 3b
My OS:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

SerialPort version: 10.5.0
Node version: 18
Would at least be interesting why exactly this happens.


Config to turn off allies in void areas

I was wondering if you could add a config to disable allies in areas they are not allowed in vanilla like the void fields?

This option is tied to the No Void Damage config option.

Oh cool! Thank you I didn't know that I'm going to go turn that off right now. Love the overhaul have a good weekend.


Uncaught TypeError: Cannot read property 'length' of undefined

[Enter steps to reproduce below:]

type @Override
begin typing whatever below that

Atom Version: 1.8.0
System: linux 4.6.2-1-ARCH
Thrown From: autocomplete-java-minus package, v0.10.0
Stack Trace
Uncaught TypeError: Cannot read property 'length' of undefined
At /home/ash/.atom/packages/autocomplete-java-minus/lib/entryfinders/method-local-fields.js:27

TypeError: Cannot read property 'length' of undefined
    at PlainEntryFinder.getMethodLocalFields (/home/ash/.atom/packages/autocomplete-java-minus/lib/entryfinders/method-local-fields.js:27:36)
    at PlainEntryFinder.getLocalFieldSuggestions (/home/ash/.atom/packages/autocomplete-java-minus/lib/entryfinders/plain-entry-finder.js:15:14)
    at PlainEntryFinder.get (/home/ash/.atom/packages/autocomplete-java-minus/lib/entryfinders/plain-entry-finder.js:60:12)
    at Provider.getSuggestions (/home/ash/.atom/packages/autocomplete-java-minus/lib/provider.js:75:36)
    at /usr/share/atom/resources/app.asar/node_modules/autocomplete-plus/lib/autocomplete-manager.js:310:56
    at Array.forEach (native)
    at AutocompleteManager.module.exports.AutocompleteManager.getSuggestionsFromProviders (/usr/share/atom/resources/app.asar/node_modules/autocomplete-plus/lib/autocomplete-manager.js:289:17)
    at AutocompleteManager.getSuggestionsFromProviders (/usr/share/atom/resources/app.asar/node_modules/autocomplete-plus/lib/autocomplete-manager.js:3:61)
    at AutocompleteManager.module.exports.AutocompleteManager.findSuggestions (/usr/share/atom/resources/app.asar/node_modules/autocomplete-plus/lib/autocomplete-manager.js:276:19)
    at /usr/share/atom/resources/app.asar/node_modules/autocomplete-plus/lib/autocomplete-manager.js:3:61

Commands
     -1:48.2.0 tree-view:toggle (atom-text-editor.editor.is-focused)
     -1:44.6.0 core:move-left (atom-text-editor.editor.is-focused)
     -0:41.1.0 core:backspace (atom-text-editor.editor.is-focused)
     -0:40.4.0 editor:delete-to-beginning-of-word (atom-text-editor.editor.is-focused)
  2x -0:36.1.0 editor:newline (atom-text-editor.editor.is-focused)
     -0:21.6.0 core:backspace (atom-text-editor.editor.is-focused)
  2x -0:11.9.0 susave:save (atom-text-editor.editor.is-focused)
  3x -0:06.4.0 editor:newline (atom-text-editor.editor.is-focused)

Config
{
  "core": {
    "disabledPackages": [
      "vim-mode",
      "highlight-selected",
      "minimap-highlight-selected",
      "tab-preview",
      "activate-power-mode",
      "linter-luacheck"
    ],
    "themes": [
      "one-dark-ui",
      "purple-haze-syntax"
    ]
  }
}

Installed Packages
# User
Sublime-Style-Column-Selection, v1.5.1
atom-cursor-indent, v0.1.1
atom-minify, v0.7.5
atom-notifier, v0.5.0
atom-yeoman, v0.3.15
atomic-chrome, v0.3.0
autocomplete-java-minus, v0.10.0
bottom-dock, v0.4.4
build, v0.64.0
build-gradle, v0.6.0
build-love, v0.1.0
busy, v0.5.0
file-icons, v1.7.15
fold-lines, v0.2.0
java-classpath-registry, v0.3.0
java-import-wiz, v0.5.0
java-plus, v0.1.0
json-schema, v0.1.15
language-gradle, v0.0.3
language-ini-desktop, v1.18.2
language-json-comments, v0.16.3
language-lua, v0.9.4
language-x86-64-assembly, v2.1.15
linter, v1.11.14
linter-lua, v1.0.2
linter-manager, v0.2.12
markdown-preview-plus, v2.4.0
minimap, v4.24.6
minimap-bookmarks, v0.3.1
minimap-codeglance, v0.4.5
minimap-find-and-replace, v4.5.1
minimap-pigments, v0.2.1
minimap-selection, v4.4.0
omnisharp-atom, v0.30.0
php-cs-fixer, v2.5.6
pigments, v0.29.2
purple-haze-syntax, v1.5.0
remember-folds, v0.1.0
susave, v0.4.0
svg-preview, v0.7.4

# Dev
No dev packages

Cannot reproduce. Please provide a minimal Java file exhibiting the issue with steps how to trigger it.


Integration Test Case between Fakes

@h1alexbel thank you for reporting this. I'll assign someone to take care of it soon.

@h1alexbel Don't forget to close this ticket before the deadline (2023-06-25T11:49:09). You are past the first half of the allowed period.

@h1alexbel Looks like you've missed the task deadline (2023-06-25T11:49:09). You are now resigned from this task.
Please stop working on it, you will not be paid. I will assign it to someone else soon.

@l3r8yJ this is your task now, please go ahead. Deadline (when this ticket should be closed) is 2023-07-05T11:59:09.024670.
Estimation here is 60 minutes, that's how much you will be paid.
Remember, you don't have to solve everything in this ticket - you can solve it partially and leave todo markers in the code, which will become future tasks.
If you have any questions don't ask me, I'm not a technical person. Open new tickets instead.
If you don't have time or simply don't want to work on this, you can always resign.

already implemented, see #392

@l3r8yJ thank you for resolving this ticket. I've just added it to your active invoice. You can always check all your invoices and more on the Contributor Dashboard.


Causes compiler build error

Version 2.0 1 causes compile errors:
OneWireESP32.cpp:44:1: warning: missing initializer for member 'rmt_transmit_config_t::::queue_nonblocking' [-Wmissing-field-initializers]
44 | };
OneWireESP32.cpp:77:9: warning: missing initializer for member 'rmt_rx_channel_config_t::intr_priority' [-Wmissing-field-initializers]
OneWireESP32.cpp:93:9: warning: missing initializer for member 'rmt_tx_channel_config_t::::invert_out' [-Wmissing-field-initializers]
Very much appreciate a fix.

Sorry - forget this - realise that there is a dependency on arduino 3.0 IDF 5.x


utils: add depth first and breadth first distinction in root pattern

Some have brought up issues with root_patterns behaving unintuitively. I've added a breadth/depth first distinction and refactored all modules to use the new helper functions.
NOTE: I did not optimize the new modules to use the simplest root function possible for each one. Comments welcome, I'm not going to merge this unless there is a strong signaling that this clarifies things.
@ahmedelgabri @lithammer I think you've both commented on root_patterns/finders before, let me know if I should bring anyone else in.

Also @hiphish

Also @hiphish

I chain calls all the time, which is why I marked this as a draft 😆 People seemed unhappy with there not being a depth/breadth option, or at least, confused because it was not clear from looking at the code (I guess) that it was breadth first. I don't think this is a strictly necessary PR, but I figured it would be easier to see if there is utility in adding this, and we can discuss it here instead of having many open issues about figuring out cleaner root pattern strategies.

I chain calls all the time, which is why I marked this as a draft 😆 People seemed unhappy with there not being a depth/breadth option, or at least, confused because it was not clear from looking at the code (I guess) that it was breadth first. I don't think this is a strictly necessary PR, but I figured it would be easier to see if there is utility in adding this, and we can discuss it here instead of having many open issues about figuring out cleaner root pattern strategies.

I think breadth_first_root_pattern is a bit verbose. Then again, you only use it like once per server, so it's not really much of an issue. But the breadth_first part of the name bothers me. I think it was me who started calling it that simply because i wanted to assign a name to it, but it's not really a breadth/depth search.
Breadth of width first is when you search through a tree starting at the root and digging down. Here however we are moving up the tree, the difference is in whether we try all the patterns at the same time during one traversal, or whether we have a separate traversal for each pattern. I don't know what to call that type of search.

I think breadth_first_root_pattern is a bit verbose. Then again, you only use it like once per server, so it's not really much of an issue. But the breadth_first part of the name bothers me. I think it was me who started calling it that simply because i wanted to assign a name to it, but it's not really a breadth/depth search.
Breadth of width first is when you search through a tree starting at the root and digging down. Here however we are moving up the tree, the difference is in whether we try all the patterns at the same time during one traversal, or whether we have a separate traversal for each pattern. I don't know what to call that type of search.

It's still (in the formal sense) a breadth first search over a tree, it's just in this case, the root node of the tree is the starting working directory, and the children belong to the parent directory.

It's still (in the formal sense) a breadth first search over a tree, it's just in this case, the root node of the tree is the starting working directory, and the children belong to the parent directory.

it's just in this case, the root node of the tree is the starting working directory

Is it? I was under the impression that the search starts at the directory containing the file and then moves up the file tree from there. I might have misunderstood the algorithm then.

it's just in this case, the root node of the tree is the starting working directory

Is it? I was under the impression that the search starts at the directory containing the file and then moves up the file tree from there. I might have misunderstood the algorithm then.

Yes, the algorithm is not specific to file tree, but is a general means of searching trees (so here, we're just defining the tree to be starting at the deepest level of the file system hierarchy)

Yes, the algorithm is not specific to file tree, but is a general means of searching trees (so here, we're just defining the tree to be starting at the deepest level of the file system hierarchy)

OK, but the algorithm only moves straight up, right? It doesn't traverse sibling directories. In that case the "tree" being traversed is just a straight line up, and if there are no turns to be taken, then there is no difference between breadth-first and depth-first. Take the following graph:
                A
                |
      +---------+---------+
      |                   |
      B                   C
      |                   |
+-----+-----+     +-------+------+
|           |     |              |
D           E     F              G
            |
            H

Starting at A, the breadth-first traversal order is A, B, C, D, E, F, G, H and the dept-first order is A, B, D, E, H, C, F, G because when we descend we have up to two choices. However, if we start at H the order is always H, E, B, A because there always exactly one choice on how to ascend.

OK, but the algorithm only moves straight up, right? It doesn't traverse sibling directories. In that case the "tree" being traversed is just a straight line up, and if there are no turns to be taken, then there is no difference between breadth-first and depth-first. Take the following graph:
                A
                |
      +---------+---------+
      |                   |
      B                   C
      |                   |
+-----+-----+     +-------+------+
|           |     |              |
D           E     F              G
            |
            H

Starting at A, the breadth-first traversal order is A, B, C, D, E, F, G, H and the dept-first order is A, B, D, E, H, C, F, G because when we descend we have up to two choices. However, if we start at H the order is always H, E, B, A because there always exactly one choice on how to ascend.

You're right in the sense that's it's technically not a tree, but a DAG since children have only one child, but can have multiple parents. The breadth and depth first still will ultimately visit the same nodes, so I think the terminology is fine.

You're right in the sense that's it's technically not a tree, but a DAG since children have only one child, but can have multiple parents. The breadth and depth first still will ultimately visit the same nodes, so I think the terminology is fine.

You be the judge then. My concern is that it might confuse new users because they expect "breadth-first" to mean something else, and the only way of finding out what it actually means is to read the source code (or documentation if it existed), which makes the name meaningless. If everyone else is fine with the name though, then so be it, I don't have any better names to suggest.

You be the judge then. My concern is that it might confuse new users because they expect "breadth-first" to mean something else, and the only way of finding out what it actually means is to read the source code (or documentation if it existed), which makes the name meaningless. If everyone else is fine with the name though, then so be it, I don't have any better names to suggest.


[Question]: How does triton avoid lds bank conflict?

Problem Description
Thank you for all the incredible work on this amazing project!
I was surprised to discover that Triton GEMM on ROCm is also conflict-free. Did we use padding or XOR-based swizzling to avoid bank conflicts?
I’m super curious how this solution is implemented! Where can I find more details on this? Thanks in advance!
Operating System
Ubuntu 22.04
CPU
AMD
GPU
MI300X
ROCm Version
ROCm 6.2.2
ROCm Component
No response
Steps to Reproduce
No response
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
No response

I've found related codes, thanks for your awesome solution, and also some discussions at #635.
thanks @zhanglx13 for your patient response :)


#147 share post

Tested this by sharing a post through the UI. Looks like it works. It will make a new post for that new Author (the id_url will include the id of the most recently shared-from Author) but with all the information from the previous post except for Source is changed.

Closes #147

This should be good, had merge conflict, but just had to merge main into this branch.


chore: upgrade vite dependency

As far as my testing shows, this works fine with Vite 6

@DylanPiercey Any chance we can release this?

Shoot, I'm used to the automated release workflow but this PR was missing a changeset (and I think the changeset bot is incorrectly configured so it didn't warn us). I'll get out a release in a few mins.


[Bug] [connector-tdengine]  ErrorCode:[COMMON-12]  tdengine  reader

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
I tried to migrate tdengine data using seatunnel, but cannot running
SeaTunnel Version
2.3.3
2.3.2
2.3.1
SeaTunnel Config
env {
  # You can set SeaTunnel environment configuration here
  execution.parallelism = 2
  job.mode = "BATCH"
  checkpoint.interval = 10000
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
        TDengine {
          url : "jdbc:TAOS-RS://<IP_ADDRESS>:6041/"
          username : "root"
          password : "taosdata"
          database : "test"
          stable : "meters"
          lower_bound : "2023-10-03 14:38:05.000"
          upper_bound : "2023-10-03 14:38:16.800"
          result_table_name = "tdengine_result"
        }
}

sink {
  Console {
  }

  # If you would like to get more information about how to configure Seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

### Running Command

```shell
./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

Error Exception
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.tdengine.exception.TDengineConnectorException: ErrorCode:[COMMON-12], ErrorDescription:[Source reader operation failed, such as (open, close) etc...] - get TDengine connection failed:jdbc:TAOS-RS://<IP_ADDRESS>:6041/test?user=root&password=taosdata
	at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.open(TDengineSourceReader.java:113)
	at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.open(SourceFlowLifeCycle.java:115)
	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:146)
	at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:89)
	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:526)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

	at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:119)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:174)
	... 2 more
2023-11-29 11:55:15,126 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

Zeta or Flink or Spark Version
No response
Java or Scala Version
openjdk version "11.0.21" 2023-10-17 LTS
Linux xxx 5.4.17-2136.323.8.1.el7uek.x86_64 #2 SMP Fri Sep 15 12:11:45 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux
Screenshots
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

reference
https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-tdengine-e2e

I had the same problem,so how u resolved it in the end?


Update battery default settings

Before the 2019 release, update battery costs and operational parameters based on recent references.

Battery costs updated for 2020.2.29 release based on NREL Annual Technology Baseline cost references.


White icons w/ transparent backgrounds aren't visible in Media Assets

Describe the bug
White icons with a transparent background are not visible in media assets because it uses a white background.
To Reproduce

Upload an image (we are experiencing this with SVG) with a white icon and a transparent bg
View the thumbnail of that icon in the Media Assets section of the site

Expected behavior
The background for the thumbnail should be something that provides adequate contrast to a wide range of images, such as a white/gray checkerboard pattern:

Screenshots

Applicable Versions:
Netlify CMS 4.2.2
CMS configuration
N/A

Agreed! Happy to take a PR for this, the only requirement is that we use a CSS-only solution for the checkered background itself.

Seems reasonable, based on things like this CodePen:
https://codepen.io/lloeki/pen/Dvkqw
I actually like the gray/dark gray pattern over the pure white checkerboard since that might 50% obscure white icons.

It'll always obscure something, eg. gray icons. The pattern used in google images seems similar to what I've seen elsewhere:

I'm not super opinionated on this though, whatever works.

Yeah I feel you. I was just thinking that I really like the UI for this SVG optimizer:
https://jakearchibald.github.io/svgomg/
That uses a darker bg which works great for white icons. Down for whatever works though. If I pick this up I'll try a few options with different icons.

Go with that one 👌🏼👍🏼
Sent with GitHawk

Opened a PR to address this!
https://github.com/netlify/netlify-cms/pull/2152


added detail

Please have more descriptive commit messages and pull request titles in the future.

Sure, Thankyou


Cannot access /keybase

VERSION:
2.0.0-20180604163023+f899a761a7
> % cd /keybase
cd: no such file or directory: /keybase

I have read through https://github.com/keybase/kbfs/issues/1368 and am unable to find the button to install the kernel extension inside of the file / folders tab in the GUI

@petergarbers what operating system are you running?  If you're on mac or Windows, look in the "Files" section of the GUI, or in the settings under "Files".

Ah ok, you already have it checked it in the settings.  This is a known bug that we'll fix soon.  You need to quit the Keybase app and relaunch it to get the Finder integration after you set it on osx.

Hi @strib
I have gone as far as to restart my machine, to no avail.

@petergarbers ok, if you do keybase log send and post the resulting log ID here, I can take a look.

Reinstalling seems to have fixed it! Thanks


A Dockerfile to run EvoDiff locally in a container (with GPU support)

@microsoft-github-policy-service agree [company="Tuple, LLC"]

@microsoft-github-policy-service agree


The commits for 1.37.5+

Not sure if you noticed, but the commits for 1.37.5+ already exist.
The branch 1.37.5_breaking is for 1.37.5+
My 3.0.0-5 release's 1.38 zip still works for 1.39.1.

You can close the pr if you forked it just to release the it for 1.39.1


Create a trail from the plugin

The user should be able to create a trail from the plugin.

Whether it be a link to the create trail depot page or a modal, the user needs to be able to create a trail directly from the plugin. User feedback has been that having to go to the depot to create a trail is too time consuming.

User should be able to type in a new Trail name.  We don't really have a place to put a description there, so should we default description to match trail name when created from the toolbar?  User can always edit that via the Depot after.

User can successfully add a new trail from the toolbar.


Compiled UI files require manual edits

All UIs in mGear import Qt like so:
import mgear.core.pyqt as pyqt
When I build a UI using the "Compile PyQt ui" option, the resulting file does imports like this instead:
from PySide2 import QtCore, QtGui, QtWidgets
I assume the python file should be manually edited after compilation to bring it in line with other UIs? The generated python is so consistent I think it would be possible for mGear to fix it automatically after compilation.

Hello @Ruuttu
sorry for the late reply.
Yes for the moment it is needed to update this manually.
Probably in the future, we will stop using qtDesigner and .ui files. So I am not sure if is worth to invest time to avoid manual edits

Aha. Would the replacement be automatically generated UIs for parameters? Or plain written UI code?
But anyway, makes sense.


Run solver in the analysis phase

This is faster, and avoids the need to run separate Docker builds on platforms for which there is no solution. More importantly, it allows us to follow changes in opam-repository more closely, since updating that no longer requires pulling a new set of base images and rebuilding everything.

This is missing support for opam's pin-depends, e.g. https://ci.ocamllabs.io:8100/job/2020-04-30/120634-ci-build-133485

Also, in opam 1.x format files it looks like the available field is ignored: https://ci.ocamllabs.io/github/0install/0install/commit/680f1b2342a3d07533179f6aaed7e26cd51c3d8a/variant/debian-10-ocaml-4.02

This seems like a big improvement even before pin-depends support. As far as I can tell, pin-depends is broken before this PR in the case of pins to git tags, since they would be cached to the first reference the tag was resolved to.

Yeah, my plan is to finish and upstream the changes to the solver libraries, then merge this, then investigate how to fix pin-depends.


Auto detect image type and SVGs?

I'm using cropit for all image upload files in our CMS to force users to upload images at exactly the right size and dimensions.
What would be the best way to check the file type being passed to cropit to allow it to export a JPG or PNG with transparency?
Also, can cropit handle SVG files?

I've solved the detecting image type between .jpg/jpeg and .png by adding the following near end of getCroppedImageData() function within jquery.cropit.js itself:
if (exportOptions.type === 'image/jpeg') {
    canvasContext.fillStyle = exportOptions.fillBg;
    canvasContext.fillRect(0, 0, canvas.width, canvas.height);
}

However, I can't find any info about cropping SVG's.  Anyone know if this is possible?

You can make use of image onload callback.


[WFLY-11876]: Upgrade Artemis to 2.7.0

Updating to Apache Artemis 2.7.0
Taking new Artemis native packaging into account
Taking advantage of ARTEMIS-2163.

Jira: https://issues.jboss.org/browse/WFLY-11876

Waiting for testing results

Are there any release notes for the changes included in this version of Artemis compared to the prior version?

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342977&styleName=Html&projectId=12315920&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED|aca26efea9a8c56184fe65141c0d1f1bf396e637|lin

FYI: there have been a 2.8.0 release to which we should upgrade https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345169&styleName=Html&projectId=12315920&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED|aca26efea9a8c56184fe65141c0d1f1bf396e637|lin

Dependency Tree Analyzer Output:
New Dependencies:

org.apache.activemq:activemq-artemis-native:jar:1.0.0:compile

Removed Dependencies:

org.apache.activemq:artemis-native:jar:2.6.3.jbossorg-00014:compile

The change is 'just' an artifact moved to a new GAV.

I'm going to merge this, but I'd like to know more details on why the org.jboss.logmanager is required. If the java.util.logging.manager system property is expected to be set this makes the logging subsystem no longer optional if the messaging subsystem is used.


UpdateBatch doesn't update all records when condition specified

Hey there!
I have a question regarding UpdateBatch with the BatchSize and update conditions specified on EF Plus v.3.1.15
Issue description
Here is an example. We have Policies table with over 90k records. 68,080 of them are not enabled. But after executing the following code we get 98 affected rows.
var affectedRows = await context.Policies
    .Where(policy => policy.IsEnabled == false)
    .UpdateAsync(
        policy => new Policy { Tooltip = policy.Tooltip },
        update => { update.BatchSize = 100; });

Generated query:
DECLARE @stop INT
DECLARE @rowAffected INT
DECLARE @totalRowAffected INT
DECLARE @ZZZ_INDEX INT

SET @stop = 0
SET @totalRowAffected = 0
SET @ZZZ_INDEX = 0

SELECT Z_Batch.[PolicyId],
       ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS ZZZ_INDEX
INTO #ZZZBatch_0817540c_1706_4fa2_80a8_09d1e5fce53f
FROM [Policies] AS Z_Batch

WHILE @stop = 0
BEGIN
    UPDATE A
    SET A.[Tooltip] = B.[Tooltip]
    FROM [Policies] AS A
    INNER JOIN ( SELECT * FROM #ZZZBatch_0817540c_1706_4fa2_80a8_09d1e5fce53f AS ZZZ_temp
                WHERE ZZZ_temp.ZZZ_INDEX > @ZZZ_INDEX AND ZZZ_temp.ZZZ_INDEX <= @ZZZ_INDEX + 100
               ) AS ZZZ_Batch ON A.[PolicyId] = ZZZ_Batch.[PolicyId]
    INNER JOIN ( SELECT [p].[PolicyId], [p].[IsEnabled], [p].[Tooltip]
FROM [Policies] AS [p]
WHERE [p].[IsEnabled] = CAST(0 AS bit)
               ) AS B ON A.[PolicyId] = B.[PolicyId]

    SET @rowAffected = @@ROWCOUNT
    SET @ZZZ_INDEX = @ZZZ_INDEX + 100
    SET @totalRowAffected = @totalRowAffected + @rowAffected

    IF @rowAffected < 100
        SET @stop = 1
END

SELECT @totalRowAffected

The issue here is that we update only the records which satisfy condition [p].[IsEnabled] = CAST(0 AS bit) out of the first 100 and stop because of this:
    IF @rowAffected < 100
        SET @stop = 1

So generated query going to update records in batches of 100 till the moment when the first record not meeting the specified condition found.
Is it an issue or made for some reason?
Solution
Having specified condition in the inserting query resolves the issue:
SELECT Z_Batch.[PolicyId],
       ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS ZZZ_INDEX
INTO #ZZZBatch_0817540c_1706_4fa2_80a8_09d1e5fce53f
FROM [Policies] AS Z_Batch
WHERE [Z_Batch].[IsEnabled] = CAST(0 AS bit)

Best regards,
Viktor

Hello @vkrylovets ,
Thank for reporting, that's indeed a very wrong behavior.
My developer will look at it and fix it asap. We already discussed the solution.
We cannot use your solution as we don't generate this part using your queries. For a simple scenario, we could probably find a way but for more scenarios with join, it could become troublesome.
Best Regards,
Jon

Performance Libraries
context.BulkInsert(list, options => options.BatchSize = 1000);
Entity Framework Extensions • Entity Framework Classic • Bulk Operations •  Dapper Plus
Runtime Evaluation
Eval.Execute("x + y", new {x = 1, y = 2}); // return 3
C# Eval Function • SQL Eval Function

Hello @vkrylovets ,
The v5.1.17 has been released.
We indeed have a very bad logic to handle the BatchSize.
Could you try this latest version and let me know if everything is now working correctly?
Best Regards,
Jon

Hello, @JonathanMagnan
Just checked v3.1.17
Now UpdateBatch works like a charm.
Generated SQL is correct. Thanks!
Best regards,
Viktor


Rewrite gateway emitter for significant performance improvement

Rewrite the Emitter for performance. The emitter is what's called when bytes, an event, or a JSON payload is being emitted by the ShardProcessor.
Emitters have internally held a DashMap of listeners; shards and clusters could create multiple streams of events via Cluster::events, Cluster::some_events, Shard::events, and Shard::some_events. This was an intentional design decision, but over time we've seen that this capability hasn't seen much purpose.
Instead, we can create a stream of events when clusters or shards are created, and simply return that from Cluster::new, Shard::new, and their builder equivalents. This means that instead of only returning the cluster or shard, a two element tuple is returned of the cluster or shard and the stream of events. Events can then be sent by the ShardProcessor directly over a channel that the stream polls. This results in removing DashMap as a dependency and a significant performance improvement, as seen in the benchmark below.
The first pair of tests is emitting the raw bytes of a GUILD_ROLE_DELETE dispatch payload. The first test has a listener that doesn't want to receive the EventTypeFlags::SHARD_PAYLOAD variant, while the second does. While the first test sees no change - the change tends to go up and down due to how small of a scale of time it is - the second sees a significant improvement.
The second pair of tests is similar to the first, but instead of emitting the raw bytes of the payload it is attempting to emit a received payload as a deserialized event. If a listener does not want an event type, then the payload is discarded. If the user does want the event type, then the payload is deserialized and then sent to the user. Much like the first pair, the first test receives little improvement, but the second receives a comparable performance improvement.
emit bytes: doesn't want
                        time:   [3.1906 ns 3.1919 ns 3.1936 ns]
                        change: [+0.1098% +0.3127% +0.5112%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

emit bytes: wants       time:   [77.733 ns 78.493 ns 79.427 ns]
                        change: [-79.574% -79.312% -79.025%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe

emit json: doesn't want time:   [7.2541 ns 7.2587 ns 7.2656 ns]
                        change: [-2.2163% -2.1091% -1.9993%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe

emit json: wants        time:   [274.46 ns 276.97 ns 279.86 ns]
                        change: [-60.786% -59.956% -58.741%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

Criterion benchmark used:

use criterion::{criterion_group, criterion_main, Criterion};

use twilight_gateway::{shard::emitter::Emitter, EventTypeFlags};

const INPUT: &str = r##"{
    "op": 0,
    "s": 2,
    "d": {
        "guild_id": "1",
        "role_id": "2"
    },
    "t": "GUILD_ROLE_DELETE"
}"##;
const INPUT_BYTES: &[u8] = INPUT.as_bytes();

fn emit_bytes(emitter: &Emitter) {
    emitter.bytes(INPUT_BYTES);
}

fn emit_json(emitter: &Emitter, input: &mut str) {
    emitter.json(0, Some(12), Some("GUILD_ROLE_DELETE"), input).unwrap();
}

fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("emit bytes: doesn't want", |b| {
        let (emitter, _rx) = Emitter::new(EventTypeFlags::empty());

        b.iter(|| emit_bytes(&emitter))
    });

    c.bench_function("emit bytes: wants", |b| {
        let (emitter, _rx) = Emitter::new(EventTypeFlags::SHARD_PAYLOAD);

        b.iter(|| emit_bytes(&emitter))
    });

    c.bench_function("emit json: doesn't want", |b| {
        let (emitter, _rx) = Emitter::new(EventTypeFlags::empty());
        let mut input = INPUT.to_owned();

        b.iter(|| emit_json(&emitter, &mut input));
    });

    c.bench_function("emit json: wants", |b| {
        let (emitter, _rx) = Emitter::new(EventTypeFlags::ROLE_DELETE);
        let mut input = INPUT.to_owned();

        b.iter(|| emit_json(&emitter, &mut input));
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

Addressed comments


npm start

error ->
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: node index.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /data/data/com.termux/files/home/.npm/_logs/2021-02-14T17_20_01_911Z-debug.log

same

Update
npm
npm install @open-wa/wa-automate@latest
yarn
yarn upgrade @open-wa/wa-automate --latest
Also you need upgrade @open-wa/wa-decrypt

Les funcionó?

Please use google translator
Funcionou sim, também usei o  comando "npm update"
Parece quem faltavam algumas pendências também, após instalar tudo corretamente funcionou :)
Thanks aymenitc <3


[CELEBORN-1487][Phase2] CongestionController support dynamic config

What changes were proposed in this pull request?
CongestionController support dynamic config
Why are the changes needed?
Currently, Celeborn only supports quota management based on disk file bytes/count, and this quota management cannot cope with sudden increases in traffic, which will cause corrupt to the cluster.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
UT.

Can you help review? @RexXiong


Use better example + typo fix

The Clock is already part of the llm-chain package

oh damn, i need to level some stuff up if you really gonna use this thing :)


[bug] sidebar not visible when canvas fills the height

Tested on iOS 16 and iPhone 12 mini screen
Example:
https://itsjavi.com/supereffective-dextracker/

I'll check that one.


Allow dynamically adding/removing tracked contract addresses

Do you want to request a feature or report a bug?
feature
What is the current behavior?
Tracked contracts are statically defined and can't be added/removed at runtime.
In some scenarios we want to be able to dynamically add/remove tracked contract addresses based on events that happen in another contract (e.g. when we have a contract that creates other contracts).

Tracked via #719.


No more useless subst

The subst of parameters of Ctor now disappear after tyck that Pat.Ctor. For example:
open data Nat
| O
| S (_1 : Nat)

def justMatch (m n : Nat) : Nat
| O, n => O
| S m', n => m'

There is a subst _1 |-> m' when we tyck the  n pattern in | S m', n => m', which is useless.

bors r+


Refactor into packages

The current implementation of the weather application is all under one package of weather. Whilst it has worked out okay so far, the organisation would improve drastically if I separate the application in to the following packages: region and site. This will so avoid the prefix with Region and sites for site.go and region.go.

version bump on the help branch to 0.7 required as well

Have to rethink on the refactor because there is a circular import problem whereby I cannot do weather ->region->weather.
Theres some things in weather that are needed for region and weather has to import back region for cmd.go. One solution is probs to separate out cmd.go?

One solution is to have an 'app' package which will contain the contents of weather.go. Seems to be appropriate because weather.go provides a client and formats URLs. This means that I can keep cmd.go in the weather package and this can avoid the circular dependency I was getting into.

Done


DoRA uses lots of GPU VRAM due to fp32 upcasting

System Info
peft 0.10.0, transformers 4.40.1, Python 3.10 on Ubuntu  22.04
Who can help?
No response
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder
[ ] My own task or dataset (give details below)

Reproduction
Doing language model fine-tuning using QLoRA with DoRA, eg. fine-tuning Meta-Llama-8-70B with https://github.com/huggingface/trl/blob/main/examples/research_projects/stack_llama/scripts/supervised_finetuning.py, with target_modules set to include all linear layers, uses much more GPU VRAM than training with ordinary LoRA.
Expected behavior
Fine-tuning a QLoRA language model using DoRA, with adapters applied to all linear layers, takes up much more GPU VRAM than ordinary LoRA and OOMed my machine. I think the issue is this line:
https://github.com/huggingface/peft/blob/608a90ded9985ee1c5912d738082bb1fd618902b/src/peft/tuners/lora/layer.py#L238
it looks like magnitude is in fp32, so the input vector x is upcast to fp32 when it gets returned as result_dora. If both MLP and attention layers are added to target_modules, that fp32 output then causes the next DoRA module (in the MLP layer) to get an fp32 vector as input. This then causes the dequantized weight matrix to get upcast to fp32:
https://github.com/huggingface/peft/blob/608a90ded9985ee1c5912d738082bb1fd618902b/src/peft/tuners/lora/layer.py#L229
which means the algebra in _get_weight_norm is done in fp32:
https://github.com/huggingface/peft/blob/608a90ded9985ee1c5912d738082bb1fd618902b/src/peft/tuners/lora/layer.py#L176
which OOMs my machine. Adding a cast back to x.dtype here: https://github.com/huggingface/peft/blob/608a90ded9985ee1c5912d738082bb1fd618902b/src/peft/tuners/lora/layer.py#L253
fixes the problem. (I also wrote a custom Triton kernel for _get_weight_norm(), but that's probably not necessary for most purposes)

Thanks for investigating this issue. Could you provide a bit more information so that we can reproduce this issue (ideally with a smaller model, so that we can create a unit test based on this). For instance, what dtype do you use, bf16?

it looks like magnitude is in fp32, so the input vector x is upcast to fp32

What was the initial dtype for you?

I also wrote a custom Triton kernel for _get_weight_norm(), but that's probably not necessary for most purposes

If you want to consider contributing this to PEFT, let us know.

@BenjaminBossan Thanks! I'm using bf16. Any broadly Llama-like model should work, eg. I've run tests with Meta-Llama-3-8B and Mistral-7B-v0.1 (the original, non-MoE Mistral).
Here's the Triton code I wrote. When I benchmarked it, it reduced total training time by 20% and reduced GPU VRAM consumption by 1,974 MB (this is comparing it to the PyTorch implementation of _get_weight_norm() after I already added return result_dora.to(x.dtype) at the end of _apply_dora()). Settings were fine-tuning Llama-3-70B with QLoRA, 2x 4090 GPUs, BNB 4-bit quantization with nf4 data type, compute_dtype bfloat16, double quantization enabled, LoRA rank r = 12, target modules set to ["q_proj", "v_proj", "o_proj", "k_proj", "gate_proj", "up_proj", "down_proj"] (all linear layers), learning rate 1e-4, batch size = 1, gradient accumulation = 4, sequence length = 1024.
(this is my first time writing Triton code, so I'm sure I did something dumb somewhere, but the tests pass and training seems stable)
import torch
import triton
import triton.language as tl

# tunable
BLOCK_SIZE_N = 64

@triton.autotune(configs=[
    triton.Config(kwargs={'BLOCK_SIZE_M': 32}, num_warps=4, num_stages=4),
    triton.Config(kwargs={'BLOCK_SIZE_M': 32}, num_warps=2, num_stages=5),
    triton.Config(kwargs={'BLOCK_SIZE_M': 32}, num_warps=1, num_stages=6),
    triton.Config(kwargs={'BLOCK_SIZE_M': 64}, num_warps=4, num_stages=4),
    triton.Config(kwargs={'BLOCK_SIZE_M': 64}, num_warps=2, num_stages=5),
    triton.Config(kwargs={'BLOCK_SIZE_M': 64}, num_warps=1, num_stages=6),
    triton.Config(kwargs={'BLOCK_SIZE_M': 128}, num_warps=4, num_stages=4),
    triton.Config(kwargs={'BLOCK_SIZE_M': 128}, num_warps=2, num_stages=3),
    triton.Config(kwargs={'BLOCK_SIZE_M': 128}, num_warps=2, num_stages=4),
    triton.Config(kwargs={'BLOCK_SIZE_M': 128}, num_warps=2, num_stages=5),
  ],
  key=['M', 'N', 'K']
)
@triton.jit
def weight_norm_kernel(
    weight_ptr, lora_B_ptr, lora_A_ptr, output_ptr,
    scaling, M, N, K,
    stride_wm, stride_wn, stride_bm, stride_bn, stride_am, stride_an, stride_om, stride_on,
    BLOCK_SIZE_M: tl.constexpr, BLOCK_SIZE_N: tl.constexpr, BLOCK_SIZE_K: tl.constexpr
):
    pid_m = tl.program_id(0)
    pid_n = tl.program_id(1)
    pid_k = tl.program_id(2)
    offm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
    offn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
    offk = pid_k * BLOCK_SIZE_K + tl.arange(0, BLOCK_SIZE_K)
    mask_m = offm < M
    mask_n = offn < N
    mask_k = offk < K

    weight_ptrs = weight_ptr + offm[:, None] * stride_wm + offn[None, :] * stride_wn
    lora_B_ptrs = lora_B_ptr + offm[:, None] * stride_bm + offk[None, :] * stride_bn
    lora_A_ptrs = lora_A_ptr + offk[:, None] * stride_am + offn[None, :] * stride_an

    weight = tl.load(weight_ptrs, mask=mask_m[:, None] & mask_n[None, :], other=0.0)
    lora_B = tl.load(lora_B_ptrs, mask=mask_m[:, None] & mask_k[None, :], other=0.0)
    lora_A = tl.load(lora_A_ptrs, mask=mask_k[:, None] & mask_n[None, :], other=0.0)

    lora_weight_row = tl.dot(lora_B, lora_A)
    weight += scaling * lora_weight_row
    squared_sum = tl.sum(weight * weight, axis=1)

    output_ptrs = output_ptr + (pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)) * stride_om + pid_n
    tl.store(output_ptrs, squared_sum, mask=mask_m)

def get_weight_norm_triton(weight, lora_B, lora_A, scaling):
    M, N = weight.shape
    K = lora_B.shape[1]

    output = torch.empty((M, triton.cdiv(N, BLOCK_SIZE_N)), device=weight.device, dtype=torch.float32)

    # minimum allowed by tl.dot()
    BLOCK_SIZE_K = max(16, triton.next_power_of_2(K))
    grid = lambda META: (
        triton.cdiv(M, META['BLOCK_SIZE_M']),
        triton.cdiv(N, BLOCK_SIZE_N),
        triton.cdiv(K, BLOCK_SIZE_K),
    )

    weight_norm_kernel[grid](
        weight, lora_B, lora_A, output,
        scaling, M, N, K,
        weight.stride(0), weight.stride(1),
        lora_B.stride(0), lora_B.stride(1),
        lora_A.stride(0), lora_A.stride(1),
        output.stride(0), output.stride(1),
        BLOCK_SIZE_N=BLOCK_SIZE_N,
        BLOCK_SIZE_K=BLOCK_SIZE_K
    )

    return torch.sqrt(torch.sum(output, dim=1)).to(weight.dtype)

def test_weight_norm(weight, lora_b, lora_a):
    torch.manual_seed(42)
    scaling = 0.1

    # Expected result using PyTorch
    lora_weight = lora_b.to(torch.float32) @ lora_a.to(torch.float32)
    expected_norm = torch.linalg.norm(weight.to(torch.float32) + scaling * lora_weight, dim=1).to(weight.dtype)

    # Triton result
    triton_norm = get_weight_norm_triton(weight, lora_b, lora_a, scaling)

    assert torch.allclose(expected_norm, triton_norm, atol=1e-5), "Test failed!"

    print("Test passed with all norms close enough!")

def run_tests():
    M, N, K = 8192, 28672, 12

    weight = torch.ones(M, N, dtype=torch.bfloat16, device='cuda')
    lora_b = torch.ones(M, K, dtype=torch.bfloat16, device='cuda')
    lora_a = torch.ones(K, N, dtype=torch.bfloat16, device='cuda')
    test_weight_norm(weight, lora_b, lora_a)

    weight = torch.zeros(M, N, dtype=torch.bfloat16, device='cuda')
    lora_b = torch.zeros(M, K, dtype=torch.bfloat16, device='cuda')
    lora_a = torch.zeros(K, N, dtype=torch.bfloat16, device='cuda')
    test_weight_norm(weight, lora_b, lora_a)

    weight = torch.randn(M, N, dtype=torch.bfloat16, device='cuda')
    lora_b = torch.randn(M, K, dtype=torch.bfloat16, device='cuda')
    lora_a = torch.randn(K, N, dtype=torch.bfloat16, device='cuda')
    test_weight_norm(weight, lora_b, lora_a)

Thanks! I'm using bf16. Any broadly Llama-like model should work, eg. I've run tests with Meta-Llama-3-8B and Mistral-7B-v0.1 (the original, non-MoE Mistral).

Thanks for the info. I still couldn't quite replicate the issue, would it be possible for you to share a code snippet? We already have some code that should take care of setting the right dtype here:
https://github.com/huggingface/peft/blob/77b7238b907c85f21ec756176351d85faf2aeaad/src/peft/tuners/lora/bnb.py#L468-L478
But apparently, you're not hitting that code. You mention QLoRa but the code you quote is from the normal LoRA layers, so I'm a bit confused.

Here's the Triton code I wrote. When I benchmarked it, it reduced total training time by 20% and reduced GPU VRAM consumption by 1,974 MB

That sounds pretty sweet. I have never worked with triton, so I'm not sure if we can just add this code and it'll work for everyone or if more work is necessary. If you're willing to contribute this but are unsure yourself, maybe I can find someone at HF with more expertise to take a look.

@rationalism gentle ping, do you have updates?

@BenjaminBossan Hi! Sorry, have been busy with work, but will look at this later this week

That would be great, thanks a lot!


Fix Animation on Safari

Fix Animation issue in the LESS file :)

Thanks. I will compile and push the changes soon.


UI_REACT: Selecting 'Reset All' when 'Mapping Details' is open leaves an empty pane.

Need to closeMappingDetails on reset all.

Fixed in the current master.


Feature: Support isort force_alphabetical_sort

The isort module provides a way to sort alphabetically using force_alphabetical_sort = true, which is very handy in large projects. I Plone we use use this option - it's part of the style-guide.
Ideally, the ruff setting would be something like this:
[tool.ruff.isort]
force-alphabetical-sort
This follows the isort option setting but with underscores replaced by dashes.

A similarly neat one is force-alphabetical-sort-within-sections: https://pycqa.github.io/isort/docs/configuration/options.html#force-alphabetical-sort-within-sections


[WIP] Add fake targets outline

This requires https://github.com/ionide/ionide-vscode-helpers/pull/29 (therefore the build is red)

(I'd recommend adding those vscode-helpers as git subtree instead of paket dependencies. I actually edited them and didn't realize they are not even under source control)


snowflake_external_table resource type not supported

Provider Version
0.23.2
Terraform Version
14.8
Describe the bug
I'm trying to use the external_table ressource with the example provided in the documentation. When I run terraform plan I get a message saying that The provider provider.snowflake does not support resource type "snowflake_external_table".
Expected behavior
I would expect Terraform to propose a plan with a create statement to add the external table.
Code samples and commands
Here is the configuration:
resource snowflake_external_table external_table {
  database = "db"
  schema   = "schema"
  name     = "external_table"
  comment  = "External table"

  column {
    name = "id"
    type = "int"
  }

  column {
    name = "data"
    type = "text"
  }
}

but when I execute terraform plan:
...
Terraform v0.14.8
Configuring remote state backend...
Initializing Terraform configuration...

Error: Invalid resource type

  on snowflake.tf line 220, in resource "snowflake_external_table" "external_table":
 220: resource snowflake_external_table external_table {

The provider provider.snowflake does not support resource type
"snowflake_external_table".

Sorry I misread my version number. I was on version 0.20.0. Switching to 0.23.2 everything works fine. I close the ticket and I take this opportunity to say a big thank you for the incredible work you provide on this provider.