Locale support, db connection closing.

Initial commit - locale support. See LocaleExample for a demonstration of usage.
Properly terminate database connection - to make sure any pending queries are executed, and to also avoid any sort of errors.

Notes: The strings directory structure is temporary, until an official plugin structure is determined. Put it there for now to keep it all in one place.
Relates to:

CraftTogether/Discussion-Issues#12.

Please only one change per pull request. :smile:

Also can we not have example classes with main(String[]) methods? If you are going to have an example please have it in the code where it would be used properly.

Aight, I'll get rid of that/delete the extra change this afternoon and reopen

Created a separate PR with the appropriate changes: #15. Locking conversation on this PR.


Create a Changelog for our sig-repos

Description
On our Guidline CheckBoard is sig-infra red marked. As a good example we should update our repo with a small
https://eclipse-tractusx.github.io/docs/release/trg-1/trg-1-3/
https://eclipse-tractusx.github.io/sig-release/
Question is what we want to share on the Page or just keep the information on the ChangeLog a bit up to date?
AC

[ ] green check mark

PR's

[x] https://github.com/eclipse-tractusx/sig-infra/pull/294
[x] https://github.com/eclipse-tractusx/sig-release/pull/199
[x] https://github.com/eclipse-tractusx/sig-security/pull/16

Visualisation

Reviewed, Approved and done


Accept Key as a string

Accept Key as a UTF-8 string for:

New-SSHSession
New-SFTPSession
Set-SCPFile
Set-SCPFolder

Code merged and added to v2.0 branch

there is something more... ;-)
 C:\> get-command -ParameterName KeyFile -Module posh-ssh

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Cmdlet          Get-SCPFile                                        1.7.6      posh-ssh
Cmdlet          Get-SCPFolder                                      1.7.6      posh-ssh
Cmdlet          New-SFTPSession                                    1.7.6      posh-ssh
Cmdlet          New-SSHSession                                     1.7.6      posh-ssh
Cmdlet          Set-SCPFile                                        1.7.6      posh-ssh
Cmdlet          Set-SCPFolder                                      1.7.6      posh-ssh

How do you specify a string instead of a path? Trying the v2.0 module and it still breaks with "New-SSHSession : Illegal characters in path."

use explicit convesion to string[]
[string[]]$key = 'here is your key string'
New-SSHSession -KeyFile $key


Source map names for context modules with parts that look like non-absolute paths are malformed

Bug report

What is the current behavior?
When a build includes context modules where the regular expression contains something which looks like a relative path, Webpack generates a source map with an incorrect source name that turns the regular expression into an absolute path.
This causes a mismatch in the getTaskForFile method of the SourceMapDevToolPlugin because the actual module id in the compilation, only has the leading path of the context module as absolute:

The consequence of which, is that rather than passing the module instance, it passes the (malformed) string module ID to ModuleFilenameHelpers.createFilename to generate the source map source name. This in turn causes the logic in createFileName to not be able to use ContextModule.readableIdentifier to create the friendly identifier that it should for context modules. Instead it will just use the mangled identifier as-is and push that into the source map.
The root of the path will be turned relative to the compiler context again; but the absolute path portion wrongly appended to the regex inside the identifier will remain and will leak into the source maps. This has the potential of changing content based on what machine actually ran the build from what directory structure, meaning caches being busted if content-hashes are added. Moreover; the fact that it leaks the directory structure of the build machine, means it also leaks the type of build agent and environment that was used - which might be information you don't want to slip to the public.
If the current behavior is a bug, please provide the steps to reproduce.
(note: still working on extracting a minimal reproducing case)

What is the expected behavior?
Webpack should not exhaustively attempt to make absolute paths from all pipe (|) separated components. It should only do so for the first component of a pipe-separated sequence.

Other relevant information:
webpack version:  5.74
Node.js version:
Operating System: Windows 10
Additional tools:

honestly, we already had a problem about it, but I can't find it, so let's mark it as bug

Thinking some more on this: most of the false positives here would be resolved if there was a better way to disambiguate context modules and treat their pipe-separated components differently.
Which to me raises the question: why are context modules building all this information into the module path, when factually it can't even be well-represented as part of an URI path to begin with. Regexes don't exactly lend themselves well to being represented as a path segment without judicious URI-encoding being applied. Worse consider these regexes are actually meant to partially match file paths, meaning slashes will be present.
Why then not go for the obvious:
Don't use the path to represent that information. Use the query or possibly even the fragment instead.
E.g. in case of the above case where things error:
./images/* ?sync|/[\\/](?:brands%7Cregion-flags)[\\/]/i or
./images/* #sync|/[\\/](?:brands%7Cregion-flags)[\\/]/i
This would have the added benefit of also the source map navigation in browser developer tools not mistaking the regex parts as part of a path anymore.

That totally worked.
My generated source map now contains what I would expect, with - of course - the regex-specific path separator characters all escaped:

I mean, it's ugly ... but I expect browser developer tools will at least show the URI-decoded result so it should be good.

Yeah, I think we can consider this, feel free to send a PR

I'm currently swamped in work leading up to a deadline, but I can take up creating a pull request out of this after.
Might take some time to get there though.

Yeah, feel free to ping me


investigate loosening the function bounds

Currently, Context::new takes an F: FnOnce() + Send + 'static. This disallows a lot of interesting use-cases. Internally, Registers::new only requires F: FnOnce().
It's not clear how Send should relate to custom threading runtimes, and this was mostly chosen as a safe bet. We might have to make decisions on how low-level this library should be exactly.
#6 covers the 'static bound.

Context::new doesn't take these bounds anymore.


Log stdOut from dotnet build along with stdErr in case of build failures

Fixes #260
cc @abpiskunov

:shipit:

Merged to dev https://github.com/aspnet/Scaffolding/commit/4ebb7b5f4c1922b47a050d0a327669c7284d8e75


Trying to make it more readable

Original document switch back and forth between digital and device twins, made it harder to read.

@leanlearns : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Minimum Viable Product

What will be your MVP? Which features are you going to validate? How?
Grading criteria (2 points): Describe what would be considered the Minimal Viable Product and how it will be tested (e.g., via implementation, prototyping, Wizard of Woz, etc.). Make clear what you are going to validate. The MVP should be coherent with the previous sections.

Completed and finalized in deliverable 2


Mapping array with objects confronting to protocol

I'm having some hard times finding right way to parse objects that confronts to a protocol.
In my model class I do have Array of items that implements Protocol FeedItemProtocol - but I want to be able to parse FeedItem class into it (which implements both FeedItemProtocol and Mappable).
I made it work via using local variable plus <- operator - but I wanted to make it more clean.
Is there a way to parse array of given elements via Mapper methods from Map object ?
Your model:
class FeedItemList: FeedItemListProtocol, Mappable {
    var total: Int!
    var items: Array<FeedItemProtocol>!

    required init?(map: Map) {

    }

    func mapping(map: Map) {
        total <- map["total"]

        var items: Array<FeedItem> = []
        items <- map["items"]
        self.items = items
    }
}

What you did:
I'm using this as workaround:
var items: Array<FeedItem> = []
items <- map["items"]
self.items = items

What you expected:
I exepected something like:
self.items = Mapper<FeedItem>().mapArray(JSONObject: map["items"].JSON)

Face the same. Any idea beside the workaround or a custom transform?


ng-show + select bug

https://plnkr.co/edit/ad6rqguvzZbrWLIGORhS?p=preview
  <button ng-click="toggle=!toggle">toggle: {{toggle}}</button>

  <div ng-show="toggle">
    <select chosen multiple ng-model="state" ng-options="s.id as s.title for s in states">
      <option value=""></option>
    </select>
  </div>

https://github.com/leocaseiro/angular-chosen/issues/145#issuecomment-194819950

ng-if is a nice workaround but i still prefer ng-show

Hi @jossef, let me know if you work around this issue. I've tried many times and I didn't succeed.
I'll be very happy looking at a PR for this feature.
Thanks


Issue with /docs/getting-started-guides/kubeadm/

This is a...

[ ] Feature Request
[x] Bug Report

Problem:
After Exeucting this command
kubeadm init --pod-network-cidr=<IP_ADDRESS>/16 --apiserver-advertise-address $MASTER_IP
it hangs at the following line
[apiclient] Created API client, waiting for the control plane to become ready
Below are the installed versions !! Is there any issue specific to kubeadm 1.6.1
Setting up kubernetes-cni (0.5.1-00) ...
Setting up socat (<IP_ADDRESS>-1) ...
Setting up kubelet (1.6.1-00) ...
Setting up kubectl (1.6.1-00) ...
Setting up kubeadm (1.6.1-00) ...
Setting up ubuntu-fan (0.9.2)
Proposed Solution:
Page to Update:
http://kubernetes.io/...

@vamsijakkula I would recommend seeking help in Kubernetes-users mailing list, IRC or Slack channels. This repository is for documentation issues.

Closing.


Optimize data copy

Data was copied byte by byte in two nested loops.
This changes the code to a single loop, using memcpy to copy one line of data at a time.
I observed a 2% performance improvement on the "decode 512x512 image" benchmark on a 2018 Macbook Pro.
This was branched from #126.

@fintelia : Ok, I updated indexing.


Reorganize scene tree dock buttons

This PR moves the "Expand/Collapse All" option back to right-click menu, after it was moved in #51413
The reason is that this option actually works on the selected node, so it can be used to expand or collapse branches selectively. Thus I renamed this option to "Expand/Collapse Branch" (the shortcut path stays the same), so it's more clear what it does.
For a more controversial change, I moved the "Instantiate Child Scene" button into the popup menu:

From my experience, it's the least used top button in the scene tree dock. And removing it allows to restore the lost space for search box.
Before:

After:

Where's the script button in "After"? 👀

Fixed the screenshot. I didn't have node selected.

I'm really not fond of hiding the instantiate button. And the remaining "Auto Expand to Selected" option is IMO awkward UX and not worth hiding a core feature such as scene instantiation for it.

That's probably personal preference, but I've never used those two buttons to the left of the filter. Since they are the first two options in the context menu, it's always quicker to use that (unlike the script button, which is somewhere in the middle). But if people do use those, makes sense to keep both "Add Node" and "Instantiate Child Scene".
Bikeshed: Maybe you should be able to select both nodes and scenes from the same list, and that would reduce the number of buttons just like that?

The instantiate button is only relevant when you are completely new to the engine. It's good for discoverability.
But we have so many ways to instantiate a scene, and the button is the least convenient one. The most popular is probably the drag and drop from file system, because you can search for the scene you want to use and then drag it precisely where you want it to be. After you get more familiar with scene instantiation, you rarely use the button.

the remaining "Auto Expand to Selected" option is IMO awkward UX and not worth hiding a core feature such as scene instantiation for it

Well, it was already discussed in #51413. There isn't really a better place for this option, because it's "global", not directly related to nodes in scene.

After you get more familiar with scene instantiation, you rarely use the button.

I only use the context menu option, which is the same as the button. You can filter it faster than anything, because it's a quick load dialog.

Personally I used the shortcut (Ctrl+Shift+A, even before it was officially added), but it was annoying that my scenes instantiate at the origin, so I implemented the "Instance Scene Here" popup. I use it almost exclusively now.

I only use the context menu option

Which is actually more convenient than the button, because you right-click at the node you want to use as parent. With button, you need to select the node and then press it, which involves more steps.

Well, to each their own, I've been using Godot for 6 years and I still use this button to instantiate scenes regularly :D
I expect that a significant proportion of Godot users might use it too and could be bothered by having it hidden (which is basically the same as removing it, if it's hidden in a "more" menu then it's even less accessible than the other options to instantiate scenes).

I am against hiding the Instantiate button, I use it all the time (I avoid ctrl-shortcuts often because of CTS). It also makes sense to have the Instantiate scene next to the Add Node button, or else why not remove both?
If anything, I would suggest hiding the Add Script button: it makes more sense to add a script by right-clicking on the node you want. Personally I just scroll down to the bottom of the Inspector and add a script from the empty slot.

Ok, I moved the instance button back:

So right now this PR only renames the Expand/Collapse All and moves it back to the RMB menu. I will open a new issue about the context menu.

Thanks!


spiral matrix solution added in c++

spiral matrix solution added in c++

Please add full program that includes the main function also.


Float64 msg Support

Hello,
I have a new feature suggestion for a new example.
Do you have any examples, using any RTOS, using topics with Float64 data?
I am using the TIMx encoder hardware from the STM32F407 on the Olimex board to populate a sensor_msgs/JointState.msg.
http://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/JointState.html
The  firmware encoder data is float, but the JointState,position and JointState.velocity are float64.
Casting doesn’t to work to assign the fw encoder data to the msg data. Any suggestions how to configure to solve?
I have an imu publisher working with float data without any problems.
Thanks for your time…

Hello! Based on the int32 demo, it should be something like:
...
#include <std_msgs/msg/float64.h>
...

rcl_publisher_t publisher;
std_msgs__msg__Float64 msg;
...
void appMain(void * arg)
{
...
	RCCHECK(rclc_node_init_default(&node, "freertos_float64_publisher", "", &support));

	// create publisher
	RCCHECK(rclc_publisher_init_default(
		&publisher,
		&node,
		ROSIDL_GET_MSG_TYPE_SUPPORT(std_msgs, msg, Float64),
		"freertos_float64_publisher"));

...

	msg.data = (double) 3.1416;

	while(1){
		RCSOFTCHECK(rcl_publish(&publisher, &msg, NULL));
		msg.data++;
	}

...
}

If you test it and it works fine, please do not hesitate to create a PR with this example for the rest of the community, thanks!

Why does casting does not work?
JointState.velocity is an array of  doubles: float64[], have you tried to initialize it like:
#define ARRAY_SIZE 1

sensor_msgs__msg__JointState msg;
msg.velocity.capacity = ARRAY_SIZE;
msg.velocity.size = ARRAY_SIZE;

// Using static memory
static double buffer[ARRAY_SIZE]
msg.velocity.data = buffer;

// Using dinamic memory
msg.velocity.data = (double*) malloc(ARRAY_SIZE*sizeof(double));

// Using just one data if ARRAY_SIZE == 1
double my_data = 3.14;
msg.velocity.data = &my_data;

Have you tried something like this?

I have not, I will give this a try and report back. Thank you for the guidance.

I verified both your static and dynamic examples compile and run. Thank you for the examples.


Add ability to disable an account

Accounts are disabled by setting a URL in their general settings. This
URL is then displayed in an iframe when they login and App Inventor is
not available to them.
Change-Id: Iebf1c268358608d29a8e000935d0b9e4458f77f3

@halatmit @wolberd Here is the account deactivation code...

@jisqyv @wolberd
How do we actually disable and reenable an account?

@halatmit I didn't write the tool to do that yet (I tested it by directly editing the UserData object in the data store)

LGTM


smartdc does not like running on node 0.12

⟩ sdc-listpackages
timers.js:156
    throw new TypeError('msecs must be a number');
          ^
TypeError: msecs must be a number
    at Object.exports.enroll (timers.js:156:11)
    at Socket.setTimeout (net.js:329:12)
    at SSHAgentClient._request (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/ssh-agent/lib/ssh_agent_client.js:290:10)
    at SSHAgentClient.requestIdentities (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/ssh-agent/lib/ssh_agent_client.js:185:15)
    at sshAgentGetKey (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:114:12)
    at Object.checkAgentForKey [as func] (/usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/lib/index.js:341:17)
    at /usr/local/lib/node_modules/smartdc/node_modules/smartdc-auth/node_modules/vasync/lib/vasync.js:190:20
    at process._tickCallback (node.js:355:11)
    at Function.Module.runMain (module.js:503:11)
    at startup (node.js:129:16)

There already is a minor upgrade for the ssh-agent dep that looks like it would address this.

Open as: https://smartos.org/bugview/PUBAPI-1053
Thanks

if I just up the version of ssh-agent to 0.2.3 it blows up in restify:
⟩ sdc-listpackages
_http_client.js:75
    throw new Error('Protocol "' + protocol + '" not supported. ' +
          ^
Error: Protocol "https:" not supported. Expected "http:".
    at new ClientRequest (_http_client.js:75:11)
    at Object.exports.request (http.js:49:10)
    at Object.exports.request (https.js:136:15)
    at rawRequest (/usr/local/lib/node_modules/smartdc/node_modules/restify/lib/clients/http_client.js:121:25)
    at FunctionCall.doCall_ (/usr/local/lib/node_modules/smartdc/node_modules/restify/node_modules/backoff/lib/function_call.js:150:20)
    at FunctionCall.start (/usr/local/lib/node_modules/smartdc/node_modules/restify/node_modules/backoff/lib/function_call.js:139:10)
    at JsonClient.request (/usr/local/lib/node_modules/smartdc/node_modules/restify/lib/clients/http_client.js:382:14)
    at JsonClient.read (/usr/local/lib/node_modules/smartdc/node_modules/restify/lib/clients/string_client.js:70:14)
    at JsonClient.get (/usr/local/lib/node_modules/smartdc/node_modules/restify/lib/clients/http_client.js:299:22)
    at CloudAPI._get (/usr/local/lib/node_modules/smartdc/lib/cloudapi.js:3951:24)

But it looks like the more recent restify versions work fine with 0.12 - so bumping the version of ssh-agent and restify makes smartdc usable on node 0.12

Dependencies bumped to work with node v0.12.0 on commit 7ad33b1e91c98e1225da74a7a779a33e434b64e7 and<EMAIL_ADDRESS>module published.
Thanks for reporting breakage!


chore: adapt to multiple goal linter 6

Another halfway split of #12381 on the way to #12352.

Filed #12850 and #12851 with the style tweaks I suggested. Reviews welcome - since I wrote that code, you should have no conflict of interest :-)

Michael, thank you very much for the continuing thorough reviews!

Thank you for writing this linter, Damiano - I would very much like to see it land!

bors merge
bors d+


operands error

@JusperLee Please tell the solution for that last issue as I'm very new to this?

I think you can try to solve this problem yourself, because the length of the data we use will be inconsistent.  The above error is prompted because the array size is inconsistent.  Since this is not a problem with the code, I hope you can patiently check the problem and solve it. Thank you

@JusperLee So is this is the problem because of the input size?


Cannot export TailwindComponent

Cannot export a component
export const MyContainer = tw.div`flex`

Getting this error:
Exported variable 'Container' has or is using name 'TailwindComponent' from external module ".../node_modules/tailwind-styled-components/dist/tailwind" but cannot be named.ts(4023)

Suggestion:
./src/tailwind.tsx
export interface TailwindComponent

I'm getting this too, following for updates

Hello guys,
I could not reproduce on my test repo.
Could you share your typescript version and your tsconfig.js ?
Many thanks

Okay I was able to reproduce, this was indeed due to not exporting the interface TailwindComponent.
It's now fixed in the version 2.1.1
thanks for reporting this issue!


Add pagination to Listings API

Part of CityOfDetroit/affordable-housing-app#76
We need to add pagination to the listings API so we can don't just have a single page with all listings.
The partner portal currently has pagination for the lists of applications and the early work for listings, so we might be able to use some of that logic. This might be a starting point
This is Done when we have a listings API with pagination.

Thanks to Abbie for getting https://github.com/CityOfDetroit/bloom/pull/133 committed!
main now has the backend pagination, but it looks like there's a small bug in the totalPages count:

I'll fix this and then close this.


networking/kvm: add flannel default gateway parsing

Continuation of #2769.

@steveeJ Thank you! I closed #2769 to let this PR be merged.

LGTM, thx @steveeJ

I see #2769 originally had several concerns from multiple sides. Does this cover everything there?

@lucab yes, it's #2769 with @steveeJ fixes, AFAIK it's all good now.

@alban how can we retrigger jenkins here?

you just did?

@iaguis hmm, the jenkins results things were tricking me, will they be overwritten when jenkins finishes? (notice they're dead links now)

@iaguis plais halp

	rkt_caps_test.go:601: Expected "net_admin=disabled" but not found: image: using image from file /home/runner/rkt/builds/build-rkt-kvm/build-rkt-1.8.0+git/tmp/functional/stage1-kvm.aci

I want to merge this but I'm not sure which jenkins tests are important to pass. However, I checked the failed jobs and can't find a relation to this change.

@iaguis I inspected the tests and everything LGTM, am I missing anything?

since we have several LGTM and "it's all good" I think we can merge it.

@tmrts

@iaguis I inspected the tests and everything LGTM, am I missing anything?

I retriggered the tests because 2 of the jenkins tests had previously failed. It was Jenkins (debian-8, coreos flavor) Success! and another one I currently don't remember or see in the checklist.


Add alternativeCallerId to Adpaters and state for outbound PSTN

What

Introduce alternativeCallerId to the CallAdapter and Stateful layer.
Why

Adding this property to these areas allows for Contoso to provide the alternativeCallerId at adapter creation so that they can leverage it from the adapter state provider. For us this allows us to use this number in our composites to introduce adding participants in the composite using this phone number.
Note: The phone number is stored in the callAgentState because the phone number is dependent on the callAgent being created to be used. When Contoso asks for the state of the adapter these properties get surfaced as well.

https://skype.visualstudio.com/SPOOL/_workitems/edit/2920717
How Tested

Built locally and ran Calling and CallWithChat sample applications to make sure that the new property didn't break anything.

Wondering if we should have the alternativeCallerId in the CallAgentState instead since it's similar to the display name?

im torn on this. I like the idea of it living in this state since the number should also be dependent on the callAgent existing, however, this callerId does behave like a identifier for the ACS user calling the PSTN user. Any thoughts here on your side?

@dmceachernmsft Possible to add Unit test to this PR to ensure that the callerId you are setting as a function argument is correctly reflected in the state?

Thanks for getting consensus on where to place the field.


add proof for proving n is a product of two safe primes

Signed-off-by: CamP<EMAIL_ADDRESS>

Don't know if I'll get another chance to look at this soon, but fwiw, tectonic bailed on an undefined control sequence. I don't see anything obviously wrong... we don't have a handy script for this like we do in anoncreds 2 so could be user error on my part.
Kinda curious why the update for anoncreds1.. figured we'd just see more work on v2.


Typescript issues when no 'dom' in tsconfig

Hello all!
I'm using jdenticon on a backend node server and therefore have no need for the "dom" inside my "lib" in the tsconfig.json
The error I'm getting is
node_modules/jdenticon/types/module.d.ts:116:43 - error TS2304: Cannot find name 'Element'.

116 export function update(elementOrSelector: Element | string, hashOrValue?: any, config?: JdenticonConfig | number): void;
                                              ~~~~~~~

node_modules/jdenticon/types/module.d.ts:132:49 - error TS2304: Cannot find name 'Element'.

132 export function updateCanvas(elementOrSelector: Element | string, hashOrValue?: any, config?: JdenticonConfig | number): void;
                                                    ~~~~~~~

node_modules/jdenticon/types/module.d.ts:148:46 - error TS2304: Cannot find name 'Element'.

148 export function updateSvg(elementOrSelector: Element | string, hashOrValue?: any, config?: JdenticonConfig | number): void;

Looking at the module.d.ts I see the three functions, update, updateCanvas and updateSvg and they all have the comments

This method is only available in the browser. Calling this method on Node.js will throw an error.

Is it possible to make these optional or something?

Thanks for reporting this issue! The module typings are shared between the browser and Node implementation of Jdenticon so unfortunately those functions will need to remain in the typings to support people importing the module into their web projects.
I researched how people have solved similar issues on DefinitelyTyped, and I found mainly two approaches:

Add a dummy interface for Element. If the consumer of Jdenticon has dom added to lib in their tsconfig.json, the interface will be merged with the real one.
Add /// <reference lib="dom"/> into the typings, but it has the side-effect of exposing the entire dom lib to the consumer of Jdenticon.

I just released a patch to NPM containing the first approach. Just update to version 3.0.1 and the compilation errors should hopefully go away.

@dmester Woah! You're awesome. Thanks for explaining the process to me as well!!


Added VS Code support to UE4's gitignore

Simple, safe change.
VS Code creates some files just like its big brother. These files can be safely ignored as they are user specific and can be generated by the engine. It's beneficial to ignore these files as they contain lots of absolute directory paths that will likely be different across computers, meaning that whenever someone loads up the project, VS code will "fix" them, causing a bunch of unnecessary changes.
I also changed "VS 2015" to just "VS" since it applies to all versions of VS supported by UE4.
Reasons for making this change:

Increase feature parity / consistency between VS and VS Code
Save others from having to make this change every time

I stand by what was mentioned already. We have an existing Global\VisualStudioCode.gitignore that covers more than just this rule.


De-SQL Tests

Changes test suite SQL to AR queries, except QueryTest.
(QueryTest is an exception because the tested Query methods generate SQL fragments;
the tests check whether those fragments comprise the correct SQL.)

Coverage remained the same at 92.207% when pulling 98f79eb7deb91fcaffe46ab60aa557026b5b3a75 on jdc-un_sql-tests into 3abee88c56d31b4249836266ae0f89cd19fa0aab on master.


[WIP] Refactoring

simplifcation, this is wip

@zaccolley should I wait till this is finished before doing a release?

I'll get this done this week


First query after reindexing takes ~30s

We need to warm up internal elasticsearch cache once everything has been imported. This can also serve as a quick test that everything works.

Is the fix still working/needed? The queries seem to fail instantly: https://github.com/NixOS/nixos-search/runs/4825472635?check_suite_focus=true#step:9:1

Turns out they were using the wrong index names. Fixing in https://github.com/NixOS/nixos-search/pull/415


[Bug] Balloon mushroom stem doesn't connect

Balloon mushroom stem placement is broken, can't place upwards and doesn't connect

Probably need to add a log rotation property

Fixed in 20.4.20


pseudo behavior test: element deletion tested properly

The timing was wrong. Element deletion was not actually being tested. The
fix is to strip away all the "smart" handling of DOM mutations. Just mark
every rule always.

:+1:


Error resolving dependencies with variables and parent POMs - here: ${pom.version}

Hello and ❤ to you for a very great piece of software!
We are using it since a few days and it really works well for our use case, but we are discovering an issue which obviously seems to have issues resolving dependencies with variables:
Failed to collect dependencies at at.compax.vn.interfaces:shop-interface:jar:49.0.0-SNAPSHOT -> at.compax.vn:commons:jar:49.0.0-SNAPSHOT -> com.mockrunner:mockrunner-all:jar:1.1.2 -> com.mockrunner:mockrunner-tag:jar:1.1.2 -> org.apache.struts:struts-taglib:jar:1.3.10 -> org.apache.struts:struts-core:jar:${pom.version}: Failed to read artifact descriptor for org.apache.struts:struts-core:jar:${pom.version}: The following artifacts could not be resolved: org.apache.struts:struts-core:pom:${pom.version} (absent): Could not transfer artifact org.apache.struts:struts-core:pom:${pom.version} from/to nexus

We have already investigated that further and we have realized that this is happening where the pom.xml is not specifying a version, but inherit it from a parent pom:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <parent>
    <groupId>org.apache.struts</groupId>
    <artifactId>struts-parent</artifactId>
    <version>1.3.10</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.apache.struts</groupId>
  <artifactId>struts-taglib</artifactId>
  <packaging>jar</packaging>
  <name>Struts Taglib</name>
  <url>http://struts.apache.org</url>

Is this a known issue or can this be resolved somehow?
Thanks and best regards,
Thorsten

Edit:
Version of mvnd is
Apache Maven Daemon (mvnd) 1.0-m6 linux-amd64 native client (47d4c6b9c399079d91312bb8ae35d1d76e7fd97a)
Terminal: org.jline.terminal.impl.PosixSysTerminal with pty org.jline.terminal.impl.jansi.linux.LinuxNativePty
Apache Maven 4.0.0-alpha-5 (26d10a4bf9a2df75feef60da01d8706f2bf77a47)
Maven home: /maven-mvnd-1.0-m6-m40-linux-amd64/mvn
Java version: 1.8.0_362, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-100-generic", arch: "amd64", family: "unix"

Ok, after investigating further it looks like this could be a maven 4.0.0-alpha6 problem, We are trying to switch to maven 3.9 to prove that.

@tmuerell do you have a reproducer ?

@tmuerell it sounds like pom.version is supposed to be a built-in Maven property? I must say I never heart of such one. Could you perhaps try to replace it with a more standard project.version ?

Ok, I think we can close this here, it's not related to mvnd at all, but I can foresee bigger to biggest issues when maven 4 will be released ;-)


lib/meta: add platform constraints algebra

Previously, we were able to express platform constraints in packages using e.g.
meta.platforms = with lib.platforms; linux ++ darwin;

These platform checks only worked on a coarse level using lists of pre-defined platform strings. It was not possible to express that only e.g. 64bit platforms or that only GNU libcs are supported previously.
This introduces an algebra based on platform patterns which allows you to express arbitrarily complex platform constraints using simple expressions without needing to rely on stdenv.*Platform.is* bools.
Once this is in wide-spread use, it can be used for accurate rendering of platform support in manuals and search sites.
It also removes the need for dependency on a package's stdenv for expressing which platforms are broken as this can now be expressed using platform constraints instead. This makes progress of potentially evaluating a package's meta information independently of the package or its inputs.
Things we still need to discuss:

[ ] The actual integration in platforms/badPlatforms
[ ] Performance testing (this is on the hot path!)
[ ] How/when to coordinate this with https://search.nixos.org?
[ ] Docs and guidelines

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

The commit I just pushed will be made redundant in a follow-up but I want to see ofBorg eval perf because @adisbladis suggested that even seemingly tiny changes can have an immense impact there because this is on the hot path for all packages.

They have also provided an alternative implementation that has much nicer properties but trades off the ability to introspect.
https://github.com/Atemu/nixpkgs/pull/1

Hm, I have also realised that their approach trades off the ability to insert arbitrary custom constraints too as all constraints must be wrapped inside of a function that takes the platform as a parameter.
This is an admittedly fringe case and could be remedied using a simple helper function around the custom constraint that applies the same wrapping.

I'm far from an expert in interpreting these stats but the eval perf report appears to show a slight increase in function activity and thunks but a slight decrease in GC heap size?
Anyhow, that's with the functionality not really in use by any packages yet. Real-world testing would likely prove to be difficult to realise. Perhaps we could migrate a critical mass of packages via automation somehow and extrapolate the effect.


Read metadata db from file

load_Corpus_metadata takes a long time, here we can load the medatata db from a previously saved pickle

pickle reads in faster, and you can't accidentally change the delimiter by opening it in libre office

could always throw in a try: read_csv except: read_pickle if you want to use csv.


Cannot build libindy-objc with iOS 14

When I tried to import Indy throws an compiler error like below;
import Indy

Module compiled with Swift 4.2.1 cannot be imported by the Swift 5.3 compiler: ....Pods/libindy-objc/libindy-objc/Indy.framework/Modules/Indy.swiftmodule/arm.swiftmodule

Will a new version be released?
Also I tried with 'pod libindy' but libindy has not enough documentation for ios. How can I use libindy in my iOS project? How to import 'libindy.a'?

@berkakkerman just look at this issue, finally got it for swift 5
https://github.com/AbsaOSS/rn-indy-sdk/issues/20#issuecomment-725611052


wireshark: 3.4.0 -> 3.4.2 (security)

Motivation for this change
https://www.wireshark.org/docs/relnotes/wireshark-3.4.1.html
https://www.wireshark.org/docs/relnotes/wireshark-3.4.2.html
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS

[ ] Tested execution of all binary files (usually in ./result/bin/)
Tested wireshark-qt.
[x] Fits CONTRIBUTING.md.

@TredwellGit should we backport this?

Yes.

Also consider backporting https://github.com/NixOS/nixpkgs/commit/251349dac30e4de0d9d0c982ccbe4e12c3217ccb.

Ported to stable as 3160ac620e2989599b8042032894dc8542759c3a, 0a94f278666871d87dc052c99fb6da2215568597, bf05444d01eae9ac965d90bcbead661f29a28c74.


”Dubova” is listed in the drop-down menu

Description:
"Dubova" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Dubova" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Dubova, MEHEDINȚI".


Please invite me to the Community Organization

Name
Codersol
Discord Username (if applicable) E.g John#412
No response
Questions & suggestions?
No response

Already a member. Closing this as duplicate.
Welcome to the community, @Codersol


version

RuntimeError: probability tensor contains either inf, nan or element < 0
There was an error during the evaluation. I felt that there was a problem with the installed package version. Could you provide the version of your installed package?

Please try building the environment with the following order:

Set up conda environment.

conda create -n ll3da python=3.8
conda activate ll3da

Install PyTorch:

pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

Install other packages:

pip install h5py scipy cython plyfile 'trimesh>=2.35.39,<2.35.40' transformers>=4.37.0

build pointnet++ and giou support:

cd third_party/pointnet2
python setup.py install
cd ../../utils
python cython_compile.py build_ext --inplace

If the issue still appears, please let me know.

conda activate ll3da

Thanks, I will try. I have a question, when I look at the script, the evaluation is also based on only the trained checkpoint  (--test_ckpt ./ckpts/opt-1.3b/ll3da-generalist/checkpoint.pth ), the tuned checkpoints are not used. What is the use of the tuned checkpoint?

We train our model on the combination of Nr3D and ScanRefer for describing objects. However, these two datasets are annotated in different styles. Thus it is required to tune on each dataset, respectively.

I'm sorry, I don't understand. Can you tell me more about it? Like how it's done. I don't see any difference with scanqa.

Since LL3DA is a 3D generalist, it can distinguish different tasks given human interactions. You can directly evaluate on ScanQA with the generalist checkpoint, or try fine-tuning it.

Since LL3DA is a 3D generalist, it can distinguish different tasks given human interactions. You can directly evaluate on ScanQA with the generalist checkpoint, or try fine-tuning it.

----------------------Evaluation-----------------------
INFO<EMAIL_ADDRESS>matched proposals: [1525 / 2068],
[BLEU-1] Mean: 0.6246, Max: 1.0000, Min: 0.0000
[BLEU-2] Mean: 0.5269, Max: 1.0000, Min: 0.0000
[BLEU-3] Mean: 0.4311, Max: 1.0000, Min: 0.0000
[BLEU-4] Mean: 0.3519, Max: 1.0000, Min: 0.0000
[CIDEr] Mean: 0.5911, Max: 5.4976, Min: 0.0000
[ROUGE-L] Mean: 0.5407, Max: 1.0000, Min: 0.1015
[METEOR] Mean: 0.2519, Max: 1.0000, Min: 0.0448
When I directly evaluate on ScanQA with the generalist checkpoint, I got the above result. I found that the result of<EMAIL_ADDRESS>is very different from that in the paper, and other metrics are similar to those in the paper. Why is this?

Hi, I tried the train.generalist.sh, but I can't reproduce a close performance as reported in the paper. the only change is the 24 batch size instead of 4 to speedup training
here are the eval logs on ScanQA, Nr3D and ScanRefer, at 20th epoch
----------------------Evaluation-----------------------
[BLEU-1] Mean: 0.3028, Max: 1.0000, Min: 0.0000
[BLEU-2] Mean: 0.1904, Max: 1.0000, Min: 0.0000
[BLEU-3] Mean: 0.1283, Max: 1.0000, Min: 0.0000
[BLEU-4] Mean: 0.0875, Max: 1.0000, Min: 0.0000
[CIDEr] Mean: 0.4818, Max: 8.0511, Min: 0.0000
[ROUGE-L] Mean: 0.2636, Max: 1.0000, Min: 0.0000
[METEOR] Mean: 0.1058, Max: 1.0000, Min: 0.0000
Evaluate [19/32]; Batch [0/1]; Evaluating on iter: 12999; Iter time 261.13; Mem 70618.97MB
----------------------Evaluation-----------------------
INFO<EMAIL_ADDRESS>matched proposals: [712 / 1214],
[BLEU-1] Mean: 0.5626, Max: 1.0000, Min: 0.0006
[BLEU-2] Mean: 0.3753, Max: 0.8165, Min: 0.0000
[BLEU-3] Mean: 0.2223, Max: 0.6583, Min: 0.0000
[BLEU-4] Mean: 0.1339, Max: 0.5756, Min: 0.0000
[CIDEr] Mean: 0.0945, Max: 1.2465, Min: 0.0000
[ROUGE-L] Mean: 0.4495, Max: 0.8299, Min: 0.1843
[METEOR] Mean: 0.2157, Max: 0.5162, Min: 0.0783
Evaluate [19/32]; Batch [0/1]; Evaluating on iter: 12999; Iter time 262.18; Mem 70618.97MB
----------------------Evaluation-----------------------
INFO<EMAIL_ADDRESS>matched proposals: [1506 / 2068],
[BLEU-1] Mean: 0.6056, Max: 1.0000, Min: 0.0000
[BLEU-2] Mean: 0.4881, Max: 1.0000, Min: 0.0000
[BLEU-3] Mean: 0.3775, Max: 0.9410, Min: 0.0000
[BLEU-4] Mean: 0.2926, Max: 0.8654, Min: 0.0000
[CIDEr] Mean: 0.3024, Max: 3.1209, Min: 0.0000
[ROUGE-L] Mean: 0.4990, Max: 0.9412, Min: 0.1015
[METEOR] Mean: 0.2349, Max: 0.5416, Min: 0.0448
the training log is here
it would be nice if the pretrained checkpoints/pre-processed point clouds can be downloaded to minimize the randomness

The actual batch size of our original configure is 4 x 8 gpus = 24 per iteration. To re-produce our results, we encourage you to train with the exact same config as we provided.
Please track the training process on the number of iterations rather than epoch numbers. Based on our experience, training LL3DA with only 13k iterations is far from convergence.
We are actively working on packing the pre-trained weights, please stay tuned.

The actual batch size of our original configure is 4 x 8 gpus = 32 per iteration. To re-produce our results, we encourage you to train with the exact same config as we provided.
Please track the training process on the number of iterations rather than epoch numbers. Based on our experience, training LL3DA with only 13k iterations is far from convergence.
We are actively working on packing the pre-trained weights, please stay tuned.

when I use actual batch size of the original configure is 4 x 8 gpus = 32 per iteration,  I find the training log:
Epoch [2/32]; Iter [11990/127936]; Loss 1.51; LR 9.79e-05; Iter time 0.46; ETA 14:48:34; Mem 18615.49MB
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Loss in not finite. Skip this training step.
Epoch [3/32]; Iter [12000/127936]; Loss 1.51; LR 9.79e-05; Iter time 0.48; ETA 15:23:59; Mem 18615.49MB
What happened?

Because of the mixed precision training, the training process might not be that stable. As long as the model is still training, you can just ignore this message.


OpenZeppeling 0.2.0 account contract have an issue with signature verification.

I have tried running the oz-account-test.ts test of starknet-plugin-exaple and ran into the following issue:

  2) OpenZeppelin account
       should handle multiple invokes through an account:
     Error: Transaction rejected. Error message:

Error at pc=0:133:
Got an exception while executing a hint.
Cairo traceback (most recent call last):
Unknown location (pc=0:746)
Unknown location (pc=0:682)
Error message: Account: invalid signature
Unknown location (pc=0:347)
Unknown location (pc=0:321)

Traceback (most recent call last):
  File "<hint14>", line 1, in <module>
  File "/app/src/starkware/starknet/services/batcher/starknet_batcher_venv-site/starkware/cairo/lang/builtins/signature/signature_builtin_runner.py", line 94, in add_signature
    ), f"Signature hint must point to the signature builtin segment, not {addr}."
AssertionError: Signature hint must point to the signature builtin segment, not 14:0.

The cause is probably this issue #378 of OpenZeppelin with contracts of version 0.2.0 on Goerli testnet.

So it happens only on alpha-goerli? Not on Devnet? Should it be fixed with the newly released OZ v0.2.1?
I still don't really understand what's the issue here.

That's my understanding, yes.
Only happens on alpha-goerli.
A fix was already merged, I assume it will be available in the next version.
This can be used to track the fix here.

I'd recommend fixing this to address https://github.com/OpenZeppelin/cairo-contracts/issues/344 too


androidenv.emulateApp: Could not connect to proxy

Describe the bug
When using the androidenv.emulateApp function I can't set an http proxy, neither via the gui nor via androidEmulatorFlags = "-http-proxy http://<IP_ADDRESS>:8080".
There is a corrosponding log entry:
DEBUG   | setting up http proxy: proxy= port=8080
proxy_resolve_server: can't resolve proxy server name ''
WARNING | Could not connect to proxy at :8080: No such file or directory !
WARNING | Proxy will be ignored !

Steps To Reproduce
Steps to reproduce the behavior:
with import (builtins.fetchGit {
  url = "https://github.com/nixos/nixpkgs.git";
  rev = "5b5e6f990070ec0c5c343ff9160554866ecd23c2";
}) {
  config.android_sdk.accept_license = true;
  config.allowUnfree = true;
};
  androidenv.emulateApp {
    name = "emulator";
    abiVersion = "x86_64";
    androidEmulatorFlags = "-http-proxy http://<IP_ADDRESS>:8080 -debug-proxy";
  }

Expected behavior
The http proxy should work when setting it to a valid ip/port
Screenshots
Proxy settings can be found here.

Additional context
Add any other context about the problem here.
Notify maintainers
@hadilq
Metadata
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.8, NixOS, 23.05 (Stoat), 23.05.20240103.70bdade`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.6`
 - channels(root): `"nixos"`
 - channels(user): `""`
 - nixpkgs: `/etc/channels/nixpkgs`

Add a :+1: reaction to issues you find important.

Issue seems to be upstream: https://issuetracker.google.com/issues/276013088


[FEATURE] Add Experimental badges and messaging

Is your feature request related to a problem?
We will first offer this feature as an Experimental Feature to gather users feedback and improvements. We will need to visually inform users this is Experimental and not ready for production use. We should offer a way for users to leave feedback as they try out the feature.
What solution would you like?
We will display an Experimental badge in selected areas, when users click on the badge, a popover will show a message:
OpenSearch Flow is experimental and should not be used in a production environment.. Clicking on the button Provide feedback on Github will open a new tab to this page, so users can provide feedback by creating an issue.

Display the badge AND popover in the following 3 areas:
For the badge, use OuiBetaBadge with the beaker icon

Search overview page (place badge on the OpenSearch Flow tile)

Project list/create page (place badge next to the header)

Inside a workflow (place badge next to the action buttons)

Display only the beaker icon in the navigation (no badge component)

When users hover over the OpenSearch Flow item, we should update the native browser label to
OpenSearch Flow – Experimental feature

I've encountered several issues with core OSD flexibility to enable these flags in many different places:

List/header page (useNewHomePage: true ONLY). The new UI header framework does not allow for an extra icon or React components to the right of the actions list.
Inside a workflow page (useNewHomePage: true ONLY). The new UI header framework does not allow for an extra icon or React components to the right of the actions list.
In the navigation bar (both new AND old home page layouts). The navigation framework does not allow for icons to be visible, let alone extra functionality such as hovering.

It looks like all of these will require some sort of framework enhancements in core OSD & the owners of the new header layouts to expose this sort of functionality.
To recap:

[ ] overview page complete
[ ] workflows page (old UI) complete
[ ] workflows page (new UI) not possible
[ ] workflow details page (old UI) not complete - we can add this next to the workflow title if we want?
[ ] workflow details page (new UI) not possible
[ ] navigation (old UI) not possible
[ ] navigation (new UI) not possible

@kamingleung  In the meantime I will open a PR with the changes possible to make today. Lmk your thoughts/priority on the other changes. Perhaps we can find a middleground and expose some labels somewhere else on the new UI, and I can open tracking issues in core OSD to address the gaps mentioned.

@AMoo-Miki
Do we supporting adding badging to the new page header, like this?

Are we able to add badging to the new Application Header? Perhaps we can add sections similar to Visualize?
Proposed Fractal Application Header with Experimental Badge:

The new Visualize:

cc. @lauralexis @ virajsanghvi – We may want to consider supporting similar patterns within our OUI components in the future.

I think we need to add a prop to page header for isExperimental=true which allows you to pass the right values.

Latest update:

[x]  overview page complete https://github.com/opensearch-project/dashboards-flow-framework/pull/472
[x]  workflows page (old UI) complete https://github.com/opensearch-project/dashboards-flow-framework/pull/472
[x]  workflows page (new UI) complete #496
[x]  workflow details page (old UI) complete #485
[x]  workflow details page (new UI) complete #485
[ ]  navigation (old UI) icon + tooltips not possible
[ ]  navigation (new UI) icon + tooltips not possible

As discussed, navigation is not possible with the current framework. Please reopen or add a standalone issue to close on this particular part.

As discussed, navigation is not possible with the current framework. Please reopen or add a standalone issue to close on this particular part.

Got it. LGTM on the latest updates. This is a temporary solution. We will open a new issue once we have the components with props to support experimental badges.


Split TXN and SSN user args

As part of the workup to fixing #2388 we want to split the TXN and SSN args. These are manipulated separated but reserved in parallel.

Fixed by #2905.


[$500] Android - Sign-in page flickers after using autofill

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.52-3
Reproducible in staging?: Y
Reproducible in production?: N
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Issue reported by: Applause - Internal Team
Action Performed:
precondition: user has saved credentials on the device

Open the app
Tap the email address field
Select any saved credentials

Expected Result:
The credentials are pasted in the field with no flickering
Actual Result:
The page flickers after pasting the credentials
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/07d43ce3-93b7-4b8b-813c-9e3524f288d9
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01c54bb0a7e9ff28d4
Upwork Job ID:<PHONE_NUMBER>693136896
Last Price Increase: 2024-03-15

We think that this bug might be related to #vip-vsp
CC @quinthar

@marcaaron  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

Production:
https://github.com/Expensify/App/assets/78819774/c79abb06-dcda-4b15-9315-48aca33e4f38

Proposal for Addressing Sign-in Page Flickering Issue
Problem Statement
We have encountered a flickering issue on the Android sign-in page when users utilize the autofill functionality to input their credentials. This behavior leads to a suboptimal user experience, as the page's elements rapidly re-render in quick succession, creating a visible flickering effect.
Root Cause Analysis
After a thorough investigation, the root cause of the flickering issue appears to be tied to two main factors within our  BaseLoginForm  component:

Rapid State Updates on Input Change: The component's state is updated every time there's a change in the input field. Autofill triggers these updates almost instantaneously for each piece of autofilled information, leading to multiple state updates and consequent re-renders in a very short time frame.
Relevant code snippet:
const onTextInput = useCallback(
    (text: string) => {
        setLogin(text);
        if (firstBlurred.current) {
            validate(text);
        }
        ...
    },
    [account, closeAccount, input, setLogin, validate],
);

Immediate Validation on Input Change and Blur: Our current logic executes validation on every input change after the first blur event. This validation, especially following an autofill action, triggers additional state updates and potentially more re-renders, exacerbating the flickering effect.
Relevant code snippet:

    if (firstBlurred.current) {
        validate(text);
    }

These factors combined result in a visible flickering effect due to the rapid and successive updates and re-renders of the component's state.
Proposed Changes
To mitigate the flickering issue and enhance the user experience on the sign-in page, the following changes are proposed:

Debounce Input Change Handler: Implement a debounce mechanism for the input change handler. This will ensure that state updates (and subsequent validation) are not triggered by every single character change but rather after the user has stopped typing for a predefined interval. This approach will significantly reduce the number of state updates and re-renders, especially during autofill.
Proposed code change:

// Utilize lodash's debounce or a similar utility function
const debouncedOnTextInput = useCallback(debounce((text) => {
    setLogin(text);
    if (firstBlurred.current) {
        validate(text);
    }
}, 300), [setLogin, validate]); // 300ms debounce interval

Optimize Validation Logic: Review and optimize the validation logic to ensure it is as efficient as possible. Consider delaying validation until a user action indicates that autofill has completed (e.g., user taps the submit button), rather than validating immediately after input changes.
Proposed adjustment:
// Adjust validation trigger to occur on form submission rather than immediate input change
const onFormSubmit = () => {
    if (validate(login)) {
        // Proceed with form submission logic
    }
};

Conclusion
By implementing the proposed changes, we aim to eliminate the flickering issue on the Android sign-in page, thereby improving the overall user experience. These adjustments will help ensure that the component handles autofill and user inputs more gracefully, with fewer unnecessary re-renders and a smoother visual experience

@omarnagy91 Thanks for your proposal. Can you also share a video of these changes in action?

Not overdue. @omarnagy91 bump!

Not able to reproduce!
I tried both on a real android device (OnePlus 7T) and emulator, both locally (dev) with latest main and also used adhoc build of recent main (18h ago) from here. No dice reproducing this 🤔

Requested a retest!

@kadiealexander Can you please ask for the tester to include steps for precondition:

user has saved credentials on the device

And also mention what device they used and whether it was a real device or emulator.

Proposal
Please re-state the problem that we are trying to solve in this issue
Sign-in page flickers when using Android 14's GBoard keyboard autofill functionality.
What is the root cause of that problem?

[!important]
I was not able to reproduce this by following the OP steps but instead I was able to precisely replicate the behaviour:
Precondition: User has saved credentials on the device.

Open the app.
Tap the email address field.
Long press on the autofill "Key".

Video (replicated behaviour)
https://github.com/Expensify/App/assets/56457735/a9684103-3a93-4355-b988-96234658558e

The reason why this flicker can be observed currently and wasn't before is because of a recent performance related issue (https://github.com/Expensify/App/issues/36645) and PR https://github.com/Expensify/App/pull/38093 introduced by me.
What happens exactly ?

In the OP video and also my own replica (reproduced with different steps), what happens is that for a split second app (state) goes into background then back to foreground, all this happens as a result of interacting with the GBoard autofill functionality (external from our app).

To illustrate this, in the following video I'm reproducing the behaviour in a slow manner by actually opening the GBoard autofill popover which causes the app (state) to go to background and return back to foreground when closing:

Video (illustrating 1.)
https://github.com/Expensify/App/assets/56457735/b83bee92-8b5a-4f05-a291-ad436a016fec

Because of this app state transition and the logic introduced recently by my PR, when app (state) is going into background we unmount the Lottie animation and instead show the empty view here:

https://github.com/Expensify/App/blob/fe0699c29437cc0e1c1829f8a895bfee80de0553/src/components/Lottie/index.tsx#L27-L29
What actually happens during the flicker is that the SignInHeroImage Lottie animation is unmounted because appState.isBackground is true which shifts the sign in page layout down for a split second, this can be observed in the OP video if played frame by frame:

OP video flicker (frame-by-frame)
https://github.com/Expensify/App/assets/56457735/b3b93017-5cf7-4732-9712-9b808da87efe

While the app state / Lottie animation unmount functionality introduced by me is the reason why this can be observed, the actual root cause of the issue is that within the Lottie component, we're not passing all the styles to the empty View which is being rendered when the Lottie animation is unmounted -> this causes the sign in page layout to move down when app goes into background briefly, which creates the flicker effect in our issue.
This happens on the sign in page because within the SignInHeroImage component, the Lottie animation has additional size related styles:
https://github.com/Expensify/App/blob/fe0699c29437cc0e1c1829f8a895bfee80de0553/src/pages/signin/SignInPageLayout/SignInHeroImage.tsx#L42
which are not passed to the Lottie component's empty View which is rendered when app goes into the background briefly.
What changes do you think we should make in order to solve the problem?
Within the Lottie component, pass all the styles from the Lottie animation to the empty View component here:
https://github.com/Expensify/App/blob/fe0699c29437cc0e1c1829f8a895bfee80de0553/src/components/Lottie/index.tsx#L28
not only the aspectRatioStyle but all the animation props.style too, like so:
-    <View style={aspectRatioStyle} />
+   <View style={[aspectRatioStyle, props.style]} />

By doing this we make sure to pass all the Lottie animation styles coming from the upstream components -> to the empty View so that when we're unmounting the animation and displaying the View, this has exactly the same size as the animation itself ensuring there won't be any layout shifting and therefore no flicker.
Videos (before / after)

Android: Native

Before
After

@allroundexperts any thoughts on the proposal above?

Thanks for the in depth explanation @ikevin127. Your proposal looks good to me.
🎀 👀 🎀 C+ reviewed

PR https://github.com/Expensify/App/pull/39374 ready for review! 🚀

⚠ Automation failed here -> this should be on Hold for payment [2024-04-28] according to 3 days ago's production deploy from https://github.com/Expensify/App/pull/39374#issuecomment-2069770445.
cc @kadiealexander

@kadiealexander Bump, payment overdue based on https://github.com/Expensify/App/issues/38347#issuecomment-2076260307.
Please let me know if there's anything required from my side to complete payment here. Thank you!

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

[ ] [@allroundexperts] The PR that introduced the bug has been identified. Link to the PR:
[ ] [@allroundexperts] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
[ ] [@allroundexperts] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
[ ] [@allroundexperts] Determine if we should create a regression test for this bug.
[ ] [@allroundexperts] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
[ ] [@kadiealexander] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

Payment Summary
Upwork Job

Contributor: @ikevin127 paid $500 via Upwork
Reviewer: @allroundexperts owed $500 via NewDot

BugZero Checklist (@kadiealexander)

[x] I have verified the correct assignees and roles are listed above and updated the neccesary manual offers
[x] I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants//hired)
[x] I have paid out the Upwork contracts or cancelled the ones that are incorrect
[x] I have verified the payment summary above is correct

Checklist

https://github.com/Expensify/App/pull/38093
Kevin is already aware of this as he fixed it. As such,  a comment is not needed.
N/A
A regression test would be helpful. Test steps given in OP look good.

https://github.com/Expensify/Expensify/issues/395764
Payment summary here.

$500 approved for @allroundexperts


No access to security features when using own defined full access admin role

Kibana version in docker: amazon/opendistro-for-elasticsearch-kibana:0.8.0
Elasticsearch version in docker: amazon/opendistro-for-elasticsearch:0.8.0
Reproduction:

Activate OpenID SSO in Kibana:

# Enable OpenID authentication
opendistro_security.auth.type: "openid"

# The IdP metadata endpoint
opendistro_security.openid.connect_url: "https://my-keycloak.com"

# The ID of the OpenID Connect client in your IdP
opendistro_security.openid.client_id: "kibana"

# The client secret of the OpenID Connect client
opendistro_security.openid.client_secret: "secret"

Activate SSO in Elasticsearch according to documentation:

authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      openid_auth_domain:
        enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://my-keycloak.com/auth/realms/master/.well-known/openid-configuration

In KeyCloak add role to user called: my_own_admin_keycloak
Define my_own_admin_kibana role in Kibana by cloning all_access role
Define role mapping my_own_admin_keycloak backend role  -> my_own_admin_kibana

After those operations when I login with user which has my_own_admin_keycloak role I have full access to all regular features in Kibana/Elasticsearch. The only one thing to which I don't have access is Security sections - all buttons disappear and I can see only Authentication & Authorization button but when I'm trying to get into that section I see javascript errors in webconsole.
To fix that I need to add "admin" role to my user in KeyCloak so then I have full access to all features but I want to avoid adding "admin" role to my users as it gives also permissions in some other services in our company.

Have you given your new user role access to the security API? Per https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/25#issuecomment-498333160

Having similar behavior while using own SSL certificates. I can not see any icon on security page of kibana. Attached is the snapshot for the same.  I do have
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
setting my elasticsearch.yaml file.

I've run into something similar as well. I've posted all the details here: https://discuss.opendistrocommunity.dev/t/authentication-and-authorization-404-error-page-not-loading/1861
Has anyone resolved this?

Same problem here

Logged in as that user

We are also facing the same issue. We have used a custom defined own all access admin role.
The security page is loading perfectly but still getting the console errors like :-
GET http://localhost:5601/app/{{actionGroupsSvgURL}} 404 (Not Found)

I found that mapping user to reserved "security_rest_api_access" group will resolve this issue. We are running OD 1.11.0

I found that mapping user to reserved "security_rest_api_access" group will resolve this issue. We are running OD 1.11.0

We are doing some "spring cleaning in the fall", and to make sure we focus our energies on the right issues and we get a better picture of the state of the repo, we are closing all issues that we are carrying over from the ODFE era (ODFE is no longer supported/maintained, see post here).
If you believe this issue should still be considered for current versions of OpenSearch, apologies! Please let us know by re-opening it.
Thanks!


Update Swift 5.3 builds to Xcode 12.2

I believe Xcode 12.2 is the version we are waiting for.

install on

[x] dev
[x] prod
[x] dtk
[x] ceres

update builder

[x] determine matching Linux version
[x] update SwiftVersion.xcodeVersion
[x] update SwiftVersion.dockerImage
run test builds on all builders

[x] dev
[x] prod
[x] dtk
[x] ceres

[x] merge https://gitlab.com/finestructure/swiftpackageindex-builder/-/merge_requests/41

update server

[x] update Xcode version string in SwiftVersion+Build.swift:25
[x] merge https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/pull/768

delete 5.3 builds (in batches, to control for build volume)

[x] write select/delete query
[ ] run deletions

remove old Xcode 12 betas

[x] Ceres
[x] DTK
[x] dev
[x] prod

Edit: review/update todos

Do we wipe 5.3 build results with this?
We definitely should when Xcode 12 final comes out, but I'm 50/50 on whether it's worth it for this upgrade.

Probably just wait for GM at this point

We're down to this being the last lonely issue in the project! 🚀

Waiting for a release later than 12.0 GM due to:

As we finalise this, I think it's also a good time to briefly look ahead at the next upcoming version, Swift 5.4. AFAIK that'll be in spring, so it's a good few months off (although betas might be appearing in Feb/March?).
The mechanism would be the same we used for 5.3's beta:

find a good enough version that'll work for most builds. That'll most likely mean not to jump on it immediately. We may want to give it a good few betas for things to settle down and there actually being a demand for 5.4 builds. (It'll depend a bit on what kind of changes come with it.)
we should aim to deploy only a limited number of 5.4 versions
not rebuilding on 5.4 changes will make this easier (as it did for 5.3 changes)
most importantly: we should drop 4.2 the minute we add 5.4, even if it's a beta

find a good enough version that'll work for most builds. That'll most likely mean not to jump on it immediately. We may want to give it a good few betas for things to settle down and there actually being a demand for 5.4 builds. (It'll depend a bit on what kind of changes come with it.)

It's possible it'll be a really minor version, but in my experience people almost always use a little library to play with/test out a new language feature. Certainly, with both the SwiftPM Library and also the SPI, we had packages that relied on the new version fairly quickly.

most importantly: we should drop 4.2 the minute we add 5.4, even if it's a beta

caveat: v4 support might be significant, so this may mean dropping a strong signal
I don't think we can afford running 5.4 in addition to all other builds though, not just from a capacity point of view. It'll also mess with the version tables.
Just to put it out there: perhaps there's a low 5.x version that we might want to drop instead? (Although that seems weird, too)

I think the tables will work 4.2 -> 5.4 is only as many columns as our current platforms matrix, which displays fine. Build capacity is the biggest issue for sure, and if we're running up against that then we need to make room. But, at the same time I'm not super keen to drop 4.x support just yet.
I don't have particularly strong feelings either way though, I think it's a minor issue.

I'm installing 12.2rc everywhere now as Xcode_12.2_rc.app with a symlink from Xcode_12.2.app to get us started. Even if there's a change before final release this will be better than keeping running on 12_beta_4.app.

Re-opening to track post-deploy cleanup tasks

I've just run the queries to get a feel for the Xcode 12b4 build count on prod.

34,591 builds remain having been run with Xcode 12-beta4
14,810 of those failed
30,382 are older than 30 days
13,158 that failed are older than 30 days

So, it looks like the vast majority of our 5.3-beta backlog is from packages that haven't seen new versions within the last 30 days, and 60% of those succeeded with 12-beta4.
Seems to me that rather than rebuild everything we should only retry the failures.

I can see the logic, in that false positives are less harmful than false negatives, but I bet there are some in there.
I agree we should do the failures first, as I think that's where we'll see the biggest impact. But let's consider doing the positives after the failure backlog is done?

Follow-up in #804


Added QLoRA support for Decoder transformers with tune_strategy "Normal"

I have added arguments under model_args to enable QLoRA support. Namely arguments are:

use_qlora: bool (default: False)
bits: int = (default 4, wtih choices between 4,8)
quant_type: str = 'nf4' default with optional choice of 'fp4'
double_quant: bool = (default: True)

Is model_args.use_qlora is set to 1/True, it also sets model_arga.lora to True so the entire pipeline works.

Hi,
Thanks for your contribution!
Let us check it.

Hi,
It is caused by the access of the repo. We can ignore this error.
I have reopened this PR so that we can review it.
Thanks!

I have further added some code to improve performance (gradient checkpointing and peft prepare_model_for_kbit_training). Need to include these...

How do I install LMFlow on colab? I have tried the attached notebook but it get's stuck at 'Running setup.py develop for lmflow'
LMFlow.ipynb.zip

It's strange. Normally it should be ok with the attached notebook. Which notebook you are working on? could you share me the link?

BUGs:
[src/lmflow/args.py ] \
line 172
Change it to
bits: int = field(
        default=4,
        metadata={"help": "The number of bits for quantization.",
                  "choices": [4 ,8],},
)

line 177
Change it to
quant_type: str = field(
        default="nf4",
        metadata={"help": "The quantization type for quantization.",
                "choices": ["nf4", "fp4"],},
)

BUGs:
[src/lmflow/args.py ]
[line 168] (https://github.com/ankitpasi/LMFlow/blob/293db00da85cdb6f96f2970272469c763af25185/src/lmflow/args.py#L168) Change it to
use_qlora: bool = field(
        default=False,
        metadata={
            "help": "Whether to use qlora."},
)

line 172 Change it to
bits: int = field(
        default=4,
        metadata={"help": "The number of bits for quantization.",
                  "choices": [4 ,8],},
)

line 177 Change it to
quant_type: str = field(
        default="nf4",
        metadata={"help": "The quantization type for quantization.",
                "choices": ["nf4", "fp4"],},
)

ERRORS
May be you need to find a proper version of transformers,deepspeed,peft and bitsandbyres
Traceback (most recent call last):
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 61, in <module>
    main()
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 54, in main
    model = AutoModel.get_model(model_args)
  File "/home/yaoguanyu/qlora/LMFlow/src/lmflow/models/auto_model.py", line 16, in get_model
    return HFDecoderModel(model_args, *args, **kwargs)
  File "/home/yaoguanyu/qlora/LMFlow/src/lmflow/models/hf_decoder_model.py", line 263, in __init__
    quant_config = BitsAndBytesConfig(
TypeError: __init__() got an unexpected keyword argument 'load_in_4bit'
Traceback (most recent call last):
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 61, in <module>
    main()
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 54, in main
    model = AutoModel.get_model(model_args)
  File "/home/yaoguanyu/qlora/LMFlow/src/lmflow/models/auto_model.py", line 16, in get_model
    return HFDecoderModel(model_args, *args, **kwargs)
  File "/home/yaoguanyu/qlora/LMFlow/src/lmflow/models/hf_decoder_model.py", line 263, in __init__
    quant_config = BitsAndBytesConfig(
TypeError: __init__() got an unexpected keyword argument 'load_in_4bit'

Hi
I have updated the requirements to the recommended versions from the official qlora source (https://github.com/artidoro/qlora/blob/main/requirements.txt)
I don't have a GPU machine to try with so I've been trying on colab but as you can see from my previous messages, I've been unable to install.
Would greatly appreciate your help in testing this please..
Thank you

Hi,
Yes, @yaoguany will test it and get to you in a day.
Thanks

Appreciate the support and help! I am very keen to continue using and
developing with LMFlow!
On Thu, Aug 10, 2023 at 11:49 PM shizhediao @.***>
wrote:

Hi,
Yes, @yaoguany https://github.com/yaoguany will test it and get to you
in a day.
Thanks
—
Reply to this email directly, view it on GitHub
https://github.com/OptimalScale/LMFlow/pull/613#issuecomment-1673692819,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJU2UHHVA4RRMSSZ325MDXUUQZ5ANCNFSM6AAAAAA3KR7LRM
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Errors
Traceback (most recent call last):
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 61, in <module>
    main()
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 57, in main
    tuned_model = finetuner.tune(model=model, dataset=dataset)
  File "/home/yaoguanyu/qlora/LMFlow/src/lmflow/pipeline/finetuner.py", line 298, in tune
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 2654, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 2679, in compute_loss
    outputs = model(**inputs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1769, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/peft/peft_model.py", line 922, in forward
    return self.base_model(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1076, in forward
    transformer_outputs = self.transformer(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 843, in forward
    inputs_embeds = self.wte(input_ids)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

When using multi-gpu training it throws this error, maybe you can google it to find some solution.Single gpu is OK.
Also, we need to update the transformers,deepspeed,peft and bitsandbyres before merge this branch.

Errors
When using multi-gpu training it throws this error, maybe you can google it to find some solution.Single gpu is OK. Besides, we need to update the transformers,deepspeed,peft and bitsandbyres before merge this branch.
Traceback (most recent call last):
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 61, in <module>
    main()
  File "/home/yaoguanyu/qlora/LMFlow/examples/finetune.py", line 57, in main
    tuned_model = finetuner.tune(model=model, dataset=dataset)
  File "/home/yaoguanyu/qlora/LMFlow/src/lmflow/pipeline/finetuner.py", line 298, in tune
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 2654, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/trainer.py", line 2679, in compute_loss
    outputs = model(**inputs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1769, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/peft/peft_model.py", line 922, in forward
    return self.base_model(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1076, in forward
    transformer_outputs = self.transformer(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 843, in forward
    inputs_embeds = self.wte(input_ids)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/home/yaoguanyu/anaconda3/envs/test/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Hi @yaoguany
Thanks so much for the test. The device_map={"":0} earlier was the issue, I suspect. I have added code to use local_rank environment variable to set the device map else leave it at "auto" which is better.
I suspect this should fix the issue.
Could you please do a quick try on multi-GPU again with the updated code? Greatly appreciate your kind support!
Thanks

The code runs well now, but we need to update lmflow requirements before merge this branch.
Thanks!

The code runs well now, but we need to update lmflow requirements before merge this branch. Thanks!

Ok, sure. Is that something I need to work on? I have updated lmflow requirements.txt to the below -
numpy==1.24.2 datasets==2.10.1 peft>=0.4.0 torch>=2.0.0 wandb==0.14.0 deepspeed==0.8.3 trl sentencepiece==0.1.99 transformers>=4.31.0 flask flask_cors icetk cpm_kernels==1.0.11 evaluate==0.4.0 scikit-learn==1.2.2 lm-eval dill<0.3.5 bitsandbytes>=0.40.0 pydantic<=1.10.9 gradio accelerate>=0.21.0 einops>=0.6.1 scikit-learn==1.2.2

Hi,
The feature QLoRA depends on the most up-to-date huggingface transformers and deepspeed. We are currently working on the upgrade of these two packages, which is expected to take 1 day.
After the upgrade, we will merge this PR soon.
Thanks for your contribution!

Hi,
Thank you for your contribution.
I encountered a bug while training with your Qlora code, specifically when using the --save_aggregated_lora=1 flag, which is intended for merging the trained lora with the base model. The error message indicates that merging lora with an int4 model isn't possible. Could you provide a script to facilitate the merging of the lora model with the base model?

Hi
I knew this bug would appear before hand since it isn’t possible to use
model.merge_and_unload() for quantised models.
We have two options -

Print out a helpful message to tell user it isn’t possible to aggregate
Lora adapters with quantised models if use_QLoRA is True

Reload the base model again in bf16 without quantisation and do the
model.merge_and_unload()

I’ll go with number 2 for now.
Thanks
Ankit Pasi
On Fri, 11 Aug 2023 at 9:32 PM, Xiang LIU @.***> wrote:

Hi,
Thank you for your contribution.
I encountered a bug while training with your Qlora code, specifically when
using the --save_aggregated_lora=1 flag, which is intended for merging
the trained lora with the base model. The error message indicates that
merging lora with an int4 model isn't possible. Could you provide a script
to facilitate the merging of the lora model with the base model?
—
Reply to this email directly, view it on GitHub
https://github.com/OptimalScale/LMFlow/pull/613#issuecomment-1675021963,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJU2W634MUXMUW5VBERA3XUZJSHANCNFSM6AAAAAA3KR7LRM
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Hi,
Thank you for your contribution. I encountered a bug while training with your Qlora code, specifically when using the --save_aggregated_lora=1 flag, which is intended for merging the trained lora with the base model. The error message indicates that merging lora with an int4 model isn't possible. Could you provide a script to facilitate the merging of the lora model with the base model?

Hi
I've updated the code to to enable merging of the lora adapters with the base model. Could you please help me out by testing it?

Question, shouldn't the following code
`def save(self, dir, save_full_model=False, *args, **kwargs):
"""
Perform generation process of the model.
    Parameters
    ------------
    dir :
        The directory to save model and tokenizer

    save_full_model : Optional.
        Whether to save full model.

    kwargs : Optional.
        Keyword arguments.

    Returns
    ------------
    outputs :
        The generated sequence output
    """
    self.get_tokenizer().save_pretrained(dir)
    if save_full_model and self.model_args.use_lora:
        self.backend_model_full.save_pretrained(dir)
    else:
        self.get_backend_model().save_pretrained(dir)`

Be this instead?
`def save(self, dir, save_full_model=False, *args, **kwargs):
"""
Perform generation process of the model.
    Parameters
    ------------
    dir :
        The directory to save model and tokenizer

    save_full_model : Optional.
        Whether to save full model.

    kwargs : Optional.
        Keyword arguments.

    Returns
    ------------
    outputs :
        The generated sequence output
    """
    self.get_tokenizer().save_pretrained(dir)
    if save_full_model and self.model_args.use_lora:
        self.get_backend_model().save_pretrained(dir)
    else:
        self.backend_model_full.save_pretrained(dir)`

because backend_model_full refers to the model without peft and backend_model refers to the PeftModel?

HI,
I have rented out a ML server with 2 x A100 80GB GPUs to test my code. I am happy to say that I ironed out a few bugs related to bnb_4bit_compute_dtype and have also fixed the above comment about saving aggregated lora by reloading the base model and merging the lora weights (disposing off the quantized model).

Hi,
Thank you for your contribution. I encountered a bug while training with your Qlora code, specifically when using the --save_aggregated_lora=1 flag, which is intended for merging the trained lora with the base model. The error message indicates that merging lora with an int4 model isn't possible. Could you provide a script to facilitate the merging of the lora model with the base model?

This had been done, per my previous comment. It is now able to train using QLoRA and then merge the lora adapters with the base model (tested on multi GPU setup)

Hi,
Thanks so much!
We have successfully upgraded the transformers and deepspeed version as planned.
Will review and merge this PR ASAP.
Thank you again!

Question, shouldn't the following code
def save(self, dir, save_full_model=False, *args, **kwargs):
        """
        Perform generation process of the model.

        Parameters
        ------------
        dir :
            The directory to save model and tokenizer

        save_full_model : Optional.
            Whether to save full model.

        kwargs : Optional.
            Keyword arguments.

        Returns
        ------------
        outputs :
            The generated sequence output
        """
        self.get_tokenizer().save_pretrained(dir)
        if save_full_model and self.model_args.use_lora:
            self.backend_model_full.save_pretrained(dir)
        else:
            self.get_backend_model().save_pretrained(dir)

Be this instead?
def save(self, dir, save_full_model=False, *args, **kwargs):
        """
        Perform generation process of the model.

        Parameters
        ------------
        dir :
            The directory to save model and tokenizer

        save_full_model : Optional.
            Whether to save full model.

        kwargs : Optional.
            Keyword arguments.

        Returns
        ------------
        outputs :
            The generated sequence output
        """
        self.get_tokenizer().save_pretrained(dir)
        if save_full_model and self.model_args.use_lora:
            self.get_backend_model().save_pretrained(dir)
        else:
            self.backend_model_full.save_pretrained(dir)

because backend_model_full refers to the model without peft and backend_model refers to the PeftModel?

Hi, Thanks for your interests. self.get_backend_model().save_pretrained(dir) is our default saving strategy. For full model, it saves full. For lora model, it saves lora. However, sometimes, when we merge lora models, we want to save the full model rather than lora model, i.e., save_full_model and self.model_args.use_lora. Our code should also be compatible with full training without peft. That is our motivation.

QLoRA is an important feature of large language model training.  We wanted to express our deepest gratitude for your outstanding contribution to implementing QLoRA to LMFlow.
Your contributions are always welcomed and will undoubtedly help us shape a more successful future for LMFlow.
Thanks!

QLoRA is an important feature of large language model training. We wanted to express our deepest gratitude for your outstanding contribution to implementing QLoRA to LMFlow.
Your contributions are always welcomed and will undoubtedly help us shape a more successful future for LMFlow.
Thanks!

I thank you and wanted to express my gratitude for the opportunity to contribute and learn along the way!
I just had a follow up question regarding the below code, and not sure where to put it -
def save(self, dir, save_full_model=False, *args, **kwargs):
        """
        Perform generation process of the model.

        Parameters
        ------------
        dir :
            The directory to save model and tokenizer

        save_full_model : Optional.
            Whether to save full model.

        kwargs : Optional.
            Keyword arguments.

        Returns
        ------------
        outputs :
            The generated sequence output
        """
        self.get_tokenizer().save_pretrained(dir)
        if save_full_model and self.model_args.use_lora:
            self.backend_model_full.save_pretrained(dir)
        else:
            self.get_backend_model().save_pretrained(dir)

The code above saves backend_model_full if save_full_model=True, but in the code for hf_decoder_model.py (lines 279-312) below self.backend_model_full actually doesn't include the lora adapters, correct? The model which includes the lora adapters and is merged is self.backend_model (not self.backend_model_full) -

 def merge_lora_weights(self):
        if self.model_args.use_lora and not self.model_args.use_qlora:
            self.get_backend_model().merge_and_unload()

 model = AutoModelForCausalLM.from_pretrained(
                    model_args.model_name_or_path,
                    from_tf=bool(".ckpt" in model_args.model_name_or_path),
                    config=config,
                    quantization_config=quant_config if model_args.use_qlora else None,
                    cache_dir=model_args.cache_dir,
                    revision=model_args.model_revision,
                    use_auth_token=True if model_args.use_auth_token else None,
                    torch_dtype=torch_dtype,
                    device_map=device_map,
                    trust_remote_code = model_args.trust_remote_code,
                )
                if model_args.use_qlora:
                    model.gradient_checkpointing_enable()
                    model = prepare_model_for_kbit_training(model)
            else:
                model = AutoModelForCausalLM.from_config(config)
                n_params = sum(dict((p.data_ptr(), p.numel()) for p in model.parameters()).values())
                logger.info(f"Training new model from scratch - Total size={n_params/2**20:.2f}M params")
            self.backend_model_full = model
            if model_args.use_lora:
                if model_args.lora_target_modules:
                    lora_target_modules = model_args.lora_target_modules
                else:
                    lora_target_modules = None
                peft_config = LoraConfig(
                    task_type=TaskType.CAUSAL_LM,
                    inference_mode=False,
                    r=model_args.lora_r,
                    lora_alpha=model_args.lora_alpha,
                    lora_dropout=model_args.lora_dropout,
                    target_modules=lora_target_modules,
                )
                model = get_peft_model(model, peft_config)

Apologies for this but it's just ringing in my mind so would appreciate if these code snippets can be validated.
Thanks
Ankit

backend_model

Hi, Thank you for your careful investigation.
In fact, backend_model_full refers to base model and backend_model refers base_model+lora. When we use lora merging, the lora weights are merged in to the base_model. So we save backend_model_full. But for backend_model, since the class of it is still peft model, so it would be some compatibility issues.
You can regard backend_model_full as backend_model.base_model, whose class is model rather than peft model.