Errors while executing zshrc

Installed it... Made a change to zshrc.. on executing, getting
/Users/xx/.oh-my-zsh/oh-my-zsh.sh: line 12: autoload: command not found
/Users/xx/.oh-my-zsh/oh-my-zsh.sh: line 31: syntax error near unexpected token `('
/Users/xx/.oh-my-zsh/oh-my-zsh.sh: line 31: `for config_file ($ZSH/lib/*.zsh); do'
Note:
echo $SHELL
/bin/zsh
Do I need to configure something>

Provide your config? Otherwise we can't track down the problem.

Check for updates on initial load...
if [ "$DISABLE_AUTO_UPDATE" != "true" ]; then
env ZSH=$ZSH DISABLE_UPDATE_PROMPT=$DISABLE_UPDATE_PROMPT zsh -f $ZSH/tools/check_for_upgrade.sh
fi
Initializes Oh My Zsh
add a function path
fpath=($ZSH/functions $ZSH/completions $fpath)
Load all stock functions (from $fpath files) called below.
autoload -U compaudit compinit
: ${ZSH_DISABLE_COMPFIX:=true}
Set ZSH_CUSTOM to the path where your custom config files
and plugins exists, or else we will use the default custom/
if [[ -z "$ZSH_CUSTOM" ]]; then
ZSH_CUSTOM="$ZSH/custom"
fi
Set ZSH_CACHE_DIR to the path where cache files should be created
or else we will use the default cache/
if [[ -z "$ZSH_CACHE_DIR" ]]; then
ZSH_CACHE_DIR="$ZSH/cache"
fi
Load all of the config files in ~/oh-my-zsh that end in .zsh
TIP: Add files you don't want in git to .gitignore
for config_file ($ZSH/lib/*.zsh); do
custom_config_file="${ZSH_CUSTOM}/lib/${config_file:t}"
[ -f "${custom_config_file}" ] && config_file=${custom_config_file}
source $config_file
done
is_plugin() {
local base_dir=$1
local name=$2
test -f $base_dir/plugins/$name/$name.plugin.zsh
|| test -f $base_dir/plugins/$name/_$name
}
Add all defined plugins to fpath. This must be done
before running compinit.
for plugin ($plugins); do
if is_plugin $ZSH_CUSTOM $plugin; then
fpath=($ZSH_CUSTOM/plugins/$plugin $fpath)
elif is_plugin $ZSH $plugin; then
fpath=($ZSH/plugins/$plugin $fpath)
fi
done
Figure out the SHORT hostname
if [[ "$OSTYPE" = darwin* ]]; then
macOS's $HOST changes with dhcp, etc. Use ComputerName if possible.
SHORT_HOST=$(scutil --get ComputerName 2>/dev/null) || SHORT_HOST=${HOST/./}
else
SHORT_HOST=${HOST/./}
fi
Save the location of the current completion dump file.
if [ -z "$ZSH_COMPDUMP" ]; then
ZSH_COMPDUMP="${ZDOTDIR:-${HOME}}/.zcompdump-${SHORT_HOST}-${ZSH_VERSION}"
fi
if [[ $ZSH_DISABLE_COMPFIX != true ]]; then
If completion insecurities exist, warn the user without enabling completions.
if ! compaudit &>/dev/null; then
# This function resides in the "lib/compfix.zsh" script sourced above.
handle_completion_insecurities
Else, enable and cache completions to the desired file.
else
compinit -d "${ZSH_COMPDUMP}"
fi
else
compinit -i -d "${ZSH_COMPDUMP}"
fi
Load all of the plugins that were defined in ~/.zshrc
for plugin ($plugins); do
if [ -f $ZSH_CUSTOM/plugins/$plugin/$plugin.plugin.zsh ]; then
source $ZSH_CUSTOM/plugins/$plugin/$plugin.plugin.zsh
elif [ -f $ZSH/plugins/$plugin/$plugin.plugin.zsh ]; then
source $ZSH/plugins/$plugin/$plugin.plugin.zsh
fi
done
Load all of your custom configurations from custom/
for config_file ($ZSH_CUSTOM/*.zsh(N)); do
source $config_file
done
unset config_file
Load the theme
if [ "$ZSH_THEME" = "random" ]; then
themes=($ZSH/themes/*zsh-theme)
N=${#themes[@]}
((N=(RANDOM%N)+1))
RANDOM_THEME=${themes[$N]}
source "$RANDOM_THEME"
echo "[oh-my-zsh] Random theme '$RANDOM_THEME' loaded..."
else
if [ ! "$ZSH_THEME" = ""  ]; then
if [ -f "$ZSH_CUSTOM/$ZSH_THEME.zsh-theme" ]; then
source "$ZSH_CUSTOM/$ZSH_THEME.zsh-theme"
elif [ -f "$ZSH_CUSTOM/themes/$ZSH_THEME.zsh-theme" ]; then
source "$ZSH_CUSTOM/themes/$ZSH_THEME.zsh-theme"
else
source "$ZSH/themes/$ZSH_THEME.zsh-theme"
fi
fi
fi

@viveksachdeva can you attach the .zshrc instead of oh-my-zsh.txt?

Attached
zshrc.txt

What version of zsh are you running? You can check by running the command 'zsh --version'.
so long as:

zshrc is located at $HOME/.zshrc
and
Your .oh-my-zsh clone is located at $HOME/.oh-my-zsh

You shouldn't have a problem. Unless your zsh version, or you have a .zlogin or .zshenv config messing something up. Those two files a read before .zshrc

To save yourself troubleshooting time you could just delete your configurations files if you haven't put time into configuring them.

version : zsh 5.2 (x86_64-apple-darwin16.0)
File location is correct.. I removed .oh-my-zsh dir and cloned again.. still same issue

Then it's a configuration profile more than likely. Backup and remove your configuration files for zsh.
Go into the .oh-my-zsh directory and re-copy the template zshrc and replace your current zshrc located at $HOME/.zshrc
If that doesn't work...
The following command will move all configuration files for zsh to your desktop as a backup, and again, copy the OMZ zshrc template to your $HOME/.zshrc
' mv $HOME/{.zshrc,.zshenv,.zprofile.,.zlogin,.zlogout '
This would rule out that other ZSH configuration files in your $HOME directory could be conflicting.

Besides checking files permissions, check if EDITOR environment variable is set. If it's not set, some scripts may fail

Run zsh -xvic 'exit' &> ~/omz-debug.log and post the omz-debug.log file as a private gist. This is a weird issue...
Check that you're really running zsh with echo $0 (should say zsh).

@Bransonlee0426 what do you mean by "when I use ~/.zshrc".
.zshrc is not an executable.
Simply start zsh by /bin/zsh or change your login shell via chsh -s /bin/bash.


Better Client use: References

Introducing API References: A cleaner wrapped utility for IWC use:
A Reference is a constructed object that links the IWC user to a specific API resource. It contains all the functions for a given API available for abstraction and low syntax footprint:
Example: Data Api : Get and Set a value
// reference the resource /foo
var fooRef = new client.data.Reference("/foo");

// set foo's value
fooRef.set(1);

// get foo's value and pass it to some functionality
fooRef.get().then(value){
   //value === 1
   someFn(value);
});

// This can be simplified further:
fooRef.get().then(someFn);

This can introduce promise function pipelines, for example getting a value, producing some modified value, then setting the value:
function someFn(value){
   return value++;
}

// Increment the value of foo
fooRef.get()
    .then(someFn)
    .then(fooRef.set);

Migrating to References is advised
The old style of accessing resources is still valid and not broken as it is an underlying component of references, but support and documentation will be catered towards References.

No more client.data()  functional access necessary
In the previous releases of IWC, a set on a data resource was done as so:
client.data().set("/foo",{...});

Accessing the data api as function isn't necessary any more, but will still work.
Now:
client.data.set("/foo",{...});

1.2.0 semver bump due to collection changes
Collection updates on watches are no longer are triggered by setting a pattern on a resource. Now, collect:true in the message to the resource will trigger it:
Before
client.data().watch("/foo",{"pattern": "/foo/"}, onChange);

Now
// the reference will receive collection updates on watch
// because collect is set to true

var fooRef = new client.data.Reference("/foo", { collect: true} );
fooRef.watch(onChange);

// disable collection updates, now watch above stops
// getting collection updates
fooRef.updateDefaults({ collect: false});

Now (non references)
NOTE: this method cannot disable collection updates for the watch once it starts. If control over receiving watch updates for collection changes is desired you must migrate to the reference style.
client.data().watch("/foo", {"collect": true}, onChange);

feat(ApiPromise): Made common functions static.
feat(ApiPromise): Added references to api functionality.
feat(ApiPromise): Added promise resolution changes for ref functionality. Now resolve with entity.
fix(node): if a node is deleted create resource should update it to not deleted.
feat(ApiPromise): References throw the error's response.
feat(ApiPromise): Added tests.
feat(ApiPromise): Added defaultPacket test for references.
feat(ApiPromise): Added documentation and clarified augmentations for future enhancing.
feat(ApiPromise): clarified augment params.
feat(gitbook): Heavy updates for common actions. Explanation of references. Terminology clarification.
feat(ApiPromise): added fullResponse/Callback flags for reference
feat(Node): Collect filter applies to read/write actions
feat(api): Deleting a resource that collects stops collection until collect set again
feat(api):Add collect action
feat(api): collection returns info on node collecting
feat(filters): refactor checkForCollect
feat(Reference): Change to prototyped
feat(Reference): collect watch behavior configurable by reference

Won't be merging this in for likely a few days, updating gitbooks further, and more importantly updating website tutorials and examples.
Once the new gh-pages content is ready, I'll merge this, release 1.2.0, then release the new website content for minimal gap between release and new learning material.
If anyone's interested in trying out the new Resources in the mean time pull down feat-apiEnhance, use the normal install/build/serve measures.
Gitbook docs can be built and read with grunt gitbook and open file:// <path to repo>/gh-pages/gitbook/index.html in a browser
Any uncertainty of this new content can be commented on this pull request while docs are updated further.

Final code review underway. Updates uncovered a few bugs that were breaking non-sharedWorker browsers.


Support table model

Query functions Q1, Q2, Q3, Q9, Q10.
1.1 After the keyword from, the table model is connected to , and the tree model is connected to ;
1.2 After the keyword where, the table model is connected to , and the tree model is connected to ;

Write function.
2.1 Mapping tree model to table model;
Retain the logic of tree model as much as possible. In root.test.g_0.d_0, test is the database, g_0_table is the table name, and d_0 is the identifier column; that is, all the data of an organization is stored in one table, and only one database is created;
2.2 Create database and table;
The database creation uses synchronization blocks to ensure that only one database is created;
2.3 Write request requires new information columnCategories
Add the attribute columnCategories in sensor. If it is a table model, create a table containing columnCategories in gentable.
2.4 Generate an identifier column
The name of the identifier column is currently fixed as "device_ID". In initworkload, if columnCategories == ID, the value of device_ID is generated.

新增 boolean 参数 ENABLE_TABLE，默认为 false。在 dbConfig 中增加 SQL_dialect ，用于为 session 指定 sql 解析。在加载配置阶段根据 ENABLE_TABLE 的值为 SQL_dialect 赋值。

目前在 initSensorCode 与 initWorkload 创建存储标识列的sensor太死板了。大家有没嘚好的 idea


Add checkbox for updating contributors.md

I think it's important to recognize our contributors!

I also think it's important to recognize contributors, but, to play devil's advocate, GitHub tracks contributors also. I think GitHub's display is more prominent and it gets tracked automatically. Does the CONTRIBUTORS.md file have an advantage over GitHub's contributors feature?

As I was creating this PR, I was also thinking of that, and was wondering the necessity of this file, or should we just do https://github.com/all-contributors/all-contributors

I'd opt for using native GitHub functionality along with specifically mentioning contributors in the notes for each release. I'd rather not add the overhead of another tool.


slowFindByPhone does not iterate over all identities

The slowFindByPhone function uses the buggy listIdentities kratos admin function that does not return the entire list.
Possible solution:

Yeah it is most likely not returning the last page of identities (kratos bug) . If we change the code to use the service wrapper in IdentityRepository it should work (because we added a workaround to get the last page)

import {IdentityRepository } from "@services/kratos"

 const identites = await IdentityRepository().listIdentities()

Originally posted by @ntheile in https://github.com/GaloyMoney/galoy/issues/2260#issuecomment-1410692551

this function has been deleted


Increasing creation time for new types/clusters, if an index is defined

In the context of a converter from SQL to graph database, I benchmarked the creation of new types in OrientDB. Strangely, when an index is defined, the time spent for creating new types/clusters increases linearly (approximately) over time.
My benchmark performs the following steps, using the Java API with OrientDB 2.1rc4:
add 2 string properties to V (someProperty1, someProperty2)
for 1 to 10 do
    insert 10 new types
    insert 10000 vertices
endfor

Concretely, in each iteration I measure only the time spent to create the 10 new types, in three different case:

no index
one index defined for someProperty1
one index defined for someProperty1 and one index defined for someProperty2

It appears that the time spent for creating new types is constant if no indices are defined. However, with an index, the creation time increases linearly to the number of existing types. If we increase the number of indices, the creation time seems to increase linearly as well.

The problem occurs with SB-Tree and Hash indices, unique or not unique. It occurs as well when the clusters are empty (no vertex insertion). Intuitively, I suspect a problem related to index allocation or index rebalacing.
Here is the quick&dirty code I runned for the benchmark: http://benjaminbillet.fr/media/OrientDBDatabaseBenchmarks.java

	for(int i = 0; i < nbSteps; i++)
	{
		long startTime = System.nanoTime();
		for(int j = nbTypes; j < goal; j++)
		{
			graph.createVertexType("MyClass" + j);
		}
		long delta = System.nanoTime() - startTime;
		totalTypeCreationTime += delta;
		System.out.println("create type: " + nbTypes + " -> " + goal + " in " + ((double) delta / NANO) + "s");

		if(nbVertices > 0)
		{
			startTime = System.nanoTime();
			for(int j = 0; j < nbVertices; j++)
			{
				graph.addVertex("class:MyClass" + (j % goal), "someProperty1", "value" + j, "someProperty2", "value" + (i * nbVertices + j));
			}
			delta = System.nanoTime() - startTime;
			totalInsertionTime += delta;
			System.out.println("insert vertices: " + (nbVertices * nbTypes) + " -> " + (nbVertices * goal) + " in " + ((double) delta / NANO) + "s");
		}

		nbTypes += nbTypesPerSteps;
		goal += nbTypesPerSteps;
	}

It appears that your "for" loop is not assigning the new nbTypes value to the loop control variable "i", so the loop limit is increasing by 10 with each iteration...10, 20, 30, 40...

I don't follow you. The variable nbTypes is not used in the first "for" loop. If we simplify my code, we have:
int nbSteps = 10;
int nbTypesPerSteps = 10;
int nbTypes = 0;
int goal = 10;
for(int i = 0; i < nbSteps; i++) // nbSteps is never increased
{
	for(int j = nbTypes; j < goal; j++)
	{
		graph.createVertexType("MyClass" + j);
	}
	nbTypes += nbTypesPerSteps;
	goal += nbTypesPerSteps;
}

The first "for" will iterate from 0 to nbSteps (10). The second "for" loop will iterate from nbTypes to goal, which are initially set to 0 and 10, and then increased by nbTypesPerSteps (10) at the end of each iteration of the first "for" loop.
Anyway, I created this benchmark because I first observed this type creation time behavior in the project I'm currently working on.

Hi,
We can mitigate this overhead for DBs of average size (in short terms) but not for big database. Reason of this overhead is that indexes are used random IO reads which are much slower than RAM reads. As result we have linear complexity during insertion because more and more pages are on disk not in cache.

But we have long term strategy for big databases too :-)

Oh I see sorry it is not a bug )) We do forcesynch when we create new types the more data you have more data to flush. so no surprise for me.

How many vertices are created once the test is finished?
Scott

@laa I don't understand why it is not a bug, because with no data (empty database) the problem is the same. In any case, this problem should be mitigated, as creating many types with many indices is not an exotic use case. I work on a project that creates a lot of types and, after some time, creating a new type takes thirty minutes... Currently, my workaround is to use a straightforward in-memory indexing instead of OrientDB indexing, given that I have less than 200000 vertices. However, this would not be suitable for large databases.
Another workaround is to create the indices after all the types are created. However, this is a "static" solution; if I want to create new types after that, the creation time will be huge.
@smolinari I performed my benchmarks first with only 100000 vertices and then with an empty database: I got nearly the same creation times. If you want to see it by yourself, you can run my benchmarking code and set the number of vertices to 0.

I agree we should fix this. Not as a bug, but a "performance" issue.

Problem of this bug is that we require to flush disk cache in case of class\cluster\index creation. Recently we decided to postpone change which allows to prevent force flush of data which in turn will fix this bug.
So I postpone this change to next version.

Fixed in 2.2 version


Hello World smart contract isn't working.

my environment is -> eosio latest version(v.1.1.0 I think) on Ubuntu 16.04.
Currently, I'm following eosio smart contract tutorials
https://developers.eos.io/eosio-cpp/docs/hello-world
But thing is, after everything done(compile to wast/abi and set contract to account), here is result of cleos push action.
root@eos-full:~/study# cleos push action hello.code hi '["user"]' -p user
executed transaction: 86e8190b16f77eaa3df27df22e2c7cae52c1a6ba4c730984835772cfb47c4cf2  104 bytes  435 us
#     hello.code <= hello.code::hi               {"user":"user"}
warning: transaction executed locally, but may not be confirmed by the network yet
root@eos-full:~/study#
Q1. What Can I do for "warning: transaction~~~" message?
Q2. Why I'm not getting smart contract return message(in this case, 'Hello, user')

Looking at the response of your cleos command, I see nothing wrong. Your action is executed successfully.
Q1. What Can I do for "warning: transaction~~~" message?

Nothing, it's just a warning that tells you it might not be confirmed by the network yet until somebody includes that transaction in the block (which most likely will be success if you are running an isolated local node)

Q2. Why I'm not getting smart contract return message(in this case, 'Hello, user')

Very likely you didn't start your nodeos with --contracts-console argument. The console is suppressed by default.

I'm still new to this myself, so someone can feel free to correct me if I'm wrong:
A1. You are running a single node, transactions can't be confirmed unless you have 15 BPs confirm it.
A2. You have to run nodeos with --contracts-console to get debug output

@andriantolie
since which eosio verison did I have to pass "--contracts-console" as an argument?

 Can't remember exactly the time, but I have been using it since last May


⚠ Spryr has degraded performance

In 90b4f00, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 3633 ms

Resolved: Spryr performance has improved in 0353f33 after 11 minutes.


How to decrypt iPhone save file?

I checked the Game Save Files Encryption/Decryption section in gddocs. I wanted to edit my save file for a video, so I decided to use the code below:
from Crypto.Cipher import AES

KEY = (
    b"\x69\x70\x75\x39\x54\x55\x76\x35\x34\x79\x76\x5d\x69\x73\x46\x4d"
    b"\x68\x35\x40\x3b\x74\x2e\x35\x77\x33\x34\x45\x32\x52\x79\x40\x7b"
)

def remove_pad(data: bytes) -> bytes:
    last = data[-1]
    if last < 16:
        data = data[:-last]
    return data

def mac_decrypt(data: bytes) -> str:
    cipher = AES.new(KEY, AES.MODE_ECB)
    return cipher.decrypt(data).decode()

with open("Savefile.txt", "rb") as io:
    array = bytearray(io.read())

decrypted = mac_decrypt(array)
print(decrypted)

Whenever I try to run my code, it gives the following error:
line 196, in decrypt
raise ValueError("Data must be aligned to block boundary in ECB mode")
ValueError: Data must be aligned to block boundary in ECB mode

Also, yes, I tried the remove_pad function, but the issue still persists there.


Roll engine 623be87e65e6..7d67e275ff82 (1 commits)

git@github.com:flutter/engine.git/compare/623be87e65e6...7d67e275ff82
git log 623be87e65e6..7d67e275ff82 --no-merges --oneline
2019-10-03<EMAIL_ADDRESS>Roll src/third_party/skia cb55010652b2..35e0a58278a3 (2 commits) (flutter/engine#12769)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

Auto-roller completed checks. About to merge.


To the implementation of the .net Core

Settings in different locations are read in different ways, eg. ScriptBundleManager (does not work), LocalTextScript, UploadSettings.
Folder names should be unified except in linux "Scripts" <> "scripts" "Site" <> "site", etc.

I think settings work as they should. You might be right about folders, can you try with 2.9.19?

ScriptBundleManager -  .Net Core - don't work, try read appsetings like in ScriptBundleManager and like in UploadHelper You'll see.
The code that reads properly tested and settings:
[SettingScope("Application"), SettingKey("ScriptBundling")]
public class ScriptBundlingSettings
    {
       public bool? Enabled { get; set; }
     public bool? Minimize { get; set; }
       public bool? UseMinJS { get; set; }
  }
.........
var settings=Serenity.Config.TryGet<ScriptBundlingSettings>();

Example implementation (MyScriptBundleManager, MyHtmlScriptExtensions) in the attached file, in the file _LayoutHeader.cshtml changed Html.Script (...) -> Html.MyScript (...) - and working properly.
MyScriptBundleManager.txt

This should be working in latest


Trev8939/animate3d graphic

Hi @priy1289 - Can I ask you to review this for me please when you have the time?
Thanks!

should the folder structure include folder 'sample' in it? similar to other samples example,
java.com.esri.arcgisruntime.sample.animate3dgraphic.

Thanks for the review @priy1289! Made those changes :)


Time fixes

Description
Backend changed the CreatedAt/UpdatedAt fields from time.Time to sql.NullTime.
This caused an Invalid date since now each field is now a struct with 2 fields, e.g:
"CreatedAt":{"Time":"2021-12-22T19:20:27.53471Z","Valid":true}

Type of change
What is it?

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[x] Tests update

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I run npm run lint:js:fix to check that my code is properly formatted

Well the PR merged not solves this. It only allows nullable time on CreatedAt/UpdatedAt.
Still those fields will have 2 fields each. This has to be merged to solve this issue.

seems like the time is now back to track, closing


Add cloudinit userdata files to bootstrap spec

Issue:  #41023

Problem

There is no way for 3rd party CAPI machine infrastructure providers to specify the provider ID of a cluster when using the embedded rke2 or an external cloud-provider, or in case where there is no corresponding cloud-provider, such as digital ocean.
Solution

Note: https://github.com/rancher/machine/pull/212 has to be merged as a prerequisite, and a mechanism has to be added onto the provisioning cluster to be able to set the bootstrap template fields for the generating handler.
Testing

Engineering Testing
Manual Testing

Manually spun up a cluster using new method, set the following in the bootstrap template:
Files: []rkev1.CloudInitFile{
  {
    Path:    "/etc/rancher/rke2/config.yaml.d/40-provider-id.yaml",
    Content: `kubelet-arg+: 'provider-id=digitalocean://{{ ds.meta_data["instance_id"] }}'`,
  },
},

Automated Testing

Test types added/modified:

Unit

Summary: Updated existing unit tests to validate expected cloud-config. Added a test case to confirm additional files are rendered as expected. Existing integration tests ensure no regressions for baseline functionality.
QA Testing Considerations

Machines should not be reprovisioned on an upgrade, but deleting machines from a machine pool should utilize the new behavior.
Regressions Considerations

Provisioning should work, but this will be caught by CI.
Existing / newly added automated tests that provide evidence there are no regressions:
TODO: Create tech debt issue to test 3rd party drivers. There should be no regressions against existing funcitonality.

Closing as we are bumping to Q4


Does it only supports Android?

As the library is with react-native, Does it supports iOS as well? or only Android support is available?

I don't think there is any equivalent API in iOS but if you know one we might Implement it

I wish it was 😅

I'm not an iOS developer, but I know that services like App Radar or AppsFlyer do something to show where your app downloads come from.
I'm now trying to find out a way to duplicate this behavior but is not being so easy.


mock: buffered producer

Async timer is hard to reasoning about, maybe we should add some logs and collect them to analyze where the flush is called

https://github.com/hstreamdb/hstreamdb-java/pull/203

@s12f


Cleanup of lookup node population

Addresses https://github.com/facebook/akd/issues/437

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.91%. Comparing base (3ce5335) to head (1a6c59a).
Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #438      +/-   ##
==========================================
+ Coverage   88.61%   88.91%   +0.29%
==========================================
  Files          39       39
  Lines        9109     9236     +127
==========================================
+ Hits         8072     8212     +140
+ Misses       1037     1024      -13

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Subsumed by #441


Why server don't have module?

I tried to test, but server don't have module.
Why, So I difficult to manage.

Did you read the Mantra specification ?

Yes, I see, so does we have solution to manage?

This is the suggested directory layout for server side with Mantra.
But, of course you can use any format. This is not a rule in Mantra.
See:  https://kadirahq.github.io/mantra/#sec-Appendix-Server-Side-Directory-Layout


Add auto-delete player from the database along with his donation if he did not go n time.

Description
I propose to add a line in the config and a command with argument checking, like /lp purge time or /lp purge time group. This command should remove all players or players from a particular group who have not logged on the server for more than n time.
Proposed Behaviour
In this case, it will help that the players who do not go more than n time will be removed from the base and not clog it.
Extra Details
Everything is described above.

players with only the default group are not stored in the database. also removing paid ranks just because they havent joined in a certain amount of time seems really scummy imo. additionally, this wouldnt really work on a bungee/velocity network, because afaik luckperms does not keep track of the last time a player was connected to the server. im not sure the proxy keeps track of that either, and that would mean no single server knows the last time any individual player connected to the server.
im not really sure why you would even want to do this anyways. your database should not be taking up that much space, and if it is, i dont think players that havent joined in a while should be your biggest concern

Suggestion is out of scope for the main plugin, but you could achieve this with an add-on using the API.


Support Static v0.7

One thing I had to add was

Base.:*(::Vec, y::Zero) = y
Base.:*(x::Zero, ::Vec) = x

But then everything worked locally

Great, thanks. I'm not sure what Windows problem is, but I get those errors on a lot of packages depending on VectorizationBase.
Maybe there is a bug on my end that only surfaces with Windows. Or perhaps it is somewhere on the Windows side, given that I've not seen it with Mac or Linux.


[json] support JSON schema draft 7

Issue Type: Feature Request
Hi,
I'm watching JSON schema specification progress for a few months as it is adding a great feature: "if", "then", "else" support.
I have noticed Draft-07 of JSON schema specification was released a few days ago: http://json-schema.org/draft-07/json-schema-release-notes.html
I hope that Draft-07 will be supported in VS Code in future :-)
Regards,
Martin
VS Code version: Code 1.21.1 (79b44aa704ce542d8ca4a3cc44cfca566e7720f1, 2018-03-14T14:46:30.761Z)
OS version: Linux x64 4.13.0-37-generic

The schema validation happens here:
https://github.com/Microsoft/vscode-json-languageservice/blob/131bfca1bf2ad90616ea6ecf720a27d01ca58e0a/src/parser/jsonParser.ts#L423

Here's an example of a test:
https://github.com/Microsoft/vscode-json-languageservice/blob/131bfca1bf2ad90616ea6ecf720a27d01ca58e0a/src/test/parser.test.ts#L435
The easiest is to develop this using the tests.

check out the vscode-json-languageservice repo, npm install
npm run test to compile and run tests
to debug, open the folding in VSCode and run the Unit Test launch config

Excellent, thanks :)

Seems like you guys are writing your own schema validation. Why not use a library like ajv (https://github.com/epoberezkin/ajv-errors)

@shawnmclean We also use it to drive code completion

This appears to have been started in:
Microsoft/vscode-json-languageservice#24.
which was merged in 8229aed327630e8aa3c2f6ac84cdae0ee1fd1d2d. I don't know what other features were added.

Does it mean that IF/THEN/ELSE part of Draft-07 is supported now?

added for august


⚠ Bofrost.at has degraded performance

In be604c9, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 977 ms

Resolved: Bofrost.at performance has improved in 9154557.


Added missing omitempty JSON tag

Fixes #25
Signed-off-by: André Martins<EMAIL_ADDRESS>

I don't really know the full repercussions of this change. I'd go for it if we could validate that everything in docker keeps working as expected.

@calavera In theory should work, other docker-go libraries do the same. But still, how can we prove that?

@aanm vendor this patch into docker/docker, run make test-integration-cli. We're good if all tests pass :smile:

@calavera So I did that but unfortunately some tests failed:
FAIL: docker_api_containers_test.go:51: DockerSuite.TestContainerApiGetJSONNoFieldsOmitted

docker_api_containers_test.go:78:
    c.Fatalf("Field %s is missing and it shouldn't", f)
... Error: Field Ports is missing and it shouldn't

I did a little more and there's some discussion about this in https://github.com/docker/docker/pull/13703
ping @duglin @runcom

I think it's pretty clear that we did that (in the linked PR) because we needed to maintain backward compatibility and do not drop the field if it was empty (IIRC)

@runcom Ah! I was missing that. Somehow the statement "To be fully compatible with 1.18, SizeRw and SizeRootFs need to be omit empty." confused me.

I'm not confident about adding all those omitempty(s), we had hassles in docker everytime we added/removed one :) not sure we should go with this PR, ping @calavera

@runcom I agree and understand your POV. But still I think it's something that needs to be addressed and the sooner the better. I just can't understand why "default" attributes should be sent. If the tests pass, there's not need to be worried :-)

Tests don't test everything I'm pretty confident this would break things
On Saturday, January 16, 2016, André Martins<EMAIL_ADDRESS>wrote:

@runcom https://github.com/runcom I agree and understand your POV. But
still I think it's something that needs to be addressed and the sooner the
better. I just can't understand why "default" attributes should be sent. If
the tests pass, there's not need to be worried :-)
—
Reply to this email directly or view it on GitHub
https://github.com/docker/engine-api/pull/28#issuecomment-172268774.

The new content-addressibility stuff is particularly important in this regard: we base the hashes on the metadata -- changing what fields are omitted will cause all of that to change.

I'm closing this PR given that a few maintainers have expressed their concerns about it. I think we should play save and don't add this if we don't really need it.
Thank you all for your comments.


Not working with TRV

HI, I bought a Shelly TRV and activated your plugin on my domoticz but in the error list I have these messages. Can you help me ? Thank you
2022-01-22 17:40:12.513 Error: (Radiatore_Camera) 'onHeartbeat' failed 'AttributeError'.
2022-01-22 17:40:12.513 Error: (Radiatore_Camera) ----> Line 315 in '/home/pi/domoticz/plugins/ShellyCloudPlugin/plugin.py', function onHeartbeat
2022-01-22 17:40:12.513 Error: (Radiatore_Camera) ----> Line 280 in '/home/pi/domoticz/plugins/ShellyCloudPlugin/plugin.py', function onHeartbeat
2022-01-22 17:40:12.513 Error: (Radiatore_Camera) ----> Line 615 in '/home/pi/domoticz/plugins/ShellyCloudPlugin/plugin.py', function updateTRV
The same plugin with a shelly1 works fine

Could you Send me the outcome when you call http:///status in a browser?

When I open the status page I have to login cause the device is passwrod protected.
After login I have this informations:
{"wifi_sta":{"connected":true,"ssid":"casamimmi","ip":"<IP_ADDRESS>","rssi":-74},"cloud":{"enabled":true,"connected":true},"mqtt":{"connected":true},"time":"09:38","unixtime":1642927135,"serial":24,"has_update":false,"mac":"60A423D3E9F2","cfg_changed_cnt":0,"actions_stats":{"skipped":0},"thermostats":[{"pos":100.0,"target_t":{"enabled":true,"value":23.0,"units":"C"},"tmp":{"value":21.6,"units":"C","is_valid":true},"schedule":false,"schedule_profile":0}],"calibrated":true,"bat":{"value":100,"voltage":4.136},"charger":false,"update":{"status":"unknown","has_update":false,"new_version":"20220115-083921/v2.1.2@3f3e56f8","old_version":"20220115-083921/v2.1.2@3f3e56f8","beta_version":null},"ram_total":97280,"ram_free":22288,"fs_size":65536,"fs_free":59508,"uptime":58355,"fw_info":{"device":"shellytrv-60A423D3E9F2","fw":"20220115-083921/v2.1.2@3f3e56f8"},"ps_mode":0,"dbg_flags":0}

found the bug and fixed it.
If you update the plugin source code then it will be fixed.
please let me know if it is working.

Sorry but I've done update but is still not working. I've the same errors on domoticz.
2022-01-23 19:15:49.711 Error: (RadiatoreCamera) 'onHeartbeat' failed 'KeyError'.
2022-01-23 19:15:49.711 Error: (RadiatoreCamera) ----> Line 320 in '/home/pi/domoticz/plugins/ShellyCloudPlugin/plugin.py', function onHeartbeat
2022-01-23 19:15:49.711 Error: (RadiatoreCamera) ----> Line 285 in '/home/pi/domoticz/plugins/ShellyCloudPlugin/plugin.py', function onHeartbeat
2022-01-23 19:15:49.711 Error: (RadiatoreCamera) ----> Line 655 in '/home/pi/domoticz/plugins/ShellyCloudPlugin/plugin.py', function updateTRV
I send too outcome of status.
{"wifi_sta":{"connected":true,"ssid":"casamimmi","ip":"<IP_ADDRESS>","rssi":-78},"cloud":{"enabled":true,"connected":true},"mqtt":{"connected":false},"time":"19:16","unixtime":1642961791,"serial":35,"has_update":false,"mac":"60A423D3E9F2","cfg_changed_cnt":0,"actions_stats":{"skipped":0},"thermostats":[{"pos":100.0,"target_t":{"enabled":true,"value":23.0,"units":"C"},"tmp":{"value":19.7,"units":"C","is_valid":true},"schedule":false,"schedule_profile":0}],"calibrated":true,"bat":{"value":100,"voltage":4.136},"charger":false,"update":{"status":"unknown","has_update":false,"new_version":"20220115-083921/v2.1.2@3f3e56f8","old_version":"20220115-083921/v2.1.2@3f3e56f8","beta_version":null},"ram_total":97280,"ram_free":22488,"fs_size":65536,"fs_free":59508,"uptime":93011,"fw_info":{"device":"shellytrv-60A423D3E9F2","fw":"20220115-083921/v2.1.2@3f3e56f8"},"ps_mode":0,"dbg_flags":0}
Thank you for your support.


Add popup with information about new stable/unstable version of engine.

Godot version:
3.1 dev
Issue description:
It would be good to see a little popup when new version of Godot Engine is released. It should be only shown once when new version comes.

I'd prefer a simple Label that says something like "Update available!", underlined, at the version number and it links to the download page when clicking on it. We have too many popups.

For me the best option, will be add a small bar(like in TV news) at the bottom of the Project Manager.
It should contain the latest news about engine, something like headers from https://godotengine.org/news, and when clicked, they should open a browser with target news
Maybe the bar should move from right to left, like in TV news
I readed #20603, and I think that ideas from this thread, take too much space
Open in new tab to see in full resolution.

Godot doesn't connect to the internet without user intent. At most a "Check for updates" button could be added to demonstrate that intent.

Would an opt-in "Check of updates on startup" setting be acceptable?

Do we have a JSON endpoint that we can use to fetch the identifier and release date of the latest stable Godot version? The GitHub Releases API might be suitable here, but we'll need to perform some checks to avoid recommending older version patches. For instance, if someone is using 3.1.2, notifying them when 3.2 is released is fine, but they shouldn't be notified if 3.0.7 is released.

Godot displays the version in the corner. Perhaps this could double as a "check for updates" button?

@aaronfranke This would conflict with #35375, unfortunately.

Godot Proposal - https://github.com/godotengine/godot-proposals/issues/1132


Cannot install with --no-binary :all:

I am a maintainer of a Homebrew package.
Homebrew's virtualenv_install_with_resources specifies --no-binary :all:.
Users are hitting errors because this package (indirectly) depends on entrypoints, which depends on flit.
I narrowed it down to the following:
~/D/brew (entrypoints) ~> pip install --no-binary :all: flit
Collecting flit
  Using cached https://files.pythonhosted.org/packages/6a/55/a25505e3b3f5d804d1a0b6a64c582f4a264697c94862ccf71b8a1c62d16d/flit-1.2.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting requests (from flit)
  Using cached https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz
Collecting docutils (from flit)
  Using cached https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz
Collecting pytoml (from flit)
  Using cached https://files.pythonhosted.org/packages/35/35/da1123673c54b6d701453fcd20f751d6a1fae43339b3993ae458875576e4/pytoml-0.1.20.tar.gz
Collecting chardet<3.1.0,>=3.0.2 (from requests->flit)
  Using cached https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz
Collecting idna<2.9,>=2.5 (from requests->flit)
  Using cached https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz
Collecting urllib3<1.25,>=1.21.1 (from requests->flit)
  Using cached https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz
Collecting certifi>=2017.4.17 (from requests->flit)
  Using cached https://files.pythonhosted.org/packages/55/54/3ce77783acba5979ce16674fc98b1920d00b01d337cfaaf5db22543505ed/certifi-2018.11.29.tar.gz
Skipping bdist_wheel for flit, due to binaries being disabled for it.
Skipping bdist_wheel for requests, due to binaries being disabled for it.
Skipping bdist_wheel for docutils, due to binaries being disabled for it.
Skipping bdist_wheel for pytoml, due to binaries being disabled for it.
Skipping bdist_wheel for chardet, due to binaries being disabled for it.
Skipping bdist_wheel for idna, due to binaries being disabled for it.
Skipping bdist_wheel for urllib3, due to binaries being disabled for it.
Skipping bdist_wheel for certifi, due to binaries being disabled for it.
Installing collected packages: chardet, idna, urllib3, certifi, requests, docutils, pytoml, flit
  Running setup.py install for chardet ... done
  Running setup.py install for idna ... done
  Running setup.py install for urllib3 ... done
  Running setup.py install for certifi ... done
  Running setup.py install for requests ... done
  Running setup.py install for docutils ... done
  Running setup.py install for pytoml ... done
  Running setup.py install for flit ... error
    Complete output from command /Users/adam/.virtualenvs/entrypoints/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/km/k06fmh5d09l068s2mnwfmf480000gn/T/pip-install-l8yb2ihv/flit/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/km/k06fmh5d09l068s2mnwfmf480000gn/T/pip-record-g1686y_c/install-record.txt --single-version-externally-managed --compile --install-headers /Users/adam/.virtualenvs/entrypoints/bin/../include/site/python3.7/flit:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'

    ----------------------------------------
Command "/Users/adam/.virtualenvs/entrypoints/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/km/k06fmh5d09l068s2mnwfmf480000gn/T/pip-install-l8yb2ihv/flit/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/km/k06fmh5d09l068s2mnwfmf480000gn/T/pip-record-g1686y_c/install-record.txt --single-version-externally-managed --compile --install-headers /Users/adam/.virtualenvs/entrypoints/bin/../include/site/python3.7/flit" failed with error code 1 in /private/var/folders/km/k06fmh5d09l068s2mnwfmf480000gn/T/pip-install-l8yb2ihv/flit/

Just in case:
$ /Users/adam/.virtualenvs/entrypoints/bin/python3.7 -u -c "import setuptools"
$

@takluyver just adding a --no-build-isolation makes pip 19.0.2 throw a pip._vendor.pep517.wrappers.BackendUnavailable.

Closing this as this now works, I guess because https://github.com/pypa/pip/issues/6222 is now resolved. Thanks for the work on this.


Feat/header

Initial implementation of the header component, alongside with the HeaderContent and HeaderSunheader components. Added tests, and provided examples.

I resolve the comment on this PR and added the shorthand example. Should we merge this, or wait for the base component to be merged first?


New binding functions should require categories to be assigned to build successfully

added to the annotation processor, which prompts the developer directly to the affected element when possible


fix: augment @nuxt/schema rather than nuxt/schema

Context: https://github.com/nuxt/nuxt/issues/28332
nuxt/schema is a re-export of @nuxt/schema for users to use. Modules should not augment it, or it may end up overwriting the inferred types from @nuxt/schema.
(We made the change in https://github.com/nuxt/module-builder/pull/295 (released in v0.8.0 of @nuxt/module-builder) to avoid doing this in @nuxt/module-builder itself.)

Thanks for the contribution (and all the work behind nuxt/nitro)!


Use tty instead of stdin

When trying to read from a process that has input piped into its stdin, inquire returns an error, saying stdin is not a TTY. In those cases, we should be reading directly from the tty device. Basically something like this:
        let stdin = if io::stdin().is_terminal() {
            process::Stdio::inherit()
        } else {
            let tty = termion::get_tty()?;
            // In that case, we use the terminal device, eg. `/dev/tty` as standard input.
            process::Stdio::from(tty)
        };

If this isn't something you want to do in inquire, an alternative would be to be able to specify a backend, or input stream, eg. prompt_with_stream(io: process::Stdio).

Hey, I've fixed that for termion, but overall cross-platform support is a long story, see #198 :)


Gtf2gtf retained introns

I've added a new method to gtf2gtf.py for identifying retained introns. I've defined retained introns as an intron within one transcript that is entirely contained within the exon of another.
To be honest, I don't know if gtf2gtf is the right place for this method. The obvious format to output retained introns in is bed, but it fit better with the other gene-model manipulation type methods in gtf2gtf than in gff2bed. Open to suggestions for a better home.
More implementation details in the documentation:
``find-retained-introns``
 Finds intervals within a transcript that represent retained-introns,
 here a retained intron is considered to be an intron in one transcript
 that is entirely contianed within the exon of another. The retained
 intron will be assigned to the transcript with the containing exon. Where
 multiple, overlapping introns are contained within a single exon of a
 transcript, the union of the introns will be output. Thus when considering
 an indevidual transcript, outputs will be non-overlapping. However,
 overlapping, or even identical feature can be output if they belong to
 different transcripts.

Its fast (the complete transcriptome finishes in 22 seconds) and I've included a test.

@sebastian-luna-valero This appears to be failing the Travis build because the build is stalling, rather than because there are any test fails. Is there any way I can try to rerun the tests? Or should I go ahead and merge without?

@IanSudbery
I have seen these errors happening before and it is normally safe to merge as the build is stalled due to problems on their end.
However, it is possible to restart a build on Travis as explained here:
http://stackoverflow.com/questions/17606874/trigger-a-travis-ci-rebuild-without-pushing-a-commit
I have already done so for your PR, so let's wait for the result before merging:
https://travis-ci.org/CGATOxford/cgat/builds/48934382
Best regards,
Sebastian.

@AndreasHeger Any views on whether gtf2gtf is the right place for this?

@IanSudbery , many thanks for this. Both gtf2gtf and gtf2gff would be good places for this.
gtf2gff transforms a "geneset" to a list of "features" and retained introns could be regarded as genomic features. However, gtf2gtf is fine as well (gene set transformation) as you might want to keep the transcript/gene information intact after processing.


docs: Fix a few typos

There are small typos in:

examples/terraform/aws/README.md
locust/test/test_runners.py

Fixes:

Should read heartbeat rather than hearbeat.
Should read distributed rather than distribuited.

Semi-automated pull request generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

https://github.com/marcosborges/terraform-aws-loadtest-distributed/raw/v0.4.0/assets/locust-home.png doesnt exist, so you cant ”fix” the url unless someone also fixes the location of the asset ;)

fixing "hearbeat" in separate PR, but thanks for bringing it to my attention.


Nameless conda environment gets run by name instead of path

Discussed in https://github.com/microsoft/vscode-python/discussions/24626

Originally posted by gandhis1 December 17, 2024
In version 2024.16.1, when I run test discovery, or really any command that ultimately invokes conda, I see the following:
2024-12-17 10:09:14.508 [info] > conda run -p ./envs/linux_main --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-x64/python_files/get_output_via_markers.py -m pytest -p vscode_pytest --collect-only .

This works, and this is what I expect, since my environments are nameless and path-based. Here is the anonymized output of conda info --env:
                         /home/myuser/git/myrepo/envs/linux_arctic
                         /home/myuser/git/myrepo/envs/linux_main
base                  *  /opt/miniconda3

In recent versions (this has been ongoing for several weeks I believe), I see this (2024.23.2024121701):
2024-12-17 10:32:03.637 [info] > ~/git/myrepo2/envs/linux_py38/bin/conda run -n linux_main --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2024.23.2024121701-linux-x64/python_files/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2024.23.2024121701-linux-x64/python_files/printEnvVariables.py
2024-12-17 10:32:03.637 [info] shell: bash
2024-12-17 10:32:04.012 [error] getActivatedEnvironmentVariables Error: Command failed: /home/myuser/git/myrepo/envs/linux_py38/bin/conda run -n linux_main --no-capture-output python /home/myuser/.vscode-server/extensions/ms-python.python-2024.23.2024121701-linux-x64/python_files/get_output_via_markers.py /home/myuser/.vscode-server/extensions/ms-python.python-2024.23.2024121701-linux-x64/python_files/printEnvVariables.py

EnvironmentLocationNotFound: Not a conda environment: /home/myuser/git/myrepo/envs/239fb791f5e1df62ec6015e65582c32020a01c5d/envs/linux_main

    at genericNodeError (node:internal/errors:984:15)
    at wrappedFn (node:internal/errors:538:14)
    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:530:35)
    at maybeClose (node:internal/child_process:1104:16)
    at Socket.<anonymous> (node:internal/child_process:456:11)
    at Socket.emit (node:events:518:28)
    at Pipe.<anonymous> (node:net:343:12) {
  code: 1,
  killed: false,
  signal: null,
  cmd: '/home/myuser/git/myrepo2/envs/linux_py38/bin/conda run -n linux_main --no-capture-output python /home/myuser/.vscode-server/extensions/ms-python.python-2024.23.2024121701-linux-x64/python_files/get_output_via_markers.py /home/myuser/.vscode-server/extensions/ms-python.python-2024.23.2024121701-linux-x64/python_files/printEnvVariables.py'
}

Fundamentally, the issue I see is that this should be using -p and not -n. It's also using a somewhat arbitrary conda executable.

@gandhis1 Can you provide the initial part of the logs:

The logs that you shared here don't contain a line that says " Conda is named/prefix". it comes from the detector.
https://github.com/microsoft/vscode-python/discussions/24626#discussioncomment-11595117

Here is everything before "discover tests":
2024-12-17 11:38:46.030 [info] Experiment 'pythonRecommendTensorboardExt' is active
2024-12-17 11:38:46.030 [info] Experiment 'pythonTerminalEnvVarActivation' is active
2024-12-17 11:38:46.030 [info] Experiment 'pythonTestAdapter' is active
2024-12-17 11:38:46.030 [info] Native locator: Refresh started
2024-12-17 11:38:47.511 [info] > pyenv which python
2024-12-17 11:38:47.511 [info] cwd: .
2024-12-17 11:38:47.534 [info] Conda environment manager found at: /opt/miniconda3/bin/conda
2024-12-17 11:38:47.550 [info] Conda environment manager found at: /home/myuser/git/myrepo2/envs/linux_py311/bin/conda
2024-12-17 11:38:47.555 [info] Conda environment manager found at: /home/myuser/git/myrepo2/envs/linux_py38/bin/conda
2024-12-17 11:38:47.562 [info] Native locator: Refresh finished in 1862 ms
2024-12-17 11:38:48.168 [info] > ~/git/myrepo2/envs/linux_py38/bin/conda info --json
2024-12-17 11:38:50.380 [info] Python interpreter path: ./envs/linux_main/bin/python

I don't see a line saying "Conda env is named/prefix"

Can you ~/git/myrepo2/envs/linux_py38/bin/conda info --json ? That seems like your in project conda binary, as opposed to running conda info --json that uses your global conda binary.

@gandhis1 Can you try setting python.condaPath setting in your user settings? Just so it prevents python extension for seeing other conda that it probably should not.

On version it runs this command,
conda run -p ./.venv --no-capture-output python ~/.vscode/extensions/ms-python.python-2024.4.1/python_files/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2024.4.1/python_files/printEnvVariables.py
On the new on it runs this command,
conda run -n .venv --no-capture-output python ~/.vscode/extensions/ms-python.python-2024.22.0-darwin-arm64/python_files/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2024.22.0-darwin-arm64/python_files/printEnvVariables.py

Do you need any more information to replicate the issue? I think replication could be as simple as creating a path-based, anonymous conda environment, and then attempting to use it as the interpreter in VS code.


cleanup kubectl wrapper and use pythons libraries

Signed-off-by: Vicente Zepeda Mas<EMAIL_ADDRESS>Why is this PR needed?
Does it fix an issue? addresses a business case?
add a description and link to the issue if one exists.
Fixes #
Reminder: Add the "fixes bsc#XXXX" to the title of the commit so that it will
appear in the changelog.
What does this PR do?
please include a brief "management" technical overview (details are in the code)
Anything else a reviewer needs to know?
Special test cases, manual steps, links to resources or anything else that could be helpful to the reviewer.
Info for QA
This is info for QA so that they can validate this. This is mandatory if this PR fixes a bug.
If this is a new feature, a good description in "What does this PR do" may be enough.
Steps to reproduce the bug
Related info
Info that can be relevant for QA:

link to other PRs that should be merged together
link to packages that should be released together
upstream issues

Status BEFORE applying the patch
How can we reproduce the issue? How can we see this issue? Please provide the steps and the prove
this issue is not fixed.
Status AFTER applying the patch
How can we validate this issue is fixed? Please provide the steps and the prove this issue is fixed.
Docs
If docs need to be updated, please add a link to a PR to https://github.com/SUSE/doc-caasp.
At the time of creating the issue, this PR can be work in progress (set its title to [WIP]),
but the documentation needs to be finalized before the PR can be merged.

Closing in favor of #603


[Cherry-pick-2.0.2[Feature-7460][]Feature-7451][Fix-7392]

Purpose of the pull request
cherry-pick PR:  #7460 #7451 #7392

Codecov Report

:exclamation: No coverage uploaded for pull request base (2.0.2-prepare@60001f8). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             2.0.2-prepare    #7483   +/-   ##
================================================
  Coverage                 ?   31.87%
  Complexity               ?     1567
================================================
  Files                    ?      432
  Lines                    ?    14568
  Branches                 ?     1444
================================================
  Hits                     ?     4644
  Misses                   ?     9478
  Partials                 ?      446

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 60001f8...39619c6. Read the comment docs.


RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by  making sure all forward function outputs participate in calculating loss.

System Info
- `Accelerate` version: 0.23.0
- Platform: Linux-5.4.0-117-generic-x86_64-with-glibc2.31
- Python version: 3.10.12
- Numpy version: 1.25.2
- PyTorch version (GPU?): 2.0.1 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 251.56 GB
- GPU type: NVIDIA A100-PCIE-40GB
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[X] My own task or dataset (give details below)

Reproduction
I can train normally without the find_unused_parameters parameter of ddp. It seems that find_unused_parameters will cause a lot of VRAM consumption
I'm using huggingface's transformers and diffusers.
The script written to imitate the diffusers training example:https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py.
raise error:
encoder_hidden_states = image_encoder(
                           ^^^^^^^^^^^^^^
 File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
   return forward_call(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/torch/nn/parallel/distributed.py", line 1139, in forward
   if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by
making sure all `forward` function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameters which did not receive grad for rank 0: vision_model.post_layernorm.bias, vision_model.post_layernorm.weight
Parameter indices which did not receive grad for rank 0: 389 390
Steps:   0%|                                                 | 1/200000 [01:06<3683:53:14, 66.31s/it, loss=0.544, lr=6e-11]

my code:
def main(args):
    logging_dir = Path(args.output_dir, args.logging_dir)

    accelerator_project_config = ProjectConfiguration(
        project_dir=args.output_dir, logging_dir=logging_dir
    )

    if args.ddp_find_unused_parameters:
        ddp_kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
        accelerator = Accelerator(
            gradient_accumulation_steps=args.gradient_accumulation_steps,
            mixed_precision=args.mixed_precision,
            log_with=args.report_to,
            project_config=accelerator_project_config,
            kwargs_handlers=[ddp_kwargs],
        )
    else:
        accelerator = Accelerator(
            gradient_accumulation_steps=args.gradient_accumulation_steps,
            mixed_precision=args.mixed_precision,
            log_with=args.report_to,
            project_config=accelerator_project_config,
        )

    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO,
    )
    logger.info(accelerator.state, main_process_only=False)
    if accelerator.is_local_main_process:
        transformers.utils.logging.set_verbosity_warning()
        diffusers.utils.logging.set_verbosity_info()
    else:
        transformers.utils.logging.set_verbosity_error()
        diffusers.utils.logging.set_verbosity_error()

    # If passed along, set the training seed now.
    if args.seed is not None:
        set_seed(args.seed)

    # Handle the repository creation
    if accelerator.is_main_process:
        if args.output_dir is not None:
            os.makedirs(args.output_dir, exist_ok=True)

        if args.push_to_hub:
            repo_id = create_repo(
                repo_id=args.hub_model_id or Path(args.output_dir).name,
                exist_ok=True,
                token=args.hub_token,
            ).repo_id

    def deepspeed_zero_init_disabled_context_manager():
        """
        returns either a context list that includes one that will disable zero.Init or an empty context list
        """
        deepspeed_plugin = (
            AcceleratorState().deepspeed_plugin
            if accelerate.state.is_initialized()
            else None
        )
        if deepspeed_plugin is None:
            return []

        return [deepspeed_plugin.zero3_init_context_manager(enable=False)]

    # Currently Accelerate doesn't know how to handle multiple models under Deepspeed ZeRO stage 3.
    # For this to work properly all models must be run through `accelerate.prepare`. But accelerate
    # will try to assign the same optimizer with the same weights to all models during
    # `deepspeed.initialize`, which of course doesn't work.
    #
    # For now the following workaround will partially support Deepspeed ZeRO-3, by excluding the 2
    # frozen models from being partitioned during `zero.Init` which gets called during
    # `from_pretrained` So CLIPTextModel and AutoencoderKL will not enjoy the parameter sharding
    # across multiple gpus and only UNet2DConditionModel will get ZeRO sharded.
    with ContextManagers(deepspeed_zero_init_disabled_context_manager()):
        if not args.train_image_encoder:
            image_encoder = CLIPVisionModel.from_pretrained(
                args.pretrained_model_name_or_path,
                subfolder="image_encoder",
            )
        vae = AutoencoderKL.from_pretrained(
            args.pretrained_model_name_or_path, subfolder="vae", revision=args.revision
        )
    clip_image_processor = CLIPImageProcessor.from_pretrained(
        args.pretrained_model_name_or_path,
        subfolder="clip_image_processor",
    )
    if args.train_image_encoder:
        image_encoder = CLIPVisionModel.from_pretrained(
            args.pretrained_model_name_or_path,
            subfolder="image_encoder",
        )
    # Load scheduler and models
    noise_scheduler = DDPMScheduler.from_pretrained(
        args.pretrained_model_name_or_path, subfolder="scheduler"
    )
    unet:UNet2DDobuleConditionModel = UNet2DDobuleConditionModel.from_pretrained(
        args.pretrained_model_name_or_path, subfolder="unet", revision=args.revision
    )

    # create custom saving & loading hooks so that `accelerator.save_state(...)` serializes in a nice format
    def save_model_hook(models, weights, output_dir):
        for model in models:
            sub_dir = (
                "unet"
                if isinstance(model, type(accelerator.unwrap_model(unet)))
                else "image_encoder"
            )
            model.save_pretrained(os.path.join(output_dir, sub_dir))

            # make sure to pop weight so that corresponding model is not saved again
            weights.pop()

    def load_model_hook(models, input_dir):
        while len(models) > 0:
            # pop models so that they are not loaded again
            model = models.pop()

            if isinstance(model, type(accelerator.unwrap_model(image_encoder))):
                # load transformers style into model
                load_model = CLIPVisionModel.from_pretrained(
                    input_dir, subfolder="image_encoder"
                )
                model.config = load_model.config
            else:
                # load diffusers style into model
                load_model = UNet2DDobuleConditionModel.from_pretrained(
                    input_dir, subfolder="unet"
                )
                model.register_to_config(**load_model.config)

            model.load_state_dict(load_model.state_dict())
            del load_model

    accelerator.register_save_state_pre_hook(save_model_hook)
    accelerator.register_load_state_pre_hook(load_model_hook)

    vae.requires_grad_(False)
    unet.requires_grad_(True)
    if args.train_image_encoder:
        image_encoder.requires_grad_(True)
    else:
        image_encoder.requires_grad_(False)

    if args.enable_xformers_memory_efficient_attention:
        if is_xformers_available():
            import xformers

            xformers_version = version.parse(xformers.__version__)
            if xformers_version == version.parse("0.0.16"):
                logger.warn(
                    "xFormers 0.0.16 cannot be used for training in some GPUs. If you observe problems during training, please update xFormers to at least 0.0.17. See https://huggingface.co/docs/diffusers/main/en/optimization/xformers for more details."
                )
            unet.enable_xformers_memory_efficient_attention()
        else:
            raise ValueError(
                "xformers is not available. Make sure it is installed correctly"
            )

    if args.gradient_checkpointing:
        unet.enable_gradient_checkpointing()
        image_encoder.gradient_checkpointing_enable()

    # Enable TF32 for faster training on Ampere GPUs,
    # cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
    if args.allow_tf32:
        torch.backends.cuda.matmul.allow_tf32 = True

    if args.scale_lr:
        args.learning_rate = (
            args.learning_rate
            * args.gradient_accumulation_steps
            * args.train_batch_size
            * accelerator.num_processes
        )

    # Use 8-bit Adam for lower memory usage or to fine-tune the model in 16GB GPUs
    if args.use_8bit_adam:
        try:
            import bitsandbytes as bnb
        except ImportError:
            raise ImportError(
                "To use 8-bit Adam, please install the bitsandbytes library: `pip install bitsandbytes`."
            )

        optimizer_class = bnb.optim.AdamW8bit
    else:
        optimizer_class = torch.optim.AdamW

    optimizer = optimizer_class(
        itertools.chain(unet.parameters(), image_encoder.parameters())
        if args.train_image_encoder
        else unet.parameters(),
        lr=args.learning_rate,
        betas=(args.adam_beta1, args.adam_beta2),
        weight_decay=args.adam_weight_decay,
        eps=args.adam_epsilon,
    )

    train_dataset, train_dataloader = make_train_dataset(
        args, clip_image_processor, accelerator
    )

    # Scheduler and math around the number of training steps.
    overrode_max_train_steps = False
    num_update_steps_per_epoch = math.ceil(
        len(train_dataloader) / args.gradient_accumulation_steps
    )
    if args.max_train_steps is None:
        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
        overrode_max_train_steps = True

    lr_scheduler = get_scheduler(
        args.lr_scheduler,
        optimizer=optimizer,
        num_warmup_steps=args.lr_warmup_steps * accelerator.num_processes,
        num_training_steps=args.max_train_steps * accelerator.num_processes,
        num_cycles=args.lr_num_cycles,
        power=args.lr_power,
    )
    # Prepare everything with our `accelerator`.
    if args.train_image_encoder:
        (
            unet,
            image_encoder,
            optimizer,
            train_dataloader,
            lr_scheduler,
        ) = accelerator.prepare(
            unet, image_encoder, optimizer, train_dataloader, lr_scheduler
        )
    else:
        unet, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
            unet, optimizer, train_dataloader, lr_scheduler
        )

    # For mixed precision training we cast the text_encoder and vae weights to half-precision
    # as these models are only used for inference, keeping weights in full precision is not required.
    weight_dtype = torch.float32
    if accelerator.mixed_precision == "fp16":
        weight_dtype = torch.float16
    elif accelerator.mixed_precision == "bf16":
        weight_dtype = torch.bfloat16

    # Move vae, unet and text_encoder to device and cast to weight_dtype
    vae.to(accelerator.device, dtype=weight_dtype)
    if not args.train_image_encoder:
        image_encoder.to(accelerator.device, dtype=weight_dtype)

    # We need to recalculate our total training steps as the size of the training dataloader may have changed.
    num_update_steps_per_epoch = math.ceil(
        len(train_dataloader) / args.gradient_accumulation_steps
    )
    if overrode_max_train_steps:
        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
    # Afterwards we recalculate our number of training epochs
    args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch)

    # We need to initialize the trackers we use, and also store our configuration.
    # The trackers initializes automatically on the main process.
    if accelerator.is_main_process:
        tracker_config = dict(vars(args))

        # tensorboard cannot handle list types for config
        tracker_config.pop("validation_prompt")
        tracker_config.pop("validation_blueprint")

        accelerator.init_trackers(args.tracker_project_name, config=tracker_config)

    # Train!
    total_batch_size = (
        args.train_batch_size
        * accelerator.num_processes
        * args.gradient_accumulation_steps
    )

    logger.info("***** Running training *****")
    logger.info(f"  Num examples = {len(train_dataset)}")
    logger.info(f"  Num Epochs = {args.num_train_epochs}")
    logger.info(f"  Instantaneous batch size per device = {args.train_batch_size}")
    logger.info(
        f"  Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"
    )
    logger.info(f"  Gradient Accumulation steps = {args.gradient_accumulation_steps}")
    logger.info(f"  Total optimization steps = {args.max_train_steps}")
    logger.info(f"  train image encoder={args.train_image_encoder}")
    global_step = 0
    first_epoch = 0

    # Potentially load in the weights and states from a previous save
    if args.resume_from_checkpoint:
        if args.resume_from_checkpoint != "latest":
            path = os.path.basename(args.resume_from_checkpoint)
        else:
            # Get the most recent checkpoint
            dirs = os.listdir(args.output_dir)
            dirs = [d for d in dirs if d.startswith("checkpoint")]
            dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))
            path = dirs[-1] if len(dirs) > 0 else None

        if path is None:
            accelerator.print(
                f"Checkpoint '{args.resume_from_checkpoint}' does not exist. Starting a new training run."
            )
            args.resume_from_checkpoint = None
        else:
            accelerator.print(f"Resuming from checkpoint {path}")
            accelerator.load_state(os.path.join(args.output_dir, path))
            global_step = int(path.split("-")[1])

            resume_global_step = global_step * args.gradient_accumulation_steps
            first_epoch = global_step // num_update_steps_per_epoch
            resume_step = resume_global_step % (
                num_update_steps_per_epoch * args.gradient_accumulation_steps
            )
            resume_progress_bar = tqdm(
                range(0, resume_step), disable=not accelerator.is_local_main_process
            )

    # Only show the progress bar once on each machine.
    progress_bar = tqdm(
        range(global_step, args.max_train_steps),
        disable=not accelerator.is_local_main_process,
    )
    progress_bar.set_description("Steps")

    image_logs = None
    for epoch in range(first_epoch, args.num_train_epochs):
        unet.train()
        if args.train_image_encoder:
            image_encoder.train()
        train_loss = 0.0
        for step, batch in enumerate(train_dataloader):
            if (
                args.resume_from_checkpoint
                and epoch == first_epoch
                and step < resume_step
            ):
                if step % args.gradient_accumulation_steps == 0:
                    resume_progress_bar.update(1)
                continue
            with accelerator.accumulate(unet):
                # Convert images to latent space
                latents = vae.encode(
                    batch["pixel_values"].to(dtype=weight_dtype)
                ).latent_dist.sample()
                latents = latents * vae.config.scaling_factor

                # Sample noise that we'll add to the latents
                noise = torch.randn_like(latents)
                bsz = latents.shape[0]
                # Sample a random timestep for each image
                timesteps = torch.randint(
                    0,
                    noise_scheduler.config.num_train_timesteps,
                    (bsz,),
                    device=latents.device,
                )
                timesteps = timesteps.long()

                # Add noise to the latents according to the noise magnitude at each timestep
                # (this is the forward diffusion process)
                noisy_latents = noise_scheduler.add_noise(latents, noise, timesteps)

                encoder_hidden_states = image_encoder(
                    batch["prompt_pixel_values"], output_hidden_states=False
                ).last_hidden_state

                blueprint = batch["blueprint_pixel_values"].to(dtype=weight_dtype)

                # Predict the noise residual
                model_pred = unet(
                    noisy_latents,
                    timesteps,
                    controlnet_cond=blueprint,
                    encoder_hidden_states=encoder_hidden_states,
                ).sample

                # Get the target for loss depending on the prediction type
                if noise_scheduler.config.prediction_type == "epsilon":
                    target = noise
                elif noise_scheduler.config.prediction_type == "v_prediction":
                    target = noise_scheduler.get_velocity(latents, noise, timesteps)
                else:
                    raise ValueError(
                        f"Unknown prediction type {noise_scheduler.config.prediction_type}"
                    )
                loss = F.mse_loss(model_pred.float(), target.float(), reduction="mean")

                # Gather the losses across all processes for logging (if we use distributed training).
                avg_loss = accelerator.gather(loss.repeat(args.train_batch_size)).mean()
                train_loss += avg_loss.item() / args.gradient_accumulation_steps

                accelerator.backward(loss)
                if accelerator.sync_gradients:
                    params_to_clip = (
                        itertools.chain(unet.parameters(), image_encoder.parameters())
                        if args.train_image_encoder
                        else unet.parameters()
                    )
                    accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
                optimizer.step()
                lr_scheduler.step()
                optimizer.zero_grad(set_to_none=args.set_grads_to_none)

            # Checks if the accelerator has performed an optimization step behind the scenes
            if accelerator.sync_gradients:
                progress_bar.update(1)
                global_step += 1
                accelerator.log({"train_loss": train_loss}, step=global_step)
                train_loss = 0.0

                if accelerator.is_main_process:
                    if global_step % args.checkpointing_steps == 0:
                        # _before_ saving state, check if this save would set us over the `checkpoints_total_limit`
                        if args.checkpoints_total_limit is not None:
                            checkpoints = os.listdir(args.output_dir)
                            checkpoints = [
                                d for d in checkpoints if d.startswith("checkpoint")
                            ]
                            checkpoints = sorted(
                                checkpoints, key=lambda x: int(x.split("-")[1])
                            )

                            # before we save the new checkpoint, we need to have at _most_ `checkpoints_total_limit - 1` checkpoints
                            if len(checkpoints) >= args.checkpoints_total_limit:
                                num_to_remove = (
                                    len(checkpoints) - args.checkpoints_total_limit + 1
                                )
                                removing_checkpoints = checkpoints[0:num_to_remove]

                                logger.info(
                                    f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
                                )
                                logger.info(
                                    f"removing checkpoints: {', '.join(removing_checkpoints)}"
                                )

                                for removing_checkpoint in removing_checkpoints:
                                    removing_checkpoint = os.path.join(
                                        args.output_dir, removing_checkpoint
                                    )
                                    shutil.rmtree(removing_checkpoint)

                        save_path = os.path.join(
                            args.output_dir, f"checkpoint-{global_step}"
                        )
                        accelerator.save_state(save_path)
                        logger.info(f"Saved state to {save_path}")

                    if (
                        args.validation_prompt is not None
                        and global_step % args.validation_steps == 0
                    ):
                        image_logs = log_validation(
                            vae,
                            image_encoder,
                            clip_image_processor,
                            unet,
                            args,
                            accelerator,
                            weight_dtype,
                            global_step,
                        )

            logs = {"loss": loss.detach().item(), "lr": lr_scheduler.get_last_lr()[0]}
            progress_bar.set_postfix(**logs)

            if global_step >= args.max_train_steps:
                break

    # Create the pipeline using using the trained modules and save it.
    accelerator.wait_for_everyone()
    if accelerator.is_main_process:
        unet = accelerator.unwrap_model(unet)
        if args.train_image_encoder:
            image_encoder = accelerator.unwrap_model(image_encoder)
        pipeline = StableDiffusionReferenceOnlyPipeline.from_pretrained(
            args.pretrained_model_name_or_path,
            vae=vae,
            image_encoder=image_encoder,
            clip_image_processor=clip_image_processor,
            unet=unet,
            revision=args.revision,
            torch_dtype=weight_dtype,
        )
        pipeline.save_pretrained(args.output_dir)

        if args.push_to_hub:
            save_model_card(
                repo_id,
                image_logs=image_logs,
                base_model=args.pretrained_model_name_or_path,
                repo_folder=args.output_dir,
            )
            upload_folder(
                repo_id=repo_id,
                folder_path=args.output_dir,
                commit_message="End of training",
                ignore_patterns=["step_*", "epoch_*"],
            )

    accelerator.end_training()

Expected behavior
find_unused_parameters are not needed and the training can be done normally

To clarify, are you saying that with find_unused_parameters=False, you get the error that you pasted above but with find_unused_parameters=True, you run out of memory? If so, could you try some steps to reduce memory, like lowering batch size, and try to run with find_unused_parameters=True?

@BenjaminBossan Is there any way to minimize VRAM overhead without finding_unused_parameters, such as changing the code of CLIPVisionModel of transformers? Is this a problem with some codes in the source code? I found out that after find_unused_parameters I had to reduce the batchsize a lot

I found out that after find_unused_parameters I had to reduce the batchsize a lot

We only need this for debugging, hopefully this allows us to identify which part of the model causes the problem, fix it, then you can turn it off again.
Alternatively, you could try running your script with the environment variable TORCH_DISTRIBUTED_DEBUG=DETAIL set and see if that avoids the memory overhead (probably not, I'm not sure).

@BenjaminBossan
I remember I added this debug parameter. It seems to be a problem with this model.vision_model.post_layernorm.bias, vision_model.post_layernorm.weight
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by
making sure all `forward` function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameters which did not receive grad for rank 0: vision_model.post_layernorm.bias, vision_model.post_layernorm.weight
Parameter indices which did not receive grad for rank 0: 389 390

Could you try if setting model.vision_model.post_layernorm.requires_grad_(False) solves the error?

@BenjaminBossan It works. Thanks. It turns out that gradient_checkpointing cannot be used with "ddp find_unused_parameters" at the same time, raising
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function . Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example , if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations. Parameter at index 385 with name vision_model.encoder.layers.23.mlp.fc2.weight has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration.
I can now use gradient_checkpointing Okay. Is the above error report normal? Does gradient_checkpointing not support simultaneous use with ddp find_unused_parameters? Is there any wrong usage in my code?

@BenjaminBossan It ran successfully and looks good. So the best way is to set this layer to have no gradient?

I don't know the model architecture, so I can't say if it is expected for this layer to have no gradients or not. If it is expected, then yes, this is valid solution. If it is unexpected, you'd need to investigate why the layer has no gradients.

@BenjaminBossan I think I know why it has no gradient anymore. I'm using last_hidden_state and it doesn't go through post_layernorm. post_layernorm is used to calculate pooler_ouput. This should be the reason. thanks


Typeahead presence bubbles appear to be a lie

https://chat.zulip.org/#narrow/stream/9-issues/topic/typeahead.20presence/near/1133743 has details, but basically it appears that something is wrong with the logic for computing the presence circles used in our mention typeahead.
I suspect some sort of caching issue (perhaps internal to the typeahead library).

I had tried debugging this when it was reported, but I was not able to find something helpful here. But for displaying presence circles in typeahead I have used helper get_user_circle_class, from buddy_data so I think it may be some problem with how it is calculated in buddy_data but another thought is that other places like pm list  also use same helper. So it may be a problem due typeahead caching(as tim pointed). I posted my observations here in case its helpful if anyone wants to look at this.

@zulibbot claim

This is likely fixed by #17777.

Closing as this has not been noticed since #17777, so we're pretty confident that fixed it.


Phantom doesn't show browser report

Page is blank and console shows:

Steps to reproduce:

meteor create app (version 1.6.1)
meteor add meteortesting:mocha
meteor npm i --save-dev phantomjs-prebuilt

$ TEST_WATCH TEST_BROWSER_DRIVER=phantomjs meteor test --driver-package meteortesting:mocha

repro: https://github.com/lorensr/meteor-mocha-chrome-repro/tree/bug/phantom

@lorensr If you open the tests in your browser, this browser will run the tests.
When defining TEST_BROWSER_DRIVER you tell the system that you want to run the client-tests in a specific browser - phantomjs in your case. This browser is then starts headless, runs all the tests and writes the report out on the console, which is caught by the browser-runner on the server and printed.
But in your case, you opened the application in - I guess it's chrome? This means that what you see in the console there is the result of the client-tests run in chrome.
Please show the full output of the command on the console. This should either include a warning of why the tests couldn't run on phantom or the test results.
@aldeed I guess we need a more declarative way of describing this step in the manual: https://github.com/meteortesting/meteor-mocha#run-client-tests-in-any-browser-manually
It should state more clearly that you actually are running the tests in the browser you're executing it in. Maybe we should also add a div showing a clear warning if TEST_BROWSER_DRIVER is defined which should state that you most-likely are running this in the wrong browser 😅

Maybe we should also add a div if TEST_BROWSER_DRIVER is defined which shows a clear warning stating that you most-likely are currently running the tests in the wrong browser

Makes sense to me

Since we do not ask the browser to click anywhere on the screen, record screenshots or the-like, this shouldn't have any affect to the outcome of any test, right?
Does this means we could actually always show it if TEST_BROWSER_DRIVER is set?

Closing this ticket because Phantom is deprecated (https://github.com/meteortesting/meteor-browser-tests#phantomjs)


Look up table Algorithm implementation for each table

input: Different table by table,
When approaching a row, the sum of cards for HARD TOTALS and SURENDER, and the combination of cards for SOFT TOTALS and PAIR SPILITING
Output: Return the KEY provided at the bottom of the table.
seperate the function for each tables.
refer this link(https://www.blackjackapprenticeship.com/blackjack-strategy-charts/)

For your information: If you think Blackjack rule is too complicated while implementation, you can assume that a player can take only a basic action, stand or hit (which means that double, split, and surrender are not allowed). In this case, you would refer only HARD TOTALS and SOFT TOTALS.

Everything has been processed and ready for use. Make sure that (pip install , opencv-python)  are available in your IDE. The photos of the output have been uploaded too within the file (Embedded Systems&IoT).

@FaltoGH @khalikovj23 For notice, do not close the issue before i check and agree.
Alright, my bad. So sorry!

The code is ready and will be uploaded with the output results. Thank you so much for the time given and kindly refer to the output results and close the issue respectively.


add ui shell component - no style

Hi ,
I am testing the UI Shell component on my local.
Mine is angular project, so I am referring the storybook  here in https://angular.carbondesignsystem.com/?path=/story/ui-shell--header
After imported UIShellModule and just copy the Header template to my html.
And see how it looks is below:

That seems the style doesn't load successfully.
But I can not find css of "bx--header" in my carbon components in  "/node_modules/carbon-components/css/carbon-components.min.css".
Any ideas about this issue?
Thanks very much!

Since it's experimental you'll need to load the carbon components scss, and set up your styles.scss like so:
$feature-flags: (
	ui-shell: true
);

@import '~carbon-components/scss/globals/scss/styles';

Our carbon angular starter repo has the style imports configured, you'd just need to add the feature-flags

Sorry for  response late.
I cloned the carbon-angular-starter repo to my local and   added the code  $feature-flags: ( 	ui-shell: true );   @import '~carbon-components/scss/globals/scss/styles';     to my styles.scss.
But the result  is that  the chrome can load the style, but not for Firefox.
Mine is Mac OS.Please help to have a look.
Thanks very much!


Maple Finance Schema: 1.3.0 Subgraph: 1.1.0 Methodology: 1.0.0 totaldepositbalanceUSD incorrect

totaldepositbalanceUSD should match totalvaluelockedUSD, reporting below that, could be reporting totalborrowedbalanceUSD

@tnkrxyz would you take on this?

@tnkrxyz would you take on this?

of course

Looking at the implementation, the difference is due to the methodology (documented in README.md) in defining totaldepositbalanceUSD and  totalvaluelockedUSD:
totalvaluelockedUSD = Cumulative Deposit USD - Cumulative Withdraw USD - Cumulative Pool Losses USD and
totaldepositbalanceUSD = Total Deposit Balance USD + Total Stake Balance USD + Total Unclaimed Supplier Interest USD
The implementation takes into account staking; not sure whether this should be included (I haven't included in the few lending protocols I implemented). It seems to me Total Unclaimed Supplier Interest USD should not be included.
Anyone can clarify whether they are intended?

@ishraq8

@tnkrxyz Yes you are correct. Maple, like all other lending protocols should not include staking as part of their TVL. This inconsistency will cause problem when making dataapps. I've put in a pull request to correct this issue in the readme. Will also need to be corrected at code level.
https://github.com/messari/subgraphs/pull/1086

@ishraq8 , are you going to make the change to the code? If not, I can do it.

@tnkrxyz Please go ahead and change the code, thank you! Dylan is giving me a crash course on subgraph coding later this week so not super comfortable making changes til then :).

OK, done.

thanks @tnkrxyz, appreciate your help!


"mtrand.pyx", line 936, in numpy.random.mtrand.RandomState.choice

I am facing the below error when I start training. There was no issue before.
Begin training!
Traceback (most recent call last):
File "/content/drive/MyDrive/yolact/./train.py", line 504, in
train()
File "/content/drive/MyDrive/yolact/./train.py", line 270, in train
for datum in data_loader:
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 630, in next
data = self._next_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 694, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/content/drive/MyDrive/yolact/data/coco.py", line 94, in getitem
im, gt, masks, h, w, num_crowds = self.pull_item(index)
File "/content/drive/MyDrive/yolact/data/coco.py", line 158, in pull_item
img, masks, boxes, labels = self.transform(img, masks, target[:, :4],
File "/content/drive/MyDrive/yolact/utils/augmentations.py", line 688, in call
return self.augment(img, masks, boxes, labels)
File "/content/drive/MyDrive/yolact/utils/augmentations.py", line 55, in call
img, masks, boxes, labels = t(img, masks, boxes, labels)
File "/content/drive/MyDrive/yolact/utils/augmentations.py", line 309, in call
mode = random.choice(self.sample_options)
File "mtrand.pyx", line 936, in numpy.random.mtrand.RandomState.choice
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (6,) + inhomogeneous part.

This issue got resolved when I downgraged numpy to 1.23.2.

thanks


test: add example for sg rule self reference

Description
Adding new security group example to the examples/default to show (and test) how to create a rule within security group that has a remote pointed to the same parent security group (self-reference).
Release required?

[ ] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

:tada: This PR is included in version 2.6.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket: