Is it possible to output Tm scores?

Hi there,
I was wondering if it's possible to output the tm-scores between proteins. Also what exactly is the distance matrix outputted?
All the best

I could be confused but is the pairwise tm score of proteins the same as the tm score over a multi alignment? I look at the reported tm score for the AAA family in Homstad computed my mtm align 0.622. They also output a pairwise tm score but this seems to be different. Is it possible to compute the tm score for the multi alignment as well as for the pairwise. Thanks again.

Any update on this question?

In general, the pairwise scores are different from the multiple alignment scores. The pairwise score is the optimum alignment score (based on the distance matrix and gap parameters) between the two structures, however, the multiple alignment algorithm uses heuristics (guide tree and progressive alignment step) to compute the multiple alignment efficiently.
For the second part of your question, yes you can get pairwise tm-scores from the multiple sequence alignment but as I mention above, this won't be the most optimum result between the pairs as the multiple sequence alignment is meant to be used to compare all structures at once.

Okay this is what  want to know. I'm trying to compare the reported mtm-align results with caretta just to verify. The homstrad dataset has reported tm score for multi alignment of proteins in the families as well m-Tm align also has these reported scores. I am trying to do the same for caretta and obviously I'm missing something. What is the function to get the multiple alignment  tm score?

Also I think I'm missing something just to clarify the pairwise score gotten from caretta isn't the same as the mtm-align pairwise matrix? As mtm-align is very time consuming and caretta is quite fast I was using caretta to get tm-scores between proteins.

Okay this is what want to know. I'm trying to compare the reported mtm-align results with caretta just to verify. The homstrad dataset has reported tm score for multi alignment of proteins in the families as well m-Tm align also has these reported scores. I am trying to do the same for caretta and obviously I'm missing something. What is the function to get the multiple alignment tm score?

You can get the pairwise tm-scores from the python api by using the StructureMultiple class method  make_rmsd_coverage_tm_matrix.  If needed, I can make an example gist to show how you can use it. I'll also take a look why there's discrepancy between tm-scores between the two tools.

Also I think I'm missing something just to clarify the pairwise score gotten from caretta isn't the same as the mtm-align pairwise matrix? As mtm-align is very time consuming and caretta is quite fast I was using caretta to get tm-scores between proteins.

Do you mean the commandline score matrix from caretta? If so, then those are caretta scores and not tm-scores. If you used the make_rmsd_coverage_tm_matrix then I will take a look at why there's a discrepancy between the outputs from the two tools (assuming the alignments are identical).

Thanks I know how to get the pairwise but https://yanglab.nankai.edu.cn/mTM-align/benchmark/homstrad.html reports a tm score from multi alignment it doesn't report a pairwise score. ATPase family associated with various cellular activities (AAA) rerports a single score of 0.513. What I want is to report a single tm score for a family not pairwise.

It's mentioned in the mtm-align paper that the single tm-score is the mean of the pairwise tm-scores. Then you could do the same to the pairwise output from caretta.

ahh it is great thank you very much I missed that I thought it was something more complex.

Apparently it isn't exactly the mean its the mean and normalised by the smaller protein.

Sorry this may be a long post with a few questions.

Caretta produces pairwise tm scores. But these scores are not all vs all pairwise alignment like mTM align. As you said this won't be the most optimum result between the pairs as the multiple sequence alignment is meant to be used to compare all structures at once. So the pairwise matrices produced by each method are different? mtm align produces the optimal scores and caretta finds the conserved and variable residues across a set of proteins.

Does normalisation occur twice in mtm align? Tm score formula divides by the length of the interested target as well by d0 the distance scales. This does occur in caretta as well as mTM align to produce the tm score between proteins. But mTM align also performs a secondary normalisation. As reported in their paper "We can calculate the number of structurally equivalent residues (Lali), the associated Root-Mean-Square Deviation (RMSD) and TM-score. Here the TM-score is normalized by the length of the smaller protein. Because the reference MSTAs are available for the HOMSTRAD dataset, we can define another metrics accuracy (ACC)." So if one wants to compare accuracy of Caretta against mTM-align and the HOMSTAD dataset the caretta results would need to be normalised by the smaller protein? If the results from caretta were not to be normalised I think they would be very bad in comparison to mTM-align but I need to be sure.

So I just want to check the ability of caretta to output a similarity score on a cluster of proteins and compare this to mTM align.

Actually going back through the code it seems like the normalisation occurs once in mTM-align with the d0.  Effectively this is the same for both mTM-align and caretta. So I guess the pairwise means are the same.

I went through the mtm align code they seem to only compute the tm score of the smaller protein in the get_TMscore_from_seqxa function. This is also what they state in the paper.
While you guys as seen in the tm_score function in the multiple alignment .py, compute the tm score of both and return the higher. These means you guys don't normalise by the smaller protein.
Also I don't understand why you guys compute the square root of the common coordinates squared in line 47 and 48 in tm_score. This appears to be different than the reported tm algorithm.

Caretta produces pairwise tm scores. But these scores are not all vs all pairwise alignment like mTM align. As you said this won't be the most optimum result between the pairs as the multiple sequence alignment is meant to be used to compare all structures at once. So the pairwise matrices produced by each method are different? mtm align produces the optimal scores and caretta finds the conserved and variable residues across a set of proteins.

mTM align also produces multiple structure alignment TM scores as far as I know, as it is a multiple structure alignment algorithm, though with a different underlying method than caretta. For pairwise TM scores you would have to use TM-align (https://zhanggroup.org/TM-align/) instead.

While you guys as seen in the tm_score function in the multiple alignment .py, compute the tm score of both and return the higher. These means you guys are normalising by both lengths and choosing the max tm score.

Choosing the so called "target" protein for TM-score calculation could be by the shortest or longest or taking the max as we do. We don't use TM score in Caretta so we didn't experiment with changing these.

Also I don't understand why you guys compute the square root of the common coordinates squared in line 47 and 48 in tm_score. This appears to be different than the reported tm algorithm.

This was a bug, I've fixed this now on master thanks for spotting!

I actually am happy with the non pairwise score, I want to get an alignment by cluster.

I just want to compare the tm score given for the homstrad families and the reported tm scores from mtm align.
https://yanglab.nankai.edu.cn/mTM-align/benchmark/homstrad.html, the mtm and the homatad families are reported here.
I've also emailed you the mean pairwise scores that caretta outputs  for each family. The scores are quite bad and I'm sure its due to the normalisation or bugs.

Yeah thanks for all the help. I realise that the homstrad scores I was comparing too were inappropriate. I would make more sense to compare to accuracy.

Okay this is what want to know. I'm trying to compare the reported mtm-align results with caretta just to verify. The homstrad dataset has reported tm score for multi alignment of proteins in the families as well m-Tm align also has these reported scores. I am trying to do the same for caretta and obviously I'm missing something. What is the function to get the multiple alignment tm score?

You can get the pairwise tm-scores from the python api by using the StructureMultiple class method make_rmsd_coverage_tm_matrix. If needed, I can make an example gist to show how you can use it.

Also I think I'm missing something just to clarify the pairwise score gotten from caretta isn't the same as the mtm-align pairwise matrix? As mtm-align is very time consuming and caretta is quite fast I was using caretta to get tm-scores between proteins.

Do you mean the commandline score matrix from caretta? If so, then those are caretta scores and not tm-scores. If you used the make_rmsd_coverage_tm_matrix then I will take a look at why there's a discrepancy between the outputs from the two tools (assuming the alignments are identical).

Okay this is what want to know. I'm trying to compare the reported mtm-align results with caretta just to verify. The homstrad dataset has reported tm score for multi alignment of proteins in the families as well m-Tm align also has these reported scores. I am trying to do the same for caretta and obviously I'm missing something. What is the function to get the multiple alignment tm score?

You can get the pairwise tm-scores from the python api by using the StructureMultiple class method make_rmsd_coverage_tm_matrix. If needed, I can make an example gist to show how you can use it.

Also I think I'm missing something just to clarify the pairwise score gotten from caretta isn't the same as the mtm-align pairwise matrix? As mtm-align is very time consuming and caretta is quite fast I was using caretta to get tm-scores between proteins.

Do you mean the commandline score matrix from caretta? If so, then those are caretta scores and not tm-scores. If you used the make_rmsd_coverage_tm_matrix then I will take a look at why there's a discrepancy between the outputs from the two tools (assuming the alignments are identical).

Hello, may I have an example for the usage of  make_rmsd_coverage_tm_matrix? Thank you :)


vulkan-loader: fix include path in pkgconfig description

Fixes wlroots (CC @primeos, @Synthetica9)
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Was this forwarded upstream? See also https://github.com/jtojnar/cmake-snips for support material for issues like this.

Good, should unblock the channel now that nixosTests.sway builds again


added icons to posts

UI looks good, I like the choices of icons!
Could you see why the Netlify deploy is failing for this PR? I suspect you might need to commit whatever is in ../assets/Icons, as well as save any new npm install you added to the package.json file (to do this, use npm install --save {NEW_PACKAGE}) to add the dependency to the package.json file.

Netlify is complaining because ../assets/Icons is missing


firefox-nightly sha256 problem?

Is this a problem in here or a problem on my side? Sorry if I'm asking noobish question.
$ brew cask install firefox-nightly
==> Downloading https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-38.0a1.en-US.mac.dmg
######################################################################## 100.0%
==> Note: running "brew update" may fix sha256 checksum errors
Error: sha256 mismatch
Expected: d0a1f9bf78de8c2b21e0e3fc90e878744699ef086e6dfa14e9551b4e1a618b98
Actual: a3bfe2e52e01c6152f31c054dc0b26128c0482c3d9ee7223e2f88a93d5582336
File: /Library/Caches/Homebrew/firefox-nightly-38.0a1.dmg
To retry an incomplete download, remove the file above.

You’re using an old version of the cask. This problem has been addressed. Please brew update and try again.


rename chest -> shack

I've decided that the name chest isn't really working so this is a proposal to rename this project to shack.

Done with #10


Replace dart.library.js with dart.library.js_interop.

This PR replaces all dart.library.js conditional imports with dart.library.js_interop to make it compatible with Wasm.
Potentially fixes #432

Thanks!


Fix for "in X day(s)" without hours or time of day

This change fixes a "Could not understand command" error that would occur when adding a reminder after a number of days without any additional hours or time of day. For example, "r in 1 day do a thing".

LGTM although note that this code will not be used in favour of the beta branch going forwards

Cool, looking forward to that.  This is easily one of the most useful workflows I've found, so thanks for your work on it.


Color icon doesn't work

If I use the color icon instead of the white icon, the splashview will change the icon to white and splash.
Is it a bug or feature?
Below is the code I use.
let revealingSplashView = RevealingSplashView(iconImage: UIImage(named: "color_icon")!, iconInitialSize: CGSizeMake(60, 60),backgroundColor: MaterialColor.grey.lighten5)

Its definitely not a intented feature. I will try to check tonight to see whats up.
Thanks!

@lancevalour Just remember that I added a iconColor property. If you don't set it it will default to white.
So, for example, you can do:
 revealingSplashView.iconColor = UIColor.blueColor()

This will change the icon color to blue, the example it will look like this:

Let me know if it works. Closing for now.


how to use "createDistributable" create single running instance of application

how to use "createDistributable" create single running instance of the application ?
like exe4j allow only single running instance of the application

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.


getter/setter interface for type-forced implementation.

Search Terms

Suggestion

support getter type in interface.
Use Cases

require implement of getter
Examples

`
interface HasValue<T = any> {
get value(): T;
}
// error: need implement value getter.
abstract class A implements HasValue {
}
// ok
abstract class B implements HasValue {
get value() {
return 123;
}
}
`
Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

As expected. You can add it as an abstract getter i your base class.

In JavaScript getters and setters are just implementation details to property gets and sets. From a external point of view they are simply properties.
So ...
interface HasValue<T = any> {
  get value(): T
}

... is same as writing ...
interface HasValue<T = any> {
  readonly value: T
}

Because that's now an external agent would see it.
So your example becomes something like this
interface HasValue<T = any> {
  readonly value: T;
}

abstract class A<T = any> implements HasValue<T> {
  abstract readonly value: T;
}

// expected error
class B extends A<number> {

}

// no error
class C extends A<number> {
  get value() { return 42 }
}

Playground

This doesn't really fit with our type system because types don't enforce implementation details

@devanshj
yes i don't want allow
class D extends A<number> {
  value = 42
}

And in js getter/setter can be seen as a special type that differ from properties/methods. I'm not pretty sure yet what it should be recognized. But it did has some differences.

There are a lot of implementation details that are theoretically observable from the outside; this is one we've chosen to say are not usually relevant in a way that matters.


boringcrypto: Not using FIPS compliant calls for SHA functions (also should we add md5? for easy proof of FIPS mode on)

Go version
go1.20.12
Output of go env in your module/workspace:
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/tmp/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/tmp/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.12"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2938776592=/tmp/go-build -gno-record-gcc-switches"

What did you do?
After building a golang binary I wanted to prove that boringcrypto was using the openssl library in FIPS mode (my kernel was already in FIPS mode) and the way I had done that for other languages was to attempt an md5 hash and see that openssl threw an error. AFter ralising the the md5() function of golang is not implemented in boringcrypto I set about patching my copy by copying the way the sha1 functions work but for md5. After getting this to work I was surprised when the md5() function returned the hash, instead of returning an erorr. Using LD_DEBUG=symbols I could see that my binary was correctly calling into openssl for both an md5() call and a sha1() call:
		result := md5.Sum([]byte("hash me"))
		fmt.Printf("%x\n", result)
		result2 := sha1.Sum([]byte("hash me"))
		fmt.Printf("%x\n", result2)

when executrd with LD_DEBUG=symbols produced:
...
	  5752:	symbol=MD5_Final;  lookup in file=/lib64/libresolv.so.2 [0]
	  5752:	symbol=MD5_Final;  lookup in file=/lib64/libdl.so.2 [0]
	  5752:	symbol=MD5_Final;  lookup in file=/lib64/libpthread.so.0 [0]
	  5752:	symbol=MD5_Final;  lookup in file=/lib64/libc.so.6 [0]
	  5752:	symbol=MD5_Final;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
	  5752:	symbol=MD5_Final;  lookup in file=/lib64/libcrypto.so.1.1 [0]
17b31dce96b9d6c6d0a6ba95f47796fb
...
	  5752:	symbol=SHA1_Final;  lookup in file=/lib64/libresolv.so.2 [0]
	  5752:	symbol=SHA1_Final;  lookup in file=/lib64/libdl.so.2 [0]
	  5752:	symbol=SHA1_Final;  lookup in file=/lib64/libpthread.so.0 [0]
	  5752:	symbol=SHA1_Final;  lookup in file=/lib64/libc.so.6 [0]
	  5752:	symbol=SHA1_Final;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
	  5752:	symbol=SHA1_Final;  lookup in file=/lib64/libcrypto.so.1.1 [0]
43f932e4f7c6ecd136a695b7008694bb69d517bd

A bit of digging around and I found this section in the openssl fips page (https://www.openssl.org/docs/manmaster/man7/fips_module.html):
Applications written to use the OpenSSL 3.0 FIPS module should not use any legacy APIs or features that avoid the FIPS module. Specifically this includes:

Low level cryptographic APIs (use the high level APIs, such as EVP, instead)

It appears that the MD5_* and SHA1_* functions are considered low level APIs by openssl and so avoid the FIPS module. I tested this theory by creating a small C program that first uses the low level MD5 api and then the haigh level EVP_ MD5 api:
#include <stdio.h>
#include <string.h>
#include <openssl/md5.h>
#include <openssl/des.h>
#include <openssl/fips.h>
#include <openssl/ssl.h>
#include <openssl/evp.h>
#include <openssl/err.h>

void compute_md5(char *str, unsigned char digest[16]);

int main(int argc, char *argv[])
{
    if (argc == 2) {
	printf("Setting FIPS mode to %s\n",argv[1]);
#if OPENSSL_VERSION_MAJOR < 3
        FIPS_mode_set(atoi(argv[1]));
#else
        EVP_default_properties_enable_fips(NULL, atoi(argv[1]));
#endif
    }
    printf("FIPS Mode: %d\n",FIPS_mode());
    char data[] = "hash me";
    unsigned char md_value[EVP_MAX_MD_SIZE];
    unsigned int md_len, i;

    printf("Attempting md5 hash via MD5_ methods.\n");
    unsigned char digest[16];
    compute_md5("hash me", digest);
    for (int i = 0; i < 16; i++)
        printf("%02x", digest[i]);
    putchar ('\n');

    printf("Attempting md5 hash via EVP_ methods.\n");
    const EVP_MD *md = EVP_get_digestbyname("md5");
    if (md == NULL) {
        printf("Unknown message digest %s\n", "md5");
        exit(1);
    }
    EVP_MD_CTX *ctx = EVP_MD_CTX_new();
        if (!EVP_DigestInit_ex(ctx, md, NULL)) {
        printf("Message digest initialization failed.\n");
	ERR_print_errors_fp (stderr);
        EVP_MD_CTX_free(ctx);
        exit(1);
    }
    EVP_DigestUpdate(ctx, data, strlen(data));
    EVP_DigestFinal_ex(ctx, md_value, &md_len);
    for (int i = 0; i < md_len; i++)
        printf("%02x", md_value[i]);
    putchar ('\n');

    return 0;
}

void compute_md5(char *str, unsigned char digest[16]) {
    MD5_CTX ctx;
    MD5_Init(&ctx);
    MD5_Update(&ctx, str, strlen(str));
    MD5_Final(digest, &ctx);
}

Which when ran with FIPS enabled (either via the cmd line parameter, or on a kernel with fips=1) gives the output:
gcc md5_fips_test.c -o md5_fips_test -lssl -lcrypto
./md5_fips_test 1
Setting FIPS mode to 1
FIPS Mode: 1
Attempting md5 hash via MD5_ methods.
17b31dce96b9d6c6d0a6ba95f47796fb
Attempting md5 hash via EVP_ methods.
Message digest initialization failed.
140030451414848:error:060800C8:digital envelope routines:EVP_DigestInit_ex:disabled for FIPS:crypto/evp/digest.c:135:

Which seems to prove that the MD5_ methods, and so presumably all the SHA*_ methods do not go via the fips module. I wasn't able to find a way to configure the system to disallow a one of the SHA* hashes to test my theory with the SHA*_ functions though.
Note I was actually using go-toolset on Redhat, but I think the changes required are in the boringcrypto code in https://github.com/golang/go/blob/master/src/crypto/internal/boring/sha.go and other places.
I also think it would be handy to add an implementation of md5 (would be very simlar to the fixed sha implementation) in boringcrypto so that you can try an md5 hash and see it refused. It would also need to use the EVP_ methods.
What did you see happen?
Covered above, sorry.
What did you expect to see?
I expected to be able to prove that the binary was running openssl in fips mode, by seeing it deny a bad algorithmn, such as md5.

Duplicate of #45565

@seankhliao Thanks for the pointer to #45565 but I don't think this is a direct duplicate of that bug. I can see the crossover where I am asking why md5() is not implemented in boringcrypto and I see https://github.com/golang/go/issues/45565#issuecomment-1379935743 indicates that is working as designed and it is up to the application to not cal md5(), but my bigger concern here is that all the functions in sha.go are using the low level openssl calls, instead of the high level ones that go via the fips provider.

I figured closed bugs won't get any attention and to remove the confusion over md5 and concentrate on the sha issue I rasued a new bug: https://github.com/golang/go/issues/66520


How can I run a previous verion?

Since the update with the delay in BIN sniping I can't make any coins, I was making a very nice profit using the previous version, but now when I download an older version I'm forced to automatically update to the new inferior version?
Is there anyway I can just use the old version?

You can checkout the tag and run locally.  Otherwise, no.  The delay was added in to help prevent bans.


 Open fish shell
2.

If the bug only appears with a specific application, the output of that application and its version:
sir@napkin ~> eval (thefuck --alias)
fish: Expected end of the statement, but instead found a pipe
function fuck -d "Correct your previous console command"   set -l fucked_up_command $history[1]   env TF_SHELL=fish TF_ALIAS=fuck PYTHONIOENCODING=utf-8 thefuck $fucked_up_command THEFUCK_ARGUMENT_PLACEHOLDER $argv | read -l unfucked_command   if [ "$unfucked_command" != "" ]     eval $unfucked_command     builtin history delete --exact --case-sensitive -- $fucked_up_command     builtin history merge ^ /dev/null   end end
                                                                                ^

sir@napkin ~ [1]>

Anything else you think is relevant:
Image in case the text misalignes above

Fish's eval uses source under the hood, so we recommend using it insted:
Quoting Shell-aliases > Fish:

Add this function to config.fish:
thefuck --alias | source

It's right there on README.md 🙂
If you'd like to keep using eval, this should be your line:
eval (thefuck --alias | string join \;)

Please, check the wiki for other tips 😉

Apologies. Thank you.

No worries at all 🙂 You're welcome.
If by any chance you're using Oh My Fish — which I'd highly recommend if you're not — you can install its thefuck plugin, which does all the setup for you 😉


[AIRFLOW-2163] Add HBC Digital to users of airflow

Make sure you have checked all steps below.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-2163

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

+1

@ashb Want to try merging this one?

@ashb Did you merge using the airflow-pr tool or manually merge? If you use the tool, the commit will say "authored by tmccartan and Ash Berlin-taylor committed...". If you manually, merge, I believe it actually creates a commit in your name, which is what I see in the commit log. Any issues running the tool? I ask as I am currently running into some TLS issue running the tool and wonder if it's broken for others as well.

I did merge with the tool, yes. I chose no to squash on this (as it was only one commit).
The Jira issue bit failed because I didn't (at the time) have the Jira python module installed.

Hmn.. I usually don't squash either. I wonder what's different. Usually, the "authored.... committed" line is preferable as it doesn't create extraneous merge commits and uses the "rebase and merge" option in github (i believe). The way it seems to have gone for you is not that route, but merge (which looks the same for single-commit PRs as a squash and merge.)

I squash most of the time. By squashing it, it will amend the commit, and generate a new commit hash. Because the hashes don't match up, the PR in Github appears to be closed instead of merged. Apparently the author was reset on the commit. Something went wrong, because I would expect having the commit under the name of the original author.


CASSANDRA-19676 - Replace Stream iteration with for-loop for StorageProxy

On profiling a write-heavy workload (90% writes) using easy-cass-stress, it became very clear StorageProxy::updateCoordinatorWriteLatencyTableMetric was a hot path that ~15% of the CPU cycles of ModificationStatement::executeWithoutCondition were taken up by (see attached async-profiler image).
We should convert this stream to a simple for loop, as has been discussed recently on the mail list.
Before

After

Great find, thanks for this profile and patch.
Do you have an “after” profile; or can you share before/after perf stats from easy-cass-stress?

Great find, thanks for this profile and patch.
Do you have an “after” profile; or can you share before/after perf stats from easy-cass-stress?

Updated the PR desc with after. This seems rather significant...


Remove any usage of nulls and any nullable values

Because they cannot be checked at compile time, the usage of nullable values is not very safe. Use Optionals instead.

Great suggestion. Let's do this from now on, and we can later refactor older code segments.

Let me move this to v1.4.


🚀 [Feature]: fiberzap accepts custom fields

Feature Description
I'd like to have a custom field that are user defined zap.Fields.

For custom structured logging
To add user defined fields
Support different entry formats, etc.

Additional Context (optional)
Currently I have an client that want's http headers response into the logging file, that's post processed later. They want all headers, now the fiberzap doesn't have an option for it. Instead of just adding a "respHeaders" field it'll be more extensible to add a custom field that are user definied fields
Code Snippet (optional)
package main

import (
	"log"

	"github.com/gofiber/contrib/fiberzap/v2"
	"github.com/gofiber/fiber/v2"
	"go.uber.org/zap"
)

func main() {
	app := fiber.New()
	logger, _ := zap.NewProduction()

	app.Use(fiberzap.New(fiberzap.Config{
		Logger: logger,
		Fields: []string{"custom"},
		CustomFieldFunc: func(c *fiber.Ctx) []zap.Field {
			var fields []zap.Field

			headers := make(map[string]interface{})

			c.Response().Header.VisitAll(func(k, v []byte) {
				headers[string(k)] = string(v)
			})

			fields = append(fields, zap.Any("headers", headers))

			return fields
		},
	}))

	app.Get("/", func(c *fiber.Ctx) error {
		return c.SendString("Hello, World!")
	})

	log.Fatal(app.Listen(":3000"))
}

Checklist:

[X] I agree to follow Fiber's Code of Conduct.
[X] I have checked for existing issues that describe my suggestion prior to opening this one.
[X] I understand that improperly formatted feature requests may be closed without explanation.

@mirusky Hi, I have literally just ran into a use case for this, have you started working on the feature at all? I don't want to intrude but I have written the code for it before I saw this open issue.

My use was using this to inject trace ids into the log, so I can link together APM traces in kibana to specific log records eg.
FieldsFunc: func(c *fiber.Ctx) []zap.Field {
      tr := apm.TransactionFromContext(c.Context())
      traceId := tr.TraceContext().Trace.String()

      return []zap.Field{zap.String("trace.id", traceId)}
},

@Jamess-Lucass I didn't started at all, busy week.
I liked your approach, I was thinking on just add a new case statement for custom and call it when user pass custom on fields values. But the way you did gives more flexibility.


Bring expo_plural_forms back into this library

See https://github.com/elixir-gettext/expo_plural_forms/issues/37.

Pull Request Test Coverage Report for Build 0a7c3b3f65bfeb20ff52764dfae2aa5f9e0055e8-PR-102

39 of 42 (92.86%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-22.2%) to 77.807%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/expo/plural_forms.ex
27
28
96.43%

lib/expo/plural_forms/syntax_error.ex
0
2
0.0%

Totals

Change from base Build a44c3e4f57b75730c52e6462b130c1874b3f34f8:
-22.2%

Covered Lines:
582

Relevant Lines:
748

💛  - Coveralls


Change "Restart Chat Session" icon and add a confirmation

This changes the current icon from reload (which is too close to a browser reload) to clear-all (which isn't perfect but looks more destructive). It also adds a confirmation dialog:

Before
After

Confirmation

Test plan

Viewed the button
Clicked the button, verify prompt cancel does nothing
Clicked the button, verify "Restart Chat Session" does the action
Typed /reset and verify it doesn't ask for confirmation

Nice!


Points/positions created are not correctly located in viewer

When p1 is created at [0,0,0], and a point is created, it appears correctly:

When p2 is created at [10,10,10], but no point is created, it appears correctly.
But when a point is created for p2 also, p1 appears to be shifted to below the origin and definitely is no longer at the world origin:

For testing:
points_misplaced.zip


Update repository after transfer

See https://github.com/senzing-garage/garage-roadmap/issues/3

Pull requests:

https://github.com/Senzing/go-sdk-abstract-factory/pull/137


No button in Figma Design

In "How to use Tkinter Designer" in https://github.com/ParthJadhav/Tkinter-Designer/blob/master/docs/instructions.md, "Formatting Your Figma Design" -> 2. Element Guide -> 7. Button:  It talks about the creation of a button but not how. In fact, and ironically enough, this is the only thing that I needed an explanation for! :) I can't find anything about buttons ...

I see, let me add some more information on it.
Sorry for inconvenience.

Hey @AlkisPis ,
Can you check the updated instructions. They more informative now.

Hi
But I just did that. I was redirected to https://github.com/ParthJadhav/Tkinter-Designer/blob/master/docs/instructions.md. Are there other instructions available elsewhere?
Alkis Piskas
Sent from Mail for Windows 10
From: Parth Jadhav
Sent: Σάββατο, 19 Ιουνίου 2021 12:54 μμ
To: ParthJadhav/Tkinter-Designer
Cc: AlkisPis; Mention
Subject: Re: [ParthJadhav/Tkinter-Designer] No button in Figma Design (#41)
Hey @AlkisPis ,
Can you check the updated instructions. They more informative now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

There is a youtube video available. I added the link to that video with timestamp in the instructions.
Check the updated elements guide.

Thanks. I checked the updated guide. I didn’t see any apparent difference. As for the “button” section. It still doesn’t tell how to do it. I wonder why’s that so difficult.
Now, about the video.  I don’t lke videos so much as a means for instructions. They are too slow as a communication form. Anyway I checked it and I had permently a large strip across the part explaining the creation of the button (quite complicated, indeed!). What’s that strip and why should it be there? It makes watching unbearable!
Parth, I am sorry I have to bring up all that. I could well avoid all that altogether and say nothing. But you have done a very good job on this, and also Figma is just great! So, I believe it would be pity to just forget about the whole Tkinter Designer. Wouldn’t it be?
Alkis
Sent from Mail for Windows 10
From: Parth Jadhav
Sent: Σάββατο, 19 Ιουνίου 2021 1:38 μμ
To: ParthJadhav/Tkinter-Designer
Cc: AlkisPis; Mention
Subject: Re: [ParthJadhav/Tkinter-Designer] No button in Figma Design (#41)
There is a youtube video available. I added the link to that video with timestamp in the instructions.
Check the updated elements guide.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Yes, I agree about the tutorial. I am going to make a new and better tutorial.
But before that I am consistently working on the experimental branch to make it more easier to work with. The naming system would be completely replaced with a plug-in. It would allow you select the type of element it is. The text if the buttons would be now dynamic unlike now which is a image. Also I am working on removing the background for button process.
So overall there would be a lot of changes and improvements in Tkinter Designer in upcoming version.
Also it's important to point out things which are not good. So any critics are welcome 😃

Thanks about all that and esp. your willingness, eagerness I would say,  to make improvements. It is a sign of a wise and intelligent person!
As for the critic, I wouldn’t put my comments into that category. I would rather put them under feedback from beta testing! Besides, we are programmers, aren’t we? 😀 (And I have done really a lo-o-o-o-t of beta testing in my life!)
BTW, have you checked the generated code I sent you?
Cheers
Sent from Mail for Windows 10
From: Parth Jadhav
Sent: Σάββατο, 19 Ιουνίου 2021 5:01 μμ
To: ParthJadhav/Tkinter-Designer
Cc: AlkisPis; Mention
Subject: Re: [ParthJadhav/Tkinter-Designer] No button in Figma Design (#41)
Yes, I agree about the tutorial. I am going to make a new and better tutorial.
But before that I am consistently working on the experimental branch to make it more easier to work with. The naming system would be completely replaced with a plug-in. It would allow you select the type of element it is. The text if the buttons would be now dynamic unlike now which is a image. Also I am working on removing the background for button process.
So overall there would be a lot of changes and improvements in Tkinter Designer in upcoming version.
Also it's important to point out things which are not good. So any critics are welcome 😃
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Thanks about all that and esp. your willingness, eagerness I would say, to make improvements. It is a sign of a wise and intelligent person! As for the critic, I wouldn’t put my comments into that category. I would rather put them under feedback from beta testing! Besides, we are programmers, aren’t we? 😀 (And I have done really a lo-o-o-o-t of beta testing in my life!)

Nice to hear that! As we already talked, If you love it then you'll spend your life fixing it.
Oh, I See. You must have good experience of beta testing. Can I ask you to test the experimental branch once it's ready ?

Hi
Re “test the experimental branch once it's ready”:  Of course
Alkis
From: Parth Jadhav
Sent: Σάββατο, 19 Ιουνίου 2021 6:03 μμ
To: ParthJadhav/Tkinter-Designer
Cc: AlkisPis; Mention
Subject: Re: [ParthJadhav/Tkinter-Designer] No button in Figma Design (#41)
Thanks about all that and esp. your willingness, eagerness I would say, to make improvements. It is a sign of a wise and intelligent person! As for the critic, I wouldn’t put my comments into that category. I would rather put them under feedback from beta testing! Besides, we are programmers, aren’t we? 😀 (And I have done really a lo-o-o-o-t of beta testing in my life!)
Nice to hear that! As we already talked, If you love it then you'll spend your life fixing it.
Oh, I See. You must have good experience of beta testing. Can I ask you to test the experimental branch once it's ready ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Thanks a lot @AlkisPis.


Adding Kinematics Python Program

Kinematics program written in python (by lwc24).

If you do 'git pull upstream master' and then do another commit and push then that might trigger it to do the checks?

Yeah, looks like this branch drifted out of sync with master.

:+1:

So this looks good to me with the following comments.

Why is "Kinematics.py" in the root of the repo? I'd have expected it to be under examples like similar scripts.
Python scripts conventionally don't use capitalisation in filenames.
There should be a docstring at the beginning of the program describing it. (See other examples.)
Ideally the docstring should include a doctest to run the program.

I'd ideally not have merged this. I'll open a PR which addresses the more serious issues.

So, it would've been a :-1: from me as is.

Good work on the script, though. It's just not where I'd have put it :).


What's the deal with Pyzor?

I receive many messages with this type of spam status:
X-Spam-Report:
 *  2.0 PYZOR_CHECK Listed in Pyzor
    *      (https://pyzor.readthedocs.io/en/latest/)

The link referenced is simply to the Pyzor documentation.  It's hard to figure out how Pyzor is involved in the spam checking of PMiaB.  It's not a cloud service, so where does Pyzor run and how does one tell it to chill and get out of the way?  All the so called Pyzor listings are false positives, that is they are messages from colleagues and well known people.
Thanks

If PMiaB is meant for "ordinary" tech-savvy people to be used, then surely we should create a resource to guide them through the spam filtering management, especially since there are no GUI options to do this?  There spamassassin, pyzor & greylisting that I'm aware of, but the level of false positives is just too high.  We can't just leave it to itself, it has to be tuned.

It's not a cloud service, so where does Pyzor run and how does one tell it to chill and get out of the way?

I have scanned for Pyzor in the config file and found this:
# cat /etc/spamassassin/pyzor/servers
public.pyzor.org:24441

So the Pyzor on PMiaB is client that reports to and/or check public.pyzor.org on port 24441.
This leads to the next question: What's the interaction between spammassassin and pyzor.  I suppose I should check the former project for details...


How to change from initially configured wifi to wired

I initially set up my Pi3 on a WiFi network. Now I'd like to run it on my wired network. Are there modifications I must make to the pivpn config or is this exclusively a RP config issue? When I attempt to use the wire, I seem to get a connection, but there is no throughput.
If I'm not mistaken your documentation talks about 2 channels. Perhaps one is working and the other stumbles when I switch??
Unbelievably easy initial setup. Thanks for creating this!
Paul

if your wifi had a static IP and you gave eth0 the same static i don't think you have to change anything. just reboot.  if you gave it a different static, we might need to talk more.

I haven't assigned a static IP at all to either. All I did was set up wifi and go. I did assign a static IP in my router however so port forwarding would go somewhere consistent. My next step is to give the pi a static IP and see how it goes. Sounds like the static IP is critical.
These are the hazards of targeting the unschooled masses.
Not sure I can dig into this till mid-week, but I'll let you know what happens.
Paul
On Thu, May 12, 2016 at 5:59 PM<EMAIL_ADDRESS>wrote:
if your wifi had a static IP and you gave eth0 the same static i don't think you have to change anything. just reboot. if you gave it a different static, we might need to talk more.
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub

you don't need a static just really preferred.  you can assign it a dhcp reservation in your router and this way you can leave the pi doing dhcp but it'll never change.
but regarding this issue at hand run
ifconfig
and look at your eth0 inet addr.
then go into the server.conf file in /etc/openvpn and you'll see a line like

your local subnet
push "route 192.168.x.x <IP_ADDRESS>"

change that IP to your inet addr.  it really just needs your subnet but can be your addr too won't matter.
finally check iptables is good
iptables -t nat -L

actually maybe just running this if your wired is eth0 will fix it for you.
sudo iptables -t nat -A POSTROUTING -s <IP_ADDRESS>/24 -o eth0 -j MASQUERADE

As of yesterday, it looked like it was working on the wired connection without my doing anything other than looking at the places you suggested.  I wanted to play with it a little more before declaring victory.
Sent from Yahoo Mail on Android
On Sun, May 15, 2016 at 8:11 AM<EMAIL_ADDRESS>wrote:
actually maybe just running this if your wired is eth0 will fix it for you.
sudo iptables -t nat -A POSTROUTING -s <IP_ADDRESS>/24 -o eth0 -j MASQUERADE
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub

@paulzak everything working?

Yes. I played with it a bit and it's working now. It's a bit over my head and it worked inconsistently but it's up right now.
I did 2 things which I'm fairly sure had no impact, but seemed right anyway.

I was getting an error in the OpenVPN log that complained about my subnet being non-canonical so I changed the IP in the push statement in /etc/OpenVPN server.conf to <IP_ADDRESS> <IP_ADDRESS> which eliminated the log entry, but really didn't seem to correct the problem.
Then, I wondered if the DHCP reservation needed to specify the MAC address of the eth0 interface. It was set to the wlan so I corrected that too.
After that, it was still inconsistently working. Just to be clear, in case you hear this from another user, it always works on wlan. When you plug in eth and reboot, the pi comes back after the boot and is managable. The VPN will connect, but no data will pass through the interface. Then, for whatever reason, if you keep trying to reboot, it will eventually connect and work.
I'm OK if you close the issue and I'll follow github to see if this comes back, but I hate to bug you anymore on this. Let me know if you still want me to try something. I did not alter the routing as you suggested in your last note.
Thanks again for all your help.

Paul
On Fri, May 20, 2016 at 5:15 AM<EMAIL_ADDRESS>wrote:
@paulzak everything working?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

Could make this an enhancement to support changing interface.  Not sure it adds a ton of value, I don't imagine most need to change their interface, but will mark as such...

@0-kaladin: wouldn't it be easier to tell users to reinstall?

I had the same issue and ran the two commands suggested above and it worked.

actually maybe just running this if your wired is eth0 will fix it for you.
sudo iptables -t nat -A POSTROUTING -s <IP_ADDRESS>/24 -o eth0 -j MASQUERADE
sudo netfilter-persistent save


[docs] Adjust markdown docs generator to move files to dedicated directory

Checklist

[x] I've run bundle exec rspec from the root directory to see all new and existing tests pass
[x] I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
[x] I've read the Contribution Guidelines
[x] I've updated the documentation if necessary.

Motivation and Context
Adjust markdown_docs_generator to move files to docs/generated directory in docs repository
This PR should be considered together with: https://github.com/fastlane/docs/pull/1095
Description
Adjust update_docs lane in Fastfile and generate! in markdown_docs_generator to move auto-generated files to docs/generated directory in docs repository
Urls in actions documentation will not change. There is a PR in docs repository to copy files from generated to correct locations
Testing Steps
Update Gemfile and run bundle install:
gem "fastlane", :git => "https://github.com/lucgrabowski/fastlane.git", :branch => "lucgrabowski-move-autogenerated-files-to-dedicated-directory"

@joshdholtz Should "Customize branch of docs repository" commit be reverted? It's not needed.
And thanks for help 🚀

@googlebot I consent.

@joshdholtz Thanks for help with this PR 💪 and fixing mkdocs config :)


Added forgotten parameter

Simplify the parsing logic of makejdk_any_platform_args (#3957) have forgot to add mandatpory parameter to usage of
removeSystemModulesHashBuilderParams in
tooling/reproducible/comparable_patch.sh#L351

CI check failures must be unrelated problem

Indeeed. The linuxes are
 { [2 bytes data]
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

windows are
 Downloading Temurin release of boot JDK version 17 failed.
Attempting to download GA release of boot JDK version 17 from https://api.adoptopenjdk.net/v3/binary/latest/17/ga/windows/x64/jdk/hotspot/normal/adoptopenjdk
Error: Process completed with exit code 4.


JSONObject of方法增加动态参数

What this PR does / why we need it?
现在JSONObject of方法最多支持5对键值，日常开发过程中往往会约到大于5对的情况，增加动态参数方便使用。
Summary of your change
Please indicate you've done the following:

[ ] Made sure tests are passing and test coverage is added if needed.
[ ] Made sure commit message follow the rule of Conventional Commits specification.
[ ] Considered the docs impact and opened a new docs issue or PR with docs changes if needed.

建议这样实现
   public static JSONObject of(
            String k1,
            Object v1,
            String k2,
            Object v2,
            String k3,
            Object v3,
            String k4,
            Object v4,
            String k5,
            Object v5,
            Object... kvArray
    ) {
          //
    }

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.lilanlong seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


[buildsystems] applocal.py incorrectly modifies the dynamic library name

Describe the bug
After installing dependencies with arm64-osx-dynamic triplet, the dynamic library name in the app bundle was incorrectly changed.

The first line displayed by otool -L should be the name of the dynamic library.

Environment

OS: macos
Compiler: revision

To Reproduce

Install libcurl using manifest mode with the arm64-osx-dynamic triplet and define the product as an app bundle:

add_executable(demo MACOSX_BUNDLE main.cpp)

After the build is complete and the app bundle is generated, vcpkg calls applocal.py to put the dependencies into the bundle.

I debugged applocal.py and printed relevant information.
I am confused about this line of code：

popen_args = ['install_name_tool', '-id', dep_rpath, binary]

The script executes the following command:

['install_name_tool', '-id'<EMAIL_ADDRESS>'/Users/runner/terminal/build/terminal.app/Contents/Frameworks/libcurl-d.4.8.0.dylib']

According to the documentation of install_name_tool:

-id name
Changes the shared library identification name of a dynamic shared library to name.  If the Mach-O binary is not a dynamic shared library and the -id option is
specified it is ignored.

Because the name was changed incorrectly, fixup_bundle did not work properly because it needs to do some logic based on the name.

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."


TableView ScrollBug

(sorry, my english is very poor)
hi jurezove
your source is very good , but i find some bug
as the last cell  and  typing ..
tableview is crazy gone...

Hello @GenoplanDennis !
Thanks for pointing this out. I haven't found a definitive solution yet, but it looks like it's an iOS bug. Disabling animations and re-setting the tableView's contentOffset seems to fix the issue. Can you try it out?
Take a look at the diff: https://github.com/jurezove/self-sizing-uitextview-in-a-table/blob/9ab3871c47e9fbd24c9a114faead5134b7ad5883/UITextView featuring Auto Layout/ViewController.swift#L35


Cleaning up Github actions tasks.

ChangeLog:

Adding a job to simply build the project
Adding a release tag on create will build a linux,mac,windows artifact and attach it to the release.

I needed a few different builds for other OSes and wrote some CICD updates that allows it to attach the binary once you create a new release.  I figured I'd share if you find it useful.  If not feel free to disregard.

I'm confused as to why you are building for linux, mac, and windows. This is a tool for the reMarkable tablet that is meant to run on the device itself. The existing build was already targeting the correct architecture.

I'm confused as to why you are building for linux, mac, and windows.

Yeah, pretty sure this is a drive-by to get some github achievement stickers.

I figured I'd share if you find it useful. If not feel free to disregard.

Thanks for your submission however it's not really needed for this tool due to it being highly specific to the reMarkable tablet operating system.  Closing this.


new 'Ready' state

since 5.5 LXD has a new user controlled 'Ready' state, any way we can handle that, it should be a quick change
Sep 28 12:00:46 lxd10.2e-systems.com lxd-snapper[726652]:         2: unknown variant Ready, expected one of Aborting, Running, Starting, Stopped, Stopping at line 1 column 2157213
more about ready state at https://discuss.linuxcontainers.org/t/lxd-instance-ready-state/14657

Sure, here you go :)


Screen capture rotated by 180 degrees

The screen capture in webrtc is rotated by 180 degrees when using xdg-desktop-portal-wlr
https://github.com/emersion/xdg-desktop-portal-wlr
Can be tested on the website below:
https://mozilla.github.io/webrtc-landing/gum_test.html

Hi @chaitan3, please recheck with latest changes. Should be fixed now.

Yes, it is fixed. Thanks a lot!


TS Error with later versions of apollo-server-express and typeorm

You can reproduce it by installing the latest version of apollo-server-express and typeorm.
const apolloServer = new ApolloServer({
    schema,
    context: ({ req, res }: any) => ({ req, res, conn }),
    plugins: [
        ApolloServerLoaderPlugin({
            typeormGetConnection: getConnection
        })
    ]
});

@geoyws I tried to run it, however did not see any errors when running tsc -b. Installed packages are:
    "apollo-server-express": "^2.18.2",
    "type-graphql": "^1.1.0",
    "type-graphql-dataloader": "^0.2.1",
    "typeorm": "^0.2.28"

What errors did you get?

Basically it's a type issue with typeormGetConnection. Had to downgrade to your tested versions of TypeORM and apollo-server-express. Didn't take a snapshot. Will try it again. Sorry, really held up with work.

I think the problem has been fixed by the new version.
https://github.com/slaypni/type-graphql-dataloader/releases/tag/v0.3.2
If it was not, please reopen the issue.


safeQueue html: change label displays:

Use the format <queue name>:<action>, or only <action> when there's no queue configured for all nodes.

[Fix #34]


Extra Space Added When Inserting New Line in HTML Document with Ctrl+Enter

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.92.2
OS Version: Windows 11 Pro 10.0.22631, Build 22631
Steps to Reproduce:

Open an HTML document.
Write some text and comment it out.
Press Ctrl+Enter to insert a new line without modifying the current line.

Actual Behavior:
The new line is inserted with an extra space (also happens when the line is indented).
Expected Behavior:
The new line should be inserted without any extra spaces.
https://github.com/user-attachments/assets/a55f0653-d5fc-492c-b9b8-1383ccd299b8

Think this happens without using just Enter also after a html comment when making a new line

https://github.com/user-attachments/assets/cd53c45a-41f9-4768-b2b2-d0e727bf872d
It seems to copy the whitespace prior to the text to the next line. The whitespace only gets carried over when there is text present.

im also having this issue. every time i newline after a comment in svelte markup, i have to backspace to delete the automatically added space. very annoying!


#2006 - Fix error not show up on large image when uploading profile picture

What does this pull request aim to achieve? 💯
#2006 - Fix error not show up on large image when uploading profile picture
Please be concise and link any related issue(s): Fix #2006
After Fix:

[x] I have checked that no one else is working on similar changes.
[x] I have read the latest README and understand the development workflow.
[x] The name of this branch is: feature/<branch_name> for a regular feature, hotfix/<branch_name> for a hotfix, and service/<branch_name> for a microservice, where <branch_name> briefly describes the issue(s) resolved and is prefixed with the issue number(s) (e.g. 1127-1130-update-git-branching-model). The branch is created in a cloned version of this repo, not a forked repo.
[x] This branch is rebased/merged with the latest staging (or the latest production if it is a hotfix).
[x] There are no merge conflicts.
[x] There are no console warnings and errors.
[x] On the right hand side of this PR, I have linked the appropriate issue(s) under "Issues" and added a project progress under "Projects".
[x] The title describes the issue(s) being addressed using format: <issue_numbers> - <brief_issue_description> (e.g. #1127/#1130 - Update Git Branching Model)

Thanks @linadinh2602
I have following observation while verifying the fix on feature branch

If avatar already exist and user try to upload new avatar without jpg preview, nothing happens.
BUT, if user does not have avatar i.e. user profile logo exist, and then user try to upload the jpg without preview, blank error message is still displayed.

Please see below GIF.

@rabbi08 : I have tried reaching all the code paths with both error images and working images. However, I cannot reproduce what you see. Do you know a specific way to reproduce? Or can you share the image that didn't work for you, so that I can try it on my end?

@rabbi08 : I have tried reaching all the code paths with both error images and working images. However, I cannot reproduce what you see. Do you know a specific way to reproduce? Or can you share the image that didn't work for you, so that I can try it on my end?

Hi @linadinh2602
I have onedrive configured on machine, so may be due to network slowness, at times it does not loads the preview of the data. so then this issue comes up. As of now as per my investigation this issue links with the slow network speed, where user just select random jpg without waiting for the preview to load and then upload in the application.
I hope that this defines the issue somewhat. Thanks

@rabbi08 : In this case of network slowness that leads to image not showing in the preview pane, then I think what you are seeing is the current expected behavior. Once the image load up, the preview will show with the correct buttons. I'm happy to work to improve this experience, but I think I'll push this PR out first to resolve the other issue that this ticket is addressing, then file a new issue for this network problem. What do you think?

@rabbi08 : In this case of network slowness that leads to image not showing in the preview pane, then I think what you are seeing is the current expected behavior. Once the image load up, the preview will show with the correct buttons. I'm happy to work to improve this experience, but I think I'll push this PR out first to resolve the other issue that this ticket is addressing, then file a new issue for this network problem. What do you think?

@linadinh2602
Actually this issue is for both, the pop up is blank:

either if the file size>5MB : Now PASS in this feature build
preview of the pic is not uploaded: still FAIL

Pease see below GIF:

So we have fixed #1 in this PR only. great if we can fix the other one here only as else we need another issue and PR. Let me know your thoughts. Thanks.

@rabbi08 sorry, I think I am confused from seeing your gif. From what I see, you do two things

Try to upload a large image => it fails and display the error message, which is correct
Then you try to upload a regular image => the preview show up => you click upload => it uploads. This is also correct.
Then, what is the problem here?

@rabbi08 sorry, I think I am confused from seeing your gif. From what I see, you do two things

Try to upload a large image => it fails and display the error message, which is correct
Then you try to upload a regular image => the preview show up => you click upload => it uploads. This is also correct.
Then, what is the problem here?

in the gif, its only for large size error, you have already fixed.
Not in this recent GIF, I am now requesting to fix/handle network issue while uploading, also. Let me know if there any queries.
Thanks @linadinh2602

@rabbi08 Seems like the only fix for the network issue is that there is some activity indicator in the view to show the user that the image is loading. I can add that in to help with the visual feedback. Do you agree with this approach?

@linadinh2602
Actually the problem is when there is no preview is not available for the selected image.
But, I am think this is very unique and corner case which can be fixed in later stage. We can merge this issue which fix for 'large file size; images. The fix is working fine when user try to upload image >5MB.
Thanks


Supress data loss warnings in VS2015

Where safe, use explicit casts to supress data loss warnings in VS2015
compiler.
This fixes #45.
Signed-off-by: Alex Radutskiy<EMAIL_ADDRESS>

Merged as 9ba4791.
Can you check if everything is working? If so, I'll make a v0.3.3 release and start working on v0.4.


expose resources for call-OptimiseAugustus

@ljyanesm @gemygk
Hitting memory for the call-OptimiseAugustus step are the resources for this exposed?

Note edited in the installed reat at (increased to 16G)
https://github.com/EI-CoreBioinformatics/reat/blob/a6003c9b87cd8156fd644a0ea30211933196a34b/annotation/prediction_module/main.wdl#L852
Can expose or just simply increase to 16G for a permanent solution later

Hi @swarbred,
Using the --computational_resources input. One can tune the call-OptimiseAugustus task cpu/memory by defining the required resources using the "ei_prediction.OptimiseAugustus.resources" parameter

Thanks @ljyanesm

yes this can be closed


Timezone for archived filename

Hi, I have a log file with dateformat filename in UTC timezone, when I roll it is changing its timezone to local so I assume that is not based on the active file name. How can I force to roll it with UTC timezone as well?

I think this isn't possible (yet).

Maybe a workaround is to change the NLog time-source to utc (AccurateUTC), and then fix the output to use local time where needed:
https://github.com/nlog/nlog/wiki/Time-Source
http://caraulean.com/2016/timestamp-accuracy-and-resolution-in-nlog/

Good one!

I assume your question has been answered, if not, please let us know!


Unable to load optimizer checkpoint when using resume_from_checkpoint

System Info
pytorch: 2.0.1+cu118
peft: 0.5.0
transformers: 4.32.0
accelerate: 0.21.0
platform: Ubuntu 22.04.2 LTS
python: 3.10.12
Who can help?
@younesbelkada @pacman100
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder
[X] My own task or dataset (give details below)

Reproduction
Here is a rough sketch of the code I use. The code is exactly this, except I skip some arguments and bookkeeping steps. The training setup is standard LoRA adapter fine-tuning for a CausalLM class. The dataset is also formatted as usual for a CausalLM task with input_ids and attention_mask.
    tokenizer = LlamaTokenizer.from_pretrained(...)
    special_token_count = tokenizer.add_special_tokens(get_special_tokens(tokenizer))

    model = LlamaForCausalLM.from_pretrained(
        ...,
        device_map={"": accelerator.local_process_index},
        load_in_8bit=True,
        ...
    )

    model.resize_token_embeddings(len(tokenizer))
    ## ... update embeddings init.

    model = prepare_model_for_kbit_training(model)

    if peft_dir is not None:
        model = get_peft_model(model, PeftConfig.from_pretrained(peft_dir))

        set_peft_model_state_dict(
            model,
            torch.load(os.path.join(peft_dir, PEFT_WEIGHTS_NAME)))
    else:
        peft_config = LoraConfig(
            task_type=TaskType.CAUSAL_LM,
            bias="none",
            r=lora_rank,
            lora_alpha=lora_alpha,
            lora_dropout=lora_dropout,
        )
        model = get_peft_model(model, peft_config)

    model.config.use_cache = False

    with accelerator.main_process_first():
        train_data, val_data, test_data = get_dataset(...)

    trainer = Trainer(
        model=model,
        args=TrainingArguments(
            fsdp=False,
            fp16=False,
            bf16=False,
            gradient_checkpointing=False,
            ddp_find_unused_parameters=False,
            max_steps=max_steps,
            eval_steps=1000,
            save_steps=save_steps,
            logging_steps=100,
            log_on_each_node=False,
            evaluation_strategy="steps",
            per_device_train_batch_size=batch_size,
            per_device_eval_batch_size=batch_size,
            loss_mode=loss_mode,
            optim="adamw_torch",
            learning_rate=learning_rate,
            lr_scheduler_type="cosine",
            warmup_steps=warmup_steps,
            weight_decay=weight_decay,
            gradient_accumulation_steps=grad_acc,
            output_dir=log_dir,
            report_to="wandb",
            dataloader_num_workers=4,
        ),
        train_dataset=train_data,
        eval_dataset=val_data,
        test_dataset=test_data,
        tokenizer=tokenizer,
    )

    trainer.train(resume_from_checkpoint=peft_dir)
    trainer.save_state()

Now here are the steps I take:

I first use this training code without the peft_dir variable above. This leads to a checkpoint folder with the following files:README.md            added_tokens.json  rng_state_1.pth  scheduler.pt             tokenizer_config.json
adapter_config.json  optimizer.pt       rng_state_2.pth  special_tokens_map.json  trainer_state.json
adapter_model.bin    rng_state_0.pth    rng_state_3.pth  tokenizer.model          training_args.bin

Now, send in exactly the same arguments, except this time passing peft_dir to the directory containing the above files.

In terms of evaluating the checkpoint, I can confirm that the adapter weights are loading correctly, since I am able to see performance improvement in test from the checkpoint (that exactly matches what I see during my training logs)
But, when loading this checkpoint for training, I get the following error:File "finetune.py", line 169, in main
  trainer.train(resume_from_checkpoint=peft_dir)
File "/ext3/.conda/envs/overlayfs/lib/python3.10/site-packages/transformers/trainer.py", line 1555, in train
  return inner_training_loop(
File "/ext3/.conda/envs/overlayfs/lib/python3.10/site-packages/transformers/trainer.py", line 1707, in _inner_training_loop
  self._load_optimizer_and_scheduler(resume_from_checkpoint)
File "/ext3/.conda/envs/overlayfs/lib/python3.10/site-packages/transformers/trainer.py", line 2499, in _load_optimizer_and_scheduler
  self.optimizer.load_state_dict(
File "/ext3/.conda/envs/overlayfs/lib/python3.10/site-packages/accelerate/optimizer.py", line 103, in load_state_dict
  self.optimizer.load_state_dict(state_dict)
File "/ext3/.conda/envs/overlayfs/lib/python3.10/site-packages/torch/optim/optimizer.py", line 390, in load_state_dict
  raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group

One way I've worked around this issue is to simply not load the optimizer.pt. I change what the transformers library looks for as: import transformers.trainer as TrainerModule
 optimizer_path = os.path.join(peft_dir, TrainerModule.OPTIMIZER_NAME)
 TrainerModule.OPTIMIZER_NAME = f"_{TrainerModule.OPTIMIZER_NAME}"

Consequently, all the code works as usual, and I know the trainer state is set correctly, because the training resumes only for the remainder of the steps. e.g. if I load a checkpoint-10000 folder, it resumes from step 10001, which I verified using the on_train_begin callback and checking state.global_step.

On manual inspection of the trainer.optimizer.state_dict(), I am able to see that the state is very different from what is stored at initialization, and therefore the conflict in loading.
However, after the first step of the optimizer, the trainer.optimizer.state_dict() looks fairly similar to the checkpoint. I believe the AdamW optimizer lazily creates the required state after the first gradient step. Since, the optimizer checkpoint is large, I was only able to verify the keys manually by traversing the state_dict. Although, by then using torch.load after the first gradient step (using a custom trainer callback), it still fails with the same error.
Loading the optimizer state is fairly important for continuing the training, since the moving averages affect subsequent training dynamics.
Could someone help with what is going wrong here, either in the API usage or otherwise?
Expected behavior
Thetrainer.train(resume_from_checkpoint=peft_dir) works by loading the optimizer state as well.

Hi @younesbelkada @pacman100, cc'ing you here again, since I did not put it in the first time (and assuming from your roles in #593). Thank you!

I tried one other thing. It turns out just cherry-picking the pretrained config works. i.e. I selectively pick the pretrained PeftConfig from the checkpoint, and then create the PeftModel as I was doing without the checkpoint.
    if peft_dir is not None:
        peft_config = PeftConfig.from_pretrained(peft_dir)
        lora_rank = peft_config.r
        lora_alpha = peft_config.lora_alpha
        lora_dropout = peft_config.lora_dropout

        if accelerator.is_main_process:
            print(f"[INFO]: Loaded PEFT config from '{peft_dir}'.")

    peft_config = LoraConfig(
        task_type=TaskType.CAUSAL_LM,
        bias="none",
        r=lora_rank,
        lora_alpha=lora_alpha,
        lora_dropout=lora_dropout,
    )
    model = get_peft_model(model, peft_config)

This seems to resume training without errors. In the trainer._load_from_checkpoint method, I see that it already loads the model adapter from checkpoint.
Could you please confirm if this is the right way to resume training correctly?

same issue here

I think I've figured out the cleanest solution:
model = PeftModel.from_pretrained(model, peft_dir, is_trainable=True)

trainer.train(resume_from_checkpoint=peft_dir)

These two lines should be enough to resume training from a checkpoint.

@activatedgeek how are you saving your peft adapter? I tried setting to the same versions as you for every library but I am getting the same error

I am not doing anything different to save. I use save_steps as usual for the checkpoints to be saved, create the base model and the LoRA adapter as above.
If you are not seeing checkpoints in your output_dir, perhaps you will need to reduce save_steps. Try save_steps=1 for a quick check.


Weaviate does not work

Describe the bug
I opened a similar issue in the Semantic Kernel repository (it was one of the reasons I came to this repository). https://github.com/microsoft/semantic-kernel/issues/8934
I can't use Weaviate Vector Store with Google Vertex AI (and, I suspect, other integrations with Weaviate may not be working either).
This is my code:
        var provider = new VertexAIProvider(new VertexAIConfiguration
        {
            GoogleCredential = GoogleCredential.FromFile("D:\\code\\my-google-cloud-project.json"),
            Location = "us-central1",
        });
        var embeddingModel = new VertexAIEmbeddingModel(provider, id: "text-multilingual-embedding-002");
        var llm = new VertexAIChatModel(provider, id: "gemini-1.5-pro-001");

        var weviateApiKey = "weaviate-api-key";
        var collection = "Test_Collection";
        WeaviateMemoryStore memoryStore = new("https://my-weaviate-endpoint.c0.us-east1.gcp.weaviate.cloud", weviateApiKey);
        var vectorDatabase = new WeaviateVectorDatabase(memoryStore);

        // Exeception is thrown here
        var vectorCollection = await vectorDatabase.AddDocumentsFromAsync<PdfPigPdfLoader>(
            embeddingModel, // Used to convert text to embeddings
            dimensions: 384, // Should be 384 for all-minilm
            dataSource: DataSource.FromUrl("https://canonburyprimaryschool.co.uk/wp-content/uploads/2016/01/Joanne-K.-Rowling-Harry-Potter-Book-1-Harry-Potter-and-the-Philosophers-Stone-EnglishOnlineClub.com_.pdf"),
            collectionName: "harrypotter", // Can be omitted, use if you want to have multiple collections
            textSplitter: null,
            behavior: AddDocumentsToDatabaseBehavior.JustReturnCollectionIfCollectionIsAlreadyExists);

        const string question = "What is Harry's Address?";
        var similarDocuments = await vectorCollection.GetSimilarDocuments(embeddingModel, question, amount: 5);
        // Use similar documents and LLM to answer the question
        var answers = llm.GenerateAsync(
            $"""
             Use the following pieces of context to answer the question at the end.
             If the answer is not in context then just say that you don't know, don't try to make up an answer.
             Keep the answer as short as possible.

             {similarDocuments.AsString()}

             Question: {question}
             Helpful Answer:
             """);

        await foreach (var answer in answers)
        {
            Console.WriteLine($"LLM answer: {answer}");
        }

I keep getting the same error:
Microsoft.SemanticKernel.HttpOperationException: Response status code does not indicate success: 401 (Unauthorized).
 ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Unauthorized).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.SemanticKernel.Http.HttpClientExtensions.SendWithSuccessCheckAsync(HttpClient client, HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.SemanticKernel.Http.HttpClientExtensions.SendWithSuccessCheckAsync(HttpClient client, HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Http.HttpClientExtensions.SendWithSuccessCheckAsync(HttpClient client, HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Connectors.Weaviate.WeaviateMemoryStore.ExecuteHttpRequestAsync(HttpRequestMessage request, CancellationToken cancel)
   at Microsoft.SemanticKernel.Connectors.Weaviate.WeaviateMemoryStore.DoesCollectionExistAsync(String collectionName, CancellationToken cancellationToken)
   at LangChain.Databases.SemanticKernel.SemanticKernelMemoryDatabase.IsCollectionExistsAsync(String collectionName, CancellationToken cancellationToken) in /_/src/SemanticKernel/src/SemanticKernelMemoryDatabase.cs:line 34
   at LangChain.Extensions.VectorDatabaseExtensions.AddDocumentsFromAsync[TLoader](IVectorDatabase vectorDatabase, IEmbeddingModel embeddingModel, Int32 dimensions, DataSource dataSource, String collectionName, ITextSplitter textSplitter, DocumentLoaderSettings loaderSettings, EmbeddingSettings embeddingSettings, AddDocumentsToDatabaseBehavior behavior, CancellationToken cancellationToken) in /_/src/Core/src/Extensions/VectorDatabaseExtensions.cs:line 42
   at Api.LangchainTest.Execute() in D:\code\Meu-Aluguel\Api\LangchainTest.cs:line 35
   at Program.<Main>$(String[] args) in D:\code\Meu-Aluguel\Api\Program.cs:line 13
   at Program.<Main>(String[] args)

I suspect that the Semantic Kernel library (responsible for the WeaviateMemoryStore class, on which this library is dependent) is not placing the necessary headers in requests managed by the Vector database classes. For instance, the Weaviate documentation (https://weaviate.io/developers/weaviate/model-providers/google/embeddings) says that, for the integration with Vertex AI to work, the Vertex AI API key must be passed in the request header in the X-Google-Vertex-Api-Key field. In the case of Open AI, it would be the X-OpenAI-Api-Key field.
Alternatively, would there be any way to use Weaviate with this Langchain library without going through the Semantic Kernel?
Steps to reproduce the bug
Execute my code
Expected behavior
No response
Screenshots
No response
NuGet package version
No response
Additional context
No response

WeaviateVectorData lanchain dotnet uses the semantic kernel implementation, you will have the same problems, the library abstraction is complex and early, surely, they will end up solving these problems, there are many vector database alternatives, my recommendation is to try another one.
https://github.com/tryAGI/LangChain.Databases/blob/main/src/SemanticKernel/src/SemanticKernelMemoryDatabase.cs
https://github.com/tryAGI/LangChain.Databases/blob/main/src/Weaviate/src/WeaviateVectorDatabase.cs

We can try to update all dependencies, and it might work if Microsoft already fixed this problem
There are some problems with dependabot here, I think it doesn't update it for some reason

P.S. I updated all SemanticKernel dependencies, please try to use latest .dev version of LangChain.Databases.Weaviate

I updated all SemanticKernel dependencies, please try to use latest .dev version of LangChain.Databases.Weaviate

Thanks for the feedback. Just upgraded all dependencies, LangChain.Databases.Weaviate is on 0.15.4-dev.5 now. Unfortunately the error persists.


Implement Array#push in Rust

Related to the thread in #2115 regarding implementing Array#push in Rust.
My understanding in the thread was that the existing implementation is actually
correct, but this operation would ideally be implemented in Rust.
I'm far from a Rust expert so I had some trouble getting types to line up when
trying to replicate the approach of calling into concat directly. Regardless,
I took an initial stab at this out of interest and am resorting to calling
push_single a bunch of times. Not sure if this is actually a bad approach so
open to feedback.
Also, am a fan of both languages so thank you for making this project which
piqued my interest enough to attempt an OSS PR for the first time :).

it looks like RuboCop isn't quite happy. It looks like running an autofix should get this PR green. You can do this locally:
bundle install
bundle exec rake lint:rubocop:autocorrect

ah I thought I had fixed that, pushed up a fix.

@b-n it looks like you've got an ❌ on this PR. This LGTM; if @Tseyang has addressed the changes you wanted to see please 👍 and merge.

@b-n bumping this, any other changes you want me to make?

@lopopolo should we keep waiting on @b-n ?

Thanks for the PR @Tseyang! and thanks for following up to poke me to merge this


Feature/Help request

It would be awesome to have a version which simply calls the engine with a configurable command, like 'stockfish bench 128 1 4 1000.epd depth mixed', instead of running a tournament manager, e.g. cutechess-cli. I think many other engines provide a similar functionality, too.
I've written a very basic Alpha-Beta search for Stockfish. Now I want to work on move-ordering but not in the usual way. I don't want to call a MovePicker object, but want to score all moves and sort them before searching. (I know this will be slow!)
I want to tune this scoring algorithm.
The nice thing with a plain AB search is that move-ordering alone decides about how many nodes it takes to finish a given depth. The best move and eval don't change.
Maybe you already have something like this, or you can at least give some advice what I need to change?

So you want to have a different objective value sent to lakas optimizer, not the match result but from the bench command or so.

Correct. Preferably the number of nodes.

I will try to implement it.

I am creating a command line interface between the engine and lakas, call it enhance.py.
command line
python enhance.py -engine cmd=sf13.exe -hashmb 32 -threads 1 -limitvalue 1000 -fenfile bench.epd -limittype nodes -evaltype mixed -concurrency 2 -posperfile 1000 -randomizefen

It will take pos file input via -fenfile.
The flag -randomizefen if called will randomize the pos in bench.epd.
If there are 2 concurrencies, 2 processes will be conducted and the mean nodes will be returned.
The '-posperfile' determines the number of pos in the file that will be used per bench run.

help
usage: Enhance v0.1.0 [-h] -engine [cmd= [option.<optionname>=value ...]] [-concurrency CONCURRENCY] [-hashmb HASHMB] [-threads THREADS] [-limitvalue LIMITVALUE] [-fenfile FENFILE] [-limittype LIMITTYPE] [-evaltype EVALTYPE]
                      [-randomizefen] [-posperfile POSPERFILE] [-v]

Run bench command.

optional arguments:
  -h, --help            show this help message and exit
  -engine [cmd= [option.<optionname>=value ...]]
                        Define engine filename and option, required=True. Example:
                        -engine cmd=eng.exe option.FutilityMargin=120 option.MoveCount=1000
  -concurrency CONCURRENCY
                        the number of benches to run in parallel, default=1
  -hashmb HASHMB        memory value in mb, default=64
  -threads THREADS      number of threads, default=1
  -limitvalue LIMITVALUE
                        a number that limits the engine search, default=15
  -fenfile FENFILE      Filename of FEN or EPD file, default=default
  -limittype LIMITTYPE  the type of limit can be depth, perft, nodes and movetime, default=depth
  -evaltype EVALTYPE    the type of eval to use can be mixed, classical or NNUE, default=mixed
  -randomizefen         A flag to randomize position before using it in bench when position file is used.
  -posperfile POSPERFILE
                        the number of positions in the file to be used in the bench, default=50
  -v, --version         show program's version number and exit

Enhance v0.1.0

log output
2021-06-07 00:57:43,894 - pid28644 -  INFO - numfiles: 2
2021-06-07 00:57:43,895 - pid28644 -  INFO - file generation elapse 1.52s
2021-06-07 00:57:44,258 - pid14392 - DEBUG - >> uci
2021-06-07 00:57:44,258 - pid17568 - DEBUG - >> uci
...
2021-06-07 00:57:44,760 - pid14392 - DEBUG - >> bench 32 1 1000 file1.fen nodes mixed
2021-06-07 00:57:44,760 - pid17568 - DEBUG - >> bench 32 1 1000 file0.fen nodes mixed
...

2021-06-07 00:57:47,049 - pid28644 - DEBUG - bench nodes: [1000910, 1000863]
2021-06-07 00:57:47,049 - pid28644 - DEBUG - mean nodes: 1000886

The process is it will read the pos file (if specified) and convert it to a list in memory. This list is then randomized (if specified). Then extract sequentially some fens specified by posperfile option. The fens extracted is written to a file which will be used in the bench.
After some debugging, I will then modify lakas.py to support this interface.

Awesome! Let me know if I can be of any help.

Help me debug enhance.py.

number of pos in a file is correct per posperfile option, random or not, note the input pos file -fenfile is not modified
check elapse time in generating big fen file (it would consume more RAM when converting the file to fen list in memory for bigger files), I only tested it with around 600k pos (43MB size)
check behaviour with high number of concurrency, I only tested it with 6
others

log enhance_log.txt is written in append mode. pos fen file is written in overwrite mode.
pos fen file are not deleted automatically after the run for inspection. You can delete it by modifying the code around line 183.
 # Delete fen files.
 for fn in fenfiles:
    pass  # self.delete_file(fn)

In the -engine option values, it does not read correctly when option has a space, and when the value is not an integer.
This one is ok.
-engine cmd=sf.exe option.FutMargin=200
This one is not ok, value true is not an integer, not supported at the moment.
-engine cmd=sf.exe option.UCI_LimitStrength=true
This one is not ok, option name has a space. I need more time to solve this, better to modify the engine option name  - remove the space.
-engine cmd=sf.exe option.Skill Level=10
The option.<option_name> in -engine option is intended for use in Lakas as the param to be optimized. While the -hashmb,  -threads, -limitvalue and others are intended for use in the bench command.
Use python 3.8 or higher, I only tested it on python 3.8. No dependent module is required.
Type python enhance.py -h for help.
See attached enhance.py and bench.epd.
enhance_pack.zip

This one also supports non-integer value including float.
-engine cmd=sf.exe option.LimitStrength=true
And without changes, it actually works on option name with space, using double quotes.
-engine cmd=sf.exe "option.Skill Level=10" "option.Slow Mover=20"
2021-06-07 12:18:10,764 - pid11596 - DEBUG - >> setoption name Skill Level value 10
2021-06-07 12:18:10,764 - pid11596 - DEBUG - >> setoption name Slow Mover value 20

But still option name without space is much better.
enhance_pack2.zip

A first, very basic test done.
This is a bench run with my modified Stockfish version:
bench 16 1 3 100.epd depth mixed
===========================
Total time (ms) : 6224
Nodes searched  : 4994456
Nodes/second    : 802451
Running python3 enhance.py -engine cmd=./stockfish -limitvalue 3 -fenfile 100.epd -posperfile 100
yields the same result.

nodes searched: 4994456
bench done

P.S. This is under Linux, Python 3.8.5

Added a bench branch, enhance.py is in interface folder.

Next test: splitting the positions with concurrency set to 4.
python3 enhance.py -engine cmd=./stockfish -concurrency 4 -limitvalue 3 -fenfile 100.epd -evaltype classical -posperfile 25
Works flawlessly. :+1:

nodes searched: 1251340
bench done

From the log file:

2021-06-07 11:43:23,005 - pid 6575 - DEBUG - bench nodes: [1091508, 1285234, 1212184, 1416436]
2021-06-07 11:43:23,006 - pid 6575 - DEBUG - mean nodes: 1251340
2021-06-07 11:43:23,006 - pid 6575 -  INFO - total elapse time: 1.94s

One suggestion: it would probably be more useful to sum up the number of nodes of the single runs.

One suggestion: it would probably be more useful to sum up the number of nodes of the single runs.

It is now in enhance v0.2.0
command
python enhance.py -engine cmd=sf13.exe -fenfile bench.epd -limitvalue 8 -concurrency 4
total nodes searched from 4 workers: 894889
bench done

log
2021-06-07 19:08:08,307 - pid 4348 - DEBUG - bench nodes: [216219, 228481, 222694, 227495]
2021-06-07 19:08:08,308 - pid 4348 - DEBUG - mean nodes: 223722
2021-06-07 19:08:08,308 - pid 4348 - DEBUG - sum nodes: 894889
2021-06-07 19:08:08,308 - pid 4348 -  INFO - total elapse time: 1.40s

Run bench with sf13 but use a different nnue file. The use of classical and nnue are alternating based from the info string. Is this a normal behavior?
PS D:\github\Lakas\interface> ./sf13
Stockfish 13 by the Stockfish developers (see AUTHORS file)
uci
id name Stockfish 13
id author the Stockfish developers (see AUTHORS file)

option name Debug Log File type string default
option name Contempt type spin default 24 min -100 max 100
option name Analysis Contempt type combo default Both var Off var White var Black var Both
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 33554432
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 10 min 0 max 5000
option name Slow Mover type spin default 100 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name UCI_ShowWDL type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
option name Use NNUE type check default true
option name EvalFile type string default nn-62ef826d1a6d.nnue
uciok
setoption name evalfile value nn48.nnue
isready
readyok
bench

Position: 1/47 (rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1)
info string classical evaluation enabled
info depth 1 seldepth 1 multipv 1 score cp 109 nodes 20 nps 6666 tbhits 0 time 3 pv e2e3
info depth 2 seldepth 2 multipv 1 score cp 121 nodes 47 nps 15666 tbhits 0 time 3 pv e2e3 b7b6
info depth 3 seldepth 3 multipv 1 score cp 171 nodes 100 nps 25000 tbhits 0 time 4 pv e2e4 d7d6 d2d4
info depth 4 seldepth 4 multipv 1 score cp 105 nodes 280 nps 56000 tbhits 0 time 5 pv e2e4 e7e6 b1c3 b7b6
info depth 5 seldepth 5 multipv 1 score cp 135 nodes 410 nps 82000 tbhits 0 time 5 pv e2e4 e7e6 b1c3
info depth 6 seldepth 6 multipv 1 score cp 58 nodes 2607 nps 325875 tbhits 0 time 8 pv g1f3 g8f6 b1c3 d7d5 d2d4 b8c6
info depth 7 seldepth 7 multipv 1 score cp 75 nodes 2928 nps 366000 tbhits 0 time 8 pv g1f3 e7e6 e2e4 d7d5 b1c3 g8f6
info depth 8 seldepth 11 multipv 1 score cp 44 nodes 7882 nps 606307 tbhits 0 time 13 pv e2e4 d7d5 e4d5 d8d5 b1c3 d5e6 d1e2 b8c6 e2e6 c8e6
info depth 9 seldepth 15 multipv 1 score cp 59 nodes 9925 nps 620312 tbhits 0 time 16 pv e2e4 d7d5 e4d5 d8d5 b1c3 d5e6 d1e2 b8c6 e2e6 c8e6
info depth 10 seldepth 15 multipv 1 score cp 125 nodes 14530 nps 691904 tbhits 0 time 21 pv e2e4 d7d5 e4d5 d8d5 b1c3 d5e6 d1e2
info depth 11 seldepth 14 multipv 1 score cp 71 nodes 32915 nps 889594 tbhits 0 time 37 pv e2e4 d7d5 e4d5 c7c6 g1f3 c6d5 b1c3 g8f6 d1e2 b8c6 d2d4
info depth 12 seldepth 16 multipv 1 score cp 66 nodes 59271 nps 1004593 tbhits 0 time 59 pv e2e4 d7d5 e4d5 d8d5 b1c3 d5f5 d2d4 b8c6 c3b5 c6b4 b5c7 e8d8
info depth 13 seldepth 17 multipv 1 score cp 86 nodes 74792 nps 1038777 tbhits 0 time 72 pv e2e4 d7d5 e4d5 d8d5 b1c3 d5f5 f1c4 f5g6 c3d5 g6g2 d1f3 g2f3 g1f3
bestmove e2e4 ponder d7d5

Position: 2/47 (r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 10)
info string NNUE evaluation using nn48.nnue enabled
info depth 1 seldepth 1 multipv 1 score cp 64 nodes 138 nps 46000 tbhits 0 time 3 pv e5g6
info depth 2 seldepth 2 multipv 1 score cp -34 nodes 323 nps 107666 tbhits 0 time 3 pv e2a6 h3g2
info depth 3 seldepth 3 multipv 1 score cp -72 nodes 600 nps 150000 tbhits 0 time 4 pv d5e6 e7e6 e2a6 b4c3
info depth 4 seldepth 4 multipv 1 score cp -101 nodes 908 nps 181600 tbhits 0 time 5 pv e2a6 b4c3 d2c3 h3g2
info depth 5 seldepth 5 multipv 1 score cp -152 nodes 1500 nps 250000 tbhits 0 time 6 pv e2a6 b4c3 d2c3 e6d5 e4d5
info depth 6 seldepth 6 multipv 1 score cp -134 nodes 1787 nps 297833 tbhits 0 time 6 pv e2a6 b4c3 d2c3 e6d5 e1g1 d5e4
info depth 7 seldepth 7 multipv 1 score cp -183 nodes 3701 nps 411222 tbhits 0 time 9 pv e2a6 b4c3 d2c3 e6d5 e4d5 f6d5 e1g1 d5c3
info depth 8 seldepth 11 multipv 1 score cp -133 nodes 5243 nps 476636 tbhits 0 time 11 pv e2a6 b4c3 d2c3 e6d5 e5g4 h3g2 f3g2 e7e4 g2e4 d5e4
info depth 9 seldepth 12 multipv 1 score cp -97 nodes 6322 nps 486307 tbhits 0 time 13 pv e2a6 b4c3 d2c3 e6d5 e5g4 h3g2 f3g2
info depth 10 seldepth 14 multipv 1 score cp -114 nodes 18248 nps 629241 tbhits 0 time 29 pv e2a6 b4c3 d2c3 f6e4 e5g6 g7c3 b2c3 f7g6 f3e4 e6d5 e4e7 e8e7 e1g1
info depth 11 seldepth 18 multipv 1 score cp -153 nodes 44185 nps 775175 tbhits 0 time 57 pv e2a6 b4c3 d2c3 h3g2 f3g2 e6d5 e5g4 e8f8 g4f6 g7f6 c3f6 e7f6 e1c1 d5e4 g2e4
info depth 12 seldepth 20 multipv 1 score cp -144 nodes 75897 nps 834032 tbhits 0 time 91 pv e2a6 b4c3 d2c3 h3g2 f3g2 e6d5 e5g4 e8f8 c3f6 g7f6 e1c1 d5e4
info depth 13 seldepth 19 multipv 1 score cp -109 nodes 86942 nps 835980 tbhits 0 time 104 pv e2a6 b4c3 d2c3 h3g2 f3g2 e6d5 e5g4 h8h4 c3f6 g7f6 e1c1 d5e4
bestmove e2a6 ponder b4c3

...

Master has new version with --enhance flag.
Command line
Example bench.bat
set optname=sfbench

python -u lakas.py --enhance --match-manager-path "python ./interface/enhance.py" --enhance-fenfile D:/github/Lakas/interface/bench.epd ^
--enhance-hashmb 64 --enhance-threads 1 --enhance-limitvalue 100 --enhance-limittype movetime ^
--optimizer bayesopt --input-data-file %optname%.dat --output-data-file %optname%.dat --optimizer-log-file %optname%.txt ^
--budget 100 --concurrency 1 ^
--engine D:/github/Lakas/engines/stockfish-modern/stockfish.exe ^
--input-param "{'RazorMargin': {'init':527, 'lower':400, 'upper':600}, 'FutMargin': {'init':227, 'lower':100, 'upper':300}, 'KingAttackWeights[4]': {'init':44, 'lower':20, 'upper':60}}"

pause

Log
2021-06-11 23:20:54,801 | INFO  | Lakas v0.40.0
2021-06-11 23:20:54,801 | INFO  | nevegrad 0.4.3
2021-06-11 23:20:54,801 | INFO  | input param: OrderedDict([('FutMargin', {'init': 227, 'lower': 100, 'upper': 300}), ('KingAttackWeights[4]', {'init': 44, 'lower': 20, 'upper': 60}), ('RazorMargin', {'init': 527, 'lower': 400, 'upper': 600})])

2021-06-11 23:20:54,802 | INFO  | total budget: 100
2021-06-11 23:20:54,802 | INFO  | games/budget: 100
2021-06-11 23:20:54,802 | INFO  | move control: base_time_sec: None, inc_time_sec: None, depth=None, nodes=None
2021-06-11 23:20:54,805 | INFO  | parameter dimension: 3
2021-06-11 23:20:54,805 | INFO  | deterministic function: False
2021-06-11 23:20:54,805 | INFO  | use best param: False, optimizer suggested param is always against the init param
2021-06-11 23:20:54,806 | INFO  | optimizer: bayesopt, initialization: Hammersley, init_budget: None, middle_point: False, utility_kind: ucb, utility_kappa: 2.576, utility_xi: 0.0, gp_parameters: {'alpha': 0.001, 'normalize_y': True, 'n_restarts_optimizer': 5, 'random_state': None}

2021-06-11 23:20:54,833 | INFO  | budget: 1
2021-06-11 23:20:54,834 | INFO  | recommended param: {'FutMargin': 183, 'KingAttackWeights[4]': 44, 'RazorMargin': 508}
2021-06-11 23:20:54,834 | INFO  | best param: {'FutMargin': 227, 'KingAttackWeights[4]': 44, 'RazorMargin': 527}
2021-06-11 23:20:54,834 | INFO  | option.FutMargin=227 option.KingAttackWeights[4]=44 option.RazorMargin=527
2021-06-11 23:20:54,835 | INFO  | best loss: 0.5
2021-06-11 23:20:54,835 | INFO  | init param: {'FutMargin': 227, 'KingAttackWeights[4]': 44, 'RazorMargin': 527}
2021-06-11 23:21:01,455 | INFO  | total nodes searched: 3422543
2021-06-11 23:21:01,464 | INFO  | budget: 2
2021-06-11 23:21:01,464 | INFO  | recommended param: {'FutMargin': 162, 'KingAttackWeights[4]': 37, 'RazorMargin': 546}
2021-06-11 23:21:01,465 | INFO  | best param: {'FutMargin': 183, 'KingAttackWeights[4]': 44, 'RazorMargin': 508}
2021-06-11 23:21:01,465 | INFO  | option.FutMargin=183 option.KingAttackWeights[4]=44 option.RazorMargin=508
2021-06-11 23:21:01,465 | INFO  | best loss: -3422543.0
2021-06-11 23:21:01,465 | INFO  | init param: {'FutMargin': 227, 'KingAttackWeights[4]': 44, 'RazorMargin': 527}
2021-06-11 23:21:07,971 | INFO  | total nodes searched: 3645699
2021-06-11 23:21:07,981 | INFO  | budget: 3
2021-06-11 23:21:07,981 | INFO  | recommended param: {'FutMargin': 194, 'KingAttackWeights[4]': 51, 'RazorMargin': 435}
2021-06-11 23:21:07,982 | INFO  | best param: {'FutMargin': 162, 'KingAttackWeights[4]': 37, 'RazorMargin': 546}
2021-06-11 23:21:07,983 | INFO  | option.FutMargin=162 option.KingAttackWeights[4]=37 option.RazorMargin=546
2021-06-11 23:21:07,984 | INFO  | best loss: -3645699.0
2021-06-11 23:21:07,985 | INFO  | init param: {'FutMargin': 227, 'KingAttackWeights[4]': 44, 'RazorMargin': 527}
2021-06-11 23:21:14,556 | INFO  | total nodes searched: 3653876

@fsmosca Thank you so much for your help!
I was busy renovating the last days. I hope I can start my first tuning session soon. I will keep you updated.


Test optional addons - Test Release KubeOne 1.6

This ticket is about testing the following optional addons:

Unattended upgrades
Backups
cluster-autoscaler

The tests should be conducted on:

[x] v1.26.1
[x] v1.25.6
[x] v1.24.10

The tests should be conducted on the following operating systems:

[x] Ubuntu
[x] CentOS/Rocky Linux
[x] Flatcar

Note: we had a reported issue with the backups addon, so please pay additional attention to it: https://github.com/kubermatic/kubeone/discussions/2472

AWS was used for testing these cases. Cluster-autoscaler and Backups were just tested with Ubuntu since they don't depend on OS. Unattended upgrades was tested with Ubuntu, Rocky Linux, and Flatcar/

Should work as expected after https://github.com/kubermatic/kubeone/pull/2685


Bug: should indicate when business is not balanced by origin currency

example:
https://accounter-prod.pages.dev/all-charges?chargesFilters=%257B%2522byOwners%2522%253A%255B%25226a20aa69-57ff-446e-8d6a-1e96d095e988%2522%255D%252C%2522sortBy%2522%253A%257B%2522field%2522%253A%2522DATE%2522%252C%2522asc%2522%253Afalse%257D%252C%2522fromAnyDate%2522%253A%25222024-01-01%2522%252C%2522toAnyDate%2522%253A%25222024-02-28%2522%252C%2522byBusinesses%2522%253A%255B%25222ee83c91-85d6-43de-9138-6c08b748d90b%2522%255D%257D

dine by #1002


Map ping fix, and show portal name in ping message

Hi! I've fixed the InvokeRoutedRPC call in MapButtonClicked(), so that it no longer throws an error. While I was there I've also changed the map ping "name" so that it shows the target portal tag. Please test as I've never in my life done any game modding of any kind. It works on my end, though!

(this addresses issue #29)

I can confirm that the fix works. Thanks a lot.

I'm closing this PR because I have created XPortal which completely replaces AnyPortal. Thanks for the inspiration and hard work, without AnyPortal's code I would never have been able to figure any of this out.