xgboost split_info differs for sparse and dense binary features

The split info output from xgboost models trained on sparse, binary data matrix does not match the output when trained on a dense matrix:
Reproducible example:
library(xgboost)

data(iris)

sparse <- sparse.model.matrix(Petal.Length~.-1, data=iris)
dense <- model.matrix(Petal.Length~.-1, data=iris)

head(sparse)

6 x 6 sparse Matrix of class "dgCMatrix"
  Sepal.Length Sepal.Width Petal.Width Speciessetosa Speciesversicolor Speciesvirginica
1          5.1         3.5         0.2             1                 .                .
2          4.9         3.0         0.2             1                 .                .
3          4.7         3.2         0.2             1                 .                .
4          4.6         3.1         0.2             1                 .                .
5          5.0         3.6         0.2             1                 .                .
6          5.4         3.9         0.4             1                 .                .

head(dense)

  Sepal.Length Sepal.Width Petal.Width Speciessetosa Speciesversicolor Speciesvirginica
1          5.1         3.5         0.2             1                 0                0
2          4.9         3.0         0.2             1                 0                0
3          4.7         3.2         0.2             1                 0                0
4          4.6         3.1         0.2             1                 0                0
5          5.0         3.6         0.2             1                 0                0
6          5.4         3.9         0.4             1                 0                0

all.equal(dense, as.matrix(sparse), check.attributes=F)

The matrices are identical except for one being sparse and the other dense. There are three binary indicator columns in each matrix. The other values are numeric.
Train an xgboost model on each
train_sparse <- xgb.DMatrix(sparse, label=iris$Petal.Length)
train_dense <- xgb.DMatrix(dense, label=iris$Petal.Length)

set.seed(100)
mod_sparse <- xgboost(train_sparse, nrounds=1)

set.seed(100)
mod_dense <- xgboost(train_dense, nrounds=1)

xgb.model.dt.tree(colnames(sparse), mod_sparse)
xgb.model.dt.tree(colnames(dense), mod_dense)

Each model is trained with the same settings and the same seed. The output for the sparse matrix is so:
 xgb.model.dt.tree(colnames(sparse), mod_sparse)
   Tree Node  ID           Feature        Split Yes  No Missing    Quality Cover
1:    0    0 0-0       Petal.Width          0.8 0-1 0-2     0-1 385.787000   150
2:    0    1 0-1              Leaf           NA  NA  NA      NA   0.282941    50
3:    0    2 0-2 Speciesversicolor -9.53674e-07 0-3 0-4     0-3  22.069500   100
4:    0    3 0-3              Leaf           NA  NA  NA      NA   1.485880    50
5:    0    4 0-4              Leaf           NA  NA  NA      NA   1.105880    50

Notice the split for Speciesversicolor. It is this very small negative value: -9.53674e-07. If I build a larger tree, the split_info for binary indicator columns is Always this value.
Compare this to the output for a dense matrix where the split value is 0.5 which is what should be expected.
xgb.model.dt.tree(colnames(dense), mod_dense)
   Tree Node  ID           Feature Split Yes  No Missing    Quality Cover
1:    0    0 0-0       Petal.Width   0.8 0-1 0-2     0-1 385.787000   150
2:    0    1 0-1              Leaf    NA  NA  NA      NA   0.282941    50
3:    0    2 0-2 Speciesversicolor   0.5 0-3 0-4     0-3  22.069500   100
4:    0    3 0-3              Leaf    NA  NA  NA      NA   1.485880    50
5:    0    4 0-4              Leaf    NA  NA  NA      NA   1.105880    50

Environment info
Operating System: os x El Capitan 10.11
Compiler: NA
Package used (python/R/jvm/C++): R
xgboost version used:  xgboost_0.6-0
If installing from source, please provide

The commit hash (git rev-parse HEAD)
Logs will be helpful (If logs are large, please upload as attachment).

If you are using python package, please provide

The python version and distribution
The command to install xgboost if you are not installing from source

If you are using R package, please provide
installed from CRAN
sessionInfo():
R version 3.3.1 (2016-06-21)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11 (El Capitan)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base
other attached packages:
[1] xgboost_0.6-0     onehot_<IP_ADDRESS>00 Matrix_1.2-7.1
loaded via a namespace (and not attached):
[1] magrittr_1.5      tools_3.3.1       Rcpp_0.12.10      stringi_1.1.3     grid_3.3.1
[6] data.table_1.10.0 lattice_0.20-34
Steps to reproduce

Train any model using a sparse matrix with binary features
dump the model or convert to DT
See that split values are very small negative numbers

What have you tried?

I have tried running on different OSs. Confirmed it also happens on Windows 7 with R 3.4

Yeah, I noticed the same thing and opened #2737. Did you figure anything out? I've been looking at the C API and I can't figure out why this is happening.

https://github.com/dmlc/xgboost/search?utf8=✓&q=sparse+dense+missing&type=Issues
I'm also adding it to FAQ https://github.com/dmlc/xgboost/pull/2732/commits/0604851f28529a4eace2922cd797572912dc9c38


Bug detecting italic syntax

Most non-alphanumeric characters that proceed a * or _ cause the remaining MD to italicize. For example
test!_test
This is a test

test!*test
This is a test

The ! can be replaced with most non-alphanumeric characters with the same result.

Looks like a duplicate :bug: :bug: of https://github.com/atom/language-gfm/issues/44, gonna close this in favor of that issue.


Should "serialized" imply JSON for member codecs?

In this example we are trying to compose the result of getCodecFromSerialized with JsonFromString. The error makes it difficult to spot which member codec is the culprit.
import * as Sum from '@unsplash/sum-types';
import { getCodecFromSerialized } from '@unsplash/sum-types-io-ts';
import * as t from 'io-ts';
import { JsonFromString } from 'io-ts-types';

type MyUnion =
  | Sum.Member<'A', string>
  | Sum.Member<'B', string>
  | Sum.Member<'C', string | undefined>;
const MyUnion = Sum.create<MyUnion>();

const Codec = getCodecFromSerialized(MyUnion)({
  A: t.string,
  B: t.string,
  C: t.union([t.string, t.undefined]),
});
const CodecFromString = JsonFromString.pipe(
  // Error: Type 'readonly ["A", string] | readonly ["B", string] | readonly ["C", string | undefined]' is not assignable to type 'Json'.
  Codec,
);

If getCodecFromSerialized required Json as the output type of all member codecs, we would have a much nicer error message:
const Codec = getCodecFromSerialized(MyUnion)({
  A: t.string,
  B: t.string,
  // Type 'undefined' is not assignable to type 'Json'.
  C: t.union([t.string, t.undefined]),
});

"Serialized" in sum-types really just means unwrapping the object with its symbols to a plain tuple, giving you the raw data in its least opinionated form. I think that's appropriate as other forms of end-to-end serialization exist besides JSON.
Given this example the second line of the error is a bit narrower:
const ab: Json = null as any as ["A", string] | ["B", undefined]

Type '["A", string] | ["B", undefined]' is not assignable to type 'Json'.
Type '["B", undefined]' is not assignable to type 'Json'.

Given this is likely a common use case then a getCodecFromJSON would probably make sense.


Enable multiple resolutions - first step

This PR is a first step towards enabling running multiple resolutions (required for transi).

properly implementing trans_end/trans_release.
remove all the flattened arrays (in a first step replace them with ASSOCIATE to make changes less intrusive, we can clean the associates up in a second step)
propagate ncur_resol to fft and gemms to be able to cache multiple plans
clean up graphs/plan caches for FFT and gemms

We have to revert the ectrans-benchmark program before merge.

Present table in my sample program (currently "ectrans-benchmark", to be removed again before merge) after trans_release and before trans_end:
host:0x73d2840 device:0x331a00000 size:1180032 presentcount:0+1 line:46 name:alloc%ptr(:)
allocated block device:0x331a00000 size:1433600 thread:1
and after trans_end
Present table dump for device[1]: NVIDIA Tesla GPU 0, compute capability 9.0, threadid=1
Hint: specify 0x800 bit in NV_ACC_DEBUG for verbose info.
...empty...

Nice work @lukasm91! This will probably take a while to review properly. In the mean time, has something gone wrong with a rebase? I see duplicated commits here from develop.

I think it is because I was merging with master. Do you prefer rebases? If yes, I might restructure the commits into three new commits or so. And yes, there are conflicts after the merge today, but I can easily resolve those.

Personally I have no preference of rebasing over merging, as long as commits aren't duplicated. It looks like that's the case, but perhaps I just don't understand how Github presents merging.

Good question, I am not sure what it was showing; maybe it was because I was starting from Willems branch (which was merged in the meantime), and then I merged develop later. In any case, I quickly restructured the commits into three commits (from which the last commit we will remove before merge; this is just for show-casing the usage). It is still correct that Willems transi branch can just be merged into this and it will work (except vordivtouv/uvtovordiv, which will we have to fix separately).

This is a huge first step! It looks really nicely done!
Question on the ASSOCIATE statements. Are they required for OpenACC to work or is it to introduce minimal changes?
The ectrans-benchmark looks a little bit too inflated to my liking, though I understand the reasoning. Could we not create a smaller test that just tests that the resolutions are properly deallocated or uniquely using their own memory?

Yes, I will revert the ectrans-benchmark. I just did that to test and show how it could be used. I think we will have to test it properly, through the transi tests, so that will happen. But I wanted to have you a look at how this can now be used.
The ASSOCIATE at this point are really to keep the changes to a minimum. In a second step, we can remove then, and we should then also check if they have a performance impact.

What would be the performance cost of having arrays like ZAA in JPRD only, and reusable for single and double precision?
Then TPM_FIELDS_GPU could be in gpu_common.
I want to gauge how much of the memory can be reused (and avoided) between all CPU/GPU/single/double precision backends for the same grid-and-truncation.

So then opposite approach, could it be in JPRM and still give desired results for double precision?

For the legendre coefficients, this could indeed be worth a test. We store one wave number in double precision for a reason, but I don't know if a "mixed precision" gemm would be sufficient in that case, too (Andreas or Sam might know the details). We do the single precision GEMM at a even lower precision, so who knows, it could be sufficient. For sure not something we should do in this PR, but I can only encourage experiments of doing certain things in lower precision.

I am happy to hear you think it's an avenue worth exploring. That's what I wanted to get to. This is of course out of scope of this PR.
@samhatfield do you think you would have time to explore this avenue?
Certainly it won't be bit-identical for double precision anymore, but much accuracy is lost...?

I think the question being asked is: what would be the implications of doing DGEMM for m=0 and SGEMM for m>0 in the dp version of ecTrans, like is already done in the sp version? Is that right?

==> took care of the comments so far.

I think the question being asked is: what would be the implications of doing DGEMM for m=0 and SGEMM for m>0 in the dp version of ecTrans, like is already done in the sp version? Is that right?

That's right.

I think the question being asked is: what would be the implications of doing DGEMM for m=0 and SGEMM for m>0 in the dp version of ecTrans, like is already done in the sp version? Is that right?

That's right.

For forecasting, it would probably be fine. The real concern is assimilation. With such a change, the tangent-linear/adjoint correspondence would be degraded, and the minimisation would potentially be compromised. I actually tried this experiment a few years ago (double-precision 4D-Var but where tangent-linear/adjoint has single-precision Legendre transforms (except for m = 0 in direct transform)). The results were ... inconclusive. It might be possible, but it needs an assimilation scientist to give the thumbs up.
So in short, no I don't think we can make this change as it would have scientific consequences for data assimilation.

I agree with you this definitely is a strong deviation. I am happy you explored this in the past. Let's park this idea somewhere far away :)

In addition to the comments above there are two remaining issues with this branch (both the Cray compiler's fault - not @lukasm91's):

When D_member (where D_member => D%member) is referenced in an OpenACC region, the parent object must also be present on device. I.e., you must add D to the list of PRESENT variables in the enclosing data region. This affects files

ledir_mod.F90
leinv_mod.F90
prfi1b_mod.F90
spnsde_mod.F90
trltom_pack_unpack.F90
trmtol_pack_unpack.F90
updspb_mod.F90
uvtvd_mod.F90
vdtuv_mod.F90

Internal compiler error when building ectrans-benchmark.F90.o:[ 97%] Building Fortran object ectrans/src/programs/CMakeFiles/ectrans-benchmark-gpu-dp.dir/ectrans-benchmark.F90.o
Global is external, but doesn't have external or weak linkage!
ptr @"str2int$ectrans_benchmark_"
Global is external, but doesn't have external or weak linkage!
ptr @"print_help$ectrans_benchmark_"
LLVM ERROR: Broken module found, compilation aborted!
ftn-2116 ftn: INTERNAL

The second error above only occurs when building the GPU version, and can be resolved by moving declarations of str2int and print_help above their first usage. Weird...

Final issue compiling with Cray on AMD:
ectrans/src/programs/ectrans-benchmark.F90, Line = 1184
Unsupported OpenACC construct INIT

This is actually an issue with PR #166. As mentioned there, ACCGPU wasn't defined previously so this code was skipped. Now that we use _OPENACC, the line is passed to the compiler, and it fails. Cray doesn't support !$ACC INIT?

https://www.openacc.org/sites/default/files/inline-files/OpenACC.2.7.pdf section 2.14.1 (p62) is about !$acc init

https://www.openacc.org/sites/default/files/inline-files/OpenACC.2.7.pdf section 2.14.1 (p62) is about !$acc init
If CrayFortran does not support this, then probably following guard should be used? #if defined(_OPENACC) && !defined(_CRAYFTN)

Yeah that should do the trick.

https://www.openacc.org/sites/default/files/inline-files/OpenACC.2.7.pdf section 2.14.1 (p62) is about !$acc init
If CrayFortran does not support this, then probably following guard should be used? #if defined(_OPENACC) && !defined(_CRAYFTN)

Yeah that should do the trick.

According to man intro_openacc, Cray supports acc_init. Would that be an alternative?

Cray supports acc_init. Would that be an alternative?
If it works, then yes :)

acc_init must take a device type argument. Can I just do
call acc_init(acc_get_device_type())

?

The second error above only occurs when building the GPU version, and can be resolved by moving declarations of str2int and print_help above their first usage. Weird...

This sounds weird that it only shows up now, but we ICE can be caused as surprizing side effects ... I moved the code now.

Following changes have been done now:

The suggestions are applied
!$acc init => acc_init is done (I am checking if this is equivalent but I am almost sure it is)
I went through the files @samhatfield  suggested and added the D/R/... as I have seen usage, this needs another test of course.

One general suggestion: If you have more comments, even if they apply to multiple places, can you just add  a comment in one case, then I can reply one-by-one and we have threads to discuss. This might be easier.

Just a couple more tweaks. PR #169 has slightly screwed up this branch - sorry about that @lukasm91. You'll have to rebase against develop again.

Unfortunately this branch (with the tweaks I just suggested) doesn't run on MI250x (LUMI-G) so we'll have to rectify that before we can merge this. Firstly I see many instances of this warning:
ACC: libcrayacc/acc_datatype.c:1799 CRAY_ACC_WARNING - Asynchronous transfer not supported for deep transfers - falling back to a synchronous transfer

and eventually the benchmark program crashes with HIPFFT_PARSE_ERROR. My suspicion is that the Cray compiler still doesn't like it when you put derived types in DATA statements. We encountered this before. Of course this branch does away with flattened derived types so it's natural this issue would come up again. I'll have a think if there's a minimally invasive workaround we can implement for Cray/AMD.

I will rebase, yes. But regarding the problem with Cray Compilers - isn't this just a warning, could we put ASYNC into ifdefs for the problematic instances? I can't reallz help with the parse error, but I don't think this can be related to derived types? Have you checked if it is actually using the not inplace FFT, because I think this caused the same error?

Yeah, I'll take some time this week to look closer and see what's going on.

And LUMI just announced a 4-day service break :)

Sorry, I'm not making any progress understanding the issue on Cray systems. I'm using Frontier to debug this. If I build ecTrans and FIAT "manually" through CMake invocations etc., everything seems to work. If I build both using an ecBundle-based system, as I usually do, I get the warning mentioned above and either this error message:
ACC: libcrayacc/acc_present.c:868 CRAY_ACC_ERROR - Error placing variable 'f' (0x1a5bf80 - 0x1a5c480) from ectrans/src/trans/gpu/generated/ectrans_gpu_sp/external/setup_trans.F90:525 on the device. Another instance of the address range (0x1a5bf80 - 0x1a5c480) is already present.

OR it does "work" but gives spectral error norms of ~ 1.0, which looks like at some point the spectral fields are just being zeroed. I've seen both these behaviours while debugging but I don't understand fully the conditions under which each one occurs. Annoying.

Finally seem to be making some progress with the Cray compiler. It seems Cray doesn't do the same thing as NVHPC when copying derived types. I think you have to explicitly copy all the members, as well as the parent itself, hence the suggestions above. With these changes I can at least run the following command successfully on MI250x:
ECTRANS_USE_MPI=0 ./ectrans/build/bin/ectrans-benchmark-gpu-sp --norms --niter 100 --vordiv

which believe me I am very happy about...
I'll report back on results from proper testing soon.

Yes 🎉 just ran on 8 MI250x. Good error norms. As before, I'm seeing thousands of these warnings
ACC: libcrayacc/acc_datatype.c:1799 CRAY_ACC_WARNING - Asynchronous transfer not supported for deep transfers - falling back to a synchronous transfer

I was able to track down a few. Surprisingly the warnings are not emitted on data statements but rather on parallel regions, even ones that don't have any explicit copys in or out. Both parallel regions in LEINV emit this warning for example, which go away if you remove ASYNC(1).
We can't have thousands of identical warnings polluting STDERR, so I'm afraid we'll have to guard all of the ones responsible with #ifdef _CRAYFTNs to remove ASYNC(1), until we understand properly what is causing the warning.

Thanks a lot for debugging this and glad you made progress! :tada:   Weird, that it happen for the parallel regions, but agree puting ifdefs seems like a good workaround. Unfortunately it is up to you to add those ifdefs, because I can't do that - I will rebase this branch onto master next week.

Absolutely - forget those suggestions above: I will instead submit a PR to cleanup-part1 for you to merge. Then I think we will finally be able to merge this.

See PR https://github.com/lukasm91/ectrans/pull/1 @lukasm91.
BTW if you rebase this branch against develop again you will pick up our new GPU-based CI tests.

I merged your branch and master. Let's see if this passes tests...  @samhatfield  may you be able to help me run the build-hpc tests?
As additional tests, I also tested doing multiple alloc/dealloc; the final present table is empty. I also enable the cleanup in transi for GPU (@wdeconinck).
@samhatfield Note I did a few minor tweaks to your changes: I removed copyin scalars in setup trans for non-Cray because they do modifications in the present table that are not ideal. Also, I modified the ifdefs to really only have the ASYNC(1) inside.

I'll take a look at that failing build-hpc suite, and also re-test everything on Cray.

Everything works with Cray (testing on Frontier now). The failing build-hpc action above is simply because we haven't yet authorised PRs from external branches to run on our HPC yet. I can trigger this action manually once the branch is merged.
Great work @lukasm91! This was a real marathon 😅 Many thanks as always for your diligence.
@wdeconinck any final objections before I merge into develop?


Irrefutable pattern Message/Parser.hs line 58

Trying to run IHaskell with Atom / Hydrogen. Running it through ipython console --kernel haskell seems to work, but when I try to evaluate a simple assignment in Atom/Hydrogen, I notice this error in the console:
kernel process received on stderr: ihaskell: src/IHaskell/IPython/Message/Parser.hs:(58,5)-(63,51): Irrefutable pattern failed for pattern Success (messageType,
                                                               username,
                                                               messageUUID,
                                                               sessionUUID)

I don't know if this is helpful, but I see this log message from Hydrogen right before that:
["<IDS|MSG>", "", "{"msg_id":"execute_3d19345d-5276-43da-8b45-4134979…":0,"msg_type":"execute_request","version":"5.0"}", "{}", "{}", "{"code":"x = 2","silent":false,"store_history":true,"user_expressions":{},"allow_stdin":false}"]

This may also be useful: the way that Hydrogen is launching IHaskell:
launching kernel: /Users/radix/Projects/IHaskell/.cabal-sandbox/bin/ihaskell ["kernel", "/Users/radix/.atom/packages/Hydrogen/kernel-configs/kernel-1e0bd2f9-92d3-4a16-a65c-09100d5cba03.json", "--ghclib", "/usr/local/Cellar/ghc/7.10.1/lib/ghc-7.10.1"]

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/24656379-irrefutable-pattern-message-parser-hs-line-58?utm_campaign=plugin&utm_content=tracker%2F332399&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F332399&utm_medium=issues&utm_source=github).

Try running ihaskell install --debug first – this should tell IHaskell to print a lot of debug info to the console.
If you do that, do you get any more debug dump info?

Ok @gibiansky I did  that. I'm not totally sure if this is new output but after doing this I also saw this in the console. Doesn't look like it has anything interesting but I don't know:
kernel process received on stdout: Header: {"msg_id":"execute_58905133-a261-4f52-ad29-9e2f84517895","username":"","session":0,"msg_type":"execute_request","version":"5.0"}
Content: {"code":"x = 2","silent":false,"store_history":true,"user_expressions":{},"allow_stdin":false}

Followed by the same irrefutable pattern on stderr.

I think it's because Atom doesn't provide a username or session field.

Is that a bug I should file against Hydrogen? or should handling lack of username/session be handled by Haskell?

Wait, perhaps not. They seem to exist on the kernel process received on stdout, but not in the original log message from Hydrogen. So perhaps hydrogen inserts them afterwards.

Ah, I think I see what's going on. The message and session IDs need to be valid UUIDs. IPython doesn't enforce this, but IHaskell requires them to at least be strings. This is a bug in Hydrogen, as they don't follow the message spec detailed here.
That said it's easy to accommodate this in IHaskell. I would accept a PR that allows numbers for UUIDs I guess; see UUID.hs.

So, to be clear, it seems that Hydrogen is sending msg_id as a string, but not a UUID. And worse, it's sending session as a number (0) instead of a string (containing a UUID), right? I just want to make sure so I can file a bug with them.

Both of these are technically not allowed and should be valid UUIDs. msg_id is almost a valid UUID, I think, except the execute_ prefix, perhaps. sessions is sent as a number.
IHaskell allows any string to be used as a UUID, which is actually slightly more general than the messaging spec says it should be.

So for the purposes of getting this fixed: session needs to be a String, not a number.
That said this is just my guess at the solution based on rereading the parsing code and looking at that request. No 100% guarantees that that's the only issue, though.

@gibiansky turns out you're right: I've fixed it in Hydrogen and filed a PR with them. Thanks for helping me debug this :) I can at least run "x = 1" and evaluate "x" to get back 1 now :)

Great!


Merge codegen

Similar process to #61, for getting a cleaner history for merging codegen.
The java codegen has been filtered out, while core codegen is kept. The build still refers to codegen-java which will need to be updated. I think the plan is to include codegen core in the SDKs currently, rather than as a dependency. We can otherwise filter out the core as well, and then update the build.
Rebased to remove merge commits. Commits all prefixed with codegen. Rebased on current main branch to create related history.

NOTE: don't squash and merge this


Some defaults are not printed in the table

Hello, I've noticed this bug while playing with this lib, it doesn't render default variables == 0, None, [], {}, False
it is related to this line https://github.com/Frefreak/mdantic/blob/master/mdantic.py#L79
default = str(f.default or "")

with a little adjustment it works fine,
default = str(f.default if f.default is not None else "")

Thanks!


[feature] Multiple api-version and version resiliency SDK

One multiple api-version client library will contain/support operations and models belonging to multiple service version.

[ ]  multiple api-version client generation

[ ] multiple api-version and resiliency operation generation:

[ ] mulitiple api-version and resiliency model generation: https://github.com/Azure/autorest.csharp/issues/2354 https://github.com/Azure/azure-sdk-for-net/issues/47658

Current We have no plan to support the comprehensive multiple-apiVersion sdk. close this issue first.


CreatedAt is type of int64，when i update the col created_at, it switch into a string of time! Why

Your issue may already be reported! Please search on the issue track before creating one.
What version of Go are you using (go version)?
go version go1.9.6 windows/amd64
Which database and its version are you using?
mysql
Please provide a complete runnable program to reproduce your issue. IMPORTANT
type ConfigRoute struct {
	RouteId        int64 `gorm:"PRIMARY_KEY"`
	NodeId         string
	Endpoint       string
	StartAccountId int64
	EndAccountId   int64
	CreatedAt      int64
	UpdatedAt      int64
	StartAt        int64
	ExpireAt       int64
	Disabled       bool
}

func (route *ConfigRoute) UpdateDisabled() error {
	nowMs :=<PHONE_NUMBER>096
	ret := db.Model(route).Updates(map[string]interface{}{
		"disabled":   route.Disabled,
		"updated_at": nowMs, // but if using int(nowMs),it would be ok.
	})
	if err := ret.Error; err != nil {
		return err
	}
	return nil
}

func main(){
 // initialize db connection.
   route := &ConfigRoute {
      /....
   }
   route.Save()
   route.Disabled = false
   route.UpdateDisabled() // err occurs!!!
}

Here is the log lines：
(D:/src/cryptobroker/trademaster/dto/route.go:55)
[2019-01-04 10:18:05]  Error 1265: Data truncated for column 'updated_at' at row 1

(D:/src/cryptobroker/trademaster/dto/route.go:55)
[2019-01-04 10:18:05]  [1919.86ms]  UPDATE `config_routes` SET `disabled` = 'true', `updated_at` = '2019-01-04 10:18:04'  WHERE `config_routes`.`route_id` = '1'
[0 rows affected or returned ]

I couldn't reproduce the problem with the code you posted, but I believe that this problem can be solved with a slight change in your code.
Rather than dealing with update timestamps, let Gorm deal with them for you by adding gorm.Model to your structure.
Change your structure from what you posted to:

type ConfigRoute struct {
        gorm.Model
	NodeId         string
	Endpoint       string
	StartAccountId int64
	EndAccountId   int64
	StartAt        int64
	ExpireAt       int64
	Disabled       bool
}

Note that this will also add a primary key ID. You can find more information here.
If you want to keep your current approach can you maybe provide the smallest program you can which reproduces this bug? Then I would gladly take a look at it 💯

created_at, updated_at are the default time type in gorm, and will set to time.now when create/update.
you can remove the callback after the db init
db.Callback().Create().Remove("gorm:update_time_stamp")
db.Callback().Update().Remove("gorm:update_time_stamp")

saved my life!


license error, adding  ViewController error

Hi @tsukisa,
I have troubles with this tool, which I think would be so useful for my apps. Please help me to understand how to use it in my app. When I install the pod, I got this error message.

I Install the pod, and I created the five viewControllers, centerViewControllers, left, right, top, bottom, (and in the mainstoryboard as well, UIviewControllers and files assigned to them) like this:

but when I run the app, this error prompts :

Please help me,
thanks in advance.

Have you figured out how to implement this? Mine looks exactly like yours right now and nothing is happening


refactor: pytest cleanup

Refactor pytest tests.
Also disable various pylint rules implemented in Ruff.
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have run pre-commit for code linting and typing (pre-commit run)
[x] I have made corresponding changes to the documentation
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes (tox -e testenv)

PR needs new tests to pass SonarCloud - will check it on Monday


Signing failed with error -2147012744.

Describe the Bug
Environments:

Windows Server 2019 Standard 64-bit (10.0, Build 17763) (17763.rs5_release.180914-1434)
Windows 10 Pro 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
.NET SDK Version: 8.0.202
Tool Version: 0.9.1-beta.24312.3 (same behavior in older versions)

When executing dotnet sign on a .nupkg file using Azure Key Vault, we encounter the following error:
Signing failed with error -2147012744.

Error Code: Hex: 0x80072EE8
Commandline output: Even when running as an administrator.
MSTest integration test that executes the dotnet.exe process with signing also fails.

Interestingly, this issue does not occur when running the code from an Azure Pipeline on a provided Windows machine. It suggests we might be missing a prerequisite on our personal computers. It does happen on our Jenkins setup running on the above specified windows 2019 server.
After some investigation, it appears the error originates from deep within the code, specifically from calls to the Windows OS to sign the assemblies inside the .nupkg.
// Example of the error code
logger?.LogTrace("Calling SignerSignEx3");
var result = mssign32.SignerSignEx3;

// Signing code snippet
try
{
    using (var ctx = new Kernel32.ActivationContext(manifestFile))
    {
        code = signer.SignFile(
            file.FullName,
            options.Description,
            options.DescriptionUrl?.AbsoluteUri,
            pageHashing: null,
            _logger);
        success = code == S_OK;
    }
}
catch (Exception e)
{
    _logger.LogError(e, e.Message);
}

if (success)
{
    _logger.LogInformation(Resources.SigningSucceeded, file.FullName);
    return true;
}

_logger.LogError(Resources.SigningFailedWithError, code);

The problem is that I'm not familiar enough with the SignerSignEx3 WinAPI call and it seems like documentation will be private to Microsoft. At this point, it's likely that I've gone too deep and am missing a more obvious issue.
Repro Steps

Create a .nupkg that includes .net462 assemblies.
Execute the following C# code (assuming dotnet is a class that starts dotnet.exe as a process with the provided string as an argument):

dotnet.Run("new tool-manifest");
dotnet.Run("tool install sign --version 0.9.1-beta.24312.3");

string command = $"sign code azure-key-vault \"{packageAbsolutePath}\" " +
"--publisher-name \"Fake Company\" " +
"--description \"Fake Signing\" " +
"--description-url \"https://www.fakecompany.be/\" " +
"--azure-key-vault-tenant-id \"123-fake-id\" " +
"--azure-key-vault-client-id \"123-fake-id\" " +
$"--azure-key-vault-client-secret \"{nugetStore.NuGetCertificatePassword}\" " +
"--azure-key-vault-certificate \"CompanyDefault\" " +
"--azure-key-vault-url \"https://common-certificates.vault.azure.net/\" " +
$"--output \"{resultDirectory}\"";

dotnet.Run(command);

Expected Behavior
Expecting to see a signed .nupkg with all content signed.
Actual Behavior
Signing starts, then it retries over and over again until it throws an exception.
Standard Output:
Tool 'sign' is up to date (version '0.9.1-beta.24312.3' manifest file C:\GIT\Solutions\Internal\CICD\SysDev\NuGetSigningIntegrationTests\bin\Debug\.config\dotnet-tools.json).
fail: Sign.Core.ISignatureProvider[0]
      Signing failed with error -2147012744.
      (repeats multiple times)
fail: Sign.Core.ISignatureProvider[0]
      Failed to sign. Attempts exceeded.
fail: Sign.Core.ISigner[0]
      Could not sign C:\Users\JANS\AppData\Local\Temp\dno54dl1.xna\lib\Newtonsoft.Json.dll.
      System.Exception: Could not sign C:\Users\JANS\AppData\Local\Temp\dno54dl1.xna\lib\Newtonsoft.Json.dll.
         at Sign.Core.AzureSignToolSignatureProvider.<>c__DisplayClass7_0.<<SignAsync>b__0>d.MoveNext() in /_/src/Sign.Core/SignatureProviders/AzureSignToolSignatureProvider.cs:line 110
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Sign.Core.AzureSignToolSignatureProvider.SignAsync(IEnumerable`1 files, SignOptions options) in /_/src/Sign.Core/SignatureProviders/AzureSignToolSignatureProvider.cs:line 104
         at Sign.Core.AggregatingSignatureProvider.SignAsync(IEnumerable`1 files, SignOptions options) in /_/src/Sign.Core/SignatureProviders/AggregatingSignatureProvider.cs:line 204
         at Sign.Core.AggregatingSignatureProvider.SignAsync(IEnumerable`1 files, SignOptions options) in /_/src/Sign.Core/SignatureProviders/AggregatingSignatureProvider.cs:line 92
         at Sign.Core.Signer.<>c__DisplayClass3_0.<<SignAsync>b__0>d.MoveNext() in /_/src/Sign.Core/Signer.cs:line 156
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Sign.Core.Signer.SignAsync(IReadOnlyList`1 inputFiles, String outputFile, FileInfo fileList, DirectoryInfo baseDirectory, String applicationName, String publisherName, String description, Uri descriptionUrl, Uri timestampUrl, Int32 maxConcurrency, HashAlgorithmName fileHashAlgorithm, HashAlgorithmName timestampHashAlgorithm) in /_/src/Sign.Core/Signer.cs:line 85

Additional Context

Sign Version: 0.9.1-beta.24312.3
dotnet info: dotnet --info

.NET SDKs Installed

2.1.526, 2.2.207, 3.1.426, 5.0.408, 5.0.416, 6.0.202, 6.0.203, 6.0.321, 8.0.106, 8.0.202

.NET Runtimes Installed

Microsoft.AspNetCore.All 2.1.30, 2.2.8
Microsoft.AspNetCore.App 2.1.30, 2.2.8, 3.1.32, 5.0.17, 6.0.5, 6.0.28, 6.0.31, 7.0.17, 8.0.6
Microsoft.NETCore.App 2.1.30, 2.2.8, 3.1.32, 5.0.17, 6.0.5, 6.0.28, 6.0.31, 7.0.17, 8.0.6
Microsoft.WindowsDesktop.App 3.1.32, 5.0.17, 6.0.5, 6.0.28, 6.0.31, 7.0.17, 8.0.6

Other Architectures Found

x86: [C:\Program Files (x86)\dotnet]

Registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment Variables

Not set

global.json File

Not found

Learn More

https://aka.ms/dotnet/info

Download .NET

https://aka.ms/dotnet/download

Hi @janstaelensskyline unfortunately, those versions of Windows are both out of the support lifecycle, so there's not a lot we can do for those.
That said Sign CLI carries its own copies of the Authenticode signing libraries and that activation context should be forcing the tool to use those versions instead of the ones the OS carries. @dtivel, looks like there may be an issue here? Can we confirm that the correct libraries are being loaded and used at runtime?

Hello @clairernovotny , understood. Do you have a list with the mainstream support for what version of windows OS is supported and part of your QA cycle for the signing of .nupkg?

Update on this issue. I've continued debugging this on my end.
It's been strange...
I tested running the AzureSign Tool on an assembly and the KeyVaultSign tool on a .nupkg directly with our same azure keyvault settings and both those tools worked on every PC and Server. Only the sign tool itself kept failing.
Up to an including 19/06/2024
Using the exact same sign command as mentioned in the first post, version of the tool and the same .nupkg package.
All the servers & pc's have .NET8 installed.
On azure pipelines using windows host: WORKS
On Github pipelines using windows host: WORKS
On 2 Pc's with Windows 10:
FAILS WITH ERROR -2147012744
On 1 PC with Windows 11:
FAILS WITH ERROR -2147012744
On Windows Server 2019:
FAILS WITH ERROR -2147012744
On Windows Server 2022:
FAILS WITH ERROR -2146762749

Today on 20/06/2024
the command suddenly started working on
2 PC's with Windows 10:
1 PC with Windows 11:
Both the servers are still failing (2019 and 2022) but with different error codes.
I spend a while checking to see what 'changed' on the PC's that didn't on the servers but I'm not finding much. I saw a windows update for microsoft defender: KB2267602. But when I installed that on the server nothing changed.
There was no reboot or restart for one of the win10 pc's for over 7 days so I'm not immediately seeing other windows updates being a factor there.
On applications I didn't see any updates that every PC had and that the servers might've been missing.
Currently I'm not able to really figure this out. I'll setup our self-hosted jenkins pipelines to use the NuGetKeyVault tool directly for now, but we'd prefer to use the same dotnet sign tool on all our CI/CD.

I think I had the same problem and in my case, installation of "Microsoft Visual C++ 14 runtime" (https://aka.ms/vs/17/release/vc_redist.x64.exe) helped. It's required component, described in readme (https://github.com/dotnet/sign?tab=readme-ov-file).
Azure Pipelines agents have some preinstalled components, that's why there signing worked.

@janstaelensskyline, see https://github.com/dotnet/sign?tab=readme-ov-file#prerequisites .  I've updated that section since this issue was opened.
If you try using the latest Sign CLI, it will raise a warning if the Microsoft Visual C++ 14 runtime" is not already installed.

Test Results for Latest .NET Sign (Version 0.9.0-beta.24469.1)
That's excellent news indeed. I've tested the latest dotnet sign version 0.9.0-beta.24469.1:
Windows 10

.nupkg: OK
.dll: OK

Windows 11

.nupkg: OK
.dll: OK

Windows Server 2019

.nupkg: OK
.dll: OK

Windows Server 2022

.nupkg: OK
.dll: OK

Thank you for the follow-up and fixes. We'll start/continue using the pre-release in our environments and will notify you if we see anything out of the ordinary.


core: transparent encrypt layer support

Many storage tools introduce transparent encryption support, like https://rclone.org/crypt/.
In the architecture of OpenDAL, it seems that this layer of transparent encryption is not a very complicated thing, and it is also a common requirement in many production projects. Therefore, do we need to plan to support this aspect?

Therefore, do we need to plan to support this aspect?

It would be preferable to begin if we have some actual use cases instead of 'good to have'.

Close until we have a user's request.


Develop into main

This merges the work of Kristen into main.

The functionality is already merged into develop and can be tested there: https://share.streamlit.io/gromopo/gromopo/develop/streamlit/GroMoPo_app.py
Currently a lot of stuff seems to be broken.

@samzipper Awesome! @kbefus you need to approve please so that I can merge.


remove codegen::Handlers

codegen::Handler is a noop trait. It's used everytime codegen::Emitter must be created and there were comments that it should be removed. So I went ahead and did that. @kdy1 if you want to use that trait in the future feel free to close this PR.

Thanks for the pr!
I previously closed a similar pr because I thought I will use it while testing, but I found better solution for #583, and I think the solution for #583 also can be used to test type engine


Feature/zod input validation

This pull request introduces input validation for various user-related actions and quiz functionalities in the backend using Zod. The following validation schemas have been created to ensure that the input data adheres to specified formats and constraints, enhancing the robustness of the application.
Fixes #32
Changes Made:

User Registration:

registrationSchema: Validates username, email, and password with requirements for minimum/maximum lengths and formats.

User Login:

loginSchema: Validates the presence of a valid email and password.

Username and Password Update:

updateUsernameSchema: Ensures that the username follows the specified length constraints.
updatePasswordSchema: Validates the password with required constraints.

Password Reset Request:

requestPasswordResetSchema: Validates that the provided email is in the correct format.

Quiz Functionality:

questionSchema: Validates the question text, options, and the correct answer, ensuring at least two options are provided.
quizSchema: Validates the title, questions (ensuring at least one), and time limit for the quiz.
questionUpdateSchema and quizUpdateSchema: Allow optional updates to questions and quizzes while maintaining necessary constraints where applicable.

Benefits:

Improves data integrity and security by ensuring that only valid data can be processed.
Provides clear error messages for invalid inputs, enhancing user experience.
Facilitates easier maintenance and readability of the validation logic.

Additional Notes:

The Zod library was used for its simplicity and type safety, making it easier to manage validation schemas.

Please review the changes and let me know if there are any questions or further modifications needed.

PR #35 adds Zod validation for better schema validation and error handling.

the response should be like this but your response is this

if I will merge this this will create lots of issues in current scenerio please send me req and res formats for login and register then I will merge it if it looks good...

and we are sending json as response not html

create new PR after resolving this issues. Thanks for your valuable efforts. I am expecting a good PR soon...
I am closing this one


PyDPF documentation search doesn't work

Description of the modifications
PyDPF documentation search doesn't work - it was resolved some time back but now it doesn't.
Useful links and references
No response

@Revathyvenugopal162 do you think you could have a look?
it seems there is no link between what's put in the search bar on the top-right and what ends-up in the actual search engine.
Also the result is cropped on the left side as we already discussed.

@Revathyvenugopal162 and @PProfizi can we please prioritize this issue? This is an important one.
cc: @rlagha, @smedikon, @MaxJPRey, @pmaroneh

Adding @jorgepiloto for visibility.

Hi @ayush-kumar-423, thanks for reporting this issue. I can reproduce, the search bar does not work. Let me investigate a bit more.

Hi @jorgepiloto, @PProfizi
Unfortunately, it still doesn't work (see screenshot below), the results are still cropped and the first search doesn't provide any results:

Hi @ayush-kumar-423. I opened https://github.com/ansys/pydpf-core/pull/1401. I am clueless about why the Meilisearch CSS and JS is not being injected in the HTML files.

Thanks @jorgepiloto, seems it is generic. Same for PyMAPDL doc.

The issue has been resolved in https://github.com/ansys/pydpf-core/pull/1401, and now the search function is working as intended.https://dpf.docs.pyansys.com/version/dev/index.html. So closing the issue.


Multi-Patient API: Client Assertion Claims Suggestions

I'm attempting to integrate the client assertion generated by the test kit with my auth provider. It's strict about the format of the generated client assertion and I have some suggestions after reviewing the IETF RFC for JWTs.

Change the iss (issuer) claim to something that is unique to the server (URL would be great) that is generating the token vs just the client id of the application. The RFC says The JWT MUST contain an "iss" (issuer) claim that contains a unique identifier for the entity that issued the JWT. The client id is not unique to the entity that issues the JWT as it's tied to the client it. I suggest Inferno::Application[:base_url].

Add the iat (issued at) / claim. My auth provider checks to ensure tokens aren't issued for longer than 61 minutes so it requires `iat. This can just be the current date/time when the assertion is generated.

Add a x5t/x5c claim for IDPs that require certificate based assertions. The kid/x5c claims can be the same.

Add a .well-known/openid-connect endpoint for IDPs that require this to confirm the ownership of the JWK.

I'm happy to submit a PR but want to ensure agreement first.
Thanks!!

According the Bulk Data v1 IG and the SMART App Launch v2 IG, the iss claim is the client id. I don't see a problem here unless that RFC has a specific definition of "entity" that conflicts with this.
Regarding iat, as a general rule, we do not include optional fields as part of the certification tests. If we did, then systems could impose additional requirements on clients by saying they must include fields which the spec says are optional. I think we would be unlikely to make a change here without input from ONC and the authors of the spec.


Structre

Ideally it should include centre as well as state election promises as well.
We are going to need elections promises either as leaflet images or urls to confirm.
More on this later.

So this is the BJP election manifesto: http://www.bjp.org/images/pdf_2014/full_manifesto_english_07.04.2014.pdf
As a non coder, I wanted to know if there would be any way I could help...

Yeah sure, we are going to need all those manifestos, and status updates on them, with news or assembly resolutions etc. Can you work on this ?

So I found this too: https://aapdelhidiary.wordpress.com/
They are doing a pretty good job of keeping track of the delhi govt. You think we can use their data as dummy data and design a website around it?
And yes, I will start working on that too

This is thorough. We can do that. But we should get structure of the website fixed first. Centre should come first, followed by states.
We need all election promises and related material first. Let's start digging.

In my opinion, let us focus on getting the BJP manifesto tracked for starters. Once we have a core team in place to monitor the development, we can branch out phase wise. Let us have a phase-wise development rather than a website with many manifestos and no tracking. We can identify people for each State who will be responsible for the task when they go live.

I agree with @kjaggu. Let's start with the Center. It should be trivial to replicate it for the states as long as we have people tracking them.

I am an AAP member/volunteer. I was quite inspired by the canadian PM tracking stuff  and therefore I created a structure in Excel based on the AAP Delhi Manifesto. I will clean it up a bit in another 15 minutes and share it. May be we can use it as a base to create a structure or atleast get some ideas.

@gitrraoind. Thank you. Maybe you could help us with Delhi state promise tracker. Can you take it on ?

PFA the sheet. Please go through all the worksheets in the attachment.
Yes, I would love to collect the AAP Delhi data, if that is what you mean. However, it will more likely than not be from public domain or AAP sources. I need ideas on how to make sure to have them verified!
Progress Tracker v1.1.xlsx

What's the current status for the dynamic website?

Count me in. Know extensive web development - FE and BE.

I am learning Web Dev, so I wouldnt be able to contribute directly unless there is a lack of volunteers and you are willing to go for someone who is learning+doing.
I can help with collating data, and my idea of an excel sheet for this is the following columns-
PromiseReferenceId;Promise;Promisee;LocationOfMakingPromise;Status-(UnderConsideration;U-Turn;Active;Fulfilled)
Each promise should link to its own page and also allow for moderated commenting under each promise.

@IndDev Check out the slack channel and join in on the data collection.. work on that has already started.
Slack channel: https://indianpromisetracker.slack.com
Get an invite: https://iptuserbot.herokuapp.com/

I have the list of AAP Delhi's manifesto promises ready and a fair amount of data on what has been achieved! I am new to github and don't know whom to connect with to have this info uploaded on the site. Any info is appreciated!
Also, could we show the 'Days in office' rather than the 'In Power Since' date? This will create a better perspective of the situation!

@rraoind first of all, welcome! :)
Secondly, can you dm me on slack? It's a messaging app that everyone here is using!
To join the slack channel, go here: https://indianpromisetracker.slack.com/
You will need an invite to join the channel, which you can get here: https://iptuserbot.herokuapp.com/
It's fairly easy. It wont take much time! And my username is @greyhound291 on slack

Basic structure is fixed, let's move to specific issues of content generation and ui design now.


Command variable substitution hangs tasks.json

Issue Type: Bug
Specify a task in tasks.json (also happens in launch.json) that uses ${command: command_id } in args or options env (places I've tried).
As an extension developer, I've made some predefined tasks.json that uses an extension command (specified by command_id) in those tasks with variable substitution. This doesn't seem to work anymore after 1.47 config and workspaceFolder variables seems to work.
Steps to reproduce:

Make a task in tasks.json that uses ${command: command_id } of some extension.
Try to execute this task with F1 -> Run task
Nothing happens. No Error and no output log.

 Does it still occur if you disable all extensions? It is possible that an extension is getting stuck during activation and is causing this (in which case you can file an issue with that extension).
My task:
    {
      "label": "My Task",
      "type": "shell",
      "command": "echo",
      "args": [
        "${command:extension.helloWorld}"
      ],
      "problemMatcher": []
    }

My command:
const disposable = vscode.commands.registerCommand('extension.helloWorld', async () => {
		return 'My command output';
	});

@alexr00 found the root of the problem and is related to the extension. Sorry for the troubles and thank you.


Draft pull

Re check

Hi @JonathanTylerCombs please let us know why you opened this PR. In the meantime, I am closing it.


move calls to request to lattice.py

Just realized this branch should actually refer to TOOLS-172. Here is the testing: https://lattice.atlassian.net/browse/TOOLS-172?focusedCommentId=18099

Thanks for catching that. I've removed the if statement, and reran LATDF477OUM just to double check.


Vehicle service multiselect

Following out a side discussion from #3517..
Now that iD supports multicombo fields (see #3034, #3080), we have a really good way to allow people to tag multiple selections.
A few fields in OSM still prefer semicolon separated strings to represent multiple values.
For example:  service=* (see shop=car_service),

service=dealer;repair;parts;tyres;electrical;glass

This one is especially bad because the service=* tag is overloaded for service highways.
(Compare to the service:bicycle:* namespace, which already exists and handles multiple values cleanly.)
So my question is, when we encounter tags like this, where

the tag has obvious usefulness, and
the existing tag is barely used if at all, and
the wiki recommendation is clearly bad

Can we just disregard what the wiki says, and create a multiselect preset that expands to tags like service:vehicle:dealer service:vehicle:repair,service:vehicle:parts, etc.
Does anybody care about this vehicle service= tag enough that changing it would cause major problems?

I agree with using the service:vehicle namespace tag for shop=car, shop=car_repair and shop=motorcycle.

Looks like service=* is only used in ~3800 of the ~100,000 shop=car_repair tags worldwide. So I agree with the change as it shouldn't cause major problems, and will make it easier to add services and bring that number up.
Though just wondering if the generic service:vehicle is correct or whether it should be service:car since it's car repair?

I was mostly thinking "vehicle" so it covers cars, trucks, motorcycles, etc.

I did this..


failed:lookup getaddrinfow:An unrecoverable error during a database search.

When I try to execute a query I got the following error

lookup @localhost: getaddrinfow: An unrecoverable error during a database search.

My code
import (
	"database/sql"
	"flag"
	"fmt"
	"log"

	_ "github.com/denisenkom/go-mssqldb"
)

var (
	debug         = flag.Bool("debug", true, "enable debugging")
	password      = flag.String("password", "436440", "password")
	port     *int = flag.Int("port", 1433, "the database port")
	server        = flag.String("server", "@localhost", "db")
	user          = flag.String("user", "sa", "user")
	connStr       = fmt.Sprintf("server=%s;Initial Catalog=Gestion;userid=%s;password=%s;port=%d", *server, *user, *password, *port)
)

func ReadVersiones(db *sql.DB) (int, error) {

	rows, err := db.Query("SELECT IdVersion, Nombre FROM dbo.VersionesApp")
	if err != nil {
		fmt.Println("Error reading rows: " + err.Error())
		return -1, err
	}
	defer rows.Close()
	count := 0
	for rows.Next() {
		var Nombre, location string
		var Id int
		location = "pepito"
		err := rows.Scan(&Id, &Nombre)
		if err != nil {
			fmt.Println("Error reading rows: " + err.Error())
			return -1, err
		}
		fmt.Printf("ID: %d, Name: %s, Location: %s\n", Id, Nombre, location)
		count++
	}
	return count, nil
}

func main() {
	flag.Parse()

	if *debug {
		fmt.Printf(" password:%s\n", *password)
		fmt.Printf(" port:%d\n", *port)
		fmt.Printf(" server:%s\n", *server)
		fmt.Printf(" user:%s\n", *user)
	}

	if *debug {
		fmt.Printf(" connString:%s\n", connStr)
	}
	conn, err := sql.Open("mssql", connStr)
	if err != nil {
		log.Fatal("Open connection failed:", err.Error())
	}
	defer conn.Close()

	count, err := ReadVersiones(conn)
	if err != nil {
		log.Fatal("ReadVersiones failed:", err.Error())
	}
	fmt.Printf("Read %d rows successfully.\n", count)
}

Im using SQL EXPRESS 2005, I've tried a variety of connectionStrings formats, but no one seems to work.

I don't think we support SQL Express 2005. Is it possible to upgrade to something newer?

I don't think we support SQL Express 2005. Is it possible to upgrade to something newer?
Yup, I just was testing it for fun. Just assumed it supports Express 2005 since it supports SQL 2005.
My bad


K8SPS-315 do not restart cluster if don't use CustomConfigKey as config name in config map.

CHANGE DESCRIPTION
Problem:
Short explanation of the problem.
According to the code we can use only my.cnf as config name in config map. If we use any other name we restart the
cluster but do not apply the changes
Cause:
Short explanation of the root cause of the issue if applicable.
Do not restart the cluster and add an error if we use not default config name.
Solution:
Short explanation of the solution we are providing with this PR.
For now we create config map anyway, but do not apply it. It might be better even do not create config map in case of a wrong data.
CHECKLIST
Jira

[ ] Is the Jira ticket created and referenced properly?
[ ] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
[ ] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

[ ] Is an E2E test/test case added for the new feature/change?
[ ] Are unit tests added where appropriate?

Config/Logging/Testability

[ ] Are all needed new/changed options added to default YAML files?
[ ] Did we add proper logging messages for operator actions?
[ ] Did we ensure compatibility with the previous version or cluster upgrade process?
[ ] Does the change support oldest and newest supported PS version?
[ ] Does the change support oldest and newest supported Kubernetes version?

Test name
Status

async-ignore-annotations
passed

auto-config
passed

config
passed

config-router
passed

demand-backup
failure

gr-demand-backup
passed

gr-demand-backup-haproxy
passed

gr-haproxy
passed

gr-ignore-annotations
passed

gr-init-deploy
passed

gr-one-pod
passed

gr-recreate
passed

gr-scaling
passed

gr-self-healing
failure

gr-tls-cert-manager
passed

gr-users
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

recreate
passed

scaling
passed

service-per-pod
passed

sidecars
passed

smart-update
passed

tls-cert-manager
passed

users
passed

version-service
passed

We run 30 out of 30

commit: https://github.com/percona/percona-server-mysql-operator/pull/558/commits/d8fe73d49e1701e85600cfe541c8c313754fd549
image: perconalab/percona-server-mysql-operator:PR-558-d8fe73d

Test name
Status

async-ignore-annotations
passed

auto-config
passed

config
passed

config-router
passed

demand-backup
failure

gr-demand-backup
passed

gr-demand-backup-haproxy
passed

gr-haproxy
passed

gr-ignore-annotations
passed

gr-init-deploy
passed

gr-one-pod
passed

gr-recreate
passed

gr-scaling
passed

gr-self-healing
passed

gr-tls-cert-manager
passed

gr-users
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

recreate
passed

scaling
passed

service-per-pod
passed

sidecars
passed

smart-update
passed

tls-cert-manager
passed

users
passed

version-service
passed

We run 30 out of 30

commit: https://github.com/percona/percona-server-mysql-operator/pull/558/commits/d8fe73d49e1701e85600cfe541c8c313754fd549
image: perconalab/percona-server-mysql-operator:PR-558-d8fe73d

Test name
Status

async-ignore-annotations
passed

auto-config
passed

config
passed

config-router
passed

demand-backup
failure

gr-demand-backup
passed

gr-demand-backup-haproxy
passed

gr-haproxy
passed

gr-ignore-annotations
passed

gr-init-deploy
passed

gr-one-pod
passed

gr-recreate
passed

gr-scaling
passed

gr-self-healing
passed

gr-tls-cert-manager
passed

gr-users
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

recreate
passed

scaling
passed

service-per-pod
passed

sidecars
passed

smart-update
passed

tls-cert-manager
passed

users
passed

version-service
passed

We run 30 out of 30

commit: https://github.com/percona/percona-server-mysql-operator/pull/558/commits/d8fe73d49e1701e85600cfe541c8c313754fd549
image: perconalab/percona-server-mysql-operator:PR-558-d8fe73d


Tokens search case sensitivity

Context information
Just a tracking of https://github.com/aeternity/ae_mdw/issues/1549 on our side
Check after implementation
Steps to reproduce
there is a difference is search of wrapped and Wrapped
https://aescan.io/search/Wrapped?type=tokens
https://aescan.io/search/wrapped?type=tokens
What is expected?
should return same result
What is actually happening?

@janmichek To check if it works as expected, after the mdw releases it.

Soleved on MDW side


TC.3.2 When pressing the button the Game should be ended,

b858b420e58b3a77886044d8f59b48e2df55c1c6

This task has been finished on the day with posted SHA comment, in the interest of correctly closing the issue (and us having to first learn how to), it has been closed a while after the finishing of the task.

We have in the meantime figured out, that we can not correctly close it once it is done (otherwise the commit is kinda useless).


Add Docker image support

I updated Gradle since the project failed to build with the latest image from Docker hub.

This change is

@danielkza if you rebase I'll look at merging this.


Stop get id by content version using the alias id

Resolve the alias to a sha when using the get content by id functions. This will do until we decide what we want to do with sha functionality.

Fixed in da7b4a89c9cc0ea106d3c7551e1bb466cfa02ae8


Feedback from a recent PR I forgot to push

I forgot to push this!
/assign @vincepri

Thanks for following up on those!


WIP: feat: Publish events via SSE

Motivation
Publishes Server Sent Events (SSE) allowing external consumers to listen for events created by Armiarma.
Using https://github.com/r3labs/sse
Description
Creates a new events package, with a Forwarder struct that listens for "internal" events, annotates them with extra data, and published a new event over SSE that is sanitized. Attempting to avoid importing "internal" structs in these events so we have a clear seperation of concerns.
Created 2 initial events:

ethereum_attestation - A raw Ethereum attestation
timed_ethereum_attestation - An Ethereum attestation with additional data like peer/timing info

Tasks
Unknown
Proof of Success
$ curl  "http://localhost:9099/events?stream=timed_ethereum_attestation" -H "accept: text/event-stream"

id:
data: {"attestation":{"aggregation_bits":"0x0000000000000000000000000000000000000000000040000000000000000080","data":{"slot":"7074121","index":"19","beacon_block_root":"0x0f45217aed1a9a492f2b5e39e95b91be3f7c50479ec8b8faed3d49f556974c4d","source":{"epoch":"221065","root":"0x4e5a9cbe289e25dd4b4307ac1040fbb422b5f701165c5670d6e025101d3e642a"},"target":{"epoch":"221066","root":"0xd71fd1327a0f4342b4bb16b2f4b7a519bc96b8df0ec82b26f608ac1c93ba7e70"}},"signature":"0x877e8383053985855862de5e581529c5c609626cbdaf9df58cee4dc47efb8abc9b1a86e69584c80d8ea74ca15c14a50404541654078113a85f07e83664786e4f9df14b8a73e2a7a7fd02937e442161fc5f3d1904737fa3bc2845b95851a614ce"},"attestation_extra_data":{"arrived_at":"2023-12-01T02:24:23.201281754Z","peer_msg_id":"OpxZmqvTwl6qGZpfDUvxL1i0Q5AbvFPSXxdhEz4DNbg=","subnet":19,"time_in_slot":9683988201281754},"peer_info":{"id":"\u0000%\u0008\u0002\u0012!\u0003oђ\u0004\ufffdu\u0013r2\ufffd\ufffd\ufffdn\ufffd\ufffd\ufffd4\ufffdn\u00029\ufffdJ\ufffd#\u000f\u0008i\u0016\ufffd\u000cv","ip":"<IP_ADDRESS>","port":9000,"user_agent":"Lighthouse/v4.5.0-441fc16/x86_64-linux","latency":276000000,"protocols":["/eth2/beacon_chain/req/ping/1/ssz_snappy","/eth2/beacon_chain/req/status/1/ssz_snappy","/meshsub/1.0.0","/eth2/beacon_chain/req/metadata/1/ssz_snappy","/ipfs/id/push/1.0.0","/eth2/beacon_chain/req/beacon_blocks_by_root/2/ssz_snappy","/eth2/beacon_chain/req/beacon_blocks_by_root/1/ssz_snappy","/eth2/beacon_chain/req/goodbye/1/ssz_snappy","/ipfs/id/1.0.0","/eth2/beacon_chain/req/beacon_blocks_by_range/2/ssz_snappy","/eth2/beacon_chain/req/metadata/2/ssz_snappy","/eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy","/meshsub/1.1.0"],"protocol_version":"eth2/1.0.0"}}

Hey @samcm, I'm so sorry to come late to check this, but it looks really good!
I'll be working on a refactor/update of the crawler over this and the next week, so I'll try to extend the number of stream events as we previously talked


FLOPs Concern

Hello, thanks for your nice work!
I test the FLOPs of GFNet using the fvcore library as following:
    from fvcore.nn import flop_count
    model_mode = model.training
    model.eval()
    fake_input = torch.rand(1, 3, 224, 224)
    flops_dict, *_ = flop_count(model, fake_input)
    count = sum(flops_dict.values())
    model.train(model_mode)
    print("fvcore FLOPs: {:.3f} G".format(count))

For gfnet-h-b model, I got the result: 8.547 G. It is slightly higher than what you mentioned in the paper, i.e., 8.4G.
My concerns are:

How do you get the FLOPs value?

From, the fvcore log, I noticed that:

Unsupported operator aten::fft_irfft2 encountered 36 time(s)

I.e., the FLOPs of the fft_irfft2 operator are not taken into account.
I wonder if you consider this operator when calculating the FLOPs?
If not, I think it would be better to consider it because this is the core operator that replaces self-attention.
Please let me know if I missed something.
Thanks.

Hi, thanks for your interest in our work.
Since the fft operations used in our model are not supported by toolkits like fvcore, we manually write down the cost of each kind of layer and add them up using a script.
The FLOPs of rfft + ifft operations should be HWD * log_2(HW) as mentioned in Table 1, which is very lightweight compared to the attention and MLP since neither (HW)^2 nor D^2 terms are not involved. Therefore, the overall computation of 36 pairs of rfft + ifft operations in gfnet-h-b is only 0.06G FLOPs.
It seems our scripts may miss some computations compared to fvcore. I think it is better to compute the FLOPs of other operations using fvcore and then add the cost of fft operations. Therefore, the overall FLOPs of gfnet-h-b should be 8.6G, which is slightly higher compared to 8.4G. We will revise the FLOPs in the next version of our paper.
Many thanks.

Thanks for your clarification.


Pagination css.icon problem

I have been trying to use the vuetable-2 with my laravel 5.4 project. I have managed to get and display the data via rest api. However when it tries to use pagination I get

[Vue warn]: Error in render function:
(found in  at /Users/username/Sites/akademi/node_modules/vuetable-2/src/components/VuetablePagination.vue)

app.js:6919 TypeError: Cannot read property 'first' of undefined

I traced the error back to the
         «
and similar lines with css.icons object within the VuetablePagination.vue and when I remove those lines it works perfectly without the glyphicons.
What am I missing?

@digitalmind There are some changes in the css prop in v1.6.0 that might cause the problem to your existing code prior to v1.6.0. Please check out the release note. And see the example on using Bootstrap CSS with Vuetable-2 in this jsfiddle.

Thanks a lot, when I followed the jsfiddle it worked perfectly.

It took me a long time to realise, the icons prop should be inside the pagination prop which is inside the css prop.
css: {
    ...
    pagination: {
        ...
        icons: {
        }
    }
}


create: allow creating casks in a specified tap

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?
[x] Have you successfully run brew man locally and committed any changes?

Allow brew create to place a new cask in any installed tap, creating the Casks folder if necessary. Also fixes an error that would occur if the specified tap did not exist.

Review period will end on 2020-12-16 at 23:28:38 UTC.

Review period ended.


Add an updated generalisation stylesheet that uses Saxon 6.5.

:-1: since we don't want to add XSLT 1.0 specific code. We have a hard dependency to XSLT 2.0 and as a result of that, to Saxon versions that support XSLT 2.0.
Is the generalize.xsl actually used anywhere? Instead, we should add generalization support to https://github.com/dita-ot/org.dita.normalize

It will be easy to update that to XSLT 2 - I assumed XSLT 1.0 is required as this is what the existing generalize.xsl used. The existing stylesheet is used only once, in: dita-ot/src/test/testsuite/testcase/unknownforeign/unknowforeign_generalize.xml. See also my comments on Slack - generalize.xsl is broken.

See https://github.com/dita-ot/dita-ot/pull/2159 for the XSLT 2.0 version. I am not sure how this should be integrated into https://github.com/dita-ot/org.dita.normalize. If you think the place for these stylesheets is not here, we can just remove them, including normalize.xsl, pretty.xsl and maybe also specialize.xsl.


JSFuck variant using "([+<])". Has anybody tried this before?

I don't know if anybody has ever done this before. I made a JSFuck variant using < instead of !.
JSFuckr: An esoteric language based on JSFuck
Try it!
Is this a novel idea, or I am reinventing the wheel? 😅

@kamil-kielczewski < also serves as a bitwise left shift operator <<. See this example: https://github.com/aemkei/jsfuck#numbers
However, I don't think using a left shift operator to generate numbers will lead to a shorter outcome:
4 = ([]<[+[]])<<([]<[+[]])<<([]<[+[]]) (34 chars)
4 = !+[]+!+[]+!+[]+!+[] (19 chars)
8 = ([]<[+[]])<<([]<[+[]])<<([]<[+[]])<<([]<[+[]]) (46 chars)
8 = !+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[]+!+[] (39 chars)
So I think, this is just a JSFuck variant made for fun.
PD: I also cosidered using the subset [(+=)]

@baboures - safety - if in future they remove this deprecated method - jsf will stop works (that 16kb alternative solve this - however I hope there is better solution)

@kamil-kielczewski Have you considered "c"["toUppercase"]() ?
Another question: do you consider solutions using escape/unescape? Those functions are not strictly deprecated as per the docs.

@wakfi as it was written in this question https://stackoverflow.com/questions/63673610/alternative-way-to-get-c-letter-in-jsfuck - btoa not works on node.js

@wakfi as it was written in this question https://stackoverflow.com/questions/63673610/alternative-way-to-get-c-letter-in-jsfuck - btoa not works on node.js

@kamil-kielczewski btoa is supported in nodejs under legacy stability level support. Apparently it was only added (as a global) in nodejs v16 though, I hadn't realized that. It was added (and will be kept) as a measure to improve interoperability between browser JavaScript and NodeJS JavaScript

@wakfi thank you for that info - I will check it

I tried making a version called JSFourk, where instead, you only get the characters []+!. Obviously you can't get as far with it as you can't call methods, but it had some interesting challenges to overcome with it:
In JSFuck you can use parentheses for the order of operations which is very useful. For example if I wanted to get the character f in JSFuck, I could just do this:
(![]+[])[+[]] // f

but without the use of parentheses, it adds a whole new aspect. The way to get around this is to simply put your expression into an array and select the first item from that array, like this:
[ /* expression */ ][+[]]

so the letter f in JSFourk would be this:
[![]+[]][+[]][+[]] // f


Problems deploying Meteor 1.6

Runtime problem after the build succeeds

Summary

Using meteor version 1.6, deploying to heroku the first time i accessed the website my chrome console asked me to meteor npm install core-js
Contents of .meteor/release
<EMAIL_ADDRESS>
Buildpack version
https://github.com/AdmitHub/meteor-buildpack-horse.git

Deployment target
heroku

@hems were you deploying a brand new application and using 1.6, or did you upgrade your application that was already deploying fine to 1.6 when it broke?

it was a new app i believe, anyway after installing core-js everything worked great. it's just weird that it only "complained" when running on my staging environment

same problem here
<EMAIL_ADDRESS>once core.js is installed
everything seems to be working fine

I can confirm this actually happened on an Ubuntu box I ran a new 1.6 meteor project, without using this build pack ( just manually installed it there )
So it seems it's actually related to running 1.6 on Linux and not about building it with this build pack!

For completeness, my experience:

Local (OSX) update from 1.5.4 to 1.6: no problems
Deploy to Heroku using buildpack-horse: blank page, Chrome console asks me to install npm core-js
Locally run meteor npm install --save core-js
Deploy again: app now runs fine on Heroku

@persac exactly, this happens even if you run the application on a linux server yourself, without using heroku or the buildpack so i reckon it's not related to this repository!


Font not set properly in Dialogs

Hello,
I'm using a separate class to make custom dialogs which uses internally the Dialog class.
This is called from an activity and the context is passed with the "this", for instance:
mDialog = new AlertDialog();
mDialog.showOKActionDialog(this, getResources().getString(R.string.warning), getResources().getString(R.string.server_error), this)
The fonts are not set as stated in the XML for the dialog.
I also tried to set the font programmatically like:
TextView tv = (TextView) findViewById(R.id.appname);
Typeface face = Typeface.createFromAsset(getAssets(),
"fonts/epimodem.ttf");
tv.setTypeface(face);
but it didn't work either.
Any idea how to get the font on the dialog using Calligraphy?
Thanks!

I would make sure you are passing the correct context though to your dialog.


[Bug]: Validator crashes on failure to obtain list of disabled validators on Kusama block #25830930

Bug Summary
Validator crashes instantly when processing block #25830930 (0x145a…36bc) on Kusama
Bug Description
Validator crashes instantly when processing block #25830930 (0x145a…36bc) on Kusama, presumably due to inability to obtain validator list for said block:
Nov 19 10:49:13 v1-oslo kagome[937259]: 24.11.19 10:49:13.373496  kagome           Critical  Babe  Can't obtain disabled validators list for block #25830930 (0x145a…36bc)

Steps to Reproduce
Mode: Validator
number of nodes: 1
Command: kagome --chain kusama -d [...] --validator --listen-addr [...] --public-addr [...] --name [...]  --rpc-port [...] --telemetry-url [...] --telemetry-url [...] --node-key-file [...]
Effects of the Bug
Instantly crashes, service restarts, hits same error again.
Example error log running with --validator:
Nov 19 10:49:11 v1-oslo systemd[1]: Started Start kusama node.
Nov 19 10:49:11 v1-oslo kagome[937259]: 24.11.19 10:49:11.692894  kagome           Info      Configuration  Parachain multi process: true
Nov 19 10:49:11 v1-oslo kagome[937259]: 24.11.19 10:49:11.692921  kagome           Info      Configuration  Listen addresses are set. The p2p port value would be ignored then.
Nov 19 10:49:11 v1-oslo kagome[937259]: 24.11.19 10:49:11.717656  kagome           Warning   chain_spec  Field 'chainType' was not specified in the chain spec. 'Live' by default.
Nov 19 10:49:11 v1-oslo kagome[937259]: 24.11.19 10:49:11.717668  kagome           Warning   chain_spec  A non-empty set of 'badBlocks' encountered! They might not be taken into account!
Nov 19 10:49:11 v1-oslo kagome[937259]: 24.11.19 10:49:11.726213  kagome           Info      Main  Kagome started. Version: 0.9.5-34-feature/nix-support-13-658dbf0d-dirty
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.102263  kagome           Info      Injector  Will use LibP2P keypair from config or 'node-key-file' CLI arg
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.103523  kagome           Info      BlockTree  Highest block: #25830930 (0x145a…36bc), Last finalized: #25830927 (0xd88f…cd2b)
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.103798  kagome           Info      PVF Executor  pvf runtime engine kWasmEdgeInterpreted
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.200893  kagome           Info      Application  Start as node version '0.9.5-34-feature/nix-support-13-658dbf0d-dirty' named as '
   Zugian Duck 🦆 Too' with PID 937259
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.200902  kagome           Info      Application  Chain path is /home/kusama/.local/share/kagome/chains/ksmcc3, storage backend is RocksDB
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.220974  kagome           Info      Timeline  Fast sync would be faster than Full sync that was selected
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.221005  kagome           Info      OpenMetrics  Listening for new connections on <IP_ADDRESS>:9615
Nov 19 10:49:12 v1-oslo kagome[937259]: 24.11.19 10:49:12.221247  statement-distr  Info      StatementDistribution  StatementDistribution subsystem started.
 Non-validator doesn't crash but stalls on block #25830930's child (see logs below).
System Information
NixOS 24.5 with kernel 6.11.5
Compiler: gcc 13.2.0
CMake: cmake version 3.25.3
Additional Context
No response

@Lederstrumpf
Thanks for reporting! Seems like the issue is related to KAGOME not being able to find runtime for some parachain. Until we figured out why is that the case, can you check if #2281 helps?

Thanks for the swift attention to the issue!

Until we figured out why is that the case, can you check if #2281 helps?

Unfortunately, get the same error on #2281 after rebuilding db (from #2271 - #2278 wasn't built on it).
Attached are logs both for validator & non-validator modes.
kagome_validator_#2281.log
kagome_non_validator_#2281.log

@Lederstrumpf we are investigating the issue. Meanwhile, you may try to sync the node with Warp or Fast sync. For some of our devs problem is not reproducing after they synced to the end of the chain.
If it does not help, you may try to Warp sync from genesis

@Lederstrumpf we are investigating the issue. Meanwhile, you may try to sync the node with Warp or Fast sync. For some of our devs problem is not reproducing after they synced to the end of the chain. If it does not help, you may try to Warp sync from genesis

Thanks, fwiw, the Full sync from my backup db from 14 days ago (mentioned here: https://github.com/qdrvm/kagome/issues/2276#issuecomment-2485838632) failed today on the same block/error.
Just restarted with a Warp sync from genesis - will update on its result once completed.

Just restarted with a Warp sync from genesis - will update on its result once completed.

Unfortunately also fails: strictly on the successor of the warp sync target block, both for validator and non-validator modes:
Nov 21 10:23:10 v1-oslo kagome[993246]: 24.11.21 10:23:10.547760  main_runner      Warning   Synchronizer  Block #25874041 (0x521d…627b) and 104 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(9) the requested block header is not found in block storage
Nov 21 10:23:35 v1-oslo kagome[993246]: 24.11.21 10:23:35.084349  main_runner      Warning   Synchronizer  Block #25874041 (0x521d…627b) and 108 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(9) the requested block header is not found in block storage
Nov 21 10:23:38 v1-oslo kagome[993246]: 24.11.21 10:23:38.752177  main_runner      Warning   Synchronizer  Block #25874041 (0x521d…627b) and 109 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(9) the requested block header is not found in block storage

I've attached some logs with the initial sync and restarts of the service: on every node restart (with --sync Warp still specified), the node manages to sync up until the current sync target and then stalls the same.
The last log (kagome_warp_sync_restart_4_validator_#2280_trace.log) is trace level.

Sorry, I couldn't find kagome_warp_sync_restart_4_validator_#2280_trace.log log.
Also, btw the issue from
kagome_validator_#2281.log
kagome_non_validator_#2281.log
could have been resolved in the master by #2285 . You may try to use master branch to check if it moves forward. (Please note we recently updated the database format, you might require --enable-db-migration flag if you run with existing database for the first time)

You may try to use master branch to check if it moves forward. (Please note we recently updated the database format, you might require --enable-db-migration flag if you run with existing database for the first time)

Will give it a shot - I didn't have #2285 in yet. DB's already migrated though.

Unfortunately still fails for me with #2285 included; even on a fresh Warp sync, see attached logs. It might be that Amforc's "The Spammening" happening atm is a factor.
kagome_warp_sync_fresh_6_validator_#2280.log
kagome_warp_sync_restart_7_validator_#2280.log

I couldn't reproduce the issue exactly. I suspect there might be some undefined behaviour either with 1) wabt instrumentation or 2) wasmedge .
To check 1) please try to use tmp/disable-wabt-instrumentation branch.
To check 2) please try to rebuild KAGOME with -DWASM_COMPILER=WAVM flag during cmake configuration and pass --wasm-execution Compiled  flag to use Wavm instead of WasmEdge

I couldn't reproduce the issue exactly. I suspect there might be some undefined behaviour either with 1) wabt instrumentation or 2) wasmedge . To check 1) please try to use tmp/disable-wabt-instrumentation branch.
To check 2) please try to rebuild KAGOME with -DWASM_COMPILER=WAVM flag during cmake configuration and pass --wasm-execution Compiled  flag to use Wavm instead of WasmEdge

Hi Kamil, sorry for my late response.
I've applied your two suggestions and get the following issue as the kagome node compiles the wasmedge wasm module - seems it's skipping compilation of a required object file:
Dec 01 20:57:45 v1-oslo kagome[1249970]: 24.12.01 20:57:45.196801  kagome           Warning   chain_spec  Field 'chainType' was not specified in the chain spec. 'Live' by default.
Dec 01 20:57:45 v1-oslo kagome[1249970]: 24.12.01 20:57:45.196811  kagome           Warning   chain_spec  A non-empty set of 'badBlocks' encountered! They might not be taken into account!
Dec 01 20:57:45 v1-oslo kagome[1249970]: 24.12.01 20:57:45.206369  kagome           Info      Main  Kagome started. Version: 0.9.5-34-feature/nix-support-13-658dbf0d-dirty
Dec 01 20:57:45 v1-oslo kagome[1249970]: 24.12.01 20:57:45.316258  kagome           Info      ModuleFactory  Start compiling wasm module /tmp/kagome/runtimes-cache/wasmedge_00c9a478de6d_e43ef38640d2c788641f3429a20a5c755ca41148c69bae1db772c9c7a60509e4_s_s2048
Dec 01 20:57:45 v1-oslo kagome[1249970]: [2024-12-01 20:57:45.331] [info] compile start
Dec 01 20:57:45 v1-oslo kagome[1249970]: [2024-12-01 20:57:45.405] [info] verify start
Dec 01 20:57:45 v1-oslo kagome[1249970]: [2024-12-01 20:57:45.487] [info] optimize start
Dec 01 20:57:53 v1-oslo kagome[1249970]: [2024-12-01 20:57:53.217] [info] optimize done
Dec 01 20:57:53 v1-oslo kagome[1249970]: [2024-12-01 20:57:53.217] [info] codegen start
Dec 01 20:57:59 v1-oslo kagome[1249970]: [2024-12-01 20:57:59.401] [info] output start
Dec 01 20:57:59 v1-oslo kagome[1249970]: ld.lld: error: cannot open /tmp/kagome/runtimes-cache/wasmedge_00c9a478de6d_e43ef38640d2c788641f3429a20a5c755ca41148c69bae1db772c9c7a60509e4_s_s2048.4508c6afe9.03586aac51.o: No such file or directory
Dec 01 20:57:59 v1-oslo kagome[1249970]: [2024-12-01 20:57:59.403] [error] link error
Dec 01 20:57:59 v1-oslo kagome[1249970]: [2024-12-01 20:57:59.403] [error] object file open error:No such file or directory
Dec 01 20:57:59 v1-oslo kagome[1249970]: terminate called after throwing an instance of 'std::runtime_error'
Dec 01 20:57:59 v1-oslo kagome[1249970]:   what():  Failed to calculate genesis state: N6kagome7runtime5ErrorE(1) Runtime module compilation failed
Dec 01 20:57:59 v1-oslo kagome[1249970]: Stack trace (most recent call last):
Dec 01 20:57:59 v1-oslo kagome[1249970]: #18   Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #17   Object "/home/shared/kagome", at 0x77fea4, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #16   Object "/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6", at 0x7f0fa8c3d1c8, in __libc_start_main
Dec 01 20:57:59 v1-oslo kagome[1249970]: #15   Object "/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6", at 0x7f0fa8c3d10d, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #14   Object "/home/shared/kagome", at 0x5dd128, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #13   Object "/home/shared/kagome", at 0x78f3e5, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #12   Object "/home/shared/kagome", at 0x86bc26, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #11   Object "/home/shared/kagome", at 0x86ab08, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #10   Object "/home/shared/kagome", at 0x849886, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #9    Object "/home/shared/kagome", at 0x83d09c, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #8    Object "/home/shared/kagome", at 0x83cbb4, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #7    Object "/home/shared/kagome", at 0x453faf, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #6    Object "/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6", at 0x7f0fa8ebc4d6, in __cxa_throw
Dec 01 20:57:59 v1-oslo kagome[1249970]: #5    Object "/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6", at 0x7f0fa8ebc284, in std::terminate()
Dec 01 20:57:59 v1-oslo kagome[1249970]: #4    Object "/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6", at 0x7f0fa8ebc219, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #3    Object "/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6", at 0x7f0fa8eacc0a, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: #2    Object "/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6", at 0x7f0fa8c3b934, in abort
Dec 01 20:57:59 v1-oslo kagome[1249970]: #1    Object "/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6", at 0x7f0fa8c52e85, in gsignal
Dec 01 20:57:59 v1-oslo kagome[1249970]: #0    Object "/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6", at 0x7f0fa8ca2efc, in
Dec 01 20:57:59 v1-oslo kagome[1249970]: Aborted (Signal sent by tkill() 1249970 1001)
Dec 01 20:58:00 v1-oslo systemd[1]: kagosama.service: Main process exited, code=dumped, status=6/ABRT
Dec 01 20:58:00 v1-oslo systemd[1]: kagosama.service: Failed with result 'core-dump'.
Dec 01 20:58:00 v1-oslo systemd[1]: kagosama.service: Consumed 14.411s CPU time, no IP traffic.

Have a suggestion how to get past this?

@Lederstrumpf we've been working on Wabt upgrade. Possibly failure was due to stack overflow issue there. Can you please check if adding
hunter_config(
    wabt
    URL https://github.com/qdrvm/wabt/archive/refs/e5c2678e806661179ea9df5238473a722e25e592.tar.gz
    SHA1 2e0af7be3aeb2aeea59590b3b54795ac0fc56a01
)

To config.cmake would solve you problems.
p.s. I would also suggest upgrading to the recent master, as there are some important fixes there (e.g. fixed deadlock)

@Lederstrumpf we've been working on Wabt upgrade. Possibly failure was due to stack overflow issue there. Can you please check if adding
hunter_config(
    wabt
    URL https://github.com/qdrvm/wabt/archive/e5c2678e806661179ea9df5238473a722e25e592.tar.gz
    SHA1 2e0af7be3aeb2aeea59590b3b54795ac0fc56a01
)

To config.cmake would solve you problems.
p.s. I would also suggest upgrading to the recent master, as there are some important fixes there (e.g. fixed deadlock)

Hey @kamilsa, thanks for the update.
Happy to confirm that with the wabt dependency change and rebasing on master, my node managed to sync back up with Kusama and remain in sync on its first launch.
However, when I restart the node, it hits the same issue:
Dec 31 18:19:36 v1-oslo kagome[1159236]: 24.12.31 18:19:36.814333  main_runner      Warning   Synchronizer  Block #26447839 (0xf83f…06be) and 53 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(1) block, which should have been added, has no known parent

Log snippet:
kagome_validator_#2280_fresh_sync_and_reboot.log
From my naive understanding of the logs, the issue appears to be related to whatever the last synced block was:
I restarted the node after it imported block #26447838, and then its first failure is on block #26447839 due to not finding its parent.
Perhaps there are issues if the node is being restarted while this has not been properly saved yet, whereas it's already registered as imported? Seems like a non-atomicity issue of block imports, but just my 2 cents.

Update: it works on some node restarts, seemingly whenever the node triggers a fresh StateSync. See for instance the last restart here (log starting prior to the first restart log shared in my last comment, with 5 restarts thereafter):
kagome_validator_#2280_fresh_sync_and_multiple_reboots.log


Load global.properties even if common.properties doesn't exist

Use global.properties even if common.properties doesn't exist

Changed! )
Thanks for the review!
Could you please check one more PR and write your notes?
https://github.com/azkaban/azkaban/pull/1727