Support for quantisation

rustformers/llm supports Q2 to Q8 quants with various varieties. Would it be possible to quantize the existing models and run them in this repo ?

Hey @okpatil4u,
We are looking at running GGML models directly. We're focusing on Q4 for now as it seems the most popular https://github.com/huggingface/candle/issues/360.

I would also very much appreciate quantization support, as it allows user with weaker hardware to run models. I'm guessing most of the work will be implementing the quantized  matmul kernels?

Pretty much yes, and probably loading GGML/GPTQ weights (GPTQ is already safetensors so shouldn't be too hard)

Is there a plan to support both GGML and GPTQ in our current project? I must admit, I'm not very familiar with GPTQ, but from what I understand, it seems to use a different quantization format compared to GGML. Would this difference necessitate the creation of separate matmul kernels?
On a related note, I believe I could contribute to the loading of GGML files. Additionally, I'm interested in assisting with the implementation of GGML quantization support, though I'm uncertain about where to begin. I presume that one of the initial steps might involve extending the TensorStorage to accommodate quantized tensors and facilitate the loading of data into these tensors.

Yes, it's on my list to look at quantization support this week. It's currently not very clear how we will do this, we already support a bunch of different backends/dtypes/ops that this will have to interact too and we would like the changes for quantization to be as non-intrusive as possible.
That's why the idea was to start by being able to load ggml weights which is almost ready, and then see how we could adapt some of the ops to use the quantized weights directly.

Concerning the operations that can directly utilize quantized weights, GGML appears to implement real quantized operations specifically for its q x f32 matmul operation. Within this operation, many quantized vector-dot operations are performed.
Integrating this functionality into Candle could be relatively straightforward if f32 is consistently the target of these quantized operations. This approach would mean that most of the weights could remain quantized during a forward call, without requiring adjustments to any of the other implemented operations.
To extend support to all other operations, we could introduce an automatic dequantization mechanism. This would be necessary for operations that don't natively support working with quantized types, ensuring seamless integration and compatibility across the system.

Right, that's exactly the plan: the initial version will stick to using f32 tensors internally and just have specific matmul using the quantized weights. It's also what Sasha has been doing in llama2.rs with good performance.
As preliminary work, I've added some simd bits within candle earlier today so that we can use them in the matrix multiply. I'm currenty adding the quantized matmul based on the llama.cpp code.
And as you mentioned, longer term we might think about having proper dtype for quantized tensors but that's a huge lift so would prefer to avoid it if we can get a large part of the benefit via the simple approach.

There has been a bit of progress on quantization though it's certainly very early days. Roughly the recent changes introduce a new QTensor type that is only used to create linear layers and there is an example that use it for a ggml llama model here.
This can be run via:
cargo run --example ggml --profile=release-with-debug

This will download some weights from the hf hub. If you want to use local weights, you can set --model ggml-model-q4_0.bin where the .bin file has been generated by the llama.cpp quantize command.
As I mentioned this is currently a proof of concept and not something polished at all:

It is very very slow, the ops are implemented in a very dummy way, no simd, no multi-thread.
When using Q4 the model is very unstable, it hasn't been lined up properly with ggml or anything so unsure where this numerical instability is coming from.
Most of the functions in k_quants.rs are todo's, we should prioritize the ones we care about.

The next steps would involve: lining things up with ggml though it's unclear to me how to do this as I haven't found how to print intermediary tensors out of ggml, complete the functions from k_quants.rs and optimize them, profile the current code to see where the time is spent, and hopefully we can make this much faster.
@LLukas22 (and others), if you're interested we could certainly use some help here, typically if you want to add some of the missing bits/add some simd optimizations/help lining things up we can just sync up on this thread so as to split the work.

Outstanding work! 👌
I took a quick glance at your commits, and I can't wait to play around with them tomorrow. SIMD stuff isn't really my strong suit, but with the ggml code as a guide, I can probably contribute a bit.
By the way, I noticed a lot of unimplemented code blocks in quantization types other than q4_0. Are you thinking of supporting all those formats, or are you mainly focusing on q4_0?

Right, most of the quantization types are unsupported, I'm planning to add the vec_dot for Q6K at the moment as it would directly be used by the last fc layer of the q4_0 llama quantization (in the current implementation this is converted to a f32 matmul which is likely a lot slower).
Certainly happy if you want to implement some of the others. Also I took a stab at simd'ing q4_0 in #474 but strangely it ended up being slower than the default implementation. My guess is that the compiler does a good work at simd out of the box (as these are mostly int types and not float types for which rust wouldn't simd stuff), so anyway no rush in having the simd version.
If you add some stuff, don't hesitate to add tests in quantized_tests.rs, typically matmul/from-float/to-float are good to cover.

I've been testing on my Ryzen 3700 machine (which doesn't support AVX512), and I must say, the results are genuinely impressive.
Initially, I ran the example without your AVX implementation, and it was somewhat sluggish, clocking in at 0.2 t/s.
Then I decided to give your AVX implementation a try, but getting it to work was a bit tricky. Rust seemed reluctant to set the avx target feature. I finally managed to force it on by adding the following to my .cargo\config.toml file:
[build]
rustflags = "-C target-cpu=native"

This change led to a massive performance boost, over 10x, reaching about 2.29 t/s.
I didn't stop there, though. I parallelized the vec_dot calls using rayon, which gave me another 3x performance increase, bringing it up to 6.53 t/s.
For comparison, running the same model through Rustformer's GGML backend gives me around 8.6 t/s. That means the current implementation is only about 25% slower than llama.cpp. In my opinion, that's quite an achievement. Well done!

Waves in windows user

Ah, feel free to make a PR with the proper target and features so that it works on your side, ideally using native so that it works for all the simd instruction sets.

I thought about adding a rayon::ThreadPoolBuilder to the QMatMul op, which could be initialized during the from_qtensor call, meaning we know the dimensions of one of the tensors used in the matmul, that could be helpful in figuring out a senseful thread count.

Sounds like a good idea. Not sure how large a threadpool object is, if it always starts the threads, maybe there is a cost to having these on a per QMatMul basis. Did you come across par_for_each, we use it at a couple place to control tightly the number of threads that are working (though it has the downside that it's less fine-grained than parallelising into lots of small tasks), you can grep in the codebase to see how we already use this.

Also I just merged the support for avx in the q6k matmul that is commonly used for the last matrix multiply when using mostly q4_0. Not sure if the model you were trying had it but if that's the case you may want to update, on my side it's a 10 to 15% speedup so should put us really close to ggml (looking only at this multiplication itself, the speedup is ~8x).

Yup, can confirm the q6k avx implementation gets the llama implementation pretty close to ggmls performance. I'll probably look into supporting q4k and q5k as this should enable us to run most of the q4-kquants which should perform a lot better than the original q4_0 format.
Regarding testing, should is just mimic the q6k tests?

Great to add q4k and q5k support, and yes mimicing the q6k tests sounds good.
We should also try to line up the inference with what llama.cpp produces, ideally when setting the temperature to 0 we should produce the same thing (up to some hopefully small numerical errors), that said it's probably not a super easy thing do but I'm currently a bit worried by some of the test that I generated in my examples which seemed a bit poor.

First on my agenda is to concentrate on implementing some of the from_float functions. I want to make sure that roundtrips are functioning correctly through thorough testing. Once that's squared away, the next step will be to validate that the qmatmuls are producing accurate results. This should be relatively straightforward if we can create test inputs from f32 tensors. However, I'll likely need some time for this part, as my goal is to make the functions more readable and easier to understand during the porting process.
As for aligning with the llama.cpp inference, it's not currently a primary focus for me. While the ability to run GGML files produced by llama.cpp would be a nice enhancement, my main interest lies in quantization support for other models like stable-diffusion or wisper. For those, I plan to use the existing inference code as a template.

Sounds good. When it comes to matmul tests, we actually already have some that you can see here for q6k for example, these were super helpful as there were some bugs in the first cut. This file also has some tests for the from_float / to_float round-tripping so hopefully we can cover all functionalities there.
Fair enough re aligning with llama.cpp and it's certainly a pretty difficult task, I feel there is some value into it, helping discovering small bugs but I'll have a go at it on my side. And certainly looking forward to you trying out quantization for the other models - one of the main selling point of candle is that it should make this experimentations a lot easier than what you can do with llama.cpp where introducing new models/architectures is a bit harder.

Re lining up with llama.cpp, it took me forever to debug this but it turned out there was some subtle bug in the way rope embeddings were computed on the candle side (totally my fault).
I've just merged #518 which fixes this. After applying the change, the generated sequences with temperature 0 seem to exactly match the llama.cpp ones (both for f16 and q4_0), and the generated sequences when using some non-zero temperature look a lot better. We definitely have to get better at figuring out this kind of issue but hopefully this makes it possible to also use the full text generation loop as a test for new quantizations.
Also with multi-threading, the generation speed seems to be within 10% of llama.cpp on my ryzen 2600x which is great.

That's great news 👍
Just a heads up they are planning to release the new GGUF file format tomorrow, meaning the current way of reading ggml files will break.
I didn't have much time but i started to port some of the k-quant quantizations over and added some more unit tests to ensure the quantization actually works as expected. q4k is already finished but I'm having some trouble with q2k and q5k which produce strange results with larger positive numbers. I'll probably setup the same testsuit in the ggml repo to help me debug this and ensure that the results match.

Ah interesting re the gguf format, thanks for pointing this out.
By "the current way of reading ggml files will break" you mean that we will need to support gguf specifically or that we'll have to do breaking changes in the way we currently read files? Looking at the discussion on the llama.cpp thread, I would have hoped that it's just a matter of supporting the new format and that the current support for the "old" file format will continue to work but maybe I'm missing some things.
And yeah the experience of lining things up is certainly tricky, more unit tests will certainly be good and would be very keen on your idea to have the same tests on both sides!

By "the current way of reading ggml files will break" you mean that we will need to support gguf specifically or that we'll have to do breaking changes in the way we currently read files? Looking at the discussion on the llama.cpp thread, I would have hoped that it's just a matter of supporting the new format and that the current support for the "old" file format will continue to work but maybe I'm missing some things.

As far as i know the Ggjt file format will be deprecated. The original author of the GGUF spec is currently working on a rust implementation, so we could probably use that to read the new files.

And yeah the experience of lining things up is certainly tricky, more unit tests will certainly be good and would be very keen on your idea to have the same tests on both sides!

Yeah, it takes some time but i'm getting there. Just another random thought how hard would it be to add CUDA support for the qmatmuls ?

As far as i know the Ggjt file format will be deprecated. The original author of the GGUF spec is currently working on a rust implementation, so we could probably use that to read the new files.

Ah that sounds good, hopefully we won't have to deprecate the old format too soon on our side (I imagine that it's more  of a burden to have multiple format in their codebase than in ours so they just want to support "one" format at a time).

Yeah, it takes some time but i'm getting there. Just another random thought how hard would it be to add CUDA support for the qmatmuls ?

Well we can just port their cuda kernels so shouldn't be that hard. That said, I think I would rather prioritize performance on M1/M2 before this as it's more appealing to a bunch of potential users.

There were some improvements to the k-quants today i guess we want sync these improvements?
I also started implementing some of the vec_dot functions but unit testing them is a bit iffy. GGML just uses this function to generate some test data:
void generate_data(float offset, size_t n, float * dst) {
    for (size_t i = 0; i < n; i++) {
        dst[i] = 0.1 + 2*cosf(i + offset);
    }
}

Should we just replicate this or is there a better way to test matmuls? Currently i find it a bit hard to debug potential errors.

Re tests, I think the current ones for matmul are actually pretty good as they generate random values with a fixed seed, see here, that said I certainly won't object to having additional testing :)
Re improvements, sure we could update. I tried to keep the code I wrote as close as possible to the original one so that it's easier to follow down the line but that's also annoying to have code that doesn't look very rust idiomatic. We could also hold off until we have plugged in everything and can play with the quantization, measure its effects, and measure that the improvements are actual improvements in the same way folks do it on the llama.cpp side.

Re GGUF, as the spec turned out to be fairly simple, I've added support for it in candle and the quantized example now works with these too (tested with both q4 and f16, #559). We'll switch to use it for the default models once they are available on the hub.

The vocab/tokens are indeed already in the GGUF files produced by ./quantize. That said these are not totally straightforward to inject in the tokenizers library so we may have to do some work if we want to rely on a single config file (currently we just also use the default llama tokenizer config which is small and most likely always cached).

GGUF was designed to contain a tokenizer.huggingface.json field which simply contains the content of a tokenizer.json file to be directly used with the tokenizers crate. I will add this filed for all models i convert but the current llama.cpp conversion script doesn't seam to add it.

Ah that's great, that will make things a lot easier.
Fwiw, you can use the tensor-tools binary to look at metadata in gguf files (or list tensor infos for any supported file format, safetensors/npz/pytorch/...) e.g.:
cargo run --example tensor-tools --release -- ls ../llama.cpp/models/7B/ggml-model-q4_0.gguf --verbose

My hope is to add more to this command line tool, e.g. conversion functions from a format to another, quantization functions etc.

I would like to contribute to candle by porting quantization and vec_dot from lama.cpp into candle. I can add that methods for q4_1, q5_0, q5_1, q8_1. And then I can add also the avx implementation.
@LaurentMazare @LLukas22  commenting before I start because I don't know if one of you has a plan to implement this. can I open a pull request for that ?

Ah actually I've started adding these and hope to have a PR ready later today (had to step out for a bit), it will also add a bunch more testing and should make lining things up with the C++ version easier. Sorry for the collision @abuelnasr0

@abuelnasr0 I only looked at the avx implementation for the k-quants but i haven't started porting anything. If you want you could also give the CUDA implementation a try.

Alright i played around a bit with the CUDA implementation but i had some trouble with the CudaDType trait and the CudaStorageSlice enum as i wanted to implement CudaDType for the QTensor but i guess that's not the right approach.
It's probably easier to work with CudaSlice<T> and CudaView<> directly, as the result will always be a f32 tensor.

Closing this now in favor of more recent issues, e.g. #1250 (and #1754 should bring some preliminary support for quantized models on cuda).


Update targetOs to target_os to fix packer errors

The targetOs version of the parameter wasn't usable by packer directly,
and I received an error attempting to use the merged version of the
changes to use the 1.7 packer release. Turns out that when I rebased
onto the windows support, I failed to directly test & ended up failing with
a recurrence of the problem reported in issue 40, and specifically, the
newer problem reported
here
The solution was the same as for the struct tags added in the 1.7 compatibility
upgrade, which was to provide a mapstructure struct tag. Apologies
for missing the new parameter when I rebased.

I did test a build with these changes locally to make sure it all worked & it's 💯

I can test to double check if you want.

@Lirt Please! I'd like to make sure it solves the issue for everyone.

I didn't test with that parameter.  @Lirt once you have a chance to test it, i can merge.

When I try to build using the source fork+branch frightenedmonkey:fix-another-struct-tag I get build errors:
$ go build -v -o packer-provisioner-goss-ova
go<EMAIL_ADDRESS>requires
<EMAIL_ADDRESS>invalid version: unknown revision aef4ced967ec

I tried to bump packer-plugin-sdk to all available versions but all of them fail. I tried to build the scaffolding project https://github.com/hashicorp/packer-plugin-scaffolding but it fails as well.
Shall we report it to packer slack or am I doing something wrong?

@Lirt weird, I haven't had any build issues. For clarity, I'm building it locally & I also have a container image where I'm building that branch from source in a CI build—that CI build is going to re-resolve all dependencies and download them at build time.
I also went and looked, and I see that SHA in the packer repository https://github.com/hashicorp/packer/commit/aef4ced967ec
I'm not sure what else to troubleshoot at the moment.

That will be my local issue. Sorry I will try as soon as I fix it.

Ok I fixed my issue and everything works well in my environment. Thank you.


Explore options for Caltrans GitHub repo for Compiler collaboration

Goal is to determine the best way to set up a GitHub repo for collaboration between Caltrans and Compiler during the period when they are still reliant on ODI infrastructure. Jason will start by discussing this with IT Ops.
This also includes exploring Snowflake access.
Zhenyu said they would like to have a repo available in 2-4 weeks (from 8/15).

Next step is to have a call with Compiler. @kengodleskidot will help with scheduling with them.
@jkarpen will write up agenda in advance.

Per @ian-r-rose , Compiler can already make pull requests from the existing Caltrans repo we are using for this project. However someone from the Caltrans team will need to do the merge. So a separate repo is not needed.
It is unclear if they need access to the project board, which would require additional step of adding them to the ODI GitHub org.

Per @ZhenyuZhu-Caltrans Compiler is planning to set up their own repo so this issue can be closed.


Update dependencies

Closes #29
Closes #28
Closes #27
Closes #26
Closes #25
Closes #24

:tada: This PR is included in version 1.1.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


⚠ BCR Bot has degraded performance

In 76e011b, BCR Bot (https://bcr.alternativasinteligentes.com/BCRApi/web/chatbot) experienced degraded performance:

HTTP code: 200
Response time: 1103 ms

Resolved: BCR Bot performance has improved in 2fd1762.


Implement React Router

Our React Web App is not using React Router right now. It will be good to implement react-router in webapp because as our component will increase, It will be easy to display them using react-router
Follow this and this tutorial to know more about react-router.

I am interested in this issue. Can I take this?

Can I take this issue?

Can I work on this issue. Please assign me this one.

Hey, I want to work on this issue

@shreyakapoor08 Hey this issue has been already assigned, please check other issues.


catch_backtrace must be called early after catch

The state fetched by catch_backtrace might get erased if IO operations occur in the same catch block before catch_backtrace is called.
(this is a workaround to https://github.com/JuliaLang/julia/issues/13947)
Test case (a less complex call to println doesn't show the same problem):
println(" ", 1, " ", Dates.Date(2016,1,1))
collect(Int64,[1.5])

on x86_64-apple-darwin15.4.0 from homebrew Version 0.4.5 (2016-03-18 00:58 UTC) this produces
 1 2016-01-01
LoadError: InexactError()
while loading In[1], in expression starting on line 2

after this PR the output contains the backtrace as intended:
 1 2016-01-01
LoadError: InexactError()
while loading In[1], in expression starting on line 2

 in collect at array.jl:256

Looks good, thanks.


Consider relying on vim-dispatch

tpope has a new vim plugin out for async jobs. Watch the introductory video.
http://vimeo.com/63116209
https://github.com/tpope/vim-dispatch#readme

@q0LoCo I tried your async branch and it seems to work with flake8 for Python but not with ESLint for JavaScript. I'd open an issue there but you don't have the issue tracker enabled on your repository.

@treyhunner it's opened now.

since syntactic relies on external program's to check syntax, what is the main hurdle in doing something like this (oversimplifying, but want to understand if this approach is fundamentally broken):

run syntax checker external command forking it to the background and redirecting the output to to a file.
use cursorhold to trigger a periodicvim  function which checks if the file (created in the above step) has changed, and if so it does whatever syntastic already is doing to parse the output and put it in the location list.

@q0LoCo your branch often fails to update the UI. I'm using powerline(if that matters) and mostly pylint as the lint checker...

@acornejo I believe the thread above answers your point: it's possible in principle, but it's less simple than you seem to believe, and the result is likely to suck.  Please feel free to prove me wrong though. :)

@s0undt3ch Can you please take this to the appropriate place? Thank you!

I've tried to use other async check plugins, but turns out I miss the features of syntastic so much.
Finally I realized the problem is not with syntastic, the actual reason is the lint tool runs so slowly.
So I made a plugin to check javascript syntax error only https://github.com/chemzqm/jscheck
it uses notify functions from syntastic which are very well  written :+1:

Out of curiosity, could we not somehow use vim's client/server model? That is, I could run my main vim session as a server, run the syntastic check in a separate client, and when checking is done have the signs and errors sent to the server.

@bendavis78: There's not much benefit to initiating the check from another instance of vim. Most of the work is done by the external checker. Vim does some work to process the checker's output (see :help efm), but I don't see how we could send that from one vim session to another.
However, using vim's client/server model could work but there are concerns about requiring it for syntastic. AsyncCommand uses vim's client/server model to do async tasks: Run the checker in the background and when it completes, use --remote to send a signal to vim (but there's only been a proof of concept syntastic integration: stgpetrovic/syntastic-async@b3e3a9b). vim-dispatch works similarly, but uses a different signal to notify vim of completion. (See the above conversation about SIGWINCH.) I think the trickiest part is figuring out how to feed the output file into syntastic (probably more complex than :cgetfile).

Just so you guys know, version 7.4.1274 of vim added job functionality.
https://groups.google.com/forum/#!topic/vim_dev/_SbMTGshzVc

Yep. So now everyone can stop pretending that job control isn't useful :)

@mtglsk The interface is still under development, please see :h job-options.  Either way, a number of things need to change in syntastic before it could take advantage of background jobs without having to rewrite the existing 202 checkers.  Neovim had a similar interface for a while, and it inspired neomake, with a different approach than syntastic (see also #1253).
The forthcoming job interface in Vim will be similar to the one in Neovim, but it probably won't be compatible to it, despite several people (including Neovim's author) trying to reconcile the two APIs.  At least that's my understanding of Bram's position, at the time of this writing.
@justinmk Working PRs are still welcome.

Neovim could parse result of async make command by using 'errorformat' option automatically, but I'm wondering vim will have this feature or not.

There is job_status() function in new job api. Vim can use autocmd to parse result if there is no way to pass a callback function to job_start().
A checker may do somethings on result, the problem is that current buffer may not be the same one it just checked.

A checker may do somethings on result, the problem is that current buffer may not be the same one it just checked.

Yup, that's one of the problems.

Vim 8 is out with "native" support for async functions via channels! Does this make sense for Syntastic to use?

Yes, async checking shall be implemented, eventually.

@lcd047 is there a separate issue for that?

@shinzui Not that I know of.

In the meantime, it looks like validator does async syntax checking. https://github.com/maralla/validator.vim

@lencioni Right.  And so does YouCompleteMe.  And vim-go.  And neomake.  And LaTeX-Box. And pyflakes-vim.  And ghcmod-vim.
And other tools mentioned in related threads.  And many others I can't remember right now.  Yet others are getting there.  All of these projects have features that syntastic doesn't.  I strongly encourage you to use them instead of syntastic.

@lcd047 as the de-facto current maintainer of Neomake I've often thought that my time would have been better spent by working on an async feature for syntastic, rather than fixing issues there all the time..
By now we have a test suite, but there are still fundamental issues, and all the groundwork already did and still does, and features it has are not available there.

@lcd047 What are your thoughts on decoupling syntastic's library of checkers (similar to how snippets are separated from Ultisnips), so that other checker "engines" can use them? That would 100x more valuable than making syntastic itself asynchronous. It seems inevitable that people are going to re-invent these types of plugins, but if a library of checkers existed it would almost certainly guarantee that the checkers themselves would no longer be re-invented.
By "decouple" I don't necessarily mean moving the checkers to a separate repo, but just structuring the checkers so that they are isolated.

@blueyed You (and many people above) assume that, now that Vim has support for jobs, what prevents syntastic from doing async checks is that it doesn't call checkers asynchronously.  That's not the case.  That is, it's true, but it's the least of the problems to be solved.  What actually prevents syntastic from doing async checks right now is that half of its functions assume that bufnr('') returns the number of the buffer currently being checked, and that b: variables are local to the same buffer.  That problem was kind of lurking in syntastic before I came to the picture, but it was cast in stone by a design mistake that I made when I allowed checkers to have per-buffer overrides.  That happened long before Neovim, and before the above discussions about async checks.
So the problem I'm facing right now is how to fix the internal API, preferably without breaking compatibility with all third-party checkers out there.  It's a design problem, and no, simply breaking backwards compatibility won't solve it. shrug

@justinmk Are we talking philosophy or logistics?
Philosophically, I think that:

Syntastic has built-in checkers; this has been a mistake;
Syntastic is using errorformat and the makeprg model; this has been a mistake;
Syntastic is trying (and failing) to solve some unreasonable parsing problems; this has been a mistake.

Logistically, I think that:

You can't meaningfully talk about a "library" without defining an API.  Syntastic did try to define such an API.  It failed, in that there are at least four methods of producing loclists, and they are only superficially integrated:

makeprg / lgetexpr (most checkers)
the preprocess machinery (used f.i. by pydocstyle)
the syntastic#c#GetLocList machinery (used by most "real" compilers, such as gcc)
generating loclists from the output of other plugins (used by vimlint).
Integrating all these in a library would involve a major rewrite.

It won't do you much good to have "decoupled" checkers, simply because syntastic goes way beyond returning a list of errors.  It also compensates for a myriad of tiny flaws in Vim, and in the design of the 200+ supported linters (despite trying hard to limit those workarounds to a minimum).  Some time ago I browsed through neomake's issues, and as far as I can tell this is what its authors failed to take into account. shrug
You could grep the code for errorformats and put them into a library.  Some people take the view that syntastic is useless, and everybody (especially new users) should learn to use the compiler features builtin to Vim.  I presume those users would find such a library useful, or at least an useful start.  Syntastic comes from there, and it's heading to the opposite direction.
VimL is a crappy language.  Among other things, it doesn't have named captures for regexps.  This could be overcome by relying on python and / or python3 for parsing errors.  That would be a worthwhile project.  It would still not result in a library; just a major, very nice overhaul of three of the four methods mentioned at 1.
The bottleneck for implementing all coupling, decoupling, async checks, and whatever other features you guys are asking for has a name: your truly.  I'm the only maintainer left (for whatever reasons), and these days I can afford to dedicate only a very small amount of resources to syntastic.  I expect to be able to afford even less in the near future, despite the pressure you guys are building here.  I encourage you to fork the project and change it to fit your needs.    Don't wait for me.

despite the pressure you guys are building here

@lcd047 I didn't intend to build any pressure for you--simply to point people who are looking for this to something that might currently fit their needs. I rely on syntastic every day and I really appreciate all of your hard work that has gone into it.

I'm the only maintainer left

I wonder if there are any folks in this thread who would be interested in helping out more?

@lcd047

it was cast in stone by a design mistake that I made when I allowed checkers to have per-buffer overrides

Neomake has those, too: the basic idea there is that the bufnr and winnr (preferably via the new win_getid function family, but otherwise using a window-local var) will be remembered for the job that is being run.  And then e.g. location list results will only be opened when you are in the matching window/buffer (in case you move around while the job runs).
Any settings that are relevant should be retrieved when the job starts, or from the corresponding buffer later.

@lcd047 Thanks for that valuable insight. Personally my interest is (was) to avoid redundant effort in the Vim ecosystem, but based on your answer here and at SO the situation appears grim.

Some people take the view that syntastic is useless, and everybody (especially new users) should learn to use the compiler features builtin to Vim.

The reason IDEs work is because they ship with the entire stack. Plugins like quickrun and syntastic attempt to use tools found in arbitrary environments, configured (or not) by clueless users, to provide a light-weight IDE. And that's a recipe for endless bugs.
My preference would be to use the "compiler" feature of Vim. But it is incomplete, as you noted...

I didn't intend to build any pressure for you--simply to point people who are looking for this to something that might currently fit their needs.

@lencioni  So you started to make a list at the ~50th message in this thread.  How inspiring.
Well, now that you mention it, a list with other linter projects could be pretty useful.  I'd suggest you create a wiki page for them and start collecting names, links, and basic characteristics (perhaps languages supported, async or not, etc.).  If / when this list starts getting substance I'll merge it into syntastic, to save GitHub the trouble of rendering it.

Any settings that are relevant should be retrieved when the job starts, or from the corresponding buffer later.

@blueyed Right, which is why bufnr('') references are fine in checkers and notifiers.  They're less than great in the checker API, in code that runs after the fact, in most of the loclist API, and in a few other autocmd-related places.  Since Vim 7.4.1640 they also boobytrap check_on_open. shrug

My preference would be to use the "compiler" feature of Vim. But it is incomplete, as you noted...

The errorformat plus makeprg model is flawed, not incomplete.  makeprg is broken because it makes it impossible to quote filenames robustly.  Syntastic found that out the hard way, and it switched to system() plus lgetexpr.  Running makeprg is also boobytrapped by QuickFixCmdPre and QuickFixCmdPost autocmds (meaning, you'll end up calling code from other plugins if you aren't careful).
errorformat is flawed because:

it doesn't expose the parsing loop to the user
it doesn't allow adding extra fields to the standard line and column numbers
it depends on isfname in a way that makes it impossible to match certain combinations of filenames and errors
the interaction between parsing rules is less than useful, and
it has arbitrary (undocumented) limitations for no good reason.

makeprg and errorformat together are also flawed because they are essentially impossible to debug.  If there is anything that syntastic has done right, it has been logging.  It isn't perfect, because you can still run into errorformat problems.  But at least it can show you Vim's idea of what a linter should be doing.
Now, if you can live with these limitations then good for you.  They're certainly fine for many simple cases.  Some of us however have to make the best out of some hairy situations.

So you started to make a list at the ~50th message in this thread. How inspiring.

@lcd047 I'm not really sure I understand what you are referring to about starting a list here. I just learned about this other project, thought "I bet the fine folks who work on syntastic are working on async too", found this issue, noticed folks talking about async, and thought they might find the thing I just learned about to be useful. I'm not trying to make any lists or inspire anyone, I just wanted to help out and share some info. 😕 I apologize if I've upset you--that was certainly not my intention.

I think any attempt to make on-the-fly checking would be a bad move, rather than having the neomake-like bufwrite execution. I saw a few different versions for on-the-fly(piping, making and writing and switching between temp buffers) and they all look flawed. vim doesn't seem to have a way to make a in memory pseudo-file for processing the errors.

Here's an update for you guys.  The plan goes something like this:

I'll make a 3.8 release, provided that no new bugs are reported the next few days.  This shall be the stable branch.  If new bugs are reported after the release I'll fix them, but I won't be adding any new features (or new checkers, for that matter).  This may eventually evolve into a stable 3.9 if needed.
Right now I'm fixing the API for the checkers.  Checkers will be able to use custom parsing functions instead of errorformat.  Right now that means moving all JSON parsers from preprocess to the new parser sections of each checker, but there will be new parsing functions based on if_python and if_perl.  This means some checkers will depend on Vim being compiled with Python or Perl.
Most internal checkers (hopefully all) will be migrated to the new API, but external third-party checkers will stop working.  In particular checkers that rely on calling a Vim function instead of running a program (such as tsuquyomi and omnisharp) will break.  I plan to allow old-style checkers to run synchronously (along with the async checkers), and I'll eventually prove some upgrade path for checkers running Vim functions, but neither would happen at the beginning.
The async functions still require a lot of bookkeeping.  I'll upload a new branch named async when I have them actually doing something useful.  They'll run with Vim 8.0, but they'll require a more recent Vim snapshot to work reliably (a few critical problems have been fixed in Vim in the mean time).  At first they won't run with Neovim.  There will be hooks for Neovim though, provided that somebody in the know is willing to fill them in.
A number of defaults will change.  If you want to suggest changes to defaults to your favourite checkers, now is the time to do it.  When the new branch gets released the defaults will again get locked for the next few years. :smile: Of course, I'll keep a detailed log with all incompatible changes, and I'll eventually update the docs, write an upgrade guide, etc.
There are a number of other problems to be addressed, how to handle the IsAvailable() functions.  This is important because some checkers (hello, rubocop) take forever and a day to return a version.  So version queries might need to be run asynchronously -- but doing so would turn the API for IsAvailable() into a pain in the rear.  Let's hope some useful approach pops up eventually.
If you plan to write new checkers for syntastic after the 3.8 release you'll have to use to the new API, based on whatever docs are available at the time.  I don't plan to offer much hand holding until the API stops changing.  Sorry about that.  Please feel free to explore using other checker engines in the mean time.
When the new API in the async branch stops changing I'll call the result syntastic 4.0, and merge it to master.  All major issues would have to be fixed for that to happen.  And you probably want to stay with the stable branch in the mean time.

That's the basic outline.  Of course, that's just a plan.  Plans do typically change.

@lcd047
Thanks for the update and plans for a bright future.
Have you thought about using Neomake as a base for the new Syntastic?
Syntastic could be kept in maintenance mode as-is, and instead its features would get ported over to Neomake.  Neomake has a quite decent testing framework already, and has support for both Neovim and Vim's async modes.
I'd rather not waste development and support effort.  So if you do not like going the Neomake route then I might just put Neomake (from my side) into maintenance mode and help you out.. :)
But please consider that Neomake had been created (not by me though) as a proof-of-concept to make Syntastic async.

@blueyed

Thanks for the update and plans for a bright future.

Oh absolutely, maintaining a healthy dose of skepticism is always a good idea. :smile:

Have you thought about using Neomake as a base for the new Syntastic?

I have.  I still think syntastic is a lot easier as a foundation for what I'm trying to do than Neomake would be.  The vast majority of it is also my code, so I'm already pretty familiar with it.

Neomake has a quite decent testing framework already,

On a side note, syntastic doesn't have tests for a single reason: I'm not the owner of this repository, and activating Travi-CI for it would require the intervention of the initial author.  I contacted him in the past and, while he was always nice, I did get a feeling he wants to put all this behind him, and he wouldn't appreciate being bothered with anything related to Vim these days.  Activating Travis-CI is simple but not completely trivial if you haven't done it in a long time, it might take a few exchanges back and forth.

and has support for both Neovim and Vim's async modes.

I'm not worrying about Neovim at this point.  With the design I'm going to use adding support for Neovim shall be either trivial, or very hard.  I'm reasonably confident it's going to be trivial rather than very hard.

But please consider that Neomake had been created (not by me though) as a proof-of-concept to make Syntastic async.

As far as I can tell the point of Neomake was to create an asynchronous :make for Neovim, completely integrated with compiler sets and directly extending them.  I think the author of Neomake considered syntastic grossly over-engineered.  I don't think Neomake was eve meant to be an async syntastic.
Now, is syntastic grossly over-engineered?  I'm sure it is.  I'm a scientist that happens to know a few things about computers, not a programmer, and when I started all this I didn't know substantially more about Vim than the average user (despite having already used Vim for 15+ years at the time).  It probably shows.  However, there is a number of things in syntastic that are the way they are for a number of reasons.  The author of Neomake was not aware of these reasons (which is by no means surprising, since errorformat and friends are a mine field), and that also shows.  I suppose people these days address this problem by writing blog posts about their projects, where they basically keep chronicles about their reasons and their designs.  But I'm an old fart, and blogs are way past my time. :smile:

On a side note, syntastic doesn't have tests for a single reason: I'm not the owner of this repository, and activating Travi-CI for it would require the intervention of the initial author. I contacted him in the past and, while he was always nice, I did get a feeling he wants to put all this behind him, and he wouldn't appreciate being bothered with anything related to Vim these days.

It might be best to create a GitHub organization and ask the original maintainer to transfer repository ownership to the new org.
GitHub redirects old URLs when you transfer ownership, so existing links (and, more importantly, git checkouts) will continue to work.

@mgedmin I'm not particularly unhappy with the current arrangement, and there are more pressing issues at the moment than enabling Travis-CI.  I might consider it later though.

Hey guys, I have also been thinking it would be a good idea to move syntastic out into an org - for a while now. @lcd047: you have been the main maintainer for a couple of years now and it is wrong for it be under my name after all this time. For one, I am getting credit for your hard work, and two, for the practical reasons you outlined above.
Thoughts?
One thing I would consider is putting the checkers and the core in different repos. Then you could have a very open policy with adding maintainers to the checker repo. For me, it was the tedium of maintaining the checkers that I burned out on. Perhaps that is a solution if you feel the same.

@scrooloose You __do__deserve the credit, syntastic has been a very useful plugin for a years when I came around.  I just made it more maintainable, by making some optimisations, improving the docs, and by adding some obvious features.  But the overall design is yours, and it's beyond my league in creativity.
I'd be fine with either a GitHub community, or with taking over the project.  I'd also be fine with leaving it as it is, but yeah, there are a few practical problems that can only be solved by the owner of the project.
As for splitting the checkers from the core, that's probably a good idea.  It would make sense to split them by groups of languages, but that would leave the less popular languages in no man's land.  In fact the long tail is probably more important than the main languages, since languages like C/C++, Python, and Go already have plugins that can do much better than syntastic, simply by being more specialised.

OK, well that was scary but seems to have worked fine.
@lcd047 I have created the vim-syntastic org and added you as an owner (i.e. the highest permissions). I have then transferred syntastic into this org. All the links are redirecting fine (AFAICS) so it looks like a win.
Thanks for the kind words :-) I wouldn't undersell your hard work and persistence though. I think the syntastic project was very lucky that you came along just as I was running out of steam.

@scrooloose Thank you, it seems to be working fine for me too.

Someone mentioned instability concerns; I use YCM and it already leaves tern stomping around 100% CPU semi-regularly.
I really would prefer syntastic to stop blocking the UI because I have tmux binds that need Vim to stay responsive for the keybinds to stay responsive.
Apologies for not reading the thread to the letter as that'd take the better half of a day which I'd be happy to devote if I could afford that. It sounds like architectural changes that will make syntastic even more amazing than it already is are in the pipes and they'll include robust async checkers. 👍

ALE is a Syntastic-like Vim plugin that works asynchronously.  AFAICT it supports fewer languages than Syntastic, but you could check it out today instead of waiting for Syntastic to get support for async.

Neomake is also an alternative

@unphased Sorry to disappoint you, but for the last few months I've been struggling with health problems.  I'm a sick, old fart, and my priorities are those of a sick, old fart.  Support for async will be ready when / if I can get myself together enough to finish it.  Until then, you shouldn't really expect more than basic support for the existing code.  In the mean time you could always:

submit a patch to add support for async,
fork syntastic, or
use something else.

Not sure what else I could tell you. shrug

@lcd047 Hope you get well soon, i'm grateful for your great work on syntastic. 🙏

@lcd047 Absolutely, your contributions to the Vim ecosystem have NOT gone unnoticed!

I know +1s are frowned upon, but I also wanted to say how much I loved and used Syntastic, it was one of the first plugins I used with vim, and we used it extensively. Love your work, and thank you.

Thank you, @lcd047 for all your work on Syntastic. I hope you feel better soon. If working on Syntastic adds stress to your life and asynchronous support will be a lot of work, you might want to try out ALE that @mgedmin suggested. I have been testing it recently and it works well -- for the languages I use it is like a drop-in replacement for Syntastic except it doesn't hang my terminal when I save. I worry that by the time Syntastic has asynchronous support that most of the userbase will have migrated to something else like it.

https://github.com/vim-syntastic/syntastic/issues/699#issuecomment-285350988
there have already been several async linter runners made. ALE is one of the newest and will have the most changes and activity, sure, but tons of people have used syntastic for a long time and there is a huge amount of linter support here.

And, last but not least, I have exactly zero use for all this, I'm pretty happy with the sync checks. shrug

I just switched from syntastic to ALE which went pretty easy. I figured someone might wonder how to do that in a reasonable manner, so here's a basic port of my config: https://github.com/ErikBjare/dotfiles/commit/2b946b5d0f95ba056ce99bf0487c3ff414ce3c51

Still blocking only, with waiting a few seconds after file saving? :(


[FLINK-19982][test] Change the log when CollectSinkFunction receives an invalid request from debug to info for future investigation

Brief change log

Change the log when CollectSinkFunction receives an invalid request from debug to info for future investigation of FLINK-19982.

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.

@flinkbot run azure


"process is not defined" erorr makes app non-responsive

I'm in the process of upgrading a CoreUI v3 project to v4. I've got things working except for some minor layout tweaks.
The problem I'm having is that the app is unresponsive when I leave the browser tab and come back to it. For example, clicking on a link results in no behavior. In the console, I see the following error. This error seems to occur either when I leave the tab or return to it. It's reproducible 100% of the time, no matter which page of my app I'm on.
Uncaught ReferenceError: process is not defined
    at 4043 (<anonymous>:2:13168)
    at r (<anonymous>:2:306599)
    at 8048 (<anonymous>:2:9496)
    at r (<anonymous>:2:306599)
    at 8641 (<anonymous>:2:1379)
    at r (<anonymous>:2:306599)
    at <anonymous>:2:315627
    at <anonymous>:2:324225
    at <anonymous>:2:324229
    at e.onload (index.js:1:1)

Has anyone else run into this? I'm not certain it is CoreUI related, but it only started happening when I upgrade to v4.

I've discovered the app is non-responsive because there is an invisible iframe covering the page. I think this might be related to this: https://github.com/facebook/create-react-app/issues/11773

Yup. The solution here fixed it for me: https://github.com/facebook/create-react-app/issues/11773#issuecomment-995933869
This was caused by some incompatibilities in React libraries and not Core UI itself. I'll close this ticket.


Prepare ledger release

Description
This PR includes various fixups needed for the next release to CHaPs, as well as the follow up task of updating Changelogs
Checklist

[x] Commit sequence broadly makes sense and commits have useful messages
[ ] New tests are added if needed and existing tests are updated
[x] When applicable, versions are updated in .cabal and CHANGELOG.md files according to the
versioning process.
[x] The version bounds in .cabal files for all affected packages are updated. If you change the bounds in a cabal file, that package itself must have a version increase. (See RELEASING.md)
[x] All visible changes are prepended to the latest section of a CHANGELOG.md for the affected packages. New section is never added with the code changes. (See RELEASING.md)
[x] Code is formatted with fourmolu (use scripts/fourmolize.sh)
[x] Cabal files are formatted (use scripts/cabal-format.sh)
[x] hie.yaml has been updated (use scripts/gen-hie.sh)
[x] Self-reviewed the diff

Looks like CI caught a bug at the very last minute. I'll have to redo the release once we fix it.

I decided to make a release regardless of this bug. We'll make a patch release for cardano-ledger-conway, once we squash it.


Add a refresh button to the source panel

This would be useful to refresh the source tree when it has changed in the source (eg. new probe added in the JRDS server).

A refresh button would be nice for another reason: to reload a time series file (mine are CSV) that keeps growing. So that you don't need to exit and restart BinJr just to reload the data files, to take a look at recent growth of the time series.

You'll find more details about this feature here: https://github.com/orgs/binjr/discussions/153#discussioncomment-10386483


jinja2.exceptions.TemplateNotFound: circos.conf.j2

Hi
I installed from the latest repository code, and get the following error:
fsnviz --png fusioncatcher final-list_candidate-fusion-genes.txt

jinja2.exceptions.TemplateNotFound: circos.conf.j2

Do you have an idea why this may fail?

Hi @mpschr,
Thanks for bringing this up. May I ask what the actual commands you executed for installation?
Yesterday I made a new release to PyPI, so you should be able to just do pip install fsnviz and get a working installation from there. There is practically no functional different (at the time of writing) between the code in the repo now and the release in PyPI :).

Closing because of no activity. Feel free to reopen if this isn't yet resolved.


How can i read the payload as ASCII or string?

How can i read the payload as ASCII or string?
Originally posted by @ItaloSa in https://github.com/melihyarikkaya/react-native-serialport/issues/8#issuecomment-485412461

You need HEXSTRING or INTARRAY  convert to utf16 with fromCharCode when  after readed.
if your returned data type is INTARRAY
onReadData(data) {
  let intArray = data.payload;
  let str = "";
  let radix = 10;
  for (var i = 0; i < intArray.length && intArray.substr(i, 2) !== "00"; i += 2) {
    str += String.fromCharCode(parseInt(intArray.substr(i, 2), radix));
  }

  console.log(str);
}

or HEXSTRING
onReadData(data) {
 let hex = data.payload;
  let str = "";
  let radix = 16;
  for (var i = 0; i < hex.length && hex.substr(i, 2) !== "00"; i += 2) {
    str += String.fromCharCode(parseInt(hex.substr(i, 2), radix));
  }
  console.log(str);
}

@ItaloSa Did this answer help you?

That is helping me a lot! I'm using it on my research in Systems and Networks of Internet class. I'm looking to read serial data from NODEMCU that's sniffing probe requests. Thanks a lot.


start.sh script missing

release 0.6.1
the scripts/start.sh script is missing in the 0.6.1 version, causing gogs_supervisord.sh to be unable to start gogs.
It was removed via the commit 6c84223f4305784fd0d13708f623c785d48221e5

I think you are confused about how you can start gogs. I believe you have downloaded gogs binary. you can start gogs server by going to your installation directory and running
$ ./gogs web
Let me know if that works.

Thank you for you response.
I already knew that It can be started like that, and it works, but in order to use gogs_supervisord.sh to start/stop/restart gogs (as detailled on this page http://gogs.io/docs/installation/configuration_and_run.md), you need the start.sh script.
use case: starting gogs at reboot from the crontab

oh man sorry for that.
I would suggest install supervisor as init.d somehow doesn't worked for me. to do that,
sudo apt-get -y install supervisor
and edit its file via
sudo nano /etc/supervisor/supervisord.conf
[program:gogs]
directory=/home/git/go/src/github.com/gogits/gogs/
command=/home/git/go/src/github.com/gogits/gogs/gogs web
autostart=true
autorestart=true
startsecs=10
stdout_logfile=/var/log/gogs/stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stderr_logfile=/var/log/gogs/stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
environment = HOME="/home/git", USER="git"
change directory's to match your installation and user to your git user. and restart the supervisor via,
sudo service supervisor restart
that's all you need to make gogs start automatically on boot.

Yes I did something like that to make it work, thanks.
I was mainly reporting the issue because the documentation suggest to use this method to control the process. Maybe the documentation should be updated to reflect the changes ?

Yeah. I see issues like that with documentation. lot of outdated documentation. infect i have asked for help on this as i wanted to improve it.

``program:gogs]
directory=/home/git/go/src/github.com/gogits/gogs/
command=/home/git/go/src/github.com/gogits/gogs/gogs web
autostart=true
autorestart=true
startsecs=10
stdout_logfile=/var/log/gogs/stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stderr_logfile=/var/log/gogs/stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
environment = HOME="/home/git", USER="git",PATH="/usr/local/git/bin"


change mergerfs to pull all releases as latest is source-only

ATTN: this also uses the available jammy release on kinetic so we don't break : it works for me, but ymmv - and you may prefer not to do this, or to tweak the grep to guess and try and catch and prefer kinetic should it appear ...

17 hours later the LTS assets appeared so latest works again.


Addition of Brazil domain

Hi!
I'm using Securitas Direct in my country (Brazil), but ApiDomain for my country is a special address (like others).
This is the address that should be added to securitas_direct_new_api/domains.py:
"br": "https://customers.verisure.com.br/owa-api/graphql",
On self.domains dict.
Thanks a lot for the good work!
Cheers!

Thanks for the suggestion. Added the endpoint to the list of endpoints.
https://github.com/guerrerotook/securitas-direct-new-api/commit/3d53f8476851027655aed5d0a93db6cf201e192a

Here is the release.
https://github.com/guerrerotook/securitas-direct-new-api/releases/tag/v<IP_ADDRESS>


behapy: A behavioural neuroscience analysis package for Python

Title
behapy: A behavioural neuroscience analysis package for Python
Leaders
Chris Nolan
Collaborators
No response
Brainhack Global 2024 Event
Brainhack Aus
Project Description
Studies using optic fibres to record real-time fluorescent biosensors in-vivo are now commonplace, yet despite an increasing literature on best practices for analysing such data, there is a surprising lack of fit-for-purpose API-level tooling. This project is a continuing effort to fill this gap by providing flexible Python-based implementations of common normalisation and artefact correction procedures for fluorescent biosensors, along with useful event-based analyses.
The goals of this project will extend beyond Brainhack Global 2024, but all are in an effort to create an open-source API and workbench for analysing fibre photometry data in a behavioural neuroscience context. Since Brainhack Global 2022, we have created a basic artefact-rejection workbench, a preprocessing stage and implemented simple linear regression for event-level analysis. This year the goal is to create a method to benchmark normalisation methods by creating data simulation functionality under different assumptions about the sources of recording noise. We are also aiming to generalise second-level linear mixed model testing for events of interest.
Link to project repository/sources
https://github.com/crnolan/behapy
Goals for Brainhack Global

Create a utility for simulating raw fibre photometry data under a variety of assumptions (for testing the toolkit).
Implement second-level linear mixed model tests for events of interest.
Develop both API-level and tutorial documentation. Docstrings are currently sparse and light on detail, and walkthroughs are virtually non-existent.
Add import scripts for MedPC datasets, and put together a skeleton framework for event summary and plotting functions.

Good first issues

Check and update installation instructions for behapy package.
Document end-user experience for running existing fibre preprocessing workbench on sample data.
Establish a configuration format / tool for importing structured MedPC files.
Add the capability in the pre-processing interface to interactively compare / select different but reasonable normalisation methods.

Communication channels
https://mattermost.brainhack.org/brainhack/channels/behapy
Skills
Primarily, some knowledge of fluorescent biosensor normalisation and analysis procedures will be useful. We'll be predominantly working in Python, but there will be tasks for all levels of Python competency.
Bonus useful skills:

Signal processing (we'll be filtering and fitting timeseries data)
Python interactive visualisation (bokeh / holoviews / panel / seaborn / matplotlib)
Working knowledge of linear regression and mixed effects models
BIDS experience - while we won't be attempting to add an official BIDS extension for fibre photometry in this project, we are trying to stay approximately in line with BIDS format

Onboarding documentation
No response
What will participants learn?

Data manipulation in Python (numpy / pandas)
Signal filtering in Python
GitHub collaboration techniques

Data to use
BYO fibre & behavioural data - we'll create a repository of useful examples.
Number of collaborators
3
Credit to collaborators
Project contributors will be listed on the project README (hosted on github).
Image
Leave this text if you don't have an image yet.
Type
data_management, pipeline_development, visualization
Development status
0_concept_no_content
Topic
data_visualisation, systems_neuroscience
Tools
BIDS, other
Programming language
Python
Modalities
behavioral, other
Git skills
1_commit_push
Anything else?
No response
Things to do after the project is submitted and ready to review.

[x] Add a comment below the main post of your issue saying: Hi @brainhackorg/project-monitors my project is ready!
[ ] Twitter-sized summary of your project pitch.

Hi @brainhackorg/project-monitors my project is ready!


Automatically highlight - bank key (ground items) (dmm)

Can you automatically highlight the bank key on the ground when a player dies in dmm?
it's super important in this game mode.
thanks

Add it to the highlight list in Ground Items

Add it to the highlight list in Ground Items

I just thought it would be beneficial if you could do that automatically for us. it would save every DMM player to add it manually when it's universally important in that game mode. Thanks anyways.


How to remove improve this page and disqus from posts

How do I remove the "improve this page" and "comments powered by disqus" from posts?

Just remove disqusShortname and gitRepo under params section from your config.yaml.

Thank you!


[HUDI-5166] Reduce test run time (Setting all shuffle par for tests to 2)

Change Logs
Describe context and summary for this change. Highlight if any code was copied.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@hudi-bot run azure

@hudi-bot run azure

Closing this for now as we've already made significant effort to reduce the CI run time to around 1 hour.


docs: fixed typo in quick start code.

Fixed typo !.

It's not a typo, since its return would infer the element possibly not being present on the page and the ! forces typescript into believing that the element is defined.


Fixes: promise usage and string operator usage for recent Sequelize

Now module is compatible with Sequelize 4.13+ (did not use string operator aliases)

Tests are passing for me

Forgot to mention minor patch:

mocha now did not exit automatically starting 4.0.0, so added --exit to force exit;

All mods are done for now.


Desktop Logo is hard to see.

Desktop Logo is hard to see on dark backgrounds. Recommend a green outline or background for the logo.

what do you think about this? (rounded borders have transparent bg's, center white-ness is opaque)

@ia Much much better for me!

I like it. Works for me.

Sweet. I'm going to close this now until more designs come in or further notice.


many to many through association on through table model return array instead of single object

many to many through association on through table model return array instead of single object :
  collections.user = Waterline.Collection.extend({
    identity: 'user',
    connection: 'foo',
    attributes: {
      id: {
        type: 'integer',
        primaryKey: true
      },
      cars: {
        collection: 'car',
        through: 'drive',
        via: 'car'
      }
    }
  });

  collections.drive = Waterline.Collection.extend({
    identity: 'drive',
    connection: 'foo',
    attributes: {
      id: {
        type: 'integer',
        primaryKey: true
      },
      car: {
        model: 'car'
      },
      user: {
        model: 'user'
      }
    }
  });

  collections.car = Waterline.Collection.extend({
    identity: 'car',
    connection: 'foo',
    attributes: {
      id: {
        type: 'integer',
        primaryKey: true
      },
      drivers: {
        collection: 'user',
        via: 'cars',
        dominant: true
      }
    }
  });

  Drive.findOne(1)
  .populate('car')
  .populate('user')

return
{
  id: 1,
  car:[ { id: 1,createdAt: '2015-09-02T08:51:37.393Z', updatedAt: '2015-09-02T08:51:37.393Z' } ],
  user:[ { id: 1,createdAt: '2015-09-02T08:51:37.393Z', updatedAt: '2015-09-02T08:51:37.393Z' } ]
}

istead of
{
  id: 1,
  car: { id: 1,createdAt: '2015-09-02T08:51:37.393Z', updatedAt: '2015-09-02T08:51:37.393Z' } ,
  user: { id: 1,createdAt: '2015-09-02T08:51:37.393Z', updatedAt: '2015-09-02T08:51:37.393Z' }
}

Anyone have time to look into this?  Failing tests are here #1134.  This will be required for proper through association support.

@devinivy that's strange, if I had to guess I would say it must have something to do with Waterline Schema flagging the Drive collection as a join table.

@particlebanana one more time you where right !

Safe to use in production?

@weyert in fact i think that waterline' actual manytomany through association is really buggy and should not be used in production...my PR correct some bugs...

Thanks for the head ups. You are managing it manually now then?
I will probably just go for the simple case of one user can have many categories.

@weyert for simple case use manytomany association

@atiertant I think I tracked this down even further. So first some background, when you have two collections pointing at each other (many-to-many) waterline-sequel generates a join table for you and flags it as a junctionTable. The current through associations implementation allows you to map a collection to an existing join table in your database. It works the same as the generated join table and expects there to be two foreign keys. Everything else works the same, the limitation being if you have additional attributes on the join table they will not be returned when populating either of the sides.
So what was happening was basically because it's tagged as a junctionTable Waterline's loader thinks it doesn't have it as an actual user defined collection - because it assumes waterline-sequel created it - and so the definition gets processed twice. This causes the definition to be corrupted, because it's a schema and not an attribute definition, which causes the populates to get mapped incorrectly.
What the proposed patch here does is attempt to fight the incorrect mapping by adding some additional metadata to the table. Unfortunately from my tests, this seems to fix the populates on the join table (Drive) and break the populates on either side (User, Car).
I'm working on a patch to Waterline-schema that adds a throughTable flag on join tables that are defined as actual collections. This will allow us to patch the collection loader and make it so that these don't get processed twice, fixing the population mapping. It should be done momentarily.
I did have a question about your models above and want to make sure I am on the same page as far as the issue is concerned.
In your User collection you define an attribute named cars that is pointing to the join table using the through mapping. However it seems your via key is incorrect. The drivers attribute on the Car collection also seems to be missing the through value and the via seems to be pointing to the User collection.
So just to clear up how that should work you can think of it as: "This population will return the collection User by looking through the Drive collection via the user key".
This would make the collections look like so:
collections.user = Waterline.Collection.extend({
    identity: 'user',
    connection: 'foo',
    attributes: {
      id: {
        type: 'integer',
        primaryKey: true
      },
      cars: {
        collection: 'car',
        through: 'drive',
        via: 'user'
      }
    }
  });

  collections.drive = Waterline.Collection.extend({
    identity: 'drive',
    connection: 'foo',
    attributes: {
      id: {
        type: 'integer',
        primaryKey: true
      },
      car: {
        model: 'car'
      },
      user: {
        model: 'user'
      }
    }
  });

  collections.car = Waterline.Collection.extend({
    identity: 'car',
    connection: 'foo',
    attributes: {
      id: {
        type: 'integer',
        primaryKey: true
      },
      drivers: {
        collection: 'user',
        through: 'drive',
        via: 'car',
        dominant: true
      }
    }
  });

I'll tag you in the patches I am submitting so you can make sure they pass. I'll cherry pick in fc0c671a98acfa997461533f9e34a812ddcd65dc so we have it tested and fix up the via's as well.

@particlebanana you are right my via are wrong ...
i think first thing to do is to define spec because i never know if it should contain a side or other ... https://github.com/balderdashy/waterline-docs/blob/master/models/associations/many-to-many-through.md

@atiertant @particlebanana should this be reopened?


save and import embeddings for faster answer generation

Hello,
Thank you for the fantastic work on PaperQA. I’ve been able to use it to ask questions by providing over 100 papers as input, and I’ve been using only local models via Ollama. Everything is working well, but I’d like to know how I can avoid reloading the same files and retraining an embedding model each time I have a new Question.
Is there a way to save the vector store and load it later, so it can be used by the LLM to generate answers? I couldn't find documentation about that, I found something re- caching, but it's unclear to me how to use it properly. Can you provide some help?
best wishes
My code so far:
from paperqa import Settings, ask

question = "Can you list all climate-sensitive pathways that affect rotavirus incidence?"

model = "ollama/llama3.1" # "ollama/llama3.2"
embedding = "ollama/nomic-embed-text" # "nomic-embed-text"

local_llm_config = {
    "model_list": [
        {
            "model_name": model,
            "litellm_params": {
                "model": model,
                "api_base": "http://localhost:11434",
            },
            "answer": {
                "evidence_k": 40,
                "evidence_detailed_citations": True,
                "evidence_summary_length": "about 100 words",
                "answer_max_sources": 5,
                "answer_length": "about 300 words, but can be longer",
                "max_concurrent_requests": 4,
                "answer_filter_extra_background": False
            }

        }
    ]
}

answer = ask(
    question,
    settings=Settings(
        llm=model,
        llm_config=local_llm_config,
        summary_llm=model,
        summary_llm_config=local_llm_config,
        agent=AgentSettings(
            agent_llm=model, agent_llm_config=local_llm_config
        ),
        embedding=embedding,
        paper_directory="papers/"
    ),
)

ok let's do a step back. Show me:

How to add my documents to the Docs object and how this is used in to generate the embeddings

Show me how to use the loaded embeddings in the code to generate the answer

Show me how to set to use the following embedding model when generating the embeddings stored into Docs
embedding = "ollama/nomic-embed-text" # "nomic-embed-text"


Ionic 2 RC2 Polyfill.js Argument not optional on Windows 10 mobile

Updated to the latest RC2, but I now get errors when deploying to the emulator/device.
Here's what I've tried

ionic start test --v2
ionic platform add windows
npm run build --dev
Open up CordovaApp.sln
Set the device to mobile emulator and deploy

And this is the error that happens in the Polyfill.js file Exception about Argument not optional

Here's what the package.json file has, but it's a vanilla project soo...
{
  "name": "ionic-hello-world",
  "author": "Ionic Framework",
  "homepage": "http://ionicframework.com/",
  "private": true,
  "scripts": {
    "build": "ionic-app-scripts build",
    "watch": "ionic-app-scripts watch",
    "serve:before": "watch",
    "emulate:before": "build",
    "deploy:before": "build",
    "build:before": "build",
    "run:before": "build"
  },
  "dependencies": {
    "@angular/common": "2.1.1",
    "@angular/compiler": "2.1.1",
    "@angular/compiler-cli": "2.1.1",
    "@angular/core": "2.1.1",
    "@angular/forms": "2.1.1",
    "@angular/http": "2.1.1",
    "@angular/platform-browser": "2.1.1",
    "@angular/platform-browser-dynamic": "2.1.1",
    "@angular/platform-server": "2.1.1",
    "@ionic/storage": "1.1.6",
    "ionic-angular": "2.0.0-rc.2",
    "ionic-native": "2.2.3",
    "ionicons": "3.0.0",
    "rxjs": "5.0.0-beta.12",
    "zone.js": "0.6.21"
  },
  "devDependencies": {
    "@ionic/app-scripts": "0.0.39",
    "typescript": "2.0.6"
  },
  "description": "test: An Ionic project",
  "cordovaPlugins": [
    "cordova-plugin-device",
    "cordova-plugin-console",
    "cordova-plugin-whitelist",
    "cordova-plugin-splashscreen",
    "cordova-plugin-statusbar",
    "ionic-plugin-keyboard"
  ],
  "cordovaPlatforms": []
}

I've even stripped out everything to this, and it still get the error.
<!DOCTYPE html>
<html lang="en" dir="ltr">
<head>
</head>
<body class="platform-windows platform-cordova platform-webview">
  <script src="build/polyfills.js"></script>
</body>
</html>

ionic serve also gives this error in RC2. The application was working fine in RC1 and when I update it to RC2, I get this error. No changes made to the code.

I am seeing the same thing...followed by:
Unable to get property '__symbol__' of undefined or null reference
in main.js

Windows 10, Edge.

Yes, I have this too. Has to do with the Zones.js somehow? Workaround is currently to revert to RC 1

Don't revert to RC1. It works fine in Chrome. Looks like an issue with Edge or non-standard behavior tailored for chrome ( which is the case these days ). If you need to test it for Windows Phone then no other option but to revert to RC1.

Yah, I don't have any other option, but reverting. I utilize WinRT API's that don't exist in Chrome, or have Cordova plugins. My workflow is make a code change, compile with cli, build in Visual Studio, and then deploy to either a device or emulator.
Until Windows 10 UWP gets more attention from developers, this is just how it is.

Related to #9025

I have this problem too. You have any ideas?

It's definitely a problem in Edge. Even viewing the online documentation doesn't work with the side preview. Either develop in chrome until it's fixed, or revert.

Which Edge version? Edge 13 seems to work fine.

Edge 14? Seems that the problem has something to do with window.fetch().

I'm running Edge 14. RC1 didn't have this issue with Edge 14, so it was something changed in Polyfill.js

@mtycholaz See my second comment.

Interesting, I'll try that.

Hello all! I am looking into this issue today. It seems that edge's implementation of the fetch api is different than other browsers (chrome, firefox, opera) which is causing an issue with the polyfills.

Is anyone here running a nightly build of edge? If so does this issue still happen in that nightly build?

I can confirm Edge 15 on Windows 10 mobile has this issue, along with Edge 14 anniversary edition.

However, in Edge 13 everything works.

Nope, just normal, actual Edge 14.14393. Doesn't work!

I  am not sure Edge has nightly build. If there is, please help me with the link. I am running Windows Insider Build 14965 ( latest ). It still does not work in the Edge version that comes with that build.

Edge 13.10586 (Threshold) - Works
Edge 14.14393 Desktop (Anniversary edition) - Error
Edge 14.14393 Mobile (Anniversary edition) - Error
Edge 15.14955 Mobile (Insider build) - Error

@jgw96
I did some digging and I found something about this problem.
Ms Edge and Chrome throws an exception at polyfill.js, line 3, col 16552. Chrome continues, but Ms Edge stops.
The code that has the problem is var N=t.fetch();
I compared the polyfill.js of RC1 (which worked) with the one of RC2.
In RC1 the piece of code (N=t.fetch();) is in a try - catch, but in RC2 is not.
Then I run gulp polyfill so I can get the polyfill.js of the latest release (RC3) of the ionic repository.
The polyfill.js file that it produced(in dist/ionic-angular/polyfills direcory), was the same as RC1 file (with the try - catch). So I think there is some mistake in the distribution that we get from the npm.
Sorry for my English. I you have any questions please ask me.

Hello everyone! We have updated to the latest Angular as of this morning (which means it will be in the next release) which includes a fix for this issue. Thanks!

Any ETA on the next release?

I've tested the conference app with RC3 and angular 2.2.1 - and as far as I can see, it still has the problem above.
@jgw96 have you tested this with Edge 14 browser?

@mpaland I think we need to wait until RC4. @jqe96 made his comment after RC3 was released.

@mtycholaz You don't need to wait for RC4. You can always use the latest branch to test actual changes.
As @jgw96 pointed out, an update of angular (to 2.2.1 as in latest) should fix the bug. But I can't confirm this.

@mpaland I'll try that once I get home.

@mpaland Just tried updating the packages.json file to Angular 2.2.1. Everything compiles, but the polyfill.js still throws an error. We'll just need to be patient. I tried this in Edge Mobile 14.14393

Hey! I got the same problem! I tried all combination with ng2.2.1, ng2.1.1, ion.rc1 and ion.rc3 without success. Using ion.rc3 with ng2.2.1 lead to #9015 so i couldn't even build.
Is there already some date or estimation for rc4? just saw that there are still 12 issues still open.
Also Offtopic: I'm a linux guy and really hate the fact that I have to use windows+vs2015 and need to reload the project in vs2015 after each build and being still not able to deploy the app to my lumia 540 with vs2015.
So I searched for alternatives and found the command ionic run windows --device -- --arch=ARM --phone to run from cmd and was able to deploy to my device. Despite the polyfills problems described earlier the app can be started. The loginscreen shows and i can put in my credentials but i can't use my login button. Is there a way to add debugging mode to the cmd command like with android or do I have to depend on vs2015?

@DeusProx Are you coding in linux, and then deploying to a device via a Windows VM?

@DeusProx If you are using VS2015, then you can always debug the app using Visual Studio debugger. In your tsconfig.json file if you declared --sourceMap flag, then you should debug the code. That's much better than debugging from the command prompt.

@jagansai I would like to not even have to start vs2015. That's why i asked for the right commands
@mtycholaz Coding on linux for android and then trying to port it on windows deploying it directly on a lumia 540 with win10.
The problem with the polyfill.js only appears to happen when I try to emulate a w10 phone(which I won't do in future because it uses Hyper-V and I need VT-x for my virtualbox....).
ATM I deploy to the lumia 540 from command line with ionic run windows -l -c -s --device -- --arch=ARM --phone. This works fine as long as I don't close the ionic server which is quite logical. When I use it without the live reload server (ionic run windows --device -- --arch=ARM --phone) it starts but pushing the login button doesn't do anything and i have no debugging on clientside and can't check the backend as I don't have access... Sry that this is all offtopic. maybe I should create a new issue for this.

@DuesProx cool. Good luck with the restrictions you imposed onto yourself 😊

@jagansai restrictions? i can't even start my app with vs2015.... it's just annoying to use it

@DeusProx Not sure if it's possible without the visual studio remote debugger. But that tool is (currently) only available on Windows. Microsoft just announced VS for Mac. It's not feature complete yet, but it may work on Linux?

@mtycholaz Sry I probably confused you. I got app debugging ionic run windows -l -c -s --device -- --arch=ARM --phone over windows cmd. Afaik the app is served by a live server on the local host to the smart phone. Through this the polyfills problems of this issue doesn't seems to appear & I have system.out.println and stack traces on the cmd. Without the flags -l -s -c everything is packed into the apk and deployed to the phone. This makes the app start but doesn't work correctly (& no debugging).
Deploying over vs2015 I have the same polyfills bug.

Hello all! Is this still an issue with RC3? The version of Angular we are now running includes fixes for the polyfills issue.

@jgw96 What version of Angular are you running?

@mtycholaz sorry, mis-spoke on that. So with the latest nightly release we are now running on 2.2.1. To update to that nightly you can change your package.json to look like this.

@jgw96 I believe I tired this already and it didn't work, but I'll try again later tonight.

@jgw96 Confirmed, I've updated my package.json to the one you provided. Everything works now. Thanks!

Confirmed! Working in nightly. Nice work!

tried ngc 2.2.x and ngc 2.1.x with  "ionic-angular": "2.0.0-rc.3-201612081620", on edge 13.10586
and I am still experiencing the issue.
any ideas ?


Compile a list of guideline

Complile a list of hiking and safety guideline that responsible hikers should follow.

Created list of general guidelines/pack essentials, CrownClown included in his html commits


Can't open tensorflow.org

I can't open https://www.tensorflow.org/ but I can open https://tensorflow.google.cn, why is that? If tensorflow has changed its URL why didn't update it? When I google tensorflow it also leads to https://www.tensorflow.org, but it cannot be accessed.
Does anyone else having this problem?

Most likely a routing issue between your exchange and tensorflow.org - contact your ISP if it hasn't been fixed in a day or two.


Use host vars of the first manager node when usig connection = local

Closes osism/issues#1058

@matofeder What do you think about this? Can you test it for the Kubernetes monitoring part? I think this should work.

My test play:
- hosts:
    - "{{ hosts_manager|default(groups['manager'][0]) }}"
  connection: local

  tasks:
    - debug: msg="{{ openstack_exporter_clouds_yml_cloud }}"

Output:
$ osism apply -e custom test
2024-06-26 10:06:45 | INFO     | Trying to run play test in environment custom
2024-06-26 10:06:45 | INFO     | Task 09037c0c-d0c6-4e26-a066-2e1e93a09c85 (test) was prepared for execution.
2024-06-26 10:06:45 | INFO     | It takes a moment until task 09037c0c-d0c6-4e26-a066-2e1e93a09c85 (test) has been started and output is visible here.
[WARNING]: Collection osism.commons does not support Ansible version 2.17.1
[WARNING]: Skipping callback 'ara_default', unable to load due to: The server
dependencies must be installed to record data offline or run the API server.

PLAY [manager.systems.in-a-box.cloud] ******************************************

TASK [debug] *******************************************************************
Wednesday 26 June 2024  10:06:47 +0000 (0:00:00.278)       0:00:00.278 ********
ok: [manager.systems.in-a-box.cloud] => {
    "msg": "admin"
}

PLAY RECAP *********************************************************************
2024-06-26 10:06:48 | INFO     | Play has been completed. There may now be a delay until all logs have been written.
2024-06-26 10:06:48 | INFO     | Please wait and do not abort execution.
manager.systems.in-a-box.cloud : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Wednesday 26 June 2024  10:06:48 +0000 (0:00:00.412)       0:00:00.690 ********
===============================================================================
debug ------------------------------------------------------------------- 0.41s


Do not incherite replication slots in branch

Problem
See
https://github.com/neondatabase/company_projects/issues/111
https://neondb.slack.com/archives/C03H1K0PGKH/p1700166126954079
Summary of changes
Do not search for AUX_FILES_KEY in parent timelines
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

I don't feel qualified to review this


docs: added dev dependencies

I've added the two requirements that need to be installed that are not obvious. however, we could also think about adding rust as requirement to be installed in the version we are using.

bors r+


Build fails if happy is not on the PATH

Even stack install happy is not enough, unless you add ~/.local/bin to the PATH.
I think it's a problem with one of the dependencies which doesn't use proper build-depends and build-tools but tries to run happy in a legacy way.
So at least the build instructions should be updated.

This is managed in the Makefile now


Make Punycode conversion optional

What is this feature about (expected vs actual behaviour)?
Currently, the isValid callback always converts the local and domain part into Punycode.
However, lots of environments don't support Punycode and we'd like to validate email addresses such that special characters like German umlauts or other accented character would be recognized as invalid.
I'll be posting a PR in a minute to fix this.
Does it take minutes, hours or days to fix?
Minutes.

@jurgenhaas There is a polyfill for punycode build-in (via https://github.com/symfony/polyfill-intl-idn/blob/main/Idn.php#L148) is that not working? Do you have an example?

@voku It's the other way round: Punycode is working but I don't want it to be involved. Because you're converting email addresses before validation with punycode, the email doesn't contain special characters anymore and therefore the validation returns the email to be correct. But in reality, the email is not correct because it contains special characters. So what we need to achieve is to not use Punycode and let the validation return with false so that we know it is invalid.

@jurgenhaas Sorry, I still do not get it. We need to use punycode so that we can check that, e.g<EMAIL_ADDRESS>(https://github.com/voku/email-check/blob/master/tests/EmailCheckTest.php#L346) is a valid e-mail address.
https://3v4l.org/d50BN

Yes, that's correct. But in our environment, this is NOT a valid e-mail address. So, when I can turn off the Punycode feature, you will still get the same behaviour as before, but users like us can turn this off and get an error, when a user comes along with such an address.


SQLservercdc将配置文件所有表读取出来了，无视配置文件

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
读取配置文件时候把库里面的所有表都读取出来了，然后按照第一张表进行增量读取
SeaTunnel Version
2.3.1
SeaTunnel Config
env {
 execution.parallelism = 1
}
source {
  SqlServer-CDC {
    hostname = "<IP_ADDRESS>"
    port = "1434"
    username = "sa"
    password = "123456aA??"
    database-names = ["testdb"]
    table-name = "testdb.dbo.sqlserver-cdc-oracle"
  }
}
transform {
}
sink {
  Jdbc {
    url = "jdbc:oracle:thin:@<IP_ADDRESS>:1521:helowin"
    driver = "oracle.jdbc.OracleDriver"
    user = "system"
    database = "HR"
    password = "helowin"
    table = SQLSERVER_CDC_TO_ORACLE
    primary_keys = ["ID"]
  }
}

Running Command
bin/seatunnel.sh --config ./config/sqlserver-cdc-oracle.conf -e local

Error Exception
2023-04-02 14:20:24,602 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.TableDiscoveryUtils - 	 list of available databases is: [master, tempdb, model, msdb, testdb]
2023-04-02 14:20:24,602 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.TableDiscoveryUtils - Read list of available tables in each database
2023-04-02 14:20:25,076 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.TableDiscoveryUtils - 	 including 'testdb.dbo.sqlserver_cdc_to_greenplum_test' for further processing
2023-04-02 14:20:25,076 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.TableDiscoveryUtils - 	 including 'testdb.dbo.sqlserver_cdc_to_oracle' for further processing
2023-04-02 14:20:25,079 INFO  io.debezium.jdbc.JdbcConnection - Connection gracefully closed
2023-04-02 14:20:25,134 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received enough reader, starting enumerator...
2023-04-02 14:20:25,179 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.source.eumerator.SqlServerChunkSplitter - Start splitting table testdb.dbo.sqlserver_cdc_to_greenplum_test into chunks

Flink or Spark Version
2.3.1
Java or Scala Version
jdk 8
Screenshots

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

good job
Can you replace the title and description in English?


[Discussions] How to retrieve the original Crop Rect from the new cropping process?

Content
The old cropping process was as follows:
Crop -> Flip -> Rotate
The new cropping process is:
Rotate -> Flip -> Crop
This means the current Crop Rect obtained via getCropRect is transformed by the flip and rotate operations. How can I retrieve the Crop Rect before any transformations are applied?
I tried the old method, but it didn't give back the proper result:
  // https://github.com/fluttercandies/extended_image/blob/03bd2ad621583f2f716da1915506f8df34ec8f52/lib/src/editor/editor.dart#L291C9-L291C20
  Rect? _getCropRect(ExtendedImageState extendedImageState, EditActionDetails? editActionDetails) {
    if (extendedImageState.extendedImageInfo?.image == null || editActionDetails == null) {
      return null;
    }

    var cropScreen = editActionDetails.screenCropRect;
    var imageScreenRect = editActionDetails.screenDestinationRect;

    if (cropScreen == null || imageScreenRect == null) {
      return null;
    }

    imageScreenRect = _paintRect(imageScreenRect, editActionDetails);
    cropScreen = _paintRect(cropScreen, editActionDetails);

    //move to zero
    cropScreen = cropScreen.shift(-imageScreenRect.topLeft);

    imageScreenRect = imageScreenRect.shift(-imageScreenRect.topLeft);

    final image = extendedImageState.extendedImageInfo!.image;
    // var size = _editActionDetails.isHalfPi
    //     ? Size(image.height.toDouble(), image.width.toDouble())
    //     : Size(image.width.toDouble(), image.height.toDouble());
    final imageRect = Offset.zero & Size(image.width.toDouble(), image.height.toDouble());

    final ratioX = imageRect.width / imageScreenRect.width;
    final ratioY = imageRect.height / imageScreenRect.height;

    final cropImageRect = Rect.fromLTWH(cropScreen.left * ratioX, cropScreen.top * ratioY, cropScreen.width * ratioX, cropScreen.height * ratioY);
    return cropImageRect;
  }

  // https://github.com/fluttercandies/extended_image/blob/03bd2ad621583f2f716da1915506f8df34ec8f52/lib/src/editor/editor_utils.dart#L154
  Rect _paintRect(Rect rect, EditActionDetails editActionDetails) {
    var resultRect = rect;

    if (!editActionDetails.hasEditAction || editActionDetails.screenCropRect == null) {
      return resultRect;
    }

    final flipOrigin = editActionDetails.screenCropRect!.center;
    if (editActionDetails.hasRotateDegrees) {
      resultRect = editActionDetails.rotateRect(resultRect, flipOrigin, -editActionDetails.rotateRadians);
    }

    if (editActionDetails.flipY) {
      resultRect = Rect.fromLTRB(
        2 * flipOrigin.dx - resultRect.right,
        resultRect.top,
        2 * flipOrigin.dx - resultRect.left,
        resultRect.bottom,
      );
    }

    return resultRect;
  }

what do you want to do?

My server-side service applies filters to the image in the following order: Crop -> Flip -> Rotate.
My goal is to retrieve the crop area as if the image hadn’t been rotated or flipped.
 The previous algorithm and the current one differ in that the current version supports arbitrary angles, which has resulted in a change in the order. If the current version does not meet your needs, you may consider using an older version.


build(bundles): inline sourcesmap and stop shipping sourcecode

Source codes are not needed for sourcemaps when having inlineSource
Yesterday it was discussed that we should probably make rxjs stop publishing src/ while still having sourceMaps to work.

With the other PR https://github.com/ReactiveX/rxjs/pull/4309 downstream users shall no longer require to compile RxJs code in order to get it working in Bazel thus there is no need to ship it and as a matter of fact we're remove Bazel.
//cc @alexeagle & @gregmagolan

Woule you elaborate for this PR's reason / what to solve? Plus with declarationmap in further, I think we will still need to include src codes anyway.

Updated description

I see reasoning, but I do plan to enable declarationMap option in latest compiler which will need ship source still. If we enable inline sources & sources, wouldn't it be duplicate? / and is there any compiler issues by shipping source but not inlining?

I think in that case in makes sense to keep the source.
@alexeagle what to do think?

My take on this is that no npm users expect to be able to compile typescript sources they fetch in the package, because doing so requires knowing how the typescript compiler should be invoked (maybe just the right tsconfig.json file but maybe some directories are compiled separately or maybe there are command line args to tsc that aren't in the tsconfig, etc)
So Angular team is not shipping any TS files to npm and we are stopping that practice with Bazel as well.
Obviously you can ship .js.map and .d.ts.map files that have the references to original TS line numbers but not ship the sources. That seems like the smallest package you can ship and doesn't harm your users?

.d.ts.map files that have the references to original TS line numbers but not ship the sources.

@alexeagle so far I know d.ts.map doesn't inline sources and source need to be placed to allow navigate (i.e in code editors go to definition) etcs, reason I'd preserve to ship source in pkg , totally agreed it's not for direct building on user's end.
does d.ts.map works without shipping sources? then shipping source obviously not required.

I was following https://github.com/Microsoft/TypeScript/wiki/What's-new-in-TypeScript#new---declarationmap Language Services can also now understand these map files, and uses them to map declaration-file based definition locations to their original source, when available. and so far when I tried with my pkgs, it seems source was needed.

Currently, RxJS must support CJS, ES5 and ES2015 users. If we suddenly switch to inline source maps, it will inflate the overall size of the bundle considerably for no real gain, and in fact, there may be people using the TS source directly. :grimacing:

I don't think anyone cares that strongly about this one.


Workflows to add to IWC

List of GTN workflows we can put into IWC

[ ] Maker annotation
[ ] Metatranscriptomics
[ ] Mothur MiSeq SOP

The maker annotation wf maybe?

The maker annotation wf maybe?

I opened a PR https://github.com/galaxyproject/iwc/pull/47


refactor /wishlists/id

refactor delete a wishlist route
refactor update a wishlist route
refactor retrieve a wishlist route

Codecov Report

Merging #122 (a9f0af6) into main (f47a00d) will increase coverage by 0.28%.
The diff coverage is 98.00%.

@@            Coverage Diff             @@
##             main     #122      +/-   ##
==========================================
+ Coverage   95.18%   95.47%   +0.28%
==========================================
  Files           7        7
  Lines         582      575       -7
==========================================
- Hits          554      549       -5
+ Misses         28       26       -2

Impacted Files
Coverage Δ

service/routes.py
95.14% <98.00%> (+2.18%)
:arrow_up:

service/error_handlers.py
70.96% <0.00%> (-9.68%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f47a00d...a9f0af6. Read the comment docs.

IBM Cloud toolchain: Delivery Pipeline deployed wishlists to prod, including commit 0e2b27207ac877248fa54efa260f5644cc662437