Module Gradle/Java/CodeQL build fails due to "Unknown javac parameter: --add-opens"

I have Gradle Java projects that I'd love to use CodeQL to analyse. However, if I enable CodeQL either in its own workflow or integrated into the main build workflow, then the build fails with error:
Error: Exception in thread "main" Unknown javac parameter: --add-opens
	at com.semmle.extractor.java.args.JavaExtractorArgs.parse(JavaExtractorArgs.java:331)
	at com.semmle.extractor.java.JavaExtractor.<init>(JavaExtractor.java:187)
	at com.semmle.extractor.java.JavaExtractor.main(JavaExtractor.java:482)

The --add-opens command line option is being introduced by the org.javamodularity.moduleplugin Gradle plugin, which the project is using as its using Java's module system (a.k.a. Jigsaw).
However, its not javac that's complaining about the option, but com.semmle.extractor.java.args.JavaExtractorArgs.parse.  The version of java the project is using supports the option. Looks like this could be an issue with the LGTM code...?

It looks like you're relying on the autobuilder to trace the build as part of building the database. Can you try commenting out https://github.com/creek-service/creek-base/pull/17/files#diff-12783128521e452af0cfac94b99b8d250413c516ec71fe6d97dbea666ff7ba27R55-R56
And then uncomment https://github.com/creek-service/creek-base/pull/17/files#diff-12783128521e452af0cfac94b99b8d250413c516ec71fe6d97dbea666ff7ba27R65-R67 and change it to run the build commands for your specific project?

I've tried both approaches.  See https://github.com/creek-service/creek-base/pull/16...

Thanks for letting me know. I heard from the team that supports the java extractor and, unfortunately, it looks like the add-opens flag is not supported. I'm waiting for clarification if there is a better workaround, but their current suggestion is to avoid using that option during code scanning.

I'm lucky, in that its the test code that uses the add-opens options. So I guess I could just not compile and run the tests.  Would CodeQL be happy with that?

Thanks for filing this issue. This is an oversight in our Java extractor (the --add-opens flag is a hidden javac option). Our Java team will take a look at fixing this next week.

So I guess I could just not compile and run the tests. Would CodeQL be happy with that?

I would expect that to work as a short-term workaround.

Thanks @yo-h.

FYI: Changing the build target to jar has the build passing: https://github.com/creek-service/creek-base/pull/17

Great. I'll close this issue. Feel free to open a new issue if you have any other issues.

@aeisenberg to be fair, the issue isn't resolved. We've just found a work around. May be more useful for others encountering this issue if the issue were left open for now...?

Good point. Let's keep this open until a fix is shipped.

Hidden options mentioned here: https://github.com/openjdk/jdk/blob/0f2113cee79b9645105b4753c7d7eacb83b872c2/src/jdk.compiler/share/classes/com/sun/tools/javac/main/Option.java
That's for Java 18 - TODO: check if there are hidden options from previous javac versions we need to tolerate as well, and make a PR to update the extractor's list of passed-through options.
Note for the OP: javac prints warning: [options] --add-opens has no effect at compile time, so you might want to check if your Gradle plugin is doing the right thing there.

Fix merged; this should work fine as of CodeQL 2.9.1

@smowton when will 2.9.1 go live?  Is there somewhere I can view this?
I see that 2.9.1 has has been build, but trying a full build still fails with the same error. (See https://github.com/creek-service/creek-test/pull/38)

It typically around two weeks for new CodeQL releases to make it into the cache we use to avoid downloading a release every time a workflow is run. You could add tools: latest to the parameters passed to the init step of your workflow if you would like to make sure it uses 2.9.1 now. This will add around a minute to your workflow's duration but will ensure you always have the latest version without the couple of weeks delay for it to get into the cache.

I'll also note that even with the above, you won't get 2.9.1 just yet because we haven't yet bumped latest to point to 2.9.1 as it is very new. That's due to happen on Monday, or Tuesday at the latest, assuming we don't uncover any issues with it.

If you wanted to try it out locally, there's a build of the CLI at https://github.com/github/codeql-cli-binaries/releases/tag/v2.9.1

All deployed and working now. Thanks all.


feat: admin can now assign users to template

Summary | Résumé
closes https://github.com/cds-snc/platform-forms-client/issues/1203

Added admin feature to assign users to template

Test instructions | Instructions pour tester la modification

Navigate to http://localhost:3000/admin/view-templates
Assign users to template using the Assign button in the forms table (you may want to create fake users to see how the UI behaves)
Using yarn prisma:studio you can verify that the assignations are saved in the database

Pull Request Checklist
Please complete the following items in the checklist before you request a review:

[x] Have you completely tested the functionality of change introduced in this PR? Is the PR solving the problem it's meant to solve within the scope of the related issue?
[x] The PR does not introduce any new issues such as failed tests, console warnings or new bugs.
[x] If this PR adds a package have you ensured its licensed correctly and does not add additional security issues?
[x] Is the code clean, readable and maintainable? Is it easy to understand and comprehend.
[x] Does your code have adequate comprehensible comments? Do new functions have docstrings?
[x] Have you modified the change log and updated any relevant documentation?
[x] Is there adequate test coverage? Both unit tests and end-to-end tests where applicable?
[ ] If your PR is touching any UI is it accessible? Have you tested it with a screen reader? Have you tested it with automated testing tools such as axe?

Weird that my review comment didn't make it through but...
Great Work!
I would just suggest that, as a security best practice, we reduce the amount of data sent to page to only the data that will be used.


Add testing through Tox and Github actions

Some tests are broken in current master so they aren't passing.
Also the linting would need to be fixed.

Amazing 😍
I will check and merge as soon as I can

@morlandi I forgot to write, that all of the problems are now fixed and everything should pass.


Added new secret history of hacking

Please review and comment

@sachininfy this is a documentary film, so it should be placed in documentary,md. Could you please make this appropriate changes?


Discarded vertex storage

Codecov Report

Merging #335 (811d428) into breaking-02 (5bc5d80) will decrease coverage by 0.99%.
The diff coverage is 32.43%.

@@               Coverage Diff               @@
##           breaking-02     #335      +/-   ##
===============================================
- Coverage        86.24%   85.25%   -1.00%
===============================================
  Files               17       17
  Lines             1868     1878      +10
===============================================
- Hits              1611     1601      -10
- Misses             257      277      +20

Impacted Files
Coverage Δ

src/utils.jl
75.86% <0.00%> (-4.32%)
:arrow_down:

src/pairwise.jl
80.00% <37.93%> (-11.16%)
:arrow_down:

src/simplex_oracles.jl
90.24% <50.00%> (-2.07%)
:arrow_down:

src/blended_cg.jl
84.00% <0.00%> (-0.91%)
:arrow_down:

src/abstract_oracles.jl
86.45% <0.00%> (-0.80%)
:arrow_down:

src/types.jl
84.69% <0.00%> (-0.36%)
:arrow_down:

src/active_set.jl
93.33% <0.00%> (-0.11%)
:arrow_down:

src/polytope_oracles.jl
92.22% <0.00%> (-0.09%)
:arrow_down:

src/norm_oracles.jl
95.53% <0.00%> (-0.08%)
:arrow_down:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bc5d80...811d428. Read the comment docs.


Simplify RepositoryList.loadAttributes()

Remove unneeded variables and loops, and use common helper function keysInt64(..).

LGTM

trusted LGTM


HeapSort added in sorting package with testcase

Hello
I have added the HeapSort in the sorting package. The added files are :
com.sorting.Heap.java that implement the Heap sort method
com.sorting.HeapSortTest.java for testcase.

Good one @MiguelSteph  Thanks. Keep contributing more!


iio-oscilloscope: init at 0.17

Description of changes
GTK+ based oscilloscope application for interfacing with various IIO devices
https://wiki.analog.com/resources/tools-software/linux-software/iio_oscilloscope
Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

By the way, there is another PR for this package: #322262


Misplaced comment from part 4 06-01

The comment "// if FEATURE_FLAG_X is not enabled" should be placed outside the if statement, since the if statement is checking if the feature flag is enabled.

What's there now is correct. It's a single sentence - code in here will be removed from the build output if FEATURE_FLAG_X is not enabled.


Sprint-3 [issue-13] Re-evaluate scikit-learn binary classifiers with v2 data

Summary

Reran the binary_spam_classifier_experiments.ipynb notebook with v2 dataset
Added the following parameters to the vectorizers:

vectorizer_params = {
    'lowercase': True,
    'min_df': 5,
    'max_features': 25000,
    'stop_words': list(set(stopwords.words('english') + list(string.punctuation)))
}

Updated the KFold loop to create all the vectors once at the start for consistent vocabulary initialization
Trained 3 NN classifier models with varying vocabulary sizes (25k, 50k, 100k)

Running MLflow

Per the Documentation page on Notion, run MLflow with the following commands:

# in models/
mlflow server --backend-store-uri sqlite:///mlruns.db

Running Classifier UI
# in models/
python binary_spam_classifier_UI.py

closes #13

When I tested on binary_spam_classifier_experiments.ipynb, it showed the following message under logreg with count_vectorizer setting:

ConvergenceWarning: lbfgs failed to converge (status=1): STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Also, how long does it take to train the neural networks? I tried to run the program for 15 min, and it still stuck at the 1st fold.


Make @push and @stack methods works

There are some Laravel Blade template's methods that don't work yet.
The stack method:
@stack('scripts')

And the push method as well:
@push('scripts')
    <script src="path/to/script.js"></script>
@endpush

The feature added


New methods MinOrDefault and MaxOrDefault

When you start playing with LINQ queries over sequences of elements (e.g. getting min / max value for enumerable source) sooner or later you will come across this one -- the InvalidOperationException (“Sequence contains no elements”).
The problem occurs as by default queries like IEnumerable<T>.Min(…) and IEnumerable<T>.Max(…) do not play nicely if you try to execute them on an empty sequence and just throw the exception described above. Unfortunately these methods do not have a corresponding counterpart like Single(…) /SingleOrDefault(…) that is smart enough to query the sequence if it is not empty or alternatively use the default value without raising an exception.
Basically you got two options now:

Either perform the check on the enumerable sequence every time you are querying it
OR integrate the logic in an extension method.

The second approach is much preferable so let’s add the missing link.
http://blogs.telerik.com/manoldonev/posts/08-10-17/linq_sequence_contains_no_elements_extension_methods_to_the_rescue.aspx

Originally reported on Google Code with ID 28
Reported by anton.georgiev on 2009-09-17 08:56:23

Isn't the idiomatic way to just prepend a DefaultIfEmpty(def) before invoking Max or Min ?
I don't think new extension methods are required here.

@fsateler whole point of linq is to reduce boilerplate code. these methods are useful and I feel its missing here...

@MkazemAkhgary That boilerplate code is precisely embodied in two methods: DefaultIfEmpty + Min/Max. It's just not in one method and that's a reasonable trade-off considering the additional code (almost per aggregate) that would have be written, tested, documented and maintained for very little gain in the end.

at boilerplate code is precisely embodied in two methods: DefaultIfEmpty + Min/Max

Its like saying Cast or OfType is embodied in Select so we don't need those.
GroupBy has 8 overloads. 2 of them can be covered via Select, why its there?
And could you explain how for example AggregateRight  is more useful than Min/Max? its strange that this library has methods for edge case usages and refuses to include more useful methods.
Imagine you had sequence of objects (reference types), and you want to get maximum of some number. here is the boilerplate code.
var max = source.DefaultIfEmpty(null).Max(obj => obj?.foo ?? 0);

vs
var max = source.MaxOrDefault(x => x.foo);

I think it definitely worth the effort, because its just two methods and saves bunch of characters, makes it more readable. its not hard to implement and document either.

var max = source.DefaultIfEmpty(null).Max(obj => obj?.foo ?? 0);

can be re-written more simply as:
var max = source.Select(obj => obj.foo).DefaultIfEmpty().Max();

Its like saying Cast or OfType is embodied in Select so we don't need those.

Cast has a slightly different raison d'être than just as a mere helper. It's used in the query syntax pattern when the source is a non-generic collection, as in from int x in xs. This translates to xs.Cast<int>().
In a similar vein, OfType really exists for dealing with derived types in IQueryable<> sources and is there on IEnumerable<> for parity. LINQ to SQL used this, for example, to support an inheritance/type hierarchy mapped over a single table.

GroupBy has 8 overloads. 2 of them can be covered via Select, why its there?

Do you mean for the result projection?

its strange that this library has methods for edge case usages…

What's an edge case to one could be common to another so it's subjective. In fact, it goes to show that MoreLINQ has had contributions that may seem uncommon and we debate & challenge this all the time; in fact, @leandromoh did so as recently as just this week!

…and refuses to include more useful methods.

No one is refusing. I hope you see that we are discussing here and I am listening and responding. The decision to include or exclude any proposal is not a simple and clear checklist. It's based on a number of factors that I admit (sigh) we should be documenting, at the very least those that can be made simple and clear.

I also would like to have those methods MinOrDefault and MaxOrDefault.


Using $parent-selector variable doesn't fully restrict styles to specified selector.

Describe the issue
There seem to be styles that do not have the value of $parent-selector prepended to them. E.g., box-sizing: border-box; is being applied to everything rather than just those elements that are descendants of the parent selector.
Current Behavior
Some elements outside the parent selector are altered by PicoCSS styles.
Expected Behavior
All PicoCSS styles should be applied within the parent selector only.

I've noticed the same here: https://github.com/picocss/pico/blob/6dc6489e69cc1f22daa938ebd38c02400032428c/scss/components/_nav.scss#L13-L16
Happy to make a little PR for this but I'd like confirmation that it's a bug rather than some deep underlying reason that I'm not seeing.


The validUntil and ID attributes are being removed by "retopped" entities, so SmeeView can't use them!

Fixed by 39beee98fe77a692de9c208c260137fe612cfb85 (well, apart from the ID, which I don't care about at present)


No background replication with V1.3.1 on Android

Avoid duplicates

[ ] Bug is not mentioned in the FAQ
[ ] Bug is specific for Android only, for general issues / questions that apply to iOS and Android please raise them in the documentation repository
[ ] Bug is not already reported in another issue

Describe the bug
Error message "Ursache 3" (39508) is coming up each time the replication is manually started.
Automatically even nothing happens. Issue seems to be connected with being mobile, before that staying in a WiFi @ home the app did run ok.

Expected behaviour

Automatic background replication
Steps to reproduce the issue

Technical details

Mobile device: SM-A300FU
Android version:6.0.1

Possible Fix

Additional context

This problem is mentioned in the FAQs and you can follow it in the open issue CAUSE: 3 Something went wrong. ApiException: 39508 https://github.com/corona-warn-app/cwa-app-android/issues/1021.

Hi @tynfr ,
could you please elaborate:

what is your mobile provider (we had seen issues in the past)
what is your type of mobile connection: edge? 3g? 4g? Ltr?

Hello @tynfr ,
Ursache 3 (59508) is a known issue and should be discussed in https://github.com/corona-warn-app/cwa-app-android/issues/1021 as @MikeMcC399 already pointed out. You can also have a look at the FAQ entry for workarounds and further instructions. If you have more questions about the bug or want to answer @vaubaehn 's questions, please continue to post in #1021, so we can better track the bug there. I'll close this issue as a duplicate.
Regards,
CH

Corona-Warn-App Open Source Team

Duplicate of #1021


Feature Request: Diagnostic Names Should Be Parameters

Scenario
When releasing between environments each of the App Insight instances that are linked should be named correctly. However if you have web-dev-ai in development when pushing to preprod it will create a second app insights instance (still pointing to preprod mind) but with the Development name.
Proposal
Make the name of the app insights logger a parameter.
Describe alternatives you've considered
No response
Additional context
No response
Contact Details
No response

Which tool are you using?  What is the command line, etc. you are using?

When using the extractor tool to create ARMs, it is not creating a logger template or moving the diagnostics (app insights, azuremonitor) to a parameter, even if the APIM has an association with app insights or azure monitor.

Which tool are you using?  What is the command line, etc. you are using?

I missed this.
Using the extractor tool in powershell.

I need the precise command you ran and the configuration file you used.

Reading this, I don't think we have enough information to set up a repro and see what is going on.  I can't tell how you are setting up your preprod and dev environments and what you are expecting to happen that isn't.  Can we get more information about the scenario please?

Closing for lack of info.


dotnet list package --vulnerable --include-transitive

Followup to #4882 + #4881
Sadly enough there is no exit-code: https://github.com/NuGet/Home/issues/11315 + https://github.com/dotnet/sdk/issues/16852 (yet)

Guess this one could be merged?


pdnsmanager and letsencrypt setup

I want to use pdnsmanager for letsencrypt new domain certificate generation by dns based challenge after installing pdnsmanegr now I am at part of letsencrypt and the part which explains to add domain to json and pem certificate to domain using pdnsmanager interface . what I am confused about is that which domain is this part suggesting is it for each new domains I want to add to dns and for that each new domain I need to create txt record with pem certificate as key .
would appreciate some info on this part mentioned below related to domain .
change to the directory of pdns-client, generate a keypair and output the public key:
cd pdns-client
./pdns-keygen
cat pdns.public.pem
Copy the generated public key to your clipboard.
In the next step, open a browser and login to your PDNS Manager instance. Add a record to your domain with the name _acme-challenge., type TXT and content none. Use a Priority of 0 and a TTL of 60.
Afterwards, click on to share icon which is the last icon in the row of the record. Click on Add Key. Enter a description like ACME and paste the public key from your clipboard into the field. Confirm with Add. You now need to remember the ID of the permission you have added, which is displayed in the table on the left.
Configure pdns-acme
Change to the directory of pdns-acme and copy the example config.
cd ../pdns-acme
cp pdns-acme.json.example pdns-acme.json
Open the file pdns-acme.json with an editor of your choice. In the config section adjust the path of your PDNS Manager installation and also the deploy-wait value. The deploy-wait parameter determines how long the script should wait for the DNS servers to get the right results. This value depends on your nameserver setup. The default of 300 should do well for most setups. After these changes, the section looks like that:
"config": {
"server": "https:///",
"pdns-client": "../pdns-client/pdns-client",
"deploy-wait": 300
}

I don't fully understand your question, but as far as I understand it is regarding the Public Key authentication against the API with pdns-client?
So what you do is the following, assuming your frontend is at https:://dns.example.com/:
The pdns-acme.json-config looks like this:
"config": {   "server": "https://dns.example.com/",   "pdns-client": "../pdns-client/pdns-client",   "deploy-wait": 300 }
Assume you want to get certificates for test.example.com and mail.example.com, you create the records:

_acme-challenge.test.example.com, type TXT, TTL 60
_acme-challenge.mail.example.com, type TXT, TTL 60

For both of those records, you add the Certificate you generated in pdns-cleint (pdns.public.pem) via "Add key".
You do not need multiple keys for pdns-client, but you need to add the key of the specific instance to all records that it should update.


”Bogdand”, ”Babța”, ”Corund”, ”Ser” are listed in the drop-down menu

Description:
When user writes ”Bogdand”, ”Babța”, ”Corund”, ”Ser” the locations appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Bogdand"

Expected results
The location appears in the drop-down menu as "Sat Bogdand, SATU MARE (Bogdand)".

Step 3
Write in the search bar "Babța"

Expected results
The location appears in the drop-down menu as "Sat Babța, SATU MARE (Bogdand)".

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Corund"

Expected results
The location appears in the drop-down menu as "Sat Corund, SATU MARE (Bogdand)".

Step 6
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 7
Write in the search bar "Ser"

Expected results
The location appears in the drop-down menu as "Sat Ser, SATU MARE (Bogdand)".


Searching does not extend into and across hyperlinks

newsboat r2.12-171-g410a-dirty - https://newsboat.org/
Copyright (C) 2006-2015 Andreas Krennmair
Copyright (C) 2015-2018 Alexander Batischev
Copyright (C) 2006-2017 Newsbeuter contributors
Copyright (C) 2017-2018 Newsboat contributors
Newsboat is free software licensed under the MIT License. (Type `newsboat -vv' to see the full text.)
It bundles JSON for Modern C++ library, licensed under the MIT License: https://github.com/nlohmann/json
It bundles an alphanum algorithm implementation licensed under the MIT license: http://www.davekoelle.com/alphanum.html
newsboat r2.12-171-g410a-dirty
System: Linux 4.18.13_1 (x86_64)
Compiler: g++ 8.2.0
ncurses: ncurses 6.1.20180127 (compiled with 6.1)
libcurl: libcurl/7.61.1 LibreSSL/2.7.4 zlib/1.2.11 libssh2/1.8.0 nghttp2/1.34.0 (compiled with 7.61.1)
SQLite: 3.25.2 (compiled with 3.25.1)
libxml2: compiled with 2.9.8
Steps to reproduce the issue:

add http://rss.slashdot.org/Slashdot/slashdot to urls list (or any rss feed with urls attached.

Go to any link

search for image 2\]\[2\]\[image

build an enterprise version of their tools (and the sales force that goes with
that) to monetize their efforts.
[image 2][2][image 4][4][image 6][6]

(nothing should be highlighted)
This bug also appears for hyperlinks that appear in the body of the text such as various articles from http://lukesmith.xyz/rss.xml

Reproduced with e88fd99b3.
We probably run RegexManager::quote_and_highlight() too early, earlier than "numbers in square brackets" are added to links. But that's counter-intuitive to me—I thought the numbers are added by HTMLRenderer, and I'm pretty sure we call quote_and_highlight on an already-rendered text. =\
Thanks for a report, @zaowen!


Proposal: MultiOwnable Smart Contract

A variation of the Ownable.sol contract.
I'd like to see a Multi Ownership Smart Contract that utilizes mapping as part of the OZ project.
I have created an implementation that may be of interest... And can send a pull request for review if open to a file such as MultiOwnable.sol. in /contracts/ownership

Hi @ChrisCates! Do you mean you'd like multiple accounts to have a role? If so, we've introduced that same thing in the 2.0 release candidate, check it out!

@nventuro the Roles.sol and it's respective files are for specific permissions.
The Ownable.sol contract acts almost as an administrative role for all account functionality.
Also, I feel the structure of the Roles.sol code could be simplified greatly as there are some redundancies in terms of functionality.
In fact, this may a good opportunity to restructure to have an admin role for verbosity purposes.
Are you open to reviewing a pull request with a proposal refactor in the v2.0.0 release?

The Ownable.sol contract acts almost as an administrative role for all account functionality.

That is precisely why we've decided to remove it in 2.0, and move towards more granular permissions (#366).
Which redundancies worry you? The distinct roles under the roles directory are, granted, the exact same contract in essence: they exist only due to limitations of Solidity, and will be auto generated (#1290) in the future.

@nventuro Ahh I see.
It seems a lot of other files depend on it.
Ah, so they will have auto-generated stubs in the future...
In that case I don't have any concerns.
Thanks for the clarifications.


CPP DBG Optimization: don't request Call Stack or Variables if window pane is hidden

Hi, I work on a fairly large repository and I notice that using GDB (cppdbg) with vscode always requests Call Stack and Variables information, which makes stepping through each line very slow. Can the cppdbg plugin not ask gdb for the Call Stack and Variables on every step if I minimize those window panes in the VSCode UI? (or alternatively have a setting to not eagerly load those)
Thanks

/extC++

It looks like this is caused by the C++ extension. Please file the issue to the C++ extension repository. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!


changefeed get stuck when lightning tidb import and kafka sink

What did you do?

Failed case: [cdc_lightning_1TBSingleTable_kafka]
lightning backend=tidb import , without any chaos, pd/tikv/cdc node is alway healthy.
checkpoint ts don't move after cdc & changefeed working for 2.5 hours:

What did you expect to see?
No response
What did you see instead?

checkpoint ts stopped in "2022-07-26 07:52:33.003", during this time, no any cdc ERROR found.
TiKV min resolved ts is normal, alway < 15s.
data flow:

tikv:

Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)

TiCDC version (execute cdc version):
(paste TiCDC version here)
[2022/07/26 05:14:13.911 +00:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v6.2.0-alpha] [git-hash=659a9a00739d41036a21940aa669b5f8cea59bf7] [git-branch=heads/refs/tags/v6.2.0-alpha] [utc-build-time="2022-07-25 11:03:21"] [go-version="go version go1.18.2 linux/amd64"] [failpoint-build=false]

v6.2.0 can reproduce this issue.
'Tammyxia' via Tools QA @.***> 于2022年7月27日周三 10:42写道：

What did you do?

Failed case: [cdc_lightning_1TBSingleTable_kafka]
lightning backend=tidb import , without any chaos, pd/tikv/cdc node
is alway healthy.
checkpoint ts don't move after cdc & changefeed working for 2.5
hours:

[image: image]
https://user-images.githubusercontent.com/78345569/181147823-18800f79-1546-455c-bb4e-88459021aa33.png
What did you expect to see?
No response
What did you see instead?

checkpoint ts stopped in "2022-07-26 07:52:33.003", during this
time, no any cdc ERROR found.
TiKV min resolved ts is normal, alway < 15s.
data flow:

[image: image]
https://user-images.githubusercontent.com/78345569/181148799-b3274414-acde-4738-b309-e28f95672ad0.png
tikv:
[image: image]
https://user-images.githubusercontent.com/78345569/181149020-a19c0f8b-ea04-4695-9742-5a7a86321ac1.png
Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL
client):
(paste TiDB cluster version here)
Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)
TiCDC version (execute cdc version):
(paste TiCDC version here)[2022/07/26 05:14:13.911 +00:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v6.2.0-alpha] [git-hash=659a9a00739d41036a21940aa669b5f8cea59bf7] [git-branch=heads/refs/tags/v6.2.0-alpha] [utc-build-time="2022-07-25 11:03:21"] [go-version="go version go1.18.2 linux/amd64"] [failpoint-build=false]
—
Reply to this email directly, view it on GitHub
https://github.com/pingcap/tiflow/issues/6481, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AT3WYSZJDDZWL7H7Q5I4CKLVWCOZBANCNFSM54X633FA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

There's two other cases also have this issue:

cdc node network loss
tiup cluster long stable test: restart cdc node every 1 hour, after working 12 hours, cdc changefeed get stuck.

/label affects-6.2

There is another case have a similar phenomenon

The logs below show that the kafka is not stable, and cause errors periodically.
[2022/07/26 11:04:23.360 +00:00] [ERROR] [processor.go:808] ["processor receives redundant error"] [error="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close"] [errorVerbose="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:183\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).flushAndNotify\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:248\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).asyncSend\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:192\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:227\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*mqSink).run.func2\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:355\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:23.553 +00:00] [ERROR] [processor.go:724] ["error on running processor"] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [changefeed=cdc-single-table-lightning-kafka] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:23.553 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:23.616 +00:00] [ERROR] [kafka.go:243] ["close async client with error"] [error="kafka: Failed to deliver 590 messages."] [duration=558.238µs] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [role=processor]
[2022/07/26 11:04:23.805 +00:00] [ERROR] [kafka_manager.go:211] ["Kafka admin client list topics failed"] [error="dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [duration=1.799286ms]
[2022/07/26 11:04:23.806 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [errorVerbose="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq.NewKafkaSaramaSink\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:439\ngithub.com/pingcap/tiflow/cdc/sink.init.0.func3\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:127\ngithub.com/pingcap/tiflow/cdc/sink.New\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:167\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).lazyInitImpl\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:666\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:561\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:498\ngithub.com/pingcap/tiflow/cdc/processor.(*managerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/manager.go:117\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:265\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:507\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:361\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:23.853 +00:00] [ERROR] [feed_state_manager.go:393] ["processor report an error"] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [captureID=47d7fa04-e529-4c2b-a8d0-fab9804a0542] [error="{\"addr\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301\",\"code\":\"CDC:ErrKafkaNewSaramaProducer\",\"message\":\"[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\"}"]
[2022/07/26 11:04:23.899 +00:00] [ERROR] [kafka_manager.go:211] ["Kafka admin client list topics failed"] [error="dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [duration=1.535666ms]
[2022/07/26 11:04:23.899 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [errorVerbose="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq.NewKafkaSaramaSink\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:439\ngithub.com/pingcap/tiflow/cdc/sink.init.0.func3\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:127\ngithub.com/pingcap/tiflow/cdc/sink.New\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:167\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).lazyInitImpl\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:666\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:561\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:498\ngithub.com/pingcap/tiflow/cdc/processor.(*managerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/manager.go:117\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:265\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:507\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:361\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:24.003 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [errorVerbose="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq.NewKafkaSaramaSink\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:408\ngithub.com/pingcap/tiflow/cdc/sink.init.0.func3\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:127\ngithub.com/pingcap/tiflow/cdc/sink.New\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:167\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).lazyInitImpl\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:666\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:561\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:498\ngithub.com/pingcap/tiflow/cdc/processor.(*managerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/manager.go:117\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:265\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:507\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:361\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:34.686 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host"] [errorVerbose="[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq.NewKafkaSaramaSink\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:408\ngithub.com/pingcap/tiflow/cdc/sink.init.0.func3\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:127\ngithub.com/pingcap/tiflow/cdc/sink.New\n\tgithub.com/pingcap/tiflow/cdc/sink/sink.go:167\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).lazyInitImpl\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:666\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:561\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:498\ngithub.com/pingcap/tiflow/cdc/processor.(*managerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/manager.go:117\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:265\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:507\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:361\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:04:34.991 +00:00] [ERROR] [feed_state_manager.go:393] ["processor report an error"] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [captureID=47d7fa04-e529-4c2b-a8d0-fab9804a0542] [error="{\"addr\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301\",\"code\":\"CDC:ErrKafkaNewSaramaProducer\",\"message\":\"[CDC:ErrKafkaNewSaramaProducer]new sarama producer: dial tcp: lookup downstream-kafka-2.downstream-kafka-peer.cdc-testbed-tps-1021883-1-361.svc.cluster.local on <IP_ADDRESS>:53: no such host\"}"]
[2022/07/26 11:36:39.650 +00:00] [ERROR] [processor.go:808] ["processor receives redundant error"] [error="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close"] [errorVerbose="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:183\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).flushAndNotify\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:248\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).asyncSend\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:192\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:227\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*mqSink).run.func2\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:355\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:36:39.653 +00:00] [ERROR] [processor.go:724] ["error on running processor"] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [changefeed=cdc-single-table-lightning-kafka] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:36:39.653 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 11:36:39.990 +00:00] [ERROR] [feed_state_manager.go:393] ["processor report an error"] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [captureID=47d7fa04-e529-4c2b-a8d0-fab9804a0542] [error="{\"addr\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301\",\"code\":\"CDC:ErrKafkaAsyncSendMessage\",\"message\":\"[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\"}"]
[2022/07/26 11:36:59.067 +00:00] [ERROR] [kafka.go:243] ["close async client with error"] [error="kafka: Failed to deliver 193 messages."] [duration=19.317471086s] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [role=processor]
[2022/07/26 12:14:36.555 +00:00] [ERROR] [processor.go:808] ["processor receives redundant error"] [error="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close"] [errorVerbose="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:183\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).flushAndNotify\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:248\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).asyncSend\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:192\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:227\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*mqSink).run.func2\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:355\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 12:14:36.754 +00:00] [ERROR] [processor.go:724] ["error on running processor"] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [changefeed=cdc-single-table-lightning-kafka] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 12:14:36.754 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 12:14:36.990 +00:00] [ERROR] [feed_state_manager.go:393] ["processor report an error"] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [captureID=47d7fa04-e529-4c2b-a8d0-fab9804a0542] [error="{\"addr\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301\",\"code\":\"CDC:ErrKafkaAsyncSendMessage\",\"message\":\"[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\"}"]
[2022/07/26 12:14:56.173 +00:00] [ERROR] [kafka.go:243] ["close async client with error"] [error="kafka: Failed to deliver 193 messages."] [duration=19.309584046s] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [role=processor]
[2022/07/26 13:14:49.761 +00:00] [ERROR] [processor.go:808] ["processor receives redundant error"] [error="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close"] [errorVerbose="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:183\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).flushAndNotify\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:248\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).asyncSend\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:192\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:227\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*mqSink).run.func2\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:355\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 13:14:49.952 +00:00] [ERROR] [processor.go:724] ["error on running processor"] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [changefeed=cdc-single-table-lightning-kafka] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 13:14:49.952 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 13:14:50.391 +00:00] [ERROR] [feed_state_manager.go:393] ["processor report an error"] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [captureID=47d7fa04-e529-4c2b-a8d0-fab9804a0542] [error="{\"addr\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301\",\"code\":\"CDC:ErrKafkaAsyncSendMessage\",\"message\":\"[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\"}"]
[2022/07/26 13:15:09.425 +00:00] [ERROR] [kafka.go:243] ["close async client with error"] [error="kafka: Failed to deliver 193 messages."] [duration=19.311982894s] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [role=processor]
[2022/07/26 14:38:50.395 +00:00] [ERROR] [processor.go:808] ["processor receives redundant error"] [error="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close"] [errorVerbose="[CDC:ErrKafkaFlushUnfinished]flush not finished before producer close\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:183\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).flushAndNotify\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:248\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).asyncSend\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:192\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*flushWorker).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq_flush_worker.go:227\ngithub.com/pingcap/tiflow/cdc/sink/mq.(*mqSink).run.func2\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/mq.go:355\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 14:38:50.552 +00:00] [ERROR] [processor.go:724] ["error on running processor"] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [changefeed=cdc-single-table-lightning-kafka] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 14:38:50.552 +00:00] [ERROR] [processor.go:542] ["run processor failed"] [changefeed=cdc-single-table-lightning-kafka] [capture=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."] [errorVerbose="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:32\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.(*kafkaSaramaProducer).run\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:310\ngithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka.NewKafkaSaramaProducer.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/mq/producer/kafka/kafka.go:367\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/07/26 14:38:50.797 +00:00] [ERROR] [feed_state_manager.go:393] ["processor report an error"] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [captureID=47d7fa04-e529-4c2b-a8d0-fab9804a0542] [error="{\"addr\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-1021883-1-361.svc:8301\",\"code\":\"CDC:ErrKafkaAsyncSendMessage\",\"message\":\"[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic cdc-canal-json-1TBSingleTable: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.\"}"]
[2022/07/26 14:39:10.156 +00:00] [ERROR] [kafka.go:243] ["close async client with error"] [error="kafka: Failed to deliver 193 messages."] [duration=19.360403889s] [namespace=default] [changefeed=cdc-single-table-lightning-kafka] [role=processor]

This may be caused by the downstream Kafka crashed during the test case running process.

close the issue, since it's caused by the downtream kafka crashed periodically, the changefeed cannot make progress in time, then cause high replication latency, which is meet the expectation at the moment.


Blog page

The following change has been implemented:

Blog page
Post
Post Category
types
Navbar
Footer

Hi @pierrelstan I will proceed to review but the deployment is failing, is that normal at this point?

Hi @pierrelstan I will proceed to review but the deployment is failing, is that normal at this point?
And please make sure to link the issue and the project with this PR so that it automatically closes once merged.

Hi @didierganthier, I think I found the problem. it should deploy successfully.


Hover animation added on all pages with responsive icons.

Issue #333 [SOLVED]
https://github.com/user-attachments/assets/1b2d7885-1d37-447f-9629-24a363ac52af
Since I have updated most of the things, thus I am only adding the video of the work done.

in recording, home page has two newsletters
refer this:
https://mansiruhil13.github.io/Bobble-AI/


fix: SendUnreliable respects ArraySegment.Offset

random find while browsing the source

thanks, nice find.
gonna keep it open until I have time to add a test :)


Refactor AbstractTableConfig (No functionality change)

Use static ObjectMapper
Use new code style
Move IndexConfig into AbstractTableConfig
Store TableType instead of String for table type

Codecov Report

Merging #1445 into master will increase coverage by 0.01%.
The diff coverage is 79.51%.

@@            Coverage Diff             @@
##           master    #1445      +/-   ##
==========================================
+ Coverage   67.92%   67.93%   +0.01%
==========================================
  Files         785      785
  Lines       38166    38141      -25
  Branches     4970     4969       -1
==========================================
- Hits        25925    25912      -13
+ Misses      10553    10549       -4
+ Partials     1688     1680       -8

Impacted Files
Coverage Δ

...pi/restlet/resources/PinotTableSegmentConfigs.java
0% <ø> (ø)
:arrow_up:

...i/restlet/resources/PinotTableIndexingConfigs.java
0% <0%> (ø)
:arrow_up:

...i/restlet/resources/PinotTableMetadataConfigs.java
0% <0%> (ø)
:arrow_up:

...i/restlet/resources/PinotTableRestletResource.java
40.42% <100%> (ø)
:arrow_up:

...kedin/pinot/common/config/RealtimeTableConfig.java
100% <100%> (+55.55%)
:arrow_up:

...nkedin/pinot/common/config/OfflineTableConfig.java
100% <100%> (+55.55%)
:arrow_up:

...ntroller/helix/core/PinotHelixResourceManager.java
58.58% <100%> (+0.08%)
:arrow_up:

...kedin/pinot/common/config/AbstractTableConfig.java
77.52% <80.26%> (+1.23%)
:arrow_up:

...pinot/core/operator/docidsets/OrBlockDocIdSet.java
87.67% <0%> (-10.96%)
:arrow_down:

...com/linkedin/pinot/transport/common/SegmentId.java
34.78% <0%> (-8.7%)
:arrow_down:

... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e649b8e...588c6f0. Read the comment docs.

@mayankshriv In the future we might want to differentiate offline/realtime table configs, so I will keep them there for now. Also, it not a small change to remove them.


videoview title

Cannot set title on embedded player in layout using videoview

call videoview#setTitle

videoview#setTitle doesn't work @UbuntuLover

videoView.videoInfo(videoInfo).getPlayer().start(); this works


OSError: Can't load the configuration of 'sentence-transformers/all-MiniLM-L6-v2'

i pulled the newest code today, when i excute "python devika.py", i got this error
24.04.01 20:16:34: root: INFO   : Initializing Devika...
24.04.01 20:16:34: root: INFO   : Initializing Prerequisites Jobs...
24.04.01 20:16:37: root: INFO   : Loading sentence-transformer BERT models...
No sentence-transformers model found with name sentence-transformers/all-MiniLM-L6-v2. Creating a new one with MEAN pooling.
Traceback (most recent call last):
File "~/devika/.venv/lib/python3.9/site-packages/transformers/configuration_utils.py", line 686, in _get_config_dict
resolved_config_file = cached_file(
File "~/devika/.venv/lib/python3.9/site-packages/transformers/utils/hub.py", line 398, in cached_file
resolved_file = hf_hub_download(
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1261, in hf_hub_download
metadata = get_hf_file_metadata(
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1674, in get_hf_file_metadata
r = _request_wrapper(
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 369, in _request_wrapper
response = _request_wrapper(
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 392, in _request_wrapper
response = get_session().request(method=method, url=url, **params)
File "~/devika/.venv/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/huggingface_hub/utils/_http.py", line 68, in send
return super().send(request, *args, **kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "~/devika/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 779, in urlopen
self._prepare_proxy(conn)
File "~/devika/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1048, in _prepare_proxy
conn.connect()
File "~/devika/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 616, in connect
self.sock = sock = self._new_conn()
File "~/devika/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "~/devika/.venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
File "~/devika/.venv/lib/python3.9/site-packages/eventlet/greenio/base.py", line 261, in connect
self._trampoline(fd, write=True, timeout=timeout, timeout_exc=_timeout_exc)
File "~/devika/.venv/lib/python3.9/site-packages/eventlet/greenio/base.py", line 206, in _trampoline
return trampoline(fd, read=read, write=write, timeout=timeout,
File "~/devika/.venv/lib/python3.9/site-packages/eventlet/hubs/init.py", line 155, in trampoline
listener = hub.add(hub.WRITE, fileno, current.switch, current.throw, mark_as_closed)
File "~/devika/.venv/lib/python3.9/site-packages/eventlet/hubs/kqueue.py", line 52, in add
self._control([event], 0, 0)
File "~/devika/.venv/lib/python3.9/site-packages/eventlet/hubs/kqueue.py", line 38, in _control
return self.kqueue.control(events, max_events, timeout)
TypeError: changelist must be an iterable of select.kevent objects
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "~/devika/devika.py", line 8, in
init_devika()
File "~/devika/src/init.py", line 28, in init_devika
SentenceBert(prompt).extract_keywords()
File "~/devika/src/bert/sentence.py", line 6, in init
self.kw_model = KeyBERT()
File "~/devika/.venv/lib/python3.9/site-packages/keybert/_model.py", line 57, in init
self.model = select_backend(model)
File "~/devika/.venv/lib/python3.9/site-packages/keybert/backend/_utils.py", line 48, in select_backend
return SentenceTransformerBackend(embedding_model)
File "~/devika/.venv/lib/python3.9/site-packages/keybert/backend/_sentencetransformers.py", line 42, in init
self.embedding_model = SentenceTransformer(embedding_model)
File "~/devika/.venv/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 199, in init
modules = self._load_auto_model(
File "~/devika/.venv/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 1134, in _load_auto_model
transformer_model = Transformer(
File "~/devika/.venv/lib/python3.9/site-packages/sentence_transformers/models/Transformer.py", line 35, in init
config = AutoConfig.from_pretrained(model_name_or_path, **model_args, cache_dir=cache_dir)
File "~/devika/.venv/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 1138, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/transformers/configuration_utils.py", line 631, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
File "~/devika/.venv/lib/python3.9/site-packages/transformers/configuration_utils.py", line 707, in _get_config_dict
raise EnvironmentError(
OSError: Can't load the configuration of 'sentence-transformers/all-MiniLM-L6-v2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'sentence-transformers/all-MiniLM-L6-v2' is the correct path to a directory containing a config.json file
befor i pulled the code, this project is functional, how can i resove this problem? btw, the fronend service is functional, i can browser in chrome, network seems no problem.

I met same problem. and solved it.
I found this stackoverflow.
https://stackoverflow.com/questions/78001556/error-while-installing-sentence-transformers

You need to use python 3.11 to install sentence-transformers

Try "python3 devika.py".
If same error exists, upgrade python3, and go through the installation process on README step by step.

upgrade my python version to 3.12 and resolved this issue.thanks for replay.

help i have this issues as well, i tried upgrading to python 3.12 but that didn't work, i also tried the python3 but it didn't work it says python3 is not recognized


Fixing Calendar issues related to DST and event organizer timezone being different than user timezone.

Does the pull request solve a related issue?
Yes

If so, can you reference the issue?
#2085

What does the pull request accomplish? Use a list if needed.

Change to use node-ical instead of ical for parsing ics files
Fixing issues because of DST offsets

If it includes major visual changes please add screenshots.

Warnings

:warning:

Please include an updated CHANGELOG.md file.This way we can keep track of all the contributions.

  Generated by :no_entry_sign: dangerJS

PTAL @rejas

@rejas, did you (by any chance) test this?

@rejas, did you (by any chance) test this?

no, just took a quick look at the code.

@MichMich - any update on whether this change can be merged or not?

Sorry for the slow response. This PR has some conflicts I can't fix. Would it be possible four you to fix these conflicts?

I will take a look this weekend.

with my all day changes (already merged) and the updated node-ical node library, we don't need this fix. the problem was in ical library..
I am waiting for the node-ical author to publish to node library before submitting pr
a sample using ical.js library is listed at the end of #1798
its expected that ical.js will reject this change (also submitted) due to requirement to be in browser only

Ok, so just to be sure: I can close this PR?

u should let him confirm, but I believe yes

It's ok to close this PR if the fixes are already in. I had a separate DST-related fix as well, but will see if it's address based on the other ical fixes - if not I'll submit a new PR.

I also want to move to node-ical, as ical.js is browser only. I put all the timezone fixes in the node-ical module, where they should be.
I also added support for ms timezone names and offset timezones
moment-tz does all the DST adjustment, so mm doesn't have to

Ok. Closing for now. Thanks for all your hard work, people!


fix check prepare&open before every VExpr execute

Proposed changes
Issue Number: close #xxx
there are many bugs. use DCHECK to find them first.


Hugo 0.124 and non basic alphanum in taxonomy terms, custom page directory name

Just tried today's freshest version of theme (pulled the repo) with Hugo 0.124 instead of 0.123. Using Windows.
It works fine but I have issues again with taxonomy term custom page issue, and fixed it because it's just a matter of logic.
My custom term can be "eté" or "ombres et lumières" or "aimé (amant)". With accents or other symbols.
Now custom page should reside in a folder whose name contains this accent and symbols, unlike with Hugo 0.122 where the cleaned term worked.
So "eté", "ombres-et-lumières" and "aimé-(amant)". Not anymore "ete", "ombres-et-lumieres" and "aime-amant".
Not intuitive and not user-friendly. Maybe this change deserves a side note in theme's documentation ?

Well, this is basically a Hugo issue and in flux since 0.123. They went thru different iterations related to accents and capitalization and I lost a bit track of the current state.
I guess the best for you is to stay with 0.122 for a while until the dust has settled. If you are interested you can take a look into Hugo's discussion forum or the repo issues and (maybe?) their relese notes.

Argh need to update theme again, layout is broken on mobile with menu left open or maybe the content custom width causes issue


0.11.x -> 0.12.x - Difference in date handling

We've observed some differences in how dates are displaying in`0.11.x and 0.12.x. For example, with the code below, the tooltip/date axis will display the first date as 2020/30/03 in 0.11.x, and 2020/29/03 in 0.12.x:
import bqplot as bqp
import pandas as pd

x_data = pd.date_range(start='2020-03-30', end='2020-03-31')

scale_y = bqp.LinearScale()
scale_x = bqp.DateScale()

mscat = bqp.Scatter(
    x=x_data,
    y=[1, 2],
    scales={'x': scale_x, 'y': scale_y},
    tooltip=bqp.Tooltip(fields=[ 'y', 'x'],formats=['', '%m/%d/%Y']))

axis_x = bqp.Axis(scale=scale_x)
axis_y = bqp.Axis(scale=scale_y, orientation='vertical')

bqp.Figure(
    marks=[mscat],
    layout={'width':'500px', 'height':'300px'},
    axes=[axis_x, axis_y])

0.11.x

0.12.x

@SylvainCorlay @maartenbreddels for awareness

So I am not seeing it locally, probably because my time zone is closer to UTC. Looking into it more.

OK, changing my time zone to PST shows the issue.

#967 seems to fix a few issues...perhaps it is worth having a look...

Yes, thanks @acelere - I was just also playing with utcFormat. I think this is the appropriate fix.

OK, this is fixed in master. Closing.


[QUERY] Generating sas for Azure IoT hub blob storage doesn't work.

Library name and version
Azure.Storage.Blobs 12.13.0
Query/Question
I'm using BlobSasBuilder from the Azure.Storage.Blob nuget package version 12.13.0 to generate blob read sas.
This works fine with blobs stored on Azure, but fails with "Signature did not match" for blobs hosted on an on-premises machine using Azure Blob Storage on IoT Edge
Interestingly, the SAS generated by Azure Storage Explorer works, and I notice that the service version (sv) in the query string is much earlier  ("2017-04-17") than that generated by v 12.13.0 of the library. Apart from the version and the signature, the SAS query strings are identical.
Any ideas?
Environment
No response

Hello @geometrikal We are looking into this!

@SatishBoddu-MSFT
We have an ASP.Net Core web api (deployed locally or dev machine or remote to Azure app services) that the client retrieves the sas key information from, and then the client downloads the blob.
This is the code
                var blobSasBuilder = new BlobSasBuilder()
                {
                    BlobContainerName = containerName,
                    BlobName = filePath,
                    Resource = "b",
                    StartsOn = DateTimeOffset.UtcNow.AddMinutes(-5),
                    ExpiresOn = DateTimeOffset.UtcNow.AddMinutes(5)
                };
                blobSasBuilder.SetPermissions(BlobSasPermissions.Read);
                var storageSharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey);
                var sas = blobSasBuilder.ToSasQueryParameters(storageSharedKeyCredential).ToString();
                ```


[FEATURE] add stop button

Description
We have a button to start searching for someone to talk to, but we don't have one for when someone wants to stop searching.
Tasks

[ ] add stop button, should have the same styles as the stop button, but smaller
[ ] when clicked should stop searching for a a user

Hi @Dun-sin , can I take this up?

Hi @Dun-sin , can I take this up?

Definitely, thanks for wanting to contribute, Make sure you follow the rules 👉🏽here👈🏽, else your PR will not be accepted and will be closed.
Good Luck

@ays14 an update on this please?

@ays14 an update on this please?

Hi @Dun-sin , I am working on it. Will finish this over the weekend.

@ays14 an update on this please?

Hi @Dun-sin , I am working on it. Will finish this over the weekend.

@ays14 any updates on this?


Fix error in github_finalize if no release exists

The script now looks for a release before continuing with the finalization steps.
Change-type: patch

@balena-ci I self certify!


Feature request: specify default move from a move list

Given a framework that forces a player to act, such as this FBG feature request related to time limits, a game needs to have a way to specify what the default action for a phase or stage is. This would often be some sort of "pass" action. My proposal here is that anywhere a move or move list can be specified, also accept a property specifying either a default move or a function with some logic for selecting a default move for a specific player.

Wouldn't it be better for the thing that “forces” action to say what it forces?
For example, in a case where a timeout does something, I would imagine it would be nice to have an onTimeout called which would advance game state rather than relying on a reference to a specific default move.
Are there use cases where this wouldn't be a good developer experience?

My thought is that there might be multiple use cases in the future for some universal default move for a particular game state, such as:

player turn times out
player chooses "do my default move in every phase/stage until I turn this option back off" to go AFK or similar
player has quit the game, but the game rules do not allow players to drop out, so their position needs to perform a move at various future stages
AI fails to choose a move
game admin intentionally triggers a player's default move without having to figure out what that move should be
... ?? Something I haven't thought of yet.

While each of these could have their own handlers, I expect they would be relatively consistent across most game states, and so it would reduce duplication and effort if there was a single place to specify this move, and then only use event-specific handlers when one of those events needs to do something exceptional.

@sparr Thanks for the extra use case examples!
Would a pattern like the following work:
function takeActionOne(G, ctx) {}
function takeActionTwo(G, ctx) {}

const game = {
  moves: {
    default: takeActionOne,
    takeActionOne,
    takeActionTwo,
  }

  phases: {
    A: {
      moves: {
        default: takeActionTwo,
        takeActionOne,
        takeActionTwo,
      }
    }
  }
}

For game implementers, the main difference between the default move and any other is presumably a reliable thing to call. Defining a default move like this works already, although you do need to remember to include default everywhere, but that would presumably also be the case for a different API.
I think this would cover all your use cases above, do you agree? Using this approach might elevate the default move name to a special status once timers or another core part of the framework started relying on it, but for external uses (AFK, admin, other UI triggered calls) this seems like a good pattern to me.

This seems good. I suspect in many cases default would be a lambda that calls an existing action with specific parameters, rather than pointing directly at the action itself, but that still fits with your approach here.


[Bug] Compilation error: when using enable-sync-stack: true

@minnieliu reported this compilation error when using the enable-sync-stack: true flag on communication-rooms sdk.
Noticing that the generate sync API expects these two parameters https://github.com/minnieliu/azure-sdk-for-java/blob/d66ad21028e650de5fe49651b5fceebc6d9dd9fb/sdk/communication/azure-communication-rooms/src/main/java/com/azure/communication/rooms/implementation/RoomsImpl.java#L286-L287
That are not defined in the method body.
But as far the async version they should be defined in the method body like such
https://github.com/minnieliu/azure-sdk-for-java/blob/d66ad21028e650de5fe49651b5fceebc6d9dd9fb/sdk/communication/azure-communication-rooms/src/main/java/com/azure/communication/rooms/implementation/RoomsImpl.java#L200-L201

@minnieliu
What is the version of autorest-java?
The one in readme is too old
https://github.com/minnieliu/azure-sdk-for-java/blob/d66ad21028e650de5fe49651b5fceebc6d9dd9fb/sdk/communication/azure-communication-rooms/swagger/README.md
Latest be<EMAIL_ADDRESS>

@weidongxu-microsoft thanks, upgrading to the latest resolved the issue! Feel free to close the bug.


Keyboard navigation for dropdowns

The current JavaScript implementation of dropdowns doesn't support keyboard navigation. Adding some event handling for keyboard events would go a long way towards aiding in 508 compliance and making power-users happy.
I hacked out a quick set of code that works for my purposes, but don't have time to turn it into a polished pull request right now. Perhaps someone else can take it the final mile:
  $('.dropdown').bind('keydown', function (evt) {
    var $this = $(this);
    switch(evt.keyCode) {
      case 13: // Enter key
      case 32: // Space bar
      case 38: // Up arrow
      case 40: // Down arrow
        $this.addClass("open");
        $this.find('.dropdown-menu a:first').focus();
        break;
      case 27: // Escape key
        $this.removeClass("open");
        $this.focus();
        break;
    }
  });

  $('.dropdown-menu a').bind('keydown', function (evt) {
    var $this = $(this);

    function select_previous () {
      $this.parent('li').prev().find('a').focus();
      evt.stopPropagation();
    }

    function select_next () {
      $this.parent('li').next().find('a').focus();
      evt.stopPropagation();
    }

    switch(evt.keyCode) {
      case 13: // Enter key
      case 32: // Space bar
        $this.click();
        $this.closest('.dropdown').removeClass('open');
        evt.stopPropagation();
        break;
      case 9: // Tab key
        if (evt.shiftKey) {
          select_previous();
        }
        else {
          select_next();
        }
        evt.preventDefault();
        break;
      case 38: // Up arrow
        select_previous();
        break;
      case 40: // Down arrow
        select_next();
        break;
      }
  });

Let's see what @fat thinks about this. I know we'll need keyboard nav for autocomplete, but I'm unsure about how best to do it for regular ol' dropdowns, if at all.

i'm going to close this in favor of this ticket i just created: https://github.com/twitter/bootstrap/issues/931 - if you have any objections, feel free to raise them :)

In general more specific tickets are easier to deal with than an abstract ticket like #931, but if you've got an idea for how to implement that one, go for it!

Yeah i hear you, but i think at this point accessibility is something we need to think about across all the plugins

@gabrielhurley I know this is ions ago, but wondering what your 'open' class looked like? As my 'display: block' is not firing off


[C++][Parquet] Crash at parquet::TypedColumnWriterImpl<>::WriteBatchSpaced

I got this crash from our customer machine: parquet_crash_WriteBatchSpaced.txt , it crashes continuously every time my application opens a parquet file and write the first row group. It happens only on one machine and runs fine on others until now. I cannot reproduce it.
I access the parquet file on only one thread, so you can eliminate the case of race condition. Do you have any idea about this crash report?
Reporter: Tham / @thamht4190
Original Issue Attachments:

parquet_crash_WriteBatchSpaced.txt

Note: This issue was originally created as PARQUET-1995. Please see the migration documentation for further details.

Micah Kornfield / @emkornfield:
Well it seems like a bug someplace. I'm not sure if we have a random generated data test that covers something like this, we can try adding one.  I think a lot has changed between 0.17.1 and 3.0.0 so if possible trying to upgrade if it is possible would be helpful to see if the bug has been patched.

Micah Kornfield / @emkornfield:
This is a little bit hard to diagnose, especially without the data in question.  Also what version of Arrow/Parquet is being used here, there have been a few bug fixes in the past for this code path.

Micah Kornfield / @emkornfield:
we've also had some other bugs related specifically to float columns, I'm not sure if this could be related.  Do you at least have something like a schema and approximate data sizes (number of nulls?)

Tham / @thamht4190:
@emkornfield  the arrow version I'm using is 0.17.1.
Sorry, I don't have the data when crash occured but I can share the scheme overall:

there are 152 columns: first column is double and others are float.

float columns may contain null value alternatively.

one row group has ~7680 rows.
When our application crashed, it tried to close the parquet file, and I found an error message printed in some log file:

[2021-03-05T15:03:49.166-04:00][WARNING][70000e71a000] parquet::arrow::FileWriter::Close(): "Column 18 had 7680 while previous column had 0"

Any idea?

Tham / @thamht4190:
Ok, thanks @emkornfield. I'll try to upgrade and run our app on the customer machine, then reply to this ticket when I have the result.


chore(js): add @hyperledger prefix

Adds the @hyperledger prefix to all js packages as aries-askar-shared is already taken.
It also updates the registartion of the native implementation to be automatic when you import the nodejs / react native as is done with indy-vdr.
It also sets the version in the package json to 0.1.0-dev.1 so we can make a first release.
@blu3beri is the React Native also ready for release? I think we still need the ios / android publishing to the release right? As the version is –dev.x and it's already integrated sort into AFJ I say we start releasing it under the official name.

@blu3beri is the React Native also ready for release? I think we still need the ios / android publishing to the release right?

The bindings should be good to go. We do indeed still need the iOS and android lib.

As the version is –dev.x and it's already integrated sort into AFJ I say we start releasing it under the official name.

Yes, that would be good.

Okay, so should we keep the react native package private then for now?

Okay, so should we keep the react native package private then for now?

Yeah I think that that would be the cleanest way. I don't think that releasing a broken version will cause issues, but setting it to private would be a good solution.

Ok, have set react native to private, removed the eslint disables.
I think we're ready to do the release now. There's been a lot of releases to main since the last release, so ideally we create a new binary release as well.
@andrewwhitehead is it okay for you to create a new release? If so we need to make sure to update the binary version in the remote_path field in wrappers/javascript/aries-askar-nodejs/package.json and set it to the github release tag beforehand.  Will it be 0.2.8 or 0.2.8-dev.1?
@blu3beri does it also look good to you?

Ok, have set react native to private, removed the eslint disables.
I think we're ready to do the release now. There's been a lot of releases to main since the last release, so ideally we create a new binary release as well.
@andrewwhitehead is it okay for you to create a new release? If so we need to make sure to update the binary version in the remote_path field in wrappers/javascript/aries-askar-nodejs/package.json and set it to the github release tag beforehand. Will it be 0.2.8 or 0.2.8-dev.1?
@blu3beri does it also look good to you?

Yes! All looks good to me.


Cannot use outFile with umd format

TypeScript Version:  3.2.2

Search Terms:
Code
// A *self-contained* demonstration of the problem follows...
// Test this by running `tsc` on the command-line, rather than through another build tool such as Gulp, Webpack, etc.
// tsconfig.json
{
    "compilerOptions": {
        "outFile": "../dist/bundle.js",
        "module": "umd"
     }
}

Expected behavior:
TypeScript lets me just compile it into one UMD file for the browser, which would eliminate my current Rollup & Rollup-Plugin-TypeScript workflow.
Actual behavior:
The compiler forces me to use AMD/SystemJS.

Duplicates #6380

@weswigham Wow thats BS

I mean, it boils down to that a chance (and thereby umd) single file output requires building a loading into the output. Webpack and friends already do this (in a very configurable way) and we didn't want to be in the bundling game, if possible. AMD and systemJS single file support exists because it's essentially just a concatenation of their multifile emit, same as non-module single-file emit.

Explicitly mentioning a workaround (using a bundler) through the error message may help to prevent users from being confused.


bg_object_segmentation

'bo_segm = anno.get("bg_object_segmentation", None)' in detectron/data/datasets/coco.py ,  does the 'bo_segm'  refer to the background?

'bo_segm' refers to the mask of the other occluding objects inside the ROI region.

Oh,I see. And  there can be multiple occluders for an object，Is it right？