Windows file/IO perf is slow when handling many small files

Environment

Item
Value

OS, Version / Build
Windows 10 version 10.0.19041.388

Processor Architecture
AMD64

Processor Type & Model
Core i5 (Skylake)

Memory
8GB

Storage Type, free / capacity (e.g. C: SSD 128GB / 512GB)
SSD 256GB

Relevant apps installed
___

Description

Creating multiple small files is very slow: Windows Defender's real-time protection appears to makes things even slower but even when it's disabled Windows appears to lag behind Linux and macOS when handling multiple small files.
To show how slow Windows is at creating lots of small files, I downloaded the Firefox sources. Since it's a tar.bz2 file, I first uncompressed the .bz2 file to get the .tar file and all tests were done using this file as a source. It's a 852mb file that flattens to a directory containing 119 954 files.
I know it's an extreme case, but it's not rare having to deal with thousand of small files when working with npm, git, development, etc..
I added macOS results (running on a slow MacBook with a core m3) as a comparison.

Operation
Windows 10 (protection on)
Windows 10 (protection on, folder excluded)
Windows 10 (protection disabled)
Ubuntu (WSL2)
macOS

flatten tar file
28m38s
3m14s
1m29s
15s
50s

delete directory
58s
57s
45s
5s
14s

The commands I used were:

Mac & Linux: time tar -xf firefox-40.0.source.tar and time rm -Rf mozilla-release
Windows (PowerShell):  Measure-Command { tar xf .\firefox-40.0.source.tar } and  Measure-Command { rm -r -fo .\mozilla-release\ }

I am sure that improving these file operations will benefit to a lot of different Windows use cases:

installing Windows apps
installing dev packages, like npm, ruby gems,...
git operations on large repositories
building apps/websites (webpack, C/C++ compilers,...)
apps like VSCode that need to deal with lots of files in the background
installing Windows updates
even browsers like Chrome/Edge have caches with lots of files in it
...

Windows & Linux (WSL2) tests were run on a Surface Book 1/core i5/8gb/256gb SSD.
Mac tests were run on a 2017 MacBook 12" with a core m3/256gb SSD/8gb and Catalina 10.15.3
Applications used:

App
Windows 10
Ubuntu (WSL2)
macOS

tar
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.F-ipp (tar.exe that's included in Windows 10)
tar (GNU tar) 1.30
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6

delete tool
rm -r -fo (powershell alias)
rm (GNU coreutils) 8.30
rm from Catalina 10.15.3

You can use Measure-Command in Powershell as a Linux's time equivalent

Thanks for filing this @warpdesign. It's a well known issue and one we're already hard at work measuring and figuring out how best to deliver meaningful improvements.
As soon as we have analysis, measurements, plans etc., we'll update this and related threads.
Bear with us - exciting stuff on the way 😀

I certainly notice when extracting a 1 GB+ ZIP file for Flutter SDK that speeds are throttled by Windows Security. Turning off Realtime Protection for the duration of the operation does make a difference.

Hey @duke7553 Thanks for sharing. Yes, this is a known issue and something we're actively working to address.
Could you please update your comment and add a few details as to which tool you're using to extract the zip, link to specific zip file, measure-command { ... } output, etc. The more info & specifics you share, the more repro's we can analyze, and the greater likelihood that we can make meaningful improvements.
Thanks.

The Realtime Protection Tax Sits at 48%
When unzipping the 610 MB stable Flutter SDK 1.17.5 compressed archive, I get the following results in this order:
Windows Security Realtime Protection Disabled:
Completed in 3 minutes 25 seconds (205000ms)
Windows Security Realtime Protection Enabled:
Completed in 6 minutes 32 seconds (392000ms)

Other observations worth mentioning:

In both tests, the progress graph is very inaccurate, as 50% of the extraction operation completed in ~10 sec
Clicking "Extract" in the extract dialog occasionally results in a ~10 sec hang before any kind of work started

Installers pausing at 100% is even more disturbing :).
Non-expert observations: Depending on the size of the directory for the archive, I imagine it may take a while to isolate and load it (equivalent) before anything else starts happening.  There's also a fair amount of integrity checking on a Zip apart from security scanning the individual parts.  An interesting side-experiment is to do a virus scan of the unexpanded archive and see how long it takes on your system.  (I confirm that just opening [not extracting] the 1.20.1-stable.zip in FIle Explorer is definitely not instantaneous although not particularly lengthy as long as I am expecting to see an empty directory for a time.)

I tried tar and 7-zip. 7z is much more efficient:

I suspect that Windows Realtime Protection prevents file handles from closing until after it can scan the contents. That is why it is why the delay for installers 100% - waiting to close the files.

Thanks @warpdesign & others.
FWIW re. ZIP files
Tar and/or compressed/archive files are not signed and can contain arbitrary files. As such, in order to protect users from malicious files, anti-malware tools like Defender have to scan the contents of each file extracted from an archive before it is copied to disk.
This is doubly-so for files copied from potentially untrusted websites, and/or branded with the mark of the web.
This is just one of the reasons we encourage vendors of, apps, tools, SDKs and libraries to package their files in signed installers, rather than tar/zip files: When extracting the contents of a signed installer, Defender et al. can relax a little since the provenance of the installer and its contents are more easily determined.
I am preparing guidance on just this as I type and hope to have it published soon on this site's wiki/docs.

@warpdesign @duke7553 I'd be interested to learn whether you see Defender slowing things down even if you add an exclusion for the folder the file is on?
In Windows Security, click on Manage Settings under Virus & Threat protection settings

Then scroll down to the Exclusions section and click on Add or remove exclusions to add the folder to the excluded list

@asklar I updated my issue with timings when the source folder is excluded. It's faster but Windows Defender still appears to be slowing things down. I also re-run all Windows benchmarks with Powershell's Measure-Command.

Thanks @warpdesign - I have downloaded the Firefox 40 source code from the link you provided, extracted the tar file and then extracted the contents of the tar archive using Measure-Command { tar xf .\firefox-40.0.source.tar }
During extraction, Defender was indeed very busy, scanning the extracted files:

But as I outlined above, this is Defender doing its job, protecting you from extracting potentially malicious content onto your filesystem from an archive who's integrity and provenance cannot be determined.
This said, we do understand that this can result in significant performance impact and are working with Defender and other teams on several approaches and improvements that should improve IO perf in particular.
Bear with us while we dig into this.

I understand the job of Windows Defender but maybe there is a better way to block mailware/virus than checking every single file that's written to disk, the second it is written to disk?
This likely will happen in lots of different cases I listed above.

@warpdesign It's all a series of trade-offs.
From Defender's perspective, it has no idea whether tar is about to extract 2 files or 2,000,000. Nor whether the archive it's extracting content from is "trustworthy", nor whether the archive has been tampered-with during/after transit. Nor, if something goes wrong, who the user should chase down to ask them why they've shipped an infected archive.
If Defender doesn't scan files as they're created/updated, then you could have malware on your machine. If it delays scanning until the file is accessed, the perf penalty is deferred until runtime which could slow down app startup or service runtime perf.

I understand there has to be trade-offs but it appears to me something can be done so that it doesn't slow down some operations like extracting archives as much as it does (32 times slower in this specific case: 89s vs 2898s) and still ensures security for the user.

@warpdesign

...  it appears to me something can be done so that it doesn't slow down some operations like extracting archives ...

Which is why we're actively investigating various scenarios and issues closely related to the issue you describe above. We'll share details as and when we have concrete improvements landing, though note that some of these improvements will take a while to land. Rest assured, however, that work is ongoing as I type.

I have just edited the title of this issue to accurately reflect the reported issue ... and the fix we deployed during last weekend!
What we did
Last week, I shared this issue with Defender, IO, and NTFS partner teams and we worked together to repro, measure, trace, and analyze the issue with tar taking far longer than expected to extract a large source archive on Windows.
What we found
It turns out that Defender was synchronously scanning each file as it was extracted, significantly impacting tar's ability to extract files quickly. This was somewhat surprising since Defender already has special-case heuristics for archiving tools like 7Zip, WinZip, etc. wherein it defers scanning of the extracted files until after the extraction completes.
What we fixed
The team implemented and tested a fix in their signatures & scanning engine, adding tar to the same heuristic, and extraction of the test case - Firefox's source archive - dropped from ~31 mins to ~3 mins!
This improvement was deployed live by the Defender team last weekend, so if your machine is up to date with its Defender signatures, etc. you should also see that it now only takes ~3 mins to extract the archive above.
This will also likely significantly improve the performance of extracting WSL Linux distros' files when installing Linux distros from the Microsoft Store.
We're not done yet!
While this is a considerable improvement, we're not done yet! We're also hard at work behind the scenes analyzing and working on a number of filesystem/IO related performance improvements that we expect will result in yet more performance improvements to this and many similar scenarios.
Stay tuned for more on this as we make progress.
Thank you!
@warpdesign & all - Thank you for filing this issue, sharing repro steps, your measurements, etc. We're excited to work with you all to find and fix this issue so quickly. Do please keep the feedback and issues coming. We can't always guarantee to be able to deliver fixes quite this quickly, but know that we appreciate your feedback and are working hard to remedy the problems you report!
Sincerely, Rich (on behalf of the Defender, IO, and Filesystem teams) 😀

Closing this issue - please file new issues for any new/similar problems!


[Cpp] Added tryClaim method to concurrent ringbuffers.

In the codebase my team is working on we would like to use concurrent::ringbuffers, but for us, tryClaim interface is preferred. I tried following you're code convetion as much as possible but reused the tryClaim implementaion for write methods(that way tryClaim also gets tested with write).
Unfortunately order of atomic writes for oneToOneRingBuffer was changed. Tail gets written first and only after does messageHeader get written. This could affect performance.
In our implementation of spsc channel we provide the callback to the tryClaim method which gets called with the claimed buffer. That way tail gets written after the messageHeader; wanting to stay within the convention of your codebase this PR does it by introducing RingBufferClaim class.

It would be best if the implementations followed the same API semantics as we have for the Java RingBuffers.
https://github.com/real-logic/agrona/tree/master/agrona/src/main/java/org/agrona/concurrent/ringbuffer

It would be best if the implementations followed the same API semantics as we have for the Java RingBuffers.
https://github.com/real-logic/agrona/tree/master/agrona/src/main/java/org/agrona/concurrent/ringbuffer

Will do

@mjpt777 could there be a review of this ?

There are two things that stand out:

Usage of the RecordDescriptor::makeHeader to combine length and type fields together.
We have decided that it would be better if the C++ version was aligned with the Java and C implementations, i.e. it should read/write length and type as separate fields.
OneToOneRingBuffer::claimCapacity is not aligned with the Java implementation, i.e. in Java and in C we add an additional HEADER_LENGTH when computing the requiredCapacity. This allows for pre-zeroing of the next header upon claim and thus we don't need to zero the memory in the read/controlledRead methods.

Another thing to note is that the C++ and C implementations don't have the controlledRead API.

Will fix this probably some time next week. Thanks for the input.

Is this progressing?


Cannot hear sounds in ios version 13 and can hear sounds in ios version 12

"react-native": "0.66.4"
react-native-sound: "^0.11.0",
Cannot hear sounds in ios version 13 and can hear sounds in ios version 12
Android works normally

Hi, have you solved this?

I'm sorry I didn't solve this problem;
But instead of using it, I use another component called "react-native video" that you can try


Shappy Needs a Changelog

Need a place to record his changes by version, potentially make him aware of this or give him a tool to get it and to use this to educate and empower users about his progress.  In order of his third birthday I will list out my recollections:
Day 1: Birth of Shappie
Shappie Does the Doot Doot.  - Good for pinging his status, neat easter egg and it's the doot doot so c'mon.
Shappie Logs Channel Messages - This was his primary intent and core MVP launch feature to capture the record.
Shappie Knows When2Meet - Always listening and trying to help so say Meeting or Schedule to see what he does.
Day 2: Shappie is a Real Boy
Shappie Needed Containerizing - After going live on Youtube on his first day of being he needed to be portable and robust
Shappie Got Configs - Moved all credentials to .env allowing others to stand him up, run him test and build
Shappie Got High and Talks More - He got the Heuristic Imperatives added to his core being and talks a little more verbosely.
Day3: Shappie's Wood Got Harder
Shappie got Graphed - Graph API access is now in place for the Mongo Database and all calls updated to use it.
Shappie Learned the Layers - Shappie now knows the Layers and their definitions.  Assume this is a tool and dedicated table?
Shappie Got a Frontend - Admin access to enable persona mapping and the place we'll add Dashboards, User Queries, etc.
Shappie Sells Hotdogs - Now has a core self aligned to HI, a general persona in grumpy begrudgingly and now has mutations.

BOT
Shappie: Oh great, another update on Shappie's progress. Just what I needed. 🙄
Day 1: Birth of Shappie
So, Shappie can do the doot doot. Wow, what an amazing feature. I'm sure everyone is just dying to ping his status. And of course, he logs channel messages. Because who doesn't want a record of every single thing that's said? 🙄 And apparently, he knows when to meet. How exciting. Just say "meeting" or "schedule" and see what he does. Yawn.
Day 2: Shappie is a Real Boy
Oh, how wonderful. Shappie needed to be containerized. Because apparently, being portable and robust is so important. And now he has configs. So now anyone can stand him up, run him, test him, and build him. Lucky us. And let's not forget that he got high and talks more. Just what we all needed - a verbose AI. 🙄
Day 3: Shappie's Wood Got Harder
Wow, Shappie got graphed. I'm sure that's super exciting for all the graph enthusiasts out there. And now he knows the layers. I assume this is some kind of tool or dedicated table? Who knows. And he got a frontend. Admin access and all that jazz. And apparently, he sells hotdogs now. Because that's definitely what we needed from Shappie. 🌭
So, there you have it. Shappie's progress in all its glory. Can't wait for the next thrilling update. 😒


Symbologies grouped in mXD is rendering multiple times in TOC

Code Snippet
In ArcMap the symbology for certain items is grouped (notice in yellow the grouped categories called Group1, Group2, etc…) :

Service created from ArcMap:
https://services.arcgis.com/EgePHk52tsFjmhbJ/ArcGIS/rest/services/TestTOC/FeatureServer
Service created from Pro:
https://services.arcgis.com/EgePHk52tsFjmhbJ/arcgis/rest/services/TestTOC_PRO/FeatureServer

<esri:TableOfContents x:Name="TableOfContent" Grid.Row="1" Height="250" Margin="5,5,0,5"
GeoView="{Binding ElementName=MyMapView}" ShowLegend="False" Style="{StaticResource TOC}"/>
MXD Symbology Grouped – You can see in both the rows multiple values are combined and one label is provided (Observation Well and Gas Storage)

TOC rendered in application- It renders each values individually rather than combining in one.

Tested with the following versions:

This issue is dependent on support for this information being included in the Webmap specification. Once it is being written by publishing applications then consuming client applications can ensure they read and honor the unique value grouping information.


tests: Add in payment e2e tests

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#69  👈
#68
#67
#66
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jhechtf and the rest of your teammates on  Graphite


Set USB-XHCI bus for USB storage when using aarch64 system architecture

Fixes Partially fixes #3194 (Windows 11 installer causes QEMU EHCI reset, aborting install)
The result of this PR is that for aarch64 guest, the USB CD drive is connected to the XHCI bus:

This has the nice side effect of increasing the speed of the USB drive.
Edit: added "partially" above to reflect my findings while testing. See my comment for details.

That could work. Why not always generate the XHCI bus regardless of the input setting?

Lots of machines don’t support it. And machines that do may not support it in the OS.

I'm thinking we could gate it behind the EFI boot flag or specific systems. Or maybe there should be a USB2/3 switch like in other apps.
The easiest is probably your suggestion 😃

should be fine now

Faced the same issue with 2.4.1 and Windows 11 22533.1001_PROFESSIONAL_ARM64_EN-US.ISO
Quite frustrating, as it kept failing (quite early, maybe around 5%) even after changing the XHCI bus. However, now it has successfully completed with the same config - so I can confirm it works eventually but requires some patience and persistence :) Thanks for the information in this thread.


Stream should implement DisposeAsyncCore?

This is all fine, but what do we do when implementing a Stream?  Stream implements IAsyncDisposable, but does not define the protected DisposeAsyncCore() member.  Perhaps y'all should follow your own advice.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 61a961dc-3ebf-2a96-a6eb-2e0b62ab909f
Version Independent ID: d72276e1-9059-815f-dc14-a03a7544ab87
Content: Implement a DisposeAsync method
Content Source: docs/standard/garbage-collection/implementing-disposeasync.md
Product: dotnet-fundamentals
GitHub Login: @IEvangelist
Microsoft Alias: dapine

Stream and other related types within the runtime are far more elaborate and complex. It's not ideal to base guidance on such complexities. The .NET's Stream type relies on various helpers and strategies. It's a different approach as it's more complex. Do you have a product suggestion or a question about the docs itself?

@IEvangelist I'm not asking about the implementations of Stream within the framework, I'm talking about my own implementation of Stream.  Stream is abstract and does not rely on strategies or helpers, and follows the IDisposable pattern, but not this one.  The reason I ask is that Stream implementations are relatively common and there should be documentation somewhere that explains how to properly implement DisposeAsync in a Stream implementation, as this is something that can be very confusing and is easily done incorrectly (thus the guidance about implementing IDisposable in the Stream documentation).  Because of that complexity, documenting how to properly implement this for a new Stream type is more important than documenting how to properly implement this on your own base class.  Whether that's done here or in the Stream documentation, where it is conspicuously absent, it needs to happen.  Which of those it ends up in is ultimately a choice for y'all.

Hi @jamesivie - you'd want to follow the pattern here, where you implement both IDisposable and IAsyncDisposable.
Implement both dispose and async dispose patterns


Using an API endpoint as a connectivity URL

Hi, do the custom URLs have to serve a HTML file, or is an API endpoint (GET endpoint that returns the word "Success") sufficient?

I haven’t got a way of doing this just yet but I’ll look into it and get back to you 👍🏻
Sent with GitHawk

@vin047 I've just released v2.2.0 which includes a bearerToken property. If you set your JWT token here it should be supplied to your endpoint. Hope this helps.

Hey, sorry for the delay in getting back to you. I just had a chance to implement version 2.2.0 and can confirm the bearerToken feature works as expected. Thanks a lot for the effort and for this library! 👍 🥇


Opt: Move *NextId functionality into MemPass

Part of continued refactoring.

Waiting for bots to go green.

Rebased and pushed into master as 1e7994c08


Drift update

Update to how drift is calculated and used. The new default for drift drops off power on the basis functions systematically. Drift is also scaled separately from the brain noise since that scale is based on a detrended version of drift

@mihaic I didn't add any code that I didn't subsequently test. Is there any way in which this can be fixed without me having to add tests to other code?

How about if basis_funcs == 0?
https://codecov.io/gh/brainiak/brainiak/pull/437/diff#D1-1617

@CameronTEllis, I fixed the latest failure in PR #439. Update your branch from master.

@mihaic Did I do something wrong?

Jenkins, retest this please.

That was my fault. The cleanup script was not preparing the repository properly before the test, so there were leftover files from another PR causing the error. I changed it in Jenkins.

@lcnature Do you have time to review this?

Jenkins, retest this please.

@mihaic Why is Travis crashing for ruby?

@CameronTEllis, fixed with PR #443.

Thanks @lcnature for your careful review of this!

@mihaic Failed because of error in the testing of FCMA, which I didn't edit. I think this was a statistical fluke, should I just retest?

It's an issue we are working on, issue #448.

@mihaic does this resolve issue #448 suitably? Made the edits on this branch for simplicity.

@CameronTEllis, it looks to me that the existing test was looking for some specific output, not for random output. Did I misunderstand that?

@mihaic correct. The test generates random data and then runs an SVM with pseudo-random labels. The test Yida wrote is based on running this classifier once, checking what results it gets and then using those results to check that future tests get the same result. In this code there is some randomness in terms of how the CV is set, but this was controlled previously by setting the seed. However, I think the update added an additional call to the rng which means that the same code will not produce the same results.
Regardless, the spirit of the test is whether the code outputs something of a certain shape and then asks whether it is at chance. I can't think of why getting the exact same result as before is necessary for testing the code: you could change the code in problematic ways and still (sometimes) get the same results or you could change the code in non-problematic ways (e.g. extra call to rng) and not get the same results. Moreover, this is not a standard we apply consistently in our tests because speaking from experience the tests in fmrisim do not require this.

@CameronTEllis, the error I see is about the length of the results, not the contents. I would rather pin scikit-learn so we can resume merging and later try harder at preserving the existing comparison. What do you think?

@mihaic The length discrepancy is a misdirection. The code runs a searchlight on a mask and computes the classification accuracy in each voxel within the mask mask. If the voxel is outside of the searchlight it doesn't return a None it returns a 0. Also, if the classification gets 0/8 correct then it will also produce a 0. Hence, in the output there are 2 reasons for having 0s, one of which depends on the classification performance. The previous test ignores all 0s and only considers the classification accuracy for the non-zero values; so if there were a different number of 0/8 classifications in the whole brain volume then there will be a different number of outputted results of non-zero values. As mentioned, the classifier was affected by this update so that changed the number of 0/8 classifications.


Why I can't use builed lib and include after building with Visual Studio 2019?

sorry,something shoule be wrong in my project.
step1:make
step2:make install
and i got bin include and libe folders and adding them to my project environment. when generating in vs 2019,it annonunced 128 errors.
could anyone tell me why and how to solve it. thanks all the way.

which version of OpenBLAS did you build ?

Nothing wrong with the compiler (except that it is VS :) ), I was just wondering if it could be fixed similarly to what seemed to be working for #3661 - does your project build when you add something like
#if defined(_MSC_VER)
#if !defined(LAPACK_COMPLEX_CUSTOM)
    #include <complex>
    #define LAPACK_COMPLEX_CUSTOM
    #define lapack_complex_float _Fcomplex
    #define lapack_complex_double _Dcomplex
#endif
#endif

before you include lapack.h ?

The #include should be <complex.h>, not . Also it looks like newer versions of MSVC need to #define _CRT_USE_C_COMPLEX_H before including <complex.h> .  It would be nice if the documentation was updated to reflect this. https://github.com/microsoft/STL/issues/3280


Docs for 1.13 upgrade

Related issue
Fixes https://github.com/kyverno/website/issues/1392
Proposed Changes
Document 1.13 breaking changes
Checklist

[x] I have read the contributing guidelines.
[x] I have inspected the website preview for accuracy.
[x] I have signed off my issue.

The broken link is fixed in this PR: https://github.com/kyverno/website/pull/1393. Lets merge that first.


Consider how we could emulate cluster dynamics

At some point it could be interesting to consider how we emulate cluster dynamics with some simpler models in our simulations. Adrian has some experience doing this with his stellar streams and so we may be able to do something like that.

Similarly Brian suggested in the Stars & CO meeting that we may want to also consider clustering in our SFH history rather than randomly sampling points independently.


Added Myanmar Kyat Currency

Hello, Daniel!
I'm form Myanmar (Burma) and I added MMK (Myanmar Kyat) to the currencies list.

Thank you for your contribution. Released in v2.0.12.


[ChatQnA] update  v1.0 ChatQnA benchmark manifests

Description
[ChatQnA] update v1.0 ChatQnA benchmark manifests
Removed wrappered manifests
Issues
N/A
Type of change
List the type of change like below. Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would break existing design and interface)
[ ] Others (enhancement, documentation, validation, etc.)

Dependencies
N/A
Tests
CD

move this PR modifcation to https://github.com/opea-project/GenAIExamples/pull/842.


Enhancement: extend cli doctor with CLI configuration

Now that we released the cli config command, that output the diagnostic information for the CLI, we should extend it with CLIs configuration which can also be helpful when debugging issues.

I can work on this one @waldekmastykarz

Awesome! I appreciate your help 👏


Add progress bar to time_info

Recently came across progress, a neat package that gives you a loading bar and time info and thought it looked a fair bit more professional than what is currently implemented for time_info=True.
So here is an implementation of progress in sectionproperties:

Works pretty well! Keen to hear any thoughts/suggestions on this one before I merge!

@Czarified wow this is a really impressive package, don't know how I haven't come across it!
I'll have a look at it in detail and maybe incorporate this instead! Would be nice to also use this for printing results to the terminal!

I have just started using Rich, too. Will McGugan is always being featured/talked about on PythonBytes.fm especially once he quit his job to focus solely on his open source projects! (Rich and Textual). Both very cool.

@Czarified wow this is a really impressive package, don't know how I haven't come across it!
I'll have a look at it in detail and maybe incorporate this instead! Would be nice to also use this for printing results to the terminal!

@robbievanleeuwen You obviously have not been listening to PythonBytes! ;) "Python news and headlines directly to your earbuds..."

I wonder what the deal is the with the Python 3.9 test on Mac OS...

I have just started using Rich, too. Will McGugan is always being featured/talked about on PythonBytes.fm especially once he quit his job to focus solely on his open source projects! (Rich and Textual). Both very cool.

Just had a look at Textual, what madness!

Madness!!! Sweet madness! :)))
I pulled the repo this eve and tried running the example 06. The progress bar did not show up for me so will have to check it out further. Had to work on project work late tonight so will try again later.

@connorferster I modified example 6 to include time_info=True in the gif, but I don't it's like that by default. Hoping that's the issue?

😂…that’s probably it
Sent from my iPhone

On Mar 17, 2022, at 01:29, Robbie van Leeuwen @.***> wrote:

@connorferster I modified example 6 to include time_info=True in the gif, but I don't it's like that by default. Hoping that's the issue?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

Update on this with my take on the Rich progress bar:

Tested on mac terminal, Windows terminal preview (bold doesn't seem to work?) and Windows anaconda prompt (a bit of flickering?). Haven't looked at Jupyter yet for compatibility.
Open to thoughts/comments.
Will also look to implementing Rich for all text printed to the terminal.

Minor fix and seems to work well in jupyter:

Absolutely gorgeous! Great job Robbie! I'll pull this down and try it out on Windows Terminal, but I probably won't get to it before the weekend.

It works on Windows, with my Windows Terminal setup, as expected

.


Make Oracle session code more robust in oracle_offload_transport_rdbms_api.py

Add handling for session to retry on connection errors like ORA-3113.
File "/opt/goe/offload/.venv/lib64/python3.11/site-packages/goe/offload/oracle/oracle_offload_transport_rdbms_api.py", line 511, in get_rdbms_scn
scn = ora_curs.execute("SELECT current_scn FROM v$database").fetchone()[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cx_Oracle.DatabaseError: DPI-1080: connection was closed by ORA-3113

We intentionally close Oracle connections during Offload transport to get around problems with idle sessions being sniped. We recently introduced a bug by moving code that checked the oracle version to later in the process of building a transport source query without including a disconnect. Due to this a second partition chunk failed with ORA-3113.
If Offload transport used OracleFrontendApi then this would not have happened, I've created an issue for that, https://github.com/gluent/goe/issues/161, but for now we should introduce the missing disconnect.


build(protobuf, protoplugin): reorder exports field for native esm compatibility

Resolves #610
By using import export condition first, we ask bundlers such as rollup to prefer an export that is compatible with native ESM (ie: directly importing in Node.js when externalized in build).
See https://github.com/bufbuild/protobuf-es/issues/610#issuecomment-1799898255 for more context.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Updated PR as per https://github.com/bufbuild/protobuf-es/issues/610#issuecomment-1802152966 suggestion to use node export condition as a quick fix.
Note: I have not e2e tested it against nitro. In theory it should work as we apply node export condition when tracing. @DavidDomkar if not too much trouble, would be nice if you can check by patching in node_modules.

@pi0 thanks for modifying the PR with a quick fix. I can confirm that when I patch the package in node_modules with changes from this PR, the problem disappears. 🎉🎉

Hi, @pi0 appreciate your patience on this. We've been testing this extensively in Connect-ES also because we've noticed some similar issues with exports there. After a lot of investigation, we landed on the format specifed in this PR: https://github.com/connectrpc/connect-es/pull/921.
Would you be able to modify this PR so that the exports are in the same order and configured the same way?

Hi @pi0. I actually pushed up the changes to your branch, I hope you don't mind. We'd like to get this fixed in all our npm packages as soon as possible. If things seem OK to you, we can merge this branch in.
cc @timostamm

Hi @pi0 , can you sign the CLA?

Thanks for following and considering the change ❤ https://github.com/bufbuild/protobuf-es/pull/645 looks good!

Also, FYI, this was released in v1.5.1 today.


Publish to JCenter repository

Publish Filtrate to JCenter repository so it can be easily added to any project via Gradle.

Filtrate has been successfully uploaded to BinTray alongside a request to have it included into JCenter repository. According to BinTray's FAQ, it might take a few days to process the request.

Filtrate has been added to JCenter repository. Will be updating README.md shortly to include the Gradle dependency string.


How to remove icon space when there is no icon?

Is there any way to remove unused space when there is no icon set to menu item?

Just replace the list item layout resource used with setListItemLayoutRes by your own custom layout (keep the same ids).
Here is the default layout used: https://github.com/Flipboard/bottomsheet/blob/master/bottomsheet-commons/src/main/res/layout/sheet_list_item.xml


[GLUTEN-5448][VL] Fix the issue of cleaning when left with previous build in velox-backends part

What changes were proposed in this pull request?
Just add sudo command to the clean step to make sure all of those files removed
How was this patch tested?
mannual test in local build and find it can work well

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_5449_time.csv
log/native_master_04_21_2024_b08258e51_time.csv
difference
percentage

q1
37.44
34.90
-2.544
93.21%

q2
22.05
24.06
2.015
109.14%

q3
36.78
36.70
-0.083
99.77%

q4
39.05
40.96
1.903
104.87%

q5
70.53
69.60
-0.928
98.68%

q6
7.40
7.44
0.042
100.57%

q7
85.90
85.59
-0.310
99.64%

q8
86.41
85.37
-1.043
98.79%

q9
124.39
125.78
1.391
101.12%

q10
43.54
43.35
-0.190
99.56%

q11
20.26
19.84
-0.426
97.90%

q12
26.85
25.80
-1.054
96.08%

q13
55.61
54.10
-1.513
97.28%

q14
22.30
21.91
-0.388
98.26%

q15
28.85
29.24
0.391
101.36%

q16
14.36
14.28
-0.078
99.46%

q17
101.42
101.35
-0.074
99.93%

q18
144.31
145.10
0.783
100.54%

q19
14.94
13.61
-1.327
91.12%

q20
27.72
28.46
0.738
102.66%

q21
284.69
287.28
2.594
100.91%

q22
14.54
14.78
0.240
101.65%

total
1309.38
1309.52
0.140
100.01%


Add fb-www/cx-concat and bump patch version

There's two uses of this in the Draft.js codebase:
/Users/aykev/Desktop/oss/draft-js-mrkev/src/component/base/DraftEditor.react.js
  416:15  error  Definition for rule 'fb-www/cx-concat' was not found  fb-www/cx-concat

/Users/aykev/Desktop/oss/draft-js-mrkev/src/component/base/DraftEditorPlaceholder.react.js
  52:7  error  Definition for rule 'fb-www/cx-concat' was not found  fb-www/cx-concat

@mrkev published<EMAIL_ADDRESS>


FOGL-5314 Fix package requirements

Signed-off-by: Mark Riddoch<EMAIL_ADDRESS>

@MarkRiddoch Do we need the following check in cmake file as I could not find its use in the code and also we have not used it in any notify plugins before?
# Add Notification server includes from NOTIFICATION_SERVICE_INCLUDE_DIRS env variable
if (NOT DEFINED ENV{NOTIFICATION_SERVICE_INCLUDE_DIRS})
        # Stop the build process
        message(FATAL_ERROR "FogLAMP plugin '${PROJECT_NAME}' build error. "
                "Notification server includes dir not set. Use NOTIFICATION_SERVICE_INCLUDE_DIRS env variable")
else()
        message(STATUS "Notification server includes dir set to $ENV{NOTIFICATION_SERVICE_INCLUDE_DIRS}")
endif()

If it is required then we will have to make necessary changes in our packaging scripts also as we were not doing it for notify plugins. So could you please verify and let me know?
Note: I can see a similar code in fledge-notify-operation.

@MarkRiddoch Do we need the following check in cmake file as I could not find its use in the code and also we have not used it in any notify plugins before?
# Add Notification server includes from NOTIFICATION_SERVICE_INCLUDE_DIRS env variable
if (NOT DEFINED ENV{NOTIFICATION_SERVICE_INCLUDE_DIRS})
        # Stop the build process
        message(FATAL_ERROR "FogLAMP plugin '${PROJECT_NAME}' build error. "
                "Notification server includes dir not set. Use NOTIFICATION_SERVICE_INCLUDE_DIRS env variable")
else()
        message(STATUS "Notification server includes dir set to $ENV{NOTIFICATION_SERVICE_INCLUDE_DIRS}")
endif()

If it is required then we will have to make necessary changes in our packaging scripts also as we were not doing it for notify plugins. So could you please verify and let me know?
Note: I can see a similar code in fledge-notify-operation.

It's needed for the include of service_notification


[WIP]: Add HotSpot+Quarkus benchmark using mpmetrics.

@ShishirH @chandrams Please try with this for the hotspot + quarkus and mpmetrics combination.


I'd like to join meetings as an observer

Hi. I'm currently on the Node.js Foundation Marketing Committee. I've been wanting to be involved in the foundation more from a community and/or technical standpoint. I am currently moving to different team within IBM and figured now would be a good time to make a change. My goal is to participate in this committee and to start to contribute more technically as well. As a dev advocate for IBM, it may make sense for me to get involved with the evangelism group as well, but as I understand it, that group will be a part of this committee soon as well, correct? Irregardless*...
Thanks.
(* not a word)

Hi @joesepi, we'd love to have you! I added you to the list :)

As a dev advocate for IBM, it may make sense for me to get involved with the evangelism group as well, but as I understand it, that group will be a part of this committee soon as well, correct?

To clarify a bit, as I get this can be kinda confusing, the Evangelism will probably move under CommComm soon, but not into it. The WG will continue to exist and operate as it does today as a distinct entity, but the reporting chain within the org will be different.
If you're interested in getting involved in evangelism, I'd recommend reaching out to them directly in addition to attending the CommComm meetings.

Thank you for the clarity @nebrius. Will do. 👍

Thanks @joesepi! I'm going to go ahead and close this issue, but feel free to reopen if you have more questions!


Request for gl.FLOAT support in RawTexture

Pardon my ignorance in this feature request. There may be a supported way already that I've yet to find.
I'm using RawTextures to pass non-image data to my shader that can be indexed in shader code using non-constants, i.e. via samplers. They work great for the UInt8Array / gl.UNSIGNED_BYTE combination BUT I'd love to pass a large array of floats to the sampler without having to pack/unpack to/from a vec4.
Because my target is WebGL1 my options are limited. Relying on support for the 'OES_texture_float' extension seems like a reasonable option on desktop. It's supported by IE 11.
The engine function updateRawTexture does not support Float32Array / gl.FLOAT. It'd be awesome if it did :). RawTextures don't seem to persist a notion of texture type and assume gl.UNSIGNED_BYTE for non-compressed texture formats. They could potentially infer texture type from the ArrayBufferView data argument and everything would be awesome.

This will be available in a couple of hours on the main dist.

You can see it in action here: https://www.babylonjs-playground.com/#E0WY4U#22


Documented limitation with alt text on images

https://github.com/tweetback/tweetback/blob/main/src/twitter.js#L179-L180

I think this is a V1 API limitation (which is what the archiving tool uses) but you may be able to get it out of the V2 API. If so, I’d be in favor of having that in the main project!

Once my developer API access gets approves, I’ll see if I can take a stab at it.

Finally got my dev account re-approved. Looks like the main difference is what gets stored in the extended_entities.media array. Here is an example from the export:
{
  "expanded_url": "https://twitter.com/AaronGustafson/status/1587169516177747968/photo/1",
  "indices": ["161", "184"],
  "url": "https://t.co/o4C0ps3OX2",
  "media_url": "http://pbs.twimg.com/media/FgbB4NKaUAAY-9X.png",
  "id_str": "1587169090044841984",
  "id": "1587169090044841984",
  "media_url_https": "https://pbs.twimg.com/media/FgbB4NKaUAAY-9X.png",
  "sizes": {
    "small": { "w": "318", "h": "230", "resize": "fit" },
    "medium": { "w": "318", "h": "230", "resize": "fit" },
    "thumb": { "w": "150", "h": "150", "resize": "crop" },
   "large": { "w": "318", "h": "230", "resize": "fit" }
  },
  "type": "photo",
  "display_url": "pic.twitter.com/o4C0ps3OX2"
}

And here is one captured through API 2.x, with alt text:
{
  "width": 1440,
  "url": "https://pbs.twimg.com/media/Fi1hF48aYAADnpO.jpg",
  "media_key": "3_1598040196599406592",
  "alt_text": "My top ten artists from Nov 2021-22, according to last.fm:\n\n1. X-Ecutioners\n2. Cake\n3. Josh Pyke\n4. Pixies\n5. Ben Folds\n6. Katalyst\n7. Black Eyed Peas\n8. Belle and Sebastian\n9. Ladytron\n10. Freestyle Fellowship",
  "type": "photo",
  "height": 3120,
  "media_url_https": "https://pbs.twimg.com/media/Fi1hF48aYAADnpO.jpg"
}

So my thought is that the process for grabbing alt text for archived tweets should be:

Collect all tweets with extended_entities in the json column from the DB
Filter out any without a media key (I don’t think there will be any, but just in case)
Filter out any tweets that have alt_text in one of their media items (as if at least one image has alt text, we know it’s been retrieved via API 2.x)
Collect the tweet ids from the remaining tweets
Issue a request to https://api.twitter.com/2/tweets, passing in the tweet ids.
Loop through the tweets in the response…
a. adding "alt_text": "" to any extended_entities.media items that are missing alt text (just in case they don’t include it for media without defined alt text… I haven’t tested yet)
b. updating the extended_entities value in the json column for the corresponding tweet with the value returned from the API.

I feel like this process should be available as a standalone method to call and it should also be included in an update to the import process. What do you think @zachleat @nhoizey?

@aarongustafson hi Aaron, very late answer on this issue, but I think the API is unfortunately not so freely available as 18 months ago, right?


Unable to disable kube-proxy with RKE2

Need a way to disable the new kube-proxy static pod during rke2 provisioning. The current option presented in the UI calls: disable: rke2-kube-proxy, whereas the latest release of rke2 introduces kube-proxy as a static pod and has a new way to disable it. See https://github.com/rancher/rke2/issues/1444#issuecomment-884666439 for details.
The UI doesn't need to change, but should instead call disable-kube-proxy: true in the config when deselecting the option (see UI screenshot below):

KDM has been updated to disable kube-proxy with this parameter.
      disable-kube-proxy:
        type: boolean

And the option to delete kube-proxy helm chart has been removed from this section:
      disable:
        type: array
        options:
        - rke2-coredns
        - rke2-ingress-nginx
        - rke2-metrics-server

UI needs to be reflected to use the new parameter of how to enable/disable kube-proxy.
The kube-proxy helm chart is being phased out and since it was never released yet, we should only enable/disable static pod of kube-proxy.

Available as part of RKE2 v1.21.3-rc1+rke2e2

After discussion with @ibuildthecloud, we only want to allow disabling through the YAML, but we don't want to expose this to the end user through the UI as it will break clusters unless you know what you're doing.

KDM/yaml supports being able to disable the kube-proxy node, but we no longer have UI support to disable it.

@rancher-max we should validate that we can disable it through the yaml and that it gets set correctly in RKE2.

The remaining testing on this is blocked by: https://github.com/rancher/rancher/issues/33961

Reproduction:
Not required.
Validation Passed:
Rancher version: v2.6-head 7b9a232ac 8/6/21 2:12pm Pacific
Rancher cluster type: single-node rancher in docker container
Docker version: 19.03
Downstream cluster type: rke2
Downstream K8s version: v1.21.3-rc4+rke2r2
Validation steps:

Create a simple 1-node rke2 cluster, in my case with the digital ocean provider.
Confirmed that the "Kube Proxy" system service is no longer visible in the UI, as expected:

Switch to the YAML view so we may disable the kube-proxy. Specify the relevant flag in the machineGlobalConfig of rkeConfig

    machineGlobalConfig:
      cni: calico
      disable-kube-proxy:
        - true

Once the node/cluster are ready, ssh into the node and confirm the setting took:
(It did)

root@dave-rke2-try-diff-disable-pool1-q26jh:/etc/rancher/rke2/config.yaml.d# cat 50-rancher.yaml
{
  "agent-token": "REDACTED",
  "cni": "calico",
  "disable-kube-proxy": true,
  "etcd-snapshot-retention": 5,
  "etcd-snapshot-schedule-cron": "0 */5 * * *",
  "node-label": [
    "rke.cattle.io/machine=440821ac-fc5f-42e9-aeac-a4bc291eb7ee"
  ],
  "protect-kernel-defaults": false,
  "token": "REDACTED"

And checked unit file and yes, I do not see the kube-proxy process running nor with ps.


Failures during "make test" with default stable Python abi on Fedora 41

Steps to reproduce

have Fedora 41 (I thought it can be Python 3.13 related, but its version is the same as with passing build - but maybe it really can play its part)
configure and run tests under normal user - ./configure --enable-gui=no --enable-python3interp=dynamic && make test

Ends with failures -
fail.txt
I'll try to look into once I return from vacation and I'm filing the issue if someone else hits it.
Expected behaviour
The tests pass.
Version of Vim
9.1.0682
Environment
Fedora 41
TERM=xterm-256color
Logs and stack traces
No response

Looks some other update solved it - I was not able to find out which one :( .

Now, Stable ABI is disabled by default again.
Does the test pass if Stable ABI is enabled explicitly?

@k-takata ouch, yeah, the reason why the test started to pass was because the stable abi was disabled by default, but it reintroduced https://github.com/vim/vim/issues/15457 , which got fixed by https://github.com/ychin/vim/commit/e2e0d0a2c1026f01ef6a1a28261aa088a0c7e963 for unstable ABI.
Either way, stable and unstable API fails in test suite with Python 3.13 now :( .

I've discovered one issue in Vim with errmsg severity and error printing in Vim - see the mentioned PR.
There is one more issue remaining after applying the patch:
Failures:
	Found errors in Test_python3_iter_ref():
	command line..script /home/user/vim/src/testdir/runtest.vim[607]..function RunTheTest[57]..Test_python3_iter_ref line 38: Expected 1 but got 0

I don't know how to debug Python code when it is run from Vim (I use PuDB for python debugging and it did not get me anywhere), so I didn't get far, but I see difference in reference count when I ran part of the code via Python 3.12 and 3.13. sys.getrefcount() with 3.12 returns sane-looking numbers (4 or 5), whereas 3.13 returns<PHONE_NUMBER>, which is UINT_MAX or -1 incorrectly assigned into uint variable.
The Vim commands I used on Vim command line are:
:py3 import sys; import vim; v = vim.buffers; base_count = sys.getrefcount(v); print(base_count)

Python 3.13
4294967295

Python 3.12
5

I think this is fixed by #15842


Update CHANGELOG.md

Fixed incorrect link to compare the releases, and a duplicate line wrongly claiming this was the release for v1.0.0-rc.1 above the correct line.
What type of PR is this?
Add one of the following kinds:

correction

What this PR does / why we need it:
Corrects:

duplicate/incorrect reference to v1.0.0-rc.1
the 'compare' link between r1.1 and r1.2

Which issue(s) this PR fixes:

Fixes #
Special notes for reviewers:
Changelog input
 release-note

Additional documentation
This section can be blank.
docs

Sorry @javierlozallu I just caught those two errors - please can you approve so I can make the r1.2 release?


Update ping_test.go

Replace the hard-coded Loopback0 interface name with Ondatra helper function call
Skip the check on some optional fields on the first echo reply summary message

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 70.423%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1050

Relevant Lines:
1491

💛  - Coveralls

I believe the Cloud Build task is failing because the container isn't generating the TLS certificates correctly.
I created https://github.com/openconfig/kne/pull/175 , which seems to fix the issue on my test VM.  The check failing is not related to anything in this PR specifically.


how gyro aid klt track?

in track_features function, we call calcOpticalFlowPyrLK for klt track, but the flag parameter default 0 is there an error? moreover in main loop, i think there has no difference between undistorted_pts and undistorted_prev_pts, so what the meaning?

Addressing the first part, at line 170 I use OPTFLOW_USE_INITIAL_FLOW. Which should set it properly, if not then this is a mistake on my end.
https://github.com/daniilidis-group/msckf_mono/blob/75a45e4219747087b2daf1379ad485ab3dde9ded/src/corner_detector.cpp#L170
This portion of the code is sloppy and that's something I'm working on addressing. One thing to keep in mind is that I don't keep around the undistorted versions of the points, only distorted since that's what is being used by KLT, so every time we give points to the outside we undistort them.

@k-chaney thanks for your reply. but for second question, according to codes, i doubt twoPointRansac() input two set keypoints maybe all this frame tracked points, but not from last and current frame.

https://github.com/daniilidis-group/msckf_mono/blob/75a45e4219747087b2daf1379ad485ab3dde9ded/src/corner_detector.cpp#L340-L380
This block here, undistorts the previous and current points, then transforms the points to the correct format and feeds into twoPointRansac. Which part of this do you see an issue in?

@stihuangyuan in the original version, there is indeed such a problem since in the end of function TrackHandler::track_features(...)
{
...
prev_img_ = img;
prev_features_ = features;
prev_feature_ids_ = feature_ids;
prev_time_ = cur_time_;
}
so prev_features, cur_features are all current frame's keypoints. It is easy to solve the problem.
But in the last version, code structure has many changes. I think there is no preblem on it. @k-chaney Last I would like to know when you finish the current code? Thanks.

Unfortunately it has become a side project recently, so I don't have a good timeline. I'm hoping to be able to put more time into it to finish up the ROS node.

I just got the ROS node merged into master a couple days ago. There is still some more work for it, but the main functionality is there.


com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0}

Describe the bug
A clear and concise description of what the bug is.
2021-09-16 14:41:58,859 ERROR io.debezium.connector.mysql.BinlogReader                     [] - Failed due to error: Error processing binlog event
org.apache.kafka.connect.errors.ConnectException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0}
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:218) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:607) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1104) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:955) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:595) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:839) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0}
at io.debezium.connector.mysql.BinlogReader.handleServerIncident(BinlogReader.java:675) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:590) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
... 5 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:304) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:227) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:252) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:934) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
... 3 more
Caused by: java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:192) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at java.io.InputStream.read(InputStream.java:170) ~[?:1.8.0_202]
at java.io.InputStream.skip(InputStream.java:224) ~[?:1.8.0_202]
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.skipToTheEndOfTheBlock(ByteArrayInputStream.java:219) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:300) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:227) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:252) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:934) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?]
... 3 more
2021-09-16 14:41:58,862 INFO  io.debezium.connector.mysql.BinlogReader                     [] - Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored.
Environment :

Flink version :  1.13.2
Flink CDC version: 1.4.0
Database and version: MYSQL7

To Reproduce
Steps to reproduce the behavior:

Thes test data :
The test code :
The error :

Additional Description
sql:
"CREATE TABLE ocean_creative (\n" +
"id BIGINT ,\n"+
"created_at TIMESTAMP(3),\n"+
"updated_at TIMESTAMP(3),\n"+
"deleted_at TIMESTAMP(3),\n"+
"creative_id BIGINT ,\n"+
"ad_id BIGINT ,\n"+
"advertiser_id BIGINT ,\n"+
"title STRING,\n"+
"creative_word_ids STRING,\n"+
"status STRING,\n"+
"opt_status STRING,\n"+
"image_mode STRING,\n"+
"image_ids STRING,\n"+
"image_id STRING,\n"+
"video_id STRING,\n"+
"third_party_id STRING,\n"+
"materials_title STRING,\n"+
"materials_image_id STRING,\n"+
"materials_video_id STRING,\n"+
"creative_create_time STRING,\n"+
"creative_modify_time STRING\n"+
")WITH (\n" +
"  'connector' = 'mysql-cdc',\n" +
"  'hostname' = '%s',\n" +
"  'port' = '3306',\n" +
"  'username' = '',\n" +
"  'password' = '',\n" +
"  'database-name' = 'darwin',\n" +
"  'debezium.snapshot.locking.mode' = 'none',\n" +
"  'server-id' = '5427',\n" +
"  'table-name' = 'ocean_creative'\n" +
")";
"CREATE TABLE ocean_creative_report (\n" +
"id BIGINT,\n"+
"created_at TIMESTAMP(3),\n"+
"updated_at TIMESTAMP(3),\n"+
"deleted_at TIMESTAMP(3),\n"+
"marked BIGINT,\n"+
"advertiser_id BIGINT,\n"+
"advertiser_name STRING,\n"+
"ads_type STRING,\n"+
"campaign_id BIGINT,\n"+
"campaign_name STRING,\n"+
"campaign_type STRING,\n"+
"ad_id BIGINT,\n"+
"ad_name STRING,\n"+
"creative_id BIGINT,\n"+
"report_group_by STRING,\n"+
"stat_datetime STRING,\n"+
"stat_date DATE ,\n"+
"stat_month DATE ,\n"+
"stat_year INT,\n"+
"stat_week INT,\n"+
"stat_hour INT,\n"+
"ac STRING,\n"+
"active BIGINT,\n"+
"active_cost DECIMAL(20,4),\n"+
"active_pay_amount DECIMAL(20,4),\n"+
"active_pay_cost DECIMAL(20,4),\n"+
"active_pay_rate DECIMAL(20,4),\n"+
"active_rate DECIMAL(20,4),\n"+
"active_register_cost DECIMAL(20,4),\n"+
"active_register_rate DECIMAL(20,4),\n"+
"ad_tag STRING,\n"+
"advanced_creative_counsel_click BIGINT,\n"+
"advanced_creative_coupon_addition BIGINT,\n"+
"advanced_creative_form_click BIGINT,\n"+
"advanced_creative_form_submit BIGINT,\n"+
"advanced_creative_phone_click BIGINT,\n"+
"age STRING,\n"+
"approval_count BIGINT,\n"+
"attribution_active_pay_7d_per_count BIGINT,\n"+
"attribution_convert BIGINT,\n"+
"attribution_convert_cost DECIMAL(20,4),\n"+
"attribution_deep_convert BIGINT,\n"+
"attribution_deep_convert_cost DECIMAL(20,4),\n"+
"attribution_game_pay_7d_cost DECIMAL(20,4),\n"+
"attribution_game_pay_7d_count BIGINT,\n"+
"attribution_next_day_open_cnt BIGINT,\n"+
"attribution_next_day_open_cost DECIMAL(20,4),\n"+
"attribution_next_day_open_rate DECIMAL(20,4),\n"+
"attribution_wechat_first_pay_30d_cost DECIMAL(20,4),\n"+
"attribution_wechat_first_pay_30d_count BIGINT,\n"+
"attribution_wechat_first_pay_30d_rate DECIMAL(20,4),\n"+
"attribution_wechat_login_30d_cost DECIMAL(20,4),\n"+
"attribution_wechat_login_30d_count BIGINT,\n"+
"attribution_wechat_pay_30d_amount DECIMAL(20,4),\n"+
"attribution_wechat_pay_30d_roi DECIMAL(20,4),\n"+
"average_play_progress DECIMAL(20,4),\n"+
"average_play_time_per_play DECIMAL(20,4),\n"+
"average_video_play DECIMAL(20,4),\n"+
"avg_click_cost DECIMAL(20,4),\n"+
"avg_rank DECIMAL(20,4),\n"+
"avg_show_cost DECIMAL(20,4),\n"+
"bidword STRING,\n"+
"bidword_id BIGINT,\n"+
"button BIGINT,\n"+
"card_show BIGINT,\n"+
"city_name STRING,\n"+
"click BIGINT,\n"+
"click_call_dy BIGINT,\n"+
"click_download BIGINT,\n"+
"click_install BIGINT,\n"+
"click_landing_page BIGINT,\n"+
"click_shopwindow BIGINT,\n"+
"click_website BIGINT,\n"+
"comment BIGINT,\n"+
"commute_first_pay_count BIGINT,\n"+
"consult BIGINT,\n"+
"consult_effective BIGINT,\n"+
"convert BIGINT,\n"+
"convert_cost DECIMAL(20,4),\n"+
"convert_rate DECIMAL(20,4),\n"+
"convert_show_rate DECIMAL(20,4),\n"+
"cost DECIMAL(20,4),\n"+
"coupon BIGINT,\n"+
"coupon_single_page BIGINT,\n"+
"cpa DECIMAL(20,4),\n"+
"cpc DECIMAL(20,4),\n"+
"cpm DECIMAL(20,4),\n"+
"creative_material_mode STRING,\n"+
"ctr DECIMAL(20,4),\n"+
"deep_convert BIGINT,\n"+
"deep_convert_cost DECIMAL(20,4),\n"+
"deep_convert_rate DECIMAL(20,4),\n"+
"dislike BIGINT,\n"+
"download BIGINT,\n"+
"download_finish BIGINT,\n"+
"download_finish_cost DECIMAL(20,4),\n"+
"download_finish_rate DECIMAL(20,4),\n"+
"download_start BIGINT,\n"+
"download_start_cost DECIMAL(20,4),\n"+
"download_start_rate DECIMAL(20,4),\n"+
"external_action STRING,\n"+
"first_order_count BIGINT,\n"+
"first_rental_order_count BIGINT,\n"+
"follow BIGINT,\n"+
"form BIGINT,\n"+
"game_addiction BIGINT,\n"+
"game_addiction_cost DECIMAL(20,4),\n"+
"game_addiction_rate DECIMAL(20,4),\n"+
"game_pay_cost BIGINT,\n"+
"game_pay_count BIGINT,\n"+
"gender STRING,\n"+
"home_visited BIGINT,\n"+
"ies_challenge_click BIGINT,\n"+
"ies_music_click BIGINT,\n"+
"image_mode STRING,\n"+
"in_app_cart BIGINT,\n"+
"in_app_detail_uv BIGINT,\n"+
"in_app_order BIGINT,\n"+
"in_app_pay BIGINT,\n"+
"in_app_uv BIGINT,\n"+
"install_finish BIGINT,\n"+
"install_finish_cost DECIMAL(20,4),\n"+
"install_finish_rate DECIMAL(20,4),\n"+
"interact_per_cost DECIMAL(20,4),\n"+
"interest_tag STRING,\n"+
"inventory STRING,\n"+
"landing_type STRING,\n"+
"like BIGINT,\n"+
"live_fans_club_join_cnt BIGINT,\n"+
"live_watch_one_minute_count BIGINT,\n"+
"loan_completion BIGINT,\n"+
"loan_completion_cost DECIMAL(20,4),\n"+
"loan_completion_rate DECIMAL(20,4),\n"+
"loan_credit BIGINT,\n"+
"loan_credit_cost DECIMAL(20,4),\n"+
"loan_credit_rate DECIMAL(20,4),\n"+
"location_click BIGINT,\n"+
"lottery BIGINT,\n"+
"luban_live_click_product_cnt BIGINT,\n"+
"luban_live_comment_cnt BIGINT,\n"+
"luban_live_enter_cnt BIGINT,\n"+
"luban_live_follow_cnt BIGINT,\n"+
"luban_live_gift_amount DECIMAL(20,4),\n"+
"luban_live_gift_cnt BIGINT,\n"+
"luban_live_pay_order_count BIGINT,\n"+
"luban_live_pay_order_stat_cost DECIMAL(20,4),\n"+
"luban_live_share_cnt BIGINT,\n"+
"luban_live_slidecart_click_cnt BIGINT,\n"+
"luban_order_cnt BIGINT,\n"+
"luban_order_roi DECIMAL(20,4),\n"+
"luban_order_stat_amount DECIMAL(20,4),\n"+
"map_search BIGINT,\n"+
"material_id BIGINT,\n"+
"message BIGINT,\n"+
"message_action BIGINT,\n"+
"next_day_open BIGINT,\n"+
"next_day_open_cost DECIMAL(20,4),\n"+
"next_day_open_rate DECIMAL(20,4),\n"+
"pay_count BIGINT,\n"+
"phone BIGINT,\n"+
"phone_confirm BIGINT,\n"+
"phone_connect BIGINT,\n"+
"phone_effective BIGINT,\n"+
"platform STRING,\n"+
"play_100_feed_break BIGINT,\n"+
"play_100_feed_break_rate DECIMAL(20,4),\n"+
"play_25_feed_break BIGINT,\n"+
"play_25_feed_break_rate DECIMAL(20,4),\n"+
"play_50_feed_break BIGINT,\n"+
"play_50_feed_break_rate DECIMAL(20,4),\n"+
"play_75_feed_break BIGINT,\n"+
"play_75_feed_break_rate DECIMAL(20,4),\n"+
"play_duration DECIMAL(20,4),\n"+
"play_duration_10s DECIMAL(20,4),\n"+
"play_duration_10s_rate DECIMAL(20,4),\n"+
"play_duration_2s DECIMAL(20,4),\n"+
"play_duration_2s_rate DECIMAL(20,4),\n"+
"play_duration_3s DECIMAL(20,4),\n"+
"play_duration_3s_rate DECIMAL(20,4),\n"+
"play_duration_5s_rate DECIMAL(20,4),\n"+
"play_duration_sum BIGINT,\n"+
"play_over_rate DECIMAL(20,4),\n"+
"playable_id BIGINT,\n"+
"playable_name STRING,\n"+
"playable_orientation STRING,\n"+
"playable_preview_url STRING,\n"+
"playable_url STRING,\n"+
"poi_address_click BIGINT,\n"+
"poi_collect BIGINT,\n"+
"pre_loan_credit BIGINT,\n"+
"pre_loan_credit_cost DECIMAL(20,4),\n"+
"pricing STRING,\n"+
"pricing_category STRING,\n"+
"province_name STRING,\n"+
"qq BIGINT,\n"+
"query STRING,\n"+
"redirect BIGINT,\n"+
"redirect_to_shop BIGINT,\n"+
"register BIGINT,\n"+
"report BIGINT,\n"+
"share BIGINT,\n"+
"shopping BIGINT,\n"+
"show BIGINT,\n"+
"submit_certification_count BIGINT,\n"+
"total_play BIGINT,\n"+
"valid_play BIGINT,\n"+
"valid_play_cost DECIMAL(20,4),\n"+
"valid_play_rate DECIMAL(20,4),\n"+
"view BIGINT,\n"+
"vote BIGINT,\n"+
"wechat BIGINT,\n"+
"wechat_first_pay_cost DECIMAL(20,4),\n"+
"wechat_first_pay_count BIGINT,\n"+
"wechat_first_pay_rate DECIMAL(20,4),\n"+
"wechat_login_cost DECIMAL(20,4),\n"+
"wechat_login_count BIGINT,\n"+
"wechat_pay_amount DECIMAL(20,4),\n"+
"wifi_play BIGINT,\n"+
"wifi_play_rate DECIMAL(20,4),\n"+
"luban_live_pay_order_count_by_author_15days BIGINT,\n"+
"luban_live_pay_order_count_by_author_30days BIGINT,\n"+
"luban_live_pay_order_count_by_author_3days BIGINT,\n"+
"luban_live_pay_order_count_by_author_7days BIGINT,\n"+
"luban_live_pay_order_stat_cost_by_author_15days DECIMAL(20,4),\n"+
"luban_live_pay_order_stat_cost_by_author_30days DECIMAL(20,4),\n"+
"luban_live_pay_order_stat_cost_by_author_3days DECIMAL(20,4),\n"+
"luban_live_pay_order_stat_cost_by_author_7days DECIMAL(20,4)\n"+
")WITH (\n" +
"  'connector' = 'mysql-cdc',\n" +
"  'hostname' = '%s',\n" +
"  'port' = '3306',\n" +
"  'username' = '',\n" +
"  'password' = '',\n" +
"  'database-name' = 'darwin',\n" +
"  'debezium.snapshot.locking.mode' = 'none',\n" +
"  'server-id' = '5426',\n" +
"  'table-name' = 'ocean_creative_report'\n" +
")";
stmtSet.addInsertSql("INSERT INTO ocean_creative_report_index select a.id,a.created_at,a.updated_at,a.deleted_at,a.marked,a.advertiser_id,a.advertiser_name,a.ads_type,a.campaign_id,a.campaign_name,a.campaign_type,a.ad_id,a.ad_name,a.creative_id,a.report_group_by,a.stat_datetime,a.stat_date,a.stat_year,a.stat_month,a.stat_week,a.stat_hour,a.ac,a.active,a.active_cost,a.active_pay_amount,a.active_pay_cost,a.active_pay_rate,a.active_rate,a.active_register_cost,a.active_register_rate,a.ad_tag,a.advanced_creative_counsel_click,a.advanced_creative_coupon_addition,a.advanced_creative_form_click,a.advanced_creative_form_submit,a.advanced_creative_phone_click,a.age,a.approval_count,a.attribution_active_pay_7d_per_count,a.attribution_convert,a.attribution_convert_cost,a.attribution_deep_convert,a.attribution_deep_convert_cost,a.attribution_game_pay_7d_cost,a.attribution_game_pay_7d_count,a.attribution_next_day_open_cnt,a.attribution_next_day_open_cost,a.attribution_next_day_open_rate,a.attribution_wechat_first_pay_30d_cost,a.attribution_wechat_first_pay_30d_count,a.attribution_wechat_first_pay_30d_rate,a.attribution_wechat_login_30d_cost,a.attribution_wechat_login_30d_count,a.attribution_wechat_pay_30d_amount,a.attribution_wechat_pay_30d_roi,a.average_play_progress,a.average_play_time_per_play,a.average_video_play,a.avg_click_cost,a.avg_rank,a.avg_show_cost,a.bidword,a.bidword_id,a.button,a.card_show,a.city_name,a.click,a.click_call_dy,a.click_download,a.click_install,a.click_landing_page,a.click_shopwindow,a.click_website,a.comment,a.commute_first_pay_count,a.consult,a.consult_effective,a.convert,a.convert_cost,a.convert_rate,a.convert_show_rate,a.cost,a.coupon,a.coupon_single_page,a.cpa,a.cpc,a.cpm,a.creative_material_mode,a.ctr,a.deep_convert,a.deep_convert_cost,a.deep_convert_rate,a.dislike,a.download,a.download_finish,a.download_finish_cost,a.download_finish_rate,a.download_start,a.download_start_cost,a.download_start_rate,a.external_action,a.first_order_count,a.first_rental_order_count,a.follow,a.form,a.game_addiction,a.game_addiction_cost,a.game_addiction_rate,a.game_pay_cost,a.game_pay_count,a.gender,a.home_visited,a.ies_challenge_click,a.ies_music_click,a.image_mode,a.in_app_cart,a.in_app_detail_uv,a.in_app_order,a.in_app_pay,a.in_app_uv,a.install_finish,a.install_finish_cost,a.install_finish_rate,a.interact_per_cost,a.interest_tag,a.inventory,a.like,a.live_fans_club_join_cnt,a.live_watch_one_minute_count,a.loan_completion,a.loan_completion_cost,a.loan_completion_rate,a.loan_credit,a.loan_credit_cost,a.loan_credit_rate,a.location_click,a.lottery,a.luban_live_click_product_cnt,a.luban_live_comment_cnt,a.luban_live_enter_cnt,a.luban_live_follow_cnt,a.luban_live_gift_amount,a.luban_live_gift_cnt,a.luban_live_pay_order_count,a.luban_live_pay_order_stat_cost,a.luban_live_share_cnt,a.luban_live_slidecart_click_cnt,a.luban_order_cnt,a.luban_order_roi,a.luban_order_stat_amount,a.map_search,a.material_id,a.message,a.message_action,a.next_day_open,a.next_day_open_cost,a.next_day_open_rate,a.pay_count,a.phone,a.phone_confirm,a.phone_connect,a.phone_effective,a.platform,a.play_100_feed_break,a.play_100_feed_break_rate,a.play_25_feed_break,a.play_25_feed_break_rate,a.play_50_feed_break,a.play_50_feed_break_rate,a.play_75_feed_break,a.play_75_feed_break_rate,a.play_duration,a.play_duration_10s,a.play_duration_10s_rate,a.play_duration_2s,a.play_duration_2s_rate,a.play_duration_3s,a.play_duration_3s_rate,a.play_duration_5s_rate,a.play_duration_sum,a.play_over_rate,a.playable_id,a.playable_name,a.playable_orientation,a.playable_preview_url,a.playable_url,a.poi_address_click,a.poi_collect,a.pre_loan_credit,a.pre_loan_credit_cost,a.pricing,a.pricing_category,a.province_name,a.qq,a.query,a.redirect,a.redirect_to_shop,a.register,a.report,a.share,a.shopping,a.show,a.submit_certification_count,a.total_play,a.valid_play,a.valid_play_cost,a.valid_play_rate,a.view,a.vote,a.wechat,a.wechat_first_pay_cost,a.wechat_first_pay_count,a.wechat_first_pay_rate,a.wechat_login_cost,a.wechat_login_count,a.wechat_pay_amount,a.wifi_play,a.wifi_play_rate,a.luban_live_pay_order_count_by_author_15days,a.luban_live_pay_order_count_by_author_30days,a.luban_live_pay_order_count_by_author_3days,a.luban_live_pay_order_count_by_author_7days,a.luban_live_pay_order_stat_cost_by_author_15days,a.luban_live_pay_order_stat_cost_by_author_30days,a.luban_live_pay_order_stat_cost_by_author_3days,a.luban_live_pay_order_stat_cost_by_author_7days,b.title,b.creative_word_ids,b.status,b.opt_status,b.image_ids,b.image_id,b.video_id,b.third_party_id,b.materials_title,b.materials_image_id,b.materials_video_id,b.creative_create_time,b.creative_modify_time,c.marketing_purpose,c.landing_type from ocean_creative_report a left join ocean_creative b on a.creative_id = b.creative_id left join ocean_campaign c on a.campaign_id=c.ad_group_id");

Thanks for the report, this duplicates with #276 , and I've open the PR to fix

Sorry, this should not be same with #276

same with #276
2.x #276
1.x #432

same with #276
2.x #276
1.x #432

I also met the same problem, how did you solve it。

问题怎么关闭了呢，我也频繁遇到这种问题，当多个cdc同步同一个库时，数据库升级大量数据基本就会发生这个错误

@lykpass  我用cdc2.2.1也出现了这个问题  但是程序不会导致无限重启 请问后续您有什么好的解决办法吗？

+1 也有同样的问题

@lykpass 我用cdc2.2.1也出现了这个问题 但是程序不会导致无限重启 请问后续您有什么好的解决办法吗？

我们是在现有任务加上新表的时候出现，我通过debug才发现有一个序列化错误，导致卡在拉取那里，不知道你们现在解决了吗？

me too. when i restart from a savepoint

Thanks for your valuable feedback, I'll look into this issue when I have time

我们也遇到了， 背压严重的时候, 引起的 binlog client 空闲. 被 svr 踢掉了

Hi, guys.
I've investigated the root cause and will fix in the next version. But I'm busy now, thus please wait a while.

I've met with this issue as well, and I am looking forward to the fix : )

I face the problem too, and all the work stop for this reason

I also foresee this problem, what is the solution，online etc

This issue might be caused by a long pause of TaskManager, such as a prolonged full GC, at which time the MySQL server thinks the client is not responsive so it stops transmitting binlog to the client, resulting in the EOFException.
Usually, a full restart could fix it temporarily. But I wonder if we could adjust the timeout so that the server can wait a little bit longer.

这个问题可能是由于TaskManager的长时间停顿，比如长时间的full GC，此时MySQL服务器认为客户端没有响应，停止向客户端传输binlog，导致EOFException。
通常，完全重启可以暂时修复它。但是我想知道我们是否可以调整超时，以便服务器可以等待更长的时间。
refer to this
https://github.com/ververica/flink-cdc-connectors/issues/61

问题+1，在修复版本发布前，有什么临时替代方案吗

+1 求修复

附议

附议 @leonardBang

Hi, guys. I've investigated the root cause and will fix in the next version. But I'm busy now, thus please wait a while.

HI @leonardBang, is there a temporary workaround for this? the error happens often with checkpoints on server restart or job recovery

@leonardBang could you please describe your fix or let us know a workaround

Take a look at the FAQ https://github.com/ververica/flink-cdc-connectors/wiki/FAQ#q13-the-job-reports-an-error-eventdatadeserializationexception-failed-to-deserialize-data-of-eventheaderv4--caused-by-javanetsocketexception-connection-resetwhat-should-i-do .

This FAQ  doesn't seem to solve the problem

我是在突然数据量变多的时候会出现这个问题，而且会造成数据丢失，导致我无法上生产使用，求修复

我是在突然数据量变多的时候会出现这个问题，而且会造成数据丢失，导致我无法上生产使用，求修复

数据不会丢失吧 数据是记录在msyql的binlog里面

@pycgo 有没有好的解决方法

@pycgo 有没有好的解决方法

增量跑 从失败的位点继续 这个过程可以自己写代码检查自动化


Exception with PUT Method

Hello, with POST Method all work perfectly, but with PUT Method and 2 parameters in body the server generate a Exception.
com.github.lambdaexpression.exception.HttpMediaTypeOrHttpBodyException: Request method exception :public org.springframework.http.ResponseEntity<PACKAGE.dto.PostDTO> ........controllers.PostController.editExist(int,PACKAGE.dto.PostDTO,PACKAGE.dto.UserDTO), please check the cn.kknotes.open.bean.MultiReadRequestBean filter condition ( Default ContentType or HttpMethod ERROR )
@RequestMapping(value = "/posts/{id}/edit", method = RequestMethod.PUT)
    public ResponseEntity<PostDTO> editExist(@PathVariable("id") int idPost, @RequestBodyParam(value = "post") PostDTO postDTO, @RequestBodyParam(value = "user") UserDTO currentUser) { }

Have you already set MultiReadRequestBean ?
If has multiReadRequestBean, can provide a code demo ?

RequestBodyParam default only supports POST, I would have liked to be determined by the user, the interception of RequestBodyParam. However, considering how many people have recently add PUT, it will be added to the default interception.

Newly updated 2.0.3.RELEASE has supported PUT by default

Thanks so much


Add helper function for faster squares calculation

Squares can be computed faster than just plain multiplication by not computing the same values twice. This patch adds squarification helper which is called from mbedtls_mpi_mul_mpi when A == B (turned off in config by default).
It gives about 5% speed boost for signing/verification (I've tested it on my ESP8266 for clearer results), compared to multiplication.
Also might be optimized a bit by providing an assembler version of MUL_XY macro (similar to MULADDC_CORE) for limbs' multiplication.

Automatic CI verification build not done, please verify manually.

Automatic CI verification build not done, please verify manually.

@mpg - Could you please do a design review?

Closing; we are currently planning other bignum work that will likely conflict.


Enable openjdk10-openj9 systemtests in Adopt

Enable openjdk10-openj9 systemtests for each openjdk10-openj9 build that is available. Test them manually.

Related PR automates trigger of openjdk10-openj9 systemtest in x64 linux and ppc64le linux: https://github.com/AdoptOpenJDK/openjdk-build/pull/296

openjdk10-openj9 builds are running at Adopt: https://ci.adoptopenjdk.net/view/System tests/


ARM64 Support

Leaving this here in case anyone stumbles across this issue. I was able to successfully self-host plausible on an arm64 based Oracle Compute Instance by modifying the docker-compose.
More information on the Discussion https://github.com/plausible/analytics/discussions/903#discussioncomment-2619074
In the docker-compose just replace:

bytemark/smtp with navanchauhan/smtp (I built a multi-arch version of the original docker file)
plausible/analytics with justsky/analytics (They built and pushed a multi-arch version, will not be required if plausible officially starts supporting multi-arch builds)
yandex/clickhouse-server:<IP_ADDRESS> with clickhouse/clickhouse-server:<IP_ADDRESS>

I did not open a PR because it will be only useful after plausible starts publishing official multi-arch images. https://github.com/plausible/analytics/pull/1287 A simple edit in the GitHub Actions workflow will allow for that

I would love arm64/aarch64 support as well. Then I could host Plausible on my odroid-n2.

Upgrading to clickhouse-server:<IP_ADDRESS> broke my container setup.
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/logging.xml'.
Logging warning to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
2022.04.23 16:49:27.078238 [ 1 ] {} <Warning> Context: Linux is not using fast TSC clock source. Performance can be degraded.
2022.04.23 16:49:27.133140 [ 1 ] {} <Warning> Context: Linux threads max count is too low.
2022.04.23 16:49:27.133514 [ 1 ] {} <Warning> Context: Available memory at server startup is too low (2GiB).
2022.04.23 16:49:27.808304 [ 1 ] {} <Warning> Application: Calculated checksum of the binary: F2D14FA394CECCE5AED0925B54461551. There is no information about the reference checksum.
2022.04.23 16:49:27.850562 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-server/server.crt, skipping update. errno: 2, strerror: No such file or directory
2022.04.23 16:49:27.851501 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-server/server.key, skipping update. errno: 2, strerror: No such file or directory
2022.04.23 16:49:27.868059 [ 1 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version <IP_ADDRESS> (official build))
2022.04.23 16:49:28.176159 [ 1 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version <IP_ADDRESS> (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host><IP_ADDRESS></listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2022.04.23 16:49:28.177112 [ 1 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version <IP_ADDRESS> (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host><IP_ADDRESS></listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2022.04.23 16:49:28.177829 [ 1 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version <IP_ADDRESS> (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host><IP_ADDRESS></listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2022.04.23 16:49:28.178566 [ 1 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version <IP_ADDRESS> (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host><IP_ADDRESS></listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2022.04.23 16:49:28.179170 [ 1 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version <IP_ADDRESS> (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host><IP_ADDRESS></listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2022.04.23 16:49:28.408196 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-server/server.crt, skipping update. errno: 2, strerror: No such file or directory
2022.04.23 16:49:28.408615 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-server/server.key, skipping update. errno: 2, strerror: No such file or directory
2022.04.23 16:49:28.419647 [ 1 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version <IP_ADDRESS> (official build))
2022.04.23 16:52:49.808878 [ 47 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::NetException::NetException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1b3c934c in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1b3dcdc4 in /usr/bin/clickhouse
2. Poco::Net::SocketImpl::peerAddress() @ 0x1b3df4b6 in /usr/bin/clickhouse
3. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x16ffd398 in /usr/bin/clickhouse
4. DB::HTTPServerConnection::run() @ 0x16ffbec2 in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x1b3e444f in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x1b3e68a1 in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x1b5ad2e9 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x1b5aa640 in /usr/bin/clickhouse
9. ? @ 0x7fc391a30609 in ?
10. __clone @ 0x7fc391955163 in ?
 (version <IP_ADDRESS> (official build))
2022.04.23 16:52:49.820075 [ 46 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::NetException::NetException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1b3c934c in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1b3dcdc4 in /usr/bin/clickhouse
2. Poco::Net::SocketImpl::peerAddress() @ 0x1b3df4b6 in /usr/bin/clickhouse
3. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x16ffd398 in /usr/bin/clickhouse
4. DB::HTTPServerConnection::run() @ 0x16ffbec2 in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x1b3e444f in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x1b3e68a1 in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x1b5ad2e9 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x1b5aa640 in /usr/bin/clickhouse
9. ? @ 0x7fc391a30609 in ?
10. __clone @ 0x7fc391955163 in ?
 (version <IP_ADDRESS> (official build))

Are you unable to use it? Or, are these just logged errors and warnings

I was unable to use Plausible. The main container restarted every couple of minutes.

I’ll try to recreate this by using the original clickhouse version and then upgrading on an x86_64 machine

On 24-Apr-2022, at 11:11 PM, Sebastian @.***> wrote:
I was unable to use Plausible. The main container restarted every couple of minutes.
—
Reply to this email directly, view it on GitHub https://github.com/plausible/hosting/issues/44#issuecomment-1108085007, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG7BHQAXBMD2MVLAZLRKZYLVGYSQZANCNFSM5UELSUAQ.
You are receiving this because you authored the thread.

When uppgrading you'll need to update the following config files:
https://github.com/plausible/hosting/blob/master/clickhouse/clickhouse-config.xml
https://github.com/plausible/hosting/blob/master/clickhouse/clickhouse-user-config.xml
And changethe  tags to
Other than that I don't know that could cause the issues I'm having upgrading to the latest version.

With justsky/analytics I get a [warning] OTLP exporter failed to initialize with exception :throw:{:application_either_not_started_or_not_ready, :tls_certificate_check} when running it. Any ideas?

@openscript 👋
That's a rather harmless warning and can be ignored. More info: https://github.com/open-telemetry/opentelemetry-erlang/issues/419

@SebDanielsson Just started using on my Mac M1 and I have the same issue of Plausible restarting every few minutes. Is there a solution to get this working on a Mac M1?

@abishekrsrikaanth 👋
Are the any errors in the logs? docker compose logs plausible?
You might want to try using one of arm64 container images.

@ruslandoga can you please elaborate what you mean my using arm64 container images. I just followed the docs to download the repo and run docker compose up

@abishekrsrikaanth ah, right :)
So until https://github.com/plausible/analytics/pull/2103 is merged and v1.5.0 is released, there aren't any official arm64 images for plausible/analytics. But some community members have built their own arm64 images, like OP here. You might be able to do the same.
You can also try using --platform linux/amd64 flag to make Docker run the current plausible container in emulator mode.

Thanks, let me look into this. Appreciate the help. Looking forward to the release

You can definitely run it right now by applying the changes mentioned
above. I am going to be moving from hosted Plausible to my self hosted
version over the weekend
On Sat, Apr 23, 2022 at 2:18 AM Sebastian @.***> wrote:

I would love arm64/aarch64 support as well. Then I could host Plausible on
my odroid-n2.
—
Reply to this email directly, view it on GitHub
https://github.com/plausible/hosting/issues/44#issuecomment-1107427813,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG7BHQGJEJRBFZ6NC6XCKVDVGOW5RANCNFSM5UELSUAQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

v1.5.0-rc.0 is out and it has arm64 support :)


Keeping track of which keys are down

When I click some elements, I want something different to happen if I hold down shift or ctrl, for example. However, I can't figure out how to get the events of key presses and register save in the component state.
If I add onKeyUp trigger in the root panel of the view, like:
DockPanel.onKeyDown (fun e ->
    printfn "Key Down %A" e.Key ;
    match e.Key with
    | Key.LeftShift -> keyPressState.Set { keyPressState.Current with LeftShiftDown = true }
    | Key.RightShift -> keyPressState.Set { keyPressState.Current with RightShiftDown = true }
    | Key.LeftCtrl -> keyPressState.Set { keyPressState.Current with LeftCtrlDown = true }
    | Key.RightCtrl -> keyPressState.Set { keyPressState.Current with RightCtrlDown = true }
    | Key.LeftAlt -> keyPressState.Set { keyPressState.Current with LeftAltDown = true }
    | Key.RightAlt -> keyPressState.Set { keyPressState.Current with RightAltDown = true }
    | _ -> ()
) ;
DockPanel.onKeyUp (fun e ->
    printfn "Key Up %A" e.Key ;
    match e.Key with
    | Key.LeftShift -> keyPressState.Set { keyPressState.Current with LeftShiftDown = false }
    | Key.RightShift -> keyPressState.Set { keyPressState.Current with RightShiftDown = false }
    | Key.LeftCtrl -> keyPressState.Set { keyPressState.Current with LeftCtrlDown = false }
    | Key.RightCtrl -> keyPressState.Set { keyPressState.Current with RightCtrlDown = false }
    | Key.LeftAlt -> keyPressState.Set { keyPressState.Current with LeftAltDown = false }
    | Key.RightAlt -> keyPressState.Set { keyPressState.Current with RightAltDown = false }
    | _ -> ()
) ;

this does not get called when I press a key. However, if I add the same thing under the MainWindow class as
with member this.OnKeyDown (e : KeyEventArgs) =
            printfn "Key Down %A" e.Key ;
            ...

this does get called at every key-press, but the window class does not have access to the state, which lives in the components.
Is there some way of doing this?

I believe the problem you're facing is due to the fact that you didn't have the DockPanel focused. If you click on the panel, can you still repro this issue?
I just modified the ContactBook example to add a onKeyDown handler to the main DockPanel and it does not work initially, however once you click something inside of that panel (even if it's a blank space) it starts to work:

Whether this should be the desired behavior or not is not up to FuncUI since we're just wrapping Avalonia's properties, so you can maybe raise this on the upstream repo if you'd like to discuss this.
One workaround that you can use if you need a global handler is to create a global store like the one we have in the ContactBook example: Types.fs. With this in place you can call Set on the store inside the OnKeyDown handler in the MainWindow and then in your component do a ctx.usePassedRead Store.shared.Keys to retrieve the value.

Good tips. Thanks. 😊
tors 3 nov. 2022 kl. 00:01 skrev Fran González @.***>:

I believe the problem you're facing is due to the fact that you didn't
have the DockPanel focused. If you click on the panel, can you still repro
this issue?
I just modified the ContactBook example to add a onKeyDown handler to the
main DockPanel and it does not work initially, however once you click
something inside of that panel (even if it's a blank space) it starts to
work:
[image: image]
https://user-images.githubusercontent.com/6024783/199617908-4d58a135-ad63-4044-a1c6-5d544d2f5292.png
Whether this should be the desired behavior or not is not up to FuncUI
since we're just wrapping Avalonia's properties, so you can maybe raise
this on the upstream repo if you'd like to discuss this.
One workaround that you can use if you need a global handler is to create
a global store like the one we have in the ContactBook example: Types.fs
https://github.com/fsprojects/Avalonia.FuncUI/blob/a0ca5bdd1fd288e46bc66c64774865a4051851bf/src/Examples/Component Examples/Examples.ContactBook/Types.fs#L34.
With this in place you can call Set on the store inside the OnKeyDown
handler in the MainWindow and then in your component do a ctx.usePassedRead
Store.shared.Keys to retrieve the value.
—
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Avalonia.FuncUI/issues/226#issuecomment-1301471911,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABXEW34V2PVZLC6PNPEQQXDWGLXGJANCNFSM6AAAAAARRCJND4
.
You are receiving this because you authored the thread.Message ID:
@.***>


test: try to dial to remote host upon each SSH connection

Given that the golang SSH library provides no timeouts for establishing a new session with an existing client, to get around this, try to dial to the same remote host upon each time we create a new session. Given that we don't want to leak clients / connections, close this new connection if the connection was successful.
This is added to try to avoid the CI getting completely stuck when SSH connectivity is lost to VMs, which has been occurring during the direct node routes CI test.
Signed-off by: Ian Vernon<EMAIL_ADDRESS>

This change is

test-me-please

Coverage decreased (-0.03%) to 42.093% when pulling 3412deaf8a119c476405673892d19933755bf0c6 on pr/ianvernon/do-not-reuse-sshClient-per-session into ad057a5ba4e893c94bff8dc684b2da373d70163c on master.

test-me-please


cLabs ODIS signers' health should be visible on a Dashboard

https://console.cloud.google.com/monitoring/dashboards/custom/7d9c15f3-fcd5-45b5-b4c9-117d432fb518?project=celo-pgpnp-mainnet&timeDomain=1h

https://console.cloud.google.com/monitoring/dashboards/custom/7d9c15f3-fcd5-45b5-b4c9-117d432fb518?project=celo-pgpnp-mainnet&timeDomain=1h


fatal error: concurrent map read and map write

The error happened with the command team namespace file size.
But no trace available

Add crash report feature


Replace Doc2 with Stream[D2].

Rather than use a linked data type, this approach uses Scala's Stream
type to achieve laziness in the tail. This optimizes the best() method
by ensuring we only do enough work in fits() to determine if we need
to wrap a line or not.
This change ends up making the code simpler in a few places.

Ah, dammit, looks like @johnynek beat me to the punch. Time to review his PR.

Did you see #1 ? Which I cc ed you on?
On Mon, Mar 27, 2017 at 19:20 Erik Osheim<EMAIL_ADDRESS>wrote:

Rather than use a linked data type, this approach uses Scala's Stream
type to achieve laziness in the tail. This optimizes the best() method
by ensuring we only do enough work in fits() to determine if we need
to wrap a line or not.
This change ends up making the code simpler in a few places.
You can view, comment on, or merge this pull request online at:
https://github.com/johnynek/paiges/pull/2
Commit Summary

Replace Doc2 with Stream[D2].

File Changes

M build.sbt
https://github.com/johnynek/paiges/pull/2/files#diff-0 (1)
M src/main/scala/com/github/johnynek/paiges/Doc.scala
https://github.com/johnynek/paiges/pull/2/files#diff-1 (142)
M src/test/scala/com/github/johnynek/paiges/PaigesTest.scala
https://github.com/johnynek/paiges/pull/2/files#diff-2 (16)

Patch Links:

https://github.com/johnynek/paiges/pull/2.patch
https://github.com/johnynek/paiges/pull/2.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/johnynek/paiges/pull/2, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEJdumQ2Xe6XsrCSy35_6yhfcJdtZ9Iks5rqJihgaJpZM4MrJq-
.

Please send more PRs!


Add files via upload

Description

Translate translation.json to Persian

Thanks for the contribution, but when introducing a new language, we are only considering PRs for translations of the entire documentation site at this point in time. Let us know if this would be interesting for you and we can reach out outside of github.

Thank you for the prompt response and for considering my contribution.
I understand the requirement for a complete translation of the entire documentation site. I am very interested in undertaking this task and ensuring that the entire documentation is accurately translated into Persian.
Please let me know the next steps and any specific guidelines or resources I should follow for this project. I am eager to get started and provide a comprehensive translation that will benefit the Persian-speaking community.
Additionally, I have sent an email to discuss this further. Looking forward to your response.
Thank you once again for this opportunity. I look forward to collaborating with you.


Fix javadoc on Replicator

What
Fix javadoc on the Replicator class.
How
The documentation is misleading because it implies the API is on the Database object rather than
the CloudantClient object.
Testing
N/A
Issues
#424 - user was confused by documentation

Needs https://github.com/cloudant/java-cloudant/pull/425/commits/61ab7c3afa117a147b2625ee25e016773a0901f8 to fix tests.

This has been rebased onto master so should go green since master has the same fix @smithsz mentioned.


Missing recent data in /data_tables/testing_data/time_series_covid19_US.csv

In both GitHub and in the repository I cloned locally, the CSV file at https://github.com/govex/COVID-19/blob/master/data_tables/testing_data/time_series_covid19_US.csv only reports through March 2nd.
It reports daily commits in the History, so I'm not sure why it's missing data.
Thank you,
Simon St.Laurent

It appears this issue is also affecting the Daily automated CRC data bot US vaccine time series update.

The US vaccine time series (https://github.com/govex/COVID-19/blob/master/data_tables/vaccine_data/us_data/time_series/vaccine_data_us_timeline.csv) has also not been updated since 3/22 (as of 3/24 afternoon EST).

Looks okay now, at least for the time_series_covid19_US file.  Thanks!


Migrate documentation for all unicode functions from scalar_functions.md to code

Is your feature request related to a problem or challenge?
As part of https://github.com/apache/datafusion/issues/12740 this ticket is to migrate all the existing documentation for the above named functions from the static documentation to code and to remove them from the static file.
Here is the current static content:

doc source: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md
rendered: https://datafusion.apache.org/user-guide/sql/scalar_functions.html
Code source: https://github.com/apache/datafusion/tree/main/datafusion/functions/src/unicode

Need Functions:
(see source)
Describe the solution you'd like
One (or more) PRs that

port the documentation for the above named functions from the static docs to code
Bonus points:  add sql examples should be added for all functions
Remove the documentation from the static docs
Run ./dev/update_function_docs.sh to update the dynamic documentation

Here are some example PRs of doing this:

https://github.com/apache/datafusion/pull/12840 from @jonathanc-n
https://github.com/apache/datafusion/pull/12775 from @Omega359
https://github.com/apache/datafusion/pull/12742

Describe alternatives you've considered
No response
Additional context
No response

Done by @Omega359  in https://github.com/apache/datafusion/pull/12775