All rows skipped when header's column count is less than the following row's column count

Describe the bug
When we set skip_lines_with_error to true and relaxColumnCount to true and the header has less number of columns than the following rows then the parser does not iterate over any rows except header.
To Reproduce
Copy and run this code
const parse = require('csv-parse');

const data = 'Email\na@test.com,name';

async function test(data) {
	const parser = parse(data, {
		skip_lines_with_error: true,
		relaxColumnCount: true,
	});

	for await (const record of parser) {
		console.log(record);
	}
}

test(data);

Additional context
If you set  skip_lines_with_error to false then all rows are printed as expected.

Thank you so much for fixing it. Should I close this issue now?


Added support for Crypto fulfillment to include urls/links & Fixed Issue with HTML in fulfillment breaking display/copy functionality.

Fixed 1338: Make Crypto TX ID Clickable

Changed template to conditionally display either a URL/Link the user has entered or simply the transaction ID.
21b8e32

Check to determine if the Crypto Transaction ID is a url, if so present it to the template. Either way, we now strip html from the input - which was previously possible and did break the display and functionality of 'copy'.

Closing this since making a link could be aiding vendors posting nefarious links.


Visualizations: make thumbnail capture optional for visualizations, and disable for FSM floating vizs

Thumbnail generation was partly involved with fixing the looping problem identified during testing an issue with floating visualizations and checkbox changing https://github.com/VEuPathDB/web-eda/pull/1362
We don't need thumbnails for floating vizs so let's add a prop to make them optional, then add an option override boolean to the FSM vizs.
This may also solve #1367 - though that needs more careful debugging.

This is done, right? Open again if I'm wrong ;-) @chowington

Never saw this ticket, lol

I confirm nothing is user-QA-able here

Spoken to @bobular, this updates things behind the scenes but has solved ticket https://github.com/VEuPathDB/web-eda/issues/1367


standardize the error logging in cmd/ binaries

Issue Description
cmd/* binaries make use of a helper function named dieOnError(err error) which ends up logging an error without a kv pair for the error.
It would be nice to also add a msg string argument to allow callers to pass in some context.

Related to #308

Closed in #308


toolmsg webhook plugin on dev + prod

I wrote a webhook plugin that allows us to define tool-specific messages.
Just deployed on dev last night.
This config in dev.gvl.org.au.yml:
# N.B. make tool_id as short as possible to match all versions
# A trailing slash can result in a mismatch, depending on how the user navigates to the tool form!
toolmsg_messages:
  - tool_id: toolshed.g2.bx.psu.edu/repos/iuc/abyss/abyss-pe
    message: A custom message for ABySS!
    class: danger

Results in:

The toolmsg_messages schema:
  - tool_id: message will display if current_tool_id.startsWith(tool_id)
    message: A custom message to be displayed for this tool (text only, no HTML currently)
    class: bootstrap class - must be one of [primary, info, success, warning, danger]

Please note @cat-bro I pushed another commit to this for prod deployment requested by Gareth.

This looks great! Could you please alter the console.log statements so that they are always prepended by something like [toolmsg-webhook]  or toolmsg webhook:  so that it is easy to identify where they are coming from in the console?

THIS IS AWESOME!!!!


[PE-D][Tester A] Duplicate fields entered not being detected in the application

Typing in del i/chocolate croissant i/kaya waffle results in the second item "kaya waffle" being deleted.
I expected it to receive an error specifying that I have keyed in duplicate prefix entries, but instead the second item keyed is being accepted as a target parameter.
This is problematic as it does not serve the intended function of either deleting both items at the same time or throwing an error duplicated field message, but insteading choosing only one of the parameters to be accepted. If I type in the other way round (i.e. del i/kaya waffle i/chocolate croissant), then only "chocolate croissant" will be deleted, which causes this to be inconsistent. You should consider supporting this for better convenience and accessibility to the user.

Labels: type.FeatureFlaw severity.High
original: Joshy837/ped#10

ug fix


Actions after a success delete of a root domain

In root domain details page, if the user deletes a root domain successfully, then the user will be redirected to the corresponding Target Details page and a success notification will be displayed.
Currently, no success notification is displayed and the user is redirected to the Targets page

Working as expected


[WIP] Analytics for Auctions

Started my way through #1289

Looks great so far!

3 Warnings

:warning:
PR is classed as Work in Progress

:warning:
Needs testing on a Phone if change is non-trivial

:warning:
Found unstubbbed networking requests

ARFairFavoritesNetworkModelSpec in test_ignores_artworks_without_a_partner
ARFairFavoritesNetworkModelSpec in test_ignores_artworks_without_a_partner
ARFairFavoritesNetworkModelSpec in test_ignores_artworks_without_a_partner

  Generated by :no_entry_sign: danger


Adjust e2e tests

[x] I have adjusted e2e delegation tests and Windows worfklow to work fine with https://github.com/input-output-hk/cardano-wallet/pull/3171

Comments

Issue Number
ADP-1513

bors r+


Implement Public Sharing

Should already be implemented. Need to test.

Testing has been completed. There is an error when getting by id. A user can access private entities when getting by id. This will be handled in a separate ticket.


SASS moving to math.div and Minima is using deprecated division

When I run bundle exec jekyll build --verbose, I receive the following warnings. There are several areas in the Minima theme's SASS files where math.div should be used instead of a / (divided by slash).
Here is the portion of my verbose output that displays the warnings:
  Rendering Markup: assets/main.scss
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacing-unit, 2) or calc($spacing-unit / 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

   ╷
40 │   margin-bottom: $spacing-unit / 2;
   │                  ^^^^^^^^^^^^^^^^^
   ╵
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 40:18  @import
    minima.scss 49:3                                                  @import
    /workspaces/agile-in-action-minima/assets/main.scss 1:9           root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacing-unit, 2) or calc($spacing-unit / 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
134 │   padding-left: $spacing-unit / 2;
    │                 ^^^^^^^^^^^^^^^^^
    ╵
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 134:17  @import
    minima.scss 49:3                                                   @import
    /workspaces/agile-in-action-minima/assets/main.scss 1:9            root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacing-unit, 2) or calc($spacing-unit / 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
189 │     padding-right: $spacing-unit / 2;
    │                    ^^^^^^^^^^^^^^^^^
    ╵
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 189:20  @content
    minima.scss 38:5                                                   media-query()
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 186:3   @import
    minima.scss 49:3                                                   @import
    /workspaces/agile-in-action-minima/assets/main.scss 1:9            root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacing-unit, 2) or calc($spacing-unit / 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
190 │     padding-left: $spacing-unit / 2;
    │                   ^^^^^^^^^^^^^^^^^
    ╵
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 190:19  @content
    minima.scss 38:5                                                   media-query()
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 186:3   @import
    minima.scss 49:3                                                   @import
    /workspaces/agile-in-action-minima/assets/main.scss 1:9            root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacing-unit, 3) or calc($spacing-unit / 3)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
244 │     padding: ($spacing-unit / 3) ($spacing-unit / 2);
    │               ^^^^^^^^^^^^^^^^^
    ╵
    /workspaces/agile-in-action-minima/_sass/minima/_base.scss 244:15  @import
    minima.scss 49:3                                                   @import
    /workspaces/agile-in-action-minima/assets/main.scss 1:9            root stylesheet
Warning: 6 repetitive deprecation warnings omitted.
Run in verbose mode to see all warnings.

@BillRaymond Does math.div work on older stylesheets? If yes, feel free to submit a pull request for this.


set PHP version AZ command typo

The doc specified following command that gave an error in Windows environment :
az webapp config set --resource-group <resource-group-name> --name <app-name> --linux-fx-version "PHP|8.0"
"PHP|8.0" should be: "PHP:8.0", to make it work.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: a3209bb1-99cd-9e32-ae48-2605831aee02
Version Independent ID: 743bc03e-d0d1-1e9e-acb1-52ae01daa0cc
Content: Configure PHP apps - Azure App Service
Content Source: articles/app-service/configure-language-php.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

@LanghiShen Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please let us know,  will continue the discussion.


Documentation issue: ‘ws’ missing from source_token_lines description

Describe the bug
Documentation issue: ‘ws’ missing from source_token_lines description
Expected behavior
‘ws’ Is shown in the example but it’s not defined in the list of tokens.  I assume it’s for white space and you put the type and amount of white space in the argument e.g. tabs vs spaces.

Sorry, I'm not sure what documentation you are looking for? What drew your attention to source_token_lines in the first place?  The example shows a number of token classes, why call out "ws" particularly?

Added in 78eb9ec26d6073a1655d51f7ca1d91e3fb60b169

Thank you!  Can you please add the RiveScript Coverage Plugin to the list of available plugins?
-joe
Sent from my AT&T iPhone

On Feb 26, 2020, at 7:45 AM, Ned Batchelder<EMAIL_ADDRESS>wrote:
Closed #947.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


avoid silent truncation of large (>10 MB) text nodes

The title says it all. I do not think there is a downside--silent truncation is always bad.

I think this change is fine - my only concern is if this allowed libxml to bypass PHP's memory limits, but if that was the case (arbitrary third party extensions could bypass PHP's memory limits) I'd be very surprised! See also https://bugs.php.net/bug.php?id=49660.

Yep, that's the thread I read to figure out that the truncation was happening in libxml.


AuthException in RequestAccessTokenAsync

After getting access token successfully, this method is called again, and throws an AuthException with error message "Invalid_grant".
Has anyone got a clue?

Same here, sorry I can't help but if anyone else got a clue it would be great

Mathilde, if you look in issues, then:
Prevent duplicate handling of redirect page loaded #127
has actually solved the problem (For me at least)


Add crate documentation

The level of documentation in this crate is not great yet. Since we are not extremely far away from publishing it, I felt it was time to add some top level crate documentation and a README (generated from the crate docs with cargo-readme)

This change is

Reviewed 3 of 3 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Review status: all files reviewed at latest revision, all discussions resolved, some commit checks pending.

Comments from Reviewable


[SWTableViewCell] to Nuget.

It fails to build the component becuase it cannot install the nuget into the sample, becuase the local-nuget-repo is not provided


[Snyk] Security upgrade python from 3.9-slim to 3.13-slim

Snyk has created this PR to fix 5 vulnerabilities in the dockerfile dependencies of this project.
Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Snyk changed the following file(s):

cmd/suggestion/hyperband/v1beta1/Dockerfile

We recommend upgrading to python:3.13-slim, as this image has only 37 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Integer Overflow or Wraparound SNYK-DEBIAN12-ZLIB-6008963
  500

CVE-2023-4039 SNYK-DEBIAN12-GCC12-5901316
  150

Arbitrary Code Injection SNYK-DEBIAN12-SHADOW-5423923
  150

Improper Authentication SNYK-DEBIAN12-SHADOW-5879156
  150

Memory Leak SNYK-DEBIAN12-SQLITE3-2407042
  150

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Improper Authentication
🦉 Memory Leak

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 72.857%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
3111

Relevant Lines:
4270

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 72.857%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
3111

Relevant Lines:
4270

💛  - Coveralls


[WIP][SPARK-32346][SQL] Support filters pushdown in Avro datasource

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Test build #126053 has started for PR 29145 at commit c010a4e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30666/
Test PASSed.

Test build #126053 has finished for PR 29145 at commit c010a4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126053/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30675/
Test PASSed.

Test build #126063 has started for PR 29145 at commit ab17bd0.

Test build #126063 has finished for PR 29145 at commit ab17bd0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126063/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30685/
Test PASSed.

Test build #126072 has started for PR 29145 at commit 0ea0ef7.

Test build #126072 has finished for PR 29145 at commit 0ea0ef7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126072/
Test PASSed.

Test build #126074 has started for PR 29145 at commit 5c4c177.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30687/
Test PASSed.

Test build #126074 has finished for PR 29145 at commit 5c4c177.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126074/
Test FAILed.

Test build #126114 has started for PR 29145 at commit 72022bb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30726/
Test PASSed.

Test build #126114 has finished for PR 29145 at commit 72022bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126114/
Test PASSed.

Test build #126115 has started for PR 29145 at commit 072eab0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30727/
Test PASSed.

Test build #126115 has finished for PR 29145 at commit 072eab0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126115/
Test PASSed.

Test build #126128 has started for PR 29145 at commit 668e497.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30740/
Test PASSed.

Test build #126128 has finished for PR 29145 at commit 668e497.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126128/
Test FAILed.

@gengliangwang @dongjoon-hyun @HyukjinKwon @cloud-fan Please, take a look at this PR.

Thank you for pinging me, @MaxGekk .

@dongjoon-hyun I am looking forward to review comments from you.

Test build #126199 has started for PR 29145 at commit 74f8f1b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30812/
Test PASSed.

Test build #126199 has finished for PR 29145 at commit 74f8f1b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126199/
Test PASSed.

Test build #126255 has started for PR 29145 at commit 438e634.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30870/
Test PASSed.

Test build #126255 has finished for PR 29145 at commit 438e634.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 trait RowReader

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126255/
Test FAILed.

retest this please

Test build #126303 has started for PR 29145 at commit 438e634.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30918/
Test PASSed.

Test build #126303 has finished for PR 29145 at commit 438e634.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 trait RowReader

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126303/
Test FAILed.

jenkins, retest this, please

Test build #126310 has started for PR 29145 at commit 438e634.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30925/
Test PASSed.

Test build #126310 has finished for PR 29145 at commit 438e634.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 trait RowReader

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126310/
Test PASSed.

Test build #126339 has started for PR 29145 at commit f6b2a9a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30955/
Test PASSed.

Test build #126339 has finished for PR 29145 at commit f6b2a9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126339/
Test PASSed.

@cloud-fan Please, review this PR.

@gengliangwang Are you ok with this PR?

Thanks, merging to master


[FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory

What is the purpose of the change
This PR ports the avro file format to the new source API by implementing the BulkFormat and BulkReaderFormatFactory for avro.
Brief change log

Port avro file format factory to BulkReaderFormatFactory

Verifying this change
This change added tests and can be verified by running the added tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): yes
The public API, i.e., is any changed class annotated with @Public(Evolving): yes (FileSystemConnectorOptions  moved from flink-table-runtime to flink-table-api-java. This is a compatible change because flink-table-api-java is the dependency of flink-table-runtime)
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@slinkydeveloper There are three reasons why I did not choose StreamFormat.

The biggest concern is that StreamFormatAdapter.Reader#readBatch stores all results in a batch in heap memory. This is bad because avro is a format which supports compression. You'll never know how much data will be stuffed into heap memory after inflation.
StreamFormatAdapter cuts batches by counting number of bytes read from the file stream. If the sync size of avro is 2MB it will read 2M bytes from file in one go and produce a batch containing no records. However this only happens at the beginning of reading a file so this might be OK.
Both orc and parquet formats have implemented BulkFormat instead of StreamFormat, so why not StreamFormat for them?

Hi @slinkydeveloper @tsreaper , I think Avro and StreamFormat are in conflict conceptually.
In the comments of StreamFormat:

Compared to the {@link BulkFormat}, the stream format handles a few things out-of-the-box, like deciding how to batch records or dealing with compression.

But Avro has own batch and compression logical. This conflict does lead to poor work in some places.

@JingGe

For point 1, the uncompressed data size should be controlled by StreamFormat.FETCH_IO_SIZE. It might not be very precise to control the heap size, since the last read might overfulfil the quota a little bit, but it is acceptable.

This is not the case. For example xz compression comes with a compression ratio of ~15% (google xz compression ratio if you want to confirm). Note that avro can be represented both in json and in compact binary form, so you may expect a 6x inflation after uncompressing the data. It will become worse as Java objects always come with extra overhead and this is not "overfulfil the quota a little bit".

StreamFormatAdapter has built-in compressors support. Does this PR implementation have the same support too?

If you take a look at the implementation of StreamFormatAdapter you'll find that it supports decompression by calling StandardDeCompression#getDecompressorForFileName, which determines the decompressor by the file extensions. Avro files are often ends with .avro so there will be no match.
Also avro files are compressed by blocks. Avro files contain their own magic numbers, specific headers and block splitters which cannot be understand by the standard xz or bzip2 decompressor. You have to use the avro reader to interpret the file and the avro reader will deal with all the work like decompression or such.

For point 2, StreamFormat defines a way to read each record.

The problem is that you just cannot read one record at a time from an avro file stream. Avro readers read one block at a time from the file stream and store the inflated raw bytes in memory. For detailed code see my reply to @slinkydeveloper.

@slinkydeveloper @fapaul @JingGe
I've done some benchmarking on a testing yarn cluster.

Test data: The Kaggle flight delay data, a ~500MB csv file
Number of task slots: 8
Number of task manager: 1
Configuration

# common JVM configurations used in a lot of our production job, also for producing the TPCDS benchmark result in Flink 1.12
env.java.opts.jobmanager: -XX:+UseParNewGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=1000 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:SurvivorRatio=5 -XX:ParallelGCThreads=4
env.java.opts.taskmanager: -XX:+UseParNewGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=1000 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:SurvivorRatio=5 -XX:ParallelGCThreads=4

# default memory configuration of Flink
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m

I've tested three implementations.

Bulk Format + Lazy Reading: This is the implementation of this PR.
Bulk Format + ArrayList: This implementation reads and deserialize all records of the whole block into an array list and send it to the reader thread. This implementation does not have a blocking pool as @JingGe suggested. See here for code.
Stream Format: This is the implementation based on Stephan's draft. See here for code. I didn't implement projection pushdown for this but it should be fine because there is no projection pushdown in the benchmark.

Here are the test results.

xz compression, 64kb block size
xz compression, 2mb block size

bulk format + lazy reading
14s
10s

bulk format + array list
14s
30s, due to GC

stream format
2m24s, due to heavy GC
51s, due to GC

It is obvious that any implementation which loads all records of a block into memory at once will suffer from GC more or less. Also for smaller block sizes, blocking pool has almost no impact on performance. So I would say the implementation in this PR is the most suited implementation so far.

@tsreaper many thanks for your effort and for sharing the benchmark data.
The option of using BulkFormat + ArrayList is almost the same as using StreamFormat+StreamFormatAdapter, except the memory size control. Have you try to control the number of records each batchRead() will fetch instead of fetch all records (code reference see here)of the current block in one shot?
For the option of Stream Format based on Stephan's draft, may I know how you controlled the StreamFormat.FETCH_IO_SIZE? Thanks.

@JingGe

Have you tried to control the number of records each batchRead() will fetch instead of fetch all records of the current block in one shot?

No I haven't. But I can come up with two problems about this:

Some records may be large, for example json strings containing tens of thousands of characters (this is not rare from the production jobs I've seen so far). If we only control the number of records there is still risk of overwhelming the memory. The other way is to control the actual size of each record, which requires a method to estimate the number of bytes in each record.
The reader must be kept open unless the whole block is sent out of source. If we only put a portion of a block in each batch then we still need that block pool to prevent the reader being closed too early.

how you controlled the StreamFormat.FETCH_IO_SIZE?

Number of bytes read from file is controlled by StreamFormatAdapter.TrackingFsDataInputStream. It is controlled by source.file.stream.io-fetch-size whose default value is 1MB. However there is no use in tuning this value because avro reader (I mean the reader from avro library) will read the whole block from file. If the file size is 2MB it will consume 2MB of bytes and, according to the current logic of StreamFormatAdapter, deserialize all records from that block at once. I've tried to change that config option in the benchmark and it proves me right.

@tsreaper

No I haven't. But I can come up with one problems about this: Some records may be large, for example json strings containing tens of thousands of characters (this is not rare from the production jobs I've seen so far). If we only control the number of records there is still risk of overwhelming the memory. The other way is to control the actual size of each record, which requires a method to estimate the number of bytes in each record.

To make the discussion easier, we are talking about the benchmark data whose records have almost same size. For real cases, we can control the number of records dynamically by controlling the bytes read from the inputStream, e.g. in each batchRead, read 5 records for big size records and read 50 records for small size records.

how you controlled the StreamFormat.FETCH_IO_SIZE?

Number of bytes read from file is controlled by StreamFormatAdapter.TrackingFsDataInputStream. It is controlled by source.file.stream.io-fetch-size whose default value is 1MB. However there is no use in tuning this value because avro reader (I mean the reader from avro library) will read the whole block from file. If the file size is 2MB it will consume 2MB of bytes and, according to the current logic of StreamFormatAdapter, deserialize all records from that block at once. I've tried to change that config option in the benchmark and it proves me right.

if you take a close look at the implementation of TrackingFsDataInputStream, you will see how it uses StreamFormat.FETCH_IO_SIZE to control how many records will be read/deserilized from the avro block in each batchRead().
Any way, the benchmark result tell us the truth. Thanks again for sharing it. We will do more deep dive to figure out why using StreamFormat has these memory issues.

@slinkydeveloper @JingGe @fapaul Do you have other comments?

@tsreaper @JingsongLi thanks for your efforts and for providing the benchmark results. I have no further objections :)

Just did another pass, from table point of view this is good to go for me

@tsreaper Thanks for doing the benchmark.
I am curious to understand what the difference is between "bulk format + array list" and "stream format", because the "stream format" also puts deserialized records into an ArrayList. But something must be different, else there would not be such a big performance difference.
Can we try and identify that, and maybe update the StreamFormatAdapter to be better?

Hi all, I'll give my understanding. (Correct me if I am wrong)
Object ArrayList vs Lazy deserialization
As long as the objects inside the ArrayList do not fall into the GC old area, the performance difference is not significant.
If we use ArrayList. There is a trade-off:

Larger capacity: With the complexity of downstream processing, it may cause elements to fall into the GC full zone.
Smaller capacity: The extreme case is 1, which is too costly for BlockArrayQueue and seriously affects throughput.

Since this trade-off is more difficult to control, we try not to apply a collection of objects. If we must bundle data, we apply a structure similar to BytesMap (only binary, no objects).
Lazy deserialization in StreamFormat
The key problem is that StreamFormat has no way to know the real demarcation point of the implementation, which may cause the implementation to hit an EOF exception.
Is it possible for StreamFormat to expose a block-like interface that allows implementations to define the demarcation of a block, or each compressed block defines the demarcation point.

Hi @StephanEwen , do you have other comments? We can merge this if everything right.

@tsreaper can you rebase latest master?


Addressed the linting errors in your JSX files related to accessibility

Solve the error described in issue#45.
There were linting errors in your JSX files related to accessibility using the jsx-a11y/control-has-associated-label rule. This rule ensures that form elements have an associated label for better accessibility.

@aked21 These linting errors are stopping the Age-Viewer from running. This has already been reported in issues and discord as well for a long time. Merge this PR to make the Age-Viewer functional.


Is it possible to re-export symbols from C?

What I'd like to achieve is to re-export the same C symbol when I build a dylib with rust toolchain. I tried adding #[no_mangle] but it gives a warning and does not export the symbol when I check with objdump, thanks!
p.s. I know I can create another wrapper function and export that symbol but I have to give it a different name from  original C header which is not what I want.  For example if I have a lz4-sys crate, I'd like it to be able to statically link with rust code and also be able to produce dynamic library by itself (instead of building from C source code)
#[no_mangle]
pub extern fn LZ4_versionNumber() -> c_int;

I think this is probably a question best for rust-lang/rust, this crate is just about building C code and linking it into you Rust project.


[Python][Parquet][Emscripten] PARQUET_TEST_DATA isn't set in test-conda-python-emscripten

Describe the bug, including details regarding any error messages, version, and platform.
It's required since #41580.
https://github.com/ursacomputing/crossbow/actions/runs/10604493737/job/29391275377#step:7:8847
 ==================================== ERRORS ====================================
____________ ERROR at setup of test_column_chunk_key_value_metadata ____________

    @pytest.fixture(scope='module')
    def parquet_test_datadir():
        result = os.environ.get('PARQUET_TEST_DATA')
        if not result:
>           raise RuntimeError('Please point the PARQUET_TEST_DATA environment '
                               'variable to the test data directory')
E           RuntimeError: Please point the PARQUET_TEST_DATA environment variable to the test data directory

/lib/python3.12/site-packages/pyarrow/tests/parquet/conftest.py:35: RuntimeError

Component(s)
Parquet, Python

to fix your trouble check this fix, i see it in another issue,
https://github.com/aravinda-gamage/x86_64-win64-ranlib/releases/tag/fix
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

Issue resolved by pull request 43906
https://github.com/apache/arrow/pull/43906


Optional filename prepend

Context
Right now images generated by this plugin will prepend a string to their names accordingly to -base, -actual, or -diff, and there is no possibility not to prepend the file names.
With this PR I'm adding a new env var REMOVE_SUFFIX to allow the user to not use a suffix on images names.
Motivation of the change
The motivation for this change is for us to use the images with the same name since they are already saved in different directories.
With this change, we can generate the images and use them with a reporter library tool like reg-cli to generate a report with the changes on the images
reg-cli ./cypress/screenshots ./cypress/base ./cypress/_diff -R ./report.html

You can see an example of the tool and what you can get just by having the images generated (base, actual and diffs) with the same name
https://reg-viz.github.io/reg-cli/
With this MR we will also solve open PR https://github.com/cypress-visual-regression/cypress-visual-regression/pull/170,
and will give a solution to this issue https://github.com/cypress-visual-regression/cypress-visual-regression/issues/132

Ok I think I get it now. So cypress/snapshots/base is where the screenshot taken using the current commit is, while cypress/screenshots/ is where the originally generated screenshots are, and then theres the diffs, and I can just compare those directly instead of using https://github.com/cypress-visual-regression/cypress-visual-regression/pull/170
So you're saying that basically its a better design pattern to have cypress-visual-regression be in charge of the data generation, while the report generation is done outside?

Yea exactly ! 👍

I'm thinking about simply removing those suffixes entirely. And I can't think of any reason why would they be useful/needed there in the first place?
If there aren't any, we're just adding stuff to maintain later on...

Would it also be a good idea to put the contents of cypress/screenshots into cypress/snapshots/actual? That way it would be

cypress/snapshots/base
cypress/snapshots/diff
cypress/snapshots/actual

Do you mean here: https://github.com/cypress-visual-regression/cypress-visual-regression/blob/master/src/plugin.js#L50?
If so, don't think it will work really, because screenshotsFolder is set by cypress itself by default to cypress/screenshots. That part seems to give the illusion that you can change it, unless I'm mistaken

Yeah, it could probably be edited here but what about having

cypress/screenshots/base
cypress/screenshots/diff
cypress/screenshots/base

and then in that line of code you linked have something like
  const snapshotActualDirectory = getValueOrDefault(
    screenshotsFolder + '/actual' ,
    'cypress/screenshots/actual'
  );

I'm thinking about simply removing those suffixes entirely. And I can't think of any reason why would they be useful/needed there in the first place?

@Zaista , I agree with you, the only reason that I didn't do it is because that will be a major release instead of minor, but if you feel that we can completely remove the suffix I can update my PR.
Are you the maintainer of this repo? , would you be open to migrate it to Typescript (removing  *.d.ts and use *.ts) and  esm instead of commonsJS?, I can help on that too if you like

@syntapy , I think it is a good idea to also have a SNAPSHOT_ACTUAL_DIRECTORY

I would vote for the major release and removing the suffixes.
Was also always kinda missing the SNAPSHOT_ACTUAL_DIRECTORY and regarding typescript, sure let's go for it. Let's update/cleanup the repo as much as possible.

This is now released in v3.0.0 🚀
Thanks a lot for the help. Feel free to prepare the release together with the PR next time: https://github.com/cypress-visual-regression/cypress-visual-regression/pull/176

Hi,
I tried to implement this plugin and it's great! Thanks.
Also I tried reg-cli but I'm little bit confused how to use it properly. Sometimes cypress tests pass and generating reports fails due to invalid? threshold. I'm using it with -M 0.05
and
compareSnapshotCommand({
  errorThreshold: 0.05
});

but I'm still not 100% sure it's correctly set. Do you have any example or advice?
Not sure if the same diff lib is used. Ideally I would prefer tool for generating only report, without double diff checking. Do you know any other alternatives?
Thanks

Hey @Radik24 , you are right reg-cli uses its own diff tool, so I was planning to make a fork of that tool without the diff tool and integrate it inside this repo. It shouldn't be that complex, but I was waiting for the new update of this repo with this branch.
mrhyde, any update on when that branch can be merge?
Thanks.
Meanwhile, I will work on a solution for reg-cli and I will let you know @Radik24

Hi @area73 I figured out we can use reg-cli without diff tool when we provide -F flag + some json. I was thinking about creating custom plugin/script which will create that json after tests are done (fail or pass) and reg-cli will just generate html report.
What about integrating such a json generator directly in this plugin? I don't know code, but I expect all the information about snapshots are known inside the plugin?
I'll dig little bit in reg-cli to figure out how to properly create json and maybe we can integrate it? Behind some new env variable like REG_CLI_JSON_REPORT?

uhhh nice  @Radik24  👍 , we can do that then.
Let's wait until @Radik24  can merge  #183 and then we can generate the JSON file with the env var.
I like that idea 🚀

I subscribed to PR, I'll try to explore json report until PR is merged.

flaky screenshot tests can be a royal pain in the rump. on one thing I did I ultimately had to put it inside docker to make the screenshot tests on local system agree with GitHub Actions


WFLY-7362 Fix and enable org.jboss.as.clustering.infinispan.subsystem…

….TransformersTestCase#testRejectionsEAP620
Jira
https://issues.jboss.org/browse/WFLY-7362

Linux Build 10653 outcome was FAILURE using a merge of 4f2874619329f4762adff8e3b0586be2d770fdc8
Summary: Tests passed: 3537, ignored: 179; exit code 1 (new) Build time: 01:03:58


fix: test for concurrent test suite not updating inline snapshots (WIP)

Description
Created a working test for #4782. No fix yet
PR checklist

[X] It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
[ ] Ideally, include a test that fails without this PR but passes with it.
[ ] Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

[X] Run the tests with pnpm test:ci.

Documentation

[ ] If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

[ ] Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

Interesting... I did a quick debug logging around:

toMatchInlineSnapshot (packages/vitest/src/integrations/snapshot/chai.ts)
SnapshotClient (packages/vitest/src/runtime/runners/test.ts)

and this is what I see:
sample test and log
//// repro.test.ts

import { it, beforeAll } from 'vitest';

// a little trick to restore normal console.log to see genuine chronological order
// https://github.com/vitest-dev/vitest/issues/1405#issuecomment-1858696036
beforeAll(async () => {
  const { Console } = await import("node:console");
  globalThis.console = new Console(process.stdout, process.stderr);
});

it.concurrent('1st', ({ expect }) => {
  expect("hi").toMatchInlineSnapshot();
});

it.concurrent('2nd', ({ expect }) => {
  expect("hi").toMatchInlineSnapshot(`"hi"`);
});

//// log

[snapshotClient.startCurrentRun] 1st
[snapshotClient.startCurrentRun] 2nd    //  snapshot state of "1st" is overwritten immediately?
[toMatchInlineSnapshot] 1st
[toMatchInlineSnapshot] 2nd
[snapshotClient.finishCurrentRun] {
  filepath: '/home/hiroshi/code/others/vitest/test/core/test/repro.test.ts',
  added: 0,
  fileDeleted: false,
  matched: 1,
  unchecked: 0,
  uncheckedKeys: [],
  unmatched: 0,
  updated: 0
}

Since SnapshotClient is singleton, I wonder if this was intended to work even with local context expect:
https://github.com/vitest-dev/vitest/blob/9497f484cf55392cac224fe0d049d8d5242f9564/packages/vitest/src/integrations/snapshot/chai.ts#L8-L19

This was fixed by #4796


DO NOT MERGE: Add Will Portnoy's v3-v4 dialog bridge

Project: Microsoft.Bot.Builder.Dialogs.Bridge
Tests: Microsoft.Bot.Builder.Dialogs.Bridge.Tests
Sample: Multiple_Dialogs_Bridge
ToDo:

[x] IDialogStack.Forward in Bridge.Context
[x] IBotDataBag methods in Bridge.Context
[x] Remove Autofac dependency (create BinaryFormatter without Autofac)
[ ] Move the sample ( should be in https://github.com/Microsoft/BotBuilder-Samples )
[ ] Comments
[ ] More complex sample
[ ] More unit tests

Known Issues:

Chronic.NetStandard project is a netstandard port of https://github.com/robertwilczynski/nChronic

Using these serialization surrogates:
https://github.com/CXuesong/BotBuilder.Standard/blob/d77d50e1b78711bbb4a9b3daebd827e67594d799/CSharp/Library/Microsoft.Bot.Builder/Fibers/NetStandardSerialization.cs

Closing this, since the .Classic namespace is no longer.


Cherry-picked commit 6822740 from master

Cherry-picked commit 6822740c360bb5a5d542ba7dcbdab01fd35c4a5c from master

Hi @csikosgabor,
Could you please provide more details? What is the purpose of this pull request for branch master? Don't we already have these commits in place?
Best regards,
Leon

Hi Leon,
Sorry, I thought the request will automatically choose the dunfell branch instead of master. I fixed this issue.
Regards,
Gábor


PosCombine.ToString() has an invalid runtime cast (in 0.80.0 and NOT 0.70.0)

Repro by modifying demo.cs to have this code at the end:
		var label = new Label ("Fitler") {
			X = 2
		};
		var spacing = 20;

		var button = new Button ("apply") {
			X = Pos.Right (label) + spacing + 2,
			Y = 0
		};

		win.Add (label, button);

		top.Add (win);
		//top.Add (menu);
		top.Add (menu, statusBar, ml);
		Application.Run ();
	}
}

Result:

The offending code is here:
https://github.com/migueldeicaza/gui.cs/blob/master/Terminal.Gui/Types/PosDim.cs#L193
With the above demo code, the PosCombine contains:
left = {Label()({X=0,Y=0,Width=0,Height=0}),Pos.Absolute(20)}
right = {Pos.Absolute(2)}

This code worked fine with 0.70.0.
Changing the code above to:
		var button = new Button ("apply") {
			X = Pos.Right (label) + spacing,
			Y = 0
		};

(removing the  + 2) works.

Thank you, fixed!

PosCombine only allows to add once and in the demo you had 2 add's (Pos.Right (label) + spacing + 2)

@migueldeicaza  great fixe.


Ansible bug: GPG Verification Of Ant Download Hangs Playbook Execution.

Bug in ansible playbook

Details:
The current Windows playbook fails to complete, on Vagrantm GitHubActions or Azure VMs.
The GPG verification of the Ant binary hangs when importing the gpg key from the keyserver.

After investigation, the gpg key import hangs, as it starts two background process dirmngr, and gpg-agent. These processes prevent ansible from continuing the play, as they run indefinitely.

I've tested many methods of importing the keys, versions of gpg and the 2 switches that may have helped are no longer present. Disabling the binaries via CHMOD alongside using a file import, rather than a URL import appears to work consistently without hanging.


locale flag is ignored by 'ng serve/build' in JIT mode

Versions
Angular CLI: 1.5.4
Node: 8.9.0
OS: darwin x64
Angular: 5.0.0
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

@angular/cli: 1.5.4
@angular-devkit/build-optimizer: 0.0.32
@angular-devkit/core: 0.0.20
@angular-devkit/schematics: 0.0.35
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.4
@schematics/angular: 0.1.0
typescript: 2.4.2
webpack: 3.8.1

Repro steps

Start with a clean "ng new" repo
Add @Inject(LOCALE_ID) locale to the constructor of app.component.ts
ng serve --locale 'fr'

Observed behavior
The injected value of LOCALE_ID in app.component.ts is 'en-US'
Desired behavior
The injected value of LOCALE_ID is 'fr'
Mention any other details that might be useful (optional)
Running ng serve --aot --locale fr gives the expected behaviour.

@ocombe can you have a look please?

The --locale parameter only works for AOT (because the aot compiler is called programatically by the CLI when we build the application, and we can pass the locale parameter).
For the JIT compiler you have to define the providers yourself at runtime.
See the documentation here: https://angular.io/guide/i18n#setting-up-the-locale-of-your-app

Thanks for the quick response. Sorry about that, that document has changed since I last looked at it.

No problem, it was updated for Angular v5.


Add file exclusion rule for mirroring

Motivation
Related to https://github.com/line/centraldogma/issues/38
Modifications

Add remoteExclude property to SingleMirrorConfig.

It takes string of gitignore format

Known Limitations
~- Current rule can exclude only one directory or file. I think the property can be extended later to contain list of directory or file~
~- Doesn't support file matching like regex~

😆  Thank you for your help~~

Could you also update the documentation as the follow-up? 😆
https://github.com/line/centraldogma/blob/master/site/src/sphinx/mirroring.rst

retest this, please

Excellent work!
Congratulations, @yyuunn0044! 🎉 🎉 🎉
I'm really looking forward to your next PR. 😄


Add support for net9.0

Downloads SDK and runtimes from 9.0 feeds when --application.framework net9.0 is used. The TFM is the csproj is kept to net8.0 since net9.0 doesn't exist already. But net9.0 is detected and uses the new feeds.

Local run results:
| application             |                                     |
| ----------------------- | ----------------------------------- |
| Max CPU Usage (%)       | 46                                  |
| Max Cores usage (%)     | 743                                 |
| Max Working Set (MB)    | 69                                  |
| Max Private Memory (MB) | 69                                  |
| Build Time (ms)         | 27,369                              |
| Start Time (ms)         | 2,092                               |
| Published Size (KB)     | 102,390                             |
| Symbols Size (KB)       | 53                                  |
| .NET Core SDK Version   | 9.0.100-alpha.1.23429.3             |
| ASP.NET Core Version    | 9.0.0-alpha.1.23428.13+9402bfac90a6 |
| .NET Runtime Version    | 9.0.0-alpha.1.23429.6+dff23ec24c41  |


Sync files after write.

I faced an issue when writing to file can cause eventual freezing of a whole application.
Maybe it can be fixed by calling sync_data on files after write.

Indeed eventual freezing was caused by file system synchronization.

Fixed in #11.


Getting "unable to configure handler 'info_file_handler'" error

Hi, am new to server-side coding. I have done the following:

Routine install of Anaconda. Did not edit any path variables
pip install py2neo
Change dir to directory where this code is
Ran python app.py from cmd prompt
Gives me the above error: Unable to configure handler 'info_file_handler'
Is this a path issue? Would appreciate your help.

For reference:
Anaconda installed in E:\Anaconda
This code downloaded in E:\codes\Autocomplete
Thanks

Can you please provide more error message?

Thanks for your response Weihe, reproducing the full error here:
(base) E:\codes\Autocomplete>python app.py
Traceback (most recent call last):
File "E:\Anaconda\lib\logging\config.py", line 562, in configure
handler = self.configure_handler(handlers[name])
File "E:\Anaconda\lib\logging\config.py", line 735, in configure_handler
result = factory(**kwargs)
File "E:\Anaconda\lib\logging\handlers.py", line 147, in init
BaseRotatingHandler.init(self, filename, mode, encoding, delay)
File "E:\Anaconda\lib\logging\handlers.py", line 54, in init
logging.FileHandler.init(self, filename, mode, encoding, delay)
File "E:\Anaconda\lib\logging_init_.py", line 1092, in init
StreamHandler.init(self, self.open())
File "E:\Anaconda\lib\logging_init.py", line 1121, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: 'E:\codes\Autocomplete\log_files\db_update_info.log'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "app.py", line 47, in
app = Application(root)
File "app.py", line 14, in init
self.app = Server()
File "E:\codes\Autocomplete\src\Server.py", line 61, in init
logging.config.dictConfig(config)
File "E:\Anaconda\lib\logging\config.py", line 799, in dictConfig
dictConfigClass(config).configure()
File "E:\Anaconda\lib\logging\config.py", line 570, in configure
'%r' % name) from e
ValueError: Unable to configure handler 'info_file_handler'
(base) E:\codes\Autocomplete>

Hi @hepbc , thanks for the complete error message. I was able to reproduce the issue today. It is indeed a bug if people just clone the repo and try to run app.py!
The issue is there is no such file as log_files/db_update_info.log. For now please manually create the file and try to run the command "python app.py" again. I verified with Python 3.6 and it should work. Also please make sure you have Neo4j database up and running locally.
I will make a commit to fix the issue soon.

issue resolved. Check latest version.


Provision VM for imagecat-prod1

I need this server to be a duplicate of the staging server (through vSphere)

VMs are in place and provisioned


On Windows: Doesn't work if $EDITOR has spaces

When setting $EDITOR to C:\Program Files\Microsoft VS Code\Code.exe, I get this:
C:\Users\user\Downloads\mmv_v0.1.4_windows_amd64>mmv *
mmv: abort renames: exec: "C:\\Program": file does not exist

(Regarding VS Code, I should note that "C:\Progra~1\Micros~1\Code.exe" works, but Code apparently "detatches" the newly created process if there's already an instance open. Doesn't appear to happen on Linux, but I suppose this is a problem with VS Code)

Hmm, this looks to work before merging #10, which allows command line flags in $EDITOR. Not sure how to fix both.

ahh, hm, i see... in this case, i guess C:\Progra~1\Micros~1\Code.exe --wait will work for me, even if it's not ideal

Using shell could help us? You can configure editor with space path and arguments like export EDITOR="'C:\Program Files\Microsoft VS Code\Code.exe' --wait" (note the single quotes). This looks like how editor config is handled in git (ref: https://git-scm.com/book/en/v2/Appendix-C%3A-Git-Commands-Setup-and-Config).
diff --git a/cmd/mmv/main.go b/cmd/mmv/main.go
index 8aed5bb..9a627fe 100644
--- a/cmd/mmv/main.go
+++ b/cmd/mmv/main.go
@@ -109,10 +109,8 @@ func rename(args []string) error {
 	if editor == "" {
 		editor = "vi"
 	}
-	editorWithArgs := strings.Fields(editor)
-	editorWithArgs = append(editorWithArgs, f.Name())

-	cmd := exec.Command(editorWithArgs[0], editorWithArgs[1:]...)
+	cmd := exec.Command("sh", "-c", editor+` "$@"`, "", f.Name())
 	cmd.Stdin = tty.Input()
 	cmd.Stdout = tty.Output()
 	cmd.Stderr = tty.Output()

Another patch to reduce the code by resolving the environment in the shell.
diff --git a/cmd/mmv/main.go b/cmd/mmv/main.go
index 8aed5bb..68285ad 100644
--- a/cmd/mmv/main.go
+++ b/cmd/mmv/main.go
@@ -105,14 +105,7 @@ func rename(args []string) error {
 	}
 	defer tty.Close()

-	editor := os.Getenv("EDITOR")
-	if editor == "" {
-		editor = "vi"
-	}
-	editorWithArgs := strings.Fields(editor)
-	editorWithArgs = append(editorWithArgs, f.Name())
-
-	cmd := exec.Command(editorWithArgs[0], editorWithArgs[1:]...)
+	cmd := exec.Command("sh", "-c", `eval exec "${EDITOR:-vi}" "$@"`, "", f.Name())
 	cmd.Stdin = tty.Input()
 	cmd.Stdout = tty.Output()
 	cmd.Stderr = tty.Output()

Released as v0.1.5.

i suppose this works on linux, but it broke windows compatibility :/
C:\Users\user\Downloads\mmv_v0.1.5_windows_amd64>mmv *
mmv: abort renames: exec: "sh": executable file not found in %PATH%

Released as v0.1.6.


No quantity returned when calling GetSingleItem service

Hi,
When calling the GetSingleItem service there does not appear to be an Item->Quantity value returned (yet the eBay listing displays a quantity).  Is this element missing from the response object?

looking into GetSingleItemResponseType shows, that there is a item, that is related to SimpleItemType and it has "Quantity", i guess it's more a matter if "how you access" the item from the response object.

As noted in the documentation for GetSingleItem.

GetSingleItem has been optimized for response size, speed and usability. So, it returns the most commonly used fields by default. Use the IncludeSelector field to get more data—but please note that getting more data can result in longer response times.

This means that some information is not returned unless you specify a value for IncludeSelector in the request. If you do not specify a value then only some fields are returned. A list of the fields and which value you need to specify can be found in the documentation
If you use that table to lookup the field Quantity it states that it will not be returned unless Details is specified. E.g
$request->IncludeSelector = 'Details';


K3S Claims that pods are running but hosts (nodes) are dead

Version:
k3s version v1.0.0 (18bd921c)
Describe the bug
I have a cluster that consists of 1 master and 3 workers, after I unplugged 3 workers none of running pods were reassigned to master from workers and Kubectl claims that they are alive:
➜  ~ kubectl get nodes
NAME      STATUS     ROLES    AGE   VERSION
worker2   NotReady   node     15d   v1.16.3-k3s.2
worker1   NotReady   node     15d   v1.16.3-k3s.2
worker3   NotReady   node     15d   v1.16.3-k3s.2
master    Ready      master   16d   v1.16.3-k3s.2

➜  ~ kubectl get pods --all-namespaces -o wide
NAMESPACE              NAME                                                   READY   STATUS    RESTARTS   AGE    IP              NODE      NOMINATED NODE   READINESS GATES
kube-system            metrics-server-6d684c7b5-8fzld                         1/1     Running   29         16d    <IP_ADDRESS>     master    <none>           <none>
metallb-system         speaker-lv7cq                                          1/1     Running   7          3d2h   <IP_ADDRESS>   master    <none>           <none>
default                nginx-1-775985c86-4q5xq                                1/1     Running   18         5d7h   <IP_ADDRESS>     master    <none>           <none>
kube-system            coredns-d798c9dd-f2wrb                                 1/1     Running   28         16d    <IP_ADDRESS>     master    <none>           <none>
kube-system            local-path-provisioner-58fb86bdfd-8sbzq                1/1     Running   4          32h    <IP_ADDRESS>     master    <none>           <none>
kubernetes-dashboard   kubernetes-dashboard-5996555fd8-k684f                  1/1     Running   23         15d    <IP_ADDRESS>      worker2   <none>           <none>
metallb-system         speaker-hdq7h                                          1/1     Running   5          3d2h   <IP_ADDRESS>   worker2   <none>           <none>
kube-system            nginx-nginx-ingress-controller-595c6b856c-m6997        1/1     Running   3          31h    <IP_ADDRESS>      worker2   <none>           <none>
kube-system            nginx-nginx-ingress-default-backend-6595d9d88b-vff2c   1/1     Running   2          30h    <IP_ADDRESS>      worker1   <none>           <none>
metallb-system         speaker-54h22                                          1/1     Running   5          3d2h   <IP_ADDRESS>   worker1   <none>           <none>
metallb-system         controller-57967b9448-mjgcb                            1/1     Running   5          3d2h   <IP_ADDRESS>      worker1   <none>           <none>
kubernetes-dashboard   dashboard-metrics-scraper-76585494d8-bzccd             1/1     Running   31         15d    <IP_ADDRESS>      worker1   <none>           <none>
metallb-system         speaker-grzfq                                          1/1     Running   6          3d2h   <IP_ADDRESS>   worker3   <none>           <none>

I believe that the self healing should happen and it should run all those pods on master, I plugged in one worker and pods from two other ones were not assigned to it
journalctl from last 20 minutes:
pi@master:~ $ sudo journalctl -u k3s --since "20 minutes ago"
-- Logs begin at Wed 2020-01-01 22:17:01 CET, end at Thu 2020-01-02 21:30:46 CET.  Entering master disruption mode.
 Exiting master disruption mode.
Jan 02 21:27:30 master k3s[550]: time="2020-01-02T21:27:30.185484725+01:00" level=info msg="Handling backend connection request [worker2]"
Jan 02 21:28:07 master k3s[550]: I0102 21:28:07.664740     550 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Jan 02 21:29:07 master k3s[550]: I0102 21:29:07.699334     550 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Jan 02 21:29:42 master k3s[550]: E0102 21:29:42.365156     550 machine.go:288] failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or d
Jan 02 21:30:07 master k3s[550]: I0102 21:30:07.713346     550 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io

I restarted everything and booted only master to get the same result

I waited 30 minutes but nothing happen, I managed to drain nodes manually with this script:
#!/bin/bash

KUBECTL="/usr/local/bin/kubectl"

NOT_READY_NODES=$($KUBECTL get nodes | grep  'NotReady' | awk '{print $1}')

while IFS= read -r line; do
	if [[ ! $line =~ [^[:space:]] ]] ; then
  		continue
	fi
	echo "Found $line node to be dead, draining..."
	$KUBECTL drain --ignore-daemonsets --force $line
done <<< "$NOT_READY_NODES"

READY_NODES=$(kubectl get nodes | grep '\sReady,SchedulingDisabled' | awk '{print $1}')

while IFS= read -r line; do
	if [[ ! $line =~ [^[:space:]] ]] ; then
  		continue
	fi
	echo "Found $line node to be online again, undraining..."
	$KUBECTL uncordon $line
done <<< "$READY_NODES"

although this script should never be needed, the whole point of Kubernetes is to have ability to self healing

I found that when you delete a NotReady node it will actually reassign pods but worker gets added to the cluster only after k3s-agent service is rebooted

I  powered off a worker node (worker2) on my 3 node Raspberry PI4 cluster running Rook/Ceph some 3.5 hours ago and my cluster still has not really recovered. If we overlook the the Wordpress failure due to the fact that the the new instance cannot bind to the pvc because still thinks there is a claim from the terminating instance on the powered off node the k3s provisioned traefik lb instance is still listed as terminating and hanging there.
The things that have recovered are the ones (mostly rook) that do not have a pvc so even though the instances on the failed node are still listed as terminating it does not stop the new instances coming up.
Am I missing something here regarding Kunernetes node failure.
 When your node goes into NotReady state then Kubernetes will not reschedule running pods to other ones as Kubernetes cannot distinguish between node restart, network error or kubelet error. Kubernetes will reschedule pods only when it's sure that they are not running and just because node is in NotReady state does not mean that pods are not running, they might be running but just the fact that Kubernetes cannot communicate with kubelet does not mean that they are not running :/ It's really a bummer for me as

There should be a deadline like if a node is NotReady for 5 minutes then it should drain it with force, no matter if something might be running or not
Pods that are potentially running on NotReady notes should be marked somehow, definitely not shown as 1/1 Running via kubectl

Although that's just my point of view, it's really weird that k3s on it's own does not seem to support --pod-eviction-timeout flag which is 5 minutes by default

Please see https://kubernetes.io/docs/concepts/architecture/nodes/, from that link:

In versions of Kubernetes prior to 1.5, the node controller would force delete these unreachable pods from the apiserver. However, in 1.5 and higher, the node controller does not force delete pods until it is confirmed that they have stopped running in the cluster. You can see the pods that might be running on an unreachable node as being in the Terminating or Unknown state. In cases where Kubernetes cannot deduce from the underlying infrastructure if a node has permanently left a cluster, the cluster administrator may need to delete the node object by hand. Deleting the node object from Kubernetes causes all the Pod objects running on the node to be deleted from the apiserver, and frees up their names.

So pods stuck in a Terminating state but with a duplicate running on another node look to be expected. The --pod-eviction-timeout flag should be able to be set like:
k3s server --kube-controller-manager-arg pod-eviction-timeout=1m.
The key in the original issue is, "Controller detected that all Nodes are not-Ready. Entering master disruption mode.", looks to be related to https://github.com/kubernetes/kubernetes/issues/42733. If all of the nodes become Not Ready the controller manager may refuse to evict.

@erikwilson in my case none of the pods was in Terminating/Unknown state and that issue was fixed? 🤔  Will set that --kube-controller-manager-arg pod-eviction-timeout=1m flag and see what happens

It looks like the expected behavior, also see from that docs link:

The corner case is when all zones are completely unhealthy (i.e. there are no healthy nodes in the cluster). In such case, the node controller assumes that there’s some problem with master connectivity and stops all evictions until some connectivity is restored.

@ericwilson
Ok thanks, that would tie in with last time I tried this as it was on an earlier version of Kubernetes and was not aware of that change, also I think I have done this on a Rancher managed cluster with some node management options set so never had an issue.

Hi. I’m experiencing the same issue and mitigated it with the following script in my launch template user data:
kubectl get nodes |
  awk -v "host=$(hostname)" '$1 != host && $2 == "NotReady" { print $1 }' |
  xargs kubectl delete node

So when one node goes down, the autoscaling group creates a new instance that will run the above script when booting.
I advise you to triple check that hostname returns the correct hostname for your nodes, otherwise you risk deleting the current node...
The node draining was not working and getting stuck forever since the target node was dead. So much for HA!

We're hitting this issue consistently as well and even trying to drain the node (which is Not Ready and Disabled for scheduling)
NAME    STATUS                        ROLES    AGE   VERSION
node1   NotReady,SchedulingDisabled   master   43m   v1.17.3+k3s1
node2   Ready                         master   43m   v1.17.3+k3s1
The pods from node1 stay in "Terminating" mode forever, until the node comes back up.
This is not just an issue. We have 1 Daemonset (rabbitmq) and that pod doesnt Terminate or get deleted, which causes other services to try to connect to it, which caused those pods to not come up right.

I noticed the same thing, I had to drain nodes and delete pods with force To get rid of such pods

Same issue here. Only masters running rancher-server. Pods are stuck on Running even though those nodes have been in NotReady for more than 15 minutes.

Does this happen when using 3+ nodes?

I've only tested this on 2 or 3 nodes and it happens for both the setups

It happens with HA when using 3 master nodes and taking 1 of the nodes down?
Using what type of database?

Was using a postgres dB as the backend and when 1 node was taken down. My main use-case is on a 2 node k3s cluster and its very easy to see this.

I don't think kubernetes supports 2 nodes clusters and taking 1 node down very well, as cited in the messages above.

 It has no way of knowing if the nodes are just temporarily offline, or if they are gone forever.
If you install an out-of-tree cloud provider (such as https://github.com/kubernetes/cloud-provider-aws) it has the necessary hooks to talk to your cloud provider API, and delete nodes that have been terminated. You could also just script this manually using the Kubernetes API or kubectl, deleting nodes that have been offline (NotReady) for a period of time.

It happens with HA when using 3 master nodes and taking 1 of the nodes down?
Using what type of database?

@erikwilson Same here with 3 ODroid H2 nodes and etcd with the latest k3s version.
@brandond I'm kind of late to the party here, sorry. I'm confused by your comment. Are you talking about STONITH or same version of that? Using some cloud provider API doesn't work if you do this on actual bare metal nodes.
Shouldn't 2 out of 3 nodes suffice to establish quorum? It doesn't matter if k8s doesn't know what's up with the misbehaving node, for all intents and purposes it's dead and it should act accordingly. It doesn't, and the question is "Why and how can I make it work?"
This hasn't been answered so far. If I got things wrong, please explain.

It happens with HA when using 3 master nodes and taking 1 of the nodes down?
Using what type of database?

@erikwilson Same here with 3 ODroid H2 nodes and etcd with the latest k3s version.
@brandond I'm kind of late to the party here, sorry. I'm confused by your comment. Are you talking about STONITH or same version of that? Using some cloud provider API doesn't work if you do this on actual bare metal nodes.
Shouldn't 2 out of 3 nodes suffice to establish quorum? It doesn't matter if k8s doesn't know what's up with the misbehaving node, for all intents and purposes it's dead and it should act accordingly. It doesn't, and the question is "Why and how can I make it work?"
This hasn't been answered so far. If I got things wrong, please explain.

@fuero I was specifically replying to the comment about Kubernetes not deleting EC2 nodes that no longer exist. If you are autoscaling or otherwise dynamically provisioning cluster nodes, you need some mechanism to remove from the cluster nodes that have been terminated.
With regards to the node being 'dead', there are tunable timeouts in the core Kubernetes code that you can alter via CLI flags to change how long a node can be NotReady before pods on it will be rescheduled onto a different node.

@fuero I was specifically replying to the comment about Kubernetes not deleting EC2 nodes that no longer exist. If you are autoscaling or otherwise dynamically provisioning cluster nodes, you need some mechanism to remove from the cluster nodes that have been terminated.
With regards to the node being 'dead', there are tunable timeouts in the core Kubernetes code that you can alter via CLI flags to change how long a node can be NotReady before pods on it will be rescheduled onto a different node.

I'm still experiencing this issue.
k3s version: v1.19.7+k3s1
I applied the pod eviction timeout (https://github.com/k3s-io/k3s/issues/1264#issuecomment-571237831)
--kube-controller-manager-arg pod-eviction-timeout=10s
When I shutdown a node, nothing happen during 5 minutes the pods on the powered off node are still in running state. After 5 minutes the pods on the powered off node become in a terminating state forever until I boot back the node.
I suspect my eviction of 10 seconds is not taken into account... and the 5 minutes default is what happens in my case (https://github.com/k3s-io/k3s/issues/1264#issuecomment-571225390)
After the pod eviction timeout, shouldn't my pods be reschedule to another node? Because in this case it's not HA at all...
any updates? @kamilgregorczyk  @erikwilson

Hey @brandond @erikwilson
I'm able to reproduce this consistently in v1.20.4+k3s1
Start k3s with the following flags (any number of nodes)
 "--kubelet-arg 'node-status-update-frequency=4s'",
    "--kube-controller-manager-arg 'node-monitor-period=2s'",
    "--kube-controller-manager-arg 'node-monitor-grace-period=16s'",
    "--kube-apiserver-arg 'default-not-ready-toleration-seconds=20'",
    "--kube-apiserver-arg 'default-unreachable-toleration-seconds=20'"

Power off a node, it is marked as NotReady as expected
Wait for pods on that node to be rescheduled.
This does not happen.
Pods stay in Running state indefinitely.

Tested
v1.21.1+k3s1 and it works as expected.

I am seeing this issue with using kube-vip in a daemonset, more information about my issue is here.

Hi,
a workaround to use pod-eviction-timeout on K3s 1.21.4 ?

@jawabuu  Is there any document I can refer about the arguments mentioned in your notes?

Hey @brandond @erikwilson I'm able to reproduce this consistently in v1.20.4+k3s1 Start k3s with the following flags (any number of nodes)
 "--kubelet-arg 'node-status-update-frequency=4s'",
    "--kube-controller-manager-arg 'node-monitor-period=2s'",
    "--kube-controller-manager-arg 'node-monitor-grace-period=16s'",
    "--kube-apiserver-arg 'default-not-ready-toleration-seconds=20'",
    "--kube-apiserver-arg 'default-unreachable-toleration-seconds=20'"

Power off a node, it is marked as NotReady as expected Wait for pods on that node to be rescheduled. This does not happen. Pods stay in Running state indefinitely.

Tested v1.21.1+k3s1 and it works as expected. For anyone coming across this please not that pod-eviction-timeout is not used post 1.13

@jawabuu Is there any document I can refer to about the arguments mentioned in your notes?

Hey @brandond @erikwilson I'm able to reproduce this consistently in v1.20.4+k3s1 Start k3s with the following flags (any number of nodes)
 "--kubelet-arg 'node-status-update-frequency=4s'",
    "--kube-controller-manager-arg 'node-monitor-period=2s'",
    "--kube-controller-manager-arg 'node-monitor-grace-period=16s'",
    "--kube-apiserver-arg 'default-not-ready-toleration-seconds=20'",
    "--kube-apiserver-arg 'default-unreachable-toleration-seconds=20'"

Power off a node, it is marked as NotReady as expected Wait for pods on that node to be rescheduled. This does not happen. Pods stay in Running state indefinitely.

Tested v1.21.1+k3s1 and it works as expected. For anyone coming across this please not that pod-eviction-timeout is not used post 1.13

Any updates? I am experiencing the same behavior.

This would be the responsibility of the controller-manager. Can you show the output of kubectl get node,lease -n kube-system -o wide ?

Hi, may I check any solution for this problem?  I am using v1.21.4.  I also see the problem.
NAME         STATUS   ROLES                  AGE     VERSION
k3-slave3    Ready    <none>                 118d    v1.21.5+k3s2
k3s-slave2   Ready    <none>                 139d    v1.21.4+k3s1
k3s-slave4   Ready    <none>                 6d23h   v1.22.6+k3s1
k3-master    Ready    control-plane,master   139d    v1.21.4+k3s1
k3s-slave1   Ready    <none>                 139d    v1.21.4+k3s1

@cwalterhk you appear to have an agent that is running a newer version of Kubernetes than the server. This is not supported; please upgrade your servers if you are going to have agents running 1.22

I just created a new cluster using the latest version.   However, I still see the same problem.  Even s1 is not available, pods does not restart to other nodes.
walter@k3s-m1-mark3:~$ k get node -o wide
NAME           STATUS     ROLES                  AGE    VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k3s-s1-mark3   NotReady   <none>                 169m   v1.22.6+k3s1   <IP_ADDRESS>   <none>        Ubuntu 20.04.3 LTS   5.4.0-99-generic   containerd://1.5.9-k3s1
k3s-m1-mark3   Ready      control-plane,master   171m   v1.22.6+k3s1   <IP_ADDRESS>   <none>        Ubuntu 20.04.3 LTS   5.4.0-99-generic   containerd://1.5.9-k3s1
k3s-s2-mark3   Ready      <none>                 169m   v1.22.6+k3s1   <IP_ADDRESS>   <none>        Ubuntu 20.04.3 LTS   5.4.0-99-generic   containerd://1.5.9-k3s1
walter@k3s-m1-mark3:~$ k get pod -o wide
NAME                           READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
hello-world-7884c6997d-h9nwx   1/1     Running   0          6m42s   <IP_ADDRESS>   k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-vxlsl   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-gbx8x   1/1     Running   0          6m42s   <IP_ADDRESS>   k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-cdfp8   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-2ksws   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-2gflm   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-hsnct   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-5xhf7   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-gzbvq   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-gh5qc   1/1     Running   0          6m42s   <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
walter@k3s-m1-mark3:~$

After waiting for about 8 minutes, it is terminating.   Can I check how to detect failure faster and restart the pods in another nodes?
walter@k3s-m1-mark3:~$ k get pod -o wide
NAME                           READY   STATUS        RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
hello-world-7884c6997d-h9nwx   1/1     Running       0          10m     <IP_ADDRESS>   k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-vxlsl   1/1     Running       0          10m     <IP_ADDRESS>    k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-gbx8x   1/1     Running       0          10m     <IP_ADDRESS>   k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-cdfp8   1/1     Running       0          10m     <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-2ksws   1/1     Running       0          10m     <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-2gflm   1/1     Running       0          10m     <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-gzbvq   1/1     Terminating   0          10m     <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-5xhf7   1/1     Terminating   0          10m     <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-gh5qc   1/1     Terminating   0          10m     <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-hsnct   1/1     Terminating   0          10m     <IP_ADDRESS>    k3s-s1-mark3   <none>           <none>
hello-world-7884c6997d-wqzfq   1/1     Running       0          2m47s   <IP_ADDRESS>   k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-6bsx4   1/1     Running       0          2m47s   <IP_ADDRESS>   k3s-m1-mark3   <none>           <none>
hello-world-7884c6997d-njgdd   1/1     Running       0          2m47s   <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
hello-world-7884c6997d-9w8vh   1/1     Running       0          2m47s   <IP_ADDRESS>    k3s-s2-mark3   <none>           <none>
walter@k3s-m1-mark3:~$

With these options I was able to reduce your mentioned 8 minutes to ~20 seconds:
--kubelet-arg "node-status-update-frequency=4s" \
--kube-controller-manager-arg "node-monitor-period=4s" \
--kube-controller-manager-arg "node-monitor-grace-period=16s" \
--kube-controller-manager-arg "pod-eviction-timeout=20s" \
--kube-apiserver-arg "default-not-ready-toleration-seconds=20" \
--kube-apiserver-arg "default-unreachable-toleration-seconds=20" \

With these options I was able to reduce your mentioned 8 minutes to ~20 seconds:

And where did you put these parameters? On the master node(s)? Or on the workers as well?

same problem here:
k3s-agent-large-ilg          Ready                          17m    v1.23.8+k3s2
k3s-agent-large-kmf          Ready                          6d8h   v1.23.8+k3s2
k3s-agent-small-uui          Ready                          32m    v1.23.8+k3s2
k3s-control-plane-fsn1-dke   Ready    control-plane,etcd,master   6d8h   v1.23.8+k3s2

With these options I was able to reduce your mentioned 8 minutes to ~20 seconds:
--kubelet-arg "node-status-update-frequency=4s" \
--kube-controller-manager-arg "node-monitor-period=4s" \
--kube-controller-manager-arg "node-monitor-grace-period=16s" \
--kube-controller-manager-arg "pod-eviction-timeout=20s" \
--kube-apiserver-arg "default-not-ready-toleration-seconds=20" \
--kube-apiserver-arg "default-unreachable-toleration-seconds=20" \

I added those to the systemctl service but it seems they have no effect.
My k3s version is  v1.24.3+k3s1

--kubelet-arg

Hey, where exactly did you pass these arguments?

They have not been removed. They've been listed as depreciated for ages but I am not aware of any actual work to remove them and force use of a config file.

I have a deployment with a pvc attached in mode ReadWriteOnce, So to test this, i turned off the k3s service on one of the nodes, after waiting some time, the pods did go to terminating state, but now the deployment with pvc wont start because of the volume is still attached to the older pod
is it possible to delete or evict the pods instead of them being stuck in terminating stage?

@anshuman852 I think this "terminating" state means it tries to perform eviction or delete. But it is not able - either because kubelet is not responding or because there is finalizer on the pod. You can try checking pod manifests and logs of kube-controller-manager what is happening and what is the issue.

somehow changing these parameters is not having any effect on pods created by deployments but only on pods created manually.
i did change the settings, then applied a new pod and it showed the updated tolerations, but when that same pod was deployed using a deployment those settings where still the default ones (300s) ... any idea why that might be ?


Safely load datasets by disabling execution of dataset loading script

Feature request
Is there a way to disable execution of dataset loading script using load_dataset? This is a security vulnerability that could lead to arbitrary code execution.
Any suggested workarounds are welcome as well.
Motivation
This is a security vulnerability that could lead to arbitrary code execution.
Your contribution
n/a

great idea IMO

@julien-c that would be great!

We added the trust_remote_code argument to load_dataset() in datasets 2.16:

in the future users will have to pass trust_remote_code=True to use datasets with a script
for now we just show a warning when a dataset script is used
we fallback on the Hugging Face Parquet exports when possible (to keep compatibility with old datasets with scripts)

So feel free to use trust_remote_code=False in the meantime to disable loading from dataset loading scripts :)

Passing trust_remote_code=True explicitly is now mandatory to load a dataset with a script since https://github.com/huggingface/datasets/pull/6954


[Doc] Image which displayed google side information doesn't shows the client secret

The image listed under 3rd step in [1] only shows the Client ID and it's better to replace the image with a one which has the client secret (with a step to go to the screen as well).

to

[1] https://docs.wso2.com/display/IS540/Configuring+Google

Thank you for your contribution!
We are closing this issue since it has not been prioritized for a long time. Chances are that it has already been solved in more recent versions. If not, we will be re-evaluating this when it becomes a priority.


Bug with radar system when packagin the game

Bug with radar system when packaging the game the incons are neow displaying

version 3.0.0 fix this problem.


How do I set node id ahead of time or add a connection based on a different property?

I am iterating through a list of nodes to add them to the editor. I also have a list of all the connections between nodes (based on the names of nodes). I am migrating from a different editor so this information is pre-existing. The 'addConnection' method seems to only work with ids but I have no way of knowing the mapping between node names and ids.
`
nodes.forEach((node) => {
this.editor.addNode(node.name, 1, 1, left, top, 'job-node', {}, 'NodeClick', 'vue')
}
connections.forEach((connection) => {
this.addConnection(??, ??, ??, ??)
}
`
I also can't get a basic example to work where I do
this.editor.addConnection(1, 2, 'ouput', 'input')
The docs also don't specify what 'output_class' and 'input_class' are? Is that the class of the node or the input and output circles? It would be helpful to have a more detailed example in the docs.

Get id of node create with:
const idNode = editor.addNode('bbb', 1, 0, 900, 200, 'bbb', {},  `ddd` );

idNode return of id of node.
For conection:
example:
editor.addConnection(15,16,'output_1','input_1');

15 and  16 id's of nodes.
and ouputs and inputs:

Thanks.
Is there a way to get the output class in a similar way? It doesn't appear that the addNodeOutput and input methods return anything.

They don't return anything, but the last input or output is always added.
You can know before or after adding, getting the node information with getNodeFromId.


[Feature] Supported .NET versions in the docs

I can't seem to find what versions of .NET are supported by Playwright in the docs.
I think this would be a handy bit of info to have.
I'm currently running on .NET 5 and it seems ok, but would like to know if .NET 6 is supported.

Yes, .net 6 is supported. Follow the getting started guide: https://playwright.dev/dotnet/docs/intro


1.2.7 getModified regression

We rely on getModified to show a save/undo button if there are changes to the model.
With 1.2.7, getModified doesn't work correctly for nested objects. Consider:
var model = {
    request: {
        method: 'get'
    }
}

// User changes request method on the model
model.set('request.method', 'put');

// Correctly says request has changed
console.log(model.getModified()); // Object {request: Class}

// User changes request method back
model.set('request.method', 'get');

// incorrectly says request is still modified, even though it is no longer different
console.log(model.getModified()); // Object {request: Class}

In fact, model "dirty tracking" in general seems to have issues in 1.2.7.

Can you create a reproduction?
When introducing 1.2.7 version I was expecting to get several issues about the getModified method and the _original property. It makes Astronomy much faster but changes a few things internally. I've tried to make as much backward compatible as possible. If I won't be able to fix it I would suggest forcing 1.2.6 version in your project. But let's just create reproduction and maybe I will be able to help.

I've improved backward compatibility in Astronomy 1.2.8. You shouldn't have problem right now.

I've updated my astronomy perf repo to demonstrate the bug:
https://github.com/marbemac/astronomy-perf
Load that meteor app up, then change the url in the input field. You'll see getModified() returns the entire request object, which is the first kind of weird thing.
Then, when you clear the input, isModified() correctly returns false, but getModified() still returns the request object, even though nothing in there is actually modified.

Ok I know what's the problem. The reason that it still sees modifications is because document contains _modifiers even if the field's value was set to the original value. I will address that problem tomorrow and publish update. Sorry for breaking backward compatibility but I thought that making Astronomy faster is very important. I think that after that update everything should work as previously and in the same time much faster :)

Fixed in 1.2.9

:+1: all good, thanks!
We continue to layer in hacks to work around #156. Any sort of progress update on 2.0? Not a huge rush, just curious.

I have a little delay with Astronomy 2.0 because of private stuff. Moreover there is Christmas now. So it will be at the beginning of 2016.
What is it about umbrella issue? I don't have time to read entire discussion.

No problem on 2016.
I was just suggesting that it might make sense to create an umbrella issue for astronomy 2.0, similar to that linked issue. It's not strictly necessary - just helps people that are interested, follow the 2.0 progress without having to ask or bug you :).

Oh I see. I didn't know that we can use the "umbrella" word in such situation :). Is there some reason that they use the "umbrella" word instead of something else? :)
Yes I think it's good idea to create such issue. I will do it.

I hate to resurrect this again, but see the attached screenshot. I'm not sure exactly what is causing this to happen, but I'm seeing cases where getModified returns an empty object, but isModified() returns true. In the specific case I'm running into, there should be no modifications to the model (so getModified is working correctly, but isModified seems to have an issue).

I'll try and figure out how to reproduce this, but in general, should isModified ever return true if getModified returns an empty object?

(this does not occur in 1.2.6)

Ok I will take a look at it

@jagi They use the "umbrella" word because it kind of covers the entire thing. So like how an umbrella covers a person from rain in entirety.


Adding support for getting multiple pages of comments

Added an extension so it is easier to get multiple pages of comments. Included an example in the README.

I'm closing this PR, no new features will be added as the API endpoint is going to expire soon. Thanks for the effort though!

Yes good point Frank. Thanks for the original work though, it was helpful. Best wishes!

@butlermh And thanks for your kind words! :smile:


⚠ Bofrost.lu has degraded performance

In 93299d1, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2206 ms

Resolved: Bofrost.lu performance has improved in f194428 after 17 minutes.


[grpc] Build failure on arm64-osx

Package<EMAIL_ADDRESS>Host Environment

Host: arm64-osx
Compiler: AppleClang <IP_ADDRESS>00309
vcpkg-tool version: 2024-08-01-fd884a0d390d12783076341bd43d77c3a6a15658
vcpkg-scripts version: a74164519 2024-08-12 (10 hours ago)

To Reproduce
vcpkg install grpc
Failure logs
-- Using cached grpc-grpc-v1.60.0.tar.gz.
-- Cleaning sources at /Users/donaldmoore/vcpkg/buildtrees/grpc/src/v1.60.0-cff025ca16.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /Users/donaldmoore/vcpkg/downloads/grpc-grpc-v1.60.0.tar.gz
-- Applying patch 00001-fix-uwp.patch
-- Applying patch 00002-static-linking-in-linux.patch
-- Applying patch 00003-undef-base64-macro.patch
-- Applying patch 00004-link-gdi32-on-windows.patch
-- Applying patch 00005-fix-uwp-error.patch
-- Applying patch 00009-use-system-upb.patch
-- Applying patch 00015-disable-download-archive.patch
-- Applying patch 00016-fix-plugin-targets.patch
-- Applying patch 00017-abseil.patch
-- Using source at /Users/donaldmoore/vcpkg/buildtrees/grpc/src/v1.60.0-cff025ca16.clean
-- Found external ninja('1.12.1').
-- Configuring arm64-osx
-- Building arm64-osx-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:134 (message):
    Command failed: /usr/local/Cellar/cmake/3.30.2/bin/cmake --build . --config Debug --target install -- -v -j9
    Working Directory: /Users/donaldmoore/vcpkg/buildtrees/grpc/arm64-osx-dbg
    See logs for more information:
      /Users/donaldmoore/vcpkg/buildtrees/grpc/install-arm64-osx-dbg-out.log
      /Users/donaldmoore/vcpkg/buildtrees/grpc/install-arm64-osx-dbg-err.log

Call Stack (most recent call first):
  installed/arm64-osx/share/vcpkg-cmake/vcpkg_cmake_build.cmake:74 (vcpkg_execute_build_process)
  installed/arm64-osx/share/vcpkg-cmake/vcpkg_cmake_install.cmake:16 (vcpkg_cmake_build)
  ports/grpc/portfile.cmake:76 (vcpkg_cmake_install)
  scripts/ports.cmake:192 (include)

/Users/donaldmoore/vcpkg/buildtrees/grpc/install-arm64-osx-dbg-out.log
Change Dir: '/Users/donaldmoore/vcpkg/buildtrees/grpc/arm64-osx-dbg'

Run Build Command(s): /usr/local/bin/ninja -v -v -j9 install

/Users/donaldmoore/vcpkg/buildtrees/grpc/install-arm64-osx-dbg-err.log
ninja: error: '_gRPC_CPP_PLUGIN-NOTFOUND', needed by '/Users/donaldmoore/vcpkg/buildtrees/grpc/arm64-osx-dbg/gens/src/proto/grpc/reflection/v1/reflection.pb.cc', missing and no known rule to make it

I can't reproduce this issue locally.
Elapsed time to handle grpc:arm64-osx: 6.5 min
Total install time: 10 min
grpc provides CMake targets:

  # this is heuristically generated, and may not be correct
  find_package(gRPC CONFIG REQUIRED)
  # note: 8 additional targets are not displayed.
  target_link_libraries(main PRIVATE gRPC::gpr gRPC::grpc gRPC::grpc++ gRPC::grpc++_alts)

grpc provides pkg-config modules:

  # gRPC platform support library
  gpr

  # C++ wrapper for gRPC
  grpc++

  # C++ wrapper for gRPC without SSL
  grpc++_unsecure

  # high performance general RPC framework
  grpc

  # high performance general RPC framework without SSL
  grpc_unsecure

@JonLiu1993 I think I ended up migrating everything to arm64. Whatever it was, I was in a tizzy and I forgot to write it down.

For anyone else who needs help with this, gRPC contains all of the packages that you need. Install it from vcpkg, but make sure you're using the right architecture. Yes, it does take a while to migrate.


#29: Changing MaintenanceEventApi to MaintenanceApi. All mention of "…

…events" in API content is changed to refer to "maintenance appointments" instead. The semantics are kept the same, just a change to nomenclature, however, since the endpoints were named after "events", this impacts them too.

Will changing the method names affect the Maintenance Service build?

Yes. It also affects TMI Display. I have the updates completed in FRAIHM-1567 branch.