In the rank objective, lambdas and hessians need to factor sigmoid_ into the computation.

Additionally, the sigmoid function has an arbitrary factor of 2 in the exponent; it is not just non-standard but the gradients are not computed correctly anyway.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: sbruch sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

I am also proposing to remove what looks like a heuristic that normalizes the gradients by the difference in document scores. This is not part of the LambdaMART algorithm and at any rate does not help with convergence rate or generalization. An alternative to this change is to make such heuristics optional.
I have also trained a model on MSLR Web30k (Fold 1) and Yahoo! Learning-to-Rank Challenge (Set 1) datasets and what I have obtained is no worse or better than a model trained from the head:
NDCG@5 on Web30k before the change is 49.20 (+/-0.07) and after the change it is 49.05 (+/-0.09)
NDCG@5 on Yahoo LTRC before the change is 74.16 (+/-0.14) and after the change it is 74.22 (+/-0.08)
where sigmoid_ is set to two (post change).

Note that in my experiments I have removed the queries that have no relevant documents to avoid NDCG inflation -- LightGBM computes an NDCG of 1 for such queries.

Update: I wanted to share an updated NDCG@5 after much fine-tuning on validation sets:
Web30K Fold 1: 49.39 (+/-0.08)
Yahoo! Set 1: 74.97 (+/-0.09)
As noted in an earlier comment, queries with no relevant documents were discarded from the test set. Here are the hyperparameters I used if you were to reproduce the results:
For Web30k (Yahoo): learning rate is 0.05 (0.05), num_leaves is 400 (400), min_data_in_leaf is 50 (100), and min_sum_hessian_in_leafis set to 200 (10), and sigmoid_ is set to 2.

@sbruch
I just remember why the normalization is added.
In our internal cases, without the normalization, some rare queries will be ignored due to the small grad/hess.
Add the normalization will enforce the equality of different queries.
So I think we should add them back.

@sbrush would you mind to create another PR for the normalization?
We can add a bool parameter for that function.
And I think it is better to open  by default, for the consistent behavious as before.

Thanks for accepting and merging the pull request. I'd certainly be happy
to send another PR and add the normalization piece back conditioned on a
boolean; but a couple of questions: (1) Would it make sense to add a flag
to the config file? (e.g., lambdarank_normalize_gradients_by_score_diff)
and (2) Could it be false by default in the public library and set to true
in internal configuration?
On Fri, Aug 16, 2019 at 8:48 PM Guolin Ke<EMAIL_ADDRESS>wrote:

@SBrush https://github.com/SBrush would you mind to create another PR
for the normalization?
We can add a bool parameter for that function.
And I think it is better to open by default, for the consistent behavious
as before.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/LightGBM/pull/2322?email_source=notifications&email_token=AKZUQSOVV3IOWHQAXYI2BDDQE5YHTA5CNFSM4ILC2X52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4QCS3I#issuecomment-522201453,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKZUQSK7FOWD6Z5GO2JGYKLQE5YHTANCNFSM4ILC2X5Q
.

@sbruch refer to #2331, besides, I add the normalization for lambdas/hessians. However, both of them are not tested.


Magisch: Watch postgresqltutorial.com

Magisch requests the watch of the watch_keyword postgresqltutorial\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

postgresqltutorial\.com has been seen in 2 true positives, 0 false positives, and 0 NAAs.

Approved by Cody Gray in Charcoal HQ


VAProfile - New Postgres Table

Value Statement
Local cache requires a way to be stored. Discussion determined a new table with two columns: ICN and value. The value will consist of stringified jsonb which will document the preferences. This allows us to store a key value pair of ICN and preferences.
Acceptance Criteria
A new table has been created with the associated columns.
Checklist

[ ] New Table created
[ ] Columns added
[ ] New version in migrations/versions/ generated through Flask migration

Cross Functional Checklist
Testing:

[ ] integration test for API

Monitoring:

[ ] statsd metrics
[ ] datadog graphs
[ ] datadog alterts

Security:

[ ] auth permissions
[ ] JsonSchema

Transform requests to business models in our API

[ ] Marshmallow model serialization / deserialization

Documentation:

[ ] Readmes
[ ] Project specific
[ ] Support
[ ] Openapi
[ ] ADR
[ ] Tech Debt wall

Mountebank

[ ] API - create mocks for any external services calls

Postman

[ ] Update collection whenever we change our API

Assumptions

Additional Info/Resources

Out of Scope

Open Questions

Hey team! Please add your planning poker estimate with ZenHub @k-macmillan @jakehova @ianperera @kalbfled

@k-macmillan Did you purposefully reopen this? CC @tabinda-syed


Question: reason to remove Pay and Close?

Hi Anton.
Any reason to remove "Pay and Close" - I find it very useful (much faster to come back after payment).
Maybe you could re-add it if app was invoked by "lightning:ln......" URL link?

To me, it was a rather distractive expirience, whereas earlier I could mindlessly tap "pay" now I have to think whether I want to pay or pay and quit.

it did not make sense for all payments - I feel the same,
but if you click on invoice in a browser (via ligthing:.... url) - that was quite a nice thing
not a big deal, but was saving some time when paying for multiple times


Add possibility to configure Jetty's minRequestDataRate

This parameter can be useful if we want to protect our server from slow http DOS attacks.

Agreed, sounds like a useful configuration for those who need it. I'd consider a PR if you or someone else would like to create one.

Hi @mgtriffid, @nickbabcock !
I also think it's a valuable configuration option, so I created a PR for that.
Could you please have a look ?
https://github.com/dropwizard/dropwizard/pull/2184


⚠ SkipTheCommericals has degraded performance

In 1607af0, SkipTheCommericals ($STC) experienced degraded performance:

HTTP code: 200
Response time: 7816 ms

Resolved: SkipTheCommericals performance has improved in 15dad84.


Fix all lint warnings and do some more formatting.

Review status: 0 of 3 files reviewed at latest revision, 5 unresolved discussions.

spec.md, line 2692 [r1] (raw file):
s/Node B/Node C/ ?

spec.md, line 2708 [r1] (raw file):
s/Node B/Node C/ ?

spec.md, line 2720 [r1] (raw file):
s/Node B/Node C/ ?

spec.md, line 2732 [r1] (raw file):
Node C ?

spec.md, line 2749 [r2] (raw file):
Node B ?

Comments from the review on Reviewable.io

Review status: 0 of 3 files reviewed at latest revision, 5 unresolved discussions.

spec.md, line 2692 [r1] (raw file):
Done.

spec.md, line 2708 [r1] (raw file):
Done.

spec.md, line 2720 [r1] (raw file):
Done.

spec.md, line 2732 [r1] (raw file):
Done.

spec.md, line 2749 [r2] (raw file):
Done.

Comments from the review on Reviewable.io


Docs app | Sanitize input

Description
Sanitize input from a command line argument flows into fs.default.writeFileSync, where it is used as a path:
https://github.com/eclipse-tractusx/portal-assets/blob/v1.8.0/src/documentation/Release.js#L90
Additional information
Maybe helpful for fixing

Close, due to comment https://github.com/eclipse-tractusx/portal-assets/issues/289#issuecomment-2042743499


[BUG]  Messages produced failed.

KOP <IP_ADDRESS>
Pulsar 2.8
kafka client 2.4.1
Message produced failed: (other topics produced successfully, but some got the error.  )
2021-11-26 10:34:19.061 [kafka-producer-network-thread | producer-1] WARN  org.apache.kafka.clients.producer.internals.Sender Sender.java:637  - [Producer clientId=producer-1] Got error produce response with correlation id 5 on topic-partition broker3_partition50_8-0, retrying<PHONE_NUMBER> attempts left). Error: NETWORK_EXCEPTION

2021-11-26 10:34:19.062 [kafka-producer-network-thread | producer-1] WARN  org.apache.kafka.clients.producer.internals.Sender Sender.java:682  - [Producer clientId=producer-1] Received invalid metadata error in produce request on partition broker3_partition50_8-0 due to org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.. Going to request metadata update now

Bookie error:
10:33:56.019 [bookie-io-1-5] ERROR org.apache.bookkeeper.proto.BookieRequestHandler - Unhandled exception occurred in I/O thread or handler on [id: 0xbbe4feaa, L:/<IP_ADDRESS>:3181 - R:/<IP_ADDRESS>:31070]
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 5387712 - discarded
        at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:503) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:489) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:376) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:419) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:332) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[io.netty-netty-codec-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [io.netty-netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:4.1.68.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [io.netty-netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:4.1.68.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:4.1.68.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]

@BewareMyPower  PTAL

@hangc0276

Can you please give an example to step by step reproduce this bug?

Adjuest maxMessageSize and nettyMaxFrameSizeBytes doesn't make sense.

Can you please give an example to step by step reproduce this bug?

KOP <IP_ADDRESS>
Pulsar 2.8
kafka client 2.4.1
1、I  tried with FLINK  to consume from topic A, and produce to topic B.
2、use kafka produce perf to produce data to topic A.
bin/kafka-producer-perf-test.sh --topic partition_3_onebroker  --num-records 10000000 --record-size 4000 --throughput -1  --producer-props bootstrap.servers=kafka-perf-test001.ys:8098 acks=1 batch.size=1048576

Could you try a smaller maxReadEntriesNum (like 1)?

Could you try a smaller maxReadEntriesNum (like 1)?

It was produce error, not consume.

The error in my env is a little different, it's just caused by sending a HTTP request to Kafka's port:
curl http://localhost:9092

Can you paste the broker configuration here? it will very helpful.

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

### --- General broker settings --- ###
#listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093

#security.inter.broker.protocol=SASL_PLAINTEXT
#security.inter.broker.protocol=PLAINTEXT
#sasl.mechanism.inter.broker.protocol=PLAIN
#sasl.enabled.mechanisms=PLAIN
#authorizer.class.name = com.didichuxing.datachannel.kafka.security.authorizer.DidiAuthorizer
#super.users=User:admin
exposeBunlesMetricsInPrometheus=true
#Kafka on Pulsar configuration
messagingProtocols=kafka
protocolHandlerDirectory=./protocols
kafkaListeners=PLAINTEXT://:8098
#kafkaProtocolMap=SASL_PLAINTEXT:SASL_PLAINTEXT
brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
brokerDeleteInactiveTopicsEnabled=false
kopClusterId=54
kopGateway=http://hidden-ip:30442
brokerId=100
saslAllowedMechanisms=PLAIN
entryFormat=pulsar
maxReadEntriesNum=1

strictBookieAffinityEnabled=true
preciseTopicPublishRateLimiterEnable=true
# Zookeeper quorum connection string
zookeeperServers=hidden-ip:2181/test
# Configuration Store connection string
configurationStoreServers=hidden-ip:2181/test
# Broker data port
brokerServicePort=6650

# Broker data port for TLS - By default TLS is disabled
brokerServicePortTls=

# Port to use to server HTTP request
webServicePort=8080

# Port to use to server HTTPS request - By default TLS is disabled
webServicePortTls=

# Hostname or IP address the service binds on, default is <IP_ADDRESS>.
bindAddress=<IP_ADDRESS>

# Hostname or IP address the service advertises to the outside world. If not set, the value of InetAddress.getLocalHost().getHostName() is used.
advertisedAddress=

# Used to specify multiple advertised listeners for the broker.
# The value must format as <listener_name>:pulsar://<host>:<port>,
# multiple listeners should separate with commas.
# Do not use this configuration with advertisedAddress and brokerServicePort.
# The Default value is absent means use advertisedAddress and brokerServicePort.
# advertisedListeners=

# Used to specify the internal listener name for the broker.
# The listener name must contain in the advertisedListeners.
# The Default value is absent, the broker uses the first listener as the internal listener.
# internalListenerName=

# Enable or disable the HAProxy protocol.
haProxyProtocolEnabled=false

# Number of threads to use for Netty IO. Default is set to 2 * Runtime.getRuntime().availableProcessors()
numIOThreads=

# Number of threads to use for ordered executor. The ordered executor is used to operate with zookeeper,
# such as init zookeeper client, get namespace policies from zookeeper etc. It also used to split bundle. Default is 8
numOrderedExecutorThreads=

# Number of threads to use for HTTP requests processing. Default is set to 2 * Runtime.getRuntime().availableProcessors()
numHttpServerThreads=

# Number of thread pool size to use for pulsar broker service.
# The executor in thread pool will do basic broker operation like load/unload bundle, update managedLedgerConfig,
# update topic/subscription/replicator message dispatch rate, do leader election etc.
# Default is Runtime.getRuntime().availableProcessors()
numExecutorThreadPoolSize=

# Number of thread pool size to use for pulsar zookeeper callback service
# The cache executor thread pool is used for restarting global zookeeper session.
# Default is 10
numCacheExecutorThreadPoolSize=10

# Max concurrent web requests
maxConcurrentHttpRequests=1024

# Flag to control features that are meant to be used when running in standalone mode
isRunningStandalone=

# Name of the cluster to which this broker belongs to
clusterName=test-pulsar

# The maximum number of tenants that each pulsar cluster can create
# This configuration is not precise control, in a concurrent scenario, the threshold will be exceeded
maxTenants=0

# Enable cluster's failure-domain which can distribute brokers into logical region
failureDomainsEnabled=false

# Zookeeper session timeout in milliseconds
zooKeeperSessionTimeoutMillis=30000

# ZooKeeper operation timeout in seconds
zooKeeperOperationTimeoutSeconds=30

# ZooKeeper cache expiry time in seconds
zooKeeperCacheExpirySeconds=300

# Time to wait for broker graceful shutdown. After this time elapses, the process will be killed
brokerShutdownTimeoutMs=60000

# Flag to skip broker shutdown when broker handles Out of memory error
skipBrokerShutdownOnOOM=false

# Enable backlog quota check. Enforces action on topic when the quota is reached
backlogQuotaCheckEnabled=true

# How often to check for topics that have reached the quota
backlogQuotaCheckIntervalInSeconds=60

# Default per-topic backlog quota limit, less than 0 means no limitation. default is -1.
backlogQuotaDefaultLimitGB=-1

# Default backlog quota retention policy. Default is producer_request_hold
# 'producer_request_hold' Policy which holds producer's send request until the resource becomes available (or holding times out)
# 'producer_exception' Policy which throws javax.jms.ResourceAllocationException to the producer
# 'consumer_backlog_eviction' Policy which evicts the oldest message from the slowest consumer's backlog
backlogQuotaDefaultRetentionPolicy=producer_request_hold

# Default ttl for namespaces if ttl is not already configured at namespace policies. (disable default-ttl with value 0)
ttlDurationDefaultInSeconds=0

# Enable topic auto creation if new producer or consumer connected (disable auto creation with value false)
allowAutoTopicCreation=false

# The type of topic that is allowed to be automatically created.(partitioned/non-partitioned)
allowAutoTopicCreationType=non-partitioned

# Enable subscription auto creation if new consumer connected (disable auto creation with value false)
allowAutoSubscriptionCreation=true

# The number of partitioned topics that is allowed to be automatically created if allowAutoTopicCreationType is partitioned.
defaultNumPartitions=1

# Enable the deletion of inactive topics
#brokerDeleteInactiveTopicsEnabled=true

# How often to check for inactive topics
brokerDeleteInactiveTopicsFrequencySeconds=30

# Set the inactive topic delete mode. Default is delete_when_no_subscriptions
# 'delete_when_no_subscriptions' mode only delete the topic which has no subscriptions and no active producers
# 'delete_when_subscriptions_caught_up' mode only delete the topic that all subscriptions has no backlogs(caught up)
# and no active producers/consumers
brokerDeleteInactiveTopicsMode=delete_when_no_subscriptions

# Metadata of inactive partitioned topic will not be cleaned up automatically by default.
# Note: If `allowAutoTopicCreation` and this option are enabled at the same time,
# it may appear that a partitioned topic has just been deleted but is automatically created as a non-partitioned topic.
brokerDeleteInactivePartitionedTopicMetadataEnabled=true

# Max duration of topic inactivity in seconds, default is not present
# If not present, 'brokerDeleteInactiveTopicsFrequencySeconds' will be used
# Topics that are inactive for longer than this value will be deleted
brokerDeleteInactiveTopicsMaxInactiveDurationSeconds=

# Max pending publish requests per connection to avoid keeping large number of pending
# requests in memory. Default: 1000
maxPendingPublishdRequestsPerConnection=1000

# How frequently to proactively check and purge expired messages
messageExpiryCheckIntervalInMinutes=5

# How long to delay rewinding cursor and dispatching messages when active consumer is changed
activeConsumerFailoverDelayTimeMillis=1000

# How long to delete inactive subscriptions from last consuming
# When it is 0, inactive subscriptions are not deleted automatically
subscriptionExpirationTimeMinutes=0

# Enable subscription message redelivery tracker to send redelivery count to consumer (default is enabled)
subscriptionRedeliveryTrackerEnabled=true

# How frequently to proactively check and purge expired subscription
subscriptionExpiryCheckIntervalInMinutes=5

# Enable Key_Shared subscription (default is enabled)
subscriptionKeySharedEnable=true

# On KeyShared subscriptions, with default AUTO_SPLIT mode, use splitting ranges or
# consistent hashing to reassign keys to new consumers
subscriptionKeySharedUseConsistentHashing=false

# On KeyShared subscriptions, number of points in the consistent-hashing ring.
# The higher the number, the more equal the assignment of keys to consumers
subscriptionKeySharedConsistentHashingReplicaPoints=100

# Set the default behavior for message deduplication in the broker
# This can be overridden per-namespace. If enabled, broker will reject
# messages that were already stored in the topic
brokerDeduplicationEnabled=false

# Maximum number of producer information that it's going to be
# persisted for deduplication purposes
brokerDeduplicationMaxNumberOfProducers=10000

# How often is the thread pool scheduled to check whether a snapshot needs to be taken.(disable with value 0)
brokerDeduplicationSnapshotFrequencyInSeconds=10
# If this time interval is exceeded, a snapshot will be taken.
# It will run simultaneously with `brokerDeduplicationEntriesInterval`
brokerDeduplicationSnapshotIntervalSeconds=120

# Number of entries after which a dedup info snapshot is taken.
# A larger interval will lead to fewer snapshots being taken, though it would
# increase the topic recovery time when the entries published after the
# snapshot need to be replayed.
brokerDeduplicationEntriesInterval=1000

# Time of inactivity after which the broker will discard the deduplication information
# relative to a disconnected producer. Default is 6 hours.
brokerDeduplicationProducerInactivityTimeoutMinutes=360

# When a namespace is created without specifying the number of bundle, this
# value will be used as the default
defaultNumberOfNamespaceBundles=16

# The maximum number of namespaces that each tenant can create
# This configuration is not precise control, in a concurrent scenario, the threshold will be exceeded
maxNamespacesPerTenant=0

# Enable check for minimum allowed client library version
clientLibraryVersionCheckEnabled=false

# Path for the file used to determine the rotation status for the broker when responding
# to service discovery health checks
statusFilePath=

# If true, (and ModularLoadManagerImpl is being used), the load manager will attempt to
# use only brokers running the latest software version (to minimize impact to bundles)
preferLaterVersions=false

# Max number of unacknowledged messages allowed to receive messages by a consumer on a shared subscription. Broker will stop sending
# messages to consumer once, this limit reaches until consumer starts acknowledging messages back.
# Using a value of 0, is disabling unackeMessage limit check and consumer can receive messages without any restriction
maxUnackedMessagesPerConsumer=50000

# Max number of unacknowledged messages allowed per shared subscription. Broker will stop dispatching messages to
# all consumers of the subscription once this limit reaches until consumer starts acknowledging messages back and
# unack count reaches to limit/2. Using a value of 0, is disabling unackedMessage-limit
# check and dispatcher can dispatch messages without any restriction
maxUnackedMessagesPerSubscription=200000

# Max number of unacknowledged messages allowed per broker. Once this limit reaches, broker will stop dispatching
# messages to all shared subscription which has higher number of unack messages until subscriptions start
# acknowledging messages back and unack count reaches to limit/2. Using a value of 0, is disabling
# unackedMessage-limit check and broker doesn't block dispatchers
maxUnackedMessagesPerBroker=0

# Once broker reaches maxUnackedMessagesPerBroker limit, it blocks subscriptions which has higher unacked messages
# than this percentage limit and subscription will not receive any new messages until that subscription acks back
# limit/2 messages
maxUnackedMessagesPerSubscriptionOnBrokerBlocked=0.16

# Tick time to schedule task that checks topic publish rate limiting across all topics
# Reducing to lower value can give more accuracy while throttling publish but
# it uses more CPU to perform frequent check. (Disable publish throttling with value 0)
topicPublisherThrottlingTickTimeMillis=10

# Tick time to schedule task that checks broker publish rate limiting across all topics
# Reducing to lower value can give more accuracy while throttling publish but
# it uses more CPU to perform frequent check. (Disable publish throttling with value 0)
brokerPublisherThrottlingTickTimeMillis=50

# Max Rate(in 1 seconds) of Message allowed to publish for a broker if broker publish rate limiting enabled
# (Disable message rate limit with value 0)
brokerPublisherThrottlingMaxMessageRate=0

# Max Rate(in 1 seconds) of Byte allowed to publish for a broker if broker publish rate limiting enabled.
# (Disable byte rate limit with value 0)
brokerPublisherThrottlingMaxByteRate=0

# Max Rate(in 1 seconds) of Message allowed to publish for a topic if topic publish rate limiting enabled
# (Disable byte rate limit with value 0)
maxPublishRatePerTopicInMessages=0

#Max Rate(in 1 seconds) of Byte allowed to publish for a topic if topic publish rate limiting enabled.
# (Disable byte rate limit with value 0)
maxPublishRatePerTopicInBytes=0

# Too many subscribe requests from a consumer can cause broker rewinding consumer cursors and loading data from bookies,
# hence causing high network bandwidth usage
# When the positive value is set, broker will throttle the subscribe requests for one consumer.
# Otherwise, the throttling will be disabled. The default value of this setting is 0 - throttling is disabled.
subscribeThrottlingRatePerConsumer=0

# Rate period for {subscribeThrottlingRatePerConsumer}. Default is 30s.
subscribeRatePeriodPerConsumerInSecond=30

# Default messages per second dispatch throttling-limit for every topic. Using a value of 0, is disabling default
# message dispatch-throttling
dispatchThrottlingRatePerTopicInMsg=0

# Default bytes per second dispatch throttling-limit for every topic. Using a value of 0, is disabling
# default message-byte dispatch-throttling
dispatchThrottlingRatePerTopicInByte=0

# Default number of message dispatching throttling-limit for a subscription.
# Using a value of 0, is disabling default message dispatch-throttling.
dispatchThrottlingRatePerSubscriptionInMsg=0

# Default number of message-bytes dispatching throttling-limit for a subscription.
# Using a value of 0, is disabling default message-byte dispatch-throttling.
dispatchThrottlingRatePerSubscriptionInByte=0

# Default messages per second dispatch throttling-limit for every replicator in replication.
# Using a value of 0, is disabling replication message dispatch-throttling
dispatchThrottlingRatePerReplicatorInMsg=0

# Default bytes per second dispatch throttling-limit for every replicator in replication.
# Using a value of 0, is disabling replication message-byte dispatch-throttling
dispatchThrottlingRatePerReplicatorInByte=0

# Dispatch rate-limiting relative to publish rate.
# (Enabling flag will make broker to dynamically update dispatch-rate relatively to publish-rate:
# throttle-dispatch-rate = (publish-rate + configured dispatch-rate).
		dispatchThrottlingRateRelativeToPublishRate=false

# By default we enable dispatch-throttling for both caught up consumers as well as consumers who have
# backlog.
		dispatchThrottlingOnNonBacklogConsumerEnabled=true

# Max number of entries to read from bookkeeper. By default it is 100 entries.
		dispatcherMaxReadBatchSize=100

# Max size in bytes of entries to read from bookkeeper. By default it is 5MB.
		dispatcherMaxReadSizeBytes=5242880

# Min number of entries to read from bookkeeper. By default it is 1 entries.
# When there is an error occurred on reading entries from bookkeeper, the broker
# will backoff the batch size to this minimum number."
		dispatcherMinReadBatchSize=10

# Max number of entries to dispatch for a shared subscription. By default it is 20 entries.
		dispatcherMaxRoundRobinBatchSize=20

# Precise dispathcer flow control according to history message number of each entry
preciseDispatcherFlowControl=false

# Max number of concurrent lookup request broker allows to throttle heavy incoming lookup traffic
maxConcurrentLookupRequest=50000

# Max number of concurrent topic loading request broker allows to control number of zk-operations
maxConcurrentTopicLoadRequest=5000

# Max concurrent non-persistent message can be processed per connection
maxConcurrentNonPersistentMessagePerConnection=1000

# Number of worker threads to serve non-persistent topic
numWorkerThreadsForNonPersistentTopic=8

# Enable broker to load persistent topics
enablePersistentTopics=true

# Enable broker to load non-persistent topics
enableNonPersistentTopics=true

# Enable to run bookie along with broker
enableRunBookieTogether=false

# Enable to run bookie autorecovery along with broker
enableRunBookieAutoRecoveryTogether=false

# Max number of producers allowed to connect to topic. Once this limit reaches, Broker will reject new producers
# until the number of connected producers decrease.
# Using a value of 0, is disabling maxProducersPerTopic-limit check.
maxProducersPerTopic=0

# Enforce producer to publish encrypted messages.(default disable).
encryptionRequireOnProducer=false

# Max number of consumers allowed to connect to topic. Once this limit reaches, Broker will reject new consumers
# until the number of connected consumers decrease.
# Using a value of 0, is disabling maxConsumersPerTopic-limit check.
maxConsumersPerTopic=0

# Max number of subscriptions allowed to subscribe to topic. Once this limit reaches, broker will reject
# new subscription until the number of subscribed subscriptions decrease.
# Using a value of 0, is disabling maxSubscriptionsPerTopic limit check.
maxSubscriptionsPerTopic=0

# Max number of consumers allowed to connect to subscription. Once this limit reaches, Broker will reject new consumers
# until the number of connected consumers decrease.
# Using a value of 0, is disabling maxConsumersPerSubscription-limit check.
maxConsumersPerSubscription=0

# Max size of messages.
maxMessageSize=10485760
nettyMaxFrameSizeBytes=10485760

# Interval between checks to see if topics with compaction policies need to be compacted
brokerServiceCompactionMonitorIntervalInSeconds=60

# Whether to enable the delayed delivery for messages.
# If disabled, messages will be immediately delivered and there will
# be no tracking overhead.
delayedDeliveryEnabled=true

# Control the tick time for when retrying on delayed delivery,
# affecting the accuracy of the delivery time compared to the scheduled time.
# Default is 1 second.
delayedDeliveryTickTimeMillis=1000

# Whether to enable acknowledge of batch local index.
acknowledgmentAtBatchIndexLevelEnabled=false

# Enable tracking of replicated subscriptions state across clusters.
enableReplicatedSubscriptions=true

# Frequency of snapshots for replicated subscriptions tracking.
replicatedSubscriptionsSnapshotFrequencyMillis=1000

# Timeout for building a consistent snapshot for tracking replicated subscriptions state.
replicatedSubscriptionsSnapshotTimeoutSeconds=30

# Max number of snapshot to be cached per subscription.
replicatedSubscriptionsSnapshotMaxCachedPerSubscription=10

# Max memory size for broker handling messages sending from producers.
# If the processing message size exceed this value, broker will stop read data
# from the connection. The processing messages means messages are sends to broker
# but broker have not send response to client, usually waiting to write to bookies.
# It's shared across all the topics running in the same broker.
# Use -1 to disable the memory limitation. Default is 1/2 of direct memory.
maxMessagePublishBufferSizeInMB=

# Interval between checks to see if message publish buffer size is exceed the max message publish buffer size
# Use 0 or negative number to disable the max publish buffer limiting.
messagePublishBufferCheckIntervalInMillis=100

# Check between intervals to see if consumed ledgers need to be trimmed
# Use 0 or negative number to disable the check
retentionCheckIntervalInSeconds=120

# Max number of partitions per partitioned topic
# Use 0 or negative number to disable the check
maxNumPartitionsPerPartitionedTopic=0

# There are two policies when zookeeper session expired happens, "shutdown" and "reconnect".
# If uses "shutdown" policy, shutdown the broker when zookeeper session expired happens.
# If uses "reconnect" policy, try to reconnect to zookeeper server and re-register metadata to zookeeper.
# Node: the "reconnect" policy is an experiment feature
zookeeperSessionExpiredPolicy=shutdown

# Enable or disable system topic
systemTopicEnabled=false

# Enable or disable topic level policies, topic level policies depends on the system topic
# Please enable the system topic first.
topicLevelPoliciesEnabled=false

# If a topic remains fenced for this number of seconds, it will be closed forcefully.
# If it is set to 0 or a negative number, the fenced topic will not be closed.
topicFencingTimeoutSeconds=0

### --- Authentication --- ###
# Role names that are treated as "proxy roles". If the broker sees a request with
#role as proxyRoles - it will demand to see a valid original principal.
proxyRoles=

# If this flag is set then the broker authenticates the original Auth data
# else it just accepts the originalPrincipal and authorizes it (if required).
authenticateOriginalAuthData=false

# Deprecated - Use webServicePortTls and brokerServicePortTls instead
tlsEnabled=false

# Tls cert refresh duration in seconds (set 0 to check on every new connection)
tlsCertRefreshCheckDurationSec=300

# Path for the TLS certificate file
tlsCertificateFilePath=

# Path for the TLS private key file
tlsKeyFilePath=

# Path for the trusted TLS certificate file.
# This cert is used to verify that any certs presented by connecting clients
# are signed by a certificate authority. If this verification
# fails, then the certs are untrusted and the connections are dropped.
tlsTrustCertsFilePath=

# Accept untrusted TLS certificate from client.
# If true, a client with a cert which cannot be verified with the
# 'tlsTrustCertsFilePath' cert will allowed to connect to the server,
# though the cert will not be used for client authentication.
tlsAllowInsecureConnection=false

# Specify the tls protocols the broker will use to negotiate during TLS handshake
# (a comma-separated list of protocol names).
# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
tlsProtocols=

# Specify the tls cipher the broker will use to negotiate during TLS Handshake
# (a comma-separated list of ciphers).
# Examples:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
tlsCiphers=

# Trusted client certificates are required for to connect TLS
# Reject the Connection if the Client Certificate is not trusted.
# In effect, this requires that all connecting clients perform TLS client
# authentication.
tlsRequireTrustedClientCertOnConnect=false

### --- KeyStore TLS config variables --- ###
# Enable TLS with KeyStore type configuration in broker.
tlsEnabledWithKeyStore=false

# TLS Provider for KeyStore type
tlsProvider=

# TLS KeyStore type configuration in broker: JKS, PKCS12
tlsKeyStoreType=JKS

# TLS KeyStore path in broker
tlsKeyStore=

# TLS KeyStore password for broker
tlsKeyStorePassword=

# TLS TrustStore type configuration in broker: JKS, PKCS12
tlsTrustStoreType=JKS

# TLS TrustStore path in broker
tlsTrustStore=

# TLS TrustStore password in broker
tlsTrustStorePassword=

# Whether internal client use KeyStore type to authenticate with Pulsar brokers
brokerClientTlsEnabledWithKeyStore=false

# The TLS Provider used by internal client to authenticate with other Pulsar brokers
brokerClientSslProvider=

# TLS TrustStore type configuration for internal client: JKS, PKCS12
# used by the internal client to authenticate with Pulsar brokers
brokerClientTlsTrustStoreType=JKS

# TLS TrustStore path for internal client
# used by the internal client to authenticate with Pulsar brokers
brokerClientTlsTrustStore=

# TLS TrustStore password for internal client,
# used by the internal client to authenticate with Pulsar brokers
brokerClientTlsTrustStorePassword=

# Specify the tls cipher the internal client will use to negotiate during TLS Handshake
# (a comma-separated list of ciphers)
# e.g.  [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256].
# used by the internal client to authenticate with Pulsar brokers
brokerClientTlsCiphers=

# Specify the tls protocols the broker will use to negotiate during TLS handshake
# (a comma-separated list of protocol names).
# e.g.  [TLSv1.2, TLSv1.1, TLSv1]
# used by the internal client to authenticate with Pulsar brokers
brokerClientTlsProtocols=

### --- Authentication --- ###

# Enable authentication
authenticationEnabled=false

# Autentication provider name list, which is comma separated list of class names
authenticationProviders=

# Interval of time for checking for expired authentication credentials
authenticationRefreshCheckSeconds=60

# Enforce authorization
authorizationEnabled=false

# Authorization provider fully qualified class-name
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider

# Allow wildcard matching in authorization
# (wildcard matching only applicable if wildcard-char:
# * presents at first or last position eg: *.pulsar.service, pulsar.service.*)
authorizationAllowWildcardsMatching=false

# Role names that are treated as "super-user", meaning they will be able to do all admin
# operations and publish/consume from all topics
superUserRoles=

# Authentication settings of the broker itself. Used when the broker connects to other brokers,
# either in same or other clusters
brokerClientTlsEnabled=false
brokerClientAuthenticationPlugin=
brokerClientAuthenticationParameters=
brokerClientTrustCertsFilePath=

# Supported Athenz provider domain names(comma separated) for authentication
athenzDomainNames=

# When this parameter is not empty, unauthenticated users perform as anonymousUserRole
anonymousUserRole=

### --- Token Authentication Provider --- ###

## Symmetric key
# Configure the secret key to be used to validate auth tokens
# The key can be specified like:
# tokenSecretKey=data:;base64,xxxxxxxxx
# tokenSecretKey=file:///my/secret.key
tokenSecretKey=

## Asymmetric public/private key pair
# Configure the public key to be used to validate auth tokens
# The key can be specified like:
# tokenPublicKey=data:;base64,xxxxxxxxx
# tokenPublicKey=file:///my/public.key
tokenPublicKey=

# The token "claim" that will be interpreted as the authentication "role" or "principal" by AuthenticationProviderToken (defaults to "sub" if blank)
tokenAuthClaim=

# The token audience "claim" name, e.g. "aud", that will be used to get the audience from token.
# If not set, audience will not be verified.
tokenAudienceClaim=

# The token audience stands for this broker. The field `tokenAudienceClaim` of a valid token, need contains this.
tokenAudience=

### --- SASL Authentication Provider --- ###

# This is a regexp, which limits the range of possible ids which can connect to the Broker using SASL.
# Default value: `SaslConstants.JAAS_CLIENT_ALLOWED_IDS_DEFAULT`, which is ".*pulsar.*",
# so only clients whose id contains 'pulsar' are allowed to connect.
saslJaasClientAllowedIds=

# Service Principal, for login context name.
# Default value `SaslConstants.JAAS_DEFAULT_BROKER_SECTION_NAME`, which is "Broker".
saslJaasBrokerSectionName=

### --- HTTP Server config --- ###

# If >0, it will reject all HTTP requests with bodies larged than the configured limit
httpMaxRequestSize=-1

# Enable the enforcement of limits on the incoming HTTP requests
httpRequestsLimitEnabled=false

# Max HTTP requests per seconds allowed. The excess of requests will be rejected with HTTP code 429 (Too many requests)
httpRequestsMaxPerSecond=100.0

### --- BookKeeper Client --- ###

# Metadata service uri that bookkeeper is used for loading corresponding metadata driver
# and resolving its metadata service location.
# This value can be fetched using `bookkeeper shell whatisinstanceid` command in BookKeeper cluster.
# For example: zk+hierarchical://localhost:2181/ledgers
# The metadata service uri list can also be semicolon separated values like below:
# zk+hierarchical://zk1:2181;zk2:2181;zk3:2181/ledgers
bookkeeperMetadataServiceUri=

# Authentication plugin to use when connecting to bookies
bookkeeperClientAuthenticationPlugin=

# BookKeeper auth plugin implementatation specifics parameters name and values
bookkeeperClientAuthenticationParametersName=
bookkeeperClientAuthenticationParameters=

# Timeout for BK add / read operations
bookkeeperClientTimeoutInSeconds=30

# Speculative reads are initiated if a read request doesn't complete within a certain time
# Using a value of 0, is disabling the speculative reads
bookkeeperClientSpeculativeReadTimeoutInMillis=0

# Number of channels per bookie
bookkeeperNumberOfChannelsPerBookie=16

# Use older Bookkeeper wire protocol with bookie
bookkeeperUseV2WireProtocol=true

# Enable bookies health check. Bookies that have more than the configured number of failure within
# the interval will be quarantined for some time. During this period, new ledgers won't be created
# on these bookies
bookkeeperClientHealthCheckEnabled=true
bookkeeperClientHealthCheckIntervalSeconds=60
bookkeeperClientHealthCheckErrorThresholdPerInterval=5
bookkeeperClientHealthCheckQuarantineTimeInSeconds=1800

#bookie quarantine ratio to avoid all clients quarantine the high pressure bookie servers at the same time
bookkeeperClientQuarantineRatio=1.0

# Specify options for the GetBookieInfo check. These settings can be useful
# to help ensure the list of bookies is up to date on the brokers.
bookkeeperGetBookieInfoIntervalSeconds=86400
bookkeeperGetBookieInfoRetryIntervalSeconds=60

# Enable rack-aware bookie selection policy. BK will chose bookies from different racks when
# forming a new bookie ensemble
# This parameter related to ensemblePlacementPolicy in conf/bookkeeper.conf, if enabled, ensemblePlacementPolicy
# should be set to org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy
bookkeeperClientRackawarePolicyEnabled=true

# Enable region-aware bookie selection policy. BK will chose bookies from
# different regions and racks when forming a new bookie ensemble
# If enabled, the value of bookkeeperClientRackawarePolicyEnabled is ignored
# This parameter related to ensemblePlacementPolicy in conf/bookkeeper.conf, if enabled, ensemblePlacementPolicy
# should be set to org.apache.bookkeeper.client.RegionAwareEnsemblePlacementPolicy
bookkeeperClientRegionawarePolicyEnabled=false

# Minimum number of racks per write quorum. BK rack-aware bookie selection policy will try to
# get bookies from at least 'bookkeeperClientMinNumRacksPerWriteQuorum' racks for a write quorum.
bookkeeperClientMinNumRacksPerWriteQuorum=2

# Enforces rack-aware bookie selection policy to pick bookies from 'bookkeeperClientMinNumRacksPerWriteQuorum'
# racks for a writeQuorum.
# If BK can't find bookie then it would throw BKNotEnoughBookiesException instead of picking random one.
bookkeeperClientEnforceMinNumRacksPerWriteQuorum=false

# Enable/disable reordering read sequence on reading entries.
bookkeeperClientReorderReadSequenceEnabled=false

# Enable bookie isolation by specifying a list of bookie groups to choose from. Any bookie
# outside the specified groups will not be used by the broker
bookkeeperClientIsolationGroups=

# Enable bookie secondary-isolation group if bookkeeperClientIsolationGroups doesn't
# have enough bookie available.
bookkeeperClientSecondaryIsolationGroups=

# Minimum bookies that should be available as part of bookkeeperClientIsolationGroups
# else broker will include bookkeeperClientSecondaryIsolationGroups bookies in isolated list.
bookkeeperClientMinAvailableBookiesInIsolationGroups=

# Enable/disable having read operations for a ledger to be sticky to a single bookie.
# If this flag is enabled, the client will use one single bookie (by preference) to read
# all entries for a ledger.
#
# Disable Sticy Read until {@link https://github.com/apache/bookkeeper/issues/1970} is fixed
bookkeeperEnableStickyReads=false

# Set the client security provider factory class name.
# Default: org.apache.bookkeeper.tls.TLSContextFactory
bookkeeperTLSProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory

# Enable tls authentication with bookie
bookkeeperTLSClientAuthentication=false

# Supported type: PEM, JKS, PKCS12. Default value: PEM
bookkeeperTLSKeyFileType=PEM

#Supported type: PEM, JKS, PKCS12. Default value: PEM
bookkeeperTLSTrustCertTypes=PEM

# Path to file containing keystore password, if the client keystore is password protected.
bookkeeperTLSKeyStorePasswordPath=

# Path to file containing truststore password, if the client truststore is password protected.
bookkeeperTLSTrustStorePasswordPath=

# Path for the TLS private key file
bookkeeperTLSKeyFilePath=

# Path for the TLS certificate file
bookkeeperTLSCertificateFilePath=

# Path for the trusted TLS certificate file
bookkeeperTLSTrustCertsFilePath=

# Enable/disable disk weight based placement. Default is false
bookkeeperDiskWeightBasedPlacementEnabled=false

# Set the interval to check the need for sending an explicit LAC
# A value of '0' disables sending any explicit LACs. Default is 0.
bookkeeperExplicitLacIntervalInMills=0

# Expose bookkeeper client managed ledger stats to prometheus. default is false
# bookkeeperClientExposeStatsToPrometheus=false

### --- Managed Ledger --- ###

# Number of bookies to use when creating a ledger
managedLedgerDefaultEnsembleSize=2

# Number of copies to store for each message
managedLedgerDefaultWriteQuorum=1

# Number of guaranteed copies (acks to wait before write is complete)
managedLedgerDefaultAckQuorum=1

# How frequently to flush the cursor positions that were accumulated due to rate limiting. (seconds).
# Default is 60 seconds
managedLedgerCursorPositionFlushSeconds = 60

# Default type of checksum to use when writing to BookKeeper. Default is "CRC32C"
# Other possible options are "CRC32", "MAC" or "DUMMY" (no checksum).
managedLedgerDigestType=CRC32C

# Number of threads to be used for managed ledger tasks dispatching
managedLedgerNumWorkerThreads=8

# Number of threads to be used for managed ledger scheduled tasks
managedLedgerNumSchedulerThreads=8

# Amount of memory to use for caching data payload in managed ledger. This memory
# is allocated from JVM direct memory and it's shared across all the topics
# running  in the same broker. By default, uses 1/5th of available direct memory
managedLedgerCacheSizeMB=

# Whether we should make a copy of the entry payloads when inserting in cache
managedLedgerCacheCopyEntries=false

# Threshold to which bring down the cache level when eviction is triggered
managedLedgerCacheEvictionWatermark=0.9

# Configure the cache eviction frequency for the managed ledger cache (evictions/sec)
managedLedgerCacheEvictionFrequency=100.0

# All entries that have stayed in cache for more than the configured time, will be evicted
managedLedgerCacheEvictionTimeThresholdMillis=1000

# Configure the threshold (in number of entries) from where a cursor should be considered 'backlogged'
# and thus should be set as inactive.
managedLedgerCursorBackloggedThreshold=1000

# Rate limit the amount of writes per second generated by consumer acking the messages
managedLedgerDefaultMarkDeleteRateLimit=1.0

# Max number of entries to append to a ledger before triggering a rollover
# A ledger rollover is triggered on these conditions
#  * Either the max rollover time has been reached
#  * or max entries have been written to the ledged and at least min-time
#    has passed
managedLedgerMaxEntriesPerLedger=50000

# Minimum time between ledger rollover for a topic
managedLedgerMinLedgerRolloverTimeMinutes=10

# Maximum time before forcing a ledger rollover for a topic
managedLedgerMaxLedgerRolloverTimeMinutes=240

# Maximum ledger size before triggering a rollover for a topic (MB)
managedLedgerMaxSizePerLedgerMbytes=2048

# Delay between a ledger being successfully offloaded to long term storage
# and the ledger being deleted from bookkeeper (default is 4 hours)
managedLedgerOffloadDeletionLagMs=14400000

# The number of bytes before triggering automatic offload to long term storage
# (default is -1, which is disabled)
managedLedgerOffloadAutoTriggerSizeThresholdBytes=-1

# Max number of entries to append to a cursor ledger
managedLedgerCursorMaxEntriesPerLedger=50000

# Max time before triggering a rollover on a cursor ledger
managedLedgerCursorRolloverTimeInSeconds=14400

# Max number of "acknowledgment holes" that are going to be persistently stored.
# When acknowledging out of order, a consumer will leave holes that are supposed
# to be quickly filled by acking all the messages. The information of which
# messages are acknowledged is persisted by compressing in "ranges" of messages
# that were acknowledged. After the max number of ranges is reached, the information
# will only be tracked in memory and messages will be redelivered in case of
# crashes.
managedLedgerMaxUnackedRangesToPersist=10000

# Max number of "acknowledgment holes" that can be stored in Zookeeper. If number of unack message range is higher
# than this limit then broker will persist unacked ranges into bookkeeper to avoid additional data overhead into
# zookeeper.
managedLedgerMaxUnackedRangesToPersistInZooKeeper=1000

# Skip reading non-recoverable/unreadable data-ledger under managed-ledger's list. It helps when data-ledgers gets
# corrupted at bookkeeper and managed-cursor is stuck at that ledger.
autoSkipNonRecoverableData=false

# Whether to recover cursors lazily when trying to recover a managed ledger backing a persistent topic.
# It can improve write availability of topics.
# The caveat is now when recovered ledger is ready to write we're not sure if all old consumers last mark
# delete position can be recovered or not.
lazyCursorRecovery=false

# operation timeout while updating managed-ledger metadata.
managedLedgerMetadataOperationsTimeoutSeconds=60

# Read entries timeout when broker tries to read messages from bookkeeper.
managedLedgerReadEntryTimeoutSeconds=0

# Add entry timeout when broker tries to publish message to bookkeeper (0 to disable it).
managedLedgerAddEntryTimeoutSeconds=0

# Managed ledger prometheus stats latency rollover seconds (default: 60s)
managedLedgerPrometheusStatsLatencyRolloverSeconds=60

# Whether trace managed ledger task execution time
managedLedgerTraceTaskExecution=true

# New entries check delay for the cursor under the managed ledger.
# If no new messages in the topic, the cursor will try to check again after the delay time.
# For consumption latency sensitive scenario, can set to a smaller value or set to 0.
# Of course, use a smaller value may degrade consumption throughput. Default is 10ms.
managedLedgerNewEntriesCheckDelayInMillis=10

### --- Load balancer --- ###

# Enable load balancer
loadBalancerEnabled=true

# Percentage of change to trigger load report update
loadBalancerReportUpdateThresholdPercentage=10

# maximum interval to update load report
loadBalancerReportUpdateMaxIntervalMinutes=15

# Frequency of report to collect
loadBalancerHostUsageCheckIntervalMinutes=1

# Enable/disable automatic bundle unloading for load-shedding
loadBalancerSheddingEnabled=true

# Load shedding interval. Broker periodically checks whether some traffic should be offload from
# some over-loaded broker to other under-loaded brokers
loadBalancerSheddingIntervalMinutes=1

# Prevent the same topics to be shed and moved to other broker more that once within this timeframe
loadBalancerSheddingGracePeriodMinutes=30

# Usage threshold to allocate max number of topics to broker
loadBalancerBrokerMaxTopics=50000

# Usage threshold to determine a broker as over-loaded
loadBalancerBrokerOverloadedThresholdPercentage=85

# Interval to flush dynamic resource quota to ZooKeeper
loadBalancerResourceQuotaUpdateIntervalMinutes=15

# enable/disable namespace bundle auto split
loadBalancerAutoBundleSplitEnabled=false

# enable/disable automatic unloading of split bundles
loadBalancerAutoUnloadSplitBundlesEnabled=false

# maximum topics in a bundle, otherwise bundle split will be triggered
loadBalancerNamespaceBundleMaxTopics=1000

# maximum sessions (producers + consumers) in a bundle, otherwise bundle split will be triggered
loadBalancerNamespaceBundleMaxSessions=1000

# maximum msgRate (in + out) in a bundle, otherwise bundle split will be triggered
loadBalancerNamespaceBundleMaxMsgRate=30000

# maximum bandwidth (in + out) in a bundle, otherwise bundle split will be triggered
loadBalancerNamespaceBundleMaxBandwidthMbytes=100

# maximum number of bundles in a namespace
loadBalancerNamespaceMaximumBundles=128

# Override the auto-detection of the network interfaces max speed.
# This option is useful in some environments (eg: EC2 VMs) where the max speed
# reported by Linux is not reflecting the real bandwidth available to the broker.
# Since the network usage is employed by the load manager to decide when a broker
# is overloaded, it is important to make sure the info is correct or override it
# with the right value here. The configured value can be a double (eg: 0.8) and that
# can be used to trigger load-shedding even before hitting on NIC limits.
loadBalancerOverrideBrokerNicSpeedGbps=

# Name of load manager to use
loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl

# Supported algorithms name for namespace bundle split.
# "range_equally_divide" divides the bundle into two parts with the same hash range size.
# "topic_count_equally_divide" divides the bundle into two parts with the same topics count.
supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide

# Default algorithm name for namespace bundle split
defaultNamespaceBundleSplitAlgorithm=range_equally_divide

# load shedding strategy, support OverloadShedder and ThresholdShedder, default is OverloadShedder
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder

# The broker resource usage threshold.
# When the broker resource usage is gratter than the pulsar cluster average resource usge,
# the threshold shedder will be triggered to offload bundles from the broker.
# It only take effect in ThresholdSheddler strategy.
loadBalancerBrokerThresholdShedderPercentage=10

# When calculating new resource usage, the history usage accounts for.
# It only take effect in ThresholdSheddler strategy.
loadBalancerHistoryResourcePercentage=0.9

# The BandWithIn usage weight when calculating new resourde usage.
# It only take effect in ThresholdShedder strategy.
loadBalancerBandwithInResourceWeight=1.0

# The BandWithOut usage weight when calculating new resourde usage.
# It only take effect in ThresholdShedder strategy.
loadBalancerBandwithOutResourceWeight=1.0

# The CPU usage weight when calculating new resourde usage.
# It only take effect in ThresholdShedder strategy.
loadBalancerCPUResourceWeight=1.0

# The heap memory usage weight when calculating new resourde usage.
# It only take effect in ThresholdShedder strategy.
loadBalancerMemoryResourceWeight=1.0

# The direct memory usage weight when calculating new resourde usage.
# It only take effect in ThresholdShedder strategy.
loadBalancerDirectMemoryResourceWeight=1.0

# Bundle unload minimum throughput threshold (MB), avoding bundle unload frequently.
# It only take effect in ThresholdShedder strategy.
loadBalancerBundleUnloadMinThroughputThreshold=10

### --- Replication --- ###

# Enable replication metrics
replicationMetricsEnabled=true

# Max number of connections to open for each broker in a remote cluster
# More connections host-to-host lead to better throughput over high-latency
# links.
replicationConnectionsPerBroker=16

# Replicator producer queue size
replicationProducerQueueSize=1000

# Replicator prefix used for replicator producer name and cursor name
replicatorPrefix=pulsar.repl

# Duration to check replication policy to avoid replicator inconsistency
# due to missing ZooKeeper watch (disable with value 0)
replicationPolicyCheckDurationSeconds=600

# Default message retention time
defaultRetentionTimeInMinutes=0

# Default retention size
defaultRetentionSizeInMB=0

# How often to check whether the connections are still alive
keepAliveIntervalSeconds=30

# bootstrap namespaces
bootstrapNamespaces=

### --- WebSocket --- ###

# Enable the WebSocket API service in broker
webSocketServiceEnabled=false

# Number of IO threads in Pulsar Client used in WebSocket proxy
webSocketNumIoThreads=8

# Number of connections per Broker in Pulsar Client used in WebSocket proxy
webSocketConnectionsPerBroker=8

# Time in milliseconds that idle WebSocket session times out
webSocketSessionIdleTimeoutMillis=300000

# The maximum size of a text message during parsing in WebSocket proxy
webSocketMaxTextFrameSize=1048576

### --- Metrics --- ###

# Enable topic level metrics
exposeTopicLevelMetricsInPrometheus=true

# Enable consumer level metrics. default is false
exposeConsumerLevelMetricsInPrometheus=false

# Classname of Pluggable JVM GC metrics logger that can log GC specific metrics
# jvmGCMetricsLoggerClassName=

### --- Functions --- ###

# Enable Functions Worker Service in Broker
functionsWorkerEnabled=false

### --- Broker Web Stats --- ###

# Enable topic level metrics
exposePublisherStats=true
statsUpdateFrequencyInSecs=60
statsUpdateInitialDelayInSecs=60

# Enable expose the precise backlog stats.
# Set false to use published counter and consumed counter to calculate, this would be more efficient but may be inaccurate.
# Default is false.
exposePreciseBacklogInPrometheus=false

### --- Schema storage --- ###
# The schema storage implementation used by this broker
schemaRegistryStorageClassName=org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorageFactory

# Enforce schema validation on following cases:
#
# - if a producer without a schema attempts to produce to a topic with schema, the producer will be
#   failed to connect. PLEASE be carefully on using this, since non-java clients don't support schema.
#   if you enable this setting, it will cause non-java clients failed to produce.
isSchemaValidationEnforced=false

### --- Ledger Offloading --- ###

# The directory for all the offloader implementations
offloadersDirectory=./offloaders

# Driver to use to offload old data to long term storage (Possible values: S3, aws-s3, google-cloud-storage)
# When using google-cloud-storage, Make sure both Google Cloud Storage and Google Cloud Storage JSON API are enabled for
# the project (check from Developers Console -> Api&auth -> APIs).
managedLedgerOffloadDriver=

# Maximum number of thread pool threads for ledger offloading
managedLedgerOffloadMaxThreads=2

# Maximum prefetch rounds for ledger reading for offloading
managedLedgerOffloadPrefetchRounds=1

# Use Open Range-Set to cache unacked messages
managedLedgerUnackedRangesOpenCacheSetEnabled=true

# For Amazon S3 ledger offload, AWS region
s3ManagedLedgerOffloadRegion=

# For Amazon S3 ledger offload, Bucket to place offloaded ledger into
s3ManagedLedgerOffloadBucket=

# For Amazon S3 ledger offload, Alternative endpoint to connect to (useful for testing)
s3ManagedLedgerOffloadServiceEndpoint=

# For Amazon S3 ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864

# For Amazon S3 ledger offload, Read buffer size in bytes (1MB by default)
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576

# For Google Cloud Storage ledger offload, region where offload bucket is located.
# reference this page for more details: https://cloud.google.com/storage/docs/bucket-locations
gcsManagedLedgerOffloadRegion=

# For Google Cloud Storage ledger offload, Bucket to place offloaded ledger into
gcsManagedLedgerOffloadBucket=

# For Google Cloud Storage ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864

# For Google Cloud Storage ledger offload, Read buffer size in bytes (1MB by default)
gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576

# For Google Cloud Storage, path to json file containing service account credentials.
# For more details, see the "Service Accounts" section of https://support.google.com/googleapi/answer/6158849
gcsManagedLedgerOffloadServiceAccountKeyFile=

#For File System Storage, file system profile path
fileSystemProfilePath=../conf/filesystem_offload_core_site.xml

#For File System Storage, file system uri
fileSystemURI=

### --- Deprecated config variables --- ###

# Deprecated. Use configurationStoreServers
globalZookeeperServers=

# Deprecated - Enable TLS when talking with other clusters to replicate messages
replicationTlsEnabled=false

# Deprecated. Use brokerDeleteInactiveTopicsFrequencySeconds
brokerServicePurgeInactiveFrequencyInSeconds=60

### --- Transaction config variables --- ###

# Enable transaction coordinator in broker
transactionCoordinatorEnabled=false
transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider

@Demogorgon314

I also encountered this problem, the phenomenon is the same as the above, fix the PR: https://github.com/streamnative/kop/pull/946, and then increase the requestQueue queue size: maxQueuedRequests=2000


Bump to version 0.4.0

What does this PR do?
Bumping version for new release.

Codecov Report

Merging #24 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #24   +/-   ##
=======================================
  Coverage   88.61%   88.61%
=======================================
  Files          19       19
  Lines         483      483
  Branches       75       75
=======================================
  Hits          428      428
  Misses         34       34
  Partials       21       21

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e11ad3...e91ad5a. Read the comment docs.


Fixes #2933 Add CacheResponseGenerateHash and CacheResponseSystemStates events

1. Why is this change necessary?
Currently it is quity hacky (override the set cookie on the response event) to add extra 'layers' to the context cookie. Adding events will make it possible for a plugin to hook into the cache key generation easily. This is needed for example when a plugin uses some custom code to alter the storefront for certain customers.
2. What does this change do, exactly?
Adds 2 events in CacheResponseSubscriber

buildCacheHash
getSystemStates

3. Describe each step to reproduce the issue or behaviour.
4. Please link to the relevant issues (if any).
https://github.com/shopware/platform/issues/2933
5. Checklist

[x] I have rebased my changes to remove merge conflicts
[ ] I have written tests and verified that they fail without my change
[x] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[x] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

🤖 Generated by Copilot at 5502b34
This pull request introduces two new events, CacheResponseGenerateHashEvent and CacheResponseSystemStatesEvent, to the CacheResponseSubscriber class, allowing plugins to customize the cache hash and the system states used for cache invalidation. It also updates the tests, the changelog and the service configuration to reflect the changes.

🤖 Generated by Copilot at 5502b34

Add custom events to the CacheResponseSubscriber class to allow modifying the cache hash and system states (link, link, link, link, link, link, link)

Define the CacheResponseGenerateHashEvent class for the cache hash event (link)
Define the CacheResponseSystemStatesEvent class for the system states event (link)

Add the event_dispatcher service as a dependency to the CacheResponseSubscriber service in the XML configuration file (link)
Add the EventDispatcher as an argument to the CacheResponseSubscriber constructor in the unit test methods (link, link, link, link, link, link, link, link, link, link, link, link, link, link)
Add a changelog entry for the pull request (link)

Can you do this in gitlab


discrete celery

Config and task changes to support running unique celery daemon per portal instance.
Added endpoint /celery-info to farm a task to celery to report its configured CELERY_BROKER_URL and SERVER_NAME - to confirm which service we're hitting.
NB - the actual setup to run a unique celery daemon per portal is driven by puppet changes in the t__celerydaemonpervhost branch.  This will also require changes on the portal's application.cfg file such as:
# Use a unique redis db (the integer value following the last '/' in the REDIS_URL)
# so named celery daemon only processes requests from this application
REDIS_URL = 'redis://localhost:6379/15'
CELERY_BROKER_URL = REDIS_URL
REQUEST_CACHE_URL =  REDIS_URL
SESSION_REDIS_URL =  REDIS_URL

@ivan-c You are welcome to merge this when ready.  We do NOT need the puppet branch merged prior to rolling this out - in fact the sooner the better, as we can test the /celery-info link to see which celery daemon is handling the request.
I would hold off on merging the puppet branch - that will likely be a bit bumpy in bringing the newly named daemons up in place of old, and having the config correctly setup beforehand.

@ivan-c as per your suggestion, we're no longer defining a REDIS_DB value, but expecting the REDIS_URL (if defined) to include the trailing /<int db> value.

Great, will merge as soon as tests pass. Go ahead if you get impatient :)


LUCENE-9907: Remove packedInts#getReaderNoHeader dependency on TermsVectorFieldsFormat

Replaces the usages of PackedInts#getReaderNoHeader with DirecReader#getInstance.

I have added the length needed to store the int array so we can retrieve it before reading it.
In the read side, I found that wrapping the IndexInput is tricky as you might be changing the current position of the index at any point. I took a different approach and I am reading the data into a byte[] which is equivalent to what we were doing before by reading it into a long[].


SPO driver: Blobs should not have epoch

Not urgent issue, but worth addressing for better perf / consistency
Blobs in the system are immutable.
We should double-check claims in this issue, and make sure that blob IDs are unique, i.e. they is no way to get duplicate ID after epoch change (which is possible if, for example, IDs were sequential numbers, and file restore would reset counter back in time).
But assuming blob IDs are unique across restores / file being overwritten, the following should be true:

Blob fetch API should not have epoch payload and epoch should not be validated on client or on server,
removeAllEntriesForDocId() should not exist, but rather changed to include only file & type properties from ICacheEntry, i.e. leave ability for driver to delete individual entries for a file (snapshots, blobs, ops), not all the data.

Even if this work can't be done because blob entries are not unique, removeAllEntriesForDocId() should be changed to at least take IFileEntry not docID, as all other APIs take ICacheEntry that uses IFileEntry as file identification, not docId.

This is work for sharepoint? Or you want to harden our systems to not rely on the epoch data?

Out system should be more performant in presence of epoch changes, and not discard immutable blobs from host caches.
IN order to do so, changes are required on SPO (likely) and SPO driver (for sure)

As per latest discussion, we should not assume anything on epoch change as blob id 'a' with epoch 'x' != blob with id 'a' and epoch 'y'. So we should clear all data in cache on epoch change. So item 1 will remain as it is.


Add files via upload

.

@unkaragoz First stage of verification passed, official domain is twitter.com as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://twitter.com/stthomasuniv/status/1230158923577073666?lang=bg

@unkaragoz I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host stu.edu not found: 2(SERVFAIL) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for your understanding!


Gatekeeper cannot get OpenID configuration

What:
I'm trying to use Gatekeeper to protect an application. I'm getting the error shown below in the log output:
1.5804951742046025e+09	info	starting the service	{"prog": "keycloak-gatekeeper", "author": "Keycloak", "version": "7.0.0 (git+sha: f66e137, built: 03-09-2019)"}
1.58049517420485e+09	info	attempting to retrieve configuration discovery url	{"url": "https://login.citygate.io/auth/realms/master", "timeout": "30s"}
1.580495184205164e+09	warn	failed to get provider configuration from discovery	{"error": "Get https://login.citygate.io/auth/realms/master/.well-known/openid-configuration: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
My Gatekeeper configuration looks like this:
{{discovery-url: https://login.citygate.io/auth/realms/master}}
{{client-id: docker-admin}}
{{client-secret: }}
{{listen: :80}}
{{enable-refresh-tokens: true}}
{{redirection-url: https://docker-admin.citygate.io}}
{{encryption-key: }}
{{upstream-url: http://app:5000}}
{{resources:}}
{{- uri: /*}}
{{methods:}}
{{- GET}}
{{- POST}}
{{roles:}}
{{- admin}}
Reference:

KEYCLOAK-12866 - Gatekeeper cannot get OpenID configuration

Hi @abstractj,
Thanks for migrating this issue! I did work around this problem by manually creating an nsswitch.conf file and bind mounting it inside the container at /etc/nsswitch.conf - file contents are below.
hosts: files dns

I assume this forces it to look at /etc/hosts first, which is the desired behaviour because I had used the extra_hosts option in docker-compose.yml to manually set the IP for my Keycloak server.
Hope this helps!

Thanks for reporting it back. For now I'm closing this issue, as this seems more like a Docker configuration issue, instead of an issue on Louketo.
@ASzc maybe the introduction of nsswitch.conf is something that we can take into consideration for #609 to prevent issues like this. But I will leave that up to you.

In #638, with UBI, there will be a /etc/nsswitch.conf file present in the image. It's host line is the following:
hosts:      files dns myhostname

Which I think is acceptable as far as this issue goes, so no further action is required, unless in the future we go to scratch or similar options.

Hi, I have a similar problem with Bitnami Image here https://github.com/bitnami/bitnami-docker-keycloak-gatekeeper/issues/11 so I'm trying Louketo too, but same problem appears.
My docker-compose.yml file.
version: '3.7'

services:
  postgres:
    image: postgres
    volumes:
      - type: bind
        source: ../postgres_data
        target: /var/lib/postgresql/data
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=Pa55w0rd
    ports:
      - 5432:5432
    restart: unless-stopped
  pgadmin:
    image: dpage/pgadmin4
    environment:
      PGADMIN_DEFAULT_EMAIL<EMAIL_ADDRESS>      PGADMIN_DEFAULT_PASSWORD: 'Pa55w0rd'
    volumes:
      - type: bind
        source: ../pgadmin_data
        target: /var/lib/pagadmin
    ports:
      - 5050:80
    restart: unless-stopped
    depends_on:
      - postgres
  ldap:
    image: i2hm/openldap:latest
  keycloak:
    image: quay.io/keycloak/keycloak:latest
    environment:
      DB_VENDOR: POSTGRES
      DB_ADDR: postgres
      DB_DATABASE: keycloak
      DB_USER: postgres
      DB_SCHEMA: public
      DB_PASSWORD: Pa55w0rd
      KEYCLOAK_USER: admin
      KEYCLOAK_PASSWORD: Pa55w0rd
    ports:
      - 8080:8080
    expose:
      - 8080
    depends_on:
      - postgres
  louketo-proxy:
    image: quay.io/louketo/louketo-proxy:latest
    command: >-
      --config /opt/louketo/config.yml
    ports:
      - 3000:3000
    volumes:
      - type: bind
        source: ../gatekeeper/config.yml
        target: /opt/louketo/config.yml
    depends_on:
      - keycloak
      - web
  web:
    build: .
    volumes:
      - type: bind
        source: ../gunicorn_log
        target: /var/log
    ports:
      - 5000:5000
    expose:
      - 5000
    depends_on:
      - postgres

My config file.
client-id: flask_example
client-secret: 7216c35c-0f0a-4dbf-adc8-beb0dc933f27
discovery-url: http://keycloak:8080/auth/realms/identite
enable-default-deny: true
listen: :3000
upstream-url: http://web:5000
resources:
- uri: /admin*
  methods:
  - GET
  roles:
  - client:test1
  - client:test2
  require-any-role: true
  groups:
  - admins
  - users
- uri: /backend*
  roles:
  - client:test1
- uri: /public/*
  white-listed: true
- uri: /favicon
  white-listed: true
- uri: /css/*
  white-listed: true
- uri: /img/*
  white-listed: true
headers:
  myheader1: value_1
  myheader2: value_2

I get this error.
louketo-proxy_1  | 2020-08-20T12:32:14.829Z	info	attempting to retrieve configuration discovery url	{"url": "http://keycloak:8080/auth/realms/identite", "timeout": "30s"}
louketo-proxy_1  | 2020-08-20T12:32:14.835Z	warn	failed to get provider configuration from discovery	{"error": "Get \"http://keycloak:8080/auth/realms/identite/.well-known/openid-configuration\": dial tcp: lookup keycloak on <IP_ADDRESS>:53: no such host"}
louketo-proxy_1  | 2020-08-20T12:32:17.835Z	info	attempting to retrieve configuration discovery url	{"url": "http://keycloak:8080/auth/realms/identite", "timeout": "30s"}
louketo-proxy_1  | 2020-08-20T12:32:17.848Z	warn	failed to get provider configuration from discovery	{"error": "Get \"http://keycloak:8080/auth/realms/identite/.well-known/openid-configuration\": dial tcp: lookup keycloak on <IP_ADDRESS>:53: no such host"}

Thanks for your help. Sorry for posting here and at bitnami too.

Hi @TcaManager ,
I had to add the extra_hosts directive to my config to overcome this. You'll want something like this:
louketo-proxy:
    image: quay.io/louketo/louketo-proxy:latest
    command: >-
      --config /opt/louketo/config.yml
    ports:
      - 3000:3000
    volumes:
      - type: bind
        source: ../gatekeeper/config.yml
        target: /opt/louketo/config.yml
    depends_on:
      - keycloak
      - web
    extra_hosts:
      - "keycloak:X.X.X.X"

Swap `X.X.X.X` for the internal or external IP address of the Keycloak server.

extra_hosts:
      - "keycloak:X.X.X.X"

Interesting @luketainton , thanks for your help. What do you mean with internal/external IP address? Do I need to get any ifconfig docker interface IP? My host has many FQDN and IPs in /etc/hosts but I'm on Linux and I wonder which IP would my container use to reach the host? I saw docker0 in ifconfig and many interfaces br-9e188c5db690 (br for bridge?)

extra_hosts:
      - "keycloak:X.X.X.X"

Interesting @luketainton , thanks for your help. What do you mean with internal/external IP address? Do I need to get any ifconfig docker interface IP? My host has many FQDN and IPs in /etc/hosts but I'm on Linux and I wonder which IP would my container use to reach the host? I saw docker0 in ifconfig and many interfaces br-9e188c5db690 (br for bridge?)

Is your Keycloak container accessible from other hosts? If so, it'll be the IP address of the host. Otherwise try copying the depends_on section but renaming it to links, then remove the extra_hosts section.

@TcaManager it'll be whatever the address is on the NIC that represents that particular Docker network - you can run docker inspect <NAME>, which will tell you the details (inc. IP address) of the container that you specify.
Otherwise, you can try this instead:
louketo-proxy:
    image: quay.io/louketo/louketo-proxy:latest
    command: >-
      --config /opt/louketo/config.yml
    ports:
      - 3000:3000
    volumes:
      - type: bind
        source: ../gatekeeper/config.yml
        target: /opt/louketo/config.yml
    depends_on:
      - keycloak
      - web
    links:
      - keycloak
      - web


Update/Install re-adds content items if they have been modified to EmbeddedResource

If you have a package with content, and you want that content then to become an embedded resource (you switch it in VS), a paket update or paket install will add the file a second time to the project (i.e. it appears twice in VS).
Just wondering if this is a bug, or if there is some paket.dependencies jigpokery I can do to stop this.
Repro steps
As described above.

add a package that has content.
open solution in VS
change content file to EmbeddedResource in properties pane
run paket update

Expected behavior
I'd expect there to be only 1 file, or there to be a way in paket.dependencies to ensure that the content is embeded, but given that you may only want to do this to certain files, maybe not adding it again would be preferable.
Actual behavior
file appears twice in the project.
Known workarounds
unknown
Related information

Operating system = Windows
Branch = Paket version <IP_ADDRESS>
.NET Runtime 4.5

yeah I guess this is not supported yet. we would need a repro sample ad then we can try to make that work

/cc @sergey-tihon

Yes, it is not supported, would be nice to know package name and type of content.
Now packet identify build action based on content type and then add to the project.
If you change build action to EmbeddedResource on the next install\update packet cannot find file (that he added) and add it once again with default build action.

Can't get you a repro right now, but will try to remember to do it over the weekend.
On a related note, this is very much like (https://github.com/fsprojects/Paket/issues/1555).  Given both those issues,is it perhaps better to not manage content files from paket, except to copy or not copy them to the solution, and add if not there?  Would solve both issues, but there may be a need/use case that I'm unaware of.


[REQUEST] Look at patches from darkxst for his multipan flasher fork of the Silabs Flasher

FYI, @darkxst has a downstream fork of Silabs Flasher add-on with patches that might be interested in looking at for your fork:
https://github.com/darkxst/multipan_flasher
https://github.com/darkxst/multipan_flasher/pulls?q=is%3Apr
https://github.com/darkxst/multipan_flasher/blob/main/multipan_flasher/DOCS.md
Thinking that perhaps him and you are also interested in collaborating on some patches from your own Silabs Flasher fork:
https://github.com/tube0013/tubeszb_addons/tree/main/tzb-silabs-flasher
https://github.com/tube0013/tubeszb_addons/blob/main/tzb-silabs-flasher/DOCS.md

I've been following the work puddly is doing in creating manifests for the different builds and will adopt that when it is finalized.


verify leadership transfer status before waiting

Type of change

Bug fix

Description
The thread which was waiting on the notifyc channel was not notified as the other thread well passed the execution before the channel gets established/initialized. Its a race condition between these 2 threads lead to the test case failure. Added additional check to check the status before waiting for notification from other go routine.
Related issues
#3629
Signed-off-by: Parameswaran Selvam<EMAIL_ADDRESS>

But even without this abdication, if the leader is removed, the other two nodes should be able to select a new leader. The test, tests this property. so it should succeed even if at the time of abdication, the follower didn't hear from the previous leader.

But even without this abdication, if the leader is removed, the other two nodes should be able to select a new leader. The test, tests this property. so it should succeed even if at the time of abdication, the follower didn't hear from the previous leader.

the test-case tests the property of whether all the nodes able to apply/commit the configuration change while rotating the current leader certificates.  https://github.com/hyperledger/fabric/blob/d18be4662d74bc3abef1b95fc703c62b4c80bfd5/orderer/consensus/etcdraft/chain_test.go#L1901 In the case of failure, one of the node was blocked on the leadership change notification condition, the node did not proceed further to make the configuration update which made the test to time out.
If we increase the test timeout value or decrease the election tick value, the blocked thread would timeout and hence the testcase would pass. This is another option. But, it does not solve the real race condition between the node Ready loop Thread and the abdicate leader thread.  As the current logic depends on the election_tick, the node is not latest wrt to the configuration changes as compared to other nodes in the quorum.


Update README.md

anything in particular you'd like it to be updated?
Pull Requests are welcome.

Thank you!


No module named... exceptions

Hi there! I have an another problem with the python predict_blend.py command. When I try to run it via anaconda shell I get an error that there is no module named "soundfile". When I try to run the same command in CMD I get an error that there is no module named "aicrowd_api".
The same kind of problem I had when tried to execute the "python download_demucs.py". In anaconda shell I got the there is no module named torch exception, but surprisingly I was able to run this command using CMD.

To be sure I deleted the old package to repeat the whole process again, so when I use conda env create -f environment.yml -n mdx-submit I get an error: Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies.  Conda may not use the correct pip to install your packages, and they may end up in the wrong place.  Please add an explicit pip dependency.  I'm adding one for you, but still nagging you. After that I following the instructions till the moment when I should execute download_demucs.py. It crashes with the same message about torch module.

Can you provide the full log to make troubleshooting easier?

Yep.
(base) PS C:\Users\elbit> cd C:\Users\elbit\mdx-net-submission
(base) PS C:\Users\elbit\mdx-net-submission> conda env create -f environment.yml -n mdx-submit
Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies.  Conda may not use the correct pip to install your packages, and they may end up in the wrong place.  Please add an explicit pip dependency.  I'm adding one for you, but still nagging you.
CondaValueError: prefix already exists: C:\Users\elbit\anaconda3\envs\mdx-submit
(base) PS C:\Users\elbit\mdx-net-submission> conda activate mdx-submit
(mdx-submit) PS C:\Users\elbit\mdx-net-submission> pip install -r requirements.txt
Collecting aicrowd_api
Using cached aicrowd_api-0.1.23-py2.py3-none-any.whl
Collecting coloredlogs
Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Requirement already satisfied: numpy in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from -r requirements.txt (line 3)) (1.22.4)
Collecting loguru
Using cached loguru-0.6.0-py3-none-any.whl (58 kB)
Collecting boto3
Using cached boto3-1.24.4-py3-none-any.whl (132 kB)
Collecting openunmix
Using cached openunmix-1.2.1-py3-none-any.whl (46 kB)
Collecting musdb
Using cached musdb-0.4.0-py2.py3-none-any.whl (29 kB)
Collecting SoundFile
Using cached SoundFile-0.10.3.post1-py2.py3.cp26.cp27.cp32.cp33.cp34.cp35.cp36.pp27.pp32.pp33-none-win_amd64.whl (689 kB)
Collecting scipy
Using cached scipy-1.8.1-cp39-cp39-win_amd64.whl (36.9 MB)
Collecting norbert
Using cached norbert-0.2.1-py2.py3-none-any.whl (11 kB)
Collecting asteroid>=0.5.0
Using cached asteroid-0.5.2-py3-none-any.whl (246 kB)
Collecting demucs==2.0.3
Using cached demucs-2.0.3-py3-none-any.whl
Collecting julius>=0.2.3
Using cached julius-0.2.6-py3-none-any.whl
Collecting tqdm
Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting torch>=1.8.1
Using cached torch-1.11.0-cp39-cp39-win_amd64.whl (157.9 MB)
Collecting torchaudio>=0.8
Using cached torchaudio-0.11.0-cp39-cp39-win_amd64.whl (372 kB)
Collecting diffq>=0.1
Using cached diffq-0.2.2-cp39-cp39-win_amd64.whl (101 kB)
Collecting lameenc>=1.2
Using cached lameenc-1.4.1-cp39-cp39-win_amd64.whl (158 kB)
Collecting requests>=2.18.4
Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting redis
Using cached redis-4.3.3-py3-none-any.whl (244 kB)
Collecting humanfriendly>=9.1
Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting win32-setctime>=1.0.0
Using cached win32_setctime-1.1.0-py3-none-any.whl (3.6 kB)
Collecting colorama>=0.3.4
Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting s3transfer<0.7.0,>=0.6.0
Using cached s3transfer-0.6.0-py3-none-any.whl (79 kB)
Collecting botocore<1.28.0,>=1.27.4
Using cached botocore-1.27.4-py3-none-any.whl (8.9 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting stempeg>=0.2.3
Using cached stempeg-0.2.3-py3-none-any.whl (963 kB)
Collecting pyaml
Using cached pyaml-21.10.1-py2.py3-none-any.whl (24 kB)
Collecting cffi>=1.0
Using cached cffi-1.15.0-cp39-cp39-win_amd64.whl (180 kB)
Collecting pandas>=0.23.4
Using cached pandas-1.4.2-cp39-cp39-win_amd64.whl (10.5 MB)
Collecting pytorch-lightning<1.5.0,>=1.0.1
Using cached pytorch_lightning-1.4.9-py3-none-any.whl (925 kB)
Collecting PyYAML>=5.0
Using cached PyYAML-6.0-cp39-cp39-win_amd64.whl (151 kB)
Collecting asteroid-filterbanks>=0.4.0
Using cached asteroid_filterbanks-0.4.0-py3-none-any.whl (29 kB)
Collecting torch-stoi>=0.1.2
Using cached torch_stoi-0.1.2-py3-none-any.whl
Collecting torch-optimizer<0.2.0,>=0.0.1a12
Using cached torch_optimizer-0.1.0-py3-none-any.whl (72 kB)
Collecting huggingface-hub>=0.0.2
Using cached huggingface_hub-0.7.0-py3-none-any.whl (86 kB)
Collecting pb-bss-eval>=0.0.2
Using cached pb_bss_eval-0.0.2-py3-none-any.whl (14 kB)
Collecting typing-extensions
Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting Cython
Using cached Cython-0.29.30-py2.py3-none-any.whl (985 kB)
Collecting packaging>=20.9
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting filelock
Using cached filelock-3.7.1-py3-none-any.whl (10 kB)
Requirement already satisfied: pyreadline3 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from humanfriendly>=9.1->coloredlogs->-r requirements.txt (line 2)) (3.4.1)
Requirement already satisfied: pytz>=2020.1 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from pandas>=0.23.4->asteroid>=0.5.0->-r requirements.txt (line 11)) (2022.1)
Collecting pesq
Using cached pesq-0.0.4.tar.gz (38 kB)
Preparing metadata (setup.py) ... done
Collecting mir-eval
Using cached mir_eval-0.7-py3-none-any.whl
Collecting pystoi
Using cached pystoi-0.3.3-py2.py3-none-any.whl
Collecting einops
Using cached einops-0.4.1-py3-none-any.whl (28 kB)
Collecting cached-property
Using cached cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Collecting pyDeprecate==0.3.1
Using cached pyDeprecate-0.3.1-py3-none-any.whl (10 kB)
Collecting tensorboard>=2.2.0
Using cached tensorboard-2.9.0-py3-none-any.whl (5.8 MB)
Collecting fsspec[http]!=2021.06.0,>=2021.05.0
Using cached fsspec-2022.5.0-py3-none-any.whl (140 kB)
Collecting future>=0.17.1
Using cached future-0.18.2-py3-none-any.whl
Collecting torchmetrics>=0.4.0
Using cached torchmetrics-0.9.0-py3-none-any.whl (418 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2<IP_ADDRESS>-py3-none-any.whl (155 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting ffmpeg-python>=0.2.0
Using cached ffmpeg_python-0.2.0-py3-none-any.whl (25 kB)
Collecting pytorch-ranger>=0.1.1
Using cached pytorch_ranger-0.1.1-py3-none-any.whl (14 kB)
Collecting deprecated>=1.2.3
Using cached Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Collecting async-timeout>=4.0.2
Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting wrapt<2,>=1.10
Using cached wrapt-1.14.1-cp39-cp39-win_amd64.whl (35 kB)
Collecting aiohttp
Using cached aiohttp-3.8.1-cp39-cp39-win_amd64.whl (554 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: protobuf>=3.9.2 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from tensorboard>=2.2.0->pytorch-lightning<1.5.0,>=1.0.1->asteroid>=0.5.0->-r requirements.txt (line 11)) (4.21.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from tensorboard>=2.2.0->pytorch-lightning<1.5.0,>=1.0.1->asteroid>=0.5.0->-r requirements.txt (line 11)) (1.8.1)
Requirement already satisfied: wheel>=0.26 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from tensorboard>=2.2.0->pytorch-lightning<1.5.0,>=1.0.1->asteroid>=0.5.0->-r requirements.txt (line 11)) (0.37.1)
Collecting absl-py>=0.4
Using cached absl_py-1.1.0-py3-none-any.whl (123 kB)
Requirement already satisfied: setuptools>=41.0.0 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from tensorboard>=2.2.0->pytorch-lightning<1.5.0,>=1.0.1->asteroid>=0.5.0->-r requirements.txt (line 11)) (62.3.2)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
Using cached tensorboard_data_server-0.6.1-py3-none-any.whl (2.4 kB)
Collecting werkzeug>=1.0.1
Using cached Werkzeug-2.1.2-py3-none-any.whl (224 kB)
Collecting google-auth<3,>=1.6.3
Using cached google_auth-2.7.0-py2.py3-none-any.whl (160 kB)
Collecting grpcio>=1.24.3
Using cached grpcio-1.46.3-cp39-cp39-win_amd64.whl (3.5 MB)
Collecting markdown>=2.6.8
Using cached Markdown-3.3.7-py3-none-any.whl (97 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
Using cached importlib_metadata-4.11.4-py3-none-any.whl (18 kB)
Collecting yarl<2.0,>=1.0
Using cached yarl-1.7.2-cp39-cp39-win_amd64.whl (122 kB)
Collecting frozenlist>=1.1.1
Using cached frozenlist-1.3.0-cp39-cp39-win_amd64.whl (33 kB)
Collecting multidict<7.0,>=4.5
Using cached multidict-6.0.2-cp39-cp39-win_amd64.whl (28 kB)
Collecting aiosignal>=1.1.2
Using cached aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
Collecting attrs>=17.3.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting zipp>=0.5
Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\users\elbit\anaconda3\envs\mdx-submit\lib\site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning<1.5.0,>=1.0.1->asteroid>=0.5.0->-r requirements.txt (line 11)) (0.4.8)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: pesq
Building wheel for pesq (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [25 lines of output]
C:\Users\elbit\anaconda3\envs\mdx-submit\lib\site-packages\setuptools\installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-39
creating build\lib.win-amd64-cpython-39\pesq
copying pesq_pesq.py -> build\lib.win-amd64-cpython-39\pesq
copying pesq_init_.py -> build\lib.win-amd64-cpython-39\pesq
copying pesq\cypesq.pyx -> build\lib.win-amd64-cpython-39\pesq
copying pesq\dsp.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesq.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqio.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqmain.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqpar.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\dsp.c -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqdsp.c -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqmod.c -> build\lib.win-amd64-cpython-39\pesq
running build_ext
cythoning pesq/cypesq.pyx to pesq\cypesq.c
c:\users\elbit\appdata\local\temp\pip-install-bl_rsrn5\pesq_4824e8fc012a42a1b80a8d78bc1ccd05.eggs\cython-0.29.30-py3.9.egg\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\elbit\AppData\Local\Temp\pip-install-bl_rsrn5\pesq_4824e8fc012a42a1b80a8d78bc1ccd05\pesq\cypesq.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'cypesq' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pesq
Running setup.py clean for pesq
Failed to build pesq
Installing collected packages: pesq, lameenc, einops, cached-property, zipp, wrapt, win32-setctime, werkzeug, urllib3, typing-extensions, tensorboard-data-server, six, scipy, rsa, PyYAML, pyparsing, pyDeprecate, pycparser, pyasn1-modules, oauthlib, multidict, jmespath, idna, humanfriendly, future, fsspec, frozenlist, filelock, Cython, colorama, charset-normalizer, certifi, cachetools, attrs, async-timeout, absl-py, yarl, tqdm, torch, requests, python-dateutil, pystoi, pyaml, packaging, norbert, mir-eval, loguru, importlib-metadata, grpcio, google-auth, ffmpeg-python, deprecated, coloredlogs, cffi, aiosignal, torchmetrics, torchaudio, stempeg, SoundFile, requests-oauthlib, redis, pytorch-ranger, pb-bss-eval, pandas, markdown, julius, huggingface-hub, diffq, botocore, asteroid-filterbanks, aiohttp, torch-stoi, torch-optimizer, s3transfer, openunmix, musdb, google-auth-oauthlib, demucs, aicrowd_api, tensorboard, boto3, pytorch-lightning, asteroid
Running setup.py install for pesq ... error
error: subprocess-exited-with-error
× Running setup.py install for pesq did not run successfully.
│ exit code: 1
╰─> [25 lines of output]
C:\Users\elbit\anaconda3\envs\mdx-submit\lib\site-packages\setuptools\installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
running install
C:\Users\elbit\anaconda3\envs\mdx-submit\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-39
creating build\lib.win-amd64-cpython-39\pesq
copying pesq_pesq.py -> build\lib.win-amd64-cpython-39\pesq
copying pesq_init_.py -> build\lib.win-amd64-cpython-39\pesq
copying pesq\cypesq.pyx -> build\lib.win-amd64-cpython-39\pesq
copying pesq\dsp.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesq.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqio.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqmain.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqpar.h -> build\lib.win-amd64-cpython-39\pesq
copying pesq\dsp.c -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqdsp.c -> build\lib.win-amd64-cpython-39\pesq
copying pesq\pesqmod.c -> build\lib.win-amd64-cpython-39\pesq
running build_ext
skipping 'pesq\cypesq.c' Cython extension (up-to-date)
building 'cypesq' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> pesq
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
(mdx-submit) PS C:\Users\elbit\mdx-net-submission> python download_demucs.py
Traceback (most recent call last):
File "C:\Users\elbit\mdx-net-submission\download_demucs.py", line 1, in
import torch
ModuleNotFoundError: No module named 'torch'

OK, it seems to work. Thank you for information. You can close this ticket.