Unable to consume messages without increasing pollDuration in consumer settings

Expected Behavior
I expected a consumer group to be created, and the topics (either topic or topicPattern) to show up in that consumer group with appropriate lag.
Actual Behaviour
What happened is the consumer group gets created, but it is always empty. And no messages get consumed from my topics. When we increased pollDuration to 10s things started working as expected. We reproduced this with the trigger and the consumer task. With topic (2 topics as a list of strings) and a topicPattern regex (that matched 81 topics, per the kafka debug logs on kestra).
Steps To Reproduce
We have a largish kafka environment with many topics and partitions. I assume this is necessary to reproduce the issue.
Environment Information

Kestra Version: v0.16.2
Plugin version: v0.16.0
Operating System (OS / Docker / Kubernetes): Kubernetes / GKE v1.28.7-gke.1026000
Java Version (If not docker): 17 as in the container

Example flow
No response

Hey, sorry for getting back to you late on this.
Could you share the flow that fails for you? are you using a Kafka trigger or task?
Regarding pollDuration: we've recently merged Realtime Trigger that will allow you to react to Kafka messages in real-time without polling https://github.com/kestra-io/plugin-kafka/commit/7c26646df3f95fb810db58de46becd65ca6219de -- it's available in Beta on the develop image

closing as:

we will release RealtimeTrigger for Kafka in ~1 week (4th of June 2024) - this should no longer have that issue as it triggers one execution per message
we cannot reproduce without a workflow example -- if you still see the issue, please reopen and add a flow example


Note for chromedriver.exe dir

Make sure that the chromedriver.exe is under C:\ChromeDriver

Do you think that the code can be improved by downloading courses from a list and not only one link ?
Thanks!

Do you think that the code can be improved by downloading courses from a list and not only one link ?
Thanks!

Yes, that can be done. Currently, I do not have an active LinuxAcademy subscription. If someone can voluntarily share their subscription with me, I shall work on the issues/requests. I'm sorry :(

Do you think that the code can be improved by downloading courses from a list and not only one link ?
Thanks!

Yes, that can be done. Currently, I do not have an active LinuxAcademy subscription. If someone can voluntarily share their subscription with me, I shall work on the issues/requests. I'm sorry :(

Hey bro, I cannot share my account, however I can send you some money via paypal so that you can resubscribe again.


[VL] Optimize the performance of hash based shuffle by accumulating batches

What changes were proposed in this pull request?
I used perf to observe the benchmark and found that the most time-consuming functions were splitFixedWidthValueBuffer and splitBinaryType. However, current column computing engines (such as starrocks) also use this idea of exchanging random read memory overhead for sequential write memory overhead to implement the split function, so I think there is not much room for optimization of the split function.

I found that when the ShuffleBatchSize is increased, the performance will be significantly improved. I think the performance benefits mainly come from the following aspects:

It can give full play to the advantages of sequential memory writing in the split stage. When PartitionNum is 10000 and ShuffleBatchSize is 4096 (the default value in the benchmark), each Partition is only allocated 1 row of data at most (the data obtained by logging statistics in the benchmark). At this time, it is obviously impossible to give full play to the advantages of sequential memory writing.

It can reduce the number of function calls and the number of memory allocations.

Therefore, I implemented this PR to cache the data to be Shuffled, which can optimize the performance of ShuffleWrite. For specific test data, please refer to the screenshot below.
I think this PR can also control whether to cache data in combination with memory usage, thereby avoiding the ShuffleWrite OOM problem.
How was this patch tested?
Command
./build/velox/benchmarks/shuffle_split_benchmark --file=/root/shuffleSplitBenchmark/cpp/velox/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet --partitions=10000 --iterations=100 --threads=1

Before optimize

After optimize

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

/Benchmark Velox

Thank you for the improvement.
The ideal case of current split function is that: the input batch size should be as large as possible but all columns can fit into L2 cache. Once the column data can't fit into L2 cache, split performance will drop dramatically.
On the other side if the batch size is too small, the overhead of preparing of split actually can't be ignored. We should have room to improve with this case.
Also in current implementation, we must flatten the rowvector to split. The flatten itself also have overhead which is not analyzed.

  template <typename T>
  arrow::Status splitFixedType(const uint8_t* srcAddr, const std::vector<uint8_t*>& dstAddrs) {
    for (auto& pid : partitionUsed_) {
      auto dstPidBase = (T*)(dstAddrs[pid] + partitionBufferBase_[pid] * sizeof(T));
      // record dstPidBase
      ...

@FelixYBW I added a log at the above location and found that the interval between dstPidBase is about 16 KB. I think this can also be optimized to make the interval between dstPidBase very small, which will help reduce cache misses.

Oops. I missed this PR before opening this for similar purpose...

The change makes sense to me. I think it's operational to merge this and use #6009 as follow-up which adds an individual Spark operator controlling this behavior for being reused for other operators in future (say, joins or aggs) by some kind of strategies. Let me know if any thoughts @XinShuoWang @FelixYBW @marin-ma

@zhztheplayer @Yohahaha Can you help me to fix the failed CI? The error message is java.lang.IllegalStateException: nativeShuffleWriter should not be -1L

@zhztheplayer @Yohahaha Can you help me to fix the failed CI? The error message is java.lang.IllegalStateException: nativeShuffleWriter should not be -1L

I think it's Uniffle's error and has been fixed on main branch, try rebase.

The change makes sense to me. I think it's operational to merge this and use #6009 as follow-up which adds an individual Spark operator controlling this behavior for being reused for other operators in future (say, joins or aggs) by some kind of strategies. Let me know if any thoughts. @XinShuoWang @FelixYBW @marin-ma
velox has a feature to merge small vector for the output of aggs,filter and join，ref https://github.com/facebookincubator/velox/pull/7899/files but the pr is not merge to master. shall we go on the pr to merge it to master?
@zhztheplayer

Sounds great to me that someone could take on the remaining work of https://github.com/facebookincubator/velox/pull/7899. This batch resizing work doesn't have to be in individual operators. It's enough to see only some extra metrics showing the resizing time from relevant operators.


Add support for Nucleo U5A5ZJ-Q

Add support for Nucleo U5A5ZJ-Q

Rebased to fix a test compilation error. Hopefully all will be green now


Fixes sentence cut-off bug in Make Circular Images with a Border Radius

When completing 'Make Circular Images with a Border Radius', the sentence is cut off at the comma (see left). This fixes it (see right).

Tested i Firefox 38 and Chromium 43.

Also, I just realized I printscreened the "after" image when I had just removed the comma. I just checked and the comma is definitely there in both browsers with my solution.

Nice catch. I hadn't realized that commas needed HTML codes in these test output.


Can not find 'ovn-k8s-overlay' file when building the ovn-kube image

hi
I'm trying rebuild the ovn-kube image in ovn-kubernetes/dist/images, there is one step in the Dockerfile
Step 10/15 : COPY ovn-k8s-overlay ovnkube ovn-kube-util /usr/bin/
lstat ovn-k8s-overlay: no such file or directory

I already built the related in ovn-kubernetes/go-controller
# make install
install -D -m 755 _output/go/bin/ovnkube /usr/bin/
install -D -m 755 _output/go/bin/ovn-kube-util /usr/bin/
install -D -m 755 _output/go/bin/ovn-k8s-cni-overlay -t /opt/cni/bin/

Seems there is NO this file ovn-k8s-overlay, is this file still need?
thanks

This is fixed.
cd ovn-kuberntes
docker build -t ovn-kube -f Dockerfile.centos .

thanks @pecameron


⚠ Jenkins Plugins Artifactory has degraded performance

In 160e100, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1380 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 81dd1ec after 39 minutes.


Character settings and selected model is reset every time the app is opened

Every time I restart the app, I have to go to the character, edit, import from json, find the file, go to settings, model settings, import from huggingface, mistral 7b instruct, Q_4, select, and only then can I chat with the chatbot normally.
Can you please update the app to save these settings between sessions, so it doesn't go back to default every time I open the app?

assign this to me wanna give this a try


[stable/redis-ha]: add redis source to restore options

What this PR does / why we need it:
This PR adds support for using a remote Redis instance as a data source for restoration. It utilizes the redis-cli --rdb command to perform the restore operation.
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

@DandyDeveloper

@hawwwdi Can we work through those inline changes your IDE has updated and undo them for now?
Also, can you provide some logging showing a successful restore operation? Backups testing locally for me is a bit of a nightmare :)

@DandyDeveloper, I have fixed those inline changes and rolled them back. Additionally, I’ve made some changes to the restoration script: it is now only executed for the initial master instance and checks the connectivity of the provided Redis endpoint to prevent getting stuck in a crashLoopBackOff state in case of a pod restart.

example logs of a successful restore operation:
restore-redis init container:
k logs -n redis test-redis-ha-server-0 -c restore-redis -f
test-redis.test-restore:6379 (<IP_ADDRESS>:6379) open
sending REPLCONF capa eof
sending REPLCONF rdb-only 1
SYNC sent to master, writing<PHONE_NUMBER> bytes to '/data/dump.rdb_'
Transfer finished with success.
'/data/dump.rdb_' -> '/data/dump.rdb'

redis instance:
k exec -it -n redis test-redis-ha-server-0 -- sh -c "echo info | redis-cli | tail -2"
Defaulted container "redis" out of: redis, sentinel, split-brain-fix, redis-exporter, config-init (init), restore-redis (init)
# Keyspace
db1:keys=107713981,expires=0,avg_ttl=0

@DandyDeveloper reminder

I'll get it checked and if all good, I'll merge tomorrow!

@hawwwdi I'll approved to get the test run but can you make this a minor rather than a patch? It's a big addition. The versioning should probably reflect this.

@hawwwdi I'll approved to get the test run but can you make this a minor rather than a patch? It's a big addition. The versioning should probably reflect this.

I fixed it.


[Umbrella] Sort-Standalone support data rollback

Describe the proposal
In some case, the down stream system break down, we need to rollback a stream or a task with certain time interval.
It require sort-standalone support consume data from a certain StartTime, and stop in StopTime
Task list

[x] #5094
[x] #5095
[x] #5418

InLong Component
InLong Sort
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

All tasks have been done.


Change format of persisted quotes

Store quotes as 3 column table:

date
symbol
price (closing)

Cool change! Quotes is so much more simple now :smile:


Fixed Travis build fail because of unused imports

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Iam role check

Check for the number of IAM roles in an account.
Fixes #22

@charleshepner Thanks for opening this PR. Unfortunately the service code you are referencing does not exist for my AWS account:
...
botocore.errorfactory.NoSuchResourceException: An error occurred (NoSuchResourceException) when calling the GetServiceQuota operation:
...
botocore.errorfactory.NoSuchResourceException: An error occurred (NoSuchResourceException) when calling the GetAWSDefaultServiceQuota operation: The request failed because the specified service does not exist.

Did you request a quota increase for the number of roles in your AWS account?

Implemented in https://github.com/brennerm/aws-quota-checker/commit/da7b983f4ee9eefd84d4ecbd1da2b0bbe1b30452


Change citizen email

Make it possible for a citizen to change their email address.
This should include #210

Not now


MatrixExclusions don't seem to work with JavaSpec

From the Cats Effect 3 build:
This is how the build used to be (GraalVM was a String).
  // Nice-to-haves but unreliable in CI
  val flakyFilters = Seq(
    MatrixExclude(Map("os" -> Windows, "java" -> GraalVM))
  )

This now results in a compile error, which is fine, calling .render should fix it.
However, that results in the following error:
sbt:root> last githubWorkflowGenerate
[error] java.lang.RuntimeException: inclusion key `java` was present in matrix, but value `temurin@11` was not in List(JavaSpec(Temurin,8), JavaSpec(Temurin,11), JavaSpec(Temurin,17), JavaSpec(GraalVM(21.3),11))
[error]         at scala.sys.package$.error(package.scala:30)
[error]         at sbtghactions.GenerativePlugin$.$anonfun$compileJob$12(GenerativePlugin.scala:355)
[error]         at sbtghactions.GenerativePlugin$.$anonfun$compileJob$12$adapted(GenerativePlugin.scala:348)
[error]         at scala.collection.immutable.Map$Map2.foreach(Map.scala:273)
[error]         at sbtghactions.GenerativePlugin$.checkMatching$1(GenerativePlugin.scala:348)
[error]         at sbtghactions.GenerativePlugin$.$anonfun$compileJob$15(GenerativePlugin.scala:378)
[error]         at sbtghactions.GenerativePlugin$.$anonfun$compileJob$15$adapted(GenerativePlugin.scala:378)
[error]         at scala.collection.immutable.List.foreach(List.scala:431)
[error]         at sbtghactions.GenerativePlugin$.compileJob(GenerativePlugin.scala:378)
[error]         at sbtghactions.GenerativePlugin$.$anonfun$compileWorkflow$8(GenerativePlugin.scala:471)
[error]         at scala.collection.immutable.List.map(List.scala:293)
[error]         at sbtghactions.GenerativePlugin$.compileWorkflow(GenerativePlugin.scala:471)
[error]         at sbtghactions.GenerativePlugin$.$anonfun$generateCiContents$1(GenerativePlugin.scala:702)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]         at sbt.Execute.work(Execute.scala:291)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error]         at java.base/java.lang.Thread.run(Thread.java:833)
[error] (githubWorkflowGenerate) inclusion key `java` was present in matrix, but value `temurin@11` was not in List(JavaSpec(Temurin,8), JavaSpec(Temurin,11), JavaSpec(Temurin,17), JavaSpec(GraalVM(21.3),11))

This is fixed in 0.14.1. I ran into it with the Cats build as well. Just need to catch up sbt-spiewak now.


Add overseerr extension

Description

Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

i'm not a dev so it's unclear to me if there's anything i need to fix in order for this to get approved. please advise 😊

@pernielsentikaer super noob question... but how do i request a new review?

You can see here how to request it 🙂

so i added axios to my json package in my main branch + requested a new review

it can be used to request any type of media that is not copyrighted
it uses sonarr and radarr. there's a raycast extention for sonarr so it can be used in combination with that
https://www.raycast.com/Aayush9029/sonarr
it's also an alternative to Jellyfin
https://www.raycast.com/d2a/jellyfin

@pernielsentikaer just checking in for an update, thx 😊

We're still checking it internally. As soon as I have something new, I'll update the thread. I should have some answers later today 🙂

Hi @iamhenry 👋
Could you look into the CI errors 🙂

i'll take a look and fix them, thank you


Documentation corrections

Some minor corrections to the README.md file:

In the "Using CLI from outside the VM section" (https://github.com/apache/incubator-openwhisk/blob/master/tools/vagrant/README.md#using-cli-from-outside-the-vm), the wsk binary should be downloaded properly from the website of openwhisk (at least for Windows machines), because the folder ../../bin/openwhisk-cli/build/ does not exist.

In the same section, the first command to set the Openwhisk Authorization Key,

wsk property set --apihost <IP_ADDRESS> --auth `vagrant ssh -- cat openwhisk/ansible/files/auth.guest`

cannot be used in Windows. The relevant command would be
wsk property set --apihost <IP_ADDRESS> --auth $(vagrant ssh -- cat openwhisk/ansible/files/auth.guest)

Hi @atsag , great that you found it!
Do you mind opening a PR to fix it?

Not at all! The changes for the PR are already done, however I first have to sign the Apache CLA.


requesting update

Any chance you could update this?
The redundancy check is being ignored, a bunch of the other settings are too, and I'm pretty sure I have multiple icons in the same location for may items I have stood looking at for a few seconds.
This mod is almost unique in the way it functions, and it seems wasteful to allow it to become defective and disused.

damn it, i accidentally removed Tekla-JsonDotNet-13.0.1
disregard this issue; mod works fine


Add cost to message list

Codecov Report

Merging #361 (8e56417) into master (c721594) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #361   +/-   ##
=======================================
  Coverage   92.49%   92.49%
=======================================
  Files          23       23
  Lines        1292     1292
  Branches      146      146
=======================================
  Hits         1195     1195
  Misses         69       69
  Partials       28       28

Impacted Files
Coverage Δ

src/views/messages.py
89.61% <ø> (ø)

src/schemas/models.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c721594...8e56417. Read the comment docs.


Add PineconeVectorStore

Adds a vector store class for Pinecone along with sample app in apps/simple/pinecone-vector-store

Hi @marcusschiesser I think I resolved all of the issues you noted.  Also re-ran the query.ts example for Pinecone to make sure everything was good end-to-end.  Let me know what else you find.

@marcusschiesser Should be ready to check again.

@marcusschiesser Should be ready to check again. I noticed an upstream conflict in pnpm-lock.yaml but it wouldn't resolve for me, even when I deleted and re-created the file with pnpm i. Let me know if there's something I should do there.

@mtutty hmm, I guess the main branch in your fork is outdated:

You can sync your fork, to prevent that from happening, see https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
Otherwise, your PR looks great!
Btw, we now have VectorDB support in create-llama, it would make sense to add Pinecone to create-llama in this PR too. Do you want to have a look into that, too?
It's just three steps:

Adding a new pinecone folder to https://github.com/run-llama/LlamaIndexTS/tree/main/packages/create-llama/templates/components/vectordbs - the package should support the factory method createChatEngine(llm: LLM), similar to the MongoDB implementation
Adding pinecone to this type:
https://github.com/run-llama/LlamaIndexTS/blob/e55d41f5df3c365c45f3b5e2d98cd679ab7a8972/packages/create-llama/templates/types.ts#L7
Add pinecone as a new option:
https://github.com/run-llama/LlamaIndexTS/blob/e55d41f5df3c365c45f3b5e2d98cd679ab7a8972/packages/create-llama/questions.ts#L255-L261

I am happy to help with that!
To test create-llama, you have to call:
cd packages/create-llama
pnpm run build
./packages/create-llama/dist/index.js

HI @marcusschiesser Yeah, I tried getting my work branch up to date with your main and found that conflict.  I'll go back and start with my main as you suggested.  Might come back to you for guidance on how to resolve the conflict (which I expect will still be there).  It wasn't a logic or code problem, it had to do with the tooling and workflow of the project.  I'm traveling today, will get back to it Wed or Thursday and let you know what I find.
In regards to the create_llama thing, I can start a new PR after this one's cleared!

Hi @marcusschiesser I was able to sync my main to run-llama/main, and then get my work branch up to date with that.  I had to run pnpm i --no-frozen-lockfile to resolve the pnpm-lock.yaml conflict, and that succeeded.  But when I went to commit the resolved changes, the local commit hook gave this failure:
llamaindex:lint: /workspaces/LlamaIndexTS/packages/core/dist/index.js
llamaindex:lint:   1686:112  error  ASTRA_DB_ID is not listed as a dependency in turbo.json         turbo/no-undeclared-env-vars
llamaindex:lint:   1687:116  error  ASTRA_DB_REGION is not listed as a dependency in turbo.json     turbo/no-undeclared-env-vars
llamaindex:lint:   1688:120  error  ASTRA_DB_NAMESPACE is not listed as a dependency in turbo.json  turbo/no-undeclared-env-vars

Not sure this is my problem to fix, or how this got into run-llama:main.  Let me know how to proceed.

@mtutty The issue was that your branch was not rebased on the main branch from this repo. I fixed that and merged your changes in https://github.com/run-llama/LlamaIndexTS/pull/296 - I saw that you merged main into your branch - for feature branches, I would recommend rebasing your branch onto main - then you'll end up having a linear commit history which is easier to deal with.
Can you please test your example with the current main branch again? Then you can close this PR.
Thanks a lot for the contribution! Let me know if you have any questions about adding pinecone to create-llama


Kotsadm/postgres images shows as vulnerability(Alpine Linux Security Update for libxml2)

Kotsadm/postgres images shows as vulnerability(Alpine Linux Security Update for libxml2) and its due date is 09/21. Can we expect the new version of kots will fix this vulnerability

@chaitutheprince can you elaborate on the specific CVE? The last vulnerability I see affecting KOTS via libxml2 was fixed 2 months ago, and any affecting Postgres were fixed earlier than that.
https://github.com/replicatedhq/kots/security/code-scanning?query=is%3Aclosed+branch%3Amain+xml

Also, it appears you might be referring to a much older image that was temporarily released and is no longer supported. We now use the upstream Alpine images.

@obfuscurity We have installed latest kotsadm version 1.85. We have the process to delete the repository first and then we will download the kotsadm tar.gz file and then we will push all the images to our container registry. and then we will install them. In this process, we saw the kotsadm/postgres has 2 images.1. postgres:10.22-alpine with (sha256:e96660aa4a5301eb16e9ba526d7cac9d943aea06670ee7afd37e87350044917c) and  2.postgres:14.5-alpine with (sha256:a093139fb76e459c3d5ff7e19956c36cf706f1a2872ce34fadf6b0b777ecf72d)
Post installing it, on the next scan vulnerability exists for those 2 Images
Vulnerability name : Alpine Linux Security Update for libxml2
Vulnerability ID: 502487
Vulnerability description : Alpine Linux has released a security update for libxml2 to fix the vulnerabilities.Affected versions:Alpine Linux 3.13Alpine Linux 3.14Alpine Linux 3.15Alpine Linux 3.16Affected Package versions prior to 2.9.14-r1.Successful exploitation of this vulnerability could lead to a security breach or could affect integrity, availability, and confidentiality.

Hi
Can you please provide an update on this. I have provided the requested details.

@chaitutheprince that CVE has an unknown severity and Postgres did not yet release a new image with the fix:
$ trivy i postgres:14.5-alpine

postgres:14.5-alpine (alpine 3.16.2)

Total: 1 (UNKNOWN: 1, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

┌─────────┬───────────────┬──────────┬───────────────────┬───────────────┬───────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├─────────┼───────────────┼──────────┼───────────────────┼───────────────┼───────┤
│ libxml2 │ CVE-2022-3209 │ UNKNOWN  │ 2.9.14-r0         │ 2.9.14-r1     │       │
└─────────┴───────────────┴──────────┴───────────────────┴───────────────┴───────┘

trivy i postgres:10.22-alpine

postgres:10.22-alpine (alpine 3.16.2)

Total: 1 (UNKNOWN: 1, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

┌─────────┬───────────────┬──────────┬───────────────────┬───────────────┬───────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├─────────┼───────────────┼──────────┼───────────────────┼───────────────┼───────┤
│ libxml2 │ CVE-2022-3209 │ UNKNOWN  │ 2.9.14-r0         │ 2.9.14-r1     │       │
└─────────┴───────────────┴──────────┴───────────────────┴───────────────┴───────┘

Once Postgres releases a new docker image with the fix, we will update KOTS to use it.
Hope that helps.

@sgalsaleh - Thanks for the update.


Is there any way to disable journald listenning ? why it always show the message in /var/log/message, even exclude it?

Hi Team,
Here is the background.
Here is my logdna-agent v2 config yaml:
  exclude:
    glob:
          - /var/log/at/*
          - /var/log/anaconda/*
          - /var/log/audit/*
          - /var/log/boot*
          - /var/log/btmp*
          - /var/log/chrony/*
          - /var/log/cloud-init*
          - /var/log/cron*
          - /var/log/dnf*
          - /var/log/falcon*
          - /var/log/firewalld*
          - /var/log/hawkey*
          - /var/log/insights-client*
          - /var/log/journal/*
          - /var/log/kdump
          - /var/log/lastlog
          - /var/log/maillog
          - /var/log/messages*
          - /var/log/openvpn.log
          - /var/log/private/*
          - /var/log/qemu*/*
          - /var/log/rhsm/*
          - /var/log/secure*
          - /var/log/spooler*
          - /var/log/sssd/*
          - /var/log/sudo.log
          - /var/log/tuned/*
          - /var/log/wtmp
          - /var/log/btmp
          - /var/log/utmp
          - /var/log/wtmpx
          - /var/log/btmpx
          - /var/log/utmpx
          - /var/log/asl/**
          - /var/log/sa/**
          - /var/log/sar*
          - /var/log/tallylog
          - /var/log/fluentd-buffers/**/*
          - /var/log/pods/**/*
          - /run/systemd/journal/*
    regex: []
  use_k8s_enrichment: ~
  log_k8s_events: ~
journald:
  paths:
    - /root/Download

I blocked most files in /var/log. and try to block all journal log in /var/log/journal/* and  /run/systemd/journal/*
But I still got these log from agent:

I searched the log content, I found some of them are in /var/log/message file. even I have exclude it.
I think maybe the root cause is the journald , Listening to journalctl , then will send log like in screenshot.
So if it was the root cause, how can I disable "Listening to journalctl" ?
If it is not  the root cause, How can I block the "init.scopr" or<EMAIL_ADDRESS>.... log  ?
Thanks

Hi @jpixy, we don't currently have a configuration option to disable the journalctl stream on Linux. If the user the agent is running on can spawn the journalctl subprocess it will read from it by default.
You should be able to filter on a line-by-line basis using the LOGDNA_LINE_EXCLUSION_REGEX env var or adding a section like the following to your config
log:
  ...
  line_exclusion_regex:
    - "init.scope"
    -<EMAIL_ADDRESS>

copy that.
I use this: run logdna-agent as nobody, and it will have NO access to journalctl.
I have tested it, its results is OK for me.
Is that OK for logdna-agent run as nobody ? is there any potential problems I need to take care ?
Thanks

Nope, there shouldn't be any issues as long as the nobody user has access to read the files you want to log for the file's whole lifecycle.

Thank you so much, close this issue.

@c-nixon I used this:
log:
  ...
  line_exclusion_regex:
    - "init.scope"
    -<EMAIL_ADDRESS>
But it does NOT work.
I just wanna make sure --- line_exclusion_regex can work on journalctl log ? According to my testing, it won't work.
And I check the log files   :
# tree /var/log/journal/
/var/log/journal/
└── de428785c490ff4c4cd01bec741c88b5
    ├──<EMAIL_ADDRESS>    ├──<EMAIL_ADDRESS>    ├── system.journal
    ├──<EMAIL_ADDRESS>    ├──<EMAIL_ADDRESS>    └── user-1001.journal

these log files  looks like binary files. I am not sure whether it is the root cause which make "line_exclusion_regex" cannot work.

Another question.
I search the redundant content in /var/log, I found the content is from file : /var/log/message. But I am so sure that logdna-agent exclude the file /var/log/message via logdna-agent's startup log.
How to explain this ? Thanks

Hi, I have the same problem as you mentioned above. My /var/log directory is still beeing logged after excluding it in dir exclusion and excluding specific lines with line_exclusion_regex. Lines like "cron.service", "sudo", "init.scope", "systemd-logind.service", "ssh.service"<EMAIL_ADDRESS>are still spamming my log. I am running on Linux (Deb 11), my config.yaml is below. What I understood from this thread it is not possible to exclude these lines on Linux systems, right? Only option is to run as "nobody". Are you planning any update on this issue? Thanks.
  host: logs.***.com
  endpoint: /logs/agent
  use_ssl: true
  timeout: 10000
  use_compression: true
  gzip_level: 2
  ingestion_key: ***
  params:
    hostname: ***
    now: 0
  body_size: 2097152
  retry_dir: /tmp/logdna
log:
  dirs:
    - /var/el/gateway/logs
    - /var/el/apps/notary/logs
    - /var/el/apps/admin/logs
    - /var/el/logs
  include:
    glob:
      - "*.log"
    regex: []
  exclude:
    glob:
      - /var/log/**
      - /var/log/wtmp
      - /var/log/btmp
      - /var/log/utmp
      - /var/log/wtmpx
      - /var/log/btmpx
      - /var/log/utmpx
      - /var/log/asl/**
      - /var/log/sa/**
      - /var/log/sar*
      - /var/log/tallylog
      - /var/log/fluentd-buffers/**/*
      - /var/log/pods/**/*
    regex: []
  line_exclusion_regex:
    - "init.scope"
    -<EMAIL_ADDRESS>    - "sudo"
    - "session-*.scope"
    - "ssh.service"
    - "cron.service"
    - "systemd-logind.service"
  use_k8s_enrichment: ~
  log_k8s_events: ~
journald: {}
startup: {}````

Could the default of monitoring /var/log and journal logs just be removed rather than these workarounds becoming the norm? I don't think this is an edge case that warrants the hacks, having the agent monitoring the exact directories as specified by the config shouldn't be this much effort, and adding 10s of config lines to ignore/workaround the default isnt a sensible way forward, it should be a clean and clear config, e.g. MZ_IGNORE_DEFAULT_LOG_DIRS=True (if using env vars)
@c-nixon

We will be giving an option to disable the spawning of the journalctl process in our next release (3.8).  I will leave this open until that is complete.

Hi, @gjanco is this already solved in current 3.8 version? I did not find any release notes. Thanks

Yes, it can now be disabled with this flag MZ_SYSTEMD_JOURNAL_TAILER

Works fine so far, thanks 🙏


Fixes Safari critical error on video element children

#2667 If the video element has any other tags than track and source, javascript errors and fails to load player

Coverage remained the same at 74.487% when pulling d5a8c1655fb5ba0dbe284f78381e1a3ccb3c343c on fmp777:fmp777-patch-1 into db3842fd816fb8ac1b55066b06be4c280e263ff0 on mediaelement:master.


error: command 'gcc' failed with exit status 1

Hello,
Trying to install ngsutils on a Linux cluster.  Downloaded the repo, unzipped and un-tarred it and went into the directory.  Typed "make."
"Make" fails with the following cryptic error:
error: command 'gcc' failed with exit status 1

Cleaning up...
Command /Users/mchiment/ngsutils-ngsutils-0.5.8/venv/bin/python -c "import setuptools, tokenize;file='/Users/mchiment/ngsutils-ngsutils-0.5.8/venv/build/pysam/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-eYZ75r-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/mchiment/ngsutils-ngsutils-0.5.8/venv/include/site/python2.7 failed with error code 1 in /Users/mchiment/ngsutils-ngsutils-0.5.8/venv/build/pysam
Traceback (most recent call last):
File "/Users/mchiment/ngsutils-ngsutils-0.5.8/venv/bin/pip", line 11, in
sys.exit(main())
File "/Users/mchiment/ngsutils-ngsutils-0.5.8/venv/lib/python2.7/site-packages/pip/init.py", line 185, in main
return command.main(cmd_args)
File "/Users/mchiment/ngsutils-ngsutils-0.5.8/venv/lib/python2.7/site-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 40: ordinal not in range(128)
make: *** [init] Error 1

Any suggestions are appreciated.  Thanks,
Michael

What version of Linux is this for? Given the Unicode error, I'm assuming that somewhere there is a non-ascii character. However, that error isn't coming from my code, so perhaps it's in a directory name someplace?
Do you have any other error messages?

Hi, thanks for the reply.  Linux is CentOS 6.4.  I didn't see any other errors.  Where should I look for non-ascii characters?  The tool installation directories?

Just to be safe, try downloading the newest version. I just pushed a new release over the weekend for some other bug fixes.
I test mainly on CentOS, so this shouldn't be the problem. Occasionally I'll see errors on Ubuntu, but rarely does it happen on CentOS.
You should also look in /Users/mchiment/.pip/pip.log to see if there are any errors in there.

I ran into the same problem.  In my case, the cause seems to have been gcc running out of memory (due to process limits rather than actual RAM limits); running on another similarly configured host where the memory limits were higher, everything worked fine.  See the following link for an explanation: http://stackoverflow.com/questions/20555761/pip-install-pandas-gives-unicodedecodeerror-ascii-codec-cant-decode-byte-0


default to user-container when selecting the pod

2 asks, can be implemented in separate PRs

[ ] Today, the UI selects numa container by default, but it should be selecting user-container.
[ ] It should remember the context (which container type is selected) when clicking on other pods on the same page.

Hi @vigith, is this issue still up for work? If yes, I would love to work!

THank you; I will assign this issue to you.


Let's Start v1.0.7 (or 8) with TW-Sounds!

Hi @BurningTreeC,
Since neither of us have been real active lately in this repo, I've decided to start work on v1.0.7/8, utilising TW-Sounds. I'll add the functions and calls, and if you want, you can look over the code and make it more "tw-like".

Hi @BurningTreeC,
Just wondering if you could push the necessary TW changes so the twsounds library can be accessed from the startup js file. I tried adding a scripts directory to my copy of the plugin, and in it a twsounds.js file (code from https://github.com/flancast90/TW-Sounds/blob/main/src/tw-sounds.js). For some reason, the plugin wouldn't call any of the twsounds functions, even after importing with the (possibly correct tiddlywiki.info file JSON) that @joshuafontany gave

Hi @flancast90 , I haven't had the chance to look at your twsounds library but I guess giving it the field module-type: library and requiring it in our startup file could do the trick ...?

That’s exactly what I was doing; After I tried what @joshuafontany, I tried
just adding it the to .info file as a tiddler with module-type: library,
but I became confused with the file structure there, as it seemed to skip
the files folder completely, and go right the the modules/startup.
What @joshuafontany recommended worked except for that it added undefined
to the end of the file name/path.
Any ideas?
On Wed, Aug 25, 2021 at 1:32 AM Simon Huber @.***>
wrote:

Hi @flancast90 https://github.com/flancast90 , I haven't had the chance
to look at your twsounds library but I guess giving it the field module-type:
library and requiring it in our startup file could do the trick ...?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/flancast90/Speech-To-Text-in-TW5/issues/17#issuecomment-905196193,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AR2EVEAZF7LBT3CVYZUXE5LT6R56TANCNFSM5CWYVEWQ
.

Still haven't gotten it to work, @BurningTreeC. If you could share the .info file on your side, as well as where I'm to include the twsounds.js file, that'd be great!


entities not reflected in webpage

Hi Matt,
I followed your steps and managed to create a working blog webapp.
However, I stuck at adding blog entities. I ran "jhipster jdl blog.jdl", say yes to all overwrites, and restart blog app.
After admin login, the entities tab is still empty.

@djsg What version of JHipster are you using? Can you try these steps and see if it works:
git clone https://github.com/mraible/jhipster8-demo
cd jhipster8-demo
rm -rf *
jhipster

This will re-generate the app with whatever version you're using and you should be able to start it with Maven.

Hi Matt,
I did as you suggested. The demo code works perfectly fine.
This time I used a computer with Ubuntu OS; previously all my experiments were tried in Windows machine.
Does OS make such a big diff?

@djsg The OS shouldn't matter, but we do tend to test things on macOS and Linux more than Windows.


Add new features

Hey, I am the maintainer for another JVC integration
https://github.com/iloveicedgreentea/jvc_projector_python
https://github.com/iloveicedgreentea/jvc_homeassistant
I have archived it since yours is in core already. I plan on extending the core version to support new sensors and other entities that my custom component had. As a precursor, I wanted to add some features to the library
Summary of changes

Added a bunch of new commands
Update state function to be modular
Fixed some misc bugs
Lowered throttling to speed up state updates (maybe this should be configurable). I found that anything over 0.2s is pretty safe. I have had users report lockups if it is too fast though. However, updates should be under 30s otherwise HA will start complaining.
added helper functions to transform values to human readable ones like version, model, mac, etc

I updated some tests and made sure all of these worked with my NZ7. I like how you set up the formatter with regex for parsing commands, that is very slick.
Please let me know what you think

I will add the new handshake procedure for new models tomorrow as well

Added the new auth protocol and a new way to send commands. It uses a send_command function that can send CMD consts. This will allow home assistant to send commands like set the laser_power to high (added a test for that)

Heres some output examples
{'model': 'ILAFPJ-B5A3', 'mac': 'E0DADC0A269A'}

Current state:
model: NZ7 <- model code lookup
version: 3.0.0 <- version is semver now
mac: E0:DA:DC:0A:26:9A <- format mac
power: on
input: hdmi2
source: signal
picture_mode: user1
low_latency: off
laser_power: high
anamorphic: off
installation_mode: mode5
hdr: none
hdmi_input_level: standard
hdmi_color_space: auto
color_profile: auto
graphics_mode: high-res
color_space: rgb
eshift: off
clear_motion_drive: off
motion_enhance: low
laser_value: 100 <- this is normalized to 0-100
laser_time: 2195 <- hours
laser_dimming: auto3
hdr_processing: None
hdr_content_type: auto

I added a functional test so you can check it out as well

Any chance this could get reviewed?

@SteveEasley Would it be possible to get this reviewed and merged?  This add-on is broken for all of the new JVC models due to a password format change.  @iloveicedgreentea has that fixed above so it would be great to implement.

It’s been a while so I might just have to fork this and make a PR to home assistant to use the fork

Hey sorry, just noticed this request after ironically just pushing up some changes to fix the auth issue. Not sure why I didn't get a git notification of the PR until I pushed up my change.
@iloveicedgreentea I notice our auth code is a little different. I think your implementation might be including the _ in the sha256 encode, which is a mistake? I also added a test to verify the functionality (I dont have a newer projector to test though).
The const changes look like a good thing. But I do want to take a day or so to test the other new commands on my own projector. I have some concerns about the fast refreshes and bogging down the projector with the new traffic. But might be ok.
How about we split this up into two HA PRs? I will get my auth changes pushed through, and lets work on the new commands as a separate (but concurrent) change. Feel free to give feedback on my changes though (https://github.com/SteveEasley/pyjvcprojector/commit/05d8e86f7294983121be9b1705fb76434f2ce61a)

Hey sorry, just noticed this request after ironically just pushing up some changes to fix the auth issue. Not sure why I didn't get a git notification of the PR until I pushed up my change.
@iloveicedgreentea I notice our auth code is a little different. I think your implementation might be including the _ in the sha256 encode, which is a mistake? I also added a test to verify the functionality (I dont have a newer projector to test though).
The const changes look like a good thing. But I do want to take a day or so to test the other new commands on my own projector. I have some concerns about the fast refreshes and bogging down the projector with the new traffic. But might be ok.
How about we split this up into two HA PRs? I will get my auth changes pushed through, and lets work on the new commands as a separate (but concurrent) change. Feel free to give feedback on my changes though (05d8e86)

Sure sounds good I copied the implementation from my library which worked for users but could have added an extra underscore there. I don’t think the underscore is supposed to be included.  I can rebase without the auth changes when it’s merged

The auth change is already merge to main here (will be good to have eyes for PRs in the future though!). So feel free to rebase when you are ready. I am starting to dig into your changes now, with my projector.

@jmery You should actually be able work around the issue until we get this into HA by doing the password encode yourself.
In fact this might be a good test of our future encoding logic, since you would be manually doing exactly what's being done in the library. Also you might be able to test an edge case I am worried about (short passwords).

Add a 10+ character to your projector if it doesn't have one already (e.g.<PHONE_NUMBER>).
Now SHA256 encode that password + the string JVCKWPJ into an online SHA256 calculator https://www.google.com/search?q=sha256+encode+online. (e.g.<PHONE_NUMBER>JVCKWPJ is encoded as cc3054cf7c701a32a883bf45a7b19cbf7f62db778208b3c058e0c6dde886996f).
Use that encoded password as your password in the JVC Projector integration of HomeAssistant. I think you might have to remove the integration first, since there is no way to change or edit the password afterward.

If that works, would you mind testing a password less than 10 characters? The current library null pads the password to at least 10 characters and I want to make sure that is still a valid thing to do with these new encoded passwords.
Thanks!

Thanks I will take a look at these this week

@jmery You should actually be able work around the issue until we get this into HA by doing the password encode yourself.
In fact this might be a good test of our future encoding logic, since you would be manually doing exactly what's being done in the library.

Add a password to your projector if it doesn't have one already (e.g.<PHONE_NUMBER>).
Now SHA256 encode that password + the string JVCKWPJ into an online SHA256 calculator https://www.google.com/search?q=sha256+encode+online. (e.g.<PHONE_NUMBER>JVCKWPJ is encoded as cc3054cf7c701a32a883bf45a7b19cbf7f62db778208b3c058e0c6dde886996f).
Use that encoded password as your password in the JVC Projector integration of HomeAssistant. I think you might have to remove the integration first, since there is no way to change or edit the password afterward.

Thanks!

This doesn't appear to work.  I reset the PJ's network password.  Adding the integration via the HA UI constantly returns "Invalid Authentication".

Hmm that’s concerning since that’s the same process we would use in the code.
Would you mind trying with a test password and share it and the sha256 value you put into the HA integration with me?

Hmm that’s concerning since that’s the same process we would use in the code.
Would you mind trying with a test password and share it and the sha256 value you put into the HA integration with me?
And/or try it the exact values I gave above.<PHONE_NUMBER> for the projector password and cc3054cf7c701a32a883bf45a7b19cbf7f62db778208b3c058e0c6dde886996f for the HA integration password.

I reset it to your exact example from the original password.  I don't see any instructions from JVC requiring any kind of reboot or anything for the password to be effective.

Ok figured this out. This was actually an unrelated bug thats always been there. Passwords over 10 characters were getting truncated to 10 characters, so it wont except the long SHA :( I submitted a fix for this here https://github.com/home-assistant/core/pull/126581. But by the time thats fixed, but SHA256 PR (https://github.com/home-assistant/core/pull/126453) should be live at the same time, so should be able to use your regular password.
Thanks for taking the time to try it, squashed a bug at least!

Ok figured this out. This was actually an unrelated bug thats always been there. Passwords over 10 characters were getting truncated to 10 characters, so it wont except the long SHA :( I submitted a fix for this here home-assistant/core#126581. But by the time thats fixed, the SHA256 PR (home-assistant/core#126453) should be live at the same time, so should be able to use your regular password.
Thanks for taking the time to try it, squashed a bug at least!

2024.9.3 shipped overnight.  I just installed and can confirm the issue is fixed.  Thanks so much for the quick fix and happy to have helped a little :-)

I rebased to main, used your connect code, and fixed the rest of the comments. Please let me know what you think

Cool will look at this over the next couple of days.

Pushed changes to address comments.  I am assuming we will leave this open until you are done on the HA side and ready for a PR on that side.

fyi looking good. I am assuming we will leave this open until you are done on the HA side and ready for a PR on that side.

Yeah I started working on that in core. I will try to get it plugged in soonish

@SteveEasley I made a draft PR for HA core here https://github.com/home-assistant/core/pull/131320/files
I added sending commands and the new sensors. I need to do more testing but at first glance it all works. I am installing the lib from git (pulling changes into my core devcontainer) with
pip install  /workspaces/pyjvcprojector/

Makes it easy to iterate
LMK what you think. I will probably disable some sensors by default because the core people will complain about it. I also want to make it easier to see what commands can be sent like "laser_power, high".
I also updated the tests.

I pushed changes to automate generating commands and for HA I added more selects and improved sensors and all that. Will do more testing tomorrow. 99% code coverage too
---------------------------------------------------------------------------------------
homeassistant/components/jvc_projector/__init__.py           31      1    97%   50
homeassistant/components/jvc_projector/binary_sensor.py      29      0   100%
homeassistant/components/jvc_projector/config_flow.py        62      0   100%
homeassistant/components/jvc_projector/const.py               5      0   100%
homeassistant/components/jvc_projector/coordinator.py        31      0   100%
homeassistant/components/jvc_projector/entity.py             17      0   100%
homeassistant/components/jvc_projector/remote.py             44      1    98%   71
homeassistant/components/jvc_projector/select.py             33      0   100%
homeassistant/components/jvc_projector/sensor.py             25      0   100%
---------------------------------------------------------------------------------------
TOTAL                                                       277      2    99%

Updated model and version display as well. I tested the selects, it all seems to work. Please test and let me know. You will probably have to edit your manifest to have this.
{
  "domain": "jvc_projector",
  "name": "JVC Projector",
  "codeowners": ["@SteveEasley", "@msavazzi"],
  "config_flow": true,
  "documentation": "https://www.home-assistant.io/integrations/jvc_projector",
  "integration_type": "device",
  "iot_class": "local_polling",
  "loggers": ["jvcprojector"],
  "requirements": ["/workspaces/pyjvcprojector/"]
}

@SteveEasley ive been running this in my HA instance and its been good the only issues I see are occasional timeouts

I suspect its an issue of trying to read an empty buffer which suggests a syncing issue but not sure. Not sure if this ever happened before my changes. In my custom component I implemented a PriorityQueue which allowed user commands to have immediate access while refresh commands took a back seat. This gave you strict command ordering, priority, and a queue system. let me know if you are interested in that I can implement it

Hey, sorry for the delay. Should be diving into this by the end of the week.

Hey, sorry for the delay. Should be diving into this by the end of the week.

Any update?  I'm waiting for this to merge before switching to the official JVC integration, since I use installation mode in my automations.  Was hoping to make the update over the holidays.  Thanks!

Any update? I'm waiting for this to merge before switching to the official JVC integration, since I use installation mode in my automations. Was hoping to make the update over the holidays. Thanks!

Yep I provided some feedback on the HA side in https://github.com/home-assistant/core/pull/131320 last weekend. It was working for me for the most part.

I pushed some changes here and to core. I do notice that user commands could get ignored during an update which is what I think causes the timeout warnings. Since this PR is already huge it would make sense to implement a priority queue after this which would also have the added benefit of making user commands instant (plus all the other stuff like order guarantees)

Hey, sorry for the delay. Should be diving into this by the end of the week.

Any update? I'm waiting for this to merge before switching to the official JVC integration, since I use installation mode in my automations. Was hoping to make the update over the holidays. Thanks!

If you know what you are doing you can install my PR in core as a custom component and upload this PR to pypy (or I suppose just reference a directory instead)

I did added some changes for consideration to https://github.com/iloveicedgreentea/core/pull/1


Property:websiteURL

Definition
URL of the Organisational Unit's main website

Dimension

Existing property

Existing class

Existing property identifier

Format
Text

Required
No

Repeatable
No

Constraints

Examples

Notes
Valid URL

Good example is needed


Add IPv6 support

Submitting a possible resolution to Issue #6. Added Google's public IPv6 DNS server address to demo.py:
addrs = ["<IP_ADDRESS>", "cnn.com", "<IP_ADDRESS>", "youtube.com", "2001:4860:4860::8888"]

When running on a host with IPv6 connectivity: the result should be this:
$ sudo python demo.py
Password:
sending one round of pings and checking twice for responses
   received responses:  ['<IP_ADDRESS>']
   no responses so far: ['<IP_ADDRESS>', '2a04:4e42:600::323', '2a00:1450:400a:805::200e', '2001:4860:4860::8888']
   --- trying another receive ---
   received responses:  ['2a04:4e42:600::323', '2001:4860:4860::8888', '<IP_ADDRESS>', '2a00:1450:400a:805::200e']
   still no responses:  []

sending again, but waiting with retries if no response received
    received responses:     ['2001:4860:4860::8888', '2a00:1450:400a:805::200e', '<IP_ADDRESS>', '<IP_ADDRESS>']
    1. retry, resending to: ['2a04:4e42:600::323']
    received responses:     ['2a04:4e42:600::323']
    all done, received responses from everyone

sending again, waiting with retries via provided send_receive()
    reponses: ['2a04:4e42:600::323', '2001:4860:4860::8888', '<IP_ADDRESS>', '2a00:1450:400a:805::200e', '<IP_ADDRESS>']

On a host without IPv6 connectivity, the result is the following:
$ sudo python demo.py
sending one round of pings and checking twice for responses
   received responses:  ['<IP_ADDRESS>']
   no responses so far: ['<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>', '2001:4860:4860::8888']
   --- trying another receive ---
   received responses:  ['<IP_ADDRESS>']
   still no responses:  ['<IP_ADDRESS>', '<IP_ADDRESS>', '2001:4860:4860::8888']

sending again, but waiting with retries if no response received
    received responses:     ['<IP_ADDRESS>']
    1. retry, resending to: ['<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>', '2001:4860:4860::8888']
    received responses:     []
    2. retry, resending to: ['<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>', '2001:4860:4860::8888']
    received responses:     ['<IP_ADDRESS>', '<IP_ADDRESS>']
    3. retry, resending to: ['<IP_ADDRESS>', '2001:4860:4860::8888']
    no response received in time, even after 3 retries: ['<IP_ADDRESS>', '2001:4860:4860::8888']

sending again, waiting with retries via provided send_receive()
    reponses: ['<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>']
    no response received in time, even after retries: ['2001:4860:4860::8888']

As excepted, the IPv4 addresses will be used and the IPv6 address is reported as unreachable.
Please review and provide feedback. Happy to incorporate any suggestions.

Thank you very much for your kind suggestions. I think I complied with all of them. Let me know what you think, in particular about the exception handling when user provides an IPv6 address but v6 is disabled. This might happen unknowingly  if DNS returns a v6 address.

Hmm, but this if statement would resolve to False when pkt_id is 0:
https://github.com/mwiget/multi-ping/blob/master/multiping/__init__.py#L292
But anyway, None is cleaner. I'm pushing another commit.

Thank you very much for your contribution! The PR is merged. I'll update the version number and the version in PyPi.

Great. Many thanks for the review and merge!


Different events using API and the tasty app

I want to subscribe to market data on options symbols and get the latest events, even when markets are closed. I'm looking for a snapshot of data rather than a real-time stream.
The tasty platform always shows the latest Greeks, quotes, etc, for the option symbols.
Looking at wss://tasty-demo-ws.dxfeed.com/delayed events in the browser, I see the usual FEED_DATA even when the options market is closed.
{"type":"FEED_DATA","channel":1,"data":["Greeks",["Greeks",".PLTR231117C15",0.859042135169914,0.679454583347139,0.0982083687887286,-0.0261660721057656,0.00609448239509639,0.01518018285523352,"Greeks",".WBA231117P27.5",0.74595897321041,-0.851562819473887,0.0539648651528887,-0.01993263273054975,-0.0167382063515774,0.01331688462849462]]}<PHONE_NUMBER>.7325706

The API client behaves differently:
this.streamer.addSubscription( '.PLTR231117C15', {subscriptionTypes: [MarketDataSubscriptionType.Trade], channelId: 1})
would not produce any data
this.streamer.addSubscription( 'AAPL', {subscriptionTypes: [MarketDataSubscriptionType.Quote], channelId: 1})
shows data arriving
I couldn't find anything obvious in the API client implementation that may prevent receiving data outside of the window for opened markets.

Hi @OperationalFallacy The api quote token has different entitlements than the platform quote tokens. Currently the api quote token is only entitled to equity quotes. We are working on adding equity options and other instrument types.

Thank you, I figured the difference.
I'll continue development and testing with a delayed stream/quotes for now.

@dmoss18 So I'm I understanding correctly that at the moment we can't use the streamer token the API gives us to stream futures options data?

Is there anywhere I can subscribe (maybe an issue or something) to know when this is available? This is a blocker for me :/

@josebalius I can open an issue here and you can subscribe to notifications. I'll come back with updates when equity options and other instrument quotes are added.

@josebalius Here is the issue https://github.com/tastytrade/tastytrade-api-js/issues/7.

@OperationalFallacy  From where are you streaming delayed data?


Datacatalog Build Failing

GoogleCloudPlatform/java-docs-samples#2102
https://source.cloud.google.com/results/invocations/9479d637-f0f9-4e16-bc22-d0d28f83d079/targets
https://source.cloud.google.com/results/invocations/0eb0ea20-3c6d-4d29-b6f0-92bc7920ae29/targets
Reaching out to sample writer

@mesmacosta

@nnegrey what is the status of this issue?


Supporter images not loading

I cannot tell if this issue is specific to our Dojo home page, but our "Supporters" img tag is not loading its src.
Frisco Coder Dojo

OS: MacOS 10.12.5 (16F73)
Browser: Chrome 59.0.3071.115 (Official Build) (64-bit)

this supporter image seems to link to https://www.call-em-all.com/ which isnt a image the supporter link needs to be changed to a link to a image

Hi m2mathew,
Unless there is an issue with how to define the url of the supporter image, I believe @butlerx is right

Hmmmm, that's odd. I added those images long ago (maybe a year ago or so), and have never updated them. I will double-check. Thanks, gents!

Yeah, my fault. I had it pointing at a picture incorrectly. It is now pointed directly to our CDN, so all fixed up!


emoji_picker: Fix status emoji picker inconsistencies.

Due to a logical bug in the process_enter_while_filtering function, the toggle_reaction was being called when pressing enter while filtering the emojis in the status emoji picker.
This PR fixes the above bug, while also adding some cleanups to the emoji_picker.js by combining multiple click handlers and aggregating all the logic related to an emoji being selected into a common function.
Fixes: #28464

Screenshots and screen captures:
Steps to reproduce: Go to "Set  Modal > Open emoji picker > Enter random words to filter > Press Enter
before-status-emoji-picker.webm
after-status-emoji-picker.webm

Self-review checklist

[X] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[X] Explains differences from previous plans (e.g., issue description).
[X] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[X] Each commit is a coherent idea.
[X] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[X] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[X] End-to-end functionality of buttons, interactions and flows.
[X] Corner cases, error conditions, and easily imagined bugs.

lgtm aside from the small comment above.

In addition to the addressing the feedback, I have also added another commit which aims to restructuring of related CSS and removing the class names -- "reaction", "status-emoji" and "composition" from the .emoji-popover-emoji div, which were previously being used for handling click events.

Merging since these changes are themselves good, and the one bug I found is present in main. Can you put up a fix for it?


node 0.10 support

We're seeing production issues updating to latest 4.4.0
ReferenceError: Symbol is not defined
    at _registerValueHandler (/var/local/cms/releases/cms-1458412259960-50d35e57-4ba2-45f9-a431-982b16f69036-60462d1/node_modules/@mic/squel/node_modules/squel/squel.js:175:38)

It seem, looking at .travis.yml it seem that you removed node 0.10 support - that's a breaking change.

Sorry about that, I wasn't aware people were still using 0.10. Am fixing now.

Fixed in 4.4.1

@hiddentao it looks like there was a possible regression of this issue in 5.x.
ReferenceError: Symbol is not defined
    at _registerValueHandler (/Users/claywalker/Workspace/API/node_modules/squel/squel.js:148:36)
    at Object._buildSquel.cls.registerValueHandler (/Users/claywalker/Workspace/API/node_modules/squel/squel.js:294:5)
    at _buildSquel (/Users/claywalker/Workspace/API/node_modules/squel/squel.js:1222:7)
    at /Users/claywalker/Workspace/API/node_modules/squel/squel.js:2971:13
    at object (/Users/claywalker/Workspace/API/node_modules/squel/squel.js:5:22)
    at Object.<anonymous> (/Users/claywalker/Workspace/API/node_modules/squel/squel.js:9:2)

Environment:
node: 0.10.44
npm: 3.8.7
squel: 5.0.4

@clayzermk1 squel 5+ is not supported on Node 0.10. I'll add a note to the README to that effect.


Update README.md

These changes make the README more engaging and easy to navigate, with a conversational and user-focused tone.

@QuickHarsh mention issue no.

I have updated the README to make it more attractive and made several improvements, aligning everything properly. However, I have not addressed older issues, as I wanted to focus on updating and enhancing the existing content."
Let me know if you need further adjustments!

@QuickHarsh we won't accept PR if you were not assigned any issue..


Global Network

I just realized that many applications allow the user to configure them for a single network which is then used in the whole program. One would normally parse the network from configuration and then pass it around in arguments. This can be a bit annoying.
Maybe we could also provide a global (internally AtomicU8) that user could set in main and then use alternative functions that don't require Network. The functions would obviously need to panic if the network is not set. To avoid accidental use by libraries (and also target_has_atomic issues) this should be behind a feature flag. When the flag is activated we could also provide an implementation of FromStr/Deserialize for Address<NetworkChecked>. (NetworkUnchecked should keep its behavior regardless of the flag.)
Note that this is intended for applications only, libraries should continue to support arguments. Also our library should support both interfaces.

I like this idea.
I wonder if we should rename Network to Chain. Or just start deprecating Network in favor of a new Chain that we think through more carefully.
In parallel to this I'd like to zoom out a bit and think about what structures here are Network specific. It seems like

Address, Extended*Key, PrivateKey all have prefix bytes in them which (mostly) imply a network; Address also has clear money-losing failure modes associated with confusing which network is involved.
Block and Transaction (and their components) are inherently tied to a specific chain because they commit to data in a specific chain. It's impossible to use these on the wrong network but a network mismatch between Address and Transaction is the issue that we're trying to prevent by having Address by network-checked. There may also be ergonomic value in the user being able to check "what network is this tx for".
Script too. Address -> Script -> Transaction needs to make sure the network doesn't change.
Similar story for Txid; inherently tied to a network and can't be confused. But the user maybe wants to keep track of this nonetheless.

It seems like there ought to be an API where the user could construct some basic data structure with the correct Network set and then magically everything else would be inferred/verified to be consistent. Doing this in general would require dependent types.

Doing this at compile-time would require dependent types (and require users produce proofs all over the place that e.g. two parsed-from-string addresses have the same network). But maybe there's a runtime solution:

Basically every structure in the library (block, header, tx, txin, txout, script, scriptbuf) gets an Option<Network> field. The "normal" constructors set this to None except if @Kixunil's magic atomic is set, in which case they are set to Some with the user's global network choice.
Constructors for things like Address which have their network encoded in the string always use that encoding[*].
Whenever things are composed, network consistency is checked. So you can e.g. convert an address to a scriptpubkey, which will carry the network field, then when you try to put that into a TxOut it'll make sure the networks are consistent.
PartialEq and Eq also check for network consistency before declaring equality.

Finally, consensus::Encode and serde::Serialize are gated on having the network field set.
[*] Ok, so the network encodings for things like Address are actaully ambiguous. So we would need to actually represent networks internally as a system of constraints and do unification, which will require a little bit of CS magic. But I think not much. So when I say Option<Network> I really mean "the empty set of network constraints".

Despite using the language of "constraints" and "unification" I actually think this could be implemented entirely in terms of bitmaps and bitwise AND. So initially the "network" field of each structure would be all-bits-1 "every network possible". Then:

If the global atomic is set (it will be a single bit, i.e. "only this specific network is possible") then we & the result with that.
If the user calls set_network explicitly on the type, then we & the result with that.
(In theory we could expose more nuanced modifiers, but we probably don't want to. Though it may be interesting to reserve some bits for e.g. bcash, and let users directly modify those if they want.)
If the type is parsed in a way that implies certain network(s) then we & with that
If a type is constructed where the network map is 0 "no network possible" this is an error.
If a type is compared where the network map &s to 0 this is an "unequal" comparsion regardless of how the other fields are set.
If a type is serialized but the network map has more than 1 bit set, this is an error.
Certain constants, e.g. genesis hashes, have their Network fixed from the beginning.
We would encourage crates like jsonrpc to somehow set the Network of incoming structures based on what daemon they're talking to. (Or maybe users need to call getblockchaininfo RPC and then set this themselves.)
We would encourage users to set the network flag explicitly when deserializing their own config files etc. So e.g. a wallet might have a testnet boolean and then the wallet-parsing logic should set all the wallet data to either mainnet or testnet, based on that.

Finally, we'd need some set of escape hatches -- a way to explicitly override the network, a way to say "ambiguous serialization is OK" (maybe we just want a bit in the map for this..), and maybe a way to directly set/clear network bits.
I think, with all this in place, "most" users would have their network usage automatically checked for consistency, without them needing to explicitly think about it except maybe at startup time, and without them being encouraged to "fix" a network at the type level so that it's impossible to use their stuff on testnet.

I wonder if we should rename Network to Chain

IDK, bitcoind calls it chain but all other applications I know of call it network. I wouldn't change it. But maybe we should use Magic instead to allow custom signets.
Your ideas seem complicated so I will have to take more time to respond to them properly. My instict is to not over-complicate but there may be good reasons. Also I think that if every input is gated on valid network it should produce valid outputs. So we really only need this in Address, Json RPC (yes, I do agree it should use our API to support checking more directly).
Raw types like Transaction don't have the tag but it seems odd to add it. Having Block contain a bunch of identical bits isn't nice.

My instict is to not over-complicate but there may be good reasons.

Right now from a user perspective, Address is extremely complicated, because our "simple" implementation completely pushes the responsibility for network-tracking onto them, and because it's not obvious what functionality is available under what conditions. (Downstream many crates have had to modify tons of unit tests, which don't care remotely about network stuff, to add .assume_unchecked() calls everywhere, which isn't obvious to do and isn't even obviously correct.)

Also I think that if every input is gated on valid network it should produce valid outputs.

I think that to achieve this we'll probably need something roughly as complicated as what I'm describing. Basically for the compiler to do it for us we'd need dependent types, and since we don't have them, we have to do our own run-time typing.

Raw types like Transaction don't have the tag but it seems odd to add it.

In your view, when converting a testnet address to a scriptpubkey, embedding this in a mainnet transaction, then broadcasting it to the mainchain, which step is the mistake and how should we determine that?

Having Block contain a bunch of identical bits isn't nice.

Hmmm, yeah, definitely agreed here. And it's even worse because every TxIn and TxOut, every Script etc., will also have the identical bits. We will need a better solution. (Maybe gating all these wasteful fields on debug_assertions?)

Hand wavey input: We should try really hard to make uses of Address be simple because its such a basic Bitcoin thing. When basic things are hard to use it puts devs off using the library, IMO. Obviously it should be correct first, simple second.

Global variables break composability and lead to action-at-distance.
I understand the sentiment, as passing such a "context" seems like pure waste. I think generally contemporary programming languages have poor support for "context" passing. (Relevant: https://tmandry.gitlab.io/blog/posts/2021-12-21-context-capabilities/)
But all in all global variables ... :vomiting_face: ... There's an already long thread of complex ideas involved, and that is only internally.
As a user now I'll have to pay attention to which APIs can be used with a flag, which can' etc. And poor developers that will have to debug issues due to someone somewhere abusing mutable global variables, etc.
IMO, if application writers want a global, they can write it themselves.

I was in between a long response. But @dpc got the most of it.
I strongly agree with dpc, I think this is bite us in the long run by break composability. This will increase more cognitive load for reviewers to think about "mutable global" things when looking at local changes.
IMO the benefits are not that impressive either.

I don't think saving the developer who can accidentally pass a wrong network is a probable error that we should care about. For example, it is also possible the developer calls p2wsh instead of p2sh etc. Or calls a address on the wrong script.

I am sure that there would be other things in global config file for application developer apart from just the network. So, adding this is not preventing them from having a global config file or network specific thing that might already do. They probably have a network dependent text like "Warning: testnet coins are not real" or "warning: Using mainnet on experimental software" etc.

If there is a need to create simple interfaces for new developers with globals and fixed reasonable defaults, we can make another crate on top of rust-bitcoin to support it. There are not that many APIs with network parameter, a simple wrapper around rust-bitcoin can do.

Right now from a user perspective, Address is extremely complicated

Yes, I think my proposal should mitigate much of the complication in vast majority of cases - you just get FromStr for Address and the only case where it gets weird is if you try to work with multiple networks.

unit tests, which don't care remotely about network stuff, to add .assume_unchecked() calls everywhere

I admit this sucks but I don't really know how to fix it. We could add this_is_test method equivalent to assume_checked just for semantics but not sure if it's a good idea.

In your view, when converting a testnet address to a scriptpubkey, embedding this in a mainnet transaction, then broadcasting it to the mainchain, which step is the mistake and how should we determine that?

Assuming the application is single-network (vast majority of applications) there are two places where the mistake could happen: when parsing the address or when connecting to the network. We have the first covered now, so only the second remains. I think JSON RPC should have some methods to either check the network by calling getblockchaininfo right after connecting or explicitly acknowledge that one doesn't want to check.
Multi-network applications don't get protection but I don't remember ever seeing one, so it's safe to assume we don't need to bother.
@dpc, @sanket1729 firstly I need to say that I hate global mutable variables with passion. But I think there is a case to be made for those which are loaded at the beginning of the program, set once and then never mutated again. Basically lazy_static. The negative effects of these are extremely limited because there's normally no "action" in "action-at-distance". I would personally support design that would outright forbid setting the value more than once (by panicking), including to the same value it already has. Yes, it still has problems around tests but that's a choice up to application developer.

As a user now I'll have to pay attention to which APIs can be used with a flag

Clearly marked in documentation and you can't use them by accident. Also you won't get bogus suggestions from RLS. Seems fine to me.

poor developers that will have to debug issues due to someone somewhere abusing mutable global variables, etc.

That can happen already. And banning setting the variable twice should make it less likely to happen. I would also be in favor of saving the Location of the initial set call so that it can be reported. ("The network was already set at {}") You'd literally get exact lines where things went wrong.

if application writers want a global, they can write it themselves

and

we can make another crate on top of rust-bitcoin to support it

unfortunately this doesn't work well due to orphan rules (can't impl FromStr for Address). The conversions required would suck even more than the solution.

saving the developer who can accidentally pass a wrong network

The current Address API already addresses that so it's not a design goal. The goal is to simplify the API for vast majority of applications, Possibly all of them.

So, adding this is not preventing them from having a global config file or network specific thing that might already do.

I don't understand what you mean, it looks like you meant "not adding"?

@Kixunil
If the goal is not have to pass Network around, users can do it themselves.
If the goal is to have FromStr ... then I just don't think it's worth it. fn parse(network: Network, s: &str) -> Address etc. is really not that much different from plain FromStr.

The conversions required would suck

If someone really needs to have the parsing of Address generic with all other types, one could have a bitcoin-network-ctx with:
pub trait NetworkCtxFromStr : Sized {
    type Err;

    fn network_ctx_from_str(s: &str) -> Result<Self, Self::Err> {
}

impl NetworkCtxFromStr for T where T: FromStr { ... }

impl NetworkCtxFromStr for bitcoin::Address { /* use the `Network` lazy static */ }

then also a normal newtype-wrapper:
struct NetworkCtx<T>(pub T);

impl FromStr for NetworkCtx<bitcoin::Address> { ... }

maybe with impl Derive<Target=bitcoin::Address> and .into() for that times where need to work directly with stuff that requires plain FromStr.
For good or bad, this is how these sorts of things are handled in Rust all the time, and people are used to it.

Assuming the application is single-network (vast majority of applications) there are two places where the mistake could happen: when parsing the address or when connecting to the network. We have the first covered now,

I really don't think we do. What happens now is that you parse an address and then find that basic functionality is missing and then you have to carefully scan the docs to figure out what generics you need, and what functions to call to get those generics.
IMO this generic solution is undiscoverable and confusing. If we want to make the user check the network at parse time we should get rid of from_str and only have from_str_with_network, although that will then make it impossible to write code that roundtrips addresses regardless of network, even though such code is perfectly safe. If we want to force a network check when converting to a scriptpubkey then we should stick a network check onto .script_pubkey even though this will probably result in redundant network checks for most users. But at least the user has some hope of finding these functions.
If we want to make it so that the user can just check the network and have that check propagated with the address from there on out, we basically cannot do it with the Rust type system. At best we can propagate a binary "I checked the network" which is what we've tried to do, but the result is noisy and hard to use.

If the goal is to have FromStr ... then I just don't think it's worth it.

Yeah, that one would be probably not enough. serde is much more interesting.
And yes, the trait approach you suggest works OK.

have to carefully scan the docs to figure out what generics you need, and what functions to call to get those generics.
IMO this generic solution is undiscoverable and confusing.

I don't think so. Reading the docs is a fundamental part of development. I read them all the time. We have a whole section about it in the docs. If people come up with specific ways the section can be improved I'd be happy to.

get rid of from_str and only have from_str_with_network

The issue is people use parse() rather than from_str. I don't remember ever seeing from_str outside of cases where it was written by a newbie (and possibly remained unchanged for a long time). But I would be open to adding a dedicated method that does .parse()?.require_method() in one step if it helps.

stick a network check onto .script_pubkey even though this will probably result in redundant network checks

It will lead to something worse: moving validation further down the line which makes error handling and debugging harder.

the user has some hope of finding these functions.

They are literally in the docs. But it just occurred to me that we could add a note to each of them that they need checked address.

I don't read documentation end-to-end and I don't think most anybody else does either. If I need a function then I scroll (or more likely search) the list of available functions. If search yields no results then I look for other functions that seem to need the missing functionality and read their source code. If searching finds something, but it's gated on some set of generics I'll get annoyed, then find source code to read. If this happens too many times I just drop the library, or revert to some old version before they started doing crazy architecture experiments. (I use time 0.1 for example rather than 0.2 for this reason. AFAICT 0.2 is actually missing critical functionality because they tried to Rustify it, while 0.1 more-or-less replicated the C API which works. In 0.3 they seem to have fixed it but have a high MSRV.)
Only for well-maintained and mature software like serde (and rust-bitcoin is definitiely not mature!) would I consider actually reading the documentation. Otherwise that sounds like a good way to waste a bunch of time without actually learning how to use the library.
Having said this, I checked the Address docs on rust-bitcoin, which I've never read (though probably I checked it for typos and rough correctness when it was being PR'd), and it does indeed describe how to deal with the marker trait. It suggests this is "for type safety" which sounds good on first reading, but after trying to use it and finding how inconvenient it is (and that "type safety" does not give me any hint what methods require validation and which ones don't) I think as a user I'd like there to be a better reason. It's especially confusing because the type doesn't actually carry any information about the network, just a boolean "did you call this API function?".

Yeah, that one would be probably not enough. serde is much more interesting.

I take your point about serde and parse(), neither of which have a sensible way for us to tack a Network parameter onto. But the current situation leads to the user being unable to serde-decode something they just serde-encoded, without doing a network check on data that was encoded in the address, and then having the code just throw away the network that was checked.

It will lead to something worse: moving validation further down the line which makes error handling and debugging harder.

It will lead to moving validation to somewhere where it's logically needed, instead of making users deal with it when they don't have to and won't understand why they have to.

nACK on the idea of adding a global to the library. Like the OP says, applications do that, I don't think libraries should introduce sidechannels in tons of functions. Our functions should be deterministic.
It's really not hard for an application to have a const NET: Network and pass that in any library function that needs a network.

@apoelstra could you hint what in time you found confusing? From quick look it seems sensible.

It suggests this is "for type safety"

I don't see any mention of type safety there. It's for user safety. When I upgraded rust-bitcoin after we forced the check I discovered several other instances missing it! Also I noticed a similar bug in LND - if they used a library forcing the check they wouldn't have it.
Note that I personally had to help numerous people recover their coins sending to a wrong address (was BTC vs LTC but still same category of issue). Not only that, I once was the one who hastily forgot to manually check the address even when being aware of the problem (was an ATM and the manufacturer was slow fixing this). So forgetting a check like this is a serious problem that is unsolvable for 99% of users.
In my eyes, this is totally worth a bit more annoying API - because it's the correct API. Programming is complex and rather than hiding this complexity under the carpet where it'll kick you in the ass later we should make tools to deal with it. Especially in security-critical software like Bitcoin.

just a boolean "did you call this API function?".

That's enough. It's similar to Rust not auto-inserting unwrap (unlike Java) whenever you pass Option<T> where T is expected and it often reminds people that it needs to be checked. Saved my butt million times.

It will lead to moving validation to somewhere where it's logically needed, instead of making users deal with it when they don't have to and won't understand why they have to.

If this is actually logical in user code (I highly doubt so) they can just pass around Address<NetworkUnchecked>.
@stevenroose we would still have the same deterministic functions, just in addition we would have simpler options. As I say single-network applications are extremely common. Actually, the only one I know that isn't is Ride The Lightning but that one doesn't directly work on Bitcoin stuff, just makes API calls into LN node. Every single wallet, Bitcoin node, LN node, ... everything else is like this. So why should everyone reinvent this when it can be in the library? Especially considering orphan rules? If not for orphan rules and annoyance of extension traits I'd be onboard with putting it into a separate library.

to have a const NET: Network

That's actually a terrible design since you have to rebuild the whole application if you want to test something. (Although it's billion times better than having it both hard-coded and scattered around the code. Yes, sadly, there's one such popular wallet with mainnet hard-coded. No, I have no clue how they test it.) Having these helper functions could actually incentivize people to design the application properly. Even if they hard code bitcoin::set_network(Network::Bitcoin) in main() it's super easy for someone to make a PR.

Note: I don't like reading documentation, read it only when I'm investigating something specific, and as a maintainer of a handful open source project noticed that just because something is documented, does not prevent large chunk of users not be aware of it. That's why I like type system forcing user's hand or at least

In my eyes, this is totally worth a bit more annoying API - because it's the correct API.

Agreed. As a user I'm happy to deal with an extra type steps if it ensures correctness. I'm only allergic to global values breaking composition, and introducing conditionally available set of APIs.
I've already mentioned 2 separate "If the goal is". This seems to be the 3rd one.
If the goal is to force users (developers) to verify network address, session types seem like the way to go:
trait VerificationStatus;

struct Verified;
struct Unverified;

struct Address<Status = Unverified>{ .. };

// Since `Address` == `Address<Unverified>, maybe an alias like this:
type VerifiedAddress = Address<Verified>;

impl Address<T> {
  fn verify(&self, network: Network) -> Address<Verified> { ... }
}

etc.
Then if the crucial places at which point the address should have been already verified, require VerifiedAddress.
A runtime flag is also an option, if the API noise of the above seem undesirable.

In my eyes, this is totally worth a bit more annoying API

For sure.

because it's the correct API

But it's not correct, since it won't actually prevent you confusing two (checked) addresses that correspond to different networks, and because it won't let you do perfectly safe things like deserializing then re-serializing an address without adding extra .assume_unchecked.
Sure, in practice this is good enough for safety, but because it's ad-hoc and "good enough" there's no mental model the user can have that will let them predict what they're able to do and not do with Address. At best this adds friction, but at worst it means they'll just learn to stick .assume_checked() everywhere to make the API work. Or switch to a different library.

it won't let you do perfectly safe things like deserializing then re-serializing an address without adding extra .assume_unchecked

Isn't this very reasonable? In a general case if you deserialize, you can't really be sure if you got correct addresses. You can assume it if you trust the source or have some other pre-existing knowledge.
As for mixing checked addresses from different networks: it's not unlike core tenants of Rust like e.g. unsafe. When using unsafe { ... } it is not required that this particular block guarantee that the usage is correct. It's a global promise that safety requirements of unsafe functions used in it will be upheld here and everywhere else. There is no practical way to express complex safety contracts in certain situations, so the language / APIs do the best they can.
So being a little bit "ad-hoc" does not seem like a big deal to me. It's just a pragmatic tradeoff. Forgetting to check the address is a common source of important bugs, so you guide the developer to do the right thing as much as practical.

@dpc I'm talking about deserializing then serializing. There is no data loss here.
@Kixunil I'd be fine with an API where you could do anything with an unchecked address except get a scriptpubkey out of it. And this should be documented.
Right now the Address API is too complicated for me to use and I really don't want to update my software to bitcoin 0.30 because of it.

@dpc the goal is to make it easier to use the Address API in contexts that are affected by orphan rules while continuing to prevent network-related bugs. And as I said, this is super-common - like literally every application except for RTL. (If you can find 10 other applications, I will close this right away.)
We already use session types, so I'm not sure what you mean by that example.

since it won't actually prevent you confusing two (checked) addresses that correspond to different networks

I can't even imagine hypothetical scenario where an application needs to check equality of networks of two addresses while at the same time not check it against some global network setting. As I said, there's at most one application I know of which can't be excluded straight away.

, in practice this is good enough for safety, but because it's ad-hoc and "good enough" there's no mental model the user can have that will let them predict what they're able to do and not do with Address

Rust can't prevent all bugs and yet people use Rust just fine. I think this is just guessing. To conclusively determine whether the API works we need to ask a bunch of people who actually use it.
Notably, we should probably also document that assume_checked is completely fine in tests.

could do anything with an unchecked address except get a scriptpubkey out of it

Yeah, this was my original idea as well. The cool thing is we can do this now without breaking anything. I'm pretty open to this, will think about it more.

Right now the Address API is too complicated for me to use and I really don't want to update my software to bitcoin 0.30 because of it.

:cry: This is a pretty strong evidence for the proposal above but note that bunch of upgrade problems were caused by lack of PartialEq between the address types and lack of reference-returning assume_checked. These were resolved in master.

Ok! Cool, I think we're (sufficiently) in agreement. Let's try to work toward a "Address<Unchecked> can do everything except script_pubkey (and maybe others; but it should be a short clearly-deliminated list)" model.

but note that bunch of upgrade problems were caused by lack of PartialEq between the address types and lack of reference-returning assume_checked. These were resolved in master.

Oh, phew :).

We already use session types, so I'm not sure what you mean by that example.

My bad, sorry. I'm out of context, got here because of "global variables", haven't even checked, and didn't use Address in the new version. blush
Looking ...
Wouldn't you want some type UncheckedAddress = Address<NetworkUnchecked>?  It eliminates the need to type <...>, and make them look more like distinct types with same set of methods.
Also - some Address::from_str_checked(&str, Network) and Address:to_string_checked(Network) could be useful?
Seems like the documentation and API insist on using the generic methods, but that's not necessary. Generic methods are required when dealing with abstractions that need them. For concrete use:
let address = Address:from_str_checked(s, self.network)?;

seems handier than:
let address = Address::from_str(s)?.require_network(Network::Bitcoin)?;

and autocompletion might suggest it due to common prefix.

Right now the Address API is too complicated for me to use and I really don't want to update my software to bitcoin 0.30 because of it.

Is there somewhere I can read about the biggest pain points? I don't understand "easier to use the Address API in contexts that are affected by orphan rules".

@dpc I'm talking about deserializing then serializing. There is no data loss here.

But there's a context-loss.

Wouldn't you want some type UncheckedAddress = Address<NetworkUnchecked>?

Yeah, that could help. Will be happy to accept a PR after I decide if I like UncheckedAddress more than AddressUnchecked. :)

Address::from_str_checked(&str, Network)

Yeah, it's a bit shorter and if people already like to say from_str instead of .parse() then this is better.

I don't understand "easier to use the Address API in contexts that are affected by orphan rules".

We don't implement FromStr or Deserialize for Address<NetworkChecked> and users who want a global can't implement it themselves due to orphan rules. And while I hate globals, this one is among the most reasonable globals one could want - all applications are single-network.
I also realized there's an additional utility in having this behind a feature flag: if you decide to refactor the code to remove the global, just turn it off and fix all compile errors.

We don't implement FromStr or Deserialize for Address<NetworkChecked> and users who want a global can't implement it themselves due to orphan rules.

But that's the whole point? When you parse/deserialize, you don't know if what you're parsing is actually valid. So you have to pause, read the docs, and come up with a solution.
You can use AddressUnchecked instead, and check later in your code.
You can use some hypothetical AssumeCheckedAddress newtype that impls these traits implicitly, and pinky promise to check later (if required).
You can use some hypothetical #[serde(deserialize_with = "bitcoin::address::deserialize_assume_checked")], and pinky promise to check later (if required).
At least 3 decent options, that are all local, and fairly standard.

Not really following entirely, but from some snipper above: it should definitely be struct Address<S = Verified> so that Address itself is verified.. The unverified variant should be the special/temporary one. So that the normal API is on impl Address {, which is on the verified type.


Scoping polymorphic counters to specific related entity

I noticed that at one point, there was a :polymorphic_associated_models param, but it was removed in favor of inferring them.  I'm only interested in countering one of the polymorphic relationships. Is the accepted pattern to proc these conditionally or is there a way to define the specific polymorphic entities that you're interested in counting?

This isn't something that has come up before; there is no way to define the specific polymorphic entities you would like to count. If using a proc to do this conditionally works for you, that sounds like a good way to approach this. I don't think we specifically test for this, though, so I'm not 100% sure it will work.

@magnusvk I don't understand why you say it hasn't come up before, when there was code to do just this in a PR that was removed. The issue with procs, in general, is that CC refuses to update the counters and you have to write manual routines. If you're going to have to write manual routines anyways, there's not much point in using CC.
Thoughts?


Added Divide Functionality to Vector3

This shouldn't be a thing, it was just convenient for a weird edge case but bad practice...

miss clicked whoopsie


fix: removed skiff (defunct and shutting down in 6 months)

You should remove Skiff since they are defunct and shutting down service in 6 months.  See https://skiff.com/data-migration.

Thanks
I will archive it by editing contents.json


Album with more than 999 pictures does not open in full screen view

Hello, I am using piwigo 12.3.0 (PHP 7.4.6, MySQL: 5.5.5-10.4.17-MariaDB) and the latest bootstrap theme 2.5.8.
I have seen the follwing issue:
Normally when using this theme and I "open" an an album and click on a picture, the picture is shown in full-screen mode and I can wipe left and right.
In case the album has more than 999 picture, the click on a picture does not open the picture in full screen. I do not have also the icon for full screen view.
Any hint is welcome.
My first idea was the plugin GThumbs+, but i do get the same issue with an disabled GThumbs+ plugin.
Many thanks.

Confirmed with Piwigo 12.3.0, PHP 7.4.30 & Bootstrap Darkroom up-to-date as of today.
Active plugins:

Admin Tools
Exif View
Extended Description
Meta Open Graph
No Stats For Robots
RV Sitemap
VideoJS

Ok, I understand the problem.
My biggest album has 1600 pictures - so can you please give me a hint, where I can "enable" this functionality ?
KR
kuzco

Have found it :-) in themecontroller.php :-).
   if (count($page['items']) > 1000)
    {
      $this->config->slick_enabled = false;
      $this->config->photoswipe = false;

      return;
    }