Add reflection to nodes

[This is a continuation of this discussion of describe]
In addition to the regular handler, each node implementation should export a reflection function that describes its inputs and outputs.
This is useful in e.g.

a visual editor, which can understand nodes in this way, especially if the inputs or outputs depend on the configuration of the node (or even incoming data!)
self describing APIs, e.g. when exposed as cloud function
as a way to express errors in a graph and iterate towards resolving those (here, missing inputs should trigger an $error output (see #129) that is the same as calling reflection with the current inputs and wired schemas, see below)

It takes three parameters, all optional:

inputs, which are inputs as if they were passed to the node
input schemas, the currently expected schema of these and other inputs
output schemas, the currently expected output schemas

The output of the reflection function contains:

input schemas, including indicating which are required – note that this can depend on the provided inputs! (see below)
output schemas
how complete the response is, which is one of

calling with the inputs is valid
supplying all inputs marked as required will be sufficient, assuming they match the schema
the inputs are not complete yet and will depend on supplying at least some of [list of missing inputs]

For nodes that have fixed inputs and outputs this boils down to schema validation and we should supply conveniences to generate reflection automatically, ideally from Typescript schemas that are used to implement the node itself.
When input or output schemas are supplied to the call, besides those actually selecting different behaviors in some cases (see below), reflection should verify that inputs conforming to those schemas are valid and/or that outputs will conform to the supplied schema. The return value should only cover the part of the schema that is actually used. This allows the input  data to contain more data / only a subset of the data from the output to be used. But it also allows a higher level validator to make sure that it's fine to ignore the other parts, which is not always the case (e.g. ignoring the second part of Restaurant { name: string, closedPermanently: boolean } might be bad). LLMs might be reasonably good at detecting such cases, FWIW.
Examples of more complex scenarios:

promptTemplate: The actual inputs depend on the {{ variables }} defined in the template. So you can imagine this sequence of calls:

empty call --> input schema describing template as required (and the output schema for prompt, which will be the same for all subsequent calls)
providing template that contains {{ foo }} --> input schema describing template and foo and indicating that supplying a string for foo will be sufficient
(optional:) calling with template and foo --> same input and output schema, but indicating that this would be a valid call

jsonata: The actual outputs depend on the query:

empty call --> input schema describing expression, json and raw + union of possible outputs (TODO: since raw has a default value, technically we should assume result unless raw = true will be provided)
calling with inputs for expression and setting raw to true --> ideally replying with both the input and output schema defined by the expression.
...

runJavascript: Function implies expected parameters
passthrough: Supplying either input or output schema implies the other
etc.

To be discussed: Clearly some of these require advanced parsing of the workload, which might be highly non-trivial to implement. Should we instead call an LLM to provide a reasonable guess? They are pretty good at understanding e.g. code or jsonata queries, but will not always be correct. And it would seem odd to hardcode a specific LLM into nodes. So maybe instead of saying "I don't know", a node can reply with a prompt to an LLM that would answer the question. And maybe we add a way to call reflection again with the answer from the LLM, so that it can turn that into a proper response for us (or even ask another question).
This can also be called in other orders, especially if $role is provided (see #62):

promptTemplate: This time the template is the missing piece:

Call with foo, bar and other given inputs --> input schema describing template with a description that says that the template can contain {{foo}} and {{bar}} (and same output as above)
Given $role and the new description ask an LLM to write a template, call again with that template --> if valid, respond as such, otherwise provide feedback such as missing inputs, e.g. if the LLM hallucinated other template variables

runJavascript, jsonata, etc. could all work the same way
We could even start with just $role and go with hallucinated inputs and then suggest nodes that provide those inputs. This would be a way to bootstrap graph generation from a few key nodes, such as a template.

This is great, loving the direction.

Clearly some of these require advanced parsing of the workload, which might be highly non-trivial to implement.

My intuition is that we should try converting a few nodes into this new world and see how what comes out. I bet we will have more clarity then.

A sketch: instead of exporting just one default export, every node handler file exports two functions: run and describe. Then, these functions are (waves hands) plumbed out.

An interactive visual editor may need more than describe. Consider the following graph:
const pt = promptTemplate();
const identity = passthrough();

identity.wire('template<-input', input());
identity.wire('*', pt);

const llm = runLLM();
pt.wire('*', runLLM());

After we run this graph, we can use describe to get the inputs of the prompt template node pt by passing it the template from identity. However, this requires the graph to be run. Running the graph is expensive (takes a long time) because it includes an LLM node.
If the user is updating their template, they might want to see the changes before they run the whole graph (I imagine some graphs might take minutes to run). For graphs with multiple LLMs, one of which may be generating the template, they could run just the LLMs needed to generate the inputs for the promptTemplate node. This would require running a subgraph up to the point at which all the input nodes to the prompt template are available.
To be even more efficient, we only actually need to run the nodes that generate the template input to the promptTemplate node. However, applying this optimization generally would require us to list the set of inputs (e.g. the prompt template) that affect a node's dynamic inputs, and do this for every node.
Additionally, we could label some nodes as cheap and others as expensive, and we could automatically re-run the set of all cheap nodes reachable without using the output of an expensive node every time the user makes a change. Furthermore, as long as we've run the graph once, we could update a cheap node that depends on the output of an expensive node and another cheap node by caching the last output of the expensive node as long as the inputs to the expensive node have not changed (although this kind of relies on nodes not having side effects).
Both of these approaches require some changes to the graph executor.


Moving to metadata.json

@nibalizer thanks for doing this!

@adrienthebo do you want to push this to forge.puppetlabs.com/puppet or keep pushing it to adrienthebo?

The last update I pushed was to my namespace because of time constraints and lack of access to someone push publishing access; I'm totally fine with this moving under the community namespace.

@adrienthebo I have push access as does @igalic, ping whenever you want a release :)

or, we establish a new way of pushing that's not based on gpg, because that's been pronounced dead :P

How about using Travis CI deploy with encrypted credentials? We do this for
camptocamp's modules so tags on GitHub automatically upload versions to the
forge.
On 2 Mar 2015 14:10, "Igor Galić"<EMAIL_ADDRESS>wrote:

or, we establish a new way of pushing that's not based on gpg, because
that's been pronounced dead :P
—
Reply to this email directly or view it on GitHub
https://github.com/puppet-community/puppet-alternatives/pull/23#issuecomment-76708804
.

i'm cool with that too… whatever gets us easy releases

I have a hard time believing that travis holding a password is secure. That being said a lot of people do it. If people think its safe and the correct way to go then I'm down.

It's either put the credentials in as a travis secret or standard up an external service that receives a webhook.  The former sounds like less of a hassle to me.

Each Travis repository has a pair a keys associated with it, that you can
use to encrypt values using the travis command line tool. The values can
only be decrypted when running a build on the given repository. I think
that's secure enough for this kind of need.
On 3 Mar 2015 01:56, "Spencer Krum"<EMAIL_ADDRESS>wrote:

I have a hard time believing that travis holding a password is secure.
That being said a lot of people do it. If people think its safe and the
correct way to go then I'm down.
—
Reply to this email directly or view it on GitHub
https://github.com/puppet-community/puppet-alternatives/pull/23#issuecomment-76865255
.

Here is an example we use in production:
https://github.com/camptocamp/puppet-openldap/blob/master/.travis.yml
See the deploy section, where the password is hashed. I can't decrypt the
value, it can only be accessed in the build triggered from our repository
(not from forks or PRs either).
On 3 Mar 2015 08:07, "Raphaël Pinson"<EMAIL_ADDRESS>wrote:

Each Travis repository has a pair a keys associated with it, that you can
use to encrypt values using the travis command line tool. The values can
only be decrypted when running a build on the given repository. I think
that's secure enough for this kind of need.
On 3 Mar 2015 01:56, "Spencer Krum"<EMAIL_ADDRESS>wrote:

I have a hard time believing that travis holding a password is secure.
That being said a lot of people do it. If people think its safe and the
correct way to go then I'm down.
—
Reply to this email directly or view it on GitHub
https://github.com/puppet-community/puppet-alternatives/pull/23#issuecomment-76865255
.

And the result of the Travis build on a tag
https://travis-ci.org/camptocamp/puppet-openldap/jobs/51198716#L780
On 3 Mar 2015 10:49, "Raphaël Pinson"<EMAIL_ADDRESS>wrote:

Here is an example we use in production:
https://github.com/camptocamp/puppet-openldap/blob/master/.travis.yml
See the deploy section, where the password is hashed. I can't decrypt the
value, it can only be accessed in the build triggered from our repository
(not from forks or PRs either).
On 3 Mar 2015 08:07, "Raphaël Pinson"<EMAIL_ADDRESS>wrote:

Each Travis repository has a pair a keys associated with it, that you can
use to encrypt values using the travis command line tool. The values can
only be decrypted when running a build on the given repository. I think
that's secure enough for this kind of need.
On 3 Mar 2015 01:56, "Spencer Krum"<EMAIL_ADDRESS>wrote:

I have a hard time believing that travis holding a password is secure.
That being said a lot of people do it. If people think its safe and the
correct way to go then I'm down.
—
Reply to this email directly or view it on GitHub
https://github.com/puppet-community/puppet-alternatives/pull/23#issuecomment-76865255
.

@nibalizer: make it so, number one.

Okay, added redhat support to metadata.json. As for deployment we have some docs here https://puppet-community.github.io/docs.html#publishing


CrossPushNotification different signature fro APIs

It seems that  CrossPushNotification.Initialize(); for Android needs different parameters. so we can not use one initialize method  for different APIs.

Where are you calling this method? should be called on each platform project

I'm going to use that in a common layer for all platforms. so If you can add methods with the same signatures it will be fine.

I need the same please


fix: flushdb may cause master-slave inconsistency

fix Issue #2708
问题简述：因为Slave端消费Binlog线程模型的缘故，原有代码中主从执行flushdb后可能出现主从不一致的现象
背景：Pika Slave在消费一条Binlog时，按事件顺序分为两步，一是Apply Binlog, 也就是将收到的Binlog也追加到自己的BinlogFile中，二是Apply DB, 也即将Binlog中抽取出来的命令写DB。具体地，Slave收到的所有Binlog都会顺序提交到一个固定的Binlog-Worker线程去消费，
在Binlog-Worker线程上，会原地完成Apply Binlog的动作（同步地追加Binlog），之后会向另外一个WriteDBWorker线程池提交一个异步的WriteDB任务（按key来分配使用池子里的哪一个DBworker）。长话短说就是：写Binlog的动作是单线程同步完成的，而写完Binlog之后，从Binlog中提取的WriteDB任务是提交到另一个线程池异步并发做的。
Case举例：

Pika的Slave在apply完binlog后，将binlog所对应的WriteDB任务分发给多个线程处理，分配时采用按key hash的方式取线程。
flushdb是一个会写binlog，却没有key的命令，每次都会固定取某个线程来执行ApplyDB/写DB的操作。
考虑这样一个case：
主节点实际落盘的顺序（同时也是Binlog的顺序）是：

set key1 a1
set key1 a2
flushdb
set key1 a3
主节点执行完上述操作，主实际写DB的顺序是1， 2， 3， 4。 最后DB的状态是：有一个pair为：{key1, a3}

但从节点在Apply完Binlog之后，会将这4条命令提交到WriteDBWorker线程池中的不同线程（按key做hash来分配，flushdb没有key，所以直接用的”flushdb“这个string来做hash），以下就是一种很有可能的分配结果:
WriteDBWorkerThread1执行：set key a1; set key1 a2; set key1 a3;
WriteDBWorkerThread2执行：flushdb
假如因为线程调度原因，thread1先执行完了这3个set，thread2才去执行flushdb, 那从节点实际落盘的顺序就是1, 2, 4, 3, 和主节点的落盘顺序不一致，进而导致主从不一致：flushdb是最后执行的（但实际上flushdb应该发生于set key1 a3命令之前），会导致整个从库DB是空的。但是按照正确预期, 主从应该一致，从库也应当有key1，值为a3。
这个case中flushdb被延后执行了，还有另一种正好相反的case：flushdb被提前执行了，就会出现主节点为空，从节点上却有数据的情况，性质和上述case几乎一样，就不具体阐述了。

解决方案：

必须确保flushdb命令之后的Binlog一定在flushdb执行完之后才被消费。具体变动为：消费flushdb的Binlog时，将写Binlog以及WriteDB两个操作都放在WriteBinlogWorker上同步执行（原本是只有写Binlog任务在这里同步执行，WriteDB任务被分发到DBWorker线程池），说白了就是flushdb特殊化一下，WriteDB的动作从异步改为同步。修改之后：能确保位于flushdb命令之后的Binlog不可能在flushdb执行完毕之前被提交到异步的DBWorker池子里，也就不可能会比flushdb更早执行。

必须确保flushdb命令在真正执行之前，位于flushdb之前的所有WriteDB任务都已经完成了（操作已经写入RocksDB, 而不是还在某个WriteDBWorker的任务队列中排队）。具体变动为：在BinlogWorker上新增一个提交计数/正在异步执行的WriteDB任务的计数。每次BinlogWorker向WriteDBWorker池子中提交WriteDB任务时，都对这个计数加1，每当有相应的WriteDB任务完成，都对这个计数减1（加一个回调函数即可）。在消费到flushdb的binlog，需要在BinlogWorker上将flushdb进行同步地WriteDB前，需要先检查这个任务计数是否为0，如果不为0，就sleep一会（等待前面的异步WriteDB任务被执行完），再次检查该计数，一直等到该任务计数归0，才去原地将flushdb的操作apply到DB。（为什么这个任务计数一定能等到归0：这个任务计数只有在BinlogWorker向WriteDBWorker池子异步提交任务时才会做自增，而这里的sleep等待实际上就是BinlogWorker在等待，换而言之，在这条同步消费的flushdb完全完成之前，下一条Binlog都不会被消费，也不会有任何的WriteDB任务被提交到WriteDBWoker的池子）。修改之后: 能确保flushdb之前的Binlog所对应的WriteDB任务一定都执行完了，才会执行flushdb，能避免出现：执行完flushdb之后（即写了binlog，也已经清空了DB），滞留在WriteDBWorker队列中的WriteDB任务才执行（即先于flushdb的命令在flushdb之后才执行)。

Problem Summary: Due to the Slave end's Binlog consumption thread model, executing flushdb on both master and slave in the original code may result in inconsistencies between master and slave.
Background: When Pika Slave consumes a Binlog, it follows two steps in order: first, Apply Binlog, which appends the received Binlog to its own BinlogFile; second, Apply DB, which extracts the commands from the Binlog and writes them to the DB. Specifically, all Binlogs received by the Slave are sequentially submitted to a fixed Binlog-Worker thread for consumption. On the Binlog-Worker thread, the Apply Binlog action (synchronously appending the Binlog) is completed in place. Subsequently, an asynchronous WriteDB task is submitted to another WriteDBWorker thread pool (the specific DBworker in the pool is assigned based on the key). In short: the Binlog writing action is completed synchronously by a single thread, while the WriteDB tasks extracted from the Binlog are submitted to another thread pool for asynchronous concurrent execution.
Example Case:

After Pika's Slave applies the Binlog, it distributes the corresponding WriteDB tasks to multiple threads, using key hash to assign threads.
flushdb is a command that writes to the Binlog but does not have a key, and it always uses a fixed thread to execute the ApplyDB/WriteDB operation.
Consider this case:
The actual write order on the master (which is also the Binlog order) is:

set key1 a1
set key1 a2
flushdb
set key1 a3
After the master executes the above operations, the actual DB write order on the master is 1, 2, 3, 4. The final DB state is: there is one pair: {key1, a3}

However, after the slave applies the Binlog, it submits these 4 commands to different threads in the WriteDBWorker thread pool (assigned by key hash, flushdb has no key, so the string "flushdb" is used for hashing). The following is a very likely assignment result:
WriteDBWorkerThread1 executes: set key a1; set key1 a2; set key1 a3;
WriteDBWorkerThread2 executes: flushdb
If due to thread scheduling reasons, thread1 completes these 3 sets first, and then thread2 executes flushdb, then the actual write order on the slave is 1, 2, 4, 3, which is inconsistent with the master's write order, leading to master-slave inconsistency: flushdb is executed last (but in fact, flushdb should occur before set key1 a3), resulting in the entire slave DB being empty. However, as expected, the master and slave should be consistent, and the slave DB should also have key1 with value a3.
In this case, flushdb is delayed. Another similar case is when flushdb is executed earlier, leading to a situation where the master DB is empty, but the slave DB has data. The nature is almost the same as the above case, so it will not be elaborated further.

Solution:

Ensure that Binlogs after the flushdb command are not consumed until flushdb is executed. Specifically: when consuming the flushdb Binlog, both the Binlog writing and the WriteDB operations are performed synchronously on the WriteBinlogWorker (originally only the Binlog writing task was done synchronously here, while the WriteDB task was dispatched to the DBWorker thread pool). Simply put, flushdb is treated specially, changing the WriteDB action from asynchronous to synchronous. After the change: it ensures that Binlogs following the flushdb command cannot be submitted to the asynchronous DBWorker pool before flushdb is completed, and thus cannot be executed earlier than flushdb.

Ensure that before the flushdb command is actually executed, all WriteDB tasks prior to flushdb have been completed (operations have been written to RocksDB, not just queued in some WriteDBWorker's task queue). Specifically: add a counter for submitted/ongoing asynchronous WriteDB tasks on the BinlogWorker. Each time the BinlogWorker submits a WriteDB task to the WriteDBWorker pool, this counter is incremented by 1. When a corresponding WriteDB task is completed, the counter is decremented by 1 (adding a callback function). When consuming the flushdb Binlog, before synchronously executing WriteDB for flushdb on the BinlogWorker, it needs to check if this task count is 0. If not, it sleeps for a while (waiting for the previous asynchronous WriteDB tasks to complete), then rechecks the count, and waits until this task count reaches 0 before applying the flushdb operation to the DB in place. (Why this task count can definitely reach 0: this task count only increments when the BinlogWorker submits tasks asynchronously to the WriteDBWorker pool. The sleep wait here means the BinlogWorker is waiting. In other words, before this synchronous flushdb consumption is fully completed, the next Binlog will not be consumed, and no WriteDB task will be submitted to the WriteDBWorker pool). After the change: it ensures that all WriteDB tasks corresponding to Binlogs before flushdb are completed before executing flushdb, avoiding the situation where: after executing flushdb (i.e., writing the Binlog and clearing the DB), the WriteDB tasks remaining in the WriteDBWorker queue are executed (i.e., commands prior to flushdb are executed after flushdb).

Summary by CodeRabbit

New Features

Introduced methods to manage asynchronous DB write tasks.
Added support for callback functions in DB task scheduling methods.

Bug Fixes

Updated method signatures to enhance callback and task handling.

Refactor

Removed obsolete methods and consolidated task handling logic.
Simplified constructor and method signatures for improved clarity and performance.

Performance Improvements

Enhanced asynchronous processing for command handling and DB operations.

其实如果把引用计数减一的操作放到队列任务的析构函数里去做比较好。但是现在队列任务并不能保证一定被执行，一定被析构，所以可能得连着一块改，比如改成unique_ptr管理，而不是用一个裸指针。

其实如果把引用计数减一的操作放到队列任务的析构函数里去做比较好。但是现在队列任务并不能保证一定被执行，一定被析构，所以可能得连着一块改，比如改成unique_ptr管理，而不是用一个裸指针。

done


feat: Use system certificate store

Acceptance criteria
Use the system CA certs when performing pull and push operations.
For pull, use rustls-native-certs to load the system certs, used here:
https://github.com/kubewarden/policy-fetcher/blob/f3212662841a5abf7d09c411fe6a4e5ca9868186/src/https.rs#L40
For push, it may need work on oci-distribution, see:
https://github.com/krustlet/oci-distribution/blob/6ef7f0582384fe7c73575a3169d03a4ebbaa5bc7/src/client.rs#L189-L205
It may be a good idea to be allow disabling this behaviour, so we can decide if we want to default to it or not in kwctl. See https://github.com/rustls/rustls-native-certs#should-i-use-this-or-webpki-roots as examples on why it is a good idea.

For https:// support on kwctl pull & kwctl push, we need to enable the feature ourselves, see https://github.com/viccuad/policy-fetcher/commit/c65d73c2b11d5a9920aab849c7e7b5511f16d9d7, and possibly take care of that TlsVerificationMode::SystemCa. TBD.
For registry:// support on kwctl pull & kwctl push, we need to make the reqwest client being used in oci-distribution to consume the system certificates.
This is done by doing:
https://github.com/viccuad/oci-distribution/commit/7c8f456a936ecb4e1aaa8b6b109c056ec579c7dd
Consuming this in kwctl and debugging it, I can see the system CAs are loaded into the client, but they don't get used later on:
$ kwctl -v 5 pull registry://core.harbor.domain/library/pod-privileged:v0.2.1
2022-12-12T16:10:39.762465Z  WARN rustls::conn: Sending fatal alert BadCertificate
Error: the policy registry://core.harbor.domain/library/pod-privileged:v0.2.1 could not be downloaded due to error: error sending request for url (https://core.harbor.domain/v2/): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Also tried making reqwest pick up the certs manually, with ENV_CERT_FILE=/etc/ssl/certs/ca-certificates.crt, with the same result.

This is more of an issue for kwctl, because with policy-server our container image doesn't have any system certificate store. Nevertheless, this is something that should be addressed

This was already done in: https://github.com/kubewarden/kwctl/pull/386


data_loader

Hello,
In train.py of Image Captioning tutorial, you import get_loader as:
"from data_loader import get_loader "
But I didn't find the corresponding file in the repos. Do I miss anything here? Thanks.

Oh, i missed to commit the data_loader.py file. Now, i uploaded data_loader.py.


Promisify The Rebound Router

The Rebound router loads a lot of content dynamically. All parts of the router should return promises.

[x] Rebound.start() starts the router, returning a promise that resolves when the first route has resolved and page has rendered.
[x] Router constructor should be of the form new Rebound.Router(options[, callback]) where callback is called when the router has started, the first route has resolved and first page has rendered.
[x] Rebound.navigate should return a promise that resolves when the route has resolved and page has rendered.

Updates here: https://github.com/epicmiller/rebound/commit/a82789dbe88f53df9b688df4937a3c9404b647b9


Clear request fields after each request

Following on from my previous pull request submitted into the Master branch:
Introduced a new public method (clearRequestFields) for clearing the request fields after each API request.
Due to BC concerns, this no longer clears the fields by default. A new constant (CLEAR_REQUEST_FIELDS_AFTER_EACH_REQUEST) has been introduced to control if the logic should change from it's existing behaviour.
By default, these changes don't affect any of the existing behaviour.

Thanks for this. Although it's an old one, I'm just going through and trying to clean things up. I'm not a fan of constants as this would require the user altering the code themselves. It should really just be a private variable with a setter.
Because it's such an old one, I'll close for now, but if you're bothered we can reopen it and add the functionality. It'll change for 2.0 anyway.


Object inspector should list function calling signature

From<EMAIL_ADDRESS>on 2010-10-12T08:49:22Z
In python interpreter you can get some help on a function by the following command:
Python 2.6.5 ( r265 :79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> help(json.dump)
Help on function dump in module json:
dump(obj, fp, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, encoding='utf-8', default=None, **kw)
Serialize obj as a JSON formatted stream to fp (a
.write()-supporting file-like object).
If ``skipkeys`` is ``True`` then ``dict`` keys that are not basic types
(``str``, ``unico

As you can see, the first line in the help message is the calling signature - a string that explains how the function is called. Many doc strings rely on this functionality, thus the calling signature is not described in the doc string, as seen in the example above.
Spyder's object inspector uses the same doc string to show the help, but do not show the signature, making the help less useful.
Please include this calling signature in object inspector.
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=409

From pierre.raybaut on 2010-10-31T06:59:13Z
This issue was closed by revision 6d9a58bdf4 .
Status: Fixed

From pierre.raybaut on 2010-10-26T04:36:28Z
Status: Accepted
Labels: -Type-Defect Type-Enhancement Milestone-Release2.0

From pierre.raybaut on 2010-10-26T05:00:59Z
issue #370 has been merged into this issue.

From pierre.raybaut on 2011-04-24T07:36:38Z
This issue was closed by revision 6d9a58bdf4db .


the attention alignment do not converge while outputs_per_step=1

the attention alignment converge in 35000 steps, while r = outputs_per_step = 2, like this:

and converge to a very good rusult after 105000 steps, like this:

but while r = 1, the alignment do not converge after 70000 steps in two separate experiments, like this:

in the paper, r = 1. Dose any one has a good result while r = 1?
by the way, I used keithito's tacotron1 and the alignment do not converge while r = 2, only converge while r = 3, 4, 5

Did you shrink the batch size? In my experience, the tacotron batch size might not be less than 32.

@begeekmyfriend hi, I used the ljspeech corpus to train the tacotron model and the batch_size is 40. I began a new training with the tacotron_initial_learning_rate=1e-4,tacotron_final_learning_rate=1e-6, but there is no sign of convergence. (Totally used the latest commit )

@Jim-Song Please wait until 40K steps. Or you might set symmetric_mels = True in hparam.py. I am using r=2 and batch size = 32.

@begeekmyfriend  thx, I will try it now

@begeekmyfriend the r=1 case alignment still do not converge after 45000 steps.
I noticed that in the tacotron2 paper, the linear loss was not included, but in the latest commit, the loss includes a loss of linear_outputs from mel_outputs by a cbhg. In your experiment ,was the linear loss included?

Of course the linear loss included. Then try symmetric_mels = True

@begeekmyfriend after 8 days' training, it finally converge

and the generated video samples are also better

Which step did it achieve first alignment?

140000-180000steps
@begeekmyfriend

@begeekmyfriend Why you can be close to ~250 of Y-axis？Mine and others just 100~160.

@wuzhonglijz It depend on the length of the training wav clips.

@Jim-Song I am knew to Tacotron2, Please I want to know where I can found r=1 ?

which attention did u use @Jim-Song ? is it Graves attention or Bahadanu attention?


add a cargo config to help macos with linkage

This lets you run cargo build on macos without having to go through the whole hoopla of maturin.

is the MacOS specific? afaik, you can't do cargo build on linux either.
With this patch, do you get a python module from cargo build?

I came across this on the pyo3 docs here. My understanding is that this is just a hack required on macs and that cargo build does succeed on linux, but I haven't tried. This shouldn't have any impact on the behavior on other platforms (because of the [target.<arch>-apple-darwin])
What you get out from running cargo build is a file target/debug/libclvm_rs.dylib, and if you make a symlink named clvm_rs.so that points to that, you can pop open a python shell and do import clvm_rs. You still probably will want to use maturin for everything -- I'm just used to working with watch cargo build -vv running in a terminal for quick feedback.


Problems with Instagram

Hi. maybe only a parametrization issue but Istagram login gives an error

My example is works for me. And also demo is works.
Please, show your code.
If you use my example, you should specify your own application ID and secret key.

Ok. I checked only the demo. I'll work on the example next week. It says "{"code": 403, "error_message": "You are not a sandbox user of this client", "error_type": "OAuthForbiddenException"}". Anyhow great job! Thanks

Yes, in my examples application keys for test. These keys have limited access. Instagram does not allow full access for the application of this type (demo, test).


Deploy web version of the validator in github gh-pages

If we deploy (in an automated way through circleCI) the web version of the BIDS validator be able to quickly test it outside of other projects (for example CRN). This way we will be also able to provide validation service to people who do not (yet!) intend to share their data.
The user interface can be very basic since the main goal is to expose/test the underlying JS library.

Here's the start of this web demo of the validator. Greg hasn't had a chance to do any styling cleanup so there are a handful of visual issues. Once we get those fixed and the repository ownership stuff figured out we can design an auto deployment strategy to keep it up to date with the validator.
https://github.com/Squishymedia/bids-demo

Is this finished? I see some CircleCI authored updates in gh-pages, but I could not find any deployment code triggered when master branch is updated.

This isn't quite finished. I started testing the deployment scripting on the gh-pages branch. Just a few more things to wrap up.

Cool! thanks for the update!

Auto deployment is up and running. This will run on any change to master but changes will only get incorporated into the gh-pages if npm is updated first. So you will only see changes on there if you bump the version number on a commit/pull-request.

Excellent! Thanks!
On Mon, Nov 16, 2015 at 11:34 AM, constellates<EMAIL_ADDRESS>wrote:

Auto deployment is up and running. This will run on any change to master
but changes will only get incorporated into the gh-pages if npm is updated
first. So you will only see changes on there if you bump the version number
on a commit/pull-request.
—
Reply to this email directly or view it on GitHub
https://github.com/INCF/bids-validator/issues/38#issuecomment-157145687.


Inject mounter into volume plugins

Split out from #6936; inject mounter into volume plugins

@vmarmol PTAL

LGTM, just the one nit.

@vmarmol Fixed

LGTM


Tracking issue: refactor versioning tests after the write operation is updated to return the version ID

Currently, the write operation does not return any metadata. As a result,  in PR #5132, all versioning tests utilize the stat operation to retrieve the current object versions.
Once the write operation is update to return the version ID, we can eliminate the need for these additional stat operations.

Hi, I think this issue is part of a bigger scope that returning metadata after writing. This will require a major refactor of our write-related public API.

Hi, I think this issue is part of a bigger scope that returning metadata after writing. This will require a major refactor of our write-related public API.

Yes. as I mentioned in: https://github.com/apache/opendal/issues/2611#issuecomment-2363304110. I'll submit a RFC for this. This issue serves as a reminder to refactor the code in #5132.


Pause all other audio playing while MindBell rings

Original issue: https://github.com/mindful-marc/mindbell/issues/42
Original issue: https://github.com/mindful-marc/mindbell/issues/19

In the moment I'm not sure whether it's possible at all because MindBell had to control your podcast playing application ... this needs some investigation ...

Both ways can be implemented using AudioManager. Suppressing bell sound by using isMusicPlaying(), pausing other audio source by using requestAudioFocus().

To be released with 3.1.0


OmniSharp server crashes on startup

Hello OmniSharp/VSCode team.
IntelliSense doesn't work in my AspNetCore 2.0 project.
OmniSharp fails with following exception copied from OmniSharp Log in VSCode.
Could you please take a look?

Extension Name: csharp
Extension Version: 1.15.2
OS Version: Windows_NT x64 10.0.17134
VSCode version: 1.24.0

System.IO.FileNotFoundException: The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
at Microsoft.VisualStudio.Setup.Configuration.ISetupInstance2.GetState()
at OmniSharp.MSBuild.Discovery.Providers.VisualStudioInstanceProvider.GetInstances()
at OmniSharp.MSBuild.Discovery.MSBuildLocator.GetInstances()
at OmniSharp.CompositionHostBuilder.RegisterMSBuildInstance(IMSBuildLocator msbuildLocator, ILogger logger)
at OmniSharp.CompositionHostBuilder.Build()
at OmniSharp.Stdio.Host..ctor(TextReader input, ISharedTextWriter writer, IOmniSharpEnvironment environment, IConfiguration configuration, IServiceProvider serviceProvider, CompositionHostBuilder compositionHostBuilder, ILoggerFactory loggerFactory, CancellationTokenSource cancellationTokenSource)
at OmniSharp.Stdio.Driver.Program.<>c__DisplayClass0_1.b__1()
at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
at OmniSharp.HostHelpers.Start(Func`1 action)

Sorry guys, this is 100% my local issue.
I upgraded Visual Studio 2017 to 15.7.3 and everything started to work like a charm!


[POAE7-2591] wrap planslice as substrait plan and do RelAlgEU generation

Why are the changes needed?
wrap planslice as substrait plan and do RelAlgEU generation
Does this PR introduce any user-facing change?
No
How was this patch tested?
UT
Which label does this PR belong to?

@yma11 Thanks for PR. Who is suitable to review this PR? Chaojun and Haiwei? I can take a quick look at it.

@jikunshang and @haiweizh can help review on this PR. I will let them know.


Switch to packs

WHY
We recently updated parse_packwerk, code_ownership and other gems in the rubyatscale ecosystem.
They all moved to make packs a more central part of it, but this gem has not been updated yet, which causes issues like the following:
VisualizePackwerk.team_graph!(teams)

# TypeError: Parameter 'package': Expected type Packs::Pack, got type ParsePackwerk::Package with value <ParsePackwerk::Package con...on" public_path="app/public/">
# Caller: ~/.rvm/gems/ruby-3.1.3/gems/visualize_packwerk-0.0.6/lib/visualize_packwerk/package_graph.rb:29
# Definition: ~/.rvm/gems/ruby-3.1.3/gems/code_ownership-1.29.3/lib/code_ownership.rb:162

WHAT

Update the gem to be compatible with the latest version of the ecosystem's gems, including code_ownership.
As a result of switching to packs, we now need to ignore the root package, but that was already in the plans based on comments I found in the code.
Update the README to reflect the changes.
~Chore: Remove Gemfile.lock from VCS.~ Update code_ownership version in Gemfile.lock

Testing
Since this repo has no tests, I manually tested the following steps:

VisualizePackwerk.package_graph! as demonstrated in the README
VisualizePackwerk.team_graph! as demonstrated in the README
The modularity-graph as explained in the d3_graph_generator/README.md

NOTE:
They were all successfully working against our codebase.
However, it's worth noting that we use an older version of parse_packwerk (0.14.0) since we're still using Packwerk 2.
It would be great to have someone else testing these changes against a Packwerk 3 codebase using the latest parse_packwerk.

Also looks like type check is failing! Let me know if I can help with that 😄

@alexevanczuk Gemfile.lock is back and sorbet looks happy when I run it locally.
Also, bumped the version in the gemspec to 0.1.0 🙌


Posts routed twice

Original problem - posts with Chinese characters in permalink go to 404
Solution - modify router in enhanceApp.js (https://github.com/vuepress-reco/vuepress-theme-reco/issues/276)
Current problem (side effect) - need to go back twice in order to go to previous page, presumably due to extra routing introduced in solution
Remark - according to https://github.com/vuejs/vuepress/issues/2716, this problem does not occur for vue-router version <= 3.4.5. But my initial test is unsuccessful, need further testing.

974eb9f removes .html from the permalink, making it different from the path to the static html file. This seems to fix the issue locally but not only Gh pages.

This issue seems to disappear in Chrome 99, but persists in Firefox 98. Maybe Chrome changes its way of handling duplicate routing.
A new solution in the aforementioned issue solves it.


chore: fixing docs - document_qa

Pleasure. Will make more of these as I find them. Hopefully will also get to code contribution at some point :)


Pa 316 fixing unit tests

I have taken following actions on this tickets:

Uncomment commented-out unit tests
Added java-doc comments in each classes and test cases
Conducted a code clean up - removed unnecessary comments, added comments where necessary and re-arrage code to make consistent across different test classes
Run test with coverage and found the classes/methods that need to cover more.
Added more test methods(i.e. count methods) to improve coverage

I made some minor additional improvements, everything looks good now. Indeed code coverage has been improved, all unit tests pass.


fixed Visually out of place drop down in pull requests views with Safari

@mjbvz @alexr00 This PR is with respect to issue #2426
@alexr00 I have done the changes suggested ,please review them ,though there was no script to run the project locally in package.json so i am unable to attach the screenshot

@alexr00  here is the screenshot !


Roll Flutter Engine from 4d5070672859 to 9b2d1923793f (5 revisions)

https://github.com/flutter/engine/compare/4d5070672859...9b2d1923793f
2023-05-04<EMAIL_ADDRESS>Disable flaky tests on arm64 (flutter/engine#41740)
2023-05-04<EMAIL_ADDRESS>Roll Skia from 6950f56ad2ec to a4828415b4af (2 revisions) (flutter/engine#41737)
2023-05-04<EMAIL_ADDRESS>Roll Fuchsia Linux SDK from LIacrdFhuTuHnYh0B... to m0-fwGyFnoDVAvxgk... (flutter/engine#41738)
2023-05-04<EMAIL_ADDRESS>Skwasm Font Loading (flutter/engine#41246)
2023-05-04<EMAIL_ADDRESS>Migrate linux_host_engine to engine v2. (flutter/engine#41715)
Also rolling transitive DEPS:
fuchsia/sdk/core/linux-amd64 from LIacrdFhuTuH to m0-fwGyFnoDV
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC jimgraham@google.com,rmistry<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Subpar decoding for transcoded jpegs

Describe the bug
Being aware that jpeg decoding is not standardized I never observed such a decoding degradation as cjxl did.
For comparison I decoded the jpeg with mozjpeg library and compared with the output of "To Reproduce" calls.
For mozjpeg I used the call:
djpeg -bmp -dct float "input.jpg" >"input.jpg.bmp"
To Reproduce
Lossless transcode with jpeg input:
cjxl "input.jpg" "input.jpg.jxl"
djxl "input.jpg.jxl" "input.jpg.jxl.png"
Expected behavior
I expect an exact, if not very close rendition of a reference jpeg decoding library, not a degradation.
Screenshots
dec_moz
dec_jxl
Environment

OS: Win7 SP1 x64
CPU type: i7-4790K
djxl version string: JPEG XL decoder v0.7.0 ab775e2 [AVX2,SSE4,SSSE3,Scalar]
cjxl version string: JPEG XL encoder v0.7.0 b432598 [AVX2,SSE4,SSSE3,Scalar]

Let me upload the images you linked to this issue, so they don't get lost.
dec_moz:

dec_jxl:

Perhaps it would also be useful to have the original jpg for comparison, since on a superficial evaluation, I don't see the degration you are talking about, even when doing a flickering comparison at 400%.

I don't know but I don't need to zoom in nor flicker test to notice the jxl decoding is blurrier.
I'm not sure where it fails but since it's more noticeable on reds it might be related to the YUV to RGB conversion, more precisely the chroma upscaling kernel.
Here's the image, I think imgur don't preprocess them.

Maybe there's is another issue with rendering the pngs somehow, for example a viewer doing some upsampling with smoothing in between?
Do you also find the differences when looking at those images in a browser?
Perhaps a screenshot could help us understand what degration you are talking about.
Thanks!

Does it still happen if you pass --jpeg_transcode_disable_cfl to cjxl when transcoding? If I decompose the images to LAB in GIMP I can see barely any changes in the L channel, but some larger differences in the A/B channels that could explain it.

Thanks @TheDecryptor
That helped quite a bit, what does it stands for? I searched and found Contextual Feature Loss, is it?
Still there's some subtle blur remaining though.

I can see a small difference between those two decoded images when I magnify enough, but I have no reason to assume that the libjxl decode is 'worse' than the libjpeg-turbo/mozjpeg decode. There's a difference, no doubt caused by differences in the IDCT implementation, the YCbCr to RGB implementation, and possibly small effects caused by Chroma From Luma (this is where --jpeg_transcode_disable_cfl can indeed help).
As far as I understand, libjxl does things more accurately than libjpeg-turbo/mozjpeg, e.g. we have YCbCr data in float32 and convert it to RGB in float32 before quantizing it to uint8, while libjpeg-turbo/mozjpeg afaiu decodes to 8-bit YCbCr first and then converts that to 8-bit RGB.
So unless proven otherwise, I think if any jpeg decoder is 'subpar', it's more likely to be libjpeg-turbo/mozjpeg than libjxl.

Yes, dithering can cause a slight blurring which might explain the remaining slight blur, but the offending one that fixed with disabling Chroma From Luma might help, maybe disabling it when 4:4:4?
Probably it isn't related to this issue because of the full chroma, but I observed that mozjpeg resamples chroma with bilinear (triangular kernel) while cjxl does some wizardry that turns into a blurry blocky mess.

Just to be sure, which of the two images is the jxl?

Just to be sure, which of the two images is the jxl?

The blurrier one

This seems to be a larger difference than expected. Can you share the original jpeg for this example? I want to try to reproduce this.

That does not do lossless jpeg recompression at all. That decodes the JPEG to pixels, downscales the image 2x, then encodes the result losslessly. Obviously downscaling an image 2x is a very lossy operation, which will of course introduce blur.

I was just replying to @jonsneyers quote.
Tomorrow I will upload a deviation test between different decoders to see how much it differs.

So, let's compare different jpeg decoders with the example image you provided.

jxl: (cjxl losslessly recompressed, then djxl),
mjpeg: using ffmpeg
mozjpeg/libjpegturbo with three different settings, namely

-dct in (the default)
-dct fast (same as -fast?) and
-dct float

rust's jpeg decoder, https://docs.rs/jpeg-decoder/latest/jpeg_decoder/

Then I compare the decompressed images along different metrics:

MSE (as a sanity check)
MAE
PAE
butteraugli
Here butteraugli is not symmetric, which is why I print the whole matrix of comparisons, even though for the other metrics half of the matrix would be enough to communicate the differences for all pairs.

MSE
jxl
mjpeg
fast
float
int
rust

jxl
0 (0)
1.64258 (2.50641e-05)
2.79644 (4.2671e-05)
1.64121 (2.50433e-05)
1.63914 (2.50116e-05)
1.64237 (2.5061e-05)

mjpeg
1.64258 (2.50641e-05)
0 (0)
1.92949 (2.94421e-05)
0.0443743 (6.77109e-07)
0.0698835 (1.06635e-06)
0.0835574 (1.275e-06)

fast
2.79644 (4.2671e-05)
1.92949 (2.94421e-05)
0 (0)
1.98651 (3.03123e-05)
2.00253 (3.05567e-05)
2.03194 (3.10054e-05)

float
1.64121 (2.50433e-05)
0.0443743 (6.77109e-07)
1.98651 (3.03123e-05)
0 (0)
0.0467264 (7.12999e-07)
0.0508508 (7.75934e-07)

int
1.63914 (2.50116e-05)
0.0698835 (1.06635e-06)
2.00253 (3.05567e-05)
0.0467264 (7.12999e-07)
0 (0)
0.0543799 (8.29784e-07)

rust
1.64237 (2.5061e-05)
0.0835574 (1.275e-06)
2.03194 (3.10054e-05)
0.0508508 (7.75934e-07)
0.0543799 (8.29784e-07)
0 (0)

PAE
jxl
mjpeg
fast
float
int
rust

jxl
0 (0)
6425 (0.0980392)
6425 (0.0980392)
6425 (0.0980392)
6425 (0.0980392)
6425 (0.0980392)

mjpeg
6425 (0.0980392)
0 (0)
1542 (0.0235294)
771 (0.0117647)
771 (0.0117647)
771 (0.0117647)

fast
6425 (0.0980392)
1542 (0.0235294)
0 (0)
1542 (0.0235294)
1542 (0.0235294)
1542 (0.0235294)

float
6425 (0.0980392)
771 (0.0117647)
1542 (0.0235294)
0 (0)
771 (0.0117647)
771 (0.0117647)

int
6425 (0.0980392)
771 (0.0117647)
1542 (0.0235294)
771 (0.0117647)
0 (0)
771 (0.0117647)

rust
6425 (0.0980392)
771 (0.0117647)
1542 (0.0235294)
771 (0.0117647)
771 (0.0117647)
0 (0)

MAE
jxl
mjpeg
fast
float
int
rust

jxl
0 (0)
208.392 (0.00317986)
310.788 (0.00474231)
208.236 (0.00317748)
208.028 (0.0031743)
208.407 (0.00318009)

mjpeg
208.392 (0.00317986)
0 (0)
259.174 (0.00395475)
8.37492 (0.000127793)
13.0649 (0.000199357)
15.6423 (0.000238686)

fast
310.788 (0.00474231)
259.174 (0.00395475)
0 (0)
264.576 (0.00403718)
265.991 (0.00405876)
268.705 (0.00410018)

float
208.236 (0.00317748)
8.37492 (0.000127793)
264.576 (0.00403718)
0 (0)
9.01121 (0.000137502)
9.9194 (0.00015136)

int
208.028 (0.0031743)
13.0649 (0.000199357)
265.991 (0.00405876)
9.01121 (0.000137502)
0 (0)
11.3722 (0.000173528)

rust
208.407 (0.00318009)
15.6423 (0.000238686)
268.705 (0.00410018)
9.9194 (0.00015136)
11.3722 (0.000173528)
0 (0)

butteraugli
jxl
mjpeg
fast
float
int
rust

jxl
0.0000000000
0.9411652088
1.1420263052
0.9466884732
0.9448851943
0.9460596442

mjpeg
1.0180765390
0.0000000000
1.1411626339
0.7881607413
0.8761280775
0.8702933788

fast
1.1666915417
1.1633328199
0.0000000000
1.1629202366
1.1629323959
1.1628748178

float
1.0231082439
0.8004256487
1.1426250935
0.0000000000
0.6474697590
0.6527773142

int
1.0192036629
0.9545516968
1.1423904896
0.6336971521
0.0000000000
0.3134871423

rust
1.0201907158
0.9508787394
1.1427843571
0.6364465952
0.3598134220
0.0000000000

Images for that comparison are here:
https://gist.github.com/mo271/f47f811dbff2f34dedd602f1f330b1f5
From this and visual inspection of the results, I conclude that the differences between jxl and mozjpeg (default) with are not greater than expected, and not greater than let's say between mozjpeg (default) and mozjpeg (-fast). Of course those are the results with this particular image, comparing on a representative corpus could give more insight on how the different ways of decoding jpeg behave.
When you write "I never observed such a decoding degradation", which ways of decoding jpeg did you observe other than mozjpeg?
Also I think it would be helpful to point to more example of degradation to better understand this issue.

Question still is which one is closer to the actual jpeg spec, i.e. using mathematically exact, infinite precision arithmetic.
As far as I understand, the existing jpeg decoders like libjpeg-turbo do iDCT with either floats or ints, but then they produce uint8 YCbCr, which then gets upsampled if needed and then converted to uint8 RGB.
By contrast, current libjxl does the whole decode pipeline with floats, and only converts to uint8 at the very end if uint8 is what is desired.
So in terms of the precision of arithmetic, I think we can assume that libjxl is more precise than the rest.
Note that even just converting 8-bit RGB to 8-bit YCbCr and back causes at least off-by-two errors in R and B (something libjxl avoids by never materializing 8-bit YCbCr buffers but keeping things in float).
The only potential source of inaccuracy could be the use of chroma-from-luma in the encoder, which might have subtle effects on Cb and Cr — not enough to change the quantization bucket (otherwise it wouldn't be reversible) but potentially enough to change the Cb and Cr values themselves quite a bit when the source JPEG is low enough quality. Note that JPEG conformance even allows causing an off-by-one in the quantization bucket, so this wouldn't make it non-conforming, but it could create noticeable differences in the decoded result. This is where the --jpeg_transcode_disable_cfl flag is useful.

So in terms of the precision of arithmetic, I think we can assume that libjxl is more precise than the rest.

More precise than mozjpeg float? I mean, when I said subpar JPEG decoding I meant exactly that, leaving model conversion out of the question.
Correct me if I'm wrong but for transcoding  JPEG -> JXL -> JPEG there's no conversion to or from RGB?
If you decoded the source JPEG and the JXL decoded JPEG both diectly to YCbCr they should give you the same output, that's how I performed the comparisons (then using the same YUV->RGB converter for both to output png). I can't tell about @mo271 tests as I don't think mozjpeg can output YUV in any form.
This leaves Photoshop and mozjpeg out of test batch. I can decode with ffmpeg in avisynth via FFMS2 directly to YCbCr, as you noted that's uint8 but checking the above tables the difference is not big between 8-bit and float.

Actually no, my comparisons were jpeg vs jpeg -> jxl -> jpeg. I decode directly to YCbCr so the degradation is visible even without conversion to RGB.
In any case, hold on, I will do a second check just in case.

If you do --strip, then it will drop the jpeg bitstream reconstruction so djxl will decode to rgb pixels and then do a reencode as q95 jpeg.
Without the --strip, the process would be file-exact reversible. With the --strip, you're introducing an extra lossy transcode...

Then I will need to run my tests with RGB conversion in mind (output to png) because --strip saves some size by removing the jbrd box.

Could you provide the resulting images, so we can compare them? At least the avisynth one, the jxl we can easily generate from the original jpg of course..
Did you use the --strip option in the initial post in this issue?

No, I didn't use --strip in OP, but the issue is JXL uses a different iDCT codepath when reconstructing jpeg so it can't be used as reference. The problem seems to lie when iDCT (not reconstruction) transcoded jpegs, rather than color model conversion I think.
Rounded
Dithered

The jbrd box allows to reconstruct the original JPEG file, so you can use any JPEG decoder you like. If the file size is a real concern, the original JPEGs can be optimized to bare JFIF before transcoding to JXL, then the jbrd box will be pretty small. The size of the jbrg box is proportional to the size of the original file, so optimizing JPEGs before transcoding, to make them as small as possible, actually makes the resulting JXL smaller too, although it affects only the jbrd data and not the actual JXL bitstream.

No, I didn't use --strip in OP, but the issue is JXL uses a different iDCT codepath when reconstructing jpeg so it can't be used as reference. The problem seems to lie when iDCT (not reconstruction) transcoded jpegs, rather than color model conversion I think. Rounded Dithered

If I understand correctly, in https://github.com/libjxl/libjxl/issues/1470#issuecomment-1149191000  you generate a use jxl to generatea a png from a jpg (going through jxl) and then compare it with another image. Could you provide the pixels of the other image also as png?
I don't understand what those two jpgs that you linked are.

I don't understand what those two jpgs that you linked are.

Hi @mo271, you asked me for the avisynth images and that's what I uploaded, to be more concise the output of this:
FFImageSource("input.jpg") # original image
ConvertBits(32)
fmtc_matrix(mats="601", col_fam="RGB", fulls=true, fulld=true)
Convertbits(8,dither=-1) # or dither=1 for floyd-steinberg

That is FFImageSource jpeg decoding (mjpeg?) then converting to RGB in float point as JXL does. I don't know what kind of dithering JXL does so I uploaded both options, or if you tell me I can upload a matching dither but I think they are no deterministic(?) I don't know.
The problem now resides that for comparison there's no framework where I can do both iDCT and RGB conversion in float, as mozjpeg only does that for iDCT (and no lossless YUV output is possible) and AviSynth only does that for RGB conversion.
In any case it doesn't take much to realize the problem is not in either of these (iDCT in int/float or RGB in int/float). JXL to JPEG code path is of reconstruction not decoding, where the issue shows only when actually decoding TRANSCODED jpeg's.

I'm trying to understand this issue, and I'm not yet convinced that jxl doing everything in float give worse or even significantly worse results than other ways of converting from jpeg. The blurriness that you mention is not the only artifact to be worried about.
We are comparing the jpeg decoding of jxl, triggered by
cjxl "input.jpg" "input.jpg.jxl"
djxl "input.jpg.jxl" "input.jpg.jxl.png"
with jpeg decoding of other decoders. So far your provided one image (the original jpeg) and one other code (mozjpeg). As pngs to do the comparisons, you provide two cropped pngs where slight differences are visible, but where the butteraugli-metric is small enough that we don't worry about any unexpected degradation.
> butteraugli_main tF7XRRL.png rhq5Z6i.png
0.7073692083
3-norm: 0.232715
> butteraugli_main rhq5Z6i.png tF7XRRL.png
0.8325679302
3-norm: 0.269613

In order to compare this with other ways of decoding the jpeg, it would be useful to have the png output of other codecs. (That is why I asked of pngs in https://github.com/libjxl/libjxl/issues/1470#issuecomment-1149808491)
Ideally we would like to compare against some decoder that does everything in float or arbitrary precision arithmetic.

As nobody can actually tell, how decoded JPEG images should look like exactly, wouldn't be better to compare decoded JPEG images with the original uncompressed images?

I'm not implying that the issue is in the float pipeline, actually I think it's in the iDCT (hopefully at decoding) of transcoded jpegs, bitdepth regardless.
I could see the same blurriness with other images, but this one so far gave me the worst case scenario, probably due to the presence of a saturated red area. If I could spot it easily (not sure if it helps having a trained eye) anyone else can also do it. A high butteraugli metric might be fine for lossy, but lossless...(?).
Also how are you running butteraugli, all three channels averaged? That is going to even things out, localize it to the offending channel and the metric should go higher, probably much worse than it probably should.
In any case butteraugli is only tested for differences between 90 and 95, something like 96 (bad in lossless terms) would pass as legit in butteraugli scores. Another aspect is metrics are usually very bad at factoring in blurriness, I don't know how many of them do but it's the reason B-SSIM was created. I will run a test later today with butteraugli and provide you with the other pngs (unfortunately with RGB conversion baked in, that's why I didn't find a point sharing them at first)
@jendalinda, yes I actually had that in mind but forgot to test. Will check as well.

So I did a little test using this command line:
f=[INPUT FILENAME]; for i in 70 80 90 95; do echo q$i:; convert $f -quality $i $f.q$i.jpg; cjxl --quiet $f.q$i.jpg $f.q$i.jpg.jxl; djxl $f.q$i.jpg.jxl $f.q$i.jpg.jxl.png --bits_per_sample 16 2>/dev/null; for m in PAE MAE PSNR; do echo -n "libjpeg-turbo: $m "; compare -metric $m $f $f.q$i.jpg null:; echo; echo -n "jxl: $m "; compare -metric $m $f $f.q$i.jpg.jxl.png null:; echo; done; done

So this is producing a few jpegs from a png original, and then computing peak absolute error (PAE, lower is better), mean absolute error (MAE, lower is better) and PSNR (higher is better) between the original and the jpeg as decoded by imagemagick (i.e. libjpeg-turbo) and as decoded by libjxl after recompressing the jpeg losslessly.
You can try this test starting from any input image and with other quality settings if you want.
This is what I got for some random example image:
q70:
libjpeg-turbo: PAE 21331 (0.32549)
jxl: PAE 22102 (0.337255)
libjpeg-turbo: MAE 947.761 (0.0144619)
jxl: MAE 924.809 (0.0141117)
libjpeg-turbo: PSNR 33.6959
jxl: PSNR 33.9103
q80:
libjpeg-turbo: PAE 12336 (0.188235)
jxl: PAE 12932 (0.19733)
libjpeg-turbo: MAE 824.314 (0.0125782)
jxl: MAE 803.689 (0.0122635)
libjpeg-turbo: PSNR 35.0689
jxl: PSNR 35.2994
q90:
libjpeg-turbo: PAE 8224 (0.12549)
jxl: PAE 9858 (0.150423)
libjpeg-turbo: MAE 602.758 (0.00919749)
jxl: MAE 586.17 (0.00894438)
libjpeg-turbo: PSNR 38.162
jxl: PSNR 38.436
q95:
libjpeg-turbo: PAE 5911 (0.0901961)
jxl: PAE 6269 (0.0956588)
libjpeg-turbo: MAE 454.982 (0.00694257)
jxl: MAE 441.292 (0.00673369)
libjpeg-turbo: PSNR 40.7147
jxl: PSNR 41.0553

Here is what I got for another random test image:
q70:
libjpeg-turbo: PAE 20046 (0.305882)
jxl: PAE 19559 (0.298451)
libjpeg-turbo: MAE 1006.07 (0.0153517)
jxl: MAE 987.158 (0.0150631)
libjpeg-turbo: PSNR 32.9006
jxl: PSNR 33.06
q80:
libjpeg-turbo: PAE 17733 (0.270588)
jxl: PAE 18204 (0.277775)
libjpeg-turbo: MAE 886.765 (0.0135312)
jxl: MAE 868.436 (0.0132515)
libjpeg-turbo: PSNR 34.1043
jxl: PSNR 34.2774
q90:
libjpeg-turbo: PAE 10023 (0.152941)
jxl: PAE 14324 (0.21857)
libjpeg-turbo: MAE 635.731 (0.00970063)
jxl: MAE 614.988 (0.00938412)
libjpeg-turbo: PSNR 37.4551
jxl: PSNR 37.7492
q95:
libjpeg-turbo: PAE 6425 (0.0980392)
jxl: PAE 7089 (0.108171)
libjpeg-turbo: MAE 473.937 (0.00723182)
jxl: MAE 462.292 (0.00705412)
libjpeg-turbo: PSNR 40.1479
jxl: PSNR 40.4159

For that same test image, this is what I get when adding --jpeg_transcode_disable_cfl to cjxl :
q70:
libjpeg-turbo: PAE 20046 (0.305882)
jxl: PAE 19559 (0.298451)
libjpeg-turbo: MAE 1006.07 (0.0153517)
jxl: MAE 987.158 (0.0150631)
libjpeg-turbo: PSNR 32.9006
jxl: PSNR 33.06
q80:
libjpeg-turbo: PAE 17733 (0.270588)
jxl: PAE 18204 (0.277775)
libjpeg-turbo: MAE 886.765 (0.0135312)
jxl: MAE 868.436 (0.0132515)
libjpeg-turbo: PSNR 34.1043
jxl: PSNR 34.2774
q90:
libjpeg-turbo: PAE 10023 (0.152941)
jxl: PAE 10127 (0.154528)
libjpeg-turbo: MAE 635.731 (0.00970063)
jxl: MAE 615.809 (0.00939665)
libjpeg-turbo: PSNR 37.4551
jxl: PSNR 37.7718
q95:
libjpeg-turbo: PAE 6425 (0.0980392)
jxl: PAE 5752 (0.0877699)
libjpeg-turbo: MAE 473.937 (0.00723182)
jxl: MAE 456.713 (0.006969)
libjpeg-turbo: PSNR 40.1479
jxl: PSNR 40.5829

So basically what this shows is that at least in terms of PSNR and MAE, jxl decoding of recompressed jpegs is better than "standard" decoding in the sense that it is closer to the original image. The PAE is sometimes better, sometimes worse; not sure if that means much; at least the numbers are in the same ballpark.
MAE and PSNR are consistently somewhat better for jxl decoding of a recompressed jpeg, confirming what was said earlier: libjxl is the more accurate decoder here, and if you see a difference, it's actually libjpeg-turbo producing a "subpar" result, not libjxl.

More tests here, Pyron.png is ground truth image, I converted to jpeg using libjpegturbo since mozjpeg was a bit blurry:
o=ImageSource("Pyron.png",start=0,end=0).ConverttoPlanarRGB().ConvertBits(32,fulls=true)
a=ImageSource("Pyronq95turbo.jpg.bmp",start=0,end=0).ConverttoPlanarRGB().ConvertBits(32,fulls=true)
b=ImageSource("Pyronq95turbo.jpg.jxl.png",start=0,end=0).ConverttoPlanarRGB().ConvertBits(32,fulls=true)

bl=o.ex_blur(0.3)
Butteraugli(o,bl) # score: 0.8894

Butteraugli(o,a)  # score: 1.524369

Butteraugli(o,b)  # score: 1.452442

bl=a.ex_blur(0.25)
Butteraugli(o,bl) # score: 1.465598

Which means that butteraugli metric is favoring blurry targets. Will run more tests.
Original, JPEG, JXL

You cannot use either MAE or PSNR as blurring introduces error spread, kind of like denoising (just tested with MAE). If any we should try with a sharpness metric, I don't know of any robust one, should research on that. I know there were some algos with laplace to detect blurring but we are talking on low decimals here.

Ok, but doesn't a less precise DCT also have the effect of introducing more (unwanted) edges, hence improving the score on that metric?
I know PSNR/MAE are quite tolerant w.r.t. blurring, but they still shouldn't give a better score to an image that is more blurred than to an image that is less blurred compared to the original.

In the images that you attached, I don't think there is any unexpected issue with problematic jpeg decoding. While there might be slight differences, they can be explained by the fact that libjxl is using floats throughout and libjpeg-turbo/mozjpeg are doing less precise computation.
Artificially doing less precise calculations in order to be more similar to existing jpeg decoding implementation, is not really an option, especially in the light of the fact that jpg-decoding is in some sense a side effect of the fact that we are able to both losslessly re-compress existing jpgs and also decode to other formats (like png).
If you prefer the (in your eyes) less blurry (more blocky) decoding of mozjpeg, then perhaps instead of going

jpg -----losslessly recompress-with-cjxl----->
jxl  ----decode-with-djxl------->
png

you could do

jpg -----losslessly recompress-with-cjxl----->
jxl  ----reconstruct-losslessly-with-djxl------->
jpg -----decode-with_djpeg ----->
png

I'm unsure what action we are supposed to take on this issue.

Ok, but doesn't a less precise DCT also have the effect of introducing more (unwanted) edges, hence improving the score on that metric?

I don't know maybe, I just tested by adding a deblocker and checking metrics. I added two different ones, a deblocker PP (Deblock_QED func in AVS) and a straight HQ iDCT decoder JPEGSource (closed source unfortunately). In both cases sharpness was still higher than the JXL decoding. I improved the code to compute over Luma instead of Red channel (practically the same), and added a gaussian type of blur instead of mean average.
There are more sharpness metrics listed here, I didn't test them.
o=o.DotClip([0.298903,0.586620,0.114477])
a=a.DotClip([0.298903,0.586620,0.114477])
b=b.DotClip([0.298903,0.586620,0.114477])
sh=ex_shape(1,mode="square",center=true)

mx = ex_luts(b, mode="max", pixels=sh)
mn = ex_luts(b, mode="min", pixels=sh)
av = b.ex_blur(0.5,mode="gaussian")
bv=ex_lutxyz(av,mn,mx,"y 1 + log x 1 + log X@ - dup * z 1 + log X - dup * + 0.5 *")

mx = ex_luts(a, mode="max", pixels=sh)
mn = ex_luts(a, mode="min", pixels=sh)
av = a.ex_blur(0.5,mode="gaussian")
av=ex_lutxyz(av,mn,mx,"y 1 + log x 1 + log X@ - dup * z 1 + log X - dup * + 0.5 *")

# ex_makediff(bv,av, metric="MAE", aug=50, dif=true, show=3) # Sharpness Diff x50 augmented

ScriptClip(function [] () {
    avg=AverageLuma()
    subtitle("Log Variance: "+string(avg))
} )

JPEGSource,Deblock_QED(quant1=30, quant2=40), |a.sharpness - b.sharpness| diff map

In the images that you attached, I don't think there is any unexpected issue with problematic jpeg decoding. While there might be slight differences, they can be explained by the fact that libjxl is using floats throughout and libjpeg-turbo/mozjpeg are doing less precise computation.

We showed before that mozjpeg float decoding against JXL was still a big difference. And that RGB float conversion wasn't a meaningful change either. My suspect was the iDCT coefficients. Maybe I can give them a try and see if the problem lies there. Do you know what file should I be looking at for them?
By the way, just tested direct encoding (-q 95) to JXL for this image and the decoded also showed less sharpness score. This wasn't apparent though on my first image example, this case here is extreme (stars and high saturation).

Reading the JXL paper  there are many steps where a low-pass is performed. I don't know if anyone is involved in lossless jpeg transcodes though.
Color correlation

In particular, the default correlation factor is 0 for the X channel and 0.935669 for the B channel: this allows
to transmit luma information using the Y channel only.
However, these factors are not necessarily optimal on areas of the image with a strong chroma component
(such as, for example, fully red regions). Therefore, JPEG XL allows altering the correlation factors locally, for
tiles of 64 × 64 pixels, with a dedicated control field.

DC handling

To further reduce banding in areas with steeper gradients, JPEG XL applies a selective smoothing algorithm
to the DC image, that is only allowed to move values inside their quantization boundaries. If the smoothed value
would be outside of the quantization boundaries, it is discarded, and the original is used.

LF predictions

After applying the a smoothing algorithm similar to the one used on the DC image, the upsampled DC is
converted back to DCT coefficients; the low-frequency values that are produced this way are then added to the
encoded low-frequency coefficients.

Loop filters

Despite the significant improvements that JPEG XL delivers to reduce artefacts, block boundaries and ringing
can still be noticed, especially at somewhat lower qualities. Ringing artefacts are spurious signals near sharp
edges, caused by quantizing or truncating high-frequency components (see Figure 7). To mitigate their impact,
JPEG XL employs two different loop filters that are applied to the image after the decompression process.
The first loop filter is a smoothing convolution. As smoothing the image inherently introduces a sharpness
loss, this effect is compensated by the encoder by means of a sharpening filter that is applied before the DCT
step. The overall effect of this procedure is that the visual impact of block boundaries gets reduced, while still
preserving sharp details present in the original image.
The second loop filter is intended to reduce ringing, while still preserving texture that is transmitted in the
image. To achieve this effect, it applies an adaptive smoothing algorithm related to Non-Local Means

What I do know is JPEG XL does things differently at the decoding stage as depicted by the slider from @jonsneyers.
Like I said in my above post unless we know what kind of dequantization coefficients it uses we are at a moot point.
As a reference I have been using very good coefficients in mozjpeg that achieves better qual/size ratio than any of the internal presets:
# with -q 82 achieves a better qual/ratio than plain -q 95
6   6   7   9  11  14  14  14
6   6   7   9  12  14  14  14
7   7   8  10  12  14  15  15
9   9  10  12  14  16  17  17
11  12  12  14  17  19  20  20
14  14  14  16  19  21  22  22
14  14  15  17  20  22  23  23
14  14  15  17  20  22  23  23

@veluca93 pointed out that the quant bias, that we have, see here: https://github.com/libjxl/libjxl/blob/a67a57006f5cd8727d528fa26fa2582cf7098952/lib/jxl/quantizer.h#L59
leads to some additional numerical inaccuracies wenn decompressing lossless recompressed jpgs to pixels.
I gave that a try and hardcoded the values in that struct to be {1.0f, 1.0f, 1.0f, 0.f} and then together with the flag --jpeg_transcode_disable_cfl, the numerical values tests similar to the ones above, are already much closer to what mozjpeg provides.
Do you think we should either somehow turn the quant bias of in the jpeg-decoding case, or introduce another flag to control this, @jonsneyers and @veluca93?

Where does the quant bias get used in the case of jpeg recompression? I thought this was an encode thing (which wouldn't apply to jpeg recompression since there the coeffs are quantized already), but this looks like it is a decode thing? Is this used for AC or for DC or both? Is it signaled?
If it causes the decoded image to be closer to the original image (as my little test with mae/psnr suggested), I don't think we should disable it by default. Getting pixels that are closer to the original seems to me to be more valuable than getting pixels that are closer to the decode result of libjpeg-turbo. Could you do a test like the one I did (https://github.com/libjxl/libjxl/issues/1470#issuecomment-1156242387) to see how it affects the pae/mae/psnr w.r.t. the original image?

Where does the quant bias get used in the case of jpeg recompression? I thought this was an encode thing (which wouldn't apply to jpeg recompression since there the coeffs are quantized already), but this looks like it is a decode thing? Is this used for AC or for DC or both? Is it signaled?

It is signaled. AC only, and actually decode only :)

Thanks for the tests @mo271, maybe the bell-shape distribution (looks like poisson but didn't check) causes the noted smoothness.
I collected a bunch of DCT quant coefs and the one posted gave me the best results on mozjpeg (had to disable trellis as it was also causing blur). It follows an equation by Jukka Ripatti.
I don't know if it can be also be applied to AC after a source quality estimation. At worst an option should come handy for testing purposes for the time being.
@jonsneyers, I don't think those metrics are suited for sharpness evaluation which is the main case here. We can run log variance as a sharpness metric or the other ones linked (https://github.com/libjxl/libjxl/issues/1470#issuecomment-1157524011)

For the one year anniversary of the issue I gave the last dev build a whirl and the output in pixels of the lossless transcoded jpg is still as blurry as it was a year ago, actually a bit blurrier since now I cannot use --jpeg_transcode_disable_cfl.
This shocks me as the solution was given back then.
So I wonder if there's a problem in implementing the fix or at least an user defined quant table, as I believe this is being a showstopper for mass adoption of the format. I had hopes this was something else than JPEG 2000 or JPEG XR.
I improved the sharpness function a bit to be more granular with little details and give more weight to the texture and not to the edges. These are the results.
JXL Sharpness Map, JPEG Turbo Sharpness Map

This methodology of using a sharpness metric to evaluate the quality of a decoder reminds me of people going to HydrogenAudio and posting pictures of spectrograms of compressed music and saying "Look how bad it sounds!".
I can't see the blurriness even zooming 450% into the JPEG/JXL Pyron files. I do notice that some of the gradient banding in some of the flat areas is slightly different, and in my opinion it doesn't classify as "loss".
(Technically it's more accurate to the underlying coefs, but that won't stop you from wanting it to look like the other decoders...)
Maybe mathematically it's different output compared to JPEG decoders, but generally the image is still, like, 99.985% the same.

Unrelated to the bug, but @Dogway it's fine to criticize the state of JPEG XL, but please try not to slander the codec, especially in conversations only tangentially-related to lossless_jpeg (issue 683) and while the major release hasn't even reached 1 yet.
It's good you've pointed out the JPEG XL decoder produces different output... Just keep the discourse reasonable

I'm going to throw my hat in the ring and say that the jpeg decoder present in djxl is fine. As @Firepal said, it's just different (in my opinion it's better) and definitely not worse.
I think what's going on is that the jpeg artifacts (particularly the mosquito noise) makes the image "look" sharper than it actually is. The djxl decoder makes the Pyron image have less mosquito noise, most notably around the stars in the bg. Psychologically it looks less sharp, but when compared to the original png, the details are kept. Compared to mozjpeg, the main details on Pyron's chest remain unchanged.

I agree with what @jonsneyers said, "Getting pixels that are closer to the original seems to me to be more valuable than getting pixels that are closer to the decode result of libjpeg-turbo." So technically, mozjpeg/libjpeg has the subpar decoding while djxl is more correct.
If this tweak leads to more accurate results relative to the original, then I'm all for it. If it's only for the purpose of being closer to mozjpeg/libjpeg then it's a bad idea.
In terms of sharpness, I do have an issue with the djpegli tool. According to my testing, the entire image has like a 1.1px gaussian blur when compared to the original. The diagonal lines on Pyron's chest is noticeably blurry when compared to the original/djxl as shown below.

@Galaxy4594 Now that jpegli decoder output, that's weird.
I don't perceive a blur though, just some sort of color shift not really bleeding into neighbors.
The difference is marginal again, but it would be cool if these "XL-era" JPEG decoders could agree on math or quantization handling that avoids perceptible differences, especially if they're intended to be "highly accurate". (Jpegli does not say it is, but it would be nice if both decoders were consistent)

Just a quick follow up. To test your theory, I converted the jpg to grayscale using jpegtran and did same testing.
It's not a color shift, it looks like blurring. It's not the kind of gaussian blurring that leads to "bleeding into neighbors" and more of a loss of local contrast. It's almost like a negative unsharp mask.


Issue-1110: fixing "Encoding error when contract generating for reque…

fixing https://github.com/DiUS/pact-jvm/issues/1110

Looks like ByteArrayOutputStream.toString(Charset) is not available in JDK 8 and 9. That is why the Travis build has failed.
We don't need to go via a String, we could just put the byte array straight into the OptionalBody.

I'm closing this PR as issue #1110 has been addressed with commits f67ecbbee3449e930a6dcc9d9c262d286fe94206 and dde272b05d419f190de3d2c170425eee79b97ed9


add --version to quickget

Enhancement Request:
Add --version command line option to display the version number of quickget.
This would be useful for those of us running the git repo version and not the release package.

You can run quickemu --version as a workaround for now, but I agree - it would be a nice addition :)


Move mod statements before use statements

As a preference, I like to have all my mod statements at the top of the file, and use statements after them. As I understand, this has no impact on the code but it makes it read better when a use statement refers to a mod statement.
For example:
Preferred
mod a;
mod b;

use a::MyA;
pub use b::MyB;

It would be great if there was some way for rustfmt to help ensure this. For example, when using the builtin tools in rust-analyzer it often puts the use statements right at the top of the file:
use a::MyA;
pub use b::MyB;

mod a;
mod b;

Which works but just doesn't read as well in my opinion.
Is this already possible? Or, if not, could it be considered as a feature request? :)
Cheers

Thanks for reaching out. Currently there isn't an option to enforce this, but I'm going to close this as it's a duplicate of #3959


JAVA-2055: Fix test failure in NodeMetadataIT

For JAVA-2055.
Change is pretty straightforward.  I suspect this was failing because the LBP hadn't been notified by the time cleanup happened (test only waits for host state to be UP, which will happen before LBP is engaged).   Since the test doesn't create schema, it doesn't really need a dedicated SessionRule.  Instead, use SessionUtils.newSession to create a standalone session for this specific test.

There are some other intermittent test failures..i'll see if there's any obvious ones I can fix here as well.

Using this ticket as an excuse to fix a number of integration tests, labeling wip while i fix things up.

Have worked out a lot of little test issues here, I set up a schedule to rerun CI every 20 minutes, will see how this goes in the morning..see if there are any other tests to fix up, and will keep at it until things run reliably.

Merging this today as there are a bunch of test fixes which make our builds pass much more reliably.
There are still two test issues that pop up now and then:

com.datastax.oss.driver.internal.core.pool.ChannelPoolResizeTest unit tests rarely fail, looks timing oriented, but looking at the test it looks like that is accounted for...needs more research. JAVA-2059

com.datastax.oss.driver.api.core.metadata.NodeStateIT.should_mark_control_node_down_when_control_connection_is_last_connection_and_dies waiting for an up event.  Needs more investigation, but is very infrequent. JAVA-2060

And of course there are even rarer occasions where for whatever reason CCM has troubles launching a cluster.  I'd like to see if we can rerun tests in these cases as it's not he drivers' fault.
I'll continue to investigate these when time permits.


Added optional shortcuts

Hi.
Added optional shortcuts.
--local is too long, so I wanted to start local mode with -l.

Sure, why not :) Could you add a changeset as well? As described in the comment above this. We've started integrating it into our workflow, and want to see how people adjust to it.

I added Changeset. Is it correct?

Looks good, thank you!


Fix logger for LockedPulpRepository

Some of the log messages were using logging, instead of logger. This
doesn't cause a python error because logging is actually valid. But
there are no handlers attached to it so the messages go nowhere. By
using logger, the code uses the module level logger which contains the
correct handlers.
Signed-off-by: Luiz Carvalho<EMAIL_ADDRESS>

Not updating release notes draft because this is bugfix to an unreleased feature.


[super_drag_and_drop] Can't read files from Microsoft Outlook

I am trying to drag and drop A file from Microsoft Outlook into my Flutter app. But I a receiving a strange error message.
Example Code:
extension on DataReader {
  dynamic getAnyFile(
    AsyncValueChanged<DataReaderFile> onFile, {
    ValueChanged<Object>? onError,
  }) {
    final formats = platformFormats;
    if (formats.isNotEmpty) {
      final format = formats.first;

      return getFile(
        SimpleFileFormat(fallbackFormats: [format]),
        onFile,
        onError: onError,
      );
    } else {
      return null;
    }
  }
}

Widget build(BuildContext context) {
    return DropRegion(
      formats: const [...Formats.standardFormats],
      onDropEnter: (event) {
        //...
      },
      onDropLeave: (event) {
        //...
      },
      onDropOver: (event) {
        return DropOperation.copy;
      },
      onPerformDrop: (event) async {
        final item = event.session.items.first;

        // data reader is available now
        final reader = item.dataReader!;
        if (reader.canProvide(Formats.fileUri)) {
          reader.getValue<Uri>(
            Formats.fileUri,
            (value) async {
              // ...
            },
            onError: (error) {
              debugPrint('Error reading value $error');
            },
          );
          return;
        }

        reader.getAnyFile(
          (value) async {
            if (value is DataReaderVirtualFileAdapter) {
              if (value.file is VirtualFileFromFile) {
                final vFile = value.file as VirtualFileFromFile;

                //...
              } else if (kIsWeb) {
                final data = await value.file.readNext();
                //...
              }
            }
          },
          onError: (error) {
            debugPrint("error $error");
          },
        );
        return;
      },
      child: Container(
          //child: ...,
          ),
    );
  }

Error Message that I get:
 ERROR [] Unexpected error Ungültige FORMATETC-Struktur (0x80040064) at src\.\win32\drop.rs:487:22
error PlatformException(super_native_extensions_error, "Windows Error: Eine Schnittstelle, die für einen anderen Thread marshalled war, wurde von der Anwendung aufgerufen. (0x8001010E)", otherError, null)
Translation:
 ERROR [] Unexpected error Invalid FORMATETC structure (0x80040064) at src\.\win32\drop.rs:487:22
error PlatformException(super_native_extensions_error, "Windows Error: An interface marshaled for a different thread was called by the application. (0x8001010E)", otherError, null)
It works fine if Drag and Dropped from Explorer and on MacOs it works fine also for outlook.
Please let me know if I can Provide further information

Just in case, which super_drag_and_drop version is this with?

super_drag_and_drop: ^0.5.0

Well, this is very disappointing. Turns out that outlook doesn't support IDataObjectAsyncCapability for attachments. I'd expect outlook being a flag ship product having spot less drag & drop implementation but it can't even download attachments in background :-/. Or, for that matter, set proper drag avatars. Quite sad.
Nonetheless, super_drag_and_drop needs to handle this properly.

I am still getting the same issue. Do I have to change anything else? I am on Windows 10 If it helps.
I am installing directly from GitHub
super_drag_and_drop:
    git:
      url: https://github.com/superlistapp/super_native_extensions.git
      path: super_drag_and_drop

What revision are you at (check pubspec.lock)?

It looks like this after I changed to git version:

I think you need to add dependency_override for super_native_extensions nad also set it to git (current revision). Otherwise it will use the old (currently published) version of super_native_extensions.

I'm hoping to publish and updated version soon though.

I updated to 0.6.4 and still get this error:
2023-08-24T15:26:49.813Z ERROR [] Unexpected error Ungültige FORMATETC-Struktur (0x80040064) at src.\win32\drop.rs:488:22

That error is normal for outlook. It is only concerning the drop preview, which is not working for outlook in any application.
This is not related in any way to actually reading the dropped data. It's the other error that you initially got:
error PlatformException(super_native_extensions_error, "Windows Error: An interface marshaled for a different thread was called by the application. (0x8001010E)", otherError, null)

That was relevant to data receiving. Are you unable to read the stream?

Thanks for the info.  I will check my code and let you know

I could fix it now had to change my code for reading Virtual Files :)
Thanks for the great help


Added 0891_Sum_of_Subsequence_Widths

IMPORTANT: Please do not create a Pull Request without an issue.
All Submissions:

[x] I have read the CONTRIBUTING document.
[x] My code is written in Python3 and is ending with .py
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[x] I have checked that my submission does pass the test on LeetCode.com
[x] Does your filename follow the naming Conventions?
[x] Have you linked your PR to an Issue?

Closing issues
closes #639
I hope you pull through the COVID19 without any damage your health.

Waiting for the review @vJechsmayr :)


Update cats-effect, cats-effect-laws to 3.3.1

Updates

org.typelevel:cats-effect
org.typelevel:cats-effect-laws

from 2.5.4 to 3.3.1.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Applied Migrations

github:typelevel/cats-effect/v3_0_0?sha=v3.0.0
github:typelevel/cats-effect/v3_3_0?sha=series/3.x

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.typelevel" } ]

labels: library-update, early-semver-major, semver-spec-major, scalafix-migrations

Superseded by #199.


[Search] Implement Preview Features for API Version 2024-03-01-Preview

This PR includes implementation and code regeneration based on the 2024-03-01-Preview preview swagger specification found at https://github.com/Azure/azure-rest-api-specs/commit/a0151afd7cd14913fc86cb793bde49c71122eb1e.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Search.Documents


additionalProperties should be false for most pod API objects

Use case: Alice creates a pod via JSON. Her JSON is flawed in that it defines a field that's not defined for the Marathon pod API. Marathon accepts the JSON as-is and run the pod. Alice queries Marathon for the pod JSON and it differs from her original. She's confused.
Alternates:
She defines a field XYZ that's required in the wrong place. Marathon reports an error that field XYZ is undefined. She's confused.
I'm strongly questioning the conclusion that we came to, when developing the pod API, that additionalProperties would be unset (true by default). This causes confusion when Marathon blindly accepts JSON with fields that don't actually map to the Marathon API.
I'm flagging this as a blocker for 1.4 until we discuss further.

For example, Marathon accepts this pod JSON (I curl'd it directly to the server):
{
  "boo": "hoo",
  "id": "/nginx",
  "containers": [
    {
      "name": "container-1",
      "resources": { "cpus": 1, "mem": 128 },
      "endpoints": [
        {
          "name": "http",
          "containerPort": 80,
          "hostPort": 0,
          "protocol": [ "tcp" ],
          "labels": { "VIP_0": "<IP_ADDRESS>:80" }
        }
      ],
      "image": { "id": "nginx", "kind": "DOCKER" }
    }
  ],
  "scaling": { "kind": "fixed", "instances": 1 },
  "networks": [ { "name": "dcos", "mode": "container" } ]
}

quoting jason, from slack: "we probably should reject additional fields"

there's probably a notable doc change that should go along with this as well: it should be clear to users up front that it is an error to specify JSON properties that aren't declared by the API

That is correct. The same problem applies to AppDefinitions.
We already scoped this for 1.5.

Any reason that we can't enforce this for pods in 1.4? Since we're
introducing pods in 1.4 it would be nice to get right before we actually
release it
On Wed, Feb 8, 2017 at 5:44 AM, Matthias Veit<EMAIL_ADDRESS>wrote:

That is correct. The same problem applies to AppDefinitions.
We already scoped this for 1.5.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mesosphere/marathon/issues/5127#issuecomment-278293721,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACPVLCH9Zvl5KN4_aXTqdI2GtCE_6W9Hks5raZyqgaJpZM4L58Fk
.

/cc @kensipe @artemharutyunyan
this actually just bit US again w/ respect to writing system integration tests. marathon accept invalid pod definitions, so test authors think there's a bug in the system when some accidentally misplaced field is accepted but the system doesn't respond in the way that they're expecting.
there's a distinct difference between the app spec and the pod spec: the app schema specifically advertises additionalProperties: false for many fields. the pod spec advertises that for NONE.
it's a valid point that Marathon doesn't enforce additionalProperties: false for apps. however, that doesn't provide justification for us to incorrectly specify pods via the new RAML API. publishing the pods API now without additionalProperties: false and then adding it later is technically a breaking API change.
PLEASE PLEASE let's address this now before we launch 1.4

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-1764. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.


add generation Number

Couple of more things:

setAllCached api also needs to be made generationNumber aware because it is possible that it is called after caching data for old gen data file which will cause corruption as currently you are updating the cache status.
We need a test that runs CachedReads in parallel to invalidations being forced


Terraform 0.12

Hi!
Here's a TF 0.12 compliant version of this module :)

In order to the pre-commit test to pass, an upgrade of lsstsqre/terraform-pre-commit image with TF 0.12 is needed.
https://github.com/lsst-sqre/docker-terraform-pre-commit/blob/master/Dockerfile

I've added an image with tf 0.12.6.  The tag is lsstsqre/terraform-pre-commit:tf-0.12.6
https://github.com/lsst-sqre/docker-terraform-pre-commit/pull/2

@seboudry It looks like recent releases of pre-commit-terraform + tf 0.12.6 will still accept the 0.11 syntax without forcing a conversion?

FYI -- I'm low bandwidth today as I'm at a conference.  What do you think of tf-0.12/0.10.x as a release branch name?

I've added an image with tf 0.12.6. The tag is lsstsqre/terraform-pre-commit:tf-0.12.6
lsst-sqre/docker-terraform-pre-commit#2

Nice, the test passes with this new image. I'll push a new commit.

@seboudry It looks like recent releases of pre-commit-terraform + tf 0.12.6 will still accept the 0.11 syntax without forcing a conversion?

I've tested lsstsqre/terraform-pre-commit:tf-0.12.6 on current 0.10.x branch:

pre-commit run -a OK after auto modify main.tf by aligning closing brace comments (I'll remove them as they don't add value)

FYI -- I'm low bandwidth today as I'm at a conference. What do you think of tf-0.12/0.10.x as a release branch name?

Why not have a 0.12.x branch name? Did you follow TF versions on your versioning?

Hi @jhoblitt!
Sorry to bother you, any update on this PR?
Have a nice day

Hi @jhoblitt !
Any change for that PR to be merged? ;)

@jhoblitt @afausti @athornton @digitizdat @rra is there any chance this will pass review and get merged? It would be great. kthxbye

Yes, we can merge this but it can't go into the master branch as it will break tf 0.11. Note that this mod isn't necessary with helm v3 / terraform-provider-helm >= 1.0.0.

Merged into a branch named tf-0.12/0.10.x.


Don't create a gcs storage client on every interaction with blobs

The following code creates a client per function call at https://github.com/langchain-ai/langchain-google/blob/main/libs/vertexai/langchain_google_vertexai/_image_utils.py
 def _blob_from_gcs(self, gcs_uri: str) -> storage.Blob:
        """Gets image Blob from a Google Cloud Storage uri.

        Args:
            gcs_uri: Valid gcs uri.

        Raises:
            ValueError if there are more than one blob matching the uri.

        Returns:
            storage.Blob
        """

        gcs_client = storage.Client(project=self._project)
        blob = storage.Blob.from_string(gcs_uri, gcs_client)
        blob.reload(client=gcs_client)
        return blob

It would make more sense to make client as a property of ImageBytesLoader

good point, would you be open to send a PR and refactor this piece, please?

hi @lkuligin , I opened a pr https://github.com/langchain-ai/langchain-google/pull/609 but i have failing tests. I don't understand how they are connected to my code. May you please take a look there and explain me ?


Oven integration

Thanks for creating this project! I was in the middle of creating a similar one when I saw yours. I have a Whirlpool smart oven that uses the same auth flow/APIs. Would you accept a pull request that adds smart oven support?

Yes, I would appreciate it!
Most of the API is the same for other appliances, so it makes sense to have them in this same lib.

Hi, I just wrote few lines to publish my whirlpool smart oven status over a mqtt server. The code is very ugly but  appears to work. I can share if it can be helpful

Hey @eskibars
I just added initial support for Washer/Dryer appliances, which may help you add the oven. Take a look at the AppliancesManager class on how I did to identify if an appliance is a aircon or washer/dryer. It may not be so straightforward for ovens though.

Thanks!  I'll have a look in a bit.  Haven't had a chance to tackle this yet, but I intend to do some work in the coming 1-2 weeks and hope to open a PR then


[Suggestion] Some command argument shouldn't suggest autocomplete

This is just a nitpick and not that important but command like scoreboard objectives add ... doesn't really make much sense to get the objective name to autocomplete.

The user might want to select an existing objective (e.g. bb.pl.enum) and then change the last part of the name to something else, so this works as intended


fix: storage.json path changes in new version of VScode

Description

VScode changed the storage.json path.
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

When can this be merged? : )

Now @telmen 🔥 😄

Hi @cy-98 👋
Thanks for the update; it works as intended! 🔥
If you want to be added as a contributor, send me your raycast username here, and I'll add it for you 😄

Thanks !
username: cy-98
nickname: 才越


Fix typo in simple.md

The following sentence:

Moved shared content into one or more partial views...

Should read:

Move shared content into one or more partial views...

Thanks @guardrex for following up! :rocket:

Thanks @guardrex for following up! :rocket:


Can't restore with forge paket restore

After creating a new project with forge, checking it out to git and then
checking it out again in a different location (same computer, so same mono version) I'm unable to run forge paket restore
Expected Behavior
Current Behavior
oivvio@blomma  forge paket restore
mono  "/tmp/minisuave/.paket/paket.bootstrapper.exe"
Cannot open assembly '/tmp/minisuave/.paket/paket.bootstrapper.exe': No such file or directory.
Error while running '/tmp/minisuave/.paket/paket.bootstrapper.exe' with args:
mono  "/tmp/minisuave/.paket/paket.exe" restore
Cannot open assembly '/tmp/minisuave/.paket/paket.exe': No such file or directory.
Error while running '/tmp/minisuave/.paket/paket.exe' with args: restore

Possible Solution
I guess there's a step I need to run before running forge paket restore to install paket in the project.
I tried forge update-version paket but that gives me
mono  "/tmp/minisuave/.paket/paket.bootstrapper.exe"
Cannot open assembly '/tmp/minisuave/.paket/paket.bootstrapper.exe': No such file or directory.
Error while running '/tmp/minisuave/.paket/paket.bootstrapper.exe' with args:

Your Environment

Version used:

Forge 1.4.1

Environment name and version (F#, .Net):

Mono JIT compiler version 5.0.0

Operating System and version:

Ubuntu 16.04

Turns out .paket/paket.exe is in .gitignore but it's needed for forge paket restore
So an easy work around for the problem I was having is to simply remove it from .gitignore

The problem is not .paket/paket.exe but .paket/paket.bootstrapper.exe which used to be in .gitignore. This was fixed some time ago, refreshing templates should fix problem for the future new projects. For existing project, just download bootstrapper from Paket webpage, and make sure it's committed to repo.


[getting-started] Trim the app.yaml down

Across languages, we're trying to trim down the number of flags we include in the default app.yaml.  In the case of Managed VMs, we should really only have:
runtime: go
vm: true
Specifically we want to avoid including resources and auto scaling, unless we're specifically showing a sample that works with those settings:
[START resources]
resources:
cpu: .5
memory_gb: 1.3
disk_size_gb: 10
[END resources]
[START scaling]
automatic_scaling:
min_num_instances: 1
max_num_instances: 5
cool_down_period_sec: 60
cpu_utilization:
target_utilization: 0.5
[END scaling]

@fbalejko here you go

Thanks!


❌  Incorrect number of secrets in Development & Pre-production

The number of secrets in the GitHub environments is not correct. This should be rectified

Confirm this is in working order by deploying both Development and Pre-production

The following secrets are now present in Development and Pre-Production:
ASSUME_ROLE_DEVELOPMENT
ASSUME_ROLE_PRE_PRODUCTION
BYOIP_POOL_ID
CORSHAM_MGMT_RANGE
FARNBOROUGH_MGMT_RANGE
NETWORK_SERVICES_CIDR_BLOCK
TRANSIT_GATEWAY_ID
TRANSIT_GATEWAY_ROUTE_TABLE_ID

This is, in short, because they are not needed. The non-production environment are not going to be attached the the transit gateway therefore all but the first two above are not required.
The two ASSUME_ROLE_XXX secrets would be required  to allow Cloudwatch exporter to pull metrics from their respective environments, we have decided that this is out of scope when running non-production environments.


refactor: use generics

LGTM 👍

@Runrioter @simonetripodi
What benefit does this change have for MyBatis users? I know If applied this change in existing implementation class, compile warning(unchecked warning) occur. I thinks have not benefit for user by only this change. Also at least, I think should wait to apply this change until next minor or major version up(Probably, we should be consider to modify the Plugin#wrap method together too).
WDYT?

@kazuki43zoo

What benefit does this change have for MyBatis users?

No. Just refactor this for coding easily without unnecessary casting

compile warning(unchecked warning) occur

Can you paste compile warning here ?

Can you paste compile warning here ?

The above source code (inner class that use in MyBatis's test case)

  @Intercepts({
      @Signature(type = Map.class, method = "get", args = {Object.class})})
  public static class AlwaysMapPlugin implements Interceptor {
    @Override
    public Object intercept(Invocation invocation) {
      return "Always";
    }

    @Override
    public Object plugin(Object target) {
      return Plugin.wrap(target, this);
    }

    @Override
    public void setProperties(Properties properties) {
    }
  }

For prevent above waring, developer need to change existing source code as follow:

Add @SuppressWaring("unchecked") on method level

    @Override
    @SuppressWarnings("unchecked")
    public Object plugin(Object target) {
      return Plugin.wrap(target, this);
    }

Change method signature and adding cast that return value of Plugin#wrap (And need to add the @SuppressWaring("unchecked") for unchecked cast...)

    @Override
    @SuppressWarnings("unchecked")
    public <T> T plugin(T target) {
      return (T) Plugin.wrap(target, this);
    }

Libraries code will polishing but the user's point of view, it seems that there is no benefit to applying this change (in my opinion).

Thanks for your clarify. This is a little break when we override this method. I will modify Plugin#wrap too.