Question about training and evaluation of RL System Policy

Hi, I have found that RL policy could be trained and evaluated with pipelines with different components occupied.
Here are several configurations I found,

Basic setting.  UserAct -> SystemRuleDST -> SystemRLPolicy -> SystemAct -> UserRLPolicy -> UserAct.
+SystemNLU.  UserUtterance  -> SystemNLU/Joint DST -> SystemRLPolicy -> SystemAct -> UserRLPolicy -> UserUtterance
+UserNLU. UserAct  -> SystemRuleDST -> SystemRLPolicy -> SystemNLG-> SystemUtterance -> UserNLU -> UserRLPolicy -> UserAct
Full setting. UserUtterance  -> SystemNLU/Joint DST -> SystemRLPolicy -> SystemNLG -> SystemUtterance -> UserNLU-> UserRLPolicy-> UserNLG -> UserUtterance.

My questions are:

For system policy training: Should I train the policy with basic setting or full setting? I think if the policy is trained with basic setting, it may not be able to adapt well in full setting for errors from other models (like DST or NLG). However, if the policy is trained with full setting, the training process may converge slowly because other models introduce too much noises, which are difficult for training. Moreover, the trained policy in full setting can only be used with the same models toghether when integrated in the pipeline. In ConvLab3 RL training tutorial, the policy is trained with basic setting.
For system policy evaluation: Should I train the policy with basic setting or full setting? I think if the policy is evaluated with basic setting, it is a component-level policy evaluation, and the evaluation results could not reflect its performance in a full pipeline. If the policy is evaluated with full setting, it is a system-level evaluation and the results are also affected by other components like NLU and NLG.
Do the setting +SystemNLU and +UserNLU meaningful for some training or evaluation cases?

If I have any mistakes, please correct me.
Looking forward to your reply, thanks!

Hi James,
thanks for your comments, you observed everything very well! Let me comment on your questions:

You correctly identified advantages and disadvantages of basic vs full setting. In my research for dialogue policy optimization using RL, I always used the basic setting. As you observed, the basic setting is a "clean" setup without any noise stemming from NLU predictions. This lets you focus on the pure RL algorithm (but without checking the robustness against noise). It also lets you train policies "from scratch", so not necessarily pre-trained on data, where performance differences are easier to observe. Moreover, first training with the basic setting can be a great sanity check to make sure that the RL optimization works properly, before you move towards full training where problems can also stem from the other components.
As you said, the full setting has the advantage that you already take the other components into account during RL training. This is preferred if your goal is to have a high performance when interacting with humans for instance. The NLG for instance is typically trained using supervised learning on the dataset. If you train the policy in the basic setting, it will likely learn to take semantic actions that the NLG has not seen in the data, which leads to a distribution shift later on for the NLG and consequently worse performance. Due to this, training both together should lead to better performance with the downside of slower training progress.
I fully agree with you on that. It is not necessarily given that component wise superiority will translate to superiority for the whole system. For instance, dialogue state trackers are often stronger than belief state trackers (which take uncertainty in the predictions into account) in state tracking performance. Nevertheless, belief state trackers can provide more information to the policy through the uncertainty, potentially boosting overall performance of the system.
Let's say you have a user policy that was trained with semantic system actions as input (such as TUS or GenTUS). If you now assume that humans have an almost perfect NLU performance (at least much better than the NLU models), the setting  +SystemNLU makes a lot of sense because adding a UserNLU in addition would just introduce additional noise.
Regarding +UserNLU, the only use case I can come up with right now is testing the noise robustness of a user simulator :)

I hope my answers were helpful! Let me know if you have further questions.
Best
Chris

Hi @ChrisGeishauser,
Thank you so much for your very very helpful and  fast reply! This helps me a lot!!!
I have a follow-up question for the point 3.
As you said,

If you now assume that humans have an almost perfect NLU performance (at least much better than the NLU models), the setting +SystemNLU makes a lot of sense because adding a UserNLU in addition would just introduce additional noise.

So if we want to remove the noises introduced by UserNLU,  does +SystemNLU is a better evaluation setting than full setting? Because we do not want the errors caused by UserNLU make the dialogue fail.

Hi. @ChrisGeishauser
Your reply perfectly answered my questions. I will try    UserPolicy with semantic act later.
Thanks a lot!!
Best regards,
James Cao


You don't seem to have a generator with the name react-webpack installed

After updating npm, yeoman, and node to 5.0.0 and running npm install -g generator-react-webpack without errors, when running "yo react-webpack", I am getting: You don't seem to have a generator with the name react-webpack installed. Any ideas?

@nategeier what's solution you found?


map': undefined method name'

He,
I tried to fix it myself but i'm not very proficient in Ruby
When i run ruby app/cryptopunks_gui.rb i get he following error:
 /Users/jefsev/Local Sites/NFT-generator/CryptoPunkGUI/cryptopunks/app/model/image.rb:4:inmap': undefined method name' for :BLUE_STOPS:Symbol (NoMethodError)
Let me know if i am doing something wrong and how i could fix this.
Thx!

Thank you for reporting this. I will look into it right away. Would you mind sharing your platform too? (e.g. Mac Catalina or Windows 10)

OK, I tested on the Mac by installing the gem and by starting with a cloned project from scratch, and both modes worked.
I will look into testing on Linux next (in case you are running on Linux).

I just tested on Ubuntu Linux 21 and had no issues either.
I will need more info from you regarding my questions above to help you further.

OK, I found the issue by running an older version of Ruby (2.7.2).
2.7.2 :001 > :hello.name
Traceback (most recent call last):
        4: from /Users/andymaleh/.rvm/rubies/ruby-2.7.2/bin/irb:23:in `<main>'
        3: from /Users/andymaleh/.rvm/rubies/ruby-2.7.2/bin/irb:23:in `load'
        2: from /Users/andymaleh/.rvm/rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
        1: from (irb):1
NoMethodError (undefined method `name' for :hello:Symbol)

The project clearly states it requires Ruby 3.0.2 to run. Please upgrade your Ruby installation to Ruby 3.0.2
The easiest way to do so is through RVM: https://rvm.io/

Good news! I just fixed compatibility with older Ruby versions down to 2.7.2 (and perhaps lower too).
I released the fix in version 0.0.13
Please give it a try and let me know if it works.
Cheers.

@AndyObtiva Thx for the quick reply! I was running 2.6.x which i updated to 3.0.2 which solved this issue.
But now when i run i get the following error:
jefsev@Jeffrey cryptopunks-gui % sudo ruby app/cryptopunks_gui.rb pixelart/1.2.1 on Ruby 3.0.2 (2021-07-07) [x86_64-darwin19] in (/Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/pixelart-1.2.1) cryptopunks/2.0.1 on Ruby 3.0.2 (2021-07-07) [x86_64-darwin19] in (/Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/cryptopunks-2.0.1)      >ac39af4793119ee46bbff351d8cb6b5f23da60222126add4268e261199a2921b< SHA256 hash matching          ✓ True Official Genuine CryptoPunks™ verified      2400x2400 (height x width) /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk.rb:1981:in _invoke': couldn't recognize data in image file "/Users/jefsev/Local Sites/NFT-generator/CryptoPunkGUI/cryptopunkGui/cryptopunks-gui/icons/cryptopunks-gui.png" (RuntimeError)
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk.rb:1981:in _ip_invoke_core'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk.rb:2016:in _tk_call_core'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk.rb:2043:in tk_call_without_enc'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:81:in initialize'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:192:in initialize'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:59:in block (2 levels) in new'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:58:in synchronize'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:58:in block in new'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:57:in instance_eval'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk/image.rb:57:in new'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/tk/widget_proxy.rb:449:in image_argument'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/tk/toplevel_proxy.rb:56:in set_attribute'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/dsl/tk/attribute_expression.rb:37:in interpret'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:181:in block in interpret_expression'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/expression.rb:64:in around'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:180:in interpret_expression'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:175:in interpret'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer.rb:78:in method_missing'
from app/cryptopunks_gui.rb:35:in block in create_gui'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/parent_expression.rb:32:in add_content'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/dsl/tk/root_expression.rb:45:in add_content'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:199:in add_content'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:182:in block (2 levels) in interpret_expression'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/facets-3.1.0/lib/core/facets/kernel/tap.rb:19:in tap'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:181:in block in interpret_expression'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/expression.rb:64:in around'
from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:180:in interpret_expression'         from /Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-2.5.1/lib/glimmer/dsl/engine.rb:65:in block (2 levels) in class:Engine'
from app/cryptopunks_gui.rb:33:in create_gui'         from app/cryptopunks_gui.rb:24:in initialize'
from app/cryptopunks_gui.rb:44:in new'         from app/cryptopunks_gui.rb:44:in '`
Looking forward to using the GUI.
If i can buy you a coffe im happy to donate some.

Looks like something with the tk library:
/Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/tk-0.4.0/lib/tk.rb:1981:in _invoke': couldn't recognize data in image file "/Users/jefsev/Local Sites/NFT-generator/CryptoPunkGUI/cryptopunkGui/cryptopunks-gui/icons/cryptopunks-gui.png" (RuntimeError)

The error is surprising. I don't get it.
You didn't tell me what platform you are running on yet. I assume you are on a Mac. If so, which Mac version? Catalina, Big Sur, Monterey, or another?
It looks like the Tk library is unable to load the image: /Users/jefsev/Local Sites/NFT-generator/CryptoPunkGUI/cryptopunkGui/cryptopunks-gui/icons/cryptopunks-gui.png
See if you can open the file yourself successfully just to make sure it is there. It has the logo of CryptoPunks GUI:

If the image works fine, then perhaps there is an issue in the code that is interacting with Tk.
In that case, open this file:
/Users/jefsev/.rvm/rubies/ruby-3.0.2/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/tk/widget_proxy.rb
Go to line 449, specifically:
the_image = ::TkPhotoImage.new(image_args)

Add require 'pd'; pd image_args; pd  in front of it to have it print out image information:
require 'pd'; pd image_args; pd the_image = ::TkPhotoImage.new(image_args)

Save the file, and re-run cryptopunks-gui
You should see output like this:
[PD] /Ruby30-x64/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/tk/widget_proxy.rb:449
   > require 'pd'; pd image_args; pd the_image = ::TkPhotoImage.new(image_args)
  => {
    :file => "C:/Users/a_mal/code/cryptopunks-gui/icons/cryptopunks-gui.png"
}
[PD] /Ruby30-x64/lib/ruby/gems/3.0.0/gems/glimmer-dsl-tk-0.0.47/lib/glimmer/tk/widget_proxy.rb:449
   > require 'pd'; pd image_args; pd the_image = ::TkPhotoImage.new(image_args)
  => #<TkPhotoImage:0x000002a85fc82e90 @type="photo", @path="i00000">

He, thx! im going to try this today. Ill let you know what happens.
Running mac Cataline 10.15.7

Similar Issue
Mac Cataline 10.15.7
ActiveTcl-8.5
pixelart/1.2.1 on Ruby 3.0.2 (2021-07-07) [x86_64-darwin19] in (/Users/wangtao/.rvm/gems/ruby-3.0.2/gems/pixelart-1.2.1)
cryptopunks/2.0.1 on Ruby 3.0.2 (2021-07-07) [x86_64-darwin19] in (/Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-2.0.1)
     >ac39af4793119ee46bbff351d8cb6b5f23da60222126add4268e261199a2921b< SHA256 hash matching
         ✓ True Official Genuine CryptoPunks™ verified
     2400x2400 (height x width)
/Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk.rb:1981:in `_invoke': couldn't recognize data in image file "/Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/icons/cryptopunks-gui.png" (RuntimeError)
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk.rb:1981:in `_ip_invoke_core'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk.rb:2016:in `_tk_call_core'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk.rb:2043:in `tk_call_without_enc'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:81:in `initialize'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:192:in `initialize'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:59:in `block (2 levels) in new'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:58:in `synchronize'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:58:in `block in new'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:57:in `instance_eval'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/tk-0.4.0/lib/tk/image.rb:57:in `new'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-dsl-tk-0.0.49/lib/glimmer/tk/widget_proxy.rb:449:in `image_argument'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-dsl-tk-0.0.49/lib/glimmer/tk/toplevel_proxy.rb:56:in `set_attribute'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-dsl-tk-0.0.49/lib/glimmer/dsl/tk/attribute_expression.rb:37:in `interpret'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:181:in `block in interpret_expression'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/expression.rb:64:in `around'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:180:in `interpret_expression'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:175:in `interpret'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer.rb:78:in `method_missing'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/app/cryptopunks_gui.rb:35:in `block in create_gui'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/parent_expression.rb:32:in `add_content'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-dsl-tk-0.0.49/lib/glimmer/dsl/tk/root_expression.rb:45:in `add_content'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:199:in `add_content'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:182:in `block (2 levels) in interpret_expression'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/facets-3.1.0/lib/core/facets/kernel/tap.rb:19:in `tap'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:181:in `block in interpret_expression'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/expression.rb:64:in `around'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:180:in `interpret_expression'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/glimmer-2.5.4/lib/glimmer/dsl/engine.rb:65:in `block (2 levels) in <class:Engine>'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/app/cryptopunks_gui.rb:33:in `create_gui'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/app/cryptopunks_gui.rb:24:in `initialize'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/app/cryptopunks_gui.rb:44:in `new'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/app/cryptopunks_gui.rb:44:in `<top (required)>'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/bin/cryptopunks-gui:3:in `require_relative'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/gems/cryptopunks-gui-0.0.15/bin/cryptopunks-gui:3:in `<top (required)>'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/bin/cryptopunks-gui:23:in `load'
	from /Users/wangtao/.rvm/gems/ruby-3.0.2/bin/cryptopunks-gui:23:in `<main>'

OK, I just updated the README with a specific version for ActiveTcl. It is 8.6. For example, I am running ActiveTcl version 8.6.9 successfully.
I also added more detailed prerequisite install instructions:
Mac:

Install the ActiveTcl 8.6 Mac package from ActiveState.com
Install RVM by running \curl -sSL https://get.rvm.io | bash -s stable (and run curl -sSL https://rvm.io/pkuczynski.asc | gpg --import - if needed for mentioned security reasons)
Run: rvm install 3.0.2 --enable-shared --enable-pthread --with-tk --with-tcl
Run: gem install tk -v0.4.0
Run: gem install glimmer-dsl-tk

Windows:

Install Ruby+Devkit (including MSYS/MINGW toolchains) through Windows Installer: https://rubyinstaller.org/downloads/
Install the ActiveTcl 8.6 Windows package from ActiveState.com
Follow this instruction: "First, Ruby needs to find the tcl86t.dll and tk86t.dll shared libraries. These are located in C:\ActiveTcl\bin. Make a copy of them somewhere Ruby can find them, e.g. C:\Ruby26\bin."
Setup environment variables TCL_LIBRARY=C:\ActiveTcl\lib\tcl8.6 & TK_LIBRARY=C:\ActiveTcl\lib\tk8.6
Run: gem install tk -v0.4.0
Run: gem install glimmer-dsl-tk

Linux:

Download the ActiveTcl 8.6 Linux package from ActiveState.com
Extract the tar gz file using command tar zxvf ActiveTcl-version-number.tar.gz
Run included install shell script ./ActiveTcl-version-number/install.sh
Install RVM by running \curl -sSL https://get.rvm.io | bash -s stable (and run curl -sSL https://rvm.io/pkuczynski.asc | gpg --import - if needed for mentioned security reasons)
Run: rvm install 3.0.2 --enable-shared --enable-pthread --with-tk --with-tcl
Run: gem install tk -v0.4.0
Run: gem install glimmer-dsl-tk

Hi, is this resolved?
cryptopunks.tech


How can I implement kerebros encryption

Closing due to the sunset of the project. Please check active maintained alternative: https://pypi.org/project/hdbcli/


Allow going from UniquePtr<T> to Pin<UniquePtr<T>>

This is safe since there cannot be any other references to the interior T and because the Deref/DerefMut implementations are sane.

You're correct. I didn't see that the pin_mut function already lets me go from a UniquePtr<T> to a Pin<&mut T>. This PR is unneeded. Sorry and thanks for the quick answer.


Image Text Extraction does not allow the storage path to be further specified

Expected Behavior
Being able to choose a sub path to look for the added files.
Actual Behavior
Only the home directory is allowed, slashes are forbidden
Steps to Reproduce the Problem

Install the experimental Image Text Extraction extension
Enter a "/subdir"
Look at the error saying invalid.

Specifications

Extension name: Image Text Extraction
Version: 0.0.1

@dackers86 lets take a look into this

I believe this is now addressed in the version published by Google Cloud, you can specify include paths and exclude paths https://github.com/GoogleCloudPlatform/firebase-extensions/tree/next/storage-extract-image-text
Closing, if this is still an issue it can be reopened in the GCP repo :)


docs: update run docs

From https://github.com/iterative/dvc/pull/1400 .

Fixed by https://github.com/iterative/dvc.org/commit/fc0df399d9969c118c1a3d6cfe158d705b1c09e4

Oops. sorry :)


Running py.test from another environment no longer works

I have installed pytest-watch globally. I used to use this to run py.test from a virtual environment.
Since I've updated to 4.0.0 this no longer works.
As a workaround I now just install pytest-watch inside of the virtualenv as well, but I prefer not having to do so for each virtualenv.

Since I've updated to 4.0.0 this no longer works.

Please elaborate: what does not work anymore?

I have installed pytest-watch using
$ pip install --user pytest-watch
$ which ptw
/home/remco/.local/bin/ptw

Now if I create a virtualenv, install requirements in that virtualenv and then run ptw, the tests will fail because of import errors.
$ mktmpenv
...
(tmp-55b2a2e395dca7) $ pip install flask
Error: Could not run --collect-only to find the pytest config file. Trying again without silencing stdout...
...
ImportError: No module named 'flask'
ERROR: could not load /home/remco/someproject/tests/conftest.py

However, when downgrading pytest-watch to ~2.0.0, I can run this just fine.
$ pip install --user -U 'pytest-watch~=2.0.0'  # outside of virtualenv

Then in the virtualenv again:
(tmp-55b2a2e395dca7) $ ptw
=== test session starts ===
platform linux -- Python 3.5.1+, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/remco/someproject, inifile:
collected 1 item

tests/test_spam.py .

=== 1 passed in 0.01 seconds ===

Sorry for the late reaction.

I have the same issue. This tool looks useful, I installed it via pipx because I only want one copy.
I thought the --runner option would solve my problem, but it does not. The program appears to import code directly rather than just executing the runner arg when files change.

I'll also chip in and say that I have a virtualenv, I used poetry to install pytest and pytest-watch. pytest works, pytest-watch seems to have the wrong path when invoked directly:
>>>  ptw
Error: Could not run --collect-only to handle the pytest config file. Trying again without silencing output...
ImportError while loading conftest '/home/myself/flask_layout/tests/conftest.py'.
tests/conftest.py:3: in <module>
    from flask_layout import create_app
E   ModuleNotFoundError: No module named 'flask_layout'

>>>  python -m pytest-watch
/home/myself/.cache/pypoetry/virtualenvs/flask-layout-j2jffhM1-py3.10/bin/python: No module named pytest-watch


add styling?

Below is styling from https://participation.usa.gov.  I don't think it'll fit in very well to the markdown file in this repo, but also would recommend that this repo focus just on content.
.custom .format_text blockquote {
	color: #666;
	background: #eee;
	margin: 0 0 1.6em .8em;
	padding: 1.4em 1em .1em;
	border-left: 5px solid #7F1315;
}

ul.frames {
	font-size: 0;
}

ul.frames li {
	font-size: 14px;
	font-size: 1.4rem;
	display: inline;
}

/* --- topnav --- */
#topnav {
	position: fixed;
	top: 0;
	left: 10px;
	right: 10px;
	padding: 5px 10px 6px;
	z-index: 3;
	box-shadow: 0 0 4px #555;
	border-radius: 0 0 8px 8px;
	background: rgb(255,255,255);
	background: rgba(255,255,255,.9);
	min-width: 390px;
	z-index: 5;
}

#topnav ul {
	margin: 0;
	float: left;
}

#topnav li {
	float: left;
	margin: 0 7px;
	font-size: 12px;
}

#topnav a {
	text-decoration: none;
	color: #555;
}

Posting at https://github.com/GSA/participation-playbook/blob/master/styling.css in case folks want to comment

Open issues

@marmar44 - I'm not sure what you meant here.  Were you wanting to reopen this issue?


New package: MaxEntropyGraphs v0.3.0

Registering package: MaxEntropyGraphs
Repository: https://github.com/B4rtDC/MaxEntropyGraphs
Created by: @B4rtDC
Version: v0.3.0
Commit: 673400ba098014ec9c0ae4ec85a22ff25d8a693b
Git reference: HEAD
Description: Julia implementation of maximum entropy graph models

@JuliaRegistrator register()

Error while trying to register: Register Failed
@B4rtDC, it looks like you are not a publicly listed member/owner in the parent organization (JuliaRegistries).
If you are a member/owner, you will need to change your membership to public. See GitHub Help

Error while trying to register: Register Failed
@B4rtDC, it looks like you are not a publicly listed member/owner in the parent organization (JuliaRegistries).
If you are a member/owner, you will need to change your membership to public. See GitHub Help

@B4rtDC please see the message above.  Once you fix the automerge issues you can register again the new revision without changing the version number and this pull request will be automatically updated.
[noblock]


[POC] simple CLI  nuth and kaab

Given that Valentin is making good progress this month, we will set up a simple CLI that allows for the coregistration of two DEMs with Nuth and Kaab.
Context
The goal of this ticket is to implement coregistration from the xdem run.

[ ] add the right function to veirfy both path exists

[ ] Load the reference and secondary DEMs using the accessor:
reference_elev, to_be_aligned_elev = geoutils.rasters.load_multiple_rasters([config["inputs"]["reference_elev"], config["inputs"]["to_be_aligned_elev"])

[ ] Execute a coregistration:

from worflows import dem_coregistration

coreg_dem, coreg_method, out_stats, inlier_mask = dem_coregistration(reference_elev, to_be_aligned_elev, config["outputs"]["path"])
coreg_dem.save("aligned_dem.tif")
print(coreg.meta)
print(out_stats)
inlier_mask.save("inlier_mask.tif")

For the tests

[ ] Retrieve test data and ground truth data
[ ] Verify that the outputs matches the ground truth from here in the file tests/test_cli.py
[ ] add tests for new entry

For the documentation

[ ] Update the quick start guide in the documentation

/estimate 3d


Update Setup Github Project

This will involve setting further GitHub project attributes see Project Settings" page .
This will involve additional tasks such as

Setup Discussions,
Setup Issue Templates,
Pull Request details
Social Preview
-- this will involve development of a python4nb logo (i.e. python logo overlayed on top of NetBeans logo)
Github Page (see https://pages.github.com/ )

Have setup basic Discussion area with a welcome and general Feature (Idea) thread.  More to follow on this as occurs.

Have updated settings for GitHub pages which are now stored in the "docs" folder and accessible via python4nb project page in the gh-pages branch


fix: Navigate correctly to notification from Activity screen

This PR resolves ONYX-853
Slack Thread

Description
Navigate correctly to notifications from the Activity screen.
PR Checklist

[ ] I have tested my changes on iOS and Android.
[ ] I hid my changes behind a feature flag, or they don't need one.
[ ] I have included screenshots or videos, or I have not changed the UI.
[ ] I have added tests, or my changes don't require any.
[ ] I added an app state migration, or my changes do not require one.
[ ] I have documented any follow-up work that this PR will require, or it does not require any.
[ ] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

Navigate correctly to notifications from the Activity screen - ole

iOS user-facing changes

Android user-facing changes

Dev changes

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Cross-platform user-facing changes (Navigate correctly to notifications from the Activity screen - ole)

  Generated by :no_entry_sign: dangerJS against b8b027b330c35c7c78f4d056bc5a4454aaca6786


GCM: record limit counter gets reset on AAD changes

It shouldn't be.  This moves the reset to the init function instead.

[ ] documentation is added or updated
[ ] tests are added or updated

In the e_aes.c code the ctrl call to  EVP_CTRL_AEAD_TLS1_AAD set this to 0, so I think it is probably wrong in earlier OpenSSL versions also?

Just 3.x and later.

Doesnt that still mean legacy paths using engines could be broken in 3.X?

This pull request is ready to merge

Merged to master and 3.0 branches. Thank you.


DRILL-7042: Fix rpm package issues

Fix the rpm package issues
-Add apache as prefix
-Remove lib folder while packaging, because lib folder does not exist.
Signed-off-by: Naresh Bhat<EMAIL_ADDRESS>

@vdiravka Thank you very much for review.  I will do suggested changes in next version and submit.

@vdiravka same case here,  Can you help me which log folder you are referring ?
[centos@centos drill]$ find . -name log
./contrib/storage-hive/hive-exec-shade/target/classes/org/apache/hadoop/hive/llap/log
./contrib/storage-hive/hive-exec-shade/target/classes/org/apache/hadoop/hive/ql/log
./contrib/storage-hive/hive-exec-shade/target/classes/org/apache/hadoop/hive/common/log
./exec/java-exec/src/main/java/org/apache/drill/exec/store/log
./exec/java-exec/src/test/java/org/apache/drill/exec/store/log
./exec/java-exec/target/classes/org/apache/drill/exec/store/log
./exec/java-exec/target/test-classes/org/apache/drill/exec/store/log
[centos@centos drill]$
[centos@centos drill]$ ls distribution/target/apache-drill-1.15.0/apache-drill-1.15.0/
bin  conf  git.properties  jars  KEYS  LICENSE  NOTICE  README.md  sample-data  winutils
[centos@centos drill]$

@nareshgbhat thank you for your contribution.
Please, combine the changes with changes found in your other commit (https://github.com/apache/drill/pull/1643) so there is one PR per JIRA issue and close this PR.

@KazydubB Thanks for suggestion. We will have a single PR for DEB/RPM changes.  This will be tracked on (#1643)


Add support for Structured Text fields

Great work guys!
It'd be great if you could also support structured text fields! You'd only need a function that converts ST nodes into a string. We've got some documentation, but here's the gist of it:
import { isText, Node } from 'datocms-structured-text-slate-utils';

function visit(
  tree: Node | Node[],
  callback: (node: Node, index: number, parents: Node[]) => void,
) {
  const all = (nodes: Node[], parents: Node[]) =>
    nodes.forEach((node, index) => one(node, index, parents));
  const one = (node: Node, index: number, parents: Node[]) => {
    if ('children' in node) {
      all(node.children, [node, ...parents]);
    }
    callback(node, index, parents);
  };
  if (Array.isArray(tree)) {
    all(tree, []);
  } else {
    one(tree, 0, []);
  }
}

export function structuredTextToString(value: Node[]): string {
  let result: string = '';

  visit(value, (node) => {
    if (!isText(node)) {
      return;
    }

    result += node.text;
  });

  return result;
}

So that in your FieldAddon.tsx you can do:
  const fieldValue: string =
    ctx.field.attributes.field_type === 'structured_text'
      ? structuredTextToString(get(ctx.formValues, ctx.fieldPath) as Node[])
      : String(get(ctx.formValues, ctx.fieldPath));
  const fieldStats: CountObject = counter(fieldValue);

Hi @stefanoverna,
Thanks for the suggestion.
We actually wanted to add the structured text but didn't seem to get it working. We based the supported fields on the PluginAttributes type where 'structured_text' isn't added in the field_types array.
We have checked how to get it working by consorting the Lorum ipsum plugin and will add structured text support in the next version (v1.2.0).

I see! But that field_types field is for legacy plugins!
Thanks, and let me know if you need any help from me!

Uh, and "previewImage": "docs/preview.gif" in package.json can now also be an MP4 file. It will render better!

When we used
manualFieldExtensions() {
    return [
      {
        id: extensionId,
        name: 'Word counter',
        type: 'addon',
        fieldTypes: [
          'text',
          'string',
          'rich_text',
          'structured_text',
        ],
        configurable: true,
      },
    ]
}

We got an typescript error on the manualFieldExtensions:  Type '"structured_text"' is not assignable to type '"string" | "boolean" | "date" | "date_time" | "float" | "integer" | "text" | "lat_lon" | "json" | "seo" | "link" | "links" | "video" | "color" | "slug" | "rich_text" | "file" | "gallery"' which got us only working with those types.
We are now using the code below based on the Lorum ipsum plugin
manualFieldExtensions() {
    return [
      {
        id: extensionId,
        name: 'Word counter',
        type: 'addon',
        fieldTypes: [
          'text',
          'string',
          'rich_text',
          'structured_text',
        ] as NonNullable<
          PluginAttributes['field_types']
        >,
        configurable: true,
      },
    ]
}

You're right! SDK v0.3.33 fixes this!

We have updated the plugin. From v1.2.0 onwards you can use this plugin together with the structured text fields.

Excellent, thanks so much! And we can wait to list you in our Partners page :) https://www.datocms.com/partners


polymc: 1.1.1 -> 1.2.1

Description of changes

1.2.0 notes
1.2.1 notes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Duplicate of https://github.com/NixOS/nixpkgs/pull/169902

Apologies, checked yesterday, did not check again when I sat down with the PR. Thanks! =D


RestAssured.useRelaxedHTTPSValidation() is ignored on http requests.

We have a web application that enforces SSL connections. So when a request is made to http://example.com:9080/myapp it will automatically be forwarded to https://example.com:9443/myapp. In our testing environments we use a self signed certificate that is generated on the fly and is not easily accessed thus our Rest Assured tests need to just ignore HTTPS Validation. I have observed the following:
Failing test:
	@Test
	public void basicPingTest() {
		RestAssured.authentication = basic("foo", "bar");
		RestAssured.baseURI = "http://example.com:9080/myapp";
		RestAssured.useRelaxedHTTPSValidation();
		given().when().get().then().statusCode(200);
	}

Fails with error:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.exampleTestServiceIT.basicPingTest(TestServiceIT.java:34)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.example.TestServiceIT.basicPingTest(TestServiceIT.java:34)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.example.TestServiceIT.basicPingTest(TestServiceIT.java:34)

Working test:
	@Test
	public void basicPingTest() {
		RestAssured.authentication = basic("foo", "bar");
		RestAssured.baseURI = "https://example.com:9443/myapp";
		RestAssured.useRelaxedHTTPSValidation();
		given().when().get().then().statusCode(200);
	}

It appears that the useRelaxedHTTPSValidation configuration is ignored when the original request is using an http scheme. Is this an oversight in the design?

Refer to SSLITest.java  for other examples as well

Example url that throws such exception
http://www.bupa.com.au/

You send an http request and you disable httpS cert checking. Of course it doesn't work.
I think your request is redirected to another address, that uses https and requires an ssl cert.
So you can use .redirects().follow(false) and grab the location header of response.

@blindenvy  We also have the same scenario as your project but in our case even with Base URI "https://localhost:8443" it is not working on tomcat server. We also have auto redirecting from http to https.
My Test case code looks like this
RestAssured.baseURI = "https://localhost:8443/some/rest/";
RestAssured.useRelaxedHTTPSValidation();
RestAssured.given().when().get("role/listAll").then().statusCode(200);
Getting below error
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.SSLSocketImpl.recvAlert(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
Please guide. Thanks in advance!

Hi! Any updates on when this is going to be fixed?

not fixed yet, just use url with https to avoid this issue


Don't slice using negative indexes to filter  bar charts

Description
We search for undefined values to filter them, but then also use the -1 to slice the arrays, resulting in wrapping around to the first element and double rendering some bins on our NotebookProfilerViewer charts. The double rendering appears darker with the use of opacity.
The fix is to not try to change the data if there are no undefined values in the findAndDeleteUndefined  function.
General Checklist

[ ] Tests added for this feature/bug
if it was a bug, test must cover it.
[ ] Conform by the style guides, by using formatter
[ ] Documentation updated
[ ] (optional) Please add a label to your PR

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 82.58%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4935

Relevant Lines:
5811

💛  - Coveralls


[TechDocs] Introduce a healthcheck endpoint for the TechDocs backend

Feature Suggestion
When deploying the TechDocs backend (as part of a more monolithic Backstage backend, or in isolation), it will be important to know that the TechDocs backend is healthy and correctly configured.
We should expose the health of the TechDocs backend service on an http endpoint for the benefit of external systems, e.g. a Kube liveness probe or a synthetic monitoring service.
Possible Implementation
Add a route /healthz to techdocs-backend, which leverages the result of a configured publisher's getReadiness method, as well as configuration, to indicate a healthy service using HTTP status codes according to kube conventions.
The current getReadiness implementations simply check that a configured storage provider is accessible, but we should add two more checks:

isReadable (indicating that the TechDocs backend is able to read from the configured storage)
isWriteable (indicating that the TechDocs backend is able to write to the configured storage)

Which we could conditionally validate based on configuration...  e.g. If techdocs.builder is external, then we could ignore the isWriteable check
Context
This is a follow-up to #5260, which added a getReadiness method to Publishers.

The backend-common package already provides a rudimentary health check endpoint for the backend: https://github.com/backstage/backstage/blob/826d50e033648c3c6950f778dac6667d053f7b36/packages/backend/src/plugins/healthcheck.ts#L24
It could be an option to investigate if we should extend this functionality so other backends could register own health checks.
On the other hand one needs to carefully select what to include in readiness/liveness checks since some issues should not result in an unreachable deployment. But the /healthz endpoint can for sure be a nice source for SLA monitoring or other synthetic monitoring strategies.

It could be an option to investigate if we should extend this functionality so other backends could register own health checks.

I was diving into this a little bit recently...  And it looks like the simplest thing to do would be to pass in custom status check logic in src/plugins/healthcheck.ts, like so:
diff --git a/packages/backend/src/plugins/healthcheck.ts b/packages/backend/src/plugins/healthcheck.ts
index 8ecae6be8..ab48716f0 100644
--- a/packages/backend/src/plugins/healthcheck.ts
+++ b/packages/backend/src/plugins/healthcheck.ts
@@ -21,5 +21,13 @@ import { PluginEnvironment } from '../types';
 export default async function createRouter({
   logger,
 }: PluginEnvironment): Promise<Router> {
-  return await createStatusCheckRouter({ logger, path: '/healthcheck' });
+  return await createStatusCheckRouter({
+    logger,
+    path: '/healthcheck',
+    statusCheck: () => {
+      return new Promise(() => {
+        // Custom status check logic here.
+      });
+    },
+  });
 }

The problem is that the types of stuff we might want to check (like the TechDocs publisher instance) are instantiated elsewhere and we don't have access to them here.
So we may want some other way to more easily compose health checks across multiple plugins in a single Backstage instance.

Noting thoughts from @freben in discord:

I mean the health endpoint could be a class that accepts a number of () => Promise callbacks and just runs them and also fails if they throw or something like that. But the value and structure added would be rather small on its own
Things could be different if there were some inversion of control in place, where plugins are able to register themselves into an environment in several aspects in one single callback. But that kind of work is still TBD
Ie, absent a more formal plugin system, this is likely staying as Just One More Thing to wire up manually (kinda bespoke)

...So I think we probably postpone this for now.


Issue with SQLite

Tutorial: https://developers.sap.com/tutorials/cp-cap-java-reusable-service.html
After running
npm install --save-dev sqlite3
as explained in step 5 of above tutorial, I try to run
cds deploy --to sqlite
The latter command can't be executed successfully. Instead, following error message is displayed:
_[ERROR] Cannot find module 'sqlite3'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Promise (/extbin/npm/globals/lib/node_modules/@sap/cds-runtime/lib/sqlite/Service.js:21:17)
at new Promise ()
at new (/extbin/npm/globals/lib/node_modules/@sap/cds-runtime/lib/sqlite/Service.js:19:10)
at SQLiteDatabase.acquire (/extbin/npm/globals/lib/node_modules/@sap/cds-runtime/lib/sqlite/Service.js:130:19)
at SQLiteDatabase.begin (/extbin/npm/globals/lib/node_modules/@sap/cds-runtime/lib/db/Service.js:48:23)
at Function.for (/extbin/npm/globals/lib/node_modules/@sap/cds/lib/srv/Transaction.js:47:26)

Hi Johannes,
Please see the comment in this issue. Maybe this solves your problem and you can continue. Otherwise please get back here.
Best regards,
Iwona

Hi Iwona,
many thanks for your support. It works when trying to run the following command as explained in the other issue:
unset NODE_PATH
Thanks and Regards,
Johannes


Long read error

Dear author,
Thanks for developing this tool! I am using the command below to genotype SVs.
jellyfish count --canonical --lower-count 2 --out-counter-len 2 --mer-len 25 --threads 8 --size 3G --output counts.jf ./hg38/no_alt/hg38.no_alt.fa

locityper add -d db -r ./hg38/no_alt/hg38.no_alt.fa -j counts.jf -v ./mc/hprc-v1.1-grch38.vcf.gz -l test_region chr22:42106499-42160865

locityper preproc -a ./GRCh38/PacBio_HiFi/HG01890.all.sorted.bam -j counts.jf -r ./hg38/no_alt/hg38.no_alt.fa -o HG01890 -t hifi

The first two commands work well, but an error raised when I run the final step.
[16:13:37 DEBUG] locityper v0.10.9-1 @ 2023-11-07 16:13:37
[16:13:37  INFO] Recognized GRCh38 reference genome, using background region chr17:72062001-76562000
[16:13:37  INFO] Detected jellyfish k-mer size: 25
[16:13:37  INFO] Calculating k-mer counts on the background region
[16:13:43 DEBUG] Loading mapped reads into memory (./GRCh38/PacBio_HiFi/HG01890
.all.sorted.bam)
[16:13:46 DEBUG]     Loaded 18201 alignments, discarded 692
[16:13:46 DEBUG]     Using 5000 bp windows, (window neighbourhood size 5000, boundary 1000)
[16:13:46 DEBUG]     Total windows:       899
[16:13:46 DEBUG]     Remove 0 windows with Ns, 672 windows with common k-mers
[16:13:46 DEBUG]     After filtering:     227
[16:13:46  INFO] Estimating read error profiles from 18201 reads
[16:13:46  INFO]         69449915 matches    (0.993567)
[16:13:46  INFO]            86444 mismatches (0.001237)
[16:13:46  INFO]           224809 insertions (0.003216)
[16:13:46  INFO]           138437 deletions  (0.001981)
[16:13:46  INFO]     Maximum allowed edit distance: 469 (for read length 15000, 99%-confidence interval)
[16:13:47  INFO]     Read depth mean = 39.73,  variance: 71.09  (at GC-content 40)
[16:13:47  INFO] Success. Total time: 0:00:10.392

Hi Darren,
I don't see the error in your log, everything looks fine?


Setting server in remix.config.js fails when using express adapter

What version of Remix are you using?
1.4.3
Steps to Reproduce

Create app using create-remix (express, just the basics, typescript)
Set server in remix.config.js to let remix produce one consolidated build.
npm run build works
npm run dev fails

Sample code here:
https://github.com/edwin177/remix-express/tree/express-server-build
Expected Behavior
Server should load
Actual Behavior
App crashes with the following error:
/home/xxx/remix-express/node_modules/@remix-run/server-runtime/routes.js:18
  return Object.entries(manifest).filter(([, route]) => route.parentId === parentId).map(([id, route]) => ({ ...route,
                ^
TypeError: Cannot convert undefined or null to object
    at Function.entries (<anonymous>)
    at Object.createRoutes (/home/xxx/remix-express/node_modules/@remix-run/server-runtime/routes.js:18:17)
    at Object.createRequestHandler (/home/xxx/remix-express/node_modules/@remix-run/server-runtime/server.js:26:25)
    at createRequestHandler (/home/xxx/remix-express/node_modules/@remix-run/express/server.js:35:28)
    at Object.<anonymous> (/home/xxx/remix-express/build/index.js:42:5)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)

This works fine on v1.3.3.

If you're using the Express adapter, you shouldn't be running remix dev. You should be running remix watch and node server.js.
remix dev is basically remix watch plus remix-serve.

@kiliman Thanks for being so active here and Discord.
You can see in the package.json that npm run dev only calls remix watch:
"scripts": {
    "build": "remix build",
    "dev": "remix build && run-p dev:*",
    "dev:node": "cross-env NODE_ENV=development nodemon ./build/index.js --watch ./build/index.js",
    "dev:remix": "remix watch",
    "start": "cross-env NODE_ENV=production node ./build/index.js"
  },

This was only slightly modified from the standard output from create-remix to call the remix build rather than the original express server.

Sorry, I misread that. Thought you meant remix dev not npm run dev.
Anyway, tried setting up server in config and it doesn't appear to be working at all in 1.4.3.

I never could get server config to work, so I punted and now use esbuild to bundle server.js. Here are my scripts. nodemon is running run-s server:*, so it runs the server build (esbuild) followed by node build/server.js. This way any changes you make to server.js will force the npm scripts to re-execute.
You'll see that esbuild outputs to build/server.js. remix watch generates the app files in build/index.js, which is imported by server.js. So any changes to route files will trigger a rebuild of build/index.js and server.js will pickup the changes after <LiveReload/>.
For production, just run npm run build. It will create the 2 files: index.js and server.js. Copy those to your host and then run npm run start. You won't need to install any node_modules.
  "scripts": {
    "build": "cross-env NODE_ENV=production remix build && npm run server:build",
    "dev": "run-p dev:*",
    "dev:node": "cross-env NODE_ENV=development nodemon --exec \"run-s server:*\"  --watch ./server.js",
    "dev:remix": "remix watch",
    "server:build": "esbuild server.js --outfile=build/server.js --platform=node --format=cjs --bundle --sourcemap",
    "server:start": "node ./build/server.js",
    "start": "cross-env NODE_ENV=production node ./build/server.js"
  },

Thanks for confirming that you can't get it to work either.
I'm currently on v1.3.3 which works fine with Express + setting server in config, but can't upgrade due to this issue.
I'll take a look at incorporating your example of handling the build directly.

In fastify  example import * as serverBuild from '@remix-run/dev/server-build'; used and then remix watch + nodemon can run together

Hmm from the docs

A server entrypoint, relative to the root directory that becomes your server's main module. If specified, Remix will compile this file along with your application into a single file to be deployed to your server. This file can use either a .js or .ts file extension.

I thought I could just throw in a server.ts but running node ./build/index.js & remix watch errors
[..]/indie-stack/node_modules/@remix-run/express/node_modules/@remix-run/server-runtime/dist/routes.js:18
  return Object.entries(manifest).filter(([, route]) => route.parentId === parentId).map(([id, route]) => ({ ...route,
                ^
TypeError: Cannot convert undefined or null to object
    at Function.entries (<anonymous>)
    at Object.createRoutes ([..]/indie-stack/node_modules/@remix-run/express/node_modules/@remix-run/server-runtime/dist/routes.js:18:17)
    at Object.createRequestHandler ([..]/indie-stack/node_modules/@remix-run/express/node_modules/@remix-run/server-runtime/dist/server.js:26:25)
    at createRequestHandler ([..]/indie-stack/node_modules/@remix-run/express/dist/server.js:34:28)
    at Object.<anonymous> ([..]/indie-stack/build/index.js:42:79)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)

Btw, still getting the same error with 1.7.0.
Sample code:
https://github.com/edwin177/remix-express-1.7

This issue is still happening.
(try upgrading from 1.2.3 to 1.7.4 Today)

After some debugging, I was able to determine that this error was happening for me because the default require(BUILD_DIR) calls were returning an empty object. Inside the remix request handler, it passes properties from the build object to various handlers. Since require(BUILD_DIR) is returning an empty object, those properties are undefined, which eventually manifests as the error message described in this issue.

I was able to fix this on my end by appending index.js (or whatever your remix build output file is) to the require(BUILD_DIR) found in createRequestHandler. As a note, I am doing an ESBuild setup for my server file, similar to what was described by @/kiliman above. I don't know if this will also fix it for setups using the default Remix 'server' configuration.

I doubt this is probably an ideal fix, but I wanted to provide what may be a potential workaround for some people, since it seems to be a common issue.
Sample code:
Original remix request handler
const BUILD_DIR = path.join(process.cwd(), "build");

// ... A bunch of express setup stuff

app.all(
  "*",
  process.env.NODE_ENV === "development"
    ? (req, res, next) => {
        purgeRequireCache();

        return createRequestHandler({
          build: require(BUILD_DIR),
          mode: process.env.NODE_ENV,
          getLoadContext,
        })(req, res, next);
      }
    : createRequestHandler({
        build: require(BUILD_DIR),
        mode: process.env.NODE_ENV,
        getLoadContext,
      })
);

A version that worked for me
const BUILD_DIR = path.join(process.cwd(), "build");
const BUILD_FILE = path.join(BUILD_DIR, "index.js");

// ... A bunch of express setup stuff

app.all(
  "*",
  process.env.NODE_ENV === "development"
    ? (req, res, next) => {
        purgeRequireCache();

        return createRequestHandler({
          build: require(BUILD_FILE),
          mode: process.env.NODE_ENV,
          getLoadContext,
        })(req, res, next);
      }
    : createRequestHandler({
        build: require(BUILD_FILE),
        mode: process.env.NODE_ENV,
        getLoadContext,
      })
);

I ran into this with Remix 1.8.2. It seems like the problem is twofold.
Problem 1: overridden entrypoint to esbuild
Setting server in remix.config.js changes the esbuild config. I think the problem may be here?
https://github.com/remix-run/remix/blob/8fb82b27bcf7174fcd83dd0fc63d99764ff264c1/packages/remix-dev/compiler/compilerServer.ts#L35-L43
As a result, build/index.js only includes things imported from the server file. As an extreme example, when the server file only includes
console.log("wat");

then build/index.js only includes that:
"use strict";

// server.ts
console.log("wat");

Problem 2: cyclic dependencies
If the server is bundled into the same file as the application, this creates a cyclic dependency.

You run node build/index.js to start the server
node requires build/index.js
While loading, it hits the createRequestHandler call, and attempts to require build/index.js
require returns {} to avoid an infinite loop
createRoutes throws because manifest is undefined

A workaround that doesn't require creating a separate esbuild step
I worked around these issues by removing server from remix.config.js and putting the following code in app/entry.server.tsx
import run from "./run.server";

if (require.main === module) {
  process.nextTick(run);
}

The run function creates the express app and calls listen. Because it runs in the next tick, requiring build/index.js does not create a loop.
Solving the problems
I think if server is specified in the config, the compiler needs to produce two files. build/index.js would be produced from the server file and build/app.js would be the application code.

I'm relatively new to remix so please correct me if I wrong. I've somehow managed to run the remix with express adapter, setting the server in remix.config.js to server.ts. Then in the server.ts I just use the import * as build from "@remix-run/dev/server-build". Here is what I did:
// rest of imports
import * as build from "@remix-run/dev/server-build";

// rest of server.ts

app.all(
  "*",
  process.env.NODE_ENV === "development"
    ? (req, res, next) => {
        purgeRequireCache();

        return createRequestHandler({
          build, // removed the require to BUILD_DIR
          mode: process.env.NODE_ENV,
        })(req, res, next);
      }
    : createRequestHandler({
        build, // removed the require to BUILD_DIR
        mode: process.env.NODE_ENV,
      })
);

// rest of server.ts

I don't know if this is the correct way to do it. I have also looked on the other adapter templates and it shows the usage of the import * as build from "@remix-run/dev/server-build".
I'm sorry if my english is bad cause it's not even my second language. Hopefully this helps as a workaround until we have the correct way to do it.

Hey folks,
for whoever is running into this issue, I've managed to circunvent this problem with a mix of the suggestion from @mfadlyramadhan above and some custom setup I did for running the Express server during local development, including support for hot-module replacement.
i've created a repo here that you can take a look or even use it in case you want to have Remix + Express + development with HMR + Docker builds.
https://github.com/brunojppb/remix-express-typescript
The secret is mostly in the remix.config.js to only add the server entry during production builds.

Hey folks,
for whoever is running into this issue, I've managed to circunvent this problem with a mix of the suggestion from @mfadlyramadhan above and some custom setup I did for running the Express server during local development, including support for hot-module replacement.
i've created a repo here that you can take a look or even use it in case you want to have Remix + Express + development with HMR + Docker builds.
https://github.com/brunojppb/remix-express-typescript
The secret is mostly in the remix.config.js to only add the server entry during production builds as you can see here which avoids this bug.
...(isProd
    ? {
        server: "server/prod.ts",
        serverBuildPath: "build/index.js",
      }
    : {}),

Then I customised the Express server to have a dynamic entrypoint where the development and production setup can be adjusted based on your needs.

In development, you have an entrypoint with Remix request handler and HMR replacement hooked up here.
In production, you just read the build files generated by the compiler here.

Ideally, would be great if the Remix team could make the server entry work smoothly with the entire Express Adapter + TypeScript compilation setup.
Last, but not least, thanks to the Remix team for the incredible work put on this framework! 👏
CC: @raulfdm @leoweigand @dge808


Implement support for Canon C-Log2.

Will Canon Log2 be supported in the configs at some point? Reading the linked Canon Whitepaper it says:

Q1: Is Canon Log 3 the successor to and an improved version of the existing Canon Log 2?
A: Canon Log 3 is not the successor to Canon Log 2 but the successor to Canon Log. It extends the dynamic
range by the equivalent of one stop compared with Canon Log.
Q2: Which log gamma curve: Canon Log, Canon Log 2 or Canon Log 3, would you recommend that
users should use?
A: For users who have a camera with all the log curves installed, we would recommend that Canon Log 2 or
Canon Log 3 be used. Users can choose to use either the Canon Log 2 or Canon Log 3 gamma curve
depending on their workflows. Canon Log 2 is suited to those workflows where image quality can be enhanced
by grading at the postproduction stage whereas Canon Log 3 is suited to cases where users want to create the
final “look” of their images by means of simple grading without investing too much time at the post-production
stage.

thanks!

Thanks for the suggestion Derek, we will definitely consider adding Canon Log 2 to a future release.

To extend on what Doug said, yes we will: Canon wanted us to release with it BUT there is not builtin yet in OCIO for it, so it would have forced us to ship with a LUT which we deemed unreasonable because a) we were striving to not have any LUTs b) we do not have code to handle that yet.

Thanks for the explanation, Thomas. What about the BuiltinTransform  CANON_CLOG2-CGAMUT_to_ACES2065-1 ?

That’s brand new! It will get released in 2.2 in a few days.
Also, thanks for the feedback - the configs working group took a “less is
more” approach to what is included in the ocio-supported configs. We did
not include everything from previous, and that was a conscious decision.
We’re relying on folks like yourself to tell us when you are
missing something!
Also - even though it’s not in the default config yet, anyone can add canon
log 2 to the generator and have it to use immediately!
On Fri, Oct 28, 2022 at 08:12 Derek Flood @.***> wrote:

Thanks for the explanation, Thomas. What about the BuiltinTransform
CANON_CLOG2-CGAMUT_to_ACES2065-1 ?
—
Reply to this email directly, view it on GitHub
https://github.com/AcademySoftwareFoundation/OpenColorIO-Config-ACES/issues/87#issuecomment-1295119613,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMPHU2DZAKUICY2YNSGZHX3WFPUORANCNFSM6AAAAAARQPT3HE
.
You are receiving this because you are subscribed to this thread.Message
ID:
<AcademySoftwareFoundation/OpenColorIO-Config-ACES/issues/87/1295119613@
github.com>

--
CAROL PAYNE
Imaging Technologist
Creative Technologies
T<PHONE_NUMBER>
5808 W. Sunset Blvd  |  Los Angeles, CA 90028

Thanks Carol. I really appreciate the collaborative and open approach that the committee is taking, and also really value you all sharing insight into the choices the commitee has made.

Sorry, I should have used "released" code earlier: We pull the Python wheels to generate the Configs artefacts on CI (GitHub Actions), so the packages the generators are using there must be available on Pypi.
Note that even though 2.2 is to be imminently released, the new BuiltinTransform will only be available when software vendors adopt that version of the library, i.e. you won't be able to use it before a few months.

Support for Canon C-Log2 is now included in the new configs.


refactor(Order/WellFounded): deprecate WellFounded.succ

This definition is somewhat of a footgun, as it can't actually be used to define a SuccOrder instance on partial orders, and in the linear order case, one can use SuccOrder.ofLinearWellFoundedLT for that purpose.

bors merge


Netty cannot build SSL Context on Android

Using: netty-all:4.1.0.Final, Android 4.4.2.
Running the following code
public void connect() {
    // Configure SSL.
    SslContext sslCtx = null;
    try {
        sslCtx = SslContextBuilder.forClient()
            .trustManager(InsecureTrustManagerFactory.INSTANCE).build();
    } catch (SSLException e) {
        System.out.println(e.getMessage());
    }
}

produces the following rather long printouts in logcat (dates and package names omitted from beginning for clarity). Note that the code works perfectly when run from a Windows environment.
I/dalvikvm: Could not find method org.slf4j.LoggerFactory.getILoggerFactory, referenced from method io.netty.util.internal.logging.Slf4JLoggerFactory.<init>
W/dalvikvm: VFY: unable to resolve static method 61515: Lorg/slf4j/LoggerFactory;.getILoggerFactory ()Lorg/slf4j/ILoggerFactory;
I/dalvikvm: Could not find method org.slf4j.LoggerFactory.getLogger, referenced from method io.netty.util.internal.logging.Slf4JLoggerFactory.newInstance
W/dalvikvm: VFY: unable to resolve static method 61516: Lorg/slf4j/LoggerFactory;.getLogger (Ljava/lang/String;)Lorg/slf4j/Logger;
I/dalvikvm: Could not find method org.apache.log4j.Logger.getLogger, referenced from method io.netty.util.internal.logging.Log4JLoggerFactory.newInstance
W/dalvikvm: VFY: unable to resolve static method 60939: Lorg/apache/log4j/Logger;.getLogger (Ljava/lang/String;)Lorg/apache/log4j/Logger;
W/dalvikvm: VFY: unable to find class referenced in signature (Lorg/apache/log4j/Logger;)
I/dalvikvm: Could not find method org.apache.log4j.Logger.getName, referenced from method io.netty.util.internal.logging.Log4JLogger.<init>
W/dalvikvm: VFY: unable to resolve virtual method 60940: Lorg/apache/log4j/Logger;.getName ()Ljava/lang/String;
I/dalvikvm: Could not find method org.apache.log4j.Logger.isTraceEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.isTraceCapable
W/dalvikvm: VFY: unable to resolve virtual method 60944: Lorg/apache/log4j/Logger;.isTraceEnabled ()Z
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
I/dalvikvm: Could not find method org.apache.log4j.Logger.isDebugEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.debug
W/dalvikvm: VFY: unable to resolve virtual method 60941: Lorg/apache/log4j/Logger;.isDebugEnabled ()Z
I/dalvikvm: Could not find method org.apache.log4j.Logger.isDebugEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.debug
W/dalvikvm: VFY: unable to resolve virtual method 60941: Lorg/apache/log4j/Logger;.isDebugEnabled ()Z
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
I/dalvikvm: Could not find method org.apache.log4j.Logger.isDebugEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.debug
W/dalvikvm: VFY: unable to resolve virtual method 60941: Lorg/apache/log4j/Logger;.isDebugEnabled ()Z
W/dalvikvm: VFY: unable to resolve static field 22387 (ERROR) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22387 (ERROR) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22387 (ERROR) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22387 (ERROR) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22387 (ERROR) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22388 (INFO) in Lorg/apache/log4j/Level;
I/dalvikvm: Could not find method org.apache.log4j.Logger.isInfoEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.info
W/dalvikvm: VFY: unable to resolve virtual method 60943: Lorg/apache/log4j/Logger;.isInfoEnabled ()Z
I/dalvikvm: Could not find method org.apache.log4j.Logger.isInfoEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.info
W/dalvikvm: VFY: unable to resolve virtual method 60943: Lorg/apache/log4j/Logger;.isInfoEnabled ()Z
W/dalvikvm: VFY: unable to resolve static field 22388 (INFO) in Lorg/apache/log4j/Level;
I/dalvikvm: Could not find method org.apache.log4j.Logger.isInfoEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.info
W/dalvikvm: VFY: unable to resolve virtual method 60943: Lorg/apache/log4j/Logger;.isInfoEnabled ()Z
I/dalvikvm: Could not find method org.apache.log4j.Logger.isDebugEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.isDebugEnabled
W/dalvikvm: VFY: unable to resolve virtual method 60941: Lorg/apache/log4j/Logger;.isDebugEnabled ()Z
W/dalvikvm: VFY: unable to resolve static field 22387 (ERROR) in Lorg/apache/log4j/Level;
I/dalvikvm: Could not find method org.apache.log4j.Logger.isInfoEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.isInfoEnabled
W/dalvikvm: VFY: unable to resolve virtual method 60943: Lorg/apache/log4j/Logger;.isInfoEnabled ()Z
I/dalvikvm: Could not find method org.apache.log4j.Logger.isTraceEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.isTraceEnabled
W/dalvikvm: VFY: unable to resolve virtual method 60944: Lorg/apache/log4j/Logger;.isTraceEnabled ()Z
I/dalvikvm: Could not find method org.apache.log4j.Logger.isDebugEnabled, referenced from method io.netty.util.internal.logging.Log4JLogger.isTraceEnabled
W/dalvikvm: VFY: unable to resolve virtual method 60941: Lorg/apache/log4j/Logger;.isDebugEnabled ()Z
W/dalvikvm: VFY: unable to resolve static field 22390 (WARN) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22389 (TRACE) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22389 (TRACE) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22389 (TRACE) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22389 (TRACE) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22389 (TRACE) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22386 (DEBUG) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22390 (WARN) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22390 (WARN) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22390 (WARN) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22390 (WARN) in Lorg/apache/log4j/Level;
W/dalvikvm: VFY: unable to resolve static field 22390 (WARN) in Lorg/apache/log4j/Level;
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5772 at 0x10 in Lio/netty/util/internal/logging/Log4JLogger;.debug
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5772 at 0x10 in Lio/netty/util/internal/logging/Log4JLogger;.debug
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5772 at 0x10 in Lio/netty/util/internal/logging/Log4JLogger;.debug
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5773 at 0x12 in Lio/netty/util/internal/logging/Log4JLogger;.error
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5773 at 0x12 in Lio/netty/util/internal/logging/Log4JLogger;.error
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5773 at 0x12 in Lio/netty/util/internal/logging/Log4JLogger;.error
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5774 at 0x10 in Lio/netty/util/internal/logging/Log4JLogger;.info
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5774 at 0x10 in Lio/netty/util/internal/logging/Log4JLogger;.info
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5774 at 0x10 in Lio/netty/util/internal/logging/Log4JLogger;.info
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5776 at 0x12 in Lio/netty/util/internal/logging/Log4JLogger;.warn
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5776 at 0x12 in Lio/netty/util/internal/logging/Log4JLogger;.warn
I/dalvikvm: DexOpt: unable to optimize static field ref 0x5776 at 0x12 in Lio/netty/util/internal/logging/Log4JLogger;.warn
I/dalvikvm: Could not find method org.apache.tomcat.jni.Pool.create, referenced from method io.netty.handler.ssl.OpenSsl.<clinit>
W/dalvikvm: VFY: unable to resolve static method 60970: Lorg/apache/tomcat/jni/Pool;.create (J)J
I/dalvikvm: Could not find method org.apache.tomcat.jni.Library.initialize, referenced from method io.netty.handler.ssl.OpenSsl.initializeTcNative
W/dalvikvm: VFY: unable to resolve static method 60969: Lorg/apache/tomcat/jni/Library;.initialize (Ljava/lang/String;)Z
E/dalvikvm: Could not find class 'org.apache.tomcat.jni.SSL', referenced from method io.netty.handler.ssl.OpenSsl.loadTcNative
W/dalvikvm: VFY: unable to resolve const-class 6540 (Lorg/apache/tomcat/jni/SSL;) in Lio/netty/handler/ssl/OpenSsl;
I/dalvikvm: Could not find method org.apache.tomcat.jni.Buffer.address, referenced from method io.netty.handler.ssl.OpenSsl.memoryAddress
W/dalvikvm: VFY: unable to resolve static method 60968: Lorg/apache/tomcat/jni/Buffer;.address (Ljava/nio/ByteBuffer;)J
I/dalvikvm: Could not find method org.apache.tomcat.jni.SSL.version, referenced from method io.netty.handler.ssl.OpenSsl.version
W/dalvikvm: VFY: unable to resolve static method 61010: Lorg/apache/tomcat/jni/SSL;.version ()I
I/dalvikvm: Could not find method org.apache.tomcat.jni.SSL.versionString, referenced from method io.netty.handler.ssl.OpenSsl.versionString
W/dalvikvm: VFY: unable to resolve static method 61011: Lorg/apache/tomcat/jni/SSL;.versionString ()Ljava/lang/String;
I/dalvikvm: Could not find method sun.misc.Unsafe.addressSize, referenced from method io.netty.util.internal.PlatformDependent0.addressSize
W/dalvikvm: VFY: unable to resolve virtual method 61528: Lsun/misc/Unsafe;.addressSize ()I
I/dalvikvm: Could not find method sun.misc.Unsafe.allocateMemory, referenced from method io.netty.util.internal.PlatformDependent0.allocateMemory
W/dalvikvm: VFY: unable to resolve virtual method 61529: Lsun/misc/Unsafe;.allocateMemory (J)J
I/dalvikvm: Could not find method sun.misc.Unsafe.copyMemory, referenced from method io.netty.util.internal.PlatformDependent0.copyMemory
W/dalvikvm: VFY: unable to resolve virtual method 61535: Lsun/misc/Unsafe;.copyMemory (JJJ)V
I/dalvikvm: Could not find method sun.misc.Unsafe.copyMemory, referenced from method io.netty.util.internal.PlatformDependent0.copyMemory
W/dalvikvm: VFY: unable to resolve virtual method 61536: Lsun/misc/Unsafe;.copyMemory (Ljava/lang/Object;JLjava/lang/Object;JJ)V
I/dalvikvm: Could not find method sun.misc.Unsafe.getChar, referenced from method io.netty.util.internal.PlatformDependent0.equals
W/dalvikvm: VFY: unable to resolve virtual method 61540: Lsun/misc/Unsafe;.getChar (Ljava/lang/Object;J)C
I/dalvikvm: Could not find method sun.misc.Unsafe.getChar, referenced from method io.netty.util.internal.PlatformDependent0.equals
W/dalvikvm: VFY: unable to resolve virtual method 61540: Lsun/misc/Unsafe;.getChar (Ljava/lang/Object;J)C
I/dalvikvm: Could not find method sun.misc.Unsafe.getByte, referenced from method io.netty.util.internal.PlatformDependent0.equals
W/dalvikvm: VFY: unable to resolve virtual method 61539: Lsun/misc/Unsafe;.getByte (Ljava/lang/Object;J)B
I/dalvikvm: Could not find method sun.misc.Unsafe.getChar, referenced from method io.netty.util.internal.PlatformDependent0.equals
W/dalvikvm: VFY: unable to resolve virtual method 61540: Lsun/misc/Unsafe;.getChar (Ljava/lang/Object;J)C
I/dalvikvm: Could not find method sun.misc.Unsafe.getChar, referenced from method io.netty.util.internal.PlatformDependent0.equals
W/dalvikvm: VFY: unable to resolve virtual method 61540: Lsun/misc/Unsafe;.getChar (Ljava/lang/Object;J)C
I/dalvikvm: Could not find method sun.misc.Unsafe.getByte, referenced from method io.netty.util.internal.PlatformDependent0.equals
W/dalvikvm: VFY: unable to resolve virtual method 61539: Lsun/misc/Unsafe;.getByte (Ljava/lang/Object;J)B
I/dalvikvm: Could not find method sun.misc.Unsafe.freeMemory, referenced from method io.netty.util.internal.PlatformDependent0.freeMemory
W/dalvikvm: VFY: unable to resolve virtual method 61537: Lsun/misc/Unsafe;.freeMemory (J)V
I/dalvikvm: Could not find method sun.misc.Unsafe.getByte, referenced from method io.netty.util.internal.PlatformDependent0.getByte
W/dalvikvm: VFY: unable to resolve virtual method 61538: Lsun/misc/Unsafe;.getByte (J)B
I/dalvikvm: Could not find method sun.misc.Unsafe.getByte, referenced from method io.netty.util.internal.PlatformDependent0.getByte
W/dalvikvm: VFY: unable to resolve virtual method 61539: Lsun/misc/Unsafe;.getByte (Ljava/lang/Object;J)B
I/dalvikvm: Could not find method sun.misc.Unsafe.getInt, referenced from method io.netty.util.internal.PlatformDependent0.getInt
W/dalvikvm: VFY: unable to resolve virtual method 61541: Lsun/misc/Unsafe;.getInt (J)I
I/dalvikvm: Could not find method sun.misc.Unsafe.getLong, referenced from method io.netty.util.internal.PlatformDependent0.getLong
W/dalvikvm: VFY: unable to resolve virtual method 61544: Lsun/misc/Unsafe;.getLong (J)J
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.getShort
W/dalvikvm: VFY: unable to resolve virtual method 61549: Lsun/misc/Unsafe;.getShort (J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.getShort
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getByte, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61539: Lsun/misc/Unsafe;.getByte (Ljava/lang/Object;J)B
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getByte, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61539: Lsun/misc/Unsafe;.getByte (Ljava/lang/Object;J)B
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.getShort, referenced from method io.netty.util.internal.PlatformDependent0.hashCodeAscii
W/dalvikvm: VFY: unable to resolve virtual method 61550: Lsun/misc/Unsafe;.getShort (Ljava/lang/Object;J)S
I/dalvikvm: Could not find method sun.misc.Unsafe.putByte, referenced from method io.netty.util.internal.PlatformDependent0.putByte
W/dalvikvm: VFY: unable to resolve virtual method 61554: Lsun/misc/Unsafe;.putByte (JB)V
I/dalvikvm: Could not find method sun.misc.Unsafe.putByte, referenced from method io.netty.util.internal.PlatformDependent0.putByte
W/dalvikvm: VFY: unable to resolve virtual method 61555: Lsun/misc/Unsafe;.putByte (Ljava/lang/Object;JB)V
I/dalvikvm: Could not find method sun.misc.Unsafe.putInt, referenced from method io.netty.util.internal.PlatformDependent0.putInt
W/dalvikvm: VFY: unable to resolve virtual method 61556: Lsun/misc/Unsafe;.putInt (JI)V
I/dalvikvm: Could not find method sun.misc.Unsafe.putLong, referenced from method io.netty.util.internal.PlatformDependent0.putLong
W/dalvikvm: VFY: unable to resolve virtual method 61559: Lsun/misc/Unsafe;.putLong (JJ)V
I/dalvikvm: Could not find method sun.misc.Unsafe.putShort, referenced from method io.netty.util.internal.PlatformDependent0.putShort
W/dalvikvm: VFY: unable to resolve virtual method 61567: Lsun/misc/Unsafe;.putShort (JS)V
I/dalvikvm: Could not find method sun.misc.Unsafe.putShort, referenced from method io.netty.util.internal.PlatformDependent0.putShort
W/dalvikvm: VFY: unable to resolve virtual method 61568: Lsun/misc/Unsafe;.putShort (Ljava/lang/Object;JS)V
I/dalvikvm: Could not find method sun.misc.Unsafe.throwException, referenced from method io.netty.util.internal.PlatformDependent0.throwException
W/dalvikvm: VFY: unable to resolve virtual method 61569: Lsun/misc/Unsafe;.throwException (Ljava/lang/Throwable;)V

This may be a duplicate of https://github.com/netty/netty/issues/4824
Can you try to run with -Dio.netty.noUnsafe=true?

How?

This should be defined as a java system property. What I specified above is how you can define the system property on the command line.

Tried it, did the exact same thing. No exception caught.

For everyone who wants to reproduce this, simply run netty's secure chat example's client on Android. No server needed. After further investigation it seems that netty's documentation is incorrect, neglecting to mention a dependency on log4j (and perhaps others), causing the errors.

How did you fixed it then for android?

Closing it as it seems to just work


Bump checkov version to 3.2.159

Why does this repository not receive automatic updates?
CC: @borchero, @pavelzw

@JanJaguschQC when there's no activity in the repo, scheduled workflows stop working at some point afaik. The feedstock was broken and did not receive any automatic updates until I fixed it last week. After merging this PR, I hope that the automation works as expected again.


Fix case. Php < PHP

What does this PR do?
Fixes case of "PHP" in Hindi booklist.
For resources
Description
Why is this valuable (or not)?
I believe it is.
How do we know it's really free?
Not applicable
For book lists, is it a book? For course lists, is it a course? etc.
Not applicable
Checklist:

[X] Read our contributing guidelines.
[X] Search for duplicates.
[X] Include author(s) and platform where appropriate.
[X] Put lists in alphabetical order, correct spacing.
[X] Add needed indications (PDF, access notes, under construction).
[X] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

agreed, but the rest of the titles don't match

I am sorry, but I don't understand. Do you mean the heading?

Don't invent titles! Use the title given by the resource.

I am really sorry for the inconvenience caused. I have followed your suggestions. Please review.

Thanks, and welcome to free-programming-books!


Error installing plugin

We have been trying to add the  amplitude histogram view to our installation of phy but keep running into an error when we launch Phy, that seems to originate with the attempt to load "AsynchCaller".

Following the instructions, we have edited the config file, and created a plugin folder in the .phy folder, into which we have placed a text file with the plugin's code, and named that file "amplitude_histogram.py".
Any guidance you can provide would be greatly appreciated.

Hi, several things you could try after activating the environment, to find out what is the problem:
python -c "import phy.gui.qt"
python -c "import PyQt4.Qt"
python -c "from phy.gui import qt"
python -c "from phy.gui.qt import AsyncCaller"

and let me know the output of these commands (if they work, they should not return anything, otherwise there will be an error message).

Thanks for the response!
I ran all four of those commands, and all but the final one ran without a error.
Running:
python -c "from phy.gui.qt import AsyncCaller"
Gave the error:
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'AsyncCaller'

What versions of phy and phycontrib are you running? If they're not the development versions from GitHub, could you try installing them?

Installing the development version fixed the AsycnhCaller issue! Thanks so much.
I have a few small questions about features present in the development version I am not familiar with from the regular version, but I will create a separate issue for those, in the interest of organization.
Thanks so much for the help!


Does not show bars

##Neofetch Version
version="5.0.0"
Description
When I used the command --battery_display after a fresh install, it doesn’t showed me any bar.
You can fix this by going into .config/neofetch/config.conf and remove the # before the line:
#info "Battery" battery. Then the program draws the battery information. The same for cpu and memory bar. Its not an important bug, but it was a bit strange in the beginning.

AFAIC it's the default behaviour to not show battery info (even after --battery_display) unless its specified in the print_info() function (by removing the comment line).
Closing for the time being, but if you're adamant this is a bug, please reopen this.


Question: Pushing MQTT events to Redis

Environment

OS: Linux (Alpine 3.7 in Docker)
Erlang/OTP: As used in emq-docker
EMQ: 2.3.3

Description
This is more a question than a bug. I tweeted you a question and was directed to start a discussion here.
We are desiging a system where events are raised on mqtt channels of the form <id>/c. Right now, we have a single subscriber which takes those mesasges using the topic subscription of +/c and puts them into Redis for processing by other parts of the system. It's important to avoid duplication of these messages so we don't have to write logic to handle this into the other components.
Now we are trying to scale up, and this single subscriber is a cause for concern - if it crashes or has any slowness for some reason, it becomes a bottleneck or single point of failure for the whole system. Clearly we need to have redundancy and scale-ability here.
It would be great if EMQ could push messages directly into Redis for us. But I have some concerns:

Have I missed something and I can already do this somehow with EMQ as-is?
If I write an EMQ plugin which picks up on topic messages and pushes them to Redis, will running this on every broker cause message duplication? We run a cluster (in k8s) of at least two brokers.
Is there some MQTT function/subscription type I've missed which allows me to have multiple subscribers to a channel but only deliver the message to
exactly one of those subscribers?

Thanks!

First for you questions:
- Have I missed something and I can already do this somehow with EMQ as-is?
A: You need to write your plugin. EMQ doesn't have a plugin fit your requirement out of box, but there're already some official plugins using redis, such as emqt-auth-redis, which can be an example for you.
- If I write an EMQ plugin which picks up on topic messages and pushes them to Redis, will running this on every broker cause message duplication? We run a cluster (in k8s) of at least two brokers.
A: No. Each client will connect to only one broker at a time, and only your plugin on that broker will be invoked upon a new event.
- Is there some MQTT function/subscription type I've missed which allows me to have multiple subscribers to a channel but only deliver the message to exactly one of those subscribers?
A: Yes, try shared-subscription
My suggestion
I recommend you to try the 'plugin' method. EMQ is designed for massive IoT devices, not stream processing. Your subscribers under a heavy load can be bottleneck of the system, and the parse of MQTT packages would lead to extra overhead.

Great, thanks for your help. I'll get started.


eachMessage timeouts and health checks

Does eachMessage have timeout?
The default session timeout is 30s,
What if my single processing takes longer? Right now I get consumer group rebalance as it seems heartbeat is only sent between messages? Perhaps heartbeat should be independent?

eachMessage does not have a timeout. If you want that, you'll need to do it in userland. However, the heartbeat interval is essentially the "timeout", since if your eachMessage has not resolved within that time, the commit will be rejected.
The reason that heartbeat is not independent from the message processing is that it is intended as a healthcheck. If your consumer is not processing messages, it is not healthy, so the heartbeat should probably fail.
If you have some long running task that requires you to heartbeat intermittently, you can currently only do that via the eachBatch interface. However, #387 proposes a way to heartbeat outside of eachBatch, but it's not available yet.

Note that when you use eachBatch, which exposes a heartbeat method, you'll have to be sure that you're only sending heartbeats when processing is actually still healthy and didn't get stuck.  Setting a setInterval for it might keep the session alive, but generally is just disabling the failure detection mechanism completely, allowing your processing to get stuck.

the heartbeat interval is very noisy as timeout as it causes all other partition consumers to rebalance as well. Right now with one bad message that processing takes over default 30s the entire consumer group falls into infinite rebalance loop.

I can add timeout and Promise.race(), but it like feels something that the existing retry mechanism  should handle first and then maybe cause consumer group rebalance.

the heartbeat interval is very noisy as timeout

I'm not sure exactly what you're saying, but I think there might be some confusion about heartbeatInterval. This does not cause automatic heartbeats, it only prevents heartbeats to happen more than whatever that is set to. You still have to manually call heartbeat (exposed in eachBatch), that value only prevents from sending hundreds of heartbeats per second. If heartbeat is called in a timely manner it should prevent a rebalance.
An example situation where we needed this was when processing a message we had to "wait" for something else to happen first, so we polled it's status every 3 seconds. After polling 10 times, the 30 second session time-out was reached and a rebalance occurred. To fix it, we called hearbeat after every time we polled the other resource for an update.

@pimpelsang feel free to re-open the issue if you need.


Introduce OptimisticNormalizedCache and API to write/rollback

Introduce OptimisticNormalizedCache to be always used as first layer in a chain
Introduce ApolloStore API to write/rollback optimistic updates
Some small refactoring related to RecordFieldJsonAdapter that required only for SQL cache
Part of #583

Maybe I'm missing something, but does the following work:
Current key value (not in optimisitc cache):  A -> 0
Optimistic update 1 A -> 1
Optimistic update 2 A -.> 2
rollback optimistic update 2:
expect A -> 1, but it seems that rolling back 2 would remove the A from the optimistic cache, forgetting about update 1, which is still pending and valid.

@BenSchwab yes that is correct, update with version 2 will override update with version 1. And when we rollback the version 2 it will rollback all. This case will happen only when second query was faster than the first one.
In order to support this case I guess we need to keep the whole history of changes for all records in mem cache. I think for now we can keep this simple version of implementation, and I will think how we can solve this case.

@BenSchwab Yeah, it will be almost similar to JS: ApolloMutationCall.optimisticUpdates(...)


NativeScript 7 Support

Could you please add support for NativeScript 7?
https://nativescript.org/blog/nativescript-7-for-plugin-authors/
https://nativescript.org/blog/nativescript-7-compatible-plugins/

Check out the 2.0.0 version I just released to npm.


Delete subway.txt

No description provided.

No description provided.

No description provided.

No description provided.


Art animation

This PR fails the Travis-CI build tests. Art, please check the build logs and see what breaks.


Separate evaluation from training

The evaluator is different from the trainer is some fundamental ways.
For instance it makes sense to run the evaluator with a batch size of 1
multiple times in the case of streaming data. While the trainer always
has a batch size and it rarely makes sense to run more than one.
Therefore what used to be Runner was separated into Evaluator and
Trainer.

@aleph7 Cool take on train/testing! I like it. Didn't dig too deep but the structure looks like it will work nicely! Once this is merged I'll get on adapting my training code to work on top of this.


Update contributors.yaml

i hope i'm doing this correctly, this is my first time. i registered for the Hacktoberfest open source event and just wanna learn and contribute.
Is your pull request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when ...
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Test plan
A good test plan should give instructions that someone else can easily follow.
Also add a link to your own Messenger experience for your reviewer to test the changes.
Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[v] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.
[ ] The title of my pull request is a short description of the requested changes.
[ ] I have added the label enhancement to my pull request.

@walidAITLHAJ I have reviewed your pull request and it has conflicts. You have to resolve these conflicts, then I will merge your pull request.


SP-1595 - Backport of PDI-12368 - Race condition using Pentaho 5 Kettle ...

...SDK and KettleLogStore.getAppender().getBuffer(...) (5.2 Suite)

@hudak
Please review.

@brosander implemented the parent fix for this ... please make sure your work is in line with his fix (details in http://jira.pentaho.com/browse/PDI-12368)

@dkincade
It is the same as https://github.com/pentaho/pentaho-kettle/pull/931 , just in one commit, not two. Should i send pr again?

Yes - and let's have @hudak review it since it reviewed the pone from @brosander
Thanks!


Anime/Manga Searches

Search for anime and manga through commands.
Made into an issue due to being an accepted item for the Beta 0.9 update.
Checklist:

[ ] Anime search command
[ ] Manga search command
[x] Search for a user's profile (Suggested by a user) | [Completed and added to Beta 0.9]
[ ] Link your MAL username with your data in Margarine

[ ] MAL profile searches error currently due to maintenance


examples: Version updates and cleanups

Bump skywalking and resolve version issues
Bump container/go versions for grpc
Bump container version for zstd
Run all verify jobs together (as disk space is less of an issue)
Use shared/echo container for lua
Use shared/echo container for wasm

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

@adisuissa this one would be good to land
im trying to clean up the examples before release - with the hope of reducing flakes on the release branch


Watch channel return immediately if revision is very large

I had encountered an unexpected scenario:

watch from, say rev:100, where the rev of etcd is < 10.
the watch response channel return immediately with empty response

Is it unexpected? I figure that it should wait until some change with larger happens.

@hongchaodeng etcd does not accept future revision. What is the use case for waiting for a future revision? I know etcd2 support that, but I really cannot think of a valid use case.

/cc @heyitsanthony

the watch response channel return immediately with empty response

Also, you should always do resp,ok:= <-ch. When there is an error, client will close the chan.

See this:
https://github.com/kubernetes/kubernetes/blob/c5260c8c7133d84f903d41a32ef6562c6c69e762/pkg/storage/etcd/etcd_watcher.go#L186
I want to get changes after specific rev:
	opts := etcd.WatcherOptions{
			Recursive:  w.list,
			AfterIndex: resourceVersion,
		}

@hongchaodeng the way I've been doing it is open a watcher wch on resourceVersion, drop the first event, then the rest of the stream is greater than resourceVersion

@magicwang-cn


Add pexpect and ptyprocess to buildInputs of aider-chat

Description of changes
I got an error that these dependencies were missing when attempting to build.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Handled in #341441

Thanks @taha-yassine I'll close this