Python error when using BlenderNC after applying material

Description of the problem:
I tried following first and second tutorials, loading GEBCO or the ssh netcdf file and as soon as I hit the "apply material" bottom, I get the errors in the attached screenshots. What fixes can I do? It seems like the blendernc is not compatible with blender 4.0?
Thanks

Please read the contributing-issues documentation before going any further:  https://blendernc.readthedocs.io/en/latest/help.html

OS: macOSventural 13.5.2
Blender Version: 4.0
BlenderNC Version: I am not sure, but I followed the tutorial page and clicked on getting the current version (so should have the latest version 0.4.0?
used BLENDERPY=/Users/asaberi1/Applications/Blender.app/Contents/Resources/4.0/python/bin/python3.10  this install it.

$ /Users/asaberi1/Applications/Blender.app/Contents/MacOS/Blender
Read prefs: "/Users/asaberi1/Library/Application Support/Blender/4.0/config/userpref.blend"
blendernc addon: Verbose is enabled
blendernc addon: Read in JSON settings from file
Registering to Change Defaults

Hello @Atousa-Saberi-NASA,
Unfortunately, BlenderNC is not yet supported in Blender 4. I'm aware of this issue and I'm working to fix it. I will recommend to install BlenderNC in a previous version of Blender.

OK, thank you!

Atousa Saberi, Ph.D.
(She/her)
<EMAIL_ADDRESS>Postdoctoral Fellow
Scientific Visualization Studio
NASA Goddard Space Flight Center

From: Josué Martínez Moreno @.>
Sent: Wednesday, February 21, 2024 11:39 AM
To: blendernc/blendernc @.>
Cc: Saberi, Atousa (GSFC-606.4)[NPP POST-DOC CONTRACT] @.>; Mention @.>
Subject: [EXTERNAL] [BULK] Re: [blendernc/blendernc] Python error when using BlenderNC after applying material (Issue #195)
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hello @Atousa-Saberi-NASAhttps://github.com/Atousa-Saberi-NASA,
Unfortunately, BlenderNC is not yet supported in Blender 4. I'm aware of this issue and I'm working to fix it. I will recommend to install BlenderNC in a previous version of Blender.
—
Reply to this email directly, view it on GitHubhttps://github.com/blendernc/blendernc/issues/195#issuecomment-1957240405, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3PQUL5BM2RWXDMXODQHZ7DYUYPNDAVCNFSM6AAAAABDTJWJFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJXGI2DANBQGU.
You are receiving this because you were mentioned.Message ID: @.***>

Is there any timeline for when you'll have it working with Blender 4.0 ? thanks

Atousa Saberi, Ph.D.
(She/her)
<EMAIL_ADDRESS>Postdoctoral Fellow
Scientific Visualization Studio
NASA Goddard Space Flight Center

From: Saberi, Atousa (GSFC-606.4)[NPP POST-DOC CONTRACT] @.>
Sent: Wednesday, February 21, 2024 12:06 PM
To: blendernc/blendernc @.>; blendernc/blendernc @.>
Cc: Mention @.>
Subject: Re: [EXTERNAL] [BULK] Re: [blendernc/blendernc] Python error when using BlenderNC after applying material (Issue #195)
OK, thank you!

Atousa Saberi, Ph.D.
(She/her)
<EMAIL_ADDRESS>Postdoctoral Fellow
Scientific Visualization Studio
NASA Goddard Space Flight Center

From: Josué Martínez Moreno @.>
Sent: Wednesday, February 21, 2024 11:39 AM
To: blendernc/blendernc @.>
Cc: Saberi, Atousa (GSFC-606.4)[NPP POST-DOC CONTRACT] @.>; Mention @.>
Subject: [EXTERNAL] [BULK] Re: [blendernc/blendernc] Python error when using BlenderNC after applying material (Issue #195)
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hello @Atousa-Saberi-NASAhttps://github.com/Atousa-Saberi-NASA,
Unfortunately, BlenderNC is not yet supported in Blender 4. I'm aware of this issue and I'm working to fix it. I will recommend to install BlenderNC in a previous version of Blender.
—
Reply to this email directly, view it on GitHubhttps://github.com/blendernc/blendernc/issues/195#issuecomment-1957240405, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3PQUL5BM2RWXDMXODQHZ7DYUYPNDAVCNFSM6AAAAABDTJWJFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJXGI2DANBQGU.
You are receiving this because you were mentioned.Message ID: @.***>

Hello @Atousa-Saberi-NASA,
I'm currently testing the PR #196 that should allow the use of the add-on in Blender >  4.0.

Hi Josue,
Thanks for letting me know.
I'd like to ask you a question. I tried it on earlier version of blender on my personal windows laptop.
I went through the first tutorial/example, and didn't get the pretty bumpy topographic look that you have in the example (even though I followed setting up exactly the same parameters listed in the example). Where do you set the exaggeration in the  topo? and how should I get the smooth look? thanks
[cid:69204ee2-5aae-45a6-a264-3b2c57302cbb]

Atousa Saberi, Ph.D.
(She/her)
<EMAIL_ADDRESS>Postdoctoral Fellow
Scientific Visualization Studio
NASA Goddard Space Flight Center

From: Josué Martínez Moreno @.>
Sent: Monday, March 11, 2024 11:30 AM
To: blendernc/blendernc @.>
Cc: Saberi, Atousa (GSFC-606.4)[NPP POST-DOC CONTRACT] @.>; Mention @.>
Subject: [EXTERNAL] [BULK] Re: [blendernc/blendernc] Python error when using BlenderNC after applying material (Issue #195)
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hello @Atousa-Saberi-NASAhttps://github.com/Atousa-Saberi-NASA,
I'm currently testing the PR #196https://github.com/blendernc/blendernc/pull/196 that should allow the use of the add-on in Blender > 4.0.
—
Reply to this email directly, view it on GitHubhttps://github.com/blendernc/blendernc/issues/195#issuecomment-1988720641, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3PQUL72FJUB2LURUUDXQ43YXXE25AVCNFSM6AAAAABDTJWJFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYG4ZDANRUGE.
You are receiving this because you were mentioned.Message ID: @.***>

Hello, if you navigate to the material of the sphere, you can increase the bump value in the bump node.  That will enhance the look of the topography.

Thanks for explaining. I tried playing with distance and strength value in the bump, but it didn't make a change. I was wondering if you could share a snapshot of your setting? or do you have the blender file examples? I tried to look for them in your github page, but wasn't able to locate them...

Atousa Saberi, Ph.D.
(She/her)
<EMAIL_ADDRESS>Postdoctoral Fellow
Scientific Visualization Studio
NASA Goddard Space Flight Center

From: Josué Martínez Moreno @.>
Sent: Tuesday, March 26, 2024 4:27 AM
To: blendernc/blendernc @.>
Cc: Saberi, Atousa (GSFC-606.4)[NPP POST-DOC CONTRACT] @.>; Mention @.>
Subject: [EXTERNAL] [BULK] Re: [blendernc/blendernc] Python error when using BlenderNC after applying material (Issue #195)
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hello, if you navigate to the material of the sphere, you can increase the bump value in the bump node. That will enhance the look of the topography.
—
Reply to this email directly, view it on GitHubhttps://github.com/blendernc/blendernc/issues/195#issuecomment-2019806926, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3PQUL7KBQQY7IJFGHHUVETY2EPPBAVCNFSM6AAAAABDTJWJFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJZHAYDMOJSGY.
You are receiving this because you were mentioned.Message ID: @.***>

Hello. How are you? I hope you are doing fine.  I installed the blenderNC add-on but when i tried to use it appeared that the xarray was not installed and there was no variables ob the netCDF file. It is blender 3.6.  Can you help me?

Hi Josue,
My name is Michel Tossaint working for the European Space Agency ESA.
I am using Blender 4.1 and get the same error as @Atousa-Saberi-NASA,
Is there any news on Blender 4.0 or 4.1 compatibility?
If not, which version of Blender do you recommend?
Thanks
Michel

Just for information, I get this error in Blender 4.1 for others to understand:

Hello, there is a branch in which I've been working to be implemented in Blender 4.1. We are aware of that error, the PR #196 solves that issue, but it seems that there are other issues that need to be fixed before being released.

Also interested in getting this solved. What is the best version of blender to use with blendernc in the meantime?

Unfortunately, I have been short in time to fully implement this, in the meantime, the all 3.X versions of blender are supported, so I will recommend to use one of those.

Hi Josue, Thanks for letting me know. I'd like to ask you a question. I tried it on earlier version of blender on my personal windows laptop. I went through the first tutorial/example, and didn't get the pretty bumpy topographic look that you have in the example (even though I followed setting up exactly the same parameters listed in the example). Where do you set the exaggeration in the topo? and how should I get the smooth look? thanks [cid:69204ee2-5aae-45a6-a264-3b2c57302cbb]
…
___________________________________ Atousa Saberi, Ph.D. (She/her<EMAIL_ADDRESS>Postdoctoral Fellow Scientific Visualization Studio NASA Goddard Space Flight Center
________________________________ From: Josué Martínez Moreno @.> Sent: Monday, March 11, 2024 11:30 AM To: blendernc/blendernc @.> Cc: Saberi, Atousa (GSFC-606.4)[NPP POST-DOC CONTRACT] @.>; Mention @.> Subject: [EXTERNAL] [BULK] Re: [blendernc/blendernc] Python error when using BlenderNC after applying material (Issue #195) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hello @Atousa-Saberi-NASAhttps://github.com/Atousa-Saberi-NASA, I'm currently testing the PR #196<#196> that should allow the use of the add-on in Blender > 4.0. — Reply to this email directly, view it on GitHub<#195 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3PQUL72FJUB2LURUUDXQ43YXXE25AVCNFSM6AAAAABDTJWJFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYG4ZDANRUGE. You are receiving this because you were mentioned.Message ID: @.***>

I too ran into this problem, I found that while changing the bump parameters helped the biggest change was changing the connection between the bump and the principled bsdf from clearcoat normal to normal. The clearcoat seems to make it smooth and glossy which we dont want! I have attached a picture of the shading editor to show all the connections.

Thanks for reporting this, probably the issue is when the nodes are automatically linked, I will raise an issue to dig more into this.

I finally managed to fix blendernc, the PR #196  adds support to blender > 4.0.0. Thus, I'm closing this issue.


Implement note priorities for MIDI/CV track

Implement the following note priorities:

lowest note
highest note
first note
last note

https://www.soundonsound.com/techniques/priorities-triggers has an overview.
Those modes are generally used for monophonic synths, however, if implemented as a stack, the same priorities could also be used for polyphonic use.

I have a work in progress implementation of this. However, as I presumed, the "lowest" and "highest" note modes are not too awesome for "extracting" the low/high note of a played chord. The note quickly changes when a chord is played before settling on the desired note. Anyway, the "last note" mode is nice and kind of what you'd expect. So it's a nice addition and will be released with 0.1.29.

Will be part of 0.1.29 release.


Please help me resolve the following issue in placement

My config.tcl file -
# User config
set ::env(DESIGN_NAME) radix2_pro

# Change if needed
set ::env(VERILOG_FILES) [glob $::env(DESIGN_DIR)/src/*.v]

# Fill this

set ::env(DESIGN_IS_CORE) "0"
set ::env(CLOCK_PORT) "clk"
set ::env(CLOCK_NET) " "

set ::env(CLOCK_PERIOD) "90.0"

#test
set ::env(FP_CORE_UTIL) "50"
set ::env(PL_TARGET_DENSITY) "0.225"
set ::env(PL_BASIC_PLACEMENT) "0"
set ::env(GLB_RESIZER_TIMING_OPTIMIZATIONS) "0"
set ::env(FP_SIZING) "absolute"
set ::env(DIE_AREA) "0.0 0.0 1000.000 1000.000"
#set ::env(CORE_AREA) "15.00 25.00 900.0 900.00"
set ::env(CORE_AREA) "11.00 16.00 900.0 900.00"
set ::env(DIODE_INSERTION_STRATEGY) "4"
set ::env(GLB_RT_MAXLAYER) "5"

set ::env(FP_PDN_AUTO_ADJUST) "0"

#set ::env(VDD_NETS) [list {vccd1}]
#set ::env(GND_NETS) [list {vssd1}]

set filename $::env(DESIGN_DIR)/$::env(PDK)_$::env(STD_CELL_LIBRARY)_config.tcl
if { [file exists $filename] == 1} {
source $filename
}

Utilization exceeds 100%
Error : or_replace.tcl

My system -
Debian OS, 16GB RAM, 1TB HDD

Your design is too big to fit in the area given.  You either need more area or fewer gates.

Even though after increasing CORE_AREA and DIE_AREA
I am getting some other issue.

Config File:
# User config
set ::env(DESIGN_NAME) radix2_pro

# Change if needed
set ::env(VERILOG_FILES) [glob $::env(DESIGN_DIR)/src/*.v]

# Fill this

#clk port #CTS error[INFO]: Running Clock Tree Synthesis...
#[ERROR CTS-0056] Error when finding -clk_nets in DB.
set ::env(DESIGN_IS_CORE) "0"
set ::env(CLOCK_PORT) "clk"

set ::env(CLOCK_PERIOD) "90.0"

#test
set ::env(FP_CORE_UTIL) "30"
set ::env(GLB_RT_ALLOW_CONGESTION) "1"
set ::env(PL_BASIC_PLACEMENT) "0"
set ::env(GLB_RESIZER_TIMING_OPTIMIZATIONS) "0"
set ::env(FP_SIZING) "absolute"
set ::env(DIE_AREA) "0.0 0.0 6937.49 6948.21"
#set ::env(CORE_AREA) "15.00 25.00 900.0 900.00"
set ::env(CORE_AREA) "5.52 10.88 6931.70 6935.04"
set ::env(DIODE_INSERTION_STRATEGY) "4"
set ::env(GLB_RT_MAXLAYER) "4"

set ::env(FP_PDN_AUTO_ADJUST) "0"

#set ::env(VDD_NETS) [list {vccd1}]
#set ::env(GND_NETS) [list {vssd1}]

set filename $::env(DESIGN_DIR)/$::env(PDK)_$::env(STD_CELL_LIBRARY)_config.tcl
if { [file exists $filename] == 1} {
source $filename
}

Is there any new contain to add.

Please attached a test case reproducing the problem

@maliberty
I don't have much Idea which test case is producing particular problem.

You seem to be confusing line in your config file with a test case.  I need something I can run that produces the same output as you are showing above.

@sumanthnimmakayala
Read and fill the issue template properly with all details and open the issue again.
Its important to know which OpenLane version currently you're working.


Random location

Can choose random location each time start docker? Please help me

Hi!
You can wrap docker invocation into some script to list available locations, select one random and run docker again with selected randomly location. Or you can add such feature into application itself, forking it and implementing feature on your own.
From my side I don't plan such feature because random location doesn't make much sense for normal usage scenario: it doesn't make sense for a user to forward their traffic over random continent, needlessly spoiling latency and throughput.


Swift 4 support

Swift 4 is supported by Alamofire 4.5.

But why is .swift-version set at 3.2?

https://github.com/Alamofire/Alamofire/blob/master/.swift-version

^ This note regarding Swift 3.2 is resolved now.


Add functions to clone private repository and ssh url

Description

Add github clone private repository authentication
Add github clone ssh link
Update README.md

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Refactoring, Maintenance
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

@soimkim #157 의 private repository clone 수정과 더불어, ssh url도 clone이 가능하도록 기능을 추가하여 새로 PR 합니다.


Question: For offline installation, are the manifest.json files no longer required?

Hi,
In v2.2.4 the release holds *-manifest.spdx.json files for each platform. Since v2.2.5 they are not deployed as part of a release. Are they no longer required as part of the release itself?
KR, Richard

Adding some more detail here, the asset files in v2.2.4 included the SBOM's for the SBOM tool. The asset files in v2.2.5 and v2.2.6 do not. Specifically, the following files did not get included in the release:

linux-x64-manifest.spdx.json
osx-x64-manifest.spdx.json
win-x64-manifest.spdx.json

@richardzaat, this was unintentional. The layout of our build pipeline changed, and the release pipeline didn't reflect that change. I've manually added the files to releases 2.2.5 and 2.2.6, and have a task to bring the release pipeline back into sync. Hopefully we'll have it in place by the time we release 2.2.7.

We have updated our release pipeline to automatically include manifest files in the release assets, so this should not be an issue going forward.


How to set up databases for tests?

I'm still new to Django, and I'm working through 2 Scoops.  I get this error when I try to run tests:
Got an error creating the test database: permission denied to create database
I found this line in /tests/engine.py : postgres_user = hitchpostgres.PostgresUser("djtasks", "password"), so I created the postgres user djtasks and gave it permission to create databases, but I still get the error.

/tests/engine.py contains the code for hitch test which are an entirely separate and new way of testing django code. It won't be run unless you tell it to run (using hitch test .) so it won't be causing your current problem.
It looks like you're trying to run the standard django tests. The user that django will be trying to use that needs the create database permission will be in your django settings file. If it's not specified I think it tries to use the username that you are logged in as.


使用glm-2b时候，跟随readme提供的例子，得到很糟糕的输出

代码如下:
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-2b", trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained("THUDM/glm-2b", trust_remote_code=True)
model = model.half().cuda()
model.eval()
inputs = tokenizer("Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai.", return_tensors="pt")
inputs = tokenizer.build_inputs_for_generation(inputs, max_gen_length=512).to('cuda')
outputs = model.generate(**inputs, max_length=512, eos_token_id=tokenizer.eop_token_id)
print(tokenizer.decode(outputs[0].tolist()))
得到输出:
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Some weights of the model checkpoint at THUDM/glm-2b were not used when initializing GLMForConditionalGeneration: ['dense.weight', 'out_proj.bias', 'out_proj.weight', 'dense.bias']

This IS expected if you are initializing GLMForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing GLMForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's attention_mask to obtain reliable results.
Setting pad_token_id to eos_token_id:50258 for open-end generation.
[CLS] Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai.<|endoftext|> <|startofpiece|> 111111111 each1 when1111111 when11 each1 each1 each1 each1 each1 each1 each1 each1 each1 each1 each1 each1 each1 each111111111111111111111111111111111111111111111111111111 when11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

同样的问题，请问解决了吗

同问


Display rating feature

Description
users can now see their average rating on their profile
Issue Resolved
fixes #94
Screenshots

Checklist
Please make sure to review the following before submitting your PR:

[x] I have read the contribution guidelines.
[x] I have read the code of conduct.
[x] I have reviewed my submission in detail.

Currently I think this would display the floor value of the average, display it upto 2 decimal places


Refactor careeropportunity

TODO:

Get kravspec (@sklirg)
Involve prokom in design phase (on wait)
Rework code (in progress)

Features:

[ ] Tags (location, type, maybe language/various)
[ ] Integration to Django rest framework
[ ] Redesign

Maybe:

[ ] Redux/Angular 2
[ ] Search

New field: type, location.

We have a project for this now. See https://github.com/dotKom/onlineweb4/projects/2


Add nullability annotations

Well... what the title says. Should be nice for those who still support iOS7!
Fixes #25

great, thanks!


Uses wrong RFCOMM channel after first opening

Bascially, I open the server, pair it up with the app via RFCOMM channel 1. Pairs up, works great. However, after closing the app via either suspending (ctrl+z) or quitting (ctrl+), the next time it opens it wants to use RFCOMM channel +1; so the second time I opened it wanted 2; then 3; then 4. When I opened the app on the phone it still acted as though it was connected, letting me bypass the connection screen (in fact I can't reconnect?) but not functioning correctly. It seems like the server needs to always open to use the same RFCOMM channel, or the app needs to recheck and sync with the server every time it is opened.
Also, simply killing the app via ctrl+c doesn't work, not sure why or if that is relevant.
system info:
System:
Kernel: 4.15.0-45-generic x86_64 bits: 64 compiler: gcc
v: 7.3.0 Desktop: Cinnamon 4.0.9 Distro: Linux Mint 19.1 Tessa
base: Ubuntu 18.04 bionic

Hi there, sorry for the late reply. I believe you are hitting a red herring. No where in the code do I specify the channel. It's a side affect of opening a channel and not closing it. I would suggest you look for the process and kill it before attempting to start another one. That should fix the issue


Select Mode of Points Layer broken in 4 dimensions

🐛 Bug Report
When displaying points 4D with the 3D view, selecting the "Select" mode of the Points layer then dragging the mouse across the canvas (not moving point, just dragging the cursor as if to create the usual selection bounding box) does nothing and raises an Assertion error. If a point is clicked instead, it can be moved normally without any error.
Here's the error I get
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
File ~/anaconda3/envs/napari-dev/lib/python3.9/site-packages/superqt/utils/_throttler.py:249, in ThrottledCallable._set_future_result(self=<superqt.utils._throttler.ThrottledCallable object>)
    248 def _set_future_result(self):
--> 249     result = self._func(*self._args[: self._max_args], **self._kwargs)
        self._func = <bound method VispyCanvas._on_mouse_move of <napari._vispy.canvas.VispyCanvas object at 0x7ff6c544fdf0>>
        self = <superqt.utils._throttler.ThrottledCallable object at 0x7ff6ac3160d0>
        self._args[: self._max_args] = (<MouseEvent blocked=False button=1 buttons=[1, 1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x7ff6c545c310> pos=[527 398] press_event=MouseEvent source=<NapariSceneCanvas (PyQt5) at 0x7ff6c5459070> sources=[<NapariSceneCanvas (PyQt5) at 0x7ff6c5459070>] time=1708521467.3701262 type=mouse_move>,)
        self._kwargs = {}
        self._args = (<MouseEvent blocked=False button=1 buttons=[1, 1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x7ff6c545c310> pos=[527 398] press_event=MouseEvent source=<NapariSceneCanvas (PyQt5) at 0x7ff6c5459070> sources=[<NapariSceneCanvas (PyQt5) at 0x7ff6c5459070>] time=1708521467.3701262 type=mouse_move>,)
        self._max_args = 1
    250     self._future.set_result(result)

File /napari/napari/_vispy/canvas.py:448, in VispyCanvas._on_mouse_move(self=<napari._vispy.canvas.VispyCanvas object>, event=<MouseEvent blocked=False button=1 buttons=[1, 1...459070>] time=1708521467.3701262 type=mouse_move>)
    436 def _on_mouse_move(self, event: MouseEvent) -> None:
    437     """Called whenever mouse moves over canvas.
    438
    439     Parameters
   (...)
    446     None
    447     """
--> 448     self._process_mouse_event(mouse_move_callbacks, event)
        event = <MouseEvent blocked=False button=1 buttons=[1, 1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x7ff6c545c310> pos=[527 398] press_event=MouseEvent source=<NapariSceneCanvas (PyQt5) at 0x7ff6c5459070> sources=[<NapariSceneCanvas (PyQt5) at 0x7ff6c5459070>] time=1708521467.3701262 type=mouse_move>
        self = <napari._vispy.canvas.VispyCanvas object at 0x7ff6c544fdf0>

File /napari/napari/_vispy/canvas.py:409, in VispyCanvas._process_mouse_event(self=<napari._vispy.canvas.VispyCanvas object>, mouse_callbacks=<function mouse_move_callbacks>, event=<ReadOnlyWrapper at 0x7ff69675b680 for MouseEvent>)
    407 layer = self.viewer.layers.selection.active
    408 if layer is not None:
--> 409     mouse_callbacks(layer, event)
        event = <ReadOnlyWrapper at 0x7ff69675b680 for MouseEvent at 0x7ff6954096d0>
        layer = <Points layer 'points' at 0x7ff6eb790c40>
        mouse_callbacks = <function mouse_move_callbacks at 0x7ff6debac940>

File /napari/napari/utils/interactions.py:178, in mouse_move_callbacks(obj=<Points layer 'points'>, event=<ReadOnlyWrapper at 0x7ff69675b680 for MouseEvent>)
    175 obj._persisted_mouse_event[gen].__wrapped__ = event
    176 try:
    177     # try to advance the generator
--> 178     next(gen)
        gen = <generator object select at 0x7ff6978be890>
    179 except StopIteration:
    180     # If done deleted the generator and stored event
    181     del obj._mouse_drag_gen[func]

File /napari/napari/layers/points/_points_mouse_bindings.py:92, in select(layer=<Points layer 'points'>, event=<ReadOnlyWrapper at 0x7ff696756100 for ReadOnlyWrapper>)
     89     layer._drag_box = np.array([layer._drag_start, coord])
     91     # update the drag up and normal vectors on the layer
---> 92     _update_drag_vectors_from_event(layer=layer, event=event)
        layer = <Points layer 'points' at 0x7ff6eb790c40>
        event = <ReadOnlyWrapper at 0x7ff696756100 for ReadOnlyWrapper at 0x7ff69675b680>
     94     layer._set_highlight()
     95 yield

File /napari/napari/layers/points/_points_mouse_bindings.py:195, in _update_drag_vectors_from_event(layer=<Points layer 'points'>, event=<ReadOnlyWrapper at 0x7ff696756100 for ReadOnlyWrapper>)
    191 if n_display == 3:
    192     # if in 3D, set the drag normal and up directions
    193     # get the indices of the displayed dimensions
    194     ndim_world = len(event.position)
--> 195     layer_dims_displayed = layer._world_to_layer_dims(
        layer = <Points layer 'points' at 0x7ff6eb790c40>
        ndim_world = 4
        event = <ReadOnlyWrapper at 0x7ff696756100 for ReadOnlyWrapper at 0x7ff69675b680>
    196         world_dims=event.dims_displayed, ndim_world=ndim_world
    197     )
    199     # get the view direction in displayed data coordinates
    200     layer._drag_normal = layer._world_to_displayed_data_ray(
    201         event.view_direction, layer_dims_displayed
    202     )

File /napari/napari/layers/base/base.py:1640, in Layer._world_to_layer_dims(self=<Points layer 'points'>, world_dims=[1, 2, 3], ndim_world=4)
   1580 def _world_to_layer_dims(
   1581     self, *, world_dims: npt.NDArray, ndim_world: int
   1582 ) -> np.ndarray:
   1583     """Map world dimensions to layer dimensions while maintaining order.
   1584
   1585     This is used to map dimensions from the full world space defined by ``Dims``
   (...)
   1638         The corresponding layer dimensions with the same ordering as the given world dimensions.
   1639     """
-> 1640     return self._world_to_layer_dims_impl(
        self = <Points layer 'points' at 0x7ff6eb790c40>
        world_dims = [1, 2, 3]
        ndim_world = 4
   1641         world_dims, ndim_world, self.ndim
   1642     )

File /napari/napari/layers/base/base.py:1652, in Layer._world_to_layer_dims_impl(world_dims=<class 'numpy.ndarray'> (3,) int64, ndim_world=4, ndim=4)
   1648 """
   1649 Static for ease of testing
   1650 """
   1651 world_dims = np.asarray(world_dims)
-> 1652 assert world_dims.min() == 0
        world_dims = <class 'numpy.ndarray'> (3,) int64
   1653 assert world_dims.max() == len(world_dims) - 1
   1654 assert world_dims.ndim == 1

AssertionError:

When the cursor is not dragged but clicked, an Axis error raises (axisa: axis -1 is out of bounds for array of dimension 0). I won't show the error stack as it is quite long.
💡 Steps to Reproduce

Display points in 4D. You can use the following snippet:

import napari
import numpy as np

np.random.seed(0)

data = np.random.random((2, 100,100,100))
points = np.random.random((3, 4)) * 100
points[:,0] = 0

viewer = napari.Viewer(ndisplay=3)
viewer.add_image(data)
viewer.add_points(points, size=10)
napari.run()

Go in 3D view mode
Make sure the "Select" mode of the Points Layer is selected. Click and drag your cursor across the canvas.

💡 Expected Behavior
Normally, the action described should draw a thin blue rectangular bounding box that allows selecting the points.
🌎 Environment
napari: 0.1.dev3244+g0747da5
Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.31
System: Ubuntu 20.04.6 LTS
Python: 3.9.0 (default, Nov 15 2020, 14:28:56)  [GCC 7.3.0]
Qt: 5.15.2
PyQt5: 5.15.9
NumPy: 1.24.2
SciPy: 1.9.1
Dask: 2023.4.0
VisPy: 0.14.1
magicgui: 0.7.2
superqt: 0.6.1
in-n-out: 0.1.7
app-model: 0.2.4
npe2: 0.7.4
OpenGL:

GL version:  4.6 (Compatibility Profile) Mesa 21.2.6
MAX_TEXTURE_SIZE: 16384
GL_MAX_3D_TEXTURE_SIZE: 2048

Settings path:

/home/jvanaret/.config/napari/napari-dev_ebc9c46b35c7ad556420cc56947aae90c9e81949/settings.yaml
Plugins:
napari: 0.1.dev3244+g0747da5 (81 contributions)
napari-console: 0.0.9 (0 contributions)
napari-svg: 0.1.10 (2 contributions)
napari-threedee: 0.0.12 (28 contributions)

💡 Additional Context
No response

Hi there, gave this a check and seems like changing the logic at _world_to_layer_dims_impl https://github.com/napari/napari/blob/1ee84b3677f8bb6b6cdd4a97754022003043c180/napari/layers/base/base.py#L1644-L1658  for something like:
    @staticmethod
    def _world_to_layer_dims_impl(
        world_dims: npt.NDArray, ndim_world: int, ndim: int
    ) -> npt.NDArray:
        """
        Static for ease of testing
        """
        offset = ndim_world - ndim
        order = np.array(world_dims)
        if offset == 0:
            return order
        if offset < 0:
            return np.concatenate(np.arange(-offset), order - offset)

        return order[order >= offset] - offset

helps.
I think the relevant PR where the logic changed is #6210 where an attempt to simplify the logic for the _world_to_layer_dims logic was done (while checking a fix for https://github.com/napari/napari/issues/6205). Maybe that logic simplification didn't took into account the points layer selection case handling on that method? 🤔

Sorry this fell through the cracks.
Thanks for looking into this @dalthviz
@brisvag do you have any thoughts on this one?


Pressing ESC + j or k will equal to Alt + j or k when using Neovim in Windows Terminal

Windows Terminal version
1.20.11381.0
Windows build number
10.0.22631.0
Other Software
NVIM v0.10.0
Lazyvim v12.13.0
Steps to reproduce
Press esc + j or k at the same time in Insert Mode of Neovim.
Expected Behavior
Back to Normal Mode, and move cursor to next or previous line.
Actual Behavior
Swap current line with next or previous line.
Another issue in other repository that may be helpful: https://github.com/LunarVim/LunarVim/issues/1857

Thanks for filing! We're going to mark this as a /dup of #13792. When it's completed, that'll allow you to work around this issue by encoding ESC+J/K differently.


Add support for tags in Graphite reporter

Statsd doesn't support tags as of yet.
Need to convert graphite reporter to carbon and understand how to report time and counts.
https://graphite.readthedocs.io/en/latest/tags.html

https://github.com/statsd/statsd/issues/619
Current solution might work with a different separator - ,, :,;


Add ` to enquoting characters

When selecting text you can currently type a variety of characters (eg ', ", (, ...etc) to have that selected text "quoted". It would be convenient to have this also work for ` (for use in mysql queries).

thanks for the suggestion


OV2640 will not work at all...SPI Interface Error (sometimes)

I have tried using both an Arduino Mega and an Arduino UNO to interface the Arducam (OV2640) and both have proven to not work. The farthest I got was with the Mega. I would open the Host application with everything configured and pinned properly and it would say "COM is open!" then "OV2640 detected" then "SPI Interface Error......".
Currently all I can get is "COM is open!" and that's it. I have tried contacting support but they have since stopped responding to my emails for some reason; Tried a completely different OV2640 unit; Tried interfacing with the Arduino UNO R3. All avenues of troubleshooting seem to not be yielding any results. I NEED HELP!!!
My pin configuration with the Mega2560 is as follows:
CS -> Communication pin10
MOSI -> ICSP pin4
MISO -> ICSP pin1
SCK -> ICSP pin3
GND -> GND
VCC -> 5V
SDA -> A5
SCL -> A4
A little background on what im doing is I need to take a picture using the camera. I then need to take the RGB values of each individual picture and create some conditional logic with it. Im running out of time and Im truly baffled as to why this has been so hard to setup as everybody else seems to be having such an easy time using the ArduCam.

After running this code (it fails) I am able to run the Host application and get to this:
COM is open!
ArduCAM Start!
SPI Interface Error!
SPI Interface Error!
SPI Interface Error!
....................
....................

Code I ran:
// ArduCAM Mini demo (C)2017 Lee
// Web: http://www.ArduCAM.com
// This program is a demo of how to use most of the functions
// of the library with ArduCAM Mini camera, and can run on any Arduino platform.
// This demo was made for ArduCAM_Mini_5MP_Plus.
// It needs to be used in combination with PC software.
// It can take photo continuously as video streaming.
//
// The demo sketch will do the following tasks:
// 1. Set the camera to JPEG output mode.
// 2. Read data from Serial port and deal with it
// 3. If receive 0x00-0x08,the resolution will be changed.
// 4. If receive 0x10,camera will capture a JPEG photo and buffer the image to FIFO.Then write datas to Serial port.
// 5. If receive 0x20,camera will capture JPEG photo and write datas continuously.Stop when receive 0x21.
// 6. If receive 0x30,camera will capture a BMP  photo and buffer the image to FIFO.Then write datas to Serial port.
// 7. If receive 0x11 ,set camera to JPEG output mode.
// 8. If receive 0x31 ,set camera to BMP  output mode.
// This program requires the ArduCAM V4.0.0 (or later) library and ArduCAM_Mini_5MP_Plus
// and use Arduino IDE 1.6.8 compiler or above
#include <Wire.h>
#include <ArduCAM.h>
#include <SPI.h>
#include "memorysaver.h"
//This demo can only work on OV2640_MINI_2MP or OV5642_MINI_5MP or OV5642_MINI_5MP_BIT_ROTATION_FIXED platform.
#if !(defined OV5642_MINI_5MP || defined OV5642_MINI_5MP_BIT_ROTATION_FIXED || defined OV2640_MINI_2MP)
#error Please select the hardware platform and camera module in the ../libraries/ArduCAM/memorysaver.h file
#endif
#define BMPIMAGEOFFSET 66
const char bmp_header[BMPIMAGEOFFSET] PROGMEM =
{
0x42, 0x4D, 0x36, 0x58, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x28, 0x00,
0x00, 0x00, 0x40, 0x01, 0x00, 0x00, 0xF0, 0x00, 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x03, 0x00,
0x00, 0x00, 0x00, 0x58, 0x02, 0x00, 0xC4, 0x0E, 0x00, 0x00, 0xC4, 0x0E, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF8, 0x00, 0x00, 0xE0, 0x07, 0x00, 0x00, 0x1F, 0x00,
0x00, 0x00
};
// set pin 7 as the slave select for the digital pot:
const int CS = 7;
bool is_header = false;
int mode = 0;
uint8_t start_capture = 0;
#if defined (OV2640_MINI_2MP)
ArduCAM myCAM( OV2640, CS );
#else
ArduCAM myCAM( OV5642, CS );
#endif
uint8_t read_fifo_burst(ArduCAM myCAM);
void setup() {
// put your setup code here, to run once:
uint8_t vid, pid;
uint8_t temp;
#if defined(SAM3X8E)
Wire1.begin();
Serial.begin(115200);
#else
Wire.begin();
Serial.begin(921600);
#endif
Serial.println(F("ACK CMD ArduCAM Start!"));
// set the CS as an output:
pinMode(CS, OUTPUT);
// initialize SPI:
SPI.begin();
while(1){
//Check if the ArduCAM SPI bus is OK
myCAM.write_reg(ARDUCHIP_TEST1, 0x55);
temp = myCAM.read_reg(ARDUCHIP_TEST1);
if (temp != 0x55){
Serial.println(F("ACK CMD SPI interface Error!"));
delay(1000);continue;
}else{
Serial.println(F("ACK CMD SPI interface OK."));break;
}
}
#if defined (OV2640_MINI_2MP)
while(1){
//Check if the camera module type is OV2640
myCAM.wrSensorReg8_8(0xff, 0x01);
myCAM.rdSensorReg8_8(OV2640_CHIPID_HIGH, &vid);
myCAM.rdSensorReg8_8(OV2640_CHIPID_LOW, &pid);
if ((vid != 0x26 ) && (( pid != 0x41 ) || ( pid != 0x42 ))){
Serial.println(F("ACK CMD Can't find OV2640 module!"));
delay(1000);continue;
}
else{
Serial.println(F("ACK CMD OV2640 detected."));break;
}
}
#else
while(1){
//Check if the camera module type is OV5642
myCAM.wrSensorReg16_8(0xff, 0x01);
myCAM.rdSensorReg16_8(OV5642_CHIPID_HIGH, &vid);
myCAM.rdSensorReg16_8(OV5642_CHIPID_LOW, &pid);
if((vid != 0x56) || (pid != 0x42)){
Serial.println(F("ACK CMD Can't find OV5642 module!"));
delay(1000);continue;
}
else{
Serial.println(F("ACK CMD OV5642 detected."));break;
}
}
#endif
//Change to JPEG capture mode and initialize the OV5642 module
myCAM.set_format(JPEG);
myCAM.InitCAM();
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_320x240);
#else
myCAM.write_reg(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);   //VSYNC is active HIGH
myCAM.OV5642_set_JPEG_size(OV5642_320x240);
#endif
delay(1000);
myCAM.clear_fifo_flag();
#if !(defined (OV2640_MINI_2MP))
myCAM.write_reg(ARDUCHIP_FRAMES,0x00);
#endif
}
void loop() {
// put your main code here, to run repeatedly:
uint8_t temp = 0xff, temp_last = 0;
bool is_header = false;
if (Serial.available())
{
temp = Serial.read();
switch (temp)
{
case 0:
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_160x120);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_160x120"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_320x240);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_320x240"));
#endif
temp = 0xff;
break;
case 1:
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_176x144);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_176x144"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_640x480);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_640x480"));
#endif
temp = 0xff;
break;
case 2:
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_320x240);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_320x240"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_1024x768);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_1024x768"));
#endif
temp = 0xff;
break;
case 3:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_352x288);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_352x288"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_1280x960);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_1280x960"));
#endif
break;
case 4:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_640x480);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_640x480"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_1600x1200);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_1600x1200"));
#endif
break;
case 5:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_800x600);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_800x600"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_2048x1536);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_2048x1536"));
#endif
break;
case 6:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_1024x768);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_1024x768"));
#else
myCAM.OV5642_set_JPEG_size(OV5642_2592x1944);delay(1000);
Serial.println(F("ACK CMD switch to OV5642_2592x1944"));
#endif
break;
#if defined (OV2640_MINI_2MP)
case 7:
temp = 0xff;
myCAM.OV2640_set_JPEG_size(OV2640_1280x1024);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_1280x1024"));
break;
case 8:
temp = 0xff;
myCAM.OV2640_set_JPEG_size(OV2640_1600x1200);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_1600x1200"));
break;
#endif
case 0x10:
mode = 1;
temp = 0xff;
start_capture = 1;
Serial.println(F("ACK CMD CAM start single shoot."));
break;
case 0x11:
temp = 0xff;
myCAM.set_format(JPEG);
myCAM.InitCAM();
#if !(defined (OV2640_MINI_2MP))
myCAM.set_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
#endif
break;
case 0x20:
mode = 2;
temp = 0xff;
start_capture = 2;
Serial.println(F("ACK CMD CAM start video streaming."));
break;
case 0x30:
mode = 3;
temp = 0xff;
start_capture = 3;
Serial.println(F("CAM start single shoot."));
break;
case 0x31:
temp = 0xff;
myCAM.set_format(BMP);
myCAM.InitCAM();
#if !(defined (OV2640_MINI_2MP))
myCAM.clear_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
#endif
myCAM.wrSensorReg16_8(0x3818, 0x81);
myCAM.wrSensorReg16_8(0x3621, 0xA7);
break;
default:
break;
}
}
if (mode == 1)
{
if (start_capture == 1)
{
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
myCAM.start_capture();
start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
Serial.println(F("ACK CMD CAM Capture Done."));
read_fifo_burst(myCAM);
//Clear the capture done flag
myCAM.clear_fifo_flag();
}
}
else if (mode == 2)
{
while (1)
{
temp = Serial.read();
if (temp == 0x21)
{
start_capture = 0;
mode = 0;
Serial.println(F("ACK CMD CAM stop video streaming."));
break;
}
if (start_capture == 2)
{
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
myCAM.start_capture();
start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
uint32_t length = 0;
length = myCAM.read_fifo_length();
if ((length >= MAX_FIFO_SIZE) | (length == 0))
{
myCAM.clear_fifo_flag();
start_capture = 2;
continue;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
temp =  SPI.transfer(0x00);
length --;
while ( length-- )
{
temp_last = temp;
temp =  SPI.transfer(0x00);
if (is_header == true)
{
Serial.write(temp);
}
else if ((temp == 0xD8) & (temp_last == 0xFF))
{
is_header = true;
Serial.println(F("ACK IMG"));
Serial.write(temp_last);
Serial.write(temp);
}
if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
break;
delayMicroseconds(15);
}
myCAM.CS_HIGH();
myCAM.clear_fifo_flag();
start_capture = 2;
is_header = false;
}
}
}
else if (mode == 3)
{
if (start_capture == 3)
{
//Flush the FIFO
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
myCAM.start_capture();
start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
Serial.println(F("ACK CMD CAM Capture Done."));
uint8_t temp, temp_last;
uint32_t length = 0;
length = myCAM.read_fifo_length();
if (length >= MAX_FIFO_SIZE )
{
Serial.println(F("ACK CMD Over size."));
myCAM.clear_fifo_flag();
return;
}
if (length == 0 ) //0 kb
{
Serial.println(F("ACK CMD Size is 0."));
myCAM.clear_fifo_flag();
return;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
Serial.write(0xFF);
Serial.write(0xAA);
for (temp = 0; temp < BMPIMAGEOFFSET; temp++)
{
  Serial.write(pgm_read_byte(&bmp_header[temp]));
}
SPI.transfer(0x00);
char VH, VL;
int i = 0, j = 0;
for (i = 0; i < 240; i++)
{
  for (j = 0; j < 320; j++)
  {
    VH = SPI.transfer(0x00);;
    VL = SPI.transfer(0x00);;
    Serial.write(VL);
    delayMicroseconds(12);
    Serial.write(VH);
    delayMicroseconds(12);
  }
}
Serial.write(0xBB);
Serial.write(0xCC);
myCAM.CS_HIGH();
//Clear the capture done flag
myCAM.clear_fifo_flag();

}
}
}
uint8_t read_fifo_burst(ArduCAM myCAM)
{
uint8_t temp = 0, temp_last = 0;
uint32_t length = 0;
length = myCAM.read_fifo_length();
Serial.println(length, DEC);
if (length >= MAX_FIFO_SIZE) //512 kb
{
Serial.println(F("Over size."));
return 0;
}
if (length == 0 ) //0 kb
{
Serial.println(F("Size is 0."));
return 0;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
temp =  SPI.transfer(0x00);
length --;
while ( length-- )
{
temp_last = temp;
temp =  SPI.transfer(0x00);
if (is_header == true)
{
Serial.write(temp);
}
else if ((temp == 0xD8) & (temp_last == 0xFF))
{
is_header = true;
Serial.println(F("ACK IMG"));
Serial.write(temp_last);
Serial.write(temp);
}
if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
break;
delayMicroseconds(15);
}
myCAM.CS_HIGH();
is_header = false;
return 1;
}

@coolb2200
Hi,
the SCk is pin 3 and in the code you have set the CS to 7 instead of 10.
Pease check it again.
Let us know if you need more help.
Regards,
ArduCAM support team.

I guess my problem is that I can't even get the Host app to do anything with the camera much less, get the code to even work. I changed the CS pin to both 7 and 10, swapping the pin configuration for both cases.
I can't even get the host app to work.
Here is a picture of my hardware setup:

Hello?

@coolb2200
Hi,
Don't worry and we will try our best to help you.
Can your camera work on other platform?
Please attach us some pictures about how do you use the host V2 software.
Let us know if you need more help.
Regards,
ArduCAM support team.

Well I got it to somehow "work" on the Arduino UNO R3.
How would I go about getting individual RGB values from each pixel?

Also, why can't I just run the example code and have it work without using the Host app? Isn't your code supposed to work after uploading it? I shouldn't have to keep using the Host app.
I have been trying to get your code to work but every time I try to use it, it spits back a bunch of errors.

@coolb2200
Hi,
Sorry for our late reply.

@coolb2200
Hi,
The hardware connnection you have attached is right.
Do you have our adapter board?We suggest you use
our adapter board which will ensure the connection is
stable.
About getting RGB values form each pixel,please refer to this code :
//Read 320x240x2 byte from FIFO
//Save as RGB565 bmp format
for (i = 0; i < 240; i++)
for (j = 0; j < 320; j++)
{
VH = myCAM.read_fifo();
VL = myCAM.read_fifo();
buf[k++] = VL;
buf[k++] = VH;
#if defined(ESP8266)
yield();
#endif
//Write image data to bufer if not full
if (k >= 256)
{
//Write 256 bytes image data to file from buffer
outFile.write(buf, 256);
k = 0;
}
}
Let us know if you need more help.
Regards,
ArduCAM support team.

Thank you! Should that code just work? I've yet to test it.
Are you talking about one of these? https://smile.amazon.com/Arducam-Multi-Cameras-Adapter-Board/dp/B01735GYWC?sa-no-redirect=1
Is there anyway I could have code JUST for the OV2640 and nothing else? I notice that a lot of your example code accommodates for both the OV2640 and the OV5640. Do you have live streaming code for JUST the OV2640? It's a lot to go through and the API doesn't really help me all that much.

@coolb2200
Hi,
Yes,if you want to use OV2640,you just should enable OV2640_MINI_2MP and disable others
in the memorysaver.h file.Then you can use our examples in MINI folder for OV2640.
IF you want to only have OV2640,you can delete other sensor by yourself.
This link is our demo .
https://github.com/ArduCAM/Arduino/tree/master/ArduCAM/examples/mini
Let us know if you need more help.
Regards,
ArduCAM suport team.

Thank you!
Anyway I could just run this code without having to use the host app every time? Is that possible?

@coolb2200
Hi,
Yes,
after you get the RGB datas,you can write it to your LCD screen or other device.
After  start capture and capture down, you can read the RGB data from the FIFO.
Let us know if you need more help.
Regards,
ArduCAM support team.

I can't read the RGB values with the given code. I keep getting an "invalid type" error when adding the code in. What is wrong with the code?
// 1. Set the camera to JPEG output mode.
// 2. Read data from Serial port and deal with it
// 3. If receive 0x00-0x08,the resolution will be changed.
// 4. If receive 0x10,camera will capture a JPEG photo and buffer the image to FIFO.Then write datas to Serial port.
// 5. If receive 0x20,camera will capture JPEG photo and write datas continuously.Stop when receive 0x21.
// 6. If receive 0x30,camera will capture a BMP  photo and buffer the image to FIFO.Then write datas to Serial port.
// 7. If receive 0x11 ,set camera to JPEG output mode.
// 8. If receive 0x31 ,set camera to BMP  output mode.
#include <Wire.h>
#include <ArduCAM.h>
#include <SPI.h>
#include "memorysaver.h"
#include <ov2640_regs.h>
//This demo can only work on OV2640_MINI_2MP or OV5642_MINI_5MP or OV5642_MINI_5MP_BIT_ROTATION_FIXED platform.
#if !(defined OV2640_MINI_2MP)
#error Please select the hardware platform and camera module in the ../libraries/ArduCAM/memorysaver.h file
#endif
#define BMPIMAGEOFFSET 66
const char bmp_header[BMPIMAGEOFFSET] PROGMEM =
{
0x42, 0x4D, 0x36, 0x58, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x28, 0x00,
0x00, 0x00, 0x40, 0x01, 0x00, 0x00, 0xF0, 0x00, 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x03, 0x00,
0x00, 0x00, 0x00, 0x58, 0x02, 0x00, 0xC4, 0x0E, 0x00, 0x00, 0xC4, 0x0E, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF8, 0x00, 0x00, 0xE0, 0x07, 0x00, 0x00, 0x1F, 0x00,
0x00, 0x00
};
// set pin 10 as the slave select for the digital pot:
const int CS = 10;
bool is_header = false;
int mode = 0;
uint8_t start_capture = 0;
#if defined (OV2640_MINI_2MP)
ArduCAM myCAM( OV2640, CS );
#endif
uint8_t read_fifo_burst(ArduCAM myCAM);
void setup() {
// put your setup code here, to run once:
uint8_t vid, pid;
uint8_t temp;
#if defined(SAM3X8E)
Wire1.begin();
Serial.begin(115200);
#else
Wire.begin();
Serial.begin(921600);
#endif
Serial.println(F("ACK CMD ArduCAM Start!"));
// set the CS as an output:
pinMode(CS, OUTPUT);
// initialize SPI:
SPI.begin();
while(1){
//Check if the ArduCAM SPI bus is OK
myCAM.write_reg(ARDUCHIP_TEST1, 0x55);
temp = myCAM.read_reg(ARDUCHIP_TEST1);
if (temp != 0x55){
Serial.println(F("ACK CMD SPI interface Error!"));
delay(1000);continue;
}else{
Serial.println(F("ACK CMD SPI interface OK."));break;
}
}
#if defined (OV2640_MINI_2MP)
while(1){
//Check if the camera module type is OV2640
myCAM.wrSensorReg8_8(0xff, 0x01);
myCAM.rdSensorReg8_8(OV2640_CHIPID_HIGH, &vid);
myCAM.rdSensorReg8_8(OV2640_CHIPID_LOW, &pid);
if ((vid != 0x26 ) && (( pid != 0x41 ) || ( pid != 0x42 ))){
Serial.println(F("ACK CMD Can't find OV2640 module!"));
delay(1000);continue;
}
else{
Serial.println(F("ACK CMD OV2640 detected."));break;
}
}
#else
while(1){
//Check if the camera module type is OV5642
myCAM.wrSensorReg16_8(0xff, 0x01);
myCAM.rdSensorReg16_8(OV5642_CHIPID_HIGH, &vid);
myCAM.rdSensorReg16_8(OV5642_CHIPID_LOW, &pid);
if((vid != 0x56) || (pid != 0x42)){
Serial.println(F("ACK CMD Can't find OV5642 module!"));
delay(1000);continue;
}
else{
Serial.println(F("ACK CMD OV5642 detected."));break;
}
}
#endif
//Change to JPEG capture mode and initialize the OV5642 module
myCAM.set_format(JPEG);
myCAM.InitCAM();
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_320x240);
#else
myCAM.write_reg(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);   //VSYNC is active HIGH
myCAM.OV5642_set_JPEG_size(OV5642_320x240);
#endif
delay(1000);
myCAM.clear_fifo_flag();
#if !(defined (OV2640_MINI_2MP))
myCAM.write_reg(ARDUCHIP_FRAMES,0x00);
#endif
}
void loop() {
// put your main code here, to run repeatedly:
uint8_t temp = 0xff, temp_last = 0;
bool is_header = false;
if (Serial.available())
{
temp = Serial.read();
switch (temp)
{
case 0:
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_160x120);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_160x120"));
#endif
temp = 0xff;
break;
case 1:
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_176x144);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_176x144"));
#endif
temp = 0xff;
break;
case 2:
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_320x240);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_320x240"));
#endif
temp = 0xff;
break;
case 3:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_352x288);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_352x288"));
#endif
break;
case 4:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_640x480);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_640x480"));
#endif
break;
case 5:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_800x600);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_800x600"));
#endif
break;
case 6:
temp = 0xff;
#if defined (OV2640_MINI_2MP)
myCAM.OV2640_set_JPEG_size(OV2640_1024x768);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_1024x768"));
#endif
break;
#if defined (OV2640_MINI_2MP)
case 7:
temp = 0xff;
myCAM.OV2640_set_JPEG_size(OV2640_1280x1024);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_1280x1024"));
break;
case 8:
temp = 0xff;
myCAM.OV2640_set_JPEG_size(OV2640_1600x1200);delay(1000);
Serial.println(F("ACK CMD switch to OV2640_1600x1200"));
break;
#endif
case 0x10:
mode = 1;
temp = 0xff;
start_capture = 1;
Serial.println(F("ACK CMD CAM start single shoot."));
break;
case 0x11:
temp = 0xff;
myCAM.set_format(JPEG);
myCAM.InitCAM();
#if !(defined (OV2640_MINI_2MP))
myCAM.set_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
#endif
break;
case 0x20:
mode = 2;
temp = 0xff;
start_capture = 2;
Serial.println(F("ACK CMD CAM start video streaming."));
break;
case 0x30:
mode = 3;
temp = 0xff;
start_capture = 3;
Serial.println(F("CAM start single shoot."));
break;
case 0x31:
temp = 0xff;
myCAM.set_format(BMP);
myCAM.InitCAM();
#if !(defined (OV2640_MINI_2MP))
myCAM.clear_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
#endif
myCAM.wrSensorReg16_8(0x3818, 0x81);
myCAM.wrSensorReg16_8(0x3621, 0xA7);
/*
 *  Start Pixel RGB Read
 */

break;
default:
break;

}
}
if (mode == 1)
{
if (start_capture == 1)
{
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
myCAM.start_capture();
start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
Serial.println(F("ACK CMD CAM Capture Done."));
read_fifo_burst(myCAM);
//Clear the capture done flag
myCAM.clear_fifo_flag();
}
}
else if (mode == 2)
{
while (1)
{
temp = Serial.read();
if (temp == 0x21)
{
start_capture = 0;
mode = 0;
Serial.println(F("ACK CMD CAM stop video streaming."));
break;
}
if (start_capture == 2)
{
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
char VH, VL, buf, k;
//Read 320x240x2 byte from FIFO
//Save as RGB565 bmp format
for (int i = 0; i < 240; i++)
for (int j = 0; j < 320; j++)
{
VH = myCAM.read_fifo();
VL = myCAM.read_fifo();
buf[k++] = VL;
buf[k++] = VH;
#if defined(ESP8266)
yield();
#endif
//Write image data to bufer if not full
if (k >= 256)
{
//Write 256 bytes image data to file from buffer
outFile.write(buf, 256);
k = 0;
}
}
  myCAM.start_capture();
  start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
  uint32_t length = 0;
  length = myCAM.read_fifo_length();
  if ((length >= MAX_FIFO_SIZE) | (length == 0))
  {
    myCAM.clear_fifo_flag();
    start_capture = 2;
    continue;
  }
  myCAM.CS_LOW();
  myCAM.set_fifo_burst();//Set fifo burst mode
  temp =  SPI.transfer(0x00);
  length --;
  while ( length-- )
  {
    temp_last = temp;
    temp =  SPI.transfer(0x00);
    if (is_header == true)
    {
      Serial.write(temp);
    }
    else if ((temp == 0xD8) & (temp_last == 0xFF))
    {
      is_header = true;
      Serial.println(F("ACK IMG"));
      Serial.write(temp_last);
      Serial.write(temp);
    }
    if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
    break;
    delayMicroseconds(15);
  }
  myCAM.CS_HIGH();
  myCAM.clear_fifo_flag();
  start_capture = 2;
  is_header = false;
}

}
}
else if (mode == 3)
{
if (start_capture == 3)
{
//Flush the FIFO
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
myCAM.start_capture();
start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
Serial.println(F("ACK CMD CAM Capture Done."));
uint8_t temp, temp_last;
uint32_t length = 0;
length = myCAM.read_fifo_length();
if (length >= MAX_FIFO_SIZE )
{
Serial.println(F("ACK CMD Over size."));
myCAM.clear_fifo_flag();
return;
}
if (length == 0 ) //0 kb
{
Serial.println(F("ACK CMD Size is 0."));
myCAM.clear_fifo_flag();
return;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
Serial.write(0xFF);
Serial.write(0xAA);
for (temp = 0; temp < BMPIMAGEOFFSET; temp++)
{
  Serial.write(pgm_read_byte(&bmp_header[temp]));
}
SPI.transfer(0x00);
char VH, VL;
int i = 0, j = 0;
for (i = 0; i < 240; i++)
{
  for (j = 0; j < 320; j++)
  {
    VH = SPI.transfer(0x00);;
    VL = SPI.transfer(0x00);;
    Serial.write(VL);
    delayMicroseconds(12);
    Serial.write(VH);
    delayMicroseconds(12);
  }
}
Serial.write(0xBB);
Serial.write(0xCC);
myCAM.CS_HIGH();
//Clear the capture done flag
myCAM.clear_fifo_flag();

}
}
}
uint8_t read_fifo_burst(ArduCAM myCAM)
{
uint8_t temp = 0, temp_last = 0;
uint32_t length = 0;
length = myCAM.read_fifo_length();
Serial.println(length, DEC);
if (length >= MAX_FIFO_SIZE) //512 kb
{
Serial.println(F("Over size."));
return 0;
}
if (length == 0 ) //0 kb
{
Serial.println(F("Size is 0."));
return 0;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
temp =  SPI.transfer(0x00);
length --;
while ( length-- )
{
temp_last = temp;
temp =  SPI.transfer(0x00);
if (is_header == true)
{
Serial.write(temp);
}
else if ((temp == 0xD8) & (temp_last == 0xFF))
{
is_header = true;
Serial.println(F("ACK IMG"));
Serial.write(temp_last);
Serial.write(temp);
}
if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
break;
delayMicroseconds(15);
}
myCAM.CS_HIGH();
is_header = false;
return 1;
}

@coolb2200
Hi,
Please attach us some picture about your phenomenon.
we send 0x30 and read the RGB data by the serial port
and we have test it.
let us know if you need more help.
Regards,
ArduCAM suppor team.

Pictures? I just sent you the code Im using and it won't work due to some "invalid type" errors. You said if I just put in the RGB data code "after start capture" it should work........Could you send me the full code that you're using since it works?

@coolb2200
Hi,
Please try this code :
#include <Wire.h>
#include <ArduCAM.h>
#include <SPI.h>
#include "memorysaver.h"
#define BMPIMAGEOFFSET 66
const char bmp_header[BMPIMAGEOFFSET] PROGMEM =
{
0x42, 0x4D, 0x36, 0x58, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x28, 0x00,
0x00, 0x00, 0x40, 0x01, 0x00, 0x00, 0xF0, 0x00, 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x03, 0x00,
0x00, 0x00, 0x00, 0x58, 0x02, 0x00, 0xC4, 0x0E, 0x00, 0x00, 0xC4, 0x0E, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF8, 0x00, 0x00, 0xE0, 0x07, 0x00, 0x00, 0x1F, 0x00,
0x00, 0x00
};
// set pin 7 as the slave select for the digital pot:
const int CS = 7;
bool is_header = false;
int mode = 0;
uint8_t start_capture = 0;
ArduCAM myCAM( OV2640, CS );
void setup() {
// put your setup code here, to run once:
uint8_t vid, pid;
uint8_t temp;
Wire.begin();
Serial.begin(921600);
Serial.println(F("ACK CMD ArduCAM Start!"));
// set the CS as an output:
pinMode(CS, OUTPUT);
// initialize SPI:
SPI.begin();
while(1){
//Check if the ArduCAM SPI bus is OK
myCAM.write_reg(ARDUCHIP_TEST1, 0x55);
temp = myCAM.read_reg(ARDUCHIP_TEST1);
if (temp != 0x55){
Serial.println(F("ACK CMD SPI interface Error!"));
delay(1000);continue;
}else{
Serial.println(F("ACK CMD SPI interface OK."));break;
}
}
while(1){
//Check if the camera module type is OV2640
myCAM.wrSensorReg8_8(0xff, 0x01);
myCAM.rdSensorReg8_8(OV2640_CHIPID_HIGH, &vid);
myCAM.rdSensorReg8_8(OV2640_CHIPID_LOW, &pid);
if ((vid != 0x26 ) && (( pid != 0x41 ) || ( pid != 0x42 ))){
Serial.println(F("ACK CMD Can't find OV2640 module!"));
delay(1000);continue;
}
else{
Serial.println(F("ACK CMD OV2640 detected."));break;
}
}
myCAM.set_format(BMP);
myCAM.InitCAM();
myCAM.wrSensorReg16_8(0x3818, 0x81);
myCAM.wrSensorReg16_8(0x3621, 0xA7);

}
void loop() {
// put your main code here, to run repeatedly:
uint8_t temp = 0xff, temp_last = 0;
bool is_header = false;
if (Serial.available())
{
temp = Serial.read();
switch (temp)
{
case 0x30:
mode = 3;
temp = 0xff;
start_capture = 3;
Serial.println(F("CAM start single shoot."));
break;
}
}
if (mode == 3)
{
if (start_capture == 3)
{
//Flush the FIFO
myCAM.flush_fifo();
myCAM.clear_fifo_flag();
//Start capture
myCAM.start_capture();
start_capture = 0;
}
if (myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK))
{
Serial.println(F("ACK CMD CAM Capture Done."));
Serial.println(F("ACK IMG"));
uint8_t temp, temp_last;
uint32_t length = 0;
length = myCAM.read_fifo_length();
if (length >= MAX_FIFO_SIZE )
{
Serial.println(F("ACK CMD Over size."));
myCAM.clear_fifo_flag();
return;
}
if (length == 0 ) //0 kb
{
Serial.println(F("ACK CMD Size is 0."));
myCAM.clear_fifo_flag();
return;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
Serial.write(0xFF);
Serial.write(0xAA);
for (temp = 0; temp < BMPIMAGEOFFSET; temp++)
{
  Serial.write(pgm_read_byte(&bmp_header[temp]));
}
SPI.transfer(0x00);
char VH, VL;
int i = 0, j = 0;
for (i = 0; i < 240; i++)
{
  for (j = 0; j < 320; j++)
  {
    VH = SPI.transfer(0x00);;
    VL = SPI.transfer(0x00);;
    Serial.write(VL);
    delayMicroseconds(12);
    Serial.write(VH);
    delayMicroseconds(12);
  }
}
Serial.write(0xBB);
Serial.write(0xCC);
myCAM.CS_HIGH();
//Clear the capture done flag
myCAM.clear_fifo_flag();

}
}
}
Let us know if you need more help.
ArduCAM support team.

I ran your code....it doesn't read or print the RGB values of each pixel and it doesn't automatically launch, let alone work. That's all I need the camera to do. I don't know what it is about this camera but it doesn't want to work. Either that or the API needs updating.

@coolb2200
Hi,
What's your baud rate?
It seems your baud rate is not right.
Regards,
ARduCAM support team.

Hi, recently purchased OV2640 and followed guide on how to connect to mega 2560 board - but can not make it work at all. SPI test passes but vid and pid are both set to 0 so the code throws an error as module is not recognized

@DigitalSocrates
Hi,
How do you connect the camera?
Please attach us some pictures for your hardware connection.
You‘d better check your I2C bus with an oscilloscope.
Arducam support team.

thank you for you response. i opened a different issue for this
see https://github.com/ArduCAM/Arduino/issues/278
will include more info there

i need help on how to connect OV2640 18 pin camera to Arduino Uno and also the code.
i try using the code for ArduCam mini OV2640 the message i got is a continues SPI interface Error


Guide + pitfalls on refactoring your existing ML code to be FL-ready

From an internal re-write of some Cognitive Services training code to leverage the Shrike federated pipeline API, there are a lot of lessons we should have learned.
Put those in a easy-to-follow guide on:

What do you need to do to turn "training script" into "federated pipeline"
Common pitfalls and things to watch out for
Best practices (e.g., log metrics to "parent" pipeline)

This has been implemented in #256


Azure CSI provider

Currently CSI wrapper only has support for ibm. I think it would be great if can implement the same for Azure.
Currently, there are 3 CSI storage drivers for Azure.

Azure Disks
Azure Files
Azure Blob storage

I think it would make sense if we start POC with Azure disks.
cc @surajssd @mkulke @bpradipt
PS: I am sorry but I don't know other people who to tag regarding this issue. But please would really appreciate other people's ideas/feedback?

@knrt10 it does make sense to me. The expectation is CSI wrapper should work without changing the native CSI drivers. But we really need to see how it works with few other CSI drivers.
Tagging few folks from IBM side who can help with any questions and clarifications during the POC.
@liudalibj @huoqifeng @yoheiueda


Cohere model not found

recently saw your project and i am pretty intrested in that, i tried running it i successfully ran the webui for sadtalker and the frontend part for this project is running good i am facing some issues running the backend
i got my trial_api keys and i added it into the apikeys.json file
{
    "api_keys": [
        "XXXXXXXXXXXXXXXX"
     ]
}

i am facing this error while running the backend saying that:
cohere.error.CohereAPIError: model not found, make sure the correct model ID was used and that you have access to the model.

please guide me through this as i am a pretty newbie to this field.
Thanks!!

it's been a while since i made this project, in that time cohere may have changed model names, so try changing the model name to only "command" in line 28 backend/functions/create_script_function.py
llm = Cohere(cohere_api_key=selected_key,
model='command-xlarge-beta', temperature=1.2, max_tokens=1700)


checking codelyzer implementation

Connected to rangle/rangle-starter#108

Also once we're happy with it, we should hook it up to npm test

yes once we are ready and finalized rules we can hook it up to npm test

Otherwise, looks like a good tool.  :shipit: at will!
Thanks Sumit


Feature suggestion - simple file explorer

Implement a basic file explorer to make it possible to:
move file from a folder to the root (vtxs firmwares...)
Play recognised format video that aren't dvr (you can play movie in the present state but only if it is in the dvr folder and named according to the dvr naming scheme)

Reject.


Roles are outdated and not well documented

i think theses are supposed to be links

but more importantly, i don't think there is a Serverless Contributor (https://docs.microsoft.com/en-us/azure/azure-signalr/authorize-access-azure-active-directory#signalr-serverless-contributor) that's mentioned.
most importantly, none of these really make a lot of sense and seem poorly documented. I'm guessing app server means you can do the negotiate phase/be a signalr hub but you can't acutally send messages?   when would i choose that over SignalR Service Owner, or is there another role that lets me send messages.  and when is it correct to use SignalR REST API Reader and why isn't there a writer equivalent of that?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c1830066-cac8-a6a7-3b31-18e51ea01cb1
Version Independent ID: 65c77d0d-08e9-891f-8444-d9f8cf211c3e
Content: Authorize access with Azure Active Directory
Content Source: articles/azure-signalr/authorize-access-azure-active-directory.md
Service: signalr
GitHub Login: @terencefan
Microsoft Alias: tefa

@drdamour Thanks for the feedback! We are currently investigating and will update you shortly.

The role links have been fixed and it seems that the SignalR Service Reader and "Serverless Contributor" have been removed. The other questions remain unresolved. We should discuss changes to this article.
#reassign:vicancy

The doc is now updated with more details provided https://docs.microsoft.com/azure/azure-signalr/signalr-concept-authorize-azure-active-directory#azure-built-in-roles-for-signalr-resources

The doc is much better.
app server vs owner is still a bit confusing..but seems to be a choice between serverless and not

Thanks for providing feedback that helps improve our documentation. Based on the previous responses, we are considering this resolved and we are closing the issue. #please-close


PREPARE statement doesn't work

What happens?
PREPARE test_query(text) AS
SELECT count(*) FROM predictions where client = $1;
EXECUTE test_query('example');

I get this warning when I run the query and it takes forever:
WARNING:  This query was not fully pushed down to DuckDB because DuckDB returned an error. Query times may be impacted. If you would like to see this query pushed down, please submit a request to https://github.com/paradedb/paradedb/issues with the following context:
Not implemented Error: Prepared statement argument types are not supported, use CAST
With this identical query it works in a few milliseconds:
SELECT count(*) FROM predictions where client = 'example';

DuckDB itself supports it: https://duckdb.org/docs/api/c/prepared.html
To Reproduce
Run the query from above
OS:
Ubuntu
ParadeDB Version:
v0.8.4
Are you using ParadeDB Docker, Helm, or the extension(s) standalone?
ParadeDB pg_lakehouse Extension
Full Name:
Janis
Affiliation:
Typewise
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include the code required to reproduce the issue?

[X] Yes, I have

Did you include all relevant configurations (e.g., CPU architecture, PostgreSQL version, Linux distribution) to reproduce the issue?

[X] Yes, I have

Currently, the utility infrastructure is facing issues that need to be addressed:

Check the meta data of foreign tables
All relation metadata in DuckDB is also stored in PostgreSQL. We need to verify that these relations exist and ensure the query is safe for pushdown to DuckDB.
Make sure relations have been loaded
DuckDB lazily loads views (relations), so it's crucial to confirm that these relations have been created before pushing DDL operations (such as PREPARE and EXPLAIN) to DuckDB.
Set search path correctly
If tables in the query are not schema-qualified, we need to set the search path correctly to ensure DuckDB can locate them.

Actually the issues mentioned above not only apply to the utility_hook but also in the executor_hook. However, they have been addressed by FDW and PostgreSQL query analysis. Now, we need to address them specifically within the utility hook.

Thank you for highlighting this. Let's handle them first as you suggest. I'm excited for us to fix these limitations in the utility hook.

Currently, the utility infrastructure is facing issues that need to be addressed:

Check the meta data of foreign tables
All relation metadata in DuckDB is also stored in PostgreSQL. We need to verify that these relations exist and ensure the query is safe for pushdown to DuckDB.
Make sure relations have been loaded
DuckDB lazily loads views (relations), so it's crucial to confirm that these relations have been created before pushing DDL operations (such as PREPARE and EXPLAIN) to DuckDB.
Set search path correctly
If tables in the query are not schema-qualified, we need to set the search path correctly to ensure DuckDB can locate them.

Actually the issues mentioned above not only apply to the utility_hook but also in the executor_hook. However, they have been addressed by FDW and PostgreSQL query analysis. Now, we need to address them specifically within the utility hook.

Is this unblocked now that the EXPLAIN PR was merged?

Yes, I'm working on pgrx to add some functions related to Prepare. You can check it out in my pull request here: https://github.com/pgcentralfoundation/pgrx/pull/1873. After that, I’ll be able to resume development.


feat: added greynoise integration incl. docs

What does this PR do?
added greynoise integration incl. docs
Type of change

[x] New feature (non-breaking change which adds functionality)

Mandatory Tasks

[x] Make sure you have self-reviewed the code. A decent size PR without self-review might be rejected.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Visual Studio unable to parse plugin.xml

I am trying to use this plugin in Visual Studio 2015 and I get the error: "Visual Studio was unable to parse plugin.xml. Please ensure plugin.xml exists and contains no errors.". I have tried downloading it by entering the Git repository location as well as personally downloading, unzipping, and pointing Visual Studio to the local repository. Both ways have resulted in the same error mentioned above.

Fixed the issue. I added a fork as well with the fix but in case anyone else runs into the issue just add one line of code to the plugin.xml file: xmlns:android="http://schemas.android.com/apk/res/android". To make it work properly you need the plugin tag to read as follows:


fix: disjunctive_scheduling example had incorrect negation constraint

Hi, another tiny fix while getting everything running.
I'm pretty sure the negation of literal => s_y - s_x <= -p_y is as follows:
$$l => s_y - s_x <= -p_y$$
$$\neg l => s_y - s_x > -p_y$$
$$\neg l => s_y - s_x >= -p_y + 1$$
$$\neg l => -s_y + s_x <= p_y - 1$$
Also the literal was not actually negated in the second case, so I added that as well in the way I think you're supposed to do it.

Great catch, however, it is still not fully correct.
We want to say that either: x before y or y before x
What we are currently saying is y before x or not y before x; note that not y before x and x before y are not equivalent! As they could now execute at the same time

So what we are really after, is the following:
$$
l_1 \rightarrow  s_x + p_x \leq s_y
l_2 \rightarrow  s_y + p_y \leq s_x
$$
Moving the variables to the left-hand side of the $\leq$, we have:
$$
l_1 \rightarrow  s_x - s_y \leq -p_x
l_2 \rightarrow  s_y - s_x \leq -p_y
$$

@maartenflippo this one was also ready


Up to latest version

I don't know if is relevant but i'm using the latest camel version (2.22.1) and the method component.createSqlSessionFactory() is now protected so you should use component.getSqlSessionFactory() instead (i did not run the test)

Ok its an error i remove


Filter PR's based on author

Currently PR are categorized using labels. It would be cool if PR's could alternatively be categorized on author. In this way, a category like "dependencies" could filter on author "renovate".

I found that this is already possible using a label_extractor.
    "label_extractor": [
        {
          "pattern": "renovate\\[bot\\]",
          "on_property": "author",
          "method": "replace",
          "target": "dependency"
        }
      ],

@paul-dingemans there is also a different feature which is a bit newer than label_extractors you can use.
Categories can have rules provided in addition to organize PRs in categories. An example test: https://github.com/mikepenz/release-changelog-builder-action/blob/af326da47fc20c69fd42234ed91d04ab68c7bbe8/tests/transform.test.ts#L553-L576
(this feature is best used with v4 though)

I like syntax above. Unfortunately, it does not seem to work in my use case. After changing the code my action flow throws and error. None of the PR's seems to be categorized anymore. Am I missing something?

It seems there is a problem in the syntax of the config.

looks there is a `,` missing.

One additional advice. Looking at the output for the shell output. you get a bunch of errors as it writes the string as non escaped string.

The reason for that is how GitHub Actions replaces the template. It basically just puts the string from the variable into that specific location. In non escaped form.
https://github.com/pinterest/ktlint/blob/master/.github/workflows/generate-changelog.yml#L80
So if you want to output it in bash or use in bash, I highly recommend you setting it into an env variable (this is not specific to this action, but general advice for GitHub actions)
E.g. something along those lines:
https://github.com/mikepenz/release-changelog-builder-action/blob/develop/.github/workflows/ci.yml#L165-L174

Note that my workflow did work before I change to using category.rules.
The label dependency changelog is just a single label and should not be separated by a comma.
Is it allowed to have both labels and rules defined for a single category? According to the documentation it is allowed. In this particular case, I would like to select all PR's for which either the label or the rule is met. After replacing the selection on label with a rule (see https://github.com/pinterest/ktlint/pull/2212) the output is generated correctly.

I'm currently on the phone so I'll only report on the first.
I see that it is only a single item, however you miss the comma afterwards, before the next item in the JSON.

Yes, was my mistake. I am so used to the editor highlight and use of trailing comma's that I completely overlooked that.

@paul-dingemans fully agree


Update dependencies

RELEASE_NOTES=n/a
Signed-off-by: Dominik Schulz<EMAIL_ADDRESS>

But you might want to correct the test failure on TestParseRule

Yes, that sounds like a good idea :)


Adding TabMonConfigSection.cs and TabMonConfig.xsd file generation project to solution

This PR adds the "TabMonConfiguration" project to the main solution.  This is a project leveraging the "Configuration Section Designer" extension for Visual Studio to provide a visual way to define the config schema and auto-generate the contents of the TabMonConfigSection.cs class and TabMonConfig.xsd schema.
This project is not included in existing build configurations and requires installation of the "Configuration Section Designer" VS extension to edit and build.

yo, thanks for the pr, appreciated

No problem, hope it helps!

Looks good. This can be shipped.


Why does #lock_tables need to be the outermost transaction?

Consider the following code:
class Invoice::Payer
  # ...

  def pay
    @invoice.lock do
      payment =<EMAIL_ADDRESS>@invoice.description, @invoice.amount)
      if payment.success?
        @invoice.update(paid: true)
      end
    end
  end
end

class CreditCard::Charger
  # ...

  def charge
    # ...

    from_account = DoubleEntry.account(:credit_card, scope: @credit_card.account)
    to_account = DoubleEntry.account(:sales, scope: @credit_card.account)

    DoubleEntry.transfer(@amount, from: from_account, to: to_account, code: :purchase)
  end
end

This seems like pretty reasonable code to me, but unfortunately doesn't work because in the Invoice class, I'm locking the invoice (which creates a transaction), and then the DoubleEntry#transfer method fails because it's transaction is not the outer-most one.
What's the reasoning behind making it the outer-most transaction? What would the world be like if it "just worked"?

Maybe there is a way to improve this API?

I don't think there is a way to improve the locking without either a) changing ActiveRecord's behaviour, b) changing MySQL's behaviour, or c) changing the database model used.

Requiring DoubleEntry.lock_tables to be the outermost transaction can result in less than ideal code.
Keith's example would be great to be able to do. I believe it is possible to change the api without a) b) or c), but by doing d) - requiring account balance records to be created before doing a double entry transfer.
The creation of the account balance records is the tricky thing that really needs that outer lock, due to the rollback and retry stuff. Looking at the locking code, it is very AccountBalance specific.
I believe Keith's example could work provided he simply guaranteed that all required double entry account records already existed. This would require more thought on the part of someone implementing a double entry solution, but it could be as simple as creating a  a matching DoubleEntry::AccountBalance every time you create a  CreditCard record. That way, when DoubleEntry tries to lock accounts, the lock won't fail due to no record existing.

The AccountBalance records exist primarily to enable locking! If you don't do per-account locks somehow, the totals in the lines table will get out of whack because of race conditions around knowing the current balance. (This is exactly the problem we had back when this code was first written, and it's what all this locking magic is there to fix.)
If I was building finance code from scratch today, I'd make it eventually consistent. i.e. Just record that money was moved inside the transaction, and calculate balances after the fact. That's a better model in many ways, but it's not the model of this library, and there's not a way to make this library use that model without fundamentally changing the underlying data structures.
.

@notahat I'm not saying do away with account balance records, or per-account locking, I'm saying the really complicated "ensure outermost transaction" locking code is due to the creation of account balance records, not locking them. ( or, trying to lock them when they don't exist ).
My hypnosis is: that if, as part of your application, you ensure that all account balance records are created separately before using them ( and you may need to "ensure outermost transaction" for that ), once you have that assurance, you can safely do DoubleEntry transactions within another transaction.

@eadz: There's definitely a lot of complexity in there to support lazy creation of account balances. And yes, you could simplify things if you assumed all account balances were created up front.
I don't think that lazy creation is the only reason for the outermost transaction requirement though.

@eadz: There's definitely a lot of complexity in there to support lazy creation of account balances. And yes, you could simplify things if you assumed all account balances were created up front.

I don't think that lazy creation is the only reason for the outermost transaction requirement though.

I think it is the only reason that really matters.
I think this is the essence of this Issue "Why does #lock_tables need to be the outermost transaction?" So we can agree that it's because of the account balance creation, but why else?
What is the other reason for being an outermost transaction? ( Other than deadlocks which IMHO having an outermost transaction requirement doesn't solve that problem on its own, although it may help ).
And the other question is, whatever the reason, is it so important that it prevents us from having a cleaner API?

Deadlocks are the problem because of ActiveRecord and MySQL having broken behaviour around them.
If you get a deadlock in a nested transaction, ActiveRecord behaves as though just the inner transaction is being rolled back, but MySQL rolls back the outermost transaction. Makes it impossible to retry after a deadlock unless you can be sure you own the outermost transaction.
So you could abandon the requirement for being the outermost transaction as long as you abandon the retrying on deadlock.

If you get a deadlock in a nested transaction, ActiveRecord behaves as though just the inner transaction is being rolled back, but MySQL rolls back the outermost transaction. Makes it impossible to retry after a deadlock unless you can be sure you own the outermost transaction.

You may be able to pass requires_new: true when starting a transaction, in addition to raising ActiveRecord::Rollback outside of the inner transaction when the inner transaction fails, to get the behaviour you're after. But I'm not suggesting this as a solution.
I would suggest removing the automatic retry instead: because of the potential benefits you get to the API, removal of database specific and very complicated transactional / locking code, and the reduction in surprises such as the following:
DoubleEntry.lock_accounts(account_a, account_b) do
  payment =<EMAIL_ADDRESS>   if payment.success?
     DoubleEntry.transfer(Money.new(20_00), :from => account_a, :to => account_b, :code => :purchase)
   end
end

It's not clear in the documentation that CreditCard::Charger.charge could be called twice in case of deadlocks... and I'm not sure the benefit you get from restarting deadlocks is worth it the downsides - maybe deadlocks should be caught and handled higher up the stack, as there are usually other SQL transactions and other code within the DoubleEntry transaction that are not the responsibility of DoubleEntry to restart or retry.

What about a :retry_on_deadlock => false or allow_outer_transaction => true option?
Retrying on deadlock/outmost transaction check stay the default, so it's safe by default, but if you know what you're doing, you can take your chances and handle retries higher up in the stack? When the option is on, deadlocks or any other errors just bubble up and fail the outer transaction, which is free to have its own retry mechanism (and even recommended so in the option's documentation)
Having use for a transaction around your financial transfer is not an edge case, I'd argue it's one of the most common and important things you may need to wrap in a transaction with other stuff. It's been a recurring issue for us. Well worth the extra-complexity in the code/interface I reckon.

What about a :retry_on_deadlock => false or allow_outer_transaction => true option?
I think we definitely need some more options.
I think there were some decisions made early on that have meant the API for using double entry is a little more difficult than it could be.
Having double entry do some "magic" ( creating account balance records as part of locking ), seems to in my opinion complicate things enough to warrant an alternative.
Some of the locking code seems to have some code smells;
Here are 3:

there is no test that shows what happens when the transaction isn't the outermost transaction
there are class variables in use ( `@@locks`` )
comments that dont make sense:

def create_missing_account_balances
  @accounts_without_balances.each do |account|
    # Get the initial balance from the lines table. <<- this comment
    balance = account.balance
    # Try to create the balance record, but ignore it if someone else has done it in the meantime.
    AccountBalance.create_ignoring_duplicates!(account: account, balance: balance)
  end
end

I don't understand how there can be a balance in the lines table, when a lock is required to create a line and therefore an account balance record.
And finally, the DoubleEntry::Locking module seems to break SRP as it does at least two major things

Creates account balance records
Locks accounts

From my reading, a potential way forward would be to make the following change:
Require AccountBalance records to exist before trying to lock them
This would remove a lot of the checks and edge cases around account balance records not existing. The creation of account balance records can happen in another place, as when you call
DoubleEntry.account(:my_account) you know then you'll need to lock that account.
From here there are a few options:

Leave it to the user implementing double entry to call something like DoubleEntry.account_for_lock(:my_account) which would ensure there is an account balance record.

make DoubleEntry.account(:my_account) ensure there is an account balance record.

The creation of DoubleEntry account balance records DOES NOT need to happen with any locks - it can't, there is nothing to lock on, that's why it tries to create the record and if it fails it ignores the error ( duplicate error ).

Based on these discussions, and also from #168 and #160. Wouldn't it be possible to eliminate locking by performing the following?

Before any transfer, ensure the AccountBalance object exists
Perform the transfer
Periodically perform a line check. I'd probably opt to do this every 15 minutes and have it automatically fix balance errors

@kschutt for most cases, you wouldn't want to eliminate locking. I guess in theory you could rely on a line check but I don't believe it will fix any error it finds. It's a sanity checker, and doesn't fix any errors it finds AFAIK. For most cases you would want your balances to be accurate all the time, hence the locking.
This thread is about the programmer's API for locking and the requirement to have double entry as the outer most transaction, not about removing locks entirely.


[bug] the attribute type is always any in generated doc when import remote kpm packages

Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)

kcl import -m crd
kcl doc generate

the crd source code uses remote kpm package k8s as dependency

2. What did you expect to see? (Required)
the metadata type in generated doc should be ObjectMeta
3. What did you see instead (Required)
any
4. What is your KCL components version? (Required)

the problem cannot be reproduced by kcl import -m crd <crd path> and kcl doc generate `
did you init a kpm package based on the crd kcl files and add k8s as dependency? please provide complete information

I tried to reproduce it, but:

kcl import -m crd ~/krm-crd.yaml -o ~/
cd ~/ && kpm init kcl-run
cd ~/kcl-run && kpm add k8s:1.28
cp ~/models/krm_kcl_dev_v1alpha1_k_c_l_run.k ~/kcl-run/
kcl doc generate ~/kcl-run

and the error is:

I tried to reproduce it, but:

kcl import -m crd ~/krm-crd.yaml -o ~/
cd ~/ && kpm init kcl-run
cd ~/kcl-run && kpm add k8s:1.28
cp ~/models/krm_kcl_dev_v1alpha1_k_c_l_run.k ~/kcl-run/
kcl doc generate ~/kcl-run

the doc generation is failed so I couldn't see the any type bug and the error is as following:

cc @zong-zhe


Update readable-stream to get asyncIterator

I would like to write this:
for await (const row of mysql.query(…).stream()) { … }

Installing readable-stream@latest adds this feature.
However, version 3 of readable-stream only supports Node 6 and later, while version 2 supports all Node.js version from 0.8. You indicate that you support Node.js 0.6 or higher. The oldest Node.js version supported by upstream is 6 (soon to be 8 in less than a month), so it's probably reasonable to drop support for very old versions.

Hi @jleedev ! Being that this is a base driver, the goal is to support all possible Node.js versions, even when the Node.js mantainers no longer support them. This is because there are vendors that will provide support for older versions of Node.js, including backporting security fixes, etc.
If you want to use the newest readable-stream with query results, a simple solution can be to just pipe the stream from this module into the newer version of a pass through stream:
for await (const row of mysql.query(…).stream().pipe(new stream.PassThrough({objectMode: true}))) { … }

Ideally the original .stream() method should have never landed by using a module like readable-stream, as it should have instead used the in-built streams directly such that the interface would actually match the version of Node.js it is running on, automatically bringing in the new features that Node.js adds to stream (for example, we use the in-built EventEmitter, not an old copy).
This would be a breaking change, but is the change that should be made, not simply upgrading the readable-stream module and then just having this same issue in another future Node.js version, if that makes sense.
That said, hopefully the code snippet above is workable for you. Otherwise, at some point I can work on removing the readable-stream module altogether (or you're welcome to volunteer to do this as well).

Yes, thanks for the snippet. I tried new Readable().wrap and it was failing mysteriously.
I'm not sure I understand why removing readable-stream would be a breaking change, but it nonetheless passes all the tests on Node 11. It would also be a matter of making sure it doesn't break on any version of Node that you support?

In fact, it apparently works (tests pass) all the way back to Node 0.10, which introduced the stream.Readable class. Before that I imagine we would still need the readable-stream library.

Hi @jleedev so the reason it is likely a breaking change is because the API returned from .stream() is currently locked to whatever readable-stream is providing, and a change to use the native stream would alter the behavior for existing users. The tests around .stream() are very basic and exercise only the very top level features.
Just as an example: if one were to swap readable-stream out for the native stream, on Node.js 6 the order in which the 'data' and 'readable' events fire on the stream are reversed, which can be really hard to debug if this breaks something. Node.js put this in a major release for this reason.
Another issue is that in readable-stream v2, all errors it generated had a message only and no code property. This meant that you have to key off the message to determine the type of stream error. The error messages from streams vary from Node.js version to version, so a fallback to the native implementation will change the errors that are being emitted, causing another point of breaking.
Ultimately that above is speculation without working on a change to this module and verifying these things to know for sure, but hopefully it is some insight as to why the test suite may still be passing even with a breaking change lurking. Test suites unfortunately only catch regressions that they were programmed to check for, and the streams tests are very lacking currently.
As for the code snippet, was the one I showed not working for you? If it wasn't, are you getting an error at all, or can you describe what happened when you run it?

It only returned the first row:
> for await (const row of conn.query("select * from t").stream().pipe(new stream.PassThrough({objectMode: true}))) { console.log(row) }
RowDataPacket { id: 1, data: '0' }
RowDataPacket { id: 2, data: '1' }
RowDataPacket { id: 3, data: '2' }
RowDataPacket { id: 4, data: '3' }
RowDataPacket { id: 5, data: '4' }
RowDataPacket { id: 6, data: '5' }
RowDataPacket { id: 7, data: '6' }
RowDataPacket { id: 8, data: '7' }
RowDataPacket { id: 9, data: '8' }
RowDataPacket { id: 10, data: '9' }
undefined
> for await (const row of new stream.Readable({objectMode:true}).wrap(conn.query("select * from t").stream())) { console.log(row) }
RowDataPacket { id: 1, data: '0' }
undefined
>

Your output shows it returned 10 rows for the snippet I gave you.

Yes, your snippet is working.
I had first tried new stream.Readable({objectMode:true}).wrap(…) to wrap the stream with the standard library version, and it emitted the first row then stopped.

Weird, I never used .wrap before, and I was just looking at the docs and it makes it sound like it only works in streams that are Node.js 0.8 and below style. Since the returned stream here is from readable-stream module I guess it is not an "old style" stream as the docs say, though they don't really say what exactly that means.

Fair enough, thanks for trying to figure it out.
I appreciate that swapping out the entire stream library would be a risky change, and a workaround exists for my use case.

No problem. So this is what I am thinking right now
(1) you got a work around (yay)
(2) i do want to just get readable-stream removed as part of the 2.0 work
(3) I'm going to investigate why .wrap is not working to see if that is expected or there is something this module can fix to make it work

I wanted to update here that I am planning to release a 2.x minor next week and then get to work on a 3.0 so this should get resolved with that one way or another (even if it is just getting readable-stream dep updated to latest).

I managed to create a working async-iterator using Readable.wrap (node 14.16.1, mysql 2.18.1). However, the first result I get is an OkPacket object that I have to manually ignore. Other than that it seems to work fine so far.
const readable = new Readable({objectMode: true}).wrap(pool.query(sql).stream());
for await (const result of readable) {
  if (result.constructor.name !== 'OkPacket') {
    console.log(result);
  }
}


How to logout from Connected firewall/panorama

Hi,
Once connected to firewall/panorama, how to logout from the same device?
I was searching throughout the doc, but couldn't help.

I had the same thought, after you completed you script, the firewall keeps the session open and the user still shows as logged in. I dont think there is an API option to explicitly log out as you do from the cli or gui. You can configure user time out on the firewall and the admin session for api user will be eventually cleared.