Can't build swig bindings for lldb

Tried following the instructions, tried everything could find in all the relevant issues reported here. Retried again in a fresh install, here's a transcript of what happened. Please help!!!
heartpunk@REDACTED-MacBook-Pro ghidra_11.1_PUBLIC % LLVM_HOME=/Users/heartpunk/llvm-project GHIDRA_INSTALL_DIR=`pwd` ./Ghidra/Debug/Debugger-swig-lldb/macos_debugger_lldb_build_from_brew.sh --verbose --debug
+ '[' -z /Users/heartpunk/ghidra_11.1_PUBLIC ']'
+ '[' '!' -z /Users/heartpunk/ghidra_11.1_PUBLIC ']'
+ pushd /Users/heartpunk/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb
~/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb ~/ghidra_11.1_PUBLIC
+ LLVM_VERSION=16
+ brew install llvm@16
Warning: llvm@16 16.0.6_1 is already installed and up-to-date.
To reinstall 16.0.6_1, run:
  brew reinstall llvm@16
++ mktemp -d
+ LLVM_TEMP_DIR=/var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R
+ brew unpack --patch --destdir /var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R llvm@16
==> Unpacking llvm@16 to<EMAIL_ADDRESS>==> Downloading https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.6/llvm-project-16.0.6.src.tar.xz
Already downloaded: /Users/heartpunk/Library/Caches/Homebrew/downloads/8cbb1836057b964f8f1bea2eee120bd27a78aa698ba2d5263996f3071234b47e--llvm-project-16.0.6.src.tar.xz
++ echo<EMAIL_ADDRESS>+ export<EMAIL_ADDRESS>+<EMAIL_ADDRESS>+ '[' '!' -d<EMAIL_ADDRESS>']'
++ brew --prefix llvm@16
+ BREW_LLVM=/opt/homebrew/opt/llvm@16
+ export 'LDFLAGS=-L/opt/homebrew/opt/llvm@16/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm@16/lib/c++,-L/opt/homebrew/opt/llvm@16/lib'
+ LDFLAGS='-L/opt/homebrew/opt/llvm@16/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm@16/lib/c++,-L/opt/homebrew/opt/llvm@16/lib'
+ export 'PATH=/opt/homebrew/opt/llvm@16/bin:/Users/heartpunk/llvm-bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2/bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2@global/bin:/Users/heartpunk/.rvm/rubies/ruby-3.1.2/bin:/Users/heartpunk/.local/bin:/Users/heartpunk/.cabal/bin:/Users/heartpunk/.ghcup/bin:/Users/heartpunk/.nvm/versions/node/v18.13.0/bin:/Users/heartpunk/.pyenv/shims:/opt/homebrew/opt/python@3.10/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.1.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/go/bin:/Users/heartpunk/.cargo/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/heartpunk/.local/bin:/Users/heartpunk/.rvm/bin:/Users/heartpunk/go/bin:/Users/heartpunk/.dirtybin/'
+ PATH='/opt/homebrew/opt/llvm@16/bin:/Users/heartpunk/llvm-bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2/bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2@global/bin:/Users/heartpunk/.rvm/rubies/ruby-3.1.2/bin:/Users/heartpunk/.local/bin:/Users/heartpunk/.cabal/bin:/Users/heartpunk/.ghcup/bin:/Users/heartpunk/.nvm/versions/node/v18.13.0/bin:/Users/heartpunk/.pyenv/shims:/opt/homebrew/opt/python@3.10/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.1.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/go/bin:/Users/heartpunk/.cargo/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/heartpunk/.local/bin:/Users/heartpunk/.rvm/bin:/Users/heartpunk/go/bin:/Users/heartpunk/.dirtybin/'
+ export CPPFLAGS=-I/opt/homebrew/opt/llvm@16/include
+ CPPFLAGS=-I/opt/homebrew/opt/llvm@16/include
++ echo /opt/homebrew/opt/llvm@16
+ export LLVM_BUILD=/opt/homebrew/opt/llvm@16
+ LLVM_BUILD=/opt/homebrew/opt/llvm@16
+ gradle buildNatives

> Configure project :Debugger-swig-lldb
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Configure project :Decompiler
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Configure project :FileFormats
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Configure project :PDB
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Task :Debugger-swig-lldb:compileMainMac_arm_64SharedLibraryMainCpp FAILED
/Users/heartpunk/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb/src/main/cpp/LLDBWrapJava.cpp:317:10: fatal error: 'lldb/API/SBScriptObject.h' file not found
#include "lldb/API/SBScriptObject.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':Debugger-swig-lldb:compileMainMac_arm_64SharedLibraryMainCpp'.
> A build operation failed.
      C++ compiler failed while compiling LLDBWrapJava.cpp.
  See the complete log at: file:///Users/heartpunk/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb/build/tmp/compileMainMac_arm_64SharedLibraryMainCpp/output.txt
   > C++ compiler failed while compiling LLDBWrapJava.cpp.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 1s
2 actionable tasks: 2 executed
<-------------> 0% WAITING
> IDLE
heartpunk@REDACTED-MacBook-Pro ghidra_11.1_PUBLIC % system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 14.5 (23F79)
      Kernel Version: Darwin 23.5.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: REDACTED's MacBook Pro
      User Name: REDACTED (heartpunk)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 3 days, 6 hours, 55 minutes

heartpunk@REDACTED ghidra_11.1_PUBLIC %

@heartpunk Am happy to provide more detailed instructions for building using SWIG, but we’re really really really recommending you use the newer and much much easier to install version of the debugger based on traceRMI and a Python3 interface.  Is there a reason you’re opting for the older JNI interface?

Nope, wasn't aware of that option. Which is it??? The frida thing?

@heartpunk Yes and no.  The class has been re-written and should be helpful, but I think I would begin with the help section for the debugger.
A couple of things that might help get you started.
First, if you were using the debugger in the past, you’ll need to reconfigure the tool to expose two new plugins.  The easiest way to do this is to simply import a new Debugger tool from the Default Tools chest.  Alternatively, you can reconfigure it by hand using the Configure dialog.
The tl;dr version:  the Targets plugin has been replaced by the Connections plugin, and the Objects plugin has been replaced by the Model plugin.  The new versions are very similar to the old ones on the surface, so you shouldn’t need too much info to get you going with them.  If both the old and new plugins are installed, you will see two “bug” buttons on the Toolbar.  I would recommend removing the old plugins just to eliminate any possible confusion.  If you end up hating the new stuff, putting back the old plugins is easy enough.
The other thing you want to check is that your default version of python3 is the same python compiled into lldb.  To get the version in lldb, start up lldb, enter “script import sys”, and then “script sys.version”.   It’s possible (although unlikely) lldb was compiled without python.  If so, let us know - can be fixed, but probably requires more details.
If the default version of python is a match, you’ll want to make sure Google protobuf and psutil are installed.  I usually use “python3 -m pip install google” and “python3 -m pip install psutil”.
All of this is one-time work.  If you’re not a python person (I’m not), it can be a bit confusing, but once it’s done it’s done.  From that point on, debugging should only involve loading a program into the debugger listing, configuring lldb via the dialog, and thereafter hitting the bug.  (The dialog will want the full path to lldb if it’s not the path, same for the executable, and the start command you prefer, i.e. do you want to break on start.)
Am quite sure I’ve left stuff out, but am available for questions as you go.  We appreciate your trying out the new versions - ultimately, we’re hoping it makes your experience with the debugger better.

I'm new to ghidra! I tried all you said as best I could manage, but still the same error. I even modified the ghidraRun script to check which python is in scope right before it invokes ghidra!!!

Excellent - welcome to Ghidra!    If it's any encouragement, you're are literally the first issue posted on the new debugger, so anything you share here will help a lot of folks.  If you are, what I would try first:
(1) Open up a terminal and run "lldb".
(2) Inside lldb, issue the command "script import ghidratrace".  This should fail with a "ModuleNotFoundError: no module named 'ghidratrace'.
(3) Exit lldb.  In the same terminal, issue the command "export PYTHONPATH=/Users/xxx/path_to/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src;/Users/xxx/path_to/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src".
(4) Now run "lldb" and "script import ghidratrace".  It should succeed this time.
If not, let us know the version of lldb you're running ("lldb --version") , the version of python ("python --version"), and the version of python in lldb ("lldb" -> "script import sys" -> "script sys.version".
If it did succeed, then we need to figure out whether the launch script is somehow mis-computing it.  The launch script lives in "/Users/xxx/path_to/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/data/debugger-launchers/local-lldb.sh".  You can modify it fairly easily.  I would suggest adding the line "echo $PYTHONPATH" around line 48.  If it doesn't agree with the path you used above (or if you don't see the echo in the Terminal window), we have some more work to do.

Alright, I'm following along with that. Did a fair bit of investigation just now and this is almost certainly an issue with pyenv (though it didn't go away when I tried to hack that outta the equation; I haven't tried a more proper try w/o it yet). Going to keep poking around w/it and modifying the launch script you mentioned and report back a bit later!

heartpunk@REDACTED ~ % lldb --version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
heartpunk@REDACTED ~ % python --version
Python 3.12.3
heartpunk@REDACTED ~ % python3 --version
Python 3.9.6
heartpunk@REDACTED ~ % lldb
(lldb) script import sys
(lldb) script sys.version
'3.9.6 (default, Mar 29 2024, 10:51:09) \n[Clang 15.0.0 (clang-15<IP_ADDRESS>)]'
(lldb)

Ok I had misunderstood, but have followed your instructions for the case where importing ghidratrace doesn't work (it didn't in any case I tried). It seems that script isn't being invoked at all, based on having modified it to print like you said, and seeing nothing printed anywhere.
Now, I'm also noticing my ghidra lives in /Applications (ok I actually have it at home too, but that's a different install ack) and wondering how lldb invocations know to look for such a launch script at all anyway??? Seems like magic to me.

(Oh, oh, I see. The PYTHONPATH setting is how it knows to make that module available.)

OK, so that helps.  The first thing I would check - make sure you installed 'psutil' and 'google' in the correct version of python, i.e. as opposed to what I said before, use "python3 -m pip install psutil" and "python3 -m pip install google".  If that's actually the problem, there's probably on error at the top of the Debug Console window to that effect.  If it's not egregiously long, maybe send us the Debug Console contents?  Often we see things that tip us off to the real error.  (Obviously, sanitize anything you're worried about out.)   Quite possible that the PYTHONPATH logic is actually working as designed but an early failure is masking the issue.

Re "how lldb invocations know to look for such a launch script at all " (just read that bit),  not magic at all as we invoking lldb, not the other way around.  In other words, when you select "lldb" from the bug pull-down, we run local-lldb.sh.  The comment tags at the top of the shell command are used to populate the dialog (ok, that part is a little bit magical) and set various variables used by the script and eventually by the python code.  The shell sets the PYTHONPATH variable and then runs lldb (line 67), passing in the commands that set-up the target and import the ghidra logic.  I am surprised you are not seeing your "echo" edits.  Suggests either an issue with having multiple versions or a failure very early in the shell script.  Maybe put one in very early, say before the "export" logic?

3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-13<IP_ADDRESS>.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
(lldb) script import ghidralldb
(lldb) target create "REDACTED"
Current executable set to 'REDACTED' (arm64
(lldb) ghidra trace connect "<IP_ADDRESS>:53664"
error: 'ghidra' is not a valid command.
(lldb)

ok so it was showing the edits i made, i had misunderstood that you meant to go and run lldb in ghidra, not on the terminal.
i double checked the packages are installed in the correct python

OK, just trying to figure out where we are now....  The example in your last post is from lldb run from a terminal, correct?  I would not expect "ghidra trace connect" to work from there, if so, because the python import does not create the corresponding ghidra commands for lldb.  This is definitey confusing will agree, but you should be able to execute "script ghidra_trace_connect('<IP_ADDRESS>:53664')" after "script import ghidralldb".  The fact that these names are matched and thus almost the same probably will throw people off - hadn't considered this.  In general, though, this is not what you want to be doing.  You really want to be executing from the Ghidra terminal - better still, you want to rely on the debugger executing all those commands for you automatically.  Does the command above fail in the Ghidra Terminal launched after you hit the bug?  or are you not getting that far?

This is the output after I click checks notes

This is the output after I click checks notes
Debugger -> Configure and Launch ... -> lldb
I didn't execute those manually!
It is the output in the ghidra terminal though, yes.

as for the differently named command, here's how that went when I gave it a try
error: 'ghidra' is not a valid command.
(lldb) script ghidra_trace_connect('<IP_ADDRESS>:53664')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'ghidra_trace_connect' is not defined
(lldb)

OK, hmmmm - that’s too bad.  If you scroll up in the Terminal window, what are the first things you see?

/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applicat
ions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-13<IP_ADDRESS>.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
(lldb) script import ghidralldb
(lldb) target create "/Applications/AntiRSI.app/Contents/MacOS/AntiRSI"
Current executable set to '/Applications/AntiRSI.app/Contents/MacOS/AntiRSI' (arm64)
.
(lldb) ghidra trace connect "<IP_ADDRESS>:51384"
error: 'ghidra' is not a valid command.
(lldb) script ghidra_trace_connect('<IP_ADDRESS>:53664')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'ghidra_trace_connect' is not defined
(lldb)

is all of it

Double hmmmm - ok, anm away from my computer right now, but, when I get back, will do the same, and see if I can spot any differences.  I do apologize - I thought this would be much simpler and certainly expected it to be easier to troubleshoot.  I really appreciated your patience.

Wow, ok, so I ran the debugger on a copy of the decompiler included with Ghidra, which I'm guaranteed to have debug access to.  It lives at ghidra_11.1_PUBLIC/Ghidra/Features/Decompiler/os/mac_arm_64/decompile.  I used the following options:

Gave me a pop-up about being unable to check fo rmalware, but then:

which looks a lot like your results except fo the error.
I will play around with this some more tonight and darg in the big guns tomorrow. I am definitely not seeing/thinking of something, because it really seems like you did all the right things.

Oh, and I goofed in my comments regarding ghidra_trace_connect.  You'd need to do something like "script from ghidralldb.commands import *"  first for it to work and even then it need a bunch of parameters.

One more possible experiment you could do if you are so inclined.  Run Ghidra using ghidra_11.1_PUBLIC/support/ghidraDebug instead of ghidraRun and make note of any lines that say ERROR in the launching console.  Mine looks like:

heartpunk@REDACTED ~ % /Applications/ghidra_11.1_PUBLIC/support/ghidraDebug ; exit;
Listening for transport dt_socket at address: 18001
openjdk version "22.0.1" 2024-04-16
OpenJDK Runtime Environment Homebrew (build 22.0.1)
OpenJDK 64-Bit Server VM Homebrew (build 22.0.1, mixed mode)
INFO  Using log config file: file:/Applications/ghidra_11.1_PUBLIC/support/debug.log4j.xml  (LoggingInitialization.java:50)
INFO  Using log file: /Users/heartpunk/Library/ghidra/ghidra_11.1_PUBLIC/application.log  (LoggingInitialization.java:51)
INFO  Loading user preferences: /Users/heartpunk/Library/ghidra/ghidra_11.1_PUBLIC/preferences  (Preferences.java:122)
INFO  Loading previous preferences: /Users/heartpunk/.ghidra/.ghidra_11.0.3_PUBLIC/preferences  (Preferences.java:175)
INFO  Searching for classes...  (ClassSearcher.java:411)
INFO  Ignoring class 'ghidra.GhidraClassLoader' from '/Applications/ghidra_11.1_PUBLIC/Ghidra/Framework/Utility/lib/Utility.jar'. Already found at '/Applications/ghidra_11.1_PUBLIC/Ghidra/Framework/Utility/lib/Utility.jar'.  (ClassSearcher.java:454)
INFO  Ignoring class 'generic.jar.GClassLoader' from '/Applications/ghidra_11.1_PUBLIC/Ghidra/Framework/Utility/lib/Utility.jar'. Already found at '/Applications/ghidra_11.1_PUBLIC/Ghidra/Framework/Utility/lib/Utility.jar'.  (ClassSearcher.java:454)
INFO  Class search complete (1420 ms)  (ClassSearcher.java:137)
INFO  Initializing SSL Context  (SSLContextInitializer.java:76)
INFO  Initializing Random Number Generator...  (SecureRandomFactory.java:37)
INFO  Random Number Generator initialization complete: NativePRNGNonBlocking  (SecureRandomFactory.java:41)
INFO  Trust manager disabled, cacerts have not been set  (ApplicationTrustManagerFactory.java:95)
INFO  User heartpunk started Ghidra.  (GhidraRun.java:75)
INFO  User settings directory: /Users/heartpunk/Library/ghidra/ghidra_11.1_PUBLIC  (GhidraRun.java:76)
INFO  User temp directory: /var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/heartpunk-ghidra  (GhidraRun.java:77)
INFO  User cache directory: /var/tmp/heartpunk-ghidra  (GhidraRun.java:78)
DEBUG Recovery snapshot timer set to 5 minute(s)  (RecoverySnapshotMgrPlugin.java:170)
INFO  Opening project: /Users/heartpunk/REDACTED  (DefaultProject.java:115)
INFO  Ghidra startup complete (4860 ms)  (GhidraRun.java:92)
INFO  Packed database cache: /var/tmp/heartpunk-ghidra/packed-db-cache  (PackedDatabaseCache.java:64)
DEBUG Using cached packed database: /Applications/ghidra_11.1_PUBLIC/Ghidra/Features/Base/data/typeinfo/mac_10.9/mac_osx.gdt  (PackedDatabaseCache.java:364)
DEBUG New Pty: /dev/ttys012 at (183,184)  (UnixPty.java:48)
INFO  local Pty session. PID = 70599  (LocalProcessPtySession.java:34)

Wow, ok, so I ran the debugger on a copy of the decompiler included with Ghidra, which I'm guaranteed to have debug access to. It lives at ghidra_11.1_PUBLIC/Ghidra/Features/Decompiler/os/mac_arm_64/decompile. I used the following options:  Gave me a pop-up about being unable to check for malware, but then:  which looks a lot like your results except for the error.
I will play around with this some more tonight and drag in the big guns tomorrow. I am definitely not seeing/thinking of something, because it really seems like you did all the right things.

just tried again w/confirming those options, and fyi the results of doing so are included in the above terminal dump, which is from the ghidra terminal run w/ghidraDebug

OK, thanks!  Although, sadly, not a single clue there.  You're using jdk 22 vice my 21.  I may download and try that, but that seems very unlikely.  Let me cogitate a bit more.  More as I know more - you may be the topic of tomorrow's team meeting.  :)

Thanks for all the help!!! Hope the meeting proves fruitful.

hard to type too much to explain what's going on but i believe the solution is explained between these two pastes. I'm pretty sure ghidra won't look in the local home directory for package installs, but system python isn't writable by default in osx!
from my terminal, before ghidra interaction
heartpunk@REDACTED ~ % sha256sum /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/commands.py
fd9562b0436095a931c6b6abecfc7d971a02d4ff5b73d060af065723302a7f3a  /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/commands.py
heartpunk@REDACTED ~ % ls /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb
__init__.py	arch.py		commands.py	hooks.py	methods.py	schema.xml	util.py
heartpunk@REDACTED ~ % ls l /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb
ls: l: No such file or directory
/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb:
__init__.py	arch.py		commands.py	hooks.py	methods.py	schema.xml	util.py
heartpunk@REDACTED ~ % ls -l /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb
total 320
-rwxr-xr-x@ 1 heartpunk  staff    614 Jun  7 14:32 __init__.py
-rwxr-xr-x@ 1 heartpunk  staff  11832 Jun  7 14:32 arch.py
-rwxr-xr-x@ 1 heartpunk  staff  68436 Jun  7 14:32 commands.py
-rwxr-xr-x@ 1 heartpunk  staff  26249 Jun  7 14:32 hooks.py
-rwxr-xr-x@ 1 heartpunk  staff  23088 Jun  7 14:32 methods.py
-rwxr-xr-x@ 1 heartpunk  staff  14552 Jun  7 14:32 schema.xml
-rwxr-xr-x@ 1 heartpunk  staff   7311 Jun  7 14:32 util.py
heartpunk@REDACTED ~ % wc -l /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/*
      16 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/__init__.py
     336 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/arch.py
    2031 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/commands.py
     714 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/hooks.py
     684 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/methods.py
     304 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/schema.xml
     258 /Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src/ghidralldb/util.py
    4343 total

from ghidra terminal
/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applica
tions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-13<IP_ADDRESS>.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
(lldb) script import ghidralldb
(lldb) target create "REDACTED"
Current executable set to 'REDACTED' (arm64
).
(lldb) ghidra trace connect "<IP_ADDRESS>:61646"
error: 'ghidra' is not a valid command.
(lldb) dir(ghidralldb)
error: 'dir' is not a valid command.
(lldb) import ghidralldb
error: 'import' is not a valid command.
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> import ghidralldb
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src
/ghidralldb/__init__.py", line 16, in <module>
    from . import util, commands
  File "/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src
/ghidralldb/commands.py", line 29, in <module>
    from ghidratrace.client import Client, Address, AddressRange, TraceObject
  File "/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src/
ghidratrace/client.py", line 27, in <module>
    from . import trace_rmi_pb2 as bufs
  File "/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src/
ghidratrace/trace_rmi_pb2.py", line 5, in <module>
    from google.protobuf.internal import builder as _builder
ModuleNotFoundError: No module named 'google'
>>> import sys.executable
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'sys.executable'; 'sys' is not a package
>>> import sys
>>> print(sys.executable)
/Applications/Xcode.app/Contents/Developer/usr/bin/lldb
>>> print(sys.version)
3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]

same terminal session as before resumed after ghidra
heartpunk@REDACTED ~ % python3 --version
Python 3.12.3
heartpunk@REDACTED ~ % /usr/bin/python3 --version
Python 3.9.6
heartpunk@REDACTED ~ % /usr/bin/python3 -m pip install psutil
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: psutil in ./Library/Python/3.9/lib/python/site-packages (5.9.6)
WARNING: You are using pip version 21.2.4; however, version 24.0 is available.
You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command.
heartpunk@REDACTED ~ % /usr/bin/python3 -m pip install google
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google in ./Library/Python/3.9/lib/python/site-packages (3.0.0)
Requirement already satisfied: beautifulsoup4 in ./Library/Python/3.9/lib/python/site-packages (from google) (4.12.3)
Requirement already satisfied: soupsieve>1.2 in ./Library/Python/3.9/lib/python/site-packages (from beautifulsoup4->google) (2.5)
WARNING: You are using pip version 21.2.4; however, version 24.0 is available.
You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command.

after checking, even sudo won't resolve this
so i think we're making progress at least???

Hmmm, you shouldn't have to sudo (and in my (very limited) experience sudo'ing with python is fraught with peril.  I get the "Defaulting to user..." messages all the time without ill effect (other than annoyance at the messages).
In your second pane, could you try "dir(ghidralldb)" and "dir(ghidralldb.commands)" after "script" and "import ghidralldb"?  You need to be in the scripting shell for those to mean anything.

And, yes, on progress - I think!

I did the import again after entering script in the paste above!

Said import failed w/a traceback around the google module not being available, hence checking the python version and packages again.

heartpunk@REDACTED ~ % /usr/bin/python3 -m pip install google
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google in /Library/Python/3.9/site-packages (3.0.0)
Requirement already satisfied: beautifulsoup4 in ./Library/Python/3.9/lib/python/site-packages (from google) (4.12.3)
Requirement already satisfied: soupsieve>1.2 in ./Library/Python/3.9/lib/python/site-packages (from beautifulsoup4->google) (2.5)
WARNING: You are using pip version 21.2.4; however, version 24.0 is available.
You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command.
heartpunk@REDACTED ~ % /usr/bin/python3
Python 3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'google'

from regular terminal

I think that's ok - try "import protobuf", which is what we're using in the google package.

Hmmm, I could be wrong about that.  Try:
lldb
(lldb) script

import google

heartpunk@REDACTED ~ % /usr/bin/python3
Python 3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import protobuf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'protobuf'
>>> ^D
heartpunk@REDACTED ~ % lldb
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> import google
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'google'
>>> import protobuf
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'protobuf'

OK, I think I am seeing your point above.
On my box, I have write permissions to ~/Library/Python/3.9/lib/site-packages.  Do you?

Sorry - lib/python/site-packages.  And in it I have:
drwxr-xr-x   3 llero  staff   96 Mar  5 10:57 google
-rw-r--r--   1 llero  staff  539 Mar  5 10:57 protobuf-3.20.0-nspkg.pth
drwxr-xr-x  10 llero  staff  320 Mar  5 10:57 protobuf-3.20.0.dist-info

heartpunk@REDACTED ~ % ls -l ~/Library/Python/3.9/lib/python
total 0
drwx------  224 heartpunk  staff  7168 Jun 12 09:04 site-packages
heartpunk@REDACTED ~ % ls -l ~/Library/Python/3.9/lib/python/site-packages | egrep google\|protobuf
sophiesmithburg@Sophies-MacBook-Pro ~ %

Any chance any of the parent dirs have less then full permissions for you?

Am stuck on this post you made:
heartpunk@REDACTED ~ % /usr/bin/python3 -m pip install google
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google in /Library/Python/3.9/site-packages (3.0.0)
Requirement already satisfied: beautifulsoup4 in ./Library/Python/3.9/lib/python/site-packages (from google) (4.12.3)
Requirement already satisfied: soupsieve>1.2 in ./Library/Python/3.9/lib/python/site-packages (from beautifulsoup4->google) (2.5)
WARNING: You are using pip version 21.2.4; however, version 24.0 is available.
You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command.
heartpunk@REDACTED ~ % /usr/bin/python3
Python 3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import google
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'google'

That seems like it's telling you that you can't "import" from the system site-packages.  Do you read it that way?

Any chance any of the parent dirs have less then full permissions for you?

heartpunk@REDACTED ~ % touch ~/Library/Python/3.9/lib/python/site-packages/foo
heartpunk@REDACTED ~ % ls -l ~/Library/Python/3.9/lib/python/site-packages | egrep foo
-rw-r--r--    1 heartpunk  staff       0 Jun 12 09:48 foo

seems no

Am stuck on this post you made:
heartpunk@REDACTED ~ % /usr/bin/python3 -m pip install google Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: google in /Library/Python/3.9/site-packages (3.0.0) Requirement already satisfied: beautifulsoup4 in ./Library/Python/3.9/lib/python/site-packages (from google) (4.12.3) Requirement already satisfied: soupsieve>1.2 in ./Library/Python/3.9/lib/python/site-packages (from beautifulsoup4->google) (2.5) WARNING: You are using pip version 21.2.4; however, version 24.0 is available. You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command. heartpunk@REDACTED ~ % /usr/bin/python3 Python 3.9.6 (default, Mar 29 2024, 10:51:09) [Clang 15.0.0 (clang-15<IP_ADDRESS>)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import google
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'google'

That seems like it's telling you that you can't "import" from the system site-packages. Do you read it that way?

yea that seems sensible

Maybe another sanity check:  /usr/bin/python3 -m pip -V

Hmmm, mine is:
pip 21.2.4 from /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/pip (python 3.9)
but I feel like that should only matter if there's a permissions issue.

OK, @nsadeveloper789  is suggesting "/usr/bin/python3 -m pip install protobuf=3.20.3"  (vs. google) and then verifying with "from google import protobuf".

heartpunk@REDACTED ~ % /usr/bin/python3 -m pip install protobuf==3.20.3
Defaulting to user installation because normal site-packages is not writeable
Collecting protobuf==3.20.3
  Using cached protobuf-3.20.3-py2.py3-none-any.whl (162 kB)
Installing collected packages: protobuf
Successfully installed protobuf-3.20.3
WARNING: You are using pip version 21.2.4; however, version 24.0 is available.
You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command.
heartpunk@REDACTED ~ % /usr/bin/python3
Python 3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from google import protobuf

OK, wow - that might have worked.  (I'm going to have to get him to explain the difference to me.)  Maybe try Ghidra now?

Seems to be working??? I've never seen it that way before so I can't be entirely sure, but this seems like it might be a debugger waiting for me to play with it!
I'd screenshot but no easy way to redact.

I'll keep poking at it and if it does seem to actually be working still, I'll close this a bit later!

/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applica
tions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-13<IP_ADDRESS>.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
(lldb) script import ghidralldb
(lldb) target create "REDACTED"
Current executable set to 'REDACTED' (arm64
).
(lldb) ghidra trace connect "<IP_ADDRESS>:62869"
(lldb) ghidra trace start
(lldb) ghidra trace sync-enable
(lldb) process launch --stop-at-entry
add listener for process failed
add initial events for cli failed
add initial events for target failed
add initial events for process failed
Process 59513 exited with status = -1 (0xffffffff) attach failed (Not allowed to at
tach to process.  Look in the console messages (Console.app), near the debugserver
entries, when the attach failed.  The subsystem that denied the attach permission w
ill likely have logged an informative message about why it was denied.)
Process 59513 launched: 'REDACTED' (arm64)
(lldb)

Yep re redaction.  At this point, I'd see if you can repeat the tutorial results.  Possibly before tackling your own target.

Ahh, good idea. I'll do that. The above paste is from my target.

(Ahh, come to think of it, the above is almost certainly just SIP being on still.)

AHHH, that looks great!  The permissions thing is standard Apple security stuff.  Basically, lldb has to have debugging permissions (which it should unless you built it from scratch) and the target has to have "allow debugging".

Yes, re SIP - although if I remember correctly, you can enable per-target debugger with SIP still enabled.   Will do some googling.

Oh if you find a pointer for that it'd be great. I'd prefer to not disable SIP unless necessary!

OK, took me a while but....
Copy your target somewhere not in a system/system-related directory.  If it's a package, you may want to extract the arm64 or x64 executable from it.  Make sure the copy is still executable. Then "codesign -s lldb_codesign -f target" (where lldb_codesign is a certificate I'd already made with debug permissions - details here https://stackoverflow.com/questions/58356844/what-are-the-ways-or-technologies-to-sign-an-executable-application-file-in-mac ).  Should (fingers-crossed) be debuggable then.

This is probably starting to get into a new issue ... oh right I forgot to do the test targets first but, since I've got the info anyway I'll share it!
The trace REDACTED.3 has no mapping to its program AARCH64-64-cpu0x0
---------------------------------------------------
Build Date: 2024-Jun-07 1416 EDT
Ghidra Version: 11.1
Java Home: /opt/homebrew/Cellar/openjdk/22.0.1/libexec/openjdk.jdk/Contents/Home
JVM Version: Homebrew 22.0.1
OS: Mac OS X 14.5 aarch64

It seems like the program is partly launching to debug but partly not? Unclear!
The ghidra terminal output for completness' sake:
/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applicat
ions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-13<IP_ADDRESS>.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
(lldb) script import ghidralldb
(lldb) target create "/Users/heartpunk/REDACTED"
Current executable set to '/Users/heartpunk/REDACTED' (arm64).
(lldb) ghidra trace connect "<IP_ADDRESS>:64173"
(lldb) ghidra trace start
(lldb) ghidra trace sync-enable
add listener for process failed
add initial events for cli failed
add initial events for target failed
add initial events for process failed
(lldb) process launch
Process 6231 launched: '/Users/heartpunk/REDACTED' (arm64)
2024-06-12 11:48:41.032869-0700 REDACTED[6231:19698698] Failed to check bundle ID.
Process 6231 exited with status = 173 (0x000000ad)
(lldb) Couldn't record page with PC: Cannot convert '$pc' to address: unable to eval
uate expression while the process is exited: the process must be stopped because the
 expression might require allocating memory.
Couldn't record page with SP: Cannot convert '$sp' to address: unable to evaluate ex
pression while the process is exited: the process must be stopped because the expres
sion might require allocating memory.

(the codesign thing worked! wouldn't be here w/o it.)

OK, so looks like your process terminated - did you use the "process launch --stop-at-entry" option?

It seems to quite self evidently be working now!!! This looks just like a debugger should.

It does die as soon as I resume it though, huh.

hmmm, well, first thing I woould check - does it behave that way in lldb w/o Ghidra or jusst in Ghidra?   If just in Ghidra, one of two (?) things is happening.  Either something went wrong in the Ghidra logic (possible) or we haven't registered a break for some exception that lldb has.  It's been a while since I've looked at the lldb commands for handling execptions and events.  I thought we default to break for most of them, but would have to check.  All of that would also have to be handled through the command-line.  Am sure we don't have GUI elements for those yet.
If Ghidra went south, there should be errors in either the Debug Console in the GUI or the ghidraDebug shell if you're using it.
If lldb's behavior is the same, you either need to set up the exception handling or some set of breakpoints.  Also, could be that the target has detected the debugger and is punting (if it's that kind of app).

"Use breakpoint set -E c++ to break on all exceptions and breakpoint set -F std::range_error  to break on a specific exception."  from stackoverflow

Giving this a try now!

two things:

the no mapping to program error remains
when trying to do the plain lldb launch of the program, i get:

heartpunk@REDACTED ~ % lldb
(lldb) script import ghidralldb
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ModuleNotFoundError: No module named 'ghidralldb'
(lldb) ^D

Regarding the missing mapping, there could be a lot of reasons for that.  Basically, the error means there's a mismatch between the name known to the OS while executing and the name of the program in Ghidra.  Could be because the program was renamed after it was compiled.  Could have been renamed on import. Could have been renamed in Ghidra after it was imported.  In all of these cases, you can open the Modules window, find the module that is a match, and right-click to  "Map Module to....", which will bring up a dialog.  Select the entry in there, hit OK, and that association will be memoized, allowing the debugger to translate between dynamic and static addresses.
On the second point, I thought we solved that, no?  Or are you running lldb from the command line?  I can't quite tell from your post.  In any case, you have to make sure that either the PYTHONPATH environment variable contains the paths to pypkg/src in Debugger-rmi-trace and Debugger-agent-lldb, or you have to install those packages in lldb's version of python using "python -m pip install" or the equivalent.

I had a similar issue where psutil and protobuf weren't installed. What's interesting is that while psutil shouted ModuleNotFound, protobuf didn't and rather returned ghidra is not a valid command.

Regarding the missing mapping, there could be a lot of reasons for that. Basically, the error means there's a mismatch between the name known to the OS while executing and the name of the program in Ghidra. Could be because the program was renamed after it was compiled. Could have been renamed on import. Could have been renamed in Ghidra after it was imported. In all of these cases, you can open the Modules window, find the module that is a match, and right-click to "Map Module to....", which will bring up a dialog. Select the entry in there, hit OK, and that association will be memoized, allowing the debugger to translate between dynamic and static addresses.

ok, cool

On the second point, I thought we solved that, no? Or are you running lldb from the command line? I can't quite tell from your post. In any case, you have to make sure that either the PYTHONPATH environment variable contains the paths to pypkg/src in Debugger-rmi-trace and Debugger-agent-lldb, or you have to install those packages in lldb's version of python using "python -m pip install" or the equivalent.

that was from the terminal, the paste you're responding to. when I run it from ghidra, the ghidra-internal terminal gives output like:
/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applica
tions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09)
[Clang 15.0.0 (clang-15<IP_ADDRESS>)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-13<IP_ADDRESS>.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-<IP_ADDRESS> clang-15<IP_ADDRESS>)
(lldb) script import ghidralldb
(lldb) target create "/Users/heartpunk/REDACTED"
Current executable set to '/Users/heartpunk/REDACTED' (arm64).
(lldb) ghidra trace connect "<IP_ADDRESS>:61685"
(lldb) ghidra trace start
(lldb) ghidra trace sync-enable
(lldb) process launch --stop-at-entry
add listener for process failed
add initial events for cli failed
add initial events for target failed
add initial events for process failed
Process 59991 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0x0000000100044b70 dyld`_dyld_start
dyld`:
->  0x100044b70 <+0>:  mov    x0, sp
    0x100044b74 <+4>:  and    sp, x0, #0xfffffffffffffff0
    0x100044b78 <+8>:  mov    x29, #0x0
    0x100044b7c <+12>: mov    x30, #0x0
Target 0: (REDACTED) stopped.
Process 59991 launched: '/Users/heartpunk/REDACTED' (arm64)
(lldb)

Perhaps this is just anti reversing stuff, and that's my own problem to look into, perhaps not.

Had a similar issue, which showed:
error: 'ghidra' is not a valid command.
I could track it down to builder.py not present in python3-protobuf-3.19.6-10.fc41.x86_64 package. I did not further investigate, just copied it from official github and it worked. Maybe it helps someone.

I also encountered the error: 'ghidra' is not a valid command.. I thought every libs has been imported properly because there were no issues when running script import ghidralldb in LLDB. However, while modifying the ghidratrace code, I met a ModuleNotFoundError: No module named 'google' error (Unlike psutil, it seems that the ImportError wasn't handled, so the error message didn’t appear.).
To resolve the issue, I created a virtual environment, installed the protobuf library, and added its path to the PYTHONPATH. After that, the error was resolved. My environment is macOS.


skip debian jobs for unsupported architectures

/merge

This may end up being all versions >= 20, but for now
/approve

/approve

/approve


Feat - add list bouncing with VirtualScroll

Short description of the problem:
List bouncing is not support when using virtual scroll
What behavior are you expecting?
List bouncing as exists in native scrolling without virtual scroll
Ionic info
Cordova CLI: 6.1.0
Gulp version:  CLI version 3.9.1
Gulp local:   Local version 3.9.1
Ionic Framework Version: 2.0.0-beta.6
Ionic CLI Version: 2.0.0-beta.25
Ionic App Lib Version: 2.0.0-beta.15
ios-deploy version: Not installed
ios-sim version: 5.0.6
OS: Mac OS X Yosemite
Node Version: v4.3.1
Xcode version: Xcode 7.2.1 Build version 7C1002

Hello! We are moving our feature requests to a new feature request doc. I have moved this feature request to the doc and because of this I will be closing this issue for now. Thanks for using Ionic!

With iOS's UIWebView implementation this feature will probably not be added. Instead we recommend using the WKWebView plugin, and we'll keep working to improve virtual scroll, thanks.
http://blog.ionic.io/cordova-ios-performance-improvements-drop-in-speed-with-wkwebview/


Fix header issue

When stress testing a new DOM method, I found an error with our rules system that creates multiple rules for the same domain, breaking our header functionality when toggling on and off. Currently, I'm thinking of putting a check in the rule creator to see if a rule with that domain already exists.

There are a few issues I am noticing here. For some reason, when I load the extension it loads with some rules in place for global-privacy-control.glitch.me. This can be solved by deleting our dynamic rules on load, but then a helper function should be in place to rebuild the desired rules. Another issue I am seeing is that sometimes the incorrect rule is deleted.
I find it odd that these issues were not found earlier. There might be a chance that work in another branch is somehow impacting other branches I travel to (perhaps through chrome caching?). I have tried to clear caches to remove this variable but it does not seem to perform differently. I have ran into other strange errors along this process like npm run start not loading the most recent version of the code, but that may be unrelated.

I find it odd that these issues were not found earlier.

It is well possible that there are bugs in the code that we introduced quite a while ago. So, it is good that you point them out. If appropriate, please open new issues on those.

This issue had NOT existed before and had instead been caused by a small mistake I made. It is resolved so this issue will be closed.


feat(mod): support udd-style fragments

Looks like my implementation was good enough to incorporate this with zero-cost :smile_cat:


fix: Update README

Changing some language in the VSCode product description

:tada: This PR is included in version 0.28.1 :tada:
The release is available on:

v0.28.1
GitHub release

Your semantic-release bot :package::rocket:


What is the purpose/function of overlap?

My understanding is that overlap just specifies how many frames overlap when splitting a video into segments. If that's correct, what's the purpose? Does it actually do anything for the user? (E.g. if I do the tracks on the first segment, are they copied across to the next segment in the overlapping region? This doesn't appear to be the case.) Put another way- should I just set overlap=0 in my video tagging examples, to avoid having to manually resolve different taggings from each segment in the overlap?
Sorry if I've misunderstood something obvious.
PS - great tool!

Hi @kodonnell ,
Thanks for your question and feedback. Your understanding is correct. There are several ways to use the parameter:

For an interpolation task (video sequence) if an object exists on overlapped segments it will be automatically merged into one track (with overlap=0 in annotation file you will have several tracks, one for each segment, which correspond to the object): https://github.com/opencv/cvat/blob/master/cvat/apps/engine/annotation.py#L824

For an annotation task (separate images) if an object exists on overlapped segments bounding boxes will be automatically merged into one (if it is possible):  https://github.com/opencv/cvat/blob/master/cvat/apps/engine/annotation.py#L897

Use overlap=n to estimate quality of the annotations (it isn't implemented)

Does it answer your question?

Thanks for the thorough response @nmanovic

For an interpolation task (video sequence) if an object exists on overlapped segments it will be automatically merged into one track ...

Hmmm - I thought I tested this by creating a task with multiple segments and overlaps. When I added a track to the first segment (I just left one box in the same place the whole time), then closed it and opened the next segment, I didn't see the track in the overlap. Is there a step I missed? (I can also try to reproduce - I'm going by memory, so might not have done it correctly.)

For an annotation task (separate images) ...

Just to clarify for others reading this: I take it by 'separate images' you just mean single-frame annotations on the video? (Given CVAT is only for video.) Otherwise I'm not sure how separate images can overlap?

Use overlap=n to estimate quality of the annotations (it isn't implemented)

OK, I'm getting a little confused - I thought overlap=n meant there were n frames overlapping between segments?
Just thinking - maybe a video illustrating this would be good? I found the annotation video really useful, so maybe something quick demonstrating how segments/overlap work (and what happens in edge case you refer to e.g. "automatically merged into one (if it is possible)").

I didn't see the track in the overlap. Is there a step I missed?

You have to annotate it explicitly on all segments. After that merge will work. It will not propagate a bounding box from one segment to another automatically because it common case it isn't correct. Thus you annotate an object on first segment. You annotate the same object on second segment and if you do it right you will have one track inside your annotation file. If annotations on different segments (on overlapped frames) are very different or match is bad or overlap is 0 you will have 2 tracks for the same object.

I take it by 'separate images' you just mean single-frame annotations on the video? (Given CVAT is only for video.)

CVAT can annotate video or images (you can upload a set of images or even an archive with images). For example, you upload 100 images, segment size is 50 and overlap is 10. In this case you should have (if I remember correctly) 3 segments: 0 - 49, 40 - 89, 80 - 99. Thus two adjacent segments have 10 images overlap. In general "overlapped" images can be annotated differently. Merge works in such cases.

Thanks for the clarification.

It will not propagate a bounding box from one segment to another automatically because in common case it isn't correct.

In our case it would be correct to do this - we'd prefer to label something only once. I.e. I add tracks in segment 1, and then when I go to segment 2, it carries across all tracks from e.g. the last 10 frames of segment 1 (for overlap = 10) to the first 10 frames of segment 2. There won't be any need to merge the tracks, as it's obvious that if you've copied track 1, then it's still track 1. However - maybe ours is a special use case (and we have no experience with big tagging tasks).
Anyway, thank @nmanovic  - feel free to close the task, or leave it open as a reminder to update the docs etc.


Uncaught TypeError: Cannot read property 'id' of null

I'm getting this with error from<EMAIL_ADDRESS>with the current dev version of talenet.
I tried to get all deps updated to muxrpc@7, here is our current npm ls.txt.
/home/user/TALEnet/node_modules/push-mux/index.js:217 Uncaught TypeError: Cannot read property 'id' of null
    at Mux._credit (/home/user/TALEnet/node_modules/push-mux/index.js:217)
    at Sub._preread (/home/user/TALEnet/node_modules/push-mux/sub.js:54)
    at Sub.DuplexStream._write (/home/user/TALEnet/node_modules/push-mux/stream.js:31)
    at writeDataToStream (/home/user/TALEnet/node_modules/push-mux/index.js:83)
    at Mux.write (/home/user/TALEnet/node_modules/push-mux/index.js:161)
    at next (/home/user/TALEnet/node_modules/push-stream-to-pull-stream/sink.js:30)
    at /home/user/TALEnet/node_modules/pull-goodbye/node_modules/pull-stream/throughs/filter.js:17
    at Object.cb (/home/user/TALEnet/node_modules/packet-stream-codec/index.js:111)
    at drain (/home/user/TALEnet/node_modules/pull-reader/index.js:39)
    at more (/home/user/TALEnet/node_modules/pull-reader/index.js:51)

thanks, fixed in 1.0.4

this should be fixed in<EMAIL_ADDRESS>

Thanks! Tried and didn't appear yet!


Other languages/ESP-IDF support question

Hi guys,
First off, thanks for putting WAMR together.
I'm working on a project called Nihilo (https://github.com/Threadnaught/Nihilo) which uses WAMR as part of a computing platform built around remote procedure calls and ECC encryption. The communication, storage and cryptography modules have all been implemented on linux so far.
I would like to run Nihilo on the ESP32, which uses the ESP-IDF and FreeRTOS. I can see that you've already done the work to support xtensa, but the IDF is not yet one of the platforms. Is this something that's on your priority list, or would I have to port it myself?
The other question is regarding other languages which compile directly into webassembly (like golang, C#, C++ etc). Is there any way to get these to use the provided c sysroot, or would a new sysroot need to be provided for each?

Hi, thanks for using WAMR in Nihilo project!
For FreeRTOS, we have some experiment code for the platform, but haven't tested it yet. I just imported the code to the project, could you please help enhance it, make it work and then submit the patch back to github? Thanks a lot.
Most of the platform related APIs should have been implemented, except os_printf, os_vprintf, os_thread_get_stack_boundary and some libc APIs, see core/shared/platform/freertos folder. And there is a sample makefile source.mk under product-mini/platforms/freertos/simple for a ref. Maybe you can implement os_printf and ref source.mk to enable the product-mini simple project firstly.
For the compiler, there are many toolchains, e.g. wasi-sdk and emcc for C/C++, tinygo for go, asc for typescript, rustc/cargo for rust, please ref to the related documents for more details. Or ref to doc/build_wasm_app.md for the detail usage of wasi-sdk.

FreeRTOS is not a full platform, but a collection of primitives for concurrency built on top of existing microcontroller APIs. Implementations vary, so a function that is present in one FreeRTOS-based API (like printf in ESP-IDF and Linux) may vary on another (like Serial.Println() on arduino-based platforms).

Yes, seems that we should re-org the code structure, how about: (1) putting all the common APIs to folder core/shared/platform/common/freertos, these APIs can be used by all FreeRTOS-based platforms, (2) and letting the platform e.g. ESP-IDF implement the platform dependent APIs, e.g. printf? And the platform should define its own platform macro, e.g. BH_PLATFORM_ESP_IDF.

I'm investigating this fork https://github.com/nickzoic/wasm-micro-runtime (branch wamr-for-esp-idf). It has the same license so depending on its quality it could be integrated back into WAMR.

Yes, there is really some useful files to ref to from the fork. But its code base is a little old, some files and codes have been updated in the latest main code base, so it might be difficult to use all its files. We re-orged the code structure in latest patch #454: (1) remove core/shared/platform/freertos folder, (2) add folder core/shared/platform/common/freertos with common-used freertos APIs, (3) and add esp-idf platform: core/shared/platform/esp-idf and product-mini/platforms/esp-idf. All the source files should have been prepared, and the CMakeLists.txt in product-mini/platforms/esp-idf was drafted ref to the fork you mentioned and the CMakeLists.txt in other platforms. We haven't compiled it yet, could you please have a look, compile it and fix the compilation issues? The efforts should be not so large as before. Thanks a lot.

I got that fork compiling for esp-idf, I am attempting to resolve some issues with it before I bringing it into this repo. I might need someone to look over the CMakeLists.txt - It's not exactly my strong suit and this one has been through three different hands.

I've been trying to get the code in the above mentioned fork working since I posted that last comment, and unfortunately I cannot; I suspect it never fully worked in the first place. I'm closing this issue for now because I need to put a lot more time into learning ESP-IDF and XTENSA before I'll be in a place to make a meaningful contribution along these lines. Thanks again for making WAMR, sorry I couldn't be more help.


required properties will trigger a AlbaJsonFormatterException when requesting a nested object

Alba 7.4.1.
.NET 8
xunit 2.5.3
When using these models:
public class RequiredModel
{
    public required string Id { get; set; }
}

public class OptionalModel
{
    public required string Id { get; set; }
}

public class Group
{
    public required RequiredModel RequiredModel { get; set; }
    public OptionalModel? OptionalModel { get; set; }
}

Calling this API:
[Produces("application/json")]
[HttpGet(Name = "GetWeatherForecast")]
public IActionResult Get()
{
    return Ok(new Group { RequiredModel = new RequiredModel() { Id = "test" } });
}

This code will fail
[Fact]
public async Task Test1()
{
    await using var host = await AlbaHost.For<global::Program>();

    // Arrange
    // Act
    var response = await host.Scenario(_ =>
    {
        _.Get.Url($"/WeatherForecast");
        _.StatusCodeShouldBeOk();
    });

    // Assert
    var result = await response.ReadAsJsonAsync<RequiredModel>();
}

The ReadAsJsonAsync will fail with this exception:
Alba.AlbaJsonFormatterException : The JSON formatter was unable to process the raw JSON:
{"requiredModel":{"id":"test"},"optionalModel":null}

There are 2 workarounds:

Removing the required from the RequiredModel.Id, everything works properly.
Requesting the Group rather than RequiredModel in the Assert:

var result = await response.ReadAsJsonAsync<Group>();

Expected
Using RequiredModel as in the code above should work with or without the required keyword.

I'm not sure why you're trying to deserialize a nested entity directly. We don't say we support this anywhere, nor is it an expected convention within .NET.

Removing the required from the RequiredModel.Id, everything works properly.

Are you sure? Unless you're running custom converters the deserializer is going to return null (STJ) or give you a default object (Newtonsoft).

I'm inheriting some code here.
However, "removing the required", I am 100% sure 😄
I thought I was stuck and found the workaround.
I don't need further support but just wanted to let you know in case this is something that was important that you might want to fix. I thought you might be using the MS libraries, but I saw some custom JSON parsing, so I'm not 100% sure if this is your bug or MS bug.
I have pushed the code here so you can give it a shot:
https://github.com/jsgoupil/Alba.Required
You're welcome to close the bug if you don't need to go further. I will use the Group in this case.

However, "removing the required", I am 100% sure 😄

I tested your example, Id returns null instead of "test" when I removed the required modifier, which aligns with my expectations of this scenario not being supported:

I don't need further support but just wanted to let you know in case this is something that was important that you might want to fix. I thought you might be using the MS libraries, but I saw some custom JSON parsing, so I'm not 100% sure if this is your bug or MS bug.

It's on the Microsoft side, for controllers we use the built-in MVC input/formatters and they in turn call whichever serializer you have configured. Alba does very little custom behavior at the serialization level.

Thanks for your support and library

Enjoy!


icloud email support

Is this platform support?
I get the following error
custom_components.mail_and_packages] Unexpected error fetching Mail and Packages (imap.mail.me.com) data: 'NoneType' object has no attribute 'split'
Traceback (most recent call last):

A bug report isn't the place to ask questions, please use the github discussions sections for Q&A.


Model update activity generated even when not updated?

Hi, probably an issue with my code here, but I'm calculating and updating a value upon a page read.
Effectively:
@this = that + something + else
Model.where(x,y,z).update(this: @this)

99% of the time, Model.this will equal @this, so no update is actually happening. However, even if the database isn't being updated, public_activity generates an activity record.
I ended up wrapping it in an if condition:
if @this != Model.this
  Model.update...
end

Is there a better way?
Thanks.

It looks like the ActiveRecord still triggers a callback even though nothing changed. Sadly it's out of the gem's scope.

Thanks. Figured as much, but thought I'd ask.

For the record, it seems update_columns is what I want.
http://apidock.com/rails/ActiveRecord/Persistence/update_columns


test flake: c_linux_ubsan_native.bins/ubsan/h2_census_nosec_test shutdown_finishes_calls GRPC_POLL_STRATEGY=epollsig

https://grpc-testing.appspot.com/job/gRPC_pull_requests_ubsan_c/181/
D0512 14:21:57.100180049    4432 test_config.c:403]          test slowdown factor: sanitizer=5, fixture=1, poller=1, total=5
I0512 14:21:57.100420302    4432 ev_epollsig_linux.c:90]     epoll engine will be using signal: 40
D0512 14:21:57.100449777    4432 ev_posix.c:121]             Using polling engine: epollsig
D0512 14:21:57.100482267    4432 dns_resolver.c:316]         Using native dns resolver
I0512 14:21:57.100577774    4432 shutdown_finishes_calls.c:53] Running test: test_early_server_shutdown_finishes_inflight_calls/chttp2/fullstack+census
E0512 14:22:47.165205567    4432 cq_verifier.c:228]          no event received, but expected:
0x65 GRPC_OP_COMPLETE success=1 test/core/end2end/tests/shutdown_finishes_calls.c:153

*******************************
Caught signal SIGABRT
bins/ubsan/h2_census_nosec_test[0x101d310]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f2c88f32890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f2c8868c067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f2c8868d448]
bins/ubsan/h2_census_nosec_test[0x5fb1d8]
bins/ubsan/h2_census_nosec_test[0x5f2c5a]
bins/ubsan/h2_census_nosec_test[0x5a1a96]
bins/ubsan/h2_census_nosec_test[0x59e8a3]
bins/ubsan/h2_census_nosec_test[0x42f689]
bins/ubsan/h2_census_nosec_test[0x429352]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2c88678b45]
bins/ubsan/h2_census_nosec_test[0x4054fd]

Duplicate of: #10653


Doesn't work in latest preview build (14328)

This no longer works in the latest insider preview build.
Thanks.

Duplicate of https://github.com/stascorp/rdpwrap/issues/79


[Channels] Sending server-side messages

With channels merged, we have a nice way to trigger server-side changes based on client-sent websocket messages, but it would be really nice to also be able to send messages from the server to a particular channel easily.
Needs:

Send a 'message to all subscribers for a particular topic
Send a 'message to a specific subscriber for a particular topic
(sorta a prereq for no. 2) List some minor metadata for all subscribers/specific topic subscribers/a single topic subscriber (metadata would include guid identifier, perhaps some subset of ctx connection data, like IP)

I think a decent mechanism for this would be to wrap the current channel state dictionary in a type that can be used to provide these methods, then register that type with the AspNetCore DI system for use in the rest of the server.
Speclet:

type SubscriberInfo =
  { Id: Guid
    IP: IpAddress }

type ChannelHub() =
  member x.SendAll(topic: string, msg: 'a): Task<unit>
  member x.SendSubscriber(topic: string, subscriber: Guid, msg: 'a): Task<unit>
  member x.Subscribers(): Task<Map<string, SubscriberInfo list>>
  member x.SubscribersForTopic(topic: string): Task<SubscriberInfo list>
  member x.Subscriber(topic: string, id: Guid): Task<SubscriberInfo option>

And the ChannelHub would be injected into the DI context at startup, and be backed by the internal sockets dictionary (or some derivative of that).
So in a handler we could do:

let saveThing next ctx = task {
  let! thing = ctx.BindModel<Thing>()
  let! saveResult = saveThingToDb thing
  let hub = ctx.ResolveService<ChannelHub>()
  let msg: NewThingMsg = { Id: saveResult.Id }
  do! hub.SendAll("thing.new", msg)
  return! setStatusCode 201 next ctx
}

An initially-untyped interface seems fine, because if a user wanted they could easily build a typed-hub on top of this:
type Hub<'msg>(hub: ChannelHub, topic: string) =
  member x.Send(m: 'msg) = hub.SendAll(topic, m)

SubscriberInfo should probably hold the HttpContext of the request that initially started the websocket.  I know that's kind of dangerous but pulling off all the possible things a user would want might be tricky (Things like User if authenticated, IPAddress, etc).  We could put some members on there to make apparent the user shouldn't access that HttpContext directly but I wouldn't want to hide it entirely.

Yeah I was thinking we might want it, but deliberately leaning on being conservative for the first pass. Worst case you could make the subscriberinfo type user-defined/generic and let them pass a generation-function to make the subscriberinfo

Additionally this gets into dotnet websockets aren't threadsafe territory.  We can either steal from https://github.com/TheAngryByrd/FSharp.Control.WebSockets/tree/master/src or reference it. It needs a bit of love though.

Sorta-related: channel topics in Phoenix are somewhat used for routing.  They can be of the following forms:

Topic
Topic:Subtopic
Topic:* // meaning wildcard, any Subtopic

Is this something that we want to explicitly support? right now the wildcard matching isn't a thing from what I can read.

Fixed by #174


Hyperlink detection?

https://crates.io/crates/supports-hyperlinks

Focus has shifted to anstream.  This doesn't fit within anstream and a crate exists for this space.


[B] Downloads dont contain correct files

I just download it and the example files have wrong links. It seems files are missing.
There is no folder dist and none of the files excist. There a 2 TOC.js files but they give errors if i link it to them.

On the page
there also link to https://raw.githubusercontent.com/jgallen23/toc/master/dist/toc.min.js
These give 404

Well i got some files going trough some of the branches, though its old. It works kind of nice with some adjusted styles.
http://www.promotiespullen.com/opmaak-en-bestanden/opmaak-en-bestanden

I have the same issue, the examples and documentation are not up-to-date.

It works only from 0.3.2... if you add jQuery yourself:
https://github.com/jgallen23/toc/releases/tag/0.3.2


Autocomplete Not Working

I am trying to set up autocomplete, I added the extension to obsidian, toggled it on, and I have tried to scan my files, toggle autocomplete with hotkey and command palette, restart obsidian, toggle off some other extensions (quick latex, languagetool, etc) but nothing seems to make autocomplete work in any capacity. Nothing comes up as I type or enter latex equations, even if i make a new note, nothing happens when I use the hotkeys or commands, the only thing that the extension seems to do is make everything freeze when I open a PDF, which I have seen in the issues log.
I am on Windows 11, with the extensions: Advanced Tables, Autocomplete, Better PDF Plugin, Calendar, Checklist, Diagrams.net, Excalidraw, LanguageTool Integration, Obsidian Tabs, Pane Relief, Paste URL into Selection, and Quick Latex for Obsidian.
Am I doing something wrong?

Same here! I just install this extension but it doesn work at all.
I am on Windows 10, no other plugins enabled.

Did you try to use the autocomplete WITHOUT the preview mode? the new editor is not supported and the project seems to be unmaintained...

Did you try to use the autocomplete WITHOUT the preview mode? the new editor is not supported and the project seems to be unmaintained...

unfortunately it still doesn't work with preview toggled off :(


AttributeError: 'OneDeviceStrategy' object has no attribute 'call_for_each_tower'

I downloaded the up-to-date source code from this page and build from source. Then I tried to train mnist using the code in https://github.com/tensorflow/models/tree/master/official/mnist
But I got the following error message:

I1203 18:58:06.548315<PHONE_NUMBER>83136 run_config.py:530] Initializing RunConfig with distribution strategies.
I1203 18:58:06.548501<PHONE_NUMBER>83136 estimator_training.py:166] Not using Distribute Coordinator.
I1203 18:58:06.548817<PHONE_NUMBER>83136 estimator.py:197] Using config: {'_model_dir': '/tmp/mnist_model', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.contrib.distribute.python.one_device_strategy.OneDeviceStrategy object at 0x7f77577a59e8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f77577a5ac8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
W1203 18:58:06.572286<PHONE_NUMBER>83136 deprecation.py:317] From /home/kathy/workspace/test/models/official/mnist/dataset.py:100: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1203 18:58:06.609055<PHONE_NUMBER>83136 deprecation.py:317] From /home/kathy/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:175: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
Traceback (most recent call last):
  File "mnist.py", line 236, in <module>
    absl_app.run(main)
  File "/home/kathy/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/kathy/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "mnist.py", line 230, in main
    run_mnist(flags.FLAGS)
  File "mnist.py", line 211, in run_mnist
    mnist_classifier.train(input_fn=train_input_fn, hooks=train_hooks)
  File "/home/kathy/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/kathy/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1181, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/home/kathy/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1287, in _train_model_distributed
    grouped_estimator_spec = self._train_distribution.call_for_each_tower(
AttributeError: 'OneDeviceStrategy' object has no attribute 'call_for_each_tower'

But if I install from tf-nightly from binary, it can be trained. Please help out!Thanks.

I got the same issue when run NCF in tensorflow-models, and I has already know that is caused by mismatching API between tensorflow and tensorflow-estimator.
TF deleted this API call_for_each_tower and used call_for_each_replica instead, but the release package of tensorflow-estimator is too old(release on Sep, 12, 2018) that still used the old API.
Tensorflow-estimator has already changed this API in their master, but they didn't release any new package and I didn't find somewhere to submit the issue, so I just answered this issue and hope someone can help to release a new estimator.

I got the same error when going through TensorFlow Keras guide to the last snippet on colab.


Add support for attributes in rolls

In GitLab by @Asacolips on Dec 19, 2020, 17:24
Currently, roll formulas (such as bond or spells) do not support attributes such as @attributes.str.value

In GitLab by @Asacolips on Dec 19, 2020, 17:43
mentioned in commit d2152044f09f0a68e210c3b8d6fc7286551ff536


"submodelDescriptors" in "shells" exist for jobs requested without aspects

Description
IRS-jobs can be started without any aspects. This jobs are expected to not contain any submodelDescriptors in shells which is currently not always the case. Detected on DEV.
Detected by tavern-test TRI-912.
Current behaviour
submodelDescriptors in shells are responsed for some shells:

Expected behavior
submodelDescriptors in shells are empty.
Steps to reproduce the Bug

Request IRS job without aspects e.g.:
{code}
{
"key": {
"globalAssetId": "urn:uuid:6d505432-8b31-4966-9514-4b753372683f",
"bpn": "BPNL00000003AVTH"
},
"collectAspects": true,
"lookupBPNs": false,
"depth": 1,
"direction": "downward"
}
{code}
Check the submodelDescriptors in shells.

Issue could not be reproduced on DEV anymore, the fix was successful. No submodelDescriptors in shells are present anymore for IRS-jobs requested without aspects. Retested with several parameter and depths.
Ticket is ready for PO-review, FYI @jzbmw.

approved


RDS: improve global clustering behaviour

Closes https://github.com/getmoto/moto/issues/6260

@bblommers could you rerun the failing tests on https://github.com/getmoto/moto/actions/runs/5268843777/jobs/9527894782?pr=6403#step:9:7096
?
It doesn't look like it's my changes, rather a flakey response from servermode tests

Done @rafcio19!

This PR caters for an example presented in https://github.com/getmoto/moto/issues/6260#issuecomment-1586361144, not sure if this is exactly what the OP needs.

Looks like this indeed covers the scenario of https://github.com/getmoto/moto/issues/6260#issuecomment-1523778464
The scenario in the first post isn't covered yet, I think, so I'm not going to close the attached issue yet, but I will merge this.
(
Create primary cluster with create_db_cluster in us-east-1
Create secondary cluster with create_db_cluster in us-west-2 with the primary cluster's arn as ReplicationSourceIdentifier
*Note that the secondary cluster is now added under the primary cluster's ReadReplicaIdentifiers

Create global database with SourceDBClusterIdentifier as the primary cluster's arn.
The global database will have the primary cluster listed as a cluster member, but the secondary cluster is missing from the list.

)


Add COMPAS and Adult dataset.

Currently, I am not able to correctly coerce the features. I will fix this problem in the future.

Thanks a lot for the pull request.
You have added a tuple COERCE_ADULT for the coercing of the scientific types, but if I understand correctly, you are facing issues to apply all of them. I will add a function to simplify that for you. It would simply take the dataframe and apply the transformations you have specified in COERCE_ADULT.

I have made a few changes. These fix the errors that you might be facing with coerce and missing data.
I am merging this Pull Request. As of now, this PR contains only one version of COMPAS Dataset. Maybe in future, we might have both: Full and reduced version of the dataset.

Thank you for your work! Although the origin COMPAS dataset has more columns, not all of them will be used in a classification task.


rebase: update dependencies in build.env

Update all the dependencies to the latest available versions.
Signed-off-by: Madhu Rajanna<EMAIL_ADDRESS>

@Madhu-1 did you miss to squash the commits as all are under same tag rebase?

@Madhu-1 did you miss to squash the commits as all are under same tag rebase?

No, as each is different update, keeping it under different commits for better tracking/understanding.

Error: initializing source docker://registry-ceph-csi.apps.ocp.ci.centos.org/rook/ceph:v1.9.4: reading manifest v1.9.4 in registry-ceph-csi.apps.ocp.ci.centos.org/rook/ceph: manifest unknown: manifest unknown

will update the image in centos/ci branch in a bit.

squashed all commits to one as it doesn't make sense to have multiple commits.

CI jobs can be restarted tomorrow, when the Rook container-image is mirrored (nightly job)

@mergifyio rebase

@mergifyio rebase

/retest all

@mergifyio rebase

Checking RBD (builtin-mgr) stats... 125s
Mon Jun  6 14:25:37 BST 2022: 'kubectl_retry -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath={.items[0].metadata.name}' done (ret=0)
Mon Jun  6 14:25:37 BST 2022: 'kubectl_retry -n rook-ceph get pod rook-ceph-tools-6944779c47-dg25d -ojsonpath={.status.phase}' done (ret=0)
Mon Jun  6 14:26:20 BST 2022: 'kubectl_retry -n rook-ceph get cephblockpools -ojsonpath={.items[0].metadata.name}' done (ret=0)
Checking RBD (builtin-mgr) stats... 130s
Mon Jun  6 14:26:25 BST 2022: 'kubectl_retry -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath={.items[0].metadata.name}' done (ret=0)
Mon Jun  6 14:26:25 BST 2022: 'kubectl_retry -n rook-ceph get pod rook-ceph-tools-6944779c47-dg25d -ojsonpath={.status.phase}' done (ret=0)
Sending interrupt signal to process

I see the above error in https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/upgrade-tests-rbd/detail/upgrade-tests-rbd/5829/pipeline/70/. @nixpanic any idea why it's existing and snapshot CRD's are not getting installed?

Checking RBD (builtin-mgr) stats... 125s
Mon Jun  6 14:25:37 BST 2022: 'kubectl_retry -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath={.items[0].metadata.name}' done (ret=0)
Mon Jun  6 14:25:37 BST 2022: 'kubectl_retry -n rook-ceph get pod rook-ceph-tools-6944779c47-dg25d -ojsonpath={.status.phase}' done (ret=0)
Mon Jun  6 14:26:20 BST 2022: 'kubectl_retry -n rook-ceph get cephblockpools -ojsonpath={.items[0].metadata.name}' done (ret=0)
Checking RBD (builtin-mgr) stats... 130s
Mon Jun  6 14:26:25 BST 2022: 'kubectl_retry -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath={.items[0].metadata.name}' done (ret=0)
Mon Jun  6 14:26:25 BST 2022: 'kubectl_retry -n rook-ceph get pod rook-ceph-tools-6944779c47-dg25d -ojsonpath={.status.phase}' done (ret=0)
Sending interrupt signal to process

I see the above error in https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/upgrade-tests-rbd/detail/upgrade-tests-rbd/5829/pipeline/70/. @nixpanic any idea why it's existing and snapshot CRD's are not getting installed?

nevermind, it's due to Rook 1.9.x added more check in Rook.sh for the new .mgr pool

Jun  7 07:45:29.594: subvolumes [[{nfs-export-eb40d884-e62b-11ec-932d-2a8066a6beb8} {csi-vol-5fac3991-e62d-11ec-8ee0-7aaed4ab8f87}]]. subvolume count 2 not matching expected count 1

Looks like we got a bug? logs at https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/mini-e2e_k8s-1.23/detail/mini-e2e_k8s-1.23/1899/pipeline

@mergifyio rebase

/retest all

/retest all

CI status is stuck.

@mergifyio rebase

@mergifyio rebase

/retest ci/centos/mini-e2e/k8s-1.22

/retest ci/centos/mini-e2e/k8s-1.21

/retest ci/centos/mini-e2e-helm/k8s-1.23

/retest ci/centos/k8s-e2e-external-storage/1.21


Improvement: Weak events Implementation in BaseLayer

This adds Weak Events to the BaseLayer, this is Step into the direction of removing the IDisposable of the Map.
Is taken from this closed pull request. https://github.com/Mapsui/Mapsui/pull/2496

@inforithmics When I run the WPF samples the map does not load when selecting the sample half of the time. I could not figure out how my EditingWidget changes could have caused this, and it turned out it also showed in other samples. I reverted the weak DataHasChanged logic to the old logic and now it works like before. I will push this revert as part of my EditingWidgetChanges.
I did not investigate the problem. One thing that comes to mind: The weak references are faster and the event arrives too early, before the mapcontrol is initialized. If this is the case there is probably an existing bug underneath it that shows due to the change in performance.


Import Error: No module named argparse

Hello, i try to use "python rcon_shell.py --host <IP_ADDRESS> --port 25765" as example, but i get this error:
Traceback (most recent call last):
File "rcon_shell.py", line 2, in
from  argparse import ArgumentParser
ImportError: No module named argparse.
any idea how to solve this ? thx

Hi,
Which operating system and Python version are you using? The argparse module is built into Python 2.7 and 3.x -- it sounds like you may be on 2.6 or an earlier version.
You may be able to solve this by running sudo pip install argparse (or look into using virtualenv to manage your Python packages).

Hello, thanks for reply, well..
I am running windows 7 & have Python 2.6, i have installed pip and argparse, after i have used "python setup.py install" and after "python setup.py build", after, i tried to use "python test.py" but i get this error:
"rcon.py:150: DeprecationWarning: catching of string exceptions is deprecated, except(error) as ret_val: False"

Hi,
I am running Windows 7 and Python 2.6, i have installed argparse and the rcon works perfectly!
###thank you so much!


How to mention someone in slack channel when a test fail

Hi, thank you for your work!
Is there a way for me to mention a user in slack or use @here or @channel when a test fail?

hi @lychanminh91
You can either try using a custom meta in the config:

Or define your own customLayout and have a text block performing the mentions inside one of the blocks


Can I save population in every 1000 evaluations?

Currently, pymoo saves the population to res.history in every generation. However, when comparing different algorithms, the number of function evaluations in each generation are usually different.
When comparing the convergence property of different algorithms, I would like to compare the population in 1000 (or other pre-specified numbers)  evaluations. It there any way to do that? Thanks in advance!

The history saves the current algorithm object by making use of the deepcopy method of Python. The population object is part of it and contains the surviving individuals if it is a GA or maybe another representative set of solutions depending on the algorithm.
One way of doing the comparison is to plot the convergence curve with the x-axis representing the number of function evaluations. The number of function evaluations is stored in the algorithm.evaluator.n_eval. See the example below.
If you only like to compare the final result you can simply set ("n_evals", 1000) to the specified number and the algorithm will terminate close to this maximum (It might be slightly more. The algorithm will not start a new generation when the limit is exceeded.)
Convergence Curve (for single-objective):
import matplotlib.pyplot as plt
import numpy as np

from pymoo.algorithms.so_genetic_algorithm import GA
from pymoo.factory import get_problem
from pymoo.optimize import minimize

problem = get_problem("sphere", n_var=10)

res = minimize(problem,
               GA(pop_size=20, eliminate_duplicates=True),
               ("n_evals", 1000),
               seed=1,
               save_history=True,
               verbose=False)

n_evals = np.array([algorithm.evaluator.n_eval for algorithm in res.history])
F = np.array([algorithm.opt[0].F[0] for algorithm in res.history])
plt.plot(n_evals, F, '--', color="black", label="GA - pop_size=20")

res = minimize(problem,
               GA(pop_size=50, eliminate_duplicates=True),
               ("n_evals", 1000),
               seed=1,
               save_history=True,
               verbose=False)

n_evals = np.array([algorithm.evaluator.n_eval for algorithm in res.history])
F = np.array([algorithm.opt[0].F[0] for algorithm in res.history])
plt.plot(n_evals, F, '--', color="green", label="GA - pop_size=50")

plt.title("Convergence")
plt.legend()
plt.xlabel("n_evals")
plt.ylabel("F")
plt.yscale("log")
plt.show()

Thank you very much!


[Feat] Download Invoice

Description
It should for now download a dummy bill with the list of products. Please assign this issue to me.
Screenshots

Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I'm a GSSoC'24-Extd contributor
[X] I'm a Hacktoberfest'24 contributor
[X] I am willing to work on this issue (optional)

@inkerton That's amazing, but can you please tell me where exactly you plan to place this feature on the website and when users can download the invoice? I think users can download the invoice after the payment, right? If that's correct, we're currently working on the payment integration. After it's completed, I'll let you know to work on it. Otherwise, please describe when users can download the invoice and other details.


Have links to the info panel take you to the right tab

Changes links in chat to "manage notifications" and "manage retention" take you to the settings tab of the info panel. r? @keybase/react-hackers

I'm glad this was that easy to fix, thanks!


Proxy tests for expect call cairo

Depends on #1363

Some installation failed on CI which is weird considering that it succeeded on the target branch and I do not see a relation between the failed test and the changes here.

closing in favor of https://github.com/software-mansion/protostar/pull/1481


Improve assignement of kafka source pipelines on several indexers

A user reported the following issue (I checked the cluster state):

He has around 50 Kafka sources. The biggest Kafka source has 4 desired pipelines, most of which have one.
He has 10 indexers, each with 4-6 pipelines. 1 indexer has all the pipelines of the biggest Kafka source.

This is bad because, as you can imagine, only 1 indexer is responsible for indexing the biggest Kafka source. And all other indexers are doing less work.

Replied offline by @fmassot : no the problem is an unbalanced load on the different nodes.


bad gemspec?

# Gemfile
source :rubygems
gem 'groupon', :git => 'https://github.com/pengwynn/groupon'

$ bundle
Fetching https://github.com/pengwynn/groupon
remote: Counting objects: 121, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 121 (delta 43), reused 103 (delta 35)
Receiving objects: 100% (121/121), 50.29 KiB, done.
Resolving deltas: 100% (43/43), done.
There was a LoadError while evaluating groupon.gemspec:
  no such file to load -- lib/groupon from
  /Users/kimball/.rvm/gems/ruby-1.9.2-p180@groupon_test/bundler/gems/groupon-18f730b5ee5d/groupon.gemspec:3:in `<main>'

Does it try to require a relative path? That doesn't work in Ruby 1.9.

Thanks for logging. We're working this week to overhaul the gem, bringing it up to date on several fronts. We hope to have the next version with this fix done in the next couple of days.


Activity Selection Problem and Left and right view of Binary tree

1.Activity Selection Problem
2.Left and right view of Binary tree
@mission-peace plz review and merge
i had removed Balanced Parentheses

Dont create a new BNode. Use existing one.