Unable to build onnxruntime-genai for iOS

Please provide us with the following information:

This issue is for a: (mark with an x)
- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce
Clone the onnxruntime-genai repository.
Ensure that ONNX Runtime is built for iOS
Run the build script for onnxruntime-genai with the following command:
python3 build.py --parallel --build_dir ./build_ios_simulator --ios --ios_sysroot iphoneos --ios_arch arm64 --ios_deployment_target 17.4 --cmake_generator Xcode

Any log messages given by the failure
CMake Error at cmake/global_variables.cmake:54 (message):
 Expected the ONNX Runtime library to be found at
 /Users/xxxx/Documents/xx/phi3/onnxruntime/onnxruntime-genai/ort/lib/libonnxruntime.dylib.
 Actual: Not found.
Call Stack (most recent call first):
 CMakeLists.txt:18 (include)

Expected/desired behavior
The build script should locate the ONNX Runtime library at the specified path and proceed with the build process without errors.
OS and Version?
macOS Sonoma 14.2.1 (23C71)
Versions
CMake: 3.28.0
Python: 3.10 (conda)
ONNX Runtime: Built from the latest source

can you confirm you have copy the dylib file in spec file from the docs ?


add dqi codes to counts for reviewed/not

Unreviewed DQI codes should be S, I, and P.
Reviewed codes should be R, Q, O, and X.
This leaves only A and U uncounted.

Done.


CloudWatch Rules not disabled, when turning off rules in Admin -> System Management

Micro services

Spring boot Applications:

Admin Service

Rules
aws_snapshot_should_not_be_publicly_accessible
aws_EC2_should_not_be_publicly_accessible_with_any_port
Webapp

OS Type: Linux
Java version:

1.8

Summary

When turning off CloudWatch rules, two of the rules stay enabled.
Reproduce steps

Under Admin -> System Management -> Rules, turn the setting to off.
Expected Results

All of the PacBot "aws_*" rules should be disabled.
Actual Results

The following rules stay enabled.
aws_snapshot_should_not_be_publicly_accessible
aws_EC2_should_not_be_publicly_accessible_with_any_port
Aha! Link: https://t-mobile1t-mobile.aha.io/features/PM-408

@grumbis  did you installed latest version of PacBot, we were unable to reproduce issue.


Issue #24 - insert DTO type ENUM as string

Issue #24
When serializing a DTO that contains a DTO_FIELD whose type is Enum<xxx>AsString, the resulting string ultimately goes out of scope. This will result in a failure when trying to insert this type of value into a database.
The solution is to utilize Serializer::OutputData::databuffer. Before the enum string goes out of scope, allocate a dataBuffer and copy the value to it.
EnumAsStringTest is included as a unit test.
If you run the unit test before applying the Serializer change you can see the failure take place.
Regarding Oatpp 1.4.0
My environment is set up using Oatpp 1.4.0.
I updated oatpp-postresql to 1.4.0 to explore.
I then discovered issue #24 and created the solution in the 1.4.0 environment.

Hey @bwengert79 ,
It's a great PR!
Let's try to run CI:
Please change docker-compose to docker compose without '-' in two places
https://github.com/oatpp/oatpp-postgresql/blob/master/azure-pipelines.yml#L16
Let's see if it builds

Got past the docker compose issue.
The build is now failing because it looks like a cmake version issue.
"CMake 3.20 or higher is required.  You are running version 3.11.1"
I changed cmake_minimum_required to 3.20 in CMakeList.txt to match what's in the oatpp repo.
Is oatpp-postgresql building in a different environment that oatpp?

Changing cmake min to 3.1 got us further in the build process. However, the next stumbling block is oatpp 140 is not available in the current CI environment. Perhaps this PR needs to wait for the CI environment to match that of oatpp 140.

Hey @bwengert79 ,
Yes, the current implementation of CI pipeline is outdated and isn't flexible.
I'll merge this PR as is, and later will rebuild the CI pipeline.


It's not allowed to transition image without first acquiring them

https://github.com/GPUOpen-LibrariesAndSDKs/HelloVulkan/blob/master/hellovulkan/src/VulkanSample.cpp#L730-L750
Here you setup a resource transition for each image in the swapchain which you run during initialize without acquiring them. This is not allowed.
https://www.khronos.org/registry/vulkan/specs/1.0-wsi_extensions/xhtml/vkspec.html#_wsi_swapchain

Use of a presentable image must occur only after the image is returned by vkAcquireNextImageKHR, and before it is presented by vkQueuePresentKHR. This includes transitioning the image layout and rendering commands.

Instead you can set the initialLayout of the renderpass attachment to VK_IMAGE_LAYOUT_UNDEFINED. Implying that you don't care about the existing data (which is made explicit by the op_load_clear).

Good catch! Just pushed a fix, thanks for reporting!


made event system file more readable

The event system I wrote some months ago is not really readable. In case someone wants to maintain it in the future, I made it more readable with shorter functions and helper functions.


discovery: Fix some cases of gunicorn detection

What does this PR do?
gunicorn doesn't always replace the command line with status info and when it's start with its wrapper, the command line starts with the binary path.  So add a pathway from the Python detector to the Gunicorn-specific detector to handle this case.
Motivation
https://datadoghq.atlassian.net/browse/USMON-1355
Describe how to test/QA your changes
Automated tests have been updated.
Possible Drawbacks / Trade-offs
Additional Notes

I'd add more inline comment in the code
it might be obvious for the writer, but I can see why the average reader won't understand the relation between gunicorn to python, or why do we need a special case

I'd add more inline comments in the code it might be obvious for the writer, but I can see why the average reader won't understand the relation between gunicorn to python, or why do we need a special case

I've added a comment for this.

/merge


Add prettier CI step

As discussed here - https://github.com/huggingface/transformers.js/pull/1043. Separated out the CI check for prettier format into a separate PR.

Thanks! I'll aim to get this merged for the upcoming minor release (Transformers.js v3.1; primarily waiting for #1001 to be finished and to merge other accepted PRs)


Expose more of the underlying apache consumer methods

Currently the KafkaConsumer exposes only a tiny subset of all the apache kafka consumer methods.
Wouldn't it be easier to provide an (albeit dangerous) withConsumer (similar to the one on KafkaConsumerActor) instead of actor style communication? Speaking of which, what's the purpose of the actor since you anyway synchronize calls to the underlying apache kafka consumer?

Thanks for raising this issue @backuitist!

Currently the KafkaConsumer exposes only a tiny subset of all the apache kafka consumer methods.

That's correct, and it's mainly because no-one has asked for or implemented the remaining functions. It would be helpful if you could either (1) list the non-implemented functions you need, or (2) list all the non-implemented functions for reference, and we could track them here.

Wouldn't it be easier to provide an (albeit dangerous) withConsumer (similar to the one on KafkaConsumerActor) instead of actor style communication?

One of the design decisions is to not expose impure functions, which you would do indirectly by providing a withConsumer function. My preference is that we expose pure versions of those functions directly on KafkaConsumer instead.

Speaking of which, what's the purpose of the actor since you anyway synchronize calls to the underlying apache kafka consumer?

The initial implementation started out with the 'actor' approach, but then the synchronized construct was added to deal with finalizer behaviour (calling close on the Java Kafka consumer). You're right in that we can get rid of the actor approach, relying solely on Synchronized, but it simply has not been done yet.

Thanks for the quick and thorough answer!
Right now my use-case is to read the last x events of a topic. Here are the methods of interest:

Set<TopicPartition> assignment();
void seekToEnd(Collection<TopicPartition> partitions); -- that one is easily implemented with endOffsets + seek

Potentially useful:

long position(TopicPartition partition);

I've opened #93 to add those functions specifically, and #94 to track remaining functions.


Convenient effects doesn't always play nice with Multi Token Status

Description:
A module conflict with Multi Token Status (https://github.com/cs96and/FoundryVTT-multistatus).
When Multi Token Status is enabled, activating the Invisible status through token controls applies the invisible dynamic effect but not the visual effect on the token. Tokens given the invisible status via token controls can still be seen by all players.
Tokens made invisible by the convenient effects window are affected as expected.
To Reproduce:
Steps to reproduce the behavior:

Activate both Convenient Effects and Multi Token Status
Right click a token, click on statuses, activate the Invisible effect
The icon for invisible appears on the token, the active effects apply to the character sheet, but the token remains visible to all players.

Expected Behavior:
Upon assigning the status from the token's status menu, the token should fade and become invisible to other players.
Screenshots:

With Multi Token Status enabled

With Multi Token Status Disabled

The workaround
Additional Context:
The console window doesn't output any error codes. It seems like it's a visual bug only, but it does affect game for the invisibility use case.
I'm not sure what other data could be useful here, but let me know and I'll provide what I can. Thanks for your time!

It's likely because we both override TokenHUD.prototype._onToggleEffect. If you open up the libWrapper settings menu, you should be able to prioritize one over the other or see what the conflicts are. Ultimately, I'm not sure there is much I can do here as I have to override this function.

I'll wait for your findings before closing this

With Convenient effects prioritized, we lose multi token status' functionality and get this warning.

With Multi Token Status prioritized, we lose the visual effects of invisibility and get this warning.

Functionally, it's not quite the same as disabling the mods. If it's unfixable, that's not a huge loss. The big workaround is just to select statuses from the convenient effects window instead of the token menu, which works for me.
Even if it can't be fixed, I'd like to thank you for looking into it. I appreciate the time!

So, Foundry STILL hasn't implemented a way for two modules to manipulate the status effects despite some feature requests. It isn't a supported API, so every module that does is patching it to do something, and they'll override each other. This can be revisited if and when Foundry adds this, or if a library is created which could somehow do this


Is there supposed to be a map?

Description
I could be mistaken, but I'd really expect some kind of map view or other data visuals.
What browser are you seeing the problem on? What version are you running?
Mozilla Firefox 132.0
The OS you're using
GNU+Linux (Gentoo)
Steps to reproduce

Visit https://explore.overturemaps.org/
Wait a bit
Stare at blank screen
?????
File bug report

GERS ID of the entity in question (if applicable)
No response
The browser URL at the time you encountered the bug
https://explore.overturemaps.org/

Yes, there very much is supposed to be a map! Yeah this isn't good- no idea what the problem could be though.  Are you able to see the site if you visit from a phone or other device?
Do you literally just see a blank screen, or do you see any text or map controls at all?  A screenshot or developer tools console log might be helpful.


"salt" option property does not work

I have two guesses: it is a bug, or it was not marked as deprecated in the types. Here is my testing:
This is one I can't explain too well.
    const salt = generateRandomString(
      {
        read(bytes) {
          crypto.getRandomValues(bytes);
        },
      },
      alphabet,
      16,
    );

This creates a salt using generateRandomString from @oslojs/crypto/random.
It then uses hash from @node-rs/argon2:
    const passwordHash = await hash(password, {
      memoryCost: 19456,
      timeCost: 2,
      outputLen: 32,
      parallelism: 1,
      salt: Buffer.from(salt),
    });

password: testtest
salt: ojq_1kIXDG5zpGx0
salt buffer: <Buffer 6f 6a 71 5f 31 6b 49 58 44 47 35 7a 70 47 78 30>

hash: $argon2id$v=19$m=19456,t=2,p=1$ZgWgG5c6yUxJWjp3QSRMmw$JJmgej/ScGPbq+RozS2zKyWZLMZixL3MV1JVsu03WI4

According to the types, salt is a correct property of Options, and is NOT marked as deprecated.
I then use the verify function from the same package, giving it the same password, and the hash created earlier, BUT i modofy the salt being passed.
    const validPassword = await verify(existingUser.passwordHash, password, {
      memoryCost: 19456,
      timeCost: 2,
      outputLen: 32,
      parallelism: 1,
      salt: Buffer.from(existingUser.salt),
    });

hash: $argon2id$v=19$m=19456,t=2,p=1$ZgWgG5c6yUxJWjp3QSRMmw$JJmgej/ScGPbq+RozS2zKyWZLMZixL3MV1JVsu03WI4
password: testtest
salt: "This has been set and should not match anything"
salt buffer: <Buffer 54 68 69 73 20 68 61 73 20 62 65 65 6e 20 73 65 74 20 61 6e 64 20 73 68 6f 75 6c 64 20 6e 6f 74 20 6d 61 74 63 68 20 61 6e 79 74 68 69 6e 67>

validPassword = true

This shows that the verify function that verifies the password against the hashed password, with a completely different salt input still passes.
Back to the fact I said it was not marked as deprecated, nowhere in the documentation nor the types does it say this. It also returns no error. However, using the secret prop instead of the salt prop now properly fails the verify function when the fake salt is given.

@Jazza-231 what version of the package are you using?
This should have been addressed by #899 raised by #893 and fixed in v2+

I couldn't tell you I'm sorry, I have no version history on the project that used it, and that has been since deleted as it was just a test.

In that case I'd assume this issue is due to using the old version which was confirmed to mistakenly ignore the provided salt and has since been addressed. And as per semantic versioning required a new major version release.
I can confirm that this does in fact now work as expected

Just ran a quick test, it seems the original issue of the hash function ignoring salt was fixed, BUT, the verify function still does not respect the provided salt. So it seems you are correct.
A quick workaround would be to use hash again instead of verify using the same argon params and comparing them manually - but that shouldn't ever be required or used as a long term solution, and totally negates the point of having a verify function in the first place.
@Brooooooklyn could you take another look please?

Actually, my last post was incorrect. I just remembered the salt is included in the encoded password hash
$<id>[$v=<version>][$<param>=<value>(,<param>=<value>)*][$<salt>[$<hash>]]

So it doesn't need to be supplied to the verify function. And as per the Rust implementation's docs
// Verify password against PHC string.
//
// NOTE: hash params from `parsed_hash` are used instead of what is configured in the
// `Argon2` instance.
let parsed_hash = PasswordHash::new(&password_hash)?;
assert!(Argon2::default().verify_password(password, &parsed_hash).is_ok());

So this is actually working as expected.. similar to any other password hasher that is encoded to the PHC format, which includes the salt in the output string. It simplifies having to store all parameters of the underlying hashing function manually and enables you to change your argon2 options at any time, without breaking previously generated passwords.
So the real solution to @Jazza-231's intended usage of the salt, would be to use it as the 'secret' param instead. But it would still be possible to use your own generated salt AND secret as options when generating password hashes, and then supplying the secret when verifying.
@Brooooooklyn perhaps it would be a good idea to change the TS types of the options parameter for the verify/verifySync methods to Pick<Options, 'secret'>. And reflect that change in the README. Hopefully that should prevent any confusion.


payloadsallthethings: init at 3.0-unstable-2024-01-21

Description of changes
PayloadsAllTheThings - A list of useful payloads and bypass for Web Application Security and Pentest/CTF.
See:

https://github.com/swisskyrepo/PayloadsAllTheThings
https://repology.org/project/payloadsallthethings/information

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 283621 run on x86_64-linux 1

1 package built:

payloadsallthethings


Add separate status for each verify job

Signed-off-by: Eli Uriegas<EMAIL_ADDRESS>

Meh a failed experiment


AttributeError: 'WebDriver' object has no attribute 'find_elements_by_class'

After running this snippet,
from bing_images import bing

bing.download_images("cat",
                      2,
                      output_dir="/path/to/imgs",
                      pool_size=10,
                      file_type="png",
                      force_replace=True,
                      extra_query_params='&first=1')

I get the following error:
Save path: /path/to/imgs
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [2], in <cell line: 3>()
      1 from bing_images import bing
----> 3 bing.download_images("cat",
      4                       2,
      5                       output_dir="/path/to/imgs",
      6                       pool_size=10,
      7                       file_type="png",
      8                       force_replace=True,
      9                       extra_query_params='&first=1')

File ~/anaconda3/envs/real_meal/lib/python3.10/site-packages/bing_images/bing.py:60, in download_images(query, limit, output_dir, pool_size, file_type, filters, force_replace, extra_query_params)
     58 # Fetch more image URLs to avoid some images are invalid.
     59 max_number = math.ceil(limit*1.5)
---> 60 urls = fetch_image_urls(query, max_number, file_type, filters, extra_query_params=extra_query_params)
     61 entries = get_image_entries(urls, image_dir)
     63 print("Downloading images")

File ~/anaconda3/envs/real_meal/lib/python3.10/site-packages/bing_images/bing.py:29, in fetch_image_urls(query, limit, file_type, filters, extra_query_params)
     27 if len(file_type) > 0:
     28     keywords = query + " " + file_type
---> 29 urls = crawl_image_urls(keywords, filters, limit, extra_query_params=extra_query_params)
     30 for url in urls:
     31     if isValidURL(url, file_type) and url not in result:

File ~/anaconda3/envs/real_meal/lib/python3.10/site-packages/bing_images/crawler.py:59, in crawl_image_urls(keywords, filters, max_number, proxy, proxy_type, extra_query_params)
     57 driver.set_window_size(1920, 1080)
     58 driver.get(query_url)
---> 59 image_urls = image_url_from_webpage(driver, max_number)
     60 driver.close()
     62 if max_number > len(image_urls):

File ~/anaconda3/envs/real_meal/lib/python3.10/site-packages/bing_images/crawler.py:26, in image_url_from_webpage(driver, max_number)
     23 img_count = 0
     25 while True:
---> 26     image_elements = driver.find_elements_by_class("iusc")
     27     if len(image_elements) > max_number:
     28         break

AttributeError: 'WebDriver' object has no attribute 'find_elements_by_class'

I tried changing find_elements_by_class("iusc") to find_elements("class", "iusc") on crawler.py as the former is deprecated, but it did not work, it resulted in new issues.
I'm using version 103.0.5060.114 of Chrome and version ChromeDriver 103.0.5060.53 of Chrome Driver. I tried with other versions unsuccesfully.
Thank you in advance.

Hello.
Looking quickly to the implementation you should be able to do this:
from selenium.webdriver.common.by import By

# change line 27 to
image_elements = driver.find_elements(By.CLASS_NAME, "iusc")

# change line
smb = driver.find_elements(By.CLASS_NAME, "btn_seemore")

Good luck.
I can create a PR if needed.

Latest update to this error Change the previous to:
`
#line 23
image_elements = driver.find_elements("name","iusc")
#line 35
smb = driver.find_elements("name", "btn_seemore")
`


tabula: pin jre version

The packaged (latest) release of tabula is quite old, and bundles a very old version of jruby which is
incapable of running on modern JREs.
See https://github.com/tabulapdf/tabula/blob/v1.2.1/.ruby-version#L1 for packaged version, and https://github.com/tabulapdf/tabula/blob/master/.ruby-version#L1 for current (but unpackaged) version.
The jruby bug which prevents use on modern JREs was apparently fixed in version <IP_ADDRESS>, but this
is newer than the packaged version (<IP_ADDRESS>), so pinning this package to jre8 suffices to get it
working again.
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

I am still getting an error in the Web UI:

(SystemCallError) Unknown error (SystemCallError) - Unknown Error (0) - ./enumerator.jar

I am still getting an error in the Web UI:

(SystemCallError) Unknown error (SystemCallError) - Unknown Error (0) - ./enumerator.jar

Did this work for you before this PR?
What platform are you on, also?
It's possibly that there is more than one runtime problem on some platforms, and this change only fixes one of them.

This is the first time I have tried to use Tabula. I am on NixOS x86_64.
On Sat, 25 Mar 2023, 20:19 Dave Nicponski, @.***> wrote:

I am still getting an error in the Web UI:
(SystemCallError) Unknown error (SystemCallError) - Unknown Error (0) -
./enumerator.jar
Did this work for you before this PR?
What platform are you on, also?
It's possibly that there is more than one runtime problem on some
platforms, and this change only fixes one of them.
—
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/222581#issuecomment-1483901755,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFMEY3S55OB23LFDSTNBO3W55AMFANCNFSM6AAAAAAWEF5IEU
.
You are receiving this because you commented.Message ID:
@.***>

Anything else needed for this to be merged?

Anything else needed for this to be merged?

Can we get this merged please?  There may be other problems with the package, but this PR solves at least one of them.
nix-repl> :l <nixpkgs>
Added 16551 variables.

nix-repl> :u tabula.override { jre = jre8; }
this derivation will be built:
  /nix/store/bqpw8wz24vr2lq7a310d07xry6vzvk9m-tabula-1.2.1.drv
this path will be fetched (83.49 MiB download, 203.32 MiB unpacked):
  /nix/store/5dvwcbqv4ih6px62kj8npv9rlf1a9gjw-zulu<IP_ADDRESS>-ca-jdk-8.0.292
copying path '/nix/store/5dvwcbqv4ih6px62kj8npv9rlf1a9gjw-zulu<IP_ADDRESS>-ca-jdk-8.0.292' from 'https://cache.nixos.org'...
building '/nix/store/bqpw8wz24vr2lq7a310d07xry6vzvk9m-tabula-1.2.1.drv'...
unpacking sources
unpacking source archive /nix/store/c6wds4a1vqbl7m5sw1pwqiv6ac0lsk93-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
mkdir: created directory '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1'
mkdir: created directory '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share'
mkdir: created directory '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share/tabula'
'AUTHORS.txt' -> '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share/tabula/AUTHORS.txt'
'LICENSE.txt' -> '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share/tabula/LICENSE.txt'
'README.txt' -> '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share/tabula/README.txt'
'tabula.jar' -> '/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share/tabula/tabula.jar'
post-installation fixup
strip is /nix/store/lx0plin5vr5p21iv9pkhhq1ify6gxgl3-cctools-binutils-darwin-949.0.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/bin
patching script interpreter paths in /nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1

dave @ laptop [nix-shell:~] $ tabula
2023-09-05 08:58:01.714:INFO::main: Logging initialized @295ms
2023-09-05 08:58:01.720:INFO:oejr.Runner:main: Runner
2023-09-05 08:58:01.841:INFO:oejs.Server:main: jetty-9.2.9.v20150224
2023-09-05 08:58:05.985:WARN:oeja.AnnotationConfiguration:main: ServletContainerInitializers: detected. Class hierarchy: empty
INFO: jruby <IP_ADDRESS> (2.3.3) 2017-12-07 929fde8 OpenJDK 64-Bit Server VM 25.292-b10 on 1.8.0_292-b10 +jit [darwin-x86_64]
INFO: using a shared (threadsafe!) runtime
DATA_DIR = /Users/dave/Library/Application Support/Tabula
DOCUMENTS_BASEPATH = /Users/dave/Library/Application Support/Tabula/pdfs
ENABLE_DEBUG_METHODS = false
running under / as root URI
============================================================
http://<IP_ADDRESS>:8080
============================================================
should we open browser?
java.lang.Boolean.getBoolean('tabula.openBrowser'): false

======================================================
Server now listening at: http://<IP_ADDRESS>:8080

You may now open a web browser to the above URL.
When you're done using the Tabula interface, you may
return to this window and press "Control-C" to close it.
======================================================

2023-09-05 08:58:11.840:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@2f410acf{/,file:/private/var/folders/vv/k0fb7yx10zs5whxtk78k_dmh0000gn/T/jetty-<IP_ADDRESS>-8080-tabula.jar-_-any-1717563567806910962.dir/webapp/,AVAILABLE}{file:/nix/store/s32lxka1vava7y12m1vljdzppar444hx-tabula-1.2.1/share/tabula/tabula.jar}
2023-09-05 08:58:11.841:WARN:oejsh.RequestLogHandler:main: !RequestLog
2023-09-05 08:58:11.866:INFO:oejs.ServerConnector:main: Started ServerConnector@6dcf7b6a{HTTP/1.1}{<IP_ADDRESS>:8080}
2023-09-05 08:58:11.867:INFO:oejs.Server:main: Started @10450ms
["_dontcare"]
Sep 05, 2023 8:58:58 AM org.apache.pdfbox.pdmodel.font.FileSystemFontProvider loadDiskCache
WARNING: New fonts found, font cache will be re-built
Sep 05, 2023 8:58:58 AM org.apache.pdfbox.pdmodel.font.FileSystemFontProvider <init>
WARNING: Building on-disk font cache, this may take a while
Sep 05, 2023 8:59:00 AM org.apache.pdfbox.pdmodel.font.FileSystemFontProvider <init>
WARNING: Finished building on-disk font cache, found 814 fonts
^C

dave @ laptop [nix-shell:~] $

...and then proceeded to successfully use the tool to extract tabular data from a PDF.

Updated PR description to note that this mitigates the problem described here: https://github.com/NixOS/nixpkgs/commit/2b14eec435ad7b4e7a3ca571458b176ef0367039

@worldofpeace any chance you can merge this?  I really need to apply for merge access; been putting that off too long.


[Bug]: The link 'potential issues' in the MOTIVATION.md is broken

Telegram Application
Other
Describe the Bug
It seems like the 'potential issues' link here leads to a section that no longer exists:
https://github.com/Telegram-Mini-Apps/tma.js/blob/master/MOTIVATION.md#unproductive-package-format
To Reproduce
Steps to reproduce the behavior:

Go to https://github.com/Telegram-Mini-Apps/tma.js/blob/master/MOTIVATION.md#unproductive-package-format
Click on 'potential issues'
Nothing happens

Expected Behavior
It should lead somewhere or it shouldn't be a link

Fixed. Thank you!


[libusb] Fix the support for Visual Studio 2019

Describe the pull request

What does your PR fix? Fixes #16832

Current version has supported Visual Studio 2019, so we should also support it.
Note: No feature needs to test.

LGTM


[NVIDIA TF] Add DTensorAllToAll and relayout implementation

This PR adds a new high-level op, DTensorAllToAll along with lowering to  CollectiveAllToAllV2.
To start, I've implemented a relayout using this new all-to-all for certain cases only. There should be many more applications for all-to-all that we can implement in the future.
Supported relayout cases
I've added support for the cases where one axis is becoming sharded to a certain mesh dimension while another axis is becoming unsharded from that mesh dimension. For example, [x,unsharded] -> [unsharded, x] or [unsharded, y, x] -> [x, y, unsharded]. You can look at the unit tests in spmd_test.py for more (non-exhaustive) examples.
Before this PR, these relayouts would be implemented using all-gather to fully replicate the tensor, followed by slice. With all-to-all, the communication overhead is reduced since only the minimal slices of data that need to be moved are exchanged across the devices.
Summary of changes

DTensorAllToAll op
DTensorAllToAllLowering pass which can lower to CollectiveAllToAllV2 for cpu and gpu, or XLA AllToAll for TPU. I did not test the TPU path at all.
Use DTensorAllToAll for certain relayouts
MLIR test
Python unit tests

@bfontain @rainwoodman @pjannaty

FYI: @RuijiaoSun

Thanks for the PR. The changes looked good to me.
I'll pull it internally to give the XLA part of the logic a try on TPUs.

Good news! The tests for the XLA code path passed on our V3 TPU machines.


When will middleman 4.3.0.rc.2 drop on RubyGems?

Hello middleman mainainters,
First, @tdreyno, thank you very much for merging in #2190. I also noticed that you bumped the version to 4.3.0.rc.2 in 64a65a73. What is the process, and when can we expect this to be released to RubyGems?
Thanks!

Done, sorry for the delay.

Changelogs are still out of date though. Hard to figure out what's new with version 4.3.3

https://github.com/middleman/middleman/blob/4.x/CHANGELOG.md


Event onChange not fired after http address is converted to link

Summernote v. 8.20, issue only when I type link manually or paste it as blank text.
When I type in Summernote for example https://summernote.org and hit space, the link in contentEditable is converted to proper a href (I can see proper tags) and styles are applied.
But the onChange callback is not fired with the change, so I need to type another space, add some text or hit enter for editor to emit this change.
So for example onChange event returns:

I type: https://summernote.org - it is still just a text
I hit a space button - onChange returns https://summernote.org&nbsp but I already see that in contentEditable it's converted to link
I hit another space button and now onChange returns https://summernote.org&nbsp&nbsp

So if someone doesn't know it and link is the last element on page, it looks link a proper link but saves as blank text. I think onChange should be called as soon as conversion has place.

I've moved this to main summernote page as it's on jquery.


[bug]: LND can't connect to peers/channels, sync to chain is lost

Background
On RoboSats mainnet node, on start up, LND gets to both synced to chain and graph. Connects to a few peers/ channels. Eventually, stops during its way to connect to all peers and channels. It might connect to only 2-30 peers/channels of 140 total channels.  Eventually, the number of active channels decays down to 0. Effectively making the node dead. After some time,
synced_to_chain goes false.
This the systematic behaviour everytime LND has been restarted for the last 12 hour.
As RoboSats uses mainly hodl invoices, in a few hours many force closures will be triggered and recovery will be painful. Hopefully we can get the node back online first :D
The only interesting thing I see in the logs is a bunch of this (when LND loses chain sync?). But I can grep it if you think I am missing something.
2023-08-24 05:15:28.633 [ERR] BTCN: Can't accept connection: unable to accept connection from <IP_ADDRESS>:45314: write tcp <IP_ADDRESS>:9735-><IP_ADDRESS>:45314: write: broken pipe                                        2023-08-24 05:15:28.959 [ERR] BTCN: Can't accept connection: unable to accept connection from <IP_ADDRESS>:45324: write tcp <IP_ADDRESS>:9735-><IP_ADDRESS>:45324: write: broken pipe                                        2023-08-24 05:15:29.169 [ERR] BTCN: Can't accept connection: unable to accept connection from <IP_ADDRESS>:45336: write tcp <IP_ADDRESS>:9735-><IP_ADDRESS>:45336: write: broken pipe

There is one of these every few miliseconds.
I did already fully reboot the machine to no avail. Also updated docker / docker-compose to latest stable.
Your environment

version of lnd: v0.16.4 Official Docker image
which operating system (uname -a on *Nix)
Dockerized on Linux 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
version of btcd, bitcoind, or other backend
Bitcoin Core 24.0.1
any other relevant environment details

Steps to reproduce
I will probably be unable to reproduce anything that might have happen to arrive to this state.
Expected behaviour
LND should start and connect to all peers/channels.

Some more context, maybe useful for debugging. RoboSats node is Tor-only. The Tor daemon runs as another container in the orchestration. Possibly these are the <IP_ADDRESS>:***** broken pipes. However, all containers can talk to each other successfully, even LND container does: after every LND container kill and restart it does connect to some peers and get's synced to chain and graph. After some minutes, the broken pipes start showing.
There is no RAM or disk limitations.
In the same machine, with identical orchestration, there is a tesnet full replicate. It is working as usual.

Hi, thanks for reporting the issue. Did this behavior occur the first time after you adjusted settings or did it start without intervention? Would you be able to provide lnd logs that contain the start up as well as showing the decay of active peers? Can also mail the logs if you don't want to attach them here.

. Did this behavior occur the first time after you adjusted settings or did it start without intervention?

No intervention. No one logged into the machine for a day prior to the issue, no setting was changed.

Would you be able to provide lnd logs that contain the start up as well as showing the decay of active peers? Can also mail the logs if you don't want to attach them here.

Collecting the logs. Would be great if I can send them by email, where can I send them to?

hieblmi at lightning dot engineering

Sent logs of the latest restart. It connected to +12 channels. Currently only 7 show as active. "synced_to_chain" is false now.

Is the wallet ever unlocked?

Is the wallet ever unlocked, can't see that from the logs.

Yes it is. Is the logs file ~1.68Mb ?

Sent new logs with Debug verbosity. This time it seems something did fail before any peer was connected.
Will rerun and let it collecting logs a little bit longer.

Have you tried restarting Tor, maybe you can double check if the socks proxy is properly working?

Can other services connect to the Tor daemon? If you restart the Tor daemon is the issue resolved? Looks like it isn't able to maintain SOCKs proxy connections.

Hey thank you @Roasbeef

Can other services connect to the Tor daemon? If you restart the Tor daemon is the issue resolved? Looks like it isn't able to maintain SOCKs proxy connections.

Every other service can keep connection to the Tor Daemon. Restarting the Tor Daemon, or the whole machine, does not help. But restarting only the LND container will grant at least those initial 5-10 minutes without connection issues where it gets to sync the graph and establish connection to a few peers.

If you check the logs of the tor daemon, do you see any relevant error logging?

It's hard to say as I am not accostumed to look at Tor Daemon logs. There is certainly a huge amount of logs, some of which look like timeouts.

If you used a normal CLI command to establish a TCP connection using the SOCKS proxy, does that work?

From inside the LND container I run curl --socks5-hostname <IP_ADDRESS>:9050 https://example.com, it prints example.com content correctly. It does so even when LND is already uncapable of connecting to peers.
I also tried deleting ./lnd/v3_onion_private_key. At next start up it created a new one. After some minutes, it failed to connect to all peers. So deleting v3_onion_private key did not help.

What does your Tor related config for lnd look like?

write: broken pipe

In the Go runtime, this means that the other side has torn down or reset the TCP connection, but then a program/goroutine tries to write to it.

Good news and bad news. Good news the node managed connecting to all channels. Bad news, it was not thanks to anything I tried, it simply happened overnight and I do not know why it works now. I have restarted the node 3 times, every single time it connects to all peers.
However, I notice that after ~37 peers, it pauses. There are many broken pipes after 1-2 minutes, it keeps connecting to all peers successfully. I can send logs of these runs if needed.
A wild guess, does LND connect to all peers at once asynchronously in no specific order? or is there a deterministic order or batches (e.g. 10 peers at a time) and if the full batch is unreachable it keeps retrying?

What does your Tor related config for lnd look like?

This is all there is in lnd.conf relating Tor.
tor.active=1
tor.v3=1

Also are you able to manually use lncli connect to create outbound connections, either to other tor nodes or clearnet nodes?

Unfortunately I cannot test now given that the node is back and running as usual.  Running lncli connect does work now, but I do not know if it had worked yesterday.

Has this been going on for some time? Or did some recent update or config change trigger the behavior?

This is the first time I observe this behavior. No recent change of settings or updates, in fact, I happened to be in holidays when it started to fail :`)
What I observe now is a few outgoing in_flight payments that were stuck when the issue started, that are still in_flight, yet, the first hop channels on which these are flying show to have 0 total HTLCs (outgoing / incoming) in node managers like Thunderhub. I must say had observe this particular thing happening 2 times in the last 8 months: payments that get stuck in_flight forever, even weeks after the first hop channel was collaboratively closed. This is probably better suited for another issue, however there is quite a few of the same stuck payments at the moment.

It happened again and I have some more information on this case.

lncli connect <pubkey@host:port> fails with rpc error: code unknown desc = write tcp <IP_ADDRESS>:.....-><IP_ADDRESS>:9050: write:broken pipe
lncli addinvoice never returns.
It sometimes stays synced to chain/graph (or at least, it indicates so), but sometimes one of them goes false.
When this issue happened again from a healthy state, the number of active channels stayed at around 120, but the number of peers dropped to 77. The node looked as offline in LN explorers like amboss.space

There is some extra context that might be of interest
The LND node of RoboSats has previously had other issues that I originally thought were unrelated to this one. For example, it sometimes cannot create new invoices (gRPC addinvoice or lncli addinvoice would never return). The first time this happened was time ~6-8 months ago. Yet, a simple restart of the LND container fixed it instantly. This issue started happening a bit more often lately, maybe every 2-5 days. Sometimes even a few times in a day, but then it might go 2-3 weeks without any occurrence. I did not observe an obvious relationship between the number of pending HTLCs or invoices and the likelihood of LND getting stuck (but there might be a relation!?). I did not report these earlier as I never saw something out of the ordinary in the logs, it simply stops replying to some RPC calls (e.g. addinvoice, newaddress, settle invoice), it does not stop replying to lncli getinfo, lncli lookupinvoice or listinvoices.
Again, the fact that simply restarting the LND container seems to temporarily fix the broken pipes and the unresponsive gRPC calls, leads me to believe there is something that LND is struggling with.
LND might typically take 4GB of RAM. The machine is not disk, RAM or CPU limited.

I can confirm that even if it connects to all peers/channels, lncli addinvoice can still never return. So these 2 issues can happen independently and together.

A wild guess, does LND connect to all peers at once asynchronously in no specific order? or is there a deterministic order or batches (e.g. 10 peers at a time) and if the full batch is unreachable it keeps retrying?

The order is random-ish, likely derived from node pubkey ordering.

However, I notice that after ~37 peers, it pauses. There are many broken pipes after 1-2 minutes, it keeps connecting to all peers successfully. I can send logs of these runs if needed.

Sounds like it maybe runs into some tor fd-like limit? I have a tor node myself with ~50 channels/peers, but haven't been able to repro anything like this. I wonder if it's some tor thing where it starts to shed connections that are lower in the priority re QoS ranking.

lncli addinvoice can still never return. So these 2 issues can happen independently or jointly.

If this happens again, are you able to grab a profile? What we want here is a snapshot of all the active goroutines, then we get all the stack traces and can see what is actually blocking.

LND might typically take 4GB of RAM. The machine is not disk, RAM or CPU limited.

That sounds rather high...particularly for the 30-ish peer range. Along side the goroutine trace, can you already get a memory profile? This won't reveal any sensitive information about the node (just what is allocating all that memory).

Sounds like it maybe runs into some tor fd-like limit? I have a tor node myself with ~50 channels/peers, but haven't been able to repro anything like this. I wonder if it's some tor thing where it starts to shed connections that are lower in the priority re QoS ranking.

The node has been running successfully for over 1 year with +100 channels/peers . Currently half of all channels have been closed, now it is down to ~70 channels with no improvement with respect to this issue.

If this happens again, are you able to grab a profile? What we want here is a snapshot of all the active goroutines, then we get all the stack traces and can see what is actually blocking.

It happens very frequently. To keep the node up I am restarting it 15-30 times a day. After each restart, there is a chance of some stability time before lncli addinvoice stops working. However, the fact that it might not connect to all peers on restart makes managing this node a real challenge.
Profile taken after ~30 mins running.  How can we deal with these? Are these safe to ignore and retry payment?
In my view the issues (unresponsive lncli commands, no connection to peers, and stuck in_flight payments) are likely to have a common cause. Given how some of these issues have shown before but they have only gotten more severe over time, it seems some sort of "node aging". What is recommended here? Should we simply create a new node and ditch the current one altogether?

@Reckless-Satoshi can you move that to a paste bin so it's easier to scroll down thru the comments here? Thx!

Some sections that pop out:
14 @ 0x43a236 0x44b5af 0x44b586 0x468ee6 0x14bb979 0x14bb955 0x1618005 0x171ddc4 0x171943e 0x171658d 0x46d261
#	0x468ee5	sync.runtime_SemacquireRWMutexR+0x25									runtime/sema.go:82
#	0x14bb978	sync.(*RWMutex).RLock+0xf8										sync/rwmutex.go:71
#	0x14bb954	github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).HasChannelEdge+0xd4				github.com/lightningnetwork/lnd/channeldb/graph.go:1128
#	0x1618004	github.com/lightningnetwork/lnd/routing.(*ChannelRouter).IsStaleEdgePolicy+0x64				github.com/lightningnetwork/lnd/routing/router.go:2658
#	0x171ddc3	github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).handleChanUpdate+0x423		github.com/lightningnetwork/lnd/discovery/gossiper.go:2702
#	0x171943d	github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).processNetworkAnnouncement+0xfd	github.com/lightningnetwork/lnd/discovery/gossiper.go:2007
#	0x171658c	github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).handleNetworkMessages+0x22c		github.com/lightningnetwork/lnd/discovery/gossiper.go:1503

4 @ 0x43a236 0x44a4be 0x14825ea 0x14858df 0x1880358 0x1889fcc 0x1888718 0x1057967 0x105491a 0x185d9d1 0x105491a 0x185d04b 0x105491a 0x185d60a 0x105491a 0x185c858 0x105491a 0x10547be 0x18885d8 0xcaf84f 0xcb3f36 0xcad3b8 0x46d261
#	0x14825e9	github.com/lightningnetwork/lnd/invoices.(*InvoiceExpiryWatcher).AddInvoices+0x89				github.com/lightningnetwork/lnd/invoices/invoice_expiry_watcher.go:265
#	0x14858de	github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).AddInvoice+0x31e					github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:605
#	0x1880357	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.AddInvoice+0x1477						github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/addinvoice.go:474
#	0x1889fcb	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.(*Server).AddHoldInvoice+0x36b				github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_server.go:367
#	0x1888717	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc._Invoices_AddHoldInvoice_Handler.func1+0x77			github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_grpc.pb.go:225
#	0x1057966	github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1+0x86			github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107
#	0x1054919	github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1+0x39					github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
#	0x185d9d0	github.com/lightningnetwork/lnd/rpcperms.(*InterceptorChain).middlewareUnaryServerInterceptor.func1+0x110	github.com/lightningnetwork/lnd/rpcperms/interceptor.go:832
#	0x1054919	github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1+0x39					github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
#	0x185d04a	github.com/lightningnetwork/lnd/rpcperms.(*InterceptorChain).MacaroonUnaryServerInterceptor.func1+0x8a		github.com/lightningnetwork/lnd/rpcperms/interceptor.go:689
#	0x1054919	github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1+0x39					github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
#	0x185d609	github.com/lightningnetwork/lnd/rpcperms.(*InterceptorChain).rpcStateUnaryServerInterceptor.func1+0x109		github.com/lightningnetwork/lnd/rpcperms/interceptor.go:781
#	0x1054919	github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1+0x39					github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
#	0x185c857	github.com/lightningnetwork/lnd/rpcperms.errorLogUnaryServerInterceptor.func1+0x57				github.com/lightningnetwork/lnd/rpcperms/interceptor.go:605
#	0x1054919	github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1+0x39					github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
#	0x10547bd	github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1+0xbd					github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34
#	0x18885d7	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc._Invoices_AddHoldInvoice_Handler+0x137			github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_grpc.pb.go:227
#	0xcaf84e	google.golang.org/grpc.(*Server).processUnaryRPC+0xcee								google.golang.org/grpc@v1.41.0/server.go:1279
#	0xcb3f35	google.golang.org/grpc.(*Server).handleStream+0xa35								google.golang.org/grpc@v1.41.0/server.go:1608
#	0xcad3b7	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x97

Looks like something is trying to grab a lock in the DB, but then that that blocks. Thanks this was super helpful, we'll look into it.

can you move that to a paste bin

Done, sorry about that! Thank you guys for the help!

Those stack traces were super helpful! We have a candidate fix here: https://github.com/lightningnetwork/lnd/issues/7928

Those stack traces were super helpful! We have a candidate fix here: #7928

The PR seems to patch a bug introduced post v0.16.4 release. We are running into this issue on v0.16.4, but also observed in  v0.16.3 and prior.

@Reckless-Satoshi yes if you're able to do so, we also plan to cut an rc2, as several other users reported that the issue was solved by the PR.

@Reckless-Satoshi yes if you're able to do so, we also plan to cut an rc2, as several other users reported that the issue was solved by the PR.

Running rc2. Can we keep this issue open until the issue is confirmed solved? I will report all findings later, but on a quick glance commands like lncli addinvoice take very long to reply (+30 seconds) so I suspect they might still get fully stuck after some time.

but on a quick glance commands like lncli addinvoice take very long to reply (+30 seconds) so I suspect they might still get fully stuck after some time.

Can you get a CPU profile of this?

Reporting some findings here after 24h running v0.17.0.rc2 . The main two issues discussed in this thread persist (1) connection to all peers is randomly lost or never achieved at all at start up and 2) unresponsive RPC for some calls (addinvoice, settleinvoice, etc))
Here (Pastebin) is a profile taken after 12 hours running perfectly (but sluggish) and then RPC became unresponsive. I sent logs to @hieblmi as well.
I also sent logs of a node start up where the node never achieved connecting to all peers, but got stuck at ~12 with broken pipe errors. Similar to the ones I sent when opening this thread. I have observed it can still eventually connect to all peers several hours after. If this is a Tor issue, I am unsure why it is surfacing now as nothing has changed in the configuration.

Can you get a CPU profile of this?
I am not familiar with Go, how can I take a CPU profile of LND?

The node still has a lot of in_flight outgoing payments that have existed for long time. How to deal with these? Any way to close them? Are these safe to ignore and re-attempt payment? Do they consume resources?

I am not familiar with Go, how can I take a CPU profile of LND?

You do it the same way you get a goroutine dump, but you use /debug/pprof/profile?seconds=30 (which will take a snapshot of 30 seconds).

You do it the same way you get a goroutine dump, but you use /debug/pprof/profile?seconds=30 (which will take a snapshot of 30 seconds).

Here the CPU profile while addinvoice is unresponsive https://pastebin.com/0NznDKYE
Another piece of data that might be of interest. We have added another LND docker service on the same docker-compose orchestration. It is a perfect clone, same lnd.conf, uses the same Tor daemon, same Bitcoind, etc. This node does not seem to have the same problems: it start up quick with easy, it can connect to all peers (no broken pipes, at the same time when the other instance can't connect to peers) and it seems to generate invoices timely. I will report if this node starts to show some of the same symptons.

@Reckless-Satoshi can you get that goroutine profile again once the node has been running for longer? At that profile, it looks like you only have 5 or so peers there.
In terms of other configuration changes, you can also look into:

Increasing the bitcoind.txpollinginterval value, you can raise this to something like 5m
Look into increasing the number of rpc threads that bitcoind uses.

Increasing the bitcoind.txpollinginterval value, you can raise this to something like 5m

Look into increasing the number of rpc threads that bitcoind uses.

The issue (hang lncli addinvoice) occured twice again today on the new node after two days working well since we replaced it. Here the CPU profile after it happened while all channels were active: https://pastebin.com/a4KuKXd9
On this new node, in addition of the fixes proposed by @roasbeef on the last comment, we disabled rpc-middleware and I we stopped running LIT and Loop to rule out any cause related to these tools. We also increased ulimit -n considerably in the machine and in the LND container. The issue still happens.

Active goroutines after ~2h running, all peers connected and gRPC functional (but veeery slugish): https://pastebin.com/hV6QKSLK

Thanks for the CPU profile and goroutine dump. Could you please take another CPU profile that covers a longer time span? Perhaps 5 minutes? Just change seconds=30 to seconds=600.
Also, a couple of additional questions:

What is the maxmempool setting of the bitcoind node?
What is the numgraphsyncpeers setting of the lnd nodes?
What is the sync-freelist and db.bolt.nofreelistsync setting of the lnd nodes?
Do you see any errors/warnings in the bitcoind log mentioning http work queue depth exceeded?

Thanks a lot!

What is the maxmempool setting of the bitcoind node?

maxmempool=6000
mempoolexpiry=700

The reason for the large mempool was to avoid the cases where outgoing onchain TXs are forgotten. I see how a maxmempool that big might have an impact. There might also be a correlation of LND issues and mempool getting larger. So after your message, we changed the values to:
maxmempool=768
mempoolexpiry=400

With these values, RPC still becomes unresponsive (but it sort of seems more stable? hard to say, sometimes it last for a couple of days before getting stuck. If we turn on the old LND node (same orquestration), it is guaranteed that the new node's RPC will become unresponsive and connection to peers will drop. It is in fact now impossible to bring up the older node, it seems to never finish connecting to peers, and it crashes the newer LND node.

What is the numgraphsyncpeers setting of the lnd nodes?
9 (default)
What is the sync-freelist and db.bolt.nofreelistsync setting of the lnd nodes?
true (default)
Do you see any errors/warnings in the bitcoind log mentioning http work queue depth exceeded?
No hits for "queue" other than [http] creating work queue of depth 16

I have also inspected the disk (errors, latency, read/write speed), everything seems to be alright.
We have increased min_htlc to 1000 Sats to avoid small holded payment parts. I've observed ~200 pending HTLCs when RPC becomes unresponsive, but it also becomes unresponsive with a smaller number. Not sure if there is any correlation.

Perhaps 5 minutes? Just change seconds=30 to seconds=600

CPU profile on the new node for 5 minutes while RPC is unresponsive https://pastebin.com/YxVSeRDW

Can you generate the svg version of those CPU profiles? It's much easier to interpret like that.
Looking at the prior goroutine dump, it looks like the goroutines are blocked on acquiring locks in the ChannelGraph. So the methods: HasChannelEdge, and UpdateEdgePolicy. This leads me to believe there's an issue in batch/scheduler.go.
Just to confirm, @Reckless-Satoshi you're not running any additional patches on top, and are running with the bbolt backend?

I've been polling my nodes for goroutine dumps for a few minutes, and never see an explicit block on fdatasync like that. If the syscall is blocking indefinately for w/e reason, then just about everything else stops, as the syscall is made in the DB while the main mutex is held.

@Reckless-Satoshi can you try to obtain a blocking profile? This can help us determine if the syscall is the issue or not, or generally where things are blocked on I/O. go tool pprof http://localhost:6060/debug/pprof/block

go tool pprof http://localhost:6060/debug/pprof/block
I will take this one the next time the RPC becomes unresponsive.

Here the 10 min CPU profile as SVG: https://user-images.githubusercontent.com/90936742/267646868-b95dea24-c627-4ce6-a9d7-804df1ba1652.svg

Hi @Reckless-Satoshi
Will the old RoboSats node come online once again or is it dead now? Would like to close the channel in a coop close if possible.
Thanks.

Will the old RoboSats node come online once again or is it dead now? Would like to close the channel in a coop close if possible.

If we manage to bring it back online we will initiate collaborative closes for every channel. We'll try again in the following days.
@Roasbeef I have tried docker exec ... curl http://localhost:6060/debug/pprof/block?seconds=600 --output /save/path/block and go tool pprof -top  /save/path/block several times in a similar fashion to the previous profiles I took, however I am always retrieving an empty set. I always tried when the RPC was unresponsive.
File: lnd
Type: delay
Time: Sep 17, 2023 at 6:21am (UTC)
Duration: 1mins, Total samples = 0
Showing nodes accounting for 0, 0% of 0 total
      flat  flat%   sum%        cum   cum%

Maybe I am doing something wrong here?

That http://localhost:6060/debug/pprof/block needs to be turned on specifically, which is only available in the latest 0.17.0-rc4 version (probably not a good idea to upgrade to that just yet).
I think the output of /debug/pprof/goroutine?debug=2 should also help us already.
By the way, I have to ask about the disk storage again.
You answered that both sync-freelist and db.bolt.nofreelistsync is true? That can't be, as they are the inverse of each other. Do you mean that both options are at their default values (meaning they are commented out in the config)?
What storage backend are you using? Some sort of network/cluster storage? What are the number IOPS allowed (if networked/provisioned storage)? I assume you're using bbolt and not Postgres as the storage backend for lnd?
Have you considered using a bitcoind instance per lnd instance? If changing the mempool size of bitcoind has an impact on how quickly lnd is getting locked up, maybe the bottleneck is actually the number of RPC threads on bitcoind.

You answered that both sync-freelist and db.bolt.nofreelistsync is true? That can't be, as they are the inverse of each other. Do you mean that both options are at their default values (meaning they are commented out in the config)?

Sorry I see now I only replied to sync-freelist is true.  In fact, we are using the default value for both, as these are not specified on our lnd.conf. I assume then db.bolt.nofreelistsync is false.

What storage backend are you using? Some sort of network/cluster storage? What are the number IOPS allowed (if networked/provisioned storage)? I assume you're using bbolt and not Postgres as the storage backend for lnd?

It's a ZFS pool of SSDs in the same machine. The LND directory is simply mounted in the docker container. We are using bbolt.

Have you considered using a bitcoind instance per lnd instance? If changing the mempool size of bitcoind has an impact on how quickly lnd is getting locked up, maybe the bottleneck is actually the number of RPC threads on bitcoind.

We increased the number of bitcoind RPC threads. In restrospect, changing the mempool size does not seem to have had much of an impact, but it's hard to say. We have been running only one LND instance for the last two weeks and still RPC becomes unresponsive. Previous LND instance was monitoring over 12000 onchain addresses, the new one just a few and RPC still hangs.

I think the output of /debug/pprof/goroutine?debug=2 should also help us already.

I will collect this one the next time the node hangs.

In the next few days we will build a new ZFS pool with new and faster drives and move to the new file system only the LND directory. This way we can finally confirm / discard if disks are the issue.

I think the output of /debug/pprof/goroutine?debug=2 should also help us already.

Here the goroutine?debug=2 while RPC is unresponsive. Splitting the pastebin into 4 parts as it exceed the 512Kb limit:
Part 1 https://pastebin.com/V49iDzyG
Part 2 https://pastebin.com/ZqcTsTFn
Part 3 https://pastebin.com/ciNTvARf
Part 4 https://pastebin.com/nkQHiVcU

Thank you! By the way, GitHub also allows uploads of text files, so I'm re-posting the full, combined file here again:
goroutinedump.txt
The full goroutine dump was very helpful!
It looks like we have a deadlock in the HODL invoice system:
--> This goroutine is trying to send on the hodlChannel, which is buffered but probably already has a message in it. This is holding the invoice registry main lock.
goroutine 1667988 [select, 32 minutes]:
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).notifyHodlSubscribers(0xc0005d83c0, {0x22ca860, 0xc015d33e20})
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:1714 +0x265
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).cancelInvoiceImpl(0xc0005d83c0, {0x4a, 0xe, 0x35, 0x49, 0xd6, 0x77, 0xec, 0xdc, 0x86, ...}, ...)
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:1371 +0x585
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).CancelInvoice(...)
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:1293
github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.(*Server).CancelInvoice(0xc003f0db40, {0x1cbd1c0?, 0x1dc2da0?}, 0x0?)
	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_server.go:309 +0xa5
.....

--> 108 goroutines are attempting to add invoices but are blocked on the main invoice registry lock:

goroutine 1690726 [sync.Mutex.Lock, 11 minutes]:
sync.runtime_SemacquireMutex(0x17ffffffffffffff?, 0x0?, 0x1f0000c017c0d050?)
	runtime/sema.go:77 +0x26
sync.(*Mutex).lockSlow(0xc0005d83c0)
	sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
	sync/mutex.go:90
sync.(*RWMutex).Lock(0xc017c0d0a8?)
	sync/rwmutex.go:147 +0x36
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).AddInvoice(0xc0005d83c0, 0xc0253b6c40, {0x4a, 0x4f, 0x96, 0x4a, 0x90, 0x63, 0x96, 0xf8, ...})
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:584 +0x3d
github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.AddInvoice({0x0?, 0x0?}, 0xc017c0d538, 0xc017c0d588)
	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/addinvoice.go:481 +0x1506
github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.(*Server).AddHoldInvoice(0xc003f0db40, {0x22e0938, 0xc01060e3f0}, 0xc028838780)
	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_server.go:367 +0x36c

I'll see if I can come up with a fix.
If there's any way for you to temporarily stop using HODL invoices then that should unclog the node. But I assume that's a main part of your application... So hopefully we can get a patch out soon.

@Reckless-Satoshi worth noting that for the blocking/mutex profiles, you only want to enable them when you want a profile, then turn it off. Otherwise it adds extra load since it continues to collect samples unlike the other profiling options.

@Reckless-Satoshi thanks for that last trace! That was exactly what we needed to get to the bottom of this. I think we have the root cause, and are discussing the best route to resolve the issue (have a few candidates of varying diff size).

Summarized my investigation and current conclusions here: https://github.com/lightningnetwork/lnd/issues/8023.

I have a candidate fix here, it applies on top of master: https://github.com/lightningnetwork/lnd/pull/8024

@Reckless-Satoshi, did https://github.com/lightningnetwork/lnd/pull/8024 fix the issue?

@Reckless-Satoshi, did #8024 fix the issue?

Since we changed the file system for a much faster one where LND is the only service, the hangs of lncli addinvoice have not happened again. It has been over a week without a crash of that kind (this is still on v0.17.0-rc2).
However, we had 3 lncli getinfo hangs in the same week. We are not running yet the fix in #8024. I collected goroutine?debug=2 while lncli getinfo was not responsive in case it is useful I could share it.


Bump cardano-api to <IP_ADDRESS> and ouroboros-* and ekg-forward as a consequence

Bumps cardano-api to <IP_ADDRESS> and ouroboros-* and ekg-forward as a consequence
Context

This is a pre-requisite for bumping cardano-cli to <IP_ADDRESS>.
This is actually very similar to the 8.39.1 release: https://github.com/IntersectMBO/cardano-node/pull/5721 This is merging back the 8.39.1 release into master

cc @bolt12, in case this is relevant to you, because you did the 8.9.1 release

8.9.1 targets a different branch so it is fine targeting master. I think


featureadd more test case

Proposed changes
Issue Number: close #xxx

create mtmv use external catalog
not allow create mtmv use other mtmv
other random function
alter mv property
not allow use mv modify property of table
not allow replace
not allow modify column
allow modify comment
not allow modify grace_period use alter table
not allow rename use alter table

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


Remove dependency from lv_conf.h to Arduino.h by using STL.

[ ] needs to be rebased on develop

I don't think there is a thread-safe way to implement this using STL. See https://github.com/Task-Tracker-Systems/Task-Tracker-Device/pull/94#discussion_r1443777642


ci: remove Ubuntu 14.04 (Trusty Tahr) builds

This distribution has been at end-of-life for 6 months, time to remove
the builds.

This change is

Codecov Report

Merging #3257 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3257      +/-   ##
==========================================
+ Coverage   90.31%   90.31%   +<.01%
==========================================
  Files         450      450
  Lines       39908    39882      -26
==========================================
- Hits        36043    36020      -23
+ Misses       3865     3862       -3

Impacted Files
Coverage Δ

google/cloud/storage/well_known_parameters.h
75.82% <0%> (-3.3%)
:arrow_down:

...oogle/cloud/bigtable/benchmarks/embedded_server.cc
97.7% <0%> (-1.15%)
:arrow_down:

google/cloud/bigtable/row_range.h
100% <0%> (ø)
:arrow_up:

google/cloud/bigtable/metadata_update_policy.h
100% <0%> (ø)
:arrow_up:

...gle/cloud/storage/internal/access_control_common.h
100% <0%> (ø)
:arrow_up:

google/cloud/bigtable/mutations.h
100% <0%> (ø)
:arrow_up:

google/cloud/storage/object_metadata.h
100% <0%> (ø)
:arrow_up:

google/cloud/storage/internal/retry_client.cc
98.36% <0%> (+0.27%)
:arrow_up:

google/cloud/storage/well_known_headers.h
81.48% <0%> (+3.7%)
:arrow_up:

...oogle/cloud/bigtable/internal/unary_client_utils.h
100% <0%> (+17.85%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d143a85...e4ef2cc. Read the comment docs.


envoy: Bump golang version

This also includes other go package upgrade as well.

/test-backport-1.14

/test-backport-1.15

/test-backport-1.14


ctrl+shift+c and ctrl-d shortcuts do not work in WSL2

Windows Terminal version
1.18.3181.0
Windows build number
10.0.19045.3803
Other Software
WSL Debian
Steps to reproduce
Open WSL terminal
Selected text
Press ctrl-shift-c
Paste text into terminal
-> Result: no text is not copied
Using ctrl-c works
Same for ctrl-d:
Open WSL terminal
Press ctrl-d
-> Result: nothing happens
Expected Behavior
ctrl-shift-c copies a selected text
ctrl-d exits the terminal
Actual Behavior
ctrl-shift-c does not copiy a selected text
ctrl-d nothing happens

wsl --version
WSL-Version: <IP_ADDRESS>
Kernelversion: <IP_ADDRESS>-1
WSLg-Version: 1.0.59
MSRDC-Version: 1.2.4677
Direct3D-Version: 1.611.1-81528511
DXCore-Version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows-Version: 10.0.19045.3803

Could you share your settings.json file/?
Are you running anything in WSL other than just your default shell? (bash/zsh/whatever)? Ultimately, that shell is supposed to be the one to handle Ctrl+D, but the Ctrl+Shift+C should be handled by the Terminal.
Actually come to think of it... what shell are you using?

Are you running anything in WSL other than just your default shell? (bash/zsh/whatever)? Ultimately, that shell is supposed to be the one to handle Ctrl+D, but the Ctrl+Shift+C should be handled by the Terminal.
Actually come to think of it... what shell are you using?

bash.
I tried to use ctrl-shift-c in powershell: same issue
The shortcuts worked before, then suddenly vanished.
settings.json

Huh, curious. I'd usually expect the user settings to include at least
    "actions":
    [
        { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" },
        { "command": "paste", "keys": "ctrl+v" },
        { "command": "find", "keys": "ctrl+shift+f" },
        { "command": { "action": "splitPane", "split": "auto", "splitMode": "duplicate" }, "keys": "alt+shift+d" }
    ]

But yours doesn't have any actions at all. That's totally fine, so long as that was intentional.
Maybe there's something else curious going on, where the Terminal thinks that shift is in the opposite of it's actual state. As if caps lock was pressed or something.
Does adding the
    "actions":
    [
        { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" }
    ]

binding at least fix ctrl+shift+c to copy/?

Does adding the
    "actions":
    [
        { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" }
    ]

binding at least fix ctrl+shift+c to copy/?

copying with ctrl+c works after adding the above lines. I also added ctrl+shift+x and it works. Tested on powershell as well as wsl bash.
Any idea what might interfere the shortcuts?

I used debug-tap to see if keystrokes are captured
␛[16;42;0;1;16;1_␛[16;42;0;0;0;1_␛[O
* `␛[16;42;0;1;16;1_␛[16;42;0;0;0;1_␛[O`
  * `Vk`: VK_SHIFT
  * `Sc`: 42
  * `Uc`: 0
  * `Kd`: down`
  * `Cs`: 16
␛[17;29;0;1;8;1_␛[17;29;0;0;0;1_␛[O
* `␛[17;29;0;1;8;1_␛[17;29;0;0;0;1_␛[O`
  * `Vk`: VK_CONTROL
  * `Sc`: 29
  * `Uc`: 0
  * `Kd`: down`
  * `Cs`: 8
␛[67;46;99;1;0;1_c␛[67;46;99;0;0;1_␛[O
* `␛[67;46;99;1;0;1_c␛[67;46;99;0;0;1_␛[O`
  * `Vk`: C key
  * `Sc`: 46
  * `Uc`: 'c'
  * `Kd`: down`
  * `Cs`: 0

To me this reads as if all three keystrokes are received from the terminal

I found the reason and I cannot resist to blame Cisco Webex client for inappropriate behaviour, capturing keystroke combinations in a selfish way.

A-ha! Thanks - that'll also help future searchers stumble on this issue. :smile:


CNF-6974: add all the missing test files with TODO unit tests

This is operator missing crucial unit tests. This is PR is so help devs to get started with all the missing tests.
/cc @jc-rh

/hold

Closing this. Jun will lead the effort for analysis!


Add handling of primitive documentation with ?doc

This implements #36.

By the way, was I doing the right thing with develop as target? I'm a bit unsure since my branch bases on main, but maybe it doesn't matter right now since they are currently the same commit.

Yes, thanks for basing on develop, and for the PR itself too :)


Can't ignore .min.js files

This command does not appear to work
ag "search" ./ --ignore=*.min.js
And yet the ignore does work with
ag "search" ./ --ignore=*.js
I can't find a solution to this issue as the regex searching is only for inclusions and not exclusions

When is the last time you pulled sources from master and rebuilt ag?
This very issue was addressed in PR #680 .

Thanks for your response but it doesn't seem to have fixed the issue.
I updated from 0.28 to 0.30 but the issue still remains.
If I add --ignore=*.min.js minified js files are still included
If I add --ignore=*min.js the js files are excluded
So it seems that it is treating the "." as a special character. I tried escapting this with \ but it still doesn't work.

$ ls
dad.min.js  mom.js  mom.min.js  sludge.js
$ egrep -n stuff *.js
dad.min.js:1:stuff
mom.js:1:stuff
mom.min.js:1:stuff
sludge.js:1:stuff
$ ag --ignore '.min.js' -l stuff
dad.min.js
mom.js
mom.min.js
sludge.js
$ ag --ignore '*.min.js' -l stuff
mom.js
sludge.js
$

The * char is needed to give fnmatch() a chance to match file wildcard patterns.

I understand the * is needed, but when I try exactly what you've listed above I do not get the same result
$ ls
test.js  test.min.js
$ egrep -n stuff *.js
test.js:1:stuff
test.min.js:1:stuff
$ ag --ignore '.min.js' -l stuff
test.js
test.min.js
$ ag --ignore '*.min.js' -l stuff
test.js
test.min.js
$ brew upgrade the_silver_searcher
Error: the_silver_searcher 0.30.0 already installed

The last ag version bump apparently occurred on April 28.  So the ag version number is not an indicator of recent changes, like this one 17 days ago .
Can you pull sources from master and manually build ag?

Let me give that a try and I'll let you know how it goes.

I can confirm that this is fixed in ag master (0.31.0). And being able to ignore .min.js files is supremely useful indeed. :)

Thanks, this works for me now, feel free to close!

Good to hear. Thanks for your help and your patience.


How do we answer: "How many people ended up collaborating because of this"

When a user clicks the collaborate button, how are we tracking it?

We can have an ajax api endpoint that logs if someone clicked the button

Who clicked collaborate
Survey "Hey did you collaborate"
3)Timeframe analysis


What is the timeframe for 0.7 release

Just curious. Trying to learn PureScript taking some time as well... lots to grasp.
Looks like most the architecture discussion is pretty settled and looks very promising!!
What about all the missing sections on the main Readme?
Cheers!!

The Readme is the last thing that needs updating before making the release, I should be done with that on Monday. There may be a few patch releases after that to improve performance, but the types and “external” API are all set now. :)

Excellent!! Thanks :)


Potential bias introduced by FilterReads.py

I just found that doing read filter using FilterReads.py may introduce bias that is not easily noticed. This may has been reported previously but I feel it may be useful to put a warning on README such that people realize such potential bias.
The bias will lead to overestimation of methylation level at lowly methylated sites (<50%) and underestimation at highly methylated sites (>50%). Suppose there are 100 reads covering one isolated CpG whose actual methylation level is 10%. We would expect 90 reads contain T basecalls and 10 reads contain C basecalls, which give us 90% methylation level. If we do FilterReads.py, then only one read with C basecall and one read with T basecall (two in each case if we consider both strands) will be kept, resulting in 50% methylation level. Such bias is more prominent on regions with high coverage and drags methylation level towards 50%. On lowly covered region, the effect is moderate.

Hi @yupenghe Yupeng, thanks for your comment.
Actrually, the FilterReads.py script is an optional one, which were added on the request of some users of BS-Seeker2.
I have added warning message in the script, and README file.
For WGBS, it is still useful to remove reads from PCR effect; and "cgmaptools rmdup" also works in similiar way.
For RRBS, it is not suggested to use this script; as more reads from the same CCGG----CCGG fragment would have similar starts and ends.
Thanks for your message. And looking forward for your further feedback!
Weilong Guo

Thanks very much, Weilong. That makes sense. Just want to share some further thoughts with you. My colleague and I found this issue because doing FilterReaD.py always gives very high bisulfite non-conversion rate (~3%) on unmethylated spike-in control (which frequently has >100x coverage without clonal removal). I tried to use picard for removing PCR duplicates and the relatively high non-conversion rate disappears (to 0.3%). I don't think it will affect much on autosomes and sex chromosomes due to much lower coverage compared to the spike-in.
Yupeng

Thanks Yupeng!
Recently, our new software package "CGmapTools : https://cgmaptools.github.io" is published in Bioinformatics ("CGmapTools improves the precision of hetero- zygous SNV calls and supports allele-specific methylation detection and visualization in bisulfite-sequencing data, Bioinformatics (2017), btx595"). This package is designed as direct downstream analysis package following BS-Seeker2. Hope you will find it be useful.
Looking forward to feedback for both BS-Seeker2 and CGmapTools.
Best,
Weilong
At 2017-12-08 11:08:02, "Yupeng He"<EMAIL_ADDRESS>wrote:
Thanks very much, Weilong. That makes sense. Just want to share some further thoughts with you. My colleague and I found this issue because doing FilterReaD.py always gives very high bisulfite non-conversion rate (~3%) on unmethylated spike-in control (which frequently has >100x coverage without clonal removal). I tried to use picard for removing PCR duplicates and the relatively high non-conversion rate disappears (to 0.3%). I don't think it will affect much on autosomes and sex chromosomes due to much lower coverage compared to the spike-in.
Yupeng
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.


Not working Cordova

Im used this code on index.html. But not working

Registering Device ID...

Failed to load resource: the server responded with a status of 403 (Forbidden)
{"status":403,"service":"Access Manager","error":true,"message":"Forbidden","payload":{"channels":["8 code id"]}}

@azrulharis it seems as though you have enabled Access Manager on this subscribe key.
According to the feature toggle:

Grant fine-grain access control for all publish and subscribe data. Supports any auth tokens including Google, Facebook or homegrown. NOTE: Once enabled, and before sending data, you will need to grant permissions first.

Either turn off Access Manager, or use your secret key to grant an auth token access to this subscribe key and channel, and then perform your device registration using that auth token.

@keyosk Thanks for your reply. Its work now.


feat(mission_planner): publish initial and goal poses to logs

Description
Sometimes, while investigating the planning module, we might find a bug that we want to report on GitHub issues.
However, if we didn't echo the initial and goal poses before the simulation, we might miss this information, which could cause difficulties for the ticket's assignee in reproducing the issues.
Since log information is stored in the .ros folder, if we printed the above information, then we could just search back through the log and add the information to the ticket accurately.
Additionally, I personally think that initial and goal poses are essential pieces of information, and it is acceptable to publish them in the terminal.
I believe a lot of man-hours can be saved if we have this information in the terminal log.
After PR
The following information are published.
When placing goal manually
[openscenario_interpreter_node-3] [component_container_mt-25] [INFO<PHONE_NUMBER>.932131958] [planning.mission_planning.mission_planner] on_set_waypoint_route(): Initial pose - x: 122.000000, y: 120.000000, z: 100.000000:(L316)
[openscenario_interpreter_node-3] [component_container_mt-25] [INFO<PHONE_NUMBER>.932201480] [planning.mission_planning.mission_planner] on_set_waypoint_route(): Initial orientation - qx: 0.000000, qy: 0.000000, qz: 0.707107, qw: 0.707107:(L318)
[openscenario_interpreter_node-3] [component_container_mt-25] [INFO<PHONE_NUMBER>.932273166] [planning.mission_planning.mission_planner] on_set_waypoint_route(): Goal pose - x: 126.000000, y: 720.000000, z: 100.000000:(L325)
[openscenario_interpreter_node-3] [component_container_mt-25] [INFO<PHONE_NUMBER>.932299006] [planning.mission_planning.mission_planner] on_set_waypoint_route(): Goal orientation - qx: 0.000000, qy: 0.000000, qz: 0.707107, qw: 0.707107:(L327)

When route file is used
[component_container_mt-25] [INFO<PHONE_NUMBER>.831081060] [planning.mission_planning.mission_planner] on_set_lanelet_route(): Initial pose - x: 89318.085938, y: 43114.648438, z: 5.952187:(L253)
[component_container_mt-25] [INFO<PHONE_NUMBER>.831137398] [planning.mission_planning.mission_planner] on_set_lanelet_route(): Initial orientation - qx: 0.001785, qy: 0.000471, qz: -0.966898, qw: 0.255157:(L255)
[component_container_mt-25] [INFO<PHONE_NUMBER>.831146385] [planning.mission_planning.mission_planner] on_set_lanelet_route(): Goal pose - x: 89428.851562, y: 43234.484375, z: 5.915613:(L262)
[component_container_mt-25] [INFO<PHONE_NUMBER>.831197853] [planning.mission_planning.mission_planner] on_set_lanelet_route(): Goal orientation - qx: 0.000000, qy: 0.000000, qz: -0.487659, qw: 0.873034:(L264)

Tests performed

Not applicable.
Effects on system behavior

Not applicable.
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[ ] I've confirmed the contribution guidelines.
[ ] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@zulfaqar-azmi-t4
Could you please make it into a function?
@TakaHoribe
Could you check whether the log level is appropriate. I think INFO is fine because it only outputs when setting the route.

@zulfaqar-azmi-t4 @isamu-takagi
Thank you for your support. Setting initialpose is appropriate as an event notification using the INFO level, following the logging rules https://autowarefoundation.github.io/autoware-documentation/main/contributing/coding-guidelines/ros-nodes/console-logging/#rules
Please follow @isamu-takagi -san's comment and his approval.

@isamu-takagi
It is now a function in https://github.com/autowarefoundation/autoware.universe/pull/6918/commits/d4af7a8a9f08ecbbe8485401aeb77baf04c28bd7


unescape url before to send it to database

For some special case, like querying accentuate string non escaped in database. I hacked it for one of my own need, but it could be useful for someone else, i guess.

oups! go fmt applied
but for test, I'm confused, I don't see where them are.
travis complained about a login and password?!

@xulien just login on travis use github account, but the  the PR is in the wrong place, please send on https://github.com/prest/template


Custom Design for UI#243

https://github.com/pdfme/pdfme/issues/243

Hey @peteward ,
If you have time, could you please check this PR?


Fix DiskReadViolation loading flipper in internal builds

Task/Issue URL: https://app.asana.com/0/1198194956794324/1207535395708803/f
Description
Move flipper loading process off main thread as it has disk io
Steps to test this PR
Flipper should still work when showing shared prefs

#4648  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @aitorvs and the rest of your teammates on  Graphite


V2: importable context type. {} as default context and also instead of every never

never is hard to work with when you mock things...
{} as default context makes sense because with any we won't get any ts warnings, it's more strict (breaking).
{} for empty arguments also makes sense since GraphQL provides an empty object when there's no arguments.

@kamilkisiela Maybe mappers can only can a mapping between a GraphQL type and a TypeScript type? and then in order to add the imports, developers can use add plugin.
What do you think?


Creating a route with an existing host name fails to report error

Create a route with an existing hostname
Request fails
No error is shown, stepper closes and user returns to routes tab
regression
Marking as P1... as this could be a common error

This could be due to an old backend?

Updated jetstream and error is successfully shown


Set new_val = old_val + 1

Sorry If I missed something. I could not find how to do this
Set new_val = old_val + 1

I meant set an attribute from another attribute
SET attribute2 = attribute1

Hi @MrNghia123!
Sorry for not having ideal documentation.
Your first example for new_val = old_val + 1 works like this:
{
  new_val: UpdateAction.set(SetValue.add('old_val', 1))
}

However, looking at my code I can interestingly not find any way to simply copy a value from one attribute to another. That's indeed something that should be added.
However, perhaps the most common case for this would be to use SetValue.ifNotExists, since it does copy another attribute but also assigns a default value if that attribute does not exist. Could that work as a workaround?
{
  attribute2: UpdateAction.set(SetValue.ifNotExists('attribute1', 0)
}

Maybe I even deliberately left out the simple copy operator since it fails if the source attribute does not exist, so wrapping such a copy in a if_not_exists is a good idea.

Hi again @MrNghia123!
My apologies, I realized the answer to your question was much easier, you simply use prefix a  referred attribute with #:
{
  attribute2: '#attribute1'
}

This is somewhat described under "pitfalls" in the README: https://github.com/henhal/dynamodb-expressions/?tab=readme-ov-file#namevalue-ambiguities
Thanks

Hi @henhal . Thank you for really helpful information, and such helpful package!
I did figure it out about the # prefix, but your tip to use the if_not_exists to avoid error is awesome.
Best regards,

Thanks!
I ended up creating some more unit tests for these special cases - and I actually found a bug also regarding special characters in attribute names, so that's been fixed now.
I'm closing this issue. Have a nice day!


Figure correct usage of kt_jvm_* rules

(edit) it is not clear that this is needed right now, and I recommend not providing kt_jvm_* implementations to generate_transitive_dependency_targets (that is, calling generate_transitive_dependency_targets() without any arguments)

Ok, I have to walk this back.  I had been led to understand that the ijar producer was substantially more kotlin aware than it actually is.  There are plans in progress, but it's not there yet.  The additional information is needed to avoid the more aggressive java ijar processor stripping method bodies (which shouldn't be stripped for inline functions).

thank you for reporting back. I've subscribed to https://github.com/square/bazel_maven_repository/issues/43 to keep a tap if you figure something out.

Thanks.  The whole problem goes away once bazelbuild/bazel#4549 happens.  At that point, you can just use a java_import and be done with it. :/   So whatever I do will be a temporary workaround while waiting for that one.

Quick update - check the convo on the bazel bug - looks like the current plan of record is to use something similar to kt_jvm_import which doesn't propagate a separate compile_jar (ijar/abi_jar) for maven imports, since the benefits aren't terribly great.  We could use kt_jvm_import, but that ends up creating an inter-workspace dependency on rules_kotlin.  I'm going to make a similar import that does the right ijar thing, but doesn't try to propagate _KtInfo - just does javainfo without ijar.   I'm making it a prerequisite for a 1.0 on my bazel/maven integration.  Feel free to yoink that code once it's done (I'm putting it into its own file anyway)

https://github.com/menny/mabel/pull/64


Bookmark to open to  the selected chapter

On clicking a bookmark, it does not open to the particular chapter.

Selecting a bookmark should be opening to the chapter associated

looks good moving to done.
attaching the recording for the same.
https://user-images.githubusercontent.com/93033055/146251357-706e98f8-f6e4-401b-900b-09f52b0a8f40.mp4


Error when I run the dkrl

Hi bxshi, when I run the dkrl.py, always raise
ValueError: Cannot feed value of shape (2, 14904) for Tensor 'static/content_matrix/initial_value:0', which has shape '(14904,)'
https://github.com/bxshi/ConMask/blob/master/ndkgc/models/dkrl.py#L202
I can't find a correct way to fix it,could you help me ?

Hi Could you try
https://github.com/bxshi/ConMask/blob/master/ndkgc/models/cnn_model.py instead?
I think this file is the correct implementation of DKRL.
I forgot to delete the old one...
On Sat, Jun 8, 2019 at 1:52 AM cxjtju<EMAIL_ADDRESS>wrote:

Hi bxshi, when I run the dkrl.py, always raise
ValueError: Cannot feed value of shape (2, 14904) for Tensor
'static/content_matrix/initial_value:0', which has shape '(14904,)'
https://github.com/bxshi/ConMask/blob/master/ndkgc/models/dkrl.py#L202
http://url
I can't find a correct way to fix it,could you help me ?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bxshi/ConMask/issues/12?email_source=notifications&email_token=AAKBAQ3VWAZ5SKPZZFSCHA3PZNXORA5CNFSM4HWGNEAKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GYMAURA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKBAQ5UUWVPIAIMWXRDYMDPZNXORANCNFSM4HWGNEAA
.

Hey bxshi,
Do you know what metric the original DKRL paper (AAAI version online)  is reporting in table 5 (Entity) and table 6 (relation) in zero-shot scenario results ?. Could you please tell if they are reporting Hits@10 - raw or filter for entities ? are they reporting Hits@1 or Hits@10 for relations (table 6). Sorry, the question is a bit irrelevant to this thread, but I thought you might have run their code and already know the answer?


Fixing the currently-inverted logic in is_new_contributor

The logic in is_new_contributor is currently backwards (i.e., it is returning True for existing contributors). You can see Highfive sending the welcome message to an existing contributor in rust-lang/rust#49795, for example.
This PR inverts the logic so that is_new_contributor returns True for new contributors and False for existing contributors.

Thanks!


Easiest way to wrap std::vector to be able to map it to C# List or int[]

Hello! Thank you for nice project!
I have a small c++ class containing a function:
vector reverseVector(vector input);
If I understand right there is no straightforward way to map vector to List or int[]. First of all I need to write some wrapper class over existing that doesn't use vector. How would this class look like? Is it possible to create such wrappers automatically with CppSharp?

std::vector is a template type and these are quite difficult to support. I had started working on them but other problems got my attention. I hope I'll be able to resume that work in a week.

As you said, there's no easy way to use vector with C#, only with C++/CLI generator.
In our parser code, we deal with it by defining some explicit get/add functions with some C macros. It's not ideal but gets the job done for now: https://github.com/mono/CppSharp/blob/master/src/CppParser/Helpers.h#L28
If you don't need too many times you can define a macro  to define a C++ class with those Add/Get methods internally with same approach. As @ddobrev said, we hope to provide some template improvements soonish.


How to download gs://vit_models/lit/bert/uncased_L-12_H-768_A-12/vocab.txt?

Thanks for the great work. How to download gs://vit_models/lit/bert/uncased_L-12_H-768_A-12/vocab.txt? Can the text encoder due with long sentence (e.g., sentence with 77 tokens? )

Thank you for your interest in our work. We have included the vocab.txt file in the repo, which can be found in this folder. By default, we followed FLIP for fine-tuning with a length of 32. However, we did not observe significant improvements when tuning with longer sentences on LAION dataset. Nevertheless, it is possible to fine-tune a text encoder with 77 tokens as input.

Thanks for your quick reply. Can the text encoder encode long sentence without fine-tuning? I want to replace the [CLIP ViT-L/14] with CLIPA. However, I am not sure whether the CLIPA can adapt to long tokens as [CLIP ViT-L/14].

If you would like to utilize CLIPA's text encoder, I recommend fine-tuning it for encoding longer sentences. Alternatively, you can also consider partially fine-tuning specific layers, such as the position embedding layer. On the other hand, if your intention is to only use CLIPA's visual encoder, it is possible to incorporate another text encoder without the need for fully fine-tuning. However,  I have not tried this before.


Add support for IBIS as a way to programmatically create SQL (and tests)

Checklist

[x] Added a CHANGELOG.rst entry

Merging without review to release 0.2.4


eval

Hello, I would like to ask, now the main function in LSTM has added the eval function? Why can't I run Eval

eval() takes no keyword arguments

It unfortunately doesn't work right now. Please track issue #26 for this


fix: fix error logs

Fix error logs.
I guess these codes are copied from the previous error handling mistakenly.
https://github.com/slsa-framework/slsa-verifier/blob/137528a257f4533d7445ec6ae1bc57011784dab0/verifiers/internal/gha/rekor.go#L81

I think this is a trivial bug fix so an issue is unneeded, but if an issue is needed please let me know.


Create an AzureContentSafetyTextTool

Even with the ContentSafetyRunnable, the clients have said a tool will still be useful in fully exposing the services potential to customers

The class is now finished with testing and docs, and will be merged with the main langchain repo


Can I mount external secrets as volume in a pod

How can I mount secrets from externalSecrets as volumes in a k8s pod instead of just as environmental variables

Hey @CourageOI, you can find that in the Kubernetes docs here: https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod.
ESO just creates Kind=Secret resources that can be consumed with the usual secret APIs, e.g.:
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

I later figured this out thanks


Drop 1 item from inventory on right click in drop region.

Fixes #2295
Adds dropping of 1 item when user presses right click while holding a stack of item on the drop region around inventory grid.

Refer to this link for build results (access rights to CI server needed):
http://jenkins.terasology.org/job/TerasologyPRs/603/
Hooray Jenkins reported success with all tests good!


Any problems with using a Seraphis library that is still in development?

I started with the following scenario: Development on the Seraphis wallet starts, on a new branch for that. We will use @UkoeHB 's Seraphis library by merging the branch that contains it into the wallet development branch.
Questions I tried to answer: If the Seraphis wallet branch changes further, will we run into any problems when getting newest changes into the wallet branch? Are there possible problems with repeated merges? Are there any things that better not happen on the library branch because that would mess up further merges?
I did some experiments to find out.
If an existing and already merged commit changes in the library branch because a commit --amend happens, this seems to be no problem: In my experiment, merge just dealt with this correctly.
I squashed 10 existing and already merged library commits into a single one and merged. Again, no problem, after merge the wallet branch just contained the new squashed commit.
I deleted an existing and already merged commit from the library branch, an older one just picked at random. Merge did not seem to notice, it told me that everything is (still) up to date in the wallet branch. I find this a little surprising, but as deleting commits in this way is probably not a realistic scenario, I don't worry.
After these results I am cautiously optimistic and see a good chance that UkoeHB will be able to just continue work on his library branch, without any restrictions as far as commits are concerned, and we will be able to repeatedly update our wallet development branch from there.
But anyway, I would welcome it if somebody else could verify my results, and maybe go through additional scenarios I did not yet test. I freely confess that I am not very good with git anyway, so I may have overlooked things.

With such updates there seems to be the eternal question whether git merge or git rebase is the better strategy. In the tests described above I did not try to work with rebasing to keep the wallet branch up to date regarding the library branch, but maybe that would be the better approach and maybe would even be able to deal with "exotic" events like random deleted comments.
If somebody has any advice, do tell!


Normalize

I'm finding that normalizing xyz before the calculation is more efficient than normalizing the spherical harmonics at the end

This was implemented as part of #2