Listener not firing when getting internet connection back.

Environnement
react-native info

System:
    OS: macOS 10.15.5
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 33.56 MB / 16.00 GB
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 13.7.0 - ~/.nvm/versions/node/v13.7.0/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.5 - ~/.nvm/versions/node/v13.7.0/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  SDKs:
    iOS SDK:
      Platforms: iOS 13.7, DriverKit 19.0, macOS 10.15, tvOS 13.4, watchOS 6.2
    Android SDK:
      API Levels: 23, 28, 29
      Build Tools: 28.0.3, 29.0.2
      System Images: android-27 | Google Play Intel x86 Atom, android-28 | Intel x86 Atom_64, android-28 | Google APIs Intel x86 Atom, android-28 | Google Play Intel x86 Atom, android-29 | Intel x86 Atom_64, android-29 | Google APIs Intel x86 Atom
  IDEs:
    Android Studio: 3.5 AI-191.8<IP_ADDRESS>10548
    Xcode: 11.7/11E801a - /usr/bin/xcodebuild
  npmPackages:
    react: 16.9.0 => 16.9.0
    react-native: 0.61.5 => 0.61.5
  npmGlobalPackages:
    react-native-cli: 2.0.1
    react-native: 0.61.5

Platforms

Android & iOS
Versions

Android: 9
iOS: 13.6.1
react-native-netinfo: 5.9.2
react-native: 0.61.5
react: 16.9.0

Description

I wired up a listener to know wether internet is reachable from the app. When there i no internet connection the listener fire as espected but when the internet connection comes back, the listener does not fire and the app still consider that internet is not available but the device has internet (email notification).
Reproducible Demo

componentDidMount() {
   ...

   this._networkInfosListener = NetInfo.addEventListener(state => {
      this.setState({isInternetReachable: state.isInternetReachable});
   });

   ...
}

Guys, any update on this bug?

I don't see any update here @muhammadhaseebsohail ? Do you have something new to add? Tracked it down to the relevant source lines and have a PR to offer perhaps? That would move things forward

This is happening for me too.  It works on a device, just not on the simulator.  Test on a device and it should work for you.

@nickmccomb I have two real devices one Android and one iOS and both of them have the issue...

Experiencing the same, but only on simulator (Android). Noticed that after a long-ish time it triggers, at least 60s delay.

Happening for me too, using real devices, and using
"@react-native-community/netinfo": "^5.9.7",

Likweise, happening to me too

iPhone X iOS 13.6, iPhone XR iOS 13.7 - both reproducible.

I am able to reproduce this on iPhone 11 Pro 14.0 under the simulator.

Network on-to-off works correctly.
Network off-to-on does not update.

Same problem for me, very troublesome...
If network come back, error message is fired.
Go offline OK :
Object {
  "details": Object {},
  "isConnected": false,
  "isInternetReachable": null,
  "type": "none",
}

Back Online KO :
Object {
  "details": Object {
    "carrier": "Orange France",
    "cellularGeneration": "4g",
    "isConnectionExpensive": true,
  },
  "isConnected": true,
  "isInternetReachable": null,
  "type": "cellular",
}
Object {
  "details": Object {
    "carrier": "Orange France",
    "cellularGeneration": "4g",
    "isConnectionExpensive": true,
  },
  "isConnected": true,
  "isInternetReachable": true,
  "type": "cellular",
}

Here is my code :
componentDidMount() {
this.handleNetworkChange = NetInfo.addEventListener(state => {
if (state.isInternetReachable !== true) {
// print an error message
}
})
}

Same issues on real devices

Hi any update on this bug?

I suspect that a lot of these are related to the known issue with the iOS simulator, as notes in the README:
https://github.com/react-native-netinfo/react-native-netinfo#issues-with-the-ios-simulator

Hi there, I'm having the same issue with a deployed build. the listener doesn't detect that the network is back. Any idea why this is happening?

+1

is there a fix for this issue?

I've tested using useNetInfo and addEventListener separately to see why connection changes aren't triggering my component to re-render. addEventListener is used underlying implementation of useNetInfo, it looks like it only fires once on initial render.
Example 1:
const netInfo = useNetInfo();
console.log(netInfo); // prints the same value regardless of network change

Example 2:
useEffect(() => {
  return addEventListener((info) => {
    console.log(info); // Prints once on mount, does not print anything after
  });
}, []);

Working Example with fetch:
useEffect(() => {
  const intervalId = setInterval(() => {
    NetInfo.fetch().then((state) => {
      console.log(state); // Prints latest information about network
    });
  }, 1000);

  return () => { clearInterval(intervalId) };
}, []);

I am experiencing similar behaviour both using the eventListner and hook. Also tried @anyamiletic's way of putting netInfo in useEffect. All result the same. What I am noticing is sometimes the network change is not triggered, but triggered on next change and overrides that change. Like online -> offline (didn't trigger) -> online(triggered both current and previous and previous overrides the current one ).
Video example:
https://user-images.githubusercontent.com/13726932/145022432-5d399a2a-63f1-4464-a023-e83c3e872313.mov
https://user-images.githubusercontent.com/13726932/145022559-869c4717-fb40-4713-a705-4a534b64ac04.mov

A release just went out with an update for android that may resolve this.
The original issue was reported for ios and android. Unsure of the testing results using current versions on ios and android, a rigorous test run with complete results posted against current versions on both platforms would be worth a lot.

@mikehardy
Problem not solved 15.12.21

I look forward to PRs, happy to collaborate but this isn't a use case I've got professionally and I'm not interested in working on it personally.

UPD @mikehardy
Guys, please test the network only on real devices, I wrote four solutions and they do not work on emulators, after a while I realized that the problem was in the emulator, the real device is working

The regular event listener in Class Component (NetInfo.addEventListener(state => ...);) works fine on Android, but utilizing the hook useNetInfo() does not fire when disabling both wifi & mobile data on my physical Android device. (I removed any other NetInfo listeners during my tests, just to be sure).
Also writing my own hook does not work on Android, but it works like a charm on iOS.
import React, { useEffect, useState } from "react";
import NetInfo from "@react-native-community/netinfo";

function useNetInfo() {
  const [netInfo, setNetInfo] = useState({
    type: "unknown",
    isConnected: null,
    isInternetReachable: null,
    isWifiEnabled: false
  });

  useEffect(() => {
    const unsubscribe = NetInfo.addEventListener((state) => {
      console.warn("new state", state);
      setNetInfo(state);
    });
    console.warn("added listener");

    return () => {
      unsubscribe();
      console.warn("removed listener");
    };
  }, []);

  return netInfo;
}

export default useNetInfo;

The subscribe and unsubscribe logs are visible, but no logs for new NetInfo state... 🤷🏼
@BLOCKMATERIAL what are these "four solutions" you are talking about if I may ask?
I'm running React Native 0.61.5
And I'm getting the feeling that this issue is to be found in there, rather than this otherwise beautiful netinfo package.
Most related issues on Github all target RN 0.61.5

I can strongly advise not to use such old software - I believe there is a problem in this library with React-native <0.64. As a general rule I would never spend time troubleshooting old software, and I will not here, with personal time. Upgrade to current stable unless you absolutely cannot (like, it won't compile). Get yarn outdated to return clean. Then try again, I bet it will have a different result here

UPDATE:
I did some investigating where the hook would break and it seems to be happening in v7.1.6. Up till v7.1.5 the hook functions as intended on both iOS and Android.
Looking into the changes for 7.1.6 got me to the following conclusion:
hasListener stays false because the React-method addListener() is not invoked when using the hook, this is the only place where hasListener is set to true.
Anyone have a suggestion on fixing this? 😅

@GunnarAK with current react-native, or still creaky old 0.61? Because those changes match changes that were required to handle underlying react-native event emitter changes. Upgrade to current stable, report findings

@mikehardy My findings are from running on RN 0.61.5
Upgrading to current stable is on my schedule for next year.

I believe you will find netinfo works for you post upgrade. Also I think the upgrade is actually just a 15-min task or so probably less than this back and forth between us :-), that's my experience from 61->66 every time I go in to example apps in repos I support and update them. https://react-native-community.github.io/upgrade-helper/

Dropping off my observation for the time being.
Env:

react-native: 0.64.0
@react-native-community/netinfo: 7.1.7
Android physical device (both debug and prod versions)

addEventListener and useNetInfo work at the first time state fetching. But the listeners aren't triggering. (only getting the correct network state when you restart the app)
Even the NetInfo.fetch gives a stale state (it gives output but it's the previous state).
Did some Java debugging and found out that  #548 has caused the issue. It has introduced hasListener flag but the flag is not being set true at the event listener registration as expected.
Ignoring the flag fixed the introduced issue.

For the time being, using this while downgrading to @react-native-community/netinfo: 7.1.5. Will try to create a PR with a proper 'hasListener' flag setting soon.

I think it's more that you need react-native 0.65+ @malithjkmt - have you tried upgrading react-native?

@mikehardy will be doing it soon. Meanwhile, we'll have to fix this issue so it can be used in not only the MOST recent RN version. Or else this library won't be of much use to the community.

I und the frustration however if I could not update rn for whatever reason I'd just pin this dependency at the old version until I could, there are no forced upgrades
That said, i welcome any well thought out PR and we're all doing our best here. Send what you've got and if it looks good I'll merge it

Hi guys, I have a problem that I don't understand I don't know did someone has already fix this issue but I have trouble dealing with it.
I wish to listen to the NetInfo event listener and I set up everything when I turn off my wifi nothing happens when I turn it back on nothing happens, but if I turn my mobile data on/off it will start the event listener.
So my question is what is the way to make NetInfo listeners listen for isWifiEnabled and when it changes to listen for the change and to do some functions.
I would love any answer because I had no luck with searching online.
I m using:
 "dependencies": {
    "@fortawesome/fontawesome-svg-core": "^1.2.36",
    "@fortawesome/free-solid-svg-icons": "^5.15.4",
    "@fortawesome/react-native-fontawesome": "^0.2.7",
    "@react-native-async-storage/async-storage": "^1.15.9",
    "@react-native-community/netinfo": "^7.1.7",
    "@react-native-community/progress-bar-android": "^1.0.4",
    "@react-navigation/drawer": "^6.1.6",
    "@react-navigation/native": "^6.0.4",
    "@react-navigation/native-stack": "^6.2.2",
    "@reduxjs/toolkit": "^1.6.1",
    "i18next": "^21.3.1",
    "jetifier": "^2.0.0",
    "js-coroutines": "^2.4.36",
    "react": "^17.0.2",
    "react-i18next": "^11.12.0",
    "react-native": "^0.66.4",
    "react-native-animate-number": "^0.1.2",
    "react-native-background-timer": "^2.4.1",
    "react-native-circular-progress": "^1.3.7",
    "react-native-collapsible": "^1.6.0",
    "react-native-dashed-progress": "^1.0.1",
    "react-native-document-picker": "^6.0.4",
    "react-native-fetch-blob": "^0.10.8",
    "react-native-file-access": "^2.0.0",
    "react-native-file-logger": "^0.2.1",
    "react-native-fs": "^2.18.0",
    "react-native-gesture-handler": "^1.10.3",
    "react-native-ico-material-design": "^3.3.1",
    "react-native-localization": "^2.1.7",
    "react-native-multithreading": "^1.1.1",
    "react-native-paper": "^4.9.2",
    "react-native-permissions": "^3.1.0",
    "react-native-pinch": "^0.1.0",
    "react-native-reanimated": "^2.2.2",
    "react-native-safe-area-context": "^3.3.2",
    "react-native-screens": "^3.8.0",
    "react-native-segmented-round-display": "^1.0.0",
    "react-native-serialport": "^1.3.1",
    "react-native-ssl-pinning": "^1.5.4",
    "react-native-svg": "^12.1.1",
    "react-native-tcp-socket": "^5.3.1",
    "react-native-threads": "^0.0.19",
    "react-native-usbserial": "^1.0.1",
    "react-native-vector-icons": "^8.1.0",
    "react-native-wifi-reborn": "^4.4.0",
    "react-redux": "^7.2.5",
    "redux": "^4.1.1",
    "redux-devtools-extension": "^2.13.9",
    "redux-saga": "^1.1.3",
    "xlsx": "^0.17.3"
  },

I have no logs for this problem, and if you need more info about my problem feel free to ask I will try to give you as detailed an explanation as I can.
I tried :

useMemo that is listening for useNetInfo.isWifiEnabled and it was working for some time like maybe one or two months and I did no change to the code but it stopped working
useEffect tried to listen with useEffect to see if it will catch isWifiEnabled but had no luck
I also tried both of the examples with :

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />

I tried installing version 6.0.0 but still didn't work.

I m starting to think that there is some problem with the Android Wifi listener so if anyone has the answer it will be helpful a lot.

Is there any fix? The listener is not firing when the internet gets back.

Same issue in 7.1.9 but I downgraded to 6.0.0 and worked for me.

@rajpootathar Works with me, But I used v6.2.1
thank you

@rajpootathar, not working properly for android in the downgraded version too.

I had to rollback to 6.2.1 to get NetInfo.addEventListener() to work at all on Android.

I have same issue. on 6.0.0 and 6.2.1

I'm having the same issue. I'm using version 8.2.0 also I've tried to downgrade version 6.0.0, 6.2.0 and etc they all didn't work...

Tried 6.0.0, 6.2.1, 7..., 8.2.0, and finally the latest 8.3.0
They all happen, unfortunately. Any workaround?

I Experienced the issue in 6.2.0, reverted to 6.1.1 and it went away.

9.3.0 - don't work listener and useNetInfo. isConnected and isInternetReachable is always true. Even when wifi, Bluetooth and mobile network are disabled and avia mode is enabled.
testing on a real android@12 device.

This doesn't work at all! We added a refresh button to let users refresh the network state when they're sure there is a connection but NetInfo.fetch() returns { isConnected: false } always!

@tsdmrfth on API request failure (or whatever reason you need network status) try the imperative method added recently to refresh state and see what happens. If the operating system doesn't call the listener I'm not sure what we can do, but now there is an imperative state refresh escape hatch at least, courtesy of a community PR here

@tsdmrfth you are also in control of the network connected method, you may substitute the native is connected check with your own check now, in the latest code, even specifying whether it is a GET or HEAD request. It does appear we are fighting the platform (Android OS APIs) here, and I'm not sure why they are not working well, but no one has been able to figure it out - there are long threads on the issue like this one, and we're still here.

This is still the issue as of today.
After disabling wifi and turning it on again the event does not trigger.
I have tried it with both event listener and via useNetInfo hook.

This is still happening on the ios simulator on version 9.3.9. Probably connection info is not propagated properly to the simulator.
Luckily it's working as expected on a physical device.

Still unresolved :(


Review for JOSS

Hi @marccanby  —
Just completing my review for JOSS. Looks like Simon's review has covered quite a bit already so just some minor notes in a single issue:

Permissions may not be sufficient

I ran into the "Open a Mac app from an unidentified developer" issue when I went to click "Run PAUP*" later. You might want to specify people to right click and open the paup executable here to trigger the Open a Mac app from an unidentified developer warning here after installation, so R can call it later down the line.

Image broken in README:

Thanks for the great advice! I have taken care of both of these points, so I am closing the issue.


Add how to re-enable SecurityAutoConfiguration doc

Resolves #18

@tzolov: Let's pair on this when you return.

Rejected in favor of #23


depthai-ros: 2.10.2-1 in 'iron/distribution.yaml' [bloom]

Increasing version of package(s) in repository depthai-ros to 2.10.2-1:

upstream repository: https://github.com/luxonis/depthai-ros.git
release repository: https://github.com/luxonis/depthai-ros-release.git
distro file: iron/distribution.yaml
bloom version: 0.12.0
previous version for package: 2.10.1-1

depthai-ros
* Fix Stereo K matrix publishing
* Fix socket ID for NN detections
* Remove catching errors when starting the device since it introduced unexcpected behavior
* Add desqueeze to NN node

Holding for Iron sync


Advanced Filtering: Dropdown cutoff

v1.1.11
Dropdown is being cutoff and is not scrollable:

@andrealperry This is the overlay issue, tracked in issue #527.  Elliot is working on this.

Needed for beta - can we close?

fixed in dropdown v4.6.x


Fix diamond tests

This enables running of collector tests in CI workflow.
Tests that have not been passing or erroring out are blacklisted.

lgtm!
Btw I'd make a wild guess that quite a few failing tests are due to our changes with cumulative counters (since before diamond did the delta calculations locally) and names etc. Of course working through all those just to fix em up would be quite tedious.

Cumulative counters and we added support for dimensions.
I agree that fixing all of these would be quite tedious but I think we can do this as we modify/touch an individual collector. Good news is - about 50% of collector tests are passing and working. So at least -  we make sure we are running tests that are good as part of the build process.

+1


Fix missing metadata directive in example

The metadata directive was missing in this example, so the code did not work as instructed.
Easiest contributor badge ever!

This is the silliest PR I've ever seen! But well done anyway!

Haha, documentation is important 😉


Line issue

i am having an issue with the line that shows where you are gonna move the pawn, when i check error this is what pops up, i am new to godot so idk if this is a common issue or not

I'll take a guess that you just copied the gdscripts files into your project. The examples depend on you defining certain actions in the input map.
Can you confirm by checking the settings by clicking on "Project" in the menu and open "Project Settings...". There is a tab called "Input Map", near the bottom there should be 2 actions mouse_left and mouse_right. If those are missing then you can add them.
It will look like this:


Remove ThePrimeagen/vim-be-good

This repo has not been updated since 2021-03-16 11:25:47 UTC.
Can @ThePrimeagen confirm whether this repo is still intended for use?

vimbegood yes
it definitely was updated 3 days ago? 5? something like that


fix(connector cluster): wrong sdk method used in delete cmd (#1555)

@rkpattnaik780 this branch "code-freeze-bugfixes" is for critical fixes during code freeze.


Research authorizing with different accounts in different windows

The task contains 3 sub-tasks:

support several sessions per one test run. Need to: change internal objects hierarchy in the SessionController, remove the connection between testRun.id and session.id, clone resources (request hooks, client scripts) from the initial session.
userRole in the multi-session mode. Need to discuss the handling of the edge case: apply role for another session, the initial session is closed, the session is closed during role initialization).
decide what action will initiate the new session creation (openWindow call, useRole call, or something else).

https://github.com/DevExpress/testcafe/compare/master...miherlosev:i5343


Cairo1 proven with Stone fails STARK verification on programs with recursion

Exploring Cairo1-run Proving
Setup:
Cairo1 program ➔ Cairo1-run (recursive layout) ➔ Stone Prover ➔ cairo0-verifier (recursive layout debug mode)

Issue:
Some programs work well, while others fail during the cairo0-verifier stage, specifically on the verify_oods that suggests layout/constraint mismatch.

List of Programs Tried:
Working:

simple.cairo - No function calls, no recursion
dictionaries.cairo - No function calls, no recursion
array_append.cairo - No function calls, no recursion
enum_match.cairo - One function call, no recursion

Not Working:

fibonacci.cairo - Function call, recursion
sample.cairo - Function call, recursion
factorial.cairo - Function call, recursion

Observation:
There seems to be an issue with regular recursion, but it's possible there's another underlying problem.
I investigate further but insights and help would be greatly appreciated.
Cheers

Hi @Okm165 !
Thanks for the report.
Yes, we are still working on the proving. Can you try running this program with this branch https://github.com/lambdaclass/cairo-vm/pull/1704/files. We still need to merge it but the Fibonacci and Factorial programs can be verified with stone prover

Hello! which branch did you run this on?

Hey it was main branch

Hi guys ;)
There seems to be a problem with proof mode run of cairo1-run:
branch: cairo-lang-2.6.3
commit: b72f5a93ffeb7de43c2b357ca1555d543c599b0d
This works as expected:
cargo run --release ../cairo_programs/cairo-1-programs/simple.cairo --print_output

Program Output : true

But proof mode makes it fail:
cargo run --release ../cairo_programs/cairo-1-programs/simple.cairo --trace_file program.trace --memory_file program.memory --air_public_input air_public_inpuy.json --air_private_input air_private_input.json --proof_mode

Error: VirtualMachine(Memory(AddressNotRelocatable))

Issue solved!
branch: cairo-lang-2.6.3
commit: b72f5a93ffeb7de43c2b357ca1555d543c599b0d
All previously failing now are successfully running -> proving ->verifying
Thanks for help guys!

Some Stats:

fibonacci.cairo - cairo0-verifier(poseidon+keccak) number of steps: 131072
sample.cairo - cairo0-verifier(poseidon+keccak) number of steps: 262144
factorial.cairo - cairo0-verifier(poseidon+keccak) number of steps: 262144

Can u tell what changed so it works now, what changes does cairo-lang-2.6.3 introduce?

@fmoletta Do u know approx. when cairo-lang-2.6.3 is going to be merged to main?


optimize policy download process in terrascan init

Currently the policy downloading flow is :

Download latest terrascan repo in the local machine and checkout to master branch in a temp location.
After the successful download, move the newly downloaded repo to the base policy path, overriding the existing repo.

There's ample scope for optmization here. We can leverage git diff and git pull for this and download the parts that are outdated or missing. Or save ourselves from downloading the policies if the existing local repo is up to date.
@jlk @patilpankaj212 may have more points to add.

What about using go-git library?

yes, I meant for applying the git diff & git pull equivalent logic using the go-git library.

I think we would be able to achieve that even with git pull as it will take care of just adding new commits, rather than cloning again. And go-git might just support it (ref: https://pkg.go.dev/github.com/go-git/go-git/v5#Worktree.Pull)

@rahulchheda have a look at refreshPolicies function here: https://github.com/patilpankaj212/terrascan/blob/0c294e9b8626591bc9f8f72ba931bc90939c6153/pkg/initialize/run.go, may be you can improve upon this?

@rahulchheda have a look at refreshPolicies function here: https://github.com/patilpankaj212/terrascan/blob/0c294e9b8626591bc9f8f72ba931bc90939c6153/pkg/initialize/run.go, may be you can improve upon this?

This looks really great! Seems like it handles all the cases that I thought of.


Investigate whether Google Apps intermediary page can be avoided

Currently, clicking the "Sign In" button on a tenant that's configured with Google Apps authentication causes you to be sent off to the general Google Auth page.

The user is then expected to enter an email address for that Google Apps domain, without any indication of them having to do so. When signing in with a different Google address, no error message is provided either.
We should investigate if we can:

Send them directly to the SSO page Google forwards them to (preferred)
Provide indication of what address they should use

This should be relatively easy for tenants that have configured a single email domain (using the hd-parameter). However, I'm not sure what we should do when a tenant has configured several domains as Google seems to only allow for one hosted domain option.

We don't encourage (maybe even allow) institutions to have multiple Google Apps domains configured, so focussing on those that have a single domain configured sounds like the right things to do.

Left some minor comments. Back to @sallakarppinen for follow-up

Fixed the style issue and added a link to comment.

Looks good. Merged.


Undefined error at Line 203

Hi
Thanks for this great enhanced version of relative time.
Most of the time it works great, but a few times a day I see this error in my log. It triggers at line 203 in relative_time_plus:
Logger: homeassistant.helpers.event
Source: helpers/template.py:570
First occurred: 09:12:31 (1 occurrences)
Last logged: 09:12:31

Error while processing template: Template<template=({% from 'relative_time_plus.jinja' import relative_time_plus %} {{ 'wait' if not has_value(config.entity) else relative_time_plus(states[config.entity].last_changed, parts=2, week=false, time=true, abbr=true, language='en') }}) renders=8572>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 568, in async_render
    render_result = _render_with_context(self.template, compiled, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2198, in _render_with_context
    return template.render(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/usr/local/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "relative_time_plus.jinja", line 203, in template
  File "/usr/local/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Undefined is not in list

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 694, in async_render_to_info
    render_info._result = self.async_render(
                          ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 570, in async_render
    raise TemplateError(err) from err
homeassistant.exceptions.TemplateError: ValueError: Undefined is not in list

Could you tell me what kind of entity config.entity is? What is the state of this entity when is has a value?

This isn't an issue with this macro.  This is an issue with config.entity being undefined.  I assume this is on a frontend card?  If yes, you need to do...
{% from 'relative_time_plus.jinja' import relative_time_plus %}
{% if config.entity is defined %}
  {{ 'wait' if not has_value(config.entity) else relative_time_plus(states[config.entity].last_changed, parts=2, week=false, time=true, abbr=true, language='en') }}
{% endif %}

Ah thanks for explaining. The config.entity is defined by the  https://github.com/thomasloven/lovelace-template-entity-row.
I expected has_value() to gard me for undefined entities. I have some doubts as an undefined entity would already fail on the state[config.entity] part.
I will try to use is defined instead and see if that solved the issue 👍

Update: it got much worse. My log is now spammed with errors and warnings during startup wrt NoneType.
Will add a check on has_value:
{% if (config.entity is defined) and has_value(config.entity)  %}

Update2:
Can this be related?!:

Ah, okay. I'll build in a safeguard for that

Should be fixed in v0.3.8 now

Will give it a try🥳

Looks great, thanks for the fast fix


view previous file version metadata

For the first task, view the previous file version metadata, both user and extractor generated.

Wanted some input here. Dealing with different file versions, I was thinking of doing something like this.
Here is the file version tab.

I was thinking of adding a button for 'select' or 'selected' next to download, where the version of the file you are viewing will default to the latest version. This would also include adding a visible version somewhere outside of the version tab as well.
I will start with this to be able to get something to test, but wanted to get some opinions .


[Feature] Support SQL direct query in Observability (#988)

Description
backport #988 to 2.x
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

attempt to resolve all conflicts for backport, still need to test if it runs


emit the data upon connect event

It is very useful to have the socket id emitted as well.

@koox00 nice, thx, bumped the version and published

Sorry, the dev tools showed that it returned an object with event and data properties, but it seems it gets consumed somewhere so the response we get is only { socket_id: xxx.xxx}.
I've made another PR.
Also what is your workflow on developing this?

I probably need to make a set up in this repo itself so it can be tested. Right now I am testing changes in a separate project where this repo is used. Not the best way...

It would be much easier for someone else to contribute also ;)


Refactor LeMUR and add missing LeMUR nodes

I refactored the existing LeMUR nodes and added the missing LeMUR endpoints as nodes.

Thank you for the suggestion. I did so in this PR: https://github.com/Ironclad/rivet/pull/90


Duplicated textInputbar when use init with scrollView

Hi when i use super.init(scrollView: asyncCollectionView) in a navigation controller the inputbar is first light grey and when the animation for seague is done is darker grey see screen:

your navigation example with tableview:

it looks like the textInputbar is rendered twice or r duplicated?

Sorry for that i thougt it is only a problem when i use pushViewController at the moment...

I don´t know why but when i wrap it into another viewcontroller it works fine ? :-)


x/exp/slog: unexpected behavior when using NewJSONHandler

What version of Go are you using (go version)?

go version go1.19rc2 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOOS="darwin"

What did you do?
Using the slog package, ran the following program, as well as one with NewTextHandler
func main() {
	log := slog.New( slog.NewJSONHandler(os.Stdout)).With("job", "Eulerian Gophers" )
	log = log.With( "id", 1 )
	log.Log( slog.InfoLevel, "", "total", 5050 )
}

func main() {
	scully := slog.New( slog.NewTextHandler(os.Stdout))
	scully.Error( "👩🦰", errors.New("🛸"))
}

What did you expect to see?
For the first program, JSON parseable by gojq
For the second program, a log message showing some error associating Agent Scully and a UFO
(I don't have the expectation that slog is anything but /x/exp, but just noticed these things)
What did you see instead?
For the first program:
"level":"INFO","msg":"","job":"Eulerian Gophers""id":1,"total":5050}

this isn't parsed because there should be a [edit] a comma here: "Eulerian Gophers","id". There is a similar glitch with TextHandler output, and I guess it's the preformatting. With("job", "Eulerian Gophers", "id", 1 ) eliminates the glitch.
For the second program, using
level=ERROR msg="👩\u200d🦰" err=🛸

Using JSON seems to avoid escaping the zero width joiner.
[edit: fixed a few details in description that were backwards]

cc @jba

The first problem is fixed at master.
The second is WAI. TextHandler uses strconv.Quote to quote strings with non-printing characters, like the zero-width joiner. You can get back the original with strconv.Unquote. JSONHandler uses the same escaping as json.Marshal, which does not escape non-printing characters. We wanted to make parsing of the text format straightforward; we don't want it to depend on anything in the encoding/json package.


Add Darizard Custom Avatar

Another avatar for the wing pile :)

For procedure's sake, would be good to have a screenshot of the avatar.
While it will break in the future, for short term screenshot sharing, just upload to Discord and share that link.

Is this a modification of a standard (non-flying) base, and if so, has the armature been changed sufficiently for it to produce a different hash?
If this is a custom you are making publicly available I (personally) like to include information on how to get it in the PR. (link to store, discord info, etc), or even a link to the public on VRC. This isn't just a nicety though.

@jennytoo This is a modification of Morghus's Nardovern edit of the Nardoragon base by Nardoiri. OpenFlight no longer detects my avatar as a Nardovern due to the changes I made to the dimensions of the Neck and Head bones to achieve a longer neck. This is a personal avatar and at this time I do not plan to make it public.
@mackandelius @Happyrobot33 No problem! Here ya go.


Reading router responses may fail using transport=system

Describe the bug
Hello,
We are using scrapli_netconf to get the configuration. It works fine on our Cisco routers, but when trying Juniper Junos-EVO routers, we have erratic behavior: sometimes it works, sometimes it doesn't.
We need to use the system transport because we are connecting through the devices through a bastion (pty proxy).
What we have identified so far, are the following (potential) issues:
Issue 1: channel is not read after client capabilities are sent (because _server_echo == None at this point - we are hitting line 204 of the scrapli_netconf sync_channel). This means that on the next read (in our case, when calling get_config), the "echo" from the capabilities command will be read by the parser of the get_config command. If the router has not replied yet (or depending on how much it has replied), the parser will stop at what he thinks it has reached the prompt. The rest of the output will not be read. This is shown below in example 2.
Issue 2: too much data is read when looking for the echo. In this case, it seems we are missing the first part of the data, and thus cannot decode the XML in the end. This is shown in example 3. I think the \n<rpc-reply xmlns:junos="http://xml.juniper.net/junos/21.4R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">\n<data>\n is lost in the process.
The currrent assumption is that Juniper devices reply faster than the Cisco devices we have, which explains why we have no issues with Cisco and intermittent issues with juniper (depending on how fast it replies).
More generally, I think there is an issue on how "read" is handled : each function should only consume what it's needs, and put the rest back in the buffer (at the beginning I guess).
To Reproduce
Steps to reproduce the behavior:

Your script:

from scrapli_netconf import NetconfDriver
import logging
import sys

formatter = logging.Formatter(
    # "%(asctime)s - %(thread)d %(threadName)s - %(name)s - %(levelname)s - %(message)s"
    "%(name)s - %(levelname)s - %(message)s"

)
stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setFormatter(formatter)
stream_handler.setLevel(logging.DEBUG)
scrapli_logger = logging.getLogger("scrapli")
scrapli_logger.addHandler(stream_handler)
scrapli_logger.setLevel(logging.DEBUG)

device = {
  "host": "bastion",
  "auth_username": "uname",
  "auth_strict_key": False,
  "auth_private_key": "./key",
  # bastion to device (egress) parameters
  "transport_options": {
      "open_cmd": ["-t", "--", "-P", "device", "--netconf", "-p", "830"]
  },
  # "timeout_transport": 10,
  # disable timeouts for debug
  "timeout_transport": 0,
  "timeout_ops": 0,
  "timeout_socket": 0,
  "channel_log": True,
  "port": 22,
}

success = 0
tries = 0
for i in range(1, 10):
  tries += 1
  try:
    with NetconfDriver(**device) as conn:
      result = conn.get_config()
      print(f"ok {len(result.get_xml_elements())}")
      if len(result.get_xml_elements()) > 0:
        success += 1
      else:
        sys.exit(1)
  except Exception as e:
    print(f"failed {e}")

print(f"success rate {success/tries:%}")

What you're connecting to : Juniper Junos-EVO

Expected behavior
Working get_config every time :)
Stack Trace
Here are the relevant traces for each case:
example 1: working
scrapli.channel - INFO - server capabilities received and parsed: ['urn:ietf:params:netconf:base:1.0', 'urn:ietf:params:netconf:capability:candidate:1.0', 'urn:ietf:params:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:netconf:capability:validate:1.0', 'urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:netconf:base:1.0', 'urn:ietf:params:xml:ns:netconf:capability:candidate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:xml:ns:netconf:capability:validate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring', 'http://xml.juniper.net/netconf/junos/1.0', 'http://xml.juniper.net/dmi/system/1.0']
scrapli.channel - INFO - sending client capabilities
scrapli.channel - DEBUG - attempting to send capabilities: NetconfClientCapabilities.CAPABILITIES_1_0
scrapli.channel - DEBUG - write: '\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.driver - INFO - connection to 'XX' on port '22' opened successfully
scrapli.driver - DEBUG - Building payload for 'get-config' operation. source: running, filter_type: subtree, filter: None, default_type: None
scrapli.driver - DEBUG - Building base element for message id 101
scrapli.driver - DEBUG - Built payload for 'get-config' operation. Payload: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>
scrapli.channel - INFO - sending channel input: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>; strip_prompt: False; eager: True
scrapli.channel - DEBUG - write: '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.channel - DEBUG - read: b'\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>\n<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>\n'
scrapli.channel - DEBUG - server echo is unset, determining if server echoes inputs now
scrapli.channel - DEBUG - server echoes inputs, setting _server_echo to 'true'
scrapli.channel - DEBUG - read: b'<rpc-reply xmlns:junos="http://xml.juniper.net/junos/21.4R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">\n<data>\n<configuration xmlns="http://xml.juniper.net/xnm/1.1/xnm" junos:commit-seconds="1662717163" junos:commit-localtime="2022-09 ... rest of the config follows

example 2: not working, _server_echo == false - response not parsed
scrapli.channel - INFO - server capabilities received and parsed: ['urn:ietf:params:netconf:base:1.0', 'urn:ietf:params:netconf:capability:candidate:1.0', 'urn:ietf:params:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:netconf:capability:validate:1.0', 'urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:netconf:base:1.0', 'urn:ietf:params:xml:ns:netconf:capability:candidate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:xml:ns:netconf:capability:validate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring', 'http://xml.juniper.net/netconf/junos/1.0', 'http://xml.juniper.net/dmi/system/1.0']
scrapli.channel - INFO - sending client capabilities
scrapli.channel - DEBUG - attempting to send capabilities: NetconfClientCapabilities.CAPABILITIES_1_0
scrapli.channel - DEBUG - write: '\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.driver - INFO - connection to 'XX' on port '22' opened successfully
scrapli.driver - DEBUG - Building payload for 'get-config' operation. source: running, filter_type: subtree, filter: None, default_type: None
scrapli.driver - DEBUG - Building base element for message id 101
scrapli.driver - DEBUG - Built payload for 'get-config' operation. Payload: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>
scrapli.channel - INFO - sending channel input: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>; strip_prompt: False; eager: True
scrapli.channel - DEBUG - write: '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.channel - DEBUG - read: b'\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>\n'
scrapli.channel - DEBUG - server echo is unset, determining if server echoes inputs now
scrapli.channel - DEBUG - server does *not* echo inputs, setting _server_echo to 'false'
scrapli.driver - INFO - closing connection to 'XX' on port '22'

example 3: configuration not parsed
scrapli.channel - INFO - server capabilities received and parsed: ['urn:ietf:params:netconf:base:1.0', 'urn:ietf:params:netconf:capability:candidate:1.0', 'urn:ietf:params:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:netconf:capability:validate:1.0', 'urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:netconf:base:1.0', 'urn:ietf:params:xml:ns:netconf:capability:candidate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:xml:ns:netconf:capability:validate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring', 'http://xml.juniper.net/netconf/junos/1.0', 'http://xml.juniper.net/dmi/system/1.0']
scrapli.channel - INFO - sending client capabilities
scrapli.channel - DEBUG - attempting to send capabilities: NetconfClientCapabilities.CAPABILITIES_1_0
scrapli.channel - DEBUG - write: '\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.driver - INFO - connection to 'XX' on port '22' opened successfully
scrapli.driver - DEBUG - Building payload for 'get-config' operation. source: running, filter_type: subtree, filter: None, default_type: None
scrapli.driver - DEBUG - Building base element for message id 101
scrapli.driver - DEBUG - Built payload for 'get-config' operation. Payload: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>
scrapli.channel - INFO - sending channel input: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>; strip_prompt: False; eager: True
scrapli.channel - DEBUG - write: '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.channel - DEBUG - read: b'\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>\n<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>\n<rpc-reply xmlns:junos="http://xml.juniper.net/junos/21.4R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">\n<data>\n'
scrapli.channel - DEBUG - server echo is unset, determining if server echoes inputs now
scrapli.channel - DEBUG - server echoes inputs, setting _server_echo to 'true'
scrapli.channel - DEBUG - read: b'<configuration xmlns="http://xml.juniper.net/xnm/1.1/xnm" junos:commit-seconds="1662717163" junos:commit-localtime="2022-09-09 09:52:43 UTC"

When scrapli returns from this example, the NetconfResponse is empty (ie. len(get_xml_elements()) == 0)
OS

OS: MacOS
scrapli version 2022.7.30.post1
scrapli_netconf 2022.7.30

Additional context
None

Hey @netixx
Thanks for opening this! Yeah I think I sorted all of this out in scrapligo but that doesn't help us much here :)
I think a simple "fix"(hack) could be to simply set _server_echo -> True and then retest everything. Obviously it'd be better to not have to do that, but that could be an easy way to confirm/make sure there is nothing else broken. Its been a while since ive been looking at this stuff, but I also think in your case it should always be on because system transport -- but I could be wrong. If that is the case an easy fix could be to just set server echo to true based on the transport type (then fallback to the current or a future improved version of the echo check).
Just a heads up that this will be pretty low priority for me because I'm not doing networking things lately and also not being able to reproduce it will make it very unpleasant to work on! But, I am happy to work through this and if you come up with a good fix get it merged in!
Carl

Hello,
Unfortunately only changing _server_echo didn't solve the issue. I had to dig a little deeper to fix the issue.
Basically, I implemented a way to save the un-parsed part of the read buffer and give it to the next read() call.
For example, when matching the prompt, only the part up to the prompt is relevant. With the previous implementation, the rest was discarded, which is what was causing the issues.
Now this part is saved for later (aka. the next call to read()).
Do you want to have a look to the proposed changes before I opened a pull request ?
I have tested this on the devices we have available (cisco ios-xr and juniper), but this could use more tests...

Awesome, that sounds good to me! Feel free to open up PR and I'll take a peak and re-run all the functional tests and such for good measure! (probably weekend after this, so may be a bit -- just a heads up)
Thanks again @netixx !

I tried to fix all the tests (changing expected output mainly) as I could, but I saw that at least one failed in the github workflow (python 3.10 - the error does not seem related to the change though).
Because this targets two repositories, the trick is that we need to update the dependencies of scrapli_netconf, but I can't do that since we don't have a version number for scrapli released yet...
For the moment we are running a custom patch on our app, so this is not urgent for us.
Your welcome, let's hope the fix works as expected :)

I haven't had a ton of time to digest this yet but had a few quick thoughts -- sharing here to have one place to put everything!

I dont think(? but could be convinced?) I would want to change scrapli "core". Since we are extending the base/sync channel and this is only affecting netconf I think we should just override read or whatever other methods here in scrapli netconf. I think we should basically never "read past" what we should be reading with "normal" ssh/telnet so it feels unnecessary in core. Again, I could be convinced I am wrong here 😃
Does this/will this also affect async? We probably will want to check on that once we have the sync stuff dialed in!
IOSXR/XR7 7.3.1 returns corrupt '<capabil\n\nity>' I hate netconf so much sometimes (ok, pretty much all the time) 😂
Looks like a bit more re.sub stuff -- possible to just use replace? I think re.sub will be a bit slower but suppose we could check that. If not, maybe collapsing some of the places where there are two re.subs in a row into one is possible?

Will try to take a deeper look this weekend or next at the latest -- so sorry if some (or all) of that doesn't quite make sense haha, just wanted to get some feedback to ya so you aren't hanging waiting on me too long!
Carl

From what I understand of the code, we are subjected to this kind of bug in regular scrapli as well (using system transport) because we use combined input/output fd. So we could run into a race condition where we read what we think is what we send, but actually start reading some of the router response. We can definitely move the code to scrapli_netconf only - but I think we'd be missing out :) (although changing a single repository would make life much easier).
I guess, unless we don't use ptyprocess for async ? Unfortunately we don't use async at the moment so I don't have a way to test it in an existing program. I can test a fix with a mwe if needed though.
yes :) All of this mess could have been avoided with http/json api
I'll look into optimizing this bit. I admit it is not directly related to this issue (it was in the code and I said why not include it here :) ) but I can also move it to a dedicated PR if you'd prefer.

we are subjected to this kind of bug in regular scrapli as well -- that may be true! I'm not 100% sure, but I don't think I've seen it be an issue before. I think also that in "normal" ssh/telnet things are a lot more... "gated" (I'm not sure the right word) -- where we have a lot more control of the flow of things. As in we never ever send a return until we see the expected prompt pattern etc. You are definitely correct that if we accidentally read ahead we will basically break scrapli though. So I will think on this and play around with it more when I get around to digging into this more deeply!
unless we don't use ptyprocess for async -- we don't use ptyprocess, all the async stuff is asyncssh. So I suppose that means we are in a better spot there perhaps. But, I guess once we are happy with how things are setup we should probably just port the changes into the async channel anyway if for no other reason than to be consistent.
yes :) All of this mess could have been avoided with http/json api 😁
I'll look into optimizing this bit -- cool, and no worries, as long as you have a fix that works for you in the short term, we have plenty of time before the next scheduled release to kick things around. After the next few weeks I should have some more time too and I can dig in so you don't need to do all the work!

Haven’t forgotten about this, just been a bit hectic — will try to check back in this weekend!

OK, finally had some time to try to get my hands on this a bit more!
Like I said before, id really prefer to have this been sorted just in scrapli netconf -- you are obviously right that this could crop up as a problem in scrpali "core", but it hasn't yet so if we can make the fixes here (which is obviously easier anyway) that would be ideal. If I could do it all over again I would probably make it more like how scrapligo is -- which is much more similar to what you are proposing... but here we are :)
The good news about trying to do this all in scrapli_netconf is we don't ever need to deal with the re.sub stuff I think -- we can just use partition, so that solves that particular gripe I had!
I pushed a branch here that I think solves some (but I think not all) of the possible cases you outlined. I plagiarized from your base channel stuff, and also the read_tail (I called it read_buf, but obviously same thing!). I think/hope that we should only ever need this behavior for the "initial" setup of things (meaning getting authed/cap exchange/determining echo) so I think I want to try to limit fixes to just that if possible.
So the current branch basically has these things:

override read method in the sync/async channels -- if we have anything in the read_buf, return that and bail, otherwise just super
the capabilities partition and re-feed to the read_buf like you had done
checking to see if we read the hello bits during the initial send input -- if we do, we then re-feed any over read to the read buf, then read (again) until our channel input

I think those changes work... I mean I know they dont break things cuz functional tests are still passing haha, but I just dont know if that fixes your issues, but I think it is a start.
If it would be possible to get the channel log (channel log, not the "normal" scrapli log) of a failure scenario I think I can use scrapli replay to more accurately help fix this. (if this is not working/not on the right path and we need to pow wow more).
I'll go ahead and close the stuff in scrapli core since I dont think I want to do that, then we can keep slowly hashing things out here if that works for you!
Carl

Hello, thank you for having a look :)
Unfortunately, this does not solve my issues. I tested your code against my juniper lab device and couldn't get the data (basically same as before I patched my version of the code).
The issue I think is the partition code which is limited to hello>]]>]]>. I my case, the greedy read happens on this read:
scrapli.channel - INFO - Read: b'\n<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>\n<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/21.4R0/junos"  message-id="101">\n<nc:data>\n<configuration xmlns="http://yang.juniper.net/junos/conf/root" junos:commit-seconds="1664806111" junos:commit-localtime="2022-10-03 14:08:31 UTC" junos:commit-user="root">\n    <version>21.4R3.13-EVO</version>\n    <groups>\n        <name>GRP_SYSLOG</name>\n        <system xmlns="http://yang.juniper.net/junos/conf/system">\n            <syslog>\n                <host>\n                    <name><*></name>\n                    <contents>\n'

more specifically, we need to split after </rpc>]]>]]>. I happens because the device replies between the code calls to read() (the device basically works too well :) ).
This is why I search for the shell pattern in my code, it's 100% reliable because we know that:

the echo will end by the prompt
the response of the server will end by the prompt
By matching for the prompt, we make sure we never over read.

Saying it out loud, I think we could simply patch read to return only the part before the prompt (and store the rest), basically something like:
if self._read_buf:
            read_buf = self._read_buf
            self._read_buf = b""
            return read_buf
read_buf, _, self._read_buf = super().read().partition(']]>]]>')
return read_buf

I'll let you decide if we should add a case for rpc> or if we should partition using ]]>]]> (which is a more general/permanent approach I think).

blah. ok I will look more this weekend or next.
I am super against a broad/generic fix because this should absolutely never happen once server echo is set/we get past this part. We always consume the inputs we write to the channel before sending enter. That is like one of the absolute core parts of scrapli (and by extension scrapli netconf) -- so I want to "solve" for this only at the initial session establishment and nowhere else.
sorry for all the back and forth. I literally have nothing to do with networking at this point so every time I try to look at things I have to spend a while trying to re-remember what the hell is going on everywhere :)
ps - what version/platform juniper is this? is it something that is repeatable in clab?

Oof. Here I go again 🤣
Ok -- so just want to summarize some of my thoughts here since I think I have been sort of scatter brained! Also want to share some history that has played into my thoughts and may or may not be relevant.

General idea of scrapli is never ever to send return until after the input we want to send has been read from the channel.

This obviously sorta fell apart in scrapli netconf due to some devices not echoing -- hence all the server echo things! But we clearly try to adhere to this pattern in scrapli netconf.

There have been several highly annoying problems, particularly, w/ juniper, which I also happen to not particularly like (understatement if we are being honest), and have the least experience with... (we can pretend this is why I dont like juniper so I dont get flamed lol)

The main issue has been despite "one Junos" juniper boxes have behaved differently. I think the first example of this was in #10
And then I had a regression and it came back up sorta in #68
TLDR for this one was that the ]]>]]> was on a newline after the payload (when we send a rpc), and mx boxes (at least?) were apparently not waiting for the delim, just simply responding as soon as the closing rpc tag was seen. Now we put the delim on the same line as the end of the rpc and this shouldn't be an issue.
Why bring this up? because it resulted in what I think is basically the same greedy/over-reading behavior

Now, my fundamental belief is that this over-reading/greedy behavior is strictly limited to session establishment. Why do I say this? Because barring an over zealous device like the MX described above and in #10 the server should never ever ever ever send a response until after we have read our inputs, because we never send return until after we have read that input off the channel. In the case of the server not echoing, this is also never a problem because we never read off our inputs and we also never send the next input until after the response has been read.
So, long story still long -- this greedy behavior has to be limited to before we have figured out the echo state (and only on devices that echo).
 But, as you obviously experienced I did not solve it! I think it solved a possible problem where we read just the capabilities back, but like you said that doesn't prevent us from reading the rpc and possible reply. This is/was all possible because we had not established the echo behavior and therefore we kinda blindly just spam stuff to the server. Thats obviously the technical definition 😎
This is all a pain to troubleshoot/reason about because I'm not involved in networking things at all at this point and I haven't really had to do much other than basic maintenance of scrapli/scrapli_netconf for a while (I guess that is a good sign of general maturity?!?).
With my excuses aside... I needed to find a way to replicate this because I am not a smart man and I gotta see/touch/set breakpoints for myself or I will never figure things out!
import asyncio
from typing import Any, Optional, Type

from asyncssh.channel import SSHServerChannel
from asyncssh.connection import create_server
from asyncssh.public_key import SSHKey
from asyncssh.server import SSHServer
from asyncssh.session import SSHServerSession

from scrapli_replay.logging import logger

BASE_SERVER_KEY = b"""-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAIEAwahUv5Tf3vWQzmz2de791K+vy2WQP9q5eOCAIlD2dFb9lTCg3CNl
kJRLMwelj4eJVdfT6YfQjRfbOkuMtGmwz+ed9ulHBVQ8Ee7JuSfxRcazWx2Wet5wzA0vkv
dohzw20jHhmLpbAi/x20Zxv5R+jK3o/+x6ciIW6sYCoQXJw88AAAIQhQL0T4UC9E8AAAAH
c3NoLXJzYQAAAIEAwahUv5Tf3vWQzmz2de791K+vy2WQP9q5eOCAIlD2dFb9lTCg3CNlkJ
RLMwelj4eJVdfT6YfQjRfbOkuMtGmwz+ed9ulHBVQ8Ee7JuSfxRcazWx2Wet5wzA0vkvdo
hzw20jHhmLpbAi/x20Zxv5R+jK3o/+x6ciIW6sYCoQXJw88AAAADAQABAAAAgQCE8ss7uz
j2GCARlzycOjaIjRRizpb5P2+VTIqrBGot9IqioX/NoX9YgnYd0mIW5zWheUpCSLskIfyf
SL6QHP8EinQ2e5VPO8sJ3So/2/a9H58ATAZX5D/Rmzjjbh9S57NqlM4y+tSaMVZbUFq53D
uhw+OTG1skt3aK4icJVoYWAQAAAEAkDdyAeM/njkl8IrcutV0Qz9uttRJ7piVGswZmwSiG
aBOVYrSvjNen23FuXXHErMTQbDSMzvI2njBB1P10rteWAAAAQQDjdvmIZGQgLaDqFIQH89
M0USeQavVKmVOjnHMOSzuKzhXoMEGaRtV6s1z6R+0FSblMju69I8KP9x8hmw8JxBn1AAAA
QQDZ86OGU3St5fz9INTz+x6wcsVVDXTywDjlU8UDrpZN9Y8WBiTSG1aNRm7IZXxOcmEJ7L
CwttJsdhYnN0En/zgzAAAAFGNhcmxAaW1wb3N0b3JlLmxvY2FsAQIDBAUG
-----END OPENSSH PRIVATE KEY-----
"""

#
CASE_ONE = (
"""<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/17.3R2/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
<nc:data>
<configuration xmlns="http://yang.juniper.net/yang/1.1/jc/configuration/junos-es/17.3R2.10" junos:commit-seconds="1665172339" junos:commit-localtime="2022-10-07 19:52:19 UTC" junos:commit-user="boxen">
    <version>17.3R2.10</version>
</configuration>
</nc:data>
</nc:rpc-reply>
]]>]]>
""",
"""<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/17.3R2/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="102">
<nc:data>
<configuration xmlns="http://yang.juniper.net/yang/1.1/jc/configuration/junos-es/17.3R2.10" junos:commit-seconds="1665172339" junos:commit-localtime="2022-10-07 19:52:19 UTC" junos:commit-user="boxen">
    <version>17.3R2.10</version>
</configuration>
</nc:data>
</nc:rpc-reply>
]]>]]>
""",
)

CURRENT_CASE = CASE_ONE
SERVER_SHOULD_ECHO = True

class BaseSSHServerSession(SSHServerSession):  # type: ignore
    def __init__(self) -> None:
        """
        SSH Server Session class

        Inherits from asyncssh and provides some extra context/setup for the mock network devices

        Args:
            N/A

        Returns:
            None

        Raises:
            N/A

        """
        logger.debug("ssh session initiated")

        self.recv_buf = ""
        self.case_index = 0

    def connection_made(self, chan: SSHServerChannel[Any]) -> None:
        """
        SSH Connection made!

        Args:
            chan: channel editor object

        Returns:
            None

        Raises:
            N/A

        """
        self._chan = chan

    def shell_requested(self) -> bool:
        """
        Handle shell requested; always return True

        Args:
            N/A

        Returns:
            bool: always True!

        Raises:
            N/A

        """
        return True

    def session_started(self) -> None:
        """
        SSH session started

        Initial SSH session started

        Args:
            N/A

        Returns:
            None

        Raises:
            N/A

        """
        print("session starting")
        self._chan.write("""<!-- No zombies were killed during the creation of this user
interface -->

<!-- user admin, class j-super-user -->

<nc:hello xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
   <nc:capabilities>
    <nc:capability>urn:ietf:params:netconf:base:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:candidate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:validate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:url:1.0?protocol=http,ftp,file</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:base:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</nc:capability>
    <nc:capability>http://xml.juniper.net/netconf/junos/1.0</nc:capability>
    <nc:capability>http://xml.juniper.net/dmi/system/1.0</nc:capability>
  </nc:capabilities>
  <nc:session-id>30111</nc:session-id>
</nc:hello>

]]>]]>
""")

    def data_received(self, data: str, datatype: Optional[int]) -> None:
        """
        Handle data received on ssh channel

        Args:
            data: string of data sent to channel
            datatype: dunno! in base class though :)

        Returns:
            None

        Raises:
            N/A

        """
        self.recv_buf += data
        if "]]>]]>" not in self.recv_buf:
            return

        if data != "\n" and SERVER_SHOULD_ECHO:
            self._chan.write(self.recv_buf)
            return

        if "</hello>]]>]]>" in self.recv_buf:
            self.recv_buf = ""
            return

        self.recv_buf = ""
        self._chan.write(CURRENT_CASE[self.case_index])
        self.case_index += 1

        return None

    def eof_received(self) -> bool:
        """
        Handle eof

        Args:
            N/A

        Returns:
            None

        Raises:
            N/A

        """
        print("eof")
        self._chan.exit(0)
        return True

    def break_received(self, msec: float) -> bool:
        """
        Handle break

        Args:
            msec: dunno, but in base class implementation :)

        Returns:
            None

        Raises:
            N/A

        """
        self.eof_received()
        return True

    def subsystem_requested(self, subsystem: str) -> bool:
        return True

class BaseServer(SSHServer):
    def __init__(self, session: Type[SSHServerSession]):  # type: ignore
        """
        Base Server class

        Args:
            session: asyncssh server session

        Returns:
            None

        Raises:
            N/A

        """
        self.session = session

    def session_requested(self) -> SSHServerSession:  # type: ignore
        """
        Session requested; return ServerSession object

        `ServerSession` set in `run` to be the appropriate SSHServerSession type for a given
        platform, i.e. `IOSXESSHServerSession`

        Args:
            N/A

        Returns:
            SSHServerSession: SSHServerSession

        Raises:
            N/A

        """
        return self.session()  # type: ignore

    def begin_auth(self, username: str) -> bool:
        """
        Begin auth; always returns True

        Args:
            username: username for auth

        Returns:
            bool: always True

        Raises:
            N/A

        """
        print("start auth")
        return True

    def password_auth_supported(self) -> bool:
        """
        Password auth supported; always return True

        Args:
            N/A

        Returns:
            bool: always True

        Raises:
            N/A

        """
        return True

    def public_key_auth_supported(self) -> bool:
        """
        Public key auth supported; always return True

        Args:
            N/A

        Returns:
            bool: always True

        Raises:
            N/A

        """
        return True

    def validate_password(self, username: str, password: str) -> bool:
        """
        Validate provided username/password

        Args:
            username: username to check for auth
            password: password to check for auth

        Returns:
            bool: True if user/password is correct (scrapli/scrapli)

        Raises:
            N/A

        """
        if username == password == "scrapli":
            print("password validated")
            return True
        return False

    def validate_public_key(self, username: str, key: SSHKey) -> bool:
        """
        Validate provided public key

        Args:
            username: username to check for auth
            key: asyncssh RSAKey to check for auth

        Returns:
            bool: True if ssh key is correct

        Raises:
            N/A

        """
        if (
            username == "scrapli"
            and key.get_fingerprint() == "SHA256:rb1CVtQCkWBAzm1AxV7xR7BLBawUwFUlUVFVu+QYQBM"
        ):
            return True
        return False

async def start(port: int = 2222) -> None:
    """
    Temporary run server entrypoint

    Args:
        port: port to run the instance on

    Returns:
        None

    Raises:
        N/A

    """

    def server_factory() -> SSHServer:
        server = BaseServer(session=BaseSSHServerSession)
        return server

    await create_server(
        server_factory,
        "localhost",
        port,
        server_host_keys=[BASE_SERVER_KEY],
    )

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(start())
    loop.run_forever()

With the above dummy ssh/netconf server I was able to replicate what I believe is the same issue(s) you are seeing. Note that this just basically sends two get config replies, nothing fancy, but did let me replicate things I think.
With the previous commit (the original commit on that stop-bein-greedy branch) I would sometimes get success, sometimes get capabilities set as a response, and sometimes get the rcp we sent as the response, and sometimes I'd get timeouts. So... even if not exactly the same as your situation, seems like things we need to fix!
Going back to my original feels that this is strictly because of echo establishment and not a more pervasive problem (this is also why I dont want to just partition every read and such), I came up w/ a kinda dumb "fiX" that seems to be working.
On channel creation we set a self._establishing_server_echo attribute to be false. Once we get to that first send_input_netconf and the initial echo check thing we flip this attribute to be true if we see echo-like behavior. This flag then causes our next read_until_input to partition on the delim (]]>]]>) like your proposed example in the read method. The only difference here is it is only happening this one time/one place (which appeases me lol).
After that, we then do the rest of the wrapping up of the echo check, and then we should always be back "in sync" where we are either sending our inputs and reading it off nicely before sending return, or the server doesn't echo and it doesn't matter.
Sorry for giant wall of text, but I know ive been a bit scatter brained on this so wanted to get some more time dedicated to share my thoughts and why I dont want to have a bigger change (one affecting things outside of the echo establishment).
If you could let me know if this sorts things, and if it doesn't maybe we could mess around with the mock ssh server so that I can have a repeatable way to see the same thing you are seeing that would be huge for helping get this dialed in!
Thanks for bearing with me on this @netixx, you rock!

hey there, going to close this out for now just to appease my ocd about open issues. defo would love to make sure we get this solved though so feel free to reopen if we can try to get this sorted. and fwiw -- im not trying to be difficult about getting fixes merged, just want to fully understand and hopefully be able to reproduce and stuff for myself!
thanks again for all the help and detailed information on this. hopefully we can get it sorted this year!

Hello again,
I took some time to test the new version of your branch.
Sorry to be the bearer of bad news, but it doesn't fix my issue :)
Here is the log output if it can help:
scrapli.driver - DEBUG - setting 'server_capabilities' value to '[]'
<redacted>
scrapli.channel - DEBUG - write: REDACTED
scrapli.channel - DEBUG - write: '\n'
scrapli.channel - DEBUG - read: b'\r\n'
scrapli.channel - DEBUG - read: b'Connecting...\r\n'
scrapli.channel - DEBUG - read: b'\x1b[?47l'
scrapli.channel - DEBUG - read: b'Password: '
scrapli.channel - DEBUG - write: REDACTED
scrapli.channel - DEBUG - write: '\n'
scrapli.channel - DEBUG - read: b'\r\r\n'
scrapli.channel - DEBUG - read: b'\r\n'
scrapli.channel - DEBUG - read: b'<!-- No zombies were killed during the creation of this user interface -->\r\r\n\r\n'
scrapli.channel - DEBUG - read: b'<!-- user fespinet, class network-admin -->\r\n'
scrapli.channel - DEBUG - read: b'<nc:hello xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">\r\n   <nc:capabilities>\r\n    <nc:capability>urn:ietf:params:netconf:base:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:netconf:capability:candidate:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:netconf:capability:validate:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>\r\n    <nc:capability>urn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf&revision=2011-06-01</nc:capability>\r\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</nc:capability>\r\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>\r\n'
scrapli.channel - INFO - found start of server capabilities, authentication successful
scrapli.channel - INFO - channel log enabled, logging channel output to 'scrapli_channel.log'
scrapli.channel - DEBUG - read: b'    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types&revision=2013-07-15</nc:capability>\r\n    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</nc:capability>\r\n    <nc:capability>http://xml.juniper.net/netconf/junos/1.0</nc:capability>\r\n    <nc:capability>http://xml.juniper.net/dmi/system/1.0</nc:capability>\r\n  </nc:capabilities>\r\n  <nc:session-id>7397</nc:session-id>\r\n</nc:hello>\r\n]]>]]>\r\n'
scrapli.channel - DEBUG - received raw server capabilities: b'Warning: Permanently added \'XX\' (ED25519) to the list of known hosts.\n*------------------------------------------------------------------------------*\n|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|\n|    ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.    |\n|                                                                              |\n|                Got trouble connecting? => http://ovh.link/pebkac <=          |\n*------------------------------------------------------------------------------*\n\xe2\x94\x82 \xe2\x9d\x97 WARN : You did not use the \'--\' delimiter to pass your remote command, maybe something crazy will happen !\n\xe2\x94\x82 <!-- user fespinet, class network-admin -->\n<nc:hello xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">\n   <nc:capabilities>\n    <nc:capability>urn:ietf:params:netconf:base:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:netconf:capability:candidate:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:netconf:capability:validate:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf&revision=2011-06-01</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types&revision=2013-07-15</nc:capability>\n    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</nc:capability>\n    <nc:capability>http://xml.juniper.net/netconf/junos/1.0</nc:capability>\n    <nc:capability>http://xml.juniper.net/dmi/system/1.0</nc:capability>\n  </nc:capabilities>\n  <nc:session-id>7397</nc:session-id>\n</nc:hello>\n'
scrapli.channel - INFO - server capabilities received and parsed: ['urn:ietf:params:netconf:base:1.0', 'urn:ietf:params:netconf:capability:candidate:1.0', 'urn:ietf:params:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:netconf:capability:validate:1.0', 'urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf&revision=2011-06-01', 'urn:ietf:params:xml:ns:netconf:capability:candidate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0', 'urn:ietf:params:xml:ns:netconf:capability:validate:1.0', 'urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file', 'urn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types&revision=2013-07-15', 'urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring', 'http://xml.juniper.net/netconf/junos/1.0', 'http://xml.juniper.net/dmi/system/1.0']
scrapli.channel - INFO - sending client capabilities
scrapli.channel - DEBUG - attempting to send capabilities: NetconfClientCapabilities.CAPABILITIES_1_0
scrapli.channel - DEBUG - write: '\n<?xml version="1.0" encoding="utf-8"?>\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\n        <capabilities>\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\n        </capabilities>\n</hello>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.driver - INFO - connection to 'XX' on port '22' opened successfully
scrapli.driver - DEBUG - Building payload for 'get-config' operation. source: running, filter_type: subtree, filter: None, default_type: None
scrapli.driver - DEBUG - Building base element for message id 101
scrapli.driver - DEBUG - Built payload for 'get-config' operation. Payload: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>
scrapli.channel - INFO - sending channel input: <?xml version='1.0' encoding='utf-8'?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>; strip_prompt: False; eager: True
scrapli.channel - DEBUG - write: '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>'
scrapli.channel - DEBUG - write: '\n'
scrapli.channel - DEBUG - read: b'\r\n<?xml version="1.0" encoding="utf-8"?>\r\n    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">\r\n        <capabilities>\r\n            <capability>urn:ietf:params:netconf:base:1.0</capability>\r\n        </capabilities>\r\n</hello>]]>]]>\r\n<?xml version=\'1.0\' encoding=\'utf-8\'?>\r\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>\r\n'
scrapli.channel - DEBUG - server echo is unset, determining if server echoes inputs now
scrapli.channel - DEBUG - server echoes inputs, setting _server_echo to 'true'
scrapli.channel - INFO - Read: b'\n<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running/></source></get-config></rpc>]]>]]>'
scrapli.driver - INFO - closing connection to 'XX' on port '22'
scrapli.transport - DEBUG - closing transport connection to 'XX' on port '22'
scrapli.transport - DEBUG - transport connection to 'XX' on port '22' closed successfully
scrapli.driver - INFO - connection to 'XX' on port '22' closed successfully

From what I read from this output, it seem the same output as "example 2" above.
Maybe I should wait for the next release of scrapli/scrapli_netconf and to a full test then ?
We are trying to move to scrapligo for this application, but we are not there yet so we are maintaining this with low effort at the moment.

Hey no worries, thanks for chiming back in! Bummer, but I sorta expected as much haha. I will try to take a peak at this again one of these days and dig into it more. I just need some solid focus time and that seems very hard to come by these days 😁


Update <IP_ADDRESS> with latest

Before creating a pull request make sure that:

[x] commit messages are meaningful and follow good commit message guidelines
[x] README and other documentation has been updated / added (if needed)
[x] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link
Change description
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

Bypass merging as service outage is blocking and there is no code changes


unknown engine

when run regexbench, it shows "unknown engine: hyperscan". And use "-e pcre2",“-e rematch”,it also shows "unknown engine:...". what's the problem?

@daveMmd Sorry for taking so long to get back to you.  I am unable to reproduce the issue you are mentioning. Do you have the correct libraries installed on your machine (pcre2 for pcre2, hyperscan 5.0 for hyperscan, etc)? Also, what system are you running on? I have successfully run the current version on Linux (4.18.0-13-generic) and Mac (10.14.1) and I have not encountered the error as described. Typically, if you get an unknown engine error it means that the desired library is not installed on the machine. When you run cmake you can look to see what engines are installed. There should be a line in the cmake output like the following:
-- AVAILABLE ENGINES: Hyperscan PCRE2 REmatch std::regex Boost::regex


datree: 1.9.8 -> 1.9.9

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/datreeio/datree/releases.
meta.description for datree is: CLI tool to ensure K8s manifests and Helm charts follow best practices
meta.homepage for datree is: https://datree.io/
meta.changelog for datree is: https://github.com/datreeio/datree/releases/tag/1.9.9
Updates performed

Golang update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.9.9 with grep in /nix/store/x1kgfv2h7pxx7q0wij2fv0y4wqyvxrk4-datree-1.9.9
found 1.9.9 in filename of file in /nix/store/x1kgfv2h7pxx7q0wij2fv0y4wqyvxrk4-datree-1.9.9

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
datree

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/x1kgfv2h7pxx7q0wij2fv0y4wqyvxrk4-datree-1.9.9 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A datree https://github.com/r-ryantm/nixpkgs/archive/36a3e8aa2165ff1a31f69e80a264866eb41874d7.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/x1kgfv2h7pxx7q0wij2fv0y4wqyvxrk4-datree-1.9.9
ls -la /nix/store/x1kgfv2h7pxx7q0wij2fv0y4wqyvxrk4-datree-1.9.9/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

datree

Maintainer pings
cc @azahi @jceb for testing.

Result of nixpkgs-review pr 241807 run on aarch64-darwin 1

1 package built:

datree


add placeholder options

This allows us to limit vouchers opening based on a placeholder response

You have to use the "dev" branch in CrazyCrew, You have some conflicts to resolve.


Updated SQLServer2000 Processor to work correctly

Fixes #171

Unfortunately I have uninstalled Sql Server 2000 and I'm not sure I want to go through that process again. This change looks fine as it does not affect anything. Did all the tests pass?

All test pass.
Unfortunately we still have a couple of customers running legacy programs on 2000 which we are still supporting, so we have a 2000 server running, which we can test on.
No I do not have commit rights


Consumer list loading fails with insufficient permissions

If any permissions issues are encountered while loading the individual consumer group, the entire request fails. That >wouldn’t necessarily be an issue, but it is when you’re loading all consumer groups, and not just the one requested

@Haarolean After investigation and attempts to reproduce I see that more details require on this issue. 1. it's not clear what kind of permission prevent the consumer list to be loaded and how to set them up. 2. What does "the entire request fails" mean? Is an exception thrown or does it just return an unexpected (incorrect or empty) list?
Also would be nice to have steps to reproduce if possible. Thx

Originally posted here. The guess is some ACL/MSK permission fails, we'll put this on hold until more details are provided.


Support .note.gnu.property

The PR implements the feature where I would like to discuss the following things:

The merging of the sections happens for all object files where the output section is present just once and should be synthesized. That's why I chose to implement it in EpilogueLayout, where the corresponding sections are identified based on the newly added SectionSlot value. Is it a good design choice?
The implementation supports (similarly to Mold) only properties with 4B property data. Do you have a hint on how to do a smarter serialization in elf_writer.rs? Theoretically, GnuProperty ^1 from object create can be converted to not only read mode.
I face an issue in testing:

Error: Unexpected memory offsets:
Part #17 (section `.note.gnu.property` alignment: 8) expected: 0x400408 actual: 0x4003d8 bumped by: 0x0 requested size: 0x30

Can you help me with that?

The output (.gnu.note.property) should be binary equal to what other linkers do. That said, will we need any special handling in linker-diff`, or is the binary diff the default one?

Thanks for review in advance!
Implements: #38

2. The implementation supports (similarly to Mold) only properties with 4B property data. Do you have a hint on how to do a smarter serialization in elf_writer.rs? Theoretically, GnuProperty 1 from object create can be converted to not only read mode.

FYI: @philipc

Error: Unexpected memory offsets:
Part #17 (section .note.gnu.property alignment: 8) expected: 0x400408 actual: 0x4003d8 bumped by: 0x0 requested size: 0x30

Allocation of addresses within the output file goes through two phases. In the first phase, objects make requests for how much space they want in a particular section. In the second phase (finalise_layout), objects take addresses in the sections in which they requested space. They do this by getting the next address for that section from the argument memory_offsets. Once they take their address, they need to increment the next address for that section so that any subsequent files that also put stuff into that section don't get overlapping addresses.
So one possible fix is to make sure that finalise_layout increments memory_offsets for the .note.gnu.property section by whatever size it is using.
Alternatively, assuming the only "file" that ever does stuff with .note.gnu.property is the epilogue, then you can just ignore that PartId, by adding it in verification::clear_ignored.

The merging of the sections happens for all object files where the output section is present just once and should be synthesized. That's why I chose to implement it in EpilogueLayout, where the corresponding sections are identified based on the newly added SectionSlot value. Is it a good design choice?

I think that's reasonable. Wherever there are things that aren't obviously "owned" by one particular input object, I usually write them from either the prologue or the epilogue. In some other cases, I pick a "winning" input object to be responsible for writing a particular thing. For example if multiple input objects define something, just one of them will be selected to write that thing. Either approach is fine, so whichever you think would be cleanest.

The implementation supports (similarly to Mold) only properties with 4B property data. Do you have a hint on how to do a smarter serialization in elf_writer.rs? Theoretically, GnuProperty 1 from object create can be converted to not only read mode.

If Object doesn't (currently) provide a struct that we can use, then perhaps just define a struct in elf.rs then derive Pod. e.g. see EhFrameHdr.

The output (.gnu.note.property) should be binary equal to what other linkers do. That said, will we need any special handling in linker-diff`, or is the binary diff the default one?

Linker-diff currently won't compare section content by default. I'm not aware of any other sections where we expect all linkers to produce exactly equal content, so this would be the first.

The implementation supports (similarly to Mold) only properties with 4B property data. Do you have a hint on how to do a smarter serialization in elf_writer.rs? Theoretically, GnuProperty from object create can be converted to not only read mode.

GnuProperty is not an appropriate struct for this. It contains a reference to a slice of data which must already be in the correct byte order. Really what you want is something that contains a u32 property and u32 value. It's pretty easy to write that yourself though in three lines of code, as already done elsewhere in object.

So one possible fix is to make sure that finalise_layout increments memory_offsets for the .note.gnu.property section by whatever size it is using.

Thanks for reminding me how the linker works :) So yes, I used the appropriate way and bumped memory_offsets.

Linker-diff currently won't compare section content by default.

Heh. I was too optimistic as we'll need something more complicated. I noticed LLD omits the section for a trivial test case where BFD produces:
Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000030	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86
	x86 ISA used:

And I also noticed the order of the properties depends on the order of the files, where stable output can be ensured by sorting the properties by their type. So a proper comparison (using ^1) would be a better approach!

If Object doesn't (currently) provide a struct that we can use, then perhaps just define a struct in elf.rs then derive Pod. e.g. see EhFrameHdr.

Yeah, I've just done that and the code seems nicer (at least I hope).

So a proper comparison (using 1) would be a better approach!

I take this back :P There are situation where probably some baseline note values are omitted:
❯ echo "int main() { return 0; }" | gcc -x c - -fuse-ld=bfd && readelf -n a.out

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000030	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86
	x86 ISA used:
❯ echo "int main() { return 0; }" | gcc -x c - -fuse-ld=lld && readelf -n a.out
(the section is omitted)
❯ echo "int main() { return 0; }" | gcc -x c - -fuse-ld=mold && readelf -n a.out

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000020	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86
❯ echo "int main() { return 0; }" | gcc -x c - -B ~/Programming/wild && readelf -n a.out

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000030	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86
	x86 ISA used:

Wild's behavior matches to what BFD does. Would it be fine to merge the PR now?


Does have gender and age detection?

Did not find the relevant code, so to confirm that, plz.

The codes for gender and age estimation are not included.

@libraries
just search the "gender estimation", "age estimation" in github, you will find many resources you can use

@chibai
Thanks. I don't need these features, just curious about this project.


Quasiquoters rendering broken in hyperlinked sources

Awesome hyperlinked sources have incorrect quasiquoters rendering, example http://hackage.haskell.org/package/yesod-form-richtext-<IP_ADDRESS>/docs/src/Yesod.Form.Summernote.html#snHtmlFieldCustomized

The problem is that @mrhania implemented a custom lexer rather than reusing GHC's lexer. I tried to port it to use GHC's lexer a few months ago but there were some bad performance problems I didn't work out.

@mpickering Do you happen to have that branch lying around somewhere? I'd love to pick up the torch on this one, but it would be nice to hear what sorts of issues you had, advice you'd like to give.
Also, how were you measuring performance? I don't see any benchmarks in the .cabal files.

@harpocrates IIRC this is the branch https://github.com/mpickering/haddock/commits/perf

I think I've managed to get @mpickering 's branch back into shape. I fixed a handful of things and it seems to be mostly working now (I haven't tried running it on anything large yet, but the performance issues I felt initially went away after rebasing).  I'll open a WIP PR when I get home from work. There are a couple fundamental issues that warrant discussion in any case though:

CPP is not handled in the GHC lexer, yet Haddock will have to handle it. Currently, the solution is to recover from all lexical errors by turning problematic lines into comments (and then restarting the lexer from after that line). That seems like an OK failure mode to me, but I think it should be discussed...

Language pragmas are not lexed as pragmas. This (probably) requires a GHC change. In the meantime, language pragmas would look like regular comments.
GHC's handling of top-level pragmas is a (IMHO) huge lexer-level hack to try to avoid loading the whole file and we are feeling that here. (If bytestring ever becomes a stage 0 package, Data.ByteString.Lazy is what that really needs - not a hand coded list-based parser.)

In any case, the actual output looks great!

This is fixed in #714. Thank you @harpocrates!

The performance problems I had were obvious trying to use the modified version of haddock at all. It would just take ages on any file and use a lot of memory. Thanks for finishing this for me.
I don't understand the comment about language pragmas which seem to get lexed to ITlanguage_prag?

@mpickering I don't think language pragmas get lexed to ITlanguage_prag using lexTokenStream, but I'd love to be proven wrong. My understanding is that the GHC lexer can be initialized with 2 states: one of those is solely for lexing top-level file pragmas (see HeaderInfo) and the other is for lexing the whole file (lexTokenStream uses that).  In the whole file version, ITlanguage_prag tokens never get generated - they just become comments.
My heuristic here was to give all {-# ... #-} comment tokens the TkPragma token style.


Create a status column per major GraalVM version

With the new GraalVM 20.1 release would be nice to have a way to check how each major release compares with the older ones.
WDYT?

Hi @rafaeldelboni
that is a great idea, but we would need a way to easily retest all the projects automatically.
I've done this for the Clojure core check here
but we would need something similar which runs all the projects.
That would be a really good contribution!!! (if you wish to make one ;-) )

out of scope

out of scope