[ BE 23.10.2 ] "Dual Stack" IPv4 + (dynamic) IPv6: Automatic fail-over to a Fallback Gateway fails

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
OPNsense Business Edition 23.10.2 :
"Dual Stack"  IPv4 + (dynamic) IPv6:
Automatic fail-over to another IPv4 Fallback Gateway fails;
Re-Boot needed
To Reproduce
Primary Network:
[ FttB ]   M-Net Premium IP MGA „100 Mbit/s"
[ VDSL ]   PPPoE via VLAN tag 40
"Dual Stack":
    IPv4: fixed
    IPv6: dynamic

providing two gateways:
MNET_PPPOE (IPv4)
MNET_DHCP6 (IPv6)

Secondary Network:
[ W-Lan Router ]
[ VDSL ] "1 und 1"  (via Deutsche Telekom)
"DS-Lite"

providing one "Fallback" gateway:
FB_GWv4 (IPv4)

The configuration was created following

"Practical_OPNsense__4th_2023"
https://docs.opnsense.org/

o) Boot - fully connected:
Lobby:Dashboard:
MNET     ^ (up)
Fallback ^ (up)

System:Gateways:Single
MNET_PPPOE (IPv4) (active) 101 (upstream) Online
MNET_DHCP6 (IPv6) (active) 102 (upstream) Offline   <--- !!!
FB_GWv4 (IPv4)    (active) 111 (upstream) Online

via ssh: Routing:
Default >>> MNET_PPPOE (IPv4)
Default >>> MNET_DHCP6 (IPv6)

Notabene:
default            fe80::aaaa:bbbb:cc UG       pppoe1

via ssh: dpinger:
/usr/local/bin/dpinger -f -S -r 0 -i Fb_GWv4 ...
/usr/local/bin/dpinger -f -S -r 0 -i MNET_PPPOE ...

but no dpinger directed at                          <--- !!!
Monitor-IP for MNET_DHCP6 (IPv6)                    <--- !!!

o) Disconnect DSL-MoDem <---> OPNsense-FW
System:Gatew<ys:Single
MNET_DHCP6 (IPv6) (active) 102 (upstream) Offline   <--- !!!
FB_GWv4 (IPv4)    (active) 111 (upstream) Online
MNET_PPPOE (IPv4) defunct (upstream)      Offline

via ssh: Routing:
Default >>> MNET_DHCP6 (IPv6)                       <--- !!!
==> DNS lookup (IPv4) fails

All the multiple "usual suspects" solutions
suggested in multiple former Forae Feeds fail;
MNET_DHCP6 does not change to "defunct"
o) Reboot helps - but strange:
Lobby:Dashboard:
MNET ^ (up)                                         <--- !!!

via ssh: Routing:
No Default >>> IPv6
Default >>> Fallback (IPv4)

o)  Re-connect:
Lobby:Dashboard:
MNET ^ (up)

System:Gateways:Single
MNET_PPPOE (IPv4) (active) 101 (upstream) Online
MNET_DHCP6 (IPv6) (active) 102 (upstream) Offline   <--- !!!
FB_GWv4 (IPv4)    (active) 111 (upstream) Online

Restore <- Fallback successful:
Default >>> MNET_PPPOE (IPv4)
Default >>> MNET_DHCP6 (IPv6)

o) Reboot:
Same as beginning - as to be expected:
|: . . . :|  ad lib

Expected behavior
In order to secure the (remote) connection,
automatic fail-over should not need a Reboot to function properly.
Describe alternatives you considered
Screenshots
Relevant log files
Additional context
Environment
OPNsense Business Edition 23.10.2-amd64

Might be relevant:
dpinger fails to restart after a PPPoE link recycle
https://forum.opnsense.org/index.php?topic=37948.msg186016#msg186016
dpinger restart required to recover WAN
https://forum.opnsense.org/index.php?action=post;quote=191750;topic=39149.0
dpinger : -> issues: none
https://github.com/dennypage/dpinger
system: adjust to dpinger reality in latency/loss handing #6231
The IPv6 gateway is reported as offline if the WAN interface has a ULA address. #6939

Hint:
In order to make the above work at least via reboot,
I had to configure fix (*) addresses into
. . . System: Settings: General: "DNS Servers"
and disable
. . .  I don't see any logs. It's probably not finding an address to monitor from?!

logs:
Please, be so kind to specify your requests.

Even after the REBOOT:
Name: MNET_DHCP6 (active) <----- ! it should not be !
Priority: 192 (upstream) <----- ! This should be "defunct" !

System: Log Files: General would probably be a good start.

grep MNET_PPPOE /var/log/gateways/latest.log | grep -v "sendto error:"
grep MNET_DHCP6 /var/log/gateways/latest.log | grep -v "sendto error:"
Both MONITOR had correctly detected
. . .  Alarm: none -> loss
. . .  Alarm: loss -> down

! Mid-Air collision !
Not wanting to copy'n'paste that here -
how am I supposed to extract all that info displayed from your web page into a file,
presumably in order to attach that with [ @ ] ?

Just need these lines to begin with:
# opnsense-log | grep skipping

Cheers,
Franco

opnsense-log | grep skipping
<12>1 2024-03-18T12:11:56+01:00 scrat.maknit-sendling.de opnsense-business 26800 - [meta sequenceId="78"] /interfaces.php: The required MNET_DHCP6 IPv6 interface address could not be found, skipping.
<12>1 2024-03-18T13:15:57+01:00 scrat.maknit-sendling.de opnsense-business 311 - [meta sequenceId="326"] /usr/local/etc/rc.bootup: The required MNET_DHCP6 IPv6 interface address could not be found, skipping.
<12>1 2024-03-18T13:15:57+01:00 scrat.maknit-sendling.de opnsense-business 311 - [meta sequenceId="329"] /usr/local/etc/rc.bootup: The required MNET_PPPOE IPv4 interface address could not be found, skipping.
system.log

Ok that's the issue... now when you have a running dpinger for MNET_DHCP6 what does it use?
# ps auxwww | grep MNET_DHCP6

Also, rerunning the monitor init probably fixes it temporarily?
# pluginctl monitor

I should bring back the missing monitor. With this out of the way we could start looking for a reason.

Cleared the log and re-booted with DSL-MoDem attached, for a clean start, IPv6 working:
. # ping -6 heise.de <--- Monitor-IP
PING6(56=40+8+8 bytes) 2001:a61:2a0a:e906:20e:cff:febc:7262 --> 2a02:2e0:3fe:1001:302::
16 bytes from 2a02:2e0:3fe:1001:302::, icmp_seq=1 hlim=58 time=10.823 ms
0:00.00 /usr/local/bin/dpinger -f -S -r 0 -i MNET_DHCP6 -B 2001:a61:2a0a:e906:20e:cff:febc:7262 -p /var/run/dpinger_MNET_DHCP6.pid -u /var/run/dpinger_MNET_DHCP6.sock -s 1s -l 4s -t 60s -d 0 2a02:2e0:3fe:1001:302::
DETACHING the cable between DSL-MoDem and OPNsense FW now:
. # date:  Mon Mar 18 15:04:56 CET 2024
0:00.04 /usr/local/bin/dpinger -f -S -r 0 -i MNET_DHCP6 -B 2001:a61:2a0a:e906:20e:cff:febc:7262 -p /var/run/dpinger_MNET_DHCP6.pid -u /var/run/dpinger_MNET_DHCP6.sock -s 1s -l 4s -t 60s -d 0 2a02:2e0:3fe:1001:302::
. # pluginctl monitor
Setting up gateway monitors...done.
RESULT: as before:
. . . "faulty" IPv6 default route
. . . no IPv4 default route
REBOOT:
. # date:  Mon Mar 18 15:11:07 CET 20
. # ps auxwww | grep dpinger
root    38220   0.0  0.0   13340    2508  -  Is   15:13   0:00.01 /usr/local/bin/dpinger -f -S -r 0 -i Fb_GWv4 -B <IP_ADDRESS> -p /var/run/dpinger_Fb_GWv4.pid -u /var/run/dpinger_Fb_GWv4.sock -s 1s -l 4s -t 60s -d 0 <IP_ADDRESS>
system__002.log

Just discovered that a reboot seems to reset "System: Log Files: General"

REBOOT
DETACH
Save the Log: --> system__003.log
RESET to re-gain a connection via Fallback
Save the Log: --> system__004.log

system__003.log
system__004.log
HTH !

It's very hard to follow again. Attached files are not helpful either :(

?
I just provided copies of the results of the commands you requested;
the attached files are the "System: Log Files: General" you requested in first place.

I know but I said I'd like to keep this focused and we use github to record all information in text. attaching files and posting other metrics while working on diagnosis is detrimental to the support flow.

M.Net contracts supply:
. . . IPv4 (fixed) + IPv6 (dynamic)
Telekom contracts supply:
. . . IPv4 (fixed) + IPv6 Präfix (fixed)
Thus I ran the following comparison using
Interfaces: [Test_IPv6]
Identifier: opt5
Device: em1
IPv6 Configuration Type: Track Interface
IPv6 Interface: MNet

COMPARISON:
A)   Interfaces: [MNet] : Request only an IPv6 prefix:  [ ]
pppoe1: No inet6 2001: ... assigned
em1:       inet6 2001: ... assigned                     <----- difference

Internet6:
default            fe80::46ec:ceff:fe UG       pppoe1
2001:AAA:BBBB:...  link#6             U           em1

... /usr/local/bin/dpinger -f -S -r 0 -i MNET_PPPOE ...
... /usr/local/bin/dpinger -f -S -r 0 -i Fb_GWv4    ...

B)   Interfaces: [MNet] : Request only an IPv6 prefix:  [x]
pppoe1: No inet6 2001: ... assigned
em1:    No inet6 2001: ... assigned                     <----- difference

Internet6:
default            fe80::c242:d0ff:fe UG       pppoe1
2001:aaa:bbbb:cccc localhost          USB         lo0   <----- in addition
2001:aaa:bbbb:cccc link#6             U           em1
2001:aaa:bbbb:cccc link#6             UHS         lo0   <----- in addition
redirector.heise.d fe80::xxxx:yyyy:zz UGHS     pppoe1   <----- in addition

... /usr/local/bin/dpinger -f -S -r 0 -i MNET_PPPOE ...
... /usr/local/bin/dpinger -f -S -r 0 -i MNET_DHCP6 ... <----- in addition
... /usr/local/bin/dpinger -f -S -r 0 -i Fb_GWv4 -B ...

Probably unrelated:
Concerning https://docs.opnsense.org/manual/radvd.html:
In my case, no "» Services » Router Advertisements" submenu is being offered.
.# ps aux | grep radvd
... /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
.# cat  /var/etc/radvd.conf
# Automatically generated, do not edit
# Generated RADVD config for dhcp6 assignment from wan on opt5
interface em1 {
	AdvSendAdvert on;
	AdvLinkMTU 1492;
	AdvManagedFlag on;
	AdvOtherConfigFlag on;
	prefix 2001:a61:aaaa:bbbb::/64 {
		DeprecatePrefix on;
		AdvOnLink on;
		AdvAutonomous on;
	};
	RDNSS 2001:a61:2a0e:5b06:20e:cff:febc:7262 { };
	DNSSL maknit-sendling.de { };
};

Digging /var/log/system (excerpt):
/usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan
/usr/local/etc/rc.newwanipv6:
list ($ip) = interfaces_primary_address6($interface);
if (!is_ipaddr($ip)) {
    log_msg("Failed to detect IP for interface {$interface}", LOG_INFO);
    return;
}

Following the path: finally leading to #7202
Furthermore:
/system_gateways_edit.php: The command '/sbin/route delete -inet6 '2a02:2e0:3fe:1001:302::'' returned exit code '1', the output was 'route: route has not been found delete host 2a02:2e0:3fe:1001:302:: fib 0: not in table'
ERGO:
1.) Tickmark  "Interfaces: [MNet]" -> "Request only an IPv6 prefix"
2.) "System: Gateways: Single" -> remove manual "Monitor IP" entry
grep /var/log/system:
none of both entries above, "Failed" as well as "delete", detectable any more.
But, DETACHING the cable between DSL-MoDem and OPNsense FW again:
/usr/local/etc/rc.newwanip: Failed to detect IP for interface wan
/usr/local/etc/rc.newwanipv6: Failed to detect IP for interface wan
unfortunately
MNET_DHCP6 (active) 	MNet 	IPv6 	102 (upstream) 			0.0 ms 	0.0 ms 	100.0 % 	Offline
still survives, as well as its corresponding routing entries
Internet6:
default            fe80::c242:d0ff:fe UG       pppoe1
2001:a61:2a12:6400 localhost          USB         lo0
2001:a61:2a12:6406 link#6             U           em1
2001:a61:2a12:6406 link#6             UHS         lo0

blocking DNS via IPv4.
All three dpinger still running.
# pluginctl monitor
does not change the situation!
REBOOT:
yields a FW working IPv4 perfectly across its Fallback gateway, with
MNET_DHCP6 (active) 	MNet 	IPv6 	102 (upstream) 			~ 	~ 	~ 	Offline
and only one dpinger on  Fb_GWv4, as to be expected.
RE-CONNECT:
Instantaneously and flawlessly delivers full connection again,
no reboot needed at all,
three dpingers running,
MNET_DHCP6 (active) 	MNet 	IPv6 	102 (upstream) 	fe80::c242:d0ff:fe91:abc0 	fe80::c242:d0ff:fe91:abc0 	6.2 ms 	2.5 ms 	0.0 % 	Online
The core problem still remains: "correctly detect and handle the broken MNET_DHCP6 gateway!"

Notabene:
As long as the broken MNET_DHCP6 keeps blocking,
even an IPv6-capable Fallback would not remedy the situation,
agnostic of that being "fixed" or "dynamic" version.
The situation might be different if the primary connection provides "fixed" IPv6.
Unfortunately, ATM, I do not have such a test bed  available.
@emesterhazy: Thank you
@fichtner: You have marked #7202 for Milestone 24.7.
Will an update for BE take till 24.10 then?

Even worse:
Expecting a solid workaround by completely restraining to
using the (fixed) IPv4 connection only:
Interfaces: [MNet] :  IPv6 Configuration Type: None
Interfaces: [Test_IPv6] : IPv6 Configuration Type: None
consistently deleting MNET_DHCP6 Gateway also.
Alas:
DETACHING the cable between DSL-MoDem and OPNsense FW,
the broken MNET_PPPOE IPv4 default route got removed,
but, again, no default route was created at all;
thus Fb_GWv4 (active) being correctly pulled up as highest priority now
rests useless.
Neither "pluginctl monitor" nor waiting patiently for 7 minutes help.
Only after a REBOOT,
the expected default route via Fb_GWv4 is available.
Very irritating:
Lobby: Dashboard: Interfaces, "MNet" still is displayed as ^ (up) !
Interfaces: Overview: -> MNet interface (wan, pppoe1):
. . . Status  up
. . . PPPoE   up

Heureka! A very minimalist Workaround,
still needing manual interference, but
. . . circumventing any need to re-boot,
. . . circumventing any need to hack via ssh:
A) "Broken Primary Connection, switch to Fallback Connection"
Remove the broken IPv6 assignments from Internet6 Routing tables:
Interfaces: Overview: MNet interface (wan, pppoe1): -> DHCP -> "Release""
Create new default routing across Fallback Interface:
System: Gateways: Single: -> Fb_GWv4 (active) -> "Edit": just -> "Save"
System: Gateways: Single: -> "Apply changes"
B) "Primary Connection working again, switch-back from Fallback Connection"
Bring back MNET_PPPOE and re-create missing IPv6 (default) routing:
Interfaces: MNet: just -> "Save"
Interfaces: MNet: -> "Apply changes"
Tested:
. . . via detaching the cable between DSL-MoDem and OPNsense FW
. . . via re-attaching the cable between DSL-MoDem and OPNsense FW
If you like, you can repeat ad libitum.
This procedure should also help if no (dynamic) IPv6 is involved.
Do not be irritated when pages like "wieistmeineip.de"
need some time to recognize the new situation.
@fichtner:
Franco, I hope this helps to further zero in on the real cause.
Thanks, and kind regards
Manfred

@AdSchellevis :
Really convinced of OPNsense BE for years now, recommending it's application for reliability and continuous development,
please, understand humbly questioning labeling non-functionality of essential Firewall capability
for "Community Support" - especially with respect to "Business Edition".
Kind regards from Munich
Respectfully yours
Manfred

@Manfred-Knick I don't mind changing or removing the label, but currently there's not enough relevant data to label this as something else than support. Sometimes strange timing issues occur due to hardware issues, (very)slow modules or half functional network cards.  When needed we do offer commercial support as well to debug specific issues.
A workaround for strange hardware related issues is sometimes to add a start hook (https://docs.opnsense.org/development/backend/autorun.html#syshook) which triggers an ifconfig XXX down && ifconfig XXX up.
If https://github.com/opnsense/core/issues/7202 includes a fix (some code has been pushed), it should be possible to try with the master branch using the latest community release (and switch to development). When this fixes it, changes will eventually flow into the business edition as well.

Further investigating multiple (IPv4) Fallback Gateways with descending GW priorities { 111 , 121 , ... },
applying (A) works - for the first fallback; but:
"pulling the plug" on the first fallback GW (prio 111) as well,
the following GW (prio 121, ...) does not get any default route;
and (A.1) does not help any more.
System: Gateways: Single:  ->  <GW>  ->  "Edit": "Mark Gateway as Down"
only 'marking' without 'pulling',
I discovered that although MNET_PPPOE and MNET_DHCP6 had been marked as Down,
it was still possible to "ping -6" e.g. heise.de or dns.isp.t-ipnet.de across a gateway being in status Offline (forced) !
These working ping only got disabled after physically 'pulling the plug'.
Sorry, Ad, denouncing tested, reliably-working server-grade equipment will not hold any water at all.
Confirmation:
Utilizing multiple gateways in order to access different (static) net segments ( System: Routes: Configuration )
is easy and works reliable.
HTH
All the best


[portaljs site][showcases][s] Merge examples into Showcases tab

This removes the examples tab and move the Showcases component to it's own tab.
The examples content is still going to be added entirely.

@lucasmbispo @anuveyatsu Shouldn't we replace Frictionlessdata.io and its website preview with Frictionless Dataset to ensure the preview is relevant (i.e. CO2 dataset screenshot)? Otherwise, it’s misleading - it gives the impression that Frictionlessdata.io was created using PortalJS.

I also noticed that the link https://frictionlessdata.io/data-packages/ in the README is showing 404.

Shouldn't we replace Frictionlessdata.io and its website preview with Frictionless Dataset to ensure the preview is relevant (i.e. CO2 dataset screenshot)? Otherwise, it’s misleading - it gives the impression that Frictionlessdata.io was created using PortalJS.

@popovayoana yes, I already suggested that to @lucasmbispo and I believe he has fixed it already. But the image hasn't been updated: https://site-portaljs-git-site-fix-showcases-datopian1.vercel.app/showcases

Shouldn't we replace Frictionlessdata.io and its website preview with Frictionless Dataset to ensure the preview is relevant (i.e. CO2 dataset screenshot)? Otherwise, it’s misleading - it gives the impression that Frictionlessdata.io was created using PortalJS.

@popovayoana yes, I already suggested that to @lucasmbispo and I believe he has fixed it already. But the image hasn't been updated: https://site-portaljs-git-site-fix-showcases-datopian1.vercel.app/showcases

@anuveyatsu updated to the dataset image. It's that what you mean?

@lucasmbispo yes, thanks!


Fix TX: Tons of duplicates

Not seeing this in the raw scrape. Only downstream in our consolidator. This might be some artifact of the older scrape that's running.


Change constant & type colours according to official screenshots

According to the official gallery here https://github.com/morhetz/gruvbox/wiki/Gallery, constants should be gruvbox-violet, and types should be gruvbox-yellow.

This PR fixes this issue: https://github.com/hlissner/emacs-doom-themes/issues/472


Create starter notebook for Colab

Create a starter notebook for people that will run the tutorial on Google Colab.
Add the link to the starter notebook on README.md

I'm struggling with creating a Google Colab notebook that installs pygmt and the other dependencies.
Because pygmtcannot be installed with pip, I tried to follow these instructions for installing Miniconda and then use condato install pygmt, but I'm getting the following error:
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... failed

InvalidVersionSpec: Invalid version '4.19.112+': empty version component


CreateOrder not able to send externalOrderId

My createOrders call works flawlessly whenever I pass my object without the externalOrderId, like:
var args = { orders: [{
	name: 'My Order Name',
	advertiserId: 86999999,
	traffickerId: dfpTraffickerID
}]};

However, I get a soap error when I pass the externalOrderId, like:
var args = { orders: [{
	name: 'My Order Name',
	advertiserId: 86999999,
	traffickerId: dfpTraffickerID,
	externalOrderId: '#100043'
}]};

The error is below and I sure would appreciate any help you could give. I've tried using v201505, v201511 and v201602.
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <soap:Fault>
      <faultcode>soap:Client</faultcode>
      <faultstring>Unmarshalling Error: cvc-complex-type.2.4.a: Invalid content was found starting with element 'externalOrderId'. One of '{"https://www.google.com/apis/ads/publisher/v201602":secondaryTraffickerIds, "https://www.google.com/apis/ads/publisher/v201602":salespersonId, "https://www.google.com/apis/ads/publisher/v201602":secondarySalespersonIds, "https://www.google.com/apis/ads/publisher/v201602":totalImpressionsDelivered, "https://www.google.com/apis/ads/publisher/v201602":totalClicksDelivered, "https://www.google.com/apis/ads/publisher/v201602":totalViewableImpressionsDelivered, "https://www.google.com/apis/ads/publisher/v201602":totalBudget, "https://www.google.com/apis/ads/publisher/v201602":appliedLabels, "https://www.google.com/apis/ads/publisher/v201602":effectiveAppliedLabels, "https://www.google.com/apis/ads/publisher/v201602":lastModifiedByApp, "https://www.google.com/apis/ads/publisher/v201602":isProgrammatic, "https://www.google.com/apis/ads/publisher/v201602":programmaticSettings, "https://www.google.com/apis/ads/publisher/v201602":appliedTeamIds, "https://www.google.com/apis/ads/publisher/v201602":lastModifiedDateTime, "https://www.google.com/apis/ads/publisher/v201602":customFieldValues}' is expected. </faultstring>
    </soap:Fault>
  </soap:Body>
</soap:Envelope>

Also, what would be the best way to log the soap requests as they're sent? I'd like to get a look at them going out to figure out what's going wrong...
Thanks a ton man,
Ferg

Same error here as with #18. Try ordering your properties to match the order in the api reference. This is an error coming from something in the WSDL library I think. Since property order is not significant in JS (and a pain to get right) I think the long term fix for this would be not have a mechanism (in either node-google-dfp or node-google-dfp-wrapper) that the input object to another object with the properties in the correct order.

Same as #18  - thanks again!


AD Login fails with "Invalid Credentials"

Rancher versions:
rancher/server or rancher/rancher: 2.0.4 (also tested in 2.0.2)
Steps to Reproduce:
We want to configure Rancher to authenticate against our AD. We've configured our AD accordingly (it's the same configuration we use for other services). When testing the connection we receive "authentication failure".
Results:
Receiving "authentication failure" and the log entry:
[ERROR] API error response 401 for POST /v3/activeDirectoryConfigs/activedirectory?action=testAndApply. Cause: LDAP Result Code 49 "Invalid Credentials": 80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580
We rechecked the credentials. They are correct and for other services they are working.

@phertweck Can you please share debug logs?

We managed to configure the AD without TLS. Removing the default login domain and only entering the user name did the trick.
Now we want to secure the authentication by enabling TLS. Our server has a self signed certificate. We provide the CA pem file through the UI, but receive the following error:
Error creating ssl connection: LDAP Result Code 200 "": read tcp 10.X.X.X:38392->10.X.X.X:636: read: connection reset by peer In the debug log there is no other message.
When we try to use ldap search using TLS and providing the CA (
env LDAPTLS_CACERT=cert.pem ldapsearch -H <ldaps://server> -x -D "<defaultLoginDomain>\<serviceAccountUsername>" -W -b "<UserSearchBase>" -s sub "(&(sAMAccountName=<testuserWhoIsLoggingin>)(objectClass=person))") it works.

Hello,
I've a similar problem on rancher 2.0.4. Leaving the default domain empty and specifing the service username in the service account field as well as in the test user field works. If I try to add an other user to the test user field I get an authentication failure.
Logs:
[DEBUG] configApplyInput &{{{ } {      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{ } {      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } activeDirectoryConfig true unrestricted []} [LDAP_IP] 389 false   SERVICE_USERNAME SERVICE_PASSWORD 2 DC=home,DC=DOMAIN,DC=TLD sAMAccountName|sn|givenName sAMAccountName person name userAccountControl  sAMAccountName group name distinguishedName distinguishedName member 5000} USERNAME MY_PW true}
2018/07/12 21:06:16 [DEBUG] Now generating Ldap token
2018/07/12 21:06:16 [DEBUG] Now creating Ldap connection
2018/07/12 21:06:16 [DEBUG] Binding username password
2018/07/12 21:06:16 [ERROR] API error response 401 for POST /v3/activeDirectoryConfigs/activedirectory?action=testAndApply. Cause: LDAP Result Code 49 "Invalid Credentials": 80090308: LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e, v3839
Running this ldap search returns the USERNAME object:
ldapsearch -H ldap://AD-IP -x -D "SERVICE_USERNAME" -W -b "DC=home,DC=DOMAIN,DC=TLD" -s sub "(&(sAMAccountName=USERNAME)(objectClass=person))"

I think @phertweck's TLS issue is ultimately the same issue we saw here in 1.6.x:
https://github.com/rancher/rancher/issues/10008
The fix was to upgrade go to 1.10

@phertweck would you be willing to test out a RC as we don't have an auth server with the same level of certs?

@deniseschannon Yes, sure. What's the tag of the image I should test?

@phertweck the fix that we had merged in to bump the go version has introduced some issues and we've had to revert that fix.
we need to do more investigation at it looks like there is a bug in golang 1.10.3 that has been fixed in golang 1.11, which is not released yet.

Thank you for this information.
Will golang 1.11 only fix the cert issue or will this also affect the "Invalid Credentials" issue, which also appears without TLS?

@phertweck My understanding was that that the invalid credentials only appears with TLS? Are you saying it appears without TLS?

@deniseschannon I think there are two problems.

Connecting to AD without TLS: I used a service account to connect to AD. Testing the login with the service account is working, but testing it with an other user account is not working. I think that's the same effect @Nephelo is facing.

Connecting to AD with TLS it is directly failing.

Please let me know if you need further information.

@Nephelo @phertweck I filed https://github.com/rancher/rancher/issues/15116
to address the first issue. I'm going to rename this issue to track the connecting AD with TLS since the history in this ticket has been around this issue.

golang is bumped up to v1.11 in rancher 2.1, marking as resolved

@sangeethah nothing specific to test, lets just make sure our validation test passes

@phertweck would you be able to test using v2.1.0-rc1 to test that you are able to enable AD authentication with your certificate type?

Thanks @phertweck!


Add aws policy for iam:TagRole to xaccount role

A workaround to add the iam:TagRole permission to the cross account role. This allows successful deployment of data services on Terraform deployed environments.

Closing this. Change will be added to the modules instead - as part of PR https://github.com/cloudera-labs/terraform-cdp-modules/pull/60


Sniff names now doubled?

I'm noticing that the sniff names being reported by CodeSniffer in these standards have recently changed, and I'm wondering if this is intentional?
For example, my most recent report mentions this sniff:
WordPress.VIP.PostsPerPage.posts_per_page_posts_per_page

(from https://travis-ci.org/MITLibraries/MITlibraries-parent/jobs/197719119 )
Some two weeks ago, those builds were coming back clean because we've explicitly excluded the old sniff name of WordPress.VIP.PostsPerPage.posts_per_page. I'm happy to update our exclusion list, but on the chance that the name change will be reverted I wanted to check in here first.

Hi @matt-bernhardt,
Yes, the change is intentional and affects most sniffs which use the FunctionRestrictions abstract as a parent class. Please see the explanation in point 2 of the PR #814. You may also want to read the "excluding a group" section in the (draft) version of a wiki page about the properties you can adjust in WPCS (issue #614).
Also, to avoid confusion: the sniff name has NOT been changed. The error code has.
The short of it is this:
All sniffs which use the above mentioned abstract provide grouped lists of functions to throw errors/warnings for.
The public exclude property allows you to disable the checks for one or more groups by setting this property in your ruleset.
Previously, you could use either the exclude property or the <exclude...> syntax to archive the same effect. However, there was no possible way to keep a group enabled, but just disable the error for one particular function within a group.
The above mentioned change fixed that.
Now, to exclude a group, you can only use the exclude property. The <exclude...> syntax for disabling a whole group no longer works.
However, you can now also exclude the message for one function while keeping the group enabled. For that you can use the <exclude... syntax with the new error codes.
The advantage of this is:

More fine-grained control over which groups and/or functions throw messages.
By leaving a group enabled and just disabling the message for one function, you still benefit from the check when new functions would be added to the group.

The errorcodes are build up of group name + function name. In your particular example the group name and the function name which triggered the error happen to be the same, which gives the appearance of the name being doubled.
However, there are two more functions being checked for in the same group.
So for the future (WPCS 0.11.0 / current develop), to disable the whole group, use the exclude property. To keep the group enabled, but just disable the message for the one function within the group, use the <exclude...> syntax.

@matt-bernhardt Just checking - did my answer help you or do you still have questions about this ?

@jrfnl Yes - thanks for clarifying the decision and why it was taken. This gives me enough information to adjust our projects accordingly. I may have further questions as I go through that process, but if necessary I'll open other tickets as they arise.
Many thanks!


Paint Editor - Design Pass 1

Tracking issue for the post-workshop design pass. If we are feeling good about all or a subset of the tools based on the workshop with kids, I will begin a pass on the paint editor UI for the GUI.
[To be fleshed out]

We should probably track this in https://github.com/LLK/scratch-paint/issues since I'm planning to mostly abandon this repo when we finish up with the prototype

Moved the issue.


Migration to Formly 6 - formlyHookConfig related issue

Hi,
We are currently migrating Formly version 5 to 6.
With V5, we were using FormlyHookConfig to capture any view changes using afterViewChecked property.
fields.hooks = {
afterViewChecked : (field: any) => {
After migration we are seeing an error on compilation. Below is the error.
**TS2322: Type '{ afterViewChecked: (field: any) => void; }' is not assignable to type 'FormlyHookConfig'. Object literal may only specify known properties, and 'afterViewChecked' does not exist in type 'FormlyHookConfig'.
afterViewChecked : (field: any) => {**
I have seen from the release notes of 5.7 version that afterViewChecked is deprecated. So how can we fix this error? Please suggest.
Thank you.

It has been removed since v6  the main reason was perf and we suggest avoid relying on it, but in case you still would like to use it in v6. Just create a custom wrapper which emit that event.


Bug: Opening Google Drive

Description
Upon Opening Drive, Files crashed.
Steps To Reproduce

Double-Click On Google Drive Icon (Z:\ in my case)
Crash

Steps to fix
Crash makes it impossible to access my Google Drive using Files, and thus makes it impossible for me to continue to use Files.
Files Version
Version: <IP_ADDRESS> OS Version: 10.0.25211.1010
Windows Version
Windows 11 Pro Insider Preview 22H2 25211.1010
Relevant Assets/Logs
debug.log

Is this an issue in the preview version? files.community/download/preview


⚠ Bofrost.lu has degraded performance

In 7256929, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1217 ms

Resolved: Bofrost.lu performance has improved in a247607 after 6 minutes.


scaleX and scaleY scale wrong axis when image is rotated at 90, -90, 270 or -270 angles

You can test this on the demo at http://fengyuanchen.github.io/cropper/
I understand the logic ... scale X and scale Y always relate to the original position of the image, but if the user rotates it to a 90 degree (or -90, or 270 or -270) the axis are now different.

Please close this ... this was for another github repo i have open in the next firefox tab :D
Your SimpleImage already rocks as is! Awesome work! Thanks a lot!


Can have text selected in multiple text fields

On a screen with multiple text fields (e.g. the Text fields demo in the Flutter gallery), select text in the first text field and then select text in the second one. Text in the first text field stays selected. Native apps don't seem to behave that way and deselect the text in the first text field.

@Hixie or @abarth did some work to make Focus unique.  somehow that's being missed here I guess.


Monitor stats not accurate

发现一个问题，在启用simple-monitor时，发现第一次调用rpc时，触发MonitorFilter的collect方法时，
Monitor monitor = monitorFactory.getMonitor(url);
if (monitor == null) {
return;
}
此时的monitor为空。后续接着调用rpc时，monitor又不为空。所以怀疑是MonitorFactory延迟加载导致。这里会导致一个问题，就是Collect的数量总是比实际少几条。

it's relevant to https://github.com/apache/incubator-dubbo/pull/777

We think monitoring data don't need to be accurate, though we will try to record every RPC statistics, it should not block the RPC process.

Feel free to reopen this issue if necessary.
Next time, please use english, also, you can subscribe to<EMAIL_ADDRESS>to discuss questions like this, just send your questions to this mail address, and we will guarantee a better and quicker reply there.
How to subscribe, see #1393.


More performance improvements

This PR aims to improve perforamance by targetting some 'easy wins'.

Mouse events are now processed once per frame. If vizia receives multiple CursorMoved events before a MainEventsCleared event then the hover system is now only called once in the MainEventsCleared. On my system I only get one CursorMoved followed by one MainEventsCleared but it was reported that sometimes this is not always the case (possibly due to a high poll rate gaming mouse).
All clones of the Tree have been removed. This required quite a bit of refactoring in some places, as well as the addition of a new kind of context, a ResourceContext, which is used for loading images with the set_image_loader method on Context.
There is now a bitflag, SystemFlags, used for skipping systems if they aren't required to run. Before we were skipping style, layout, and draw if not needed but now we skip the other systems which run in process_visual_updates.
Modifiers and the style system now selectively decide which systems need to be run. Before many of the modifiers would trigger relayout and redraw even if they didn't need to.

TODO

[x] Cache or find a way to remove the cloning + sorting of the tree to produce the draw tree.

Added a DrawIterator which iterates the tree in draw order, taking in to account the z-order while iterating. This removes the need for a z-order system as well as the need to clone and sort the tree by z-order. However, this now means that children can not be placed behind their parent using z-order.

[ ] Run systems incrementally on entities. For example, when an entity has its transform changed we currently iterate the whole tree when we only need to iterate down from the changed entity. This isn't trivial though, because if multiple entities change, and some of the those are descendants of others, we need a good way to figure out the minimal set of entities that need updating and then to just update those. This becomes more complicated when we have systems such as layout and style where changing an entity could affect its ancestors (and the ancestors descendants) as well.

I think I've gone about as far as I want to go on optimisation for now. Once 0.1 is released we can come back to optimisation work and focus on making everything as incremental as possible.

That fix seems to have done it.


[doc] fixes in validator doc from discord comments

@Alberto has already fixed this in:
https://github.com/MystenLabs/sui/pull/1994

@sblackshear and @asonnino see https://mysten-labs.slack.com/archives/C02JTC4QJBU/p1653059720308319


"Undefined method" for every iteration of setup_and_run

Each time the metrics are pulled and reported, the log file throws this out:
undefined method `[]' for nil:NilClass: mongodb_agent.rb:128:in `block in report_lock_counter_metric'
   mongodb_agent.rb:128:in `each'
   mongodb_agent.rb:128:in `inject'
   mongodb_agent.rb:128:in `report_lock_counter_metric'
   mongodb_agent.rb:71:in `poll_cycle'
   /var/lib/gems/2.1.0/gems/newrelic_plugin-1.3.1/lib/newrelic_plugin/agent.rb:139:in `run'
   /var/lib/gems/2.1.0/gems/newrelic_plugin-1.3.1/lib/newrelic_plugin/run.rb:143:in `block in run_agent_data_collection'
   /var/lib/gems/2.1.0/gems/newrelic_plugin-1.3.1/lib/newrelic_plugin/run.rb:141:in `each'
   /var/lib/gems/2.1.0/gems/newrelic_plugin-1.3.1/lib/newrelic_plugin/run.rb:141:in `run_agent_data_collection'
   /var/lib/gems/2.1.0/gems/newrelic_plugin-1.3.1/lib/newrelic_plugin/run.rb:127:in `loop_forever'
   /var/lib/gems/2.1.0/gems/newrelic_plugin-1.3.1/lib/newrelic_plugin/run.rb:15:in `setup_and_run'
   mongodb_agent.rb:150:in `<module:MongodbAgent>'
   mongodb_agent.rb:9:in `<main>'

I suspect it might be due to an update to the newrelic_plugin gem, and thus also related to issue #14

any news on this? basically all the calls that are 'report_lock_counter_metric' throw this error.

I'm getting this error because some of the metrics the agent expects are simply not reported in my instance - specifically, the cursor metrics. Adding the appropriate:
if stats['cursor']
...
end

makes the error go away for me.


⚠ Panoramas has degraded performance

In fe37dbc, Panoramas ($SITE_URL_PANORAMAS) experienced degraded performance:

HTTP code: 200
Response time: 9460 ms

Resolved: Panoramas performance has improved in 525ad13 after 8 minutes.


CRTLogSystem.cpp:60:13: too many initializers for ‘aws_logger_vtable’

Confirm by changing [ ] to [x] below:

[X ] I've gone though Developer Guide and API reference
[ X] I've searched for previous similar issues and didn't find any solution

Platform/OS/Hardware/Device
Ubuntu 18.04
Describe the question
When building core  from source, I get the following error. For building I use cmake .. -DBUILD_ONLY="core" -DCMAKE_BUILD_TYPE=Release (inside build directory) and the error occurs when then calling make -j16.
Logs/output
[ 90%] Building CXX object aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/ub_core.cpp.o
In file included from XXX/aws-sdk-cpp/build/aws-cpp-sdk-core/ub_core.cpp:28:0:
XXX/aws-sdk-cpp/aws-cpp-sdk-core/source/utils/logging/CRTLogSystem.cpp:60:13: error: too many initializers for ‘aws_logger_vtable’
             };
             ^
aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/build.make:866: recipe for target 'aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/ub_core.cpp.o' failed
make[2]: *** [aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/ub_core.cpp.o] Error 1
CMakeFiles/Makefile2:804: recipe for target 'aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/all' failed
make[1]: *** [aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/all] Error 2
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2

I tried using both cmake 3.18.2 and cmake 3.19.6. My OpenSSL version is 1.1.1l

Can you make sure that you are downloading the submodules before you try to build this sdk?
If you’ve already downloaded source code for SDK for C++, e.g. in directory aws-sdk-cpp, you can update the git submodule by:
cd aws-sdk-cpp
git checkout main
git pull origin main
git submodule update --init --recursive


Performance issue

Hi there.
Great lib, but it has really big problems with optimizations on long-range setup (e.g. -100 years, +100 years) because of generateWeekDays. It calls for each month and loads main thread. On a weak device it could take up to 5-10 seconds, which looks not very good. Seems like you need to review this approach.
You can find it in Example3 if you set up it like this:
 setup(currentMonth.minusYears(100), currentMonth.plusYears(100), daysOfWeek.first())

You are right. I have been considering a setupAsync() alternative to the setup() method that basically does the date generation in the background and updates the calendar when ready. Just haven't got the time to actually implement it. Hopefully this weekend or the next one 🤞.

It would really help. Moreover, I want to suggest to you, check generating months dynamically after user's swipe. It could also become a part of infinite calendar. But it seems that it need to rewrite lot of logic, so mb think about it for far future

First of all, thank you @kizitonwose for such a wonderful library. I appreciate the time that you spent on making so many things customizable. Unfortunately, I must say that the performance is a dealbraker for me. I have a BottomNavigationView in my app, and when I switch to the CalendarView, it takes a whole lot for the tab to be displayed. I did not dive into the code so I don't know what is the problem here, but I may assume that you're generating all the months and days that are not visible (I concluded this because the scrolling is super smooth). Is there a way for me to configure this? The scrolling isn't important for me. Lazy loading is definitely what you should do here.

I want to suggest to you, check generating months dynamically after user's swipe. It could also become a part of infinite calendar. But it seems that it need to rewrite lot of logic, so mb think about it for far future

Yep, this is on my TODO, just need to find the time.

Is there a way for me to configure this? The scrolling isn't important for me. Lazy loading is definitely what you should do here.

Not at the moment but the async setup should come this weekend(hopefully), and that should be sufficient until lazy loading is possible.

Would be really great to get this feature

Happy to announce that setupAsync is now available in 0.4.1.
There are also other async methods for related tasks like updating the start and end months, etc.


why is tensorflow.map_fn slow, what is wrong with following code?

I am trying to use tensorflow map_fn to do parallel computation. However it seems to me that the performance gain is not significant.
Here are example code running Python 3.6.5, Tensorflow version 1.12.0 on Ubuntu 14.04 LTS, 28 duo cores (Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz) = 56 processors
These same codes running on Amazon AWS SagerMaker ml-p3-xlarge even took longer time, 227 seconds.
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
b'v1.12.0-0-ga6d8ffa' 1.12.0
import tensorflow as tf
import time
version 1
tic = time.time()
elems = np.array(range(1,1000000), dtype=np.float64)
output = tf.map_fn(lambda x: x**6 , elems, dtype=tf.float64,  parallel_iterations=56)
sess = tf.Session()
res = sess.run(output)
toc = time.time() - tic
print("elapsed=", toc)  # 29.47 (seconds)
version 2
tic = time.time()
elems = np.array(range(1,1000000), dtype=np.float64)
output = tf.map_fn(lambda x: x**6 , elems, dtype=tf.float64,   parallel_iterations=56)
n_cpus=28
with  tf.Session(
config=tf.ConfigProto(log_device_placement=True,
device_count={ "CPU": n_cpus },
inter_op_parallelism_threads=n_cpus,
intra_op_parallelism_threads=1,
)) as sess:
res = sess.run(output)
toc = time.time() - tic
print("elapsed=", toc)  # 29.26 (seconds)
version 3
tic = time.time()
elems = np.array(range(1,1000000), dtype=np.float64)
x6 = [ x**6 for x in elems]
toc = time.time() - tic
print("elapsed time=", toc) # 0.5 seconds
What is problem with the above codes? without map_fn, sequential execution version 3 only 0.5 (seconds).

I'm having a similar issue where map_fn is the bottleneck!..

Yes. I am experiencing almost the same issue on my side.  Is this happen because tf.map_fn or tf.while_loop is not designed for parallelization?

I am leaving an additional comment because the former post is just grunting not suitable for asking for help. In my case, I am using multiple tf.map_fn methods within call() method of my custom layer implementation using "tf.keras.layers.Layer" class. I am experiencing very slow graph construction and training time too. Since graph construction may have its own overhead, I can deal with slow graph construction. However, my custom layer is optimized for some computation. It has to be fast if tf.map_fn is really doing parallelization inside. I even optimize the code so the GPU has almost over 90 percent of utilization, but it gives only a slight speed up. The same algorithm on the custom ASIC is verified working and it's fast. So it's not related to the algorithmic problem. Could you give me any clue on this? Maybe using tf.map_fn in the custom layer is a bad idea?

@Han-sok - Can you please provide full information about your issue according to the issue template?

tf.map_fn is really slow.

I experience the same problem with tf.map_fn being slow. Is there any way to accelerate this function?

I experience the same problem with tf.map_fn being slow. Is there any way to accelerate this function?

For me, the best solution is to avoid using it. Use vectorization as many as possible.

For my problem i have to use the tf.boolean_mask-function, which can produce vectors of different lenghts. This output is further reduced to a single vector.
Thats the reason i have to use the tf.map_fn since it is possible to compute  the tf.boolean_mask for a single element, then reduce it to a scalar and when the computation is done it gets combined to a vector.

I experience the same problem with tf.map_fn being slow. Is there any way to accelerate this function?

For me, the best solution is to avoid using it. Use vectorization as many as possible.

I implemented a customized loss similar to yours, and tf.map_fn is running very slow as well. is it possible to share how you vectorized your computation? thanks a lot.

I am also facing this issue. I wrote a custom layer to process an color image. Input tensor for 16 batch size is(16, 128, 128, 32). Using tf.map_fn is taking indefinite time.

Please suggest a workaround

mark, I also meet this problem

@alextp - do you have any thoughts on a workaround for tf.map_fn performance?

Does tf.vectorized_map work here instead of map_fn?

map_fn will create a branch for every element, your code creates 1000000 graph branches which will cause the performance issue. You could try vectorized_map or custom op kernel.

@fsx950223 I wanted to change the data type as well inside the map_fn. This change isn't allowed in vectorized_map, hence the choice

Whoever is coming here, here are some of the pitfalls to be aware of while using vectorized_map: https://www.tensorflow.org/api_docs/python/tf/vectorized_map

I had the same issues. Vectorized map works faster but still subpar compared with numpy alternatives. In the same dataset size  numpy.apply_along_axis takes 50microseconds (over a numpy array) while tf.vectorized_map takes 25ms over a tensor (numpy is 500x faster), if comparing with tf.map_fn over a tensor, things go even worse, since it takes 300ms (numpy is 6000x faster).

I have same issue as well, trying to map a floor approximation function  to image pixels. Its horribly slow. The Only reason for using TF is faster parallel computation, its strange that Map_fn doesn't do that.

@nikhil1008 It looks like your code is also measuring the "graph construction cost" which is likely unintentional.
Here is my measurement, using TF 2.x:
def diff_floor(x):
xp = 10
a = tf.range(-xp, xp + 1, 1, 'float32')
b = tf.tile(x, (2 * xp + 1,))
b = b - a
b = 5000 * b
b = 1 + tf.math.exp(-b)
b = 1.0 / b
b = tf.reduce_sum(b) - xp -1
return b
@tf.function(experimental_compile=True)
def f(a):
return tf.map_fn(diff_floor, tf.reshape(a, (-1, 1)), parallel_iterations=8)
f(tf.constant(np.random.rand(224, 224, 3), dtype=tf.float32))  # warmup
%timeit f(tf.constant(np.random.rand(224, 224, 3), dtype=tf.float32))
-> 10 loops, best of 3: 21.8 ms per loop
Replacing the map_fn with a vectorized_map gives:
->100 loops, best of 3: 5.02 ms per loop

@tejaslodaya Does tf.RaggedTensor work for you case instead of string encoding ? Another common approach is to pad the ragged data and then loop over it carefully to avoid the paddings.
Also vectorization support for control flow is better now and it can handle tf.while_loop and tf.cond. i.e. the fn passed to tf.vectorized_map can have loops and branches.
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/parallel_for/control_flow_ops_test.py#L1429
for example that uses an internal pfor API for testing.

@igormorgado A common pitfall with vectorized_map usage is that the call includes the cost of code vectorization in addition to code execution. So it should typically be placed within a tf.function
Example:
def f(x):
return tf.vectorized_map(lambda z: tf.math.log(z), x)
inp = tf.random.uniform([200, 200])
f(inp)  # warmup
Slow call since it vectorizes code in each call!
%timeit f(inp)
-> 10 loops, best of 3: 28.9 ms per loop
compiled_f = tf.function(f)
compiled_f(inp)  # warmup
Vectorization process is done once
%timeit compiled_f(inp)
-> 1000 loops, best of 3: 328 µs per loop
NumPy:
def np_f(x):
return np.apply_along_axis(lambda z: np.log(z), 0, arr=x)
np_inp = inp.numpy()
np_f(np_inp)  # warmup
%timeit np_f(np_inp)
-> 1000 loops, best of 3: 961 µs per loop
Manually vectorized NumPy
np.log(np_inp)  # warmup
%timeit(np.log(np_inp))
-> 1000 loops, best of 3: 249 µs per loop

@vidursatija
Here are some benchmarks:
def f(elems):
return tf.map_fn(lambda x: x**6 , elems, dtype=tf.float64, parallel_iterations=56)
elems = tf.range(1,1000, dtype=np.float64)
f(elems)
%timeit f(elems)
-> 10 loops, best of 3: 162 ms per loop
compiled_f = tf.function(f)
compiled_f(elems)
%timeit compiled_f(elems)
->100 loops, best of 3: 12.8 ms per loop
@tf.function
def vectorized_f(elems):
return tf.vectorized_map(lambda x: x**6 , elems)
vectorized_f(elems)
%timeit vectorized_f(elems)
->1000 loops, best of 3: 297 µs per loop

@agarwal-ashish thanks for the code examples, would you mind formatting it to make it more readable?

tf.map_fn was slow for me as well, which is why I was switching to tf.vectorized_map, however some TensorFlow Ops seem to be not supported / vectorized with tf.vectorized_map.
For example:
dataset = tf.data.Dataset.from_tensor_slices([b'\x00\x00\x00\x00']*512)

def f(x):
    x1 = tf.strings.substr(x, 0, 2)
    x2 = tf.strings.substr(x, 2, 2)

    return tf.io.decode_raw(x1, tf.uint8), tf.io.decode_raw(x2, tf.uint8)

dataset = dataset.batch(32).map(lambda x: tf.vectorized_map(f, x))

WARNING:tensorflow:Using a while_loop for converting Substr
WARNING:tensorflow:Using a while_loop for converting DecodeRaw
WARNING:tensorflow:Using a while_loop for converting Substr
WARNING:tensorflow:Using a while_loop for converting DecodeRaw

Which results in execution times of vectorized_map being similar to map_fn.
Any ideas to solve this?

vectorized_map does require registering a "converter" for each op that defines how that op is vectorized. Please see comments for RegisterPFor that describes how these can be defined, in case you want to contribute these.
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/parallel_for/pfor.py#L844

@jsimsa for dataset map calls which has its own vectorization support

Thanks @agarwal-ashish, but I fear that registering the a converter for the above ops is beyond my skill set, as I just started with TensorFlow.

@agarwal-ashish in my case, padding it with 0s and carefully parsing them out using raggedtensors would mean padding all the features till the max of length of all rows.
For anyone having the same problem, we overcame this requirement by creating an explicit training-prediction skew with transformed features used for training (till model function) and applying transformations only for prediction.
Considering our models are served on tensorflow serving, we have single, one-off predictions rather than a bulk call. Might not work if your models are offline, batch requests.

Hi @minhg! We see that you are using old version of Tensorflow which is officially considered as end of life, We recommend that you upgrade to 2.6 version and let us know if the issue still persists in newer versions .Please open a new issue in case you face any errors, we will get you the right help .Thanks!


Improve pod_spec and pod_patch documentation; include specific usage examples

There are some gotchas in using pod_spec and pod_patch (discussed in #3319; added in #3427) that should be clarified, and having specific examples of the most common use cases (which seems to mostly be adding volumes) would be helpful.
In particular, the most obvious way of adding a volume (using an add operation with an array of volumes at that location) doesn't actually work, unless you know pachyderm already uses some  volumes and append to the list instead of adding it. More details here: https://github.com/pachyderm/pachyderm/issues/3319#issuecomment-471478256
Specific example of the problem from #users:
Matt Goldberg

I have another question about using pod_patch. After I run create-pipeline, my pipeline is not being created and I see errors in the pachd logs.
This is the pod_patch I’m trying to apply:
[
  {
    "op": "add",
    "path": "/volumes",
    "value": [
      {
        "name": "config",
        "emptyDir": {}
      }
    ]
  },
  {
    "op": "add",
    "path": "/affinity",
    "value": {
      "nodeAffinity": {
        "requiredDuringSchedulingIgnoredDuringExecution": {
          "nodeSelectorTerms": [
            {
              "matchExpressions": [
                {
                  "key": "kops.k8s.io/instancegroup",
                  "operator": "In",
                  "values": [
                    "nodes"
                  ]
                }
              ]
            }
          ]
        }
      }
    }
  },
  {
    "op": "add",
    "path": "/dnsConfig",
    "value": {
      "options": [
        {
          "name": "ndots",
          "value": "2"
        }
      ]
    }
  }
]

These are the logs I’m seeing in pachd:
2019-03-09T07:51:48Z INFO moving pipeline test-secrets to PIPELINE_STARTING
2019-03-09T07:51:48Z ERROR master: error running the master process: ReplicationController "pipeline-test-secrets-v1" is invalid: [spec.template.spec.containers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.containers[0].volumeMounts[1].name: Not found: "pachyderm-worker", spec.template.spec.containers[0].volumeMounts[2].name: Not found: "pachyderm-storage-secret", spec.template.spec.containers[0].volumeMounts[3].name: Not found: "docker", spec.template.spec.containers[1].volumeMounts[0].name: Not found: "pach-dir-volume", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "pachyderm-storage-secret", spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.initContainers[0].volumeMounts[1].name: Not found: "pachyderm-worker"]; retrying in 20.004449691s
2019-03-09T07:51:52Z INFO Launching PPS master process
2019-03-09T07:51:52Z INFO pfs.API.GetFile {"request":{"file":{"commit":{"repo":{"name":"__spec__"},"id":"655e62c792ec408bbd5e4375e3a5fc3a"},"path":"spec"}}} []
2019-03-09T07:51:52Z INFO authclient.API.WhoAmI {"request":{}} []
2019-03-09T07:51:52Z INFO authclient.API.WhoAmI {"request":{}} []
2019-03-09T07:51:52Z INFO authclient.API.WhoAmI {"request":{}} []
2019-03-09T07:51:52Z INFO pfs.BlockAPI.Obj.GetObjects {"request":{"objects":[{"hash":"84e57bf24a8d24b81d2dced5a9c9a33604288718b9440ed61cee82345d55d5504672eeebac565ae37e4826d9650cf7bcfd8c728f3cde328de22b4d35c5fd53db"}],"total_size":743}} []
2019-03-09T07:51:52Z INFO pfs.BlockAPI.Obj.InspectObject {"request":{"hash":"84e57bf24a8d24b81d2dced5a9c9a33604288718b9440ed61cee82345d55d5504672eeebac565ae37e4826d9650cf7bcfd8c728f3cde328de22b4d35c5fd53db"}} []
2019-03-09T07:51:52Z INFO pfs.BlockAPI.Obj.InspectObject {"duration":0.000018182,"request":{"hash":"84e57bf24a8d24b81d2dced5a9c9a33604288718b9440ed61cee82345d55d5504672eeebac565ae37e4826d9650cf7bcfd8c728f3cde328de22b4d35c5fd53db"},"response":{"object":{"hash":"84e57bf24a8d24b81d2dced5a9c9a33604288718b9440ed61cee82345d55d5504672eeebac565ae37e4826d9650cf7bcfd8c728f3cde328de22b4d35c5fd53db"},"block_ref":{"block":{"hash":"6d0ddfb72c45410b8ceecb3b07b6b41e"},"range":{"upper":743}}}} []
2019-03-09T07:51:52Z INFO pfs.BlockAPI.Obj.GetObjects {"duration":0.000462571,"request":{"objects":[{"hash":"84e57bf24a8d24b81d2dced5a9c9a33604288718b9440ed61cee82345d55d5504672eeebac565ae37e4826d9650cf7bcfd8c728f3cde328de22b4d35c5fd53db"}],"total_size":743}} []
2019-03-09T07:51:52Z INFO pfs.API.GetFile {"duration":0.00602323,"request":{"file":{"commit":{"repo":{"name":"__spec__"},"id":"655e62c792ec408bbd5e4375e3a5fc3a"},"path":"spec"}}} []
2019-03-09T07:51:52Z INFO PPS master: creating/updating workers for pipeline test-secrets
2019-03-09T07:51:52Z INFO enterprise.API.GetState {"request":{}} []
2019-03-09T07:51:52Z INFO enterprise.API.GetState {"duration":0.000002547,"request":{},"response":{}} []
2019-03-09T07:51:52Z ERROR error creating workers for pipeline test-secrets: ReplicationController "pipeline-test-secrets-v1" is invalid: [spec.template.spec.containers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.containers[0].volumeMounts[1].name: Not found: "pachyderm-worker", spec.template.spec.containers[0].volumeMounts[2].name: Not found: "pachyderm-storage-secret", spec.template.spec.containers[0].volumeMounts[3].name: Not found: "docker", spec.template.spec.containers[1].volumeMounts[0].name: Not found: "pach-dir-volume", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "pachyderm-storage-secret", spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.initContainers[0].volumeMounts[1].name: Not found: "pachyderm-worker"]; retrying in 623.768688ms
2019-03-09T07:51:52Z INFO enterprise.API.GetState {"request":{}} []
2019-03-09T07:51:52Z INFO enterprise.API.GetState {"duration":0.000002166,"request":{},"response":{}} []
2019-03-09T07:51:52Z ERROR error creating workers for pipeline test-secrets: ReplicationController "pipeline-test-secrets-v1" is invalid: [spec.template.spec.containers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.containers[0].volumeMounts[1].name: Not found: "pachyderm-worker", spec.template.spec.containers[0].volumeMounts[2].name: Not found: "pachyderm-storage-secret", spec.template.spec.containers[0].volumeMounts[3].name: Not found: "docker", spec.template.spec.containers[1].volumeMounts[0].name: Not found: "pach-dir-volume", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "pachyderm-storage-secret", spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.initContainers[0].volumeMounts[1].name: Not found: "pachyderm-worker"]; retrying in 404.080357ms
2019-03-09T07:51:53Z INFO enterprise.API.GetState {"request":{}} []
2019-03-09T07:51:53Z INFO enterprise.API.GetState {"duration":0.000002691,"request":{},"response":{}} []
2019-03-09T07:51:53Z INFO moving pipeline test-secrets to PIPELINE_STARTING
2019-03-09T07:51:53Z ERROR master: error running the master process: ReplicationController "pipeline-test-secrets-v1" is invalid: [spec.template.spec.containers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.containers[0].volumeMounts[1].name: Not found: "pachyderm-worker", spec.template.spec.containers[0].volumeMounts[2].name: Not found: "pachyderm-storage-secret", spec.template.spec.containers[0].volumeMounts[3].name: Not found: "docker", spec.template.spec.containers[1].volumeMounts[0].name: Not found: "pach-dir-volume", spec.template.spec.containers[1].volumeMounts[1].name: Not found: "pachyderm-storage-secret", spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "pach-bin", spec.template.spec.initContainers[0].volumeMounts[1].name: Not found: "pachyderm-worker"]; retrying in 8.617200437s

Gabriel Grant

@goldberg i believe the problem you're running into is that pachyderm already uses several volumes internally, but your patch is replacing the list of volumes with the one you've defined. instead you'll need to append to that list, so your patch's path should be "/volumes/-" instead of "/volumes"

see full thread: https://pachyderm-users.slack.com/archives/C3UN0S0Q0/p1552117931505200

More issues around adding k8s directives via 'pod_spec' & pod_patch.  https://pachyderm-users.slack.com/archives/C3UN0S0Q0/p1558352661036200
{
  "pipeline": {
    "name": "mypipeline"
  },
  "transform": {
    "cmd": [
      "R",
      "-e yyy::generate_data()"
    ],
    "image": "xxx/yyy:v0.1.6-pachyderm"
  },
  "pod_spec": "{\n\t\"containers\": [\n\t\t{\n\t\t\t\"env\": [\n\t\t\t\t{\n\t\t\t\t\t\"name\": \"S3_SECRET\", \n\t\t\t\t\t\"valueFrom\": {\n\t\t\t\t\t\t\"secretKeyRef\": {\n\t\t\t\t\t\t\t\"name\": \"xxx-basics\", \n\t\t\t\t\t\t\t\"key\": \"API_SECRET\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}, \n\t\t\t\t{\n\t\t\t\t\t\"name\": \"APP_DEBUG\", \n\t\t\t\t\t\"valueFrom\": {\n\t\t\t\t\t\t\"configMapKeyRef\": {\n\t\t\t\t\t\t\t\"name\": \"xxx-basics\", \n\t\t\t\t\t\t\t\"key\": \"APP_DEBUG\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t]\n\t\t}\n\t]\n}",
  "input": {
    "pfs": {
      "repo": "images",
      "glob": "/*"
    }
  }
}

When we move to CRDs, pod spec/patch go away. Depending on the estimates delivery date of CRD pipelines by @msteffen (which seems like it's not in the super near future), it may or may not makes sense to even bother with this. Marking at low priority


Reflow should remove trailing spaces except the last line

Hi there,
When I use the reflow feature, I also end up with two spaces between the ending word of a line and the starting word of next line. E.g.,
test<space>
test

will become
test<space><space>test

This is very inconvenient to me. I really love this feature (I use this feature frequently in TextWrangler). So I expect that it can become
test<space>test

In addition, we should leave the spaces for the last line of this block. E.g.,
test<space>
test<space><space><space>

should become
test<space>test<space><space><space>

So I made this change and hope it can be added to the package.
Sorry for repeatedly trying because this is my first time to submit a pull request.
Cheers,
Hengfeng Li

Hi @HengfengLi, thanks for sending this PR!
The Atom team is currently migrating many core Atom packages back into the main Atom repository.  autoflow is now being moved as a part of this migration!  Since this PR has conflicts with master we can't currently merge it so I'm closing it for now with the hope that you'll be willing to reopen it on atom/atom at some point in the future.
Thanks again for contributing to Atom! :sparkles: :heart: :atom:


Add Nuget dependency and remove deprecated methods

Thanks for sending this. Now I just need to go through the actual tutorials and copy the changes over.

Great! 😄
I have some problems running OpenTKTutorial8-2, nothing shows in the windows.


Setting JsonConvert.DefaultSettings changes the behavior of ToObject

I have a scenario where I'm accessing the value of a JProperty by calling JProperty.ToObject<string>(). This has worked for me before so I always did it this way, though I could be a fool. Then I discovered that when some other code was introduced that changed the default serializer settings, it would fail. I was able to workaround it by calling ToObject<string>() on the Value instance instead.
I'm not sure if this behavior is a bug in the library, or if I've always been doing it wrong and just got lucky up until now.
Here's a sample function that reproduces it:
    [Test]
    public void Weird()
    {
        var json = "[{\"something\":[{\"property\":\"value\"}]}]";

        // Try to obtain the "value" ... this works
        {
            var array = JArray.Parse(json);
            var first = (JObject)array.First;
            var value = (first["something"] as JArray)
                .Select(x => x as JObject)
                .Select(x => x.Property("property").ToObject<string>())
                .First();
            Assert.AreEqual("value", value);
        }

        // Set default serializer settings; the options don't seem
        // to matter, just that it is set
        JsonConvert.DefaultSettings = () =>
        {
            return new JsonSerializerSettings();
        };

        // Now try to obtain the "value" in the same way again ...
        try
        {
            var array = JArray.Parse(json);
            var first = (JObject)array.First;
            var value = (first["something"] as JArray)
                .Select(x => x as JObject)
                .Select(x => x.Property("property").ToObject<string>())
                .First();
        }
        catch
        {
            // Whoops, failed with: Error reading string. Unexpected token: PropertyName. Path '', line 1, position 27.
        }

        // This method, however, still works
        {
            var array = JArray.Parse(json);
            var first = (JObject)array.First;
            var value = (first["something"] as JArray)
                .Select(x => x as JObject)
                .Select(x => x.Property("property").Value.ToObject<string>())
                .First();
            Assert.AreEqual("value", value);
        }

        // This works too
        {
            var array = JArray.Parse(json);
            var first = (JObject)array.First;
            var value = (first["something"] as JArray)
                .Select(x => x as JObject)
                .Select(x => (string)x.Property("property").Value)
                .First();
            Assert.AreEqual("value", value);
        }
    }

Here's the full stacktrace:
Newtonsoft.Json.JsonReaderException: Error reading string. Unexpected token: PropertyName. Path '', line 1, position 27.
   at Newtonsoft.Json.JsonReader.ReadAsStringInternal()
   at Newtonsoft.Json.Linq.JTokenReader.ReadAsString()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject[T]()
   at Shipvine.Badger.Engine.Test.Integration.SmartyStreetsStateLookupTest.<>c.<Weird>b__2_4(JObject x) in C:\Users\npiaseck\Desktop\Working Directories\Badger\Src\Shipvine.Badger.Engine.Test\Integration\Addressing\State Lookup\SmartyStreetsStateLookupTest.cs:line 55
   at System.Linq.Enumerable.<>c__DisplayClass2`3.<CombineSelectors>b__3(TSource x)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Shipvine.Badger.Engine.Test.Integration.SmartyStreetsStateLookupTest.Weird() in C:\Users\npiaseck\Desktop\Working Directories\Badger\Src\Shipvine.Badger.Engine.Test\Integration\Addressing\State Lookup\SmartyStreetsStateLookupTest.cs:line 53

That's by design.

@JamesNK, Can you please clarify idea behind this intention?
I faced exactly the same situation and I don't understand why JToken.ToObject method has 2 branches:
JsonConvert.DefaultSettings == null and the rest.
I'm confused that if I just want to override some settings (let's say DateTimeFormatString), I can no longer use default logic (used in "JsonConvert.DefaultSettings == null" if branch)

The branch for when DefaultSettings is null is optimized for performance and matches Json.NET's default serialization rules. When DefaultSettings have been changed then it is possible the fast version no longer matches the serializer with the new settings.

Understood, but how about making it obvious that it's not safe to call JProperty.ToObject and call JProperty.Value.ToObject to avoid break-downs.
For example making JToken.ToObject virtual and overriding it in JProperty class:

with throwing NotSupportedException (I understand that can break backward compatibility and it breaks Liskov Substitution Principle, but if it's used like that risk is already in place)
mark JProperty.ToObject override with ObsoleteAttribute to draw attention

Ideal case would be that JProperty is not inherited from JToken at all and does not have this method IMHO.


Enforce that name passed to AmplifyError has desired suffix

From https://github.com/aws-amplify/amplify-backend/pull/1883#issuecomment-2297768931 .
If AmplifyUserError or AmplifyError is created we should suffix name with Error or Fault. Failure to do so might cause wrong classification.
This should be enforceable by custom linter rule.

https://github.com/aws-amplify/amplify-backend/pull/1889

Done!


Unable to track ".changed?" for arrays

What follows is just an example to illustrate my current situation. All requests shown as examples were actually made.
Creating users with tags worked fine:
client.users.create(email<EMAIL_ADDRESS>name: "John Example", verified: true, tags: ["first_tag", "second_tag"])

Logger's output:
I, [2015-05-21 14:29:40#44533]  INFO -- : post https://xxx.zendesk.com/api/v2/users
D, [2015-05-21 14:29:40#44533] DEBUG -- :   Accept: "application/json"
    Accept-Encoding: "gzip;q=1.0,deflate;q=0.6,identity;q=0.3"
    User-Agent: "ZendeskAPI API 1.7.4"
{"user"=>#<Hashie::Mash<EMAIL_ADDRESS>name="John Example" tags=["first_tag", "second_tag"] verified=true>}
I, [2015-05-21 14:29:41#44533]  INFO -- : Status 201

A end-user called John Example was successfully created in my Sandbox with both tags. Later on, I tried to update John. Every field was updated, except for the tags.
john = client.users.search(query: "john@example.com").first
john.name += " updated"
john.tags.push("third_tag")
john.alias = "jexmpl"
john.save

Checking again on Zendesk John now is John Example updated, it has the "jexmpl" alias but the tags remain the first two that were set upon creation, and third_tag was not added.
Checking Logger's outputs, I've noticed that the client's PUT request only sends the updated (a.k.a. changed) fields to Zendesk API.
I, [2015-05-21 14:35:26#46888]  INFO -- : put https://xxx.zendesk.com/api/v2/users/984876129.json
D, [2015-05-21 14:35:26#46888] DEBUG -- :   Accept: "application/json"
  Accept-Encoding: "gzip;q=1.0,deflate;q=0.6,identity;q=0.3"
  User-Agent: "ZendeskAPI API 1.7.4"
{"user"=>#<Hashie::Mash alias="jexmpl" name="John Example updated">}
I, [2015-05-21 14:35:27#46888]  INFO -- : Status 200
D, [2015-05-21 14:35:27#46888] DEBUG -- :   server: "nginx"

Notice that only alias and name where send, while tags was left aside. The client didn't understand that tags attribute had changes, so it was left out of the PUT request.
Digging into the Gem's code, I've found out that the problem lies within the changes method in lib/zendesk_api/track_chages.rb. The tags attribute validates at the first check on line 67 (v.is_a?(Array)) but fails when checked upon v.any? {|val| val.respond_to?(:changed?) && val.changed?}.
That was the farthest I went. Unfortunately I don't actually know how to fix it, otherwise I'd be sending a pull request instead of opening an issue.

Additional information: on another test, if I set tags as a comma-separated string, the client track the change and sends the attribute in the PUT request. While this might work as a workaround, it would be better to work with tags as an array.
Let me know if I can help somehow!

Yes, unfortunately this has been a caveat for a while. I still haven't figured out a good way to fix it without injecting into the broader array class. Since, this has really only been a problem with a select few fields perhaps a small shim would work.

Ops, my bad! Didn't see that on the caveats section. Thanks for the reply @steved .
Even tough it only affects only a few things, I believe tags to be an important one. I'll be looking for a solution as well and will PR if I find any suitable solution.


Update userData.ts

add discuss, views, and solution

@alfaarghya so,its not possible to fetch all discussion for each user?

@mahdiehmortazavi , I don't know if the queries are available or not, but if u want to get the data u need to add graphQl queries, because in my code I don't have those queries.
So first try to add those queries in GQLQueries folder, and check if it's possible or not And don't forget to add types

@alfaarghya Thank you for your reply, but since I am not a professional in writing queries with graphql, could you please give me more guidance?

Yeah Sure!! But I am not professional also 😅
Try to visit src/GQLQueries! Here are all of the queries that me and contributors found. Please try to understand them.
You need to change src/GQLQueries/userProfile.ts.

Tnx a lot


[FEATURE REQ] Make message converter of Azure Spring Cloud Event Hubs configurable

To expose message deserializers/serializers of Azure Spring Cloud Event Hubs libraries, we should modify the current behavior of hardcoding an EventHubMessageConverter obejct in AbstractEventHubTemplate and make it configurable.
Ref: Auto-configuration Azure Spring Cloud Service Bus

Done in #26200


Bug: Patches deselected if "Both XL" patch is selected

Describe the bug
If selecting "Both XL" under "Drive patches" and options "UsbdSec" and "Key Fix" under "Other patches" then trying to Create XeBuild the "Other patches" are deselected, but only if "Both XL" has been selected.
This behavior makes it uncertain if other patches has been applied or not.
To Reproduce

Load source
Select "Both XL" and both patches under "Other patches"
Create XeBuild
Click OK on prompt asking about "Both XL"
Other patches has been deselected according to console

Expected Behavior
Other patches should not be automatically deselected after build if "Both XL" has been applied.
Screenshots
Before creating XeBuild:

After creation:

System version
Windows 11
J-Runner version
3.3.0 r3
Additional context
Console log
Initializing updflash-17599.bin, please wait...
Glitch2 selected
RGH3 selected
Nand Initialization Finished
Both XL selected
UsbdSec selected
Corona Key Fix selected
XeBuild Initialized
Kernel Selected: 17559
KV Info saved to file
---------------------------------------------------------------
     xeBuild v1.21.810
---------------------------------------------------------------
base path changed to C:\Users\Rob\Downloads\J-Runner-with-Extras_2\xeBuild
---- { Image Build Mode } ----
building glitch2 image

done! fsroot found ok!
---------------------------------------------------------------
C:\Users\Rob\Downloads\J-Runner-with-Extras_2\#\updflash.bin image built, info:
---------------------------------------------------------------
Kernel    : 2.0.17559.0
Console   : Corona
NAND size : 16MiB
Build     : Glitch (v2)
Xell      : power on console with console eject button
Serial    : #
ConsoleId : #
MoboSerial: #
Mfg Date  : 11/21/2013
CPU Key   : #
1BL Key   : #
DVD Key   : #
CF LDV    : 3
KV type   : type2 (hashed - unchecked, master key not available)
---------------------------------------------------------------
    xeBuild Finished. Have a nice day.
---------------------------------------------------------------
Converting Image to RGH3...
RGH3 Conversion Finished!
Saved to C:\Users\Rob\Downloads\J-Runner-with-Extras_2\#
Image is Ready
Initializing updflash.bin, please wait...
Nand includes Both XL patches
UsbdSec deselected
Corona Key Fix deselected
Nand Initialization Finished

Parameter name: startIndex
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at System.BitConverter.ToUInt32(Byte[] value, Int32 startIndex)
at JRunner.Nand.PatchParser.getPatches(Byte[] data) in D:\Scratch\J-Runner-with-Extras\J-Runner\Nand\PatchParser.cs:line 53
at JRunner.Nand.PatchParser.parseAll(Int32 i) in D:\Scratch\J-Runner-with-Extras\J-Runner\Nand\PatchParser.cs:line 158
at JRunner.MainForm.nandinit(Boolean nomove, Boolean dontUpdateHackType) in D:\Scratch\J-Runner-with-Extras\J-Runner\MainForm.cs:line 2088
That's why. Fixing.

Fixed.
Kind Regards,
Josh


added math operations using bash dc , an older unix language

What type of PR is this? (check all applicable)

[ ] ♻ Refactor
[x] ✨ Feature
[ ] 🐛 Bug Fix
[ ] 👷 Optimization
[ ] 📝 Documentation Update
[ ] 🚩 Other

Description
Related Tickets & Documents
Added to documentation?

[ ] 📜 readme
[ ] 🙅 no documentation needed

[optional] What gif best describes this PR or how it makes you feel?

First commit and first PR.


Add clip command

Make a command that take a clip (with API) & return in the chat the link of the clip

No need for the command, the /clip is already running


fix(deps): update all non-major

This PR contains the following updates:

Package
Type
Update
Change

axum
dependencies
patch
0.7.3 -> 0.7.4

base64
dependencies
patch
0.21.5 -> 0.21.7

Release Notes

tokio-rs/axum (axum)
v0.7.4: axum - v0.7.4
Compare Source

fixed: Fix performance regression present since axum 0.7.0 (#2483)
fixed: Improve debug_handler on tuple response types (#2201)
added: Add must_use attribute to Serve and WithGracefulShutdown (#2484)
added: Re-export axum_core::body::BodyDataStream from axum

marshallpierce/rust-base64 (base64)
v0.21.7
Compare Source

Support getting an alphabet's contents as a str via Alphabet::as_str()

v0.21.6
Compare Source

Improved introductory documentation and example

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.2.11 :tada:
The release is available on:

GitHub release
v1.2.11

Your semantic-release bot :package::rocket:


fix: Update Cloud Pak for Data to 4.6.4

closes: #227
Description of changes:

Update CP4D version to 4.6.4
Parameterize location of common services

Output of argocd app list command or screenshot of the ArgoCD Application synchronization window showing successful application of changes in this branch.
argocd app list
NAME                 CLUSTER                         NAMESPACE         PROJECT               STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                    PATH                                  TARGET
argo-app             https://kubernetes.default.svc  openshift-gitops  argocd-control-plane  Synced  Healthy  Auto-Prune  <none>      https://github.com/IBM/cloudpak-gitops  config/argocd                         227-cp4d-464
cp-shared-app        https://kubernetes.default.svc  ibm-cloudpaks     default               Synced  Healthy  Auto-Prune  <none>      https://github.com/IBM/cloudpak-gitops  config/argocd-cloudpaks/cp-shared     227-cp4d-464
cp-shared-operators  https://kubernetes.default.svc  ibm-cloudpaks     default               Synced  Healthy  Auto-Prune  <none>      https://github.com/IBM/cloudpak-gitops  config/cloudpaks/cp-shared/operators  227-cp4d-464
cp4d-app             https://kubernetes.default.svc  openshift-gitops  default               Synced  Healthy  Auto-Prune  <none>      https://github.com/IBM/cloudpak-gitops  config/argocd-cloudpaks/cp4d          227-cp4d-464
cp4d-platform        https://kubernetes.default.svc  cp4d              default               Synced  Healthy  Auto-Prune  <none>      https://github.com/IBM/cloudpak-gitops  config/cloudpaks/cp4d                 227-cp4d-464

:tada: This PR is included in version 0.20.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


replaced 1:length(x) by seq_len statements

Hi
I just replaced deprecated 1:length(x) R statements by seq_len(length(x)) statements as suggested by
python setup.py test --installation

@chrisdane any news here?

seq_along is also the fastest option
> mb <- microbenchmark(times=nbenchmark,  `1:length(x)`=1:length(x),
        seq_len=seq_len(length(x)), seq_len2=seq_len(len) ,  seq_along=seq_along(x))
> print(mb)
Unit: nanoseconds
        expr min    lq   mean median    uq  max neval
 1:length(x) 219 272.5 381.42  337.0 428.0 1688   100
     seq_len 194 244.0 343.70  312.5 358.5 2678   100
    seq_len2 122 182.5 244.45  203.0 279.5 1093   100
   seq_along 115 176.0 271.04  197.5 246.0 5829   100

Hi guys
Sry for the late reply. I updated with seq_along but I am stuck in a cannot-push-because-need-to-pull-first-but-this-does-not-work-loop:
$ git status
On branch small_R_fixes_seq_len
nothing to commit, working tree clean
$ git push --set-upstream origin small_R_fixes_seq_len
Updates were rejected because the tip of your current branch is behind its remote counterpart
$ git pull
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
$ git pull origin master
From https://github.com/chrisdane/ESMValTool
 * branch                master     -> FETCH_HEAD
Already up to date.
$#start again with `git push`

It looks like you're pulling in the wrong branch, i.e. the master branch, but your branch is called small_R_fixes_seq_len . Maybe you could try something like this
git pull origin small_R_fixes_seq_len

?

Yes that worked, ty!

Thanks! @mattiarighi Could you please test?

[ ] ./examples/recipe_r.yml
[ ] ./recipe_capacity_factor.yml
[ ] ./recipe_combined_indices.yml
[ ] ./recipe_diurnal_temperature_index.yml
[ ] ./recipe_extreme_events.yml
[ ] ./recipe_extreme_index.yml
[ ] ./recipe_heatwaves_coldwaves.yml
[ ] ./recipe_hyint.yml
[ ] ./recipe_miles_block.yml
[ ] ./recipe_miles_eof.yml
[ ] ./recipe_miles_regimes.yml
[ ] ./recipe_modes_of_variability.yml
[ ] ./recipe_multimodel_products.yml
[ ] ./recipe_quantilebias.yml
[ ] ./recipe_spei.yml
[ ] ./recipe_toymodel.yml

recipe_extreme_events.yml is running since more than 3 hours.
@jhardenberg is that normal?

Ok, it's finished. Sorry, I didn't remember it takes so long.


Re-do mongo CLI execution so it does not use temp files to execute cmds

We were seeing periodic errors running migrations where the mongo CLI was returning exit code 253.  This is an exit code indicating that it couldn't parse the script file in some way or another.
The tricky thing was that the migrations didn't always fail on the same migration.  I suspected that this had to do with some file lock or buffered write ephemeral-ness that was causing these periodic failures.
The mongo CLI supports providing the mongo commands via a parameter without having to write the commands to a temp file at all.  This is the crux of this PR.
Notice the HEREDOC surrounding the command.  This is to handle multi-line migration scripts.

@.ProjectMaintainer or @.reviewer: if you choose to merge this PR note that I branched off of the 0.3.2 tag because the project I'm targetting is on Play 2.3.  If possible can you make sure to build a 0.3.x package along that same dependency tree?

@tmbo thoughts?  Not sure what the next steps might be.

Sorry, did not notice this PR - if you still would like to get this merged, please let me know.


WIP: add fault-domain

High Level Description
What features does this change enable? What bugs does this change fix? High-Level description of how things changed.
Related Issues

DCOS_OSS- Foo the Bar so it stops Bazzing.
DCOS- Foo the Bar so it stops Bazzing.

Checklist for all PR's

[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[ ] Read the DC/OS contributing guidelines
[ ] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

PLEASE FILL IN THE TEMPLATE ABOVE / DO NOT REMOVE ANY SECTIONS ABOVE THIS LINE
Instructions and Review Process
What is the review process and when will my changes land?
All PRs require 2 approvals using github's pull request reviews.
Reviewers should be:

Developers who understand the code being modified.
Developers responsible for code that interacts with or depends on the code being modified.

It is best to proactively ask for 2 reviews by @mentioning the candidate reviewers in the PR comments area. The responsibility is on the developer submitting the PR to follow-up with reviewers and make sure a PR is reviewed in a timely manner. Once a PR has 2 ship-it's, no red reviews, and all tests are green it will be included in the next train.

Can you fill out the template please?

@mesosphere-mergebot label Ready For Review

@mesosphere-mergebot label Work In Progress

@mesosphere-mergebot label Ready For Review

@mesosphere-mergebot bump-ee

@mesosphere-mergebot bump-ee

Gotcha. Thanks.
On Wed, Sep 13, 2017 at 5:03 PM, Maksym Naboka<EMAIL_ADDRESS>wrote:

@darkonie commented on this pull request.
In packages/mesos/extra/dcos-mesos-slave.service
https://github.com/dcos/dcos/pull/1874#discussion_r138769714:

@@ -23,4 +23,4 @@ ExecStartPre=/bin/ping -c1 leader.mesos
ExecStartPre=/opt/mesosphere/bin/bootstrap dcos-mesos-slave
ExecStartPre=/opt/mesosphere/bin/make_disk_resources.py /var/lib/dcos/mesos-resources
ExecStartPre=/bin/bash -c 'for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 2 > $i; echo -n "$i: "; cat $i; done'
-ExecStart=$PKG_PATH/bin/mesos-agent
+ExecStart=$PKG_PATH/bin/start_mesos.sh $PKG_PATH/bin/mesos-agent

@vinodkone https://github.com/vinodkone sorry for confusion, i did not
understand what you meant.
Ok, so currently there is no a good way to run a script from systemd unit
and set the output if this script as environment variable.
ExecStartPre=/opt/mesosphere/bin/make_disk_resources.py
/var/lib/dcos/mesos-resources this like does not set any environment
variables. It might read MESOS_RESOURCES
https://github.com/dcos/dcos/blob/master/packages/mesos/extra/make_disk_resources.py#L152,
but it does not set it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dcos/dcos/pull/1874#discussion_r138769714, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAC8A2TvlXZ5LnxbUJF8cR1jPsJZeONAks5siG1SgaJpZM4PBtot
.

@mesosphere-mergebot label ship it


Add support for request/response transform for Wasm plugin

[approve ci autest]

@serrislew would you pls review again? Thanks.

@serrislew ready for another review

LGTM


docs(operator-docs): Added MergeAll operator documentation

Closes: https://github.com/ReactiveX/rxjs-docs/issues/67

Codecov Report

Merging #141 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #141   +/-   ##
=======================================
  Coverage   88.46%   88.46%
=======================================
  Files           7        7
  Lines          78       78
  Branches        7        7
=======================================
  Hits           69       69
  Misses          6        6
  Partials        3        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b46319...a28ea40. Read the comment docs.

Looks like the branch is out of sync @sumitarora

@ladyleet done

@sumitarora oh noes. otra vez por favor.

LGTM once single quote issue is fixed

@sumitarora please rebase to master.

@ashwin-sureshkumar done


3rd party arpa LM to asr-crdnn-rnnlm-librispeech

Hi,
Is there any example of attaching 3rd party arpa LM to existing model asr-crdnn-rnnlm-librispeech?
I see speechbrain.lm.arpa module, but there is no any examples of BackoffNgramLM usage.

Hi,
In the current version n-gram can easily be integrated with ctc_decode or pyctcdecode (external libraries). By the beginning of Fall, we will also release our own n-gram restorer :-)

Great!
is there any example of such integration?

I think Aku @.***) can provide some examples. In general I think LMs
and integration of ARPA LMs can be the object of a tutorial. What do you
think about Aku?
On Thu, Jun 23, 2022 at 4:47 PM Andrey-847-2 @.***>
wrote:

Great!
is there any example of such integration?
—
Reply to this email directly, view it on GitHub
https://github.com/speechbrain/speechbrain/issues/1467#issuecomment-1164858819,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEA2ZVQSDRO5Q5KF4OHD24DVQTEPXANCNFSM5ZR22I7Q
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

I think @Moumeneb1 can give an example as well. Please provide a short piece of code showing how to connect the CTC output to a n-gram rescoring if you can :-)

@Moumeneb1 , did you have a chance to look at this? (see above)

Hi !
It's made simple to utilize pyctcdecode with a speechbrain trained EndoderASR model; all you have to do is encode the batch and import the right labels from the tokenizer that has already been exported. Following that, you must use pyctcdecide to load the arpa/bin file and do the decoding; batching is another option (multiprocessing not gpu parallelism). This as an easy example.
import torchaudio
import torch
from pyctcdecode import build_ctcdecoder
from speechbrain.pretrained import EncoderASR

#load the model
asr_model = EncoderASR.from_hparams(source="speechbrain/asr-wav2vec2-commonvoice-fr", savedir="pretrained_models/asr-wav2vec2-commonvoice-fr")
#load audio
audio, sr  = torchaudio.load('/content/speechbrain/example1.wav')
#load wav length
rel_length = torch.tensor([1.0])

#encode the wav with wav2vec2+vanilla_enc+ctc head
encoder_out  = asr_model.encode_batch(audio,rel_length)

#get the labels from the used tokenizer
labels  = [asr_model.tokenizer.id_to_piece(id).lower() for id in range(asr_model.tokenizer.get_piece_size())]

labels[1]=' '
labels[0] = '<pad>'

decoder = build_ctcdecoder(labels,
    "model.arpa",
    alpha=0.6)

#decode the batch
decoder.decode(encoder_out[0].cpu().numpy())

@TParcollet @Moumeneb1 Hello, I'm trying to implement pyctcdecode too but I have an issue with the logits and vocab size even if I'm implementing pyctcdecode the same way as you : ValueError: Input logits shape is (179, 1000), but vocabulary is size 1001. Need logits of shape: (time, vocabulary)
Have you ever met that issue?

Hi @VitoPepe you please share your labels list ? Did you correctly change the unk Token to pad ? maybe it's that ?

Great that it worked ! You're welcome ✌

Hello,
It seems that the question has been solved. Therefore, I am closing this issue. Please, feel free to reopen it if necessary! Thank you very much. 😃


Feature/roecct 173 bugfix amend urls

JIRA link
https://companieshouse.atlassian.net/browse/ROECCT-173
Change description
Amend URLs bug fix
Work checklist

[ ] Tests added where applicable
[ ] UI changes meet accessibility criteria

Merge instructions
We are committed to keeping commit history clean, consistent and linear.  A BREAKING CHANGE can be part of commits of any type,
types other than fix: and feat: are allowed, for example build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others,
footers other than BREAKING CHANGE: <description> may be provided.

Closing this PR as I have raised new one https://github.com/companieshouse/overseas-entities-web/pull/1598


⚠ Processors has degraded performance

In 7ec9ffb, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 118 ms

Resolved: Processors performance has improved in 2222693.


"menu" in InputControl has no effect on the TV

Thanks for this repo, I've been using it for quite a while now.
Today (21/06/27) I upgraded to the latest version with pip, and the "menu" command doesn't seem to work anymore.
However I have an older version, in which the "menu" command of InputControl opens a menu on my TV.
Has anything changed in the past months, which might have caused this?

That could a real issue. Just to confirm what you are saying: You have version 1 and version 2 such that something works in v1 but not in v2 when you call the same API in an identical fashion against the same TV.
If this is correct, it could very well be a bug that got introduced recently. Can you help me with the actual versions (git commit hashes) of these two instances? Essentially, I'd like to know how to reproduce this (git commit hashes, version strings, or even the diff of the actual installed file across these version). Thanks!

Yes, your confirmation is correct.
I've last updated my local version in June 2019, so it should be commit c6236e599645d14626c4d15fad56eac9f4cbacd2. This version didn't offer the "menu" command yet. I cheated and added that entry in "controls.py" at line 289, like this "menu": {             "command": [["type", "button"], ["name", "MENU"]]         },
and that made it work.
Currently I'm using the latest pip version, which should be 22f2f19d8958d158a5dc41349212c4b56c6c8cce. This version offers the "menu" command, but it has no effect on my tv, while the old version still works fine.

Thanks! I'll check it locally and post the results back here.

Could you verify the problem?

Sorry, last few weeks were busy and I couldn't get back to this one.
TL;DR: I couldn't reproduce it.
TV being tested against: LG CX 55 OLED (2020 model), WebOS 5.0, v03.23.15
State A: July 2019, git checkout c6236e5, patch the source with "menu": { ...}, install the library in a virtualenv
client = ...  # discover, connect and register

inp = InputControl(client)
inp.connect_input()
inp.menu()  # Triggers the menu correctly

State B: Checkout the HEAD, no patches, install the library in another virtualenv.
client = ...  # discover, connect and register

inp = InputControl(client)
inp.connect_input()
inp.menu()  # Triggers the menu correctly

In both the cases, this is what I observe:

@kilian579 try pressing a button on your physical remote control before executing the command. I still don't quite understand why this happens, but in my case, if I don't press a physical button before, it doesn't work.

Closing this out, feel free to re-open if you still need help.


Nested function information is missing

info functions command does not output information if the program is stopped in a nested function.
(gdb_dev) vperevez@ansatnuc05:/localdisk/work/vperevez/numba-dppy/numba_dppy/examples/debug$ NUMBA_OPT=0 gdb-oneapi -q python
Reading symbols from python...
(gdb) break simple_dppy_func.py:22
No source file named simple_dppy_func.py.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (simple_dppy_func.py:22) pending.
(gdb) run simple_dppy_func.py
Starting program: /localdisk/work/vperevez/miniconda3/envs/gdb_dev/bin/python simple_dppy_func.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 2736192]
[Detaching after fork from child process 2736193]
[Detaching after fork from child process 2736194]
[New Thread 0x7fffdb97d700 (LWP 2736196)]
intelgt: gdbserver-gt started for process 2736186.
intelgt: attached to device 1 of 1; id 0x9bca (Gen9) on inferior 2.
[New Thread 0x7fffca8c9700 (LWP 2736200)]
[Detaching after fork from child process 2736201]
[Detaching after fork from child process 2736202]
[Detaching after fork from child process 2736203]
[Detaching after fork from child process 2736204]
[Detaching after fork from child process 2736205]
[New Thread 0x7fffbb1d5700 (LWP 2736206)]
[New Thread 0x7fffbadd4700 (LWP 2736207)]
[New Thread 0x7fffba9d3700 (LWP 2736208)]
[New Thread 0x7fffba5d2700 (LWP 2736209)]
[New Thread 0x7fffb9dd0700 (LWP 2736211)]
[New Thread 0x7fffba1d1700 (LWP 2736210)]
[New Thread 0x7fffb99cf700 (LWP 2736212)]
[New Thread 0x7fffb91cd700 (LWP 2736214)]
[New Thread 0x7fffb8dcc700 (LWP 2736215)]
[New Thread 0x7fffb95ce700 (LWP 2736213)]
[New Thread 0x7fffb89cb700 (LWP 2736216)]
[Detaching after fork from child process 2736217]
[Detaching after fork from child process 2736218]
[New Thread 1.1073741824]
[New Thread 1.1073742080]
[Switching to Thread 1.1073741824 lane 0]

Thread 2.2 hit Breakpoint 1, with SIMD lanes [0-7], __main__::func_sum () at simple_dppy_func.py:22
22          result = a_in_func + b_in_func
(gdb) info func func_sum
All functions matching regular expression "func_sum":
(gdb) info func __main__::func_sum
All functions matching regular expression "__main__::func_sum":
(gdb) bt
#0  __main__::func_sum () at simple_dppy_func.py:22
#1  __main__::kernel_sum () at simple_dppy_func.py:29
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000fffeafe0 in __main__::func_sum(float, float) at simple_dppy_func.py:22 inf 2
        breakpoint already hit 1 time

(gdb) info kernel_sum
Undefined info command: "kernel_sum".  Try "help info".
(gdb) info func kernel_sum
All functions matching regular expression "kernel_sum":

File simple_dppy_func.py:
26:     void __main__::kernel_sum(DPPYArray<float, 1, C, mutable, aligned>, DPPYArray<float, 1, C, mutable, aligned>, DPPYArray<float, 1, C, mutable, aligned>);

@diptorupd
Do you have any ideas about it?

Need to check on the latest version and have test for it.