Extending information for HAL+JSON-responses

The best HAL+JSON-spec I could find is this expired draft.
Point Link Objects lists a few more props we have not yet implemented which are easily feasible, such as type for the content-type of the linked resource.

For now only type seems feasible.
Prop title could be done with introducing a new combinator, see #31.
The others as well but they seem uncommon and useless.


If create pr is called and the PR is already deployed it needs to rebuild

When PR's are made clusternator makes a deployment.  Currently if the PR is updated through git Clusternator will attempt to re-deploy and fail.  This needs to be fixed.  First implementation can just check if PR exists then tear down and re-deploy

This issue is coupled to #161 in that teardowns fail with security group issues sometimes


Add INNER as a JoinOption

I've run into a use case that could be made simpler with another JoinOption available.
I'm dynamically generating SQL queries and conditionally change from JOIN to LEFT JOIN. Since an unmodified JOIN is equivalent to INNER JOIN then I realized that I could use JoinWithOption, and conditionally change the JoinOption.
There are a few obvious options Ito do this:

Conditionally call either Join or JoinWIthOption, with a JoinOption to the second
Create my own const JoinOption that's just the word "INNER". Then I'll conditionally just use a package constant Inner or the imported constant sqlbuilder.LeftJoin. It's a little inconsistent, but not bad
Add a new const Inner JoinOption = "INNER" to the library

Any thoughts on the last option? I figured it's an easy and reasonable addition.
I can create a PR if you think it's a good path forward for the library.

I agree. Adding a new const InnerJoin can improve readability. Please submit a PR for it. Welcome to contribute.

Will do.
I just recently started with this library and I'm enjoying it. Thanks!

👍


Missing FEC IDs

We're using your data downstream. Thanks! A few hundred legislators serving post-2000 have no FEC IDs listed, and for our purposes we have to join legislators to donation data, so this is a necessary crosswalk. Our downstream pull of your data is slightly out of date, so some of these might already be fixed, but I figured I'd contribute back the results of an automated attempt to populate these fields.
Attached is a CSV with the following format:
Column 1: bioguide_id
Column 2: first_name
Column 3: last_name
Column 4: thomas_id
Column 5: state
Column 6: fec_id
(You can probably just use 1 and 6)
legislators_additional_fec_ids.csv
Note that many entries have multiple FEC IDs. I manually screened for same-named father-son pairs to remove those. Hope this is useful.

Sorry, one additional omission there:
J000110,Christopher,John,01495,LA,S4LA00040
J000110,Christopher,John,01495,LA,H6LA07131

Thanks. I opened a PR to add this in #934.
For posterity, I wrote this script to import it:
import csv
import rtyaml

count = 0

for fn in ("current", "historical"):
	with rtyaml.edit(f"../legislators-{fn}.yaml") as P:
		PID = { }
		for p in P:
			PID[p["id"]["bioguide"]] = p
		for rec in csv.DictReader(open("/home/user/Downloads/legislators_additional_fec_ids.csv")):
			if rec["bioguide_id"] not in PID: continue
			p = PID[rec["bioguide_id"]]
			if "fec" not in p["id"]: p["id"]["fec"] = []
			if rec["fec_id"] not in p["id"]["fec"]:
				p["id"]["fec"].append(rec["fec_id"])
				count += 1
print(count)

But our tests caught some FEC IDs that now occur duplicated on legislators:
H0MO01066 is duplicated: C000488 C001049
H4IL03077 is duplicated: L000342 L000563

I would appreciate if you could resolve this (so that my effort to import this isn't wasted...).

These are father/son pairs, good catch and my apologies for the wasted time.
The correction:
bioguide_id,fec_id
C000488,H6MO01022
L000342,H2IL05019

Source for Lipinski: https://www.fec.gov/data/candidate/H2IL05019/ (the other bioguide ID is Lipinski's father, who has a different FEC ID)
Source for Clay: https://www.fec.gov/data/candidate/H6MO01022/  (the other bioguide ID is Clay Jr., who has a different FEC ID)
File with correction:
legislators_additional_fec_ids.csv

Thank you! All fixed & merged.


iOS 11

Was this plugin tested on ios11? Somebody was able to get notifications there?

I have tested with IOS 11 (builded with xcode9) and it works! however, the first app open crashes after the user grant notification access. The permission was set and after that the app opens and works normally.

We get notification on iOS 9 & iOS 10 but not on iOS 11.
Swizzling is enabled (no setting), we get a FCM token. Seems that Firebase does not know the APNs token
I found issues that could be releated on iOS Firebase SDK.
https://github.com/firebase/firebase-ios-sdk/issues/254
https://github.com/firebase/firebase-ios-sdk/issues/314
We tried to copy the iOS SDK v4.4.0 instead of v4.0.0 inside this plugin but it does not work straight away
https://github.com/Loisirsencheres/cordova-plugin-firebase/commit/6c118fa8ac9900f69a948a5d97473b123bc7ceb9
Has anyone got Firebase notification on iOS 11 ?

@ironbone
Yes, also when app is closed. All works fine, the only issue is in the first app open.
@Jeremlicious, you tried to use cordova IOS platform 4.5.1 and xcode 9?

I just did some tests. The newest xcode, newest iOS, newest cordova, newest ionic.
When application is closed (not in background but not working at all) and I send a Data (only) Message I never obtain it when I open the application. Is I see it arives on the phone nut there is never called the function onNotificationOpen()
If the application is in foreground or backgroud it works ok.

We ended up by using phonegap-plugin-push + cordova-plugin-fcm it works.

@Cicciokr Yes I have a similar issue - I get a token fine however nothing else happens

Hi, maybe this can help you http://francescopantisano.it/complete-guide-ionic-3-firebase-cordova-plugin-firebase/  it's important the last point, to install the ios certs for send notification: https://firebase.google.com/docs/cloud-messaging/ios/certs

closing as resolved

I was having this issue too, but it was only happening with iOS. On Android it was fine. I realised that I wasn't getting the token when connected with an iOS device even with the APN certs already configured. I just add this to make it work:
window.FirebasePlugin.grantPermission();
It seens that by default any iOS app is set to deny push notifications, so you have to grant permission it when the app initializes.

@Cicciokr try to debug with device console, maybe there are some details logged.
i encountered a similar problem on my ios 11 relates to a GoogleService.plist problem but no one log in dev console of Safari (now called 'Safari Technologies Preview' wtf)
Hope it can helps


1012: Implementing OB Domestic Payment Consents API

Adding DomesticPaymentConsentsApi/Controller for v3.1.10

This controller implements the OBIE Domestic Payment Consents API in RS
Adding OBWriteDomesticConsent4Validator to the validation framework
NOTE IG impl of this API was generating charges for domestic payments, this new impl does not generate charges (this is more realistic for a UK Bank)

Further work can be done to inject charges generation logic if desired

x-api-client-id header added to DomesticPaymentConsentsApi and DomesticPaymentsApi to identify the ApiClient (TPP)

Gateway must supply this header

Using RCS Consent Store API for Consent CRUD operations

Using secure-api-gateway-ob-uk-rcs-consent-store-client library to call Consent Store API
Configuring rcs.consent.store.api.baseuri Spring property via environment variable supplied from the deployment config map
Updating existing DomesticPaymentApiController to use the RCS Consent Store
Updating GlobalExceptionHandler to handle ConsentStoreClientException

Fixed BasePaymentRequestValidator to validate that consent is Authorised prior to submitting the payment

Raised a new PR for this as the rebase was getting messy: https://github.com/SecureApiGateway/secure-api-gateway-ob-uk-rs/pull/200


[WIP] added cache to drupal search

Drupal cannot handle the search request that come from volunteers module through spark.
It was decided to cache drupal searches in spark.
Proposed solution

userdata is stored in memory with TTL of one day.
USE_DRUPAL_CACHE=true env vairable enables the cache usage.
node-cache package is used.
Tradeoffs

drupal changes will be reflected only after cache invalidation
Testing Done

added env variable for TTL:
USE_DRUPAL_CACHE_TTL
default is 3600*24


Error when updating/log

I am trying to resolve an error when trying to update or see logs.
It seems it is a user/permission problem but I am not able to find a solution.
I tried these 2 solutions which are very similar to my problem but with no luck:
https://github.com/oznu/homebridge-config-ui-x/issues/106#issuecomment-395956896
https://github.com/oznu/homebridge-config-ui-x/issues/87#issuecomment-385610290
This is the error I get when updating
USER: homebridge
DIR: /usr/lib/node_modules
CMD: sudo -E -n npm --no-update-notifier install --unsafe-perm homebridge-webos-tv@latest

sudo: unable to change to root gid: Operation not permitted
sudo: unable to initialize policy plugin

Command failed. Please review log for details

this is the error I get when checking the log
Using systemd to tail logs
CMD: sudo -n journalctl -o cat -n 500 -f -u homebridge

sudo: unable to change to root gid: Operation not permitted
sudo: unable to initialize policy plugin

The log tail command "sudo -n journalctl -o cat -n 500 -f -u homebridge" exited with code 1.
Please check the command in your config.json is correct.

See https://github.com/oznu/homebridge-config-ui-x#log-viewer-configuration for instructions.

If I type that log tail command via ssh if works without a problem.
This is my config
{
   "name": "Config",
   "port": 8080,
   "theme": "red",
   "restart": "sudo -n systemctl restart homebridge",
   "tempUnits": "c",
   "sudo": true,
   "platform": "config",
   "log": {
      "method": "systemd",
      "service": "homebridge"
   }
},

I have also activated Password-less sudo required
Let me know if you need any other information.
Thanks

Hi yepecece, I solved this error changing the service file of HomeBridge:
/etc/systemd/system/homebridge.service
the parameter that chenge is "CapabilityBoundingSet".
my file, that working, is:
[Unit]
Description=Node.js HomeKit Server
After=syslog.target network-online.target

[Service]
Type=simple
User=homebridge
EnvironmentFile=/etc/default/homebridge
ExecStart=/usr/bin/homebridge -I $HOMEBRIDGE_OPTS
Restart=on-failure
RestartSec=10
KillMode=process
CapabilityBoundingSet=CAP_IPC_LOCK CAP_SETGID CAP_NET_RAW CAP_NET_ADMIN CAP_SETUID CAP_SYS_ADMIN
AmbientCapabilities=CAP_NET_RAW

[Install]
WantedBy=multi-user.target

I hop that helps.

Thanks @Desio90, nice work figuring that out.
I'll do some testing / research and if it's all good I'll add a "best practice" systemd service file to the wiki.

Wow.
Thanks @Desio90. Works right away.

Thanks again, I've added a guide to the wiki that contains the updated systemd service file.
https://github.com/oznu/homebridge-config-ui-x/wiki/Homebridge-&-Systemd-(Raspbian,-Ubuntu,-Debian)
While testing on Ubuntu 18.04 and Raspbian I found I needed to add a few more items to the CapabilityBoundingSet list to get all the features working:
/etc/systemd/system/homebridge.service:
[Unit]
Description=Homebridge
After=syslog.target network-online.target

[Service]
Type=simple
User=homebridge
EnvironmentFile=/etc/default/homebridge
ExecStart=/usr/bin/homebridge $HOMEBRIDGE_OPTS
Restart=on-failure
RestartSec=3
KillMode=process
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE CAP_AUDIT_WRITE CAP_SYS_ADMIN
AmbientCapabilities=CAP_NET_RAW

[Install]
WantedBy=multi-user.target

/etc/default/homebridge:
# Defaults / Configuration options for homebridge
# The following settings tells homebridge where to find the config.json file and where to persist the data (i.e. pairing and others)
HOMEBRIDGE_OPTS=-U /var/lib/homebridge -I

# If you uncomment the following line, homebridge will log more
# You can display this via systemd's journalctl: journalctl -f -u homebridge
# DEBUG=*

# To enable web terminals via homebridge-config-ui-x uncomment the following line
# HOMEBRIDGE_CONFIG_UI_TERMINAL=1


Can't Add Weapon Qualities to Weapons (Ver. 11)

So I made a new weapon and new weapon quality just to make sure it wasn't working. Every time I drag the item quality over nothing shows up on the weapon. I also made sure I wasn't pulling from compendiums or anything.
Foundry: Ver 11, Build 300
Star Wars FFG: 1.801

Ok I think I figured it out. For some reason I tried adding the weapon qualities that were from an earlier build and they didn't show up. So I tried adding a completely new one, and it still didn't work. Which prompted this bug report. But started working out of nowhere which leads me to guess that it had to do with my old version weapon qualities. It seems to work every time with a freshly made quality if I don't drag the old one onto it 1st. Not sure if this helps you at all but there it is.


No error message printed when not using an AP supporting wifi card

(Forgive me the meaningless title, I hit Enter mistakenly ;) )
Tried to use the tool on a Thinkpad T500, Ubuntu Bionic, e1000e driver of kernel 4.15.0-73.
The Wifi HW is:
03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300 (8086:4236)
Obviously this adapter doesn't allow AP mode... it's been set to master, but the rest doesn't seem to work
console:
Starting smart config pairing procedure
Waiting for the device to install the intermediate firmware
Put device in EZ config mode (blinking fast)
Sending SSID                  vtrust-flash
Sending wifiPassword
Sending token                 00000000
Sending secret                0101
Traceback (most recent call last):
File "./smartconfig/main.py", line 26, in
smartconfig( passwd, ssid, region, token, secret )
File "/home/to/tuya-convert/scripts/smartconfig/smartconfig.py", line 46, in smartconfig
sock = SmartConfigSocket()
File "/home/to/tuya-convert/scripts/smartconfig/smartconfig.py", line 29, in init
self._socket.bind((address, 0))
OSError: [Errno 99] Cannot assign requested address
.......................................
smarthack-wifi.log
Attempting to stop wpa_supplicant
Stopping NetworkManager...
Configuring AP interface...
RTNETLINK answers: File exists
Starting DNSMASQ server...
Starting AP on wlp3s0...
Configuration file: /dev/stdin
nl80211: Could not configure driver mode
nl80211: deinit ifname=wlp3s0 disabled_11b_rates=0
nl80211 driver initialization failed.
wlp3s0: interface state UNINITIALIZED->DISABLED
wlp3s0: AP-DISABLED
hostapd_free_hapd_data: Interface wlp3s0 wasn't started
AP closed
Stopping DNSMASQ server...
Restarting NetworkManager...
I later used a USB stick (idVendor=0bda, idProduct=8176
....
rtl8192cu: Chip version 0x10
rtl8192cu: Board Type 0
....
rtl8192cu: Loading firmware rtlwifi/rtl8192cufw_TMSC.bin
....
which worked ok, thus I suspect the Intel not to like the master mode
Pre-Condition was to "rfkill unblock" the devices previous to starteing the script. This could be captured in the script, I think ;)
Nice work!!! /To

Hi @tmuehlhoff, are you saying that tuya-convert is not working with a wifi adapter that does not support AP mode? This is expected as AP mode is required to, well, create an AP.

Yeah sure, but what I'm after is an according error output....
nl80211: Could not configure driver mode
Could cause to output one, or ?
Also it could issue an rfkill unblock....

Sorry I don't understand what you are asking. If your card does not support AP mode, you will get errors if you try anyway. None of the services can start if that first step fails.

What about an error message if the wifi chip isn't supported ?

We do this already. Can you specify what you think it should do differently?

well, as you can see from above, I didn't receive any, only took that info from the logs...
my shell output:
Starting smart config pairing procedure
Waiting for the device to install the intermediate firmware
Put device in EZ config mode (blinking fast)
Sending SSID vtrust-flash
Sending wifiPassword
Sending token 00000000
Sending secret 0101

obviously my intel card doesn't produce that error message you're matching on... Let me try to reproduce it, I will come back with some more info on why this isn't catched.

Got it: if you just grep for "AP" in iw list it will find the string "AP" even though it's not supported by the adapter:
tomas@thinky:~$ iw list | grep AP
* AP: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0
* AP/VLAN: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0
* AP: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0
* AP/VLAN: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0
Driver supports full state transitions for AP/GO clients
tomas@thinky:~$
but:
    Supported interface modes:
             * IBSS
             * managed
             * monitor

does not list AP mode.
Sure you can now argue this is maybe an interface fault, but it's an Intel "Ultimate N Wifi Link 5300", not the most recent, but pretty common still ;)
On top of that, if I plug in the 2nd Wifi adapter, "iw list" will print all capabilities for BOTH adapters!
Attaching my iw list output for only the Intel adapter as reference.
BR/To
Wiphy phy0 	max # scan SSIDs: 20 	max scan IEs length: 195 bytes 	max # sched scan SSIDs: 0 	max # match sets: 0 	max # scan plans: 1 	max scan plan interval: -1 	max scan plan iterations: 0 	Retry short limit: 7 	Retry long limit: 4 	Coverage class: 0 (up to 0m) 	Device supports RSN-IBSS. 	Supported Ciphers: 		* WEP40 (00-0f-ac:1) 		* WEP104 (00-0f-ac:5) 		* TKIP (00-0f-ac:2) 		* CCMP-128 (00-0f-ac:4) 		* CCMP-256 (00-0f-ac:10) 		* GCMP-128 (00-0f-ac:8) 		* GCMP-256 (00-0f-ac:9) 	Available Antennas: TX 0 RX 0 	Supported interface modes: 		 * IBSS 		 * managed 		 * monitor 	Band 1: 		Capabilities: 0x107e 			HT20/HT40 			SM Power Save disabled 			RX Greenfield 			RX HT20 SGI 			RX HT40 SGI 			No RX STBC 			Max AMSDU length: 3839 bytes 			DSSS/CCK HT40 		Maximum RX AMPDU length 65535 bytes (exponent: 0x003) 		Minimum RX AMPDU time spacing: 4 usec (0x05) 		HT Max RX data rate: 450 Mbps 		HT TX/RX MCS rate indexes supported: 0-23 		Bitrates (non-HT): 			* 1.0 Mbps 			* 2.0 Mbps (short preamble supported) 			* 5.5 Mbps (short preamble supported) 			* 11.0 Mbps (short preamble supported) 			* 6.0 Mbps 			* 9.0 Mbps 			* 12.0 Mbps 			* 18.0 Mbps 			* 24.0 Mbps 			* 36.0 Mbps 			* 48.0 Mbps 			* 54.0 Mbps 		Frequencies: 			* 2412 MHz [1] (15.0 dBm) 			* 2417 MHz [2] (15.0 dBm) 			* 2422 MHz [3] (15.0 dBm) 			* 2427 MHz [4] (15.0 dBm) 			* 2432 MHz [5] (15.0 dBm) 			* 2437 MHz [6] (15.0 dBm) 			* 2442 MHz [7] (15.0 dBm) 			* 2447 MHz [8] (15.0 dBm) 			* 2452 MHz [9] (15.0 dBm) 			* 2457 MHz [10] (15.0 dBm) 			* 2462 MHz [11] (15.0 dBm) 			* 2467 MHz [12] (15.0 dBm) (no IR) 			* 2472 MHz [13] (15.0 dBm) (no IR) 	Band 2: 		Capabilities: 0x107e 			HT20/HT40 			SM Power Save disabled 			RX Greenfield 			RX HT20 SGI 			RX HT40 SGI 			No RX STBC 			Max AMSDU length: 3839 bytes 			DSSS/CCK HT40 		Maximum RX AMPDU length 65535 bytes (exponent: 0x003) 		Minimum RX AMPDU time spacing: 4 usec (0x05) 		HT Max RX data rate: 450 Mbps 		HT TX/RX MCS rate indexes supported: 0-23 		Bitrates (non-HT): 			* 6.0 Mbps 			* 9.0 Mbps 			* 12.0 Mbps 			* 18.0 Mbps 			* 24.0 Mbps 			* 36.0 Mbps 			* 48.0 Mbps 			* 54.0 Mbps 		Frequencies: 			* 5180 MHz [36] (15.0 dBm) (no IR) 			* 5200 MHz [40] (15.0 dBm) (no IR) 			* 5220 MHz [44] (15.0 dBm) (no IR) 			* 5240 MHz [48] (15.0 dBm) (no IR) 			* 5260 MHz [52] (15.0 dBm) (no IR, radar detection) 			* 5280 MHz [56] (15.0 dBm) (no IR, radar detection) 			* 5300 MHz [60] (15.0 dBm) (no IR, radar detection) 			* 5320 MHz [64] (15.0 dBm) (no IR, radar detection) 			* 5500 MHz [100] (15.0 dBm) (no IR, radar detection) 			* 5520 MHz [104] (15.0 dBm) (no IR, radar detection) 			* 5540 MHz [108] (15.0 dBm) (no IR, radar detection) 			* 5560 MHz [112] (15.0 dBm) (no IR, radar detection) 			* 5580 MHz [116] (15.0 dBm) (no IR, radar detection) 			* 5600 MHz [120] (15.0 dBm) (no IR, radar detection) 			* 5620 MHz [124] (15.0 dBm) (no IR, radar detection) 			* 5640 MHz [128] (15.0 dBm) (no IR, radar detection) 			* 5660 MHz [132] (15.0 dBm) (no IR, radar detection) 			* 5680 MHz [136] (15.0 dBm) (no IR, radar detection) 			* 5700 MHz [140] (15.0 dBm) (no IR, radar detection) 			* 5745 MHz [149] (disabled) 			* 5765 MHz [153] (disabled) 			* 5785 MHz [157] (disabled) 			* 5805 MHz [161] (disabled) 			* 5825 MHz [165] (disabled) 	Supported commands: 		 * new_interface 		 * set_interface 		 * new_key 		 * start_ap 		 * new_station 		 * new_mpath 		 * set_mesh_config 		 * set_bss 		 * authenticate 		 * associate 		 * deauthenticate 		 * disassociate 		 * join_ibss 		 * join_mesh 		 * set_tx_bitrate_mask 		 * frame 		 * frame_wait_cancel 		 * set_wiphy_netns 		 * set_channel 		 * set_wds_peer 		 * probe_client 		 * set_noack_map 		 * register_beacons 		 * start_p2p_device 		 * set_mcast_rate 		 * connect 		 * disconnect 		 * set_qos_map 		 * set_multicast_to_unicast 	Supported TX frame types: 		 * IBSS: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0x b0 0xc0 0xd0 0xe0 0xf0 		 * managed: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0  0xb0 0xc0 0xd0 0xe0 0xf0 		 * AP: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0  0xc0 0xd0 0xe0 0xf0 		 * AP/VLAN: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0  0xb0 0xc0 0xd0 0xe0 0xf0 		 * mesh point: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0 xa0 0xb0 0xc0 0xd0 0xe0 0xf0 		 * P2P-client: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0 xa0 0xb0 0xc0 0xd0 0xe0 0xf0 		 * P2P-GO: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0  0xb0 0xc0 0xd0 0xe0 0xf0 		 * P2P-device: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0 xa0 0xb0 0xc0 0xd0 0xe0 0xf0 	Supported RX frame types: 		 * IBSS: 0x40 0xb0 0xc0 0xd0 		 * managed: 0x40 0xd0 		 * AP: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0 		 * AP/VLAN: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0 		 * mesh point: 0xb0 0xc0 0xd0 		 * P2P-client: 0x40 0xd0 		 * P2P-GO: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0 		 * P2P-device: 0x40 0xd0 	software interface modes (can always be added): 		 * monitor 	interface combinations are not supported 	HT Capability overrides: 		 * MCS: ff ff ff ff ff ff ff ff ff ff 		 * maximum A-MSDU length 		 * supported channel width 		 * short GI for 40 MHz 		 * max A-MPDU length exponent 		 * min MPDU start spacing 	Device supports TX status socket option. 	Device supports HT-IBSS. 	Device supports SAE with AUTHENTICATE command 	Device supports scan flush. 	Device supports per-vif TX power setting 	Driver supports full state transitions for AP/GO clients 	Driver supports a userspace MPM 	Device supports static SMPS 	Device supports dynamic SMPS 	Device supports configuring vdev MAC-addr on create.

Got it: if you just grep for "AP" in iw list it will find the string "AP" even though it's not supported by the adapter

How about iw list | grep -A 10 "Supported interface modes" | grep -e "\* AP$"?

On top of that, if I plug in the 2nd Wifi adapter, "iw list" will print all capabilities for BOTH adapters!

Yes, this is a known issue with how iw list works. I decided it was better to catch some bad configurations than none at all. If you know of a way to list the capabilities of only the specified interface, please let us know.

Yes, suggestion works well here, thanks for fixing....
The other thing: if the device is RFKILLED... you could run an rfkill unblock  over it ?!
Or do you want me to file another ticket ?

You do not need to file another ticket. A wifi device wouldn't normally be rfkilled, unless done explicitly by the user, in which case it would not be the responsibility of tuya-convert to change this. Can you explain why a device might be rfkilled without the user knowing?

well, MY device WAS rfkilled, the only thing I did is enable Ethernet in NetworkManager (which implicitly disables Wifi, thus rfkills it).


Upgrade to 2.13.8

https://gerrit.googlesource.com/homepage/+/refs/heads/md-pages/releases/2.13.md#2.13.8

@CoRfr
Someone has submitted a PR for 2.14. I wonder if it's right time for you to move 2.13.x to a new 2.13.x branch and use master branch to track 2.14.x?

@thinkernel
Oh cool I didn't knew that 2.14 had a release, looking forward to try PolyGerrit!
I will probably continue to use 2.13.8 internally until 2.14.1 or 2.14.2 but I think it's a great idea to make it available as a Docker image, so your proposition works for me!


Adds proxy support

Given that the person has exported $http_proxy and $https_proxy (which they need to if they are using Internet over a proxy server), their proxy settings will be used by requests. The program won't run without this.
Please try this on a non-proxy Internet connection.

This error.

will fix it

Yeah. Seems fine. Do some PEP-8 checks before the final commit.
You may want to use this https://github.com/PyCQA/pycodestyle

Fixed most of the issues caused by me. Leaving the long line length issue, which you don't seem to care about

Long line is acceptable to an extent. Anyways, merging it now. Thanks :)


Recent History STF Test Vectors (GP Section 7)

README

[x] ASN.1 schema
[x] README
[x] MMR computation using keccak-256
[x] JAM Codec (*.bin files)
[x] Delete *.scale vectors (old compact)

I am not getting the same MMR hash. Can you clarify how it is calculated?
For progress_blocks_history-2, the payload is 0x8720b97ddd6acc0f6eb66e095524038675a4e4067adc10ec39939eaefc47d842 ++ 0x7507515a48439dc58bc318c48a120b656136699f42bfd2bd45473becba53462d and the hash I am getting is 0x 7076c31882a5953e097aef8378969945e72807c4705e53a0c5aacc9176f0d56b
I am getting the same number using https://emn178.github.io/online-tools/keccak_256.html with input of 8720b97ddd6acc0f6eb66e095524038675a4e4067adc10ec39939eaefc47d8427507515a48439dc58bc318c48a120b656136699f42bfd2bd45473becba53462d

@xlc there was a deviation from the spec in our impl. Thank you

the values are matching with our implementation now 👍

@davxy
progress_with_bad_signatures-2.json this file will trigger two errors I think
in culprit "key": "0x3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29"
it's in the pre_state's offender too
    --- FAIL: TestDisputesTiny/progress_with_bad_signatures-2.json (0.00s)
        disputes_test.go:139: Failed in File progress_with_bad_signatures-2.json
            except error Bad signature within the culprits sequence.
            failed to verify disputes: got Culprit offender already in the offenders list.

Temporarily closed in favor of https://github.com/w3f/jamtestvectors/pull/28


Visual query builder from Graphql schema

Hello,
I am quite new at graphql, but I love the way we can query the graphql API with the graphiql interface so easily. I would like to push this concept forward, and make an interface designed for non-developers who don't like to see a single line of code:
I was wondering if it would make sense to you, and would be technically possible, to automatically create a visual query builder similar to the JQuery Builder (https://querybuilder.js.org/) based on graphql schemas? And what would be the limitations of such tool?
It looks feasible to me, but I have searched for some implementations but found nothing.
I was wondering if anyone is aware of such visual query builder interface or would like to contribute to build one?

I've been meaning to open source this https://youtu.be/C6-Uu9Ljh9U?t=61 (try it here https://graphiql.onegraphapp.com)
What's holding me up is that it doesn't hand fragments, unions, interfaces, variables, or lists of arguments - and visually it's not great. I'll work on the visual design a bit next week, but the other items on the list also need to be handled (also, it's written in Reason, and the implementation is technically wonky).

Thank you, that is a good start 👍 ! I will definitively have a look and come back to you.
If I am not mistaken, looks like https://graphqleditor.com/ is also promising some features in its roadmap, but not sure when we will have something to test to build queries (currently they only support schema creation).

I'd love to see this added to graphiql as well. It would take the DX to a new level!

OneGraph was acquired by Netlify and it looks like the explorer plugin hasn't been updated in quite a while. Is there any chance that plugin could be forked and merged into the main project? Having a query builder is the entire reason I want to use a GUI for GraphQL.


Credential name doesn't work since 0.9.0

Description
juju_model resource is breaking with value sunbeam-microk8s-creds (valid credentials registered in juju controller)
Urgency
Blocker for our release
Terraform Juju Provider version
0.9.0
Terraform version
1.3.7
Terraform Configuration(s)
https://github.com/canonical/sunbeam-terraform/
Reproduce / Test
Create a machine controller
Add a microk8s cloud to the controller
Reference its credentials in sunbeam-terraform
terraform init
terraform apply

Debug/Panic Output
When applying changes to juju_model.sunbeam, provider
                    "provider[\"registry.terraform.io/juju/juju\"]" produced an unexpected new
                    value: .credential: was cty.StringVal("sunbeam-microk8s-creds"), but now
                    cty.StringVal("sunbeam-microk8s/micro1.project.serverstack/sunbeam-microk8s-creds").

Notes & References
This is a regression introduced in 0.9.0.
Deploying with 0.8.0 works fine.

@gboutry can you please point to where juju_model.sunbeam is defined in the GitHub repo provided.

Of course: https://github.com/canonical/sunbeam-terraform/blob/8ec04e0dc6d76fa3ea406c5a23bb88d349a2101a/main.tf#L38C1-L38C1

@gboutry my understanding is that sunbeam hardcodes the version of the terraform provider used by pulling it into the snap, please confirm.

At snap build time, snapcraft will pull the providers respecting the plans' constraints.
We don't set upper constraints on terraform provider and 0.9.0 got pulled in our latest edge.

This job failed because credentials "" got translated to microk8s/admin/microk8s
Error message:
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to juju_model.sunbeam, provider
│ "provider[\"registry.terraform.io/juju/juju\"]" produced an unexpected new
│ value: .credential: was cty.StringVal(""), but now
│ cty.StringVal("microk8s/admin/microk8s").
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵


Feat: [OR-170] TextArea Rebrand

TextArea's active border is not ok as well

Stale PR, everything is getting updated in other pr, so this one goes to the trash


Maine data way off

Data for Maine is way off on number of deaths.  Looks like maybe hospitalizations are being counted instead as deaths.

I looked at the data closer and that's exactly what's happening.  The death data is not in the file and the column marked deaths is actually hospitalizations.

Table format on Maine CDC website changed on April 2, breaking the scraper.  Will try to submit a pull request to fix.

Thanks @kb1ujs, please do send along a PR!

Hi @kb1ujs , thanks for the PR.  This was recently merged, so if this problem is resolved in the next release or so please close this issue.  Thanks again! jz

three dates have not yet been fixed: 03-30, 03-31, and 04-01

Looks like the format change occurred a few days earlier than I thought.  I'll fix it and submit a new pull request.

Closed by #718


generate base test config files, add config for sonar using credo

Features:

Generate sonarqube reports using sobelow and credo it closes #20

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Growth statistics for "past day" showing 0

When I go to http://<url>/supabase and select all repos, the growth statistics for "Past day" shows 0 even though I don't think it should be 0.
I feel like we could simply remove the whole section now that we have "daily new counts"? Or, alternatively, I feel like we should show "Past week" and "Past 30 days", and then compare them to the previous 1 week period and 1 month period, respectively.

Ooh can i check - cause for me its showing 7 when all repos are selected

Gotcha on the second point too - good one! yeah we can compare with the previous "periods" and show the indication like issues

Ooh can i check - cause for me its showing 7 when all repos are selected

Gotcha on the second point too - good one! yeah we can compare with the previous "periods" and show the indication like issues

Hmm I'm not sure why it's not showing the correct info on my end. It looks like this (the monthly info's also different):

Hmm I'm not sure why it's not showing the correct info on my end. It looks like this (the monthly info's also different):

Anyway I think I can look into this issue tomorrow!

Anyway I think I can look into this issue tomorrow!

oooh thats really odd 🤔
but can do! thanks so much @ykdojo!

oooh thats really odd 🤔
but can do! thanks so much @ykdojo!

I just checked repository.surf, and I still see the same thing:

I'm assuming @joshenlim doesn't have a problem with it. Maybe it's a timezone thing? Will look into it.

I just checked repository.surf, and I still see the same thing:

I'm assuming @joshenlim doesn't have a problem with it. Maybe it's a timezone thing? Will look into it.


Bug: asserting an interface and using non-interface members

https://github.com/confluentinc/confluent-kafka-dotnet/blob/376a3250c06f347c6492e90b9921f55eec7a5676/src/Confluent.SchemaRegistry.Serdes.Avro/SpecificDeserializerImpl.cs#L54-L57
I think that this must be a bug.
The code asserts ISpecificRecord but then goes and reads some random field on the type that has nothing to do with the interface. Which is not really correct conceptually, and doesn't work with all types that legally implement ISpecificRecord but do not expose that field.
Since the deserialiser already requires a parameterless constructor, maybe it'd be cleaner to once create an empty instance of the type and read ISpecificRecord.Schema from it? (this is what Protobuf does).

I noticed that the Serialiser is doing the same, which is a subject of the same bug:
https://github.com/confluentinc/confluent-kafka-dotnet/blob/376a3250c06f347c6492e90b9921f55eec7a5676/src/Confluent.SchemaRegistry.Serdes.Avro/SpecificSerializerImpl.cs#L124-L127


validate_assignment=False has no effect

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
It appears that setting model_config = Config(validate_assignment=False) doesn't work. Reprex below:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/private/tmp/test/test.py", line 11, in <module>
    settings = Settings(a=None)
               ^^^^^^^^^^^^^^^^
  File "/private/tmp/test/.venv/lib/python3.11/site-packages/pydantic/main.py", line 192, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
a
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.8/v/string_type

Example Code
from pydantic import BaseModel, ConfigDict

class Settings(BaseModel):
    a: str = "foo"

    model_config = ConfigDict(validate_assignment=False)

if __name__ == "__main__":
    settings = Settings(a=None)

Python, Pydantic & OS Version
pydantic version: 2.8.0
        pydantic-core version: 2.20.0
          pydantic-core build: profile=release pgo=true
                 install path: /private/tmp/test/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.9 (main, Apr 15 2024, 17:28:11) [Clang 17.0.6 ]
                     platform: macOS-14.4.1-arm64-arm-64bit
             related packages: pydantic-settings-2.3.4 typing_extensions-4.12.2
                       commit: unknown

@baggiponte,
Thanks for reporting this. Definitely a bug. Will roll out a fix for v2.8.1!

@baggiponte,
Thanks for your question. This is actually the expected behavior - validate_assignment refers to validation when an instance attribute is changed after initialization.
So:
from pydantic import BaseModel, ConfigDict

class Settings(BaseModel):
    a: str = "foo"

    model_config = ConfigDict(validate_assignment=False)

if __name__ == "__main__":
    settings = Settings(a=None)

Is expected to error.
from pydantic import BaseModel, ConfigDict

class Settings(BaseModel):
    a: str = "foo"

    model_config = ConfigDict(validate_assignment=False)

settings = Settings()
print(repr(settings))
#> Settings(a='foo')

settings.a = None
print(repr(settings))
#> Settings(a=None)

Doesn't fail, because the assignment isn't validated.

Oh shoot, thanks! Is there a way I can disable validation on instance?


Remove the AppArmor only when the injector package is removed

What does this PR do?
Remove the AppArmor only when the injector package is removed, not when the instrumentation is disabled.
It was causing issues because the AppArmor profile was not restored when the instrumentation was re-enabled.
AppArmor support was added in https://github.com/DataDog/datadog-agent/pull/29697
Motivation
Describe how to test/QA your changes
Tested manually on a vm & by E2E tests
Possible Drawbacks / Trade-offs
Additional Notes

/merge


"No Access-Control-Allow-Origin" when using node-auth in browserify

I've had node-oauth working nicely in a node/express environment, but now I'm trying to integrate it into a single page app using browserify. Using the same API endpoints that worked correctly, I'm getting the following error in dev tools:
Fetch API cannot load http://mytestapi.com. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8000' is therefore not allowed access. The response had HTTP status code 400

This might well be related to this issue as the problem might well go away if I could set the Access-Control-Allow-Origin header to "*"

It seems like module was created to work with https://github.com/substack/http-browserify but it is outdated replaced by https://www.npmjs.com/package/stream-http . There is the same method which generates request object, but API is different.


Python 3 support

python 3 is out for almost 10 years, please don't release python 2 only libraries.

+1

So sad. I'm a Python 3 user but this looks awesome!

Let me test really quick. Thanks for the quick PR.


fix NPE when a result is Null

When an attribute in JmxResultProcessor#attributes  is null, it throws NPE and leads to other attributes missing。So，we should judge whether the current attribute is null or not. If current attribute is null, just ignore it.

In my company's practice, this commit is useful.


support trigger and diagnosis ops

support trigger add and list
support diagnosis add and list
update web logo

lgtm


Auto Login not worked when login by access token

What steps will reproduce the problem?
Login by access token.
What is the expected result?
auto login by cookie.
What do you get instead?
auto login not worked.
I found the reason.
The identity and duration is not written into the cookie when login by access token.
Since the cookie is written by switchIdentity function in yii/web/user.php, but it is not applied in LoginByAccessToken function?

It shouldn't work. the propose of token based authentication is to not use session and being able to remove it completely so you can build a stateless environnement like RESTful Web Service APIs. see docs for more info.


Question: Is there an example for handling payments somewhere ?

I am trying to understand how to proceed after sending an invoice to a user, but the process using this library is not very clear to me.

Found out myself, I needed to do b.Accept(*PrecheckoutQuery)

@alkanna, how did you handle PrecheckoutQuery?

It's been a while, but I had to look for it in the source of this lib to
get it to work, sorry I don't really have anything related to this anymore.

I just posted my question and found out answer myself, it's just b.Handle(telebot.OnCheckout,...
@alkanna thank you for the reply


Update FUNDING.yml

Description

Checklist
~- [ ] My changes follow the contributing guidelines of this project.~
~- [ ] I have performed a self-review of my code.~
~- [ ] I have made corresponding changes to the documentation.~
~- [ ] I have added tests that prove my fix is effective or that my changes work.~
~- [ ] New and existing unit tests pass locally with my changes.~

Coverage: 94.7%. Remained the same when pulling 142a86875f98cf492e6f2ca3f1644f6e54ea9e2e on edno-patch-1 into dcf14ea0fcc0ef5c764a943c5c6b8d0fe6b29a25 on main.


Paper 5a: Real-Time Middlewarte for Cyber-Physical Event Processing

Please add feedback on the paper

Reviewer: Akinori Kahata
Review type: Comprehensive

The problem being solved

As cyber-physical systems connected with many sensors and actuators, the systems receive many inputs and processes many events concurrently based on the priorities of each process.
Because the priorities and processes of each event are diverse, event processing management of cyber-physical systems is difficult but important for the quality of the systems’ output.
Existing stream processing software cannot provide prioritized processing and share with time consistency; then, the authors try to develop the software which can manage the event processes based on priorities of each process.

The main contributions

The paper introduces new type software for event processing, “CPEP”, which can manage the event with four features; configurable processing operations, processing prioritization and sharing, enforcement of temporal validity and shedding, and efficient concurrent processing.
As a result of the empirical experiments, CPEP with TAO software combination is superior to Apache Flink software from perspectives; the total timely-event latency, thorough put, effectiveness of using memory.

Questions

What is happen when the high-quality event comes to the event processor a few milliseconds after the middle-quality event occurred? I mean, some processes should be processed concurrently, but the start timing of the event is not the same. I cannot understand the algorithm on how to decide the real priority.
The differences between CPEP and existing software, such as Apache Flink, are not apparent for me. In §2, there is an explanation about Apache Flink has not featured of time consistency. However, I cannot understand other differences. Then my question is, what is the most significant difference with Apache Flink. In other words, what is the most difficult challenge of CPEP development?

Reviewer: Lily Shpak
Reviewer Type: Critical
Main Problem Being Solved
For systems that have both hardware and software components, real time data processing is essential. It is important for data to be processed in real time so it can be actuated correctly. Another issue they are attempting to solve is making sure that new and old data do not get processed together.
Main Contributions
The authors solve this issue by creating a system that they call Cyber-physical Event Processing (CPEP). A main attribute of this system is that it separates the event processing into three levels; low, medium, and high priority. This separation allows events to be processed concurrently.
Questions

The authors implemented this system within TAO, which they say is a widely used middleware, if someone using this system wanted to use something other than TAO would they be able to?
They bring up related work, but what sets apart their research from research that has already been done?
They process different priorities separately but then how do they make sure that after these events are processed the low priority processes can run at the same time as high processes?

Critiques

I think that they could have explained the related work at the end of the paper. It confused me a bit because they introduce their research and then they jump right into related work instead of fully explaining their work.
I do like that they have all these experiments that they ran to test their system. I think it is good that they laid out each experiment and what they were testing.

Reviewer: Rick Sear
Review Type: Comprehension
Problem being solved
The authors say there are 4 issues to do with CPS systems:

Many little events that build together into complex operations
Event prioritization
Time consistency and enforcing time validity of events
Concurrent data processing while maintaining efficiency

Important contributions
The paper introduces the CPEP middleware system which attempts to be customizable depending on use case and provide a good (space-efficient, fast, accurate) solution to all four of these issues.
Questions

The paper runs performance metrics against Apache Flink. Not knowing much about middleware, is this a fair comparison? Are there other middleware platforms besides Flink which might outperform CPEP?
CPEP is nice because it's very configurable. Experiment Set 4 compares some combinations of its cross-priority sharing option and its event shedding option. I'm wondering what examples of actual contexts might need these different combinations? Is one much more prevalent than the others?
How reliant is CPEP on using the TAO software (i.e. could I easily implement it in something other than TAO if I wanted to)?

Reviewer: Graham Schock
Review Type: Comprehensive
Problem being solved
Constraints are important in cyber physical systems. It is vital to the success of a cyber physical system that we are able to process events efficiently, safely and within the time constraints necessary to the application of the physical system. The paper gives an example of an electric grid that requires a latency time to be less than 50ms. To make things worse multiple sensor integration is required for certain applications which adds more stages and layers of complexity.
Main Contributions
In order to meet this need the paper introduces CPEP which stands for cyber physical event processing. This CPEP system allows for configurable processing operations (which allow for simple and complex event processing), a level of prioritization and resource sharing which allow for important events to be processed sooner, an enforcement of constraints and removal of outdated data, and concurrent processing which leads to more efficiency.
Questions

How do we determine what is high priority in a system like this? It seems the middle ware relies on a configuration file for reliable information. Can this configuration file always be trusted?

It seems security was not discussed much. I wonder how we can add to our middle ware to ensure that everything is safe and secure and what trade offs that has?
_

In figure 2 it talks about how we can generate priority through event processing. It also talks about "operators" and how execution of an operator produces an event. What is an "operator" and how does it create an event?

Reviewer: Michael Hegarty
Review Type: Comprehensive
Problem Being Solved
Cyber-Physical systems are becoming more and more complicated in terms of large numbers of inputs and outputs, and the data that needs to be processed often has complex operations that might involve multiple inputs such as sensor combination. In addition some inputs have higher priorities than others and should be processed before lower priority ones.  In addition events have absolute timing constraints they need to meet(the real world system needs to respond to an event within a certain amount of time otherwise the data isn't relevant anymore) as well as relative timing constraints(if 2 inputs are used together to decide something they need to happen at around the same time).
Main Contributions
The authors create a middle ware to solve these problems called CPEP. It is implemented off of a middle ware called TAO to create a configurable system capable of processing events with different levels of priority(maintained via preemption) while maximizing concurrency of processing. In addition the system strives to maintain absolute and relative time consistency between events and can validate if these constraints are met or not.
Questions

I understand the concept of a system with multiple inputs and outputs but I was wondering what real world examples of this would look like? Is this something that would be on an embedded linux system?
The system abstracts the idea of sampling the input data sources by having suppliers that give it the data. Are there trade offs to a system that does this instead of one that includes the timing requirements to sample these data steams at their various frequencies?
The idea of checking the timing validity of an event and marking/discarding it if doesn't make those requirements is interesting to me. Is this level of unreliability of acceptable in most IoT systems?

Reviewer: Henry Jaensch
Review Type: Comprehension Review
Problem Being Solved
Cyber physical systems pose unique challenge in the diversity of components that need to work together to achieve a task. Physical sensors all generate different kinds of data at different periods. A CPS is also usually responsible for completing more than one task. The more complexity added means that more work needs to be done to guarantee any constriant. A wide variety of sensor events and actuator requirements paired with variable task prioritization can result in a complex system with many points of failure.
Main Contributions
This paper proposes a system that prioritizes event based processing called CPEP. This system uses the idea of an event to compartmentalize processing and assign priority. Events can vary greatly in complexity and timliness. CPEP attempts to maximize concurrency between events to improve the efficiency of the system while still meeting deadlines and maintaining proper prioritization.
Questions

The priority section discusses handling cross-priority sharing and how individual threads are assigned adjacent priorities, but how does this system handle concurrency issues created by different processes of differing priorities interact with the same memory?

The system was built on top of TAO middleware and designed to emulate some of the existing TAO mechanisms. Without the constraints of attempting backwards compatability would CPEP be implemented in a different way?

Reviewer: Ratnadeep Bhattacharya
Review Type: Critical
Problem Being Solved:
In real time event processing for cyber-physical systems (CPS), events need to be processed in a timely manner and with regard to temporal validity of the data being used. This work implements a middleware for the same that reduces redundant work, maintains temporal validity of data and provides a way for the system to accommodate user level priorities.
Contributions
The main contribution of this work is that it provides a middleware that can combine simple operations to build up more complex workflows. It improves over message passing systems like Kafka, which does not provide end-user defined priority based processing, over Data Distribution Service, which does not process events and over Flink, which does not provide temporal consistency.
Figure 2, 3 and 4 in the paper provide an excellent overview of the design of the system. The overall design is a directed acyclic graph of operators.  Any operator might receive events from multiple operators (upstream). Any operator can be scheduled once all of its input events have arrived (upstream operators have completed their job).
Equation 1 describes the event's lifetime. Equations 2 and 3 describe absolute validity of internal events (events produced by upstream operators and not suppliers). These flow down from the event interval constraints of the supplier events. Similarly, relative time consistency is defined by Equations 4 and 5.
The workers itself can be divided into multiple priorities. Priority level of operators flow in the reverse direction than that of timing consistencies. That is, consumers provide priority levels for each end operator. Any upstream operator is given the highest priority amongst the priorities of its downstream operators.
Each priority level has its own event processor which in turn has a bunch of worker threads and a bunch of mover threads. There are also three queues - InputQ -> buffers all supplier events at the same priority level, PendingQ -> holds events for the downstream operators, MovingQ -> for cross priority sharing. When cross-priority sharing is required, the worker thread that produced the event places it in the PendingQ and any free mover thread places it in the MovingQ.
Questions

The system does not bind threads to CPUs. This is good for resource utilization but how much context switching is happening?
This is a multi-threaded event processing system with a bunch of data structures per process. How does it scale? What kind of coordination is required and how does it slow the system down when using multiple CPUs?

Reviewer: Pat Cody
Review Type: Comprehensive
Problem Being Solved
Cyber-physical event processing is difficult because applications may have different temporal and latency requirements. Furthermore, event processing operations have varying degrees of complexity, and having these operations work concurrently is challenging.
Main Contributions
This paper introduces CPEP, a middleware for cyber-physical event processing designed to address the 4 problems described in the introduction. CPEP provides relative and absolute time consistency for events, and schedules based on event priorities.
Questions

If the system is designed to handle multiple applications, what kind of security/isolation does this provide, if any?
How do the authors envision deploying this system to the real-world?
The authors provide some base discussion on making the system distributed. How/when do they envision this being useful, and what kind of tradeoffs would this present?

Reviewer: Tuhina Dasgupta
Review Type: Comprehension
Problem:
The author identifies four primary issues with cyber-physical systems (event prioritization, time validity, concurrent data operations, and complexity of the system)
Contribution:
They propose a solution, called CPEP, which is a customizable middleware system that can solve all of the above-identified issues while having good space & time complexity by maximizing concurrency.
Questions:

After doing some basic research (Googling) there seems to be a vast number of types of middleware systems. The only comparison made in the paper is against Apache Flink. What this because Apache is the most comparable or for some other reason?
Surprisingly, this paper is one of the few so far to not address security and privacy issues. Are there any with this system?
I understand that the system is multi-threaded but don't fully understand the cost of switching between flows of execution. What's the overhead like?

@AkinoriKahata, Akinori Kahata, Comprehension:

Summary: Aki questions about the scheduling algorithm as a whole and how it handles scheduling high and low priority tasks. He also inquires about the usefulness in comparing CPEP to Apache Flink. Aki shows clear understanding of the benefits that CPEP has over other real-time middle-wares, such as prioritization with time scheduling.
As far as I’m able to gather, this system simply provides a conduit for task scheduling, such as the supplier, worker, and mover threads. It does not reinvent any core scheduling properties and uses a fixed priority\algorithm. I believe that a higher-priority event that comes later than a lower-priority event would still be treated as more important. Such high priority events tend to be time-sensitive where missed deadlines are unacceptable. CPEP gives me every reason to believe that the system will still function as such.

@lrshpak, Lily Shpak, Critical:

Lily brings up the middleware TAO and the complexity of moving to a different middleware framework. She asks about how CPEP handles processes running at the same time in different priority levels. Lily shows an understanding of the benefits of a real-time middleware such as CPEP and Flink, and brings up valid concerns over its integrity.
The data structures used in CPEP are native to C, which is a very versatile language that can run miles on embedded, low memory, and power constrained systems. TAO is an open-source software, probably a good motivation as to why CPEP decided to work with it, but the beauty of what CPEP outlines is its core concepts, which seem independent of platform. The threading and hierarchy model used in CPEP should be portable to other technologies. Still, it would have been interesting to see speed comparisons using another middleware, although presents another huge layer of complexity.
Each Event Level has separate worker threads that can run concurrently, and furthermore multiple worker threads within these levels can be initialized.

@searri, Rick Sear, Comprehension

Rick brings up the four main features of a real-time data streaming middleware, as well as many valid concerns for how the platform is managed. How CPEP falls into TAO was another concern brought up here.
Apache Flink is one of the leading data streaming platforms, providing high throughput at super low latencies. This is absolutely a fair comparison, as it seems CPEP was built to directly compete with this platform. If CPEP can really perform as well as its graphs indicate, it could provide quite a useful service.

@grahamschock, Graham Schock, Comprehension

Graham takes a unique mindset on this paper, bringing up the time constraints of CPEP. The time constraints are one of the most important factors that a middleware needs to address, and Graham nails it on the head when he says that multiple sensors causes greater complexity.
For large scale data streaming applications that would want to utilize CPEP or Apache Flink, it is assumed that a hierarchy of these priorities are thoroughly specified by the administrator. I would assume these would be well thought out, but CPEP is completely reliant on this to run properly.
Good question. Right off the bat, a huge security concern for any scheduling platform is making sure that priorities aren’t starved. Low priority does not mean “run if possible.” All events need to be serviced. Unfortunately, this paper does little to clarify this issue, and makes use of Fixed Priority Scheduling, which by nature does not protect against starvation.
I see “operator” as an extension of “event.” Imagine an operator is a thread running Fourier Transform on a set of data. This operator gets passed data, and starts going to work on one of the worker threads. This then becomes and “event,” running a process on said data. Operators are separated by types of actions occurring, and events are literally the process. Make sense?

@mjhegarty, Michael Hegarty, Comprehension

Michael focuses on the timing portion of this paper, such as the relative and absolute constraints for meeting a deadline. He also wonders about the real-world applications of needing such a scheduling system.
Good question! Any embedded system that needs fasting-acting processing and low latency can make use of CPEP essentially. Image processing, sensor readings of various kinds. All of these readings are likely coming from distributed devices that work together to make some kind of decision. Having faster throughput and higher microprocessor efficiency can make these decisions happen faster based on a hierarchy of priorities. It may take a memory hit when sharing events between priorities (although many of their graphs seem to indicate otherwise astonishingly, not sold on this one), but often times this is the price of speed.
Supplying timing boundaries is still included in CPEP’s design. It makes decisions to schedule these processes more efficiently to produce lower latency, much like Apache Flink, but it shouldn’t compromise the core principles of Fixed priority scheduling.
If data is so old that processing it would require input from sensors that are miles ahead, then it stands to reason that it was either low priority or that a ton of readings came in at the same time. It’s important that the entire system be acting in real-time, otherwise the data is useless, and could potentially result in an accident if old. Fantastic questions, really giving me some cool stuff to think about.

@hjaensch7, Henry Jaensch, Comprehension

Henry brings up the deadline constraint problem that CPEP attempts to mitigate. The interesting part of his discussion is the notion of failure, that is, the system CPEP sets up is not designed to solve the problem of scheduling, it simply gets closer to an optimal CPU efficiency, and expects some data to be shedded/failed.
Great observation! Standard C data structures are used in these threads, and as you might expect, CPEP uses built in thread locking mechanisms as well.
Yes. I assume this was done to cut down on production time to get a working product. It stands to reason that sufficient effort would be required to port this to another framework, but the concept of what CPEP does is platform independent as long as they provide the correct concurrency structures.

@ratnadeepb, Ratnadeep Bhattacharya, Critical

Deep shows a fundamental understanding of how CPEP works overall, calling in example of the thread design and how events are moved between Event Processors and how operators "trickle down" to each other to avoid operators working with expired data. He also inquires about the overhead incurred from designing such a system that does not assign processes to threads.
Woah, interesting. I assume it’s going to be huge amount of the time. If you were to assign worker threads to cores, this would sufficiently cut down on the number of switches, but I think it would fundamentally oppose what the middleware is trying to achieve, which is CPU efficiency. Fantastic observation.

@pcodes, Pat Cody, Comprehension

Pat shows understanding of how CPEP works at a high level, citing its core features. He also questions about the real-world utility of this system and how the company envisions using this at scale.
Locking mechanisms are used when editing data structures. This can be all be handled within CPEP. However, just like normal processing, there’s no guarantee that a programmer writes code that stops. Still very possible for one thread to be waiting on a broken sensor that never sends required data. Certain guards can be put in place but a level of unpredictability is expected.
If we scale this system, I believe a certain amount of latency will be incurred. Also, if this is going to be deployed on a network of sensors, safeguards against broken systems need to be put in place. As long as events are correctly taking this into account and have conditions for this, the scheduling of CPEP should take care of the rest. Certain suppliers will have higher priorities than others, but CPEP should be able to handle this regardless of user specifications.

@tuhinadasgupta, Tuhina Dasgupta, Comprehension

Tuhina brings up the main contributions CPEP has to real-time processing, and inquires about the security concerns of this system as well as middle-ware competitors.
The cost of context switching will definitely be high here, especially since worker threads are not assigned to a single CPU. This is compounded even further when event sharing is not present, as more memory is needed to compensate for Event Processors not distributing workloads efficiently.

Reviewer: Jacob Cannizzaro
Reviewer Type: Critical
Main Problem Being Solved
Cyber-physical systems absolutely need to be able to make strict deadlines. On constrained systems this can be hard to do as there are other difficulties such as starvation for threads of different priority as well as validity of data that must be kept timely. It is important to meet these other demands for all the complex events being operated on while not losing the ability to stay within low latency requirements.
Main Contributions
This paper introduces the real-time middleware for cyber-physical event processing (CPEP). This system helps to integrate the complex event processing while allowing process prioritization to effectively meet high priority thread needs while not starving lower priority ones. It also helps to maintain temporal validtity of data so that the system is not acting on outdated event information.
Questions

Can we go over the event processing stream diagrams in figure 7? I do not understand the difference between (a) and (b) and why the high priority threads are potentially jumping all the way down and appear to stay at a low priority.
Is there a more comparable framework to CPEP than Apache Flink?

Critiques

I don't know much about Apache Flink but they say that it is an open source stream processing framework. They don't really say that it is meant for middleware purposes. Looking them up, I see it is meant for event driven applications but doesn't mention edge computing or middleware, or even embedded devices. Maybe there isn't anything out there exactly like CPEP but then I think that they should have expressed that and expaned upon that as their contribution. This just seems like too easy of a comparison.
I do not have much experience with the exact subject here, but think the paper is just poorly written, as if it needed one or two more draft cycles. One blaring example is the repeated pattern of listing a related project without really describing its goals, saying what it can't do, and then: "In Contrast, CPEP" (which appears 4 times in the text). Maybe I'm being overly critical, but three of them are on the same page.


[blog] Add Material UI v6 stable release

A blog post for Material UI v6 stable release.
👉  https://deploy-preview-41932--material-ui.netlify.app/blog/material-ui-v6-sneak-peek/

[ ] I have followed (at least) the PR section of the contributing guide.

The structure looks good. We discussed previously that because the target for the beta and stable release are so close, we might skip the beta blog. I started this discussion in the GTM page and tagged you. I think we can repurpose this for the stable release.

@zanivan For the first demo that you proposed, I think it'd much simpler to use images may be with bento Grid layout. What do you think?

For the first demo that you proposed, I think it'd much simpler to use images may be with bento Grid layout. What do you think?

I think it might work too, but wouldn't it be too small space for a bento grid with templates?
Or are you suggesting a static image like this one?

Or are you suggesting a static image like this one?

👍 Something like that. I can stretch the section wider to make the image bigger.

👍 Something like that. I can stretch the section wider to make the image bigger.

Does this looks good?

👍 Something like that. I can stretch the section wider to make the image bigger.

Does this looks good?

Perfect! can you add a light mode and send me the Figma link

Here it is @siriwatknp  😊

@zanivan I will update the PR based on your comment.

@zanivan I updated the templates and custom theme preview, please check again.

@siriwatknp I added some tweaks to the demos, specially the CSS variables table—I think we can show this information in a different way, I don't like the idea of having a table like this with both vertical and horizontal scroll

@siriwatknp I added some tweaks to the demos, specially the CSS variables table—I think we can show this information in a different way, I don't like the idea of having a table like this with both vertical and horizontal scroll

What do you have in mind? The reason I use table is to show a quick view of the tokens without taking up too much space vertically, so user can skip to other sections if they want to.
The other option is to not show the preview at all.

What do you have in mind? The reason I use table is to show a quick view of the tokens without taking up too much space vertically, so user can skip to other sections if they want to.

I think it's ok for the blog post, I added a slight gradient to the bottom and right sides, so it's a bit clearer that it can be scrolled horizontally and vertically.
Maybe, we could make the width a bit larger, though, like the bento templates image.

Maybe, we could make the width a bit larger, though, like the bento templates image

Done. Does it look better to you?

Done, I use the same width as the template section. Does it look better to you?

I think users will be able to spot both the vertical and horizontal scrolling more easily. What do you think?

@prakhargupta1 @bharatkashyap I moved the Toolpad section from h3 to h2

The content here is RTM from my perspective.
Important note: Please wait until https://github.com/mui/material-ui/pull/43073 is merged to merge this one, as the upgrade guide URL changes in that PR (migrating-to-v6 -> upgrade-to-v6). After that PR is merged, we have to modify the links in this PR as well.


not support translate character encoding

if my java code contains Chinese character, then the result like this:
this.logger.error("\u6267\u884c\u7528\u6237\u6570\u636e\u540c\u6b65\u51fa\u9519")
I can't read what's it mean
so I must to translate it by myself ,my god!

That's my question too, have u resolved?


bayesian optimization draft

This is a draft for bayesian optimization (work in progress) #15 @gaocegege @YujiOshima

/assign @gaocegege @YujiOshima

/hold

Thanks for your awesome contribution! I will take a look soon.

When the PR is ready to merge, please use /hold cancel to cancel the hold.

/ok-to-test
Please rebase the master and I think the test should pass.

@libbyandhelen Please move suggestion/python/api/ directory to api/python and store your bayesian-optimization source codes under suggestion/bayesianoptimization directory in conformity to other services.

@libbyandhelen And please dockerize your suggestion service and add k8s manifest.
I'm sorry there is no nice documents. Please refer other services.
https://github.com/kubeflow/hp-tuning/blob/master/suggestion/random/Dockerfile
https://github.com/kubeflow/hp-tuning/tree/master/manifests/vizier/suggestion/random

@libbyandhelen If you have any question about dockerize your suggestion service and add k8s manifest please at me and I can help you.

@gaocegege I wrote both the dockerfile and k8s manifest. I tested the dockerfile and it works fine, but I do not know exactly how to test the k8s manifest. Could you take a look of my code to see whether it is OK?

/hold cancel

@gaocegege OK, got it.
@YujiOshima
Do you think the api proto is OK now? I did not change the protocol in api.proto.
I used SetSuggestionParameters to set all the parameters needed for the suggestion service, and StopSuggestion for deleting the relevant information of the specific study stored in the service.
completed_trials in GenerateTrials is used to pass information of trials for now, since the interface for DB is not ready.

@libbyandhelen OK, we will add DB interface for python later.
Finally, please add build and deploy test for this suggestion to test script.
Since this is in python, you don't need go build test. Only gcloud container builds submit and deploy.
https://github.com/kubeflow/hp-tuning/blob/master/test/scripts/build.sh
https://github.com/kubeflow/hp-tuning/blob/master/test/scripts/run-tests.sh

I think it is the last step! 🎉
Thanks for your contribution!

@libbyandhelen Thank you for your great contribution!
/lgtm


HTTP Publish API authentication does not work with emqttd_auth_http plugin

I'm successfully using the new emqttd_auth_http plugin for user authentication and subscribe/publish access control. However, with the plugin loaded, the HTTP Basic Authentication for the HTTP Publish API does not seem to work. Specifically, all the POST requests get rejected with 401 without any attempt to validate the HTTP user:password combination through the emqttd_auth_http plugin.
Is this a design choice? If so, how can I use the HTTP Publish API while the emqttd_auth_http plugin is loaded?

@teifip, If the emqttd_auth_http plugin is loaded, the HTTP Publish API will be authenticated by the plugin.

@emqplus, great! Thank you!


v6.5 oom, long checkpoint/resovlets lag on 50k tables

What did you do?
test 50k tables via benchbot
python3 main.py gen-benchbot-case --email<EMAIL_ADDRESS>--token 'xxx'  --testbed_size "2xl" --benchmark_id 6000048  --resource_pool benchbot-cdc --bench_type cdc_workload --bench_sub_types "bank" \
--cdc_workload_options  "-row-count 10000000 -table-count 50000 -tps 14000" \
--duration=30m --threads 64 --versions v6.1.3,v6.5.0 --tidb_replicas 4 --tikv_replicas 6 --cdc_replicas 4 \
--tikv_configs "{raftstore: {store-io-pool-size: 1}, raft-engine: {enable: true}, import: {num-threads: 128}, cdc: {min-ts-interval: "1000ms"}}" \
--cdc_configs "{per-table-memory-quota: 33554432, owner-flush-interval: "200ms", processor-flush-interval: "100ms"}" \
--sink_options "?protocol=canal-json&kafka-version=3.1.0&partition-num=6&max-message-bytes=67108864&replication-factor=1" \
--warmup False --tidb_configs "{new_collations_enabled_on_first_bootstrap: false}" \
--storage_url s3://perftest/cdc_workload_bank_50k_t_20m_r --db_name bank50k --sink_target kafka --br_options "--ddl-batch-size 64"  --staging True --description "kafka-50k-table"

[2022/12/10 16:29:29.015 +08:00] [INFO] [utils.go:353] ["tiup cdc:v6.5.0 cli changefeed create  --pd=http://pd-peer.benchbot-cdc-kafka-amd64-2xl-cdc-workload-tps-1325828-1-198:2379 --sink-uri=\"kafka://d
s-kafka-kafka-peer:9092/cdc-topic-1670660969?protocol=canal-json&kafka-version=3.1.0&partition-num=6&max-message-bytes=67108864&replication-factor=1\" --changefeed-id=\"simple-replication-task\" --no-con
firm --config=/tmp/2022-12-10T16:29:28+08:00.cf.toml"]

What did you expect to see?
no oom, log checkpoint/resolvets lag
What did you see instead?
v6.5 oom, long checkpoint and resovlets lag, which v6.1.3 is fine

Versions of the cluster
v6.5 release test

the qps on upstream cluster on v6.5 is much higher, not sure if the higher qps on v6.5 is causing CDC oom

/severity critical

/affects-6.5

is the first phase  on  v6.1.3 and the second on 6.5 ?

is the first phase on v6.1.3 and the second on 6.5 ?

yes, first phase is v6.1.3

a heap profile when the memoy usage is ~30GB

the oom is caused by timeout to connect with the pd periodically ,which would cause the processor restart.
we can set the the value as for timeout more bigger(such as 30s) as a workaround.

/remove severity/critical

with PR #8699 , cdc 6.5 can handle about 100k tables with 4 nodes


give the cluster fuzzer tests more resources

backport of https://github.com/arangodb/arangodb/pull/17446

@dothebart please re-run Jenkins.


Exception occurred when getting the state dictionary（state_dict）.

In TorchSharp, I defined a model that contains nn.Sequential cv4. However, when I obtained the state_dict of the entire model, the dictionary for cv4 was missing, which is very strange. Other models also have nn.Sequential, and they can all be correctly obtained, but not the last layer.

I suspect it's because of this API that it fails to recognize all the modules.
this.add_module(nameof(model), this.model);

When the elements of ModuleList are nn.Module<Tensor, Tensor> instead of nn.Sequential, the state_dict of the ModuleList elements can be captured correctly.

When calling model.add_module, it traverses its sub-items and calls RegisterComponents() for each sub-item. However, Sequential does not perform such an operation. As a temporary measure, in a custom model, after initialization, force the call to RegisterComponents().

Comparing Sequential and ModuleList, it can be observed that ModuleList overrides RegisterComponents, whereas Sequential does not. Because ModuleList overrides RegisterComponents, it gains the ability to automatically invoke the RegisterComponents of its child items.

Is there a smallish repro case that I can debug?

public class AModel : TorchSharp.torch.nn.Module
{
    public TorchSharp.Modules.Sequential cv1;
    public TorchSharp.Modules.ModuleList<torch.nn.Module<torch.Tensor, torch.Tensor>> cv2;
    public AModel():base(nameof(AModel))
    {
        cv1 = nn.Sequential(Enumerable.Range(0, 3).Select(x => new BModel()));
        cv2 = nn.ModuleList<torch.nn.Module<torch.Tensor, torch.Tensor>>(Enumerable.Range(0, 3).Select(x => new BModel()).ToArray());
        RegisterComponents();
    }
}

public class BModel : torch.nn.Module<torch.Tensor,torch.Tensor>
{
    public Tensor stride;
    public Tensor stride2;

    public BModel() : base(nameof(BModel))
    {
        stride = torch.ones(100, 100);
        stride2 = torch.ones(100, 100);
    }

    public override torch.Tensor forward(torch.Tensor input)
    {
        throw new NotImplementedException();
    }
}

var a = new AModel();
var aStateDict = a.state_dict();

In this case, the state_dict of cv1 cannot be obtained.
public class AModel : TorchSharp.torch.nn.Module
{
    public TorchSharp.Modules.Sequential cv1;
    public TorchSharp.Modules.ModuleList<torch.nn.Module<torch.Tensor, torch.Tensor>> cv2;
    public AModel() : base(nameof(AModel))
    {
        cv1 = nn.Sequential(Enumerable.Range(0, 3).Select(x => new BModel()));
        cv2 = nn.ModuleList<torch.nn.Module<torch.Tensor, torch.Tensor>>(Enumerable.Range(0, 3).Select(x => new BModel()).ToArray());
        RegisterComponents();
    }
}

public class BModel : torch.nn.Module<torch.Tensor, torch.Tensor>
{
    public Tensor stride;
    public Tensor stride2;

    public BModel() : base(nameof(BModel))
    {
        stride = torch.ones(100, 100);
        stride2 = torch.ones(100, 100);
        RegisterComponents();
    }

    public override torch.Tensor forward(torch.Tensor input)
    {
        throw new NotImplementedException();
    }
}

var a = new AModel();
var aStateDict = a.state_dict();

In this case, the state_dict of cv1 can be obtained.

So I guess the problem is that Sequential.RegisterComponents won't call RegisterComponents on its submodules.

Sequential and ModuleList have different implementation methods for RegisterComponents.

I suppose the issue could be simply solved by adding a call to the submodule's RegisterComponents when adding it.
Actually in my opinion, all the modules should always call RegisterComponents themselves, so there is no need to deal with the submodules (because they will do that on their own) but it seems not because even ModuleList and Sequential is not doing that...
I'm a bit confused now... I can't even use:
using static TorchSharp.torch.nn;

var l = ModuleList(Linear(1, 1));
Console.WriteLine(l.state_dict().Count); // 0

I believe this is ill designed...

 protected override void RegisterComponents()
 {
     if (_registered) return;

     for (int i = 0; i < _list.Count; i++) {
         register_module($"{i}", _list[i]);
     }
     _registered = true;
 }

This is the implementation of ModuleList, and I think adding similar code in Sequential should be able to fix it.

 protected override void RegisterComponents()
 {
     if (_registered) return;

     for (int i = 0; i < _list.Count; i++) {
         register_module($"{i}", _list[i]);
     }
     _registered = true;
 }

This is the implementation of ModuleList, and I think adding similar code in Sequential should be able to fix it.

However you will find that it's unable to use var l = ModuleList(BModel()).state_dict();, since RegisterComponents of Linear will not be called by itself, so your models' RegisterComponents is also not invoked. That's because Linear allows models to be dynamically appended, so we have to register them dynamically, instead of calling RegisterComponents only once.
So one solution might be to call the submodule's RegisterComponents in Sequential.Add. However it might make RegisterComponents be called too early, especially when the submodules are also mutable. So I'm not sure what the expected behavior should be.
And let me repeat my suggestion, perhaps all the modules should always call RegisterComponents themselves, and the parent modules should not care about the submodules.

We can call RegisterComponents once in the top-level model, so that other models will be registered automatically, which is the most convenient.

I wonder if this could serve as a relatively good solution.

To facilitate the use of pre-trained weights in PyTorch, it is advisable to maintain consistency with PyTorch as much as possible.

To facilitate the use of pre-trained weights in PyTorch, it is advisable to maintain consistency with PyTorch as much as possible.

That is what I mean. A module should register the parameters by themselves. In PyTorch it is done by __setattr__. However it's impossible for csharp. Perhaps we need a source generator? So complicated it could be :(

I'm not sure if this is feasible or not, but the idea is to call RegisterComponents within the parameterless constructor of nn.Module. This way, when you create a custom model that inherits from nn.Module, it will automatically register itself.

I'm not sure if this is feasible or not, but the idea is to call RegisterComponents within the parameterless constructor of nn.Module. This way, when you create a custom model that inherits from nn.Module, it will automatically register itself.

Unfortunately, that's impossible. The constructor of nn.Module is runned before the constructor of your model. Thus no values have been assigned to the properties and fields, so we can't register them...

Switching gears, we could declare properties and then mark them with a custom attribute that has a “name” which will be used as the registration name. Subsequently, we could employ Fody to inject code into the getter and setter methods to handle the registration process. However, this approach is somewhat cumbersome.

Yes I suppose Fody/SourceGenerator could be a beautiful solution.
@NiklasGustafsson Could you please take a look at this?

I have made a demo here: https://github.com/yueyinqiu/TorchSharp.AutoRegister

I'm not sure if this is feasible or not, but the idea is to call RegisterComponents within the parameterless constructor of nn.Module. This way, when you create a custom model that inherits from nn.Module, it will automatically register itself.

Unfortunately, that's impossible. The constructor of nn.Module is runned before the constructor of your model. Thus no values have been assigned to the properties and fields, so we can't register them...

That is exactly right. That's why RegisterComponents exists and needs to be called last in the (custom) module constructor.

Switching gears, we could declare properties and then mark them with a custom attribute that has a “name” which will be used as the registration name. Subsequently, we could employ Fody to inject code into the getter and setter methods to handle the registration process. However, this approach is somewhat cumbersome.

That capability already exists. For example, in the rewrite we're working on for some of the standard modules, which will enable more attributes to be exposed, the parameters of Linear are defined as:
            const string WeightComponentName = nameof(weight);
            const string BiasComponentName = nameof(bias);

            public Parameter? bias {
                get => _bias;
                set {
                    _bias?.Dispose();
                    _bias = value?.DetachFromDisposeScope() as Parameter;
                    ConditionallyRegisterParameter(BiasComponentName, _bias);
                }
            }

            public Parameter weight {
                get => _weight!;
                set {
                    if (value is null) throw new ArgumentNullException(nameof(weight));
                    if (value.Handle != _weight?.Handle) {
                        _weight?.Dispose();
                        _weight = (value.DetachFromDisposeScope() as Parameter)!;
                        ConditionallyRegisterParameter(WeightComponentName, _weight);
                    }
                }
            }

            [ComponentName(Name = BiasComponentName)]
            private Parameter? _bias;
            [ComponentName(Name = WeightComponentName)]
            private Parameter? _weight;

Perhaps all the modules should always RegisterComponents themselves, and the parent modules should not care about registering the sub-submodules of its submodules (that is, not to call RegisterComponents on its submodules).

Yes, this is exactly the intended protocol, and the documentation says so:
https://github.com/dotnet/TorchSharp/wiki/Creating-Your-Own-TorchSharp-Modules
The exception from this rule will be modules (such as the Linear module shown above, where the parameters may be altered, and the property setter needs to conditionally register a component, which allows you to assign it 'null' as well as overwrite an already existing parameter.

I have made a simplified demo here: https://github.com/yueyinqiu/TorchSharp.AutoRegister
PS: It's still impossible to get rid of the traditional constructors (and use a primary constructor instead), because we have to access the generated property. So sad :(

As much as I dislike relying on reflection, which the current scheme does (I dislike it because it prevents AOT), having to use source code generation adds complexity and something that has to be automated. That would be a last resort, I think.
The current scheme works fairly well as long as you follow the instructions very closely and don't so advanced stuff like the Linear module above. I don't know why you would allow setting the parameters after the module has been constructed, but PyTorch does, so TorchSharp should, too.

So, to address the issue of Sequential not registering its sub-modules, for the time being, should we also rewrite Sequential’s RegisterComponents, just like we did with ModuleList?

Perhaps all the modules should always RegisterComponents themselves, and the parent modules should not care about registering the sub-submodules of its submodules (that is, not to call RegisterComponents on its submodules).

Yes, this is exactly the intended protocol, and the documentation says so: https://github.com/dotnet/TorchSharp/wiki/Creating-Your-Own-TorchSharp-Modules
The exception from this rule will be modules (such as the Linear module shown above, where the parameters may be altered, and the property setter needs to conditionally register a component, which allows you to assign it 'null' as well as overwrite an already existing parameter.

So it is my understanding that custom modules should not relies on others calling its RegisterComponents. But why we are doing that in register_module?

So, to address the issue of Sequential not registering its sub-modules, for the time being, should we also rewrite Sequential’s RegisterComponents, just like we did with ModuleList?

We can certainly reconsider the protocol for module registration for the future. However, if the guidelines for custom modules described in the Wiki article are followed, the current protocol works.

[...] why we are doing that in register_module?

As unsatisfying as this answer is -- I don't recall.

haha... Perhaps we should check all the modules provided by TorchSharp, and remove this call if nothing depends on that? I believe this should be done as early as possible, to avoid more projects' relying on it by mistake.

oh well there is at least one thing (ModuleList) that depends on that:
using static TorchSharp.torch.nn;

var l = ModuleList(Linear(1, 1));
Console.WriteLine(l.state_dict().Count); // 0

So, to address the issue of Sequential not registering its sub-modules, for the time being, should we also rewrite Sequential’s RegisterComponents, just like we did with ModuleList?

We can certainly reconsider the protocol for module registration for the future. However, if the guidelines for custom modules described in the Wiki article are followed, the current protocol works.

Now it’s calling RegisterComponents() in the custom module, it just feels a bit verbose, haha.


Support of transposed convolutions through pixel padding

Gradually adding classes and functions to support the "pixel padding" implementation of transposed convolutions.

[x] Custom_op for PixelPadding nodes
[x] Test bench to validate the full deconvolution (PixelPadding +Conv) [Ongoing]
[x] InferPixelPadding transformation to convert ONNX ConvTranspose node to FMPadding_Pixel + (optional) FMPadding_Batch + ConvInpGen + MVAU
[x] Test bench using InferPixelPadding transformation instead of manual graph creation.

Thanks @auphelia for taking the time to review this and sorry if there were things I forgot. I don't know which tests you are talking about but don't hesitate to ask if you need help running them.


[ELE-585] on-run-end hook crashes on Redshift in concurrent dbt runs

Describe the problem
We override the dbt artifacts at the on-run-end hook. If there are multiple dbt invocations that run concurrently, hence the artifacts attempt to upload at the same time, this transaction might run concurrently and cause the following error: "Serializable isolation violation on table in Redshift."
Describe the solution
It seems that adding a lock to the transaction would solve the issue.
Additional context

Thread 1
Thread 2

From SyncLinear.com | ELE-585

Hey @elongl, first of all, thanks for tackling this, we are happy users of the package but we've experienced this issue.
We upgraded it to the last version (v0.7.4.) since it contains the https://github.com/elementary-data/dbt-data-reliability/pull/298 fix. However, even with this version, we've still faced the same issue.
Shall I create an issue for this?
Just in case it helps, we're using dbt 1.2.5.
Thanks!


Makefile - install golangci-lint and codespell tools

Adding codespell and golangci-lint installation to Makefile.
Signed-off-by: Navid Yaghoobi<EMAIL_ADDRESS>

LGTM


Require nasm for AV1 decoding, unless on arm64 macOS

What

Closes https://github.com/rerun-io/rerun/issues/7669

Closes https://github.com/rerun-io/rerun/issues/7671

Disable AV1 decoding unless the nasm feature is turned on

Turn on the nasm feature when building pre-built binaries

Print enabled video features with rerun --version to make testing of the above easy

Checklist

[x] I have read and agree to Contributor Guide and the Code of Conduct
[x] I've included a screenshot or gif (if applicable)
[x] I have tested the web demo (if applicable):

Using examples from latest main build: rerun.io/viewer
Using full set of examples from nightly build: rerun.io/viewer

[x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
[x] If applicable, add a new check to the release checklist!
[x] If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

PR Build Summary
Recent benchmark results
Wasm size tracking

To run all checks from main, comment on the PR with @rerun-bot full-check.

@rerun-bot full-check

@rerun-bot full-check


new feature: Line lengths and auto-wrapping

This PR is a work in progress towards some new scalariform rules to help auto-format overly long lines.
I've started by introducing two new settings: "rightMargin" (default 100) and "reflowComments" (default "true").
So far this will reflow Scaladoc style comments if they are over 100 chars wide.
This works and is tested, so is ready to merge now if you like, or fine to wait for the extra features discussed below.
It includes #57 and #58 as it would conflict if submitted separately.
I'd like to add support for reflowing / autobreaking of:

"//" style comments
string literals, i.e.

  val example = "a very long string literal, for example an error message or simliar"
  // would become:
  val example = "a very long string literal, " +
    "for example an error message or simliar"

method calls:

  foo.example(example1, example2, example3, example4, example5, example6, example7)
  // would become:
  foo.example(
    example1,
    example2,
    example3,
    example4,
    example5,
    example6,
    example7)

Please could you let me know:

If you are happy with the proposed new features, and if you'd merge them in if I were to finish up and submit them here
If you are happy with the changes so far
If you are happy with these new features to be on by default

Thanks,
Rich

Is the rightMargin feature intended to completely format anything that is over 100 characters onto multiple lines, or is it going to be scoped to strings, comments, and method calls? How will the method call reformatting work relative to the alignMethodCalls feature, which can shorten a line indirectly by placing arguments on multiple lines? One resolution to this may be to perform the reflowing first, and then have the behavior of alignMethodCalls kick in afterwards.

It might be simpler to just always reflow comments instead of having it be an option -- my experience with having extra scalariform options is that it makes maintaining the codebase more difficult when trying to combine and merge the behavior of overlapping features. What kind of scenario would someone want the rest of their code formatted to be less than 100 characters, but have their comments be longer?

I am okay with these features being on by default.

Is the rightMargin feature intended to completely format anything that is over 100 characters onto multiple lines, or is it going to be scoped to strings, comments, and method calls?

I was planning to have a single int-valued "rightMargin" setting and several boolean-valued settings like "reflow strings", "reflow comments", "reflow method calls" etc.
Do you think we need the option to have different margins for strings, comments and method calls? How might that config look?

How will the method call reformatting work relative to the alignMethodCalls feature, which can shorten a line indirectly by placing arguments on multiple lines? One resolution to this may be to perform the reflowing first, and then have the behavior of alignMethodCalls kick in afterwards.

I'm not sure yet. I was expecting to do all the other formatting first, then add extra breaks only when the line was over-long. After adding extra breaks, I'd then re-apply all the normal formatting.
For example, a one-line method call like
foo(a,b,c)

might get reformatted to just add spaces:
foo(a, b, c)

But only if it went over the right margin would the "reflow method calls" format come into play:
foo(
  a,
  b,
  c)

and then that "reflown" method call would need reformatting according to the current preferences like "alignArguments".

It might be simpler to just always reflow comments instead of having it be an option -- my experience with having extra scalariform options is that it makes maintaining the codebase more difficult when trying to combine and merge the behavior of overlapping features.

Currently it's well isolated, so it's easy to have an on/off flag. I think this is likely to stay the same, as there will always be a single "entry point" to the reflowing where the current text is compared to the current margin.
I think it's likely to be a setting where individual preferences will differ. Some people are very keen on strict 80-char lines, and some people think that's pedantic nonsense and are happy with 150-char lines. It's also a more invasive reformatting than many of the rules. IMO this is one that does need an option to control it.

What kind of scenario would someone want the rest of their code formatted to be less than 100 characters, but have their comments be longer?

My plan was to have all the "reflow" formatters controlled by a single "rightMargin" setting, so I think this question doesn't arise, unless I've misunderstood you?

I am okay with these features being on by default.

OK, thanks.
You didn't directly address "if you'd merge them in if I were to finish up and submit them here", but it sounds like you would?

(I've added support for reflowing "//" style comments to the PR. I'm still working on the other TODOs listed above.)

Great feature addition.  Can you add a setting for the closing paren placement?
I would love to see the outcome of a split turn out to be
foo(
a,
b,
c
)

Can you add a setting for the closing paren placement?

I agree that this needs a setting.
I think that this should be covered by #29 and should be independent of any changes here (although we'd need to ensure that they work together).

Interesting prior work re auto-breaking lines and how hard it is:

http://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/
https://news.ycombinator.com/item?id=10195091

Hey @RichardBradley, scalariform has new maintainers. Are you interested in bringing this PR back to live? You would have to rebase against the master branch of https://github.com/scala-ide/scalariform

I'm not working on Scala at the moment (boo), but I might come back to this at some point in the future. My work so far is here if anyone wants to pick this up or use parts of it.
I'll reopen at that location if I return to this.
I think this would be a very valuable feature to some users (but not all), but after reading the posts I linked to above, I'm worried that this might be very difficult or impossible to get right in all cases. Reflowing comments (the work so far) is useful and more approachable than reflowing code.


pre-commit: Add hook to check metaschemas

Add a pre-commit hook to check that JSON schemas are correct.
Sorry for this piddling little PR, but I thought this should be submitted by itself.

It only validates the actual schemas themselves are correctly formed (i.e. the files in schemas/). There's actually a schema for validating JSON schemas so it can use that.
We probably also want to verify that our input files themselves work with our schemas at some stage.


Add support for global common properties on each request

Hi,
I just added a new method on the service to allow changing a set of common properties after the service has been configured so we can track so custom data. Please review my commit and let me know what other improvements need to be done.
Best,
Lucian

Thanks lucassp for the contribution. This is a great idea. I will merge it into the master branch soon.


EPN3D.json

Hello, could you please provide the EPN3D.json file? I encountered a lack of this file during runtime. Thank you!

added

thank you very much！

Received, thank you very much！
------------------ 原始邮件 ------------------
发件人:                                                                                "CuiRuikai/Latent-Transport-UPCN"                                                                                @.>;
发送时间: 2023年6月27日(星期二) 中午1:33
@.>;
<EMAIL_ADDRESS>主题: Re: [CuiRuikai/Latent-Transport-UPCN] EPN3D.json (Issue #4)
added
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>