CLOUDSTACK-9067 - As I developer I want to remove all the unused router-shell scripts from ACS

This PR removes the unused shell scripts that were present in the ACS project. Those script were replaced by the.
Some of the scripts are used by the HyperV Resource, which were hardcoded. I took the opportunity to use the Java constants over there as well, so the next one touching the code will know they exist and won't hardcode anything.
The following task were applied:

Remove the shell files and the Java constants that were mapping them;
Apply the use of the Java constants to the HyperV Resource class;
Wrap the String.format() method in the StringUtils so we can test the changes in the HyperV Resource class.

The last point was added because I do not have a HyperV test environment. Hence, I wanted to make sure the tiny code I changed is covered at least by unit tests.

Ping @remibergsma @miguelaferreira @karuturi @DaanHoogland @borisroman
Partial tests results, but I will still run some tests against XenServer 6.2
Cheers,
Wilder
:: Environment 1 ::

Hardware required: TRUE
Management Server + MySQL on CentOS 7.1
One KVM Host on CentOS 7.1
Agent + Common RPMs built from source

:: Tests Suites Executed ::
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-kvm1-ISOLATED.cfg -s -a tags=advanced,required_hardware=true component/test_vpc_redundant.py component/test_routers_iptables_default_policy.py component/test_routers_network_ops.py component/test_vpc_router_nics.py component/test_password_server.py component/test_router_dhcphosts.py smoke/test_loadbalance.py smoke/test_internal_lb.py smoke/test_ssvm.py smoke/test_vpc_vpn.py smoke/test_network.py

:: Environment 2 ::

Hardware required: FALSE
Management Server + MySQL on CentOS 7.1
One KVM Host on CentOS 7.1
Agent + Common RPMs built from source

:: Tests Suites Executed ::
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-kvm1-ISOLATED.cfg -s -a tags=advanced,required_hardware=false smoke/test_routers.py smoke/test_reset_vm_on_reboot.py smoke/test_vm_life_cycle.py component/test_vpc_routers.py smoke/test_service_offerings.py component/test_vpc_offerings.py smoke/test_network_acl.py smoke/test_privategw_acl.py smoke/test_network.py

:: Summary ::

Tests executes: 75
Successfull tests: 74
Skipped tests: 1(*)
Failed tests: 0

(*) VM migration, I had only 1 host.
:: Test results for Environment 1 ::
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_isolate_network_FW_PF_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_isolate_network_FW_PF_default_routes_egress_false | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 33 tests in 14862.694s

OK
(END)

:: Test results for Environment 2 ::
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... SKIP: At least two hosts should be present in the zone for migration
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 42 tests in 7912.350s

OK (SKIP=1)
(END)

less is more! reviewed java code only, didn't check the removed scripts. LGTM

@miguelaferreira discussed with me the over-engineering wrapper method just to add 1 test and we agreed on getting rid of it for the following reasons:

It could lead other committers to add simple tests - like the one I added - and forget about the real tests that we need;
An experienced Software Engineer reviewing the code will spot problems with the string formats used.

I will retest the PR and post the results here.
Cheers,
Wilder

good points, I do want to have a more generic way of creating complex log messages though. might be useful to construct womething like this for it anyway.

Ping @DaanHoogland @remibergsma @miguelaferreira @bhaisaab @karuturi @borisroman
First round of tests done and got the same results! I will run some XenServer tests as well, but if you can help testing that will be much appreciated.

Environment

Management Server on CentOS 7.1
One KVM Host on CentOS 7.1
Agent + Common RPMs built from source
Advanced Zone
Requires Hardware: TRUE

Tests Executed

nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-kvm1-ISOLATED.cfg -s -a tags=advanced,required_hardware=true component/test_vpc_redundant.py component/test_routers_iptables_default_policy.py component/test_routers_network_ops.py component/test_vpc_router_nics.py component/test_password_server.py component/test_router_dhcphosts.py smoke/test_loadbalance.py smoke/test_internal_lb.py smoke/test_ssvm.py smoke/test_vpc_vpn.py smoke/test_network.py

Results

Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_isolate_network_FW_PF_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_isolate_network_FW_PF_default_routes_egress_false | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 33 tests in 15096.741s

OK
(END)

@wilderrodrigues maybe we can ask @anshul1886 to run the test wuite on hyperv?

If @anshul1886 can test it, just to make sure all is fine, would be very nice!
@karuturi, do you have a HyperV test environment? :)
Cheers,
Wilder

More test results:

Environment

Management Server on CentOS 7.1
One KVM Host on CentOS 7.1
Agent + Common RPMs built from source
Advanced Zone
Requires Hardware: FALSE

Tests Executed

nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-kvm1-ISOLATED.cfg -s -a tags=advanced,required_hardware=false smoke/test_routers.py smoke/test_reset_vm_on_reboot.py smoke/test_vm_life_cycle.py component/test_vpc_routers.py smoke/test_service_offerings.py component/test_vpc_offerings.py smoke/test_network_acl.py smoke/test_privategw_acl.py smoke/test_network.py

Results

Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... SKIP: At least two hosts should be present in the zone for migration
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 42 tests in 8016.294s

OK (SKIP=1)
(END)

@wilderrodrigues nope. No HyperV environment.

Thanks, @karuturi!
I'm testing against XenServer 6.2 now. Given the changes on the HyperV Resource class, I'm not expecting problem there. The changes a very clear.
Is there anyone willing to test this PR or the +1 from @DaanHoogland  and all the tests I already executed will be enough to get a second +1?
What do you think, @remibergsma ?
Cheers,
Wilder

Ping @DaanHoogland @miguelaferreira @karuturi @borisroman @bhaisaab @remibergsma
More test results, guys! It looks good to me... but my vote doesn't count. Still need 1 LGTM.

Environment

Management Server on CentOS 7.1
One XenServer 6.2 Host
Advanced Zone
Requires Hardware: FALSE

Tests Executed

nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-xen1-ISOLATED.cfg -s -a tags=advanced,required_hardware=false smoke/test_routers.py smoke/test_reset_vm_on_reboot.py smoke/test_vm_life_cycle.py component/test_vpc_routers.py smoke/test_service_offerings.py component/test_vpc_offerings.py smoke/test_network_acl.py smoke/test_privategw_acl.py smoke/test_network.py

Results

Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... SKIP: At least two hosts should be present in the zone for migration
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 42 tests in 4785.363s

OK (SKIP=1)
(END)

@wilderrodrigues LGTM :+1:
Always in support of removing old/dead code! :)
Testing:
CentOS 7.1 Setup Managment/Hypervisor
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-kvm1-ISOLATED.cfg -s -a tags=advanced,required_hardware=true component/test_vpc_redundant.py component/test_routers_iptables_default_policy.py component/test_routers_network_ops.py component/test_vpc_router_nics.py component/test_password_server.py component/test_router_dhcphosts.py smoke/test_loadbalance.py smoke/test_internal_lb.py smoke/test_ssvm.py smoke/test_vpc_vpn.py smoke/test_network.py

Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_isolate_network_FW_PF_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_isolate_network_FW_PF_default_routes_egress_false | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 33 tests in 11263.519s

OK
(END)

@wilderrodrigues @karuturi
VR scripts got stabilised over years. If any thing wrong in new implementation there is chance to look into the VR scripts and update/correct the python methods. If the scripts are there in code it is easy for the developer to compare.  So I feel keeping the scripts in code base is better.

Instead of removing the script files move those into another folder and DO NOT include in these in systemvm.iso

@jayapalu That's we're git's versioning system comes into play. We want to keep the current working directory as clean as possible. Meaning, without unused/dead code. If you want to compare how it was, use the history that git provides.

@jayapalu, I'm with @borisroman on that one. We are no longer using shared file systems to keep files stored. We have a good version control system and the commit message is clear about what was done. We also have a very clear message in the Jira ticket, which was used in every commit message.
The current Master/4.6 is stable and have been tested. if we face problems, we check the history.
Cheers,
Wilder

I also agree we shouldn't keep unused files in the active tree. We already got some PRs against files that were not in use and that is a waste of time for everybody. Will merge soon.


SAKIII-4918 - Add Sakai var to templaterenderer

https://jira.sakaiproject.org/browse/SAKIII-4918

@christianv if I'm reading this right, the solution here is to make sakai a global variable -- is that right?

Yes, I've added some thoughts in the jira.
I'm open to suggestions/alternatives.


chore: Automate content generation of some Priority docs

Description

User visible changes:

The file format samples for Priority emojis are now machine-generated in:

user docs
Tasks-Demo vault

A side-effect is that some minor white-space errors were fixed

Code changes:

Added helper functions in PriorityField to convert Priority enum values to string

Tests

New file tests/TaskSerializer/DocsSamplesForTaskFormats.test.ts generates *.approved.md files showing how Priority values are represented in Emoji and dataview file formats.

These files are designed to be embedded in to documentation and markdown files
When new formats are added, new sample files will be created automatically
When priorities are renamed, or more priorities are added, when the tests are run, the

withAllPriorities() is exported, and now includes a useful description for each task, so the tasks are suitable for inclusion in documentation
Verify functions in tests/TestingTools/VerifyMarkdownTable.ts now use Visual Studio Code's diff tool for showing and approving differences in approval test files

This was the only diff tool I could find, in the ones I have available, that handles Emojis correctly, not corrupting them when editing lines containing Emojis.
WARNING: this may need more thought if there are any contributors who don't have VSCode installed, and are reviewing Approval Test diffs

Notes

There are other locations in the docs that list the available priorities, such as the list of group names when grouping by priority. These may may be updated in separate steps.

Motivation and Context

Start of work towards reducing many repetitious edits in the user docs and sample vault

How has this been tested?

With new tests
Viewing the changed markdown files in Obsidian, in Source and Reading modes

Screenshots (if appropriate)
Types of changes

Changes visible to users:

[x] Documentation (prefix: docs - improvements to any documentation content for users)
[x] Sample vault (prefix: vault - improvements to the Tasks-Demo sample vault)

Internal changes:

[x] Refactor (prefix: refactor - non-breaking change which only improves the design or structure of existing code, and making no changes to its external behaviour)
[x] Tests (prefix: test - additions and improvements to unit tests and the smoke tests)

Checklist

[x] My code follows the code style of this project and passes yarn run lint.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] My change has adequate Unit Test coverage.

Terms

[x] My contribution follow this project's contributing guide
[x] I agree to follow this project's Code of Conduct

@ilandikov You might be interested in this as:

it's the start of what I was describing when we last paired
it changes the behaviour of withAllPriorities(), which might conflict with what you are working on
if you had time to look at it, I would welcome any feedback you had

Looks good =)
So the idea here is to use approvals as snippets for docs? Did I get it right?

So the idea here is to use approvals as snippets for docs? Did I get it right?

Yes, nice summary. Also for including in markdown files in the Test Vault.


Feature request: Request for better timer debug -- at least on local scripthost

There is what I would say is a annoyance level 2 on a scale of 1-5 that when I am debugging a timer function in Visual Studio with the Azure Function Core tools / local scripthost, using the "class library" / new function types,, I have to keep messing with the CRON setting before running it to either

Keep it from firing when I don't want it to fire
Keep it from firing twice and screwing up my first test that is in progress
Make sure it fires

I wonder if you guys could consider adding some syntax in the source file, or host.json setting that says to fire the timer ONCE immediately, and then don't fire it again.  This would allow people testing local functions to reliably test something that counts on timers.
This is analogous to the website which "runs" whenever you press "run", and doesn't wait for the timer time.
What do you guys think?
-thanks
-e

Have you tried configuring the timer function to be disabled and then using the functions core tools func run command to run the function?

I should have explained that I'm looking for something that is a quick iteration for F5 debugging in Visual Studio.  I don't see under that constraint that using the func run command would apply.
I updated my original post to make that more clear.
If you think that the func run command still applies to that scenario, please let me know how it would set that up.  Perhaps having that set up would make any other changes low priority, or no priority ;)

I see. While you're debugging, you could change the schedule to be very infrequent and then take advantage of the RunOnStartup property:

okay, sounds like we already have what I was asking for.  I was unaware of that property.  Please excuse the interruption.

Note that I used RunOnStartup locally, which worked well.  When that setting made it to Azure on publish, it appeared that the timer fired more often than expected.   The CRON spec is "0 0 3 * * *", which would say to me 0300 UTC, but the timer fired every 10 minutes or so.
Either RunOnStartup was the problem, or the Timer fired each time a queue function ran out of time.  I'm happy to start another issue on that if it is expected that is the cause instead.

Don't publish with RunOnStartup set to true. Just set it while you're debugging.

Yeah, that's what I found out!
NP
-thanks
-e


conda-forge depenencies

I just tried to install using the command conda install -c conda-forge pyironflow.
The following required packages did not get installed:

esbuild
nodejs

Also just FYI, I need to use pyironflow by from pyironflow.pyironflow import PyironFlow
Is the second pyironflow needed? Should we rename the file or something?

They should only be required for recompiling pyironflow but they should be required for using pyironflow as far as I understand.

Should be required for using ``


Read timeout on wait()

I'd like to be able to wait for any number of messages but timeout after a time.
$nc->setStreamTimeout(1);
$nc->wait(0);

function to set timeout on the stream:
    /**
     * Set Stream Timeout
     *
     * @param int $seconds before timeout on stream
     */
    public function setStreamTimeout($seconds) {
        if ($this->isConnected()) return stream_set_timeout($this->streamSocket, $seconds);
        return false;
    }

new receieve function that can return boolean false like fgets:
    /**
     * Receives a message thought the stream.
     *
     * @param integer $len Number of bytes to receive.
     *
     * @return string
     */
    private function receive($len = null)
    {
        if ($len) {
            $line = fgets($this->streamSocket, $len + 1);
        } else {
            $line = fgets($this->streamSocket);
        }
        if ($line === false) return $line;
        else return trim($line);
    }

new wait function that respects the timeout, by checking for false from receive:
    public function wait($quantity = 0)
    {
        $count = 0;
        while (!feof($this->streamSocket)) {
            $line = $this->receive();
            if ($line === false) {
                return;
            }
            // PING
            if (strpos($line, 'PING') === 0) {
                $this->handlePING();
            }
            // MSG
            if (strpos($line, 'MSG') === 0) {
                $count = $count + 1;
                $this->handleMSG($line);
                if (($quantity != 0) && ($count >= $quantity)) {
                    return $this;
                }
            }
        }
        $this->close();
        return $this;
    }

https://github.com/the-control-group/phpnats/blob/master/src/Connection.php

Fixed. Thanks @nickcarenza and @adriacidre! :facepunch:


Add compatibility checker to /serverinfo

While it's possible to manually do this inside a given implementation, this would more easily allow tools to determine if implementations contain the minimum implemented calls necessary for compatibility without having to parse the /serverinfo results. We discussed this a bit at PAG, so I'm hoping that @BrapiCoordinatorSelby can add to the details.

The Problem:
Two systems are reported to be BrAPI compatible, but support different sets of endpoints and are therefore not compatible with each other. For example, Field Book is BrAPI compatible requiring Studies, OUs, Variables, and Observations, and GIGWA is also BrAPI compatible supporting the Studies, AlleleMatrix, and Variants endpoints. Both BrAPI compatible, but not with each other. This causes confusion and frustration.
Solution:
The current solution is for every client application to access the /serverinfo endpoint of a server it is pointed at. This happens before any other calls are made. The data returned by /serverinfo should be enough information to determine compatibility, but it is clumsy to do, and the logic needs to be done on the client side.
The proposed solution is to make a modification to /serverinfo or make a new endpoint to push this logic to the server side. Something where the client can submit the endpoints it requires, and the server can respond with a simpler 👍 or 👎. I have some ideas of what that might look like, including method (GET, POST, etc), endpoint, BrAPI version, and criticality(required vs optional).
This seems like an easy, low hanging fruit to improve the standard. Always open to ideas and opinions.


Release 1.12.0

Release checklist:

[x] Create a new Release Issue

[x] Replace 1.11.0 with the former release, and 1.12.0 with THIS

[x] Update the Roadmap calling out shipped features
[x] Create and update a Milestone for the release

[x] Assign this task
[x] Assign all the shipped roadmap Issues
[x] Assign the relevant recently closed Isssues

Code changes:

[x] Create a release branch 'release-x.y.z': git checkout -b release-1.12.0
[x] Create a temporary tag git tag v1.12.0 && git push opensource --tags
[x] Create a New Draft GitHub Release, and generate the automated changelog (for new contributors)
[x] Update the release version in package.json, and npm i
[ ] Update in-app News src/apps/news/news.data.tsx
[x] Update the in-app News version number
[ ] Update the README.md with the new release
[ ] Copy the highlights to the docs/changelog.md

Release:

[ ] merge onto main git checkout main && git merge --no-ff release-1.12.0
[ ] re-tag `git tag -f v1.12.0 && git push opensource --tags -f'
[ ] verify deployment on Vercel
[ ] verify container on GitHub Packages
[ ] update the GitHub release
[ ] push as stable git push opensource main:main-stable

Announce:

[ ] Discord announcement
[ ] Twitter announcement

Links

Milestone: https://github.com/enricoros/big-AGI/milestone/12
GitHub release: https://github.com/enricoros/big-AGI/releases/tag/v1.12.0
Former release task: #335

Artifacts Generation
You help me generate the following collateral for the new release of my opensource application
called big-AGI. The new release is 1.12.0.
To familiarize yourself with the application, the following are the Website and the GitHub README.md.

paste the URL: https://big-agi.com
drag & drop: README.md

I am announcing a new version, 1.12.0.
For reference, the following was the collateral for 1.11.0 (Discord announcement,
GitHub Release, in-app-news file news.data.tsx, changelog.md).

paste the former: discord announcement,
GitHub release,
news.data.tsx,
changelog.md

The following are the new developments for 1.12.0:

paste the link to the milestone (closed) and each individual issue (content will be downloaded)
paste the git changelog  git log --pretty=format:"%h %an %B" v1.11.0..v1.12.0 | clip

news.data.tsx
I need the following from you:

1. a table summarizing all the new features in 1.12.0 (description, significance, usefulness, do not link the commit, but have the issue number), which will be used for the artifacts later
2. after the table score each feature from a user impact and magnitude point of view
3. Improve the table, in decreasing order of importance for features, fixing any detail that's missing, in particular check if there are commits of significance from a user or developer point of view, which are not contained in the table
4. I want you then to update the news.data.tsx for the new release

Readme (and Changelog)
I need you to update the README.md and the with the new release.
Attaching the in-app news, with my language for you to improve on, but keep the tone.

GitHub release
Please create the 1.12.0 Release Notes for GitHub.
Use a truthful and honest tone, understanding that people's time and attention span is short.
Today is 2024-1-1.

Now paste-attachment the former release notes (or 1.5.0 which was accurate and great), including the new contributors and
some stats (# of commits, etc.), and roll it for the new release.
Discord announcement
Can you generate my 1.12.0 big-AGI discord announcement from the GitHub Release announcement?
Please keep the formatting and stye of the discord announcement for 1.11.0, but with the new messaging above.

More misc info:

Call App
Mobile UI
updated tiktoken
large speed improvements
updated ollama models
150 commits, 7,000+ lines changed

Screenshots
https://github.com/enricoros/big-AGI/assets/32999/95ceb03c-945d-4fdd-9a9f-3317beb54f3f

news.data.tsx
  {// 🆕
    versionCode: '1.12.0',
    versionName: 'AGI Hotline',
    versionMoji: '✨🗣',
    versionDate: new Date('2024-01-26T12:30:00Z'),
    items: [
      { text: <>Voice Call Personas: save time, recap conversations</>, issue: 354 },
      { text: <>Updated OpenAI Models to the 0125 release</>, issue: 364 },
      { text: <>Chats: Auto-Rename and assign folders</>, issue: 222 },
      { text: <>Link Sharing makeover and control</>, issue: 356 },
      { text: <>Accessibility for screen readers</>, issue: 358 },
      { text: <>Export chats to Markdown</>, issue: 337 },
      { text: <>Paste tables from Excel</>, issue: 286 },
      { text: <>Large optimizations</> },
      { text: <>Ollama updates</>, issue: 309 },
      { text: <>Over 150 commits and 7,000+ lines changed for development enhancements</>, dev: true },
    ],
  },

Readme.md
## What's New in 1.12.0 · Jan 26, 2024 · AGI Hotline

https://github.com/enricoros/big-AGI/assets/32999/95ceb03c-945d-4fdd-9a9f-3317beb54f3f

- **Voice Calls**: real-time voice call your personas out of the blue or in relation to a chat [#354](https://github.com/enricoros/big-AGI/issues/354)
- Support **OpenAI 0125** Models. [#364](https://github.com/enricoros/big-AGI/issues/364)
- Rename or Auto-Rename chats.  [#222](https://github.com/enricoros/big-AGI/issues/222), [#360](https://github.com/enricoros/big-AGI/issues/360)
- More control over **Link Sharing** [#356](https://github.com/enricoros/big-AGI/issues/356)
- **Accessibility** to screen readers [#358](https://github.com/enricoros/big-AGI/issues/358)
- Export chats to Markdown [#337](https://github.com/enricoros/big-AGI/issues/337)
- Paste tables from Excel [#286](https://github.com/enricoros/big-AGI/issues/286)
- Ollama model updates and context window detection fixes [#309](https://github.com/enricoros/big-AGI/issues/309)

GitHub Announcement
🚀 big-AGI 1.12.0 - AGI Hotline
Introducing big-AGI 1.12.0, a significant step in refining AGI interaction and management. Who can it be now? Your phone is ringing - it's your AGI. Voice Call, with new AI models, a snappier and juicier UI, and more. Here's the essence of what's new:

Voice Calls: real-time voice call your personas out of the blue or in relation to a chat #354
Updated OpenAI 0125 Models. #364
Rename and Auto-Rename chats.  #222
Link Sharing Revamped, with more user control #356
Accessibility to screen readers #358
Export chats to Markdown #337
Paste tables from Excel #286
Ollama model updates and context window detection fixes #309

For Developers:

Over 150 commits and 7,000+ lines changed, emphasizing performance optimizations and under-the-hood improvements for a smoother experience.
Dive into the details: v1.11.0...v1.12.0

Upgrade Now
Upgrade to big-AGI 1.12.0 today (code, docker, or the hosted site).
Join us on Discord and explore our public roadmap. Together, we're shaping its future! 🚀

Discord Annoucement
@everyone ## :sparkles: big-AGI 1.12.0 - AGI Hotline (Bling) Released
Massive drop :partyblob: . Update to 1.12.0 and get closer to your AGI personas than ever before:

Voice Calls: Dial-up your AGI personas for real-time calls, out of the blue or over a chat
Updated OpenAI Models: just announced by OpenAI the 0125 models
Chats renaming, auto-renaming, exporting and folder selections
Link Sharing makeover: share with more control
Accessibility for screen readers - fully reforged the structure
Paste from Excel, and export to Markdown,
Ollama models and context updates

For Developers: This release is larger than usual, with over 150 commits and 7,000+ lines changed, in features, React optimizations, and fixes across the board :capybarathink:
Quick Links:

Update now at https://get.big-agi.com or grab the latest Docker image https://github.com/enricoros/big-AGI/pkgs/container/big-agi
Dive into what's next on our roadmap: https://github.com/users/enricoros/projects/4/views/2

:rocket: So much more coming, and lots of exciting innovation - ping the devs, join us, so much to do! :partyblob:


Attribute error: 'nonetype' object has no attribute 'keys'

this error is coming after downloading inception package at command Retrain.py

Can you please tell me which version of Python and Tensorflow you are running?

I resolved this issue, my python verson 3.6 and tensorflow is latest version, sorry forgot to  query, the error occured due to path error

The program was not behaving in the right way as it is supposed to be, it didn't create a  folder named tf_"data" or something, inspite of that in the temp directory, .pb model and attribute text file were changed after retaining, so i changed path in the final .py executable file and it ran, took around 2-3 hrs extra but worked!!!
Thanks for making this simple code!!

Sure, not a problem! Great that you were able to make it work.

if this error again occur, try this:
python F:\Tensorflow\retrain.py --image_dir C:\training_data --how_many_training_steps 500 --model_dir F:\Tensorflow\inception --output_graph=F:\Tensorflow\retrained_graph.pb --output_labels=F:\Tensorflow\retrained_labels.txt


Create script for setting correct #import statement in JXcoreExtension.m

At this moment in JXcoreExtension.m we have hardcoded import
#import "ThaliTest-swift.h"
It makes impossible to use this library with other projects. Therefore we need to have during install a script that will edit JXcoreExtension.m and will add appropriate import according iOS Product name.

The plan is to change the installation process a bit to change jxcore extension file with app name.

There is a plugin with which help get app name from cordova.
Will try to use today.


Hope to support database transactions for the webflux model

Why you need it?
希望可以支持webflux模型的数据库事务
How it could be?
A clear and concise description of what you want to happen. You can explain more about input of the feature, and output of it.
Other related information
Add any other context or screenshots about the feature request here.

@slievrly, @xingfudeshi, @zhengyangyong , @arnkore , @cmonkey , please help to give a reponse.

What will happen if you use current seata?Is there any exception?

What will happen if you use current seata?Is there any exception?

It seems that it does not support webflux reactive programing mode (base on netty, epoll model).

Can you provide more details?like error logs.


Introduce {ci_project}-{git_repo}-fabric8-analytics-build-master template

The idea here is to be able to deploy more services from the same image, at the same time.

Example: https://github.com/openshiftio/saas-analytics/blob/master/bay-services/cvejob.yaml
3 cronjobs, deployed always from the same image, but under different names (and with different params).

I think it would be safer to have another template for your particular case since this change affects a huge amount of jobs, WDYT?

I think you're right. I wanted to avoid "forking" the default template as the YAML file is already huge and complicated. But it will be safer to create a new template.

@riuvshin WDYT now?

CI fails with following error:
-------------------------------------------------------------------------
JOB_DIFF:

new_jobs:
- devtools-cvejob-fabric8-analytics-build-master
removed_jobs:
- devtools-cvejob-build-master

That is expected as I am introducing new template here and I am modifying configuration of an existing job to use the new template.
@riuvshin is there a process how to request removal of an old job?

@riuvshin Please re-review when you find a minute :wink: Thanks!

CI fails. The reason is that I am modifying an existing job: https://github.com/openshiftio/openshiftio-cico-jobs/pull/977#issuecomment-473802006

LGTM

[test]

@skryzhny @riuvshin Thanks guys :)


Debian 12 support

Description of the Issue
On Debian 12, the installation of the google-cloud-ops-agent package is failing. The exact error message being received is:
E: Unable to locate package google-cloud-ops-agent
[2023-06-20T14:42:22+0000] google-cloud-ops-agent installation failed.

Steps to Reproduce

On a Debian 12 system, attempt to install the google-cloud-ops-agent package via terminal.

Expected Behavior
The google-cloud-ops-agent package should install successfully without any error messages.
Actual Behavior
The installation fails, throwing an error message that the package cannot be located.
Environment

OS: Debian 12
Repository: https://packages.cloud.google.com/apt google-cloud-ops-agent-bookworm-all InRelease

Any help resolving this issue would be greatly appreciated.

Hi @de4ps thank you for opening an issue. The team is currently actively working Debian 12 support, we are hoping to have it ready ASAP.

Unfortunately we don't have an ETA right now, but we will keep this issue updated when it is ready.

I've also been tricked... and I'm wondering: why offer VM installations in debian 12 when few things are compatible with it.
Anyway.
Let's hope the patch doesn't take 8 months like with debian 11.

Debian 12 support is available in Ops Agent v2.35.0: https://github.com/GoogleCloudPlatform/ops-agent/releases/tag/2.35.0
Installing the latest agent release by following https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/installation should do the trick.


Unhandled Exception: System.ComponentModel.Win32Exception: The system cannot find the file specified

The following error is occurring on Windows 10 x64.
Unhandled Exception: System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithShellExecuteEx(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Supervisor.MonitorThread.DoMonitor()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

@Testerboy21 its an open source software. You're free to fix it and make a pull request to merge the changes.

@esidegallery I had a similar issue. One thing to check for is if the path to the executable has spaces that you need to put 3 quotation marks before the path and two after the end of the path to the executable.

Try running supervisor with an elevated (administrator) level shell.


Proposal: Evolution

Exploration on various aspects of the project so it can evolve, for the better.
(May get updated upon receiving further feedback)

Observation
Proposed direction
Examples of possible new direction

Rendering

With @frint/react-dom
With ReactDOM

Higher-order component

With RxJS

Dependency Injection

With React

Props composition

Current state
Subscription
With RxJS
With React

Root Apps
Blocks

Set up window.blocks
Render Regions
Register a Block

Blocks and props from Region
Accessing Blocks

Synchronously
Promised
Callbacks
Waiting limit

Block to Block communication
Cascaded and scoped providers

Considerations

Pros
Cons

Note about Travix

Observation
From the perspective of a regular JavaScript developer:

One may not want the concept of Apps when doing UI development
Dependency Injection is not a common use case
Child Apps is not a common use case
Loading Region component for any kind of React usage is not ideal
Giving up control of React's rendering to DOM may discourage some
RxJS-first approach is not helping beginners
One may want to use only certain features of FrintJS without having to buy rest of the features

Proposed direction
To tackle some of the problems, we can aim to be:

Less of a framework, and more of a suite of libraries
Allow selective composition
Don't lock developers in with opinions
Be flexible and allow easier interop with other libraries (like RxJS)
Keep serving minimum building blocks, allowing other tools to be built around us

Examples of possible new direction
Key differences:

Providers data structure defined by the developer (not opinionated)
App is not a requirement
DI is not a requirement
RxJS is not a requirement
Component does not need to be defined in DI Container
Taking over React's rendering to DOM is not a requirement
No need to load Child Apps' features

Rendering
With @frint/react-dom
Either render with render for ease of use:
import { render } from '@frint/react-dom';

render(
  MyComponent,
  document.getElementById('root'),

  // third argument is providers data
  {
    foo: 'foo value',
    bar: 'bar value',
  },
);

The last argument can also be a function, returning providers data:
render(
  Component,
  domNode,

  () => ({
    foo: 'foo value',
    bar: 'bar value',
  }),
)

With ReactDOM
OR render manually with React's new Context API:
import {
  // created with `React.createContext()`
  FrintContext
} from '@frint/react';
import { render } from 'react-dom';

render(
  <FrintContext.Provider value={{
    foo: 'foo value',
    bar: 'bar value',
  }}>
    <MyComponent />
  </FrintContext.Provider>,

  document.getELementById('root'),
);

Can also happen without exposing the React Context directly:
import {
  // regular Component
  FrintProvider
} from '@frint/react';
import { render } from 'react-dom';

render(
  <FrintProvider providers={{
    foo: 'foo value',
    bar: 'bar value',
  }}>
    <MyComponent />
  </FrintProvider>,

  document.getELementById('root'),
);

Higher-order component
import React from 'react';

import {
  // higher-order component
  // capable of emitting props
  // while having access to providers
  hoc,
} from '@frint/react';

function MyComponent(props) {
  return (

Hello World

  );
}

const EnhancedComponent = hoc({
  // regular props generation,
  // called only once
  props({ foo, bar }, cb) => {
    // props via callback
    cb({ foo: 1 });
    cb({ foo: 2 });

    // optional unmount handler
    return () => console.log('unmounted');
  },

  // optional: access props passed from parent,
  // called as many times new props are passed
  parentProps(parentProps, providers) => {
    return parentProps;
  },

  // optional: default props (sync),
  // called only once
  defaultProps({ foo, bar }) {
    return {};
  },

  // optional: merge all of them,
  // called as many times as needed
  mergeProps({ props, parentProps, defaultProps }) {
    return {
      ...defaultProps,
      ...props,
      ...parentProps,
    };
  },

  // optional: should emit props for updating?
  // called as many times as needed
  shouldUpdate(prevProps, mergedProps, providers) {
    return true;
  },
})(MyComponent);

With RxJS
import React from 'react';
import { hoc } from '@frint/react-rx'; // separate package

function MyComponent(props) {
  return (

Hello World

  );
}

const EnhancedComponent = hoc(({ providers, parentProps$ }) => {
  return Observable.of({}); // or plain object
})(MyComponent);

Dependency Injection
DI is completely optional, and a standalone library:
import { createContainer, resolveAndProvide } from '@frint/di';

const Container = createContainer([
  {
    name: 'foo',
    useValue: 'foo value',
  },
  {
    name: 'bar',
    useFactory({ foo }) {
      return 'bar value',
    },
    deps: ['foo'],
  }
]);

const providers = resolveAndProvide(Container);
console.log(providers.foo); // `foo value`
console.log(providers.bar); // `bar value`

With React
To use the resolved providers when rendering:
import { render } from '@frint/react';

render(
  Component,
  domNode,

  // function that will be called to return providers object
  resolveAndProvide(Container),
);

Props composition
import {
  compose, // higher-order function
  withDefaults,
  withStore,
  shouldUpdate
} from '@frint/props';

const propsComposition = compose(
  withDefaults({
    foo: 'foo value',
  }),
  withStore(
    mapState,
    mapDispatch
  ),
  shouldUpdate(props => true),
)(providers = {});

Current state
Current state of props (sync):
const props = propsComposition.getValue();

Subscription
Regular subscription (async):
const unsubscribe = propsComposition.subscribe(
  props => console.log(props)
);

unsubscribe();

With RxJS
Interop:
const props$ = Observable.from(propsComposition);

With React
import React from 'react';
import {
  withDefaults,
  withStore,
  shouldUpdate,
} from '@frint/props';

import {
  compose // higher-order component
} from '@frint/props-react';

function MyComponent(props) {
  return

;
}

export default compose(
  withDefaults(...),
  withStore(...),
  shouldUpdate(...),
)(MyComponent);

Root Apps
There is nothing called Root Apps any more.
Blocks
The new "Blocks" are the old "Child Apps".
Blocks:

Have lifecycle hooks (initialize and beforeDestroy)
They can be registered dynamically
Rendered in certain places via predefined <Region>s
Can have its own set of providers (with access to those cascaded from top)
Primary app never needs to know about them early

Set up window.blocks
import { createContainer, resolveAndProvide } from '@frint/di';
import { BlocksRegistry } from '@frint/blocks';

const Container = createContainer([
  {
    name: 'foo',
    useValue: 'foo value',
  },
  {
    name: 'bar',
    useValue: 'bar value',
  },
  {
    name: 'blocks',
    useFactory({ container }) {
      return new BlocksRegistry(container, {
        cascade: ['foo', 'bar'],
      });
    },
    deps: ['container'],
  },
]);
const providers = resolveAndProvide(Container);

window.blocks = providers.blocks;

Render Regions
import React from 'react';
import { render } from '@frint/react-dom';
import { Region } from '@frint/blocks-react';

function Root() {
  return (

      <Region name="sidebar" />

  );
}

render(
  <Root />,
  document.getElementById('root'),
  providers,
);

Register a Block
function BlockComponent() {
  return
Hello World from Block.
;
}

// register
window.blocks.register({
  name: 'Block1',
  provide(cascadedProviders = {}) {
    return {
      baz: 'baz value',
    };
  },
  initialize() {
    // can access `this.providers`
  },
  beforeDestroy() { },
  component: BlockComponent,
  regions: ['sidebar'],
  multi: false, // multi-instance: optional, defaults to `false`
});

Blocks and props from Region
RegionService is dropped. Functionality lives in React-land.
Regular usage:
import React from 'react';
import { withRegionProps } from '@frint/blocks-react';

function BlockComponent(props) {
  const { regionData } = props;

  return

;
}

export default withRegionProps((regionProps, providers) => {
  return (
    <BlockComponent regionData={regionProps.data} />
  );
});

Accessing Blocks
Registering new Blocks:
// first Block
window.blocks.register({
  name: 'FirstBlock',
  provide({ blocks }) {
    return {
      blocks,
      foo: 'foo value',
    };
  },
  component: Block1Component,
  regions: ['sidebar'],
});

// second Block
window.blocks.register({
  name: 'SecondBlock',
  provide({ blocks }) { return {}; },
  component: Block2Component,
  regions: ['footer'],
});

Access Block:
Synchronously
// Block instance or null
const block = window.blocks.getSync('FirstBlock');

Promised
// no cancellation - dangerous for memory leaks
window.blocks.get('FirstBlock').then(block => {
  console.log(block.name); // `FirstBlock`
  console.log(block.providers); // `{ blocks, foo }`
});

Callbacks
// with cancellation for listening
const unlisten = window.blocks.get('FirstBlock', (error, block) => {
  // emits once
  console.log(block.name);
});
unlisten();

Waiting limit
// waiting limit (Promise - no cancellation)
window.blocks.waitFor('FirstBlock', 2000)
  .then(block => console.log(block));
  .catch(() => console.log('Not available within 2 seconds'));

// waiting limit (callback - with cancellation)
const unlisten = window.blocks.waitFor('FirstBlock', 2000, (error, block) => {
  if (error) {
    console.log('Not available within 2 seconds');
    return;
  }

  console.log(block);
});
unlisten();

Block to Block communication
// FirstBlock
import React from 'react';
import { hoc } from '@frint/react';

function FirstBlockComponent(props) {
  const { secondBlockState } = props;

  return

;
}

export default hoc({
  props({ blocks }, cb) => {
    let unsubscribeBlockStore = null;

    const unsubscribeBlock = blocks.waitFor(
      'SecondBlock',
      2000,
      (error, block) => {
        unsubscribeBlockStore = block.providers.store.subscribe(
          state => cb({
            secondBlockState: state,
          })
        );
      });

    return () => {
      unsubscribeBlock();
      unsubscribeBlockStore && unsubscribeBlockStore();
    };
  }
})(FirstBlockComponent);

Cascaded and scoped providers
We currently have "cascaded scoped providers", which allow you to set a Provider from Root App, and have it cascaded to all the Child Apps too, but the classes will be instantiated again with the App in scope (Child App).
We can achieve something similar by doing this:
class Foo {
  constructor(args) {
    const { block } = args;
  }

  // return's a new instance of self
  // scoped by given Block
  forBlock(blockInstance) {
    return new Foo({
      block: blockInstance,
    });
  }
}

Render initially with foo in providers:
import { render } from '@frint/react-dom';
import { BlocksRegistry } from '@frint/blocks';

const providers = {
  foo: new Foo (),
};

window.blocks = new BlocksRegistry(providers, {
  cascade: ['foo'],
});

render(
  Component,
  domNode,
  providers,
);

Register a new Block:
window.blocks.register({
  name: 'MyBlock',
  // ...
  provide({ blocks, foo }) {
    // `foo` is returned value of `Foo.forBlock(myBlockInstance)`
    return {
      foo,
    };
  },
});

Considerations
Before making any decision, we can take a look at the pros and cons of all these proposed changes from various perspectives.
Pros
For all:

This makes FrintJS more approachable
The learning curve goes down significantly for regular React developers
You don't have to learn about multiple packages at once
You aren't forced to use RxJS (choose it at will)
You don't have to load too many features in your application, unless you really need them (you have a choice)

Technically:

Better API support with React's new (since v16.3):

Async rendering
Context API
Safer server-side rendering

Cons
For new users:

N/A

For existing users and maintainers:

This involves a LOT of breaking changes
This involves a LOT of work that needs to be done in almost all the packages that we currently have across multiple monorepos on GitHub
Changes the very core principles of how FrintJS itself works (for e.g., no more Apps)
Backwards compatibility would be quite hard to achieve
Migrating existing applications would involve a significant amount of work
After establishing all the common practices in last couple of years, we may be opening up a whole new wormhole

Note about Travix

The project is indeed open source and has been developed very publicly and openly, it was built primarily for the needs of Travix
We don't have a good track record of upgrading FrintJS in our production websites in time (irrespective of the amount of migration work, even though the maintainers always provide detailed guides with each major semver release)
Personally, I feel it will take more than a year to get all these things to happen (if they happen at all), including further development of FrintJS and rolling out the changes in production

According to the points above, I can understand if we decide not to go for this proposal in this project.

Now that most of the info is laid out in the open, we need a decision. The process of decision making is quite conflicting, since two sides can be involved:

Point of view from a generic purpose open source project
Point of view from a company for which this project was initialy built

I would highly appreciate all of your feedback on this.
Cheers!

@fahad19, great job reimagining a less opinionated version of FrintJS moving forward.
I certainly agree strongly with the worries around the following:

Huge breaking changes and forced rewriting of applications that use FrintJS make it very difficult to agree to continue development of FrintJS applications given how much upgrade work will be required in 1-2 years time.
Sheer scope of the project such that this next version of FrintJS could take over 1 year to transition to

Given the project I'm using FrintJS for is still in it's early days it would be frustrating to put in this effort to use the framework only to have all of it in substantial ways be broken in the next update.
Specifically to the worries expressed about rigidness and approachability, at least for our application those have been a positive for our development process. In building a modular dashboard, the current version of FrintJS has been instrumental in allowing us to have on demand loading of modules and tabs to an admin dashboard. I don't see the vision that initiated the project as obsolete by any means.
I think a framework as outlined above would be valuable but the migration/upgrade process should hopefully be top of mind when release approaches or should be released as a totally different framework name.

Thanks for such a thoughtful response, @andrewparadi!

I think a framework as outlined above would be valuable but the migration/upgrade process should hopefully be top of mind when release approaches or should be released as a totally different framework name.

I think that's what's gonna happen in the end. FrintJS will continue to maintain its API stability for as long as possible, while upgrading to latest ReactJS + RxJS periodically.
Other new ideas can possibly be expressed in fresh new libraries :)

FrintJS will not go for any major breaking changes in near future. So closing it.


Merge master into dev-1.21

Merge master into dev-1.21 to keep in sync - SIG-Release 1.21 Docs team 3/5/21
/cc @reylejano @mvortizr @ChandaniM123 @tengqm

Build error with commit 0ccba55, fixed in same PR kubernetes/website#26817 with commit 012fcb0

/lgtm

/approve


Error using alpinejs-ray via CDN

Description
Unable to use alpinejs-ray via CDN shows the following error in the web inspector:
TypeError: Cannot read properties of undefined (reading 'randomUUID')
Steps To Reproduce

Using the following file:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
    <script<EMAIL_ADDRESS>    <script<EMAIL_ADDRESS>    <script<EMAIL_ADDRESS>defer></script>
</head>

<body>
    <div x-data>
        <button @click="$ray('hello from alpine')">Send to Ray</button>
    </div>
</body>

</html>

Open it and click on Send to ray
See the following screenshot of the web inspector

Expected behavior
The web page should not have any errors and should communicate with Ray
Environment

OS: macOS 14.5
Browser: Chrome 126.0.6478.127 (Official Build) (arm64)

Additional context
Setting up a project with alpinejs-ray installed via npm works correctly.

@bnzo  thank you for reporting this, I'll look into it.

Wondering if there's an update on this issue as I have the same error message - very new to Ray so not sure whether installed correctly or how to resolve. Thank you :)

Having the same issue as well. Can't seem to get it working in a Magento 2 Installation with a Hyva Theme.

@longestdrive @stijntrinos @bnzo I'm actively working on a v3.0 which should resolve the issue. Thank you for your patience :+1:

You can see the progress in this PR.

@patinthehat When do you think this version is availible?


Documentation

Updated the documentation.

The updated guide can be found here: http://<IP_ADDRESS>/biojs-frontend/dist/#/guide
(make sure to clear your browser's cache or open in incognito)

Merging into develop rather than master?

@Megh-Thakkar, yes, all the branches are being merged to develop right now. Develop will later be merged into Master.
@rowlandm, noted. I will be going through the whole documentation again as soon as I complete writing thr unit tests. Thanks for the review!


Fatal Exception: java.lang.SecurityException

com.apnaklub.apnaklub: One of RECEIVER_EXPORTED or RECEIVER_NOT_EXPORTED should be specified when a receiver isn't being registered exclusively for system broadcasts

Facing the same issue. @Yadav-Sunil Please fix this or upgrade the SDK support to resolve this.

Resolve this issue

Any update on this ?


Update .gitmodules

Users cannot install submodules with a git address.  changed robofleet_client_lib to an http address.  is that acceptable?

Seems fine to me, though with this method people may need to type in their GitHub credentials to update the submodule. The nice thing about using ssh keys for the clone is that you don't need to do this step


TensorMetric not updated to cuda device

🐛 Bug
When tensors located on the GPU are passed through a TensorMetric forward method, they are transfered to the CPU by this method. It seems that the 'cpu' device is not updated when training on a gpu. The cpu device seems to be hardcoded here
To Reproduce
Steps to reproduce the behavior:
import torch
import torch.nn.functional as F
from pytorch_lightning.metrics import TensorMetric

class FocalTverskyMetric(TensorMetric):
    def __init__(self, alpha: float, beta: float, gamma: float, smooth=1e-8):
        super().__init__("FocalTversky")
        self.alpha = alpha
        self.beta = beta
        self.gamma = gamma
        self.smooth = smooth

        # This line works as a workaround
        # self._device = torch.device('cuda')

    def _tversky_index_c(self, p: torch.Tensor, g: torch.Tensor):
        c = p.shape[1]
        p = p.permute(0, 2, 3, 1).reshape((-1, c))
        g = F.one_hot(g.flatten().long(), c)

        tp = torch.sum(torch.mul(p, g), dim=0)
        fn = torch.sum(torch.mul(1. - p, g), dim=0)
        fp = torch.sum(torch.mul(p, 1. - g), dim=0)
        return (tp + self.smooth) / (tp + self.alpha * fn + self.beta * fp + self.smooth)

    def forward(self, x, y):
        ti = self._tversky_index_c(x, y)
        res = (1 - ti).pow(1 / self.gamma)
        return torch.sum(res, dim=0)

if __name__ == '__main__':
    metric = FocalTverskyMetric(alpha=0.5, beta=0.5, gamma=1.)

    preds = torch.Tensor([[[[1.]], [[0.]]], [[[1.]], [[0.]]], [[[0.]], [[1.]]]]).cuda()
    assert preds.is_cuda  # Passes
    labels = torch.Tensor([[[1]], [[1]], [[0]]]).cuda()
    assert labels.is_cuda  # Passes

    loss = metric(preds, labels)
    assert loss.is_cuda  # Fails

When training in DDP mode with 16-bit precision, these metrics throws the stack trace below. This disappears in 32-bit mode since it's the amp grad_scaler asserting that the loss value is a cuda tensor.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0,1,2,3]
Using 16bit precision.
Using 16bit precision.
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
Using 16bit precision.
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4
Using 16bit precision.
initializing ddp: GLOBAL_RANK: 3, MEMBER: 4/4
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 4 processes
----------------------------------------------------------------------------------------------------

  | Name      | Type               | Params
-------------------------------------------------
0 | model     | DeepLabV3          | 60 M
1 | calc_loss | FocalTverskyMetric | 0
2 | calc_iou  | IoU                | 0
Epoch 1:   0% 0/493 [00:00<?, ?it/s] Traceback (most recent call last):
  File "/opt/code/deeplabv3_lightning.py", line 269, in <module>
    'pred': predict
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/opt/code/deeplabv3_lightning.py", line 224, in train
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 895, in fit
    self.ddp_train(task, model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 526, in ddp_train
    self.run_pretrain_routine(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1091, in run_pretrain_routine
    self.train()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 374, in train
    self.run_training_epoch()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 457, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 633, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 611, in optimizer_closure
    model_ref.backward(self, closure_loss, optimizer, opt_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/hooks.py", line 179, in backward
    self.trainer.scaler.scale(loss).backward()
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 156, in scale
    assert outputs.is_cuda
AssertionError
Traceback (most recent call last):
  File "/opt/code/deeplabv3_lightning.py", line 269, in <module>
    'pred': predict
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/opt/code/deeplabv3_lightning.py", line 224, in train
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 895, in fit
    self.ddp_train(task, model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 526, in ddp_train
    self.run_pretrain_routine(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1091, in run_pretrain_routine
    self.train()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 374, in train
    self.run_training_epoch()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 457, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 633, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 611, in optimizer_closure
    model_ref.backward(self, closure_loss, optimizer, opt_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/hooks.py", line 179, in backward
    self.trainer.scaler.scale(loss).backward()
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 156, in scale
    assert outputs.is_cuda
AssertionError
Traceback (most recent call last):
  File "deeplabv3_lightning.py", line 269, in <module>
    'pred': predict
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "deeplabv3_lightning.py", line 224, in train
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 910, in fit
    self.spawn_ddp_children(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 442, in spawn_ddp_children
    self.ddp_train(local_rank, model, is_master=True)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 526, in ddp_train
    self.run_pretrain_routine(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1091, in run_pretrain_routine
    self.train()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 374, in train
    self.run_training_epoch()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 457, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 633, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 611, in optimizer_closure
    model_ref.backward(self, closure_loss, optimizer, opt_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/hooks.py", line 179, in backward
    self.trainer.scaler.scale(loss).backward()
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 156, in scale
    assert outputs.is_cuda
AssertionError
Traceback (most recent call last):
  File "/opt/code/deeplabv3_lightning.py", line 269, in <module>
    'pred': predict
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/opt/conda/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/opt/code/deeplabv3_lightning.py", line 224, in train
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 895, in fit
    self.ddp_train(task, model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 526, in ddp_train
    self.run_pretrain_routine(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1091, in run_pretrain_routine
    self.train()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 374, in train
    self.run_training_epoch()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 457, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 633, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 611, in optimizer_closure
    model_ref.backward(self, closure_loss, optimizer, opt_idx)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/hooks.py", line 179, in backward
    self.trainer.scaler.scale(loss).backward()
  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 156, in scale
    assert outputs.is_cuda
AssertionError
Exception ignored in: <function tqdm.__del__ at 0x7ff26f244a70>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1086, in __del__
  File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1293, in close
  File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1471, in display
  File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1089, in __repr__
  File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1433, in format_dict
TypeError: cannot unpack non-iterable NoneType object

Expected behavior
The TensorMetric should have self._device updated to equal the current Trainer device during initialization.
Environment

CUDA:

GPU:

GeForce GTX 1050

available:         True
version:           10.2

Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.6.0.dev20200618
pytorch-lightning: 0.8.0
tensorboard:       2.1.1
tqdm:              4.46.1

System:

OS:                Linux
architecture:

64bit
ELF

processor:         x86_64
python:            3.8.3
version:           #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020

The bug is also present on AWS p3.8xlarge instances using the same environment but with 4 Nvidia Tesla V100s

Just discovered that this seems to be fixed in release 0.8.1
Thanks for the great work!

Nevermind, issue still persists

I understand now that the metrics must be moved to the appropriate device, but it's not clear what the best way to do this is or if it should be handled automatically by the LightningModule somehow.
Currently I'm just setting self.calc_loss = FocalTverskyLoss(0.7, 0.3, 4./3.) in my LightningModule's __init__ function. Is there a way to register these metrics so that they'll be placed on the correct device when training with DDP etc.?

Can metrics not be used for the loss computation?

they should be ablet to...
@justusschock @SkafteNicki

This shall be possible after you call super().__init__() , since after that assignments of nn.Modules (which is the baseclass for Metrics) should be registered automatically

Based on the stack-trace above though, this movement off of the cpu and onto the cuda device doesn't seem to be happening. I'll try to post a minimal example using a metric as an attribute of a lightning module soon. The stacktrace is from a module though that uses the above TverskyLossMetric code, which also calls super().__init().

>>> from pytorch_lightning.metrics import Accuracy
>>> from pytorch_lightning import LightningModule
>>> class DummyModule(LightningModule):
...     def __init__(self):
...         super().__init__()
...         self.acc = Accuracy()
...     def forward(self, x, y):
...         return self.acc(x, y)
...
>>> dummy = DummyModule()
>>> dummy.device
device(type='cpu')
>>> dummy.acc.device
device(type='cpu')
>>> dummy.to('cuda')
DummyModule(
  (acc): Accuracy()
)
>>> dummy.device
device(type='cuda')
>>> dummy.acc.device
device(type='cpu')

Unfortunately I could also reproduce this behaviour for LightningModules. So with nested modules only the device (and type) of the main model is updated, but not on the inner models.

I guess this can be fixed by introducing a buffer to each metric, which is a feature we need to implement regardless of this problem if we want to accumulate stats over multiple batches. For now, simply changing the base class to something like:
class Metric(DeviceDtypeModuleMixin, torch.nn.Module, ABC):
    """
    Abstract base class for metric implementation.

    Should be used to implement metrics that
    1. Return multiple Outputs
    2. Handle their own DDP sync
    """

    def __init__(self, name: str):
        """
        Args:
            name: the metric's name

        """
        super().__init__()
        self.name = name
        self._dtype = torch.get_default_dtype()
        self._device = torch.device('cpu')

        self.register_buffer('state', torch.zeros(1))
        self.reset_state()

    @abstractmethod
    def forward(self, *args, **kwargs) -> torch.Tensor:
        """
        Implements the actual metric computation.

        Returns:
            metric value

        """
        raise NotImplementedError

    def reset_state(self):
        self.state.zeros_()

    @property
    def device(self):
        """ From https://github.com/pytorch/pytorch/issues/7460#issuecomment-623200989 """
        devices = ({param.device for param in self.parameters()} |
                   {buf.device for buf in self.buffers()})
        if len(devices) != 1:
            raise RuntimeError('Cannot determine device: {} different devices found'
                               .format(len(devices)))
        return next(iter(devices))

Should work.

I think I fixed this recently in #2657

The cpu device seems to be hardcoded here

It just looks like that, but this is just the initial value there. The DeviceDtypeMixin takes care of updating the device.
I agree with @SkafteNicki. All metrics that save tensors must register them as buffers.


Why is it worth to add this package?
Dependencies pulseaudio
Home page URL
No response
Source code URL
https://github.com/alsa-project/alsa-lib
Packaging policy acknowledgement

[X] I have read the Termux-Pacman Package Addition Policy and understand that my request will be denied in case of violation.

Additional information
Dependencies pulseaudio

build with
-D alsa=disabled
because https://github.com/termux/termux-packages/issues/17692#issuecomment-1690419567


Fixed page crash (Merged with chrisccerami's) and resulting other resulting bugs

Tried to fix #355 to pass the test

You can force push to get travis to re-run tests, you don't have to open a new PR git push --force
As to why it's failing? I have no idea, you changed a ton of stuff in this PR. I merged in #333 which included parts of #336 maybe? idk. This is all a bit strange

Oh. Sorry, I'm kinda new to git. It's nice to learn about that. Thanks!


Wait Command Support

Is the wait command not yet supported in Redisson?
https://redis.io/commands/wait

Will be implemented in https://github.com/redisson/redisson/issues/317


Package page - multiple error messages on the console

These errors seem to be related to the integration with Adobe Analytics

These are all warnings, not errors so nothing is breaking. Additionally, this on the owners of the analytics library to set the sameSite attribute with secure, we do not control this

Do they know they need to set it?

This is likely a wontfix from the analytics team at this time - they've explained this is only a warning present on Firefox due to stricter tracking settings and won't be present on other browsers. Doesn't impact functionality


Implement google/grpc.

Implement google/grpc module to make grpc calls from snet-admin-dapp.

google/grpc could not be implemented because it has dependency with docker.


[Backport v2.7-branch] boards: arm: stm32l562e_dk and nucleo_l552ze_q add openocd support

OpenOCD was partially implemented in board.cmake,
it is now functional.
Backport the https://github.com/zephyrproject-rtos/zephyr/pull/39406 to v2.7 branch
fixes https://github.com/zephyrproject-rtos/zephyr/issues/44352
Signed-off-by: Alexandre Bourdiol<EMAIL_ADDRESS>

@stephanosio, @carlescufi  - do you have superpowers to Update Branch?

Couldn't update "backport-39406-to-v2.7-branch": Cannot force-push to this protected branch

There is a slight issue with the way that the branch protection rules are configured at the moment. I will take a look at fixing those.

Rebased


MOD: CheckPoint - RollBack

The following functionalities have been  completed. Namely

Coding all Functionalities
Tested all Functionalities
If Job/View is not present on the Interim. Delete the same on the Production.

This is the checkpoint before I go ahead and add the rich module for a more interactive console output.

Push again


There is no issue template  of this project "

Let me know if you want to add

Not required

Not required


Build error for BL702

When building any of the example programs with make CHIP=bl702 BOARD=bl702dk, the following error occurs:
/home/tony/bl_mcu_sdk/drivers/lhal/config/bl702/device_table.c:205:19: error: 'BFLB_DEVICE_TYPE_KYS' undeclared here (not in a function); did you mean 'BFLB_DEVICE_TYPE_AES'?
  205 |       .dev_type = BFLB_DEVICE_TYPE_KYS,
      |                   ^~~~~~~~~~~~~~~~~~~~
      |                   BFLB_DEVICE_TYPE_AES
make[3]: *** [build_out/drivers/lhal/CMakeFiles/lhal.dir/build.make:368: build_out/drivers/lhal/CMakeFiles/lhal.dir/config/bl702/device_table.c.obj] Error 1
make[2]: *** [CMakeFiles/Makefile2:351: build_out/drivers/lhal/CMakeFiles/lhal.dir/all] Error 2

This error does not occur for any other target (BL602/BL616/BL808).

That's strange, I'm using mcu sdk with BL706 just fine. I will take a look.

@gamelaster the last commit that builds is 3c19b600a0cb8f618b448e2514515dbed5da8a8e , anything later gives me that error.

@kaofishy ah then makes sense, I'm on older commit. Well, just rollback to the working commit, everything should work fine.

Has fixed.


Customizable text

for the customizable text,

maybe it will be done with shadow <slot> elements?
maybe it could be component html attributes?
most likely: it might have to be an installation option for namaka specified in prepare-components.ts and passed into a special "options model" or some such madness
maybe it actually can be a shadow <slot> of the whole modal system, (<namaka-modals>).. but it'd still have to be specified during the modal installation in install-modal-system.ts

Originally posted by @chase-moskal in https://github.com/haumea-group/namaka/issues/83#issuecomment-1130266114

@zenkyuv closing this ticket in favor of newer #119


sink(ticdc): rename eventsink and remove useless type declaration

What problem does this PR solve?

Issue Number: ref https://github.com/pingcap/tiflow/issues/8132
What is changed and how it works?

Rename eventsink to dmlsink
Remove useless type declaration

Check List
Tests

Unit test

Questions
Will it cause performance regression or break compatibility?
No
Do you need to update user documentation, design documentation or monitoring documentation?
No
Release note
None

/run-all-tests

/run-all-tests

/merge

Thanks for your review!  💚 💙 💜 💛 ❤

/run-kafka-integration-test
network issue