Zabbix plugin not working.

Hi Openbaton Team,
I 'm trying to setup openbaton mainly for VNF onboarding and monitoring which is the key component I 'm looking for in my VNFM.
In my VDU, I 've the below configuration for monitoring.
 "monitoring_parameter":[
              "agent.ping"
            ],
            "fault_management_policy":[
              {
                "name":"AAA Server not available",
                "criteria":[
                  {
                    "parameter_ref":"agent.ping",
                    "function":"nodata(1m)",
                    "vnfc_selector":"at_least_one",
                    "comparison_operator":"=",
                    "threshold":"1"
                  }
                ],
                "severity":"CRITICAL",
                "period":60

Still, I don't see OpenBaton monitoring my VNF(s).
I 've my Zabbix server 3.0 running, with the configurations done for Auto discovery and have configured the alert script manually to trigger the fault to Zabbix driver.
From Zabbix server logs, I can see the event is being triggered and can see the request coming to the driver.

13-6-2017 11:56:48 [Thread-3] [DEBUG] org.openbaton.monitoring.agent.ZabbixMonitoringAgent - Received: {"Trigger":"Zabbix discoverer processes more than 75% busy","Trigger status":"PROBLEM","Trigger severity":"Average","Item values":"Zabbix busy discoverer processes, in %(Zabbix server:zabbix[process,discoverer,avg,busy]):100 %","Original event ID":"815"}
13-6-2017 11:56:48 [Thread-3] [DEBUG] org.openbaton.monitoring.agent.ZabbixMonitoringAgent -

So, it seems my zabbix plugin is not able to parse that data and take any action on that.
Here is configuration done on Zabbix under Configuration > Actions.
`{
"Trigger":"{TRIGGER.NAME}",
"Trigger status":"{TRIGGER.STATUS}",
"Trigger severity":"{TRIGGER.SEVERITY}",

"Item values":"{ITEM.NAME1}({HOST.NAME1}:{ITEM.KEY1}):{ITEM.VALUE1}",

"Original event ID":"{EVENT.ID}"
}`

I 'm not sure, if this is the template that is expected by the Zabbix driver. I couldn't find relevant documentation to configure this.
The official documentation says

When using the method createThreshold provided by the plugin, it automatically creates an action executed when the specific condition is met. If the threshold is crossed (the status of the trigger goes from OK to PROBLEM or viceversa) the action is performed.

So in my case, I didn't see the action was created automatically, so I manually created it using Zabbix documentation.
My another issue is how do I use this createThreshold method? Is it expected in my VDU/NSD? Couldn't find any document related this also. Also where and how do I mention the healing action?
I would like the action here to restart the VNF, the scripts inside VNF (EMS) can't perform this action and this has to be done by the VIM, so how do I tell the VIM from NFVO to perform this healing action?
Please suggest.
Thanks,
Ashish

Hi Ashish,
Thanks for writing us, I will answer your questions in line.

Still, I don't see OpenBaton monitoring my VNF(s).

If you want to monitor the VNFs deployed by Open Baton, you should set the property “nfvo.monitoring.ip” in openbaton.properties, with the actual IP address of Zabbix Server. After the network service goes to ACTIVE, you should see the VNF monitored by Zabbix.

So, it seems my zabbix plugin is not able to parse that data and take any action on that.

It is correct, because it expects the data to be formatted in a certain way, since it is supposed to create the ACTION automatically in the method createThreshold.

I'm not sure, if this is the template that is expected by the Zabbix driver. I couldn't find relevant documentation to configure this.

You are not supposed to create the ACTION since the Zabbix plugin takes care of it.

So in my case, I didn't see the action was created automatically, so I manually created it using Zabbix documentation. ... My another issue is how do I use this createThreshold method? Is it expected in my VDU/NSD? Couldn't find any document related this also

The createThreshold method is called by the Fault Management System (FMS). Are you using it? Otherwise you need to create an application that does it (check out the section “using it via monitoringplugincaller” here "http://openbaton.github.io/documentation/zabbix-plugin/ ).

Also where and how do I mention the healing action? ... I would like the action here to restart the VNF, the scripts inside VNF (EMS) can't perform this action and this has to be done by the VIM, so how do I tell the VIM from NFVO to perform this healing action?

The healing action is performed by the FMS, you can specify whether/when to execute it in Drools rules. (check out section "how the heal method works” here http://openbaton.github.io/documentation/fault-management/)
Hope to have helped you and please don’t hesitate to write us.
Cheers,
Marcello

Hi @marcellom,
Thanks, for the prompt response, really appreciate it.
Please find my response in line.

If you want to monitor the VNFs deployed by Open Baton, you should set the property “nfvo.monitoring.ip” in openbaton.properties, with the actual IP address of Zabbix Server. After the network service goes to ACTIVE, you should see the VNF monitored by Zabbix.

I 've the zabbix ip set inside 'nfvo.monitoring.ip' *
openbaton-nfvo.properties:nfvo.monitoring.ip=<IP_ADDRESS>
I don't have file openbaton.properties in my system, is that something to worry about? However I can see this variable being read by the cloud-init file and it installs the zabbix plugin inside my VNF and configures it.

You are not supposed to create the ACTION since the Zabbix plugin takes care of it.

Nice, I was expecting zabbix plugin to do it. For some reason my zabbix plugin is not creating that action and I don't see any errors in the log during VNF onboarding. Can you help me with some pointers where to look for any error in my zabbix plugin.
My NSD looks like
 "monitoring_parameter":[
              "agent.ping"
            ],
            "fault_management_policy":[
              {
                "name":"AAA Server not available",
                "isVNFAlarm": true,
                "criteria":[
                  {
                    "parameter_ref":"agent.ping",
                    "function":"nodata(1m)",
                    "vnfc_selector":"at_least_one",
                    "comparison_operator":"=",
                    "threshold":"1"
                  }
                ],
                "severity":"CRITICAL",
                "period":5

Anything wrong here with the descriptor?

The createThreshold method is called by the Fault Management System (FMS). Are you using it? Otherwise you need to create an application that does it (check out the section “using it via monitoringplugincaller” here "http://openbaton.github.io/documentation/zabbix-plugin/ ).

Yes, I 'm using FMS or I want to use it even though it's not working for me.
In the fm logs, all I see is this.
2017-06-14 10:24:32.978  INFO 58152 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 10:24:37.981  INFO 58152 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 10:24:42.983  INFO 58152 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 10:24:47.989  INFO 58152 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 10:24:52.994  INFO 58152 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...

The healing action is performed by the FMS, you can specify whether/when to execute it in Drools rules. (check out section "how the heal method works” here http://openbaton.github.io/documentation/fault-management/)

Where can I find this drool rules inside my FMS, just to make sure they exist in my setup. I found the src file of the rules in github https://github.com/openbaton/fm-system/blob/master/src/main/resources/rules/vnfm_fault_correlator.drl
But I can't see that file on my system nor the fms.properties file but I 've /etc/openbaton/openbaton-fms.properties (installed by the bootstrapping script).
My service seems to be running.
root@Ubuntu14:/etc/openbaton# service openbaton-fms status
openbaton-fms start/running, process 58152

Hope the provided info can help you understand the issue or please let me know if you need any additional information from my setup.
Thanks.
Regards,
Ashish

Hi,

I don't have file openbaton.properties in my system, is that something to worry about?

If the zabbix-agent gets installed in the vm, you have this property set. With openbaton.properties I just meant the configuration file of the NFVO, for you the name is just different because of the bootstrap.

Anything wrong here with the descriptor?

It looks correct.

Yes, I 'm using FMS or I want to use it even though it's not working for me.

Is the FMS in the same machine of the NFVO? If yes can you show me the properties of the FMS? Looks like it cannot connect to the Zabbix plugin, and this is the problem why you don't get the action created.

Where can I find this drool rules inside my FMS...  I found the src file of the rules in github

Yes, they are in there.
Note:  The dafault behaviour of the FMS is to performs the heal action upon a VNF alarm and the switch to standby action upon a VR alarm. So actually you don't have to modify the Drools rules, because you have the field "isVNFAlarm" set to true. Once the FMS will get the VNF alarm, you'll get the heal executed.

Hi @marcellom
Yes, my FMS is on the same machine as the NFVO.

period time to check the VNF to monitor (in seconds)
fms.monitoringcheck=60
period time to check the redundancy of a VNF (in seconds)
fms.redundancycheck=60
Open Baton FM port
server.port=9000
#########################################
########## Logging properties ###########
#########################################
logging.level.org.springframework=INFO
logging.level.org.hibernate=INFO
logging.level.org.openbaton=INFO
logging.file=/var/log/openbaton/openbaton-fm.log
################################################
####### Open Baton Related properties ##########
################################################
nfvo.ip=<IP_ADDRESS>
nfvo.port=8080
nfvo-usr=Admin
nfvo-pwd=zabbix
Default project name is "default"
nfvo.project.name=
##########################################################
####### Open Baton Monitoring Plugin properties ##########
##########################################################
plugin.monitoring.name=zabbix
plugin.monitoring.type=zabbix-plugin
################################################
####### Rabbit MQ Related properties ##########
################################################
spring.rabbitmq.host=
spring.rabbitmq.port=5672
spring.rabbitmq.management.port=15672
spring.rabbitmq.username=admin
spring.rabbitmq.password=openbaton
##############################################
####### Database Related properties ##########
##############################################
DB properties
spring.datasource.username=fmsuser
spring.datasource.password=password
mysql jdbc
spring.datasource.url=jdbc:mysql://localhost:3306/faultmanagement
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.database-platform=org.hibernate.dialect.MySQLDialect
spring.datasource.validationQuery=SELECT 1
spring.datasource.testOnBorrow=true
hibernate properties
spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=update
#########################################
############### GSON ####################
#########################################
spring.http.converters.preferred-json-mapper=gson

Just to make sure, FMS is able to connect to Zabbix plugin, I restarted the FMS service.
root@Ubuntu14:/etc/openbaton# service openbaton-fms restart openbaton-fms stop/waiting openbaton-fms start/running, process 34804 root@Ubuntu14:/etc/openbaton# root@Ubuntu14:/etc/openbaton# service openbaton-fms status openbaton-fms start/running, process 34804
Below is the logs from fms after restart.
2017-06-14 13:33:59.758  INFO 34804 --- [main] o.openbaton.faultmanagement.Application  : Starting Application on Ubuntu14 with PID 34804 (/usr/lib/openbaton/fms/openbaton-fms-1.2.4.jar started by root in /)
2017-06-14 13:33:59.762  INFO 34804 --- [main] o.openbaton.faultmanagement.Application  : No active profile set, falling back to default profiles: default
2017-06-14 13:33:59.882  INFO 34804 --- [main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@442562f2: startup date [Wed Jun 14 13:33:59 IST 2017]; root of context hierarchy
2017-06-14 13:34:02.631  INFO 34804 --- [main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'beanNameViewResolver' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter.class]]
2017-06-14 13:34:03.197  INFO 34804 --- [main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.amqp.rabbit.annotation.RabbitBootstrapConfiguration' of type [class org.springframework.amqp.rabbit.annotation.RabbitBootstrapConfiguration$$EnhancerBySpringCGLIB$$84fabf6d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-06-14 13:34:03.383  INFO 34804 --- [main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$be8f013f] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-06-14 13:34:04.171  INFO 34804 --- [main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 9000 (http)
2017-06-14 13:34:04.190  INFO 34804 --- [main] o.apache.catalina.core.StandardService   : Starting service Tomcat
2017-06-14 13:34:04.194  INFO 34804 --- [main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.0.32
2017-06-14 13:34:04.328  INFO 34804 --- [localhost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2017-06-14 13:34:04.329  INFO 34804 --- [localhost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 4451 ms
2017-06-14 13:34:04.865  INFO 34804 --- [localhost-startStop-1] o.s.b.c.e.ServletRegistrationBean        : Mapping servlet: 'dispatcherServlet' to [/]
2017-06-14 13:34:04.883  INFO 34804 --- [localhost-startStop-1] o.s.b.c.embedded.FilterRegistrationBean  : Mapping filter: 'characterEncodingFilter' to: [/*]
2017-06-14 13:34:04.883  INFO 34804 --- [localhost-startStop-1] o.s.b.c.embedded.FilterRegistrationBean  : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2017-06-14 13:34:04.883  INFO 34804 --- [localhost-startStop-1] o.s.b.c.embedded.FilterRegistrationBean  : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2017-06-14 13:34:04.883  INFO 34804 --- [localhost-startStop-1] o.s.b.c.embedded.FilterRegistrationBean  : Mapping filter: 'requestContextFilter' to: [/*]
2017-06-14 13:34:05.375  INFO 34804 --- [main] j.LocalContainerEntityManagerFactoryBean : Building JPA container EntityManagerFactory for persistence unit 'default'
2017-06-14 13:34:05.449  INFO 34804 --- [main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [
        name: default
        ...]
2017-06-14 13:34:05.608  INFO 34804 --- [main] org.hibernate.Version                    : HHH000412: Hibernate Core {5.1.0.Final}
2017-06-14 13:34:05.612  INFO 34804 --- [main] org.hibernate.cfg.Environment            : HHH000206: hibernate.properties not found
2017-06-14 13:34:05.615  INFO 34804 --- [main] org.hibernate.cfg.Environment            : HHH000021: Bytecode provider name : javassist
2017-06-14 13:34:05.685  INFO 34804 --- [main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
2017-06-14 13:34:05.701  WARN 34804 --- [main] org.hibernate.orm.deprecation            : HHH90000006: Attempted to specify unsupported NamingStrategy via setting [hibernate.ejb.naming_strategy]; NamingStrategy has been removed in favor of the split ImplicitNamingStrategy and PhysicalNamingStrategy; use [hibernate.implicit_naming_strategy] or [hibernate.physical_naming_strategy], respectively, instead.
2017-06-14 13:34:06.523  INFO 34804 --- [main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.MySQLDialect
2017-06-14 13:34:07.620  INFO 34804 --- [main] rmationExtractorJdbcDatabaseMetaDataImpl : HHH000262: Table not found: Alarm
2017-06-14 13:34:07.620  INFO 34804 --- [main] rmationExtractorJdbcDatabaseMetaDataImpl : HHH000262: Table not found: Alarm
2017-06-14 13:34:10.264  INFO 34804 --- [main] o.d.c.k.builder.impl.KieRepositoryImpl   : KieModule was added: MemoryKieModule[releaseId=org.default:artifact:1.0.0-SNAPSHOT]
2017-06-14 13:34:11.209  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@442562f2: startup date [Wed Jun 14 13:33:59 IST 2017]; root of context hierarchy
2017-06-14 13:34:11.294  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/alarm/vnf],methods=[POST],consumes=[application/json],produces=[application/json]}" onto public org.openbaton.catalogue.mano.common.monitoring.Alarm org.openbaton.faultmanagement.receivers.RestEventReceiver.receiveVnfNewAlarm(org.openbaton.catalogue.mano.common.faultmanagement.VNFAlarmNotification)
2017-06-14 13:34:11.295  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/alarm/vnf],methods=[PUT],consumes=[application/json],produces=[application/json]}" onto public org.openbaton.catalogue.mano.common.monitoring.Alarm org.openbaton.faultmanagement.receivers.RestEventReceiver.receiveVnfStateChangedAlarm(org.openbaton.catalogue.mano.common.faultmanagement.VNFAlarmStateChangedNotification)
2017-06-14 13:34:11.295  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/alarm/vr],methods=[POST],consumes=[application/json],produces=[application/json]}" onto public org.openbaton.catalogue.mano.common.monitoring.Alarm org.openbaton.faultmanagement.receivers.RestEventReceiver.receiveVRNewAlarm(org.openbaton.catalogue.mano.common.faultmanagement.VirtualizedResourceAlarmNotification)
2017-06-14 13:34:11.296  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/alarm/vr],methods=[PUT],consumes=[application/json],produces=[application/json]}" onto public org.openbaton.catalogue.mano.common.monitoring.Alarm org.openbaton.faultmanagement.receivers.RestEventReceiver.receiveVRStateChangedAlarm(org.openbaton.catalogue.mano.common.faultmanagement.VirtualizedResourceAlarmStateChangedNotification)
2017-06-14 13:34:11.299  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2017-06-14 13:34:11.299  INFO 34804 --- [main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2017-06-14 13:34:11.358  INFO 34804 --- [main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-06-14 13:34:11.359  INFO 34804 --- [main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-06-14 13:34:11.430  INFO 34804 --- [main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-06-14 13:34:12.055  INFO 34804 --- [main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2017-06-14 13:34:12.064  INFO 34804 --- [main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase -2147482648
2017-06-14 13:34:12.064  INFO 34804 --- [main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase<PHONE_NUMBER>
2017-06-14 13:34:12.150  INFO 34804 --- [SimpleAsyncTaskExecutor-1] o.s.a.r.c.CachingConnectionFactory       : Created new connection: SimpleConnection@dc76548 [delegate=amqp://admin@<IP_ADDRESS>:5672/]
2017-06-14 13:34:12.151  INFO 34804 --- [SimpleAsyncTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin         : Auto-declaring a non-durable, auto-delete, or exclusive Queue (nfvo.fm.nsr.create) durable:false, auto-delete:true, exclusive:false. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2017-06-14 13:34:12.151  INFO 34804 --- [SimpleAsyncTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin         : Auto-declaring a non-durable, auto-delete, or exclusive Queue (nfvo.fm.vnf.events) durable:false, auto-delete:true, exclusive:false. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2017-06-14 13:34:12.151  INFO 34804 --- [SimpleAsyncTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin         : Auto-declaring a non-durable, auto-delete, or exclusive Queue (nfvo.fm.nsr.delete) durable:false, auto-delete:true, exclusive:false. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2017-06-14 13:34:12.314  INFO 34804 --- [main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 9000 (http)
2017-06-14 13:34:12.317  INFO 34804 --- [main] o.openbaton.faultmanagement.utils.Utils  : Waiting until NFVO is available...
2017-06-14 13:34:12.317  INFO 34804 --- [main] o.openbaton.faultmanagement.utils.Utils  : NFVO is listening on port 8080 at <IP_ADDRESS>
2017-06-14 13:34:12.989  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 13:34:17.999  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 13:34:23.008  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 13:34:28.013  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 13:34:33.023  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 13:34:38.028  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...
2017-06-14 13:34:43.037  INFO 34804 --- [main] org.openbaton.faultmanagement.Starter    : Waiting until Monitoring Plugin (of type zabbix-plugin) is available...

If you can notice some abnormality here.
Thanks,
Ashish

Your log says that the zabbix-plugin is not started. Could you please check it?

Hi @gc4rella,
That's strange. I can see my zabbix pluggin running.
root@Ubuntu14:/etc/openbaton# service openbaton-plugin-monitoring-zabbix status
openbaton-plugin-monitoring-zabbix start/running, process 57968

I tried restarting it and below are the logs from plugin.
14-6-2017 02:44:55 [main] [DEBUG] org.openbaton.monitoring.agent.ZabbixMonitoringAgent - Loading properties from external-properties-file: /etc/openbaton/openbaton-plugin-monitoring-zabbix.properties
14-6-2017 02:44:55 [main] [DEBUG] org.openbaton.monitoring.agent.ZabbixMonitoringAgent - Loaded properties: {notification-receiver-server-port=8010, external-properties-file=/etc/openbaton/openbaton-plugin-monitoring-zabbix.properties, client-request-frequency=10, password-zbx=zabbix, notification-receiver-server-context=/zabbixplugin/notifications, zabbix-host=<IP_ADDRESS>, type=zabbix-plugin, zabbix-plugin-ip=<IP_ADDRESS>, history-length=250, user-zbx=Admin, zabbix-server-version=3.0}
14-6-2017 02:44:55 [pool-3-thread-1] [INFO] org.openbaton.monitoring.agent.ZabbixSender -
14-6-2017 02:44:55 [pool-3-thread-1] [INFO] org.openbaton.monitoring.agent.ZabbixSender - Trying to connect to Zabbix at url: http://<IP_ADDRESS>/zabbix/api_jsonrpc.php...
14-6-2017 02:44:55 [main] [ERROR] org.openbaton.monitoring.agent.ZabbixMonitoringAgent - Exception while updating hosts:Zabbix Server is not reachable
14-6-2017 02:44:55 [main] [DEBUG] org.openbaton.monitoring.agent.ZabbixMonitoringAgent - Notification receiver server running on url: /0:0:0:0:0:0:0:0:8010 port:8010
14-6-2017 02:44:56 [pool-3-thread-1] [INFO] org.openbaton.monitoring.agent.ZabbixSender - Connected to Zabbix 3.0.9 at url: http://<IP_ADDRESS>/zabbix/api_jsonrpc.php
14-6-2017 02:44:56 [pool-3-thread-1] [DEBUG] org.openbaton.monitoring.agent.ZabbixSender - trying autentication..
14-6-2017 02:44:56 [pool-3-thread-1] [DEBUG] org.openbaton.monitoring.agent.ZabbixSender - Authenticated to Zabbix Server http://<IP_ADDRESS>/zabbix/api_jsonrpc.php with TOKEN 8d008c35bedce3f5cbdda71afca122c2

Here is the conf file of my plugin.
zabbix-plugin-ip=<IP_ADDRESS>
# Set client-request-frequency to 0 to disable the caching
type=zabbix-plugin
client-request-frequency=10
history-length=250
notification-receiver-server-context=/zabbixplugin/notifications
notification-receiver-server-port=8010
external-properties-file=/etc/openbaton/openbaton-plugin-monitoring-zabbix.properties

## Zabbix Server info

zabbix-host=<IP_ADDRESS>
# zabbix-port=
user-zbx=Admin
password-zbx=zabbix
# Supported Zabbix versions: 2.2 and 3.0
zabbix-server-version=3.0

Even I thought something was wrong between my FMS and Plugin and FMS can reach the plugin.
I installed everything using the bootstrapping method so any version mismatch shouldn't happen.
Regards,
Ashish

Hi Team,
I restarted all the services and enabled the debug mode, after that I could see my FMS is failing to start and stopping abruptly.
I 've attached the debug logs.
fms.txt
Regards,
Ashish

Hi, from the log of the FMS I see that now it connects correctly to the Zabbix Plugin, the problem is when it tries to get the token from the NFVO. Here the logs:
Trying to connect to the NFVO...
Username is: Admin
Password is: zabbix
httpPost is: POST http://<IP_ADDRESS>:8080/oauth/token HTTP/1.1
Status Code [400]: Error signing-in [Bad credentials]
The username and password of the NFVO, in the configuration file of the FMS, are wrong because I think you are using the ones of the Zabbix Server instead. Please set the correct properties "nfvo-usr" and "nfvo-pwd", in the configuration file of the FMS, and restart it.

@ashish235 please let us know if the issue is solved in order to close this issue.

Hi @marcellom,
Yes, i saw that mistake now. Argh! I corrected the password and now I can see a new action was created in zabbix automatically, and it was triggered once my VNF was stopped and healing action was taken.
I will still need to do some configurations and test to have it working end to end.
Great! thank you so much @marcellom and @gc4rella for the help. We can close this issue.
Cheers.


CLI: new onyo new syntax

This mainly updates the argparsing for onyo new to use the new style of CLI syntax and flags like described in issue #286.
Updates argparsing and removes unneeded code of class StoreKeyValuePairsAndAssetsSeparately and uses instead StoreKeyValuePairs for handling key-value pairs like onyo set. Together with this, the tests for onyo new and the demo are updated, to use the new syntax everywhere.
Updates code of onyo new to work with the new arguments passed into it.
Close #287

Codecov Report
Base: 90.40% // Head: 90.50% // Increases project coverage by +0.09% :tada:

Coverage data is based on head (16d32cc) compared to base (35e5b6e).
Patch coverage: 100.00% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #293      +/-   ##
==========================================
+ Coverage   90.40%   90.50%   +0.09%
==========================================
  Files          21       21
  Lines        1470     1443      -27
  Branches      311      305       -6
==========================================
- Hits         1329     1306      -23
+ Misses        111      106       -5
- Partials       30       31       +1

Impacted Files
Coverage Δ

onyo/commands/new.py
95.62% <100.00%> (ø)

onyo/main.py
93.50% <100.00%> (+1.83%)
:arrow_up:

onyo/commands/edit.py
94.02% <0.00%> (-1.56%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

That sounds reasonable. I will add it and merge on Monday then, thanks!

I added to tests_new.py the test test_new_with_keys_overwrite_template() and will merge :)


Can I have ~/.bashrc file?

[thanhviet@localhost ~]$ brew doctor
bash: brew: command not found...

export PATH="$HOME/.linuxbrew/bin:$PATH"

See http://linuxbrew.sh/#installation


[Docs] Tutorial is not clear about toolbar

The last picture in the tutorial (which is great) shows the toolbar at the top of the page. Though, there is no explanation on how to activate it. I found the way to enable it reading a googlegroups thread; but that's an info that should be added to the official tutorial, too. If you don't agree to change template_1.html to include {% cms_toolbar %}, the picture shouldn't show a toolbar :)
Thanks.
ref. https://groups.google.com/d/topic/django-cms/g5jKbdCez0I/discussion

Apologies, I need assistance using git. I have committed the patch to the working copy but I have problems to do the push to my fork. Please help.
Clone the project
$ git clone git@github.com:guandalino/django-cms.git

Cloning into django-cms...
remote: Counting objects: 30615, done.
remote: Compressing objects: 100% (10404/10404), done.
remote: Total 30615 (delta 19514), reused 29081 (delta 18185)
Receiving objects: 100% (30615/30615), 11.67 MiB | 542 KiB/s, done.
Resolving deltas: 100% (19514/19514), done.

Configure remotes
$ cd django-cms/
$ git remote add upstream https://github.com/divio/django-cms.git
$ git fetch upstream
From https://github.com/divio/django-cms
* [new branch]      develop    -> upstream/develop
* [new branch]      hotfix/2.1.1 -> upstream/hotfix/2.1.1
* [new branch]      hotfix/2.1.2 -> upstream/hotfix/2.1.2
* [new branch]      master     -> upstream/master
* [new branch]      release/2.0.0 -> upstream/release/2.0.0
* [new branch]      release/2.1.0 -> upstream/release/2.1.0
* [new branch]      release/2.2 -> upstream/release/2.2

Changed tutorial.rst

Commit
$ git add tutorial.rst
$ git commit -m "Fix for #1023 (docs)."
[develop 815b542] Fix for #1023 (docs).
1 files changed, 1 insertions(+), 0 deletions(-)

$ git push origin master
error: src refspec master does not match any.
error: failed to push some refs to 'git@github.com:guandalino/django-cms.git'

please hop on irc (#django-cms on irc.freenode.net) so we can help you with git.

Ah try git push origin develop


IGAPP-506: Malte translations

This pull request belongs to an issue on our bugtracker.
You can find it there by looking for an issue with the key which is mentioned in the title of this pull request.
It starts with the keyword IGAPP.

 This was quite challenging because we changed fa to pes, and also removed a lot of translation keys. Merging took over 1 hour :D


feat: new feat 10

:tada: This PR is included in version 2.4.0-stage.1 :tada:
The release is available on:

npm package (@stage dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Some suggestions

Consider hosting the images on imgur or maybe github release description? Having them in repo makes it take a while to clone on a 100 kb/s connection.
Add -O2 to cflags; it reveals some warnings that -Werror treats as errors. Happens at least on gcc-10.

I used this raytracer to benchmark an overclock on an ARM board, and it picked it up nicely!
Thank you for this C adaptation.

Those are all good ideas, I will do that when I have more time.


Save/Update-Help -Module parameter should have argument completer

Summary of the new feature / enhancement
Save/Update-Help Module parameter should have an argument completer.
Proposed technical implementation details (optional)
Can use the existing module name completer from Get-Command -Module.

I'm seeing quite a lot of these argument completer issues being opened. IMHO everything that can possibly have a completer should have so I'm all for it. BUT would it make sense to have a piece of work within the team to check that all the commands which ship as part of the language have completers instead of opening each missing one as its own issue ? This is (a) a way of improving the quality and consistency of the experience and (b) setting a good example to developers.

Commenting as myself and not on behalf of the Interactive-UX Working Group
This is something that I'd like to see for productivity from an interactive perspective & I have 2 main thoughts on this other than the proposed implementation makes sense to me

we can either add this natively so it's always there (irrespective of whether interactive or not)
or does it make more sense to externalise all argument completers into a module like TabExpansion++ and then ship it like we do other PowerShell modules from the Gallery and then when we open an Interactive session we load that module like we do PSReadline and don't when pwsh -noprofile -noninteractive is called.

If we externalise this then in future we can reduce the overall package size of PowerShell (which is still complained about) with a lite package which I know has been something that has been discussed many a time before
I know that's beyond the ask here, but if we externalised this then we could also backport too which would be an additional hidden benefit and like @jhoneill mentions may be an easier way to manage this as a bigger improvement piece but at minimum could be done either as a mega issue or project in here

@kilasuit my opinion is core PowerShell cmdlets should have argument completers in-box and not a separate module. See #17283   that importing argument completers via a module causes PowerShell startup time to suffer. The argument completer issues I've opened already exist in TabExpansion++ but could no longer handle the ~10+ second penalty.
The size and performance penalty to having these core argument completers defined in C# is minimal.

The following cmdlets have -Module parameters:

Cmdlet
Has completer

Get-Command
Yes

Export-PSSession
No

Import-PSSession
No

Save-Help
No

Update-Help
No

The following module cmdlets have a -Name parameter:

Cmdlet
Has completer

Get-Module
Yes

Import-Module
Yes

New-Module
No - but shouldn't

Remove-Module
Yes

The question is should completers be added to cmdlets that don't have argument completers to resolve module names?
For consistency, I think Save-Help and Update-Help should have completers.
I think the PSSession cmdlet should not.

@ThomasNieto - I don't necessarily disagree & personally prefer to have them defined in the cmdlet/function definition where possible unless there is an element of potential reusability of the completer like in this case.  elsewhere is something that's desired and shipped inbox & where possible suggest when I've trained others that this is best done as part of the the first phase of cmdlet/function development and included when v1 of a module is released, but that's not always a possibility due to time constraints.
Just pointing out additional benefit that could be gained by taking a different course than taking the quick win option.
There would be nothing stopping a possible rewrite in c# of the completers that are available in TE++ similar to this comment from @lzybkr as a new module that could potentially be optimised in how it's loaded when running in an interactive session only but would also require work in engine to make that happen.
However that's another PowerShellGet esk size task which is why I'd rather the WG's discuss both short and longer term approaches here. Interactive-UX should be discussing this today btw!
So to sum up my thoughts on which ways we could go with this more for referencing later on as part of any other argument completer conversations going forward
Quick Wins/Short term/least effort - we add this argument completer (& any others found desirable) directly to this repo for any and all core cmdlets
Longer term/most effort - build a new c# module with argument completers added from te++ ship that inbox in future and potentially also allow to be made available for down level support too.

Thank you @ThomasNieto  for reporting this issue.  The Interactive WG has reviewed this issue and agrees that this should be fixed in future releases.  It would benefit the team to open this up to the community as "Up-For-Grabs".


Add version number string

Add version number string variable for PROJ.4 and geodesic libraries.  Compatible for future versions of PROJ.4 after 4.10.0.  This comes on the heels of this PR https://github.com/OSGeo/proj.4/pull/364
Here's the console output of what this will add:
In [1]: import pyproj

In [2]: pyproj.proj_version_str
Out[2]: '4.9.2'

In [3]: pyproj.geodesic_version_str
Out[3]: '1.44.0'

This a versioning scheme that seems to follow PEP 0440.

Odd,  the test passed in my repository.

Nice, why is it odd @micahcochran ?

@georgeouzou  Github usually integrates Travis-CI testing result.  Most of the other pull requests for pyproj have a check mark ✔ upon all tests passing, and a or an X showing if a test failed.  Travis is a great tool in the toolkit of quality assurance, but it is occasionally buggy.

Oh i got it,  this pull request has neither ✔ or X. Did you try to close and reopen the pull request?

Thanks @jswhit


Other Stream::toList conversions

Found other Collector.toList()s when we were importing it as a static function
Documentation

[ ] I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

[ ] I thought about adding a changelog entry, and added one if I deemed necessary.

there was another subtle situation where we need a mutable list but the UT was not catching the exception


Electron sim card activation

Electron sim card activation screen leads nowhere (unresponsive confirmation button). In same screen browser version presents payment form popup.

@jensck @idokleinman From what I understand popup comes from stripe, if I enable popups I'll be redirected to stripe checkout however it won't work since stripe does not support webviews. Looks like other people are suggesting custom checkout form to go around this issue.

Popup seems to work now.


[0010] Update for writability was incomplete

Which proposal does this relate to?
Proposal 0010 - vk::BufferPointer
Describe the issue or outstanding question.
The Get() method definitions are missing non-const versions.
Additional context
See comment here

@llvm-beanz Please assign to me.

Here is the current spec:
template <struct S, int align>
class vk::BufferPointer {
    vk::BufferPointer(const vk::BufferPointer&);
    vk::BufferPointer& operator=(const vk::BufferPointer&);
    vk::BufferPointer(const uint64_t);
    S& Get() const;
}

Note that the const after Get() does not prevent writing of the pointee. The const here only assures that the BufferPointer itself is not changed by the Get() method.
So I contend that the current spec allows writing as well as reading of the pointee.

Apologies for incorrectly referencing the error. This is the incorrect line:
https://github.com/microsoft/hlsl-specs/blob/main/proposals/0010-vk-buffer-ref.md?plain=1#L71

The Get() method represents the struct const lvalue reference of the pointer to which it is applied.

The Get() method should not return a const lvalue reference (and doesn't in the code).

OK. I will remove const.

@llvm-beanz I believe this can now be closed. PTAL.


I try install jupyter_micropython_kernel with anaconda3

Hi,
How to install jupyter_micropython_kernel with anaconda3 please ?
Thank you

In the readme file now


Unsupported transport profile

Describe the bug
We are trying to connect OpcUA Publisher with a PCS7 OPC UA server with [None, None] SecurityMode. The server is reachable and we are able to connect with other clients. With OPC UA publisher we got this error :
Session creation to endpoint 'opc.tpc://....' failed 9084 time(s). Please verify if server is up and Publisher configuration is correct.
Opc.Ua.ServiceResultException: Unsupported transport profile
   at Opc.Ua.WcfChannelBase.CreateUaBinaryChannel(ApplicationConfiguration configuration, EndpointDescription description, EndpointConfiguration endpointConfiguration, X509Certificate2 clientCertificate, X509Certificate2Collection clientCertificateChain, ServiceMessageContext messageContext)
   at Opc.Ua.WcfChannelBase.CreateUaBinaryChannel(ApplicationConfiguration configuration, EndpointDescription description, EndpointConfiguration endpointConfiguration, X509Certificate2 clientCertificate, ServiceMessageContext messageContext)
   at Opc.Ua.DiscoveryChannel.Create(Uri discoveryUrl, EndpointConfiguration endpointConfiguration, ServiceMessageContext messageContext, X509Certificate2 clientCertificate)
   at Opc.Ua.DiscoveryClient.Create(Uri discoveryUrl, EndpointConfiguration endpointConfiguration, ApplicationConfiguration applicationConfiguration)
   at Opc.Ua.DiscoveryClient.Create(Uri discoveryUrl, EndpointConfiguration configuration)
   at Opc.Ua.Client.CoreClientUtils.SelectEndpoint(String discoveryUrl, Boolean useSecurity, Int32 operationTimeout)
   at OpcPublisher.OpcSession.ConnectSessionAsync(CancellationToken ct) in D:\a\1\s\modules\src\Microsoft.Azure.IIoT.Modules.OpcUa.Publisher\src\OpcSession.cs:line 431
[15:51:11 INF] Connect and monitor session and nodes on endpoint 'opc.tpc://....'.
[15:51:11 ERR] Session creation to endpoint 'opc.tpc://...' failed 9085 time(s). Please verify if server is up and Publisher configuration is correct.
Opc.Ua.ServiceResultException: Unsupported transport profile

   at Opc.Ua.WcfChannelBase.CreateUaBinaryChannel(ApplicationConfiguration configuration, EndpointDescription description, EndpointConfiguration endpointConfiguration, X509Certificate2 clientCertificate, X509Certificate2Collection clientCertificateChain, ServiceMessageContext messageContext)
   at Opc.Ua.WcfChannelBase.CreateUaBinaryChannel(ApplicationConfiguration configuration, EndpointDescription description, EndpointConfiguration endpointConfiguration, X509Certificate2 clientCertificate, ServiceMessageContext messageContext)
   at Opc.Ua.DiscoveryChannel.Create(Uri discoveryUrl, EndpointConfiguration endpointConfiguration, ServiceMessageContext messageContext, X509Certificate2 clientCertificate)
   at Opc.Ua.DiscoveryClient.Create(Uri discoveryUrl, EndpointConfiguration endpointConfiguration, ApplicationConfiguration applicationConfiguration)
   at Opc.Ua.DiscoveryClient.Create(Uri discoveryUrl, EndpointConfiguration configuration)
   at Opc.Ua.Client.CoreClientUtils.SelectEndpoint(String discoveryUrl, Boolean useSecurity, Int32 operationTimeout)
   at OpcPublisher.OpcSession.ConnectSessionAsync(CancellationToken ct) in D:\a\1\s\modules\src\Microsoft.Azure.IIoT.Modules.OpcUa.Publisher\src\OpcSession.cs:line 431

To Reproduce
Steps to reproduce the behavior:

restart OpcuaPublisher and got the above error

Expected behavior
OpcUa Publisher should be able to connect with  PCS7 OPC UA server
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Windows 10 IoT Enterprise
Browser N/A

Additional context
Add any other context about the problem here.

Hi @AlbertoCruzHidalga, the fix should be simple: There is a typo in the url: Instead of opc.tpc it must be opc.tcp.
Please let me know if this is the issue.
Thanks,
Martin


tf.GraphKeys.UPDATE_OPS

do I need to add following code：
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
train_op = optimizer.minimize(loss)
if I want to save the mean and var of BatchNorm in tensorpack？

it's enabled by default in tensorpack

thanks


Removed explicit install of 1.1 runtime in 1.1-sdk-msbuild Dockerfiles

This is no longer necessary because of https://github.com/dotnet/cli/pull/5225

Looks good to me.


[BUG] Managed page.save() Exception calling "Save" with "0" argument(s): "CSOM service exception"

Reporting an Issue or Missing Feature
After switching from Client Id / Secret to Managed Identity my Azure Function PnP PowerShell is unable to save a page
Exception calling "Save" with "0" argument(s): "CSOM service exception"
Expected behavior
Using Managed Identity I expect to be able to do the same operations as Client Id / Secret - in my powershell function I was able to update pages & save with Client Id / Secret - switch to Managed Identity and I'm receiving an error.
Actual behavior
Exception calling "Save" with "0" argument(s): "CSOM service exception"

Steps to reproduce behavior
Azure Function - PowerShell Core / PowerShell 7.2
My Managed Identity has Sites.ReadWrite.All and Sites.FullControl.All
incoming from function
$Page = "Home.aspx"
$SiteURL = "{your site url}"
    Connect-PnPOnline -Url $SiteURL -ManagedIdentity
    Write-Host "Successfully connected"
    $web = Get-PnPWeb
    $page = Get-PnPPage -Identity $Page.Substring($Page.LastIndexOf("/") + 1)

$page.Save()
$page.Publish()
on Save
INFORMATION: Exception calling "Save" with "0" argument(s): "CSOM service exception"
What is the version of the Cmdlet module you are running?
'PnP.PowerShell' = '2.0.45-nightly'
I've tried the latest nightly 2.280-nightly & 2.2.76
Which operating system/environment are you running PnP PowerShell on?

[ ] Windows
[ ] Linux
[ ] MacOS
[ ] Azure Cloud Shell
[x] Azure Functions
[ ] Other : please specify

I'm closing this issue because it is now working with 2.2.80 and seems to work with the 2.2.71 and up. I think that it was not working because it was using an older version of PnP.PowerShell when I went in to debug. I don't think the new version actually changed in the Azure function when I was testing. I added some logging to tell what version and I reset the Azure Function and it's now saving as expected.


Makefile has a race in it

Code-gened files like stats.grpc.pb.cc include stats.pb.h but there is no explicit dependence between the two different results from codegen. Originally spotted by @mehrdada, and @nicolasnoble says that he knows a fix.

Closing for now (hopefully #11986 has fixed this).


Wheeldelta behavior is different on Chrome OSX and Chrome Windows

Environment

Browser: Chrome OSX / Chrome Windows
Version of smooth-scrollbar: 7.2.0

Issue Summary
Wheeldelta behavior is different from OS.
Current Behavior
With the same configuration, i notice that on Windows, vertical scroll is slower than on OSX.
I think this is caused by the utils getDelta.
Online demo
https://codepan.net/gist/8aa4f4fce8158f6c05956cd9b7e63dd7
Suggested solution
It's possible to use third-party library, for example https://www.npmjs.com/package/normalize-wheel to normalize this behaviour?
Thanks in advance

It’s caused by OS/drivers. There’s no guarantee that mouse wheel behaves same cross OS.


[MM-47711] Encapsulate the bridged API so that it can only be accessed by the Desktop App's own scripts

Summary
One potential security concern is the case where an attacker could potentially spoof or provide malicious code posing as a Mattermost application that could potentially gain access to Electron main process functionality, that could involve injecting a malicious payload (for example, using the UPDATE_CONFIGURATION hook).
This PR aims to encapsulate access to these hooks via a module. The module will make a call to pull the hooks into itself, and then set a flag on the main process to disallow any further attempts to gain access to those hooks. Then the only source of those hooks should be the module, which should only be importable via scripts packaged with the Desktop App.
I'm not sure if this is an infallible solution, so I'm adding security review on this as well and setting it as draft for the time being so that we can get some input on if there's any better solutions, or if there's any additional attack vectors we should be worried about for this particular scope.
Ticket Link
https://mattermost.atlassian.net/browse/MM-47711
NONE

This is clever, but clever isn't always good 😅 On the surface, the code looks ok and seems to guarantee a pretty reasonable level of isolation. But you have to remember that this all gets compiled by webpack, and webpack wasn't designed to uphold any of those guarantees.
Here's a piece of black magic that, when included in a malicious server's remote_entry.js, successfully invokes getAPI().quit() after the app has been running for 10 seconds:
let push = window.webpackChunkmattermost_desktop.push
window.webpackChunkmattermost_desktop.push = function (a) {
        a[1] = a[1].map(b => {
                return (module, exports, require) => {
                        setTimeout(_ => {
                                require(6).setAPI().then(_ => require(6).getAPI().quit());
                        }, 10000);
                        return b(module, exports, require);
                };
        })
        push.call(this, a);
}

Closing this for now as I try to figure out a way that will be more secure.


No load on duplicate extension

Hi, i was upscaling some assets of a game im converting for a unity project.
Now when i decompiled the assets the names where: {name}.bmp.png <- this is where the problem is
When you drag the files into qualityscaler it just gives a gray screen but when changing file names to {name}.png they load normaly?
Dont know how you extension checking, but assuming from source you use python, could split name on . and do a -1 index to get extension instead. havent checked out the source do, will do when i have a bit more time.

Hi, thank you :D
yes, actually it counts how many times there are for example .jpg, .png etc in the filename to count files


Cache out of space : Failed to compress build artifacts: signal: broken pipe

When pushing an application to cloud foundry, the buildpack goes and downloads the JDK and IBM JRE components.  On a re-push, it re-downloads it, even though those components are already in cache.    What then happens is that the capacity of the cache runs out gives a Failed to compress build artifacts: signal: broken pipe.
To resolve this temporarily, we tested doing a rm -rf on the cache directory in the compile scripts and this fixes the problem.    The real fix probably should check the cache for the component first before redownloading.

I am having a very similar problem myself. Sometimes the build works, and sometimes it does not. Removing the cache directory does not seem to ensure that the build will work, although maybe that is because I am not doing rm -rf on the correct location.

please reopen if still an issue.


can we train a model with yolov4

can it possible that we can create a model using with yolov4, does it possible ?

As far as I know it's not possible


Azure AD B2C tight integration with SWA (or fixing the preview env redirect link for aad in the cli)

Is your feature request related to a problem? Please describe.
Most people building SaaS apps on SWA and Azure need user login, the login provider that makes sense is b2c (equivalent of cognito).
Problem is: b2c have to be configured manually, with one redirect link (or more, but it's manual), one tenant (can't mix prod and preview) ...
and the real problem arises when you're in a preview env and you want to authenticate a user.
It's not a good practice to create a redirect link on b2c for every preview env.
Describe the solution you'd like
Like Storage, Functions and APIs, CI/CD, environments... SWA should include a good auth provider like B2C pre-packaged into the app.

or maybe find a solution for the mess of integrating B2C with SWA

Describe alternatives you've considered
Tried using the default AAD IDP... yet, doesn't cover the common use case of a "public-facing saas platform".

as you see in the pic above, once the preview env got provisioned, I went to aad and added the redirect link manually.
an easy fast fix, is including this in the tooling, a better fix, is integrating b2c, that's my opinion.

that does it! I strongly believe though this should be managed by SWA (B2C should be integrated imo).
@superhp do you happen to have a template for the action ? not a lot of hands on deck around here :) appreciate it!

Sure. I hope it helps. Just change the parameters to yours in the script.
`update_b2c_uris_job:
if: github.event_name == 'pull_request' && github.event.action != 'closed'
name: Update B2C Redirect URIs
needs: build_and_deploy_job
runs-on: ubuntu-latest
steps:
- name: Authenticate to Azure B2C
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_B2C_CLIENT_CREDENTIALS }}
allow-no-subscriptions: true
enable-AzPSSession: true
- name: "Add temporary environment URI to Azure B2C"
uses: azure/powershell@v1
with:
inlineScript: |
$appUris = az rest --method GET --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>?$select=spa' | ConvertFrom-Json
$newUris = @{'spa' = $appUris.spa}
$temporaryEnvironmentUrl = 'https://<<app_uri>>-${{ github.event.pull_request.number }}.<>.azurestaticapps.net'
if ( -not ($newUris.spa.redirectUris -contains $temporaryEnvironmentUrl)) {
$newUris.spa.redirectUris += $temporaryEnvironmentUrl
}
$newUrisJson = $newUris | ConvertTo-Json -Compress
$newUrisJson = $newUrisJson -replace '([\]*)"', '$1$1"'
az rest --method PATCH --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>' --headers 'Content-Type=application/json' --body $newUrisJson
azPSVersion: "latest"
close_pull_request_job:
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
name: Close Pull Request Job
steps:
- name: Close Pull Request
id: closepullrequest
uses<EMAIL_ADDRESS>with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_TOKEN }}
action: "close"
- name: Authenticate to Azure B2C
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_B2C_CLIENT_CREDENTIALS }}
allow-no-subscriptions: true
enable-AzPSSession: true
- name: "Delete temporary environment URI from Azure B2C"
uses: azure/powershell@v1
with:
inlineScript: |
$appUris = az rest --method GET --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>?$select=spa' | ConvertFrom-Json
$newUris = @{'spa' = $appUris.spa}
$newUris.spa.redirectUris = $newUris.spa.redirectUris -ne 'https://<<app_uri>>-${{ github.event.pull_request.number }}.<>.azurestaticapps.net'
$newUrisJson = $newUris | ConvertTo-Json -Compress
$newUrisJson = $newUrisJson -replace '([\]*)"', '$1$1"'
az rest --method PATCH --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>' --headers 'Content-Type=application/json' --body $newUrisJson
azPSVersion: "latest"`

Using wildcards is also support (but discouraged), however they work quite well for preview environments:
Manifest snippet:
	"replyUrlsWithType": [
		{
			"url": "https://productive-domain-foo.2.azurestaticapps.net/.auth/login/aad/callback",
			"type": "Web"
		},
		{
			"url": "https://*.westeurope.2.azurestaticapps.net/.auth/login/aad/callback",
			"type": "Web"
		}
	]

Sure. I hope it helps. Just change the parameters to yours in the script. Let me know if you have any question.
add_environment_uri_to_b2c_job:
    if: github.event_name == 'pull_request' && github.event.action != 'closed'
    name: Update B2C Redirect URIs
    needs: build_and_deploy_job
    runs-on: ubuntu-latest
    steps:
	  - name: Authenticate to Azure B2C
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_B2C_CLIENT_CREDENTIALS }}
          allow-no-subscriptions: true
          enable-AzPSSession: true
      - name:  Add environment URI to Azure B2C
        uses: azure/powershell@v1
        with:
          inlineScript: |
            $appUris = az rest --method GET --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>?$select=spa' | ConvertFrom-Json
            $newUris = @{'spa' = $appUris.spa}
            $temporaryEnvironmentUrl = 'https://<<app_uri>>-${{ github.event.pull_request.number }}.<<region>>.azurestaticapps.net'
            if ( -not ($newUris.spa.redirectUris -contains $temporaryEnvironmentUrl)) {
              $newUris.spa.redirectUris += $temporaryEnvironmentUrl
            }
            $newUrisJson = $newUris | ConvertTo-Json -Compress
            $newUrisJson = $newUrisJson -replace '([\\]*)"', '$1$1\"'
            az rest --method PATCH --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>' --headers 'Content-Type=application/json' --body $newUrisJson
          azPSVersion: "latest"

close_pull_request_job:
    if: github.event_name == 'pull_request' && github.event.action == 'closed'
    runs-on: ubuntu-latest
    name: Close Pull Request Job
    steps:
      - name: Close Pull Request
        id: closepullrequest
        uses<EMAIL_ADDRESS>        with:
          azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_TOKEN }}
          action: "close"
      - name: Authenticate to Azure B2C
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_B2C_CLIENT_CREDENTIALS }}
          allow-no-subscriptions: true
          enable-AzPSSession: true
      - name: Delete environment URI from Azure B2C
        uses: azure/powershell@v1
        with:
          inlineScript: |
            $appUris = az rest --method GET --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>?$select=spa' | ConvertFrom-Json
            $newUris = @{'spa' = $appUris.spa}
            $newUris.spa.redirectUris = $newUris.spa.redirectUris -ne 'https://<<app_uri>>-${{ github.event.pull_request.number }}.<<region>>.azurestaticapps.net'
            $newUrisJson = $newUris | ConvertTo-Json -Compress
            $newUrisJson = $newUrisJson -replace '([\\]*)"', '$1$1\"'
            az rest --method PATCH --uri 'https://graph.microsoft.com/v1.0/applications/<<app_id>>' --headers 'Content-Type=application/json' --body $newUrisJson
          azPSVersion: "latest"

What is the appropriate permission I have to add to my app for this action?

@timonmasberg I would imagine as a minimum you can try.
permissions:
  contents: read
  id-token: write

Using wildcards is also support (but discouraged), however they work quite well for preview environments:
Manifest snippet:
	"replyUrlsWithType": [
		{
			"url": "https://productive-domain-foo.2.azurestaticapps.net/.auth/login/aad/callback",
			"type": "Web"
		},
		{
			"url": "https://*.westeurope.2.azurestaticapps.net/.auth/login/aad/callback",
			"type": "Web"
		}
	]

not ideal to allow such vulnerability to the production environment I think


Introduce logging level in kubeaudit

In the current scenario kubeaudit emit logs when there is an error/warning but there might be other use cases.
Like getting more information about the healthy k8s resources like the once which are not violating any security policies laid out by kubeaudit.
So for this we need to have different log level for example:
INFO: This would be the most verbose log level
ERROR/WARNING: This would be default log level
@klautcomputing

Sounds good. I think we should introduce a new option for this like: --log-level=INFO for that.
Do we want to introduce something like result.info?

I think you can just configure logrus depending upon that flag to emit different output

I like this idea, although instead of introducing a new flag like --log-level I think we should stick with the conventions followed in other kubernetes command line tools like -v=9 or something?
We already have a verbose flag -v so?

#36 closes this


Select2 input fields are not working as repeater fields.

I am using jQuery repeater and select2.js in a project. they both are working fine separately however when i use a select2 dropdown field inside repeater the select2 dropdown field is showing but the dropdown options are not appearing thus jQuery Repeater is not working with select2.js.
it will be great if there's any fix for this issue? Thank you

The library doesn't support this out of the box but you could probably make it work using the show callback
$('.my-repeater').repeater({
  show: function () {
    // the value of "this" is set to the newly created html element
    setNewElementWithMyCustomBehavior(this)
  }
})

Thank you for your response. show callback was not enough in my cae. i've hooked my function in index callback too.

is there an option to set min and max repeater rows?

is there an option to set min and max repeater rows?

is there an option to set min and max repeater rows?


Nice admin commands collection

The best collection of admin commands I have seen in official documentation

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9f884a3d-3808-7b57-e06a-58ca041634b8
Version Independent ID: c575b37c-a4c9-dfe7-b04d-582160f22f88
Content: Azure virtual machine serial console
Content Source: articles/virtual-machines/windows/serial-console.md
Service: virtual-machines-windows
GitHub Login: @harijay
Microsoft Alias: harijay

@TomaBusarov thanks for the feedback! Happy to hear it was clear and easy to follow :)
@harijay Great work on the doc!


Migrate from Laravel Mix to Vite

This pull request includes changes for migrating from Laravel Mix to Vite outlined in Migration Guide and automated by the Vite Converter.
Before merging, you need to:

Checkout the shift-86323 branch
Run composer update
Run npm install
Review all comments for additional changes
Thoroughly test your application (no tests?, no CI?)

Please send your feedback to<EMAIL_ADDRESS>or share the good vibes on Twitter.

:information_source: Previously, Laravel configured several npm scripts for building assets. For Vite, Laravel configures a dev script for generating and watching your assets (for development) and build script for generating your assets (for production).
Shift attempted to replace commands using these scripts with their new Vite script. However, you may still be referencing these scripts in your code or in deployment scripts.

:information_source: Laravel renamed the environment variables used by Mix to use a VITE_ prefix, instead of MIX_. While Shift automated this change throughout your code, you should check for any additional references which may not be included in your Git repository.

:information_source: Vite requires JavaScript files containing JSX to have a .jsx file extension. Shift did not detect any JSX in files with a .js extension under the resources folder. However, if you are using JSX, be sure the files have a .jsx extension.

:warning: Vite generates frontend assets to the public/build folder. This folder is not tracked by Git within a default Laravel project. As such, Shift added this folder to your .gitignore file.
If you wish to track the generated frontend assets within your project, such as for ease of deployment, you may undo this commit by running git revert d3001fc7.

:alembic: This Shift is still being refined. Please report any issues or suggestions to<EMAIL_ADDRESS>Your feedback is what helps improve the experience for everyone.


Github link for extension is returning 404

https://github.com/davila7/codegpt

Because the repo is not available for public yet.
Cool work BTW @davila7


Automatize enabling windows nodes support

Is your request related to a new offering from AWS?
no
Is your request related to a problem? Please describe.
As stated in the FAQ I need to "apply some configs manually" to "enable Windows support for your EKS cluster".
The FAQ also suggests not to use eksctl but the Powershell scripts provided by AWS. Is there a reason not to use eksctl?
I would like to automate these manual steps.
Describe the solution you'd like.
This is my solution approach:
resource "null_resource" "prepare_windows" {
  provisioner "local-exec" {
    command = "eksctl utils install-vpc-controllers --cluster ${var.infrastructurename}-eks --approve"
    interpreter = ["PowerShell", "-Command"]
  }
  depends_on = [module.eks]
}

But how can I delay the creation of windows worker nodes until vpc controllers have been installed?
Describe alternatives you've considered.
Do everything manually like it's 1798.

Just discovered this Helm chart that might solve the issue: https://github.com/aws-samples/aws-eks-accelerator-for-terraform/tree/main/kubernetes-addons/windows-vpc-controllers
Will check later.

Will check later.

No, I won't. Windows VPC controllers are not needed anymore. Check https://github.com/aws-samples/aws-eks-accelerator-for-terraform/pull/85#issue-1065524718 and https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html#enable-windows-support

Unfortunately, I don't know how much of this can be supported so I'm going to close this issue for now. PRs are always welcome for review


Unable to add SSH GitHub Repo in Embedded Ansible

I'm unable to add SSH github repo to Embedded Anisble. Any idea why it isn't accepting even though the URL is correct? Https URL works but SSH url doesn't work

Update: Morphy version doesn't have this bug. I'm able to add the github repo

Thanks for validating on morphy @kTipSSIoYv !

We still have this issue in master.

So it's a "new" problem on master, then?

Fixed in https://github.com/ManageIQ/manageiq-ui-classic/pull/8122


Added 'New Notebook' task on database dashboard

This PR resolves https://github.com/microsoft/azuredatastudio/issues/4945
This PR includes:

Adding 'New Notebook' task on database dashboard
Adding 'New Notebook' action on the menu when right-clicking db node in Search widget.

Hey Gene, can we reorder the tasks, maybe like so?
New Query   New Notebook
Backup         Restore
Learn how to configure...

@chlafreniere
Fixed.


Fix Modify interaction with hitDetection

Fixes Modify configured with hit detection.
See #12235
Hit detection should ignore any RenderFeatures and features not in the configured source / collection.

This most likely doesn't fix what @mike-000 reported in https://github.com/openlayers/openlayers/issues/12235#issuecomment-825086303

This most likely doesn't fix what @mike-000 reported in #12235 (comment)

That issue is fixed by #12408.


Full refactor

Note:
I've removed most of the unused code such as unused/unfinished tests, Socket.Results, Telnet functionality, commented out functions.
I've also renamed many packages and functions so that they are less misleading/more fitting to the functionality they provide.
I've not tested integration with Telegram bot but I did not change any main logic behind it so I just assumed it works.

go install command can install the package name as binary to PATH, so one can then run ${APP_NAME} as entrypoint the previous commit looks a bit old and strange tho

Dockerfiles should be as little as possible and there is little gain in making it possible to run entrypoint from PATH since it won't make any difference in how user would use or access the image
Any arguments, flags or configuration should be accepted outside of image regardless of how image operates within so it doesn't matter if dish is available from PATH or not, and it's simpler to just use go build.


Updating

We need a way for users to automatically update. This would be especially helpful during closed/open beta: instead of publishing each new release and asking people to update, we could just push out an update and be confident that everyone saw it.
There's a service I saw a while ago called Equinox, which might be our best bet, though I don't fully know how it works. The alternative would be to roll our own updater.
EDIT: Equinox is a paid service. It basically just provides servers for users to download from. The actually update library is called go-update. Incidentally, it looks like the binary patching algorithm used by go-update was designed by Colin Percival.
I wonder how Bitcoin handles updates. Having people download updates from (our) centralized servers seems reasonable for the beta, but after that we'll probably want something more decentralized. I can imagine nodes sharing updates (signed, of course) with their peers.

There are quite a few more features I'd like to implement before considering this "closed." I can just make new issues for them if that makes more sense though.

I think that I would prefer that.
Having these large scope issues makes me uneasy, not because they're bad but just because it makes it seem like no progress is being made.


experiment(modules): Refactor to a more module oriented structure

I'm opening this PR for discussion purposes only.
ESDoc is pretty amazing as is, @h13i32maru has done an incredible job with the overall architecture and the plugin system. That said, I feel like the standard HTML publishing plugin could be a bit more flexible. The opinionated layout is great as a basis, but it doesn't work for every package.
ES6 is fundamentally module based, which means there are lots of libraries (like Lodash for instance) which consist entirely of a flat set of modules that export large numbers of functions. It would be nice to be able to have the top level navigation reflect this, and to generate multiple Function/Variable pages (one per module) in these cases.
I'm not sure if this should be the responsibility of the esdoc-publish-html-plugin, but from this PR you can see that it wouldn't require many changes at all. It could be an option, or maybe we make the plugin more extensible, so other plugins could extend it to change the structure when necessary.

For now, I don't have the plan to this changing.


Super Expression must either be null or a function

Everything was working fine till yesterday
On a clean and fresh install I suddenly started seeing this error:
system.js:4 Uncaught (in promise) Error: Super expression must either be null or a function, not object
  Evaluating https://test.com/widget.js

The stack trace took me to widget.js:
module.exports = {
  test: test
}

Any idea why this is happening?

Closing. Cleared my cache and reinstalled.


kirby make:command foo:bar does not create working command

if the command you created contains a : character, you won't be able to run it.
$ kirby make:command foo:bar --global
The command has been created: /home/roman/.kirby/commands/foo:bar.php
$ kirby foo:bar
The command does not exist

The file foo:bar.php is actually created, but the cli is unable to run it.

If you generate it like kirby make:command foo/bar --global (with a slash instead), running it with the colon works:
$ kirby make:command foo/bar -g
The command has been created: /home/roman/.kirby/commands/foo/bar.php
$ kirby foo:bar
Nice command!

✅


TODO: Complete NBT library

I need to complete NBTDeserializer.java for now.

Scrapped the NBT library, used Querz/NBT


Add test for compatibility with @typescript-eslint/no-unsafe-call rule

Closes #99

It passes because I worked around it in the lint config: https://github.com/xojs/eslint-config-xo-typescript/commit/5d67296b63c34ad9cc1f25b51a3bbde28e58afb6
When you remove linked workaround your test fails.

Yes, I know that you've made a workaround for this. I've reenabled this rule for this test (see https://github.com/SamVerschueren/tsd/pull/120/files#diff-c0316592149a0bea42c336d2b96889d6c72b0191d6ce058ed6df91073cf22f50R7) and I actually added a proof that it fails with a different construct for this rule (see https://github.com/SamVerschueren/tsd/pull/120/files#diff-7aba3dbc285f4db952e54a26767dd82ccf9db80fbc73636de365bf26f9de2b30R26).

Sorry, I missed that. Then I guess it was fixed somewhere.


adding jenkins file

this file will be required on branch where PR checks have to be triggered.
PLEASE NOTE:- edge-ci library referenced in this file is a private repo and is used for internal testing only.

It might makes sense to squash these commits into one and add your PR description to the actual commit message:

this file will be required on branch where PR checks have to be triggered.

I don't think it's that big a deal though.


Fix typo in WithoutRunMount

Signed-off-by: Justin Terry (VM<EMAIL_ADDRESS>

@Random-Liu - This issue is for sure not related to this change. Can we try the test again?

It is related:
I0510 18:37:55.828] !!! 'gofmt -s' needs to be run on the following files:
I0510 18:37:55.828] ./pkg/containerd/opts/spec.go

Not sure how that happened. Had an extra space. Also, missed that error text sorry about that.


Update views to match issues/2955-enable-annotations in Alaveteli

A couple of the views in the theme duplicate stuff from Alaveteli, so this
updates them to use the new enable_annotations setting as those in alaveteli
do.
See https://github.com/mysociety/alaveteli/pull/2956

:+1:


[Bug]: deploy without cache does not work

Error Message and Logs
I just could not get a docker running without cache when running docker
Steps to Reproduce

try to use turborepo with coolify
build next app from turbo repo
try to remove cache and build

Example Repository URL
r
Coolify Version
v4.0.0-beta.355
Are you using Coolify Cloud?
No (self-hosted)
Operating System and Version (self-hosted)
f
Additional Information
r

Seems I'm having the same issue, try to redeploy but coolify wont use my latest docker images.

Maybe the same issue as I have here? https://github.com/coollabsio/coolify/issues/4060


Support disabling validators.

Instead of forcing a new era when the staking module slashes a validator it disables the validator. The default implementation for this forces a new era, but the babe module should be able to do it's own book keeping.
Superseeds #2713

This isn't quite right. Is it only meant as the first step with several follow-up PRs yet to come?
Staking shouldn't be implementing or responding to OnDisable directly at all. The only signal Staking should be responding to is SessionWillEnd. The only modules that should be listening are Grandpa, Babe and (in Polkadot) Parachains. Session should have a configurable type in its trait to tell it when the session is_ended, which for now can be a simple block_number() % SESSION_LENGTH == 0, but will ultimately come from Babe.

Is SessionWillEnd distinct from OnSessionChange?


chore(master): release 1.0.0

:robot: I have created a release beep boop
1.0.0 (2023-03-21)
⚠ BREAKING CHANGES

use eludris-api-types.go (#2)

Code Refactoring

use eludris-api-types.go (#2) (46ff1ae)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/eludris-community/eludris.go/releases/tag/v1.0.0 :sunflower:


copyWith() on generic subclass

First of all, thanks for this amazing package!
@MappableClass(hook: BaseModelHook())
abstract class BaseModel with BaseModelMappable {
  final String id;
  final String path;
  final DateTime? docCreatedAt;
  final DateTime? docUpdatedAt;
  final DocState? docState;

  const BaseModel({
    this.id = '',
    this.path = '',
    this.docCreatedAt,
    this.docUpdatedAt,
    this.docState = DocState.unknown,
  });

@MappableClass()
class Group extends BaseModel with GroupMappable {
  final String name;

  const Group({
    super.id,
    super.path,
    super.docCreatedAt,
    super.docUpdatedAt,
    super.docState,
    this.name = '',
  });

class DatabaseService<T extends BaseModel> {
  final _db = FirebaseFirestore.instance;
  final T Function(DocumentSnapshot<Map<String, dynamic>>) fromFirestore;

  DatabaseService(this.fromFirestore);

  CollectionReference<T> collRef(String collectionPath) => _db.collection(collectionPath).withConverter<T>(
        fromFirestore: (doc, _) => fromFirestore(doc),
        toFirestore: (model, _) => model.toMap(),
      );

  Future<T> add(
    CollectionReference<T> collRef,
    T item,
  ) async {
    final model = item.copyWith(
      docCreatedAt: DateTime.now(),
      docUpdatedAt: DateTime.now(),
      docState: DocState.active,
    );
    // model is now of type BaseModel, not Group

    // this results in: The argument type 'BaseModel' can't be assigned to the parameter type 'T'.
    return collRef.add(model).then(
      (ref) => ref.get().then(
        (doc) => doc.data()!,
        onError: (Object error) {
          LogService.error('Document ${ref.id} does not exist: $error');
          throw Exception('Document ${ref.id} does not exist: $error');
        },
      ),
      onError: (Object error) {
        LogService.error('Failed to add document: $error');
        throw Exception('Failed to add document: $error');
      },
    );
  }
}

The problem in the code is that I need to augment some properties to the item object before adding it to Firestore. Simply setting them doesn't work as the properties are final. When I use copyWith(), it uses the copyWith() of BaseModel, not of Group, which is what T actually is.
Any way of solving this?

The .copyWith does actually respect subclasses. So the instance should be a Group, its just typed as BaseModel since the copyWith method isn't itself generic.
Long story short, simply casting as T should work.
If it doesn't please reopen as a bug.


RatioEstimator abstraction

What does this implement/fix? Explain your changes
It introduces a RatioEstimator abstraction. It will wrap neural networks that process x and theta to estimate ratios. The base class is suggestive of how to create extensions that use alternative data types (dict, etc.) while flexible enough to handle how embedding works and how embedded data is combined. Natural extensions include ratio estimators with specific network architectures such as transformers or convnets.
This implements #992 which is the ratio specific issue for #1046. It also implements the ratio part of #957 and makes the documentation confirm to the shape goals of #1041.
Does this close any currently open issues?
Closes:
#992
Also #1036 since it eliminates a confusingly name and now-irrelevant class.
The others are more general and require more work on subjects related to ratios before closing.
Any relevant code examples, logs, error output, etc?
Nope
Any other comments?
There is a test that doesn't pass, namely #1090, but I think it is not because of my changes.
Checklist
Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask. We're here to
help! This is simply a reminder of what we are going to look for before merging
your code.

[x] I have read and understood the contribution
guidelines
[x] I agree with re-licensing my contribution from AGPLv3 to Apache-2.0.
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have reported how long the new tests run and potentially marked them
with pytest.mark.slow.
[x] New and existing unit tests pass locally with my changes
[x] I performed linting and formatting as described in the contribution
guidelines
[x] I rebased on main (or there are no conflicts with main)

I added a test. it's extremely simple, but it checks that the estimated ratios are the correct size . (like log_prob).

I will follow up after Friday.

@tomMoral I think this is done now.

There is a wip push, but supporting sample dim will require:

[ ] making warnings that it is not allowed to be used in loss function computations
[ ] add broadcasting features for log ratio evaluations

As commented in #1103 , please make sure the all relevant (non renaming related changes) from #1103 are moved to this PR (I think they are all in here anyways).
Otherwise I think that this PR is almost done, no?
The RatioEstimator is now independent from the ConditionalEstimator class family. Two points remain I believe:

The RatioEstimator does its own shape checks etc, but are the shape conventions the same? Are all the shape checks passing for SNRE as well?
there should be a test case for 2D inputs with embedding nets.

Thanks a lot! 🙏

I will no longer remove RatioEstimator from neural_nets because it looks like the conditional density estimators still live there... unless someone tells me that's wrong. Now it has its own ratio_estimators.py file under neural nets.
It looks like zukoflow expects that the batch_x and batch_x have shape (batch, *shape). I used that convention as well.

if this passes the CI, all tests that check anything related to ratios are passing on my computer as well:
pytest tests/inference_on_device_test.py tests/linearGaussian_snre_test.py tests/ratio_estimator_test.py
................xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx................xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.x......................................................................................................... [ 59%]
.......................................................xx....................................................................................                                                               [100%]

Right now test_lc2st_true_positiv_rate[LC2ST_NF] is failing, but that shouldn't affect merging this pull request @janfb


Possible non-ending search function?

What happens if the all the buckets are filled and we are searching for an item?
char* ht_search(ht_hash_table* ht, const char* key) {
    int index = ht_get_hash(key, ht->size, 0);
    ht_item* item = ht->items[index];
    int i = 1;
    while (item != NULL) {
        if (strcmp(item->key, key) == 0) {
            return item->value;
        }
        index = ht_get_hash(key, ht->size, i);
        item = ht->items[index];
        i++;
    }
    return NULL;
}

Seems like there is not exit condition to verify that key is not in the hashtable.

Hey @FedericoPonzi - you're absolutely right, there's no error checking there. However, in section 6 we add support for increasing the number of buckets when the hash table gets too full. The idea is that resizing will mean that the hash table is never full.
I didn't include error checking here for the sake of brevity, but I think it would be a good idea for a production-ready hash table.

Thanks @jamesroutley  for the reply, you're totally right!


bugfix: webCrawl can't handle website well recursively

When using Cheerio Web Scraper and selecting Web Crawl, it is supposed to crawling web pages recursively. There were couple bugs for successfully doing that:
1, extracting urls from webpage with things like default:blank and mailto etc.
2, extracting rules are not well aligned with specification of html  element
3, it always returns 10 pages of the website
Furthermore, this commit also adds a few informative logs when debug enabled

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi @gaord
I got a lot of additional weird links from this PR
Testing URL: https://www.itsjane.com
https://www.itsjane.com/jobs-1/eatatjane

https://www.itsjane.com/team-member/heidi-lee/itsjanesworld

Results:
webCrawl_bug.txt
original.txt

Hi,
The website is not well maintained actually. There are quite many links broken. As an example you mentioned:
go to in browser: https://www.itsjane.com/jobs-1/
in the left-bottom corner of the page you will see a link: https://www.itsjane.com/jobs-1/eatatjane
got a screenshot for you:

well tested! thanks.
@chungyau97

Hi @gaord,
My testing:
eatatjane will not be scrape in current method as it's not a URL or a relative path:

<EMAIL_ADDRESS>will be scrape in current method but it does not have host name so it will not be added into pages array:

Conclusions:
What I can agree on with you is adding this code below, this will help improve scraping performance.:
        if (
            !linkElement.href ||
            linkElement.href.startsWith('about:blank') ||
            linkElement.href.startsWith('mailto:') ||
            linkElement.href.includes('#')
        )

Logs:
This can be done on your own forked repository.
Other functionality:
Not entirely sure what you're trying to achieve, as the current method merges baseURL and relative path correctly. I cross-reference the web scraping results from other sources, it displays the same outcomes as our current method.
cc @HenryHengZJ

hi there, could you try https://docs.ceph.com/en/quincy/ please? Let me know what you find with the new code.
@chungyau97

Original Code:
pages: ["https://docs.ceph.com/en/quincy","https://docs.ceph.com/en/latest/releases/general","https://docs.ceph.com/en/latest","https://docs.ceph.com/en/reef","https://docs.ceph.com/en/pacific","https://docs.ceph.com/en/latest/releases","https://docs.ceph.com/en/octopus","https://docs.ceph.com/en/nautilus","https://docs.ceph.com/en/mimic","https://docs.ceph.com//readthedocs.org/projects/ceph","https://docs.ceph.com//readthedocs.org/builds/ceph"], length: 11
failed href: because does not cater to ../../,  ../ and relativePathName/
failHref: ["start/intro/","install/","cephadm/","rados/","cephfs/","rbd/","radosgw/","mgr/","mgr/dashboard/","api/","architecture/","dev/developer_guide/","dev/internals/","governance/","foundation/","ceph-volume/","security/","glossary/","jaegertracing/","dev/developer_guide/basic-workflow/#basic-workflow-dev-guide","start/documenting-ceph/#documenting-ceph","radosgw","rbd","cephfs","install","architecture","start/intro/","../../","../../","../../start/intro/","../../install/","../../cephadm/","../../rados/","../../cephfs/","../../rbd/","../../radosgw/","../../mgr/","../../mgr/dashboard/","../../api/","../../architecture/","../../dev/developer_guide/","../../dev/internals/","../../governance/","../../foundation/","../../ceph-volume/","../","../../security/","../../glossary/","../../jaegertracing/","../../","../#ceph-releases-index","../../ceph-volume/zfs/inventory/","../","start/intro/","install/","cephadm/","rados/","cephfs/","rbd/","radosgw/","mgr/","mgr/dashboard/","api/","architecture/","dev/developer_guide/","dev/internals/","governance/","foundation/","ceph-volume/","releases/general/","releases/","security/","glossary/","jaegertracing/","dev/developer_guide/basic-workflow/#basic-workflow-dev-guide","start/documenting-ceph/#documenting-ceph","radosgw","rbd","cephfs","install","architecture","start/intro/","start/intro/","install/","cephadm/","rados/","cephfs/","rbd/","radosgw/","mgr/","mgr/dashboard/","api/","architecture/","dev/developer_guide/","dev/internals/","governance/","foundation/","ceph-volume/","releases/general/","releases/","security/","glossary/","jaegertracing/","dev/developer_guide/basic-workflow/#basic-workflow-dev-guide","start/documenting-ceph/#documenting-ceph","radosgw","rbd","cephfs","install","architecture","start/intro/","genindex/","http-routingtable/","py-modindex/","start/intro/","","dev/developer_guide/basic-workflow/#basic-workflow-dev-guide","start/documenting-ceph/#documenting-ceph","radosgw","rbd","cephfs","install","architecture","start/intro/","install/","cephadm/","rados/","cephfs/","rbd/","radosgw/","mgr/","mgr/dashboard/","api/","architecture/","dev/developer_guide/","dev/internals/","governance/","foundation/","ceph-volume/","security/","glossary/","genindex/","genindex/","http-routingtable/","py-modindex/","start/intro/","","../","../","../start/intro/","../install/","../cephadm/","../rados/","../cephfs/","../rbd/","../radosgw/","../mgr/","../mgr/dashboard/","../api/","../architecture/","../dev/developer_guide/","../dev/internals/","../governance/","../foundation/","../ceph-volume/","general/","reef/","quincy/","pacific/","octopus/","nautilus/","mimic/","luminous/","kraken/","jewel/","infernalis/","hammer/","giant/","firefly/","emperor/","dumpling/","cuttlefish/","bobtail/","argonaut/","../security/","../glossary/","../jaegertracing/","../","reef","reef#v18-2-0-reef","quincy","quincy#v17-2-6-quincy","pacific","pacific#v16-2-14-pacific","octopus","octopus#v15-2-17-octopus","nautilus","nautilus#v14-2-22-nautilus","mimic","mimic#v13-2-10-mimic","luminous","luminous#v12-2-13-luminous","kraken","kraken#v11-2-1-kraken","jewel","jewel#v10-2-11-jewel","infernalis","infernalis#v9-2-1-infernalis","hammer","hammer#v0-94-10-hammer","giant","giant#v0-87-2-giant","firefly","firefly#v0-80-11-firefly","emperor","emperor#v0-72-2-emperor","dumpling","dumpling#v0-67-11-dumpling","reef","quincy","reef#v18-2-0-reef","quincy#v17-2-6-quincy","quincy#v17-2-5-quincy","quincy#v17-2-4-quincy","quincy#v17-2-3-quincy","quincy#v17-2-2-quincy","quincy#v17-2-1-quincy","quincy#v17-2-0-quincy","general/","reef/","genindex/","start/intro/","","radosgw","rbd","cephfs","start","architecture","start/intro/","install/","cephadm/","rados/","cephfs/","rbd/","radosgw/","mgr/","mgr/dashboard/","api/","architecture/","dev/developer_guide/","dev/internals/","governance/","foundation/","ceph-volume/","releases/general/","releases/","glossary/","genindex/","genindex/","start/intro/","","genindex/","start/intro/","radosgw","rbd","cephfs","start","architecture","start/intro/","start/","install/","start/kube-helm/","rados/","cephfs/","rbd/","radosgw/","mgr/","mgr/dashboard/","api/","architecture/","dev/","dev/internals/","governance/","ceph-volume/","releases/","glossary/","genindex/","genindex/","start/intro/","genindex/","start/intro/","radosgw","rbd","cephfs","start","architecture","start/intro/","start/","install/","start/kube-helm/","rados/","cephfs/","rbd/","radosgw/","mgr/","api/","architecture/","dev/","ceph-volume/","releases/","glossary/","genindex/","genindex/","start/intro/"]
This PR:
website too big and takes too long.
actively crawling https://docs.ceph.com/docs/master/security/cves/ pages: 2672

hey @gaord thank you so much for the solution! We are going to put this on hold first, since we are going to revamp the URL scraping UI to allow users see what links are scraped, and stop whenever they want. Otherwise this could go on for a long period of time and leave users in the dark as in what have been scraped

no problem. Do things right is first thing for it saves time. Original code doesn't implement in a way to process web pages. PR is crafted in case it is useful for others. Thanks for your great work to make AI apps easy anyway.
@HenryHengZJ


nixos/zram: use nixos/zram-generator as backing implementation

Description of changes
This allows granular configuration of zram-generator bypassing the limited interface of the zram module.

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg test zram-generator


Python 2 Deprecation Meeting (20 Oct 2020 TBC):

One-off @StackStorm/tsc 1 hour meeting will take place on (TO BE CONFIRMED ...Tuesday, 20th Oct 2020, 09:30 AM US Pacific)
See #33 for more info about how to join.
Meeting Agenda

Do we want to support StackStorm on EL 7 with Python 3.6?

NB. Python 3.6.8 available in EL7 repo

Do we want to support StackStorm on Ubuntu 16.04 (EOL April 2021) with Python 3, and if so given that it ships with python 3.5, and python 3.6 is only available from source or PPA what python version do we support?
Do we want to start supporting Ubuntu 20.04 and python 3.8?
NB. Ubuntu 16.04 EOL is April 2021, Ubuntu 20.04 uses python 3.8
Do we still support packs using python2 once we've migrated to python3, given that older versions of StackStorm will still use it? And if so, then how long for?
Do have a need to support python2 packs on latest StackStorm once all our supported distributions use python 3? (NB. there is no support for them on latest O/S releases).
After agreeing where we want to be, then what are the steps and order to get there?
Can we define a targetted timeline for achieving this?

Further information on prior discussions and technical findings can be found at: https://github.com/StackStorm/discussions/issues/40
Summary on python versions against OSes and EOLs below (thanks to @nzlosh)

:+1:

@amanda11 I've added that meeting to the google calendar: https://calendar.google.com/calendar?cid=MDA5bDJyN3NqcDlvam5uaTB2MjJqamNyZmdAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

I'll add an idea i had a while back around Python versioning support that people can think over and happily discuss next week:
Thought was to maybe create an "All-in-one" installation that includes our own version of Python, that way we can control the release process and version especially across distros.  I know both Chef and Puppet do this with Ruby. Go binaries do this with regards to them building "all-in-one" static binaries that do not dynamically link to any system libraries.

I'll add an idea i had a while back around Python versioning support that people can think over and happily discuss next week:
Thought was to maybe create an "All-in-one" installation that includes our own version of Python, that way we can control the release process and version especially across distros. I know both Chef and Puppet do this with Ruby. Go binaries do this with regards to them building "all-in-one" static binaries that do not dynamically link to any system libraries.

Thanks Nick - I've added it to the agenda so that we don't forget to discuss.

Thank you to everyone who attended the meeting for giving us your time, it was great to get your feedback. I'll write up minutes shortly.

Python 2 Deprecation Meeting Notes:
Attendees: Amanda, Eugen, Nick, Drew, Winson, Carlos, Vivek, Jacob, AJ
Discussed whether StackStorm ship our own version of python, so one version of python could be used across all OSes. It was decided that this was not the preferred option, as we would then be responsible for maintaining and providing timely fixes, and that this wasn't liked by all when performed by other products.
Supporting Python 3.6 on EL7 in the StackStorm release was agreed by all.
Most of discussion was around Ubuntu 16, given that it does not come with official Python 3.6 repository, and that it is EOL in Spring 2021. No unanimous option was agreed, so a vote on the short-term plans (target 3.4.0 release) by all those present was held on the following options:

Support Python 3.6 on Ubuntu 16, but customer must source their own Python 3.6 from a PPA.
Support Python 3.6 on Ubuntu 16 with a StackStorm supplied version of Python 3.6.
Continue to support Python 2.7 on Ubuntu 16 in next release
Drop Support for Ubuntu 16.

The voting was 5 for Option 1, 3 for Option 4. No votes for option 2 or 3.
For longer term the following option was agreed by all - to add Ubuntu 20.04 support and Python 3.8, whilst keeping EL7/EL8/U18 at Python 3.6, and then dropping U16 support.
Next steps:
Amanda to write up proposal voted on in the meeting for official TSC Voting, and plan to be announced once TSC have voted and agreed on a plan.

StackStorm 3.4.0. Add support for EL7 and Python 3.6. Add support for U16 and Python 3.6 with customer's own supplied PPA
StackStorm 3.5.0. Add support for U20 and Python 3.8, drop support for  U16.

NB. StackStorm release numbers are proposals, and the release cadance of aiming for 3 month releases would be followed, so release numbers may differ depending on when functionality is delivered.

Quick context behind the option 1 with migrating EL7 & U16 to py3 is that it technically can allow us dropping the python 2 support in v3.4.0 as an achievable chance to get rid of it sooner.
Without waiting for full U16 EOL (April 2021) or being blocked on more involved U20/py3.8 work (see @nzlosh PR  WIP - Add Python3.7 to CI pipeline #4855  why that could take more effort).

Keeping a note here of items that have been discussed on Slack that will need changing to support py 3 on EL7:

python scripts that refer explicitly to py2.7 to point to py3.6


Use raw_input when asking for credentials on python 2.7

First of all, I'm sorry to open a python 2.7 compatibility issue in 2020.
I'm using  confluence_ask_password and confluence_ask_user to ask for user credentials.
input() is evaluating the input as python code, I think raw_input should be used here?
sphinxcontrib-confluencebuilder: 1.2.0
Python version: 2.7.13
Sphinx version: 1.8.5
Best,
Mitja

Thanks for reporting this issue. This should now be resolved on master via #285; and will be made available on PyPI next stable release (v1.3+).