Webserver not reachable when deploying on eu-central-1

After running your CloudFormation template I am unable to access the webserver running on 8080 port. It just does not connect. I tried running the template two times to no avail

@villasv I am also unable to SSH into webserver even with EC2 Instance Connect

Hi, @ilyanoskov. I don't remember testing with eu-central-1, but I'm going to check it.
So it looks like the CloudFormation deploymed finished successfully, then? And the Webserver not responding is what you're reporting?
As per SSH, SSH is not enabled by default. You would have to use Session Manager.

I could not reproduce. I successfully deployed the stack on eu-central-1 and was able to access the webserver UI. Can you post here the creation parameters?

@villasv thanks a lot for getting back to me, I am desperate at this point haha. I am going to try to deploy it again with the button and let you know here.

@villasv yes, I have recreated it again and I am unable to access the webserver UI. All I did was press CloudFormation button, wait for it to finish, wait 3-5 mins for airflow to startup on ec2, and then try to access ec2-x-x-x-x-x.eu-central-1.compute.amazonaws.com:8080 to no avail.

I can see that you have AllowHTTPAccessCIDR = <IP_ADDRESS>/16, which means that you can only access the webserver if you're inside the VPC. If you don't have a VPN set up, I guess you can't do it. You can try to change it to <IP_ADDRESS>/0 temporarily to see if you can access it. If you're inside the webserver shell, you can test it with a curl localhost:8080
Those logs are indeed troubling, though. But it doesn't provide sufficient details. IIRC you can see the setup logs in /var/log/cfn-init.log and /var/log/cfn-init-cmd.log, and consult the service logs with journalctl -u airflow-webserver.service.

@villasv let me check these logs, will update you in 15 mins.

@villasv this is what cfn-init.log gives me:
ws s3 sync s3://turbine-quickstart/quickstart-turbine-airflow/scripts /opt/turbine
chmod +x /opt/turbine/webserver.setup.sh
/opt/turbine/webserver.setup.sh
) failed
2020-05-17 20:48:53,692 [DEBUG] Command setup output: fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
chmod: cannot access '/opt/turbine/webserver.setup.sh': No such file or directory
/bin/sh: line 9: /opt/turbine/webserver.setup.sh: No such file or directory

2020-05-17 20:48:53,693 [ERROR] Error encountered during build of config: Command setup failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command setup failed
2020-05-17 20:48:53,694 [ERROR] -----------------------BUILD FAILED!------------------------
2020-05-17 20:48:53,696 [ERROR] Unhandled exception during build: Command setup failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command setup failed

and cfn-init-cmd.log:
aws s3 sync s3://turbine-quickstart/quickstart-turbine-airflow/scripts /opt/turbine
chmod +x /opt/turbine/webserver.setup.sh
/opt/turbine/webserver.setup.sh
) failed
2020-05-17 20:48:53,692 [DEBUG] Command setup output: fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
chmod: cannot access '/opt/turbine/webserver.setup.sh': No such file or directory
/bin/sh: line 9: /opt/turbine/webserver.setup.sh: No such file or directory

2020-05-17 20:48:53,693 [ERROR] Error encountered during build of config: Command setup failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command setup failed
2020-05-17 20:48:53,694 [ERROR] -----------------------BUILD FAILED!------------------------
2020-05-17 20:48:53,696 [ERROR] Unhandled exception during build: Command setup failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command setup failed
sh-4.2$ cat /var/log/cfn-init-cmd.log
2020-05-17 20:48:52,551 P2658 [INFO] ************************************************************
2020-05-17 20:48:52,552 P2658 [INFO] ConfigSet default
2020-05-17 20:48:52,552 P2658 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-05-17 20:48:52,552 P2658 [INFO] Config config
2020-05-17 20:48:52,553 P2658 [INFO] ============================================================
2020-05-17 20:48:52,553 P2658 [INFO] Command setup
2020-05-17 20:48:53,691 P2658 [INFO] -----------------------Command Output-----------------------
2020-05-17 20:48:53,692 P2658 [INFO]    fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
2020-05-17 20:48:53,692 P2658 [INFO]    chmod: cannot access '/opt/turbine/webserver.setup.sh': No such file or directory
2020-05-17 20:48:53,692 P2658 [INFO]    /bin/sh: line 9: /opt/turbine/webserver.setup.sh: No such file or directory
2020-05-17 20:48:53,692 P2658 [INFO] ------------------------------------------------------------
2020-05-17 20:48:53,692 P2658 [ERROR] Exited with error code 127

Thanks @ilyanoskov, it looks like it's a permissioning problem in the s3 buckets as it all starts here:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

I'll check if the eu-central-1 has the right public access permissions.

@villasv thanks a lot! Please let me know when you do and I will try again with Cloudformation.

There you go. Not only the files had to be public, but there was a missing bucket policy for the bucket to make the s3 sync api work.

@villasv now I am able to connect to UI, thank you very much.
There are some errors I see in the session manager though:
-1-243.eu-central-1.compute.internal systemd[1]: Reloading.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Started cfn-hup.service.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Starting cfn-hup.service...
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Reloading.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Started Airflow webserver daemon.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Starting Airflow webserver daemon...
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Started airflow-confapply-agent.path.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Starting airflow-confapply-agent.path.
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,166] {cli_action_loggers.py:107} WARNING - Failed to log action with (psycopg2.errors.Undef
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [parameters: {'dttm': datetime.datetime(2020, 5, 18, 11, 36, 51, 908287, tzinfo=<Timezone [UTC]>), 'dag_id': Non
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal rsyslogd[2442]: imjournal: journal reloaded... [v8.24.0-41.amzn2.2.1 try http://www.rsyslog.com/e/0 ]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: (Background on this error at: http://sqlalche.me/e/f405)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ____________       _____________
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ____    |__( )_________  __/__  /________      __
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,379] {__init__.py:51} INFO - Using executor CeleryExecutor
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,379] {dagbag.py:396} INFO - Filling up the DagBag from /airflow/dags
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,422] {dagbag.py:239} ERROR - Failed to import: /usr/local/lib/python3.7/site-packages/airfl
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: psycopg2.errors.UndefinedTable: relation "slot_pool" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM slot_pool
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: The above exception was the direct cause of the following exception:
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 236, in process_file
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: m = imp.load_source(mod_name, filepath)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/lib64/python3.7/imp.py", line 171, in load_source
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: module = _load(spec)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap>", line 696, in _load
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap_external>", line 728, in exec_module
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_subdag_operator.py", line 50, in <modu
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: dag=dag,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return func(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/decorators.py", line 98, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: result = func(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/operators/subdag_operator.py", line 77, in __init__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: .filter(Pool.pool == self.pool)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3375, in first
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ret = list(self[0:1])
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return list(res)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return self._execute_and_instances(context)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: result = conn.execute(querycontext.statement, self._params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return meth(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return connection._execute_clauseelement(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: distilled_params,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: e, statement, parameters, cursor, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy_exception, with_traceback=exc_info[2], from_=e
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: raise exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "slot_pool" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM slot_pool
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [SQL: SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: FROM slot_pool
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: WHERE slot_pool.slots = %(slots_1)s AND slot_pool.pool = %(pool_1)s
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LIMIT %(param_1)s]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [parameters: {'slots_1': 1, 'pool_1': 'default_pool', 'param_1': 1}]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: (Background on this error at: http://sqlalche.me/e/f405)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: psycopg2.errors.UndefinedTable: relation "connection" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM connection GROUP BY connection.conn_id
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: The above exception was the direct cause of the following exception:
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/bin/airflow", line 37, in <module>
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: args.func(args)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return f(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 900, in webserver
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 232, in cached_app
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: app = create_app(config, testing)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 87, in create_app
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: from airflow.www import views
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2477, in <module>
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2568, in ChartModelView
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: 'conn_id': _connection_ids()
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return func(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2474, in _connection_ids
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: .group_by(Connection.conn_id))]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return self._execute_and_instances(context)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: result = conn.execute(querycontext.statement, self._params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return meth(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return connection._execute_clauseelement(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: distilled_params,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: e, statement, parameters, cursor, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy_exception, with_traceback=exc_info[2], from_=e
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: raise exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "connection" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM connection GROUP BY connection.conn_id
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [SQL: SELECT connection.conn_id AS connection_conn_id
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: FROM connection GROUP BY connection.conn_id]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: (Background on this error at: http://sqlalche.me/e/f405)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: airflow-webserver.service: main process exited, code=exited, status=1/FAILURE
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Unit airflow-webserver.service entered failed state.
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: airflow-webserver.service failed.```

and this error as well:
May 18 11:37:02 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: + /opt/aws/bin/cfn-signal -e 0
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: Error: No WaitConditionHandle URL or logical resource id specified

May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: May 18 11:37:03 cloud-init[2647]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: May 18 11:37:03 cloud-init[2647]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: May 18 11:37:03 cloud-init[2647]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: ci-info: no authorized ssh keys fingerprints found for user ec2-user.
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal ec2[3436]:
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal ec2[3436]:
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: Cloud-init v. 19.3-2.amzn2 finished at Mon, 18 May 2020 11:37:03 +0000. Datasource DataSourceEc2.  Up 166.91
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: cloud-final.service: main process exited, code=exited, status=1/FAILURE
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Failed to start Execute cloud user/final scripts.
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Unit cloud-final.service entered failed state.
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: cloud-final.service failed.

There are some errors I see in the session manager (journalctl) though

That might happen temporarily, if the migrations are still ongoing. It shouldn't stay happening forever, though.
About your second logs paste:

Error: No WaitConditionHandle URL or logical resource id specified

This is an error indeed, but it's not relevant. There's no WaitConditionHandle in use by the stack at the moment, this is likely to be removed later on.

Let me know if the errors in the journal are persisting, or if your tasks are not being scheduled/executed properly.

ok, thank you very much! Will update you soon as I will have some test DAGs running.
You provided tremendous support man, the coffee is on me ;)


Correct misspelling of Lombok

See diff for details - minor spelling correction.

@shahzaibj Thanks for reviewing. I don't think this change warrants a changelog update, but am hesitant to merge/override branch requirements.
If you agree, could you do the honors of throwing the merge lever?


CI fail intermittently on docs pipeline

Describe intermittently
Sometimes CI builds fail on the docs pipeline with an error.
/go/pkg/mod/google.golang.org/grpc@v1.43.0/status/status.go:35:2<EMAIL_ADDRESS>reading<EMAIL_ADDRESS>403 Forbidden

Failed builds:

https://drone.owncloud.com/owncloud/ocis/22861/96/3
https://drone.owncloud.com/owncloud/ocis/22815/78/3

@micbar @kulmann

failing here too : https://drone.owncloud.com/owncloud/ocis/22894/78/3

This is happening on almost every run now. We need to find the root cause.
How much time do we have between the assignment of the "Admin" role and the space creation?

@micbar maybe you're talking about this issue https://github.com/owncloud/ocis/issues/6431 ? This one's related to docs pipeline

@grgprarup is working on the e2e intermittent failure


Feature request: Filter notifications client-side based on relationship

The official Mastodon iOS app has a setting for notifications like so:
Notify me when [anyone / a follower / anyone i follow / no one]:

favorites my post
follows me
reblogs my post
mentions me

One way to do this better than the official Mastodon iOS app would be to have a policy per type, and to support mutuals too. I think you can do this with client-side filtering. When ST receives a push notification, check the type, then check the notification policy.
For example, you receive a favorite notification, then check the app settings / configuration to see that the policy is "anyone i follow". You then show the notification if it is indeed from someone you follow, or you do not show it if it is from someone you do not follow.

do you talk about this?

Yes, but separated by notification type. For example:

Type
Policy

Favorites
Anyone I follow

Boosts
Mutuals

Mentions
Anyone

Follows
None

I am willing to support it, but the time will be delayed due to private reasons.

that's fine. Thank you for everything you do for this client, it remains
and continues to be the best way to use the fediverse on any platform.
On Sun, Aug 22, 2021 at 6:40 PM tateisu @.***> wrote:

I am willing to support it, but the time will be delayed due to private
reasons.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tateisu/SubwayTooter/issues/161#issuecomment-903349248,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACQ5OX7XY3ZVUX7DPDG2DQ3T6GDG7ANCNFSM5CI3KWCA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

With ST 5.512 I've revamped our push notification support, and I've noticed:

API accept policy parameter for all alerts, not policy per alert.
WebPush message does not contain acct of notification.user. then app can't know who triggered about notification.

related mastodon issue: https://github.com/mastodon/mastodon/issues/22943


show forced encryption state

Task/Issue URL: https://app.asana.com/0/392891325557410/518282040737111
Tech Design URL:
CC:
Description:
Shows forced encryption state after upgrading a http request to https.
Have recreated TLDs logic in the app otherwise redirects made the app lose the forced state for a given site. As such the data in tlds.js is now in a json file read in by the app and injected in to the tlds.js script when loading content blocking files.
Steps to test this PR:

Go to http://booking.com
After it finishes redirecting, check the privacy dashboard - should say Forced Encryption  on the overview and encryption detail screen

Other sample sites:
http://happyearth.marketing -> Unencrypted
http://falkirkrpg.org.uk -> Encrypted (redirects to https but not upgraded/forced because not in the list)
https://falkirkrpg.org.uk/noxwall/mixed.html -> Mixed

Internal references:
Software Engineering Expectations
Technical Design Template

LGTM


Do no use subprocess for the ScanPackage (scan_package) pipeline

Using subprocess is problematic for multiple reasons:

Introduce security issues
Cannot log the progress
Cannot catch errors and exceptions while running
Cannot interact with the running process
Cannot use mock in the unit test context
Cannot use the status system to flag and exclude resources from scan
....

From https://github.com/nexB/scancode.io/issues/556#issuecomment-1307401548

The underlying issue is that the ScanPackage pipeline depends on a subprocess call to the scancode command.
If scancode fails, there's no easy way to save this as a ProjectError and continue the pipeline run.

A better approach would be to replace this approach with proper API calls, as we did for the Scanners in the ScanCodebase pipeline (For example: get_copyrights, get_licenses, get_package_data)

The 2 missing pieces as callable API for the ScanPackage pipeline are:

--classify: FileClassifier(PostScanPlugin)
--summary: ScanSummary(PostScanPlugin)

Following the merge of https://github.com/nexB/scancode.io/pull/855/ , we are not relying on a subprocess call anymore as we call directly the scancode.cli.run_scan function.
This is better but there's still the following limitations:

Cannot catch errors and exceptions while running
Cannot interact with the running process
Cannot use the status system to flag and exclude resources from scan


Web Accessibility

I love this but it creates a possible accessibility trap for keyboard-only users. You cannot tell when the close button is in focus and the Esc key doesn't work when tabbing through content.

Confirmed that this is the case. Having trouble currently with both YouTube and Vimeo videos. If you tab into the video controls, it's a keyboard trap and there's no way to exit from the video and close the lightbox without using the mouse. Breaks WCAG 2.1.2 Keyboard Trap.


Do not fail if .info is called on an empty holomap

Looks good and thanks for the fix!
I've tested it and it works fine -  I'll merge it once you confirm that you are done with it.

I think I'm happy with it. :)


tests: Fluent badge bunit tests

Pull Request
📖 Description
bUnit tests covering FluentBadge component rendering, meaning we check if the HTML is valid generated by the Blazor component.
🎫 Issues
No related issues.
👩💻 Reviewer Notes

I renamed previous tests files as they didn't have "Render" in their name
the additional functionalities (dealing with extra style, css class and unmatched attributes) are exercised at component level
Appearance input value is checked as the enum contains more value than the component supports (the enum itself is widely used across the components)
I took the liberty and added FluentAssertions nuget package. I don't know what is the policy when it comes to an additional 3rd party package. If this package cannot be added, please let me know. I'll remove it and assert the exception in a different way.

📑 Test Plan
Testing is not necessary here.
✅ Checklist
General

[ ] I have added tests for my changes.
[ ] I have tested my changes.
[ ] I have updated the project documentation to reflect my changes.
[x] I have read the CONTRIBUTING documentation and followed the standards for this project.

Component-specific

[ ] I have added a new component
[ ] I have modified an existing component

⏭ Next Steps

I don't think adding the extra package is an issue. It is only being used in this test project, right?

I don't think adding the extra package is an issue. It is only being used in this test project, right?

Yes, only in the test project.


[WIP] Cheatsheet for Pytorch to Gluon

The goal is to provide a concise set of reference for existing pytorch user to adopt gluon.
How to contribute

pick a section that's not being worked on (fork subsections if necessary, discuss in this issue first)
comment on this issue stating that you're working on section x.x
write a paragraph to describe high-level similarity and differences. For example:

Placeholder for description, highlights of similarity and differences
- Lorem ipsum dolor sit amet, consectetur adipiscing elit.
- Vestibulum commodo est et diam volutpat molestie.

write a table that looks like the following. fill in the table with code snippets.

| Function               | Pytorch                           | MXNet Gluon                              |
|------------------------|-----------------------------------|------------------------------------------|
| Copy to GPU            | `y = torch.FloatTensor(1).cuda()` | `y = mxnet.nd.ones((1,), ctx=mx.gpu(0))` |
| Convert to numpy array | `x = y.cpu().numpy()`             | `x = y.asnumpy()`                        |

keep everything short and concise.

The above will render to:
Placeholder for description, highlights of similarity and differences

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Vestibulum commodo est et diam volutpat molestie.

Function
Pytorch
MXNet Gluon

Copy to GPU
y = torch.FloatTensor(1).cuda()
y = mxnet.nd.ones((1,), ctx=mx.gpu(0))

Convert to numpy array
x = y.cpu().numpy()
x = y.asnumpy()

The file to edit is https://github.com/zackchase/mxnet-the-straight-dope/blob/master/cheatsheets/pytorch_gluon.md
Progress

[ ] 1. Pytorch Tensor and MXNet NDArray

[ ] 1.1. Tensor operation
[ ] 1.2. Functional
[x] 1.3. GPU (by @szha)
[x] 1.4. cross device (by @ZiyueHuang)

[x] 2. Autograd (by @ZiyueHuang)

[x] 2.1. variable wrapper vs autograd scope
[x] 2.2. gluon can supply head grad
[x] 2.3. scope override (pause, train_mode, predict_mode)
[x] 2.4. batch-end synchronization is needed

[ ] 3. Pytorch module and Gluon blocks

[ ] 3.1. usage of existing blocks look alike
[ ] 3.2. for new block definition, gluon needs name_scope
[ ] 3.3. HybridBlock can be hybridized, and allows partial-shape info
[ ] 3.4. SymbolBlock
[ ] 3.5. Parameter and Initializer

[ ] 4. Pytorch optimizer vs Gluon Trainer

[ ] 4.1. for gluon zero_grad is not necessary most of the time
[ ] 4.2. for gluon zero_grad only needed when grad_req is 'add'
[ ] 4.3. Multi-GPU training
[ ] 4.4. Distributed training

[ ] 5. Monitoring

[ ] 5.1. MXNet has pre-defined metrics
[ ] 5.2. Data visualization

[ ] 6. I/O and deploy

[ ] 6.1. Data loading
[ ] 6.2. Serialization

I worked on 1.3 in #302

I will work on 1.4 cross device.

I will take 6.

Great! Thanks guys!
On Fri, Oct 20, 2017 at 1:24 AM, Joshua Z. Zhang<EMAIL_ADDRESS>wrote:

I will take 6.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/zackchase/mxnet-the-straight-dope/issues/301#issuecomment-338042296,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACR4zlvf_wrxJ-qhV0xzZCLo0E3kEniTks5st731gaJpZM4P8hjL
.

working on 1.1 & 1.2

working on section 2 Autograd

working on section 3 Pytorch module and Gluon blocks

maybe can add an item about initializer and shape infer between pytorch and gluon @szha

Good idea. I added a section 3.5 on parameter and initializer. Partial shape is included in 3.3.
Suggestions are welcome.

I will take the initializer section first.

Ping

working on sec 4.1 and sec 4.2


Kong accepts plugins that lack required fields, but validation rejects them

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version ($ kong version)
Kong 2.5
Current Behavior
Per Kong/docs.konghq.com#3368, if we create a KongClusterPlugin or KongPlugin manifest using an authentication-related plugin that defines a "config.hide_credentials" field in its schema, but our manifest omits that field, the validating admission Webhook rejects the proposed object as invalid—due to it lacking that field—but if we bypass the Webhook, Kong accepts the same proposed object and supplies a default value for the omitted field.
This example is about this set of plugins whose schema treats this "hide_credentials" field as required—as of Kong/kong#6785—but supplies a default value, but I suspect that the problem is more general, applying to any plugins with schema fields with this same required-but-with-default arrangement.
It would be clearer for users if either the validating admission Webhook accepts what Kong ultimately accepts, or if Kong would reject proposed KongClusterPlugin and KongPlugin objects that lack such a required field. In other words, the Webhook and Kong should be equally fastidious.
Why does this matter? We have KongPlugin manifests that lack this required "config.hide_credentials" field, and everything was working fine for us until we put the validating admission Webhook into service. That Webhook blocks our use of these same manifests—perhaps rightly so. However, we would have preferred if Kong would have been rejecting these manifests long before we starting using the Webhook. At present, if we disable the Webhook, we can use these same manifests without any apparent problem.
Expected Behavior
Either Kong's validating admission Webhook should accept proposed KongClusterPlugin and KongPlugin objects lacking fields that are required but have a default value in the schema, or Kong should reject such proposed objects because they lack this required field.
Steps To Reproduce
1. Write a KongPlugin manifest for a plugin like "key-auth" that lacks the "config.hide_credentials" field.
2. Disable Kong's validating admission Webhook.
3. Apply that manifest to the target cluster.
4. Confirm that Kong's proxy container accepted the plugin.
5. Delete the KongPlugin Kubernetes object.
6. Enable Kong's validating admission Webhook.
7. Apply that manifest to the target cluster.
8. Observe that the Webhook rejects the creation request.

Anything else?
We see this with KIC version 2.0.5.

For now, we're working around this problem via this Kyverno policy:

Kyverno "kong-plugin-fields" policy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: kong-plugin-fields
spec:
  rules:
  - name: populate-missing-required-fields
    match:
      resources:
        kinds:
        - configuration.konghq.com/v1/KongClusterPlugin
        - configuration.konghq.com/v1/KongPlugin
    preconditions:
      all:
      - key: "{{ request.object.plugin }}"
        operator: In
        value:
        - basic-auth
        - hmac-auth
        - key-auth
        - ldap-auth
        - oauth2
      any:
      # NB: This case isn't strictly necessary, as the second case
      # will still evaluate to true if the field is missing, but it
      # aids in clarity.
      - key: "{{ contains(keys(request.object.config), `\"hide_credentials\"`) }}"
        operator: Equals
        value: false
      - key: "{{ request.object.config.hide_credentials == `null` }}"
        operator: Equals
        value: true
    mutate:
      patchStrategicMerge:
        config:
          # Of all the plugins, only the OAuth2 plugin uses true as
          # its default value for this field.
          hide_credentials: "{{ request.object.plugin == `\"oauth2\"` }}"

cc @Kong/team-k8s

Can you share an example plugin manifest that's rejected by this and the admission error that comes back? I'm not able to replicate:
15:48:36-0800 esenin $ kubectl apply -f /tmp/kvalid.yaml
kongclusterplugin.configuration.konghq.com/key-auth-valid created
15:48:39-0800 esenin $ kubectl delete -f /tmp/kvalid.yaml
kongclusterplugin.configuration.konghq.com "key-auth-valid" deleted
15:48:43-0800 esenin $ kubectl apply -f /tmp/kinvalid.yaml
Error from server: error when creating "/tmp/kinvalid.yaml": admission webhook "validations.kong.konghq.com" denied the request: plugin failed schema validation: schema violation (config.garbage: unknown field)
15:48:47-0800 esenin $ kubectl apply -f /tmp/knohide.yaml
kongclusterplugin.configuration.konghq.com/key-auth-nohide created
15:48:55-0800 esenin $ cat /tmp/knohide.yaml
---
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: key-auth-nohide
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
config:
  key_in_header: true
plugin: key-auth

That part of plugin validation is just sending a POST /schemas/plugins/validate with the proposed plugin to the admin API and regurgitating whatever it returns. The controller has no logic to validate plugin config of its own, only for the parts of the manifest it understands (e.g. ensuring that you have a name and plugin set. Kong and the controller should already be equally fastidious for config validation, but all indications are that Kong is just okay validating without those fields when I test.

The error we see when trying to create such an object lacking the "config.hide_credentials" field looks like this:

admission webhook "validations.kong.konghq.com" denied the request: plugin failed schema validation: schema violation (config.hide_credentials: required field missing)

Here is an example manifest that includes the field, but with a null value:
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: basic-auth
  annotations:
    kubernetes.io/ingress.class: kong
plugin: basic-auth
config:
  hide_credentials:

If I recall correctly, we see the same behavior with one completely lacking the field.
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: basic-auth
  annotations:
    kubernetes.io/ingress.class: kong
plugin: basic-auth

It looks like it only happens when it's present but empty. I need to check exactly what this ends up serializing to, but I wouldn't expect that to work regardless:
17:38:44-0800 esenin $ kubectl apply -f /tmp/basicnull.yaml
Error from server: error when creating "/tmp/basicnull.yaml": admission webhook "validations.kong.konghq.com" denied the request: plugin failed schema validation: schema violation (config.hide_credentials: required field missing)
17:38:50-0800 esenin $ kubectl apply -f /tmp/basicnothing.yaml
kongclusterplugin.configuration.konghq.com/basic-auth created
17:38:54-0800 esenin $ cat /tmp/basicnull.yaml /tmp/basicnothing.yaml
---
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: basic-auth
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
plugin: basic-auth
config:
  hide_credentials:
---
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: basic-auth
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
plugin: basic-auth

Thank you for sussing that out. The manifests in question within my company aren't ones I've written, or even seen in most cases. I was on the receiving end of a bunch of complaints after I activated the Webhook, and focused first on disabling the Webhook temporarily, then writing that Kyverno policy to fix up the malformed KongPlugin manifests that are in play.
I agree that the field being present with a null value is weird. However, from what I hear, Kong accepts it. Maybe that's not true after all.

Yep, so the actual request to Kong when the field is present but not set includes it with value null:
POST /schemas/plugins/validate HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Content-Length: 56
Content-Type: application/json
Accept-Encoding: gzip

{"name":"basic-auth","config":{"hide_credentials":null}}

Omitting it omits it:
POST /schemas/plugins/validate HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Content-Length: 21
Content-Type: application/json
Accept-Encoding: gzip

{"name":"basic-auth"}

The Kong admin API normally interprets null value fields as an instruction to delete that field. If you have a plugin with some optional field (e.g. anonymous for basic-auth) set, sending a PATCH with field=null edits the plugin to remove that configuration. That behavior is kinda meaningless for POST, but I don't think it's unexpected.
I don't think there's anything else to do here, but will leave that up to the core team.

Thank you for the thorough investigation.
Not that we were trying to accomplish this, but I can imagine a user wanting to delete a field in order to "set it back to its default value," which is again difficult to reason about for fields that are both required and have a default value.

That does appear to be the de facto behavior I get when editing a plugin, though I'm not sure entirely what's going on there: if I start with hide_credentials: true and key_in_body: true (to see what happens with non-required fields) and then edit (or kubectl apply) to use empty fields, I get a POST to /validate that omits them, and a PUT after that restores the defaults.
There appears to be a bit different behavior in kubectl itself when creating a new object versus editing an existing one, and the latter is more or less what we want.

I appear to have stumbled upon the same issue, with a different plugin, due to the kong chart enabling the webhook by default.
The plugin is https://docs.konghq.com/hub/kong-inc/http-log/
Here is the kongClusterPlugin, for reference:
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
  name: kong-http-log-plugin
config:
  http_endpoint: "http://logs.cloud.internal:2000/indexes.kong"
  method: "POST"
  timeout: 1000
  keepalive: 1000
  flush_timeout: 2
  retry_count: 15
  custom_fields_by_lua:
    request.headers: 'return {["user-agent"]=kong.request.get_header("user-agent")}'
    request.querystring: 'return nil'
plugin: http-log

We get the following error when trying to apply / edit this plugin:
cannot patch "kong-http-log-plugin" with kind KongClusterPlugin: admission webhook "validations.kong.konghq.com" denied the request: plugin failed schema validation: schema violation (config.custom_fields_by_lua: expected a string)

Where the admitionWebhook requires the field to be a string, when the docs state this field is a map. As such, the webhook will reject it, but kong accepts it without issue.

@skylt The issue you're experiencing should be related to https://github.com/Kong/kong/issues/10654

Yes, as I understand it both issues are the same, the admission webhook can reject configurations that kong would accept.

My understanding is that this will be fixed with https://github.com/Kong/kong/commits/3.4.0.

It seems this PR: https://github.com/Kong/kong/pull/11091 has already fixed the issue. Closing the ticket. Please feel free to reopen if that's not the case or you have further concerns.


Binary release still current?

Hi, is the plugin binary in Gdrive still current? I can't get it to work for the life of me.
Meshtastic firmware: 1.2.45
Meshtastic app: 1.2.31
ATAK: <IP_ADDRESS>
It will connect and setup the channel, but nodes won't see each other. Both my test EUDs keep showing "x repeated errors..." ie. all received packets are in error. Even the Meshtastic app won't talk/see each other any more outside of ATAK, despite the channel setup seemingly correct. Meshtastic worked correctly prior to running the plugin on ATAK.
What am I missing?

Try a different speed / mode, I have had similar issues using some of the modes before.

I've switched to Medium and it worked... for about 5 minutes. Then they stopped talking to each other again.
The worrying thing is, even after I close ATAK and change channels/device ID in Meshtastic app, the same issue persists in Meshtastic itself. Lights are blinking as normal and app shows devices are connected via Bluetooth, but text messages fail instantly. I can't say for sure of course, but it seems that something about using the ATAK plugin has caused something to change in Meshtastic. Flashing the device firmware again fixes it. I've done this whole sequence twice now and reproduced the same problem.
Unfortunately I'm not familiar with development and don't know how to access logcat. No idea where the problem lies. Maybe I should post this on Meshtastic forum as well?

I've uploaded a beta build of 1.0.2 to the Drive link under the beta/ folder, could you try it out and let me know if it fixes the issue?

Here's my report for 1.0.2:
Starting again with freshly flashed TBEAMS, all was well using Medium range mode. When I tried to change to Long mode, again they stopped talking (100% message error). Meshtastic app shows correct channel info and connected, but no messages getting through.
Resetting the channel in Mesh app to default allows the app to work and exchange messages again, but then ATAK forwarder stops working completely. As in, setting a new channel via ATAK will never get any messages any more.
Just as I'm writing this, I tried it again and now ATAK forwarder doesn't even generate a complete QR image any more. The front and end of the URL is missing a lot of bytes. I checked ATAK and I'm now on <IP_ADDRESS> apparently.
Sorry, I don't know how to help this situation.

There is a new release, 1.0.4, that may be worth trying out, please let me know how it goes.

I am seeing the same issues on the latest releases.  I would be happy to do any requested troubleshooting.  I have access to 3 android devices and 4 t-beams.

Closing since there are no updates here.


ReferenceError: exports is not defined in ES module scope

This file is being treated as an ES module because it has a '.js' file extension and '/Users/wangchunyu40/workspace/global-express-m/node_modules/vue-clipboard3/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.

I had the same issue:


knot-dns: 3.1.0 -> 3.1.1

https://gitlab.nic.cz/knot/knot-dns/-/tags/v3.1.1
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

This version fixes several bugs and reverts one recent change in libzscanner.  It emerged that the previous implementation of how omitted TTL is interpreted is closer to what the users expect. I'm sorry for this confusion!


Add ability to search for method/field usage and use custom update sites

This adds three new features:

A new option to override the update center URL to fetch plugins from.
A new option to search for qualified method names used by plugins (minus the rest of the method signature).
A new option to search for qualified field names used by plugins.

Example output from the example given in USAGE.adoc (and a field lookup):

@jeffret-b @Wadeck @daniel-beck

Major update applied. Note that the work directory layout has changed slightly in order to support multiple update centers.

Forgot to add. I tested this with a variety of inputs. I didn't get very fancy with specifying methods and fields. I tried several different basic searches. The more complicated ones should work just fine based on standard specifications. I tried it with a different update center and that worked fine.

There seems to be no deduplication of core releases. Given the following command indicates the same weekly release of Jenkins at the same URL is analyzed twice:
mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json'

I'm at about 1.5% of downloads failing. This indicates that we should have a retry feature, or at the very list a summary at the end of output listing which plugins failed to download, with a note that re-running the tool will re-attempt downloads.

There seems to be no deduplication of core releases. Given the following command indicates the same weekly release of Jenkins at the same URL is analyzed twice:
mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json'

Analyzing deprecated APIs in work/59a92455/core/2.243/jenkins.war
Analyzing deprecated APIs in work/6ba2a29d/core/2.243/jenkins.war
Finished deprecated API analysis in work/6ba2a29d/core/2.243/jenkins.war
Finished deprecated API analysis in work/59a92455/core/2.243/jenkins.war

There's also no deduplication of plugin downloads. As the majority of plugins will be identical between common update centers, that's a pretty big waste of time, and the support for multiple update sites brings minimal benefit over two separate executions.
I think the assumption that, between update sites, identical versions of the same component are the same is reasonable, and could be used to speed this up.

I can certainly return and try to implement this again, but I had previously given up on the approach as writing multithreaded code to mass download potentially overlapping things was getting fairly non-trivial. My "complicated" approach that seemed to work was to acquire FileChannel locks on the output files so that two download requests for the same plugin don't clobber one another. Alternatively, I can figure out some sort of processing queue to deduplicate the HTTP requests in the first place so that file locking won't be necessary. One of these approaches should work.

@jvz I would look at the JSONs as a whole to determine what to download, and if any number of them tell you to download $plugin-$version you only do that once and reuse. Existing $plugin/$version/ directories would be good enough, maybe remove the code that deletes obsolete directories there.

Yeah, I think that could work. I'll have some changes ready later for further review.

I'll note that some of the errors you'll see during downloads are because either the plugin download file's checksum doesn't actually match the published checksum, or some artifacts seem to be corrupted. The HttpClient library I added performs automatic retries by default, hence why this PR seems to generally work better now at downloading everything in one go. I could potentially change this to simply emit warnings for plugins that don't match the update center metadata (any of which I'd expect to fix itself with a new plugin release).

I'll note that some of the errors you'll see during downloads are because either the plugin download file's checksum doesn't actually match the published checksum, or some artifacts seem to be corrupted

Most of them (all except 1) were "Connection reset by peer" and gone by the third execution of the same command.
Just one, the error I quoted in a line comment, was a very recent release not yet on mirrors.

I'll note that some of the errors you'll see during downloads are because either the plugin download file's checksum doesn't actually match the published checksum, or some artifacts seem to be corrupted

Most of them (all except 1) were "Connection reset by peer" (or similar, don't exactly remember) and gone by the third execution of the same command.
Just one, the error I quoted in a line comment, was a very recent release not yet on mirrors.

Alright, I'll take a closer look while refactoring.

Still failing 20/1800 plugin downloads, more than 1% without an explanation 😦
If the degree of concurrency is too high, maybe allow manually overriding that? IIUC, the concurrency is based on CPUs, but uses bandwidth as resource, which seems weird.
Not a huge problem, just not great.
Error downloading http://updates.jenkins-ci.org/download/plugins/azure-dev-spaces/3.0.0/azure-dev-spaces.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/azure-dev-spaces/3.0.0/azure-dev-spaces.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/azure-keyvault/2.0/azure-keyvault.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/azure-keyvault/2.0/azure-keyvault.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/changelog-history/1.7/changelog-history.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/changelog-history/1.7/changelog-history.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/configuration-as-code-groovy/1.1/configuration-as-code-groovy.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/configuration-as-code-groovy/1.1/configuration-as-code-groovy.hpi
Error downloading http://updates.jenkins-ci.org/download/plugins/config-rotator/2.0.1/config-rotator.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/config-rotator/2.0.1/config-rotator.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
Error downloading http://updates.jenkins-ci.org/download/plugins/cucumber-living-documentation/2.2.2/cucumber-living-documentation.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/cucumber-living-documentation/2.2.2/cucumber-living-documentation.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/generic-webhook-trigger/1.67/generic-webhook-trigger.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/generic-webhook-trigger/1.67/generic-webhook-trigger.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/gitlab-oauth/1.6/gitlab-oauth.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/gitlab-oauth/1.6/gitlab-oauth.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/ezwall/0.3/ezwall.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/ezwall/0.3/ezwall.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/heavy-job/1.1/heavy-job.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/heavy-job/1.1/heavy-job.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/hyper-commons/0.1.5/hyper-commons.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/hyper-commons/0.1.5/hyper-commons.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/google-git-notes-publisher/0.3/google-git-notes-publisher.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/google-git-notes-publisher/0.3/google-git-notes-publisher.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/kubernetes-client-api/4.9.2-2/kubernetes-client-api.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/kubernetes-client-api/4.9.2-2/kubernetes-client-api.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/jdk-tool/1.4/jdk-tool.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/jdk-tool/1.4/jdk-tool.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/jenkins-design-language/1.23.2/jenkins-design-language.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/jenkins-design-language/1.23.2/jenkins-design-language.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/label-linked-jobs/5.1.2/label-linked-jobs.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/label-linked-jobs/5.1.2/label-linked-jobs.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/no-agent-job-purge/1.2/no-agent-job-purge.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/no-agent-job-purge/1.2/no-agent-job-purge.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/pipeline-multibranch-defaults/2.1/pipeline-multibranch-defaults.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/pipeline-multibranch-defaults/2.1/pipeline-multibranch-defaults.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/pry/1.1/pry.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/pry/1.1/pry.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
…
Error downloading http://updates.jenkins-ci.org/download/plugins/testweaver/1.0.6/testweaver.hpi -> /Users/danielbeck/Repositories/github.com/jenkins-infra/usage-in-plugins/work/testweaver/1.0.6/testweaver.hpi
org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer

Behavior with multiple update sites tested with
$ cat classes.txt
org/apache/http/client/methods/HttpPost

and
mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json -C classes.txt -i'
mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'

Usage in audit-trail is new in a release requiring Jenkins newer than 2.176.x, so the first call doesn't list it, the second does.

Concurrency is controllable via -D number or --downloadConcurrent number

It would be nice if the log output contained explicitly the values chosen for some arguments so I can be sure it's effective. -D 5 doesn't feel much different from the default of 48 😆

Ah, I think I figured out the issue for the parallelism. I was using a ForkJoinPool that used the available processors as its default limit. The way I've written this code would make more sense to simply use a cached thread pool instead of a work stealing one as this is mostly waiting on IO in an asynchronous API that isn't based on CompletableFuture in the first place. This seems to greatly increase performance correlating with the -D value. I'll push some updates around that.

Ooh, and I can add some manual code to handle retrying the HTTP request for ConnectionClosedException as that doesn't seem to be included in the library's auto-retry strategy.

Alright, few small changes since your last review, @daniel-beck:

Changed from a work-stealing thread pool (with parallelism=Ncpus) to a cached thread pool which makes more sense with a Semaphore since they're blocking on threads. (The equivalent way of doing that in a ForkJoinPool involves some other APIs I didn't feel like figuring out right now which likely won't improve the performance of this all that much)
Only prints a warning message when the checksum of a downloaded file doesn't match.
HTTP retry strategy configuration updated to use up to 3 retries spaced out by five seconds; this seemed to be sufficient in my local testing to get the super fat plugins to download more reliably.
Changed the log message for connection timeouts to specify that it gave up trying to download that URL; other errors will still show the generic exception error message.

WARNING! Checksum of http://updates.jenkins-ci.org/download/plugins/echarts-api/4.8.0-2/echarts-api.hpi does not match update center value!

Huh. Well, let's see.
$ curl -LI http://updates.jenkins-ci.org/download/plugins/echarts-api/4.8.0-2/echarts-api.hpi
HTTP/1.1 302 Found
Date: Thu, 09 Jul 2020 21:54:50 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: http://mirrors.jenkins-ci.org/plugins/echarts-api/4.8.0-2/echarts-api.hpi
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 302 Found
Date: Thu, 09 Jul 2020 21:54:50 GMT
Server: Apache/2.4.29 (Ubuntu)
X-MirrorBrain-Mirror: archives.jenkins-ci.org
X-MirrorBrain-Realm: other
Link: <http://mirrors.jenkins-ci.org/plugins/echarts-api/4.8.0-2/echarts-api.hpi.meta4>; rel=describedby; type="application/metalink4+xml"
Link: <http://archives.jenkins-ci.org/plugins/echarts-api/4.8.0-2/echarts-api.hpi>; rel=duplicate; pri=1; geo=us
Location: http://archives.jenkins-ci.org/plugins/echarts-api/4.8.0-2/echarts-api.hpi
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 404 Not Found
Date: Thu, 09 Jul 2020 21:54:50 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1

Still an inappropriate message, now even more panic inducing 😦

I am almost certain that the retry strategy isn't used. Even with ridiculously large numbers of retries and intervals, the failed downloads appear in the log immediately (as the plugin names are iterated in alphabetical order), which makes no sense.
In fact, I created an anonymous subclass of DefaultHttpRequestRetryStrategy to log some stuff, and there isn't a single call to #retryRequest with a execCount that isn't 1.

You did remember to compile it, right? mvn exec:java doesn't recompile the code when you change things; I learned that the hard way recently :)

Yes, and my changes were visible, otherwise I wouldn't have claimed

In fact, I created an anonymous subclass of DefaultHttpRequestRetryStrategy to log some stuff, and there isn't a single call to #retryRequest with a execCount that isn't 1.

25195  09.07.20 23:50:15 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25196  09.07.20 23:50:32 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json -C classes.txt -i'
25197  09.07.20 23:50:57 rm -rf work/
25198  09.07.20 23:51:01 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
…
25200  10.07.20 00:03:40 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25201  10.07.20 00:03:49 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
25202  10.07.20 00:09:36 rm -rf work/
25203  10.07.20 00:09:40 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25204  10.07.20 00:09:47 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
25205  10.07.20 00:13:08 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25206  10.07.20 00:13:13 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
25207  10.07.20 00:13:52 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25208  10.07.20 00:13:57 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
…
25210  10.07.20 00:21:04 rm -rf work/
25211  10.07.20 00:21:08 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25212  10.07.20 00:21:19 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
25213  10.07.20 00:21:59 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25214  10.07.20 00:22:04 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'
25215  10.07.20 00:24:39 /Applications/apache-maven-3.6.2/bin/mvn clean verify
25216  10.07.20 00:24:44 /Applications/apache-maven-3.6.2/bin/mvn exec:java -Dexec.args='--updateCenters=https://updates.jenkins.io/2.176/update-center.json,https://updates.jenkins.io/update-center.json -C classes.txt -i'

I think I figured out the problem with the retry strategy. It seems I misinterpreted the default exception list there as the ones to retry on, but those are the ones not to retry on! Seems like we'd actually like to retry on at least one of these scenarios.

Alright, who can merge this now?


add dynamic_settings

The goal is that at least some settings used after project initialization can be customized by plugin code.
I though about replacing the django.conf.settings object we use everywhere, but that caused circular-import-hell. Now it's very explicit when the dynamic mechanism is used. I like that we can use any prefix or other mechanism to provide defaults in settings.py using defaults from the env-file (feel free to suggests other names/mechanisms).
TODO

[x] add dynamic settings object
[x] migrate SITE_URL and USERCONTENT_URL
[ ] migrate brand_logo signal to brand_static_path
[ ] add platform name and primary color

coverage: 83.871% (-0.003%) from 83.874%
when pulling 34ba26273a8c9c6c6798d7a929246c11b1086af2 on dynamic-settings
into 9e8f868b670c7f24b7c77692bc616b8c477ed4f1 on main.


Fix protocol violation error happening due to Const node in append node

Description
Previously, certain driver was running into an protocol violation error when Const node in target list appears under Append node. This was happening because resolve_numeric_typmod_from_exp was not implemented to handle Const node properly for decimal data types. This commit fixes that issue by properly handling decimal data type for Const node under Append node.
Task: BABEL-5086
Signed-off-by: Dipesh Dhameliya<EMAIL_ADDRESS>Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

9 of 9 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.009%) to 73.428%

Totals

Change from base Build<PHONE_NUMBER>:
0.009%

Covered Lines:
42177

Relevant Lines:
57440

💛  - Coveralls


Fix/update models: make homePhone optional, add Class type, add Telemetry rowData

Follow-up to #44

Sorry, I closed #44 by mistake.
Question: why do you use None types for some fields? (e.g. home_phone, postal_code)
Is it because you only saw null values before and don't know yet what data to expect?

Correct, I generated the pydantic models from the raw API response data using datamodel-code-generator, wherever my own data had things missing the classes were generated with None values. I suppose I could also go back from the apk and look at the Java classes to see what they define but that code is quite messy so I'm not sure how much work that would involve


submitFieldName not setting name of hidden field

According to documentation
// Specify the name attribute for the hidden field that will contain the formatted date for submission.
submitFieldName: "date",

It seems that it sets the name only for the 3 input fields, but not for the formatted hidden input.
Looking at the sources
    // Build a hidden input and set this.$element as the wrapper
    hiddenField = $("<input/>", {
        type: "hidden", // Set to type hidden after development
        name: this.internals.submitFieldName
    });

Should probably be:
     name: this.config.submitFieldName

+1

Nice catch, fixed in the above commit :+1:


add telemetry for kernel launcher and finder

For #10897

[x] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR).
[x] Title summarizes what is changing.
[ ] Has a news entry file (remember to thank yourself!).
[ ] Appropriate comments and documentation strings in the code.
[ ] Has sufficient logging.
[x] Has telemetry for enhancements.
[ ] Unit tests & system/integration tests are added/updated.
[ ] Test plan is updated as appropriate.
[ ] package-lock.json has been regenerated by running npm install (if dependencies have changed).
[ ] The wiki is updated with any design decisions/details.

Codecov Report

Merging #11345 into master will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #11345      +/-   ##
==========================================
- Coverage   61.00%   60.98%   -0.02%
==========================================
  Files         606      606
  Lines       33448    33452       +4
  Branches     4730     4730
==========================================
- Hits        20404    20402       -2
- Misses      12021    12025       +4
- Partials     1023     1025       +2

Impacted Files
Coverage Δ

src/client/telemetry/index.ts
87.38% <ø> (ø)

src/client/datascience/constants.ts
99.71% <100.00%> (+<0.01%)
:arrow_up:

...client/datascience/kernel-launcher/kernelFinder.ts
62.24% <100.00%> (+0.78%)
:arrow_up:

src/datascience-ui/react-common/arePathsSame.ts
75.00% <0.00%> (-12.50%)
:arrow_down:

src/client/common/utils/platform.ts
64.70% <0.00%> (-11.77%)
:arrow_down:

src/client/linters/pydocstyle.ts
86.66% <0.00%> (-2.23%)
:arrow_down:

src/client/datascience/debugLocationTracker.ts
76.56% <0.00%> (-1.57%)
:arrow_down:

src/client/common/process/proc.ts
14.49% <0.00%> (-0.73%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8fd21c...daf356a. Read the comment docs.


Proposal: Adding a prune command/flag to the cargo CLI

Problem
Dependencies may be left orphaned or unused in a project as it grows and changes. This can increase build times and storage requirements for developer workstations. While it is up to developers to ensure their dependencies are in order, issuing a cargo command to remove these dependencies would be beneficial.
This seems like a common enough use case among developers to integrate it with the Cargo CLI.
Proposed Solution
Implementing a prune command cargo prune --[flags] to automatically mark unused dependencies. Some flags to consider would be a dry-run flag, dev dependencies only, and more.
Notes
Docker has a similar command docker image prune which will remove all unused images in a local registry. docs
Npm has a similar solution. docs
There is an existing solution to this problem. github

Huh, I thought we had an issue for this but I can't find it.
Existing solutions

https://crates.io/crates/cargo-machete
https://crates.io/crates/cargo-udeps

The big challenge with this is combining reliability with being able to detect unused deps across all build configurations.
If we did this, I imagine this would work better to be a lint controlled by #12235 that would support cargo fix removing the dependency.

I think the primary issue tracking this is https://github.com/rust-lang/rust/issues/57274, which contains some commentary about why this is very hard and some ideas for solutions.

By the way, the link npm and docker commands are not removing the unused dependency requirement they are instead cleaning up the built artifacts. To translate cargo they are cleaning the target directory, not the cargo.toml. So the work on GC is relevant, at least based on links if not the description.

Ah okay, thank you. I knew docker's prune is just to remove images without running containers, I thought npm was tidying up unused dependencies.


some errors are not raised, but rather printed

Simple code that reproduces your problem
when requesting a non-existing ticker symbol, the history method does not handle it as an error, but just a print output.
Therefore, since no error is raised, error handling using try and except is not possible.
For example:

import yfinance as yf

# trying a ticker that does not exist
ticker = yf.Ticker('T1XN')
try:
    ticker.history(period='max')
except:
    print("not found")

The error message

Got error from yahoo api for ticker T1XN, Error: {'code': 'Not Found', 'description': 'No data found, symbol may be delisted'}
- T1XN: No data found for this date range, symbol may be delisted

The error message is not actually an error message!
right now the error is a print output rather than raise

@ValueRaider okay thank you for quick response!
I guess this issues can be closed now!

For me, raise_errors has no effect here at all. Errors are just printed, not great when using tqdm to show progress.

@misantroop What ticker prints an error? Any ticker so we can reproduce.


Revert "Remove PaC secret reaper CronJob (#4399)"

This reverts commit 6392437d78053d06e7663b0913bc1f27e123ce7c.
The original commit removed a workaround for an issue considered fixed. The issue is still affecting Konflux, so I add the workaround back until a proper fix is deployed.

/test appstudio-e2e-tests

/test appstudio-e2e-tests

/test appstudio-hac-e2e-tests

/lgtm

/test appstudio-upgrade-tests


feat: track trade and donation amounts in usd

Description
Adds tracking of the USD amounts for trade and donations during trade both for the current Swapper and the not yet enabled MultiHop feature.
New properties sent to Mixpanel for the trade events: fiatAmountUsd and donationAmountFiatUsd
The existing fiatAmount and donationAmountFiat are preserved as we may use the non-USD amounts in other charts at some point, so I figure it's better to keep this data since we have it.

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[x] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #4866
Risk
None. No user facing change.

Testing
Make a trade, check the Mixpanel event sent when it is confirmed/fails.

Engineering
☝

Operations
☝

Screenshots (if applicable)

FYI, I've wondered about storing the user currency in that event data too, but it is retrievable from the user profile on Mixpanel (we track this setting), so I figure we shouldn't duplicate this data.

Seems like we've been working on the same thing at the same time unknowlingly this morning 😆
Converting to draft as it's very likely to be either discarded or cherrypicked to the benefit of #4893 !

LGTM! Closing as superseeded by https://github.com/shapeshift/web/pull/4893 which brings the vernacular across the swapper domain


Fixed: ListAgreementItemSupplierProducts doesn't show products OFBIZ-11682)

OFBIZ-11682)
fixed: wrong relation defined to get agreementProducts

Reworking required


Youtube video causes a crash when getting metadata

https://www.youtube.com/watch?v=nJ6tidrzxqM
Gets added to the queue, but soon crashed when trying to get metadata.
Calling yt-dlp from the command-line works with no issues.
Windows 10 64 bit, latest versions of everything.

Thanks for reporting this, I've released v1.6.4 which fixes the crash.