PostgreSQL error: PAM authentication failed with AWS RDS Token

Summary
We're using AWS RDS Postgres  DB as Kong DB. For security-wise, we're trying to use AWS RDS access token to connect to Kong DB.
It's successful to connect to DB with injected environment variables, but failed to start Kong with kong.conf file-based configuration.
Steps To Reproduce

Set up and start Kong with kong.conf, and pg_password = "AWS RDS Access Token", eg: pg_password = kong-service.xxxxxxxxxxx.aws_region.rds.amazonaws.com:5432/?Action=connect&DBUser=kong&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=aws_key%2F20200728%2Fus-west-2%2Frds-db%2Faws4_request&X-Amz-Date=20200728T083013Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Start Kong without any KONG_XXX_XXX environment variables
Run kong start --vv: it fails to connect to DB due to PAM authentication failure

Additional Details & Logs

Kong version 2.0.5
Kong debug-level startup logs

2020/08/04 05:32:30 [verbose] Kong: 2.0.5
2020/08/04 05:32:30 [debug] ngx_lua: 10015
2020/08/04 05:32:30 [debug] nginx: 1015008
2020/08/04 05:32:30 [debug] Lua: LuaJIT 2.1.0-beta3
2020/08/04 05:32:30 [verbose] reading config file at /home/kong/kong.conf
2020/08/04 05:32:30 [debug] reading environment variables
2020/08/04 05:32:30 [debug] admin_access_log = "/dev/stdout"
2020/08/04 05:32:30 [debug] admin_error_log = "/dev/stderr"
2020/08/04 05:32:30 [debug] admin_listen = {"<IP_ADDRESS>:8001","<IP_ADDRESS>:8444 ssl"}
2020/08/04 05:32:30 [debug] anonymous_reports = true
2020/08/04 05:32:30 [debug] cassandra_consistency = "ONE"
2020/08/04 05:32:30 [debug] cassandra_contact_points = {"<IP_ADDRESS>"}
2020/08/04 05:32:30 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2020/08/04 05:32:30 [debug] cassandra_keyspace = "kong"
2020/08/04 05:32:30 [debug] cassandra_lb_policy = "RequestRoundRobin"
2020/08/04 05:32:30 [debug] cassandra_port = 9042
2020/08/04 05:32:30 [debug] cassandra_refresh_frequency = 60
2020/08/04 05:32:30 [debug] cassandra_repl_factor = 1
2020/08/04 05:32:30 [debug] cassandra_repl_strategy = "SimpleStrategy"
2020/08/04 05:32:30 [debug] cassandra_schema_consensus_timeout = 10000
2020/08/04 05:32:30 [debug] cassandra_ssl = false
2020/08/04 05:32:30 [debug] cassandra_ssl_verify = false
2020/08/04 05:32:30 [debug] cassandra_timeout = 5000
2020/08/04 05:32:30 [debug] cassandra_username = "kong"
2020/08/04 05:32:30 [debug] client_body_buffer_size = "8k"
2020/08/04 05:32:30 [debug] client_max_body_size = "0"
2020/08/04 05:32:30 [debug] client_ssl = false
2020/08/04 05:32:30 [debug] cluster_control_plane = "<IP_ADDRESS>:8005"
2020/08/04 05:32:30 [debug] cluster_listen = {"<IP_ADDRESS>:8005"}
2020/08/04 05:32:30 [debug] database = "postgres"
2020/08/04 05:32:30 [debug] db_cache_ttl = 0
2020/08/04 05:32:30 [debug] db_cache_warmup_entities = {"services","plugins"}
2020/08/04 05:32:30 [debug] db_resurrect_ttl = 30
2020/08/04 05:32:30 [debug] db_update_frequency = 5
2020/08/04 05:32:30 [debug] db_update_propagation = 0
2020/08/04 05:32:30 [debug] dns_error_ttl = 1
2020/08/04 05:32:30 [debug] dns_hostsfile = "/etc/hosts"
2020/08/04 05:32:30 [debug] dns_no_sync = false
2020/08/04 05:32:30 [debug] dns_not_found_ttl = 30
2020/08/04 05:32:30 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2020/08/04 05:32:30 [debug] dns_resolver = {}
2020/08/04 05:32:30 [debug] dns_stale_ttl = 4
2020/08/04 05:32:30 [debug] error_default_type = "text/plain"
2020/08/04 05:32:30 [debug] go_plugins_dir = "off"
2020/08/04 05:32:30 [debug] go_pluginserver_exe = "/usr/local/bin/go-pluginserver"
2020/08/04 05:32:30 [debug] headers = {"server_tokens","latency_tokens"}
2020/08/04 05:32:30 [debug] kic = false
2020/08/04 05:32:30 [debug] log_level = "notice"
2020/08/04 05:32:30 [debug] lua_package_cpath = ""
2020/08/04 05:32:30 [debug] lua_package_path = "/custom-plugins/?.lua;;"
2020/08/04 05:32:30 [debug] lua_socket_pool_size = 500
2020/08/04 05:32:30 [debug] lua_ssl_trusted_certificate = "/cache/rds-combined-ca-bundle.pem"
2020/08/04 05:32:30 [debug] lua_ssl_verify_depth = 1
2020/08/04 05:32:30 [debug] mem_cache_size = "128m"
2020/08/04 05:32:30 [debug] nginx_admin_directives = {}
2020/08/04 05:32:30 [debug] nginx_daemon = "off"
2020/08/04 05:32:30 [debug] nginx_events_directives = {{name="worker_connections",value="auto"},{name="multi_accept",value="on"}}
2020/08/04 05:32:30 [debug] nginx_events_multi_accept = "on"
2020/08/04 05:32:30 [debug] nginx_events_worker_connections = "auto"
2020/08/04 05:32:30 [debug] nginx_http_client_body_buffer_size = "8k"
2020/08/04 05:32:30 [debug] nginx_http_client_max_body_size = "0"
2020/08/04 05:32:30 [debug] nginx_http_directives = {{name="client_max_body_size",value="0"},{name="ssl_prefer_server_ciphers",value="off"},{name="client_body_buffer_size",value="8k"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_session_tickets",value="on"},{name="ssl_session_timeout",value="1d"}}
2020/08/04 05:32:30 [debug] nginx_http_ssl_prefer_server_ciphers = "off"
2020/08/04 05:32:30 [debug] nginx_http_ssl_protocols = "TLSv1.2 TLSv1.3"
2020/08/04 05:32:30 [debug] nginx_http_ssl_session_tickets = "on"
2020/08/04 05:32:30 [debug] nginx_http_ssl_session_timeout = "1d"
2020/08/04 05:32:30 [debug] nginx_http_status_directives = {}
2020/08/04 05:32:30 [debug] nginx_http_upstream_directives = {{name="keepalive_requests",value="100"},{name="keepalive_timeout",value="60s"},{name="keepalive",value="60"}}
2020/08/04 05:32:30 [debug] nginx_http_upstream_keepalive = "60"
2020/08/04 05:32:30 [debug] nginx_http_upstream_keepalive_requests = "100"
2020/08/04 05:32:30 [debug] nginx_http_upstream_keepalive_timeout = "60s"
2020/08/04 05:32:30 [debug] nginx_main_daemon = "off"
2020/08/04 05:32:30 [debug] nginx_main_directives = {{name="daemon",value="off"},{name="worker_rlimit_nofile",value="auto"},{name="worker_processes",value="auto"}}
2020/08/04 05:32:30 [debug] nginx_main_worker_processes = "auto"
2020/08/04 05:32:30 [debug] nginx_main_worker_rlimit_nofile = "auto"
2020/08/04 05:32:30 [debug] nginx_optimizations = true
2020/08/04 05:32:30 [debug] nginx_proxy_directives = {{name="real_ip_header",value="X-Real-IP"},{name="real_ip_recursive",value="off"}}
2020/08/04 05:32:30 [debug] nginx_proxy_real_ip_header = "X-Real-IP"
2020/08/04 05:32:30 [debug] nginx_proxy_real_ip_recursive = "off"
2020/08/04 05:32:30 [debug] nginx_sproxy_directives = {}
2020/08/04 05:32:30 [debug] nginx_status_directives = {}
2020/08/04 05:32:30 [debug] nginx_stream_directives = {}
2020/08/04 05:32:30 [debug] nginx_supstream_directives = {}
2020/08/04 05:32:30 [debug] nginx_upstream_directives = {{name="keepalive_requests",value="100"},{name="keepalive_timeout",value="60s"},{name="keepalive",value="60"}}
2020/08/04 05:32:30 [debug] nginx_upstream_keepalive = "60"
2020/08/04 05:32:30 [debug] nginx_upstream_keepalive_requests = "100"
2020/08/04 05:32:30 [debug] nginx_upstream_keepalive_timeout = "60s"
2020/08/04 05:32:30 [debug] nginx_worker_processes = "auto"
2020/08/04 05:32:30 [debug] pg_database = "kong"
2020/08/04 05:32:30 [debug] pg_host = "ei-service.cnw4oga6wzpl.us-west-2.rds.amazonaws.com"
2020/08/04 05:32:30 [debug] pg_max_concurrent_queries = 0
2020/08/04 05:32:30 [debug] pg_password = "******"
2020/08/04 05:32:30 [debug] pg_port = 5432
2020/08/04 05:32:30 [debug] pg_semaphore_timeout = 60000
2020/08/04 05:32:30 [debug] pg_ssl = true
2020/08/04 05:32:30 [debug] pg_ssl_verify = true
2020/08/04 05:32:30 [debug] pg_timeout = 5000
2020/08/04 05:32:30 [debug] pg_user = "kong"
2020/08/04 05:32:30 [debug] plugins = {"bundled","kong-plugin-template-transformer","api-transformer"}
2020/08/04 05:32:30 [debug] prefix = "/usr/local/kong/"
2020/08/04 05:32:30 [debug] proxy_access_log = "/dev/stdout"
2020/08/04 05:32:30 [debug] proxy_error_log = "/dev/stderr"
2020/08/04 05:32:30 [debug] proxy_listen = {"<IP_ADDRESS>:8000","<IP_ADDRESS>:8443 ssl"}
2020/08/04 05:32:30 [debug] real_ip_header = "X-Real-IP"
2020/08/04 05:32:30 [debug] real_ip_recursive = "off"
2020/08/04 05:32:30 [debug] role = "traditional"
2020/08/04 05:32:30 [debug] router_consistency = "strict"
2020/08/04 05:32:30 [debug] router_update_frequency = 1
2020/08/04 05:32:30 [debug] ssl_cipher_suite = "intermediate"
Error:
/usr/local/share/lua/5.1/kong/cmd/start.lua:31: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: FATAL: PAM authentication failed for user "kong"
stack traceback:
	[C]: in function 'assert'
	/usr/local/share/lua/5.1/kong/cmd/start.lua:31: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45>
	/usr/local/bin/kong:9: in function 'file_gen'
	init_worker_by_lua:49: in function <init_worker_by_lua:47>
	[C]: in function 'xpcall'
	init_worker_by_lua:56: in function <init_worker_by_lua:54>
2020/08/04 05:32:30 [debug] ssl_ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
2020/08/04 05:32:30 [debug] status_access_log = "off"
2020/08/04 05:32:30 [debug] status_error_log = "logs/status_error.log"
2020/08/04 05:32:30 [debug] status_listen = {"off"}
2020/08/04 05:32:30 [debug] stream_listen = {"off"}
2020/08/04 05:32:30 [debug] trusted_ips = {}
2020/08/04 05:32:30 [debug] upstream_keepalive = 60
2020/08/04 05:32:30 [verbose] prefix in use: /usr/local/kong

Operating System: Alpine 3.11.6, docker image kong:2.0.5-alpine

Curious: would it be more adventagious / possible to utilize a AWS instance profile role as well as username / password?

Hi hutchic,
we'll try it .
Many thanks for you reply.

Curious: would it be more adventagious / possible to utilize a AWS instance profile role as well as username / password?

Hi hutchic,
it didn't work as well, but still works with environment variables.

Hi wtfyucheng, I'm closing this issue since it seems it was resolved with environment variables. Please reopen if this was not the case.


Project clone should deep copy notebook files

Currently cloning a project copies all the data in the data model but leaves individual GCS bucket files in place. This makes sense - we don't want people to pay for it twice! But notebook ipynb files in the /notebooks folder should be copied
child of #353

Technically, the right answer here appears to do this via the API. We should decide whether it belongs on Content + Discovery (likely) or Cloud Accounts. Brad will take this to the appropriate PMs

Notebooks are not the last analysis artifact we will have (python scripts, Rstudio session histories, perhaps others). So how generic do we need to go with our cloning rules?

Idea: UIs keep paths they know about and can specify to clone/not-clone them. We need to know where notebooks are to display, localize, save them.
So clone could become "clone with file pattern whitelist"

@ahaessly There's an update needed to workspace clone, which is properly on the Rawls endpoint. Do you think this a C+D issue?

We're going to drive the copy from the client. We may not be able to handle all the rough edges, but it's the fastest way to get needed functionality in the platform.
Behavior is to implicitly copy all ipynb files in the /notebooks directory.

Talk to Vikram before starting this ticket!

This is being done by Leonardo in time for ASHG. @vkumra-broad Do you have a ticket number?

The priority for this got pushed back, this is not being done for ASHG.
https://github.com/DataBiosphere/leonardo/issues/601

@vkumra-broad Thanks for the update! I see that it's in progress. If not for ASHG, do you have an idea when it might land?

I am hoping to get it in for the release after ASHG.

Blocking ticket in progress.

This appears to be in review in Rawls. PR here:
https://github.com/broadinstitute/rawls/pull/1014


RuntimeError: storage has wrong size: expected 67123990 got 1024

When I run "python tools/train_net_step.py --dataset coco2017 --cfg configs/baselines/e2e_mask_rcnn_R-50-C4_1x.yaml --use_tfboard --bs 16 --nw 4"
I meet error about torch.load()
INFO train_net_step.py: 234: Takes 124.40 sec(s) to construct roidb
Traceback (most recent call last):
File "tools/train_net_step.py", line 454, in
main()
File "tools/train_net_step.py", line 256, in main
maskRCNN = Generalized_RCNN()
File "/home/detection/mask-rcnn.pytorch/lib/modeling/model_builder.py", line 124, in init
self._init_modules()
File "/home/detection/mask-rcnn.pytorch/lib/modeling/model_builder.py", line 128, in _init_modules
resnet_utils.load_pretrained_imagenet_weights(self)
File "/home/detection/mask-rcnn.pytorch/lib/utils/resnet_weights_helper.py", line 29, in load_pretrained_imagenet_weights
pretrianed_state_dict = convert_state_dict(torch.load(weights_file))
File "/home/anaconda2/envs/tf/lib/python3.6/site-packages/torch/serialization.py", line 267, in load
return _load(f, map_location, pickle_module)
File "/home/anaconda2/envs/tf/lib/python3.6/site-packages/torch/serialization.py", line 427, in _load
deserialized_objects[key]._set_from_file(f, offset)
RuntimeError: storage has wrong size: expected 67123990 got 1024
So ,how can I solve it
Thank you very much

have you fixed this bug right now? I encountered this error as you~


Address npm audit issues

Addressed on issue in a PR the rest appear to be mostly devDependency issues via webpack / glup / postcss. Would be good to resolve during debt week.


callbacks in grpc.cc

Hi,
I am new in distributed learning. I am wondering what is the 'callback' in the FnType::operator() arguments used for in the grpc.cc. It seems that the python code only passes 'args' into the function, but there are quite many codes handling the callbacks. Is the 'callback' argument used in seed_rl?

Callbacks are used on the server side for asynchronous grpc and also internally when calling the registered tensorflow function.
https://github.com/google-research/seed_rl/blob/master/grpc/ops/grpc.cc#L721
https://github.com/google-research/seed_rl/blob/master/grpc/ops/grpc.cc#L716

Thank you so much for your quick response! The callback is used for 'timeout' and shape/status checking in f_callback, but I am still curious about what the callback  could be for asynchronous grpc, i.e. what callback function would be passed into the operator() function.
Another quick question, I am wondering if multiple batches will be inferred on a same device in parallel.
https://github.com/google-research/seed_rl/blob/eff7aaa7ab5843547fbf383fcc747b7a8ca67632/agents/vtrace/learner.py#L387-L390
For example, if we only have one inference device and an inference task is running on it when another batch received by the grpc, will the coming batch be assigned to the same inference device and run in parallel with the original task or waits for the former one until it completes?

I'm not sure I understand the first part of the question so can you rephrase it?
Whenever there is a batch ready the registered function 'inference' will be called, so there is no waiting. I have experimented with adding features to grpc to restrict how many can run at any given time. It's easy to restrict it in TF as well with e.g. tf.CriticalSection

Sorry for the implicitness.  What I would like to know is whether there are any callbacks used in the SEED-RL other than the callbacks that appeared in the grpc.cc code.

No, the only callbacks are in the grpc.cc

Thank you so much for your patience! That really helps! I am closing this issue.


Check the result of SSL certificate verification

It is unsafe to use a certificate without checking if it is valid.

plan to switch the openssl to mbedtls


feat: filter by service and city

As a user, I would like to filter services based on the city and the service type. see https://trello.com/c/TnFo7mCd/310-filter-by-city & https://trello.com/c/5To9j8Z2/311-filter-by-service
Acceptance Criteria

[ ] route /services displays all services
[ ] route /services/:location?/ displays services that match the location name
[ ] route /services/:location?/:service?/ displays services that match the service type
[ ] spinner is shown when data is loading

Demo
https://deploy-preview-76--wehelpvip.netlify.app/services/

route /services displays all services
route /services/:location?/ displays services that match the location name
spinner is shown when data is loading
Above mentioned acceptance criteria is working fine  in android device ,ipad and laptop.
/services/:location?/:service?/ displays services that match the service type
only the above one is not working fine.


Export fromShelleyBasedScript from Cardano.Api

New PR based off of a branch in this repo. Old PR here https://github.com/input-output-hk/cardano-node/pull/4386

bors r+


Wrong default value for fp16

https://github.com/Megvii-BaseDetection/YOLOX/blob/8ba73adbad0415f56378eb77f8d28ea7969fb87c/tools/train.py#L60

Thx. We actually set fp16 at true by default, as it saves GPU memory and has no influence on the accuracy.

then perhaps the action should be store_false ?

then perhaps the action should be store_false ?

then the name might be --no_fp16 or --no_amp


Google Analytics user journey measurements

[ ] can pages be grouped, for example editorial community pages in the Behaviour Flow?
[ ] are Exit Pages interesting? e.g. lots of users exiting the flow at a certain point
[ ] how do we measure outgoing links to Biorxiv being clicked?
[ ] typing search terms or scrolling through the search page: can we track them with some kind of custom events and JavaScript?
[ ] should have a cookie notice?
[ ] how do we differentiate someone doing a demo from end users? Do UTM parameters help?

We should be able to do something around https://support.google.com/analytics/answer/7478520?hl=en to track outbound links.

Content Grouping for aggregating e.g. all Editorial communities: https://support.google.com/analytics/answer/2853423

Added a Content Grouping to test tomorrow after it has taken effect:
https://analytics.google.com/analytics/web/?authuser=1#/a167311475w233457116p219087825/admin/content-groupings/m-content-contentGroupingTable.rowShow=10&m-content-contentGroupingTable.rowStart=0&m-content-contentGroupingTable.sortColumnId=dateModified&m-content-contentGroupingTable.sortDescending=true

More on events:
    Category: "Articles"
    Action: "ClickToFullText"
    Label: "10.1101/845081"

typing search terms or scrolling through the search page: can we track them with some kind of custom events and JavaScript?

If you can subscribe a event listener in JavaScript, you can the send an event:
gtag('event', <action>, {
  'event_category': <category>,
  'event_label': <label>,
  'value': <value>
});

This is likely overkill for the cases mentioned here, but could be useful in other contexts.

Should try to use utm_campaign when demoes are done, so that this can be tracked separately from organic traffic:
http://prc.libero.pub/?utm_campaign=demos-to-editorial-communities
When we give out the link to a certain group of people, we can even put a separate utm_campaign or other parameters.

Assumptions I am trying to verify:

Campaign in All Pages can be used a secondary dimension with these values
I thought that utm_campaign is tracked for a whole user session (as soon as you click another internal link it wouldn't be in the URL anymore)

When you add parameters to a URL, you should always use utm_source, utm_medium, and utm_campaign.

More like this then:
http://prc.libero.pub/?utm_source=paul-shannon-friends&utm_medium=demo&utm_campaign=psychoceramics-demo-2020-07-13

List of campaigns
A report with campaign as a secondary dimension:

I think we need a cookie notice, that loads Google Analytics only after consent.

Cookiebot report


Packaged ApplicationInsights.xml contains hard coded instrumentationKey

Environment

Spring boot starter:
azure-spring-boot-starter  2.1.3 (and from at least 2.0.8)

Summary
The packaged jar contains an ApplicationInsights.xml that with the following instrumentationKey:
    <InstrumentationKey>d3278c59-22d7-4697-af8c-ff0077e3b4e9</InstrumentationKey>

Reproduce steps
When used together with the logback appender (from applicationinsights-logging-logback) that instrumentationKey was used instead of the one that we wanted.
Expected Results
Should only use our instrumentationKey - not any hard coded ones.
Actual Results
No logs arrived to our application insight - and after a long debugging session we found out that another instrumentationKey was actually being used.

@LennartHaggkvistHM
Thanks for your issue. It may related to ai sdk configuration mechanism. We are working on it.

@Incarnation-p-lee yes I think this is correct. It would be interesting to investigate how this behaves when customer's app has one of the azure starter, and the user himself also adds the Application Insights SpringBoot starter to configure their app.
you should ensure that telemetry is not polluted.

User telemetry goes to his or her ikey.
The usage telemetry goes to your ikey.

Also, it is very tricky to provide the hardcoded ikey inside the config file. This can cause potential security issues also.

@dhaval24 @LennartHaggkvistHM
Thanks for your issue here. We have hotfix for this issue and remove the ApplicationInsights.xml for now. Could you please help to try the 2.1.4 version.
To avoiding less impact to user, we would like to file another issue to track this.

@Incarnation-p-lee We are still on spring boot 2.0 and as far as I can see there has been no release for that branch?

@LennartHaggkvistHM
Yes, still under working. The release 2.1.x for spring 2.1.x may have higher priority.

The 2.0.12 release seems to fix the problem for us. Thanks!

Thanks @LennartHaggkvistHM for issue this. Any other problem please feel free to contact us.


Add option to select fields in email report

Add checkboxes to allow the users to select the fields to be included in the generated report.

Included option to include/exclude the 'Notes' and 'Tags' fields.


Optimize ResolveGatewayName

In large clusters this has been shown to uses ~20% of CPU and ~10%/60% of
allocations  in terms of byes/number leading to huge GC trashing
BenchmarkResolveGatewayName/old-8               10342688
1222 ns/op             320 B/op         16 allocs/op
BenchmarkResolveGatewayName/new-8               39173893
322 ns/op              64 B/op          4 allocs/op
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
4e0c42a4072596039c017e434a75b39e6b80a939
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

/cherrypick release-1.6

@howardjohn: new pull request created: #25611

In response to this:

/cherrypick release-1.6

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Having errors as modal screens (generalized)

Description
Currently we have got different ways to show errors. Errors should get shown as modal screens everywhere.
Acceptance criteria

Tasks

[ ]

Suggested next steps

Collect all error messages that we currently display
Decide the nature of each error message

In most cases they will be displayed within a modal similar to the newly implemented transfer modal in issue https://github.com/raiden-network/light-client/issues/426#issuecomment-560426394
Worth concidering are also error messages such as the one show when a user provides invalid input


Modified route_guide sample app to show RPC log

Modified the screens to show meaningful information about RPC progress.
When you click on each tab the screen now shows log messages that previously
only went to the debug console.

Please use camelCase for names

@jcanizales The test failures are unrelated since we don't build this sample. Anything else needs doing?

Just one style issue, and two ideas to remove boilerplate.

LGTM!

test this please

test this please


Add @tag support to the table for schema contracts

It would be helpful to know which libraries support schema contracts @tag directive.  Would also be a good signal to maintainers to add support for it

I believe it 'just works so long as you declare the schema directive in your SDL file as per the tag spec
There was no need for any actual directive implementation for the use cases I've done with GraphQL Java / DGS.  Rover + Apollo Studio does the rest of the magic.

Ah yeah I guess some gql libs do support directives out of the gate.  I know Elixirs absinthe does not support custom directives and some other libraries prob dont but your right prob isnt a concern for this project, unless we make this more generic and have a column for supports custom schema directives or something

I think this issue can be closed now that the compatibility matrix has been updated to show fed1 / fed2


dynamic state

will use it insted of all other states

    mapping(uint256 => uint256[]) public data;

In case of using an array, we will pay more gas.

let's close this issue then.


🐛 BUG: Stoping Nebula does not free up port :4242

What version of nebula are you using?
1.7.12
What operating system are you using?
all
Describe the Bug
When running Nebula programmatically, it does not free up port :4242
This is causing some issues, for example re-connecting on the Android app requires restarting the app completely to bypass the error complaining about the port being already busy.
You can test this easily like this:
package main

import (
	"github.com/slackhq/nebula"
	"github.com/slackhq/nebula/config"
	"github.com/sirupsen/logrus"
	"os"
	"fmt"
	"time"
)

func startNebula(configRaw []byte) *nebula.Control {
	logrus := logrus.New()
        logrus.Out = os.Stdout
	logrus.Info("## Starting nebula")

	c := config.NewC(logrus)
	err := c.LoadString(string(configRaw))
	if err != nil {
		fmt.Println(err)
		return nil
	}

	// Adjust PKI paths to be absolute

	l, err := nebula.Main(c, false, "", logrus, nil)

	if err != nil {
		fmt.Println(err)
		return nil
	}

	l.Start()

	logrus.Info("## Started Nebula")

	return l
}

func main() {
	// read config from test.yml
	config, err := os.ReadFile("test.yml")
	if err != nil {
		fmt.Printf("Failed to read config: %v", err)
		return
	}

	l := startNebula(config)

	time.Sleep(2 * time.Second)

	l.Stop()

	time.Sleep(2 * time.Second)

	startNebula(config)
}

Logs from affected hosts
No response
Config files from affected hosts
No response

Hi @azukaar -
AFAIK, calling nebula.Main multiple times is unsupported and listen.port is not a reloadable setting. In other words, this is expected behavior. Maybe you can describe more clearly what you're trying to achieve?

This is causing some issues, for example re-connecting on the Android app requires restarting the app completely to bypass the error complaining about the port being already busy.

If you're experiencing a bug with Mobile Nebula, please file a ticket at https://github.com/DefinedNet/mobile_nebula. This isn't something I've seen reports of, nor do I experience myself. In the mobile app, we kill the Nebula process and create a new one when restarting.

I was writing a OS service was a desktop application connecting to Nebula
Having to restart the service on every disconnection is a little overkill in what you would normally expect to be doing (just stop() and start() the service)
Feel free to close the ticket if that's not something you plan on improving

Hi @azukaar -
Nebula was written as an application first and so the code was written in such a way that it expects a systemctl stop nebula (for example), shutting the entire process down and bringing it back up.
Some initial support for using Nebula like a library was first added in order to support Mobile Nebula, IIRC.
Some configuration can be reloaded via the reload() call. Does this achieve what you're looking for?

Thanks for your input!
I will just restart the service for now (reload() does not work as I would like to change the config)

@azukaar I think that's the best option at the moment...
But I didn't mean to imply this isn't something we would like to support. Just not something we seem to have tackled yet.
Can you share which platform you're running on?

I was mistaken. This should be doing what you want already...
In interface.go we do try to close all listeners:
https://github.com/slackhq/nebula/blob/280fa026ead8693abb6f04fa687920fda19a57df/interface.go#L410-L415
The UDP listener is passed to Interface here:
https://github.com/slackhq/nebula/blob/280fa026ead8693abb6f04fa687920fda19a57df/main.go#L299
The Interface should be closed when the Control is closed:
https://github.com/slackhq/nebula/blob/280fa026ead8693abb6f04fa687920fda19a57df/control.go#L82-L84

Yes I saw this code that is why I was confused and marked it as a bug,
if you try the code I sent earlier you will see that the port is indeed not closed (tested on both Android and Windows with 1.7.2)
I closed the ticket after your previous comment but if you agree it is actually a bug it can be re-opened

@azukaar It looks like this issue was fixed in a3e59a38eff2a4942fb42938a8243fe73aa62650. Can you re-test with your code pointed at master instead of v1.7.2?

@azukaar Were you able to re-test with master?

Sorry for delay @JohnMaguire the issue cannot be reproduced on @master branch

No worries, thanks for checking. Closing as fixed by a3e59a38eff2a4942fb42938a8243fe73aa62650


Expose new Admin endpoint on orderer for the channel participation API

Type of change

New feature

Description
First commit

split out common system logic for http endpoints from existing operations system code

Second commit

expose new Admin endpoint on the orderer and move the channel participation API to this endpoint

Related issues
FAB-18246

Opened a new flake, FAB-18251, for the earlier flake (and pushed a likely fix: https://github.com/hyperledger/fabric/pull/1941). Current flake is FAB-16233. Grabbed the logs for that one and I'll take a look to see if anything jumps out.

Update commit message to reflect package names after last commit.

Update commit message to reflect package names after last commit.

Thanks for catching this. Noticed it last week but forgot to get back to updating it.

Looks good to me.

Just a rebase on master since this has been sitting around a while.


Cannot go through the guide

Hello,
I follow the instructions the reach the step "Training my own", and met an error below.
plz help me to solve it.
Thanks!
https://github.com/tensorflow/magenta/tree/master/magenta/models/melody_rnn
melody_rnn_train
--config=attention_rnn
--run_dir=/tmp/melody_rnn/logdir/run1
--sequence_example_file=/tmp/melody_rnn/sequence_examples/training_melodies.tfrecord
--hparams="batch_size=64,rnn_layer_sizes=[64,64]"
--num_training_steps=20000
ERROR:
Traceback (most recent call last):
File "/root/anaconda3/bin/melody_rnn_train", line 11, in
load_entry_point('magenta', 'console_scripts', 'melody_rnn_train')()
File "/eviluess/magenta/magenta/models/melody_rnn/melody_rnn_train.py", line 108, in console_entry_point
tf.app.run(main)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/root/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/root/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/eviluess/magenta/magenta/models/melody_rnn/melody_rnn_train.py", line 104, in main
checkpoints_to_keep=FLAGS.num_checkpoints)
File "/eviluess/magenta/magenta/models/shared/events_rnn_train.py", line 84, in run_training
is_chief=task == 0)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/training/python/training/training.py", line 549, in train
loss = session.run(train_op, run_metadata=run_metadata)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 861, in exit
self._close_internal(exception_type)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 899, in _close_internal
self._sess.close()
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1166, in close
self._sess.close()
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1334, in close
ignore_live_threads=True)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/root/anaconda3/lib/python3.7/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Name: , Key: inputs, Index: 0.  Number of float values != expected.  values size: 38 but output shape: [74]
[[{{node ParseSingleSequenceExample/ParseSingleSequenceExample}}]]
(base) ➜  ~

Looks like you did not set the config correctly when following the steps for sequenceExamples generation there. That could be causing the difference in shapes it is complaining about. Could you paste the command you used to generate the sequenceExample?

Thanks for your quick reply.
The config is lookback_rnn.
For the detail of what I have done, I'll show you all my steps (from the very beginning to error):

I tested the environment using "Generate a melody":

BUNDLE_PATH=/eviluess/res/lookback_rnn.mag
CONFIG=lookback_rnn
melody_rnn_generate
--config=${CONFIG}
--bundle_file=${BUNDLE_PATH}
--output_dir=/tmp/melody_rnn/generated
--num_outputs=10
--num_steps=128
--primer_melody="[60]"
And it's fine and shows:
Wrote 10 MIDI files to /tmp/melody_rnn/generated

To build a dataset, I followed "Building your Dataset" and actually do the following things:
2.1. I downloaded "Building your Dataset" and unpacked it with tar -xf and no error occurs.
2.2. Convert a subset of it to NoteSequences:
INPUT_DIRECTORY=/eviluess/res/lmd_matched/A/A
SEQUENCES_TFRECORD=/tmp/notesequences.tfrecord

convert_dir_to_note_sequences
--input_dir=$INPUT_DIRECTORY
--output_file=$SEQUENCES_TFRECORD
--recursive
And it ran fine. The last line of the result is:
INFO:tensorflow:Converted MIDI file /eviluess/res/lmd_matched/A/A/S/TRAASKZ128F9308820/886270e42b69e0983dfd9a591e66f214.mid.
I1102 17:43:05.389676<PHONE_NUMBER>02656 convert_dir_to_note_sequences.py:133] Converted MIDI file /eviluess/res/lmd_matched/A/A/S/TRAASKZ128F9308820/886270e42b69e0983dfd9a591e66f214.mid.
2.3. Create SequenceExamples -- the step you concert:
melody_rnn_create_dataset
--config=lookback_rnn
--input=/tmp/notesequences.tfrecord
--output_dir=/tmp/melody_rnn/sequence_examples
--eval_ratio=0.10
It succeeded with the following results:
INFO:tensorflow:DAGPipeline_TranspositionPipeline_training_transpositions_generated: 780
I1102 17:48:18.414772<PHONE_NUMBER>97824 statistics.py:141] DAGPipeline_TranspositionPipeline_training_transpositions_generated: 780
And I checked the directory, the files are there indeed.
2.4. Train and Evaluate the Model
I ran the command directly:
melody_rnn_train
--config=attention_rnn
--run_dir=/tmp/melody_rnn/logdir/run1
--sequence_example_file=/tmp/melody_rnn/sequence_examples/training_melodies.tfrecord
--hparams="batch_size=64,rnn_layer_sizes=[64,64]"
--num_training_steps=20000
And I got errors like the post.

So you cannot use look back_rnn in sequence example generation and attention_rnn in training. Both need to be same as input sizes for both configurations are different. Right now, you're generating input for lookback and trying to pass it into attention rnn. Change one of those as per your choice and it should work.

This blog has details on why sizes of both would differ: https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn

Hey, Sorry for my late response.
I don't have the machine that has the environment to run the code.
I'll try your solution asap.
Thank you very much.

Now I have a new issue that seems to be the hardware configuration, or could you plz give me some advice to fix it, thanks!
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 725, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1207, in init
_WrappedSession.init(self, self._create_session())
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1212, in _create_session
return self._sess_creator.create_session()
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 878, in create_session
self.tf_sess = self._session_creator.create_session()
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 647, in create_session
init_fn=self._scaffold.init_fn)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/session_manager.py", line 290, in prepare_session
config=config)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/session_manager.py", line 194, in _restore_checkpoint
sess = session.Session(self._target, graph=self._graph, config=config)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1585, in init
super(Session, self).init(target, graph, config=config)
File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 699, in init
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid device ordinal value (3). Valid range is [0, 2].
while setting up XLA_GPU_JIT device number 3

More logs before the previous post:
2019-11-05 20:25:52.957209: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/anaconda3/lib/:
2019-11-05 20:25:52.960330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-11-05 20:25:52.960357: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-11-05 20:25:52.960723: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-11-05 20:25:52.969081: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-11-05 20:25:52.971085: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c1e97be5f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-11-05 20:25:52.971110: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2019-11-05 20:25:52.974750: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:3: failed initializing StreamExecutor for CUDA device ordinal 3: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
2019-11-05 20:25:52.975185: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:4: failed initializing StreamExecutor for CUDA device ordinal 4: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
2019-11-05 20:25:53.142180: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:7: failed initializing StreamExecutor for CUDA device ordinal 7: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2019-11-05 20:25:53.154143: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:5: failed initializing StreamExecutor for CUDA device ordinal 5: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2019-11-05 20:25:53.163928: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:6: failed initializing StreamExecutor for CUDA device ordinal 6: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2019-11-05 20:25:54.909159: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c1e93af0c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2019-11-05 20:25:54.909198: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2019-11-05 20:25:54.909207: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Tesla V100-PCIE-16GB, Compute Capability 7.0
2019-11-05 20:25:54.909215: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (2): Tesla V100-PCIE-16GB, Compute Capability 7.0
Traceback (most recent call last):
File "/root/anaconda3/bin/melody_rnn_train", line 11, in
load_entry_point('magenta', 'console_scripts', 'melody_rnn_train')()
File "/eviluess/magenta/magenta/models/melody_rnn/melody_rnn_train.py", line 108, in console_entry_point

This looks like possibly an issue of Cuda version mismatch/ not added to path correctly.

Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Certainly points to that.
What's the output of
tf.test.is_gpu_available()?


build: minify protos.json in the build directory

Slightly reduces the size of the unpacked files (from 58MB to 55MB). We do it for some other packages (e.g. Firestore) and we need to start doing it for all packages.

cool


added parallel pytest

Codecov Report

Merging #42 (06fddab) into master (978eaf2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #42   +/-   ##
=======================================
  Coverage   69.79%   69.79%
=======================================
  Files           7        7
  Lines         384      384
=======================================
  Hits          268      268
  Misses        116      116

Flag
Coverage Δ

unittests
69.79% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 978eaf2...06fddab. Read the comment docs.


feat(@angular-devkit/core): ng help/--help in JSON format

This change prints out the help texts in JSON format.
It is meant for CLI tools to process the output, e.g. to transform it into bash or zsh completion instructions.
In other words, this is the precondition for ng completion 2.0.

@hansl I cannot view the review findings.
My bad for not considering the impact on schematics and architect.
Will the JSON format be discussed on slack or in your "internal" project meetings ?

Internal project meetings. We'll make the design available. It will most likely be very close to the way we declare Schematics and Architect inputs.

@catull ,
ng <command> --help=json will be supported in today's 7.0 beta (I'm waiting for my PR to go in). Closing this as it covers this PR. Cheers!


Feature Request : HTTP MPEG-TS & SRT custom port caller and custom port Play

Describe the feature
HTTP MPEGTS  SRT custom port caller and custom port Play
I appreciate the recent updates I have a couple of feature requests in mind
1 HTTP MPEGTS Playback It would be fantastic if you could introduce the capability to play HTTP MPEGTS streams This feature would enhance the versatility of the system
http://localhost/streamname.ts or something. I hope mpeg2/h262 supported
2 SRT Publishing Without Stream Name It would be convenient if we could publish SRT streams without specifying a stream name just by using the format srt://ip:port Additionally it would be great if we could define custom ports for SRT playback for example srt://ip:customport These enhancements would make working with SRT more flexible
To add a section for predefined SRT streams with custom ports for publishing and playing in a mediamtx.yaml configuration file you can modify the YAML file as follows
Add a section for SRT configuration in existing mediamtx.yaml
srtstreams
name SRT Stream 1
publishurl srt://localhost:2001
playurl srt://localhost:3001
name SRT Stream 2
publishurl srt://localhost:2002
playurl srt://localhost:3002
Existing configuration
Each stream is defined with a name a descriptive name for the stream publish url the SRT URL for publishing and read/play url the SRT URL for playing
Thank you for considering these suggestions

Hello,
First of all, instructions say

Please create a request FOR EACH feature. Do not report multiple features in a single request, otherwise they'll probably never get ALL implemented.

Regarding the two features you're suggesting:

1 HTTP MPEGTS Playback It would be fantastic if you could introduce the capability to play HTTP MPEGTS streams This feature would enhance the versatility of the system
http://localhost/streamname.ts or something. I hope mpeg2/h262 supported

This is a legacy format that is used by less than 1% of live streams. Adding it wouldn't bring any benefit and increase the efforts to maintain the code.

2 SRT Publishing Without Stream Name It would be convenient if we could publish SRT streams without specifying a stream name just by using the format srt://ip:port Additionally it would be great if we could define custom ports for SRT playback for example srt://ip:customport These enhancements would make working with SRT more flexible
To add a section for predefined SRT streams with custom ports for publishing and playing in a mediamtx.yaml configuration file you can modify the YAML file as follows

This server works by opening a fixed number of ports, in order to allow users behind a NAT to map those ports and accept connections from outside the LAN. Opening a port for every single SRT stream would go against this design pattern, and would also limit the overall number of streams to 65536. Furthermore, allocating streams without an explicit configuration would not be possible anymore, since their port would be random and therefore impossible to determine without looking at the server logs.


Move system-dependant logic to load dynamic library into minimal module, and use geos-config to get development headers

The main thing done here is to move the system-dependant loading of the GEOS dynamic library into a separate module shapely.libgeos, which can be used by setup.py for checking versions before install, and by shapely.geos for run-time use the GEOS of dynamic library.
A TODO item is to figure out a different way to specify the dynamic library (#234). Someone else might need to take the lead on this, but at least it will be in one place, rather than repeated in setup.py and shapely/geos.py. CC @kynan for ideas.
Secondly, geos-config, where available, is used to obtain libraries and headers for developing Cython extensions. This is useful where GEOS is installed on a non-standard prefix. An alternative location for this utility can be proved for setup.py using a GEOS_CONFIG environment variable. Note that this is only needed for setup.py, so I don't think it's suitable for setting dynamic library prefix. Also, geos-config is a shell script, and is not available for MS Windows.
Other edits use the logging module for setup.py, and fixing the requires metadata. Extra debugging info can be shown e.g. python -W all setup.py .... Also add TODO that free is not used anywhere.
Please test and comment, as these files are rather sensitive components.

@mwtoews Fantastic, I really appreciate this. Could you retarget this at master? It's a little much for maint. I may cherry pick to address #234 or I might just unwind the breaking changes I made in order to release 1.5.6.

Here's something I can't figure out how to do: is it possible to change the target branch from Toblerity:maint to Toblerity:master? Or do I need to make a new PR?

It looks like this is a limitation of GitHub, so see PR #239 for re-targeting on Toblerity:master

Changes Unknown when pulling 6aa26d85c7c3564b9ccadabee02ba2a0780170ce on mwtoews:setup into ** on Toblerity:maint**.


code can't select lang tag.

Hi @cm16
Tell me more about the issue.

Hi @cm16
Tell me more about the issue.

I want to apply like this action.

Sorry, this function is not implemented in the editor not yet.

thank your editor.  I will waiting this function. 👍

Thank you.
However, I don't know when language support will be added.
I will add it to the todo list in suneditor project and close this issue.


Strange behaviour with integer argument, potential bug

Hi,
It seems like integer are not properly decoded for some reason. For example the following:
 Verk.enqueue(%Verk.Job{
      queue: :algolia_indexer,
      class: "Jumpn.AlgoliaIndexUpdater",
      args: [%{"activity_id" =><PHONE_NUMBER>29462191}],
      max_retry_count: 2
    })

is failing with:
[error] GenServer #PID<0.874.0> terminating
** (stop) :failed
Last message: {:"$gen_cast", {:perform, %Verk.Job{args: [%{"activity_id" => 3.9308728862946e17}], class: "MyWorker", enqueued_at:<PHONE_NUMBER>, error_backtrace: "    :erlang.binary_to_integer(393087288629462191)\n    (backend) lib/jumpn/activity/algolia_index_updater.ex:20: Jumpn.AlgoliaIndexUpdater.perform/1\n    (verk) lib/verk/worker.ex:35: Verk.Worker.handle_cast/2\n    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4\n    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6\n", error_message: "argument error", failed_at:<PHONE_NUMBER>, finished_at: nil, jid: "4749953972661461251", max_retry_count: 2, original_json: "{\"error_message\":\"argument error\",\"max_retry_count\":2,\"jid\":\"4749953972661461251\",\"class\":\"Jumpn.AlgoliaIndexUpdater\",\"retried_at\":null,\"queue\":\"algolia_indexer\",\"finished_at\":null,\"enqueued_at\":1530103850,\"error_backtrace\":\"    :erlang.binary_to_integer(393087288629462191)\\n    (backend) lib\\/jumpn\\/activity\\/algolia_index_updater.ex:20: Jumpn.AlgoliaIndexUpdater.perform\\/1\\n    (verk) lib\\/verk\\/worker.ex:35: Verk.Worker.handle_cast\\/2\\n    (stdlib) gen_server.erl:637: :gen_server.try_dispatch\\/4\\n    (stdlib) gen_server.erl:711: :gen_server.handle_msg\\/6\\n\",\"args\":[{\"activity_id\":3.9308728862946e+17}],\"failed_at\":1530103850,\"retry_count\":1}", queue: "algolia_indexer", retried_at: nil, retry_count: 1}, #PID<0.756.0>}}
State: nil

2 things to notice from the errror and stacktrace:

Somehow in the last message, the integer has been casted to a float 3.9308728862946e17
The root issue seems to be with the argument error being emitted by :erlang.binary_to_integer(393087288629462191). I dont understand why and how a casting from String to Integer is happening.

Any idea what is going on ?

@edgurgel any though ? Thanks in advance !

cjson strikes again?
<IP_ADDRESS>:6379> EVAL 'return cjson.encode(cjson.decode(\'{"args":[{"activity_id":393087288629462191}]}\'))' 0
"{\"args\":[{\"activity_id\":3.9308728862946e+17}]}"

Comparison with Poison:
iex(13)> Poison.decode!(Poison.encode!(%{"activity_id" =><PHONE_NUMBER>29462191}))
%{"activity_id" =><PHONE_NUMBER>29462191}

-.- @krasio haha cjson is basically the reason of all problems :)

Yeah it's because this is not an "integer" anymore. It's a "Big Number" that cjson won't support.
For numbers that are larger than 32 bits you will need to use a string I think and then convert

More here: https://github.com/DaveGamble/cJSON/issues/162
I'm not sure if we can do much apart from documenting that we should avoid numbers greater than 2^32

Ha thanks for pointing out cJson, I did not realise that the LUA scripts needed to decode the job to move it around on redis side.
The thing is the LUA side does not care at all about the job args so it seems we only need to wrap it in a way that it will be treated as a string as far as cJson is concerned but on elixir side when we decode the job, we could unwrap it.
Simply doing a 2 pass Json encoding/decoding would do the trick.
What do you think ?

Ok so the reason why we need to encode again is for this:
https://github.com/edgurgel/verk/blob/master/priv/enqueue_retriable_job.lua#L7-L12
We had enqueued_at to the JSON.
A really really nasty solution?
Add "manually" to the original json the key "enqueued_at" : ...
Then we won't suffer of cjson encoding rules?

Ya i saw that and it's fine, my point is as far as Redis / LUA is concerned, the args can be opaque, it never needs them. So we can protect them from cJson by simply encoding them as a json string using Poison meaning that in Redis, the args is a Json encoded string.
We simply need to have a custom decoder / encoder with Poison. Might be a good opportunity to switch to Jason as well.

That's the problem. It needs them.
It needs to know which queue will be moved to and it needs to populate the enqueued_at. We can't remove from the sorted set before we add to the queue so we don't have jobs out of Redis at any point in time.

Or do you mean double encoding just args?

Now I understood your point! :)

Yeah I'm completely fine doing this approach + using Jason if it makes sense 👍

The only downside that I can think of is that it won't have the property of accepting "Sidekiq/Resque" clients sending jobs to Verk. They would need to double encode args before enqueuing the job 🤔

Hmm I did not think about that.. We should be able to detect this easily enough, the args should never be a string to begin with so I can unwrap when I detect one (a job being enqueued by Verk) or otherwise bypass this step (a job enqueued by Sidekiq/Resque).

I have pushed the commit to take care of that and fixed the 2 credo issues.


Wandb local server start error

Current Behavior
Hi Wandb team,
I am a wandb user and currently encountering the following error when we run: wandb server start
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/12/19 23:47:02 Created default user
2022/12/19 23:47:02 Generating new session key for auth...
2022/12/19 23:47:02 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 48
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully local/local-files.
mc:  Unable to set 'notify_redis address=<IP_ADDRESS>:6379 format=access queue_limit=10000 queue_dir=/etc/minio/events key=filemetadata' to server: one or more targets are offline. Please use mc admin info --json to check the offline targets.
mc:  Unable to enable notification on the specified bucket. A specified destination ARN does not exist or is not well-formed. Verify the destination ARN.
Above is the output from docker logs wandb-local
Please help. Many thanks.
Expected Behavior
No response
Steps To Reproduce
No response
Screenshots
No response
Environment
OS: ubuntu linux 16.04
Browsers: Chrome and Firefox
Version: 16.04, pip 22.2.2, Docker version 18.09.7, build 2d0083d
Additional Context
No response

Hi @liswp, can you double check that minio is set up properly by referring to these docs please?

Checking now!

@lesliewandb I am checking the docs now, is there a particular cli command you'd like me to run to confirm minio is correctly set up?

@lesliewandb And just to be clear, does wandb require minio server, client, or both?

@lesliewandb I can confirm that minio server has always been restored, according to the screenshot below:

@liswp do you have any runs yet or is this an entirely new server you are trying to start? If this is a new server, I'd recommend deleting the docker volume and image, then trying wandb server start again.
Let me know if that helps or if you get stuck on the same error. Also, if there is anything unique about your environment that may be helpful information for our side.

Hi @liswp, I just wanted to follow up on this and see if this was still and issue?
Thank you,
Nate

@liswp I'm closing for now since we haven't heard back but feel free to ping me here if this is still an issue. Thank you!

I am having the same issue as @liswp.
Removing volume and image and running wandb server start once again does not help.


Add new git command to Unipipe CLI

Add support of Git into Unipipe CLI
New command

git  [repo]

Parameters

 required parameter to specify Git operation. Possible values 'pull' or 'push'
[repo] optional parameter to specify Git repository path. The current directory is used if a parameter is empty

Operations

'pull' does a pull with fast forward merge
'push' adds changes into a staged area, makes a commit, and pushes it to a remote repository

Examples

unipipe git pull /some/repo/path
unipipe git push /some/repo/path

I unassigned myself as a reviewer. After @sttomm comments are resolved, we should be good.

just a fyi, collie got quite an extensive subsystem to handle running shell commands with debug/verbose flags and visibility into what happens. it uses that for running terraform/terragrunt etc.

Closing this PR because of an unuseful rebase merge which moved all changes into this branch.
The reason was in a missed force-push flag.


Emote-Picker context menu like in chat

What should be added?
Right now, right clicking on an emote in the emote picker gives the option to copy all emotes in that category. I would like it to work like in chat, where it's possible to copy/open the image link and the bttv/ffz/twitchemotes link.
Why should it be added?
I often want to link somebody an emote from another chat, and this change would make it a lot easier. Generally a nice quality of life change.

duplicates #1125


TCP establishment timeouts

Environmental Info:
K3s Version: v1.18.4+k3s1 (97b7a0e9)

Node(s) CPU architecture, OS, and Version: Debian Buster 64bits Linux k00 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

Cluster Configuration: 1 master, 3 workers

Describe the bug:

When running Gitlab runners on the K3s cluster I experience timeouts. See a full report here: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25380
Steps To Reproduce:

Installed K3s: here

Expected behavior:

The docker image build process is running successfully.
Actual behavior:

The docker image build process crashes due to TCP connection establishment fail or unknown DNS host error.
Additional context / logs:

Step 7/14 : RUN bundle install --retry=4
 ---> Running in f118d3bcb3df
Fetching gem metadata from https://rubygems.org/...........
Fetching public_suffix 4.0.5
Installing public_suffix 4.0.5
Fetching addressable 2.7.0
Retrying download gem from https://rubygems.org/ due to error (2/5): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/addressable-2.7.0.gem)

Upstream kubernetes just released a fix for a long-standing issue that triggered a kernel bug when using vxlan. I suspect this is root cause for your problem. This fix has been backported and is available in the most recent k3s releases of 1.16, 1.17, and 1.18. Can you try updating and see if things work better?

@brandond I just installed the newest version v1.18.6+k3s1 and the issue is not solved yet. After installing the new version a reboot is required or should it just work with restarting the k3s service?
The kernel version I have is: Linux k00 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

Hmm, from the message it's hard to tell if it's a timeout or DNS failure or both. What is your node using for its DNS server?

$ cat /etc/resolv.conf
search infrastructure.svc.cluster.local svc.cluster.local cluster.local
nameserver <IP_ADDRESS>
options ndots:5

Not in the container, on the node.

Sorry,
Direct name server of our colocation provider:
$ cat /etc/resolv.conf
nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>

My job failed at a other position, hope it might help:
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://security.debian.org/debian-security buster/updates/main amd64 Packages [212 kB]
Err:3 http://deb.debian.org/debian buster InRelease
  Connection failed [IP: <IP_ADDRESS> 80]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [7868 B]
Fetched 337 kB in 60s (5575 B/s)
Reading package lists...
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Connection failed [IP: <IP_ADDRESS> 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...

I notices other services having difficulties communicating:
E0720 17:50:13.072142       1 controller.go:131] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="error creating new order: Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: i/o timeout" "key"="blog-32-production/production-auto-deploy-tls-1681138177"

@brandond can I provide you with more information?

It seems that it is package related within the DockerInDocker environment: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25380#note_393553417

I'm too experiencing this problem. Another user reported that switching to Cilium resolved their issue.

I believe this should now be resolved upstream.

FYI, specifically fir Gitlb runners you can solve it like this: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25380#note_456368774


💡 [FEATURE] -

Idea Contribution

[X] I have read all the feature request issues.
[X] I'm interested in working on this issue
[X] I'm part of GSSOC organization

Explain feature request
To Enhance the Documentation reader and to provide more valuable links. To make the words more easy and Catchy so that the Visitor must understand about what the Project is Mainly about and what is the code for.
Explain your solution
I will update the document.
Any alternative approaches/features
No response
Additional Context
No response

which document? README.md CONTRIBUTING.md?

Your Documentation of this Repo. I want to Add feautres in it.

i d like to create a static documentation site using mk docs.


[ENG-448][ENG-447] Fix password-strength-bar double validation message and failure to display message for weak passwords

Purpose
To remove double validation messages on the password-strength-bar and always provide a message when the password does not pass validation.
Summary of Changes

Move logic from the password-strength-bar to the change-password widget.
Refactor logic to determine when to show the password-strength-bar's message.
Add a message to be shown when password strength is weak, but the checker does not provide a warning.
Display password-strength-bar with the warning color (orange) when validation passes, but the password is mediocre.

Side Effects
N/A
Feature Flags
N/A
QA Notes
This should fix the double validation messages on the password-strength-bar.  Please make sure that it works when editing the new password field as well as when editing the existing password field.
This also add a validation message to be displayed when the password is too weak, but the strength checker does not provide a particular reason. Additionally, when the password strength is high enough to pass validation, but the strength checker provides a warning that the password is not of high strength, the message will be shown in orange (instead of red) to match the password strength bar and not make it look like there is a validation error.
Ticket
https://openscience.atlassian.net/browse/ENG-448
https://openscience.atlassian.net/browse/ENG-447
Reviewer Checklist

[ ] meets requirements
[ ] easy to understand
[ ] DRY
[ ] testable and includes test(s)
[ ] changes described in CHANGELOG.md

Pull Request Test Coverage Report for Build 2888

21 of 21 (100.0%)  changed or added relevant lines in 2 files are covered.
4 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.06%) to 71.498%

Files with Coverage Reduction
New Missed Lines
%

app/router.ts
1
86.6%

lib/osf-components/addon/components/osf-navbar/component.ts
3
89.19%

Totals

Change from base Build 2857:
0.06%

Covered Lines:
6020

Relevant Lines:
7561

💛  - Coveralls


Ollie: Watch ikamachinery.com

Ollie requests the watch of the watch_keyword ikamachinery\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

ikamachinery\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Spevacus in Charcoal HQ


Loosen the bounds sdk has on exporter-otlp, so v0.0.2 (containing bugfixes of the last two years) can actually be used

When using the libraries by depending on in hs-opentelemetry-sdk, you'll end up with a very outdated version of hs-opentelemetry-exporter-otlp, as the current bound was still ==0.0.1.*. However version <IP_ADDRESS> contains many important bugfixes like for instance #42 and #89.
This PR loosens this bound.

@iand675 What do you want me to do with the failing format spec? It is complaining about a lot (all?) files, none of which I touched as I obviously only touched the .yaml and .cabal file in ./sdk.
Maybe the version of fourmolu that is loaded inside the Github action was recently bumped?

I also bumped the actual versions set in the package configs themselves as curiously some of them were behind their changelogs.


Platform Collection Search (WIP)

collection_search() Allows users to retrieve list of mission (collections) for the following platforms:

UAVSAR
AIRSAR
SENTINEL-1 INTERFEROGRAM (BETA)

ASFCollection() acts as a wrapper for asf_search.search() using the ASFCollection.mission as the CollectionName

Codecov Report

Merging #79 (c5f3ebc) into master (9974e45) will decrease coverage by 3.42%.
The diff coverage is 37.50%.

@@            Coverage Diff             @@
##           master      #79      +/-   ##
==========================================
- Coverage   78.73%   75.31%   -3.43%
==========================================
  Files          23       26       +3
  Lines         442      482      +40
==========================================
+ Hits          348      363      +15
- Misses         94      119      +25

Flag
Coverage Δ

unittests
75.31% <37.50%> (-3.43%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

asf_search/ASFCollection.py
0.00% <0.00%> (ø)

asf_search/search/collection_search.py
0.00% <0.00%> (ø)

asf_search/CMR/MissionList.py
81.25% <81.25%> (ø)

asf_search/constants/INTERNAL.py
100.00% <100.00%> (ø)

asf_search/exceptions.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9974e45...c5f3ebc. Read the comment docs.


Globs in the bower.json main array are not supported

See http://bower.io/docs/creating-packages/#main

@kenwheeler Can you please merge this to help solve the issue with bower-installer.
Here is Stackoverflow thread with more info:
http://stackoverflow.com/questions/31351375/how-to-change-bower-installer-path-for-one-component

I won't merge this yet, as the resource you've linked from Bower states no Font/Image files as main .. if you can clean up the PR to accomodate this, I will merge :)

@simeydotme On my machine there is no fonts/images - there is only 1 extra file, namely slick/ajax-loader.gif. So, from what I understand main in bower.json should be:
"main": [
    "slick/slick.min.js",
    "slick/slick.css",
    "slick/slick-theme.css",
    "slick/ajax-loader.gif",
    "slick/fonts/slick.eot",
    "slick/fonts/slick.svg",
    "slick/fonts/slick.ttf",
    "slick/fonts/slick.woff"
]

Correct?

from www.bower.io:

Image and font files may be used or referenced within the JS or Sass files, but are not main files as they are not entry-points.

Use source files with module exports and imports over pre-built distribution files.
Do not include minified files.
Do not include assets files like images, fonts, audio, or video.
Filenames should not be versioned (Bad: package.1.1.0.js; Good: package.js).
Globs like js/*.js are not allowed.

Currently the highlighted parts are wrong, there's no point me accepting a PR which doesn't fix the thing completely :D -- if you can fix the PR to cover all those points, I will merge, else give me some time to do it myself.
Si.

For future reference: https://github.com/bower/bower.json-spec/issues/47


App: zone title card shader

We need a title card shader that will introduce gameplay areas. It has both fonts and decorative elements.
It should meet the style of our UI, and be inspired by the Sonic 3 title cards: https://www.youtube.com/watch?v=q9NHdh9t-ow.
There you can see a possible good effect and the timing of the elements coming in and out. Because we will draw this quickly, many times, and in many different contexts, it needs look good on top of many backgrounds. It will also serve as our loading screen.
I spent a few minutes on some shape primitives and animation structure to draw as a full screen quad if you load this scene on a local dev server: https://local.webaverse.com/?src=.%2Fscenes%2Ftitle-card-zone.scn
https://user-images.githubusercontent.com/6926057/141464124-1bc9522a-02d7-4017-be46-3a29c44290eb.mp4
Ideally we would draw this as a pure shader, but because of the requirement to draw fonts it is probably best to use our text components support: https://github.com/webaverse/app/blob/8e4b03e0f62ac6a132bec11ffcdae8369b74c32d/scenes/gunroom.scn#L43
Internally the text rendering uses uses troika-three-text: https://github.com/webaverse/app/blob/8e4b03e0f62ac6a132bec11ffcdae8369b74c32d/metaversefile-api.js#L10
The font to use here is called Plaza and we already have it available: https://github.com/webaverse/app/blob/master/public/fonts/Plaza Regular.ttf

This should draw on top of the scene, so we will need to bring that plane forward to the camera.


MINOR: Eliminate unnecessary Topic(And)Partition allocations in Controller

Eliminated all the unnecessary allocations of TopicPartition and
TopicAndPartition in the Controller. We now use the former
in the Controller (bringing it inline with the rest of the non legacy
code).
Fixed missed Listener -> Handler renames for companion
objects.
More String.format -> String interpolation conversions (the former
is roughly 5 times more expensive).
Some other minor clean-ups.

FAILURE
No test results found.
--none--

@onurkaraman, would you be able to review this?

SUCCESS
8052 tests run, 5 skipped, 0 failed.
--none--

Overall looks good. My only concern is the change to ControllerChannelManager.addStopReplicaRequestForBrokers and StopReplicaRequestInfo in which the PR no longer checks for a null callback and now manually fills in no-ops as needed.
If we're not careful in later patches, this could lead to an NPE. So it seems a bit brittle.
Some other options:

leave it as it was before the PR.
change CallbackBuilder.stopReplicaCallback to default to a no-op instead of null and use the Callbacks.stopReplicaResponseCallback method everywhere.
overload ControllerChannelManager.addStopReplicaRequestForBrokers with another method that excludes the callback param and internally provides a no-op to the other method variant.

SUCCESS
8052 tests run, 5 skipped, 0 failed.
--none--

FAILURE
No test results found.
--none--

@onurkaraman, thanks for the review.
Good point about CallbacksBuilder, I removed the class altogether since it served little purpose.
With regards to future NPEs, that is the case for pretty much all of our Scala code. nulls should not be used in Scala, but unfortunately one can use them anywhere due to the JVM. Reviews and tests should catch them generally and NPEs are reasonably rare in our Scala code as far as I can see.

One more thing, I did notice that we are allowing nulls in the new ZooKeeper code in some cases still, it would be good to fix that.

LGTM.
There are some minor inconsistencies with how we do callbacks in the controller today but these difference can be ironed out in a separate PR.

SUCCESS
8053 tests run, 5 skipped, 0 failed.
--none--

SUCCESS
8053 tests run, 5 skipped, 0 failed.
--none--

Merging to trunk.


Added opLRN documentation corrected few testcases

changes for global pool tests and comments in LRN (Local Response Normalization) tests.


upload to codecov hanging in github actions

This job is taking a long time to uplodad.
I noticed in my .github/workflows/tests.yaml that we are still using v1 .. maybe this is related?
- name: "Upload coverage to Codecov"
      uses: codecov/codecov-action@v1
      with:
        fail_ci_if_error: false

I saw this message as well:
Please upload with the Codecov repository upload token to resolve issue
@jcapriot I tried following the instructions here, but it seems that maybe I don't have permission to access the token via simpeg codecov account -- is that possible?

Yeah, codecov no longer supports tokenless upload. I’ve just added the CODECOV_TOKEN as a secret variable for this repo. You should probably update to the new version of the uploader action (v4).
You’ll need to pass this token to the action as well:
- name: Upload coverage reports to Codecov
    uses: codecov/codecov-action@v4
    with:
      token: ${{ secrets.CODECOV_TOKEN }}

Thanks @jcapriot -- that's got it fixed!


How to get image from URL??

roughly http://stackoverflow.com/questions/17876015/image-from-url-to-stream plus some coding

You can use the URIAdaptor.

have any examples to download image direct to control via url ?
Thanks

i'm create new uri adapter with link and then getthumbnail form uriadapter then add new item in listview but i got error in this line
LsiImgOnline.Items.add(new ImagelistviewItem, "image", pbxadptrImage.image,_uriadapter)
the error description is
Unable to cast object of type 'Manina.Windows.Forms.ImageListViewItem' to type 'System.String'.
at the same time below code is working.. without error 👍
LsiImgOnline.Items.add(new ImagelistviewItem, pbxadptrImage.image,_uriadapter)


Change basic example to contain backward two losses.

There is a warning when running doc tests:
=============================== warnings summary ===============================
tests/doc/test_rst.py::test_basic_usage
  /home/runner/work/torchjd/torchjd/.venv/lib/python3.10/site-packages/torch/nn/modules/loss.py:535: UserWarning: Using a target size (torch.Size([16, 1])) that is different to the input size (torch.Size([16])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
    return F.mse_loss(input, target, reduction=self.reduction)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================== 18 passed, 1 warning in 2.86s =========================

I think this can be fixed very easily by using targets of shape (16,) instead of (16, 1)

Looking good! A few minor fixes to do, then we can merge.


chore: add missed telemetry

Codecov Report

Merging #1406 (00af5f5) into master (70aaf4d) will decrease coverage by 1.84%.
The diff coverage is n/a.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1406      +/-   ##
==========================================
- Coverage   29.44%   27.59%   -1.85%
==========================================
  Files         128       93      -35
  Lines       10516     9972     -544
  Branches       99        0      -99
==========================================
- Hits         3096     2752     -344
+ Misses       7240     7040     -200
  Partials      180      180

Flag
Coverage Δ

be_unit_test
27.59% <ø> (ø)

e2e_test
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70aaf4d...00af5f5. Read the comment docs.

wait for other PRs merged, then upgrade packages version


Custom sidecar mako templates

Is your feature request related to a problem? Please describe.
I have a large variety of semi-structured data that I currently embed in the Photos caption field as a catch-all. Some of these are tag-like and could be keywords. However, the bulk is either hierarchical or key-value in nature. I'd like to be able to parse and clean this data so that I can map it to more appropriate structure in the XMP export.
Describe the solution you'd like
I've been thinking about this for a while and custom sidecar templates via mako (like the existing XMP ones) seem like a reasonable approach. This would definitely be an advanced feature and possible footgun. However, I suspect it would be relatively simply to implement, aligns with existing templating functionality, and provides all the flexibility one could need. I could even imagine a repository user-contributed templates to handle specific quirks of all the various photo apps out there.
Describe alternatives you've considered
I've considered re-writing the exported XMP/JSON sidecar files post-export. However, I worry about that not playing nice with --update.
There's also properties like the photo UUID which aren't available. I currently work-around this by including the UUID in the file name. However, I'd like to avoid this because it makes the names annoyingly long.

I like this idea.

There's also properties like the photo UUID which aren't available.

I don't understand this -- not available from where?
Assuming you don't also want also the current XMP sidecar, you could rig this now by editing the file xmp_sidecar_beta.mako in your site-packages/osxphotos/templates/ directory of wherever osxphotos is installed (~/.local/pipx if using pipx) and then passing the --beta flag which causes osxphotos to use this template instead of the normal one. This is how I test changes to the XMP file. Of course, the file will be named .xmp and it may not be a valid XMP but it would let you play around with the template.
I'll add this to the queue -- should be fairly straightforward to implement.  Will need to add the files generated to the update list as you noted and provide a template for the filename so you can customize the output sidecar name.

I like this idea.

Awesome 😄

There's also properties like the photo UUID which aren't available.

I don't understand this -- not available from where?

I meant there's no way to get the UUID of the photo post-export unless you explicitly include it in the filename template. Somehow I had overlooked --keyword-template and --description-template so that's a misunderstanding on my part.

I'm close to having this implemented. Here's the interface so far.  Does this meet your use case? Basically, you can provide one or more mako templates, a custom osxphotos template which will be used to render the filename of the sidecar, and flags to either strip excess whitespace and/or excess new lines (which happens with Mako sometimes).
--sidecar-template MAKO_TEMPLATE_FILE SIDECAR_FILENAME_TEMPLATE STRIP_WHITESPACE STRIP_LINES
                                Create a custom sidecar file for each photo exported
                                with user provided Mako template (MAKO_TEMPLATE_FILE). MAKO_TEMPLATE_FILE
                                must be a valid Mako template (see https://www.makotemplates.org/). The
                                template will passed the following variables: photo (PhotoInfo object for
                                the photo being exported), sidecar_path (pathlib.Path
                                object for the path to the sidecar being written), and photo_path
                                (pathlib.Path object for the path to the exported photo.
                                SIDECAR_TEMPLATE_FILENAME must be a valid template string (see Templating
                                System below) which will be rendered to generate the filename of the sidecar
                                file. The `{filepath}` template variable may be used in the
                                SIDECAR_TEMPLATE_FILENAME to refer to the filename of the photo being
                                exported. STRIP_WHITESPACE and STRIP_LINES are boolean values (true/false,
                                yes/no, 1/0 are all valid values) and indicate whether or not to strip
                                whitespace and blank lines from the resulting sidecar file. For example, to
                                create a sidecar file with extension .xmp using a template file named
                                'sidecar.mako' and strip blank lines but not whitespace: `--sidecar-template
                                sidecar.mako '{filepath}.xmp' no yes`. To do the same but to drop the photo
                                extension from the sidecar filename: `--sidecar-template sidecar.mako
                                '{filepath.parent}/{filepath.stem}.xmp' no yes --sidecar-drop-ext`. For an
                                example Mako file see https://raw.githubusercontent.com/RhetTbull/osxphotos/
                                main/examples/custom_sidecar.mako

I believe this is now working. I also updated report writer to indicate which files are user generated sidecars in the report. Works with --update and --cleanup. I need to add tests then will get this pushed out soon.

Looks awesome! I'll be porting my hacks that were patching the beta template file to use this instead.
Happened to notice you have a typo in the docs, you reference both SIDECAR_FILENAME_TEMPLATE and SIDECAR_TEMPLATE_FILENAME.

I also left a comment about potentially caching the compiled template.

This is implemented in v0.61.0. I added caching of the template as you suggested and fixed the typos. Please let me know what you think and if this meets your use cases. The Mako sidecar is passed three variables:

photo: a PhotoInfo object for the photo being exported
photo_path: a pathlib.Path object for the photo file being exported
sidecar_path: a pathlib.Path object for the sidecar file being written

There are three example Mako sidecars in the examples directory:

custom_sidecar.mako: creates a simple text file with a few details.
custom_sidecar_xmp.mako: creates an XMP sidecar
custom_sidecar_json.mako: creates a JSON sidecar with the full JSON dump of the PhotoInfo object for the photo (everything osxphotos knows about the. photo).

Because the PhotoInfo object is available to the sidecar template, this last example is as simple as:
${photo.json(shallow=False, indent=4)}

This is definitely a niche option but as you suggested, it might be very useful for creating and sharing sidecars for specific apps.

Please let me know what you think and if this meets your use cases.

This is great and covers everything I need so far.

Because the PhotoInfo object is available to the sidecar template, this last example is as simple as:
${photo.json(shallow=False, indent=4)}

The trivial JSON output that I have full control over is super nice. In my case I'm joining photos to some external CSV files that I can lookup records via keys (mostly URLs) that I've added as part of description field.
This makes for a great plugin system since you get the full power of python for every photo being processed. I bet you could even do some crazy stuff like convert videos to GIF and emit that.
The other thing I'm wondering, should there be a way to conditionally write the sidecar, even for files that are not skipped. E.g. a flag that says, if your template renders to an empty buffer, skip writing the empty file. That would have to be another boolean option to --sidecar-template. At that point it might be better to collapse WRITE_SKIPPED STRIP_WHITESPACE STRIP_LINES into a final option of comma separated string flags, e.g. --sidecar-template my-template.mako '{filename}.ext' write_skipped,skip_lines. That would make it easier to add new flags and more obvious at the command line what's happening than the trailing true false true. Food for thought

I like the idea of comma separated flags. I'd thought about doing a bit mask but figured that might not be intuitive for some users. Then option would still need a fixed number of arguments so if no flags used there would still need to be something like none for the flags.

Click, which osxphotos uses for option parsing, does not support variable number of arguments to an option. Wondering which of these is better:
--sidecar-template my-template.mako '{filename}.ext' write_skipped,strip_lines
--sidecar-template my-template.mako '{filename}.ext' none
or
--sidecar-template my-template.mako '{filename}.ext' --sidecar-options write_skipped,strip_lines
--sidecar-template my-template.mako '{filename}.ext'
As much as I hate adding yet another option, am leaning towards moving these to --sidecar-options which makes simple use case easier and allows options to be changed in the future without changes to the basic --sidecar-template option.

I'd slightly lean to the latter option as well for simplicity. The one minor downside to that approach is that if you use multiple custom sidecars in a single option you get the same custom set. Unless there's a way to say each --sidecar-options ... applies only to the previous --sidecar-template ....

Unfortunately you'd be stuck with a single set of options and click doesn't provide a way to group options. That's a good point that keeping the options with the sidecar is important (and why I originally included those with --sidecar-template

In that case I think an explicit none is still better than a series of extra bools.

Any error created by the template should be logged. Add user_sidecar_error field and catch errors to log them.  There could be an option added for catch_error or raise_error to define behavior -- catch/log the error and keep writing sidecars or abort the export.

Completed in v0.62.0. --sidecar-template now accepts one or more named options


update matrix-discord-parser

it received a few bugfixes + bridges custom matrix emoji kinda well to discord now

Woop :tada:


Add support for k8s 1.9

k8s versions 1.7, 1.8 and 1.9 are now supported.

This change is

So with the same 1.9 api all earlier are also supported?
That looks like great simplification!

Reviewed 25 of 29 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Yes, 1.9 CRI API is backward compatible with 1.7/1.8

Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

Comments from Reviewable


Policy: confirm identifier schema for SDR & TACO

Can the policy team confirm this is the identifier schema wanted? https://github.com/sul-dlss-labs/taco/wiki/Identifier-schema
(Note, if so, we will be required to have internal identifiers that vary within TACO from what we present out in the world given versioning considerations & filename presence / changes over time issues).

Confirmed for SDR schema from my point of view.

Confirmed for me too. I like it not only is it logical, but because it also is consistent with the convention often used for Source IDs. A nice parallel.

We added notes to the above link based on the outcomes of our discussion

Thanks! Closing.


response.apparent encoding throws error after using iter_lines

When using iter_lines with stream=True, and later trying to get the response.apparent_encoding, I get an exception as follows:
RuntimeError: The content for this response was already consumed
So my question is: Is this the proper behavior? I mean, shouldn't I be able to get the apparent_encoding even after consuming the content using stream=True?
I was thinking on running the chardet on the buffer result of consuming the iteration but I'm thinking that this beats the purpose since the result is already encoded.
Am I missing something?
Thanks.

Thanks for opening this issue!

Is this the proper behavior?

Yes. :smile:

I mean, shouldn't I be able to get the apparent_encoding even after consuming the content using stream=True?

No. :smile: When you set stream=True, you are telling Requests that you do not want us to buffer the content for any reason. That means we do not store it, which means we cannot hand it off to chardet to process. apparent_encoding cannot be used in conjunction with stream=True. If you are going to consume all the content into a single buffer, you can just safely use stream=False.

Thanks for your answer. Thought would be that.


How to use getHalfedges()?

Is find this kind of vertex indexing very confusing. How do I use getHalfedges() to ultimately draw each edge (once) from the triangulation?
Furthermore, how do I find the 3 half edges that make up each triangle?
And is there an easy way to get the triangle that shares a half-edge with a given triangle (or given a half-edge, get the (up-to) two triangles that share it) via this indexing scheme?

@micycle1 sorry for the very late reply.. Please check the code here it has what you're looking for. This port follows the original delaunator, therefore these methods are not included.


adds checks for the new radio buttons and amends commenting

we have 2 new radio buttons in the cops modal (pdf and distribution preview). added tests for these and amended some commentary in the code.

The failed test is nothing I touched, not related to my changes.

@omehes - Just FYI, that failure is something we're aware of and looking into (see cnx 1168).

@rnathuji thanks for letting me know


How to playback paused recording?

Version of react-native-audio-recorder-player
3.2.0
Version of React Native
0.63.0
Platforms you faced the error (IOS or Android or both?)
Android
Expected behavior
User can call startRecorder, then pauseRecorder, then startPlayer with the same file path to be able to hear the recording thusfar before continuing to record
Actual behavior
startPlayer throws an error when trying to do this
Steps to reproduce the behavior
See expected behavior

This is not possible in the current version "^3.2.0"

It's someone working on this issue?


Duva Connector Support Issues

Environment (local, stage, preview, production)
ona.io
Describe the bug
There are four issues raised by Caritas with the Duva Connector:
Issue No. 1 - Autosync not working
Currently, the sync does not work. When you make a submission on Ona, it should reflect on Tableau but this is not the case. Even the manual refresh doesn't seem to work.
Issue No. 2 - Advanced Export Options not all displaying when Beta is Enabled
If the new data view is enabled, some of the choice options seem to be missing. Please see below screenshot on the same:

Issue No. 3 - Tableau token expires or is removed after some time
Tableau token is removed after a certain time. Even after creating and recreating it, the new connection is not shown after refreshing the page:

This happens even though the created Tableau Prep (hyper) connection is shown when clearing the cookies of https://duva.ona.io.
Issue No. 4 - The name of Tableau connection seems to be missing after creation
Most of the time, after the connection is created, its name doesn't appear.

To be addressed here


Support setupProxy.mjs, cjs, ts, tsx, etc

Uses @babel/register and moduleFileExtensions so that you get just as much love and support in src/setupProxy as you would in, say, src/index.
Fixes #8046 and #6794
Try it out by cloning this branch and then running this:
yarn run create-react-app --template file:packages/cra-template-typescript ../pr-10228
cd ../pr-10228
echo "export default function(a) {throw new Error('It worked!')}" > src/setupProxy.mjs; yarn start
rm src/setupProxy.mjs; echo "module.exports=a=>{throw new Error('It worked!')}" > src/setupProxy.cjs; yarn start
rm src/setupProxy.cjs; echo "export default function(a: any) {throw new Error('It worked!')}" > src/setupProxy.ts; yarn start

Notice how all three times that yarn start are run, you get an error saying "It worked" - this shows that cra paid attention to all the files, parsed their syntax correctly, and ran them (if they weren't run, you wouldn't see that error, and if they weren't parsed properly, you'd get a syntax error).
Additionally, you can drop the --template file:packages/cra-template-typescript bit in line 1, and lines 4 and 5 (cjs and mjs) will work.

Now that ES is stable in Node 14, one cannot use setupProxy and "type": "module" together without this patch. Anything I can do to help unblock this PR?

I would very much like this to be merged. I'm trying to do typescript and it's bothering me that I can't do setupProxy.ts

I also would like this being merged. Thank you very much for your time if you end up reviewing this :)


Get genesis epoch committee assignment

In current validator doc, there's the responsibility look ahead which is implemented by get_next_epoch_committee_assignment. Similarly we should have   get_gensis_epoch_committee_assignment for genesis epoch assignment. So when validators try to get their genesis assignment they don't end up getting slots" genesis_epoch + 64 .. genesis_epoch + 127 😛
Let me know if this is already specified somewhere and I'd be happy to open a PR

Ah, you're right. They would need to use get_crosslink_committees_at_slot directly for hte current epoch. I'll either mod the helper or add something new to make it explicit how to do so


APT whitelist request for qemu-kvm

Would travis-ci/apt-package-whitelist/issues/483 suffice for this?

Yes, thanks for reference.


[BUG]: Sometimes can't get subscription details after successful approve

Is there an existing issue for this?

[X] I have searched the existing issues.

🐞 Describe the Bug
Sometimes we can't get subscription details after creating it, and we see that paypal is creating subscription and charge. Event if we try request data with some sleep function after 3 seconds. We can't catch this issue, and wrote some logging code inside try catch block with logging JSON.stringify(e) and it's just empty {}
😕 Current Behavior
When our client is trying to subscribe on some plan and approves payment on checkout page, we trying to get subscription details:
const onApprove = (data: OnApproveData, actions: OnApproveActions) => {
  try {
    const details = await actions?.subscription?.get();
  } catch (e) {
    console.log(JSON.stringify(e));
  }
};

<PayPalScriptProvider
  options={{
    'client-id': 'CLIENT-ID',
    'disable-funding': 'credit,card',
    vault: true,
    currency: 'USD',
  }}
>
  <PayPalButton
    onError={() => { /* handle error code */  }}
    onApprove={onApprove}
    createSubscription={() => { /* handle creating subscription code*/  }}
  />
</PayPalScriptProvider>

🤔 Expected Behavior
When the client approves the payment, we should be able to get the subscription details
🔬 Minimal Reproduction
Somehow we can't reproduce it, usually it happens randomly

🌍 Environment
| Software         | Version(s) |
| ---------------- | ---------- |
| react-paypal-js  |   7.3.3    |
| Browser          |    any     |
| Operating System |    any     |

Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

➕ Anything else?
No response

@devchristina please, take a look to this issue

Sent this to the product team to research.

@devchristina thanks, will be waiting update

Hi, we are all in the same situation @devchristina.
I need to take a decision on it at the end of the day (if change PayPal to another payment gateway or not).
Please, we are all waiting for an update.

@devchristina Hi, are there any updates?

still affecting

We had to change to Stripe the whole checkout process to avoid this issue.
PayPal 👎

@devchristina hi, any updates?

still affecting


DragOverlay dropAnimation only animates / applies certain css properties

I'm trying to make a DragOverlay item animate seamlessly back into place once dropped.
I'm running into trouble with the DropAnimation, it seems that only certain css properties will animate - eg. background and border
While other properties will not animate - eg. boxShadow, borderRadius
CodeSandbox example

You can specify custom keyframes to animate those properties, or define other sideEffects you would like to perform:
https://github.com/clauderic/dnd-kit/blob/5c58f0fe5d19b5aaa5cf93572f3435f4a0a6e54f/stories/3 - Examples/Tree/SortableTree.tsx#L75-L88