Catch HTTP 409 error for Code Engine

Recently, I have been seeing frequent HTTP 409 errors when processing data in bulk. I spoke with the IBM customer support to clarify where the source of the problem is and how it can be fixed. So far, all I've gotten from them is this response:
This looks like that you have some application code which is interacting with a Code Engine project using the Kubernetes API endpoint.
There is a quite famous bug in Kubernetes that concurrent create operations can lead to 409 errors of this kind.
You will have to retry them in your application code. A retry would look like this in pseudo code:

const MAX_RETRY = 5

  main() {
    myObject = { ... }
    createWithRetry(myObject)
  }

  createWithRetry(object) {
    for (i = 0; i < MAX_RETRY; i ++) {
       sleepSeconds(i)
       try {
          return kubeClient.create(object)
       } catch (e) {
          if (!isRetryable(e) || i == MAX_RETRY - 1) {
            logError(e)
            throw e
          }
          logWarning("Retrying due to", e)
       }
    }
  }

  isRetryable(exception) {
    if (exception is ConflictError) {
      return true
    }

    // It makes sense to retry more like network failures

    return false
  }

The original stacktrace I get:
Traceback (most recent call last):
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/lithops.py", line 81, in _callback
    self._manager.annotate_lithops(ds=ds, del_first=msg.get('del_first', False))
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/dataset_manager.py", line 114, in annotate_lithops
    ServerAnnotationJob(executor, ds, perf).run()
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/annotation_job.py", line 321, in run
    self.db_formula_image_ids = self.pipe.store_images_to_s3(self.ds.id)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/pipeline.py", line 199, in store_images_to_s3
    return store_images_to_s3(
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/store_images.py", line 67, in store_images_to_s3
    results = executor.map(_upload_png_batch, [(cobj,) for cobj in png_cobjs], runtime_memory=512)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 292, in map
    raise exc
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 325, in run
    futures = executor.map(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 288, in map
    futures = self.invoker.run_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 266, in run_job
    futures = self._run_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 205, in _run_job
    raise e
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'audit-id': 'a082f7b4-70d3-4708-a29e-8ec22f965220', 'cache-control': 'no-cache, private', 'content-length': '326', 'content-type': 'application/json', 'date': 'Sat, 03 Sep 2022 03:19:13 GMT', 'x-kubernetes-pf-flowschema-uid': '05dc5f92-6c95-416b-b707-da00a57e0adc', 'x-kubernetes-pf-prioritylevel-uid': '8d40cc3d-a5c4-47ee-8978-ec5f7a197920'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on resourcequotas \"5s4f5qcqf4f\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"5s4f5qcqf4f","kind":"resourcequotas"},"code":409}

Is it possible to add handling of such exceptions to lithops? I use lithops 2.7.0.

@kpavel can you handle it? :)

As far as I understand, the code responsible for the retry goes here and I need to explicitly insert the connection_retries value into the code engine config.

Hello @sergii-mamedov, is it a full stacktrace? No other traces from code_engine level?
regarding connection_retries  - yes, you need to specify in the config file retries number, e.g. 10.
the issue here is that we are currently handling connection retries when 409 exception happens with the reason "AlreadyExists", while the one you get has a reason "Conflict".
We can add another reason in the filtering to let it retry.

@kpavel
Traceback (most recent call last):
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 325, in run
    futures = executor.map(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 277, in map
    futures = self.invoker.run_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 269, in run_job
    futures = self._run_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 208, in _run_job
    raise e
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 205, in _run_job
    self._invoke_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 252, in _invoke_job
    activation_id = self.compute_handler.invoke(payload)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 406, in invoke
    self._run_job(jobrun_res)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 48, in decorated_func
    return func(*args, **kwargs)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 414, in _run_job
    self.custom_api.create_namespaced_custom_object(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 225, in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 344, in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/rest.py", line 274, in POST
    return self.request("POST", url,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/rest.py", line 233, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'audit-id': 'a082f7b4-70d3-4708-a29e-8ec22f965220', 'cache-control': 'no-cache, private', 'content-length': '326', 'content-type': 'application/json', 'date': 'Sat, 03 Sep 2022 03:19:13 GMT', 'x-kubernetes-pf-flowschema-uid': '05dc5f92-6c95-416b-b707-da00a57e0adc', 'x-kubernetes-pf-prioritylevel-uid': '8d40cc3d-a5c4-47ee-8978-ec5f7a197920'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on resourcequotas \"5s4f5qcqf4f\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"5s4f5qcqf4f","kind":"resourcequotas"},"code":409}

@sergii-mamedov Does #995 solve your issue?

@JosepSampe
Sorry for a long answer. I deployed this changes on production. I will get back to you in a couple of days with an answer.

@kpavel @JosepSampe
Unfortunately, the patch didn't change the situation (which is a bit strange for me). I added some logging, stacktrace below:
2022-10-13 18:36:00,920 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:172 - ExecutorID 77ece4-14 | JobID M005 - Starting function invocation: process_centr_segment() - Total: 277 activations
2022-10-13 18:36:01,360 - INFO - lithops.serverless.backends.code_engine.code_engine[Thread-1-ex] - code_engine.py:54 - connection_retries=10
2022-10-13 18:36:01,361 - INFO - lithops.serverless.backends.code_engine.code_engine[Thread-1-ex] - code_engine.py:58 - HTTP409
2022-10-13 18:36:11,376 - ERROR - engine.lithops-wrapper[Thread-1] - executor.py:286 - process_centr_segment raised an exception. Failed activation(s): [] ID(s): []
Traceback (most recent call last):
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 325, in run
    futures = executor.map(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 288, in map
    futures = self.invoker.run_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 266, in run_job
    futures = self._run_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 205, in _run_job
    raise e
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 202, in _run_job
    self._invoke_job(job)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/invokers.py", line 249, in _invoke_job
    activation_id = self.compute_handler.invoke(payload)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 396, in invoke
    self._run_job(jobrun_res)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 67, in decorated_func
    raise e
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 52, in decorated_func
    return func(*args, **kwargs)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 404, in _run_job
    self.custom_api.create_namespaced_custom_object(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 225, in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py", line 344, in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/rest.py", line 274, in POST
    return self.request("POST", url,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/kubernetes/client/rest.py", line 233, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'audit-id': 'd4735826-f7ab-4244-91ec-43b59d77b169', 'cache-control': 'no-cache, private', 'content-length': '326', 'content-type': 'application/json', 'date': 'Thu, 13 Oct 2022 16:36:01 GMT', 'x-kubernetes-pf-flowschema-uid': '05dc5f92-6c95-4$
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on resourcequotas \"5s4f5qcqf4f\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","det$

@kpavel
Did it, I will let you know when this problem occurs again.

@kpavel
I understood what the problem was. There is no condition in the code to handle the block for HTTP 409. You should either add `continue' to this block
if e.status == 409:
    some_logic
    continue
if e.status == 500:
    some_logic
else:
    raise e

or perhaps it is better to make a solid if - elif - else block:
if e.status == 409:
    some_logic
elif e.status == 500:
    some_logic
else:
    raise e

I tested first option - works well.

@JosepSampe
Thanks, I will be grateful for the release of a new version when possible.

@sergii-mamedov @JosepSampe I will draft new relase

@sergii-mamedov We released 2.7.1

thanks


feat(#major); stargate; add stargate subgraph

Multichain Bridge Subgraph

networks: mainnet, fantom, avalanche, bsc, arbitrum, optimism, matic
schema: bridge

Methodology

TODO

deployments, status

mainnet:

https://thegraph.com/hosted-service/subgraph/dhruv-chauhan/stargate-mainnet-2
https://okgraph.xyz/?q=dhruv-chauhan%2Fstargate-mainnet-2
https://subgraphs.messari.io/subgraph?endpoint=dhruv-chauhan/stargate-mainnet-2&tab=protocol

fantom:

https://thegraph.com/hosted-service/subgraph/dhruv-chauhan/stargate-fantom
https://okgraph.xyz/?q=dhruv-chauhan%2Fstargate-fantom
https://subgraphs.messari.io/subgraph?endpoint=dhruv-chauhan/stargate-fantom&tab=protocol

Hey @dhruv-chauhan I added this recently, a new subgraph template: https://github.com/messari/subgraphs/blob/master/.github/PULL_REQUEST_TEMPLATE/new_subgraph_template.md
It gets kinda lost when you make the PR, but feel free to copy pasta the new subgraph checklist!

@steegecs this is good to go from the QA side. Can you deploy this on the messari hosted when this PR gets merged into master?


2107 allow private ssh when building

Issue #, if available:
Description of changes:
Added the following configurable variables to
1. allow private ssh to private subnet:

ssh_username
temporary_security_group_source_cidrs
ssh_interface
subnet_id

2. allow assignment of public ip address when default is disabled

associate_public_ip_address

3. customize ami description

ami_description

Also allowed ./install-worker + ./files source paths to be relative to template dir.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm


Multiple coordinates

Add a database (plus associated api+wiki) solution for multiple coordinates of an object.
At its easiest this could be a coord_link_table with "main_id, lat, lon" to be added only if object already has a "lat/lon".
Wiki-wise simply add support for lat2/lon2.

This would complicate the simply synking routines


retry Attest provenance step

Requests to download prime registry manifests appear to fail intermittently (see https://github.com/rancher/ingress-nginx/actions/runs/12260568096/job/34224296299).
This PR adds a retry capability.

@brooksn good job! Overall this should be safe, apart from the cases whereby the image was not correctly pushed to the Prime registry - on those cases the errors are not intermittent.


Introduce staged timeout system

All timeouts are now done on a stage by stage basis, failing the build with a helpful message if the timeout is reached:
TOP LEVEL PIPELINES*

Adopt api requests = 1 hour
Copy artifacts = 6 hours
Archive artifacts = 6 hours
Publish binary = 3 hours

DOWNSTREAM JOBS

Overall (test, build, everything) = 18 hours
Adopt api requests = 1 hour
Clean workspace = 1 hour
Checkout scm = 1 hour
Pull docker image = 2 hours
Main build script (make-adopt-build-farm.sh) = 6 hours
Archive artifact = 3 hours
Sign build job = 2 hours
Test jobs = 16 hours
Installer jobs = 2 hours

Closes: #2120
Signed-off-by: Morgan Davies<EMAIL_ADDRESS>

run tests

Latest tester failed due to ongoing aarch64 build issues. Otherwise, all builds passed without timing out or failing the gradle download step. I'll param the error messages, squash and edit the commits and move this into review ready

run tests

lgtm, needs a good test though

Agreed although I'm wondering what the best way to test it is. In the meantime (seeing as people seem happy with the timeout values), I'll put this back into draft and figure out a way to test it

@andrew-m-leonard @karianna Would we prefer to have the timeouts defined on a platform to version basis instead of a general timeout for every platform and version? I'm thinking that we could add in a Map attribute into each platform's build_configuration_map that contains the timeouts for that specific platform. That way we could have longer timeouts for slower systems like AIX and ARM and shorter ones for others?

run tests

Tester failed due to https://github.com/AdoptOpenJDK/openjdk-build/pull/2212. This is blocked until that PR is merged

Conflicts as well

Conflicts as well

I'm waiting till 2212 is merged before resolving them since it means I only have to do it once 🙂

run tests

@andrew-m-leonard @karianna Would we prefer to have the timeouts defined on a platform to version basis instead of a general timeout for every platform and version? I'm thinking that we could add in a Map attribute into each platform's build_configuration_map that contains the timeouts for that specific platform. That way we could have longer timeouts for slower systems like AIX and ARM and shorter ones for others?

I'm wary of over complicating things here, and also for example a particular AIX machine maybe slow now, but a simple H/W upgrade could easily change that. I would stick with a "course" set of timeouts that roughly covers the given stage.

Thanks Andrew. In that case, I'll leave as it is for now and let others add in platform specific timeouts in the future should they prove necessary. This is ready for review again (and merge, pending PR tester result)

run tests

run tests

run tests

run tests

🟠 PR TESTER RESULT 🟠
❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

Windows failed due to https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1573
aarch64/JDK16/j9 failed due to unknown error. Rebuild passed https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk/job/jdk-linux-aarch64-openj9/384/
13:02:34  CCACHE_COMPRESS=1          CCACHE_SLOPPINESS=pch_defines,time_macros CCACHE_BASEDIR=/home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src /usr/local/gcc/bin/ccache  /usr/local/gcc/bin/gcc-7.5 -O -fgnu89-inline -g -DLINUX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -fpic -DIPv6_FUNCTION_SUPPORT -DJ9AARCH64 -fstack-protector -Wimplicit -Wreturn-type -Werror -I. -I../include -I../oti -I../gc_include -I../omr/gc/include -I../gc_glue_java -I../nls -I../omr/include_core    -DTR_HOST_ARM64   -E vmcheck.c | sed -n -e '/^DDRFILE_BEGIN /,/^DDRFILE_END /s/^/@/' -e<EMAIL_ADDRESS>> vmcheck.i
13:02:35  gmake[6]: Leaving directory '/home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src/build/linux-aarch64-server-release/vm/vmchk'
13:02:35  gmake[5]: Leaving directory '/home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src/build/linux-aarch64-server-release/vm'
13:02:35  [2020-11-09 13:02:34] Scraping anotations from preprocessed code ...
13:02:41  [2020-11-09 13:02:40] Restoring annotated files ...
13:02:41  [2020-11-09 13:02:41] All done.
13:02:41  Running ddrgen to generate j9ddr.dat and superset.dat
13:03:54  Blob written to file: ../j9ddr.dat
13:03:54  Superset written to file: ../superset.dat
13:03:54  gmake[4]: Leaving directory '/home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src/build/linux-aarch64-server-release/vm/ddr'
13:03:54
13:03:54  ERROR: Build failed for targets 'product-images legacy-jre-image test-image debug-image' in configuration 'linux-aarch64-server-release' (exit code 2)
13:03:54  Stopping sjavac server
13:03:54
13:03:54  === Output from failing command(s) repeated here ===
13:03:54  * For target jdk_modules_jdk.jshell__the.jdk.jshell_batch:
13:03:54  * For target jdk_modules_jdk.security.auth__the.jdk.security.auth_batch:
13:03:54
13:03:54  * All command lines available in /home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src/build/linux-aarch64-server-release/make-support/failure-logs.
13:03:54  === End of repeated output ===
13:03:54
13:03:54  === Make failed targets repeated here ===
13:03:54  CompileJavaModules.gmk:609: recipe for target '/home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src/build/linux-aarch64-server-release/jdk/modules/jdk.security.auth/_the.jdk.security.auth_batch' failed
13:03:54  make/Main.gmk:197: recipe for target 'jdk.security.auth-java' failed
13:03:54  CompileJavaModules.gmk:609: recipe for target '/home/ubuntu/workspace/build-scripts-pr-tester/build-test/jobs/jdk/jdk-linux-aarch64-openj9/workspace/build/src/build/linux-aarch64-server-release/jdk/modules/jdk.jshell/_the.jdk.jshell_batch' failed
13:03:54  make/Main.gmk:197: recipe for target 'jdk.jshell-java' failed
13:03:54  === End of repeated output ===

@karianna Assuming @smlambert is happy with these changes, this should be good to merge


Files placed in public - Documents directory in web server does not show in browser

I exported the project using yarn export and published them on my web server. I placed some files in Documents directory under users public. Nothing shows up when I navigate in file explorer in web browser.

It requires running the build to update the JSON file.

That works. Thank you Dustin. Great project. I love daedalOS
Another question. How do I change the default wallpaper with my own wallpaper jpg.

Glad it worked!
To store the wallpaper setting you would also need to update the default session.json to have the wallpaper settings you want. Ex:
{
  "wallpaperFit": "fill",
  "wallpaperImage": "/Users/Public/Pictures/Dustin/My Eye.jpg",
}

This works. Thank you Sir!


Added docker-cloud.yml to supported files

I've added docker-cloud.yml to supported files with docker icon.

@patrikulus Could you please rebase the branch to resolve conflicts? If it's too hard we can do it for you, just let us know.

Already implemented in #602.
Credits will be present in the Changelog


Change defaults for requester to be non-daemon and have a shorter timeout

Pull Request Etiquette

[x] I have checked the PRs for upcoming features/bug fixes.
[x] I have read the contributing guidelines.

Changes

[x] Internal code
[ ] Library interface (affecting end-user code)
[ ] Documentation
[ ] Other: _____

Closes Issue: NaN
Description
Now the rate limit pool will keep doing requests until all requests are exhausted. I also decided to add an info log about this behavior so people know why the process doesn't stop immediately when calling JDA#shutdown.

Just tested this branch, it fixed #1212 for me.
Got the note [main] INFO RateLimiter - Waiting for 1 bucket(s) to finish. Average queue size of 16 requests and the program immediately terminated after the last message was sent.
Great job!


Radium breaks getDefaultProps on IE

I've tested this on IE9/10 (internet is too slow to download an 11 VM right now) and created some examples on my fork that demonstrate that it is Radium that is responsible.
https://github.com/bobbyrenwick/radium/tree/test-out-ie
I can't see anything that makes it immediately obvious why this is happening

This doesn't repro in IE11, the only one I have access to currently:
var Example = React.createClass({
  getDefaultProps() {
    return {
      text: 'hello'
    };
  },

  render() {
    return (
      <div style={{
        ':hover': {
          fontSize: '200%'
        }
      }}>
        {this.props.text || 'default not working'}
      </div>
    );
  }
});

Example = Radium(Example);

React.render(<Example />, document.body);

@ianobermiller I had a brief look at this between talks at ReactEurope today and tracked it down to the Enhancer which creates a subclass of the class you're wrapping.
This is because defaultProps becomes a static property of the component before you wrap it, and a caveat of babel means that doesn't get copied to the subclass you create. I spoke to @sebmck about this and he suggested using the optional spec.protoToAssign in order to try and work around the issue. We tried that and it didn't work. So it seems that that in order to for Radium to support IE9/10 then we need to move away from using classes in the enhancer.
What do you think about that?


MapDB: use atomic longs rather than UUIDs when generating IDs

UUIDs are ugly and we do present them to the user (for instance in the route view, although maybe that needs rethinking as well. Nonetheless, we probably don't want UUIDs used as entity IDs in the GTFS export either . . . )
In order to reduce transaction overhead, we can put the atomic long in a separate database with a single global transaction. We wrap that in call that makes a commit in versioned data store, so that an ID is not returned until it has been written to disk. That way, in a crash, the worst thing that can happen is that we skip a few ids - not the end of the world.

Oops, that wasn't supposed to close this.


[TO VALIDATE] Split appleseed.lib into multiple libraries

Like I said on Discord:
What do you think about splitting appleseed.lib into multiple libs (for instance foundation and renderer libs) so that building gets faster ?
Because it has to link appleseed.lib each time and it takes multiple seconds.
@FrancoisBeaune, I would like your opinion on this :)

Too much of a challenge for nothing much. Incremental build is already pretty fast.


windowsservercore: urllib SSL errors

I can't get SSL to work in the container, any ideas welcome:
docker run -it python:3.7.1-windowsservercore
>>> import urllib.request; urllib.request.urlopen("https://letsencrypt.org/")
Traceback (most recent call last):
  File "C:\Python\lib\urllib\request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Python\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Python\lib\http\client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Python\lib\http\client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Python\lib\http\client.py", line 1016, in _send_output
    self.send(msg)
  File "C:\Python\lib\http\client.py", line 956, in send
    self.connect()
  File "C:\Python\lib\http\client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "C:\Python\lib\ssl.py", line 412, in wrap_socket
    session=session
  File "C:\Python\lib\ssl.py", line 853, in _create
    self.do_handshake()
  File "C:\Python\lib\ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Python\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Python\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Python\lib\urllib\request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Python\lib\urllib\request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)>
>>>

I've verified that this is indeed an issue, but I've no idea where to turn to figure out what's going on. :disappointed:

As another data point, whether the error drops seems to depend on the site you're trying to hit; for example, doing urllib.request.urlopen("https://bootstrap.pypa.io/get-pip.py"), I do not get an error. :man_shrugging:

I have the same issue, maybe this whould be forwarded to the Python people?

Has anyone been able to reproduce outside the minimal Docker environment
Microsoft provides? The most likely culprit is something the image is
missing that a normal install isn't.
(I'm not aware of anyone having filed anything upstream yet, but it's worth
a search before doing so.)

Well I have the issue also with mcr.microsoft.com/windows which is not so minimal anymore. Not sure what else to test.
I'll have a look and file a bug on the Python tracker.

Created https://bugs.python.org/issue36137

As a (limited) workaround, maybe this can be added to the images:
certutil -generateSSTFromWU roots.sst && certutil -addstore -f root roots.sst && del roots.sst

But this really needs to be fixed in Python...

What is the solution to this.

As a (limited) workaround, maybe this can be added to the images:
certutil -generateSSTFromWU roots.sst && certutil -addstore -f root roots.sst && del roots.sst

But this really needs to be fixed in Python...

I am not getting the solution.Can you please elaborate please.

As a (limited) workaround, maybe this can be added to the images:
certutil -generateSSTFromWU roots.sst && certutil -addstore -f root roots.sst && del roots.sst

But this really needs to be fixed in Python...

How to add this into my docker image.

In your Dockerfile:
RUN certutil -generateSSTFromWU roots.sst && certutil -addstore -f root roots.sst && del roots.sst

Looks like the latest on this is in https://bugs.python.org/issue36011 (and that it's still a problem :disappointed:).


Docs: Fix for #110

The configuration file (/etc/systemd/system/etcd2.service.d/40-listen-address.conf) needs to be created and configured.
Fixes #110

Thanks for your work. This page has recently been rewritten. Closing this PR.


Saving an already created pipeline is broken

Task

When a pipeline has been saved, it should be able to be edited, instead it sometimes appears greyed out

find out why its greyed out

Correct functionality is:

the pipeline should always be editable
when the pipeline is set to active on the consortia pipeline page, editing the pipeline
should un-map the current users who are mapped to the pipeline. This is already done if the
active pipeline is switched on the consortia pipeline page, look how this is done

Perhaps relevant error:
Error:
collections.js:421 Uncaught (in promise) TypeError: col.associatedConsortia.indexOf is not a function
at eval (collections.js:421)
at Array.forEach ()
at eval (collections.js:420)
at Promise.js:801
at callListener (Promise.js:469)
at endMicroTickScope (Promise.js:555)
at IDBTransaction. (Promise.js:624)

hi, @rssk
I checked the codebase and found out what makes the inputs grey.
All inputs are disabled if runId in params are not provided to this component.
What is rundId and what is this for?
Also, it seems that only allow pipeline editing for its creator.
But I could not find creator/owner field in the database.


Explain how the statistics are created

Please add some explanation about how the statistics are generated by statistics.php. In  the docs/scripts/README.md there's a link to a documentation statistics.md that does not exist.

Duplicate of #27


broken should_receive expectations with do blocks don't cause the test to go red on classes which handle exceptions

If you have some code which rescues from exceptions, and you use the should_receive(:foo) do |arg| syntax to expect that class to make a call to a mock, if the assertions inside the do block fail, the test doesn't go red as expected.
Test-case which demonstrates the bug:
require 'spec_helper'

module BugReport203
  describe "an expectation on a class with a rescue block" do
    class TestClass
      def test_dependency
        @dependency ||= TestDependency.new()
      end

      def test_method
        begin
          self.test_dependency.some_call("I'm calling this method with unexpected arguments - THIS SHOULD CAUSE THE TEST TO GO RED!  But doesn't.")
        rescue
          # Exception-handling logic here
        end
      end
    end

    class TestDependency
      def some_call(arguments)

      end
    end

    # This test should be red!
    it "should know the method was called with wrong arguments" do
      mock_dependency = mock(TestDependency)

      test_object = TestClass.new()
      test_object.stub(:test_dependency).and_return(mock_dependency)

      # Obviously in this trivial case of checking string equality, I could use
      #
      # mock_dependency.should_receive(:some_call).with("expected arguments")
      #
      # which will go red as expected.
      # This bug applies to the more complex case where I actually need the do block to make my assertions
      mock_dependency.should_receive(:some_call) do |arg|
        arg.should == "expected arguments"
      end

      test_object.test_method()
    end
  end
end

We ran into this problem as well. To work around it, we just handled exception in a method, then monkeypatched that method in spec/support. It's working well for now.
Something like this:
# my_class.rb
class MyClass
  def my_method
    begin
      do_something
    rescue Exception => e
      handle_exception(e)
    end
  end

  def handle_exception(e)
    do_something_else
  end
end

# spec/support/helpers.rb

class MyClass
  original_handle_exception = instance_method(:handle_exception)

  define_method(:handle_exception) do |e|
    raise e if e.is_a?(RSpec::Mocks::MockExpectationError)

    original_handle_exception.bind(self).call(e)
  end
end

Another solution: rspec 3.3 (to be released ASAP) contains a new aggregate_failures feature that allows multiple failures within a single example and raises a single exception at the end.

Why didn't that throw/catch solution suit that @moll suggested above? Seems to be most natural way of fixing this problem and then you could say something in the lines of all other testing frameworks are fragile :)

@jarmo -- changing to throw/catch would be a breaking change to the API, and rspec-mocks and rspec-expectations are both usable with other test frameworks (e.g. cucumber and minitest) which expect the current behavior (raising an exception).  The best we can do for now is make it configurable.
...which, as of RSpec 3.3 it is :).  It's not a documented public API (and therefore may have breaking changes in later 3.x releases) but there's now this:
RSpec::Support.failure_notifier = lambda do |error, options|
  # do something with the error
end

So I believe you could do this:
RSpec::Support.failure_notifier = lambda do |error, options|
  throw :rspec_failure, error
end

RSpec.configure do |c|
  c.around do |ex|
    error = catch(:rspec_failure) do
      ex.run
      nil
    end

    raise error if error
  end
end

RSpec.describe "Rescuing exception failures" do
  it "does not prevent it from failing" do
    begin
      expect(1).to eq(2)
    rescue Exception
    end
  end

  it 'allows passing examples to pass' do
    expect(1).to eq(1)
  end
end

@myronmarston: instead of "fixing" bad design (tests not failing due to bad rescue Exception clause smells) you see that adding new features/configuration parameters is a better solution?
I hope RSpec is not heading Java's way where backwards compatibility is the most important thing to have so all that crap from it's beginning still exists.

I hope RSpec is not heading Java's way where backwards compatibility is the most important thing to have so all that crap from it's beginning still exists.

We made lots of backwards incompatible changes when we released 3.0.  4.0 will include more backwards incompatible changes (although much fewer, I believe -- and no major syntax changes).
We do follow SemVer, which means we've promised to our users not to intentionally break backwards compatibility in a minor or patch release, so making that kind of change in a 3.x release simply isn't an option.
All that said: I plan to think more about this to see if there's a backwards-compatible way we can do this in a 3.x release.
The snippet I pasted above was simply meant to show that for people who want this sooner, there is a way to do it.

Actually, I didn't think that this change as a backwards-compatible change. I'd be really fine if that would end up in 4.x series instead of not happening ever and being broken in future major versions as well.
I know what SemVer is and I'm following that with my own OSS libraries as well.
Thank You for all the good work so far!


mdbook: 0.1.7 -> 0.1.8

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

cc @Havvy

Success on aarch64-linux (full log)
Attempted: mdbook
Partial log (click to expand)

installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/cn3xdd18lffp15sm33ps8iv9j517swpc-mdbook-0.1.8
shrinking /nix/store/cn3xdd18lffp15sm33ps8iv9j517swpc-mdbook-0.1.8/bin/mdbook
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/cn3xdd18lffp15sm33ps8iv9j517swpc-mdbook-0.1.8/bin
patching script interpreter paths in /nix/store/cn3xdd18lffp15sm33ps8iv9j517swpc-mdbook-0.1.8
checking for references to /build in /nix/store/cn3xdd18lffp15sm33ps8iv9j517swpc-mdbook-0.1.8...
/nix/store/cn3xdd18lffp15sm33ps8iv9j517swpc-mdbook-0.1.8

Success on x86_64-linux (full log)
Attempted: mdbook
Partial log (click to expand)

installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/jphx6c1avnbvj374lsj1jbdw7879kx0z-mdbook-0.1.8
shrinking /nix/store/jphx6c1avnbvj374lsj1jbdw7879kx0z-mdbook-0.1.8/bin/mdbook
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/jphx6c1avnbvj374lsj1jbdw7879kx0z-mdbook-0.1.8/bin
patching script interpreter paths in /nix/store/jphx6c1avnbvj374lsj1jbdw7879kx0z-mdbook-0.1.8
checking for references to /build in /nix/store/jphx6c1avnbvj374lsj1jbdw7879kx0z-mdbook-0.1.8...
/nix/store/jphx6c1avnbvj374lsj1jbdw7879kx0z-mdbook-0.1.8

LGTM


Calendar Pages

Once we have #59 solved, it would be nice to have 2 calendar pages:

https://endoflife.date/calendar for all future EOL dates
https://endoflife.date/dates for all recent and future dates (Support/Releases/EOL)

See https://kevinl.info/just-the-class/calendar/ for inspiration.

I’d love to hear if this project also intends to support future releases? For example on https://endoflife.date/django, I see the image coming from the Django project has 4 releases that haven’t been released yet (but for which the release date is already decided). And the table underneath only shows past releases.

Hi @thibaudcolas, here are my 2 cents (I am only speaking for myself):
I do not think because:

endoflife.date is mostly about presenting EOL versions / dates for various product, so mostly about the past.
Most of the product don't have predictable or decided release date, so this use case looks pretty "niche".

-1 from my side too about adding incoming release information
On Thu, Sep 28, 2023, 23:17 Marc Wrobel @.***> wrote:

Hi @thibaudcolas https://github.com/thibaudcolas, here are my 2 cents
(I am only speaking for myself):
I do not think because:

endoflife.date is mostly about presenting EOL versions / dates for
various product, so mostly about the past.
Most of the product don't have predictable or decided release date,
so this use case looks pretty "niche".

—
Reply to this email directly, view it on GitHub
https://github.com/endoflife-date/endoflife.date/issues/2148#issuecomment-1739948494,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA3R4UCZXYXFRWYS7MJI63X4XLM5ANCNFSM6AAAAAATJYRQOU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

👍 seems fair to me! Just to be clear the info is already there for projects like Django. It’s in the image as I mentioned, and the "Active support" and "Security Support" dates are also based on when the next relevant releases are.


Invisible entity for client superior to 1.8.9 server

Describe the bug, provide any errors
Entity are invisible in some rare cases for 1.12.2 client that login a server with 1.8.8 viaversion
A clear and concise description of what the bug is. Can you https://hastebin.com the error?
Nope
How can we reproduce it?
Steps to reproduce the behavior:

Login into a 1.8.9 server with viaversion by using 1.12.2
entity like armor stand, villager, animals, sometimes are invisible in rare cases, you can hear only sounds for them

Expected behaviour
Entity must stay visibile
Screenshots
None
ViaVersion Dump:
https://dump.viaversion.com/47d8ae6906d9baec00bc4e8ced5166dc6432b7164d17fee4944b1d39e8cd68b2
Additional server info
Waterfall
Paperspigot 1.8.8

Unless we can consistently replicate this / others can replicate, we unfortunately can't fix it.
Closing this for now, but if anyone can provide a scenario for this will happily investigate.


Added itasa provider

Hi this is like my previous PR, just with all the checks you done before already applied, and some bit of improvements, with better support for SickRage

Please reopen


fixed cstyle error on src/istgt_log.h

Signed-off-by: Jayadeep KM<EMAIL_ADDRESS>

My bad. It was istgt_log.h , not isgt.h. ./cstyle.pl src/istgt.h is giving blank output


Possible typo in primitive-recursion.tex

The current passage from lines 99 to 102 reads as follows:

The general pattern then is this: to give a primitive recursive definition of a function , we provide two equations. The first defines the value of   without reference to .

It seems like the reference to  might be a typo or ambiguous, since  is last mentioned on line 50. The passage seems to make more sense if it were to say that the initial value of  is defined without reference to itself, instead of defined without reference to .

Thanks!


fix: pass jwt secret to studio container

What kind of change does this PR introduce?
relates to https://github.com/supabase/supabase/pull/26858
What is the new behavior?

[ ] Bump studio version once upstream is merged.

Additional context
Add any other context or screenshots.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.01%) to 60.185%

Files with Coverage Reduction
New Missed Lines
%

internal/gen/keys/keys.go
5
11.76%

Totals

Change from base Build<PHONE_NUMBER>:
-0.01%

Covered Lines:
6890

Relevant Lines:
11448

💛  - Coveralls

@sweatybridge I'm trying to debug an issue related to the endpoint /api/projects/default/config, but I am getting a - instead of a valid JWT.

I stumbled across this PR while researching and debugging; I noticed that in this PR, you put "AUTH_JWT_SECRET=" + utils.Config.Auth.JwtSecret under utils.Config.Studio.PgmetaImage instead of under utils.Config.Studio.Image.
I was slightly confused, only because pg_meta does not seem to rely on the env var AUTH_JWT_SECRET: https://github.com/search?q=repo%3Asupabase%2Fpostgres-meta AUTH_JWT_SECRET&type=code
I am using CLI 1.178.2 if that helps (which I believe wouldv'e included this PR, which seemed to ship around 1.174).
I just have no idea where the - would be coming from for the /api/projects/default/config endpoint.

Ah yes, it was an oversight on my side. I will add the env var to studio instead.

awesome, thank you @sweatybridge !


Update spatial function to sf

The package current depends on sp and geos, which will be deprecated in Oct 2023. We need to update all spatial functions to sf.

I went through the code and removed any direct calls to sp or raster. The ll_ratio function still depends on geosphere::distVincentyEllipsoid, which imports sp.


Custom EC Point Formats option

Would it be possible to add an option to change the supply custom ec point format list?
Current defaults are https://github.com/openssl/openssl/blob/e4162f86d7fd92058e5558bd81ce9dbc615fec3f/ssl/t1_lib.c#L197,
but I would only like to use TLSEXT_ECPOINTFORMAT_uncompressed

It wouldn't be too hard to do. As far as I can tell all the underlying code to achieve this exists - it just that (for some reason) its never been exposed via an API call. Whether the underlying code actually works is another question since its almost certainly not being exercised in any test given the lack of API.
So, the  implementation work consists of creating a function to manipulate s->ext.ecpointformats and s->ext.ecpointformats_len, as well as the equivalent SSL_CTX fields. Probably also a test would need to be written in test/sslapitest.c.
Unfortunately, this feature request is unlikely to receive much attention from the main committer team since I consider it quite niche and all current focus is on the 3.0 release. However if someone wanted to contribute a PR for this then it would be considered. I've added the "help wanted" label accordingly.
Note that this would not be eligible for backport to 1.1.1 since it is a new feature and the 3.0 feature freeze is hopefully later this month, so its  unlikely to appear in any release until 3.1/4.0 (whatever the next release after 3.0 is going to be).

Point format negotiation has been more-or-less removed at this point, so changing the default in OpenSSL may also make sense.
https://tools.ietf.org/html/rfc8422#section-5.1.2

And in terms of US government consumers (e.g., those that care about FIPS 140 validation), the use of (only) uncompressed point format is going to be required by the CNSA profile for (D)TLS 1.2: https://tools.ietf.org/html/draft-cooley-cnsa-dtls-tls-profile ("as required by [RFC8422] and [RFC8446]")
I'd be +1 to changing the default

And in terms of US government consumers (e.g., those that care about FIPS 140 validation), the use of (only) uncompressed point format is going to be required by the CNSA profile for (D)TLS 1.2: https://tools.ietf.org/html/draft-cooley-cnsa-dtls-tls-profile

Changing the default to conform to RFC 8446 seems like a good idea. However, for most U.S. government consumers (i.e., those not operating National Security Systems), CNSA does not apply. A more relevant reference for TLS would be NIST Special Publication 800-52 Rev. 2, Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations.

And in terms of US government consumers (e.g., those that care about FIPS 140 validation), the use of (only) uncompressed point format is going to be required by the CNSA profile for (D)TLS 1.2: https://tools.ietf.org/html/draft-cooley-cnsa-dtls-tls-profile

Changing the default to conform to RFC 8422 seems like a good idea. However, for most U.S. government consumers (i.e., those not operating National Security Systems), CNSA does not apply. A more relevant reference for TLS would be NIST Special Publication 800-52 Rev. 2, Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations.

"The use of compression may enable attackers to perform attacks using compression-based side
channels (e.g., [60], [11]). To defend against these attacks, the null compression method shall be
enabled, and all other compression methods shall be disabled. "
This suggests that the default should indeed be changed but I would also like to see an option for manually choosing them.

This suggests that the default should indeed be changed but I would also like to see an option for manually choosing them.

That text is referring to compression of the TLS session data (see compression_methods in https://tools.ietf.org/html/rfc5246#section-<IP_ADDRESS>).
This discussion is about the EC Point Formats extension, which is discussed in Sections <IP_ADDRESS> and <IP_ADDRESS> in SP 800-52 Rev. 2.


[OOBE] Updated assets and margins

Summary of the Pull Request
Recropped ImageResizer, ColorPicker and PowerRename .gifs: #9997
Moved hardcoded margins to resource dictionary, resolving: https://github.com/microsoft/PowerToys/issues/9993#issuecomment-789086919
Quality Checklist

[X] Linked issue: #9993 #9997
[ ] Communication: I've discussed this with core contributors in the issue.
[ ] Tests: Added/updated and all pass
[ ] Installer: Added/updated and all pass
[ ] Localization: All end user facing strings can be localized
[ ] Docs: Added/ updated
[ ] Binaries: Any new files are added to WXS / YML

[ ] No new binaries
[ ] YML for signing for new binaries
[ ] WXS for installer for new binaries

Contributor License Agreement (CLA)
A CLA must be signed. If not, go over here and sign the CLA.

/azp run

I think this is better but I think for 0.35, we need to redo them, a lot falls off.
For CP, the click point goes here


npm registry error 'no valid versions available'

hello,
somehow the npm entry for this plugin isn't valid anymore. npm install produces this error:
 fetchPackageMetaData error for<EMAIL_ADDRESS>No valid versions available for com.passslot.cordova.plugin.passbook
Is there still somebody maintaining this repo? otherwise i would maybe fork it and make a new npm registry entry

maybe the problem is that there are 2 entries for this plugin in package.json.
one is cordova-plugin-passbook, the other is com.passslot.cordova.plugin.passbook

We will have look into this.
We think you should still be able to use the plugin when you add it directly from github via
cordova plugin add https://github.com/passslot/cordova-plugin-passbook
If you already found a solution, we would be happy for a pull request

Any update on this issue ?

Hi @malua , were you able to achieve the functionality?

Yes, adding it directly via Github works! just use the the command passslot has posted. Unfortunately I don't have time to look at the issue and resolve it at the moment

Adding it via Github does indeed, work, but not out of the box. I had to make some manual updates to config.xml and package.json to get this to work, as well as build in ionic.
package.json:
"dependencies": {
...
"cordova-plugin-passbook": "https://github.com/passslot/cordova-plugin-passbook.git",
},
"plugins": {
...
"cordova-plugin-passbook": {}
}
config.xml:


Open source the release and changelog feature

What
In the current Manomano internal project, homer offers the option of publishing and displaying a project's change log using the following commands :

/homer changelog Generate changelogs for any Gitlab project using release tags.
/homer release Create a Gitlab release.
/homer release cancel Cancel a Gitlab release.
/homer release end End a Gitlab release.

These functionalities should be reviewed and modified to make them available in the open-source project.
How
In order to add a releasable project, an internal configuration file must be modified. This behaviour needs to be reviewed to simplify the process and avoid the need to redeploy homer each time.
Firstly, we should externalise this configuration, and write documentation about it.
Then we could add a command like /homer release enable to register a new releaseable project.
The current configuration is as follows:
{
    notificationChannelIds: [
      IT_DEPLOY_CHANNEL_ID,
      FT_SELLER_CHANNEL_ID,
    ],
    projectId: 1,
    releaseChannelId: FT_SELLER_CHANNEL_ID,
    releaseManager: defaultReleaseManager,
    releaseTagManager: semanticReleaseTagManager,
  },

projectId : Gitlab project ID
notificationChannelIds : ID of the Slack channels that receive notifications about the release
releaseChannelId : ID of the slack channel where the release of the project is enabled
releaseManager : change depending on the type of the project (library, micro-service, frontend ..)
releaseTagManager : defines the tag naming (semantic, stable date ...)
The modal could be like this :

I opened this MR before someone is working on this topic

This one can be closed


Format .rake files on save

My current setup doesn't seem to autformat .rake files on save, (I use :Rufo manually). Maybe I am missing something, but is there an easy way to configure this?

It's a bug. Now autosave works only for files with .rb extension.

https://github.com/ruby-formatter/rufo-vim/issues/32#issuecomment-2294814610


Opening web wallet with mnemonic phrase (Fatal error)

undefined
The above error occurred in task Za
created by Xa
created by takeEvery(staking/fetchAccount, Xa)
created by er
Tasks cancelled due to error:
takeEvery(staking/fetchAccount, Xa)
takeEvery(staking/validatorSelected, Qa)
undefined

Duplicate of https://github.com/oasisprotocol/oasis-wallet-web/issues/403


Handles custom roles

Users with custom roles were returned with just the role id so I added a method to get roles and then used them in the createUserListCSV sample to translate the id to a name.

Everything looks good @SpatialStrout. I went through and added a few comments. If you can address those in a few new commits then I can go ahead and accept the PR. Also, please pull the latest commit from ago-tools:master so that I can easily merge this in. You should be able to do pull --rebase upstream master to get the missing commit (it's a simple readme edit) and then automatically replay your commits on top of it.


[jvm-packages] change DeviceQuantileDmatrix into QuantileDMatrix

To fix https://github.com/dmlc/xgboost/issues/8453

So you mean we also change the CPU way to build QuantileDMatrix instead of plain DMatrix?

Yes, similar to pyspark interface, we can have QuantileDMatrix as default for both hist and gpu_hist.

Ok, I will file another PR for it.

@trivialfis please help to check the CI why the xgboost-gpu tests run on windows and mac?

There's an undefined symbol: XGQuantileDMatrixCreateFromCallbackImpl when compiling without CUDA.

Thx, fixed it.
But I'm still curious why the test can pass in ubuntu? and I locally build CPU version with USE_CUDA=OFF, and it still pass

Thank you for the quick fix! triggered the CI.

@hcho3 Could you please help check whether the CI is working alright?

@trivialfis please help to trigger the CI. Thx

Done.

@trivialfis Seems the CI is not triggered.

I triggered it again.

Does the python test have some issues?
E           ImportError: /opt/python/envs/cpu_test/lib/python3.8/site-packages/_cffi_backend.cpython-38-x86_64-linux-gnu.so: symbol ffi_type_uint32 version LIBFFI_BASE_7.0 not defined in file libffi.so.7 with link time reference

@trivialfis @hcho3 please help to trigger the CI. Thx

Done

@hcho3 @trivialfis please help to trigger the CI again, I just merged the latest commits into this PR. Thx

@trivialfis  please help to check why python test failed in this CI, I suppose this PR won't touch any python code.

@wbo4958 I re-created the machine env and re-triggered the tests. The CI is passing on the master branch, so hopefully it will remove the error.

Thx @hcho3, @trivialfis please help to review again. Thx


Update Header on each request

I'm using Laravel, and Laravel JWT to authenticate users. The jwt is set to refresh on each request, so that means that the previous token will be blacklisted.
How can I update each vuetable request header with the new token?
Tried to set axios config globally but that doesn`t affect vuetable requests.
I managed to update the token when receiving the request by using the below functionality, but still need to send the new token on each request.
  <ax-vuetable ref="vuetable"
     @vuetable:load-success="loadData"
  >
  ......

  ......
  loadData(success = this.loadSuccess, failed = this.loadFailed) {
      let token = success.headers.authorization;
      token = token.replace("Bearer ", "");
      token = token.trim();

      localStorage.setItem('token', token);
   }
   ......

Have you check out http-fetch prop? Will that solve your problem?

@ratiw I managed to add a function for http-fetch that will update the header with the new token. Thanks!

Can u please post a code for that? Cuss Im getting same issue and only first call works for vuetable


Update chart reloader to 1.1.0

This PR contains the following updates:

Package
Update
Change

reloader
minor
1.0.121 -> 1.1.0

Release Notes

stakater/Reloader (reloader)
v1.1.0
Compare Source
Announcement :loudspeaker: :loudspeaker: :loudspeaker:
Discussions are being done internally on how to release helm charts, it is highly likely that the release versioning of helm chart and images will become divergent from next release onwards.
(Different versioning for both artifacts, for example git release of v1.1.1 might not have a chart released with the same version)
Pls edit your automated workflows to accommodate for this change before the new release to have seamless updates in your clusters.
What's Changed

Manual release workflows by @MuneebAijaz in https://github.com/stakater/Reloader/pull/722
fixing go vet issues with logrus by @bnallapeta in https://github.com/stakater/Reloader/pull/727
Fixing Rollout restart, Previously would trigger rollout deployment strategy by @jangeja in https://github.com/stakater/Reloader/pull/724
Misc fixes to workflows by @MuneebAijaz in https://github.com/stakater/Reloader/pull/728
docs: update wording in readme by @deggja in https://github.com/stakater/Reloader/pull/730
chore(deps): update dependency stakater/vale-package to v0.0.33 by @renovate in https://github.com/stakater/Reloader/pull/721
Reduce footprint of UBI images  by @bainsy88 in https://github.com/stakater/Reloader/pull/717
Update dependencies by @MuneebAijaz in https://github.com/stakater/Reloader/pull/735

New Contributors

@jangeja made their first contribution in https://github.com/stakater/Reloader/pull/724
@deggja made their first contribution in https://github.com/stakater/Reloader/pull/730
@bainsy88 made their first contribution in https://github.com/stakater/Reloader/pull/717

Full Changelog: https://github.com/stakater/Reloader/compare/v1.0.121...v1.1.0

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.


extend the example in getting started page

This PR extends the example of the "getting started" page to add some contracts definition, contracts usage, doc usage, default usage, functions, merge, ...
EDIT: this is a draft, because it needs a fix in nickel, already implemented in PR 587, actually waiting for merge.

The idea was not just to add code to the initial example, but rather to add new sections to the getting started page (or, possibly, a new page) to complete the tour of the language, building on the same example. For example, keep things as they were, but add a new section "Validation", explaining that you may want to ensure now that your configuration respect some schemas, and add a contract definition, explain how it works, etc. Sorry if this hasn't been clear. I imagine you can still keep your big example, but break it into incremental addition to the original one.


[INIT] Create data model from UML diagram

Create the data model via classes and their associated tables in an SQLite database
https://github.com/Juansero29/Loop/blob/main/docs/uml.svg

Folders per modules
modules/models.ts
modules/services.ts
const create = ...
const update = ...
export default { create, update }
For exporting useful methods from services

https://ionicframework.com/docs/native/sqlite
https://github.com/storesafe/cordova-sqlite-storage
https://github.com/storesafe/cordova-sqlite-storage#android-database-provider
https://github.com/alex-steinberg/ionic-react-sqlite-example/blob/master/src/pages/Home.tsx

https://github.com/typeorm/typeorm

Dependency Injection of repositories: https://thomasburlesonia.medium.com/https-medium-com-thomasburlesonia-universal-dependency-injection-86a8c0881cbc

That dependency injection system looks quite nice.
npm install @mindspace-io/utils --save
It allows decoupling of use and construction, and easy testing via mocks. I didn't fully understand however how to tell when to use the mock and when to use the "real"  implementation. Do I need to comment out the real one to use the mock? Seems weird => need to investigate further this library

Also:

Don’t forget to build a custom hook to make DI lookups super easy!

Looks nice

Updated typeorm site: https://typeorm.io/
Also, updated dependency injector is this one: https://www.npmjs.com/package/@mindspace-io/react - oddly enough it requires use of --legacy-deps in order to be installed


Fix: carbon site works in IE (fixes #169)

What is the purpose of this pull request? (Check any that's applicable)

[ ] Feature
[X] Bug fix
[ ] Dependency updates
[ ] Documentation updates
[ ] Build related changes
[ ] Changes an existing functionality
[ ] Other, please explain:

Does this introduce a breaking change?

[ ] Yes
[X] No

What changes did you make? (Give an overview)

Updated Carbon-site-helpers to the latest version to fix the carbon site work in IE

Add any screenshots

Issue:
https://github.com/cerner/carbon-graphs/issues/169

:tada: This PR is included in version 2.4.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


"Green arrow" at timed traffic light junctions

@patrickjpeterson
https://upload.wikimedia.org/wikipedia/commons/thumb/a/a9/Grüner_Pfeil.jpg/220px-Grüner_Pfeil.jpg

http://steamcommunity.com/workshop/filedetails/discussion/583429740/1470840994968932782/

We can already to that manually by separating right turn lights and make them always green.
In my eyes it should not be not be a "separate feature", maybe just a convenience function, to speed up the setup.

Not quite. The green arrow is like a Stop sign for right turns. You habe to stop first, then obey yield rules and only then you are allowed to turn right at a red traffic light.

moved


BUG: Apphash on block 21627396 do we still have non-determinism in Paloma?

What is happening?

Section description
Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

KingSuper Apphashed on block 21627396
Paloma and pigeon versions and logs

Section description
Write down paloma version.
Write down pigeon version.

Section description
Please write detailed steps of what you were doing for this bug to appear.

NO repro
What is the expected behaviour?

Section description
If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

Non-determinism resolved.

The last block they signed before getting apphash error was https://paloma.explorers.guru/block/21627372 so their problems might have started a bit earlier.
This was also the only validator to report apphash error in this version.

Closing as won't fix.


Administrative function to remove peer

It's not currently possible to instruct weave to forget about a peer that was specified via weave launch or weave connect. This causes problems for users with dynamic environments where peers can go away forever; their only recourse currently is to restart the weave network leaving out the decommissioned peer.
Add a script command that instructs the router to forget a specified peer:
weave forget <CIDR>
'Forget' is suggested for the command name as 'disconnect' implies simply closing a connection; forget removes the peer from weave's internal list of known seeds. Any advance on a better name @rade?

Invite @fermayo

The original motivation for this feature was to be used on one peer to forget about one address previously specified to that peer. But what if it's inconvenient to remember which peer was told about which other peer?  Maybe 'forget' should work from any peer in the network.
This would require a way to uniquely identify a peer; IP addresses do not do this in the presence of NAT. Perhaps nicknames could be used?
A 'distributed forget' as I have described might have complex semantics under network partition.

I think in the long term that could be useful - perhaps we could split it off as a separate ticket? I'm keen to get an MVP of this out in the next stable release if possible...

'connect' adds to the "list of IP addresses this peer should always attempt to connect to". It makes sense to have an inverse operation for that. Let's not boil the ocean here.

How is this feature supposed to work? Suppose if I have 10 nodes in my weave network and I want to remove one from the network, do I need to run 'weave forget ' from each of the other 9 nodes?
It would be helpful, as mentioned by @bboreham, if i could run 'weave forget ' from only one of the other nodes.
Following is the experiment I did. Note: Auto discovery is ON (default)
I started 'weave launch' from 3 machines m1, m2 & m3.
On m1: I ran 'weave connect m2' and 'weave connect m3'. Now all were part of the network.
Then I ran 'weave reset' on m2 followed by 'weave forget m2' on m1.
Then on m2: I ran 'weave launch'. Immediately m2 joined the network automatically as if 'forget' had no effect.
In the second run, I 'weave reset' on m2 and 'weave forget m2' on both m1 and m3 and then I ran 'weave launch' on m2. This time m2 did not join the network. I got what I desired. But this is too tedious if have more peers.
In the first case I am wondering if what I think below is happening.
Even though m1 forgot m2, m3 did not and due to dynamic discovery m3 brought m2 into the network and propagated it to m1 as well. If this indeed is the case, can't we propagate what m1 forgot? Or if both tried propagating conflicting things - ie m1 saying I should foget m2 and m3 saying m2 is my peer, then can't the latest action (forget m2 in this case) take priority?

This still appears to be the case. If I have 10 nodes, this becomes nearly impossible without doing full restarts.

It doesn't require full restarts. You remove a node with weave reset on the node itself, followed by weave forget \<old-node\> on all the remaining nodes.


Puppeteer helper, proceedClick should return _waitForAction

Since click return the value from proceedClick.
I think proceedClick should return waitForAction just like other actions.

In this case, this should not be a difference.
The idea is the same: wait for action after the action is performed,


[Automation] Generate SDK based on TypeSpec 0.21.1

Major change is for generated TestBase to use getHttpClientOrUsePlayback from azure-core-test lib
.httpClient(getHttpClientOrUsePlayback(getHttpClients().findFirst().orElse(null)))

for configuring HttpClient.
Formatter changed back to spotless, hence lots of diff but not affecting API.
[Automation] Generate SDK based on TypeSpec 0.21.1

API change check
API changes are not detected in this pull request.

/azp run java - devcenter - ci

/azp run java - eventgrid - ci


clarification of featureParams

In spite of current wording, some readers are having difficulty understanding that non-null offsets and description of feature parameter tables is in the descriptions of specific features. Might help to give a specific example.
Also, in the table describing the Feature table, the description for the featureParams field does say "= NULL (reserved for offset to FeatureParams)". (The examples at the end of the chapter also show "featureParams, reserved, null".) This is misleading as it suggests that there is no current usage for this field, which is not the case.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1cb42967-20ca-9270-4572-65315127fcc5
Version Independent ID: 1fa413f1-71cf-ac2a-0999-70dd5d8ac4cd
Content: OpenType layout common table formats - Typography
Content Source: typographydocs/opentype/spec/chapter2.md
Product: typography
GitHub Login: @PeterCon
Microsoft Alias: PeterCon

There's also the sentence, "If a Feature Parameters table is not needed, the FeatureParams field must be set to NULL."  The word 'needed' should be replaced by something clearer, such as 'wanted', or 'wanted and defined', or perhaps weakened to 'to be supplied'.

And an explicit reference to the current registered tags, somewhat along to line of "see registered tag section on details of individual feature tags for which FeatureParams are used..." , may be?

The comments so far captured more adequately what I cannot quite express clearly. I have looked at the text again, and I think I would like to add two points:

one "naturally" expects FeatureParams being part of the FeatureTable being part of the GSUB/GPOS tables, more details are where those tables are described, at least where an overview is, instead of at the festure tags (and their individual description and implementation) section.

in where it says "null", "reserved", the surrounding paragraphs do not point to where it might not be - e.g. "...see the description and implementation section of individual feature tags for details..." would be nice?... And "reserved" definitely seems to sound like it is the "norm".

I now understand the list of feature tags are (1) somewhat in flux and more are being added over time, (2) and somewhat vendor-specific too - vendor's software have better support to specific groups of tags vs others.

There's also the sentence, "If a Feature Parameters table is not needed, the FeatureParams field must be set to NULL." The word 'needed' should be replaced by something clearer, such as 'wanted', or 'wanted and defined', or perhaps weakened to 'to be supplied'.

I would reworded it as "If a Feature Parameters table is not needed for the specific feature tag, the FeatureParams field must be set to NULL. See the registered tag section on description and implementation details of individual tags." . The not-explicit part about "is not needed", is "under what circumstances is it needed or not needed?"
Hindsight is  wonderful thing :-) .

That first sentence should be, "If a Feature Parameters table is not defined for the specific feature tag, the FeatureParams field must be set to NULL."  That avoids the issue of whether a table is an unnecessary luxury.

Changed for the next version:

A Feature table consists of an offset to a Feature Parameters (FeatureParams) table (if one has been defined for this feature - see note in the following paragraph), a count of the lookups listed for the feature (LookupCount), and an arbitrarily ordered array of indices into a LookupList (LookupListIndex). The LookupList indices are references into an array of offsets to Lookup tables.
Feature Parameters tables can only be used for certain features. The format of the Feature Parameters table is specific to a particular feature, and must be specified in the feature’s entry in the Feature Tags section of the OpenType Layout Tag Registry. Currently, Feature Parameter tables are defined only for the following features:

'cv01' – 'cv99'
'size'
'ss01' – 'ss20'

A Features Parameters table may be required or optional, according to the specifications for a given feature. The length of the Feature Parameters table must be implicitly or explicitly specified in the Feature Parameters table itself. The FeatureParamsfeatureParamsOffset field in the Feature Table recordsgives the offset relative to the beginning of the Feature Table. If a Feature Parameters table is not neededdefined for a given feature, or if a Feature Parameters table is defined but not used in a given font, the FeatureParamsfeatureParamsOffset field must be set to NULL.

In the table describing the Feature table:

Offset16featureParamsOffset= NULL (reserved for offset to FeatureParams)Offset from start of Feature table to FeatureParams table, if defined for the feature and present, else NULL

Thanks a lot!


Extending Isofit to handle multisurface classifications

This is a first shared draft of changes to the code base to handle an external classification file that is mapped to specific configurations of surface models and executions varying per-pixel or per-superpixel.
The goals of the changes:

Utilize the external classification of surface configurations via apply_oe, directly in a manually configured config, and able to run both in per-pixel mode and in superpixel mode.

Remain efficient both in terms of computation time and memory requirements.

How the code does this:

The isofit config file now carries information for all surface types present within the image.

When a classification file is present, isofit.init now indexes the image by surface classification

The n-number of state-pixel-groups are run in serial, which includes steps to persist forward model, set up workers and run_per_spectra.

Some outstanding considerations:

Pytest: The first version of these commits includes some files I used to extend Pytest. I need to reconcile my tests with other pytest elements so that we can eliminate all these extra (in most cases repeat) files.

Input: When running via apply_oe, there is a component of explicit (hard-coded) mapping in the form of a dict between the surface pixel value/label name and the surface model to use. This resides within the template_construction workflow. Some discussion could go into what the best way is the make this functionality easy to use.

Analytical line applicability: now with an open-ended number of potential surface state elements, the universal applicability of the analytical line subroutine to the surface-components of the statevector is an outstanding question. If the analytical line (and empirical line for that matter) is used, I need to add some functionality to only operate on rfl-components of the statevector (and flexible enough to allow for glint elements once we finalize the math).

Analytical line performance: The other analytical line consideration is in terms of computation. If the AL has to allow for both glint and non-glint pixels, it likely has to be run on a per-spectra basis, in serial similar to isofit.run. My initial fix has a bottleneck in saving the spectra.  need to add a step to aggregate per-spectra results similar to isofit.

Need to edit the updated config to be a non-breaking change. Utilize the multi-surface flag to accept both a single surface and multi-surface style.

Looking through there's a bunch of cleaning I need to do, but I wanted to get it out there.  I'll also attempt to add comments with context below.

This is an example of the output of a multisurface run with the currently implemented glint_model for water pixels. I'm curious what others think of the best strategy is to handle boundary (and likely mixed) pixels that clearly show up when using multiple surface models.

Updated surface config:
Uses flag to specify if isofit should look for multi-surface or use backwards compatible config
class SurfaceConfig(BaseConfigSection):
    """
    Surface configuration.
    """

    def __init__(self, sub_configdic: dict = None):
        self._multi_surface_flag_type = bool
        self.multi_surface_flag = None

        self._sub_surface_class_file_type = str
        self.sub_surface_class_file = None

        self._surface_class_file_type = str
        self.surface_class_file = None

        self._Surfaces_type = dict
        self.Surfaces = None

        self._surface_params_type = dict
        self.surface_params = None

        self.set_config_options(sub_configdic)

Backwards compatible config:
    "surface": {
      "surface_category": "additive_glint_surface",
      "surface_file": "../data/prm20151028_surface_coastal_model.mat"
    },

Multi-surface config:
    "surface": {
        "Surfaces": {
            "0": {
                "surface_category": "multicomponent_surface",
                "surface_file": "../remote/ang20170228_surface_model.mat",
                "surface_type": "base"
            },
            "1": {
                "surface_category": "glint_model_surface",
                "surface_file": "../remote/ang20170228_surface_model.mat",
                "surface_type": "test"
            }
        },
        "multi_surface_flag": true,
        "surface_params": {
            "select_on_init": true,
            "selection_metric": "Euclidean"
        }
    },

Isofit now handles the multi-surface run.
isofit.init will index image by state and construct the scene-wide statevector
class Isofit:
    def __init__(self, config_file, level="INFO", logfile=None):
        ...
        ...
        # Set up the multi-state pixel map
        if self.config.forward_model.surface.multi_surface_flag:
            self.state_pixel_index = index_image_by_class(
                self.config.forward_model.surface
            )
        else:
            self.state_pixel_index = []

        # Construct and cache the full statevector (all multistates)
        self.full_statevector, *_ = construct_full_state(self.config)

The multisurface run is handled in sequence within isofit.run.
Iterables are set up that map to the number of scene-wide surface classes.
    def run(self, row_column=None):
        ...
        ...
        # If multistate, split into class
        if len(self.state_pixel_index):
            index_pairs_class = []
            for class_row_col in self.state_pixel_index:

                if not len(class_row_col):
                    continue

                index_pairs_class.append(np.delete(np.array(class_row_col), 2, axis=1))
            index_pairs = index_pairs_class

        # Else it's not a multistate run
        else:
            index_pairs = [index_pairs]

The worker pool is then set at the level of the shared pixel-states and multi-processing is kicked off
        # Loop through index pairs and run workers
        class_loop_start_time = time.time()
        for i, index_pair in enumerate(index_pairs):

            # Don't want more workers than tasks
            n_iter = index_pair.shape[0]
            n_workers = min(n_workers, n_iter)

            # The number of tasks to be initialized
            n_tasks = min(
                (n_workers * self.config.implementation.task_inflation_factor), n_iter
            )

            # Get indices to pass to each worker
            index_sets = np.linspace(0, n_iter, num=n_tasks, dtype=int)
            if len(index_sets) == 1:
                indices_to_run = [index_pair[0:1, :]]
            else:
                indices_to_run = [
                    index_pair[index_sets[l] : index_sets[l + 1], :]
                    for l in range(len(index_sets) - 1)
                ]

            # Construct full fm
            self.fm = fm = ForwardModel(self.config, str(i))

            logging.debug(f"Pixel class: {str(i)}")
            logging.debug(f"Surface: {self.fm.surface}")

            # Put worker args into Ray object
            params = [
                ray.put(obj)
                for obj in [
                    self.config,
                    self.fm,
                    self.full_statevector,
                    self.loglevel,
                    self.logfile,
                    n_workers,
                ]
            ]
            # Initialize Ray actor pool (Worker class)
            self.workers = ray.util.ActorPool(
                [Worker.remote(*params, n) for n in range(n_workers)]
            )

            start_time = time.time()
            logging.info(
                f"Beginning {n_iter} inversions in {n_tasks} chunks"
                f"using {n_workers} cores"
            )

            # Kick off actor pool
            res = list(
                self.workers.map_unordered(
                    lambda a, b: a.run_set_of_spectra.remote(b), indices_to_run
                )
            )

All pixel level outputs are matched against the full_statevector, which is carried by the isofit object:

def match_statevector(
    state_data: np.array, full_statevec: list, fm_statevec: list, null_value=np.nan
):
    """
    A multi-class surface requires some merging across statevectors
    of different length. This function maps the fm-specific state
    to the io-state that captures all state elements present in the
    image. The full_state will record a Non

    Args:
        state_data: (n,) numpy array with the fm-specific state vector
        full_statevec: [m] list of state-names of the image-universal combined statevector
        fm_statevec: [n] list of state-names of the fm-specific state vector
        null_value: (optional) value to fill in the statevector elements that aren't present at a pixel
    """
    full_state = np.zeros((len(full_statevec))) + null_value
    idx = []
    for i, state in enumerate(full_statevec):
        if state in fm_statevec:
            idx.append(i)
    full_state[idx] = state_data

    return full_state


Issue with spring cloud aws and secrets manager, (random suffix expected for secrets manager id)

(Bug | Misconfiguration ?)
I'm using spring-boot 2.1.2 with spring-cloud-starter-aws-secrets-manager-config (2.1.1, tried
with 2.1.2 also) and Greenwich release of spring cloud.
I have followed the docs as described in documentation to setup the authentication from my service to postgres instance on aws rds:
https://cloud.spring.io/spring-cloud-static/spring-cloud-aws/2.1.0.RELEASE/single/spring-cloud-aws.html#_integrating_your_spring_cloud_application_with_the_aws_secrets_manager.
On my local environment I do manage to connect using the secrets manager id e.g.
/hercules_prod,
but when I run the same service inside aws beanstalk (as docker container) it fails on startup with reason:
AwsSecretsManagerPropertySourceLocator : Fail fast is set and there was an error reading configuration from AWS Secrets Manager: AWSSecretsManagerException: User: arn:aws:sts::xxxxxxxxxx:assumed-role/hercules-SecretsManagerAccessRole/i-xxxxxxxxxxxx is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:eu-west-1:xxxxxxxxxxxx:secret:/hercules_prod-QovDQp (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException;
The exception from logs is kind of misleading from aws side, because it is not about authorization, but non-existing id.
Why does the secrets manager id get suffix concatenated in this case when run on EC2 instance (/hercules_prod-QovDQp) and on each run there is new random string ?. When I debug the app locally it is /hercules_prod and works ok.
Looking at the code of AwsSecretsManagerPropertySourceLocator.java there is no reference to any suffix like this added, so Im not sure am I missing some additional config ?
Contents of my bootstrap.yml:
aws:
secretsmanager:
prefix:
defaultContext: service
failFast: true
name: hercules
enabled: true

Hi @sm-slavko-lozic ! Did you find any reason?
We have same problem.

Hi @sm-slavko-lozic ! Did you find any reason?
We have same problem.

Unfortunately no, as an alternative I switched to usage of kms for the time being, which works fine: https://github.com/zalando/spring-cloud-config-aws-kms

Hi @sm-slavko-lozic ! Did you find any reason?
We have same problem.

Unfortunately no, as an alternative I switched to usage of kms for the time being, which works fine: https://github.com/zalando/spring-cloud-config-aws-kms

Thanks!

Hi Everyone,
Happy New Year! I'm also getting the same issue and working on doing it myself through aws-sdk but I have one query here. Even if we set failFast as false I don't think it will be able to fetch the secrets from SM ever. It will just help us to start our application but that wouldn't be sufficient for any service which wants secrets. If secrets are not cardinal for an application then there is no point of even having it.
Please correct my understanding here.

Hey I figured out issue at my end. The account which I was using to create aws access key and the account through which I was trying to access AWS SecretManager were different. Later was not having permission to access the secrets and hence Spring cloud was appending some random characters. When I used same account to access both, the issue is resolved now.
I have kept failFast true and my service is up and running with this.
Cheers!!

Hello, i am using spring boot 2.2.27 and latest spring cloud version.  I set a secret in aws - /secret/user and store it. I used the jdk code from aws sdk to getString.
I now delete the secret in aws secrets manager. There is a time period of min 7 days.
Now i try to run my project without the secret manager,  i get run time error 400. InvalidRequest, cannot access secret set for delete.
I use bootstrap to override
aws.secretmanager.enable=false
aws.secretmanager.failfast=false
all combination did not work.
Is this a bug with aws or spring cloud?
My entire project is come to halt. I checkedout the previous source code,  even then ita throwing error.
Any help much appreciated.
Thanks
Govind

@govindrk the prefix is aws.secretsmanager (you missed "s"). Once you put this property there integration will be turned off.
400 when getting accessing secret set for delete is AWS Secrets Manager behavior, not Spring Cloud AWS.

I was able to get a secret value using an IAM role which the secret wasn't created with.
For testing I created an EC2 machine with the IAM role attached. The role had an IAM policy with secretsmanager:GetSecretValue action allowed. The trickiest thing was to figure out what the Resource value is valid.
For example, I have a secret with this ARN: arn:aws:secretsmanager:eu-west-1:12345678912:secret:super-duper-secret-Zm8sMf.
This
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:eu-west-1:12345678912:secret:super-duper-secret-Zm8sMf"
        }
    ]
}

And this
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:eu-west-1:12345678912:secret:super-duper-secret*"
        }
    ]
}

...didn't work.
But this policy works with no problems:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        }
    ]
}

I don't know why Spring Cloud requires access to all the secrets, though  aws secretsmanager get-secret-value CLI command works with no problems with even the first policy, but at least it works.
Hope this information will help.
P. S. Also we can make a conclusion that resource based policies won't work with Spring Cloud at all, because the policy is attached to a particular secret.

@badyag I ran into this problem in the past and it may be this very specific limitation in secrets manager:

If you specify an ARN, we generally recommend that you specify a complete ARN. You can specify a partial ARN too—for example, if you don’t include the final hyphen and six random characters that Secrets Manager adds at the end of the ARN when you created the secret. A partial ARN match can work as long as it uniquely matches only one secret. However, if your secret has a name that ends in a hyphen followed by six characters (before Secrets Manager adds the hyphen and six characters to the ARN) and you try to use that as a partial ARN, then those characters cause Secrets Manager to assume that you’re specifying a complete ARN. This confusion can cause unexpected results. To avoid this situation, we recommend that you don’t create secret names ending with a hyphen followed by six characters.
https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html


Add method to set DBOptions from string.

it would be great if you can add a test case for this to make sure it's doing what it's supposed to do


Documentation example for formatJson and parseJson

Congratulations on the 1.0.0 release. I find Objection has achieved to be both simple & powerful.
Plus, the documentation is quite good with lot of examples.
But when I tried to use $formatDatabaseJson and $parseDatabaseJson, I did not find much.
My problem was quite simple : convert boolean to integer value.
I use SQLite3, so boolean values are stored as integers.
And I get an integer value (0 or 1) on my Json when I do a query, even if I specify boolean in the knex schema migration builder or in the objection model,
So, I thought Objection and format/parseDatabaseJson were the perfect place to do this :
$formatDatabaseJson(json) {
  let db = super.$formatDatabaseJson(json);
  db.enabled = (db.enabled) ? 1 : 0;
  return db;
}

$parseDatabaseJson(db) {
  let json = super.$parseDatabaseJson(db);
  json.enabled = (json.enabled === 1) ? true : false;
  return json;
}

First, this seems to work but I'm not sure if there is a better way to do it or to answer my problem ?
I'm also wondering if this can be improved and made "generic" ? (without specifying each boolean field).
Second, it would be nice to have a real life example for those functions in the documentation.
I did find one here : #60, but this is not really about conversion but about computed field.

Thank you!
You're absolutely right, there should be an example for those. I'll add this to my todo list.

First, this seems to work but I'm not sure if there is a better way to do it or to answer my problem ?
I'm also wondering if this can be improved and made "generic" ? (without specifying each boolean field).

You could implement those methods in a common base class (for example BaseModel) and use jsonSchema to determine the type. If you don't use jsonSchema, you can add a static property for your model classes.

Thank you @koskimas.
I did also forget to mention that your (meaning you and the community :-) issue treatment and resolution is also one of the most efficient.
I do like the idea of using a BaseModel class.
It's pretty clean and I will probably use it anyway to have modelPaths or other stuff in there.
I tried a bunch of stuff to test my code in the Person model and I ended up with something like this :
  $parseDatabaseJson(db) {
    print('parse');
    let json = super.$parseDatabaseJson(db);
    if (Person.jsonSchema && Person.jsonSchema.properties) {
      var keys = Object.entries(Person.jsonSchema.properties);
      for (var i = 0 ; i < keys.length ; i++) {
        if (keys[i][1].type) {
          if (keys[i][1].type === 'boolean') {
            json[keys[i][0]] = (json[keys[i][0]] === 1) ? true : false;
          }
        } else {
          print('Missing type key for property ' + keys[i][0] + ' on jsonSchema');
        }
      }
    }
    else {
      print('Missing jsonSchema or jsonSchema.properties');
    }
    return json;
  }

I check for errors and use some 'print' function to customize with your taste (console.log, debug, logfile).
One thing I don't handle is a property of type: ['boolean', 'null'] which I don't use and I'm not sure is relevant.
I have two questions :

how do I get this up to the BaseModel class ? I get some errors and I can't seem to find my way around.
the parseDatabaseJson seems to parse more than it should : I got 18 "parse" for 16 db rows.
What are those extras pass in the parseDatabaseJson ? Where could this come from ?

how do I get this up to the BaseModel class ? I get some errors and I can't seem to find my way around.

Each javascript class instance has a constructor property. Simply replace Person.jsonSchema by this.constructor.jsonSchema. You should probably also only access jsonSchema once, since it returns a new object each time it is accessed (if a getter is used).

the parseDatabaseJson seems to parse more than it should : I got 18 "parse" for 16 db rows.
What are those extras pass in the parseDatabaseJson ? Where could this come from ?

$parseDatabaseJson is used to map column names to property names. The calls are cached though, so you should only see one "extra" call per property during the lifetime of the process.

You can throw whatever you like from there and the execution of the query will be halted and your error is thrown from the query. Let's keep this open. We'll close this when the example is written.

Hi @koskimas
Just to let you know, I use Typescript now and I got everything working nice with a BaseModel.
I just wanted to point out that I struggled to access the jsonSchema static variable without any error and I ended up using this line :
let schema = (<typeof BaseModel> this.constructor).jsonSchema;
The "extra calls" I had were coming from my request.
Basically, if if I just do Animal.query(), I got as many calls as I have rows.
If I do Animal.query().eager('owner'), I got some extra calls the first time (as you mentioned, this is cached).
If anybody might be interested with some code, here it is :
import { Model } from 'objection';

export default class BaseModel extends Model {
  static modelPaths = [__dirname];

  $formatDatabaseJson(json: any) {
    let db: any = super.$formatDatabaseJson(json);
    let schema = (<typeof BaseModel> this.constructor).jsonSchema;
    if (schema && schema.properties) {
      let properties: any = schema.properties;
      let keys: any = Object.keys(properties);
      for (let i = 0 ; i < keys.length ; i++) {
        if (properties[keys[i]].type) {
          if (properties[keys[i]].type === 'boolean') {
            db[keys[i]] = (db[keys[i]]) ? 1 : 0;
          }
        } else {
          throw(new Error('Missing type information for property \"' + keys[i] + '\" on jsonSchema'));
        }
      }
    } else {
      throw(new Error('Missing jsonSchema or jsonSchema properties on Model'));
    }
    return db;
  }

  $parseDatabaseJson(db: any) {
    let json: any = super.$parseDatabaseJson(db);
    let schema = (<typeof BaseModel> this.constructor).jsonSchema;
    if (schema && schema.properties) {
      let properties: any = schema.properties;
      let keys: any = Object.keys(properties);
      for (let i = 0 ; i < keys.length ; i++) {
        if (properties[keys[i]].type) {
          if (properties[keys[i]].type === 'boolean') {
            json[keys[i]] = (json[keys[i]] === 1) ? true : false;
          }
        } else {
          throw(new Error('Missing type information for property \"' + keys[i] + '\" on jsonSchema'));
        }
      }
    } else {
      throw(new Error('Missing jsonSchema or jsonSchema properties on Model'));
    }
    return json;
  }
}

Actually the docs have examples on how to override these. I don't know what other kind of examples I would add.


Add to snapcraft store

[ ] add .snapcraft.yml which runs this project as daemon
[ ] allow usage of a custom feeds.yml
[ ] run snap on ci and request a feed (similar to docker)
[ ] on master branch, push new snap version
[ ] add instructions to readme

I've started working on this. Sadly the snap-plugin for ruby can't compile the latest ruby version. So it's stuck at that stage.
However, I'd love to see this, as snap basically is made for rolling releases and does not require the setup of an 'auto updating' mechanism on the server.