Add support for alternate scroll escape

Closes #2727

I think this is ready for review. A couple of questions:

Could you point out to examples of tests that applicable for this feature? I think ref tests won't work for this or I just don't understand.
Is naming good?
The removal of scrolling.faux_multiplier setting is still undecided. Do we need it? Personally, I'd remove it for simplicity.

@chrisduerr
I've rebased my commits. The CI is green. Please take a look.


Permission denied for the downloading URL of FALCON binaries

Hello,
I noticed that the downloading URL of FALCON binaries (https://downloads.pacbcloud.com/public/falcon/) has suddenly become inaccessible recently. Is this a temporary problem?
Thanks!
Best,
Jia-Xing

hmm, it looks like the URL has just become accessible again. So no more problem. :-)
Best,
Jia-Xing


Consider removing flake8 support

the less code, the better!

@kennethreitz is it on the roadmap?
If so, I would be happy to submit a PR.

@sobolevn yes, we'll be happy to review a PR removing this feature.

I am on it.

It struck me as odd too that linting was part of the project.


Failing to build on Travis with :all

Hi,
Not sure if it is a bug or a configuration error on my part.
Failing Case
With this line in my spec_helper:
Combustion.initialize! :all

The build fails on travis with the chief complaint being:
activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:274:in `require': cannot load such file -- action_mailer/railtie (LoadError)

Passing Case
However, with this line:
Combustion.initialize! :action_controller, :action_view

It succeeds.
In both cases, I did not experience this issue on my local machine (using RVM as well) so I am not sure what to make of it.

Combustion is a library to help you test your Rails Engines

I think you gem is not an engine.

Do you think that the fact it is not an engine is the cause for it failing on travis and not failing locally?
And why not make Combustion work for any rails gem development project, and not just engines? I mean it looks like its practically working already.

I use Combustion with non-engine gems (Thinking Sphinx in particular, which has a railtie, but no engine).
I'm not quite sure why, but when bundler is installing gems in the failing build, it uses a very old version of actionmailer - 0.6.1. I'm quite sure railties weren't a thing then (10 years or so ago!). I'd recommend being clear about your runtime dependencies in your gemspec instead of your Gemfile, and anything essential to the gem working should be a runtime dependency, not just a test/development dependency.


Failed to start containers on the agent

Component
agent
Describe the bug
The builds i start all fail even before the clone step. I tested the clone step manually and it worked.
My Problem is that the agent can't even start the container.
Its similar to the error mentioned in https://github.com/woodpecker-ci/woodpecker/issues/85#issuecomment-1146854548_.
The containers run in an bridged network, because of some issues of podman with nftables.
System Info
{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-e08e1e1b"}

Additional context
journald log:
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"received execution"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"listen for cancel signal"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","Repo":"sebastian/website","Build":"16","ID":"46","time":"2022-07-10T09:27:20Z","message":"Executing 2 stages, in order of:"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","Repo":"sebastian/website","Build":"16","ID":"46","Stage":"0_7617429746464150374_clone","Steps":"0_7617429746464150374_clone","time":"2022-07-10T09:27:20Z","message":"stage"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","Repo":"sebastian/website","Build":"16","ID":"46","Stage":"0_7617429746464150374_stage_0","Steps":"0_7617429746464150374_step_0","time":"2022-07-10T09:27:20Z","message":"stage"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"Error response from daemon: no container with name or ID \"0_7617429746464150374_clone\" found: no such container","time":"2022-07-10T09:27:20Z","message":"could not kill container '0_7617429746464150374_clone'"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"Error response from daemon: no container with name or ID \"0_7617429746464150374_step_0\" found: no such container","time":"2022-07-10T09:27:20Z","message":"could not kill container '0_7617429746464150374_stage_0'"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"Error: No such network: 0_7617429746464150374_default","time":"2022-07-10T09:27:20Z","message":"could not remove network '0_7617429746464150374_default'"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","error":"Error response from daemon: could not find free subnet from subnet pools","exit_code":1,"time":"2022-07-10T09:27:20Z","message":"pipeline complete"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"uploading logs"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"uploading logs complete"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","error":"Error response from daemon: could not find free subnet from subnet pools","exit_code":1,"time":"2022-07-10T09:27:20Z","message":"updating pipeline status"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"rpc error: code = Unknown desc = Proc finished with exitcode 1, Error response from daemon: could not find free subnet from subnet pools","time":"2022-07-10T09:27:20Z","message":"grpc error: wait(): code: Unknown: rpc error: code = Unknown desc = Proc finished with exitcode 1, Error response from daemon: could not find free subnet from subnet pools"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"warn","repo":"sebastian/website","build":"16","id":"46","error":"rpc error: code = Unknown desc = Proc finished with exitcode 1, Error response from daemon: could not find free subnet from subnet pools","time":"2022-07-10T09:27:20Z","message":"cancel signal received"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"pipeline done"}
Jul 10 09:27:21 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:21Z","message":"updating pipeline status complete"}
Jul 10 09:27:21 echidna woodpacker-agent-0[100253]: {"level":"debug","time":"2022-07-10T09:27:21Z","message":"polling new jobs"}
Jul 10 09:27:21 echidna woodpacker-agent-0[100253]: {"level":"debug","time":"2022-07-10T09:27:21Z","message":"request next execution"}

podman version:
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

I used socat to get into the "conversation" between the agent and podman: conn.log
i used this command to start socat and capture the output: socat -t100 -x -v UNIX-LISTEN:podman.sock,mode=777,reuseaddr,fork UNIX-CONNECT:podman.sock.orig 2>&1 | tee conn.log
The post request for creating the network (/v1.41/networks/create) after the volume is a bit strange for me because i set WOODPECKER_BACKEND_DOCKER_NETWORK and from what i understood, it should use the already existing network.
and after that it cleans up everything.

look's like a podman issue witch can not create networks :thinking:

i fixed the  problem by using an non routed default network, using podman 3.0.1 and letting woodpecker create the network. the last one was the final working fix.
After it woodpecker created the network without problems and everything was working.

Should that be documented?

that it works with podman 3: yes the documentation can be changed
the behaviour of letting woodpecker create the network or it fails: that should definitly be changed and until its changed it should be documented.

@stop5 wana create a pull for the documentation?

i fixed the problem by using an non routed default network, using podman 3.0.1 and letting woodpecker create the network. the last one was the final working fix. After it woodpecker created the network without problems and everything was working.

What non-routed default network means? Is it impossible to use Podman inside of Podman?
I'm actually struggling with that.


Remove rating from hiring staff feedback form

Jira ticket URL:
https://dfedigital.atlassian.net/secure/RapidBoard.jspa?rapidView=21&projectKey=TEVA&modal=detail&selectedIssue=TEVA-253

I was just looking at this on my local. Main thought is the form looks a bit odd.

vs
The general feedback form:

Yes I am aware of that, would like to do that work in another PR.


pkg/constants: improve GetMsgByCode test

Complete tests in other situations.
Signed-off-by: Zhou Hao<EMAIL_ADDRESS>
Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #1214 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1214   +/-   ##
=======================================
  Coverage   47.89%   47.89%
=======================================
  Files         117      117
  Lines        7320     7320
=======================================
  Hits         3506     3506
+ Misses       3534     3533    -1
- Partials      280      281    +1

Impacted Files
Coverage Δ

supernode/daemon/mgr/scheduler/manager.go
21.91% <0%> (-0.69%)
:arrow_down:

pkg/constants/dfget_super_code.go
100% <0%> (+5.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0508f23...78504d7. Read the comment docs.

LGTM


Sorted class list in getSelector() to reduce duplicated selectors

Fixes #514
Adding this to v5 branch due reasons given in https://github.com/GoogleChrome/web-vitals/issues/514#issuecomment-2277072435

We really need unit tests for this...

We really need unit tests for this...

I was just going to add that we should have a test for this behavior.

I decided to just add them because I wanted to look at this code for some other v5 PRs as well.


Recovering from Worker Crash?

This is related to #209
If a Worker Crash occurs, my server just stop serving requests. Is there a way to recover from this?

Hm, it should  not to stop serving after catching exceptions.
Adding a while loop around https://github.com/ipkn/crow/blob/master/include/crow/http_server.h#L124 will do that.


Hover Effect on Header Section

When I hover over the top nav bar of the site, it doesn't feel much smooth cause there is no hover effect on them. It would be great if there was a hover effect on the top nav bar
I want to make a hover effect on the top nav bar
### Tasks

Kindly assign this issue to me, to get started

I am interested to take this  issue. Please assign me this issue.

I want make this awesome with a special effect according to theme that's why I would like to work on this

Please assign me this issue.

I would like to fix this issue, leave it to me .

@Its-Aman-Yadav thanks, let me know if you need anything :)

@blumamir Thank you so much for assigning me this issue, I have started my work and @alonkeyval would let you know when I need any help :)


ERROR "Undefined function 'ordfilt2' for input arguments of type 'double'."  when trying to run Depth2HHA.m

Greetings. Congratulations on your work. I was trying to run Depth2HHA.m.
I ran into the following error, I'm pasting the error traceback.
---Traceback Starts----
video = 1  frame = 1 video_name = dataset/test/signer14_sample100_depth.mp4
Unrecognized function or variable 'ordfilt2'.
Error in computeNormalsSquareSupport>filterItChopOff (line 107)
minSP = ordfilt2(sp, 1, B, 'symmetric');
Error in computeNormalsSquareSupport (line 52)
AtA = filterItChopOff(cat(3, AtARaw, AtbRaw), R, superpixels);
Error in processDepthImage (line 19)
[N1 b1] = computeNormalsSquareSupport(z./100, missingMask, normalParam.patchSize(1),...
Error in saveHHA (line 14)
[pc, N, yDir, h, pcRot, NRot] = processDepthImage(D*100, missingMask, C);
Error in CVPR21Chal_convert_HHA (line 30)
hha = saveHHA(['frame', mat2str(frame_loop)], matrix, save_folder, frame_gray, frame_gray);
----Traceback Ends----
Any insights will be appreciated.
Thanks.

Install  ”Image Processing Toolbox“


FilterTypes produce schema errors when filters properties are nullable.

Describe the bug
FilterTypes produce schema errors when filters properties are nullable.
To Reproduce
public class FooNullable
{
    public short? BarShort { get; set; }
}

[Fact]
public void Model_With_Nullable_Properties()
{
    // arrange
    // act
    var schema = CreateSchema(
        new FilterInputType<FooNullable>(
            d => d.Filter(f => f.BarShort)));

    // assert
    schema.ToString().MatchSnapshot();
}

Expected behavior
The schema correctly is created without any schema error.
Desktop (please complete the following information):

OS: macOS
Version: HotChocolate Version 10.0.0

This one is now fixed with #1170


Custom CSS?

It's a bit difficult to follow the code on how roam-server implements the CSS for org-mode when previewing (hovering over a node).  It does not seem to pull from orgcss via a link.  Is the code in /assets/org.css just "copy and pasted" CSS from orgcss?
Seems like the only way to change the CSS for preview is to edit the /assets/org.css file directly?  Is that the file holding the CSS for preview?  Or is there another mechanism?

You can change the CSS with the org-roam-server-style variable, see this snippet from my config:
(setq org-roam-server-style
	  (concat
	   "button#toggle-preview { margin: 4px; }"
	   "div#view-menu { margin: 4px; }"
	   "div#controls { right: 4px; left: 4px; bottom: 4px; }"
	   "button#toggle-list-type-button { margin: 0 4px; }"
	   "label#colormode { transform: translate(-25%, 0); }"
	   "label.toggle-off.btn-sm { padding-left: 0px; }"
	   ))


Allow JsonString to be used in Retrofit

This is for those cases where you want the json string of a response.
The 'usual' way would be to define your service method return as ResponseBody and call ResponseBody.string() at the call site to get the json string but this slight indirection is not necessary
interface Service {

  @GET("the/good/stuff") suspend fun goodStuff(): ResposeBody
}
// myService.goodStuff().string()

Being able to just annotate the method with @JsonString with a return type of String is much better.
Right now the @JsonString annotation targets are PROPERTY, FIELD. Adding FUNCTION to that should fix this with no other changes.

That seems reasonable to me. That would also allow this to work in auto value models. Want to send a PR?

Yeah I'll do that. Gimme a couple mins.


Bad request 400

Using Rails 4.2.0, I'm having an issue when uploading the file with the newest version. I believe the plugin does not set the correct request header :
Error occurred while parsing request parameters.
Contents:

------WebKitFormBoundaryHdAgqpA1RySFsj4l
Content-Disposition: form-data; name="media[file]"; filename="10189462.jpeg"
Content-Type: image/jpeg

In one of my app I'm still using an old version of the plugin (~3.0.7) which actually does works. By comparing the request made (between the new version that does not work and the old one that works), I can see that in the new version, the request header are being sent as Content-Type: application/json which I think is causing the error. However, in the old version, the request header is correct Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryzZASLhyCDGddt6Wq.
As note, I simply use the code in the demo in order to try it :
vm.upload = function (file) {
  Upload.upload({
    url: env.api + '/users/' + vm.user.id + '/attachments',
    data: {
      media: { // Rails is expecting this => params.require(:media).permit(:file)
        file: file
      }
    }
  }).then(function (resp) {
    console.log('Success ' + resp.config.data.file.name + 'uploaded. Response: ' + resp.data);
  }, function (resp) {
    console.log('Error status: ' + resp.status);
  });
}

The Rails backend is pretty straight forward, use carrierwave and looks like (again, this code does works with the old version) :
module Api
  module V1
    class AttachmentsController < Api::ApiController
      response_type :json

      def create
        media = attachable.new(media_params)
        respond_to do |format|
          if media.save
            format.json { render json: media, status: :created }
          else
            format.json { render errors_for(media) }
          end
        end
      end

      private

      def media_params
        params.require(:media).permit(:file)
      end
    end
  end
end

I've tried to play around a bit using the plugin options, but none of them seems being able to fix the issue.

mmm, ok my bad, I was hard setting Content-Type: application/json within a RequestInterceptor which causes the issue. Sorry about that and thanks again :)


[Question] Help converting ONNX to TensorRT with graphsurgeon and HPS plugin

Hi,
I'm attempting to convert an existing ONNX model with embeddings into a TensorRT model that uses HPS to perform the embedding lookup. I have extracted the embeddings from the ONNX model into files on disk in the HPS format, then used graphsurgeon to replace the embedding Gather node with an HPS_TRT node. Then I use trtexec to convert the new ONNX model into a TRT model and get this error:
2024/05/23 05:56:51 [stderr] [05/23/2024-05:56:50] [I] [TRT] Successfully created plugin: HPS_TRT
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.594][ERROR][RK0][main]: Expression check failed!
2024/05/23 05:56:51 [stderr] 	File: /hugectr/hps_trt/hps_plugin/hps_plugin.cpp:74
2024/05/23 05:56:51 [stderr] 	Function: getOutputDimensions
2024/05/23 05:56:51 [stderr] 	Expression: inputs[0].nbDims == 2
2024/05/23 05:56:51 [stderr] 	Hint: The dimensions of inputs[0] should be 2

I think I am probably not performing the graph surgery correctly, but am having trouble figuring out how to swap in the HPS_TRT node into the existing model.
Here is the relevant section from the original model. Not pictured is the categorical input feeding into this section. The upper Gather node selects a single categorical feature and sends it to the lower Gather node which does the embedding lookup.
After performing the graph surgery to replace the lower Gather node (the embedding lookup) with the HPS_TRT node, here is the relevant section in the new model. Here are details for the upper Gather node and the lower HPS_TRT node.
The specific changes I make using graphsurgeon are:

Create a new HPS_TRT node and set its inputs and outputs to the inputs and outputs of the embedding Gather node.
Change the input and output shapes to include the batch size and embedding vector size as documented here.
Remove the outputs from the old embedding Gather node (so it gets removed from the graph).

Here is the code I'm using to do this.
import numpy as np
import onnx
from onnx import shape_inference
import onnx_graphsurgeon as gs
import sys

model = onnx.load_model("model.onnx")
graph = gs.import_onnx(model)

input = next(n for n in graph.nodes if n.name == "/input_layer/Gather_77")
assert len(input.outputs) == 1
assert len(input.outputs[0].outputs) == 1
old = input.outputs[0].outputs[0]
embedding = next(t for t in old.inputs if t.name.endswith("embedding.weight"))
in_var = input.outputs[0]
in_var.shape = ("batch_size", 1)
in_var.dtype = np.int32
out_var = old.outputs[0]
out_var.shape = ("batch_size", 1, 512)
out_var.dtype = np.float32
new = gs.Node(
    name=old.name,
    op="HPS_TRT",
    attrs={
        "ps_config_file": f"hps.json\0",
        "model_name": "mymodel\0",
        "table_id": 0,
        "emb_vec_size": embedding.shape[1],
    },
    inputs=[in_var],
    outputs=[out_var],
)
graph.nodes.append(new)
old.outputs.clear()

graph.cleanup().toposort()
onnx.save(gs.export_onnx(graph), "model_hps.onnx")

Here is the contents of my hps.json configuration file, if it's relevant, but these are mostly initial values just to try to get something working.
{
  "supportlonglong": false,
  "models": [{
    "model": "mymodel",
    "sparse_files":["embeddings/input_layer.pretrained_embeddings.0.layers.embedding.weight.496328x512.tritonhps"],
    "deployed_device_list": [0],
    "maxnum_catfeature_query_per_table_per_sample": [1],
    "embedding_vecsize_per_table": [512],
    "default_value_for_each_table": [0]
  }]
}

Here is the full HCTR output when running trtexec.
2024/05/23 05:56:51 [stderr] =====================================================HPS Parse====================================================
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: fuse_embedding_table is not specified using default: 0
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: max_batch_size is not specified using default: 64
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: gpucache is not specified using default: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: hit_rate_threshold is not specified using default: 0.9
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: gpucacheper is not specified using default: 0.2
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: dense_file is not specified using default:
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: num_of_worker_buffer_in_pool is not specified using default: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: num_of_refresher_buffer_in_pool is not specified using default: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: cache_refresh_percentage_per_iteration is not specified using default: 0
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: maxnum_des_feature_per_sample is not specified using default: 26
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: refresh_delay is not specified using default: 0
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: refresh_interval is not specified using default: 0
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: use_static_table is not specified using default: 0
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: use_context_stream is not specified using default: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: use_hctr_cache_implementation is not specified using default: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: HPS plugin uses context stream for model mymodel: True
2024/05/23 05:56:51 [stderr] ====================================================HPS Create====================================================
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: Creating HashMap CPU database backend...
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][DEBUG][RK0][main]: Created blank database backend in local memory!
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: Volatile DB: initial cache rate = 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.074][DEBUG][RK0][main]: Created raw model loader in local memory!
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.581][INFO][RK0][main]: Table: hps_et.mymodel.sparse_embedding0; cached 496328 / 496328 embeddings in volatile database (HashMapBackend); load: 496328 /<PHONE_NUMBER>3709551615 (0.00%).
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.585][DEBUG][RK0][main]: Real-time subscribers created!
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.585][INFO][RK0][main]: Creating embedding cache in device 0.
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Model name: mymodel
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Max batch size: 64
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Fuse embedding tables: False
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Number of embedding tables: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.200000
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Embedding cache type: dynamic
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Use I64 input key: False
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: Configured cache hit rate threshold: 0.900000
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: The size of thread pool: 16
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: The size of worker memory pool: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: The size of refresh memory pool: 1
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.586][INFO][RK0][main]: The refresh percentage : 0.000000
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.594][INFO][RK0][main]: LookupSession i64_input_key: False
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.594][INFO][RK0][main]: Creating lookup session for mymodel on device: 0
2024/05/23 05:56:51 [stderr] [05/23/2024-05:56:50] [I] [TRT] Successfully created plugin: HPS_TRT
2024/05/23 05:56:51 [stderr] [HCTR][05:56:50.594][ERROR][RK0][main]: Expression check failed!
2024/05/23 05:56:51 [stderr] 	File: /hugectr/hps_trt/hps_plugin/hps_plugin.cpp:74
2024/05/23 05:56:51 [stderr] 	Function: getOutputDimensions
2024/05/23 05:56:51 [stderr] 	Expression: inputs[0].nbDims == 2
2024/05/23 05:56:51 [stderr] 	Hint: The dimensions of inputs[0] should be 2
2024/05/23 05:56:51 Process exited with exit status 1

What's confusing to me is that I'm getting the assertion failure inputs[0].nbDims == 2, even though the input to the HPS_TRT node is tensor: int32[batch_size, 1]. I suspect I am not performing the graph surgery correctly.
Is it valid to perform step (2) above, where I manually change the existing inputs and outputs to include the batch size and embedding vector size? Will the upper Gather node know to pass the batch size down to the HPS_TRT node, or does it not work that way? Do I instead need to be setting max_batch_size = 0 in this situation? (When I tried that I got Runtime error: Invalid value for unique_op capacity and haven't yet debugged further.)
Please let me know if I'm not thinking about this correctly or if there's more information I could provide to illuminate what's going wrong.
Thanks!

Thanks for the feedback, @KingsleyLiu-NV could you help check this out?

Hi @dmac , thanks for reporting this issue. Can you please share the minimal code to reproduce this? I think we need these files here:

The surgeon onnx model with HPS_TRT plugin
The HPS JSON file
The embedding table in HPS format
The script to build TRT engine

You don't need to provide the original onnx model and embedding table. Instead, you can provide the onnx model only with the plugin node that triggers the assertion error, and provide a mini embedding table to help us reproduce.

Hi @dmac ,
I create an ONNX model has the same layers as yours:

And then use your script to do graph surgery to use HPS TRT plugin:

After that, I use the follow script to build TRT engine:
import tensorrt as trt
import ctypes

plugin_lib_name = "/usr/local/hps_trt/lib/libhps_plugin.so"
handle = ctypes.CDLL(plugin_lib_name, mode=ctypes.RTLD_GLOBAL)

TRT_LOGGER = trt.Logger(trt.Logger.INFO)
EXPLICIT_BATCH = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)

def build_engine_from_onnx(onnx_model_path):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(EXPLICIT_BATCH) as network, trt.OnnxParser(network, TRT_LOGGER) as parser, builder.create_builder_config() as builder_config:
        model = open(onnx_model_path, 'rb')
        parser.parse(model.read())

        profile = builder.create_optimization_profile()
        profile.set_shape("keys", (1, 26), (1024, 26), (1024, 26))
        profile.set_shape("numerical_features", (1, 13), (1024, 13), (1024, 13))
        builder_config.add_optimization_profile(profile)
        engine = builder.build_serialized_network(network, builder_config)
        return engine

serialized_engine = build_engine_from_onnx("dlrm_tf_with_hps.onnx")
with open("dlrm_tf_with_hps.trt", "wb") as fout:
    fout.write(serialized_engine)
print("Successfully build the TensorRT engine")

Here is the log of successful building:
[05/28/2024-01:58:08] [TRT] [I] [MemUsageChange] Init CUDA: CPU +18, GPU +0, now: CPU 26, GPU 423 (MiB)
[05/28/2024-01:58:14] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1667, GPU +310, now: CPU 1770, GPU 733 (MiB)
[05/28/2024-01:58:14] [TRT] [W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[05/28/2024-01:58:14] [TRT] [I] No importer registered for op: HPS_TRT. Attempting to import as plugin.
[05/28/2024-01:58:14] [TRT] [I] Searching for plugin: HPS_TRT, plugin_version: 1, plugin_namespace:
=====================================================HPS Parse====================================================
[HCTR][01:58:14.412][INFO][RK0][main]: fuse_embedding_table is not specified using default: 0
[HCTR][01:58:14.412][INFO][RK0][main]: dense_file is not specified using default:
[HCTR][01:58:14.412][INFO][RK0][main]: num_of_refresher_buffer_in_pool is not specified using default: 1
[HCTR][01:58:14.412][INFO][RK0][main]: maxnum_des_feature_per_sample is not specified using default: 26
[HCTR][01:58:14.412][INFO][RK0][main]: refresh_delay is not specified using default: 0
[HCTR][01:58:14.412][INFO][RK0][main]: refresh_interval is not specified using default: 0
[HCTR][01:58:14.412][INFO][RK0][main]: use_static_table is not specified using default: 0
[HCTR][01:58:14.412][INFO][RK0][main]: use_context_stream is not specified using default: 1
[HCTR][01:58:14.412][INFO][RK0][main]: use_hctr_cache_implementation is not specified using default: 1
[HCTR][01:58:14.412][INFO][RK0][main]: thread_pool_size is not specified using default: 16
[HCTR][01:58:14.412][INFO][RK0][main]: init_ec is not specified using default: 1
[HCTR][01:58:14.412][INFO][RK0][main]: enable_pagelock is not specified using default: 0
[HCTR][01:58:14.412][INFO][RK0][main]: fp8_quant is not specified using default: 0
[HCTR][01:58:14.412][INFO][RK0][main]: HPS plugin uses context stream for model dlrm: True
====================================================HPS Create====================================================
[HCTR][01:58:14.413][INFO][RK0][main]: Creating HashMap CPU database backend...
[HCTR][01:58:14.413][DEBUG][RK0][main]: Created blank database backend in local memory!
[HCTR][01:58:14.413][INFO][RK0][main]: Volatile DB: initial cache rate = 1
[HCTR][01:58:14.413][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0
[HCTR][01:58:14.413][DEBUG][RK0][main]: Created raw model loader in local memory!
[HCTR][01:58:15.068][INFO][RK0][main]: Table: hps_et.dlrm.sparse_embedding0; cached 2600 / 2600 embeddings in volatile database (HashMapBackend); load: 2600 /<PHONE_NUMBER>3709551615 (0.00%).
[HCTR][01:58:15.071][DEBUG][RK0][main]: Real-time subscribers created!
[HCTR][01:58:15.071][INFO][RK0][main]: Creating embedding cache in device 0.
[HCTR][01:58:15.093][INFO][RK0][main]: Model name: dlrm
[HCTR][01:58:15.093][INFO][RK0][main]: Max batch size: 1024
[HCTR][01:58:15.093][INFO][RK0][main]: Fuse embedding tables: False
[HCTR][01:58:15.093][INFO][RK0][main]: Number of embedding tables: 1
[HCTR][01:58:15.093][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 1.000000
[HCTR][01:58:15.093][INFO][RK0][main]: Embedding cache type: dynamic
[HCTR][01:58:15.093][INFO][RK0][main]: Use I64 input key: False
[HCTR][01:58:15.093][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000
[HCTR][01:58:15.093][INFO][RK0][main]: The size of thread pool: 256
[HCTR][01:58:15.093][INFO][RK0][main]: The size of worker memory pool: 3
[HCTR][01:58:15.093][INFO][RK0][main]: The size of refresh memory pool: 1
[HCTR][01:58:15.093][INFO][RK0][main]: The refresh percentage : 0.200000
[HCTR][01:58:15.109][INFO][RK0][main]: Initialize the embedding cache by by inserting the same size model file with embedding cache from beginning
[HCTR][01:58:15.109][DEBUG][RK0][main]: Created raw model loader in local memory!
[HCTR][01:58:15.109][INFO][RK0][main]: EC initialization on device 0 for hps_et.dlrm.sparse_embedding0
[HCTR][01:58:15.178][INFO][RK0][main]: Initialize the embedding table 0 for iteration 0 with number of 512 keys.
[HCTR][01:58:15.248][INFO][RK0][main]: Initialize the embedding table 0 for iteration 1 with number of 512 keys.
[HCTR][01:58:15.316][INFO][RK0][main]: Initialize the embedding table 0 for iteration 2 with number of 512 keys.
[HCTR][01:58:15.385][INFO][RK0][main]: Initialize the embedding table 0 for iteration 3 with number of 512 keys.
[HCTR][01:58:15.452][INFO][RK0][main]: Initialize the embedding table 0 for iteration 4 with number of 512 keys.
[HCTR][01:58:15.473][INFO][RK0][main]: LookupSession i64_input_key: False
[HCTR][01:58:15.473][INFO][RK0][main]: Creating lookup session for dlrm on device: 0
[05/28/2024-01:58:15] [TRT] [I] Successfully created plugin: HPS_TRT
[05/28/2024-01:58:15] [TRT] [I] Graph optimization time: 0.00860746 seconds.
[05/28/2024-01:58:15] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 5950, GPU 833 (MiB)
[05/28/2024-01:58:15] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 5952, GPU 843 (MiB)
[05/28/2024-01:58:15] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[05/28/2024-01:58:17] [TRT] [I] Detected 2 inputs and 1 output network tensors.
[05/28/2024-01:58:17] [TRT] [I] Total Host Persistent Memory: 7088
[05/28/2024-01:58:17] [TRT] [I] Total Device Persistent Memory: 0
[05/28/2024-01:58:17] [TRT] [I] Total Scratch Memory: 0
[05/28/2024-01:58:17] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 4 MiB
[05/28/2024-01:58:17] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 6 steps to complete.
[05/28/2024-01:58:17] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 0.0212ms to assign 3 blocks to 6 nodes requiring 1102336 bytes.
[05/28/2024-01:58:17] [TRT] [I] Total Activation Memory: 1102336
[05/28/2024-01:58:17] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 6138, GPU 931 (MiB)
[05/28/2024-01:58:17] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 6138, GPU 941 (MiB)
[05/28/2024-01:58:17] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +4, now: CPU 0, GPU 4 (MiB)
Successfully build the TensorRT engine

Maybe there is something wrong with your ONNX model after graph surgery. Can you please check the output tensor of /input_layer/Gather_77 in your model is really a two-dimensional tensor of shape (batch_size, 1)?
The annotation tensor: int32[batch_size, 1] in ONNX visualization may not reflect he actual tensor shape during runtime. You can run your original ONNX model with onnxruntime to check the tensor shape in related layers.

@KingsleyLiu-NV Thanks for taking a look at this, your response helped put me on the right track.
There were two issues I needed to fix:

The /input_layer/Gather_77 was using scalar for its indices input. I changed that from 100 to [100] so it would output the correct shape that the HPS_TRT node expects.
I needed to add a Reshape node between HPS_TRT and Concat to reshape ["batch_size", 1, 512] to ["batch_size", 512].

After making these adjustments to my script I was able to succesfully convert it to TensorRT. Thanks again for your help!

For posterity, this is the script I ended up with.
import numpy as np
import onnx
import onnx_graphsurgeon as gs

model = onnx.load_model("model.onnx")
graph = gs.import_onnx(model)

def replace_embedding_node(gather_node_name: str):
    # Change the Gather input shape from i to [i] to match the expected input
    # shape for the HPS_TRT node.
    gather = next(n for n in graph.nodes if n.name == gather_node_name)
    indices = next(t for t in gather.inputs if "Constant" in t.name).inputs[0]
    indices.attrs["value"].values = np.array([indices.attrs["value"].values])

    # Replace the inline weights embedding lookup with HPS_TRT.
    embedding = gather.outputs[0].outputs[0]
    weights = next(t for t in embedding.inputs if t.name.endswith("embedding.weight"))
    concat_in = embedding.outputs[0]
    hps_out = gs.Variable(
        name=embedding.name+".HPS_output_0",
        dtype=np.float32,
        shape=("batch_size", 1, weights.shape[1]),
    )
    hps = gs.Node(
        name=embedding.name+".HPS",
        op="HPS_TRT",
        attrs={
            "ps_config_file": f"hps.json\0",
            "model_name": "mymodel\0",
            "table_id": 0,
            "emb_vec_size": weights.shape[1],
        },
        inputs=[gather.outputs[0]],
        outputs=[hps_out],
    )

    # Reshape the HPS_TRT output from ["batch_size", 1, N] to ["batch_size", N]
    # to match the expected shape of the Concat inputs.
    new_shape = gs.Constant(
        name=hps.name+"/Reshape_input_shape",
        values=np.array([-1, weights.shape[1]]),
    )
    reshape = gs.Node(
        name=hps.name+"/Reshape",
        op="Reshape",
        inputs=[hps_out, new_shape],
        outputs=[concat_in],
    )

    graph.nodes.append(hps)
    graph.nodes.append(reshape)
    embedding.outputs.clear()

replace_embedding_node("/input_layer/Gather_77")
graph.cleanup().toposort()
onnx.save(gs.export_onnx(graph), "model_hps.onnx")


ALC256 partition corruption

Got corrupted partition after first reboot of my freshly installed macOS. Error 8 while checking partition with diskutility and have to reinstall the whole system after.
Don't know why it cause some interference with my NVMe ssd. Also I'm getting high CPU load using the kext during the first boot.
However no such problems using CloverHDA. And it's a little bit strange since as far as I can see you're sharing the same sources.
I want leave my system partition intact, so let get things fixed together. I can perform several reinstall to help debugging and fixing. How can I help you with it and what should I provide you?

Hi,
The issues you describe are quite weird. And I am a bit at a loss what to think of.
I do not think AppleALC could really cause any filesystem corruption, since it only performs read operations, and this should be safe. NVMe SSD could be related but I am afraid I do not know anything to suggest in this direction.
As for high CPU load, could you clarify which process actually causes it? Is it kernel_task? Does the issue persist after a reboot? After all we know that Spotlight takes quite some time to index everything right after the installation, and this produces pretty high CPU load.
Also note that AppleALC does hardly anything after AppleHDA is loaded. So if you see kernel_task occupying loads of your CPU time after you booted, it is likely AppleHDA itself.
We did get reports of similar issues with different resources, and I would suggest starting with decompressing the resources and comparing them to the ones that work fine for you. AppleALC uses optimised packages, so as a result there could be differences and these differences may theoretically cause high CPU load.

@vit9696 could it be due to the AppleHDA patching made by the kext and not by clover itself? CloverHDA also uses five patches and uses same zlibs made by @insanelydeepak . However the AppleHDA patches remain in the config.plist and proceeded by clover. Is there a way to make AppleHDA patches external for your kext the same way? I mean making it the way so the zlibs could be injected by the kext and AppleHDA loaded, while patching made on the clover side.

@nos1609 could you please answer the questions in the previous? I am not even sure when exactly you have high CPU load, and what causes it, "during first boot" is not clear at all.
Having patches present in Clover's config.plist will cause issues, you should not do that. This is because AppleALC will have to fully rescan the kext binary instead of finding and replacing a few bytes in the beginning. To remove patches from AppleALC you could edit codec plist and recompile, but it is very unlikely that something like that is related.
I would not be so sure that the zlibs are exactly the same even if the source is mentioned. Please follow what I told you and compare the contents of the files. We have source releases.

@vit9696 yep, sorry about that, I'm aware of sierra's spotlight first boot high load due to siri's requirements, but the kernel task cpu load remains high after spotlight process got finished. Also sometimes I get some kind of distortion after 5-10 minutes of video preview. I'm currently using this kext to get output working. They are some issues with the jack, but speakers and internal mic are working fine.

@vit9696 maybe trying the DEBUG version might help to find out the root cause of the issue?

So, if you have kernel_task eating CPU resources after system is loaded, it is definitely not AppleALC, but AppleHDA. Since it is AppleHDA, the issue is in the resources, and obviously changing anything in AppleALC code or enabling its debug mode will simply not help.
Since the resources were provided by @insanelydeepak, I think you might want to talk to him, and not to us. Plus I repeat… there might be certain substantial differences between what we have in AppleALC, and what exists in cloverHDA. Please check that first of all.

i had this too on my Dell 7559 when I installed the kext to /Library/Extensions. Now I have it on EFI/Clover/kexts/other/ which works but sometimes it does not load (reboot fixes it). I have one question though, is it possible to add HDMI audio out support for mobile HD530 (0x191b0000)?

@vit9696 Finally checked everything and hopefully fixed the issue. For now. It's not finished as they're bug in the layout itself, but at least I've fixed the 1500% load caused by kerneltask and the partition deterioration during sleep. As I do not know how to creat these zlib files from scratch I've simly replaced them from scratch I had simply to replace the layout13.xml.zlib & PlatformsID.xml.zlib with the versions from a working kext provided in the open issue by @insanelydeepak. Audiojack is not yet fixed, but at least we now the cause of the issue for now.
I'm not a git' specialist, so I'll simply leave you the needed files to correct the error just right here in attach.
Архив.zip
I'm gonna try to find some working layout files during the week, so please keep the issue open till that moment.
And no, that was not caused by NVMe patches😉
To find out that I've tested using both the 2 known approaches to bring up the ssd, and both showed the same final 🚫 at loading the system after few reboots.
Hope we might fix this soon, now aware of the culprit in this issue.

@nos1609 that's interesting. In theory the resources we use should be not different from the original ones.
@vandroiy2013 could you check what's wrong with what we have? These freezes are not too funny. Might those be the optimisations?

I'll contact @insanelydeepak for the clarification. In my opinion existing resources in AppleALC was a kind of test version. Too many changes in it...
Left - existing resources, Right - new resources provided by @nos1609

Hi
Interesting I did too many fixes to ALC256 to work with many systems  , I don't remember for which system Left Side resource were made Right I assumes this is perfect for your system and if other don't issue can for them too in my repo too
BTW good catchUp @nos1609
issue should be closed

@insanelydeepak the only issue is a strange device in the input called "External speakers". However  seems like @syscl had solved this mysterious input...

@nos1609 @insanelydeepak But we lost External speaker.
On ALC 298, there's ext mic on both XPS 15 9550 and Surface Pro 4.
syscl

@syscl

But we lost External speaker
what ??

On ALC 298, there's ext mic on both XPS 15 9550 and Surface Pro 4.
I was unable to detect ext mic on XPS 9550 and it was present in surface pro


Validate VCN Security List Rules for File Storage

Oracle notifies us:

The Oracle     Cloud Infrastructure team has identified an issue where customers without     stateful VCN security list egress rules may experience a temporary     disruption during Mount Target failover.
To function correctly, File Storage requires stateful ingress to TCP ports     111, 2048, 2049, and 2050 and stateful ingress to UDP ports 111 and 2048.     File storage also requires stateful egress from TCP ports 111, 2048, 2049,     and 2050 and stateful egress from UDP port 111.
If you do not have the correctly configured stateful egress rules your     Mount Target(s) can become temporarily unavailable during a planned or     unplanned Mount Target failover. Please validate your security list rules     as soon as possible. Instructions can be found in the additional     information section of this message.
Review the Configuring VCN Security List Rules for File Storage documentation found here: https://docs.cloud.oracle.com/iaas/Content/File/Tasks/securitylistsfilestorage.htm

Default Security List
The default security list does not allow NFS traffic:

Stateless
Source
IP Protocol
Source Port Range
Destination Port Range
Type and Code
Allows

No
<IP_ADDRESS>/0
TCP
All
22

TCP traffic for ports: 22 SSH Remote Login Protocol

No
<IP_ADDRESS>/0
ICMP

3, 4
ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don't Fragment was Set

No
<IP_ADDRESS>/16
ICMP

3
ICMP traffic for: 3 Destination Unreachable

[usre@host ~]$ oci network security-list get --security-list-id=${SEC_LIST_ID}

{
  "data": {
    "compartment-id": "ocid1.compartment.oc1..aaaaaaaap76l4pmddjqc4xs3foytrlxjaemhsteenmvxgcile43nmjtmm44q",
    "defined-tags": {},
    "display-name": "Default Security List for ClusterVCN",
    "egress-security-rules": [
      {
        "destination": "<IP_ADDRESS>/0",
        "destination-type": "CIDR_BLOCK",
        "icmp-options": null,
        "is-stateless": false,
        "protocol": "all",
        "tcp-options": null,
        "udp-options": null
      }
    ],
    "freeform-tags": {},
    "id": "ocid1.securitylist.oc1.iad.aaaaaaaaapgg7m7nkb5dnx7n6ohj5ztjc2k6adjwqetrlknfdsvheoz3eklq",
    "ingress-security-rules": [
      {
        "icmp-options": null,
        "is-stateless": false,
        "protocol": "6",
        "source": "<IP_ADDRESS>/0",
        "source-type": "CIDR_BLOCK",
        "tcp-options": {
          "destination-port-range": {
            "max": 22,
            "min": 22
          },
          "source-port-range": null
        },
        "udp-options": null
      },
      {
        "icmp-options": {
          "code": 4,
          "type": 3
        },
        "is-stateless": false,
        "protocol": "1",
        "source": "<IP_ADDRESS>/0",
        "source-type": "CIDR_BLOCK",
        "tcp-options": null,
        "udp-options": null
      },
      {
        "icmp-options": {
          "code": null,
          "type": 3
        },
        "is-stateless": false,
        "protocol": "1",
        "source": "<IP_ADDRESS>/16",
        "source-type": "CIDR_BLOCK",
        "tcp-options": null,
        "udp-options": null
      }
    ],
    "lifecycle-state": "AVAILABLE",
    "time-created": "2019-06-18T10:59:38.045000+00:00",
    "vcn-id": "ocid1.vcn.oc1.iad.aaaaaaaaxzh7u7tonlrvp4th7wh743auhvvyxmbeuejyryloqrk4vr7doija"
  },
  "etag": "51ce8358"
}

CitC  Security List
The  firewall for the cluster network is set in
https://github.com/ACRC/oci-cluster-terraform/blob/e8dcbb41d2f2d47f7ede8d12322acf41206840c1/network.tf#L36-L59
And this defines an ingress security list

Stateless
Source
IP Protocol
Source Port Range
Destination Port Range
Type and Code
Allows

No
<IP_ADDRESS>/8
TCP
All
All

TCP traffic for ports: All

No
<IP_ADDRESS>/0
TCP
All
3000

TCP traffic for ports: 3000

[user@host ~]$ oci network security-list get --security-list-id=${SEC_LIST_ID}

{
  "data": {
    "compartment-id": "ocid1.compartment.oc1..aaaaaaaap76l4pmddjqc4xs3foytrlxjaemhsteenmvxgcile43nmjtmm44q",
    "defined-tags": {},
    "display-name": "ClusterSecurityList",
    "egress-security-rules": [],
    "freeform-tags": {},
    "id": "ocid1.securitylist.oc1.iad.aaaaaaaalj4an5ghttm4gohw6yyogq2pfyukcr6g6n2mxp4oixptzhovxkbq",
    "ingress-security-rules": [
      {
        "icmp-options": null,
        "is-stateless": false,
        "protocol": "6",
        "source": "<IP_ADDRESS>/8",
        "source-type": "CIDR_BLOCK",
        "tcp-options": null,
        "udp-options": null
      },
      {
        "icmp-options": null,
        "is-stateless": false,
        "protocol": "6",
        "source": "<IP_ADDRESS>/0",
        "source-type": "CIDR_BLOCK",
        "tcp-options": {
          "destination-port-range": {
            "max": 3000,
            "min": 3000
          },
          "source-port-range": null
        },
        "udp-options": null
      }
    ],
    "lifecycle-state": "AVAILABLE",
    "time-created": "2019-06-18T10:59:38.725000+00:00",
    "vcn-id": "ocid1.vcn.oc1.iad.aaaaaaaaxzh7u7tonlrvp4th7wh743auhvvyxmbeuejyryloqrk4vr7doija"
  },
  "etag": "ac4e45d8"
}

There are no egress security lists.
Verify
As I read it the required ports are covered by the broad list in  ClusterSecurityList. I don't thing we need the egress security list as we are not mounting the NFS mount target from outside the cluster VCN.


KODI not loading EPG data.

Hello,
I am using KODI as a client and using the simple IPTV plugin and I have the global list loading but EPG data isn't being loaded, tested this on multiple devices, and it's the same issue.

prajwal

Please precise "global". The full EPG file doesn't exist since some time ago.

Please precise "global". The full EPG file doesn't exist since it was removed some time ago.

How would I get the data if I am using the main list?

Please precise "global". The full EPG file doesn't exist since it was removed some time ago.

How would I get the data if I am using the main list?

You don't. If it doesn't exist, it doesn't get any data.
However if you use only a specific set of channel, just look at the README to get the EPG link dedicated to your set.

Please precise "global". The full EPG file doesn't exist since it was removed some time ago.

How would I get the data if I am using the main list?

You don't. If it doesn't exist, it doesn't get any data.
However if you use only a specific set of channel, just look at the README to get the EPG link dedicated to your set.

Why was it removed?
Also any plans to create a new global list?

Please precise "global". The full EPG file doesn't exist since it was removed some time ago.

How would I get the data if I am using the main list?

You don't. If it doesn't exist, it doesn't get any data.
However if you use only a specific set of channel, just look at the README to get the EPG link dedicated to your set.

Why was it removed?
Also any plans to create a new global list?

Many difficulties due to certain sources having the same channels (so it's basically a choice), EPG size being too big.
No plans are known as of today concerning a global list creation.

Closed in favor #310


Support for TPM Using PyOpenSSL with Azure IoT SDK

First, this is a proof of concept of the implementation. It is stable and we are actively using it in our current project. However, we believe it is necessary to contribute these changes upstream, as they affect two projects: azure-iot-sdk-python to support PyOpenSSL and paho.mqtt.python.

This Draft is mainly for early feedback purpose.

Reference/Link to the issue solved with this PR (if any)

https://github.com/Azure/azure-iot-sdk-python/issues/1093
https://github.com/Azure/azure-iot-sdk-python/issues/278
https://github.com/Azure/azure-iot-sdk-python/issues/1066
https://github.com/Azure/azure-iot-sdk-python/issues/339
https://github.com/Azure/azure-iot-sdk-python/issues/254
https://github.com/Azure/azure-iot-sdk-python/issues/1179

Description of the problem
There is currently no straightforward way to use a TPM with the azure-iot-sdk-python.
Description of the solution
This PR leverages PyOpenSSL to provide a custom context with OpenSSL PKCS#11 support, enabling the use of a TPM. This requires changes to both the azure-iot-sdk-python and paho.mqtt.python to support the PyOpenSSL context.

See https://github.com/eclipse/paho.mqtt.python/pull/849 for related changes in paho.mqtt.python.

@microsoft-github-policy-service agree [company="RMDS"]

+1


Update documentation for change in ci_framework location

Change in location was implemented in [1]
Code "$" were removed to make copy/paste easier but maybe [2] would be
better and it supports striping prompts like $ and >>>.
[1] https://github.com/openstack-k8s-operators/install_yamls/pull/525
[2] https://sphinx-copybutton.readthedocs.io/en/latest/
As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running
[x] Appropriate documentation exists and/or is up-to-date:

[x] Content of the docs/source is reflecting the changes

@lewisdenny Hi. Do you think we should consider all the doc?
[2023-09-11 09:06:24] {afuscoar@afuscoar} (~/.Redhat/Projects/github/ci-framework) (docs_update)$ -> rgrep '$ '
CONTRIBUTING.md:$ make create_new_role ROLE_NAME=my_wonderful_role
CONTRIBUTING.md:$ make run_ctx_pre_commit
CONTRIBUTING.md:$ make run_ctx_ansible_test
CONTRIBUTING.md:$ make run_ctx_molecule MOLECULE_CONFIG=.config/molecule/config_local.yml
CONTRIBUTING.md:$ make enable-git-hooks
ci_framework/roles/copy_container/files/copy-quay/README.rst:    $ go build -o copy-quay main.go copy.go utils.go quayapi.go
ci_framework/roles/copy_container/files/copy-quay/README.rst:    $ podman login quay.io
grep: docs/source/images/logo_cifmw_800.png: binary file matches
grep: docs/source/images/logo_cifmw_200.png: binary file matches
docs/source/quickstart/99_FAQ.md:$ virsh -c qemu:///system list
docs/source/quickstart/99_FAQ.md:$ oc get pods -n openstack
docs/source/quickstart/99_FAQ.md:$ oc get pods -n openstack-operators
docs/source/quickstart/99_FAQ.md:$ oc get pods -n openstack
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system destroy crc
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system undefine crc
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system net-list --all
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system net-destroy <NETWORK>
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system net-undefine <NETWORK>
docs/source/cookbooks/reproducer.md:$ ansible-playbook -K -i inventory_hypervisor.yml \
docs/source/cookbooks/reproducer.md:$ ssh controller-0
docs/source/cookbooks/reproducer.md:$ ansible-playbook -i inventory_hypervisor.yml reproducer-clean.yml
docs/source/cookbooks/reproducer.md:$ virsh net-list --all
docs/source/cookbooks/reproducer.md:$ virsh list --all
docs/source/cookbooks/speed-up-dev.md:$ ansible-playbook deploy-edpm.yml \
docs/source/cookbooks/speed-up-dev.md:$ groups
docs/source/cookbooks/speed-up-dev.md:$ ansible-playbook deploy-edpm.yml \
docs/source/usage/01_usage.md:$ ansible-playbook deploy-edpm.yml -K --tags bootstrap,packages [-e @scenarios/centos-9/some-environment -e <...>]
docs/source/usage/01_usage.md:$ ansible-playbook deploy-edpm.yml -K --skip-tags bootstrap,packages [-e @scenarios/centos-9/some-environment -e <...>]
docs/source/internals/02_testing.md:$ sudo dnf install -y aspell-en
docs/source/internals/02_testing.md:$ pip install pyspelling
docs/source/internals/02_testing.md:$ pyspelling -c .spellcheck.yml
docs/source/internals/02_testing.md:$ cat docs/dictionary/en-custom.txt >> your_temporary_file
docs/source/internals/02_testing.md:$ cat temporary_file | tr '[:upper:]' '[:lower:]' | sort -u > en-custom.txt

@lewisdenny Hi. Do you think we should consider all the doc?
[2023-09-11 09:06:24] {afuscoar@afuscoar} (~/.Redhat/Projects/github/ci-framework) (docs_update)$ -> rgrep '$ '
CONTRIBUTING.md:$ make create_new_role ROLE_NAME=my_wonderful_role
CONTRIBUTING.md:$ make run_ctx_pre_commit
CONTRIBUTING.md:$ make run_ctx_ansible_test
CONTRIBUTING.md:$ make run_ctx_molecule MOLECULE_CONFIG=.config/molecule/config_local.yml
CONTRIBUTING.md:$ make enable-git-hooks
ci_framework/roles/copy_container/files/copy-quay/README.rst:    $ go build -o copy-quay main.go copy.go utils.go quayapi.go
ci_framework/roles/copy_container/files/copy-quay/README.rst:    $ podman login quay.io
grep: docs/source/images/logo_cifmw_800.png: binary file matches
grep: docs/source/images/logo_cifmw_200.png: binary file matches
docs/source/quickstart/99_FAQ.md:$ virsh -c qemu:///system list
docs/source/quickstart/99_FAQ.md:$ oc get pods -n openstack
docs/source/quickstart/99_FAQ.md:$ oc get pods -n openstack-operators
docs/source/quickstart/99_FAQ.md:$ oc get pods -n openstack
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system destroy crc
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system undefine crc
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system net-list --all
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system net-destroy <NETWORK>
docs/source/cookbooks/reproducer.md:$ virsh -c qemu:///system net-undefine <NETWORK>
docs/source/cookbooks/reproducer.md:$ ansible-playbook -K -i inventory_hypervisor.yml \
docs/source/cookbooks/reproducer.md:$ ssh controller-0
docs/source/cookbooks/reproducer.md:$ ansible-playbook -i inventory_hypervisor.yml reproducer-clean.yml
docs/source/cookbooks/reproducer.md:$ virsh net-list --all
docs/source/cookbooks/reproducer.md:$ virsh list --all
docs/source/cookbooks/speed-up-dev.md:$ ansible-playbook deploy-edpm.yml \
docs/source/cookbooks/speed-up-dev.md:$ groups
docs/source/cookbooks/speed-up-dev.md:$ ansible-playbook deploy-edpm.yml \
docs/source/usage/01_usage.md:$ ansible-playbook deploy-edpm.yml -K --tags bootstrap,packages [-e @scenarios/centos-9/some-environment -e <...>]
docs/source/usage/01_usage.md:$ ansible-playbook deploy-edpm.yml -K --skip-tags bootstrap,packages [-e @scenarios/centos-9/some-environment -e <...>]
docs/source/internals/02_testing.md:$ sudo dnf install -y aspell-en
docs/source/internals/02_testing.md:$ pip install pyspelling
docs/source/internals/02_testing.md:$ pyspelling -c .spellcheck.yml
docs/source/internals/02_testing.md:$ cat docs/dictionary/en-custom.txt >> your_temporary_file
docs/source/internals/02_testing.md:$ cat temporary_file | tr '[:upper:]' '[:lower:]' | sort -u > en-custom.txt

Hi @adrianfusco, thanks for checking my PR!
I did not touch others yet as I would like to do that in an immediate follow up PR after some discussion about what route we would like to go.

Leave it as is (In which case I will just add the $ prompts back in)
Remove all $ for easy copy/paste
Add the  copy button plugin [1] and leave the $ as it can be configured to auto strip them

[1] https://sphinx-copybutton.readthedocs.io/en/latest/

@pablintino Could you please remove the hold on this one now that [1] has merged?
[1] https://github.com/openstack-k8s-operators/install_yamls/pull/525


Zookeeper management interface in monitoring center displays abnormal…

… #2547
Tips

Thanks very much for contributing to Apache DolphinScheduler.
Please review https://dolphinscheduler.apache.org/en-us/community/index.html before opening a pull request.

What is the purpose of the pull request
develop issue #2547:Zookeeper management interface in monitoring center displays abnormal.
Brief change log

add zookeeper environment variable ZOO_4LW_COMMANDS_WHITELIST in docker-compose.yml file.
change the data type of minLatency,avgLatency and maxLatency from int to float.

Verify this pull request
(Please pick either of the following options)
This change added tests and can be verified as follows:

Manually verified the change by testing locally.

Codecov Report

Merging #2595 into dev will increase coverage by 0.08%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev    #2595      +/-   ##
============================================
+ Coverage     36.29%   36.37%   +0.08%
- Complexity     2476     2480       +4
============================================
  Files           431      431
  Lines         19838    19838
  Branches       2418     2418
============================================
+ Hits           7201     7217      +16
+ Misses        11996    11977      -19
- Partials        641      644       +3

Impacted Files
Coverage Δ
Complexity Δ

...che/dolphinscheduler/api/utils/ZooKeeperState.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...e/dolphinscheduler/api/utils/ZookeeperMonitor.java
6.25% <0.00%> (ø)
2.00 <0.00> (ø)

...e/dolphinscheduler/dao/entity/ZookeeperRecord.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...he/dolphinscheduler/common/thread/ThreadUtils.java
68.25% <0.00%> (-9.53%)
13.00% <0.00%> (-1.00%)

...nscheduler/server/entity/TaskExecutionContext.java
79.20% <0.00%> (+5.94%)
56.00% <0.00%> (+3.00%)

.../server/worker/processor/TaskExecuteProcessor.java
39.28% <0.00%> (+28.57%)
3.00% <0.00%> (+2.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e08b29...19eb3b5. Read the comment docs.


Improvements to pulling all branches

Right now, the tool doesn't do a great job of getting all of the branches, which as an archival tool is really important to me.
We should extract the functionality to determine remote branches: https://github.com/excellalabs/konmaripo/blob/774ddccd06a6d4e10ede68f4f39bdd3af758c0f0/src/Konmaripo.Web/Services/RepositoryArchiver.cs#L59-L73
And we definitely need to extract and make the name transformation more robust and add tests: https://github.com/excellalabs/konmaripo/blob/774ddccd06a6d4e10ede68f4f39bdd3af758c0f0/src/Konmaripo.Web/Services/RepositoryArchiver.cs#L92-L95
We can use the ithelpbot repo as an example -- here are its remotes:

Based on this, I would expect to see branches in the local repo called:

master
scheduler-script-tweaks
all the dependabot branches, starting with dependabot/.

 I may be able to mirror a repo in a simpler way and get all the tracking branches.


Docs.MS Release Updates for azure_template

Update docs metadata and targeting for release of azure_template

Docs Build status updates of commit 538acd3:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs-ref-services/preview/template-readme.md
:warning:Warning
View (azure-python-preview)
Details

ci-configs/packages-preview.json
:white_check_mark:Succeeded

docs-ref-services/preview/template-readme.md

Line 2, Column 1: [Warning-ms-prod-and-service] Only one of the following attributes can exist: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 9, Column 16: [Warning-ms-prod-technology-invalid] Invalid value for 'ms.technology': 'azure' is not valid with 'ms.prod' value 'azure'.
Line 11, Column 13: [Warning-ms-service-subservice-invalid] Invalid value for 'ms.service': 'template'.
Line 16, Column 1: [Warning-h1-not-first] Markdown content is not allowed before H1 'Azure Template Package client library for Python - Version 0.1.0b687560'.
Line 25, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Getting started') are not allowed. You can only have one top-level heading.
Line 29, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Key concepts') are not allowed. You can only have one top-level heading.
Line 33, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Examples') are not allowed. You can only have one top-level heading.
Line 37, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Troubleshooting') are not allowed. You can only have one top-level heading.
Line 41, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Next steps') are not allowed. You can only have one top-level heading.
Line 45, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Contributing') are not allowed. You can only have one top-level heading.
Line 2, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 25, Column 1: [Suggestion-duplicate-h1s] H1 'Getting started' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(21,1), docs-ref-services/preview/communication-chat-readme.md(22,1), docs-ref-services/preview/communication-sms-readme.md(21,1), docs-ref-services/preview/template-readme.md(25,1)'. First level headings must be unique within a docset.
Line 29, Column 1: [Suggestion-duplicate-h1s] H1 'Key concepts' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(33,1), docs-ref-services/preview/communication-chat-readme.md(85,1), docs-ref-services/preview/communication-sms-readme.md(37,1), docs-ref-services/preview/template-readme.md(29,1)'. First level headings must be unique within a docset.
Line 33, Column 1: [Suggestion-duplicate-h1s] H1 'Examples' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(65,1), docs-ref-services/preview/communication-chat-readme.md(141,1), docs-ref-services/preview/communication-sms-readme.md(42,1), docs-ref-services/preview/template-readme.md(33,1)'. First level headings must be unique within a docset.
Line 37, Column 1: [Suggestion-duplicate-h1s] H1 'Troubleshooting' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(181,1), docs-ref-services/preview/communication-chat-readme.md(398,1), docs-ref-services/preview/communication-sms-readme.md(74,1), docs-ref-services/preview/template-readme.md(37,1)'. First level headings must be unique within a docset.
Line 41, Column 1: [Suggestion-duplicate-h1s] H1 'Next steps' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(184,1), docs-ref-services/preview/communication-chat-readme.md(402,1), docs-ref-services/preview/communication-sms-readme.md(78,1), docs-ref-services/preview/template-readme.md(41,1)'. First level headings must be unique within a docset.
Line 45, Column 1: [Suggestion-duplicate-h1s] H1 'Contributing' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(193,1), docs-ref-services/preview/communication-chat-readme.md(406,1), docs-ref-services/preview/communication-sms-readme.md(82,1), docs-ref-services/preview/template-readme.md(45,1)'. First level headings must be unique within a docset.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 538acd3:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs-ref-services/preview/template-readme.md
:warning:Warning
View (azure-python-preview)
Details

ci-configs/packages-preview.json
:white_check_mark:Succeeded

docs-ref-services/preview/template-readme.md

Line 2, Column 1: [Warning-ms-prod-and-service] Only one of the following attributes can exist: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 9, Column 16: [Warning-ms-prod-technology-invalid] Invalid value for 'ms.technology': 'azure' is not valid with 'ms.prod' value 'azure'.
Line 11, Column 13: [Warning-ms-service-subservice-invalid] Invalid value for 'ms.service': 'template'.
Line 16, Column 1: [Warning-h1-not-first] Markdown content is not allowed before H1 'Azure Template Package client library for Python - Version 0.1.0b687560'.
Line 25, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Getting started') are not allowed. You can only have one top-level heading.
Line 29, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Key concepts') are not allowed. You can only have one top-level heading.
Line 33, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Examples') are not allowed. You can only have one top-level heading.
Line 37, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Troubleshooting') are not allowed. You can only have one top-level heading.
Line 41, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Next steps') are not allowed. You can only have one top-level heading.
Line 45, Column 1: [Warning-multiple-h1s] Multiple H1s(H1 'Contributing') are not allowed. You can only have one top-level heading.
Line 2, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 25, Column 1: [Suggestion-duplicate-h1s] H1 'Getting started' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(21,1), docs-ref-services/preview/communication-chat-readme.md(22,1), docs-ref-services/preview/communication-sms-readme.md(21,1), docs-ref-services/preview/template-readme.md(25,1)'. First level headings must be unique within a docset.
Line 29, Column 1: [Suggestion-duplicate-h1s] H1 'Key concepts' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(33,1), docs-ref-services/preview/communication-chat-readme.md(85,1), docs-ref-services/preview/communication-sms-readme.md(37,1), docs-ref-services/preview/template-readme.md(29,1)'. First level headings must be unique within a docset.
Line 33, Column 1: [Suggestion-duplicate-h1s] H1 'Examples' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(65,1), docs-ref-services/preview/communication-chat-readme.md(141,1), docs-ref-services/preview/communication-sms-readme.md(42,1), docs-ref-services/preview/template-readme.md(33,1)'. First level headings must be unique within a docset.
Line 37, Column 1: [Suggestion-duplicate-h1s] H1 'Troubleshooting' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(181,1), docs-ref-services/preview/communication-chat-readme.md(398,1), docs-ref-services/preview/communication-sms-readme.md(74,1), docs-ref-services/preview/template-readme.md(37,1)'. First level headings must be unique within a docset.
Line 41, Column 1: [Suggestion-duplicate-h1s] H1 'Next steps' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(184,1), docs-ref-services/preview/communication-chat-readme.md(402,1), docs-ref-services/preview/communication-sms-readme.md(78,1), docs-ref-services/preview/template-readme.md(41,1)'. First level headings must be unique within a docset.
Line 45, Column 1: [Suggestion-duplicate-h1s] H1 'Contributing' is duplicated with other articles: 'docs-ref-services/preview/communication-administration-readme.md(193,1), docs-ref-services/preview/communication-chat-readme.md(406,1), docs-ref-services/preview/communication-sms-readme.md(82,1), docs-ref-services/preview/template-readme.md(45,1)'. First level headings must be unique within a docset.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Increment decimal

Create a new question "Increment decimal".
Fix #127

Let me know when @arbiter527 has finished reviewing and I'll take a look.

It looks like this functionality is duplicated in #175. Which of your groups claimed the question first?

Oops. But I think we created the issue first at  #127 .  And #175 is linked to #133 .

@eyurko please take another look at the changes :)

Also, you should check to make sure the Travis build has succeeded before you LGTM -- currently, you're failing some linter checks.

@eyurko @arbiter527  Let's close this pr. We have merged our test cases to #175


Which token is necessary?

Hi,
I have doubts about which token set in plugin to make it work. According to the documentation, for versions higher than 4.11.0 in Android and 12.2.0 in iOS, you have to use the sdk tokens and you cannot use the api token. However, for the plugin the api token is necessary, if not, when putting a generated token (sdk token) it fails to take the photos.
Do you know if plugin oonly works with API token?

Finally I've found a solution. You can generate the token with the onfido endpoint for generate sdk tokens (https://documentation.onfido.com/#generate-sdk-token) and once you've generated token you can start the plugin with this token. But you have to change the way builder starts. It is necessary to use the 'withSDKToken()' function instead of 'withToken()' and remove the function 'withApplicatn()' because of is required by sdk.


Release v0.1.4

Release Notes
Another day, another release. These are the release notes for the version v0.1.4.
Other Changes

ci: version bump to v0.1.4-dev (Lacework)(3c8803b)

nope, this was a misclick


Generation speed

Thanks for your cool work, I tried some samples and the performance is really good and love the token-saving. But seems the decoding process is too low compared to others, is there any possible method to accelerate the decoding into <1s for decoding around 10s speech?  Many thanks!

@yaotingwangofficial That might need some extra optimization on the engineering side. Since this is only a research-based project, so I do not conduct thorough optimization on the inference speed. You are most welcome to contribute on that if you want of course.

Thanks!


What do I do when my web scrobbler isn't scrobbling?

It shows it's scrobbling but it doesn't show in my profile, I used scrobbler last month and it was working, but I changed my last.fm account and I know I used to do something when I change my account but I don't remember what to do. Can someone help me?

Have you tried re-authenticating your LastFM account? Click on the scrobbler icon top right of browser and select Options. Scroll all the way to the bottom of the page and click on the button labelled "Re-authenticate with Last.fm"

@kyurawx1 as @RadianM said before, you should reauthenticate via a button placed in the options page.
I also made a notification of service errors, so users will see a message when scrobbing isn't working due to authentication errors. This feature will be available in the next release. I hope it will help users to understand why the extension doesn't work properly.


'Session is closed' errors

Hi,
I am a really happy user of telepot. Amazing library, amazing code. I am using it for a async DelegatorBot-based bot.
The thing is that since three days ago I am getting random 'Session is closed' errors. The bot works normally until without any warning starts prompting this errors:

RuntimeError: Session is closed
Traceback (most recent call last):
File "/home/ubuntu/politibot/lib/python3.5/site-packages/aiohttp/client.py", line 187, in _request
yield from resp.start(conn, read_until_eof)
File "/home/ubuntu/politibot/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 606, in start
message = yield from httpstream.read()
File "/home/ubuntu/politibot/lib/python3.5/site-packages/aiohttp/streams.py", line 591, in read
result = yield from super().read()
File "/home/ubuntu/politibot/lib/python3.5/site-packages/aiohttp/streams.py", line 446, in read
yield from self._waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in iter
yield self  # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 297, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
aiohttp.errors.ServerDisconnectedError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/politibot/lib/python3.5/site-packages/telepot/async/init.py", line 228, in get_from_telegram_server
result = await self.getUpdates(offset=offset, timeout=20)
File "/home/ubuntu/politibot/lib/python3.5/site-packages/telepot/async/init.py", line 168, in getUpdates
return await self._api_request('getUpdates', _rectify(p))
File "/home/ubuntu/politibot/lib/python3.5/site-packages/telepot/async/init.py", line 38, in _api_request
return await api.request((self._token, method, params, files), **kwargs)
File "/home/ubuntu/politibot/lib/python3.5/site-packages/telepot/async/api.py", line 108, in request
async with fn(*args, **kwargs) as r:
File "/home/ubuntu/politibot/lib/python3.5/site-packages/aiohttp/client.py", line 539, in aenter
self._resp = yield from self._coro
File "/home/ubuntu/politibot/lib/python3.5/site-packages/aiohttp/client.py", line 150, in _request
raise RuntimeError('Session is closed')

Do you have any idea of what can it be? Is anyone around having the same issue?

I do have similar errors.
File: /usr/lib/python3.5/ssl.py, Line: 571, Method: read, Exception: ('Connection aborted.', ConnectionResetError(104, 'Die Verbindung wurde vom Kommunikationspartner zurückgesetzt'))
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 595, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 393, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 389, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.5/ssl.py", line 925, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.5/ssl.py", line 787, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 571, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
self._bot.sendMessage(receiverID, message)
File "/usr/local/lib/python3.5/dist-packages/telepot-8.1-py3.5.egg/telepot/init.py", line 235, in sendMessage
return self._api_request('sendMessage', _rectify(p))
File "/usr/local/lib/python3.5/dist-packages/telepot-8.1-py3.5.egg/telepot/init.py", line 224, in _api_request
return api.request((self._token, method, params, files), **kwargs)
File "/usr/local/lib/python3.5/dist-packages/telepot-8.1-py3.5.egg/telepot/api.py", line 130, in request
r = fn(*args, **kwargs)  # fn must be thread-safe
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/request.py", line 151, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/poolmanager.py", line 248, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 640, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/util/retry.py", line 238, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 595, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 393, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/usr/local/lib/python3.5/dist-packages/urllib3-1.16-py3.5.egg/urllib3/connectionpool.py", line 389, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.5/ssl.py", line 925, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.5/ssl.py", line 787, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 571, in read
v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Die Verbindung wurde vom Kommunikationspartner zurückgesetzt'))
After this i got this in my log:
2016-06-30 12:29:12 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 13:02:45 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 13:36:18 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 14:09:51 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 14:43:24 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 15:16:57 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 15:50:29 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 16:24:03 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 16:57:36 INFO     Thread-6     Resetting dropped connection: api.telegram.org
2016-06-30 17:31:09 INFO     Thread-6     Resetting dropped connection: api.telegram.org

My Prob seems to be like in #87
Using 7.1. now....

This is happening to me. My bots run for a day or two until it happens.

As detailed in #87, it may be the version of urllib3. Try downgrade it to 1.9.1, and see what happen.

As mentioned in #87, making retries=3 solves the problem.


Segmentation fault when trying to enter the shell of the main branch.

Issue statement
Greetings.
I'm trying to get into the shell of the main branch and the command pscale shell <database> main started throwing
Error: signal: segmentation fault (core dumped)

Although, the command seem to work fine with other branches.
For some context, I'm using the CLI inside a newly spun container based on the Ubuntu 20:04 image. Authentication and deploy request were the commands made prior to this error and those were successful.
Screenshot

Version
pscale version 0.68.0 (build date: 2021-08-19T11:23:27Z commit: 6c41383)

Hi @kasipavankumar
It looks like something with your container is wrong. Can you please run it with our official docker container and let us know if you see any issues? :
$ docker pull planetscale/pscale:v0.68.0
$ docker run -it
  --env PLANETSCALE_SERVICE_TOKEN= <token>
  --env PLANETSCALE_SERVICE_TOKEN_NAME=<name>
  --env PLANETSCALE_ORG=<org>
  planetscale/pscale  shell links main

Hi. Sure let me

Hi.
Sure, let me try it.

Error cannot be reproduced anymore.
Maybe, it was some container configuration issue.
Thanks.


Documentation improvements

Any issues with the readme or suggestions about how to improve documentation in general please chuck them here!
Info needed:

[ ] Launch options (debug, skip intro, etc)
[ ] Deployment / dev workflow

Need to document how the animation queue works.

Reference this excellent README - https://github.com/gkjohnson/three-bvh-csg?tab=readme-ov-file


Does Ergo have a plan to support OTP25/OTP26?

Discussed in https://github.com/ergo-services/ergo/discussions/123

Originally posted by leonlee2013 December  4, 2022
OTP-26 is scheduled to be released on May 1, 2023
If use Ergo to communicate with Erlang, Erlang cannot be upgraded to OTP25 or above, which is a limitation to Ergo's development.
Is there any plan for Ergo to support OTP25/26?

OTP 25 is now supported in v2.2.1


Fix type conversions.

They were not correct before. Also clarified the two types of conversions:

TryConvertImplicit implements the implicit conversions allowed by the C# language
TryConvert tries every means at its disposal to convert a value to a type

AvaloniaObject uses only implicit conversions. This allows one to write:
var control = new TextBlock
{
    [Canvas.TopProperty] = 10
}

Without implicit conversions, this would fail because Canvas.Top is a double whereas 10 is an int, however only implicit conversions should be used here, otherwise the following would pass, which is probably not what would be wanted:
var control = new TextBlock
{
    [TextBlock.TextProperty] = observable, // Text is now the type name of `observable`
}

DefaultValueConverter uses TryConvert, i.e. every conversion possible.
Fixes #972.

@kekekeks any idea why the skia tests are failing?


feat: support arbitrary length async contracts

BREAKING CHANGE: the various AsyncContract2, AsyncContract3, ... types have all been replaced with a single AsyncContract type that takes a tuple of Runtype meaning parameters and a Runtype meaning a return type of Promise.

Coverage increased (+0.005%) to 99.196% when pulling 76cb6ab5a60bc06e28661996a0c4c036117084f4 on yuhr:feat/arbitrary-length-async-contracts into<PHONE_NUMBER>79fba77a709050b4d0b393fb1dadc6 on pelotom:master.


firewall related issues and network connectivity with latest git version

Server running on Ubuntu 14.01, I installed docker following docker installation document https://docs.docker.com/installation/ubuntulinux/ for latest version in docker
Got a 1.4 docker version
The jenkins official image is not able to access external world when the firewall is based on DROP main policy.
Everything is fine when  I get back to earlier versions of docker

Seems to be a docker issue then, please ask on docker mailing list


”Podeni” is listed in the drop-down menu

Description:
"Podeni" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Podeni" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Podeni, MEHEDINȚI".


feat: permanently save logs/events

Can we update the scope to modules/blockchain_connector here?

It touches multiple scopes

Yes good point. Ignore my comment! :)

@jrainville exactly, keep the event between stop and start and also low memory.
The problem of switching to not async is concurrent write.
So if we switch to async I feel we should do the queue too.

Ah ok, I understand your point with the async. But I feel like it would be cleaner and better to do it anyway. Especially with the async library, it's really simple.
Just use a cargo to "queue" the write jobs for you, then you write to the file in one go: http://caolan.github.io/async/docs.html#cargo

Fair enough, I ll make the changes

async cargo may be the way to go, but I was wondering whether this might be a good use case for fs.createWriteStream.

@michaelsbradleyjr I had thought about fs.createWriteStream and I wonder what would be the algorithm you have in mind?
1 - close the stream each time you finish the write?
I wasn't use this solution is really a win
2 - keep it open while embark is running
My problem with this one is that that I need to have a json format.
3 - something else?
IMO createWriteStream would be an alternative if just using append.
Anything better?

@jrainville @michaelsbradleyjr I have added the queue (works better than cargo)
If you have an idea for createWriteStream let me know an I will change.
I also added a commit for start:embark

@alaibe, @jrainville: here's my attempt at an implementation with a Writable stream:
https://github.com/embark-framework/embark/compare/feature/permanent-save-log-event...feature/permanent-save-log-event-WriteStream
A few things to note:

Per my discussion with @alaibe, there seems to be a problem with event logging on this PR branch, so my branch only has changes for the console listener.
Node's docs for the write method indicate a stream will buffer if it's not draining; so, I'm not sure if implementing n queue with an array was really necessary or provides any benefits for us, but that's just how I did it.
Node's docs for fs.createWriteStream indicate that autoClose is enabled by default, and will trigger if there's an error, closing the file descriptor. My code doesn't listen for the error event, but maybe that should be added and on error we could create a fresh file, not sure what's best.
I'm not super happy with the loops of 1 ms timers (to avoid reader/writer collision). It would be better to have logFileWriting and logFileReading implemented as observables and await on a promise that resolves when the observable has an event... something like that. But that's something to improve in the future, I think.
If we're streaming the writes, maybe we can stream the reads too, piping them to the res object for more efficient delivery?
The log file isn't really valid JSON or NDJSON. Should we give it a different file extension, maybe just .log?

I made some important changes in _saveLog so that the queue is used much more effectively, i.e. pending writes don't pile up looped timers, they just write to the queue. In fact, I was able to get rid of the looped time _saveLog, it's only needed in _readLogs now.

@jrainville PR updated with cargo
@michaelsbradleyjr I had a good look at your implementation.
IMO it is way to much complexity for the usage. Also with the file queue you have re-implemented exactly what cargo does. The boolean: this.logFileReading won't work you need a lock strategy here in order to be really efficient I think.
I also feel that json makes it more easy, it was the main reason why I didn't go for Stream in the first place.

@alaibe The queue I implemented is definitely similar. I think going with a simple, straightforward solution like cargo is worthwhile, but I still have some questions.

_readLogs is implemented with readFileSync, and every call of _saveLog() results in a call of _readLogs(). Is the idea with readFileSync to avoid reader-writer collision?

If so, and if it's the only approach that's practical for low complexity, then maybe it's just as well that we use writeJsonSync, since every call of _readLogs() will block the process anyway.

If the time stamps are tossed in the end, and since Object.values() doesn't guarantee key ordering (though maybe it works in practice), how about encoding as an array of objects (JSON.stringify('[{},{},{}]'))?

New entries would simply be pushed onto the end of the array.

With every call of _saveLog() we're reading and writing the entire log. It can definitely work at a small scale, but if the log sticks around for awhile and/or if the logging rate is high for some reason, won't that become a bottleneck?

Perhaps at some point we should investigate use of fs-ndjson or similar.

For my own understanding, can you explain why this.logFileReading doesn't work?

I experimented in VS Code's debugger and it did seem to work (reading blocking writing, and vice versa). It's a given that my simple locking strategy can't work across multiple instances of ConsoleListener accessing the same file, and it provides no protection re: other code/processes that might read/write to the same file.

I have learned that in the ES6 specification, object property traversal became well defined:
http://2ality.com/2015/10/property-traversal-order-es6.html
So please that aspect of my question/concern about using an array vs. an object.

@michaelsbradleyjr

_readLogs is implemented with readFileSync, and every call of _saveLog() results in a call of _readLogs(). Is the idea with readFileSync to avoid reader-writer collision?

Definitively

If so, and if it's the only approach that's practical for low complexity, then maybe it's just as well that we use writeJsonSync, since every call of _readLogs() will block the process anyway.

The only advantage of cargo is that if you have 10 simultaneous request, it will result in 1 read/1write
because it will pack the request together. See this link for an explanation of cargo:
https://camo.githubusercontent.com/f4810e00e1c5f5f8addbe3e9f49064fd5d102699/68747470733a2f2f662e636c6f75642e6769746875622e636f6d2f6173736574732f313637363837312f36383130312f38346339323036362d356632392d313165322d383134662d3964336430323431336266642e676966

If the time stamps are tossed in the end, and since Object.values() doesn't guarantee key ordering (though maybe it works in practice), how about encoding as an array of objects (JSON.stringify('[{},{},{}]'))?
New entries would simply be pushed onto the end of the array.

You reply to this one yourself ;)

With every call of _saveLog() we're reading and writing the entire log. It can definitely work at a small scale, but if the log sticks around for awhile and/or if the logging rate is high for some reason, won't that become a bottleneck?
Perhaps at some point we should investigate use of fs-ndjson or similar.

I definitively agree that it won't work very well if we start to have Gb of data.
However those logs are save only in development when there is tx. I doubt we will have someone with Gb of data any time soon.
Also if you look at the read, it reads the full file, this is definitively not scalable.
We would need to introduce pagination/cursor IMO.
I didn't do it because the way we handle the storage is temporary. If we introduce sqlite for example, most of the problems are solved.

For my own understanding, can you explain why this.logFileReading doesn't work?
I experimented in VS Code's debugger and it did seem to work (reading blocking writing, and vice versa). It's a given that my simple locking strategy can't work across multiple instances of ConsoleListener accessing the same file, and that it provides no protection re: other code/processes that might read/write to the same file.

Here we are talking about race condition. I have personally never being able to reproduce a race condition with a debugger. The reason for that is that usually those errors happens at micro second speeds and when you debug you reduce the speed.
Also note that with one user (the developer), the race condition is unlikely to happen.
Here is why I think the locking strategy won't work:

save_log is called and go to line 150
read_log is called and go to line 176
From there you can see that whatever happens next you can have a read and write simultaneously