KeyError when Running the "run_lm_finetuning.py" Script

I encountered an issue while trying to run the "run_lm_finetuning.py" script from the repository. The error message indicates a KeyError, which means that a specified key was not found in the "reverse_label_dict" dictionary.
I suspect that the issue may be related to the initialization of the "reverse_label_dict" or the data used as input to the script. However, I'm not entirely sure what's causing the problem.
Any help in identifying the cause of this issue and suggesting possible solutions would be greatly appreciated. Let me know if you need any further information to investigate the problem. Thank you!
  File "/style-transfer-paraphrase/style_paraphrase/run_lm_finetuning.py", line 505, in <module>
    main()
  File "/style-transfer-paraphrase/style_paraphrase/run_lm_finetuning.py", line 417, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/style-transfer-paraphrase/style_paraphrase/run_lm_finetuning.py", line 68, in load_and_cache_examples
    dataset = InverseParaphraseDatasetText(
  File "/style-transfer-paraphrase/style_paraphrase/style_dataset.py", line 169, in __init__
    self.examples = limit_styles(self.examples, args.specific_style_train, split, self.reverse_label_dict)
  File "/style-transfer-paraphrase/style_paraphrase/data_utils.py", line 189, in limit_styles
    logger.info("Preserving authors = {}".format(", ".join([reverse_label_dict[x] for x in specific_style_train])))
  File "/style-transfer-paraphrase/style_paraphrase/data_utils.py", line 189, in <listcomp>
    logger.info("Preserving authors = {}".format(", ".join([reverse_label_dict[x] for x in specific_style_train])))
KeyError: 1```

Hi @BenKabongo25 which dataset were you running this on? What did you set --specific_style_train to? The value of --specific_style_train should an integer less than the number of total styles in your dataset file.

Thanks for your answer, @martiansideofthemoon.
I use a custom dataset.
The first small dataset I run on has 5221 items in train, 294 in test and 294 in evaluation. I therefore set the --specific_style_train parameter to 5221.
Do you think I should change it?
PS: I'm currently doing an internship with the MLIA/ISIR team and I'm working on personalized data-to-text: we want the output of our models to be close to the style of a particular user.
There is no such thing as a personalized data-to-text dataset. And to create one, I use your STRAP model.
So I'm training STRAP models on user reviews, and then I want to use the trained models to generate custom descriptions for the data-to-text.

How many different styles did you have in the training data? --specific_style_train refers to the index of the style. So if you have two styles in the training data, the only possible values for this are 0 and 1.

I run different scripts with numbers of styles varying between 2 and 5.
Indeed, that was the problem. I fixed it and the current scripts seem to run correctly.
Last question: does it make sense to run the same script for a single style?
Thank you very much.

Yes you can run the script for a single style as well, it will learn an inverse paraphraser for that style in isolation.


(Jubilee 2021.0.0-M1 + 2.6.0-M1) to (Jubilee 2021.0.0-M1 + 2.6.0-M2) The dependencies of some of the beans in the application context form a cycle: org.springframework.cloud.config.server.environment.vault.SpringVaultClientConfiguration

A small issue observed after migration.
Disclaimer: I am filing this issue on Spring Cloud Config Server, but if you believe this issue should be for Spring Cloud, or Spring Boot, please let me know, I will close and amend.
Background:
I had a very simple Spring Cloud Config Server application: with version Jubilee 2021.0.0-M1 + 2.6.0-M1. Working fine, very happy.
@EnableConfigServer
@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class);
    }

}

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.6.0-M1</version>
        <relativePath/>
    </parent>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>2021.0.0-M1</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

The M2 got announced here: https://spring.io/blog/2021/08/19/spring-boot-2-6-0-m2-is-now-available
Therefore, I went to do the version bump up to Jubilee 2021.0.0-M1 + 2.6.0-M2 a bit excited!
Unfortunately, after the version bump, application cannot startup, with this error
***************************
APPLICATION FAILED TO START
***************************

Description:

The dependencies of some of the beans in the application context form a cycle:

   pathMappedEndpoints defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/WebEndpointAutoConfiguration.class]
      ↓
   healthEndpoint defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]
      ↓
   healthContributorRegistry defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]
      ↓
   configServerHealthIndicator defined in class path resource [org/springframework/cloud/config/server/config/EnvironmentRepositoryConfiguration$ConfigServerActuatorConfiguration.class]
      ↓
   org.springframework.cloud.config.server.config.CompositeConfiguration
      ↓
   vaultEnvironmentRepository defined in class path resource [org/springframework/cloud/config/server/config/SpringVaultRepositoryConfiguration.class]
      ↓
   vaultEnvironmentRepositoryFactory defined in class path resource [org/springframework/cloud/config/server/config/EnvironmentRepositoryConfiguration$SpringVaultFactoryConfig.class]
┌─────┐
|  org.springframework.cloud.config.server.environment.vault.SpringVaultClientConfiguration
└─────┘

Action:

Relying upon circular references is discouraged and they are prohibited by default. Update your application to remove the dependency cycle between beans. As a last resort, it may be possible to break the cycle automatically be setting spring.main.allow-circular-references to true if you have not already done so.

Fore sure, it is the version bump, as I just change one number in the whole project (revert back to 2.6.0-M1 = work 2.6.0-M2 = does not work)
May I ask what is the issue? Maybe some workaround?
Hope this issue is helpful, thank you for your time.

Can you try 2021.0.0-SNAPSHOT?

Hello,
What kind of magic is that!
Indeed, issue resolved.
Thanks @ryanjbaxter

(If I may ask, what is the root cause, and can the fix of this issue be expected for 2.6.0-M3 please?)

There was a change in Spring Boot
https://github.com/spring-projects/spring-boot/issues/27652
Our next Spring Cloud 2021.0.0 milestone will be compatible.
You can try using Spring Cloud 2021.0.0-M1 and set spring.main.allow-circular-references=true and see if that makes things work temporarily.

Hello @ryanjbaxter,
As announce today https://spring.io/blog/category/releases  "Spring Boot 2.6.0-M3 available now"
This is super exciting, thank you Spring Team.
Unfortunately, even after bumping to
<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.6.0-M3</version>
        <relativePath/>
    </parent>

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>2021.0.0-M1</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

The problem still persists.
May I ask what is the correct way please? I was hoping this M3 release would fix the issue.
Thank you for your time

The next Spring Cloud Milestone will address this, until that is released you will need to continue to use the property above.
The milestone should be released this week.

Hello @ryanjbaxter,
Thank you for your input.
Confirmed, I just updated and the issue is resolved.
Thank you!


Project doesn't build when using Nix + Cabal?

Tried building the project from the Nix shell using Cabal:
[nix-shell:~/projects/haskell/cardano-db-sync]$ cabal build all
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-b_-c97d51a3264adedf'...
remote: Enumerating objects: 103, done.
remote: Counting objects: 100% (103/103), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 1268 (delta 36), reused 57 (delta 20), pack-reused 1165
Receiving objects: 100% (1268/1268), 303.02 KiB | 954.00 KiB/s, done.
Resolving deltas: 100% (640/640), done.
HEAD is now at 1222078 Update dependency on cardano-prelude (#86)
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-c_-613c120a593cd275'...
remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 1457 (delta 5), reused 20 (delta 2), pack-reused 1422
Receiving objects: 100% (1457/1457), 461.02 KiB | 263.00 KiB/s, done.
Resolving deltas: 100% (611/611), done.
HEAD is now at 2547ad1 Merge pull request #65 from dcoutts/dcoutts/drop-vrf-p256-openssl
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-l_-eee106153f3e350d'...
remote: Enumerating objects: 187, done.
remote: Counting objects: 100% (187/187), done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 14119 (delta 77), reused 126 (delta 56), pack-reused 13932
Receiving objects: 100% (14119/14119), 3.87 MiB | 1.20 MiB/s, done.
Resolving deltas: 100% (8611/8611), done.
HEAD is now at 22e89a5 Update to latest cardano-ledger-specs (#754)
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-l_-afc0f56298b7d554'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 22507 (delta 13), reused 19 (delta 2), pack-reused 22459
Receiving objects: 100% (22507/22507), 21.55 MiB | 1.23 MiB/s, done.
Resolving deltas: 100% (13583/13583), done.
HEAD is now at 28b43814 Merge pull request #1335 from input-output-hk/jc/store-prev-proto-params
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-n_-695a8c1c408f8118'...
remote: Enumerating objects: 148, done.
remote: Counting objects: 100% (148/148), done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 12773 (delta 50), reused 83 (delta 19), pack-reused 12625
Receiving objects: 100% (12773/12773), 4.50 MiB | 1.21 MiB/s, done.
Resolving deltas: 100% (8976/8976), done.
HEAD is now at b5b0a29 Merge #707
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-p_-6616435bbe3d1976'...
remote: Enumerating objects: 1201, done.
remote: Total 1201 (delta 0), reused 0 (delta 0), pack-reused 1201
Receiving objects: 100% (1201/1201), 289.69 KiB | 840.00 KiB/s, done.
Resolving deltas: 100% (534/534), done.
HEAD is now at 3ac22a2 Merge #103
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/cardano-s_-40b8d61132c3cbd1'...
remote: Enumerating objects: 139, done.
remote: Counting objects: 100% (139/139), done.
remote: Compressing objects: 100% (99/99), done.
remote: Total 2176 (delta 69), reused 71 (delta 30), pack-reused 2037
Receiving objects: 100% (2176/2176), 2.37 MiB | 1.19 MiB/s, done.
Resolving deltas: 100% (1154/1154), done.
HEAD is now at bc3563c Update CODEOWNERS (#357)
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/goblins-f4f7645a78b07334'...
remote: Enumerating objects: 410, done.
remote: Counting objects: 100% (410/410), done.
remote: Compressing objects: 100% (181/181), done.
remote: Total 423 (delta 167), reused 395 (delta 153), pack-reused 13
Receiving objects: 100% (423/423), 119.14 KiB | 709.00 KiB/s, done.
Resolving deltas: 100% (167/167), done.
HEAD is now at 26d35ad Add SeedGoblin instance for `Maybe a`
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/iohk-moni_-c4029568d4836e27'...
remote: Enumerating objects: 340, done.
remote: Counting objects: 100% (340/340), done.
remote: Compressing objects: 100% (123/123), done.
remote: Total 8725 (delta 194), reused 256 (delta 154), pack-reused 8385
Receiving objects: 100% (8725/8725), 34.76 MiB | 1.21 MiB/s, done.
Resolving deltas: 100% (4955/4955), done.
HEAD is now at 10877fb Merge pull request #534 from input-output-hk/cad-667-multiple-trace-acceptors
Cloning into '/home/ksaric/projects/haskell/cardano-db-sync/dist-newstyle/src/ouroboros_-86f4da3e33dbe3ab'...
remote: Enumerating objects: 2983, done.
remote: Counting objects: 100% (2983/2983), done.
remote: Compressing objects: 100% (660/660), done.
remote: Total 51096 (delta 2395), reused 2803 (delta 2265), pack-reused 48113
Receiving objects: 100% (51096/51096), 23.34 MiB | 1.23 MiB/s, done.
Resolving deltas: 100% (30957/30957), done.
HEAD is now at e71aa72a Merge #1844
Warning: No remote package servers have been specified. Usually you would have
one specified in the config file.
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: cardano-binary-test-1.3.0 (user goal)
[__1] unknown package: quickcheck-instances (dependency of
cardano-binary-test)
[__1] fail (backjumping, conflict set: cardano-binary-test,
quickcheck-instances)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cardano-binary-test,
quickcheck-instances

Had a similar problem building with Stack, will submit a PR for Stack.

cc @angerman

I think we've agreed to make sure the cabal works (and stack is a bonus).  We need to do this for the new Shelley version (#66)

I think this is due to a mis-feature of haskell,nix. I have quizzed @angerman .

There is a possible solution in https://github.com/input-output-hk/cardano-db-sync/pull/68 .

This is obviously not "high priority" nor is is "shelley testnet" specific.

The possible solution in #68 has been merged. I am assuming this is now fixed.


Email / other notification sends when job is approved

Investigate and integrate a way for job posters to receive an email / other notification which triggers as soon as their job is approved/merged. This notification would ideally include the live job post and the forum discussion thread that is created along with some guidance on how to make their job a successful one.

Blocked by: https://github.com/opensourcedesign/opensourcedesign.github.io/issues/393


Added a few popular CDN providers

Added popular "other" CDN providers in the service matrix section.

These are good additions - thank you @lobster1234!

Glad to help!


Add Chinese

Signed-off-by: Sunbreak<EMAIL_ADDRESS>

@Sunbreak @bagaswastu
You still need to add the chinese option in lib/main.dart.

@Sunbreak @bagaswastu
You still need to add the chinese option in lib/main.dart.

@Sunbreak @bagaswastu
You still need to add the chinese option in lib/main.dart.

Didn't know that. flutter run stops at splash page. Maybe VPN is needed to access Library Genesis

@Sunbreak @bagaswastu
You still need to add the chinese option in lib/main.dart.

Didn't know that. flutter run stops at splash page. Maybe VPN is needed to access Library Genesis

@Sunbreak @bagaswastu
You still need to add the chinese option in lib/main.dart.

Didn't know that. flutter run stops at splash page. Maybe VPN is needed to access Library Genesis

I've already fixed the splash screen issue, it should be work now

@Sunbreak @bagaswastu
You still need to add the chinese option in lib/main.dart.

Didn't know that. flutter run stops at splash page. Maybe VPN is needed to access Library Genesis

I've already fixed the splash screen issue, it should be work now

@Sunbreak
I didn't get Chinese to work on my emulator. Maybe it's because there are multiple Chinese options?
On the Flutter documentation there is more info. Maybe you can look into it.

@Sunbreak
I didn't get Chinese to work on my emulator. Maybe it's because there are multiple Chinese options?
On the Flutter documentation there is more info. Maybe you can look into it.

@Sunbreak
I didn't get Chinese to work on my emulator. Maybe it's because there are multiple Chinese options?
On the Flutter documentation there is more info. Maybe you can look into it.

Checkout https://github.com/bagaswastu/librebook/pull/7

@Sunbreak
I didn't get Chinese to work on my emulator. Maybe it's because there are multiple Chinese options?
On the Flutter documentation there is more info. Maybe you can look into it.

Checkout https://github.com/bagaswastu/librebook/pull/7


fix(askvm): call

Closes #46
Closes #68
TODO:

[ ] fix lambda-04 test

Abandoned in favour of https://github.com/xFAANG/askql/pull/178


convert the model to onnx and torchscript

Hello, I have some difficulties converting your model both to onnx and to torchscript. I've read closed issues already but there isn't any help. Could you help me? Or, may be, someone succeeded to do that. Below I'll show code and errors I get after I try to convert the model.
2nd_model.pt  is the full model saved in pytorch. Works fine after loading in python.
Converting the model to onnx
model = torch.load("2nd_model.pt").to(device)
model.eval()
torch.onnx.export(
    model,
    sample,
    "model.onnx",
    opset_version=11,
    export_params=True)

Error:
/content/Regression-model/lib/models/laneatt.py:230: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  n_proposals = len(self.anchors)
/content/Regression-model/lib/models/laneatt.py:235: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for batch_idx, img_features in enumerate(features):
/content/Regression-model/lib/models/laneatt.py:82: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  attention = softmax(scores).reshape(x.shape[0], len(self.anchors), -1)
/content/Regression-model/lib/models/laneatt.py:83: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  attention_matrix = torch.eye(attention.shape[1], device=x.device).repeat(x.shape[0], 1, 1)
/content/Regression-model/lib/models/laneatt.py:87: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  batch_anchor_features = batch_anchor_features.reshape(x.shape[0], len(self.anchors), -1)
/content/Regression-model/lib/models/laneatt.py:116: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for proposals, attention_matrix in zip(batch_proposals, batch_attention_matrix):
/content/Regression-model/lib/models/laneatt.py:127: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if proposals.shape[0] == 0:
/usr/local/lib/python3.7/dist-packages/torch/onnx/symbolic_opset9.py:2766: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  "If indices include negative values, the exported graph will produce incorrect results.")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-d8ff0aee41a0> in <module>()
      6     "model.onnx",
      7     opset_version=11,
----> 8     export_params=True)

8 frames
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py in _graph_op(g, opname, *raw_args, **kwargs)
    888     if _onnx_shape_inference:
    889         from torch.onnx.symbolic_helper import _export_onnx_opset_version as opset_version
--> 890         torch._C._jit_pass_onnx_node_shape_type_inference(n, _params_dict, opset_version)
    891
    892     if outputs == 1:

RuntimeError: input_shape_value == reshape_value || input_shape_value == 1 || reshape_value == 1INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp":520, please report a bug to PyTorch. ONNX Expand input shape constraint not satisfied.

 If indices include negative values, the exported graph will produce incorrect results.
  "If indices include negative values, the exported graph will produce incorrect results.")
Traceback (most recent call last):
  File "converting_model.py", line 29, in <module>
    export_params=True)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/__init__.py", line 208, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 530, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 384, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 188, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/__init__.py", line 241, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 787, in _run_symbolic_function
    symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 745, in _find_symbolic_in_registry
    return sym_registry.get_registered_op(op_name, domain, opset_version)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/symbolic_registry.py", line 109, in get_registered_op
    raise RuntimeError(msg)
RuntimeError: Exporting the operator eye to ONNX opset version 11 is not supported. Please open a bug to request ONNX export support for the missing operator.

Scripting the model
model = torch.load("2nd_model.pt").to(device)
model.eval()
traced_script_module = torch.jit.script(model)

Error:
---------------------------------------------------------------------------
FrontendError                             Traceback (most recent call last)
<ipython-input-17-59a3d0112d6d> in <module>()
      1 model = torch.load("2nd_model.pt").to(device)
      2 model.eval()
----> 3 traced_script_module = torch.jit.script(model)

4 frames
/usr/local/lib/python3.7/dist-packages/torch/jit/annotations.py in check_fn(fn, loc)
    132     if len(py_ast.body) == 1 and isinstance(py_ast.body[0], ast.ClassDef):
    133         raise torch.jit.frontend.FrontendError(
--> 134             loc, f"Cannot instantiate class '{py_ast.body[0].name}' in a script function")
    135     if len(py_ast.body) != 1 or not isinstance(py_ast.body[0], ast.FunctionDef):
    136         raise torch.jit.frontend.FrontendError(loc, "Expected a single top-level function")

FrontendError: Cannot instantiate class 'Softmax' in a script function:
  File "/content/Regression-model/lib/models/laneatt.py", line 80

        # Add attention features
        softmax = nn.Softmax(dim=1)
                  ~~~~~~~~~~ <--- HERE
        scores = self.attention_layer(batch_anchor_features)
        attention = softmax(scores).reshape(x.shape[0], len(self.anchors), -1)

Here is the colab link also of the above code link

Maybe you should convert the NMS function to python code first. I can convert the model to onnx after rewriting that function.
My problem is I get this error when I use Onnxruntime to inference onnx model:
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Node (Reshape_914) Op (Reshape) [ShapeInferenceError] Invalid position of 0
Also I'm new to ONNX, so I have no idea about how to fix this.==

Hey @sevocrear, how exactly have you managed to export the model to ONNX? I've tried removing NMS and exporting the model without it (I'd do it after computing the output), but I keep getting the error:
RuntimeError: input_shape_value == reshape_value || input_shape_value == 1 || reshape_value == 1INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp":520, please report a bug to PyTorch. ONNX Expand input shape constraint not satisfied.

Hey @sevocrear, how exactly have you managed to export the model to ONNX? I've tried removing NMS and exporting the model without it (I'd do it after computing the output), but I keep getting the error:
RuntimeError: input_shape_value == reshape_value || input_shape_value == 1 || reshape_value == 1INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp":520, please report a bug to PyTorch. ONNX Expand input shape constraint not satisfied.

Hi  @tersekmatija , have you removed NMS totally? In my implementation, I just changed the CUDA NMS to python NMS function (which, of course, is much slower). And then I could convert it to ONNX. But the inference speed was awful so I use python original code with some changes regarding my project. Me and @Lllllp93  described the changed parts and requirements above

Hello, I have some difficulties converting your model both to onnx and to torchscript. I've read closed issues already but there isn't any help. Could you help me? Or, may be, someone succeeded to do that. Below I'll show code and errors I get after I try to convert the model.
2nd_model.pt is the full model saved in pytorch. Works fine after loading in python.
Converting the model to onnx
model = torch.load("2nd_model.pt").to(device)
model.eval()
torch.onnx.export(
    model,
    sample,
    "model.onnx",
    opset_version=11,
    export_params=True)

Error:
/content/Regression-model/lib/models/laneatt.py:230: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  n_proposals = len(self.anchors)
/content/Regression-model/lib/models/laneatt.py:235: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for batch_idx, img_features in enumerate(features):
/content/Regression-model/lib/models/laneatt.py:82: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  attention = softmax(scores).reshape(x.shape[0], len(self.anchors), -1)
/content/Regression-model/lib/models/laneatt.py:83: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  attention_matrix = torch.eye(attention.shape[1], device=x.device).repeat(x.shape[0], 1, 1)
/content/Regression-model/lib/models/laneatt.py:87: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  batch_anchor_features = batch_anchor_features.reshape(x.shape[0], len(self.anchors), -1)
/content/Regression-model/lib/models/laneatt.py:116: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for proposals, attention_matrix in zip(batch_proposals, batch_attention_matrix):
/content/Regression-model/lib/models/laneatt.py:127: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if proposals.shape[0] == 0:
/usr/local/lib/python3.7/dist-packages/torch/onnx/symbolic_opset9.py:2766: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  "If indices include negative values, the exported graph will produce incorrect results.")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-d8ff0aee41a0> in <module>()
      6     "model.onnx",
      7     opset_version=11,
----> 8     export_params=True)

8 frames
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py in _graph_op(g, opname, *raw_args, **kwargs)
    888     if _onnx_shape_inference:
    889         from torch.onnx.symbolic_helper import _export_onnx_opset_version as opset_version
--> 890         torch._C._jit_pass_onnx_node_shape_type_inference(n, _params_dict, opset_version)
    891
    892     if outputs == 1:

RuntimeError: input_shape_value == reshape_value || input_shape_value == 1 || reshape_value == 1INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp":520, please report a bug to PyTorch. ONNX Expand input shape constraint not satisfied.

 If indices include negative values, the exported graph will produce incorrect results.
  "If indices include negative values, the exported graph will produce incorrect results.")
Traceback (most recent call last):
  File "converting_model.py", line 29, in <module>
    export_params=True)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/__init__.py", line 208, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 530, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 384, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 188, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/__init__.py", line 241, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 787, in _run_symbolic_function
    symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 745, in _find_symbolic_in_registry
    return sym_registry.get_registered_op(op_name, domain, opset_version)
  File "/home/sevocrear/.local/lib/python3.6/site-packages/torch/onnx/symbolic_registry.py", line 109, in get_registered_op
    raise RuntimeError(msg)
RuntimeError: Exporting the operator eye to ONNX opset version 11 is not supported. Please open a bug to request ONNX export support for the missing operator.

Scripting the model
model = torch.load("2nd_model.pt").to(device)
model.eval()
traced_script_module = torch.jit.script(model)

Error:
---------------------------------------------------------------------------
FrontendError                             Traceback (most recent call last)
<ipython-input-17-59a3d0112d6d> in <module>()
      1 model = torch.load("2nd_model.pt").to(device)
      2 model.eval()
----> 3 traced_script_module = torch.jit.script(model)

4 frames
/usr/local/lib/python3.7/dist-packages/torch/jit/annotations.py in check_fn(fn, loc)
    132     if len(py_ast.body) == 1 and isinstance(py_ast.body[0], ast.ClassDef):
    133         raise torch.jit.frontend.FrontendError(
--> 134             loc, f"Cannot instantiate class '{py_ast.body[0].name}' in a script function")
    135     if len(py_ast.body) != 1 or not isinstance(py_ast.body[0], ast.FunctionDef):
    136         raise torch.jit.frontend.FrontendError(loc, "Expected a single top-level function")

FrontendError: Cannot instantiate class 'Softmax' in a script function:
  File "/content/Regression-model/lib/models/laneatt.py", line 80

        # Add attention features
        softmax = nn.Softmax(dim=1)
                  ~~~~~~~~~~ <--- HERE
        scores = self.attention_layer(batch_anchor_features)
        attention = softmax(scores).reshape(x.shape[0], len(self.anchors), -1)

Here is the colab link also of the above code link

have you found the way of  convert model to torchscript

Hey @sevocrear, how exactly have you managed to export the model to ONNX? I've tried removing NMS and exporting the model without it (I'd do it after computing the output), but I keep getting the error:
RuntimeError: input_shape_value == reshape_value || input_shape_value == 1 || reshape_value == 1INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/passes/onnx/shape_type_inference.cpp":520, please report a bug to PyTorch. ONNX Expand input shape constraint not satisfied.

have you sloved this problem?

hey, i used you code try to convert ,but i got some error,and i can not find the reason,please help!
/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py:170: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
r_filters = np.zeros(len(scores))
/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py:178: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if i == (len(scores) - 1):  # break if indice is the last one
/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py:199: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
end = torch.min(torch.min(end_a, end_b), torch.tensor(self.n_offsets - 1))
Traceback (most recent call last):
File "main.py", line 63, in
main()
File "main.py", line 59, in main
runner.pytorch2onnx(epoch=args.epoch or exp.get_last_checkpoint_epoch())
File "/home/hw/project/LaneDetection/lane-att/lib/runner.py", line 191, in pytorch2onnx
torch.onnx.export(model,  # model being run
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/onnx/init.py", line 203, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/onnx/utils.py", line 86, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/onnx/utils.py", line 526, in _export
graph, params_dict, torch_out = _model_to_graph(model, args, verbose, input_names,
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/onnx/utils.py", line 366, in _model_to_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/onnx/utils.py", line 319, in _trace_and_get_graph_from_model
torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/jit/init.py", line 338, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/jit/init.py", line 421, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/jit/init.py", line 412, in wrapper
outs.append(self.inner(*trace_inputs))
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 720, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/home/hw/anaconda3/envs/laneatt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 704, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 109, in forward
proposals_list = self.nms2(reg_proposals, attention_matrix, nms_thres, nms_topk, conf_threshold)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 157, in nms2
keep, num_to_keep, _ = self.Lane_nms(proposals, scores, overlap=nms_thres, top_k=nms_topk)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 182, in Lane_nms
r_filter = self.Lane_IOU(proposals[indice, :], proposals[sub_indice, :], overlap)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 199, in Lane_IOU
end = torch.min(torch.min(end_a, end_b), torch.tensor(self.n_offsets - 1))
RuntimeError: Expected object of scalar type float but got scalar type long int for argument 'other'

@Lllllp93 , thanks for the code! But I noticed that one of your function didn't have return so I added it. Finally, I could convert the model to onnx. And it runs with onnxruntime. I use:

python3.8
torch1.8.2
cuda10.2

Here is the related part of laneatt.py file code
<\>
    def forward(self, x, conf_threshold=0.2, nms_thres=50.0, nms_topk=5):
        print(nms_thres, conf_threshold)
        batch_features = self.feature_extractor(x)

        batch_features = self.conv1(batch_features)
        batch_anchor_features = self.cut_anchor_features(batch_features)
        # Join proposals from all images into a single proposals features batch
        batch_anchor_features = batch_anchor_features.view(-1, self.anchor_feat_channels * self.fmap_h)

        # Add attention features
        scores = self.attention_layer(batch_anchor_features)
        attention = self.softmax(scores).reshape(x.shape[0], len(self.anchors), -1)
        # attention_matrix = torch.diag(torch.ones(attention.shape[1], device=x.device)).repeat(x.shape[0], 1, 1)
        attention_matrix = torch.eye(attention.shape[1], device=x.device).repeat(x.shape[0], 1, 1)
        non_diag_inds = torch.nonzero(attention_matrix == 0., as_tuple=False)
        attention_matrix[:] = 0
        attention_matrix[non_diag_inds[:, 0], non_diag_inds[:, 1], non_diag_inds[:, 2]] = attention.flatten()
        batch_anchor_features = batch_anchor_features.reshape(x.shape[0], len(self.anchors), -1)
        attention_features = torch.bmm(torch.transpose(batch_anchor_features, 1, 2),
                                       torch.transpose(attention_matrix, 1, 2)).transpose(1, 2)
        attention_features = attention_features.reshape(-1, self.anchor_feat_channels * self.fmap_h)
        batch_anchor_features = batch_anchor_features.reshape(-1, self.anchor_feat_channels * self.fmap_h)
        batch_anchor_features = torch.cat((attention_features, batch_anchor_features), dim=1)

        # Predict
        cls_logits = self.cls_layer(batch_anchor_features)
        reg = self.reg_layer(batch_anchor_features)

        # Undo joining
        cls_logits = cls_logits.reshape(x.shape[0], -1, cls_logits.shape[1])
        reg = reg.reshape(x.shape[0], -1, reg.shape[1])

        # Add offsets to anchors
        reg_proposals = torch.zeros((*cls_logits.shape[:2], 5 + self.n_offsets), device=x.device)
        reg_proposals += self.anchors
        reg_proposals[:, :, :2] = cls_logits
        reg_proposals[:, :, 4:] += reg

        # Apply nms
        proposals_list = self.nms(reg_proposals, attention_matrix, nms_thres, nms_topk, conf_threshold)
        return proposals_list

    def nms(self, batch_proposals, batch_attention_matrix, nms_thres, nms_topk, conf_threshold):
        softmax = nn.Softmax(dim=1)
        proposals_list = []
        for proposals, attention_matrix in zip(batch_proposals, batch_attention_matrix):
            anchor_inds = torch.arange(batch_proposals.shape[1], device=proposals.device)
            # The gradients do not have to (and can't) be calculated for the NMS procedure
            with torch.no_grad():
                scores = softmax(proposals[:, :2])[:, 1]
                if conf_threshold is not None:
                    # apply confidence threshold
                    above_threshold = scores > conf_threshold
                    proposals = proposals[above_threshold]
                    scores = scores[above_threshold]
                    anchor_inds = anchor_inds[above_threshold]
                if proposals.shape[0] == 0:
                    proposals_list.append((proposals[[]], self.anchors[[]], attention_matrix[[]], None))
                    continue
                # keep, num_to_keep, _ = nms(proposals, scores, overlap=nms_thres, top_k=nms_topk)
                keep, num_to_keep, _ = self.Lane_nms(proposals, scores, overlap=nms_thres, top_k=nms_topk)
                keep = keep[:num_to_keep]
            proposals = proposals[keep]
            anchor_inds = anchor_inds[keep]
            attention_matrix = attention_matrix[anchor_inds]
            proposals_list.append((proposals, self.anchors[keep], attention_matrix, anchor_inds))

        return proposals_list

    def Lane_nms(self,proposals,scores,overlap=50, top_k=4):
        keep_index = []
        print(scores)
        sorted_score, indices = torch.sort(scores, descending=True) # from big to small
        r_filters = np.zeros(len(scores))

        for i,indice in enumerate(indices):
            if r_filters[i]==1: # continue if this proposal is filted by nms before
                continue
            keep_index.append(indice)
            if len(keep_index)>top_k: # break if more than top_k
                break
            if i == (len(scores)-1):# break if indice is the last one
                break
            sub_indices = indices[i+1:]
            for sub_i,sub_indice in enumerate(sub_indices):
                r_filter = self.Lane_IOU(proposals[indice,:],proposals[sub_indice,:],overlap)
                if r_filter: r_filters[i+1+sub_i]=1
        num_to_keep = len(keep_index)
        keep_index = list(map(lambda x: x.item(), keep_index))
        return keep_index, num_to_keep, None

    def Lane_IOU(self,parent_box, compared_box, threshold):
        '''
        calculate distance one pair of proposal lines
        return True if distance less than threshold
        '''
        start_a = (parent_box[2] * self.n_strips + 0.5).int() # add 0.5 trick to make int() like round
        start_b = (compared_box[2] * self.n_strips + 0.5).int()
        start = torch.max(start_a,start_b)
        end_a = start_a + parent_box[4] - 1 + 0.5 - (((parent_box[4] - 1)<0).int())
        end_b = start_b + compared_box[4] - 1 + 0.5 - (((compared_box[4] - 1)<0).int())
        end = torch.min(torch.min(end_a,end_b),torch.tensor(self.n_offsets-1))
        # end = torch.min(torch.min(end_a,end_b),torch.FloatTensor(self.n_offsets-1, device = torch.device('cpu')))
        if (end - start)<0:
            return False
        dist = 0
        for i in range(5+start,5 + end.int()):
            # if i>(5+end):
            #     break
            if parent_box[i] < compared_box[i]:
                dist += compared_box[i] - parent_box[i]
            else:
                dist += parent_box[i] - compared_box[i]
        return dist < (threshold * (end - start + 1))

    def loss(self, proposals_list, targets, cls_loss_weight=10):
      <\>

Here is the part of code how I export it and run with onnx
torch.onnx.export(model,               # model being run
                images,                         # model input (or a tuple for multiple inputs)
                "laneatt.onnx",   # where to save the model (can be a file or file-like object)
                export_params=True,        # store the trained parameter weights inside the model file
                opset_version=11)

onnx_model = onnx.load("laneatt.onnx")
onnx.checker.check_model(onnx_model)

ort_session = onnxruntime.InferenceSession("laneatt.onnx")

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()
# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(images)}
ort_outs = ort_session.run(None, ort_inputs)
# compare ONNX Runtime and PyTorch results
onnx_output = [tuple(map(lambda x: torch.tensor(x), ort_outs))]
prediction = model.decode(onnx_output, as_lanes=True)

But the results are different from the original nms implementation
This is the visualization of the model output with original NMS:  This is the visualization of the model output with custom NMS given by @Lllllp93 :  conf_thrresh and nms_thresh are the same.
So, now I'm investigating the code why results are so different (but actually 3 lines are almost totally the same). Would be great if you point on differences between your nms implementation and @lucastabelini one. Also, it's 20 times slower than original one (just for notion). Tested on my pc with NVIDIA RTX 2080Ti 11Gb, Intel® Core™ i9-9820X CPU @ 3.30GHz × 20, 62,5 GiB RAM

hey, @sevocrear ,i used you code try to convert ,but i got some error,and i can not find the reason,please help!
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 109, in forward
proposals_list = self.nms2(reg_proposals, attention_matrix, nms_thres, nms_topk, conf_threshold)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 157, in nms2
keep, num_to_keep, _ = self.Lane_nms(proposals, scores, overlap=nms_thres, top_k=nms_topk)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 182, in Lane_nms
r_filter = self.Lane_IOU(proposals[indice, :], proposals[sub_indice, :], overlap)
File "/home/hw/project/LaneDetection/lane-att/lib/models/laneatt.py", line 199, in Lane_IOU
end = torch.min(torch.min(end_a, end_b), torch.tensor(self.n_offsets - 1))
RuntimeError: Expected object of scalar type float but got scalar type long int for argument 'other'

hey @lucastabelini sorry to bother you,  i have some difficulties in covert the model to onnx,so i have an idea that remove the nms program in the project,because in my project just need detect one lane . do you think this idea is ok ?

Hi @tersekmatija , have you removed NMS totally? In my implementation, I just changed the CUDA NMS to python NMS function (which, of course, is much slower). And then I could convert it to ONNX. But the inference speed was awful so I use python original code with some changes regarding my project. Me and @Lllllp93 described the changed parts and requirements above

Yes @sevocrear . I just tried removing the NMS in the forwards pass, this is the snapshot:
        # Add offsets to anchors
        reg_proposals = torch.zeros((*cls_logits.shape[:2], 5 + self.n_offsets), device=x.device)
        reg_proposals += self.anchors
        reg_proposals[:, :, :2] = cls_logits
        reg_proposals[:, :, 4:] += reg
        # nms removed here
        return reg_proposals

But I am still getting the error. Do you know which version of Python and Torch did you use if it might depend on this?

hey @lucastabelini sorry to bother you, i have some difficulties in covert the model to onnx,so i have an idea that remove the nms program in the project,because in my project just need detect one lane . do you think this idea is ok ?

If you only need to detect one lane boundary (i.e., one line), then you can do that. Just get the one with the highest confidence score.

hey @lucastabelini sorry to bother you, i have some difficulties in covert the model to onnx,so i have an idea that remove the nms program in the project,because in my project just need detect one lane . do you think this idea is ok ?

If you only need to detect one lane boundary (i.e., one line), then you can do that. Just get the one with the highest confidence score.

@lucastabelini ok,i will try in my project,thanks!

@hwang12345 It seems that @sevocrear uses it in his forward method and he managed to export it.

@tersekmatija i had try @sevocrear push code ,but i did not succeed.

@hwang12345  It seems like you use torch.min() to compared different data type tensors as the error message shows, self.n_offsets in your code may be a long tensor. You can just  convert it to Float or just use torch.FloatTensor(self.n_offsets-1).


GCP CloudSQL: Proxy connections

What problem are you facing?

This is related to connecting to CloudSQL instances from an application managed by Crossplane.
@ichekrygin and I were talking about different connectivity models for accessing CloudSQL instances from managed applications, and I want to capture some of the notes from that discussion. Maybe it makes sense to turn this into a one-pager in the future. This is all early thinking, and is not necessarily well thought out yet : ).
The question is: how do we connect to a Crossplane-managed CloudSQL instance from a managed application? Because it's a managed application and a managed database, the following requirements apply:

The specification should not be provider-specific; it should be portable
The specification should be simple

There is also a consideration of supporting connections between an application in a non-GCP provider and a database in GCP. But I consider this more of a tradeoff between approaches than a requirement.
Proposal
The discussion here will explore the CloudSQL Proxy option. The reason for focusing on the proxy approach is the following positive tradeoffs:

On the application side, the proxy is portable across all environments and providers, and it sidesteps any network connectivity questions
The proxy model is similar to access models for other services (e.g. Redis), and could inform our thinking around other proxy types.

From the application configuration side, the naive approach is to configure the proxy next to the application, which is provider-specific and not portable. The portable way to do this would be for the application to receive a connection string which points it to the cloudsql proxy. This is also a better developer experience.
To determine the connection string, the most likely scenario is that the crossplane controller would need to set up a cloudsql proxy container for the application, and would need to return a connection string that could be injected into the application container's environment. In order for the controller to know how to configure the proxy container, it would need to know which database the application is trying to connect to.
To summarize, the proposed model for interacting with a CloudSQL instance using a proxy would be:

The application configuration specifies which database it depends on (at the crossplane level, so for example, the name of a crossplane-managed database claim)
The controller managing KubernetesApplications (or maybe KubernetesApplicationResources; I'm not sure at this time) spins up a CloudSQL proxy for an application which declares that it depends on a particular database (if that database is bound to a CloudSQL instance). This could also be a controller which is separate from those other controllers. Maybe it goes as far as being modeled as a claim on a database connection, for example. Or maybe the same controller which manages a database claim handles this.
The aforementioned controller exposes the connection string for the application to consume
The application is now able to consume the connection string in the regular way

Further reading and related issues

About the CloudSQL Proxy
Resource connectivity model: #606

I also have not thought this through particularly thoroughly, but I've previously imagined a supplement to Crossplane that was capable of automatically injecting proxies (CloudSQL, service mesh sidecars, etc) into workloads (perhaps using a mutating webhook) by intelligently analysing their connectivity needs as declared (somehow) by Crossplane. My suspicion is that this functionality would be best if it were not baked into Crossplane but was instead an optional addon.


(gdk): Directories are not cleaned up when the --name option is specified but the template does not exist

Describe the bug
Directories are not cleaned up when the --name option is specified but the template does not exist
To Reproduce
Run gdk component init -l java -t template -n nonexistent
Expected behavior
The directory named nonexistent should not be created because the template does not exist.
Actual behavior
gdk will first create the directory nonexistent before determining that the template does not exist.

Hi Tim,
Thanks for trying our CLI tool. I see room for improvement in the customer experience here but we'll need to prioritize and work on it.  However, this can be categorized as an improvement to the customer experience rather than a bug. Let us know what you think.

Fixed this as part of GDK 1.3.0. With this change, a new directory is created only when a template or repository exists when a project is initiated with --name argument. Closing this out.


Can't use "Options pattern" with generic host

Can't understand how to grab data using  "Options pattern" from appsettings.json file.
I have next code (part of Main method):
var environment = EnvironmentName.Development;

            var config = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json", optional: true)
                .AddJsonFile(
                        $"appsettings.{environment}.json",
                        optional: true)
                .AddCommandLine(args)
                .AddEnvironmentVariables()
                .Build();

            var host = new HostBuilder()
                .ConfigureAppConfiguration(configApp => {
                    configApp.AddConfiguration(config);
                })
                .ConfigureLogging((hostContext, configLogging) =>
                {
                    configLogging.AddConsole();
                    configLogging.AddDebug();
                })
                .ConfigureServices((hostContext, services) => {
                    services.AddLogging();
                    services.AddHostedService<SomeService>();
                    services.AddSingleton<ISomeHelperService, SomeHelperService>();
                    services.Configure<MyOptions>(config);
                })
                .UseEnvironment(environment)
                .Build();

My app settings file contains
{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "AllowedHosts": "*",
  "option1": "value1_from_json",
  "option2": -1
}

Options class
public class MyOptions
    {
        public string Option1 { get; set; }
        public int Option2 { get; set; }
    }

SomeService looks like
internal class SomeService : IHostedService
    {
        private readonly ILogger _logger;
        private readonly MyOptions _myOptions;
        private readonly ISomeHelperService _someHelperService;

        public SynchronizationService(ISomeHelperService someHelperService, ILogger<SomeService> logger, IOptions<MyOptions> optionsAccessor)
        {
            _someHelperService = someHelperService;
            _logger = logger;
            _myOptions = optionsAccessor.Value;
        }

        public Task StartAsync(CancellationToken cancellationToken)
        {
            return Task.CompletedTask;
        }

        public Task StopAsync(CancellationToken cancellationToken)
        {
            return Task.CompletedTask;
        }
    }

When I run app _myOptions' fields have default values, but have to hold values from settings file.
What can be the reason of this issue and how to solve it ?
Thanks a lot for any answer !

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c5f1eeee-d500-82bf-0d0a-b4e0d5d5c175
Version Independent ID: 2373a3df-6ea3-50c7-d63d-61c51d572511
Content: .NET Generic Host
Content Source: aspnetcore/fundamentals/host/generic-host.md
Product: aspnet-core
GitHub Login: @guardrex
Microsoft Alias: riande

Ping @HaoK

@andriialbumentormate I am able to repro the behavior you're seeing here with one of my test apps.

@Tratcher Are u aware of any gotchas for options and the Generic Host?

Nevermind ... I think this is just the 'old base path situation.
@andriialbumentormate Can you set ...
.SetBasePath(Directory.GetCurrentDirectory())

... and try again.

I'd like to surface this in the options topic where we show the use of the appsettings.json file for the sample app.

Everything all figured out here @guardrex ?  Or did you have something you wanted me to look at still?

Thanks @HaoK ... I've done the same thing as @andriialbumentormate ... twice! I think we're good here. I'm just waiting on confirmation from @andriialbumentormate that the app is working there with the base path set.

@guardrex , hi !
I've added SetBasePath to my app:
var config = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: true)
                .AddJsonFile(
                        $"appsettings.{environment}.json",
                        optional: true)
                .AddCommandLine(args)
                .AddEnvironmentVariables()
                .Build();

in configure services I have the same
.ConfigureServices((hostContext, services) => {
                    // add services
                    services.Configure<MyOptions>(config);
                })

But still can't retrieve values from appsettings.json file.
Probably I do something wrong ?

This project is just console app and in main method I configure host and run it.
appsettings.json and appsettings.environment.json files are in root of my project.

I run this app using VS 2017 (the latest update)
The path when I use Directory.GetCurrentDirectory() points to D:\....\Proj\bin\Debug\netcoreapp2.1

The only one way in which I can retrieve some data from appsettings.json is to set Copy to Output Directory property to Copy Always.

If I do like this I can get option1 and option2 from settings even without using of SetBasePath(Directory.GetCurrentDirectory())
{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "AllowedHosts": "*",
  "option1": "value1_from_json",
  "option2": -1
}

My question - did I do something wrong when I read https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-2.1 ?
Thanks for help and any answers !

Oh ... I thought u were working with the Web SDK (<Project Sdk="Microsoft.NET.Sdk.Web">). SetBasePath is effective with the Web SDK but not a console app that sets the SDK to Microsoft.NET.Sdk. You'll probably need move files (settings files or any other types of developer-added files) manually. You may have noticed that when you set the build action that it adds the following to to your project file ...
<ItemGroup>
  <None Update="appsettings.json">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
</ItemGroup>

We don't document this requirement in the ASP.NET Core docs because this doc set is very focused on the Web SDK and web apps in general. If you change your SDK over to the Web SDK, then you'll see that you won't need to move the settings file manually when you have SetBasePath present.

@guardrex  thanks a lot for quick answer and clarification !

@guardrex want to mention that in example https://github.com/aspnet/Docs/blob/master/aspnetcore/fundamentals/host/generic-host/samples/2.x/GenericHostSample/GenericHostSample.csproj you use Sdk="Microsoft.NET.Sdk".
Thanks a lot !

Good point. @Tratcher SetBasePath isn't effective without the Web SDK AFAICT. What's our guidance for the Generic Host in this circumstance where the appsettings file(s) need to be moved and the app SDK is Microsoft.NET.Sdk?

Same deal with the ConfigJson sample ... https://github.com/aspnet/Docs/blob/master/aspnetcore/fundamentals/configuration/index/sample/ConfigJson/Program.cs . That sample project has a project SDK of Microsoft.NET.Sdk. It doesn't run in VS ...

Change the SDK to Microsoft.NET.Sdk.Web and :tada: ...

@guardrex probably better to mention this somewhere in generic host article or in another one. Because this issue (don't know how to name it correctly) can confuse.

The short answer is that we haven't given any guidance here yet. I suggest manually marking those config assets as <CopyToOutputDirectory>Always</CopyToOutputDirectory>.

In that case, I'll add a bit to the Generic Host topic to cover this. The project SDK is Microsoft.NET.Sdk, so I agree with @andriialbumentormate that readers need a little coverage on this.
As for the config topic, I'll add it there, too. We're planning a major config overhaul where the sample will go to RP and this won't be an issue, but I don't know exactly when that overhaul will occur.


Ignore undefined values rather than throw exceptions

Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":

Search the issues already opened: https://github.com/googleapis/nodejs-firestore/issues
Search the issues on our "catch-all" repository: https://github.com/googleapis/google-cloud-node
Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js

If you are still having issues, please be sure to include as much information as possible:
Environment details

OS: Mac
Node.js version: 12
npm version: 6.13.6
@google-cloud/firestore version: 3.7.4"

Steps to reproduce

Insert using fstore.collection(blah).doc(foo).set({"hello": undefined})

Expected
Like JSON.stringify, undefined keys are ignored.
Actual
Error: Value for argument "data" is not a valid Firestore document. Cannot use "undefined" as a Firestore value (found in field "hello").
Explanations
undefines are a routine part of js. Not gracefully handling undefines means firestore becomes a brittle library that barfs every now and then. The developer experience could be much better.
A quick dirty workaround is doc.set(JSON.parse(JSON.stringify(val)), which is yuck. The library can handle this elegantly and preserve JSON.stringify like behaviour.

@nojvek Thanks for this request! This has come up a couple of times. I will talk with the team to see if this is a behavior we can support.

If you're open to a PR, I can submit one. I did a log of debugging yest to see where this was failing.

This is more of a political challenge rather than a technical one, as this might also lead to unexpected behavior. Stripping undefined breaks the assumption that data the read from Firestore is the same as the data written to, and as such I am expecting some pushback. If we do decide to change the behavior, then I would be more than happy to accept a PR.

That's definitely a two edged sword - I personally like the current behavior and I saw a couple of instances where it saved people from writing invalid data to the DB. We could also expose it as an option, so that people can decide for themselves.

Yeah mongodb exposes an ignoreUndefined option as part of its connection
settings and I’ve been a big fan of that.
https://mongodb.github.io/node-mongodb-native/2.2/reference/connecting/connection-settings/
I can understand this being a polarized discussion by making it default

We will add an option to the SDK to turn on the requested behavior.

@nojvek If you have time, can you take a look at https://github.com/googleapis/nodejs-firestore/pull/1062 and let us know if this addresses your needs?

As of today May 29, 2020, Firebase Added support for calling FirebaseFiresore.settings with { ignoreUndefinedProperties: true }. When this parameter is set, Cloud Firestore ignores undefined properties inside objects rather than rejecting the API call.

I'm posting this is a work-around that if you're for whatever reason unable to update your legacy firebase version and can't access the ignoreUndefinedProperties argument.
Here's a similar custom version set that does the same thing:
export const ignoreUndefinedSet = async (
  // depending on your namespace & /types version, you can use firebase.firestore.<name> instead
  reference: FirebaseFirestore.DocumentReference,
  data: FirebaseFirestore.DocumentData,
  options?: FirebaseFirestore.SetOptions,
) => {
  const onlyDefinedProperties = Object.entries(data).reduce(
      (result, [key, value]) => (
           value === undefined
                ? result
                : Object.assign(result, { [key]: value })
       ),
      {}
  )
  await reference.set(onlyDefinedProperties, { ...options })
}

So essentially these two statements are equivalent
await reference.set(data, { ignoreUndefinedProperties: true }) // newer firebase version

ignoreUndefinedSet(reference, data) // my polyfill

As of today May 29, 2020, Firebase Added support for calling FirebaseFiresore.settings with { ignoreUndefinedProperties: true }. When this parameter is set, Cloud Firestore ignores undefined properties inside objects rather than rejecting the API call.

YAY

wait ... this only works for root level properties? what in the ........... google u mad

This should work for nested undefined values as well.

For anyone using the v9 API:
import { getFirestore, connectFirestoreEmulator, initializeFirestore } from 'firebase/firestore'; // Firebase v9+

  // Must be called before getFirestore()
  initializeFirestore(app, {
    ignoreUndefinedProperties: true
  });
  const firestore = getFirestore(app);

initializeFirestore(app, {
ignoreUndefinedProperties: true
});

Uncaught FirebaseError: initializeFirestore() has already been called with different options. To avoid this error, call initializeFirestore() with the same options as when it was originally called, or call getFirestore() to return the already initialized instance.


A few portal gun enhhancements

Added animations and sounds
Uses Portal or Portal 2 view/worldmodels when mounted
Prediction

this looks sick, will accept pr when I get the time to test it


Bypass unsupported Tag Exceptions through config if implementation pending

I came across situation where one of the m3u8 had tag EXT-X-PROGRAM-DATE-TIME which is part of spec [https://tools.ietf.org/html/draft-pantos-http-live-streaming-16#section-<IP_ADDRESS>]
Since library is not complete it ends up throwing exception.  Adding parsing for all such tags is going to take a while. Till then is there better alternative than throwing  exception?
Can one bypass that [through config maybe]? Can you help with example of LENIENT mode.
Here us sample m3u8 which fails to parse
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:NO
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:11
#EXT-X-PROGRAM-DATE-TIME:1970-01-01T00:00:00Z
#EXTINF:10.010010,no-desc
Fragments(video=0,format=m3u8-aapl-v3,audiotrack=aac_UND_2_127)
#EXTINF:10.010010,no-desc
Fragments(video=100100100,format=m3u8-aapl-v3,audiotrack=aac_UND_2_127)
#EXTINF:10.010010,no-desc
Fragments(video=200200200,format=m3u8-aapl-v3,audiotrack=aac_UND_2_127)
#EXTINF:10.010010,no-desc
Fragments(video=300300300,format=m3u8-aapl-v3,audiotrack=aac_UND_2_127)

Ok. Got the LENIENT mode. RTFM :)

:thumbsup:


TypeError: Object of type KeycloakRole is not JSON serializable

Whenever I try to create an user with an initial role, I get the error: "TypeError: Object of type KeycloakRole is not JSON serializable"

Hi @fherrera124, unfortunately I wasn't able to reproduce the error. Could you please provide the whole user you're trying to register and at which point you get the serializing error?

Sure, this is my function:
@app.post("/users", tags=["user-management"])
def create_user(first_name: str, last_name: str, email: str, password: SecretStr, initial_roles: List[str]):
    return idp.create_user(first_name=first_name, last_name=last_name,
                           username=email, email=email, password=password.get_secret_value(),
                           initial_roles=initial_roles, send_email_verification=False)

This is the list containing a valid role:
[
  "SUBCOORDINADOR_REGIONAL"
]

The output of the error:
File "/home/server/dockers/xxxx/app/app.py", line 57, in create_user
    return idp.create_user(first_name=first_name, last_name=last_name,
  File "/home/server/.cache/pypoetry/virtualenvs/xxxx-7DESS4RH-py3.8/lib/python3.8/site-packages/fastapi_keycloak/api.py", line 57, in wrapper
    result: Response = f(*args, **kwargs)  # The actual call
  File "/home/server/.cache/pypoetry/virtualenvs/xxxx-7DESS4RH-py3.8/lib/python3.8/site-packages/fastapi_keycloak/api.py", line 686, in create_user
    response = self._admin_request(url=self.users_uri, data=data, method=HTTPMethod.POST)
  File "/home/server/.cache/pypoetry/virtualenvs/xxxx-7DESS4RH-py3.8/lib/python3.8/site-packages/fastapi_keycloak/api.py", line 913, in _admin_request
    return requests.request(method=method.name, url=url, data=json.dumps(data), headers=headers)
  File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type KeycloakRole is not JSON serializable

Seems like the response by Keycloak isn't valid JSON or mismatches the Pydantic model. Could you please debug into the Keycloak response (E.g. breakpoint at line 59 of fastapi_keycloak/api.py) and provide a (safe) version of the response provided by Keycloak?

The object returned:
[KeycloakRole(id='xxxxxx2', name='SUBCOORDINADOR_REGIONAL', composite=False, clientRole=False, containerId='Test')]

This was indeed an issue. The initial roles were directly dumped to JSON from the Pydantic model which didn't work quite well. Should be fixed in 1.0.2 I just released

Thanks, great work.

This was indeed an issue. The initial roles were directly dumped to JSON from the Pydantic model which didn't work quite well. Should be fixed in 1.0.2 I just released

Just tested, it always returns error 500 :/
So, for now, first I'm creating the user, and then adding the role to that user (yesterday it worked).
PD: New bug introduced, if there are no initial roles, now fails with this:
File "/usr/local/lib/python3.8/site-packages/fastapi_keycloak/api.py", line 664, in create_user
    initial_roles_json = [role.dict() for role in initial_roles]
TypeError: 'NoneType' object is not iterable

Turns out I tested the wrong version 🥲 Also, the Keycloak documentation was misleading so the clientRoles field was totally wrong, the realmRoles field is the correct one. Should be fixed once and for all in v1.0.3.

Thanks, the errors are gone now. But from the KeyCloak Admin Console I'm seeing that the new users don't get the initial roles. It's no big deal for me, I will stick with the procedure of creating users without roles, and then add them.

Seems like a old know bug

Hmm, seems like that's the case... thanks for pointing that out. For convenience, I'll change the API to make additional requests for the initial roles for now

I'll not say "it's fixed" anymore. But I think it's finally working as expected. Though I had to add the initial roles in additional requests.

May also be related to https://github.com/keycloak/keycloak/issues/9342. Further introspection required.


fix: correct preOrderIndex and postOrderIndex

What kind of change does this PR introduce?
close #18008
This PR recalculates preOrderIndex and postOrderIndex if chunkGroup is polluted, detail see #18008
Did you add tests for your changes?
Yes

Does this PR introduce a breaking change?
No

What needs to be documented once your changes are merged?

Done

Thank you, hope we will not have a perf regression :smile:


Create Dataset from SQL Query Example uses Deprecated Code that fails on newer versions

https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-register-datasets#create-datasets
In the "Create Datasets" section of this article, we do a great job of showcasing how to create various types of datasets.  However, the SQL Query example doesn't work.  It's deprecated and throws an error on SDK v1.8 or higher.  Here's the new code to accomplish this task.
from azureml.core import Dataset, Datastore
from azureml.data.datapath import DataPath
create tabular dataset from a SQL database in datastore
sql_datastore = Datastore(workspace, 'mssql')
sql_query = DataPath(sql_datastore , 'SELECT * FROM my_table')
sql_ds = Dataset.Tabular.from_sql_query(sql_query)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e258128c-b871-72b0-34e2-e5157977aa17
Version Independent ID: 08a2917b-e921-2a39-9385-ca4ef4bb25d9
Content: Create Azure Machine Learning datasets to access data - Azure Machine Learning
Content Source: articles/machine-learning/how-to-create-register-datasets.md
Service: machine-learning
Sub-service: core
GitHub Login: @MayMSFT
Microsoft Alias: sihhu

So perhaps this feedback should change slightly, the code does work if it is written EXACTLY as specified in the article.  The double parenthesis is incredibly important and was easily overlooked.  It may be worth adding some verbiage or commenting around double parenthesis vs. using DataPath.
This command works:
sql_ds = Dataset.Tabular.from_sql_query((sql_datastore, 'SELECT * FROM my_table'))
This command fails with a very unhelpful error message:
sql_ds = Dataset.Tabular.from_sql_query(sql_datastore, 'SELECT * FROM my_table')

@BreakingBI we'll review your feedback and get back to you shortly. Thanks.


PlanBuild seems to popup it's clipboard upon booting up the game and resuming a world. May also interfere with tooltips showing up.

[Warning:Jotunn.EventExtensions] Exception thrown at event InvokeOnCustomGUIAvailable in BlueprintGUI.Register:
System.NullReferenceException: Object reference not set to an instance of an object
at PlanBuild.Blueprints.BlueprintGUI.ShowTab (PlanBuild.Blueprints.BlueprintLocation location) [0x0001c] in :0
at PlanBuild.Blueprints.BlueprintGUI.Register () [0x00431] in :0
at Jotunn.EventExtensions.SafeInvoke (System.Action events) [0x00018] in D:\Projects\Jotunn\JotunnLib\Extensions\EventExtensions.cs:26
[Warning:Jotunn.EventExtensions] Exception thrown at event InvokeOnCustomGUIAvailable in BlueprintAssets.g__InitTooltipGUI|30_0:
System.MissingMethodException: Method not found: UnityEngine.Transform .Utils.FindChild(UnityEngine.Transform,string)
at Jotunn.EventExtensions.SafeInvoke (System.Action events) [0x00018] in D:\Projects\Jotunn\JotunnLib\Extensions\EventExtensions.cs:26

This happens to me also, i think thats beacuse the new update of Valheim on Steam.

fixed it in v0.14.6


ENYO-6135: Not scroll horizontally via 5-way down in horizontal scroller

Checklist

[X] I have read and understand the contribution guide
[X] A CHANGELOG entry is included
[ ] At least one test case is included for this feature or bug fix
[X] Documentation was added or is not needed
[ ] This is an API breaking change

Issue Resolved / Feature Added
When 5-way Down on the first item in a horizontal scroller, the scroller scrolls horizontally.
Resolution
If 5-way down is pressed and there is no more spottable items to the direction in a vertical scroller, the scroller scrolls vertically. But if it is a horizontal scroller, we don't have to scroll. So I guard this action with a condition not to scroll.
Links
ENYO-6135

LGTM

The reported bug was fixed incidentally by #2430 but this change prevents entering a function that shouldn't be called in this case so I'm merging as well!


Improve flexibility of email contents

The email contents values suit the three examples well, but are a little limiting if you're trying to create more generic emails.
The first thing I noticed on this was name being output after the language greeting (e.g. Hi ). Perhaps this could just be title and allow the entire contents of this title to be managed in one value?
The second thing I noticed was the lack of flexibility with paragraphs. Mail contents need to fit within an action or table which may not suit simple emails with a few paragraphs. Supporting an array of paragraphs would be great, and perhaps a definition list for blocks like this:

Hi @AlecRust,
These are excellent suggestions!
Regarding the title -- this should be easy enough to add:
var email = {
    body: {
        title: 'Your Monthly Report' // this will override "{greeting} {name}" in the templates if specified
   }
};

We do want to keep supporting the existing users of Mailgen and I do think that it's still practical to simply pass in a name, so I think it should still be possible to use the name and greeting keys.
Regarding the paragraphs -- good point. I'm thinking that intro and outro can become arrays (strings will still work) and the templates will traverse them, inserting line breaks in between them or wrapping them in <p> tags, depending on the template.
var email = {
    body: {
        intro: ['Welcome to Mailgen!', 'We\'re glad you joined us.'],
        outro: 'We hope you enjoy the package!'
   }
};

And finally, regarding the blocks, should be pretty easy to implement as an additional property for body, something like this:
var email = {
    body: {
        intro: ['Welcome to Mailgen!', 'Here are your account details:'],
        data: {
                    name: 'Mr Test',
                    email<EMAIL_ADDRESS>                    telephone: '0123456789',
        }
        outro: 'We hope you enjoy the package!'
   }
};

I think data is a bit ambiguous, would love to hear a better name for that key.
Let me know what you think, and also if you'd like to help with this (if not, that's okay).

Sounds good to me. I'd probably go with definitionList if that's the HTML that will be output.
Happy to take a look at this, might not get to it for a few days though.

@AlecRust cool, let me know if you have any questions. I think dictionary might be more appropriate (as those are key-value pairs).


Middleman question

Hey all. I'm still learning the ropes with GitHub and Slate, but I managed to get middleman up and running, made changes to the index.md, and was able to publish my changes.
All was going well, but now I can't get middleman to start. Whenever I try to start the middleman server, it says it's loading and then does nothing. What am I doing wrong?
Thanks in advance!

:/ hmmm...does 'bundle exec middleman server --verbose` give any more information?

Here's what it says:

Hmm...doesn't tell us much. I apologize, but I'm not really sure how else to diagnose the problem. :disappointed: If you can't find the source of the problem yourself, but still desperately want to use Slate, you could run Linux in a virtual machine, or use Docker with the Dockerfile in the repo.
I'm going to close this since it's not actionable, but feel free to continue posting if you find out anything more.

Sure, thanks for the help. I'm still using Slate, it's just that I can't preview changes to the document using the Middleman server. I just have to do a rake build any time I want to see the changes. A bit cumbersome, but it works.

Just to let you know, I am facing the same issue on Win7 (and the same ruby version)

Having the same problem on Windows 8.


invite-user-modal: Update CSS for select dropdowns and expires_in_values keys for snakecase.

These are the first two prep commits from #26109 updated for changes in the code base:

The first commit cleans up some CSS rules and fixes the small issue in the screenshot below.
The second commit updates some settings config values to be snake case vs camel case.

Notes:

There's still a slight height difference between the time input and the dropdown for time units, which is not addressed in these changes.

Screenshots and screen captures:

Before
After

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

@timabbott I haven't tested.

Merged, thanks @laurynmm!


Fixed an issue where folder provider icons would not show

Closes #5274
The new logic is as follows:

If we don't have any folder mapping info, we show the standard folder svg
If we have in the resource manager an svg for the provider, we use that one (usually nicer and we have one for our own Azure profider connector)
If the provider has one, we use the one the provider dictates If all else fails, we use the standard svg one.

In addition to this, I also added a button so users can create new folder mappings (totally forgot about that part when implementing the feature).

Summary

/azp run

Last commit should handle all the provided feedback and appears to work for all scenarios:


fix: verify chain id

Verify the circuit's constant chain ID is included in the messages signed by the validators.
Thanks @evan-forbes for pointing this out!

@evan-forbes The sign bytes for each validator are different as they're all timestamped (and so the message isn't shared). This means we'll probably need to check for the type of the commit as well.


Batch script to build project from command line

Quick script to build project from command line

Hi @GoodOlClint, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.
TTYL, MSBOT;

Thanks for the updates. Will merge the script


Roundup: [oss-security] kio vulnerability: need CVE

Here is a report from the oss-security mailing list for Vulnerability Roundup 27.
Skip to First Email
Instructions:
Identification
Identify if we have the software, in 16.09, 17.03, and unstable. Then determine if we are vulnerable, and make a comment with your findings. It can also be helpful to specify if you think there is a patch, or if it can be fixed via a general update.
Example:
unstable: we are not vulnerable (link to the package)
17.03: we are vulnerable (link to the package)
16.09: we don't have it packaged

IMPORTANT: If you believe there are possibly related issues, bring them up on the parent issue!
Patching
Start by commenting on this issue saying you're working on a patch. This way, we don't duplicate work.
If you open a pull request, tag this issue and the master issue for the roundup.
If you commit the patch directly to a branch, please leave a comment on this issue with the branch and the commit hash, example:
fixed:

release-16.09: abc123

Skip to First Email
Upon Completion ...

[ ] Update Graham's database

Info
Triage Indicator:
-needs-triage +roundup27 thread:0000000000003efc

File Search: https://search.nix.gsc.io/?q=kio&i=fosho&repos=nixos-nixpkgs
GitHub Search: https://github.com/NixOS/nixpkgs/search?utf8=✓&q=kio+in%3Apath&type=Code

Should the search term be changed from kio? Suggest a new package search by commenting:
-suggested:kio +suggested:correctPackageName thread:0000000000003efc

Known CVEs:

Skip to End
Tue, 28 Feb 2017 19:44:38 +0100 Albert Astals Cid <aacid-at-kde.org>, 1501907.mtIKE32cx4@xps
Hi, Albert from KDE, can we get a CVE assigned for kio (input/output library)?

advisory is here https://www.kde.org/info/security/advisory-20170228-1.txt

Thanks,
  Albert

Skip to End

Wed, 1 Mar 2017 12:03:55 +0530 (IST) P J P <ppandit-at-redhat.com>, alpine.LFD.2.20.1703011202510.1553@wniryva
+-- On Tue, 28 Feb 2017, Albert Astals Cid wrote --+
| Hi, Albert from KDE, can we get a CVE assigned for kio (input/output library)?
| advisory is here https://www.kde.org/info/security/advisory-20170228-1.txt

Please see:
  -> http://cveform.mitre.org/

--
 - P J P
47AF CE69 3A90 54AA 9045 1053 DD13 3D32 FE5B 041F

Skip to End

Our version of kio is recent enough.


WCAG 2.0 failure on buttons

The replacement of div controls with buttons causes Colorbox to fail WCAG 2.0 validation because they do not have a title or label.  I've tested adding labels successfully, is this something you would be willing to add in?  Because the controls are all image replaced this shouldn't cause any conflicts.

Duplicate of #765.


Minor docs improvement

..and build changes.

coverage: 98.484%. remained the same when pulling 8a11c6cc7291d3d533f710fe9ef57ab9f72fa839 on minor-docs into 135e364c967228310120838c9e5e86f0a27d212d on main.


Empty Validation Rule

I guess such a validation rule is pretty common.
Although this could be as simple as ValidationRuleLength(min: 1, failureError: error) having a shortcut method for it would be awesome.
Useful and fast enhancement

Check out ValidationRuleRequired
https://github.com/adamwaite/Validator/blob/master/Validator/Validator/Rules/ValidationRuleRequired.swift
See the readme for usage.
Ty

I've tried that. And no, it doesn't solve my problem because when using UITextFields, they always have at least an empty string.
That validator checks against nil. Thus, it will always be valid.

Extend the library as you like. Maybe:
extension Validator {

  static var err = SomeError()

  static var blankRule: ValidationRuleLength {
    ValidationRuleLength(min: 1, failureError: err)
  }
}

I think creating this sort of convenience method litters the convention so i don't really want to put it in.

Yea, I already did that.
My main point was that if it's a drop-in validation library, it should cover a fair amount of use cases.
But cool, sure.


Testing the dev feed requires the bearer token (#29723)

Pull e00634b7c6433cf700dc3157f5dba8dcbd28d72d from main into this branch which fixes the retry logic with the dev feed.

I'm going to override check enforcer, there's nothing in any pipeline outside of a maven failure that tries to republish to maven and the java dev feed that will actually test this.


Naive implementation to fix double sender check

Fix #382

We first reproduce the bug
Then we see the gas cost increase of the naive approach

Signature check Gas Cost
This PR

Transfer

senders all different

operation gas cost: 4664005
transaction gas cost: 6377081

some same senders

operation gas cost: 4798157

Mass Migration

senders all different. operation gas cost: 4614020
some same senders: operation gas cost: 4550813

Create2Transfer

senders all different

operation gas cost: 5406666
transaction gas cost: 8009615

some same senders

operation gas cost: 5438038

Master (senders all different)

Transfer

operation gas cost: 4610066
transaction gas cost: 6323012

Mass Migration

operation gas cost: 4603308

Create2Transfer

operation gas cost: 5338076
transaction gas cost: 7941113

The naive approach to calculate the nonce

This PR: 4716079
Master: 4552102

Thanks for the review, created #423 to address that later. So far we can focus on making things work.

Thanks for the review, created #423 to address that later. So far we can focus on making things work.


Paginate dir/ls

Allow pagination on dir/ls to serve the apis faster

Ls is now async, So the will be faster to load entries. Closing it for now. We can add this feature if future for ux


Update learning-resources.md with links to video material

Update learning-resources.md with links to video material
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @liferoad

retest this please


Hash Index Out of Bounds

Hey so when running the following in debug mode:
https://github.com/RealTimeChris/Json-Performance,
it seems that this index here goes out of bounds:
https://github.com/stephenberry/glaze/blob/main/include/glaze/util/hash_map.hpp#L314
I suppose we could check to see if index is > sizeof(hashes) but that may not be ideal.

Thanks so much for pointing this out! Fixed in #615


[HELP] Cannot start K3s Server

K3s Version:
k3s version v1.30.2+k3s1 (aa4794b3)
go version go1.22.4

Node(s) CPU architecture, OS, and Version:
Linux k8s-master 5.14.0-427.26.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jul 17 15:51:13 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration: N/A
Describe the bug:
I downloaded the packages k3s, k3s-airgap-images-amd64.tar.gz, install.sh at https://github.com/k3s-io/k3s/releases/tag/v1.30.2%2Bk3s1 to install K3s locally.

mv k3s /usr/local/bin; chmod +x /usr/local/bin/k3s
mkdir -p /var/lib/rancher/k3s/agent/images/; cp ./k3s-airgap-images-amd64.tar.gz /var/lib/rancher/k3s/agent/images/
INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

After the installation, I used kubectl get nodes to get nodes; however, I failed to get it as below:

I also downloaded the log with journalctl -u k3s > k3s.log and I found several errors in the log; attached the log here...
k3s.log
Steps To Reproduce:

I disabled the firewall and allow all traffic in iptables
systemctl disable --now firewalld
iptables -F; iptables -t nat -F; iptables -t mangle -F; iptables -X
iptables -P INPUT ACCEPT; iptables -P FORWARD ACCEPT; iptables -P OUTPUT ACCEPT

Disabled SELinux

Did the steps to install K3s offline as stated in the section above.

What do the actualy logs say journalctl -u k3s?
What does your K3s config look like cat /etc/rancher/k3s/config.yaml?
Are you using a seperately installed kubectl, or the one that comes with K3s (i.e. make sure your KUBECONFIG is pointing at /etc/rancher/k3s/k3s.yaml?

The logs seem to stop partway through normal startup, however I don't see any errors.

After the installation, I used kubectl get nodes to get nodes; however, I failed to get it as below:

Are you sure you're using the k3s kubeconfig? K3s isn't listening on <IP_ADDRESS>:43397. Make sure you don't have the KUBECONFIG environment variable set and pointing at some other kubeconfig. You should be using the kubeconfig at /etc/rancher/k3s/k3s.yaml. If you're using the kubectl that comes with k3s, it will use this file by default.

Hi @dereknola, @brandond - thank you for reply!
After the error's happening, I uninstalled the whole K3s. As you all mentioned kubectl - I found out I did not use the default kubectl...instead, I used kubectl installed by dnf install -y kubectl by my own.
I removed the kubectl and directly executed the command as described in K3s website - curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh - (the version of K3s seems not up to date; it is 1.29.x), finally K3s started well! :-)
I will take a look at the documentation for knowing more.


chore(deps): upgrade dev dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-dev-dependencies-1.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 1596

Questions ?
Please refer to the Backport tool documentation


Fixed Auth Type name

I think the correct name is 'WebAuthn' , not 'WebAutn'.

I'm sorry, I'm late. Thanks a lot for the fix.


Mamba and inference at any frequency

@albertfgu , in your S4 work we could infer models trained on audio data for example at certain frequency on higher ones, I was thinking can this still be done with Mamba along with bandlimiting?
if self.bandlimit is not None:
    freqs = dt / rate * A.imag.abs() / (2 * math.pi)  # (H N)
    mask = torch.where(freqs < self.bandlimit * 0.5, 1, 0)
    C = C * mask

I think the connection is not as strong because the model is no longer linear-time invariant. I think of Mamba as a discrete model and not really a continuous model.


Enable JitterRetry RetryController by default

Currently the retry code submitted in #202 is disabled. This PR enables the code by defaulting to using the JitterRetry controller.
The Jitter retry calculates the wait time between retry requests in the recommended way from AWS. It adds a random element to the retry time calculation to ensure that all the requests that failed at the same time do much keep retrying at the same time.

I really like this. The only thing i‘m not sure about is the controller name part in retrycontroller. But i cant come up with anything better that isn‘t totally generic.

@fabianfett The one thing I'm not keen on is the return value of getRetryWaitTime where nil means dont retry. What do you think of changing that from an optional TimeAmount to an enum
enum RetryStatus {
    case retry(wait: TimeAmount)
    case dontRetry
}

I really like this. The only thing i‘m not sure about is the controller name part in retrycontroller. But i cant come up with anything better that isn‘t totally generic.

Yeah I struggled with a name, this was the best I could come up with

In go the protocol is called RequestRetryer. I think i like that! @adam-fowler wdyt?
https://github.com/aws/aws-sdk-go/blob/d7b5372b5b7583d5a30a3d7440d9d684aba581a9/aws/config.go#L108


context is not protected, then is made sync

In github actions, cp semaphore tests are failed sometimes. In windows dump file, we saw that the seg fault is occured at then continuation. As we review the code, context lifecycle is not protected. As there is no user code, instead of extending lifecycle, it is also made sync. So that the client will not destroyed until this continuation.

Linux test FAILed.

Could you run git-clang-format for this change ?

done

Linux test PASSed.

Linux test PASSed.

Windows test PASSed.

This PR will be closed, solution to the problem will be changed.

Windows test FAILed.

Linux test PASSed.


Improving Dependency Parsing Keyword Extraction

Issue Description
Currently, the dependency parsing does not include conjunction links to catch verb-object pairs. An example is given below:

The custom pipeline needs to be updated to capture the verb-object pairs with conjunction depth.
Step Details
Steps that will be performed:

[x] Implement deeper dependency link search
[x] Include to returned keywords

Final Actions
The trace capture must be re-run with new verb object pairs.
Deadline of the Issue
08.05.2023

preposition catch implemented.


[Bug]: "ALTER TABLE" can collect entities but entities don't have belongStmt

Version
4.0.1
Which SQL?
Postgre
SQL content
ALTER TABLE my_table DROP a_column|;

JavaScript/TypeScript code
No response
What happened?
my_table can be collected but it has no belongStmt, then people cannot use isContainCaret or other properities

my_table被收集了但是其属性里没有belongStmt, 基于isContainCaret或其他StmtContext属性的功能可能会无法正常工作。

I find that maybe there's no definition of altertablestmt. I will create a PR and try to cover it. If there is already a roadmap, ignore the PR.

看了下好像是没有定义'altertablestmt'这个StmtContext，我这边尝试添加了一下，已测试通过。如果这在你们的roadmap上，请忽略此PR
Relevant log output
No response

此issue先关闭，实体收集相关问题请追踪 https://github.com/DTStack/dt-sql-parser/issues/325