Several questions about using Cranelift as a backend for a programming language

I'm looking into using Cranelift as a backend for a programming language that I'm working on. Other options I have been looking into are compiling to C or using LLVM, but both come with their own set of trade-offs I'm not a fan of. For example, LLVM is generally quite slow, while C compilers tend to not give you enough control (i.e. they may reorder things unless you carefully insert compiler barriers). Cranelift is promising because of the amount of control it gives, and because it's written in Rust.
Unfortunately, the Cranelift documentation is limited, to a point where I'm overwhelmed/confused about where to even start. In this issue I'll be asking several questions to hopefully clear this up. If there's a better place for asking long-form questions like this, please let me know :smiley:
To set the stage: Inko is a language similar to Go/Erlang in that it has lightweight processes, and these can suspend at more or less any place (technically these spots are known, e.g. when performing IO, but to the user it may as well be "anywhere"). This means I need to be able to maintain and swap the stacks, like stackful coroutines. I also need to be able to grow the stack, and maybe shrink it (though figuring out when to do that is really tricky). Bare-metal performance is not a main goal, rather I want something that compiles fast and performs better than a well optimised interpreter (without a JIT), something I think one should be able to achieve with Cranelift.
Which brings me to the questions I have, in no particular order:

Cranelift offers no way of injecting assembly directly into the IR. Instead it seems you have to define a C/assembly function then bind to that. Is Cranelift able to jump to a naked function defined in C and exposed using #[no_mangle] and extern "C"? The idea here is to make these stack fiddling routines naked functions so they don't introduce extra assembly overhead, and to give you full control. By jumping to them directly they essentially act as assembly macros, avoiding the function call overhead.

Defining function names in Cranelift is a bit confusing. UserFuncName basically takes two arbitrary u32 values, and it's not clear what you'd use those for. Then there's Module.declare_function which takes a &str, and looking at some existing Cranelift projects this seems to be what they use. So which one should I use if I want to expose my function names (potentially mangled)?

EBBs take variables as arguments, but it's not clear how you'd determine what variables to pass down to sub blocks. For example, if I have a graph where blocks A and B both jump to C, and C depends on a result produced by these blocks, how would I know what variable to pass? My own IR is already a graph but not in SSA, thus blocks can just use variables defined in some earlier block. My graph does ensure that in the above case the blocks A and B write to any variable needed by C where its value can differ per branch.

How would one associate debugger information with their generated code? For example, I would like for a debugger to know about my source code, line numbers, etc. The Cranelift API doesn't seem to offer anything to do things like "map these instructions to lines X, Y, Z in the source code". Do I have to fiddle with the likes of e.g. gimli (and whatever the equivalent is for Windows)?

Does Cranelift have any routines for detecting integer overflows? Instructions such as iadd seem to wrap around on overflow, so detecting it manually is doable if needed; I just don't want to reinvent this if already provided by Cranelift.

Cranelift seems to have this notion of heaps for functions, and the ability to restrict functions to a certain portion of the heap. For my language there are no such restrictions (i.e. processes don't use fixed size heaps. How would I express this in Cranelift?

Related to that, I'm seeing various hints about Cranelift performing heap bounds checking in various places. It's not clear to me if this would be needed in my case, and if not if there's a way of opting out (i.e. I don't want redundant heap bounds checking in my generated code). Is there more information on this somewhere?

Related to swapping the stacks: if I insert an assembly routine in a function prologue (i.e. the function just starts with a call/jump to such a routine), is there a way to guarantee Cranelift won't move/inject anything before the call/jump? The assembly that might be needed for the system's calling convention should be fine as I can just add some padding to my stacks to account for that, but I don't want Cranelift to think my routine has no side effects and what not and just move it elsewhere.

Thanks for answering any of these questions, and again let me know if it's better to post this elsewhere! :smiley:

@bjorn3 Thanks for the answers so far!

Yes, you can jump to functions defined in C.

In the documentation of cranelift_codegen the closest I could find is https://docs.rs/cranelift-codegen/latest/cranelift_codegen/ir/trait.InstBuilder.html#method.jump, but this jumps to a block and not a random memory address. Should I perhaps look in a different crate for this?

You can use the cranelift-frontend crate to generate SSA form on the fly.

Looking at the example on https://docs.rs/cranelift-frontend/latest/cranelift_frontend/#example, it seems to use jumps but leaves out the arguments. Am I correct in thinking that if you use use_var() in a sub block, Cranelift automatically passes its input as an argument when jumping to said block? That is, if you have a graph like this:
A    assign a to 42
|
+--------->----------> C use(a)
|
B    assign a to 50

Does Cranelift construct the graph such that it passes a in blocks A and B as an argument to block C, such that a in C is whatever value was produced in the branch leading up to it? Or do I have to explicitly set a in A/B as an argument for the jump(s) to C?

No, Cranelift will always put a prologue first before doing anything. One option I guess would be to insert raw bytes representing the call before the function. You can use context.compile(), get the compiled code and relocations, insert your custom prologue and then call module.define_function_bytes instead of module.define_function.

I suspect this would probably get a bit tricky, and maybe break across versions. If I just insert a function call as the first instruction after the prologue, is there anything I can do to prevent Cranelift from moving it around, i.e. is there something like a compiler barrier? If so I think that should be good enough.

https://bytecodealliance.zulipchat.com/#narrow/stream/217117-cranelift might be a better place.

I thought about the Zulip, but it's too easy to lose track of things in a chat channel unfortunately. But if desired I can ask things over there instead :smiley:

Looking at the example on https://docs.rs/cranelift-frontend/latest/cranelift_frontend/#example, it seems to use jumps but leaves out the arguments. Am I correct in thinking that if you use use_var() in a sub block, Cranelift automatically passes its input as an argument when jumping to said block?

Correct. Cranelift-frontend automatically appends basic block arguments as necessary to handle SSA form.

@cfallin

We currently don't have a way of injecting assembly at all -- #1041 goes over some of the reasons why. Basically, while it seems conceptually simple ("just emit the assembly I give you!"), it actually is enormously complex for all the reasons listed there (see especially this comment).

I had indeed seen the issue, and I figured it wouldn't be as easy as just dumping assembly in the middle of the IR :smile:

When you say "jump to a function", not directly; we can call one though, which is actually probably what you'd want. (Otherwise, how does this snippet know where to jump back to?) Perhaps you were thinking the jump would somehow inline the snippet (we don't have this)?

Ah yeah good point, I forgot about actually having to jump back; heh.
Either way, the function call cost isn't really something to worry about so calling a helper function should be fine.

Popping up a level (in the X-Y problem sense): the goal here is stack-switching and/or segmented stacks, goroutine-style, no?

Pretty much. I was planning on taking the approach outlined in https://graphitemaster.github.io/fibers/ where I'd just define a bunch of routines in assembly (instead of using setjmp/longjmp), then call those in the appropriate places.

Combining the two, I could see a small modification to the Cranelift prologue generator that would say "if limit reached, call this runtime function" and that function would essentially allocate a new stack, switch to it, place a little trampoline on the stack to return back to the previous segment on return, and then jump back to the return address in the function that hit the limit. I'd be happy to help design this feature if you're interested.

I can see this being useful, though I currently don't know enough about Cranelift or how exactly I would implement the routine in a cross-platform way. I think it would probably be best to first implement this in my language, then maybe come back to Cranelift to see how/if it makes sense for it to provide a generalised solution.

For segmented stacks following the same abi as https://gcc.gnu.org/wiki/SplitStacks may work. LLVM also implements this and Rust itself used to use it back when it still used green threads.

@bjorn3 @cfallin Thanks for the info! I'll move this to Zulip and close the issue :smiley:

As other folks have said, welcome! Although Cranelift is co-developed with Wasmtime which gives Wasmtime a little bit of a privileged position, we explicitly want to support projects like yours. So we're excited to hear from you!
I especially want to emphasize that we're open to making changes to Cranelift to support new use cases. We have to balance that against making sure we have the resources for ongoing maintenance of every change we merge. But if you find that it's impossible to do something you need using Cranelift, just keep in mind that fixing Cranelift is an option.

Looking at the example on https://docs.rs/cranelift-frontend/latest/cranelift_frontend/#example, it seems to use jumps but leaves out the arguments. Am I correct in thinking that if you use use_var() in a sub block, Cranelift automatically passes its input as an argument when jumping to said block?

Correct. Cranelift-frontend automatically appends basic block arguments as necessary to handle SSA form.

To clarify, "as necessary" is doing double-duty here. Cranelift allows you to use a value in one block that was defined in another block, as long as the definition dominates the use. So cranelift-frontend will avoid adding a block parameter if there's exactly one definition of the variable that reaches the use. You shouldn't have to think about that as long as you use cranelift-frontend but it's good to be aware of if you're reading the generated Cranelift IR.

If I just insert a function call as the first instruction after the prologue, is there anything I can do to prevent Cranelift from moving it around[...]?

I believe, and @cfallin can correct me if I'm wrong, that currently 1) Cranelift won't move side-effecting instructions, and 2) it treats all function calls as side-effecting. It may move pure instructions past effectful ones, but I think it'll never move a pure instruction any earlier in the function. So I think the current state is that if a function call is the first instruction, it will still be the first instruction after all optimizations, and only the prologue will precede it in the final machine code.
Whether you can rely on that behavior in the future is another question, of course.
I don't entirely understand what you're trying to do, but I'm curious whether the get_stack_pointer instruction added in #4573 helps your use case at all, and we can always talk about adding more instructions if we have a good motivating use case for them. Since there's ongoing work in the WebAssembly standards world on proposals for stack-switching, it could even be that Wasmtime is going to need something similar to whatever you're looking for. If there are two projects both looking for the same features from Cranelift, then that's especially strong motivation to come up with a good design.

@jameysharp Thanks for the reply!

I don't entirely understand what you're trying to do, but I'm curious whether the get_stack_pointer instruction added in https://github.com/bytecodealliance/wasmtime/pull/4573 helps your use case at all, and we can always talk about adding more instructions if we have a good motivating use case for them.

Essentially I'd need routines for the following:

Allocating memory that can be used as stack memory. This is out of the scope of Cranelift I think, and OS/calling convention specific (i.e. Windows of course does things its own way compared to everybody else). This I'd implement in my Rust based runtime library, using the appropriate OS functions.
A routine that takes a pointer to such a chunk of memory, and turns it into the current OS thread's stack memory. I think this involves mostly changing the stack pointer to point to this chunk of memory.
A routine for writing all registers to a buffer, using the steps described in https://graphitemaster.github.io/fibers/#getting-the-context. For this I'd use an assembly function, or maybe a naked Rust function with inline assembly; I haven't decided yet. This being provided in a reusable manner is something I could see being useful, but I'm not sure it would fit within the scope of Cranelift.
A routine that takes the current stack pointer and stores it somewhere, so that we can restore this stack later on.

This is basically what https://graphitemaster.github.io/fibers/#user-space-context-switching discusses, but in Rust/Cranelift instead of C.

A routine for writing all registers to a buffer, using the steps described in https://graphitemaster.github.io/fibers/#getting-the-context.

On second though, if Cranelift were to provide this it would be able to only save the registers actually in use at that point, rather that pessimistically having to assume any of them may be used, and thus having to save all of them.
The inverse would also be true: if we have known wake-up points, Cranelift would know which registers are in use afterwards, and only restore those. This may be less useful though, as based on the register allocator and the code that follows it might just be easier to restore all of them.

On second though, if Cranelift were to provide this it would be able to only save the registers actually in use at that point, rather that pessimistically having to assume any of them may be used, and thus having to save all of them.

That linked code only saves callee saved registers which have to be saved anyway for ABI reasons.


To enable merged cells

Proposal
Sometimes I have spreadsheets with this kind of table:
+---------------------+----------+
| Property            | Earth    |
+=============+=======+==========+
|             | min   | -89.2 °C |
| Temperature +-------+----------+
| 1961-1990   | mean  | 14 °C    |
|             +-------+----------+
|             | max   | 56.7 °C  |
+-------------+-------+----------+

When I need to use in quarto, it's difficult to do it, because I do not have a tool to convert in example a xlsx file to this kind of pandoc grid_tables. There is no way to keep the merged cells.
Using python, a tool to do it could be great-tables: I could read the xlsx table using pandas and then use great-tables to render properly an HTML table with rowspan.
Thank you to @cderv for the idea.

Thanks for opening this issue. I've added the data from your example as an xlsx file, so we can explore a bit!
example-gt.xlsx
I think that in general, it seems like there are two challenges in this issue:

reading  the spreadsheet into a DataFrame and tidying it
outputting a grid table from that DataFrame

Reading and tidying
Here we read the excel spreadsheet using pandas.
import pandas as pd

df = pd.read_excel("https://github.com/posit-dev/great-tables/files/13821956/example-gt.xlsx")

                Property Unnamed: 1     Earth
0  Temperature 1961-1990        min  -89.2 °C
1                    NaN       mean     14 °C
2                    NaN        max   56.7 °C

Note that the merged rows in the Property column ended up with NaNs in them, and there is an unnamed column. To fix this, we can forward fill and do a rename.
tidy = df.assign(Property = df["Property"].ffill()).rename(columns={"Unnamed: 1": "Measure"})
tidy

                Property Measure     Earth
0  Temperature 1961-1990     min  -89.2 °C
1  Temperature 1961-1990    mean     14 °C
2  Temperature 1961-1990     max   56.7 °C

Outputting as a grid
With the data tidied, we can produce something like a grid table, by setting the rowname_col="Measure", and groupname_col="Property".
import pandas as pd
from great_tables import GT

df = pd.read_excel("https://github.com/posit-dev/great-tables/files/13821956/example-gt.xlsx")
tidy = df.assign(Property = df["Property"].ffill()).rename(columns={"Unnamed: 1": "Measure"})

gt = GT(tidy, rowname_col="Measure", groupname_col="Property")
gt

Note that it differs a bit from you example output though. I wonder if there's another way this is supported in the R gt library?

Thank you very much @machow very useful and didactic.
By also doing it directly in excel, I was somehow already able to.
I like this result of yours and it is very readable, but I would like to see if it is useful (also for this great project of yours), to introduce something like groupname_row, something that allows you to group rows.
Just a thought.

I wonder if there's another way this is supported in the R gt library?

I do not know if there is a gt way, but I know this other R way created by @Steinthal.

Well, @machow, I’d like to caution you that the example given be @aborruso is quite specific.
What if there’s another set of combined cells for another period again with min/mean/max values and those values shall be presented directly above or below those of the earlier period? Then the arrangement presented or requested by @aborruso would make more sense than placing the period above the values.
What I mean is: There are different possible ways to present data depending on the amount of data and requirements – and a solution should be flexible enough to support them all (or at least most of them).

@Steinthal if I understand correctly, are you saying that something like this...
import pandas as pd

df = pd.DataFrame({"Property": ["Temp 1961"] * 2 + ["Temp 1962"] * 2, "Measure": ["min", "max"] * 2, "Earth": [1, 2, 3, 4]})
GT(df, rowname_col="Measure", groupname_col="Period")

Isn't ideal because the property values (e.g. Temp 1961) shouldn't go across the full dataframe? I wonder if the gt for R row_group_as_column argument could make this look a bit better (even if it doesn't provide a general solution for spanning rows)?

Note that above, the groupings (e.g. grp_a) are their own column, and span rows.


[boost-variant2] Build error on x64-linux

Package<EMAIL_ADDRESS>Host Environment

Host: x64-linux
Compiler: GNU 11.4.0
vcpkg-tool version: 2024-04-23-d6945642ee5c3076addd1a42c331bbf4cfc97457
vcpkg-scripts version: dee924de7 2024-05-27 (8 hours ago)

To Reproduce
vcpkg install
Failure logs
-- Downloading https://github.com/boostorg/variant2/archive/boost-1.85.0.tar.gz -> boostorg-variant2-boost-1.85.0.tar.gz...
[DEBUG] To include the environment variables in debug output, pass --debug-env
[DEBUG] Trying to load bundleconfig from /home/sivachandran/Workspace/personal/vcpkg/vcpkg-bundle.json
[DEBUG] Failed to open: /home/sivachandran/Workspace/personal/vcpkg/vcpkg-bundle.json
[DEBUG] Bundle config: readonly=false, usegitregistry=false, embeddedsha=nullopt, deployment=Git, vsversion=nullopt
[DEBUG] Metrics enabled.
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] Feature flag 'dependencygraph' unset
[DEBUG] 1000: execute_process(curl --fail -L https://github.com/boostorg/variant2/archive/boost-1.85.0.tar.gz --create-dirs --output /home/sivachandran/Workspace/personal/vcpkg/downloads/boostorg-variant2-boost-1.85.0.tar.gz.1408568.part)
[DEBUG] 1000: cmd_execute_and_stream_data() returned 0 after   651367 us
[DEBUG] Trying to hash /home/sivachandran/Workspace/personal/vcpkg/downloads/boostorg-variant2-boost-1.85.0.tar.gz.1408568.part
[DEBUG] /home/sivachandran/Workspace/personal/vcpkg/downloads/boostorg-variant2-boost-1.85.0.tar.gz.1408568.part has hash f4f79ab208c534a7e7a285e2722dbe14c7d810a8ad9e73dd2a6de51862b3b08928802251e8661815e5bf9f82eefd10125a984724ae4670823307edc93cd679a8
error: Failed to download from mirror set
error: File does not have the expected hash:
url: https://github.com/boostorg/variant2/archive/boost-1.85.0.tar.gz
File: /home/sivachandran/Workspace/personal/vcpkg/downloads/boostorg-variant2-boost-1.85.0.tar.gz.1408568.part
Expected hash: 1a5fc662247f3e8437dfeec552fc3c5c380112db4a2e6b0c9e7abad1c06ec0d07dcbb528c244884f2507ac14174f4db3fadfd211e84d9bcd26a5ab5f33cb4d97
Actual hash: f4f79ab208c534a7e7a285e2722dbe14c7d810a8ad9e73dd2a6de51862b3b08928802251e8661815e5bf9f82eefd10125a984724ae4670823307edc93cd679a8
[DEBUG] /source/src/vcpkg/base/downloads.cpp(997):
[DEBUG] Time in subprocesses: 651367us
[DEBUG] Time in parsing JSON: 3us
[DEBUG] Time in JSON reader: 0us
[DEBUG] Time in filesystem: 848us
[DEBUG] Time in loading ports: 0us
[DEBUG] Exiting after 655 ms (652891us)

CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:32 (message):

      Failed to download file with error: 1
      If you are using a proxy, please check your proxy setting. Possible causes are:

      1. You are actually using an HTTP proxy, but setting HTTPS_PROXY variable
         to `https://address:port`. This is not correct, because `https://` prefix
         claims the proxy is an HTTPS proxy, while your proxy (v2ray, shadowsocksr
         , etc..) is an HTTP proxy. Try setting `http://address:port` to both
         HTTP_PROXY and HTTPS_PROXY instead.

      2. If you are using Windows, vcpkg will automatically use your Windows IE Proxy Settings
         set by your proxy software. See https://github.com/microsoft/vcpkg-tool/pull/77
         The value set by your proxy might be wrong, or have same `https://` prefix issue.

      3. Your proxy's remote server is out of service.

      If you've tried directly download the link, and believe this is not a temporary
      download server failure, please submit an issue at https://github.com/Microsoft/vcpkg/issues
      to report this upstream download server failure.

Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:270 (z_vcpkg_download_distfile_show_proxy_and_fail)
  scripts/cmake/vcpkg_from_github.cmake:106 (vcpkg_download_distfile)
  buildtrees/versioning_/versions/boost-variant2/71aeea8a9326c7b978940c83970790090ddcf8f9/portfile.cmake:3 (vcpkg_from_github)
  scripts/ports.cmake:175 (include)

Additional context
vcpkg.json
{
  "dependencies": [
    "boost-headers",
    "boost-optional",
    "boost-lexical-cast",
    "boost-algorithm",
    "xerces-c",
    "openssl",
    "gtest",
    "nowide",
    "log4cplus",
    "udt",
    "cpp-netlib"
  ],
  "builtin-baseline": "829c9e3def84d777b609784d0820dbfa3284d342"
}

duplicate of #38974

Duplicate of https://github.com/microsoft/vcpkg/issues/38974, PR #38979 will fix this issue.


Update Portability Service Gateway routes

SF doesn't allow the tags to be parameters so we'll have to do some token substitution ourselves.

@mjrousos Is this ready to be merged?

Should we file a feature request to allow that to be parametized? It would make deployment much easier


Moving between tenants

can this be used to move content between different SharePoint Online Tenants?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 85a08784-543a-32b7-1e83-4f3ebf5bc6d4
Version Independent ID: 7234f9f2-a3a1-b37b-4153-f86ec483993c
Content: Migrate to SharePoint Online
Content Source: migration/migrate-to-sharepoint-online.md
Product: sharepoint-server-itpro
GitHub Login: @JoanneHendrickson
Microsoft Alias: jhendr

Hi Dean:  At this time, it cannot be used to content between SPO tenants.
Joanne

@DeanGross and @Lalith09 - There is a similar thread here: https://github.com/MicrosoftDocs/OfficeDocs-SharePoint/issues/84
We found out from the product team that the tool does not support moving between SharePoint Online tenants right now. But there is an active User Voice thread to up-vote and request the feature. You can find a link on the thread I linked above. Please up vote! :)

Hi, Is it possible to move files between sharepoint online sites? Lets say you have a old sharepoint site and just created a new one and want to move the files there?


SQL error when trying to attach role - Insert value list does not match column list

I just installed jeremykenedy/laravel-roles by doing the following:

composer require jeremykenedy/laravel-roles
php artisan vendor:publish --tag=laravelroles
php artisan migrate
(edited database\seeders\DatabaseSeeder.php per the instructions at database\seeds\DatabaseSeeder.php)
composer dump-autoload
php artisan db:seed

After doing all that the roles table now has 3x entries in it: Admin, User and Unverified. So it seems like that all worked out.
But then when I do php artisan tinker and try to add a role to a user I get this error:
$adminRole = config('roles.models.role')::where(['name' => 'Admin']);
$user = User::findOrFail(1);
$user->attachRole($adminRole);

   Illuminate\Database\QueryException  SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 4 (Connection: mysql, SQL: insert into `role_user` (`created_at`, `role_id`, `updated_at`, `user_id`) values (2024-02-10 11:00:45, ?, 2024-02-10 11:00:45, 1), (2024-02-10 11:00:45, [], 2024-02-10 11:00:45, 1), (2024-02-10 11:00:45, \*\eagerLoad, 2024-02-10 11:00:45, 1), (?, 2024-02-10 11:00:45, ?, ?, ?, \*\localMacros, 2024-02-10 11:00:45, 1, ?, ?), (2024-02-10 11:00:45, ?, 2024-02-10 11:00:45, 1), (2024-02-10 11:00:45, \*\propertyPassthru, 2024-02-10 11:00:45, 1, from), (2024-02-10 11:00:45, \*\passthru, 2024-02-10 11:00:45, 1, aggregate, average, avg, count, dd, ddrawsql, doesntexist, doesntexistor, dump, dumprawsql, exists, existsor, explain, getbindings, getconnection, getgrammar, implode, insert, insertgetid, insertorignore, insertusing, max, min, raw, rawvalue, sum, tosql, torawsql), (?, 2024-02-10 11:00:45, \*\scopes, 2024-02-10 11:00:45, 1), (2024-02-10 11:00:45, \*\removedScopes, 2024-02-10 11:00:45, 1)).

Here's the SHOW CREATE TABLE  role_user output:
CREATE TABLE `role_user` (
  `id` int unsigned NOT NULL AUTO_INCREMENT,
  `role_id` int unsigned NOT NULL,
  `user_id` bigint unsigned NOT NULL,
  `created_at` timestamp NULL DEFAULT NULL,
  `updated_at` timestamp NULL DEFAULT NULL,
  `deleted_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `role_user_role_id_index` (`role_id`),
  KEY `role_user_user_id_index` (`user_id`),
  CONSTRAINT `role_user_role_id_foreign` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE,
  CONSTRAINT `role_user_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

I'm running Laravel Framework 10.42.0.
Any ideas?

I do not not know with Tinker


Add similar album or artist recommendations?

This is by far the best Jellyfin music player, but it’s missing this feature from Jellyfin. Possible to add it in? Thanks for the amazing app!

Hey there! Thanks for your feedback. This could be possible to implement, depending on the availability of an API for this data (shouldn't be a big problem). There is other stuff higher on the priority list right now, but if you're willing to submit a Pull Request I could give you some pointers with regards to implementation ;).

Last.fm provides an API, that’s an option I think

There is an API in Jellyfin already that generates suggestions for a particular media item. This is something that looks quite straightforward to implement. Again, if anyone needs pointers how to get started on it, please get in touch!

This is implemented in v2.1.0!

Hi, sorry to bring this in again, not sure whether to make a new for this, but will it be possible to support 3rd party APIs for recommendations? I'm currently brainstorming a project related to generating recommendations it will be awesome if it this client (or jellyfin) is able to support it...

I would strongly prefer the recommendations to come from Jellyfin, so that there is only one datasource for the app. It's possible to create plugins for Jellyfin that might hook into the recommendation logic.
If a third party API is the only way to do it and there is strong support for the username, I would consider merging an implementation for it. But it would need strong privacy guarantees, easy setup and a de facto standard for it.


Update Research Statement

We should ask folks to create a new updated research statement & images!

I'll ask for a research statement and possibly updated images in the Slack, but I think it would be good to have an example. Minsik, would you mind writing your research statement and sending to me so that I can give yours as an example when I ask?


Allow to define operators without safe

Pull Request Test Coverage Report for Build 1611

12 of 15 (80.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 42.993%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/helpers/web3Providers/HDWalletSafeProvider.js
12
15
80.0%

Totals

Change from base Build 1608:
-0.03%

Covered Lines:
2179

Relevant Lines:
4672

💛  - Coveralls


Refactor/typestate connections

Very basic draft implementation for a typestate Connection type.
Note that the inner methods modifications are minimal for now, as the first step was to transpose the nested enums to typestate.

Drafted myself a small diagram while reviewing this, so I'll just drop it here
My thoughts:

The new implementation is nice in terms of structuring. In yesterday private discussion we it was suggested that state type pattern would remove the need for the "handlers" layer but it seems to be that Connection<I, T, S> would become equivalent of the current handlers layer, eg  the currentimpl Connection { (src/handlers/connection/connection.rs).

My main problem with the new implementation is following:

With existing implementation, the presence of states is hidden internally behind impl Connection handler/facade. Altough this enables consumers of Connection to call wrong methods (and be punished with error :-)), it on the other hand makes the client code simpler - and especially in terms of de/serialization. Lets' take following example:

let connection_serialized = database.load_connection_by_id("alice");
let connection = Connection::from_string(connection_serialized);
connection.some_operation()?; // we might get error if `some_operation` did not
                              // make sense in whatever state `connection` was at

However, with the new approach, assuming Connection<I, T, S> would present the public API for aries-vcx consumers, it gets trickier, but let me try anyway: (note I just ignore the transport / T generic arg)
let connection_serialized = database.load_connection_by_id("alice");
let connection: Connection <Inviter, T, CompletedState> = Connection::from_string(connection_serialized);
connection.some_operation(); // this is safer to call now, but we had to know the state ahead of time

well that works, but the developer needs to already know if the serialized connection was inviter or invitee and what state is that connection at, as it's now required at compile time. Which seems kinda impractical.
Perhaps the state pattern can be still adopted internally, but impl Connection<I, T, S> { } would be internal model, and this would be wrapped by "handler" like the impl Connection in /handlers today.

More of a side note: I like that this PR is addition of alternative implementation instead of mutating and disrupting existing code. The new Connection<I, T, S> structure would serialize obviously different and though possible to retrofit on old serialization format, I am not even sure if that would be worthy the effort. If this will move forward, I'd suggest continue in this fashion (perhaps even more structurally separate old/new approach), deprecate old style, let people migrate, delete old code.

@Patrik-Stas That's a very good observation and yes, that's the disadvantage described by me as well.
However, the serialization/deserialization scenario can still be worked around by introducing helper structs and perhaps even a trait for making things easier for users.
An oversimplified example is:
// User code:
enum MyStoredConnection {
    Http(StorableConnection<HttpClient>),
    Websocket(StorableConnection<Websocket>)
}

// Library code:
pub enum StorableConnection<T> {
    Invitee(StorableInviteeConnection<T>),
    Inviter(StorableInviterConnection<T>)
}

pub enum StorableInviteeConnection<T> {
    Initial(CommonConnectionData<T>),
    Invited(CommonConnectionData<T>),
    Requested(CommonConnectionData<T>),
    Responded(CommonConnectionData<T>),
    Complete(CommonConnectionData<T>)
}

pub enum StorableInviterConnection<T> {
    Initial(CommonConnectionData<T>),
    Invited(CommonConnectionData<T>),
    Requested(CommonConnectionData<T>),
    Responded(CommonConnectionData<T>),
    Complete(CommonConnectionData<T>)
}

pub struct CommonConnectionData<T> {
    marker: PhantomData<T>
}

pub trait StoredConnections {
    type StorableConnection;

    fn new() -> Self;

    fn store_inviter_initial<T>(&mut self, con: Connection<Inviter, T, InitialState>) {}

    fn store_inviter_invited<T>(&mut self, con: Connection<Inviter, T, InvitedState>) {}

    fn store_inviter_requested<T>(&mut self, con: Connection<Inviter, T, RequestedState>) {}

    fn store_inviter_responded<T>(&mut self, con: Connection<Inviter, T, RespondedState>) {}

    fn store_inviter_complete<T>(&mut self, con: Connection<Inviter, T, CompleteState>) {}

    fn store_invitee_initial<T>(&mut self, con: Connection<Invitee, T, InitialState>) {}

    fn store_invitee_invited<T>(&mut self, con: Connection<Invitee, T, InvitedState>) {}

    fn store_invitee_requested<T>(&mut self, con: Connection<Invitee, T, RequestedState>) {}

    fn store_invitee_responded<T>(&mut self, con: Connection<Invitee, T, RespondedState>) {}

    fn store_invitee_complete<T>(&mut self, con: Connection<Invitee, T, CompleteState>) {}

    fn store_inviter<T>(&mut self, con: StorableInviterConnection<T>) {
        match con {
            Initial(con_data) => self.store_inviter_initial(con_data.into()),
            Invited(con_data) => self.store_inviter_invited(con_data.into()),
            Requested(con_data) => self.store_inviter_requested(con_data.into()),
            Responded(con_data) => self.store_inviter_responded(con_data.into()),
            Complete(con_data) => self.store_inviter_complete(con_data.into()),
        }
    }

    fn store_invitee<T>(&mut self, con: StorableInviteeConnection<T>) {
        match con {
            Initial(con_data) => self.store_invitee_initial(con_data.into()),
            Invited(con_data) => self.store_invitee_invited(con_data.into()),
            Requested(con_data) => self.store_invitee_requested(con_data.into()),
            Responded(con_data) => self.store_invitee_responded(con_data.into()),
            Complete(con_data) => self.store_invitee_complete(con_data.into()),
        }
    }

    fn store_connection<T>(&mut self, con: StorableConnection<T>) {
        match con {
            Invitee(invitee_con) => self.store_invitee(invitee_con),
            Inviter(inviter_con) => self.store_inviter(inviter_con)
        }
    }

    fn handle_storable_connection(&mut self, con: Self::StorableConnection);
}

The idea is that the Connection<I, T, S> would serialize to StorableConnection (can be done under the hood by serde) and users would just wrap it to provide transport distinction (not mandatory if only one transport is used throughout the agent - StorableConnection<Transport> can be used in this case). Deserialization would happen to the user defined type (or StorableConnection<Transport> and could be uniformly handled through a StoredConnection trait impl.
An implementor of StoredConnection can decide which connections it wants to store (as all default to no-op) and how to store them and use them throughout the rest of the code (could use some type-based HashMap, Box<dyn Any>, multiple HashMap or Vec, channels for sending the connections to other parts of the code, etc.). The only "mandatory" method to implement would be handle_storable_connection where users implement the transport distinguishing logic.
NOTE: An idea might be to return errors from the trait methods, but this was merely written as an example.
This is indeed more advanced but definitely not extreme.

Codecov Report

Merging #739 (8fff813) into main (66ee076) will decrease coverage by 0.14%.
The diff coverage is 17.35%.

:exclamation: Current head 8fff813 differs from pull request most recent head ad9a6fe. Consider uploading reports for the commit ad9a6fe to get more accurate results

@@            Coverage Diff             @@
##             main     #739      +/-   ##
==========================================
- Coverage   54.73%   54.59%   -0.14%
==========================================
  Files         363      374      +11
  Lines       37122    37218      +96
  Branches     8089     8096       +7
==========================================
+ Hits        20317    20321       +4
- Misses      10832    10926      +94
+ Partials     5973     5971       -2

Flag
Coverage Δ

unittests-aries-vcx
54.49% <17.35%> (-0.14%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

 Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Hi @gmulhearn have a look at this, I am interested to hear your thought on this. Consider waiting until today's video recording is uploaded on https://wiki.hyperledger.org/display/ARIES/2023-01-26+Aries-vcx+community+call where @bobozaur goes through the approach taken here.

Hey @Patrik-Stas  sorry for delays, just got around to catching up on the recording and this PR now.
This is a very exciting POC 👌.
@bobozaur  I share similar concerns around what ser/de in agents which store connections persistently. In the recording, I believe you mention how depending on the situation, you could assume that dev would know at compile time what the expected state is. But ocourse that's not always the case. E.g. imagine an ACA-py-like agent, where the agent persistently stores connections in permutations of different states, ACA-py fetches those connection records from DB without knowing at all what state permutation they are in - this is the most common pattern in DI agents from what I've been exposed to.
I know you've covered the ser/de concerns several times. However, maybe a good way for you to make the solution clearer is if you try to build onto the rust agent (within agents/rust/aries-vcx-agent) to store these new connection structs persistently (or perhaps even storing the string serealized connections to further drive home the solution).

Thinking out loud, but...
IMO, the main style of DI agents is where;

an agent takes a step in a protocol, 2. stores the protocol state/handler in some persistent DB, 3. then waits for a new message 4. handles that message (which may involve fetching a protocol state/handler from the DB), 5. repeat.

so perhaps when you're at step 4 of the above, the type of message that you receive is enough to infer the exact type of the handler. e.g. if you receive a ConnectionRequest message, when you fetch the appropriate Connection from the DB, your 'connection request' handler code could infer that the type should be Connection<Inviter, InvitedState>. So maybe it's generally not a problem to expect codebases to know the expected type of their serialized handler.
I do however share the concern about this increasing complexity for consumers. But I'm not sure how much of a concern that should be.. Rust is generally more complex, so you could argue that it's good we are taking advantage of rust features. Further, this concern can be mitigated with very good docco and some good code examples.

@gmulhearn There's still work to be done in this, but the serialization/deserialization part would essentially happen like this:

Concrete connection type (e.g: Connection<Inviter, Requested>) -> Some intermediate type (used for alignment between serialization/deserialization) -> String: This is all under-the-hood. The user has access to call Connection<Inviter, Requested>::serialize(serializer) as you would with any type and serialize directly.
String -> VagueConnection (type that can deserialize any connection, exposed by the API) -> Concrete connection type (e.g: Connection<Invitee, Complete>): There's TryFrom implementations for all connection states to attempt conversion from VagueConnection if the state is the expected one. This would allow users that know they expect a certain connection state to deserialize into that directly.
String -> VagueConnection -> Runtime pattern matching -> Handle concrete connection type: Since the VagueConnection is exposed, users can take advantage of the type for things like storing all connections in a Vec or any sort of thing. They can then either use TryFrom or pattern matching on the state to handle concrete connection types based on their state.

Is this more complex than it used to be? Definitely. Is it worth it? I wholeheartedly believe it is. As you pointed out, Rust is more complex. Regardless of how user-friendly an API we expose, it will never be easier for people to develop a Rust agent than, for instance, a Python one (assuming a similar level of knowledge of the two languages).
Don't get me wrong, we should definitely make the library as user-friendly as possible, and you also pointed out other important aspects such as examples and documentation, but on the other hand nobody writes Rust because of how easy it is to code in. So I'd say that we should focus on the language's strengths, such as performance, safety and robustness at the cost of being somewhat more verbose and complex.
Another important idea that I also mentioned in the community call was that while this kind of feels like forcing users to handle and deal with certain aspects of the library, I'd say that we do that because they should do that anyway. We enforce the idea of being aware of the state a connection is in and handle it appropriately because... they should be aware.
We make them handle connection states separately because, again, they should. I'd argue that we're in fact helping them a bit by pointing out "Hey, these connections do different things because of the different states. Handle them separately".
@Patrik-Stas mentioned the same assumption that you pointed out regarding the ConnectionRequest example. You get a ConnectionRequest and you assume the connection will be in the invited state, but there's no guarantee. So with the current API you could try to handle the request, and progress the state machine if the state is the expected one or return an error if it isn't. Or, you check the state before even trying, and handle that instead.
With the proposed API you always know the state you should be in to take a specific action, and simply try to get into that state and handle whether that succeeded or not. After that it's all compile-time guarantees and a lot less pattern matching.
Regarding the example agent, we could probably do that too after I finish the libvcx implementation. You could look at that for an example of how you can take advantage of the VagueConnection in the general case and then attempt to convert a connection to the expected state in order to take a specific action.

Hey @bobozaur could you please let me know as well when this is in a final review-ready state (if not already). Would love to review these changes

@gmulhearn Almost there. I still need to integrate some stuff from #163 for backwards compatibility with agents built on top of the wrapper. I'll update the PR when it is done.

Necessary changes from #741 to get the wrapper working were done. I'd say this PR is in it's final state, so please feel free to review it.

Just finished a pass thru and couldn't really find anything other than a few small things. Looks really good. I'm not too versed with the napi stuff, so I have no comments there.

I may have noticed a potential bug. After the invitee calls send_request, if the invitation was pairwise, the thread_id for the invitee Requested state is not set as the request message's ID. Shouldn't it be set as that, such that when the verify_thread_id is called in handle_response it can verify that the Response message thread_id matches the request ID that we sent?
My reading of RFC 0160 is that the invitee's request ID should == the inviter's response thid.
I'm a little confused about how this works, could you guys shed some light on if I'm misunderstanding? a test could be nice.
Also two other things;
it would be nice if the Request message's ID is set as a UUID rather than using a default ("testid") - like how the generate_invitation does now.
is it necessary to set the thread_id of the request message? rfc 0160 does not seem to require or imply this.

@gmulhearn I replicated the old behavior, so I'm not 100% certain of the implementation details, but from my limited understanding a pairwise invitation is for a pair of agents. You wouldn't expect a second invitee to accept it. So the thread ID can remain the invitation ID, as it leads to contiguous context tracking.
For the Request ID, it's out of scope for this PR. It's the old behavior and while I agree it should change, probably many things could/should 😅 .

The current failure on nodejs serialization in CI is fixed in this PR
https://github.com/hyperledger/aries-vcx/pull/749

@gmulhearn is testid  actually ever used as value in non-test context?
@gmulhearn About the logic of setting @id as response to pairwise invitation, I agree with @bobozaur in scope of this PR I'd just stick to original behaviour, for starter.
We can discuss this separately, but I'll briefly share my opinion - I think this is actually right behaviour, as in case of connection protocol kicked-off by inviter sending pairwise invitation, that's esentially the first agent2agent (altough out of band) message kicking off the protocol with certain @id. Then the connection-request just continues in that thread (dictated by invite @id).

Kk no worries, I might open an issue where further discussion can happen. But ye agreed, not in scope here


Crash on Android

Describe the bug
I am trying to run react-native-zendesk-messaging on Android. It's works fine on iOS but issue on Android.
To Reproduce
Steps to reproduce the behavior:
1 - Open terminal
2 - Navigate to project
3 - Run command yarn android
Expected behavior
A clear and concise description of what you expected to happen.
**Environment **

Platform: [iOS, Android]
React Native Version [0.68.6]
Zendesk SDK Version [2.8.0]

Can I get logs that run with --verbose flag? @Numan-Munir

failing for me as well:
@leegeunhyeok here is the verbose logs:
FAILURE: Build failed with an exception.
* What went wrong:
Could not determine the dependencies of task ':app:processDebugResources'.
> Could not resolve all task dependencies for configuration ':app:debugRuntimeClasspath'.
   > Could not find zendesk.messaging:messaging-android:2.13.0.
     Searched in the following locations:
       - https://oss.sonatype.org/content/repositories/snapshots/zendesk/messaging/messaging-android/2.13.0/messaging-android-2.13.0.pom
       - https://repo.maven.apache.org/maven2/zendesk/messaging/messaging-android/2.13.0/messaging-android-2.13.0.pom
       - file:/Users/ankurankur/Documents/Float/mobile-app/node_modules/jsc-android/dist/zendesk/messaging/messaging-android/2.13.0/messaging-android-2.13.0.pom
       - https://dl.google.com/dl/android/maven2/zendesk/messaging/messaging-android/2.13.0/messaging-android-2.13.0.pom
       - https://www.jitpack.io/zendesk/messaging/messaging-android/2.13.0/messaging-android-2.13.0.pom
     Required by:
         project :app > project :react-native-zendesk-messaging

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

i found a fix in latest react native version it does not allProjects section in android/build.gradle so instead of adding maven link in repositories directly I added it as below
allprojects {
    repositories {

        maven {
            url "https://zendesk.jfrog.io/artifactory/repo"
        }

    }
}

make sure this block of code is outside of buildscript block.
@leegeunhyeok I am not sure if this can be fixed in package itself

@Ankur-Float This is already covered in the android section of getting started guide.
(In root build.gradle, add repository to allprojects)
https://github.com/leegeunhyeok/react-native-zendesk-messaging/blob/master/docs/getting-started.md#android

we faced a similar issue that was related to some conflicting build issue with Firebase performance https://github.com/firebase/firebase-android-sdk/issues/3155
we have fixed the issue by upgrading AGP to v7.2 and firebase:perf-plugin to v1.4.2


HHH-13374: workaround for JDK bug

https://hibernate.atlassian.net/browse/HHH-13374
Complete with testcase. Forced update to fix formatting.

It just hit me that once the bug gets fixed, then this will invert it. I know how to fix it. Will do it tomorrow on my workstation and rebase after HHH-13357 is fixed by #2835 or something, so all the tests will pass.

Many thanks @coladict ! Do you know if the JDK bug got assigned an identifier? Would love to track this.

I reported it months ago, but never got confirmation of it being assigned. Another bug I reported after it did get a number, and is already fixed in JDK-14.

I'll close this as out of date since this appears to be fixed in newer JDK versions already.


Added the -r flag to userfile upload docs 21.2

Closes #10875
Added the new -r / --recursive flags to 21.2 docs, including:

Flag included in cockroach userfile upload flag table + description
Added to synopsis section on cockroach userfile upload page
Added usage example
Updated notes on Userfile for bulk operations page

This change is

Hey @adityamaru, adding you as engineer review as you reviewed Nihal's PR - lmk if someone else needs to be tagged here for review. Thank you!

TFTR Aditya, feedback incorporated 👍


Docker build returning incorrect results

We utilize the docker image using @latest and the 2.2.0 is not returning correct results.  Here are some examples:

curl http://geo2tz:2004/tz/43.74206/-87.73337
{"coords":{"lat":43.74206,"lon":-87.73337},"tz":"America/Detroit"}

Should be returning tz: America/Chicago

curl http://geo2tz:2004/tz/44.50144/-88.0625889
{"coords":{"lat":44.50144,"lon":-88.0625889},"tz":"America/Detroit"}

Again, tz: America/Chicago is correct.

curl http://geo2tz:2004/tz/50/40
{"coords":{"lat":50,"lon":40},"tz":"Europe/Kyiv"}

Should be returning tz: Europe/Moscow

Please let me know if any further information is needed.  Thanks!

thanks for reporting, yes this version fixes errors where location were not found at all but it is less precise than before. I will work on a fix, meanwhile use version 2.1.5:

@LarrySonar should be fixed in https://github.com/noandrea/geo2tz/releases/tag/v2.3.0 can you give it a go?

This is indeed resolved in that release, all of the previous issues have been corrected.  Thanks for the rapid fix!

While fixed in 2.3.0 it is broken in 2.4.0.  The coordinates 0, 0 are returning Pacific/Honolulu and should be returning Etc/GMT.

I cannot reproduce, running:

I cannot reproduce, here are the results I get running it locally:
version request
➜ curl -s http://localhost:2004/tz/version | jq
{
  "version": "2024a",
  "url": "https://github.com/evansiroky/timezone-boundary-builder/releases/tag/2024a",
  "geo_data_url": "https://github.com/evansiroky/timezone-boundary-builder/releases/download/2024a/timezones-with-oceans.geojson.zip"
}

coordinates request
➜ curl -s http://localhost:2004/tz/0/0 | jq
{
  "coords": {
    "lat": 0,
    "lon": 0
  },
  "tz": "Etc/GMT"
}

server logs
➜ docker run --pull=always -p 2004:2004 ghcr.io/noandrea/geo2tz:latest
latest: Pulling from noandrea/geo2tz
Digest: sha256:6f4db352cbe26ecb415a3ab600f106a9c3cbf4feddea05732f76502a880b21cb
Status: Image is up to date for ghcr.io/noandrea/geo2tz:latest

  _____           ___  _
 / ____|         |__ \| |
| |  __  ___  ___   ) | |_ ____
| | |_ |/ _ \/ _ \ / /| __|_  /
| |__| |  __/ (_) / /_| |_ / /
 \_____|\___|\___/____|\__/___| version 2.4.0
⇨ http server started on [::]:2004
{"time":"2024-07-01T14:59:22.315462477Z","id":"","remote_ip":"<IP_ADDRESS>","host":"localhost:2004","method":"GET","uri":"/tz/version","user_agent":"curl/7.81.0","status":200,"error":"","latency":20320,"latency_human":"20.32µs","bytes_in":0,"bytes_out":234}
{"time":"2024-07-01T14:59:28.355996568Z","id":"","remote_ip":"<IP_ADDRESS>","host":"localhost:2004","method":"GET","uri":"/tz/0/0","user_agent":"curl/7.81.0","status":200,"error":"","latency":67025,"latency_human":"67.025µs","bytes_in":0,"bytes_out":44}

closing as cannot reproduce, @LarrySonar please reopen if you have more details about the issue you are facing


Add support for exporting Assets Trends on a date range

https://mint.intuit.com/trend.event Allows filtering based on date.  I would like to be able to get the account data based on a specified date.
searchQuery: {"reportType":"AA","chartType":"P","comparison":"","matchAny":true,"terms":[],"accounts":{"groupIds":["AA"],"accountIds":[],"count":25},"dateRange":{"period":{"label":"This month","value":"TM"},"start":"8/1/2021","end":"8/29/2021"},"drilldown":null,"categoryTypeFilter":"all"}

Have you tried the --start-date parameter, and does that accomplish what you are looking for? Or are you looking for an end date as well?

I am looking for the end date.  I tried --start-date and it doesn't change anything in the output of --accounts.

duplicate of #208

Unless someone beats me to it, once I finish getting everything going with mintapi, I'll take a crack at this.

I'd just like to get some clarification here on what the expectation is when filtering accounts based on date.  Is the expectation that the results are filtered based on the accounts opened during that date, the accounts that have activity during that date, etc?

@timdau thanks for the clarification.

So I went in and reviewed this a little bit more.  The section of Mint that you're referencing is really looking at trends and uses a different API call than the current handling of the account data.
Presently, we use bundledServiceController.xevent to get the account data, while the trend section uses trendData.xevent with a different set of input parameters.  It could be something we explore, but with as major of a change as I'd imagine that would be, I'd want to get @mrooney and @jaron-l to give thoughts on this first.

@jaron-l agreed.  I'll add this to my list of tasks.  Thanks!

Noting on here that I had started to work on this and ran into some roadblocks and ended up pivoting.  Will revisit when I get a chance.

Thanks @burkematthew, Happy New Year

@burkematthew Who do I owe the $200USD as the thank you gift?

@timdau this was made possible due to the contributions of @eyadgaran !

Thanks Matthew.  @eyadgaran please let me know how you want me to send you the thank you gift.  You can reach out to me paul{at} hessels(.) ca.    Just let me know.

lol no need! i was happy to do it for my own selfish needs. feel free to send it to @burkematthew and @mrooney for putting in the work to maintain this project. cheers!

@burkematthew spent a billion hours to make a ton of work happen, I nominate him :)

@timdau if you have Venmo, you can send it to @matthew-burke-26.

@burkematthew I just tried to sign up for venmo and it wouldn't let me because I am not in the USA.

@timdau just messaged you on Discord.


CNAME Record IP Address

Hello,
I'm wondering what to set my CNAME record for my www name to? When I tried setting it to <IP_ADDRESS> Cloudflare gives an error DNS Validation Error (Code: 1004) Content for CNAME record is invalid.

Ah I figured it out, I had to add the worker URL to support *yourdomain.com/*. I also added an A record with www set to <IP_ADDRESS> rather than a CNAME.

Ah I figured it out, I had to add the worker URL to support *yourdomain.com/*. I also added an A record with www set to <IP_ADDRESS> rather than a CNAME.


DB/ Pick Data Access Technology

Since we use Reactive approach, in particular Project Reactor and WebFlux Spring R2DBC was picked as Database access primary tool.


Chef Server 12: View Public Keys of all Users, including clients

We have been using chef-vault to deploy secrets to our Chef clients.  Specifically, we often generate secrets in recipes, then encrypt them for other nodes and users by reading their public keys via the Chef API.
This is not currently working in Chef 12, because clients cannot read public keys for users.
This issue was originally reported here:
https://github.com/chef/chef-server/issues/20
A partial fix was implemented in PR 244:
https://github.com/chef/chef-server/pull/244
The changes in PR #244 allow users to read client keys.  Unfortunately, it still doesn't work the other way around: clients cannot read user keys.
Authenticated as a client, I would like to be able to read users' public keys via the API.

Issue #20 was re-opened, so this is now a duplicate issue.

Thanks @http-418 sorry for the confusion!

A team I work with mentions:

The team is looking to encrypt chef-vault items with keys during a Chef run. They are using the Chef Client runs and likely executing some kind of search to update a vault item. I would need to get the specifics, but we’re considering this a regression as it is something that was working prior.

Yes, this worked fine in Chef 11.  The chef-vault resources could read users' public keys.
I've been moving the codebase to Chef 12, and this feature is now broken.

It's worth mentioning that under Open Source Chef Server 11, there was no RBAC , so nothing would prevent this.
Did it work under Enterprise Chef Server 11?

I don't know whether it works with Enterprise Chef Server 11.   I have not attempted it.

This PR adds GET /organizations/:organization/users/:user/keys(/:key) to the API, adds a new group public_key_read_access that controls its permissions, and changed the permissions on GET /organizations/:organization/clients/:client/keys(/:key) to also be controlled by that group.
Now, GET on both those endpoints can be accessed by members of the group public_key_read_access. This group can be modified by admin users per org but defaults to containing the users and clients groups. So now, by default, org scoped user and client key GETs will be accessible by users and clients within the same org as the requested key.
Update for Chef Vault to leverage this work coming soon.

Being fixed for vault in this PR: https://github.com/chef/chef-vault/pull/209
Note that you will need to be on Chef Server 12.5+ to see the benefits of this though.


JSON everywhere

Most error HTTP status returned from the API have a non-JSON body. Everything should respond with JSON since it's a RESTful API!

Turns out Rocket will already respond with JSON when handling errors, providing the request supplies a Accept: application/json header. I've began work on a Fairing, DefaultJSON in feat/json-everywhere, which overrides the Accept header (or provides if it's not supplied) to application/json, which has the effect of making all error responses JSON whilst still letting the main application code respond in whatever it wants (i.e plain/text for the health-check and YAML for the docs).
There is however, the 406 HTTP status code, which is explicitly for when the server cannot respond with any of the MIME types specified in Accept headers, so we could only return with JSON and respond with a 406 if the client won't accept it, however the issue here is that Rocket currently doesn't support letting Fairings terminate or respond to requests in on_request (see the overview for Fairings in the Rocket docs. This is something planned for v0.6 of Rocket, but it's currently blocked with some hacky workarounds suggested (see https://github.com/rwf2/Rocket/issues/749).


OS Error: Permission denied, errno = 13 when writing  a File in Temp Directory

Steps to Reproduce

Trying to create the Directory if doesn't exist
  final Directory systemTempDir = Directory.systemTemp;

      // created theory directory if doesn't exists
      final Directory theoryDir = Directory('${systemTempDir.path}$theoryDirectoryName');
      if (!theoryDir.existsSync()) {
        await theoryDir.create();
      }

Logs

FileSystem Exception : Creation failed , path '/data/local/tmp/theory/ OS Error: Permission denied, errno = 13

[✓] Flutter (Channel dev, v0.10.1, on Mac OS X 10.13.6 17G65, locale en-DE)
    • Flutter version 0.10.1 at /Users/driveddy_android/Applications/flutter
    • Framework revision 6a3ff018b1 (13 days ago), 2018-10-18 18:38:26 -0400
    • Engine revision 3860a43379
    • Dart version 2.1.0-dev.7.1.flutter-b99bcfd309

[✓] Android toolchain - develop for Android devices (Android SDK 28.0.2)
    • Android SDK at /Users/driveddy_android/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1136-b06)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.0, Build version 10A255
    • ios-deploy 1.9.2
    • CocoaPods version 1.5.3

[✓] Android Studio (version 3.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 29.1.1
    • Dart plugin version 181.5656
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1136-b06)

[!] VS Code (version 1.28.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (1 available)
    • HTC M10h • FA65ZBN04140 • android-arm64 • Android 8.0.0 (API 26)

! Doctor found issues in 1 category.

In Flutter you need to use https://pub.dartlang.org/packages/path_provider
Please consider asking support questions in one of the other channels listed at http://flutter.io/support .


Add the ability to run specific tests with the invoke-unit-tests task

What does this PR do?

Add the ability to run specific tests with the invoke-unit-tests task
Motivation

Previously I had to use the Python interpreter directly, for instance:
❯ python -m unittest tasks/unit-tests/linter_tests.py tasks/unit-tests/junit_tests.py
...................
----------------------------------------------------------------------
Ran 19 tests in 0.455s

OK

Now:
❯ deva invoke-unit-tests -t tasks/unit-tests/linter_tests.py -t tasks/unit-tests/junit_tests.py
...................
----------------------------------------------------------------------
Ran 19 tests in 0.445s

OK

Or even:
❯ deva invoke-unit-tests -t tasks.unit-tests.linter_tests.TestIsGetParameterCall.test_no_get_param
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

Additional Notes

invoke-unit-tests without parameters works the same way as before
Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


importccl: support foreign keys in mysqldump IMPORT

PR: https://github.com/cockroachdb/cockroach/pull/27861
From release notes:

Added foreign key support to IMPORT ... MYSQLDUMP. [#27861][#27861] {% comment %}doc{% endcomment %}

Fixed by  #3766


add chatie landing page

preview: https://lijiarui.github.io/chatie-coming-soon/index.html

Sorry to confuse you.
Make it work:

Add github page for master branch.
Change id for mailchimp as I show above.

Btw, do you know how to get mailchimp form id? If not, l will add information for mailchimp

It seems that you did not get my point at all ...
What I said is that you should understand how this repository works.

Well, to be honest,I'm not sure how this repo work because it doesn't looks like a static page and I cannot find any related  intro about this repo.
After read all of the code more, still not pretty sure, l guess you mean I should put all files in bin folder?

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Close this PR because it's already published at https://chatie.github.io/www


add: expose self as pouchdbUpsert and bind to self

so we can use this lib in webworks =)
ref nolanlawson/pouchdb-find#200

See my comment in https://github.com/nolanlawson/pouchdb-find/pull/200#issuecomment-229388923 ; we could definitely try to detect all global usages of PouchDB in every possible env, but I try to avoid doing so because it's a slippery slope, thanks for understanding. :)

well, if do not generate/pack the script in "standalone" mode, there is no way to importScripts and apply it to PouchDB as far as i can see.
i fully understand your concertn, but maybe browserify the script in standalone with with -s pouchdbUpsert ?

Yes, just browserify and insert into your worker, that's what I would do. TBH we should have made our own bundle a standalone UMD bundle, and the fact that we didn't is a bug. I'll fix that.

fixed in 2.0.2 now


Do not allow mockery >= 1.6.3

With mockery version 1.6.3 the location of the helper script changed to src/helpers.php.

https://github.com/mockery/mockery/compare/1.6.2...1.6.3#diff-7827dcf8c28ba9d97397f03b4ca75c132540f5c377e253fbc47ef064a373ce73
https://github.com/mockery/mockery/pull/1290

This is why the coding-standard currently fails with:
Running PHPStan
Scanned file /app/vendor/mockery/mockery/library/helpers.php does not exist.

For now we do not use mockery version 1.6.3 to prevent this error, but we should find a better solution in the long run.
(I would also suggest not to get the dependencies dynamically for every coding-standard run in the CI, but instead use fixed versions of the dependencies.
FYI: The src/helpers.php, it is automatically autoloaded by composer.

Many thanks for fixing it @ghostwriter :smiley:
So we should be fine again and can close this PR :+1:
We do not seem to use the autoloading at this point, but @rdss-sknott knows more about it.. :wink:


FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged.js'

I'm executing:
python appmon.py -a com.example.test --spawn 1 -p android -s scripts/Android -o d:/app_mon_dumps -r dump.db
It gives me an error:
Now Spawning com.example.test
[INFO] Attached to com.example.test
[ERROR] [Errno 2] No such file or directory: '/tmp/merged.js'
Traceback (most recent call last):
  File "appmon.py", line 361, in <module>
    script = session.create_script(generate_injection())
  File "appmon.py", line 194, in generate_injection
    with codecs.open(merge_scripts(script_path), 'r', 'utf-8') as f:
  File "appmon.py", line 143, in merge_scripts
    with codecs.open(merged_script_path, "w", "utf-8") as f:
  File "C:\Users\........................\Python37\lib\codecs.py", line 904, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merged.js'

I found same error at .
It was solved by adding to PATH:
merge_script_path = ‘C:/Users/<nombre_usuario>/AppData/Local/Temp/merged.js’
But it doesn't helped. Please help, what should I do?

Everything is works FINE!


Key instanceof Array is failing

Eventually, we are trying to use your library in our GWT project and have the following problem:
Line https://github.com/vibornoff/asmcrypto.js/blob/master/src/rsa/rsa.js#L17
if ( key instanceof Array ) {
Is failing the check and thus, we get an exception constantly. However, we pass a key as an Array.
After googling a while, I've found out the following:

This approach will fail if obj was created in a different frame. Array.isArray doesn't have that issue if you can target EcmaScript5.

And indeed, GWT does some work in a different frame.
I've put the following code for an expirement:
if ( Array.isArray(key) ) {
And it did work fine for us.
So, the question is if you can use Array.isArray for the checks?

Does it compatible with IE10?

Array methods are compatible with all browsers listed in the compatibility table:
http://kangax.github.io/compat-table/es5/

Thanks, looks legit. But the problem is not only with instanceof Array.
Suitable replacement is need also for:

instanceof ArrayBuffer: https://github.com/vibornoff/asmcrypto.js/blob/master/src/utils.js#L135-L137
instanceof Uint8Array: https://github.com/vibornoff/asmcrypto.js/blob/master/src/utils.js#L139-L141
instanceof <TypedArray>: https://github.com/vibornoff/asmcrypto.js/blob/master/src/utils.js#L143-L149 (ArrayBuffer.isView doesn't work with IE10 and probably Safary).

For those methods, I can't say anything. Either they are not used in our case, or they work.
We use RSA encryption, SHA512 hashing and HMAC_SHA signing.
What are the system requirements?

IE10+, FF22+, Opera12+, Safari5.1+


Terms query can accept encoded terms input as bitmap

Description
Document the newly added bitmap filtering feature in 2.17.
Issues Resolved
Closes #8121
Version
2.17
Checklist

[x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@bowenlan-amzn I have reviewed and pushed my edits into your PR. Please verify that everything is still technically correct (especially the expanded Python example: please make sure I'm using the correct library). Also, for the value_type, please provide the valid values and default value. Thank you!


Projects referencing Nuget v1.0.2 doesn't compile with MediatR nuget package 8.1.0

The sample classes in the wiki complain and look for v8.0.1 of MediatR and not the latest version although dependencies say >=8.0.1

I updated the package references. Let me know if v1.0.3 works.

The sample classes in the wiki complain and look for v8.0.1 of MediatR and not the latest version although dependencies say >=8.0.1

I updated the package references. Let me know if v1.0.3 works.

The sample classes in the wiki complain and look for v8.0.1 of MediatR and not the latest version although dependencies say >=8.0.1

I updated the package references. Let me know if v1.0.3 works.


Hani hassan

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: dac10c25-1625-4f0d-b113-a1fdd7a7a900
Version Independent ID: dac10c25-1625-4f0d-b113-a1fdd7a7a900
Content: Get started - Overview - Teams
Content Source: msteams-platform/get-started/get-started-overview.md
Service: msteams
GitHub Login: @surbhigupta
Microsoft Alias: lajanuar

@hassanhani-44 - Could you please share more details about your issue?

This looks like an invalid issue.
We are closing this issue for now, please feel free to reopen it, if required.


RStudio debugger broken

The RStudio version I'm getting from the current container is 1.2.5033, it crashes whenever I try to debug.
f <- function() {
    1
}

debug(f)
f()

This has been reported as a RStudio bug https://github.com/rstudio/rstudio/issues/5888, and supposedly fixed in RStudio 1.2.5036.

We'll have to wait for rocker to update their rstudio version for this to be fixed i'm afraid @Shians


Graceful shutdown of HTTP server and worker threads

This change implements graceful shutdown to HTTP server and MultiThreadAgent threads.

Undertow creates XnioWorker automatically but here doesn't use it. If we use it, Undertow.stop calls xnioWorker.shutdownNow immediately. However, it shouldn't be called until GracefulShutdownHandler.shutodwn is called to make sure that the HTTP handlers are not processing requests. Therefore, here creates XnioWorker and controls shutdown order manually.
Upgraded Undertow to 1.4.x because Undertow.Builder.setWorker is not available in 1.3.
A new shutdownHandler waits for completion of ServerControl.destroy(). It calls @PreDestroy methods. Thus shutting down digdag server will wait for completion of @PreDestroy LocalAgentManager.shutdown method.
LocalAgentManager.shutdown calls MultiThreadAgent.shutdown. It waits for completion of all running tasks.

@danielnorberg PTAL
Multi-process agent is a next step that might not be implemented soon :-p

Does this mean that the digdag rest api becomes unavailable until all currently running operators finish?

Does this mean that the digdag rest api becomes unavailable until all currently running operators finish?

Yes. It's another reason why we should run agents on different machine.

Ok, please add a test verifying that the rest api is available while the server waits for task completion.

@danielnorberg can I ask you to check new test?
The test runs only on unix systems because there're no good ways for Java to send signals to a PID on Windows.

Undertow 1.4.0.Final seems released but not on maven central yet somehow.
https://github.com/undertow-io/undertow/commits/1.4.x
Release note:
https://issues.jboss.org/secure/ReleaseNote.jspa?projectId=12312521&version=12330950

👍

Now I could download undertow 1.4.0.Final, somehow... Updated.

👍  LGTM


DID Context Hack MD as Input Document

https://hackmd.io/U4aRb1QOQsqjNDWA_kKhnQ
this hackmd describes concepts like "connections" and "metadata" that are in the current spec, it's 2 years old and has no comments, but might be useful as we attempt to refactor / restructure the concept of "correlation".

related:

https://github.com/transmute-industries/universal-wallet/issues/11
https://github.com/transmute-industries/universal-wallet/issues/7


Two suggested improvements to the pseudo instruction table

I would like to suggest two improvements to the table with pseudo instructions:

For pseudo instructions that maps to more than one real instruction, put the real instructions on separate lines. Today the real instructions are separated with ';', and the second instruction is wrapped half way onto the next line. This makes it quite hard to see the real instructions and their operands, fields.

Somewhere, either before or after the table explain 'myriad sequences', an expression which only appears in the table. For non-native English speakers, this short expression may be hard to understand what it implies.

Both suggestions sound good.
Are you interested in providing PRs as well?

Fixed with #101.

Looks really good!


streaming ping pong benchmarks send 0 length replies

This seems wrong?

Ping ?

Obsolete?

Indeed obsolete. Fixed by an earlier PR.
On Tue, Apr 26, 2016 at 1:20 PM Craig Tiller<EMAIL_ADDRESS>wrote:

Obsolete?
—
You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
https://github.com/grpc/grpc/issues/4082#issuecomment-214873644


Implementation details and request for comments

iowatcher-ng initiative
Rationale

Main aim is replacing blktrace/blkparse/iowatcher et al.
In order to upgrade to our today's landscape: things don't happen all time in your shell.
Using TCP sessions to send sensor data might not be optimal.
Graphing tools can be replaced by Grafana/Kibana/<choice of $user>.
Following the UNIX concept:  do one thing and do it well.

Stack of choice
#3 issue related.

Golang is huge in the cloud ecosystem.
Golang provides access to C ABI somewhat painlessly.
Golang is GC so less prone to segmentation faults.

Golang is trendy :smile:
GoEBPF is enough stable on further iterations

Limitations of v1 proof of concept

Depending on blktrace output following blktrace_api.h

Some workarounds on the basic DebugFS and blktrace api

eBPF bpftrace

Data collection algorithm
Incoming from blktrace output
Like a stream of this.
Event filtering
Looking at this.
Output JSON payload
#2 issue related.

[ ] Host id
[ ] Blockdevice id
[ ] PID
[ ] Event category
[ ] Event payload encoded in JSON

@sbates130272 I will come back later and take it from here in order to expand this issue/and/or answer comments :smile:

@escanda some feedabck

Forget about eBPF for v1. We will look at this after v1. v1 will be blktrace based.
The Prometheus scrape metrics are not JSON based. The format for them is more simple. As we work together to define item #2 this will become clearer.

We do have though, @sbates130272 , to decide on a metrics library to gather info

Yes this looks like the right library. Please work with this one.

I am closing this now since we are closing in on a v0.1.


.NET Standard support?

I have been waiting for this to exist for years! Finally a way to step into and debug NuGet packages.
I am trying to get SourceLink to work with our in-house NuGet package.
I am running a netcore2.1 web application which references a netstandard2.0 NuGet package hosted on our Azure Devops feed.
Question 1: Are .NET Standard packages supported?
I have followed the instructions in this post.
The Build pipeline Publish symbols path appears to be working - in the logs I see:
Succeeded processing D:\a\1\s\src\MyCompany.Core.Services.SnapshotClient\bin\release\netstandard2.0\MyCompany.Core.Services.SnapshotClient.pdb:

When I start debugging I see a bunch of output in the Output window:
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.1.4\Microsoft.AspNetCore.Hosting.dll'. Cannot find or open the PDB file.

But for my NuGet package I see "Symbols loaded" which seems promising. BTW, my NuGet package does not include .pdb files Question 2 - should it?
However, I never see the prompt "Source Link will download from the internet" in your screenshot above.
Consequently, when I debug and attempt to Step-In to my NuGet package, it just steps over it.
I have tried:

Following the instructions on this page
When that didn't work, I headed over to https://github.com/dotnet/sourcelink and followed their instructions and installed the Microsoft.SourceLink.Vsts.Git package (Question 3 is that necessary?)
When that didn't work, I upgraded every darn package in my application, which forced me to install .NET Core SDK 2.1.403 which forced me to reboot my machine which caused me to lose this post and have to type it in again!
Update: Tried adding some stuff to the .csproj of my NuGet package, after trawling GitHub issues

<PublishRepositoryUrl>true</PublishRepositoryUrl>
<AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder>

and
<DebugType>portable</DebugType>
<ci>true</ci>

Now my .nupkg includes .pdb files too. Still doesn't help me step in debug though.
5. installed the sourcelink cli tools from https://www.nuget.org/packages/sourcelink/ and ran sourcelink print-urls on the .pdb from my .nupkg. Looks correct, I think? URLs are present.
6. Disabled indexing after seeing a comment https://github.com/MicrosoftDocs/vsts-docs/issues/1336#issuecomment-414415049 from @mitchdenny . Still doesn't work.
And now I'm stumped as to why it's not working. Am I missing something? The help page here makes it seem as though all you need to do is Enable a VSTS task, but I have spent hours trying to get this working to no avail.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 728b7c25-2ef7-3807-b898-b9024ba297e2
Version Independent ID: 4b2eda64-1f32-fdf8-ee09-420dbb8b9f4d
Content: Setting up GitHub Source Linking with Azure DevOps Services Build & Azure DevOps Services Symbol Server
Content Source: docs/artifacts/symbols/setting-up-github-sourcelinking.md
Product: devops
GitHub Login: @elbatk
Microsoft Alias: elbatk

Hi @mattfrear ! I'm glad you're excited about this! These types of questions (re: "how do I do xxxx") are better suited for StackOverflow. However there could be a docs consequence that comes out of that discussion and if that's the case I'd love to hear it so I can make helpful changes.
Closing this issue here for now, but to try to help, I'm adding @mitchdenny to take a look for you if he can help you quickly here. I still suggest you post on SO as well.

OK, I've asked over on stackoverflow too.
https://stackoverflow.com/questions/52889238/debug-nuget-package-with-azure-devops-and-source-link


Editing of multipolygons does not work

When clicking the edit button I can not select a polygon to edit.
When clicking the delete button I can not select a polygon to delete.
This is the case in my local app and also at http://examples.flask-admin.org/geo/admin/multipolygon/edit/?url=%2Fgeo%2Fadmin%2Fmultipolygon%2F&id=3

I had this problem in latest chromium 64.0.3282.167 and latest firefox on Linux

This reproduces for me today. I'm running the latest code checked out from master, 46d5a6ee5dd and ran the example following examples/geo_alchemy/README.rst .
When clicking the edit button on the map the rendering of the polygons changed to dashed-line outlines, but clicking them doesn't show the edit mode, unlike the non-multi polygon.
When I click the delete icon on the map the rendering of the polygons change. When I click on one, both disappear.


benchmark for chatglm2-6b failed

I build chatglm2-6b engine with following command:
python3 build.py --model_dir=./pyTorchModel \
                 --dtype float16 \
                 --use_gpt_attention_plugin float16 \
                 --use_gemm_plugin float16

Then I tried to run benchmark:
python benchmark.py \
    -m chatglm2-6b \
    --mode plugin \
    --batch_size 4 \
    --engine_dir /code/tensorrt_llm/examples/chatglm2-6b/trtModel\
    --input_output_len 200,200

It returns this error:
[10/26/2023-11:28:17] [TRT] [E] 3: [executionContext.cpp::setInputShape::2252] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setInputShape::2252, condition: engineDims.nbDims == dims.nbDims )
Traceback (most recent call last):
  File "/code/tensorrt_llm/benchmarks/python/benchmark.py", line 325, in <module>
    main(args)
  File "/code/tensorrt_llm/benchmarks/python/benchmark.py", line 305, in main
    raise e
  File "/code/tensorrt_llm/benchmarks/python/benchmark.py", line 284, in main
    benchmarker.run(inputs, config)
  File "/code/tensorrt_llm/benchmarks/python/gpt_benchmark.py", line 503, in run
    self.decoder.decode(inputs[0], inputs[1], self.sampling_config)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 514, in wrapper
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 1887, in decode
    return self.decode_regular(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 1659, in decode_regular
    should_stop, next_step_buffer, tasks, context_lengths, host_context_lengths, attention_mask, logits = self.handle_per_step(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 1430, in handle_per_step
    self.runtime._set_shape(context, ctx_shape)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 202, in _set_shape
    raise ValueError(
ValueError: Couldn't assign position_ids with shape torch.Size([4, 2, 200]), engine supports [min, opt, max] = [(1, 1), (2, 1), (4, 4096)]

Is this because the engine I built does not support the bs dimension? How should I solve this problem?
Any helpful suggestions would be greatly appreciated！

@shaunxiong it is mainly about that the shape info used in build phase is inconsistent with the Tensor shape set during the runtime phase. Have you tried with the run.py provided in ChatGLM2-6b directory to see whether it can work or not?

@shaunxiong also, pls refer this and try with the latest main branch to see whether the issue still exist.

Close this bug because the issue is inactivated. Feel free to ask here if you still have question/issue, we will reopen the issue.