Stack overflow

The "int16_t audio_buffer[16000]=WAVE_DATA;" should not be defined in the main function because it will result in stack overflow when running with some other boards such K64F.
This variable should defined as global variables and store in heap.

Hi @Hust-ShuaiWang,
It worked fine on K64F board using mbed-cli and GCC compiler. What compiler/tool-chain did you use?
I agree, some other boards may face this issue if the stack space is small. In such cases, it is recommended to store in heap by allocating the memory in main function.

Thank you for your reply and I want to ask you some questions which puzzle me.
Could you tell me the  meaning and effect of parameters of bias_shift and out_shift in arm_fully_connected_q7(...,  const uint16_t bias_shift,  const uint16_t out_shift, ........)?(No ARM_MATH_DSP)
Thank you very much.
The format of bias ( ....const q7_t *bias..... )is q7_t and you convert it to q31_t ( int ip_out = ((q31_t)(bias[i]) << bias_shift) + NN_ROUND(out_shift);).Inorder to prevent errors, we have to move (q31_t)bias[i] 24 bits left,  but you move it by parameters as bias_shift and I can not understand the meaning of this parameter.
Maybe I have did some wrong understanding of the parameter and this parameter has some other functions.
Could explain me the role of these two parameters?
Thank you very much and I wish to get your apply.😊

These two parameters are used to adjust (shift) the weights and data ranges so that they can be represented in 8-bit fixed-point representation without much loss in accuracy.
Please see this discussion.
The basic idea is as follows:
Consider this equation, which is common for fully-connected layers or convolution layers:
Layer_output = [sum(layer_input x weights) + (bias<<bias_left_shift)] >> out_right_shift.
The key is to make sure that values that we are summing up have same fixed-point representation (i.e. same number of bits for fractional part).
For example, the arm_fully_connected_q7 function call here, from the quantization sweeps we get the input is of Q5.2 in fixed-point format or 2 bits after decimal point (i.e. ranges from -32,31.75) and the expected output format for maximum accuracy is Q5.2 (i.e. ranges from -32,32). Using quant_test.py, we get the quantized weights are of the format Q0.7 (i.e. in range -1,1) and the quantized biases have 8 bits for fractional point (i.e. range -0.5,0.5). So,  the product (layer_input x weights) will have 9 bits in the fractional part (Q5.2 x Q0.7 = Qx.9) and biases need to be shifted left by 1 to get to the same representation (i.e. Qx.8<<1 = Qy.9). The expected layer output has 2 bits for fractional part (Q5.2), so the product (layer_input x weights) needs to be shifted right by 7 to get to Q5.2 format.
Hope it helps (a bit 😊).

Thank you very much for your reply.
The input format of first layer is Q5.2 (which is gotten from mfcc),the weights of first layer is Q0.7 ,so the product of (input * weights) is Qx.9.The bias is Q0.8,the expected format of output is Q5.2.
So the bias_shift is 1,out_shift is 7 and the format of first layer's output is Q5.2 which is the input of second layer(ReLU has no effect on data format ).The main purpose of the shift operation is to make 2 parameters in same format before operation( + - * /).
The second layer has the same format of input data as first layer, but the bias_shift and out_shift is 2 and 8.In my understanding, weights and bias in different layers has different ranges and the second layer's weights and bias has different formats as first layer. So the bias_shift and out_shift is different in different layers.
If I have done right understanding in that and your weights and bias in different layers are surely different, would you like to mark formats of weights and formats in dnn_weights.h ? I think that will help people make understanding more easily.
Maybe I have did some wrong understanding in the 2 parameters again and please forgive my slowness haha😄~  Would you like to tell me why those 2 parameters are different in layer 1 and 2?
Thank  you very much~

Hi @Hust-ShuaiWang,
You got it all right. The shift parameters may be different in different layers (even if they have same input output ranges) because of the different ranges of bias/weights. (Too many differents in the same sentence 😄). The output of quant_test.py shows the ranges of bias/weights.

Closing the issue due to inactivity, please reopen it if you still face the issue.

@Hust-ShuaiWang
In the quantization process, how to get the value of the quantization ranges -- act_max in quant_test.py .
how to set the specific value of the parameter act_max ( x 0 0 0 0 0 0 0 0 0 0 0 ) ?

@Hust-ShuaiWang
Thanks., you explained your understanding very nicely,
Could you please tell me how did you find out the input layer format? From where we can get to know the formats of weights, bias?
Can you tell me am I right or wrong about this formats?
**fc1_W_0 number of wts/bias: (3920, 128) dec bits: 6 max: (1.875,1.87343) min: (-1.53125,-1.53759)
Q1.3 is format of weights of first layer (I have only considered max from (1.875,1.87343)), how to make use of min (-1.53125,-1.53759)**
**fc1_b_0 number of wts/bias: (128,) dec bits: 6 max: (0.71875,0.723896) min: (-1.09375,-1.09377)
Q1.6 is the bias format (only considered the max(0.723896))**


Button not working for android

ive tried the repo but the button only works for ios and web
is there any fix?

@vip0708 @BielPlugner I'm not clear understand the code can you explain it with demo or YouTube video please

@vip0708 @BielPlugner I'm not clear understand the code can you explain it with demo or YouTube video please

the buttons no longer work, WhatsApp has discontinued this feature regardless of your language or library, you won't be able to use it, only companies using the official API


Add Homesteads data

Add:

[ ] http://services.ga.gov.au/site_7/rest/services/NM_Labelling_and_Boundaries/MapServer/23 (it looks like this one is working)

[ ] http://services.ga.gov.au/site_7/rest/services/NM_Labelling_and_Boundaries/MapServer/24

Both datasets work, but 24 only has imagery at high zoom levels. Waiting on which group to put these datasets in

I think we should add 23 instead of 24.


Link to github repository

I feel like there should be a link to the github repository?
https://github.com/ipfs/dnslink.dev/blob/master/index.html#L410

Partially addressed with eb4b1c92a23e57c7bdf71d3c1da004178b0eb392


Improve doc and readme

Made some changes for a, what I think is, better README/documentation.
This focuses the readme to be centered around:

What is kubehound
How do I use it

and not on technical choices of the backend / architecture.

/merge

/remove


ci: adding detekt workflow

This workflow is not required, as detekt is already run by gradle builds during PR


Training Workflow

The workflow for the training pages should be:
If no training, bring to user training page
If training is completed, bypass training page and pre-populate info from previous training
If new training has been added, prompt user to verify new training dates

what does "bypass training page with pre-populated previous training" mean?

How would we determine that new training has been added?


Rename Vector4.components -> coord for consistency

This brings it in line with all the other vectors:

Vector2.coord
Vector2i.coord
Vector3.coord
Vector3i.coord
Vector4i.coord

I'm guessing this naming happened when Vector4 was copied from Color, as it's essentially the same struct layout, just with different functions.
This is obviously a potentially breaking change. I didn't find any other references in core to 'components', though at least godot-cpp has named their Vector4.components property after this one (see https://github.com/godotengine/godot-cpp/pull/1608), so godot-cpp users may be affected.

For the engine itself this isn't compatibility breaking as we don't provide compatibility guarantees for the engine internals (only the exposed API)

From the perspective of godot-cpp, if we want to maintain source compatibility, we could keep both declarations in the union and add a note about the older one being deprecated. Vector4 is new enough, though, that I think an argument could be made that we don't need to do that.
In any case, I think it's fine to move forward with this on the Godot-side, and we can have the "maintain source compatibility for Vector4.components" discussion in the follow-up godot-cpp PR

I'd be ok with switching to coordinates for all the types too, I'm not a big fan of shorthands. But I'll let other reviewers decide.

I think this change is fine as is, I wouldn't change all Vector types further. If we're to change I'd go one step further and make this a private member so we don't even need to bother about its name or compatibility, but that'd be a Godot 5 change :)

Thanks!


Cors Problem with Userinfo

Hello
I have an app in angular 2 that gets an access token from Identity server 4, after that i use the same token to get information about the user from the userinfo endpoint
i have no problems with the token endpoint but when i use the userinfo endpoint i get a 204 error and

XMLHttpRequest cannot load http://localhost:5000/connect/userinfo. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:52970' is therefore not allowed access.

anything i'm doing wrong or missed?
my startup http://pastebin.com/HSbwvC3v
and the calls i make from angular 2 http://pastebin.com/gbfvzT8V
if any more info needed just say

Have you configured that client's origin in the client configurations Allowed CORS Origin collection?

didn't had anything... right now it's like this http://pastebin.com/mYeiNT9g still same problem

Any update on the issue? closing for now - feel free to re-open if it needs further discussion.

Hi jjoaoclaro, I had the same problem, but solved it by adding AllowAnyMethod to .UseCors. Here is my .UseCors in Startup.Configure :
app.UseCors(builder =>
{
builder.AllowAnyOrigin()
.AllowAnyHeader()
.AllowAnyMethod();
});

Hello @leastprivilege, hello @jjoaoclaro, we seem to be having the same issue. We have a very basic JS client that goes ahead and authenticates implicitly (meaning that XHRs on .well-known/openid-configuration and .well-known/openid-configuration/jwks are going through with expected response header access-control-allow-origin with a value from Client.AllowedCorsOrigins). Things start to fail when we (after being redirected back with access+id token) try to get connect/userinfo. No/some different CORS handling seem to happen for that endpoint.
In our Startup.cs we app.UseCors where we set a few headers and methods but we tried AllowAnyHeader and AllowAnyMethod as well. As soon as we manually/redundantly add WithOrigins things work as expected. We try to stay clear of @RasmusRummel 's suggestion of allowing any origin.
I tested this on 1.3.1.
Kind regards, Simon

This works for us - compare with the JS quickstart in the docs.

I'm having exactly the same issue using JS quickstart - the only difference to the docs is that js app is on a different domain compared to identity server.

Thank you @leastprivilege for the swift reply. I had a closer look at the JS quickstart and it looks like our own UseCors seemed to interfere with the CORS-handling setup by IdentityServer.
Maybe that helps others facing the same situation.
I am still not certain why it only fails for connect/userinfo but it is probably related to POST or the authorization header even though I set those in my CORS settings.
Is there any guidance (for or against) having another logical application (controllers configured differently from IdentityServer) hosted in combination with IdentityServer? This is where our initial CORS settings came from.

We use the built-in CORS plumbing yes: https://identityserver4.readthedocs.io/en/release/reference/options.html#cors

Hi folks, just run into a similar situation. All the suggestions so far havent worked for me.

tldr - make sure you lowercase your urls.
I found this thread the other day, where I had the same issue, could not access userinfo.  Fun part was, it was only when I went to production...everything worked on my desktop, everything failed on the server.
Ultimately I disabled all cors checking in IdentityServer, and only relied on asp.net cors checking...still the same issue, worked at my desk, failed on the server.
Started reviewing asp.net cors code and found 'a something'.
I logged an issue with the asp.net cors people - https://github.com/aspnet/CORS/issues/124, they are doing StringComparer.OrdinalIgnoreCase in two out of three places, and a StringComparer.Ordinal in the last one.
In my appsettings.json for development I had:
"ServerOptions": {
    "AllowedCorsOrigins": "http://localhost"
  }

In production I had:
"ServerOptions": {
    "AllowedCorsOrigins": "http://MyCoolProductionServer"
  }

Once I lowercased MyCoolProductionServer everything worked as expected.

scopelt solution worked for me.
I've commented out the services.AddCors and app.UseCors and /userinfo started to return Access-Control-Allow-Origin

@brockallen and @leastprivilege. First, thanks for IdentityServer! Great stuff.
I am having the same issue reported above. The exact error is:
Failed to load http://identityserver.dsldhomes.com/connect/userinfo: Request header field Authorization is not allowed by Access-Control-Allow-Headers in preflight response.
Background.
NOTE: This works both locally while testing as well as on any machine within the same domain as the server where the applications 1. (web services with client app) and 2. (dentity server) are hosted.
To reiterate, it works on any machine within the same domain where the Production Server hosts the applications. While on any machine in that domain, I use same urls to go to my application as if I were navigating to it remotely. Remotely however is where my problem exists.
Project and Deployment Details:
1. Angular 6 Client hosted in www of web services application
2. http://www.mywebapp.mydomain.com -> Asp.Net Core 2 with full 4.7 framework for web services application- (Angular Client 6 client deploys to www here)
3. http://www.identityserver.mydomain.com -> Asp.Net Core 2 for Identity Server  Application
I am using SQL Server and EntityFramework quickstart.
My ClientCorsOrigins table contains the following entry for my client
http://www.mywebapp.mydomain.com
Here are the steps that cause the error.

navigate to http://www.mywebapp.mydomain.com/
get redirected to http://identityserver.mydomain.com/account/login?returnUrl=%......
enter my credentials and click login
get redirected to http://optionsandcolors.dsldhomes.com/auth.html#id_token=ey...&token_type=Bearer&expires_in=3600&scope=openid profile role SCOServicesAPI&state=bedd9e51cab4413fb0c34f7b1cea814c&session_state=FZbc7T5JLhFdo6-ChpTmiHMP7fOynrSl1K2q_XpYHNM.c5bc7e8249a32fac785efdcc6c0e0923
Then get simply Network Error displayed in Chrome. When I open dev tools I get the error: Failed to load http://identityserver.dsldhomes.com/connect/userinfo: Request header field Authorization is not allowed by Access-Control-Allow-Headers in preflight response

_ I would appreciate anyone's assistance on this. I will continue my own investigating and post the answer here if I find it first._
Here is the content of my Configure and ConfigureServices if that helps
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc();
var migrationsAssembly = typeof(Startup).GetTypeInfo().Assembly.GetName().Name;
services.AddIdentityServer()
.AddSigningCredential(new X509Certificate2(Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location),
"Resources\JABIdentityServer4Auth.pfx"), "mypassword"))
// Configures EF implementation of IClientStore, IResourceStore, and ICorsPolicyService with IdentityServer.
.AddConfigurationStore(options =>
{
options.ConfigureDbContext = builder => builder.UseSqlServer(Config.ConnectionString,
sql => sql.MigrationsAssembly(migrationsAssembly));
})
// Configures EF implementation of IPersistedGrantStore with IdentityServer.
.AddOperationalStore(options =>
{
options.ConfigureDbContext = builder => builder.UseSqlServer(Config.ConnectionString,
sql => sql.MigrationsAssembly(migrationsAssembly));
                options.EnableTokenCleanup = true;
                options.TokenCleanupInterval = 3600; // 1 Hour, also default value.
            })
            // Adds the resource owner validator, using Identity Server extension methods
            .AddResourceOwnerValidator<ResourceOwnerPasswordValidator>()
            // Adds the profile service, using Identity Server extension methods
            .AddProfileService<ProfileService>();

        services.AddAuthentication("SOME_COOKIE")
            .AddCookie("DSLD_COOKIE", options =>
            {
                options.ExpireTimeSpan = TimeSpan.FromHours(12);
            })
            .AddOpenIdConnect("oidc", "OpenID Connect", options =>
            {
                options.SignInScheme = IdentityServerConstants.ExternalCookieAuthenticationScheme;
                options.SignOutScheme = IdentityServerConstants.SignoutScheme;

                options.Authority = "http://identityserver.mydomain.com";

                options.RequireHttpsMetadata = false;

                options.ClientId = "some_client";

                options.ResponseType = "code id_token"; // determines which grant is being used (hybrid here)
                options.SaveTokens = true; // Save Access Token in cookie

                // TODO: Don't think this is needed but verify.
                options.Scope.Add("roles");
                options.GetClaimsFromUserInfoEndpoint = true;
                options.TokenValidationParameters = new TokenValidationParameters
                {
                    NameClaimType = JwtClaimTypes.Name,
                    RoleClaimType = JwtClaimTypes.Role
                };
            });
        services.AddDbContext<UserDbContext>(builder =>
            builder.UseSqlServer(Config.ConnectionString, options => options.MigrationsAssembly(migrationsAssembly)));
        services.AddDbContext<UserRoleDbContext>(builder =>
            builder.UseSqlServer(Config.ConnectionString, options => options.MigrationsAssembly(migrationsAssembly)));

        // Adds a scoped service of the type specified in serviceType with an implementation
        // of the type specified in implementationType to the specified Microsoft.Extensions.DependencyInjection.IServiceCollection.
        services.AddScoped<UserRepository>();
        services.AddScoped<UserRoleRepository>();
        services.AddScoped<UserStore>();
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseIdentityServer();
        app.UseStaticFiles();
        app.UseMvcWithDefaultRoute();
    }

I had the same issue and the fix was adding the url to the client's "Allowed CORS Origins".

@EYScudeler, What did you add to the client's origins?
In my client (i.e.  http://www.mywebapp.mydomain.com from explanation above) in
ConfigureServices I create this policy to allow identityserver to make requests against my client:
options.AddPolicy("default", policy =>
{
policy.WithOrigins("http://identityserver.mydomain.com")
.AllowAnyHeader()
.AllowAnyMethod();
});
In Configure I add:
app.UseCors("default");
It does not seem that the error I am getting would require a cors policy by the client but rather a cors policy by the identity server. From the error posted above it looks like the Client is trying to call into IdentityServer, so it would be IdentityServer that needs to allow the origin from the client, which it does.

To be honest, I think many of us would agree, answers from you guys are just confusing and not helpful at all.

@billyjacobs2014 I think it was not a firewall. I have the same issue, but only in Firefox. in Chrome it works.

this helped for me:
[IdentityServer4] Startup.cs
public void ConfigureServices(IServiceCollection services) { //... services.AddCors(options =>             {                 options.AddPolicy("api", policy =>                 {                     policy.AllowAnyOrigin().AllowAnyHeader().AllowAnyMethod();                 });             }); } public void Configure(IApplicationBuilder app) { //... app.UseCors("api"); }

.AllowAnyOrigin() on the Cors Policy looks like a work around.

IS4 registers a custom ICorsPolicyProvider. When you have multiple additional CORS policies in your system, make sure to add the middlewares for them after IS4 added its middleware.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory) { 	... 	app.UseIdentityServer(); 	app.UseCors("mycustomcorspolicy"); 	... }
That is important since the custom ICorsPolicyProvider by IS4 decorates the DefaultCorsPolicyProvider by ASP.NET. If the request is not for one of IS4's paths, it lets the decorated policy provider handle that request.
That means if the first middleware is not IS4's, then IS4's ICorsPolicyProvider asks the decorated provider, ASP.NET's ICorsPolicyProvider, to determine the fate of the request for IS4's paths. This leads to rejections like that of /connect/userinfo.

@halllo thanks this worked for me. I changed order it worked great.

@halllo Thank you. It helped. This is Something you wouldn't notice immediately.


Seeing "NaN mins ago" while card is just starting up / after HA restart

and have you...

[X] I updated to the latest version available
[X] I cleared the cache of my browser

Version of RPi Monitor Card
v1.3.4 - likely introduced with code update of card age calc
Is this a regression?
No or unsure. This never worked, or I haven't tried before.
Browser and Operating system
Browser: N/A
OS: N/A
Description of problem
The card is showing the unknown values are being retrieved while HA is just setting up.
The Card should be suppressing this output until HA is completely set up.
NOTE: This is also true for the values found in the Daemon update field in the card... again, suppress this too until the card backing state is known.
WORKAROUND: refresh the card after HA completes its setup.
Errors in Browser Console
No response
Additional information
No response

Changes committed and appear in v1.4.0 and later


"Visible todos" should not take into account properly hidden (h:1) items

Hi,
I'm wondering between bug and feature request, could be both but as it's confusing users I'd side more with bug I think... feel free to contradict me :)
Describe the bug
If users have created items with the h:1 tag to hide them AND intentionally selected in the options that "Hidden todos" is unchecked (=really hidden), then they should not be counted in the "Visible todos" counter or the user believes a search/filter is still active.
Example : if we have 3 items and 1 hidden item

We should not see "Visible todos 3 of 4" when "Hidden todos" is unchecked (=really hidden)

Expected behavior
With same example : if we have 3 items and 1 hidden item

We should see either nothing (if all 3 non-hidden items are visible), or "Visible todos X of 3" (but not 4) if a search/filter is activated

Cheers,
O.

@Oroumov can be tested here: https://github.com/ransome1/sleek/releases/tag/v2.0.0-dev9


chore: initial implementation copied from @japa/assert

🔗 Linked issue
n/a
❓ Type of change

[ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[x] ✨ New feature (a non-breaking change that adds functionality)
[x] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
This adds an initial version of the code required for this japa plugin.  It is exported as a openapi module, and exposed on the test context via openapi.
There's a few minor changes:

Exposed as openapi on test context instead of on assert
Renamed assert.isValidApiResponse to openapi.isValidResponse

📝 Checklist

[ ] I have linked an issue or discussion.
[x] I have updated the documentation accordingly.

Documentation PR: https://github.com/japa/japa.dev/pull/26

Exposed as openapi on test context instead of on assert?

Can we have it on assert. A couple of reasons.

Easier to migrate to this package, as I will only have to register a new package vs replacing all usages.
assert reads a little better in terms of running assertions

WDYT?

Exposed as openapi on test context instead of on assert?

Can we have it on assert. A couple of reasons.

Easier to migrate to this package, as I will only have to register a new package vs replacing all usages.
assert reads a little better in terms of running assertions

WDYT?

I wasn't sure how to get it on to assert, hence going this route

Created a PR for that. https://github.com/ThisIsMissEm/japa-openapi-assertions/pull/1

@thetutlage okay, I think this is good to go now.

Thanks 👍


Use https for Avalon and Inchi downloads

Those URLs are available as https so no reason to stick to legacy http URLs.

nice catch, thanks


isMobilePhone: additional prefix for KE locale

Ref: https://ca.go.ke/new-series-of-mobile-number-introduced-in-kenya/

@profnandaa, I am working on this.

@me-x-mi -- great, go ahead sir 🌮

Closed by #1035


Support React.Component

Codecov Report

Merging #12 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #12      +/-   ##
==========================================
+ Coverage   99.46%   99.47%   +<.01%
==========================================
  Files           1        1
  Lines         188      191       +3
==========================================
+ Hits          187      190       +3
  Misses          1        1

Impacted Files
Coverage Δ

index.js
99.47% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aabfba6...e33fb56. Read the comment docs.


Update APIs to use OAuth user details instead of cookies

Is your feature request related to a problem? Please describe.
We need to use the OAuth user details to get the profile id
Describe the solution you'd like
Update the endpoint with @AuthenticationPrincipal annotation
Add unit tests
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
N/A
Additional context
Add any other context or screenshots about the feature request here.
N/A

Solved with #109


[TEST] test_csi_umount_when_longhorn_block_device_is_disconnected_unexpectedly failed on v1.4.x pipeline

What's the test to develop?  Or the test case is actually failed? @khushboo-rancher

@shuo-wu Please help check this.

This is a test teardown failure, but I'm curious to know if something is preventing the pod to get deleted?
@nitendra-suse Did the pod stuck in terminating state or it got deleted eventually?

This is a test teardown failure, but I'm curious to know if something is preventing the pod to get deleted?
@nitendra-suse Did the pod stuck in terminating state or it got deleted eventually?

Pod stuck in terminating state
block-device-disconnect-unexpectedly-test-0   0/1     Terminating   0          69m

Seems this test case has never passed in v1.4.x since it's been implemented.
Is it possible to use longhorn-tests master image to run this test case in v1.4.x? If it passed, it probably means some code/commits are missing in longhorn-tests v1.4.x.

This test case with v1.4.x images will fail when kubernetes version is v1.25.3+k3s1(v1.4.x pipeline currently using), pass when kubernetes version is v1.27.1+k3s1(which is currently v1.5.x and mater pipeline using)

I think there is an issue here.
In the step 3 and step 4 of this test case, it should wait for pod actually terminated before go to the next step:
    3. Delete the workload pod
    4. Verify that the pod is able to terminated and a new pod is able
        start

But in the current implementation, it doesn't wait for the pod termination:
    crash_engine_process_with_sigkill(client, core_api,
                                      sspod_info['pv_name'])
    delete_and_wait_pod(core_api,
                        pod_name=sspod_info['pod_name'],
                        wait=False)

If we remove the wait=False option in the delete_and_wait_pod to let it wait for the pod termination, we can see this test case failed at this step instead of the tear down function:
=================================== FAILURES ===================================
___ test_csi_umount_when_longhorn_block_device_is_disconnected_unexpectedly ____

client = <longhorn.Client object at 0x7f4f88be3640>
core_api = <kubernetes.client.api.core_v1_api.CoreV1Api object at 0x7f4f88a03a30>
statefulset = {'apiVersion': 'apps/v1', 'kind': 'StatefulSet', 'metadata': {'name': 'block-device-disconnect-unexpectedly-test', 'na...usybox:1.34.0', 'imagePullPolicy': 'IfNotPresent', 'name': 'sleep', ...}], 'terminationGracePeriodSeconds': 10}}, ...}}
storage_class = {'allowVolumeExpansion': True, 'apiVersion': 'storage.k8s.io/v1', 'kind': 'StorageClass', 'metadata': {'name': 'longhorn-test'}, ...}

    @pytest.mark.csi  # NOQA
    def test_csi_umount_when_longhorn_block_device_is_disconnected_unexpectedly(client, core_api, statefulset, storage_class):  # NOQA
        """
        Test CSI umount when Longhorn block device is disconnected unexpectedly

        GitHub ticket: https://github.com/longhorn/longhorn/issues/3778

        1. Deloy a statefulset that has volumeClaimTemplates with
            volumeMode: Block
        2. Crash the engine process of the volume to simulate Longhorn block
            device is disconnected unexpectedly
        3. Delete the workload pod
        4.
Unless this test case is not applicable in k8s version <= 1.25, otherwise there is an issue here.
cc @longhorn/qa @innobead

Could related to https://github.com/longhorn/longhorn/issues/6919

My gut reaction is that this is not #6919, because that one requires a kubelet restart to hit.

My gut reaction is that this is not #6919, because that one requires a kubelet restart to hit.

Thanks for the clarification! So it's back to https://github.com/longhorn/longhorn/issues/6533#issuecomment-1777176535. Still need investigation to know what happened.

@yangchiu @chriscchien do we have a support bundle?

Saw the error messages in the kubelet log (v1.25.3+k3s1)
Oct 26 03:14:58 ubuntu2004 k3s[3704]: E1026 03:14:58.522314    3704 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/driver.longhorn.io^pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445 podName:f3d02095-1d37-4704-842f-a6bc63d9efb4 nodeName:}" failed. No retries permitted until 2023-10-26 03:17:00.522301451 +0000 UTC m=+2359.758012633 (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pod-data" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445") pod "f3d02095-1d37-4704-842f-a6bc63d9efb4" (UID: "f3d02095-1d37-4704-842f-a6bc63d9efb4") : blkUtil.DetachFileDevice failed. globalUnmapPath:/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445/dev, podUID: f3d02095-1d37-4704-842f-a6bc63d9efb4, bindMount: true: failed to unmount linkPath /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445/dev/f3d02095-1d37-4704-842f-a6bc63d9efb4: unmount failed: exit status 1
Oct 26 03:14:58 ubuntu2004 k3s[3704]: Unmounting arguments: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445/dev/f3d02095-1d37-4704-842f-a6bc63d9efb4
Oct 26 03:14:58 ubuntu2004 k3s[3704]: Output: umount: can't unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445/dev/f3d02095-1d37-4704-842f-a6bc63d9efb4: Resource busy

This is an upstream issue, https://github.com/longhorn/longhorn/issues/3778#issuecomment-1572999855.
@chriscchien you verified it before :), the fix was just fixed into K8s 1.27, so the failure here is expected.
ref: https://github.com/longhorn/longhorn-tests/pull/1475

@yangchiu probably we can also update the K8s version for 1.4?

Workaound

For the pod stuck in Terminating  state
kubectl delete pod <name>> --force

Detach the problematic block device

Find the problematic block device# losetup -a
/dev/loop1: [1048679]:7340510 (/var/nfs-data/data.img)
/dev/loop2: [0006]:426 (/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-3d454e11-5fce-4b0a-86fb-4d2698b84445/dev/f3d02095-1d37-4704-842f-a6bc63d9efb4 (deleted))
/dev/loop0: [64515]:6816043 (/home/vagrant/blockfile)

Detach the devicelosetup --detach /dev/loop2

@yangchiu probably we can also update the K8s version for 1.4?

https://github.com/longhorn/infra/pull/146


How do I fit a ":then" in the use-package example?

This is what I have:
(use-package ryo-modal
  :commands ryo-modal-mode
  ;;; :bind ("<C-SPC>" . ryo-modal-mode)
  :init
  :config

(ryo-modal-keys

   ("1" delete-other-windows)
   ("2" split-window-below)
   ("3" split-window-right)
   ("4" switch-window)
   ("t" transpose-frame)
   (";" comment-line)
   ("," ryo-modal-repeat)
   ("q" save-buffers-kill-terminal)
   ("a" mark-whole-buffer)
   ("s" my-save-buff)
   ("k" kill-buffer)
   ("j c" cfg_cfgorg)
   ("j k" cfg_cfgkb)
   ("j r" cfg_ryo)
   ("K" python-nav-backward-defun)
   ("y" end-of-line)
   ("c" easy-kill)
   ("v" consult-yank-pop)
   ("x" kill-region)
   ("g" my-escape)
   ("u" beginning-of-line)
   ("f" consult-line-multi)
   ("S" consult-line)
   ("R" anzu-query-replace)
   ("m" bm-menu)
   ("+" text-scale-adjust 2)
   ("-" text-scale-adjust 2)))

But cannot work it out.  I can work it out in other cases:
(ryo-modal-key
 "SPC" '(("s" save-buffer)
         ;; ("g" magit-status)
         ("b" myswitchbuff)
         ("k" describe-key)
         ("m" menu-switch)
         ("j c" cfg_cfgorg)
         ("j k" cfg_cfgkb)
         ("v" consult-yank-pop)
         ("x" kill-region)
	 ("z" undo-fu-only-undo)
         ("p" package-list-packages)
         ("t" straight-pull-all)
         ("SPC" ryo-toggle))
         :then '(ryo-off) :exit t)

What I am trying to do is change the background colour when ryo is active, ryo-off resets it back.  I could write little wrapper defuns, but that would be very long winded.  Thanks!

Got it!  would have been quicker, but I went down a rabbit hole caused by a corrupt history file
(use-package ryo-modal
  :commands ryo-modal-mode
  ;;; :bind ("<C-SPC>" . ryo-modal-mode)
  :init
  :config

(ryo-modal-keys
   ("1" delete-other-windows :then '(ryo-off))
   ("2" split-window-below :then '(ryo-off))
   ("3" split-window-right :then '(ryo-off))
   ("4" switch-window)
<snip>

I am sure I tried this, but didn't quote it - doh!.

Happy to hear you figured it out! :)


Import Solution EnvironmentVariablesNotSupported

I asked 2 days ago here https://github.com/microsoft/powerplatform-build-tools/discussions/898 but got no answer. I dont find a solution so i think it's an issue.
We get the Error (sry German):
CanvasApp import: FAILURE: Code: EnvironmentVariablesNotSupported Message: Die Umgebung unterstützt keine Umgebungsvariablen. Diese Umgebung hat die Version „<IP_ADDRESS>“, es ist jedoch Version „<IP_ADDRESS>“ erforderlich. Client Request Id: 505bda21-02b3-40b2-adab-90a4c68fde64.
it talks about the enviroment version. but i don't know were i can see this.
OUR solution has the version <IP_ADDRESS> (packed feb 2024 old) and <IP_ADDRESS> (packed Monday new)
in the GUI we see the Error: 80072352
tested with two different tenants. 1 worked on feb 2024. same error
tested with old and new solution. same error

Hey @carenggli
Thanks for the message. Good to know.

We also created a ticket. They talked about fixing for yesterday. Today same error.
Mail from last week:

We asked today again.

We got an answer, it can take up 1 or 2 weeks from now.

Today it worked. we got an E-Mail. Microsoft fixed it. Thx pls close.


Allow plugins to create new windows

I created (ing?) a new hyperterm-visor plugin that allows you to activate the HyperTerm terminal globally (and other features feature (but I plan on fleshing it out a bit)), but if you close all the windows, it doesn't look like you can create a new window from a plugin.
Perhaps I'm missing something, but if not, could we refactor app to in someway allow for creating new windows?

This is great! 👍  I use visor mode in iTerm alot so would definitely push me over edge in terms of adopting hyperterm

@CWSpear Looks like this was merged with #248?

Yeah, I have been waiting for a new release (which apparently has happened!).
I'm gonna tweak a little bit of how some things work (i.e. gonna change how the dedicated visor window works) and hopefully get a new release myself up soonish.

Alright, cool. Could you close the issue then @CWSpear? 😄

I think we should document that you can do like this in plugins aswell, maybe another issue.
https://github.com/zeit/hyperterm/issues/398#issuecomment-235131717


Usage purely offline

Quick question -
can this be used purely offline without google cloud integration, as a hotword trigger?
thanks.

I'm assuming you found Snowboy 😃


Where's the buy-now link???

Nice stuff! 💥

Thanks, you can export the gerber files from the KiCAD project files. If you don't have KiCAD, I can probably export the files and add them to each folder.


Fix DeferView logic not properly exiting

Somehow I broke it after making changes yesterday.

Fixed with https://github.com/macadmins/nudge/commit/50c71caae73d5177e8edc527c6ed2cd6d292bf7e


Update mochiweb to the latest from mochi/mochiweb

The primary motivation for updating mochiweb is to pull in this fix: https://github.com/mochi/mochiweb/issues/149
In local testing mochiweb appears as version 2.13.0 and passes a full pedant test (chef-server-ctl test --all)

@stevendanna per your feedback,  I traced the mochiweb dependency:

One afterthought/question:   Should I update the rebar config and lock for bookshelf and bifrost in this same PR, a different PR, or leave them alone?

wilson build:  http://wilson.ci.chef.co/job/chef-server-12-build/1509/

One afterthought/question: Should I update the rebar config and lock for bookshelf and bifrost in this same PR, a different PR, or leave them alone?

We've been trying to keep core deps like this consistent across projects, so I would definitely be +1.

This means we're on the wrong webmachine too.  We should be using https://github.com/webmachine/webmachine

hey @marcparadise @stevendanna since the scope of this PR is creeping bigger and bigger, I propose that we keep the current PR as-is in an effort to be GEFN  (fix the issue that’s affecting customers)  and then work on the opscoderl_wm changes in another PR?

New wilson build containing latest mochiweb for erchef, bifrost and bookshelf:  http://wilson.ci.chef.co/job/chef-server-12-build/1517/

Update:   Validated the new mochiweb in erchef, bifrost and bookshelf (build 1517) in ec-metal's loadtest environment -  performed as expected.
cc: @sdelano

Replaced by the changes in #592 , which pull in the new deps via updated https://github.com/chef/opscoderl_wm.


proxy: better error messages for postgres connection limits

We have customers exhausting active connection limits of postgres. Errors include

remaining connection slots are reserved for non-replication superuser connections
sorry, too many clients already
no more connections allowed (max_client_conn)

All 3 mean similar things. 1 and 2 refer to the postgres connection limit. 3 refers to the pgbouncer connection limit.
All three should be prefixed with a consistent greppable error message. 1 and 2 should additionally include a suggestion to use pooled connections.

It's not what I thought it was.
So the first error remaining connection slots are reserved for non-replication superuser connections comes from here, which is part of InitPostgres. So it deserves investigation.
The second error sorry, too many clients already comes from here, which is part of ProcessStartupPacket, which makes sense (this could be a legit client error).
The third error no more connections allowed (max_client_conn) I can't even find in the code, so idk where it's coming from.

Wait it might actually be what I thought it was, explain by the comment here. Postgres is being optimistic that some connections won't need a backend because they won't auth. If they actually auth, we get error 1 later. If there's enough connections to cross the optimistic threshold, then we get error 2 early. We can double-check by reproducing the behavior locally

To attempt to reproduce:
Test 1: Make enough connections to see error 1
Test 2: While already at the limit, make enough concurrent connection attempts to get error 2. Adding some sleeps during connection from the client-side can make it easier to trigger the behavior (if the error is what I think it is)

The third error no more connections allowed (max_client_conn) I can't even find in the code, so idk where it's coming from.

It's from PgBouncer, not Postgres

Wait it might actually be what I thought it was, explain by the comment here. Postgres is being optimistic that some connections won't need a backend because they won't auth. If they actually auth, we get error 1 later. If there's enough connections to cross the optimistic threshold, then we get error 2 early. We can double-check by reproducing the behavior locally

Ok. As far as I can tell then the action from the end user is the same in either case, right? They need to use less connections, get a larger DB, or use pooling

Not actionable by proxy team, closing.


notebook: block content menu usability

fixes LAND-598 by:

updating the menu style to match the rest of our app
adding a "+" button to access it
updating the placeholder text
ensuring that the placeholder text appears on each new node

Also:

updates tiptap to the latest stable version (2.0 is out of beta). Required pinning some specific prosemirror deps to get typescript to be happy.
fixes an issue where keyboard up/down keys worked in reverse.

Looks like this:
https://github.com/tloncorp/landscape-apps/assets/1221094/f4d3bed8-12bc-404f-b561-2052ac06849b

We have to kill tiptap


added the new twitter logo(x)

This pull request introduces the updated Twitter logo(x) to the project. The new logo(x) enhances brand representation and aligns with the latest design guidelines provided by Twitter. The changes include updating the logo(x) asset files and integrating them into the project's codebase.
Changes:
Added the new Twitter logo(x) asset files.
Updated references to the Twitter logo(x) in relevant project files.
Ensured consistency with the project's design standards.

Reasoning:
Keeping the project up-to-date with the latest branding assets is essential for maintaining a cohesive user experience. By incorporating the new Twitter logo(x), we ensure that our application remains visually appealing and in line with current design trends.
Testing:
Verified that the new Twitter logo(x) displays correctly across various screen resolutions.
Checked for any rendering issues or distortions caused by the logo(x) update.
Tested logo(x) accessibility and visibility on different backgrounds and themes.

Hello @akshaykaluse35, The preview of your PR is working fine. But, icon changed by you seems to be blank to me. PTAL!

Also, Please do changes in your forked repository instead of organization repository

@shubham-kshetre I've tested it and it works fine for my end

@shubham-kshetre yeah sure ;) next time i'll make changes to forked repo

@akshaykaluse35 can you PTAL for why icon is not visible in the commit?

@SAYUK09 PTAL


Is there a way to use n-locks?

I wonder if I can use multiple locks? The main reason - to be able to limit how many same job workers can go in parallel. So say I want to limit 1 job to be run concurrently with only 2 worker threads - can I somehow set 2 locks and then acquire first or second?

Hey @sandric!
I'm not sure I understand exact use case, but it seems this gem might not be a right solution for your problem - if you want to limit number of works per job (and by job I assume you mean queue here?) you might simply spin-up multiple sidekiq processes and configure each process for each queue (consume only specific queue(s) per process and set number of threads for your needs) or maybe use something like ?

Hey @sandric!
I'm not sure I understand exact use case, but it seems this gem might not be a right solution for your problem - if you want to limit number of works per job (and by job I assume you mean queue here?) you might simply spin-up multiple sidekiq processes and configure each process for each queue (consume only specific queue(s) per process and set number of threads for your needs) or maybe use something like ?

@emq Yeah, I tried sidekiw-limit_fetch, but for some reason my redis k8s pod just dies right after I start sidekiq worker :slightly_frowning_face: Yes, I could spawn other process just to run specific queue, but the thing is doing this I will load the same sidekiq + activerecord etc two times and I would like to avoid it. So I was thinking mb solution might be to use smth like mutex but instead of true/false check for acquisition - make it integer - and on acquire - check for limit and increment, while on release - decrement..

@emq Yeah, I tried sidekiw-limit_fetch, but for some reason my redis k8s pod just dies right after I start sidekiq worker :slightly_frowning_face: Yes, I could spawn other process just to run specific queue, but the thing is doing this I will load the same sidekiq + activerecord etc two times and I would like to avoid it. So I was thinking mb solution might be to use smth like mutex but instead of true/false check for acquisition - make it integer - and on acquire - check for limit and increment, while on release - decrement..

I wouldn't worry about spinning multiple processes unless you're running your app under some heavy resources limits - but then again I would rethink using ruby in the first place.
If allocating some extra hardware resources solves your problem - I would say go for it; I'm almost sure it will be cheaper in the long-term than maintenance of some custom 'trickery' logic.
I'm gonna close this issue as it seems you have more of an architectural choice to make - and as usual it depends ;) good luck!

I wouldn't worry about spinning multiple processes unless you're running your app under some heavy resources limits - but then again I would rethink using ruby in the first place.
If allocating some extra hardware resources solves your problem - I would say go for it; I'm almost sure it will be cheaper in the long-term than maintenance of some custom 'trickery' logic.
I'm gonna close this issue as it seems you have more of an architectural choice to make - and as usual it depends ;) good luck!


Update dependencies

It seems like smol 0.3 is the latest version compatible with rust 1.40.

It seems like smol 0.3 is the latest version compatible with rust 1.40.


Add a short header style

Homepage

Everywhere else

Still not sure what to do about the image, don't really want to use a separate small image instead ( then the image is cached between pages ) but it's better than before


Getting sprite with optimized svgs

Do you want to request a feature, report a bug or ask a question?
ask
What is the current behavior?
seems like svgo loader doesn't work
What is the expected behavior?
expects to use svgo optimization on output sprite
gist
Please tell us about your environment:

Node.js version: v9.11.1
webpack version<EMAIL_ADDRESS>svg-sprite-loader version<EMAIL_ADDRESS>OS type & version: Linux 420afcde5ebb 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 15:12:16 UTC 2018 x86_64 Linux

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)
as you can see in gist, i try to use round in svgo, but in result got not rounded numbers in sprite
also, i'm always getting this warning svg-sprite-loader exception. Some loaders will be applied after svg-sprite-loader in extract mode

Could you please create repo with minimal setup to demonstrate a problem (package.json, webpack config, SVG image and piece of your code). If you don't want to create a repository - you can create a gist with multiple files.

I'll close it because I figured out the situation


selector() missing from readme

looks like it'd be part of the API, no?

Ye it is but I just have't documented it yet.


grafana-toolkit "command not found" when using "yarn dev"

To get "yarn dev" to work, I added grafana-toolkit (currently under alpha) to the package dependencies with
yarn add @grafana/toolkit
After that I was able to successfully compile the typescript code with the commands specified in package.json

@grafana/toolkit has been added to the dependencies a number of months ago, so I suppose it's safe to close this.


Support for webpack 2.x

Is webpack 2.x support planned in the near future?

Is this project still active?

@lewebsimple did you find a solution? Did this package work alright for you?


OS X, Will not start if Application installed by another user

The application will not start, console shows
15/08/2015 11:55:25.619 am mcedit[4558]: Running in fixed mode. Support files are in your App Support Folder (Available from the main menu of MCEdit)
15/08/2015 11:55:26.224 am mcedit[4558]: Splash load...
15/08/2015 11:55:26.690 am mcedit[4558]: Traceback (most recent call last):
15/08/2015 11:55:26.690 am mcedit[4558]:   File "/Applications/mcedit.app/Contents/Resources/__boot__.py", line 399, in <module>
15/08/2015 11:55:26.690 am mcedit[4558]:     _run()
15/08/2015 11:55:26.690 am mcedit[4558]:   File "/Applications/mcedit.app/Contents/Resources/__boot__.py", line 336, in _run
15/08/2015 11:55:26.690 am mcedit[4558]:     exec(compile(source, path, 'exec'), globals(), globals())
15/08/2015 11:55:26.690 am mcedit[4558]:   File "/Applications/mcedit.app/Contents/Resources/mcedit.py", line 13, in <module>
15/08/2015 11:55:26.691 am mcedit[4558]:     import splash
15/08/2015 11:55:26.691 am mcedit[4558]:   File "splash.pyc", line 49, in <module>
15/08/2015 11:55:26.691 am mcedit[4558]: IOError: [Errno 13] Permission denied: u'/Applications/mcedit.app/Contents/Resources/splash'
15/08/2015 11:55:26.755 am mcedit[4558]: mcedit Error

To reproduce

Download installation zip file
Copy Application package from the zip file to Applications Folder
Log in as a different user, not the user that installed it
Start the application

Workaround

Open command shell as an Administrator
touch /Applications/mcedit.app/Contents/Resources/splash
chmod o+w /Applications/mcedit.app/Contents/Resources/splash

Hmm, looks like the splash folder is being generated with the first user's permissions.

Where in code is the splash directory created?

The splash folder is not created at runtime. It is embeded in the application during the compilation, AFAIK. (I'm not the one who compiles on Mac, but this is done like this on Linux and Windows.)
By the way, OSX is derived from a Unix system (actually BSD), and may not parmit a user to access other users apps/data. Depending the policies, Windows can behave like this too.
IMO, this is not a MCEdit bug, but an OS side issue, due to the structure of the OS and its security policies.

IMO, this is not a MCEdit bug, but an OS side issue, due to the structure of the OS and its security policies.

Yet, here you are writing to files inside the application bundle.

So, I'm sure you have the solution.

Yep. It's Don't write files inside the application bundle.

This is obvious.
But with 'solution', I meant a code piece wich can permit a Mac user to run MCEdit without delaying the splash display, without workaround and without installing the app for another user.
I can't do this, because I don't have a Mac...

You should be able to solve this. The same problem is present on Linux if the user has no write permission to the file given by splash_name.

On Linux the user is the owner of the application directory, and other users will not be able to run MCEdit for the reason they can't... this is the base policy on these systems.
Moreover, Linux users are warned at install time that MCEdit is installed for the current user only.

Yes. That is the same problem. If you can't solve it, then you will have to also warn OS X users that MCEdit is installed for the current user only.

sed 's/directories.getDataDir()/directories.docsFolder/' splash.py

Better yet, don't even write the file at all. It serves no purpose.

If not present, the file is re-created. Look at readme (and the source) for more information.
Another idea is to change the app rights bits to permit other users to read/write/execute it, and link it in the other users Application folder.

Yes, I read the source. The file does nothing. Just don't write it.

The file contains the name of the splash to be used. This name is changed everytime MCEdit starts.

The file is written but never read.
Regardless, it was an incredibly bad decision to write a file to the application folder to begin with. That is basically the first rule of writing apps for multi-user systems: Do not write files to the application folder. You may not have permission to write files to the application folder.

The file is written but never read.

Right. Someone disabled it: https://github.com/Khroki/MCEdit-Unified/commit/02d7683d69c2b9a31b7c51d6482f26fb8dcde47c
It will be reenabled.

Would it be possible to have that file written with other support files (such as the usercache) instead of inside the .app? That would also make it easier for users to add their own splashes if they so desired.

i also ran into a similar problem, mcedit.log is saved to the read-only application folder and crashes it

As said, OSX is a Unix based system and have some policies regarding using other users stuff.
I recommend to install MCEdit for each user who want to use it on the machine.


V6.0.0 - Strike Permission Level fix

@klerikdust reason being is, looking at the perm level file, you have moderator listed as 2 and 3 is admin level, 1 is beta user and 0 is general user...

also, @klerikdust is it easier to contact you on here, discord, twitter, or Instagram

Ah my bad, I forgot admin-level does exist lol. And yep, I guess I can merge this.
@BaitGod01 you can always dm me on discord, I'll be occasionally login there.


update v3 and leaflet, fix a chartmarker bug

I heard from Connor that react-leaflet-v3 is a high priority so I worked for this. This is branched out from react-leaflet-v3-ref to test several things:
a) update v3
b) update leaflet to the latest version, v1.9.3
c) fix a chartmarker bug (please see https://github.com/VEuPathDB/web-components/pull/444)
These updates seem to work fine with storybook as well as web-eda's mapViz. However, please note that somehow npm-pack-here at web-eda did not work natively. It throws an error:
An unexpected error occurred: "expected hoisted manifest for \"@veupathdb/components#react-leaflet\"".
I tried to find a solution and the one I could make it work is to explicitly specify react-leaflet v3 at dependenciesat web-eda's package.json, like:
  "dependencies": {
    "debounce-promise": "^3.1.2",
    "fp-ts": "^2.9.3",
    "io-ts": "^2.2.13",
    "lodash": "^4.17.21",
    "monocle-ts": "^2.3.11",
    "react-cool-dimensions": "^2.0.7",
    "react-draggable": "^4.4.3",
    **"react-leaflet": "^3.2.5",**
    "react-resizable": "^3.0.4",
    "uuid": "^8.3.2"
  },

IDK if there is another solution without specifying it. @dmfalke?
Anyway, I hope anyone can test this PR to make sure if everything works well.
p.s. Actually this branch can be directly merged into main as well, I suppose.

Thanks, @moontrip. I will test this out soon.

Hi @moontrip
Without fully understanding what's going on, I followed some googled advice on the "expected hoisted manifest" issue, and deleted/renamed yarn.lock and node_modules, then did yarn and yarn add -D file:local_modules/@veupathdb/components and it built without that error.
I'm not sure what this means going forward though. A PR on the web-eda side with these major changes to yarn.lock?

@bobular wow, it sounds great 👍 I did try to remove yarn.lock but didn't try to delete node_modules.

But then it doesn't actually compile :sob:
There are lots of these very low-level looking React errors/warnings
webpack compiled with 24 warnings
ERROR in src/index.tsx:131:16
TS2786: 'DefaultComponent' cannot be used as a JSX component.
  Its element type 'ReactElement<any, any> | Component<Props, any, any> | null' is not a valid JSX element.
    Type 'Component<Props, any, any>' is not assignable to type 'Element | ElementClass | null'.
      Type 'Component<Props, any, any>' is not assignable to type 'ElementClass'.
        The types returned by 'render()' are incompatible between these types.
          Type 'React.ReactNode' is not assignable to type 'import("/home/maccallr/Desktop/EDA/web-eda/node_modules/@types/react-router/node_modules/@types/react/index").ReactNode'.
    129 |           >
    130 |             <SnackbarProvider styleProps={{}}>
  > 131 |               <DefaultComponent {...props} />
        |                ^^^^^^^^^^^^^^^^
    132 |             </SnackbarProvider>
    133 |           </UIThemeProvider>
    134 |         </DevLoginFormContext.Provider>

Probably due to mixed @types/react ?
$ yarn why @types/react | grep Found
=> Found "<EMAIL_ADDRESS>=> Found "@types/react-resizable#<EMAIL_ADDRESS>=> Found "@types/react-router-dom#<EMAIL_ADDRESS>=> Found "@types/react-table#<EMAIL_ADDRESS>=> Found "@testing-library/react-hooks#<EMAIL_ADDRESS>=> Found "@types/react-router#<EMAIL_ADDRESS>=> Found "@types/react-test-renderer#<EMAIL_ADDRESS>=> Found "@types/react-transition-group#<EMAIL_ADDRESS>=> Found "@visx/gradient#<EMAIL_ADDRESS>=> Found "@visx/group#<EMAIL_ADDRESS>...

I'm guessing there are some major React version issues here...
If I set the @types/react version to 18.0.27 (and a few others while I'm at it)
diff --git a/package.json b/package.json
index 2d10f0b1..f219d3ae 100644
--- a/package.json
+++ b/package.json
@@ -71,14 +71,14 @@
     "@types/lodash": "^4.14.182",
     "@types/luxon": "^3.0.1",
     "@types/node": "^12.0.0",
-    "@types/react": "^16.9.53",
-    "@types/react-dom": "^16.9.8",
-    "@types/react-resizable": "^1.7.2",
+    "@types/react": "^18.0.27",
+    "@types/react-dom": "^18.0.10",
+    "@types/react-resizable": "^1.7.4",
     "@types/react-router-dom": "^5.3.3",
-    "@types/react-table": "^7.7.12",
+    "@types/react-table": "^7.7.14",
     "@types/uuid": "^8.3.4",
     "@veupathdb/browserslist-config": "^1.1.0",
-    "@veupathdb/components": "^0.19.17",
+    "@veupathdb/components": "file:local_modules/@veupathdb/components",
     "@veupathdb/coreui": "^0.18.18",
     "@veupathdb/eslint-config": "^2.0.0",
     "@veupathdb/http-utils": "^1.1.0",

Then the compiler warnings change to something a bit different
webpack compiled with 24 warnings
ERROR in src/lib/core/__test__/hooks/useAnalysis.test.tsx:72:41
TS2339: Property 'children' does not exist on type '{}'.
    70 | } as AnalysisClient;
    71 |
  > 72 | const wrapper: React.ComponentType = ({ children }) => (
       |                                         ^^^^^^^^
    73 |   <WorkspaceContext.Provider
    74 |     value={{
    75 |       analysisClient,

Yarn is still installing<EMAIL_ADDRESS>so I doubt the 18.x types are going to work...

@bobular well I didn't see such an error so I don't know what to say.

@bobular well, I didn't try your method honestly as I am working on other stuff. As I said in my first description (https://github.com/VEuPathDB/web-components/pull/445#issue-1563388756), I could make it by manually adding react-leaflet at dependencies

@bobular well, I didn't try your method honestly as I am working on other stuff. As I said in my first description (#445 (comment)), I could make it by manually adding react-leaflet at dependencies

Sure - I was looking for a longer-term solution that we can merge to main - to save Dave some time.
I have found the following:
Forget everything about react-leaflet v3 and npm-pack-here etc.
If I delete yarn.lock and do a yarn again, we get the same compile warnings as in https://github.com/VEuPathDB/web-components/pull/445#issuecomment-1410633997
Something is causing @types/react to go to v18.0.27 (as in https://github.com/VEuPathDB/web-components/pull/445#issuecomment-1410642948)  - yarn why says that a bunch of things are "asking" for that, for example @types/react-router-dom which has this dependency:    "@types/react": "*",
Unfortunately, I think we're going to have to bother Dave to get a solution where we don't have to delete yarn.lock!

But maybe we don't need a permanent solution...  Maybe it will work fine once the web-components module is merged and built via npm.

I did some careful testing of the maps (FSM and viz) and thumbnails in EDA. All seems to be working fine.
My only comment is to be careful merging the mosaic changes (but maybe they are already in main - I notice that this isn't a PR to merge into main)

@bobular yes those are merged from main so they would not make a conflict. I think that it was because I didn't commit the merge of the main to react-leaflet-v3-ref. Now they are not shown.

@bobular as you suggested at https://github.com/VEuPathDB/web-components/pull/444, I have changed the default color to silver, rgb(192,192,192) here too.

:tada:


Problem with tailing textfile in Windows

Hi, I'm not sure if I missed something or some extra step, but same code which gives good response on all of my linux systems doesnt give same result in windows

that's literally all of the code
that console.log doesnt output anything at all unless I close the program which is writing to the log file.As soon as I quit the program thats writing to the log file, all of the log data gets printed to console at once. That behavior is exhibited only on windows.
Same writing program on linux. And console.log output is perfect.
Please let me know if I'm doing anything wrong. Thanks

Hey,
I can't see anything wrong.
It's probably caused by fs.watch implementation, the nodejs core function used by tail, that doesn't have 100% compatibility across different OSes:
https://nodejs.org/dist/latest-v14.x/docs/api/fs.html#fs_caveats
On Windows, you could try to use WatchFile instead of watch, by useWatchFile as a constructor parameter:
https://github.com/lucagrulla/node-tail#constructor-parameters
I hope it helps.

thanks for fast response
my program will mostly be running on linux, so its fine
I'll try switching to watchFile() for windows use, although it would be good if you perfected the method from your end since I'm not really experienced in the thing

Do you think we need to mention fs.watchFile is actually polling the file and has a default interval of 5007ms? Cost me another 20min to figure this out lol.


using react-font-awesome  in webpack config

I'm trying to using this font-awesome css icons from this github repo.
https://github.com/andreypopp/react-fa
Despite its instructions to use, I'm not entirely sure it's applicable for this static boilerplate I'll be using since the boilerplate comes preconfigured with the css loaders already.
Thus I'm just curious to know how to configure webpack config to import font-awesome?  I'm not familiar with webpack environment setup btw vs gulp/grunt.

The simplest solution is just add the following line to components/Html/Html.js:
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" />

Then you can use those icons in React components as described in FontAwesome tutorial:
<i className="fa fa-camera-retro fa-5x />

Great! Will try that out.


Chunk Index inconsistency between hover and x-axis

Describe the bug
The x-axis chunk index and the hover chunk index do not have the same numbers when plotting both reference and analysis. x-axis when using chunk_count (and other chunkers) without a timestamp column specified.

To Reproduce
Run the following code sample with nannyml 0.10.1
import nannyml as nml
from IPython.display import display

reference_df, analysis_df, _ = nml.load_synthetic_car_loan_dataset()
display(reference_df.head())

column_names = ['car_value']

calc = nml.UnivariateDriftCalculator(
    column_names=column_names,
    treat_as_categorical=['y_pred'],
    continuous_methods=['jensen_shannon'],
    categorical_methods=['jensen_shannon'],
    chunk_number=6
)

calc.fit(reference_df)
results = calc.calculate(analysis_df)

figure = results.plot(kind='drift')
figure.show()

Expected behavior
Indexes will be consistent.

Main idea here was that the "index" represents the number of the chunk in its respective dataset, i.e. reference and analysis. I'm not sure if that makes sense though, in the light of an "ever growing" analysis dataset and repeat NannyML runs...
Options for fixing this:

Check if we can use custom X-axis labels that show the chunk index according to the interpretation as listed above
Change the chunk index in the hover to match the one used for plotting on the X-axis.

What do you think is better?


iOS long music skip to next

Version
4.17.1
Description
When I load a playlist with 1 hour musics, on iOS (Safari, Chrome) after 1 minute skip to the next music (onAudioPlayTrackChange hook ) without any error.
Anyone has encountered this problem before?

I don't understand your mean, please input by issue template


Remove JWT Validation Module entirely because docs pipeliene doesn't.

The pipeline removed examples but all the other stuff is still there :/
I'm not sure if this will get clobbered by future pipeline runs if we merge it. Maybe better to change this and merge to main now??

I'm not sure if this will get clobbered by future pipeline runs if we merge it.

yea I think it probably would get overridden by new runs of the pipeline if we merged into the dev branch. I think it makes sense to go to main with this


PReLU activation implementation

I wanted to gauge interest on adding a PReLU activation. I noticed that flax.linen.activations are simply aliasing jax.nn activation functions which also doesn't have a PReLU implementation.
To add some background, PReLU is simply Leaky ReLU where the alpha (slope) parameter is trainable and not fixed. This makes it simple to implement as a Module if desired.
Here's an example implementation from another project of mine.
from functools import partial
from typing import Any, Sequence

import jax.numpy as jnp
import flax.linen as nn

# This is nearly identical to jnp.ones however multiplies the output of jnp.ones by the constant value
def constant(key, shape: Sequence[int], value: Any, dtype: Any = jnp.float32) -> jnp.ndarray:
    value = jnp.asarray(value, dtype)
    return jnp.ones(shape, dtype) * value

class PReLU(nn.Module):
    negative_slope_init: float = 0.01
    dtype: Any = jnp.float32

    @nn.compact
    def __call__(self, x: jnp.ndarray) -> jnp.ndarray:
        x = jnp.asarray(x, self.dtype)
        negative_slope = self.param(
            "negative_slope",
            partial(constant, value=self.negative_slope_init, dtype=self.dtype),
            (1,)
        )
        return jnp.where(x >= 0, x, negative_slope * x)

Given that all current activation functions reside in JAX, it seem more fitting to add this JAX. Do you want to file an issue against their repo?

Thanks for the suggestion. The main reason I filed the issue here was because it seems like PReLU is a special case where it has a trainable param and, if I'm not mistaken, all other jax activations do not.

@isaaccorley - hey so sorry for the slow feedback on your suggestion here.
2 points:

instead of defining a constant init func, we can just declare a jnp scalar array of the correct dtype.
I think an -activation- "function" should strictly follow the dtype of its argument, so no dtype attribute, just derive it from x

So what if we added something like this?
class PReLU(nn.Module):
    negative_slope_init: float = 0.01
    @nn.compact
    def __call__(self, x: jnp.ndarray) -> jnp.ndarray:
        negative_slope = self.param(
            "negative_slope",
            lambda k: jnp.array(self.negative_slope_init, x.dtype)
        )
        return jnp.where(x >= 0, x, negative_slope * x)

I'm indifferent on the implementation. I think the only thing to point out would be since we are inheriting from Module and other Modules have a dtype param, should we stray from that standard even though it is an activation function?
I created a constant init func because jax itself seemed to be lacking one, however I haven't received a response to the issue I posted in the jax repo requesting to add it so I'm fine with just using a lambda.

Other Modules have a dtype param to control the precision of their -intermediate- values, and a simple activation function like this doesn't have intermediates.  We don't require modules to surface a dtype= attribute - it's just convention for the core layers to do so to give users the ability to control the floating-point types of the "insides"

The "constant" functions you're looking for already exist: jnp.full and jnp.full_like

Makes sense thanks for clarifying that.
Thanks for pointing me jnp.full. I wasn't aware of that.

Shall I make a PR then?

Yeah if you'd like to make a PR we could add the above to activations.py I think (after all the passthrough function imports).  (but no pressure - if you don't have time we can add it soon ourselves.)

I'll try to take a first stab at it since it will be my first time contributing to flax.

The current implementation of PReLU does not work as an activation function.
The following example code raises an error at initialization
class MLP(nn.Module):
    """ Definition of MLP

    attributes
    ----------
    :param hidden_sizes: list of int corresponding to the number of neurons in each hidden layer.
    :param output_size: int corresponding to the number of neurons in the output layer.
    """
    hidden_sizes: Sequence[int]
    out_size: int

    @nn.compact
    def __call__(self, x, **kwargs):
        name = kwargs.pop('name', 'fc')
        for e, size in enumerate(self.hidden_sizes, 1):
            x = nn.Dense(size, name=name + str(e))(x)
            # x = nn.silu(x)
            x = nn.PReLU(x)
        x = nn.Dense(self.out_size, name=name + '_output')(x)
        return x

    186 @compact
    187 def __call__(self, inputs: Array) -> Array:
    188   """Applies a linear transformation to the inputs along the last dimension.
    189
    190   Args:
   (...)
    194     The transformed input.
    195   """
    196   kernel = self.param('kernel',
    197                       self.kernel_init,
--> 198                       (jnp.shape(inputs)[-1], self.features),
    199                       self.param_dtype)
    200   if self.use_bias:
    201     bias = self.param('bias', self.bias_init, (self.features,),
    202                       self.param_dtype)

IndexError: tuple index out of range

PReLU is not following the same definitions as the other activations.

Because PReLU initializes a trained scalar parameter, it has to be treated as a layer.  I've added clarification and example usage to the docs in #3122


[v6] Upgrade min version of Node.js from 12 to 16

Duplicates

[X] I have searched the existing issues

Summary 💡
Node v12 has reached end-of-life rather its ended. https://nodejs.org/en/about/releases/
Examples 🌈
Similar to dropping support for node v10 in #25306.
Motivation 🔦

Upgrade CI docker node image etc.

cc: @mui/core

We can use a recent Node version for building the library, but I'd leave Node 12 for tests as we still support it. We will likely drop support for it with the next major version.

We will likely drop support for it with the next major version.

Maybe we should link this issue in the next major version roadmap. Or somewhere else.

Just a quick update,  Node.js 16 is now unsupported, so we probably want to bump this up to 18.
https://nodejs.org/en/about/previous-releases

v18 is in maintenance mode, so I'd say we should go for 20.

What's the value of this change? For example React has
{
node: ">=0.10.0"
},

I fail to see how removing support for older version of node is more helpful than paiful.

For example, being able to use native ES modules.

I think we should follow this strategy: https://github.com/mui/mui-x/pull/10997#issuecomment-1924227287, meaning:

It's not our role to tell developers which Node.js version to use.
Get a sense of the ecosystem version distributions
Have the Node.js version as wide as possible, cover 80-95% of the active version installation
Only upgrade a Node.js version if it falls too low under the above 5% of active version installation or if it's necessary to solve something.

There doesn't seem to be an agreement in https://github.com/mui/material-ui/issues/40958.

it's not really our role to tell developers which Node.js version to use

Agree, it's not. Node.js itself tells developers which versions not to use by putting them out of maintenance. We can follow their guidance.
I'm not advocating for increasing the minimum version just for the sake of it, but using older versions comes with a couple of issues:

We won't be able to use some supporting packages. For example Chai requires v18
Old Node versions don't support ESM (or support them with a flag)
If we don't test our packages in old versions of Node, how can we say we support them?

We won't be able to use some supporting packages. For example Chai requires v18

Agree, we need to increase the version of the CI to match Node.js LTS

Old Node versions don't support ESM (or support them with a flag)

We will see, but I suspect that the ecosystem isn't ready for a ESM only package, so I suspect that v6 would be a hybrid version: https://twitter.com/wooorm/status/1663129796195364865.

If we don't test our packages in old versions of Node, how can we say we support them?

So far, we have relied on bug reports, and fixing them.
I imagine that having one smoke server-side rendering test could be enough. Is it worth the effort? If there are users using older versions of Node.js I think so. If there aren't enough developers, then it might not be worth the effort to maintain. Now, honestly, I don't see this as being a lot of work. Create a new npm package, use the build output, run a simple node server that render a few components, see if it fails or if it works.

Webpack: there are 50% of the users on v4 supports node down to v6

Not sure who's using Node.js v6 and at the same time is serverside rendering React 😄, but I'd love to see how they've got that set up. Webpack is used as well for other purposes than React and those are more than likely the ones v6 support exists for.

it's not really our role to tell developers which Node.js version to use

We do tell them which browsers to use though. When they want to use older than supported browsers they'll have to transpile the packages themselves. One could argue that the same should apply to node.js versions.

For example, being able to use native ES modules.

According to their docs v12 is the first version that declares ESM stable. So I guess there is no real problem on this front?
Anyway, I'm personally with @oliviertassinari here, also leaning towards keeping support for Node.js 12 version for our packages. I don't think at the moment it blocks us from moving forward with anything, but:

I don't think this restriction should apply to our tooling, i.e. workspace root should be at LTS minimum. At least for security sake.
I wouldn't phrase this as "We support Node.js 12", I'd phrase this as "We actively support Node.js v18, but we're not going to ruin your day if you're on node 12 and try to install our package". Mainly because it's highly likely you're installing it in Node.js v12 without the intention to also run it on node 12. This is still a frontend lib.

I'm moving this out of the v6 milestone per the last comment. Upgrading the minimum version of node at this moment doesn't seem to bring any benefit to users, and it's not blocking anything either.
Please let me know if this should be reconsidered.

Being lazy, waiting to see the problem, makes sense to me.
I'm amazed by how quickly people moved "Today, the ecosystem seems to be" https://github.com/mui/material-ui/issues/32546#issuecomment-1928032163 in 6 months https://tools-public.mui.com/prod/pages/npmVersion?package=
-Next.js: the most used version is v12. That Next.js version works down to node v12
+Next.js: the most used version is v14. That Next.js version works down to node v18

-Webpack: there are 50% of the users on v4 supports node down to v6
+Webpack: there are 24% of the users on v4 supports node down to v6. v5 supports node down to v10

-Vite: most of the users are on v4 which supports node down to v14
+Vite: most of the users are on v5 which supports node down to v18

-pnpm latest<EMAIL_ADDRESS>supports node down to v16
+pnpm: most of the users are on v8 which supports node down to v18

I have just realized that MUI X is on Node.js v14 as their minimum version: https://github.com/mui/mui-x/pull/6966. I think we should sync those two,  so developers have a more consistent experience.

Updated https://github.com/mui/material-ui/pull/42920 to sync version with MUI X

Merged https://github.com/mui/material-ui/pull/42920, Material UI is now synced with MUI X, closing this one.


Added license section

Description
I have added a license section to the README file, specifying the licensing terms for this project.
Added unit tests?

[ ] No
[ ] Yes

[optional] Random GIF to +1 someone's day
closes #175

@saurabhdaware done


Test dir not part of crate

Hi,
I'm packaging this crate for Guix System. Part of the build is also to run the test suite. As this crate is available on crates-io I just download it from there.
The only issue is when I execute the package build it fails on the tests as the test dir is not part of the crate tar.gz file. Could you please add the test directory to the crate manifest?

Sorry for the delay in responding to this issue.
I am not familiar with Guix, so I'm curious whether running the tests is a requirement of the package management system or if this is something you have decided to do? Just trying to understand the motivation behind this, as if it is the latter case then the obvious short-term solution would just be not to run these tests :)
I'm a little apprehensive of adding this directory mainly for two reasons:

the tests directory contains binaries, and will increase the size of the distribution artifact by a non-negligible amount
these tests do not actually cover all supported chips at present, so they are far from comprehensive; if we were to include this data I would like to see better coverage, personally

Hi @jessebraham,
Thanks for explaining the reasons. Well, Guix is little bit different GNU/Linux(or Hurd) distro, similar to Nix (and Nixos). They have novel way of packaging stuff to name the most important feature.
When porting a package you write a pkg description. Part of the that is the build system. For rust it's pretty obviously cargo. There are some additional changes not important here to get Rust code working on non FHS compliant distro.

these tests do not actually cover all supported chips at present, so they are far from comprehensive; if we were to include this data I would like to see better coverage, personally
However, the most important feature is that the build system attempts to run tests by default. Only if they pass will you build the package. IMHO the idea is verify the package is build and runs correctly if the author(s) already provide the test suite even though if it's incomplete - better to run some tests than none.

This differs from other distros which usually in general just configure, build and install the package.

the tests directory contains binaries, and will increase the size of the distribution artifact by a non-negligible amount

Now I understand why Rust and also Python packages on pypi.org don't have any test directories packaged in their respective formats. That's why the tests always fail as the contents of the git repository is not the same as the content of the pypi or cargo package.
I do understand it's additional size and the need not to run test suits - probably applicable to many users. It's just nobody stressed this enough anywhere.
Then again it's not requirement as Guix can clone the whole repo instead of downloading the crate itself.
So you can mark this ticket as solved :-)


Error: unable to parse anomalies

In the Visualizing Data section - "Any metric from the Integration on your Database with the anomaly function applied."
Running this through the API throws the following error when trying to use anomalies with a metric
{"errors": ["Invalid query in widget at position 2. Error: unable to parse anomalies(avg:mysql.performance.cpu_time{host:i-0697fbf09f4a9c384}, basic, 2): Rule 'scope_expr' didn't match at ', 2)' (line 1, column 74)."]}
This anomaly can be added manually via the GUI, but if I then pull the JSON through a GET request to see what it looks like and POST it via cURL it throws the above error. Here is the snippet of the definition. Same goes if I look at the JSON in the editor vie the DataDog GUI, it does not work when passed in via the API.
Note I can successfully create other dashboards, just not when the anomaly function is used.
"definition":{   			"requests":[   			   {   				  "q":"anomalies(avg:mysql.performance.cpu_time{host:i-0697fbf09f4a9c384}, 'basic', 2)", 				  "style":{   					 "line_width":"normal", 					 "palette":"dog_classic", 					 "line_type":"solid" 				  }, 				  "display_type":"line" 			   } 			], 			"type":"timeseries", 			"title":"CPU Time with the anomaly function applied." 		 }

This was a user error, closing issue


[SCB-780] sessionstickrule add the judgement of whether lastServer can be accessed.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.02%) to 86.699% when pulling 466425669b3bfa10419bb0dffe8938f5e47aadfc on weichao666:fixsessionstick into 40981a8e93e6c9e135959074b1897950eda4b59e on apache:master.


Feature request: Integration of Code Formatting and Linting Tools for the improve Function Workflow

Policy and Info

Maintainers will close issues that have been stale for 14 days if they contain relevant answers.
Adding the label "sweep" will automatically turn the issue into a coded pull request. This works best for mechanical tasks. More info/syntax at: https://docs.sweep.dev/
Consider adding the label "good first issue" for interesting, but easily approachable features.

Feature Description
I propose the integration of code formatting and linting tools into the workflow of our improve function. This would involve two key stages: pre-processing and post-processing. In the pre-processing stage, before the code is sent to LLMs for improvement, the code would be formatted and linted to ensure it adheres to standard coding practices and is free of syntax errors. In the post-processing stage, after the LLMs have applied changes, the code would again be subjected to formatting and linting to refine the improvements, rectify any introduced errors, and standardize the code style.
Motivation/Application
The integration of these tools serves several purposes:

Pre-processing checks: By checking the code for errors before sending it to LLMs, we can reduce the computational load on LLMs, allowing them to focus more on the complexity of the tasks rather than correcting basic syntax errors. This not only enhances efficiency but also improves the quality of the improvements suggested by LLMs.

Post-processing refinement: After LLMs have applied their changes, a second round of formatting and linting can significantly enhance user experience by ensuring that the improvements adhere to standard coding practices, and any errors inadvertently introduced by LLMs are corrected. This step is crucial for maintaining code quality and readability, especially for dynamically typed languages where indentation and formatting are vital, like Python.

Suggested Tools:

Python: Utilize Black for code formatting and Pylint or Flake8 for linting to ensure adherence to PEP 8 standards and to catch common errors.
JavaScript: Employ Prettier for formatting and ESLint for linting, allowing for customizable rules to fit our project's coding standards.
Java: Implement Google Java Format for formatting to ensure the code is consistent with Google's Java Style Guide.

These tools are widely recognized for their robustness and flexibility, making them suitable for integration into our existing workflow. This feature would not only streamline the development process but also significantly enhance the quality and reliability of code improvements generated through the improve function.

Hey! Sounds like a useful feature. I had some questions;
I wonder whether it’s a good idea to run the linter if we can’t locate its config-file. ChatGPT has seen way more code than we have, so it probably has a pretty good idea of how to produce code with popular styling. So got-engineer selecting a default may just get in the way.
One question is who should fix the errors that the linter can't fix itself, user or LLM? Or should we ignore them altogether and just focus on --fixing?
Some suggestions

I think it would be helpful to make it easy to disable autolinting for a project, for example where the rules don’t interface well with GPTs output. Or enable use of a separate config, for the same reason. However, I don't remember there being a .gpt dir or some other place we put config. Is that possible somehow yet? Otherwise I suggest env-var and cli option to disable/set linting config file.
I suggest creating a generalizable lint-config class and add a manual on how contributors can easily add parsers for other languages

CC @ATheorell @similato87

@Styren Thanks for the insightful suggestions! Your idea of managing linter configs and the approach to handle non-auto-fixable errors are crucial. I'm inclined to let LLMs address these errors, while we should ensure users are informed about this process.
The concept of toggling autolinting and introducing a flexible configuration system, perhaps through a .gpt directory or environment variables, is something we should pursue. While we haven't set up a config system yet, it's a direction worth exploring.

I want to add some context here and I'll speak mainly for python, as I don't have experience with the linters in the other languages.
First of all, for the cases where the linter fails, I agree with the previous comments.
The reason this thing came up was that, particularly in improve mode, chat-gpt doesn't even get the indentation right. The LLM actually isn't that good at following PEP-8. Reason for this is probably that it has seen so much code which doesn't. The linter fixes the indentation problem in 100% of the cases I've seen. Whenever  we can use a deterministic system, like a linter, which: is not random, is super fast, very reliable and very cheap, we should prefer that over using the LLM. I can honestly not think of a case where the LLM would right better style than the linter. Additionally, in many projects, like gpt-engineer, a standard linter is part of the CI-pipeline and I think this is how it should be. When gpt-engineer generates code from scratch, I really cannot see any reason why it should not be linted. In improve mode, I can imagine that linting could be a problem in rare cases. I would vote to have linting by default and the possibility to turn it off with some flag, for the few cases where people really don't want neat code or follow some outdated code standard.

We have implemented this for https://gptengineer.app, and are in the process of adopting a gpt-engineer.toml configuration file for projects we generate, which will contain configurable entrypoints for build/test/lint/format tools.
I will present a proposal regarding this at the meeting later today :)


[improvement](stream load) http_stream support max_filter_ratio

Proposed changes
curl -v --location-trusted -H "expect:100-continue" -u root: -H "sql: insert into test.t2(id, name, score) select c1,c2,c3 from http_stream(\"format\" = \"csv\", \"column_separator\" = \",\", \"max_filter_ratio\" = \"0.7\") where c2 = 'a'" -T test_stream_load3.csv http://<IP_ADDRESS>:8032/api/_http_stream

{
    "TxnId": 25236,
    "Label": "insert_f04aca0e4da63609_11494640e9295ac",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 5,
    "NumberLoadedRows": 3,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 46,
    "LoadTimeMs": 37,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 8,
    "ReadDataTimeMs": 9,
    "WriteDataTimeMs": 9,
    "CommitAndPublishTimeMs": 18,
    "ErrorURL": "http://<IP_ADDRESS>:8042/api/_load_error_log?file=__shard_68/error_log_default_cluster:test_insert_f04aca0e4da63609_11494640e9295ac_f04aca0e4da63609_11494640e9295ad"
}

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Add the correct links for G-Cloud 12

https://trello.com/c/Y8E1lAQM/339-update-g-cloud-12-framework-metadata-govuk-links
As confirmed by Nora. When we get the final agreement PDF tomorrow, we'll update this PR and then merge and release.

FYI we'll need a (patch) version bump in package.json.


Dependency error occurred in Chapter 6 of Google Colab.

Which is.
!CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python

This error msg came out.
Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 2.9 MB/s eta 0:00:00
Building wheels for collected packages: llama-cpp-python
  error: subprocess-exited-with-error

  × Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for llama-cpp-python (pyproject.toml) ... error
  ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (llama-cpp-python)

I was working around it, but it's painfully slow to compile
!CMAKE_ARGS="-DLLAMA_CUDA=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir --verbose

Thank you for sharing this! I think we can speed up all of this by installing pre-built wheels instead:
pip install --no-cache-dir llama-cpp-python==0.2.78 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122

It will still take a while but at least it should be a bit faster and more stable by providing the extras.


[DO NOT MERGE]Experimental print dependencies and transitive dependencies in buildlog

This is an experimental PR to print dependencies and transitive dependencies in build log.
Please do not merge.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

[test]

[test]


say thanks is broken

Tha say thanks button seems to be broken

Thank you for reporting this! Say thanks changed their link format. I changed it to be a mailto link in #723 for now.


Update the ABI dumps.

We decided to update the ABI dumps to learn more about how these tools
work. I updated the ABI dumps for google_cloud_cpp_common,
storage_client (was missing) and bigtable_client. These files are huge,
so I am committing them compressed, that also keeps the line number
stats more sensible.

This change is

In case we are interested in the output, here is the (text formatted) API report for the latest changes:
https://travis-ci.com/googleapis/google-cloud-cpp/jobs/183088441#L4097
The specific section around source-level compatibility is here:
https://travis-ci.com/googleapis/google-cloud-cpp/jobs/183088441#L4637
I found some of the source level reports interesting:
https://travis-ci.com/googleapis/google-cloud-cpp/jobs/183088441#L5392

PTAL. We said we would be interested in the results of the ABI checker.  Here are some:
https://travis-ci.com/googleapis/google-cloud-cpp/jobs/183088441#L4097
for the changes since more or less yesterday.  I think the formatting as text is kind of terrible, it would be much better if we could capture the HTML output and host that somewhere (I believe we can do that with Kokoro).
In any case, the report tells us about stuff we changed (changed a bunch of return types, added some functions, etc.) Which I think would be nice to know about.


mutex lock/unlock mismatch

Received via email:

mbedTLS version 2.0.0
In function mbedtls_rsa_public:
if mbedtls_mpi_read_binary fails, the macro MBEDTLS_MPI_CHK does goto cleanup where ctx->mutex is unlocked without having been locked first.
Same problem is in mbedtls_rsa_private.

I also audited our source code looking for other places where control could jump to and unlock before the corresponding lock, an found none. We added a note in our backlog about testing that using an dummy mutex implementation.