Remove compute-rw scope from GCE nodes

AFAIK this is currently necessary only for mounting GCE PDs onto a kubelet. If we moved the PD mounting from the kubelet to the master then we could drop this scope from the node VMs.

Agree with goal, but not with proposed solution. My preferred solution here is to declare that the cloud provider library is responsible for getting auth tokens for any operations it needs to perform, and write that logic there. Reasoning:

There's no convenient master component to stick this logic into.
Counting on the default scopes that come with VM creation is a non-solution anyway, since e.g. some parties want to do cross-project VMs and scopes in GCE at least don't work that way AFAIK.
Making the cloud provider library responsible for this provides a model that should work in all environments.
This will be a lot less code churn (i.e., no api change) than moving the mount call out of kubelet.

Would that be the cloud provider library that is part of the kubelet? If so, how does that allow us to remove the overly broad scopes? Or are you saying that something on the master would get the auth token and pass it to the kubelet (maybe as part of the pod spec)?

We already have a mechanism to pass a configuration file to the cloud
provider; we could use that. Master could make a secret for kubelets with
an auth token. Or we could do that at start up time.
On May 12, 2015 3:24 PM, "Robert Bailey"<EMAIL_ADDRESS>wrote:

Would that be the cloud provider library that is part of the kubelet? If
so, how does that allow us to remove the overly broad scopes? Or are you
saying that something on the master would get the auth token and pass it to
the kubelet (maybe as part of the pod spec)?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/8074#issuecomment-101440838
.

On GCE, we have to end up with a Google trusted (not just k8s master trusted), compute-rw scoped access token at the point where we make the AttachDisk call.
If we want this call to happen in the kubelet (as it does today), I see three options:

Node VM has compute-rw scope (status quo). Google client libraries know how to get access token from metadata server.

This means that everything running on that node also has compute-rw capabilities (trivially).

Node VM gets refresh token passed to cloudprovider config at startup. Configure Google client calls to exchange refresh token for access token (or use cached access token).

This would mean that anything that can see that config has permanent (until revoked) compute-rw access.
I'm also not sure if we have a way to dish out service account tokens, so these might have to be end-user credentials, which is bad.

Node VM gets a master-created secret at startup, uses that secret to request access tokens from master, then makes Google client calls with those tokens.

Anything that can see that secret can get (short-lived) compute-rw access.
We are putting the master in the loop, why not just make it responsible.

If we had a PersistentDiskController in the master, it could just find scheduled pods that need disks attached, and find attached disks that are no longer needed by the current set of scheduled pods.
If we say that the cloudprovider inside kubelet is responsible for being able to make authorized requests, I think we are also saying that anything within the same security boundary as the kubelet will be able to make the same authorized requests.

From a security standpoint, I like the idea of a PersistentDiskController
which runs wherever its runs (let's try not to use the word master), and
which has a GCE token to let it mount disks.
Need to analyze what happens in bootstrapping and failure situation.
Also, wonder how many other types of cluster filesystems support a remote
attachment approach.  If it is just GCE, maybe not worthwhile.
Another option is to wait until GCE has some sort of finer grained
permissions.
On Tue, May 12, 2015 at 11:43 PM, CJ Cullen<EMAIL_ADDRESS>wrote:

On GCE, we have to end up with a Google trusted (not just k8s master
trusted), compute-rw scoped access token at the point where we make the
AttachDisk call.
If we want this call to happen in the kubelet (as it does today), I see
three options:

Node VM has compute-rw scope (status quo). Google client libraries
know how to get access token from metadata server.
- This means that everything running on that node also has compute-rw
capabilities (trivially).
2.
Node VM gets refresh token passed to cloudprovider config at startup.
Configure Google client calls to exchange refresh token for access token
(or use cached access token).
- This would mean that anything that can see that config has
permanent (until revoked) compute-rw access.
- I'm also not sure if we have a way to dish out service account
tokens, so these might have to be end-user credentials, which is bad.
3.
Node VM gets a master-created secret at startup, uses that secret to
request access tokens from master, then makes Google client calls with
those tokens.
- Anything that can see that secret can get (short-lived) compute-rw
access.
- We are putting the master in the loop, why not just make it
responsible.
If we had a PersistentDiskController in the master, it could just find
scheduled pods that need disks attached, and find attached disks that are
no longer needed by the current set of scheduled pods.
If we say that the cloudprovider inside kubelet is responsible for being
able to make authorized requests, I think we are also saying that anything
within the same security boundary as the kubelet will be able to make the
same authorized requests.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/8074#issuecomment-101535501
.

And if we move to a reconciler-style controller for attaching the disk, we'd decouple that from the local mounting that kubelet has to do, hopefully preventing issues like #7972.

Having the Attach/Detach Controller landed in 1.3 (https://github.com/kubernetes/kubernetes/pull/26351), does it mean we can remove the compute-rw scope from the nodes?

That (and moving the ingress controller to the master) were the last remaining things that I knew of blocking us, but we'd need to run e2e tests on a cluster w/o scopes to see what might have picked up an inadvertent dependency on having the scope available.
For example, @vishh found (in https://github.com/kubernetes/kubernetes/pull/28871) that the kubelet had picked up a dependency on having a service account attached to the VM. There may be other dependencies that we haven't found yet.

Auto-configuring gcr.io credentials require service accounts as of now. I
wonder if there is a way to work around that?
On Sat, Jul 23, 2016 at 9:53 PM, Robert Bailey<EMAIL_ADDRESS>wrote:

That (and moving the ingress controller to the master) were the last
remaining things that I knew of blocking us, but we'd need to run e2e tests
on a cluster w/o scopes to see what might have picked up an inadvertent
dependency on having the scope available.
For example, @vishh https://github.com/vishh found (in #28871
https://github.com/kubernetes/kubernetes/pull/28871) that the kubelet
had picked up a dependency on having a service account attached to the VM.
There may be other dependencies that we haven't found yet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/8074#issuecomment-234757738,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKLdhqopu3oVwe8SBI56JUM-C7ethks5qYu_VgaJpZM4EXEK5
.


Add support for eth_newPendingTransactionFilter full rpc function

Describe the feature
the eth_newPendingTransactionFilter takes an optional bool parameter that determines whether the filter is for hashes or full transactions:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/filter.rs#L238-L238
this is similar to the pubsub function parameters:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc-types/src/eth/pubsub.rs#L99-L109
TODO

add new parameter type in rpc-types/filter.rs that is an enum PendingTransactionFilterKind {Hashes, Full}
add Serialize and Deserialize impls that correspond to Option<bool>, similar to: https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc-types/src/eth/pubsub.rs#L125-L138
add this as an parameter to the EthFilter function
depending on the input parameter either install a listener for pending hashes like it is now:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/filter.rs#L240-L242
or install a listener for full transactions:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/transaction-pool/src/traits.rs#L147-L147
see pubsub for reference: https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/pubsub.rs#L270-L280

add an enum for both variants where one variant for hashes is: https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/filter.rs#L481-L485
add a new variant for Transaction to the filterchanges type:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc-types/src/eth/filter.rs#L827-L833
Additional context
@DoTheBestToGetTheBest
No response

Thank you so much. I take this

add this as an parameter to the EthFilter function

Hey Mat, sorry to ping you for this but please can you tell me what is the EthFilter function ?

ah sorry, I see how this was worded badly
it's this one

async fn new_pending_transaction_filter(&self) -> RpcResult {

ah sorry, I see how this was worded badly

No don't be sorry, it's normal. Thank you so much for the fast answer

it's this one

async fn new_pending_transaction_filter(&self) -> RpcResult {

Ty lot !

Describe the feature
the eth_newPendingTransactionFilter takes an optional bool parameter that determines whether the filter is for hashes or full transactions:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/filter.rs#L238-L238
this is similar to the pubsub function parameters:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc-types/src/eth/pubsub.rs#L99-L109
TODO

add new parameter type in rpc-types/filter.rs that is an enum PendingTransactionFilterKind {Hashes, Full}

add Serialize and Deserialize impls that correspond to Option<bool>, similar to: https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc-types/src/eth/pubsub.rs#L125-L138

add this as an parameter to the EthFilter function

depending on the input parameter either install a listener for pending hashes like it is now:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/filter.rs#L240-L242
or install a listener for full transactions:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/transaction-pool/src/traits.rs#L147-L147
see pubsub for reference: https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/pubsub.rs#L270-L280

add an enum for both variants where one variant for hashes is:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc/src/eth/filter.rs#L481-L485
add a new variant for Transaction to the filterchanges type:
https://github.com/paradigmxyz/reth/blob/0f254e16a50a6d39543fe59590227a09fd0e21bb/crates/rpc/rpc-types/src/eth/filter.rs#L827-L833
Additional context
@DoTheBestToGetTheBest No response

hey mat, i have a problem here if you can help please
when i add the type in parameter of the function, i need to also add it to into the trait where the function is defined. But i can't import PendingTransactionFilterKind where the trait function is defined. Any idea how can i solve this please?

here : https://github.com/paradigmxyz/reth/blob/fc4fc936807870b07acb91e1d50c3920b55ff055/crates/rpc/rpc-api/src/eth_filter.rs#L18

hmm, I don't think this is an issue if you

define PendingTransactionKind in reth-rpc-types
use it in reth-rpc-api

because reth-rpc-api already depends on it
feel free to open a draft anytime

hmm, I don't think this is an issue if you

define PendingTransactionKind in reth-rpc-types
use it in reth-rpc-api

because reth-rpc-api already depends on it
feel free to open a draft anytime

oh very good like that. I thought you wanted this type  in rpc-rpc
Thank you so much for your help Mat, realy appreciate this

add an enum for both variants where one variant for hashes is:

hey , sorry for this but just wanna to be sure if you are okay with this name :
are you okay with this type name ?
enum PendingTransactionsReceiverKind {
    Hashes(PendingTransactionsReceiver),
    Full(NewSubpoolTransactionStream<Transaction>),
}```

if you don't like this name for this type i can take anything you suggest

sounds good


VB.NET docs: Factual error in IIf function description

Describe the bug
The VB.NET IIf function documentation: https://docs.microsoft.com/en-us/dotnet/api/microsoft.visualbasic.interaction.iif?view=netcore-3.1
In the Remarks section, it claims that

The IIf function provides a counterpart for the ternary Conditional Operator: ? : in Visual C++.

This is very incorrect, because the main feature of the ternary operator ? : is that it short-circuits, only evaluating one of its branches and not both.
VB.NET does have a ternary operator of its own, it is called If() operator. That is indeed a counterpart for the ternary Conditional Operator in C++.
The IIf function, on contrary, evaluates both its arguments at all times. It had this behaviour in VBA and in VB6, and it still has it in VB.NET, which can be easily verified:
Sub Main()
    Dim result = IIf(Rnd() > 0.5, SideEffect1(), SideEffect2())
End Sub

Private Function SideEffect1() As Integer
    Console.WriteLine("Side effect 1 caused")
    Return 1
End Function

Private Function SideEffect2() As Integer
    Console.WriteLine("Side effect 2 caused")
    Return 2
End Function

Which outputs:
Side effect 1 caused
Side effect 2 caused

as expected.
The current documentation makes it sound like there is no difference between the If() operator and the IIf() function.
This is a very important omission that will lead to incorrect expectations and wrong code.
Expected behavior
Please fix the documentation to reflect the fact that IIf does not short-circuit, is not therefore a counterpart for the ternary operator, and evaluates its arguments at all times, leading to potentially unwanted side effects. Respective wording can be taken from the VBA documentation Remarks section where it is correct.

This issue was moved to dotnet/dotnet-api-docs#4523


Support description

https://docs.microsoft.com/pl-pl/dotnet/api/system.componentmodel.descriptionattribute?view=net-6.0
You can enter support description. Information from the Attribute can be displayed in the following form:

You can also enter a description here, it will facilitate the database management by people who did not create it. Such a mini documentation

Hello.
We support description via out MetaEntityAttribute and MetaEntityAttrAttribute
[MetaEntity(Description="The orders")]
class Order
{
      [MetaEntityAttr(Description="The order date")]
      public DateTime OrderDate {get; set;}
}

Or via our fluent API
app.UseEndpoints(endpoints => {
    endpoints.MapEasyData(options => {
        options.UseDbContext<ApplicationDbContext>(opts => {
            opts.CustomizeModel(model => {
                model.Entity<Order>()
                    .SetDescription("The orders")
                    .Attribute(o => o.OrderDate)
                           .SetDescrition("The order date");
            });
        });
    });
});

Ant the description will be displayed exactly as you proposed

We are open to contribution if you want to add support of DisplayAttribute as well :)

@antifree Thanks for writing back, in theory you can use Display, there is name, description and whatever you need.
https://github.com/KorzhCom/EasyData/blob/6a8ecec02923ee10cb4026c1432dee62c018a906/easydata.net/src/EasyData.EntityFrameworkCore.Relational/MetaDataLoader/DbContextMetaDataLoader.cs#L395-L466
if (propInfo.GetCustomAttribute(typeof(DisplayAttribute)) is DisplayAttribute displayAttr) {
                    entityAttr.Caption = displayAttr.Name;
                    entityAttr.Description = displayAttr.Description;
}

If I understand the assumption of your code correctly, just do something like this and it should work. I understand well? 😄

Yes. You are right, thank you. I'll add with next release

Add 3b49c869d785b4c2422f6ffc5c1e6bea2a31788e. Will release it tomorrow

Done in version 1.4.4

Works great, thanks!
In theory, you can still make a description here.

We will consider this, thank you!


Dev

small changes for virtual + lookup plugin

This is ready to go @lts-po want to do the honors?


[FLINK-35358][clients] Reintroduce recursive JAR listing in classpath load from "usrlib"

What is the purpose of the change
With https://github.com/apache/flink/commit/e63aa12252843d0098a56f3091b28d48aff5b5af in Flink 1.19 an unintentional change were introduced, that now the DefaultPackagedProgramRetriever only loads JARs from the root level of the usrlib directory.
This change reverts that behavior to search directories in a recursive manner, and adds a test case to cover such scenario.
Brief change log

Flies#list -> Files#walk in DefaultPackagedProgramRetriever.
Add test case in DefaultPackagedProgramRetrieverITCase.

Verifying this change
Added relevant IT case.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
The S3 file system connector: no

@mbalassi @ferenc-csaky can we merge this and https://github.com/apache/flink/pull/24792 ?

@mbalassi @ferenc-csaky can we merge this and #24792 ?

Yes, they should be merged.


Confirm Postgres database support

This PR fixes #1.
It includes:

improve Postgres tests

Currently, I am unable to MariaDb or MySql to work.
There seems to be an issue with using volumes.
However, I was also unsuccessful using ADD and COPY in docker files.
@see https://github.com/MariaDB/mariadb-docker/issues/62

Issues open for

#2
#3


Map/connectivity separation

I saw that my recent PR https://github.com/ForestClaw/forestclaw/pull/271 on map/connectivity separation just got merged and wanted to ask about the status for the remaining new_conn_map occurences. Since @donnaaboise mentioned that she worked on this too, I want to make sure that I do not interfere with your progress on this topic.
I am happy to help with any future work regarding this subject.

No problem - I'll take a look at what you have.  I was making a few more cosmetic changes that I can take care of.  I am going to work on this over the next few days

It seems that there are new functions added.  Before, we had :
) {
    case 0:
        conn = p4est_connectivity_new_cubed();
        cont = fclaw2d_map_new_cubedsphere(fclaw_opt->scale,rotate);

followed by
fclaw2d_domain_t *domain =
        fclaw2d_domain_new_conn(glob->mpicomm, fclaw_opt->minlevel, conn);

Looking at the latest by @hannesbrandt , this has been changed to :
domain  = fclaw2d_domain_new_cubedsphere (glob->mpicomm, fclaw_opt->minlevel);
cont = fclaw2d_map_new_cubedsphere (fclaw_opt->scale, rotate);

Is there a reason new domains are now "connectivity specific"? It does reduce the number of calls, and doesn't expose the connectivity to the user.  Anything else?

It seems that there are new functions added.  Before, we had :
) {
    case 0:
        conn = p4est_connectivity_new_cubed();
        cont = fclaw2d_map_new_cubedsphere(fclaw_opt->scale,rotate);

followed by
fclaw2d_domain_t *domain =
        fclaw2d_domain_new_conn(glob->mpicomm, fclaw_opt->minlevel, conn);

Looking at the latest by @hannesbrandt , this has been changed to :
domain  = fclaw2d_domain_new_cubedsphere (glob->mpicomm, fclaw_opt->minlevel);
cont = fclaw2d_map_new_cubedsphere (fclaw_opt->scale, rotate);

The previous call fclaw2d_domain_new_conn is still valid and will remain.
It is the most general way to construct a domain.  Please feel free to use
that one if you prefer it.  It allows, for example, to switch an example on
the connectivities and program the domain construction only once.

Is there a reason new domains are now "connectivity specific"? It does reduce the number of calls, and doesn't expose the connectivity to the user.  Anything else?

You're right, this is just a convenience function for common cases, and the
user does not see any dependence on p4est functions.

In the PR https://github.com/ForestClaw/forestclaw/pull/280 @scottaiton  separated mappings in the Geoclaw, ThunderEgg, and CudaClaw examples. Since we plan to eventually remove the dependency of  fclaw2d_convenience.{c,h} on the mappings I checked if we can already remove the map dependency in fclaw2d_convenience.{c,h} .
After adjusting applications/clawpack/advection/2d/swirl_rays/swirl.cpp in the same way as used in the PR https://github.com/ForestClaw/forestclaw/pull/280 the code compiled with the configure flags --enable-paper and --enable-clawpack. However, there are still calls of fclaw2d_domain_new_conn_map in the code, which are not compiled in this configuration. Furthermore, it seems to me that these files are not compiled for any build configuration because the respective files are commented out in the build, e.g. in applications/geoclaw/geoclaw.apps.
A special case of a not compiled file is applications/clawpack/advection/2d/filament_swirl/filament/filament.cpp since this file seems to duplicate the functionality of the file applications/clawpack/advection/2d/filament_swirl/filament.cpp.
Is the map separation still work in progress or is the plan to leave the remaining files as they are now?


Empty test class doesn't show warning, but breaks CI by passing on non-zero STDERR from PHPHUnit

If there are no tests in a test class file, PHPUnit raises a warning (Event\Facade::emitter()->testRunnerTriggeredWarning() - 'no tests found in class') and will pass on a non-zero value to STDERR. Collision will not display this warning, but does pass on the STDERR value, which led to some confusion when it appeared that all tests were passing, but our CI pipeline was stopping at the test step in failure. However, narrowing it down to this issue took a while as at first I thought it was a bug in the CI runner.
You can recreate this with a default Laravel install:
composer create-project laravel/laravel empty-class
cd empty-class
php artisan test
# All tests pass
echo $?
# STDERR is 0
cat <(echo "<?php\nnamespace Tests\\\\Unit;\nuse PHPUnit\Framework\TestCase;\n\nclass EmptyClassTest extends TestCase{}") > tests/Unit/EmptyClassTest.php
php artisan test
# All tests pass
echo $?
# STDERR is 1
vendor/bin/phpunit
# Warnings!

It's not clear to me if this is limited to this one 'empty class' issue, or if all testRunnerTriggeredWarning events are ignored.

This also affects downstream test packages like Dusk.
https://github.com/laravel/dusk/issues/1063

This also affects the --coverage option. No coverage output is generated with empty test classes present.

Hi there,
Just wanted to follow up on this. I've also encountered the same issue.
Do we have any updates on the plans for a fix? Additionally, I'm curious about the status of merging/releasing #294. Could someone also explain why this warning was initially hidden/suppresed? Thanks!

Bump 👀


Star tables

Simple star tables support.
No additional tests yet. Only failing test is currently failing in master.
Will change commit messages from WIP and clean up the code, but keen to hear feedback. I'm running a parallel PR against our fork since time is tight for me but the idea is to get this merged upstream and keep our fork consistent with upstream.
join_column should be in the fact_table.
Simple optimisation for listing members of a dimension - it won't join with the fact table if the dimension table is the only one referenced in the query.
The binding tuples should really become objects, I think.
Assumes referencing a column adds it to q.froms, which seems to be the case from experimentation.

I've tidied up a bit. I won't be force-pushing to this branch any more - now only additional commits.
Next step is adding tests.
Keen to hear feedback or explain this better if it isn't clear when you have a chance.

@jbothma this is great - I'll do a proper review in a couple of days and we'll merge this into upstream.
Thanks!

Hold on before merging this - I think I found a bug where the join isn't always restricted via the join column - will update when refuted or fixed.

Fixed a couple of bugs:

It wasn't joining on the fact table if one table was mentioned in one of the parser instances and one other table was mentioned in another instance
It wasn't joining on the fact table if it wasn't explicitly mentioned (perhaps by default of all fields) for the facts query while you'd expect cardinality from matching fact table rows even if only a non-fact-table field was requested


docker-compose up not working

I tried the latest docker support, by using 'docker compose up' , and got this exception: => ERROR [devika-backend-engine  8/20] RUN $HOME/.cargo/bin/uv venv

The docker-compose command represents the legacy version of the tool. Try without the '-' e.g docker compose up should work.


strip command output for multiple files incorrect in 8.2.0

As of v8.2.0, the --strip option no longer generates a valid strip(1) output:
strip build/Release/webgl.node,build/Release/obj.target/webgl.node --strip-all

which results in No such file.

Before (in v8.1.2), I believe the output was simply either one of:
# it may only be stripping the first file
strip build/Release/webgl.node --strip-all

or
# note the space
strip build/Release/webgl.node build/Release/obj.target/webgl.node --strip-all

Both of which ultimately run successfully. I'm can't confirm what the actual strip command output is, however.

I tested v8.1.2 and it works as expected.
This may be the result of the switch to recursive-readdir
https://github.com/prebuild/prebuild/blob/3b8401da1f0363156d4249d9dc7f12436a756aec/collect-artifacts.js#L3

Yes, it's the first case ("it may only be stripping the first file").

This may be the result of the switch to recursive-readdir

Correct, and apologies for the breaking change. Before 8.2.0, files in nested folders were not included. The default include regex is \.node$, which before only matched 1 file in your case, now 2.
There are 2 problems to solve here:

Should we not scan nested folders by default? Or tweak the default regex to only include top-level .node files? I'm not sure how to to best satisfy everyone's needs.
Fix stripping to correctly handle multiple files.

about 1. I think either solution is fine. I would personally go for the regex limiting to only top-level files.

Hello, I've opened PR #247 to temporarily revert these breaking changes.

@zaro We've reverted the recursive feature in v8.2.1. You're more than welcome to open a new PR - taking the above into account. Thanks!

Thought: what if we used a glob pattern rather than a regex? The default would be *.node and if you need nested files you could specify e.g. **/*.node.


Set CI tests on both architectures.

I checked build settings of repo on travis, and I see that CI runs test only on iPhone 6, so only 64bit architecture.
I think CI should test also 32 bit architecture (eg. iPhone 5).

:+1:


Lo table report is not displaying in textbook_distribution

In configuration file We included only lo_table  report and tried to process it.
Configuration file is successfully processed and data processing also successfully processed.
Issue : Ui report is not showing lo table

While processing of the configuration file, Found below error
ExecuteSQLRecord[id=94c0402e-7d16-3300-9fcf-ef66272d7a35] Unable to execute SQL select query select distinct report_name,report_type,description from configurable_datasource_properties
for StandardFlowFileRecord[uuid=25d0db75-8356-4853-8423-c53c76fd2f15,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1665669856060-15, container=default, section=15], offset=64500, length=25],offset=0,name=textbook_distribution.csv,size=25] due to ERROR: relation "configurable_datasource_properties" does not exist
Position: 58; routing to failure: org.postgresql.util.PSQLException: ERROR: relation "configurable_datasource_properties" does not exist
Position: 58

@JalajaTR have u completed this? the estimated time was yesterday?

@htvenkatesh It was raised yesterday night Am checking on it now

@JalajaTR this we discussed in 4.30 call and that time only we have estimated this timeline, not raised night

@devu1454 could you confirm

@devu1454 and @htvenkatesh The issue is fixed and the code is available in github
Assigning it to @devu1454 for retesting

Started the re-testing of the defect

Data processing is working as expected

Data processing is working as expected

Instead of showing Lo_table, Map is showing in the textbook_distribution

Instead of showing Lo_table, Map is showing in the textbook_distribution

Instead of showing Lo_table, Map is showing in the textbook_distribution

@devu1454 if its still a issue, please reassign this ticket to developer

@htvenkatesh This ticket is assigned to @FaizanMohammed326

@FaizanMohammed326 wts the update on this?

Fixed @htvenkatesh

Textbook_distribution tabs are working as expected

1. tags are displaying on the cells.

above defect is re-tested, working as expected


Introduce "Spinnaker-compatible" as an option for deployment

By adding a few details to the CF deployment phase, it's possible that apps deployed from a Concourse pipeline would be visible as proper "server groups" inside a Spinnaker installation monitoring the same space.
Env Variables

This screenshot shows full complement of variables, that when set, makes a server group visible to Spinnaker.

SPINNAKER_LOAD_BALANCERS is the name of the route from which the CF app would be visible, i.e. it's public URL (minus the domain). This is required.

However, just about everything else is optional, and provides extra information, but isn't needed for the app to be visible to Spinnaker (though if avaiable, let's populate!)
App name
The other thing needed to make the CF app visible to Spinnaker as a "server group" is the actual name of the application. There is a precise format that must be followed or it doesn't work.
While it might be old habit to name some demo pushed to CF spring-a-gram-demo-for-jug, Spinnaker has a library to enforce a format that looks like this => springagram-dev-demo-for-jug-v023.
The app name is split up by -'s, and tokenized as follows:

springagram - app name. Think of github repo where the code comes from. Or if you have a set of microservices that work together, this could be common element between all of them.
dev - stack. It's a logical label, often using things like "dev", "staging", "production", "gh123". This can also be how you distinguish a suite of microservices running in staging vs. the same stuff in production.
demo-for-jug - details. A free form where you can put anything, even dashes. When you have a set of microservices, it's great for describing each one, e.g. "circuit-breaker", "service-discovery", "email-service", "frontend", etc.
v023 - version. The last piece which denotes the exact version. Has to start with "v", and has to contain exactly three digits. Instead of "blue" and "green", they simply bump up the number and NEVER update an existing CF app, but instead push out a newer copy. For an out-of-the-box pipeline, we merely need to start with "v000".

NOTE: None of this dictates what org/space an app is deployed into.
Issues with this

This can introduce complexity. That's why I'm leaning toward making it an option. The extra metadata and formatting of the app can add confusion we don't want if the user doesn't intend to use Spinnaker.
If someone starts without Spinnaker-compatible, they can't expect to convert already deployed stuff into Spinnaker-compatible.
It also introduces a wrinkle if you intend to support blue/green deployments. Blue/green deployments that are Spinnaker-compatible would require a poor man's version of what Spinnaker does. When pushing, instead of finding the inactive side and updating it, instead find the latest, bump up the version by 1, and push things out, then move the route. Issues arise when you see potential roll over scenarios, e.g. v999 and v000 currently deployed.

Create on a branch separate jobs (no pipeline) for everything but the deployment. That will be taken by Spinnaker. @gregturn - should we add the env vars like presented in this issue?

I suggest we start with the Jenkins build job (no pipeline) and the pipelines themselves. When that is operational, we can steer toward linking the two together.

On hold.


Commands with tables

Added Help command (help is a valid subcommand)

can be invoked by main_class help
Help to subcommands would be invoked via main_class help subcommand eg: main_class help list (gives help about list command)

Added details subcommand
Added table to display results on cli

awesome! seems working very well! one last thing is to add some unit tests.

Unit test should be quite straightforward by changing the @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, classes = GeoweaverApplication.class) to @SpringBootTest(classes = GeoweaverCLI.class)

Unit tests specific to command line or Basetool and others?

Unit tests specific to command line or Basetool and others?

Unit tests for the newly added code. Some simple tests for list and detail function.


CU 99000787 "Create Prod. Order lines" Publisher "OnCheckMakeOrderLineBeforeInsert" add new Parameter InsertNew

is it possible to add the Variable InsertNew on the Publisher OnCheckMakeOrderLineBeforeInsert
OnCheckMakeOrderLineBeforeInsert(ProdOrderLine, ProdOrderComp, var InsertNew);

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


[PDI-13980] Google Analytics NPE with "Get Fields"

Prevent an NPE exception when "Get Fields" is used in the dialog, but no results are returned for the given query parameters

Bump

Code is merged in another PR


Uncaught ReferenceError: CKEDITOR is not defined(anonymous function)

Uncaught ReferenceError: CKEDITOR is not defined(anonymous function) @ ng-ckeditor.js:16invoke @ angular.js:4709(anonymous function) @ angular.js:4517forEach @ angular.js:321createInjector @ angular.js:4517doBootstrap @ angular.js:1751bootstrap @ angular.js:1772angularInit @ angular.js:1657(anonymous function) @ angular.js:31468i @ jquery.min.js:2fireWith @ jquery.min.js:2ready @ jquery.min.js:2J @ jquery.min.js:2

I'm trying to use it inside of a directive but for some reason I'm not able to find the CKEDITOR

Apologies, this was a duplicate.


New GPG key for @lafriks

Latest release is signed by a new GPG key for @lafriks, that was apparently generated today (and not the former one). Any reason why so?

Also I think the 1.7.2 tag + binaries were deleted and recreated, received two notifications

There was a PR that wasn’t backported to the 1.7.2 release and so @lafriks was able to sort that out. I can confirm this is ok. You can verify yourself that the commit signed is ok, and the binaries are signed by our official gpg key.

@ArchangeGabriel I created new GPG key for use from my work as I had no laptop with me, all is ok :)

OK, thanks to both of your for those precisions.


Built assets should be cache-busted

As a developer, I want my built assets file names to include version hashes and replace the asset references with these names, so I can cache-bust new deploys.
@josephabrahams we're going to skip on generating an asset manifest for now. Let me know if you think we need it.

Use https://github.com/sindresorhus/gulp-rev to add version hashes to all assets except *.html
Use https://github.com/jamesknelson/gulp-rev-replace to replace references to these assets with their cache-busted counterparts.

:+1:

Fixed by #7


docs(README): remove <a name... since it's done by default

This will cause the readme to render correctly on GitHub

Thanks 🍻

With pleasure!


add ivf_flat_cc index

Issue #823
Add IVF_FLAT_CC to support running build & search on IVF index concurrently by following change

Add IVF_FLAT_CC index inherited with IVF_FLAT
Replace the InvertedLists implementation from ArrayInvertedLists into ConcurrentArrayInvertedLists

/lgtm


Generation of 2 apk on building the integrated project

After the integration of unity exports/project in the React Native project, when we generate the apk using assembleRelease/assembleDeploy, it results in generation of 2 different APKs

react native unity integrated apk
APK which consists of only the unity project

Could you please suggest a solution to get rid of 2 APKs
How do we alter the configuration of the react-native-unity-2 package to cancel out of the generation of unity project APK

Generation of two APKs can be avoided if you build an AAR library from your Unity project beforehand, and linking the AAR library in your React Native project.

https://github.com/fusetools/react-native-unity2/blob/main/docs/unity-artifacts.md

The included example app demonstrates how to use the AAR library in React Native.


[FEATURE] Docker image

Are there any plans to release a Docker image?

You mean docker image for build? It's quite easy to get the build working as long as one has Node.js installed. Only requirement for build is Node.js. We can create a Build.md with docker setup.

I mean that it would be nice to have an official Docker image on Docker hub. It will help to use wildduck-ui in docker projects like https://github.com/nodemailer/wildduck-dockerized

This is UI repo there is no server aspect of this repository. We are not recommending what web server to use to serve static files. People often use CDN to host their static files or use nginx server to serve static files. At our internal usage we use s3 to store these static resources. This is one reason there is no Dockerfile for this repo.
We may consider providing Dockerfile for build but not for running.


Add support for 'tabix' formatted genomics/genetics data

Geneticists and genomics scientists cannot natively get data from S3 when it's compressed using the most common format for that data type
Tabular data in genetics and genomics is often compressed using the so-called "tabix" format. This is a block compressed, gzip-compatible compression that allows indexing into a file by genome position. While the file suffixes are .gz, and gunzip can be used to decompress them from local files (via pandas.read_csv with compression=gzip) AWS data wrangler cannot fetch these data from S3 (by awswrangler.s3.read_csv).
What I'd like to see
I'd like to be able to get data via awswrangler.s3.read_csv(S3_uri, compression="bgzip"). While guzip will decompress the file on local storage,bgzip -d is actually the preferred method. I believe that subtle differences between gzip and bgzip corrupt the reading of the data from S3.
Alternatives
Transferring a file to local storage (aws s3 cp <uri> ./) and then using the pandas read_csv function works, but involves an extra copy step.
It is likely that the genomics community's use of bgzip for tabix' files is idiosyncratic, but there is a large and growing number of users in this space.
Adding support for this compression method would support the genetics field and the biopharma industry.

Hi @MattBrauer can you help us replicate your scenario? You mention using pandas' read_csv method, but I get an error when using bgzip as the compression value.
ValueError: Unrecognized compression type: bgzip
Valid compression types are ['infer', None, 'bz2', 'gzip', 'xz', 'zip', 'zstd']

Hello. The bgzip format is compatible with gzip, so pandas' read_csv can be used successfully with gzip compression. I'd like it to be possible to do the same with awswrangler.

OK, so you are using pandas.read_csv(.., compression="gzip") after copying file to local today?

That is correct. That's a reasonable workaround, but it would be nice to have wr.s3.read_csv behave the same way, if possible.
Thanks for the attention on this. If necessary I can get you a file that demonstrates the problem, but since they contain restricted data I'd have to do some obfuscation work first.


Unable to mount volumes or bind mount on Windows

Issue:
When using nerdctl + containerd on Windows, I am unable to mount volumes or bind mount to a new windows container. I've tried using the following command with no success:
PS C:\Users\Administrator\nerdctl\_output> nerdctl.exe --debug-full run -d -v C:\bind_volumes\v1:c:\src hello-world:nanoserver-1809
time="2022-01-30T02:04:37-08:00" level=debug msg="verification process skipped"
time="2022-01-30T02:04:37-08:00" level=debug msg="Calling proc (1)" spanID=7599fa456674789d traceID=6d976231e769a37073cadb49b4a21362
time="2022-01-30T02:04:37-08:00" level=debug msg="Calling proc (2)" spanID=7599fa456674789d traceID=6d976231e769a37073cadb49b4a21362
time="2022-01-30T02:04:37-08:00" level=fatal msg="failed to parse \"C:\\\\bind_volumes\\\\v1:c:\\\\src\""

I've also tried experimenting with the the -v option, by trying various ways of single/double quoting an escaping special characters, but again I did not have any success.
Code Investigation:
Now, I do not know how to program in Go (so please excuse me if I'm totally wrong here). But starting at line 56 in mountutil.go, it looks like that whole switch statement is really only designed to handle Linux paths. I say this because on line 56, the value passed to -v is being split on the ":" character and the switch statement doesn't look like it would be able to handle bind mount (eg: -v c:\hostDir:c:\guestDir) or just mounting volumes to a windows container (eg: -v vol1:c:\guestDir).
Also, line 56's parent function (generateMountOpts), returns back up to run_mount.go on line 195. Just a few lines below that on line 203, it looks like the SecureJoin may encounter issues if a full windows path containing a colon character were to be joined with tempDir on that line.
That's what I've been able to observe to the best of my ability. If I'm overthinking or if I've made some very obvious mistake when trying to use the -v option, please let me know.
Thanks!
My Environment:

OS: Windows Server 2019
OS Version: 1809 (OS Build 17763.1282)
CPU arch: Intel x86-64
nerdctl version: v0.16.0
containerd version v1.5.9
Other information:

Is the Windows Container feature enabled: Yes

I would like to make a PR to fix this

I would like to make a PR to fix this

Thanks for taking a look!

Use instead: -v /bind_volumes/v1:/src
For me in Windows it works using Linux paths style.

I am unable to mount volumes or bind mount to a new windows container. I've tried using the following command with no success:

Hey @textminer, if I use this approach, do you have a recommendation on how to target different windows drive letters? For example, if I wanted to bind-mount "f:\foo\bar", how would I accomplish this using nerdctl?

@wtfacoconut There is a bug in containerd for different drive letters in Windows: https://github.com/containerd/containerd/issues/6589. Once that is fixed and this parsing error here is fixed, we would be able to support the other driver letters

@wtfacoconut There is a bug in containerd for different drive letters in Windows: https://github.com/containerd/containerd/issues/6589. Once that is fixed and this parsing error here is fixed, we would be able to support the other driver letters

Understood. Thanks for the info.

@wtfacoconut There is a bug in containerd for different drive letters in Windows: containerd/containerd#6589. Once that is fixed and this parsing error here is fixed, we would be able to support the other driver letters

Looks like containerd/containerd#6589 fixed by containerd v1.6.6 (PR https://github.com/containerd/containerd/pull/6651)

PS C:\> containerd.exe -v
containerd github.com/containerd/containerd v1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

PS C:\> nerdctl.exe -v
nerdctl version 0.21.0

Get "failed to parse" error message:
PS C:\> nerdctl.exe --debug-full run -it -v C:\tools\:c:\tools --rm --name test mcr.microsoft.com/windows/nanoserver:ltsc2022
time="2022-07-22T08:55:21Z" level=debug msg="verification process skipped"
time="2022-07-22T08:55:21Z" level=debug msg="Calling proc (1)" spanID=0fbc95fd368421cb traceID=751b7e5859dbdbba7510ddbf9c44f373
time="2022-07-22T08:55:21Z" level=debug msg="Calling proc (2)" spanID=0fbc95fd368421cb traceID=751b7e5859dbdbba7510ddbf9c44f373
time="2022-07-22T08:55:21Z" level=fatal msg="failed to parse \"C:\\\\tools\\\\:c:\\\\tools\""

@kkbruce  this won't work, there is a patch in progress at https://github.com/containerd/nerdctl/pull/924
work around is to use /path/ without the drive : https://github.com/containerd/nerdctl/issues/759#issuecomment-1036215481

@jsturtevant @textminer
I've been trying for a long time and can't come up with the alternatives you mentioned. Can you provide a complete run description of the -v or -mount parameters?
Local c:\\data folder:
PS C:\> dir .\data\

    Directory: C:\data

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         7/25/2022   3:36 PM              8 test.txt

Test mount function:
PS D:\> nerdctl.exe run -it --rm -v "/data/:c:\Windows\Temp" mcr.microsoft.com/windows/servercore:ltsc2022
time="2022-07-25T15:49:09+08:00" level=warning msg="expected an absolute path, got a relative path \"/data/\" (allowed for nerdctl, but disallowed for Docker, so unrecommended)"
time="2022-07-25T15:49:09+08:00" level=fatal msg="expected an absolute path, got \"c\""

PS D:\> nerdctl.exe run -it --rm -v "/data/:/Windows/Temp" mcr.microsoft.com/windows/servercore:ltsc2022
time="2022-07-25T15:49:23+08:00" level=warning msg="expected an absolute path, got a relative path \"/data/\" (allowed for nerdctl, but disallowed for Docker, so unrecommended)"
time="2022-07-25T15:49:23+08:00" level=fatal msg="expected an absolute path, got \"/Windows/Temp\""

I am running nerdctl from gitbash on Windows machine and this command is working for me:
nerdctl run -v c\:/folder/anotherfolder:/build/somefolder <image_id> ls build/somefolder

Any progress on this? I've tried a lot of combinations of path specification. Nothing works.

Same problem there. I tried in all possible ways but I have to deal with Docker CE yet. Any ETA regarding this issue, or which version will have a fix available?
Thanks!

Any updates ? Thanks

@AkihiroSuda this is apparently fixed by #2661 so I guess we should close this.


Node.js SDK - Launch multiple tests in parallel for image provision

The test is currently skipped. It pass locally but failed with ./hack/make sdk:nodejs:test
Signed-off-by: jffarge<EMAIL_ADDRESS>

This PR is out of date. The provisioning approach has changed.
More context here : #4194


Gh 2630 example real federatedstore

Related Issue

Resolve #2630

Adding the missed new lines request.

Codecov Report

:exclamation: No coverage uploaded for pull request base (v2-alpha@20413bf). Click here to learn what that means.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             v2-alpha    #2790   +/-   ##
===========================================
  Coverage            ?   64.98%
  Complexity          ?      259
===========================================
  Files               ?       59
  Lines               ?      774
  Branches            ?       62
===========================================
  Hits                ?      503
  Misses              ?      237
  Partials            ?       34

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


GPU support Table & VRAM usage

It would be great to get the instructions to run the 3B model locally on a gaming GPU (e.g. 3090/4090 with 24GB VRAM).
GPU Support Table - confirmed working

GPU Model
VRAM (GB)
Tuned-3b
Tuned-7b

GTX 1080 Ti
11

RTX 2080 Ti
11

RTX 3060
12

RTX 3080
10

RTX 3080 Ti
12

RTX 3090
24
✅
✅

RTX 4070 Ti
12
✅

RTX 4080
16

RTX 4090
24

✅

A100
40

✅

Update: VRAM usage analysis
Weights

model name
parameters
W (fp32)
W (fp16)
weights (VRAM)
load time (s)
works

stablelm-tuned-alpha-3b
3637321728
13.55
6.78
7.03
18.62
✅

stablelm-tuned-alpha-7b
7868755968
29.31
14.66
14.91
50.28
✅

weights (fp32, GB): that's the minimum required RAM to load the model (before calling .half())
weights (fp16, GB): that's the minimum VRAM when transferring the model to the GPU
weights (fp16, VRAM): reported VRAM increase after loading the model

Activations
Empyrical (numbers in bytes, fp32):

stablelm-tuned-alpha-3b: total_tokens * 1,280,582
stablelm-tuned-alpha-7b: total_tokens * 1,869,134

The regression fits at 0.99999989. For instance, with 32 input tokens and an output of 512, the activations are: 969 MB of VAM (almost 1 GB) will be required. Haven't tested with Batch not equal 1.
Examples of a few recorded activations numbers:

model
input_tokens
out_tokens
total_tokens
VRAM (MB)

3b
3072
1024
4096
5003

3b
1024
512
1536
1875

3b
64
1
65
78.19

3b
8
1
9
9.77

7b
3072
1024
4096
7304.22

7b
2048
512
2560
4564.47

7b
8
64
72
126.64

7b
8
1
9
14.27

3B runs on 2080ti

Does it run on AMD? I can try AMD RX 480 😎

Tesla P40 (24GB)  - works

Best bet is 4bit quantization. 7B will likely run in 6gigs of VRAM at that level, as that's about the requirement for 7b with LLaMa.

Got 7B models working on my Tesla M40 w/ 24GB ram

Got the 7B running fine on my 4090.

Keep eyes on this issues

Not a gaming PC, but I just tried the Colab notebook with 83.5GB of System RAM and A100 with 40GB.
It's insanely fast to initialize and the prompts on the tuned-alpha-7B model took around 2 seconds to complete.

# 3B f16
>>> (3_638_525_952 * 2) / 1024 / 1024 / 1024
6.77728271484375
# 3B f32
>>> (3_638_525_952 * 4) / 1024 / 1024 / 1024
13.5545654296875
# 7B f16
>>> (7_869_358_080 * 2) / 1024 / 1024 / 1024
14.657821655273438
# 7B f32
>>> (7_869_358_080 * 4) / 1024 / 1024 / 1024
29.315643310546875

This is only the size of the weights tho, right?
Does anyone know what's the ratio of actvations/weights in this sort of network?

Able to run the tuned-alpha-3b on a 4070 Ti (12GB)

For the sake of convenience (2x less download size/RAM/VRAM), I've uploaded 16-bit versions of tuned models to HF Hub:
https://huggingface.co/vvsotnikov/stablelm-tuned-alpha-7b-16bit
https://huggingface.co/vvsotnikov/stablelm-tuned-alpha-3b-16bit

For the sake of convenience (2x less download size/RAM/VRAM), I've uploaded 16-bit versions of tuned models to HF Hub: https://huggingface.co/vvsotnikov/stablelm-tuned-alpha-7b-16bit https://huggingface.co/vvsotnikov/stablelm-tuned-alpha-3b-16bit

Would you mind showing how you made the conversion? I'm new to this and would like to do the same for the base 7B model. Thanks.

@cduk it's pretty much straightforward:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList

tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-tuned-alpha-3b")
model = AutoModelForCausalLM.from_pretrained("stabilityai/stablelm-tuned-alpha-3b")
model.half().cuda()
model.save_pretrained('vvsotnikov/stablelm-tuned-alpha-3b-16bit')
tokenizer.save_pretrained('vvsotnikov/stablelm-tuned-alpha-3b-16bit')

It will save the model and the tokenizer locally, then you will have to upload them to Hub. Good luck!

Nvidia T4 (16 GB) runs out of memory when trying to load the fp16 7B model. The 3B model runs smoothly in fp16.

I only have 40GB of RAM. So the default code did not work for me for 7B.
By changing the first lines to this, RAM is limited to 17GB and the model loads in 9:50 min.
tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
model = AutoModelForCausalLM.from_pretrained(
    "StabilityAI/stablelm-base-alpha-7b",
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)
model = model.to("cuda")

You need to pip install accelerate as well. So make those changes to avoid loading a 32 bit version of the model (34 gb), then the weights separately (another 34 gb). You still download 2x the size, unlike with @vvsotnikov 's images, but my VM has gigabit so I don't mind.
However, then it crashes, because I have a T4 which only has 15.3GB with the following :/:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 592.00 MiB (GPU 0; 14.62 GiB total capacity; 14.33 GiB already allocated; 185.38 MiB free; 14.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I'm confident that with some messing around it will fit on the T4. It's so close!
For now I will try running with device_map="auto" and report back.
https://huggingface.co/docs/transformers/v4.28.1/en/model_doc/flan-ul2#running-on-low-resource-devices

@antheas So close! Have you considered quantizing to 8-bit and seeing how well that works?

So I came up with the following, to use 8 bit quantization:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, StoppingCriteria, StoppingCriteriaList

quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,
)

tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
model = AutoModelForCausalLM.from_pretrained(
    "StabilityAI/stablelm-tuned-alpha-7b",
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    load_in_8bit=True,
    quantization_config=quantization_config,
    device_map={'': 0}
)

Loading takes around 13GB of RAM peak. I record the VRAM before running a prompt.
Then I run the default script prompt and record the time and VRAM. Then I keep rerunning the script prompt and I record the run times. These results are on a T4. I will play with llm_int8_threshold to see if there are other savings possible. Again, no direct SSD storage = slow load times.
Running with 10.0 now, will update this table with results as they become available.

Threshhold
Initial VRAM
After first Prompt
Loading
First Prompt
Next ones

6
9618MiB
10031 MiB
10m35s
43.8s
3-6s

However, with 4 prompts run a bit faster (?)

Isn't this expected? The lower the threshold, the more weights are converted to int8 (hence less compute to do).

However, with 4 prompts run a bit faster (?)

Isn't this expected? The lower the threshold, the more weights are converted to int8 (hence less compute to do).

The way I read it it's the opposite. According to its description, values follow a normal distribution, with most being less than [-3.5, 3.5]. llm_int8_threshold is the bound at which if a value is lower, it is converted to int8. This is because higher values are associated with outliers and can destabilize the model.
Might be wrong though.
Built myself a little chat bot with ipywidget. I'm playing a bit with the model now, it's quite fun.
By adding streamer=TextStreamer(tokenizer=tokenizer, skip_prompt=True) to model.generate(), responses are streamed.

I was able to get 3B parameter to work on CPU with 16GB of ram.

I was able to get 3B parameter to work on CPU with 16GB of ram.

Did you use any tricks such as the dtyle or similar?

I had to disable torch.backends.cudnn and convert to float.
check out my repo https://github.com/astrobleem/Simple-StableLM-Chat
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)
    model.float().to(device)
    torch.backends.cudnn.enabled = False

I am using Radeon 6900xt (16GB VRAM) and quick start code on README works well! (Using stabilityai/stablelm-tuned-alpha-7b)
I used rocm/pytorch docker with rocm5.4.2_ubuntu20.04_py3.8_pytorch_2.0.0_preview version.
https://hub.docker.com/r/rocm/pytorch

Works on P6000 24gb.. up to 3000 context before it OOM.

I can confirm Tuned-7B works on my A6000 Ada / 48Gb GPU :)

Have any of you run into this error when you have the model running?

RTX 3080 Ti Tuned-3B ✅
RTX 3080 Ti Tuned-7B 🚫  (CUDA OOM)

RTX 3070 (8GB VRAM) Tuned-3B (fp16) ✅
RTX 3070 (8GB VRAM) Tuned-3B (fp32) 🚫
RTX 3070 (8GB VRAM) Tuned-7B (fp16) 🚫
RTX 3070 (8GB VRAM) Tuned-7B (fp32) 🚫

RTX 3060 (12GB VRAM) Tuned-3B (fp16) white_check_mark

stablelm-tuned-alpha-3b (fp16) works on a Tesla K80. I load it on GPU2 because it runs cooler.

7900XTX 24GB is OK with tuned-7B, based on docker-based ROCm5.5-rc5 and PyTorch2.0

Have any of you run into this error when you have the model running? I've attempted method where the model is quantized to an 8-bit version but it seems to cause this problem with the probability tensor/tokens.

For those of you who are using the 8-bit version of StableLM how did you get the ChatBot up and running?

I had the same error, I discovered checking around that there is a parameter that you can add to the generate() function called remove_invalid_values, if you put it in True it should work :) I leave here the parameters that I put:
tokens = model.generate(
**inputs, max_new_tokens=64,
temperature=0.7, do_sample=True,
stopping_criteria=StoppingCriteriaList([StopOnTokens()]),
remove_invalid_values=True
)
PD:
RTX 3080 (12GB VRAM) Tuned-7B (fp16) OK


[Searchable Snapshots] Add integration test to validate cache files are closed after use

Description

Add an integration test for Searchable Snapshots to validate all cache files are closed / no resource leak after an index which backed by remote snapshot is deleted.

The cache directory is supposed to exist in the parent directory for the shard data of the index (https://github.com/opensearch-project/OpenSearch/blob/2.5.0/server/src/internalClusterTest/java/org/opensearch/snapshots/SearchableSnapshotIT.java#L367-L370)
final Path file = PathUtils.get(path)
    .resolve("indices")
    .resolve(index.getUUID())
    .resolve(Integer.toString(shardRouting.getId()))
    .resolve("LocalCache");

Based on that, I iterator all the files in the parent directory to validate all the files are closed.
I used Files.move() to check whether the file is closed / locked / in-use or not, if the file can be moved, then it proves the file has been closed properly. (reference: https://stackoverflow.com/questions/1500174/check-if-a-file-is-locked-in-java/13706972#13706972)
The steps for the whole test are based on the idea from the original issue https://github.com/opensearch-project/OpenSearch/issues/5244

creating remote backed index, do some searches, and then delete the index. And monitor the file descriptor held by process affecting the caching directory.

Issues Resolved
https://github.com/opensearch-project/OpenSearch/issues/5244
Check List

[ ] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #6215 (7e53d6e) into main (1f4cdd2) will increase coverage by 0.03%.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #6215      +/-   ##
============================================
+ Coverage     70.72%   70.75%   +0.03%
- Complexity    58791    58887      +96
============================================
  Files          4782     4782
  Lines        281484   281484
  Branches      40642    40642
============================================
+ Hits         199072   199158      +86
+ Misses        65984    65979       -5
+ Partials      16428    16347      -81

Impacted Files
Coverage Δ

...java/org/opensearch/client/indices/DataStream.java
0.00% <0.00%> (-76.09%)
:arrow_down:

...n/indices/forcemerge/ForceMergeRequestBuilder.java
0.00% <0.00%> (-75.00%)
:arrow_down:

.../opensearch/client/indices/CloseIndexResponse.java
17.50% <0.00%> (-73.75%)
:arrow_down:

.../indices/forcemerge/TransportForceMergeAction.java
25.00% <0.00%> (-58.34%)
:arrow_down:

...a/org/opensearch/client/cluster/ProxyModeInfo.java
0.00% <0.00%> (-55.00%)
:arrow_down:

...cluster/coordination/PublishClusterStateStats.java
33.33% <0.00%> (-37.51%)
:arrow_down:

...ndex/query/functionscore/DecayFunctionBuilder.java
33.45% <0.00%> (-32.37%)
:arrow_down:

.../java/org/opensearch/index/reindex/RemoteInfo.java
50.45% <0.00%> (-29.36%)
:arrow_down:

...opensearch/snapshots/SnapshotMissingException.java
28.57% <0.00%> (-28.58%)
:arrow_down:

...rc/main/java/org/opensearch/ingest/IngestInfo.java
51.72% <0.00%> (-27.59%)
:arrow_down:

... and 484 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


fix: FunctionValueExpressionParser should support NodeListFunction

Motivation:
Make FunctionValueExpressionParser support NodelistFunctionDefinition too

No, a nodelist is not a value.  This parser handles only functions which return values.

But it's been evaluated

Add a test that fails without your change, please.  I don't think this is necessary.

Let me check the noop first.

I try to run parse $[?count(noop(@.*))>0)] but it fails,

Okay.  Good.  Please create a test, show that it fails without your changes and then apply your changes to show that they fix the problem.

Closing in favor of #835


💡[FEATURE]: Competition to Find the Best Font for the Website

:fire:  Is your feature request related to a problem? Please describe.
Organizing a competition to find the best font that enhances the readability and aesthetic appeal of our website.
:ballot_box_with_check:  Tasks:

[ ] Integrate the winning font into the website.
[ ] Ensure consistency across all web pages.
[ ] Test the website with the new font on various devices and browsers.

:memo:  Attachments:

[ ] Include any relevant mockups or design guidelines.

Fonts used - Hedvig & Cabin
NavBar + Hero:

Services:

Love:

If this looks good, I can add these fonts to the entire website and do a PR.

Nunito Font

Used for Title


unable to download all receipts - some months get skipped

Issue:
No receipt for online transactions
Working Transaction ID:<PHONE_NUMBER> (In-store)
Not Working Transaction ID:<PHONE_NUMBER> (Online)
Output Reference:
In-store transaction works fine:
Item { id: "1732416902", receipt: Some(...) }
Online transaction gets skipped due to no receipt:
Item { id: "1738356113", receipt: None }
Skipping<PHONE_NUMBER> as there is no associated receipt.
JSON for Working Transaction (In-store):
 {'id': '1732416902', 'displayDate': 'Fri 23 Feb', 'description': '$37.49 at Neutral Bay', 'message': '', 'displayValue': '+ 38 pts', 'displayValueHandling': 'NORMAL', 'icon': 'woolworths', 'iconUrl': 'https://prod.mobile-api.woolworths.com.au/zeus/mnemosyne/v1/public/activity/supermarkets_division_logo.png', 'transaction': {'origin': 'Neutral Bay', 'amountAsDollars': '$37.49'}, 'highlights': [], 'receipt': {'receiptId': 'U2FsdGVkX1+UnHRzqGXiA963dcd+pflUSI4kgLphDirnVa1EOINXPj1zeXR/u4crP/8CkhBbOXE3TBrZ+7BMj/7igzJ+24jsRa5Me+Fx3Bw=', 'receiptSource': 'INSTORE'}, 'transactionType': 'purchase', 'actionURL': 'com.woolworths.rewards://activityDetails?id=eyJhY3Rpdml0eUlkIjoiMTczMjQxNjkwMiIsInJlY2VpcHRJZCI6IlUyRnNkR1ZrWDErVW5IUnpxR1hpQTk2M2RjZCtwZmxVU0k0a2dMcGhEaXJuVmExRU9JTlhQajF6ZVhSL3U0Y3JQLzhDa2hCYk9YRTNUQnJaKzdCTWovN2lnekorMjRqc1JhNU1lK0Z4M0J3PSIsInJlY2VpcHRTb3VyY2UiOiJJTlNUT1JFIn0=', 'showChevron': True}

JSON for failing Transaction (Online):
{'id': '1738356113', 'displayDate': 'Mon 26 Feb', 'description': '$138.51 online at Woolworths Supermarket', 'message': "Here's $10 Everyday Rewards Dollars", 'displayValue': '+ 336 pts', 'displayValueHandling': 'NORMAL', 'icon': 'woolworths', 'iconUrl': 'https://prod.mobile-api.woolworths.com.au/zeus/mnemosyne/v1/public/activity/supermarkets_division_logo.png', 'transaction': {'origin': 'Woolworths Supermarket', 'amountAsDollars': '$140.51'}, 'highlights': [{'iconUrl': 'https://prod.mobile-api.woolworths.com.au/zeus/mnemosyne/v1/public/activity/activity_list_piggybank.png', 'description': "Here's $10 Everyday Rewards Dollars", 'value': None, 'style': 'HIGHLIGHT'}, {'iconUrl': 'https://prod.mobile-api.woolworths.com.au/zeus/mnemosyne/v1/public/activity/activity_list_booster.png', 'description': '198 Bonus points collected', 'value': None, 'style': 'STANDARD'}], 'receipt': {'receiptId': 'U2FsdGVkX1+4EvL26v4unJN+NUs2VswNOPM7k/iL84lyviokW1TKyV2cD4HocL2WRAlufbTDzU5Mr7RsqVJY6l5kehxac5EiEdyYiXeKQ4g=', 'receiptSource': 'ONLINE'}, 'transactionType': 'purchase', 'actionURL': 'com.woolworths.rewards://activityDetails?id=eyJhY3Rpdml0eUlkIjoiMTczODM1NjExMyIsInJlY2VpcHRJZCI6IlUyRnNkR1ZrWDErNEV2TDI2djR1bkpOK05VczJWc3dOT1BNN2svaUw4NGx5dmlva1cxVEt5VjJjRDRIb2NMMldSQWx1ZmJURHpVNU1yN1JzcVZKWTZsNWtlaHhhYzVFaUVkeVlpWGVLUTRnPSIsInJlY2VpcHRTb3VyY2UiOiJPTkxJTkUifQ==', 'showChevron': True}

Major Differences:

receiptSource: The working transaction is INSTORE, whereas the problematic one is ONLINE.
receiptId: Both have valid receiptId values, but the application fails to process the online transaction's receipt.

Hope this helps pinpoint the problem. Let me know if you need more info!

I updated the graphql query to the latest version used on the official website, which now has receipt(enableOnlineReceipt: true) as the receipt selector. Hopefully this should fix your issue.
Unfortunately I have never done an online order so I cannot test this myself.

Thanks for looking into it
i am getting the below fatal crash with the update
./target/release/everyday-rewards-receipts --token EmgxfglQyDPwVT2ifuw1ktOpqX73
optOutOfPaperBanner
This_Month
    Item { id: "1763045859", receipt: Some(ItemReceipt { receipt_id: "U2FsdGVkX1/+aAvSp4wfW2G17v/6jYvzUiA1v1KmfHKg3dily0aThed65yflFx6Ku5AzaiMPtKE4w1UYvrGMyEGtXlKxAUyKUjIrR+EG7F0=" }) }
Error: UnknownError("Unknown response: {\"status\":500,\"code\":\"16007\",\"message\":\"Unable to retrieve receipt details\"}")

Finally found some time to get back around to this. I updated the main function to not panic anymore and simply print out errors as a hack solution.
When it comes to some items not having receipts, I noticed when downloading mine that there are some entries in December 2023 which are duplicated, one entry with the receipt information and one without. I have created an issue for this #9


unable to run the program

seeing this error
Traceback (most recent call last):
File "main.py", line 2, in
from credcheck.core.cred_check_utils import CredUtils
File "/opt/credcheck/credcheck/core/init.py", line 1, in
from .cred_check_active import DynamicTest
File "/opt/credcheck/credcheck/core/cred_check_active.py", line 3, in
from credcheck.core.cred_check_utils import CredUtils
File "/opt/credcheck/credcheck/core/cred_check_utils.py", line 81
raise ParametersRequiredError(_helper.get("help"), _helper) from None
^
SyntaxError: invalid syntax

are you running it using python 2.7.* ?

Run it using 3.6+, it should work. @ankushgoel27


Server side pagination

How server side pagination will work with react-bootstrap-table? As I see bootstrap table supports server side pagination using sidePagination but react-bootstrap-table is not supporting this.
Below are the link for supporting server side pagination on bootstrap table
http://bootstrap-table.wenzhixin.net.cn/documentation/
https://github.com/wenzhixin/bootstrap-table-examples/blob/master/options/server-side-pagination.html#L22

@smondal, this component does not support server side pagination and remote data fetching. I think this feature will not be implemented in near future. But PR is welcome

@AllenFang,
can just add a prop 'dataTotalSize' which is total size of data from remote server in BootstrapTable component?
then component can use the 'dataTotalSize' not the size of 'data' prop.

HI @yybear, what's your purpose, or any trouble you encountered?

Hi @AllenFang
now react-bootstrap-table is not support for server side pagination, i think it can add  a prop 'dataTotalSize' which is the total size of data got from server side.
<BootstrapTable dataTotalSize={this.state.total} data={this.state.items} selectRow={this.selectRowProp} pagination>
the reponse from server side is not only returning data items but also the total size of data. for example:
{'total':100, "items":[item1, item2, item3]}
now u must use the size of 'data' prop to paginate in client side. if add the 'dataTotalSize' prop, u can use it to paginate in client side.  when page changed we can use the of 'sizePerPageListChange' or 'onPageChange' to get the new data items from server side.
i think it's a way to support server side pagination for react-bootstrap-table.

@yybear, react-bootstrap-table has been supported remote pagination from v2.3.0, but not yet update document, you can check this example. Hope this feature can help you :)

@AllenFang good job! i will try it, thks.

Hello @AllenFang, thanks for your great package on React, I'm using React Bootstrap Table and I have small problem. When using pagination, how can I show all page depend on totalSizeData without get all data. It's because if I have about more thousands records and I have to get all for the first request, it will be take a long time, could you please help me?

@VuongTran21 Please refer this comment above. This is a similar issue.
Thanks.

Thanks @prajapati-parth , I'm fresher with React and I really don't understand exactly how to apply it to pagination in server side, could you please help me?

@VuongTran21 With all humbleness, it would be really great for this question to be asked on StackOverflow. The issues tab of the repo is more focused to solve repo specific bugs and request for new features.
Thanks.

For everybody who will face this problem to pass total datasize as number, just go here https://github.com/AllenFang/react-bootstrap-table/blob/master/examples/js/remote/remote-paging.js .
You will find something like this :
fetchInfo={ { dataTotalSize: this.props.totalDataSize } }


Proposal: XamlApplication: Application implementing IXamlMetadataPorvider interface

Proposal: XamlApplication: Application implementing IXamlMetadataPorvider interface

Summary

XamlApplication is a Application implementing IXamlMetadataProvider interface and loading WinUI theme. This API provides an easy way to create an application integrating XAML.
runtimeclass XamlApplication : Microsoft.UI.Xaml.Application, Microsoft.UI.Xaml.Markup.IXamlMetadataProvider

Rationale

WinUI 3 apps need Application implementing IXamlMetadataProvider.
This API makes a code simpler.

Scope

Capability
Priority

This proposal will allow developers to make a code simpler
Could

Important Notes

This API was proposed by @Scottj1s in Meet C++/WinRT 2.0: Faster and smarter in the open - BRK4009. A similar API XamlApplication for XAML Islands is provided by Windows Community Toolkit.
The following code would be simpler using XamlApplication;
// Application and IXamlMetadataProvider
struct App : public ApplicationT<App, IXamlMetadataProvider>
{
	void OnLaunched(LaunchActivatedEventArgs const&)
	{
		Resources().MergedDictionaries().Append(XamlControlsResources());
		...
	}
	IXamlType GetXamlType(TypeName const& type)
	{
		return provider.GetXamlType(type);
	}
	IXamlType GetXamlType(hstring const& fullname)
	{
		return provider.GetXamlType(fullname);
	}
	com_array<XmlnsDefinition> GetXmlnsDefinitions()
	{
		return provider.GetXmlnsDefinitions();
	}
private:
	...
	XamlControlsXamlMetaDataProvider provider;
};

// XamlApplication
struct App : public XamlApplicationT<App>
{
	void OnLaunched(LaunchActivatedEventArgs const&)
	{
		...
	}
};

As it is, the application does implement IXamlMetadataProvider. This is defined in the generated C++ file for the app class. Furthermore, as I understand it, IXamlMetadataProvider must be able to return full type information, including callbacks to get or set properties by name, for every type used in XAML in your application. If it doesn’t, those types will not be usable from XAML as then WinUI will have no idea how to configure the object used to match the values in the XAML. If you do as you describe, then you may end up being unable to use any types defined in your project from XAML, including code-behinds; the only types you could successfully use are the ones supported by XamlControlsXamlMetaDataProvider. Of course there is no documentation for any of this.
IXamlMetadataProvider and the objects it returns are how the C++-based WinUI 3 core library is able to manipulate objects implemented in any programming language. (C++ types call through to other C++ code, C# types go through a CCW and are handled in managed code, and so on.) You could use these APIs to create your own type metadata system, which unfortunately is intended to be used only when you are porting a new language to WinRT. It is assumed that C# and C++ developers would always use the built-in system, and that the built-in system would be sufficient for all cases.
In short, your proposed change would actually cause widespread breakage (as far as I can tell). Sorry about that!

Thank you for providing a lot of information. Actually, I want to create WinUI 3 apps without XAML and use IXamlMetadataProvider only for instantiating XamlControlsResources. If we can instantiate XamlControlsResources without XAML integration, it would be nice.

While I understand what you are going for here, I think the better option would be for Microsoft to document how it works and provide a sample.
Interestingly, the Xaml Islands application that you referenced is close to what you would need for this to work normally. Here is a repository that shows this at work. This should be extensible to a certain extent, where if you have a custom control that also provides a metadata provider then these controls can also be used.

Thank you for providing the implementation. I updated the issue info with your code.

I updated the sample code with Kenny Kerr's implementation.


Failure to deal with tableMap events

It looks like CacheableTableMapBehaviour doesn't properly deal with tableMap events.
We have a table Foo that was mapped to id 71 for a while. We then had a failover, and the next binary log file has it mapped to 81. The table-map events are correct in the bin-log, and are being processed by the trait, but the tableCache doesn't register the new id->name and continues to return Foo for id=71
I had to work around in my code by adding:
... extends AbstractMySQLBinaryLogConsumer {
    var tableCacheBinLogPosition: Option[String] = None
    override protected def handleEvent(binaryLogEvent: MEvent): Unit = {
      if (Some(getBinaryLogPosition.get.filename) != tableCacheBinLogPosition) {
        logger.info("resetting table cache")
        this.tableCache = new TableCache(hostname, port, username, password)
      }
      super.handleEvent(binaryLogEvent)
      tableCacheBinLogPosition = Some(getBinaryLogPosition.get.filename)
    }
}

but it would be nice to have that logic go into the CacheableTableMapBehaviour behaviour

It seems like adding a layer of inderiction to TableCache where we go from (db,table_name) -> table and table_id->(db,table_name) would fix this pretty cleanly


IE9/10/11, Windows 7: Line break on the Gray 7 palette

On the Content page, the 7th and darkest grey (#757575) wraps down to the next row. The expected behavior is having that palette line up with the rest of the grey palettes.
Current:

Expected:

This affects all browsers in some way. Noticed on Chrome at sub-976px this is pushing elements to the next line.

@briansaycocie I think this is better than the alternative - http://cl.ly/342b2r0A030A

That works for me. Closing out.