Failed to compile - since v0.5

Hi and thanks for the module,
I updated to v0.5, but now I get :
Failed to compile.

./node_modules/react-popover/index.js
Module not found: Can't resolve './build/main' in '<my-project-path>/node_modules/react-popover'

I'm using create-react-app. To reproduce just create-react-app . && npm install react-popover then add a <Popover/> in src/App.js, then npm start.
I am missing something ?
Cheers

There is a PR for this already: #145
I hope it gets merged quickly too

0.5.2 is available now and should fix this and a few others issues.


Remove credential file from spec

Resolve #206

Do we need to create another issue to update the CLI baseline accordingly?

Do we need to create another issue to update the CLI baseline accordingly?

Yeah, I can make the code change to resolve it. Do we also need to update the cli spec?


made some changes so that we can create a parent/index sitemap

Accidentally created a PR, will reopen it later when its ready


Evaluate Specification for one Entity

It would be nice to use a specification to make sure an object is still valid.
I've given an example of why you'd want to share this logic in one place.
@fiseni proposed the following:

I was thinking of this, and we might introduce a new type of specification, or perhaps just a specific builder for this purpose. Conceptually, all we need is the entity as a parameter, and bool as a result, right? So, we might hold Func<T, bool> delegates.
We can have something as follows:
public class ValidVerificationCodeSpec : Specification<VerificationCode>
{
    public ValidVerificationCodeSpec()
    {
        Validators.Add(entity =>
        {
            if (entity.DateUsed == null && entity.DateCreated.AddMinutes(30) > DateTime.Now) return true;

            return false;
        }
    }
}

And the specification might expose IsValid method as follows

public bool IsValid(T entity)
{
    bool result = true;
    foreach (var validator in specification.validators)
    {
        result = result && validator(entity);
    }

    return result;
}

This is still not refined, but it might go along with this idea.

Let's say we add a Validator builder. This should be translatable to a query on the database right?
Or do you guys see it so they live together next to each other as following?
public class ValidVerificationCodeSpec : Specification<VerificationCode>
{
    public ValidVerificationCodeSpec()
    {
        Query
            .Where(vc => vc.DateUsed == null)
            .Where(vc => vc.DateCreated.AddMinutes(30) > DateTime.Now);

        Validators.Add(entity =>
        {
            if (entity.DateUsed == null && entity.DateCreated.AddMinutes(30) > DateTime.Now) return true;

            return false;
        }
    }
}

That will result in some duplication of the logic.

Let's start by clarifying few concepts and give some context here.
Specifications, as the name suggests, are just constructs that hold some information for a given purpose. The specifications by themselves don't bother how this information will be utilized, but the main focus is just to structure the "knowledge" it contains.
Having said that, we can identify two types of specifications in our case

Query specification - It holds information for querying a set/collection of data. The constructed query can be executed against some database, or against some in-memory collection, nevertheless. The important point is that these specifications have to do with some sort of filtering operation.
Validation specification - This specification holds information for a particular business rule. By leveraging this information we try to validate a given entity or any construct against those rules. So, this type of specification has to do with validation and should operate against in-memory data that you might have. Its job is not to retrieve data, but to ensure the correctness of the data you already have.

Once we categorize it this way, then we should keep these concerns a bit separate, and allow them to evolve independently. I'd rather have duplicated logic (anyhow, you can keep both in the same spec) than to be constrained to the "cross-section" of these concerns. For an instance, in your example, what if tomorrow you want to add some additional condition for the validation logic, which has nothing to do with the query itself?
Initially, I see two options here.
Option 1
We can simply have a new interface, let's say IValidationSpecification, which defines bool Validate() method. The entity to be validated along with any other inputs that the validation logic might require will be defined as constructor parameters
public interface IValidationSpecification
{
    public bool Validate();
}

public class CompanySpecification : Specification<Company>, IValidationSpecification
{
    private readonly Company company;
    private readonly int someOtherParameter;

    // This will be used for query information
    public CompanySpecification()
    {
        Query.Include(x => x.Stores);
    }

    // This will be used for validation logic
    public CompanySpecification(Company company, int someOtherParameter)
    {
        this.company = company;
        this.someOtherParameter = someOtherParameter;
    }

    public bool Validate()
    {
        // Do some validation logic here against company and someOtherParameter fields.
        return true;
    }
}

The reason I don't like this approach is that it keeps the state in a very verbose way. The specification no longer feels simple, but it gives an impression of some complex construct with states and stuff. The other reason is that this can lead to improper usages. In this case, you can create an instance with the parameterless constructor, and then try to use the Validate method. Nothing prevents you from doing that, right? We can define the entity input as part of the method, like bool Validate(T entity), but still you might require additional conditions, and you'll need a constructor for that.
Options 2
We keep validation logic as a delegate Func<T, bool> Validate. I'm not sure if we need a list, or maybe just one delegate is enough.
public class ValidVerificationCodeSpec : Specification<VerificationCode>
{
    public ValidVerificationCodeSpec()
    {
        Query
            .Where(vc => vc.DateUsed == null)
            .Where(vc => vc.DateCreated.AddMinutes(30) > DateTime.Now);
    }

    public ValidVerificationCodeSpec(int minutes)
    {
        Validator(entity =>
        {
            if (entity.DateUsed == null && entity.DateCreated.AddMinutes(minutes) > DateTime.Now) return true;

            return false;
        };
    }
}

The base specification will expose IsValid(T entity) method as follows
public bool IsValid(T entity)
{
	if (Validate== null) return false;

	return Validate(entity);
}

If we have not defined a validator, it should always return false or throw an exception. So, in case we insatiate it with the parameterless ctor ValidVerificationCodeSpec, it will never return true. Maybe throwing an exception is a better way.
Note I still don't like either of these approaches. It feels clumsy somehow :)

As for your original idea of utilizing the Evaluate feature for this purpose, you can easily do that even today.
All you need is to create your own base specification as follows. Then, you use it as a base for all your specs in the application.
public class AppSpecification<T> : Specification<T>
{
    public bool Evaluate(T source)
    {
        return base.Evaluate(new[] { source }).Any();
    }
}

If we have two types of specifications in our case (Query specification, Validation specification) maybe we can introduce these.
QuerySpecification
It holds information for querying a set/collection of data.
public class ValidVerificationCodeSpec: QuerySpecification<VerificationCode>
{
    public ValidVerificationCodeSpec()
    {
        Query
            .Where(vc => vc.DateUsed == null)
            .Where(vc => vc.DateCreated.AddMinutes(30) > DateTime.Now);
    }
}

ValidationSpecification
This specification holds information for a particular business rule. By leveraging this information we try to validate a given entity or any construct against those rules.
public class ValidVerificationCodeSpec: ValidationSpecification<VerificationCode>
{
    public ValidVerificationCodeSpec()
    {
         Validate
            .That(vc => vc.DateUsed == null)
            .And(vc => vc.DateCreated.AddMinutes(30) > DateTime.Now);
    }
}

The ValidationSpecification has the protected builder Validate so we can use it in the constructor to show our intentions.
The ValidationSpecification exposes an IsValid(T entity) method to check for the validity of an entity.
If the user uses the IsValid method and there are no rules defined it should throw an exception because it doesn't make sense to have a ValidationSpecification without any rules.
The last type of specification could be some super type as lack of better wording.
Specification
This specification holds information for querying a set/collection of data and has the rules defined for validating the entity.
public class ValidVerificationCodeSpec: Specification<VerificationCode>
    {
        private Expression<Func<VerificationCode, bool>> itIsNotUsed = code => code.DateUsed == null;
        private Expression<Func<VerificationCode, bool>> itIsStillValid = code => code.DateCreated.AddMinutes(30) > DateTime.Now;

        public ValidVerificationCodeSpec()
        {
            Query
                .Where(itIsNotUsed)
                .Where(itIsStillValid);

            Validate
                .That(itIsNotUsed)
                .And(itIsStillValid);
        }
    }

Also here will an IsValid method be exposed that throws an error if it is used when there are nog validation rules defined.
The user can leverage this type to reuse the logic used in the filtering as in the validation and is still able to have different validation rules than filtering rules. Also if they want to have the validation and querying logic in the same specification they could use this type.
I still have mixed feelings about the super type Specification because we are having the querying and validation logic in the same place.

These are valid approaches. But, implementation wise can get tricky. I'll share my concerns here
Separate specifications
What I meant by keeping these concerns independent is to just use different builders (or methods, properties, whatever). It's ok to have separate specification types as long as we keep them completely separate. But, if we define those separate specs and on top of it the "SuperSpecification" too, then the implementation and the maintenance can be a headache. The "SuperSpecification" can inherit either one or another, so we'll have to duplicate the implementation for the other one. There is already some level of complexity just because we have Specification<T, TResult>, and introducing additional complex inheritance chains will just create a mess. Also, we'd like to make the infrastructure extensible. Even at this point, extending it is a bit tricky (just because of the Select feature), so if we complicate even further we can forget of extensibility :)
I vote for having just one specification type, "SuperSpecification" in your case. We'll keep Query as it is, and just introduce a new mechanism to add validation logic.
Validation builder
Your intention here is great. But, trust me, it's a pain to implement that :) You'll start with And, then you will need Or, Andnot, OrNot, Not, etc, then you will define additional sub-builders to group them. It's a rabbit hole. Historically, we did all that in order to implement "composite specifications", which is not our requirement at all here. I personally dislike them and did not have great experience in the past. If you need a combination of several specs, then just write a new one, it makes stuff much cleaner. So, since we're not doing composition, we really have no reason to introduce some complex builder mechanisms. We have Linq, let the users write whatever they want, we just need the bool as a result, right?
It's important to emphasize that the success and the adoption of any tool is inversely proportional with the learning curve that the tool requires. That's the reason we introduced Query, not to introduce new concepts and rely on what users already know. It's the same with the validation logic, we should keep things familiar. In order to make it more flexible, we can start with a simple builder Validator which returns IValidationBuilder, and write an extension method Define which accepts Func<T, bool>. In the future, if we change our minds, we'll just add new extension methods.
So, it would be something like
public class ValidVerificationCodeSpec : Specification<VerificationCode>
{
    public ValidVerificationCodeSpec()
    {
        Query
            .Where(vc => vc.DateUsed == null)
            .Where(vc => vc.DateCreated.AddMinutes(30) > DateTime.Now);
    }

    public ValidVerificationCodeSpec(int minutes)
    {
        Validator.Define(entity =>
        {
            if (entity.DateUsed == null && entity.DateCreated.AddMinutes(minutes) > DateTime.Now) return true;

            return false;
        };
    }
}

Of course, users can organize their specs however they want, and based on your example they can write the above spec same as follows
public class ValidVerificationCodeSpec: Specification<VerificationCode>
{
    private Expression<Func<VerificationCode, bool>> itIsNotUsed = code => code.DateUsed == null;
    private Expression<Func<VerificationCode, bool>> itIsStillValid = code => code.DateCreated.AddMinutes(30) > DateTime.Now;

    public ValidVerificationCodeSpec()
    {
        Query
            .Where(itIsNotUsed)
            .Where(itIsStillValid);
    }

    public ValidVerificationCodeSpec()
    {
        Validator.Define(entity =>
        {
            if (itIsNotUsed(entity).Compile() && itIsStillValid(entity).Compile()) return true;

            return false;
        };
    }
}

Hey @JelmarVA ,
I just created a validations branch. If you wanna work on this, you can PR on that branch. If you agree, we can start only with the  Validator builder and the Define extension. Then we'll see how it goes.
@ardalis any thoughts about this thread?

I've just caught up on this thread and have an alternative design to propose for consideration, though I'm not sure how to structure it. Based on the initial requirement of simply allowing a simpler way to check whether an individual element is returned as by a specification, could we not create a wrapper around a specification that would provide this functionality? It's late and I don't have much time to commit to this but if I were custom coding it myself I could certainly write something like this:
public class Customer
{
  public int OrdersPlaced { get; set; }
}

public class CustomerWithOrderCountSpec : Specification<Customer>
{
  public CustomerWithOrderCountSpec(int minOrderCount)
  {
    Query.Where(c => c.OrdersPlaced >= minOrderCount);
  }
}

public class CustomerValidator
{
  public CustomerValidator(CustomerWithOrderSpec spec)
  {
    _spec = spec;
  }
  private CustomerWithOrderSpec _spec;

  public bool IsValid(Customer customer)
  {
    return _spec.Evaluate(new[] { customer}).Any();
  }
}

Now the trick to make this part of the framework would be to give CustomerValidator a constraint of some sort that made its design enforce the implicit rules given here, mainly that you pass a valid spec into it via its constructor.
Another option would be an extension method that would do the same work.
Both of these seem cleaner than adding a whole new concept of validators into the base spec class itself, to me. I don't know that too many people will use this feature instead of just using a more full-featured library like FluentValidation.
Thoughts?

Actually, that was the original request by @JelmarVA. In one of my answers, I provided the following.

As for your original idea of utilizing the Evaluate feature for this purpose, you can easily do that even today.
All you need is to create your own base specification as follows. Then, you use it as a base for all your specs in the application.
public class AppSpecification<T> : Specification<T>
{
    public bool Evaluate(T source)
    {
        return base.Evaluate(new[] { source }).Any();
    }
}

You can name it IsValid, and then use it like this (you don't even need a separate validator class, right?)
new CustomerSpec().IsValid(customer);

The arguments against implementing this as part of the package are:

The evaluator will evaluate also Take, Skip, Order, and you might get inconsistent results. It's a subtle issue and it's hard to spot it. So, you'll end up with some implicit rules "Ok, if this spec will be used for validation, then you should remember to use only Where".
Validations are in-memory actions. They operate against some in-memory constructs and can contain complex rules, right? But, in this case, you'll have to be careful what you're writing in Query.Where, since if you write something complex the ORM won't be able to evaluate it. So, either you'll restrict yourself to only simple rules, or you'll have another implicit rule "Ok, you can write complex rules here, but you should remember not to use this spec with your repository".
I guess conceptually is wrong. Where is a filtering operation against collections. We can use it, but it's just a hack, right? Do we wanna embed it in the package?

To summarize, it's OK approach as long you are aware of these caveats. So, users can adopt it in their codebases if they want (simple base class as shown above). But, I'm not so sure of including it as part of the package. It will confuse and create ambiguity for the rest 90% of the users.

I get @fiseni's arguments against implementing it in the package. For me, it will be clear because I introduced this hack and know the caveats. For other users (and colleagues), it would create ambiguity indeed.
If I want to remove the ambiguity in my own solutions about the validation I can introduce the FluentValidation library.
I can have an abstract class ValidationSpecification
 public abstract class ValidationSpecification<T> : IValidationSpecification<T>
{
    public Specification<T> Specification { get; internal set; }
    public AbstractValidator<T> Validator { get; internal set; }

    protected ValidationSpecification(Specification<T> specification, AbstractValidator<T> validator)
    {
        Specification = specification;
        Validator = validator;
    }

    public ValidationResult Validate(T entity)
    {
        return Validator.Validate(new ValidationContext<T>(entity));
    }

    public IEnumerable<T> Evaluate(IEnumerable<T> entities)
    {
        return Specification.Evaluate(entities);
    }
}

With the ValidationSpecification class, I can hold both a complex validation and the Specification I want to use against the ORM.
This way I can leverage both powerful libraries.
public class ValidVerificationCodeValidationSpec : ValidationSpecification<VerificationCode>
{
    private static int minutes = 30;
    public ValidVerificationCodeValidationSpec() :
        base(
            new ValidVerificationCodeSpec(minutes),
            new ValidVerificationCodeValidator(minutes)
            )
    { }
}

Let me know how you guys feel about this design.
I don't know if this should be implemented in the package because it would become dependant on the FluentValidator library. But it can be a nonissue.

I have no strong opinion on this. Honestly, I don't see myself using this feature very much. Usually, I use different constructs for validations, I need error messages, localizations, etc. So, whatever the community likes 😀
Actually, we can publish all this as a separate addon package Ardalis.Specifications.Validations. So whoever wants it, can pick it up explicitly.
@ardalis has the last call, I'm OK either way.

I'd be fine with having a separate package as @fiseni suggests, which could simply hold the types shown in @JelmarVA 's last comment. It can live in this repo but have a separate nuget package and build pipeline.

While I don't use this for validation, I do have a similar use-case for evaluating a specification in memory for a single entity.
I have an extension method similar to what was suggested:
public static bool IsSatisfiedBy<T>(this ISpecification<T> specification, T entity) =>
    specification.Evaluate(new[] { entity }).Any();

So I can do things like:
var spec = new PremiumCustomerSpecification();

if (spec.IsSatisfiedBy(customer))
{
    ShowOfferOnlyPremiumCustomersGet();
}

I'm not sure if I'm doing something completely wrong here but just wanted to add this to the discussion.

Nothing wrong here. This is a perfectly valid approach. 👍

We're getting this request quite often in one form or another. So, let's try to implement it. But, instead of utilizing the above-mentioned hack, we should try to do it properly. We can use the predicates directly, and not apply it through LINQ Where at all.
The API should be simple, you provide the entity and get back information whether it satisfies the specification. We can discuss the name, it can be named "IsSatisfiedBy" or "IsValid", or anything else.
public interface ISpecification<T>
{
	...

	bool IsSatisfiedBy(T entity);
}

As discussed in several threads, our specifications primarily are intended for query generation and don't hold only criteria information. So, during this validation, we have to account only for the state which represents a "criteria" (Where and Search expressions in our case). We already have something similar for the specification evaluator (used for the CountAsync action), and we can extend the same idea for the in-memory evaluators too. To recap, we have two types of evaluators

ISpecificationEvaluator which iterates through a list of partial IEvaluator evaluators, and is intended to be used for query generation through various ORMs.
IInMemorySpecificationEvaluator which iterates through a list of partial IInMemoryEvaluator evaluators, which evaluates the provided in-memory collections.

public interface ISpecificationEvaluator
{
    IQueryable<TResult> GetQuery<T, TResult>(IQueryable<T> inputQuery, ISpecification<T, TResult> specification) where T : class;
    IQueryable<T> GetQuery<T>(IQueryable<T> inputQuery, ISpecification<T> specification, bool evaluateCriteriaOnly = false) where T : class;
}
public interface IEvaluator
{
    bool IsCriteriaEvaluator { get; }

    IQueryable<T> GetQuery<T>(IQueryable<T> query, ISpecification<T> specification) where T : class;
}

public interface IInMemorySpecificationEvaluator
{
    IEnumerable<TResult> Evaluate<T, TResult>(IEnumerable<T> source, ISpecification<T, TResult> specification);
    IEnumerable<T> Evaluate<T>(IEnumerable<T> source, ISpecification<T> specification);
}
public interface IInMemoryEvaluator
{
    IEnumerable<T> Evaluate<T>(IEnumerable<T> query, ISpecification<T> specification);
}

The idea is to extend the IInMemorySpecificationEvaluator as follows. The Evaluate method will iterate and check against all partial evaluators which are defined as criteria evaluators.
public interface IInMemorySpecificationEvaluator
{
    IEnumerable<TResult> Evaluate<T, TResult>(IEnumerable<T> source, ISpecification<T, TResult> specification);
    IEnumerable<T> Evaluate<T>(IEnumerable<T> source, ISpecification<T> specification);
    bool Evaluate<T>(T entity, ISpecification<T> specification);
}
public interface IInMemoryEvaluator
{
    bool IsCriteriaEvaluator { get; }
    IEnumerable<T> Evaluate<T>(IEnumerable<T> query, ISpecification<T> specification);
    bool Evaluate<T>(T entity, ISpecification<T> specification);
}

Then, the IsSatisfiedBy implementation in the specification becomes quite trivial
public virtual bool IsSatisfiedBy(T entity)
{
    return Evaluator.Evaluate(entity, this);
}

This is implemented a bit differently. Utilizing and extending the IInMemoryEvaluator won't be nice, since we will force the omitted evaluators to implement bool Evaluate<T>(T entity, ISpecification<T> specification);. Instead, we'll use a separate infrastructure for this feature. We'll have ISpecificationValidator construct which will iterate through a list of partial IValidator validators. Thus, we'll be able to define only the required validators, in this case for Where and Search. More details in the related PR.
@JelmarVA This is exactly what you wanted. You can use the same specification without the need to duplicate your expressions. The same spec can be utilized for query generation, in-memory evaluation, and also for entity validation.


K8SPXC-586 Some tests fixes

 ❨?❩

Monitoring 2.0 test was adapted for openshift and latest PMM server
Upgrade proxysql test was updated with missing file for openshift

Test name
Status

upgrade-haproxy
passed

big-data
passed

scaling
passed

recreate
passed

haproxy
passed

storage
passed

self-healing
passed

init-deploy
passed

scaling-proxysql
passed

security-context
passed

upgrade-proxysql
passed

limits
passed

restore-to-encrypted-cluster
passed

monitoring-2-0
passed

smart-update
passed

affinity
passed

demand-backup
passed

self-healing-advanced
passed

one-pod
passed

auto-tuning
passed

proxysql-sidecar-res-limits
passed

scheduled-backup
passed

users
passed

operator-self-healing
passed

tls-issue-self
passed

demand-backup-encrypted-with-tls
failed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

upgrade-consistency
passed

validation-hook
passed

Test name
Status

upgrade-haproxy
passed

big-data
passed

scaling
passed

recreate
passed

haproxy
passed

storage
passed

self-healing
passed

init-deploy
passed

scaling-proxysql
passed

security-context
passed

upgrade-proxysql
passed

limits
passed

restore-to-encrypted-cluster
passed

monitoring-2-0
passed

smart-update
passed

affinity
passed

demand-backup
passed

self-healing-advanced
passed

one-pod
passed

auto-tuning
passed

proxysql-sidecar-res-limits
passed

scheduled-backup
passed

users
passed

operator-self-healing
passed

tls-issue-self
passed

demand-backup-encrypted-with-tls
failed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

upgrade-consistency
passed

validation-hook
passed

Test name
Status

storage
passed

big-data
passed

scaling
passed

haproxy
passed

upgrade-haproxy
passed

recreate
passed

self-healing
passed

init-deploy
passed

scaling-proxysql
passed

security-context
passed

restore-to-encrypted-cluster
passed

upgrade-proxysql
passed

limits
passed

self-healing-advanced
passed

monitoring-2-0
passed

demand-backup
passed

smart-update
passed

operator-self-healing
passed

affinity
passed

one-pod
passed

auto-tuning
passed

proxysql-sidecar-res-limits
passed

scheduled-backup
passed

users
passed

upgrade-consistency
passed

tls-issue-self
passed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

validation-hook
passed

demand-backup-encrypted-with-tls
failed

Test name
Status

storage
passed

big-data
passed

scaling
passed

haproxy
passed

upgrade-haproxy
passed

recreate
passed

self-healing
passed

init-deploy
passed

scaling-proxysql
passed

security-context
passed

restore-to-encrypted-cluster
passed

upgrade-proxysql
passed

limits
passed

self-healing-advanced
passed

monitoring-2-0
passed

demand-backup
passed

smart-update
passed

operator-self-healing
passed

affinity
passed

one-pod
passed

auto-tuning
passed

proxysql-sidecar-res-limits
passed

scheduled-backup
passed

users
passed

upgrade-consistency
passed

tls-issue-self
passed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

validation-hook
passed

demand-backup-encrypted-with-tls
failed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-712-50769a24

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-712-50769a24

Test name
Status

upgrade-haproxy
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring-2-0
passed

affinity
passed

recreate
passed

one-pod
passed

storage
passed

scaling
passed

restore-to-encrypted-cluster
passed

auto-tuning
passed

scaling-proxysql
passed

self-healing
passed

demand-backup
passed

proxysql-sidecar-res-limits
passed

security-context
passed

self-healing-advanced
passed

scheduled-backup
passed

users
passed

operator-self-healing
passed

demand-backup-encrypted-with-tls
passed

tls-issue-self
passed

big-data
passed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

validation-hook
passed

upgrade-proxysql
passed

smart-update
passed

upgrade-consistency
passed

pitr
passed

Test name
Status

upgrade-haproxy
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring-2-0
passed

affinity
passed

recreate
passed

one-pod
passed

storage
passed

scaling
passed

restore-to-encrypted-cluster
passed

auto-tuning
passed

scaling-proxysql
passed

self-healing
passed

demand-backup
passed

proxysql-sidecar-res-limits
passed

security-context
passed

self-healing-advanced
passed

scheduled-backup
passed

users
passed

operator-self-healing
passed

demand-backup-encrypted-with-tls
passed

tls-issue-self
passed

big-data
passed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

validation-hook
passed

upgrade-proxysql
passed

smart-update
passed

upgrade-consistency
passed

pitr
passed


early stop worker got Exception Error

🐛 Describe the bug
dp = ...
dp = dp.sharding_filter()
rs = MultiProcessingReadingService(num_workers=4)
dataloader = DataLoader2(dp, reading_service=rs)

for _ in dataloader:
    pass

dataloader.shutdown()

Exception: Can not expect any response without submitted request
This exception is thrown by __iter__ of _IterateQueueDataPipes(datapipes=[QueueWrapper, QueueWrapper, QueueWrapper, QueueWrapper])

related to https://github.com/pytorch/data/commit/901b48312179865657b90792aa03f0775ec50971
Versions
[pip3] numpy==1.26.4
[pip3] pytorch-wpe==0.0.1
[pip3] rotary-embedding-torch==0.6.4
[pip3] torch==2.1.1
[pip3] torch-complex==0.4.4
[pip3] torchaudio==2.1.1
[pip3] torchdata==0.7.1

Hi @kamirdin, please see https://github.com/pytorch/data/issues/1196 as we won't be supporting dataloader2 nor datapipes in torchdata moving forward

@kamirdin have you fix it? I get same issue


VS Code 1.79.0 use root user can not open with cmd --no-sandbox --user-data-dir

Type: Bug

VS Code Version: 1.79.0
OS Version: rockylinux9.2

Steps to Reproduce:

use root user
code --no-sandbox --user-data-dir=/home/vscode #Display an empty window
vim /var/log/messages #show some err msg
Jun  9 15:27:06 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:06 R9 systemd[1]: Started Process Core Dump (PID 6287/UID 0).
Jun  9 15:27:06 R9 systemd-coredump[6288]: Resource limits disable core dumping for process 6286 (code).
Jun  9 15:27:06 R9 systemd-coredump[6288]: Process 6286 (code) of user 0 dumped core.
Jun  9 15:27:06 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:06 R9 systemd[1]: Started Process Core Dump (PID 6293/UID 0).
Jun  9 15:27:06 R9 systemd-coredump[6294]: Resource limits disable core dumping for process 6292 (code).
Jun  9 15:27:06 R9 systemd-coredump[6294]: Process 6292 (code) of user 0 dumped core.
Jun  9 15:27:06 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:06 R9 systemd[1]: Started Process Core Dump (PID 6299/UID 0).
Jun  9 15:27:06 R9 systemd-coredump[6300]: Resource limits disable core dumping for process 6298 (code).
Jun  9 15:27:06 R9 systemd-coredump[6300]: Process 6298 (code) of user 0 dumped core.
Jun  9 15:27:06 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:06 R9 systemd[1]: Started Process Core Dump (PID 6305/UID 0).
Jun  9 15:27:06 R9 systemd-coredump[6306]: Resource limits disable core dumping for process 6304 (code).
Jun  9 15:27:06 R9 systemd-coredump[6306]: Process 6304 (code) of user 0 dumped core.
Jun  9 15:27:06 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:06 R9 systemd[1]: Started Process Core Dump (PID 6311/UID 0).
Jun  9 15:27:06 R9 systemd-coredump[6313]: Resource limits disable core dumping for process 6310 (code).
Jun  9 15:27:06 R9 systemd-coredump[6313]: Process 6310 (code) of user 0 dumped core.
Jun  9 15:27:06 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:32 R9 systemd[1]: virtqemud.service: Deactivated successfully.
Jun  9 15:27:44 R9 systemd[1]: Crash recovery kernel arming was skipped because of an unmet condition check (ConditionKernelCommandLine=crashkernel).
Jun  9 15:27:49 R9 kernel: show_signal: 30 callbacks suppressed
Jun  9 15:27:49 R9 kernel: traps: code[6431] trap int3 ip:56500835a0de sp:7ffee17641e0 error:0 in code[565005131000+70a8000]
Jun  9 15:27:49 R9 systemd[1]: Started Process Core Dump (PID 6432/UID 0).
Jun  9 15:27:49 R9 systemd-coredump[6433]: Resource limits disable core dumping for process 6431 (code).
Jun  9 15:27:49 R9 systemd-coredump[6433]: Process 6431 (code) of user 0 dumped core.
Jun  9 15:27:49 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:49 R9 kernel: traps: code[6439] trap int3 ip:55fc8b24f0de sp:7ffdc2d3cd70 error:0 in code[55fc88026000+70a8000]
Jun  9 15:27:49 R9 systemd[1]: Started Process Core Dump (PID 6441/UID 0).
Jun  9 15:27:49 R9 systemd-coredump[6442]: Resource limits disable core dumping for process 6439 (code).
Jun  9 15:27:49 R9 systemd-coredump[6442]: Process 6439 (code) of user 0 dumped core.
Jun  9 15:27:49 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:49 R9 kernel: traps: code[6464] trap int3 ip:564e6e8100de sp:7ffe4a79e980 error:0 in code[564e6b5e7000+70a8000]
Jun  9 15:27:49 R9 systemd[1]: Started Process Core Dump (PID 6465/UID 0).
Jun  9 15:27:49 R9 systemd-coredump[6466]: Resource limits disable core dumping for process 6464 (code).
Jun  9 15:27:49 R9 systemd-coredump[6466]: Process 6464 (code) of user 0 dumped core.
Jun  9 15:27:49 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:49 R9 kernel: traps: code[6470] trap int3 ip:55953e5650de sp:7ffd8aa79710 error:0 in code[55953b33c000+70a8000]
Jun  9 15:27:49 R9 systemd[1]: Started Process Core Dump (PID 6471/UID 0).
Jun  9 15:27:49 R9 systemd-coredump[6472]: Resource limits disable core dumping for process 6470 (code).
Jun  9 15:27:49 R9 systemd-coredump[6472]: Process 6470 (code) of user 0 dumped core.
Jun  9 15:27:49 R9 systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Jun  9 15:27:49 R9 kernel: traps: code[6476] trap int3 ip:5641a50f00de sp:7ffc1ea51e40 error:0 in code[5641a1ec7000+70a8000]
Jun  9 15:27:49 R9 systemd[1]: Started Process Core Dump (PID 6477/UID 0).
Jun  9 15:27:49 R9 systemd-coredump[6478]: Resource limits disable core dumping for process 6476 (code).

VS Code version: Code - Insiders 1.79.0-insider (b380da4ef1ee00e224a15c1d4d9793e27c2b6302, 2023-06-07T14:26:54.152Z)
OS version: Windows_NT x64 10.0.22621
Modes:

Same issue after updating to 1.79.0 on Windows 10 (19045.2965) and trying to launch Code as admin

Logs
[main 2023-06-09T08:46:43.113Z] IPC Object URL: Registered new channel vscode:aba3ed2f-13a2-43fe-8cd1-7c38f6cb798b.
[main 2023-06-09T08:46:43.113Z] window#validateWindowState: validating window state on 2 display(s) { mode: 0, x: 2381, y: 0, width: 1459, height: 846 }
[main 2023-06-09T08:46:43.113Z] window#validateWindowState: multi-monitor working area { x: 1920, y: 0, width: 1920, height: 1040 }
[main 2023-06-09T08:46:43.113Z] window#ctor: using window state { mode: 0, x: 2381, y: 0, width: 1459, height: 846 }
[main 2023-06-09T08:46:43.158Z] window#load: attempt to load window (id: 1)
[main 2023-06-09T08:46:43.172Z] windowsManager#open used window count 1 (workspacesToOpen: 0, foldersToOpen: 0, emptyToRestore: 1, emptyToOpen: 0)
[main 2023-06-09T08:46:43.172Z] lifecycle (main): phase changed (value: 3)
[main 2023-06-09T08:46:43.177Z] resolveShellEnv(): skipped (Windows)
[main 2023-06-09T08:46:43.177Z] update#setState idle
Renderer process launch-failed - see https://www.electronjs.org/docs/tutorial/application-debugging for potential debugging information.
[main 2023-06-09T08:46:43.194Z] CodeWindow: renderer process gone (reason: launch-failed, code: 18)
[33572:0609/104643.231:INFO:electron_api_web_frame_main.cc(235)] Attempt to setup WebFrameMain connection while render frame is disposed
[main 2023-06-09T08:46:43.227Z] [File Watcher (node.js)] Started watching: 'c:\Users\grinenkoa\AppData\Roaming\Code\User'
Renderer process launch-failed - see https://www.electronjs.org/docs/tutorial/application-debugging for potential debugging information.
[main 2023-06-09T08:46:43.230Z] CodeWindow: renderer process gone (reason: launch-failed, code: 18)

@sjwyz can you start with --no-sandbox --disable-gpu-sandbox --user-data-dir=/tmp/vscode on your linux setup and see if the issue persists.
@Arko109 thanks for the logs, exit code 18 is generic process launch error from the chromium sandbox layer https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:sandbox/win/src/target_process.cc;l=181-191 so looks like sandbox did not get disabled in your case. To which executable did you pass the flags ? Maybe a screen recording would help.

@deepak1556 use arguments --no-sandbox --disable-gpu-sandbox --user-data-dir=/tmp/vscode and the problem still exists.
There are some changes in the phenomenon.Previously, the window remained on the desktop,add arguments --disable-gpu-sandbox, the window flickered and disappeared.

@deepak1556 And my Rockylinux9.2 is installed on Windows version vmware Workstation 17

@deepak1556  I have a shortcut that points to "C:\Program Files\Microsoft VS Code\Code.exe" --no-sandbox --disable-gpu-sandbox, but I also tried running Code --no-sandbox --disable-gpu-sandbox in PowerShell, same results. Here's a video recording:
https://github.com/microsoft/vscode/assets/16703367/f74a67c3-6e4b-4bdb-a6ed-faad76728949

Microsoft Office developers are experiencing this issue as well.
Things tried:
1 - --no-sandbox --disable-gpu-sandbox
Issue persists.
2 - I tried creating a dump, but it only creates a dat file.
D:\dumps>code --crash-reporter-directory D:\dumps\

D:\dumps>tree /F
Folder PATH listing for volume Storage space
Volume serial number is DED7-F5BA
D:.
│   settings.dat
│
├───attachments
└───reports

3 - I have recorded time travel traces of the different processes.
D:\dumps>C:\debuggers\TTD\tttracer.exe -parent * -onlaunch code.exe -out D:\dumps
Microsoft (R) TTTracer 1.01.10
Release: <IP_ADDRESS>
Copyright (C) Microsoft Corporation. All rights reserved.

Recording process Code.exe(46564)        From parent process cmd.exe(55072)
    Recording process (PID:46564) on trace file: D:\dumps\Code01.run
Recording process Code.exe(21876)        From parent process Code.exe(46564)
    Recording process (PID:21876) on trace file: D:\dumps\Code02.run
(x64) (PID:46564): Process exited with exit code 0 after 13890ms
  Full trace dumped to D:\dumps\Code01.run
Recording process Code.exe(42108)        From parent process Code.exe(21876)
    Recording process (PID:42108) on trace file: D:\dumps\Code03.run
Recording process Code.exe(29728)        From parent process Code.exe(21876)
    Recording process (PID:29728) on trace file: D:\dumps\Code04.run
Recording process Code.exe(24444)        From parent process Code.exe(21876)
    Recording process (PID:24444) on trace file: D:\dumps\Code05.run
(x64) (PID:24444): Process exited with exit code 0 after 14891ms
  Full trace dumped to D:\dumps\Code05.run
(x64) (PID:29728): Process exited with exit code 0 after 20156ms
  Full trace dumped to D:\dumps\Code04.run
(x64) (PID:21876): Process exited with exit code 0 after 31422ms
  Full trace dumped to D:\dumps\Code02.run
(x64) (PID:42108): Process exited with exit code 0 after 21656ms
  Full trace dumped to D:\dumps\Code03.run

Things tried:
1 - --no-sandbox --disable-gpu-sandbox Issue persists.
2 - I tried creating a dump, but it only creates a dat file.
3 - I have recorded time travel traces of the different processes.

I have some feeling that update has broke argument parsing/saving on no-sandbox flag because I have this output:
Verbose output
`# code --user-data-dir="/root/.vscode" --no-sandbox --verbose
[102303:0610/020244.034319:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
[main 2023-06-09T23:02:44.176Z] [File Watcher (node.js)] Request to start watching: /root/.vscode/User (excludes: , includes: ),/root/.vscode/User/settings.json (excludes: , includes: )
[0610/020244.198923:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[main 2023-06-09T23:02:44.201Z] Starting VS Code
[main 2023-06-09T23:02:44.201Z] from: /usr/share/code/resources/app
[main 2023-06-09T23:02:44.202Z] args: {
  _: [],
  diff: false,
  merge: false,
  add: false,
  goto: false,
  'new-window': false,
  'reuse-window': false,
  wait: false,
  'user-data-dir': '/root/.vscode',
  help: false,
  'list-extensions': false,
  'show-versions': false,
  'pre-release': false,
  version: false,
  verbose: true,
  status: false,
  'prof-startup': false,
  'no-cached-data': false,
  'prof-v8-extensions': false,
  'disable-extensions': false,
  'disable-gpu': false,
  'ms-enable-electron-run-as-node': false,
  telemetry: false,
  debugRenderer: false,
  'enable-smoke-test-driver': false,
  logExtensionHostCommunication: false,
  'skip-release-notes': false,
  'skip-welcome': false,
  'disable-telemetry': false,
  'disable-updates': false,
  'disable-keytar': false,
  'disable-workspace-trust': false,
  'disable-crash-reporter': false,
  'crash-reporter-id': 'd3f810e8-d4bf-4c6c-b47c-cef319bf1b60',
  'skip-add-to-recently-opened': false,
  'unity-launch': false,
  'open-url': false,
  'file-write': false,
  'file-chmod': false,
  force: false,
  'do-not-sync': false,
  trace: false,
  'force-user-env': false,
  'force-disable-user-env': false,
  'open-devtools': false,
  'disable-gpu-sandbox': false,
  '__enable-file-policy': false,
  'enable-coi': false,
  'no-proxy-server': false,
  'no-sandbox': false,
  nolazy: false,
  'force-renderer-accessibility': false,
  'ignore-certificate-errors': false,
  'allow-insecure-localhost': false,
  'disable-dev-shm-usage': false,
  'profile-temp': false,
  logsPath: '/root/.vscode/logs/20230610T020244'
}
[main 2023-06-09T23:02:44.206Z] Resolving machine identifier...
[main 2023-06-09T23:02:44.206Z] Resolved machine identifier: 9ca044ac56a200b25f97a24709fdad525caa02c9d29e3134ef3c2c0e8e50f74f
[main 2023-06-09T23:02:44.206Z] Main->SharedProcess#connect
[main 2023-06-09T23:02:44.211Z] [File Watcher (node.js)] Started watching: '/root/.vscode/User'
[main 2023-06-09T23:02:44.213Z] [File Watcher (node.js)] Started watching: '/root/.vscode/User/settings.json'
[main 2023-06-09T23:02:44.216Z] StorageMainService: creating application storage
[main 2023-06-09T23:02:44.225Z] ElectronURLListener: waiting for window to be ready to handle URLs...
[main 2023-06-09T23:02:44.225Z] lifecycle (main): phase changed (value: 2)
[main 2023-06-09T23:02:44.226Z] windowsManager#open
[102333:0610/020244.226525:WARNING:sandbox_linux.cc(380)] InitializeSandbox() called with multiple threads in process gpu-process.
[main 2023-06-09T23:02:44.226Z] windowsManager#open pathsToOpen [
  {
    backupPath: '/root/.vscode/Backups/1686313915326',
    remoteAuthority: undefined
  }
]
[main 2023-06-09T23:02:44.227Z] windowsManager#doOpenEmpty {
  restore: true,
  remoteAuthority: undefined,
  filesToOpen: undefined,
  forceNewWindow: true
}
[main 2023-06-09T23:02:44.229Z] IPC Object URL: Registered new channel vscode:254a97f4-1783-438f-b576-add0d3e00854.
[main 2023-06-09T23:02:44.230Z] window#validateWindowState: validating window state on 1 display(s) { mode: 0, x: 0, y: 29, width: 1920, height: 1011 }
[main 2023-06-09T23:02:44.230Z] window#validateWindowState: 1 monitor working area { x: 0, y: 0, width: 1920, height: 1040 }
[main 2023-06-09T23:02:44.231Z] window#ctor: using window state { mode: 0, x: 0, y: 29, width: 1920, height: 1011 }
[main 2023-06-09T23:02:44.320Z] window#load: attempt to load window (id: 1)
[main 2023-06-09T23:02:44.337Z] windowsManager#open used window count 1 (workspacesToOpen: 0, foldersToOpen: 0, emptyToRestore: 1, emptyToOpen: 0)
[main 2023-06-09T23:02:44.337Z] lifecycle (main): phase changed (value: 3)
[main 2023-06-09T23:02:44.337Z] resolveShellEnv(): skipped (VSCODE_CLI is set)
[main 2023-06-09T23:02:44.338Z] update#setState idle
[102303:0610/020244.444472:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020244.486842:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020244.673278:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020244.707520:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020244.878822:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020244.909962:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020245.105310:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020245.139476:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020245.311264:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020245.345518:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020245.528944:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020245.560666:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020245.750638:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020245.797133:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020245.914979:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020245.938484:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020246.108599:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020246.148474:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020246.182080:ERROR:service_worker_storage.cc(1933)] Failed to delete the database: Database IO error
[102303:0610/020246.279736:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020246.316049:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[102303:0610/020246.507002:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.
[0610/020246.552562:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.`

Look at no-sandbox element in args dict, it has false value, but --no-sandbox flag presented in run command. Also there are a lot of FATAL messages on not presented --no-sandbox flag

This seems to break quite a few people's CI's, as in the referenced issue: https://github.com/microsoft/vscode-test/issues/221

Seeing the same thing on Windows Server 2019.

Getting the same issue when using @vscode/test-electron to run test-cases for an extension.

My headless CI tests works with 1.78.2 ,  but as soon as I use .79, it starts crashing continuously with below messages repeated.
[0608/193640.183415:FATAL:electron_main_delegate.cc(304)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[454:0608/193640.247475:ERROR:network_service_instance_impl.cc(521)] Network service crashed, restarting service.

Thanks @ITAxReal for confirming the args are not being passed down to the child processes. The issue is that runtime removes the --no-sandbox switch when the following API is called https://github.com/microsoft/vscode/blob/89ec834df20d597ff96f7d303e7e0f2f055d2a4e/src/main.js#L31

Oh, nice find, that's on me 🤦

Oh, nice find, that's on me 🤦

Yeah. No argument value checking before doing argument-controlled action. Just happy because it's not a big problem to fix it quickly :)

@deepak1556 Will the fixed build be published soon on stable channel?

It will be available with 1.79.2 later this week, in the meantime recommended course of action is to roll back to v1.78

/closedWith 4f95907

Steps for verification:

From command line launch code-insiders <path-to-folder>
Open process explorer and inspect the workbench process command line has --enable-sandbox flag in it
Exit the application
Now launch from command line with code-insiders <path-to-folder> --no-sandbox --disable-gpu-sandbox
Open process explorer and inspect the workbench process command line has --no-sandbox flag instead of --enable-sandbox flag in it

This bug has been fixed in the latest release of VS Code Insiders!
@sjwyz, you can help us out by commenting /verified if things are now working as expected.
If things still don't seem right, please ensure you're on version f3aa9a201b2455e6556797e8ffb7145d77adb792 of Insiders (today's or later - you can use Help: About in the command palette to check), and leave a comment letting us know what isn't working as expected.
Happy Coding!

@tiagomacarios can you also verify if the issue is resolved with the above insiders link on your setup ?

@deepak1556 No. The problem persists.

@tiagomacarios is the application launched with code-insiders --no-sandbox --disable-gpu-sandbox, from the screenshot I don't see the flags passed.

@deepak1556 yeah that works. Is there a way to run elevated without using those flags?

can you also verify if the issue is resolved with the above insiders link?

/verify
Yeah, yesterday version of insiders vscode is working on Linux under root user with --no-sandbox flag.

@deepak1556
I use cmd code-insiders --no-sandbox --user-data-dir=/home/lzl/vscodeii
Now the program seems to be running normally,but in cmd ps aux|grep sandbox , --enable-sandbox still exists

2023-06-16 10:53:23 root@control-tower:~# apt -y install ./code-insiders_1.80.0-1686909910_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'code-insiders' instead of './code-insiders_1.80.0-1686909910_amd64.deb'
The following NEW packages will be installed:
  code-insiders
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/95.7 MB of archives.
After this operation, 375 MB of additional disk space will be used.
Get:1 /root/code-insiders_1.80.0-1686909910_amd64.deb code-insiders amd64 1.80.0-1686909910 [95.7 MB]
Selecting previously unselected package code-insiders.
(Reading database ... 127565 files and directories currently installed.)
Preparing to unpack .../code-insiders_1.80.0-1686909910_amd64.deb ...
Unpacking code-insiders (1.80.0-1686909910) ...
Setting up code-insiders (1.80.0-1686909910) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for shared-mime-info (2.2-1) ...
N: Download is performed unsandboxed as root as file '/root/code-insiders_1.80.0-1686909910_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
2023-06-16 10:54:05 root@control-tower:~# code-insiders --no-sandbox --user-data-dir=~
2023-06-16 10:54:47 root@control-tower:~# cat /proc/version
Linux version 6.1.0-9-amd64<EMAIL_ADDRESS>(gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
2023-06-16 11:13:31 root@control-tower:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
2023-06-16 11:13:35 root@control-tower:~#

insider works again! Is this a fix that will get backported to 1.79.x release or will I need to wait for the official 1.80.x release?

@reukiodo it's in 1.79.2

Hi,
I have VSCode 1.79.2 running on Ubuntu 20.04 which I installed using snap. When I try to start VSCode as root using the following command:
sudo code --verbose --user-data-dir=/root/.config/Code --no-sandbox
I just get a blank dialog as shown in the attachment below. I am also attaching my logs. Any help would be extremely grateful.
Thank you.

VerboseLogs1.txt

@Satya-Sidhu are you able to launch the application without sudo ?

As a non-root user I can launch it without sudo. But cannot launch it with sudo as a non-root user. Also cannot launch it if I switch over to root account.

@Satya-Sidhu thanks for confirming, in your previous comment https://github.com/microsoft/vscode/issues/184687#issuecomment-1598098862 I missed that you are not launching with --disable-gpu-sandbox, does the issue happen when you add this flag as well sudo code --user-data-dir=/root/.config/Code --no-sandbox --disable-gpu-sandbox ?

Hi @deepak1556, this
sudo code --user-data-dir=/root/.config/Code --no-sandbox --disable-gpu-sandbox
did work for me.
Thank you very much for your help.

@deepak1556 Is there a way to persist --no-sandbox --disable-gpu-sandbox ? registry key or config file?
Our development environment runs elevated and engineers now need to start adding these flags to all their commands...

I  use a shell script to start VSCode as I have to do a lot of configurations before.
Below is what I do to start VSCode.

@tiagomacarios we will provide a setting to persist this option with 1.80 release.

@tiagomacarios In linux I can use 'alias' command like alias code='code --no-sandbox
In win I think you can override it with bat script with same name placed in folder that placed in path. Note that you should use absolute path of 'code' in bat script.

What is the equivalent of disable-gpu-sandbox for settings.json in windows?

@ppgovekar please refer to https://code.visualstudio.com/updates/v1_80#_disable-chromium-sandbox


BUG: Unable to create project when using litmusctl with latest (v3.0.0) version of Litmus

When trying to create a project using litmusctl we get following error:
Unmatched status code:{"error":"invalid_request","errorDescription":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed"}

When checking the logs for auth-server I see this in the logs:
{"file":"/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/logger.go:192","func":"github.com/sirupsen/logrus.(*Logger).Log","level":"warning","msg":"json: unknown field \"project_name\"","time":"2023-10-16T11:28:38Z"}

This is because the project_name param has been renamed here to projectName. Changing payload to use projectName fixes the problem. Here is an example curl request.
curl --location 'http://localhost:9003/create_project' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <JWT TOKEN>' \
--data '{
    "projectName": "awesome-project"
}'

Thanks @smitthakkar96 for pointing out this issue. This issue has already been addressed through PR #156 ( and  i think it's going to be merged soon)

Fixed with PR: https://github.com/litmuschaos/litmusctl/pull/156


issue: RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 32 but got size 31 for tensor number 1 in the list.

(workflow attached)
Images are generated normally at 2048x2048 and 1920x2048 resolutions. If you try to generate an image at 1984x2048, an error occurs, and once an error occurs, changing the resolution continues to cause errors. (event if 2048x2048, 1920x2048)
got prompt
It seems like you have activated model offloading by calling `enable_model_cpu_offload`, but are now manually moving the pipeline to GPU. It is strongly recommended against doing so as memory gains from offloading are likely to be lost. Offloading automatically takes care of moving the individual components vae, text_encoder, tokenizer, unet, scheduler, safety_checker, feature_extractor, image_encoder to GPU when needed. To make sure offloading works as expected, you should consider moving the pipeline back to CPU: `pipeline.to('cpu')` or removing the move altogether if you use offloading.
  0%|          | 0/25 [00:00<?, ?it/s]
!!! Exception during processing!!! Sizes of tensors must match except in dimension 1. Expected size 32 but got size 31 for tensor number 1 in the list.
Traceback (most recent call last):
  File "/mnt/teratera/git/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/git/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/git/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-ELLA-wrapper/nodes.py", line 422, in process
    images = pipe(
             ^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 971, in __call__
    noise_pred = self.unet(
                 ^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
    result = forward_call(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/diffusers/models/unets/unet_2d_condition.py", line 1281, in forward
    sample = upsample_block(
             ^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv-stable/lib/python3.11/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 2512, in forward
    hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 32 but got size 31 for tensor number 1 in the list.

Prompt executed in 0.24 seconds
#2 [diffusers_sampler]: 0.24s

Seems to be a bug in the HiDiffusion code, couldn't find it myself, probably should report to their repository. There was another dimension specific bug they fixed, but it didn't fix this one.


add german translation

This PR adds a German translation.
PS: It would be nice if you would add the hacktoberfest-accepted label to this issue if you find it worthy merging.

For sure it is, thank  you!


README link to remote debugging info is broken

When I visit https://jsconsole.com/remote-debugging.html (linked from README) I see:

it's not just that link, the whole remote debug is broken.

Not broken, removed. There's no replacement for the time being. Sorry.


chore(): add support of default translation in missingTranslationHandler

We can both define a default translation and a missing translation handler. In my case, I need to treat the default translation in the missing handler. This PR add the defaultTranslationText parameter to the handler.

Just a general question: Do you have several DefaultTranslations? And are they language dependent? And what do you do further with them after being in the result of the missingHandler?

I'm late!
My purpose is only to allow us to use both a missing handler and the default text.
I needed to add logic on default translation and return a modified version.
This PR gives more scope to the missingTranslationHandler.
I hope I answer your question. Thanks

@LoicMahieu thank you, I've picked this as 8c5044c.


Introducing template strings and strconcat function

This PR introduces the ability to define static strings inside a template ({{ }}). This feature allows for the passing of static values in functions.
{{ strconcat('Bearer ', input:token ) }}

Codecov Report

Merging #122 into master will decrease coverage by 0.14%.
The diff coverage is 43.75%.

@@            Coverage Diff             @@
##           master     #122      +/-   ##
==========================================
- Coverage   86.64%   86.50%   -0.15%
==========================================
  Files          89       89
  Lines        5144     5157      +13
==========================================
+ Hits         4457     4461       +4
- Misses        426      434       +8
- Partials      261      262       +1

Impacted Files
Coverage Δ

pkg/specs/template/template.go
85.48% <0.00%> (-14.52%)
:arrow_down:

pkg/lookup/lookup.go
89.67% <100.00%> (+0.27%)
:arrow_up:


Supported ability to consume jekyllFrontMatter as part of project properties

👀 @minasmart @mikkoh @tessalt
With this, we don't need to loop through all the files here to add the layout information to the top of every yuidoc generated files after they are generated.

This PR is part of the work in progress of improving Shopify/js-buy-sdk docs. See #Shopify/js-buy-sdk#202


Ignore editor's directories in .gitignore file

This pull request will add idea and vscode directories to .gitignore file.

Thanks, but these entries belong in the global .gitignore file instead.


[feature] Provide real-time updates #147

Tasks

[ ] Fix the failing tests
[ ] Add new tests

closes #147

Coverage decreased (-2.2%) to 96.998% when pulling a3356484566cfece664f607e9430c0bc4d5bcc1d on issues/147-provide-realtime-updates into f7422560c747641cf400bcc4ef1e3bdd23eea2db on master.


Updates to retraction status checker

[x]  Updated formatted citation with retraction status.
[x] Moved download_retracted_dataset to crossref.py
[x] Ignore downloaded retraction.csv

closes #366

ope didn't rebase, sorry!

Hi @geemi725  - open this into main now that we've done release


Typo

Ensure that the storage path ends with %username% to ensure that eah (!) user gets a unique folder.
Ensure that the storage path ends with %username% to ensure that each user gets a unique folder.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f735c74e-f4aa-9839-81fe-050782e09e22
Version Independent ID: 94ceabf5-70ec-8a81-6eee-8f9aa2671ad6
Content: Get Started with UE-V
Content Source: windows/configuration/ue-v/uev-getting-started.md
Service: unspecified
GitHub Login: @MaggiePucciEvans
Microsoft Alias: jdecker

Closing issue as fixed.


[BUG] Intermittent test failure RestApiComplianceAuditlogTest.testAutoInit

What is the bug?
Seeing intermittent test failures from org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testAutoInit [1]  The test passed on 5 of 6 platform/jdk combinations.
How can one reproduce the bug?
Unclear how to reproduce the bug, consult https://github.com/opensearch-project/security/actions/runs/3381396002/jobs/5615256769 for detailed failure information.
What is the expected behavior?
Tests should only fail when an error was introduced in the codebase, or network infrastructure was unavailable.
Do you have any additional context?
Impacted https://github.com/opensearch-project/security/pull/2224

This test was stabilized in https://github.com/opensearch-project/security/pull/3733


Use one floating window instead of two.

Description
As I described in https://github.com/neovide/neovide/issues/2375#issuecomment-1969194912
It is embarrassing to see two shadow floating windows in mutigrid
Neovim version
0.10
Expected behavior
one glass contain line number
Actual behavior
two glass
Minimal config
local plugins = {
  ts         = 'https://github.com/nvim-treesitter/nvim-treesitter',
  ts_context = 'https://github.com/nvim-treesitter/nvim-treesitter-context',
  -- ADD ADDITIONAL PLUGINS THAT ARE _NECESSARY_ TO REPRODUCE THE ISSUE
}

for name, url in pairs(plugins) do
  local install_path = '/tmp/nvim/site/'..name
  if vim.fn.isdirectory(install_path) == 0 then
    vim.fn.system { 'git', 'clone', '--depth=1', url, install_path }
  end
  vim.o.runtimepath = install_path..','..vim.o.runtimepath
end

-- ADD INIT.LUA SETTINGS THAT IS _NECESSARY_ FOR REPRODUCING THE ISSUE

Steps to reproduce

nvim --clean -u minimal.lua
...

No

In case someone have the same request, here is a fork version using one window
https://github.com/xzbdmw/nvim-treesitter-context

@lewis6991 Can you add more context? Is there any issue you foresee?
If someone has an implementation with 1 window I'd be happy to accept a PR.

We can accept a good PR if it doesn't add any complexity. The user issue is just that Neovide adds a shadow which I don't think is an issue enough for adding complexity.
I looked at the fork and all it does it prepend the line numbers to each line in the context window (with lots of unnecessary formatting changes), which is a broken way of doing this since it breaks highlighting. So we won't accept that as a PR.
We would need to use either inlay-extmarks, or try and pack the information into the signcolumn (somehow?). Both of those will introduce unnecessary complexity, so I'm not interested in exploring those.

I'd be happy to accept a PR.

Do you plan on maintaining it? I'm happy to see you back contributing to the project, but with respect, you seem to have a tendency to take long hiatuses.

Fair points, the tradeoff with complexity is also something that I'm wary of when receiving a PR.
Wdyt about using inlay hints for line numbers? Same window, same buffer, and afaict it wouldn't affect highlights.

Fairs point, I'm also wary of complexity when received a PR.
Why do you think inlay extmarks would introduce too much complexity? Doesn't sound like that much work. I don't have enough bandwidth to implement that but if someone is interested in doing that exploration, do you foresee any issue there?

Why do you think inlay extmarks would introduce too much complexity?

Inlay hints are a 0.10 feature so we would instantly drop support for 0.9, and it's likely (as with all new extmark features) that there's still probably some unknown issues with them. Using a second floating window is just closer to what we're trying to emulate, so we don't need to worry about any weird edge cases like alignment and such.
Arguably, Neovide should just be able to render this properly despite there being two windows, provided they have the same z-index.
Lots of plugins use floating windows for all sorts of stuff (e.g. https://github.com/dstein64/nvim-scrollview and https://github.com/lewis6991/satellite.nvim), so the shadow they are rendering is going to generally cause issues.
So really, I'm not convinced this is something we need to address.

Arguably, Neovide should just be able to render this properly despite there being two windows, provided they have the same z-index.

Valid argument. With that in mind I'm ok with treating this as a neovide issue rather than a bug here. Thanks for the comments.

We can accept a good PR if it doesn't add any complexity. The user issue is just that Neovide adds a shadow which I don't think is an issue enough for adding complexity.
I looked at the fork and all it does it prepend the line numbers to each line in the context window (with lots of unnecessary formatting changes), which is a broken way of doing this since it breaks highlighting. So we won't accept that as a PR.
We would need to use either inlay-extmarks, or try and pack the information into the signcolumn (somehow?). Both of those will introduce unnecessary complexity, so I'm not interested in exploring those.

I'd be happy to accept a PR.

Do you plan on maintaining it? I'm happy to see you back contributing to the project, but with respect, you seem to have a tendency to take long hiatuses.

Oh I see the broken highlight, and simply add gutterwidth to query's start col and end col fix that, I'm not familiar with highlight stuff so linenumber highlight simply using context window's one.


Support incremental reconfiguration

Right now in the yang branch we can statically configure an lwaftr process via a YANG-modelled configuration.  Great.  As part of this work we incorporated the binding table into the configuration, which is what our users expect; however, this removed the ability to reload just the binding table.  Indeed, the lwaftr no longer knows what file the configuration came from, so it can't do this on its own.
However the previous situation wasn't that great either.  You could reload the binding table, but you couldn't re-set the MTU or the next-hop IPv6 address or anything like that.  You just had to restart the application (which is fine enough of course).  But it would be nice to support more fine-grained reconfiguration.
So, that's what this bug is about: supporting reconfiguration of a running Snabb lwAFTR process that doesn't involve just restarting the whole thing.  First I wrote down a bit of design, then I see how the existing code works with this design and what's needed to get there.
Design
The basic design is going to be that we will add an app to the graph: the config app.  The config app will have the configuration for the program as a whole.  When it is run by the engine, it will see if it has any incoming requests to change the program configuration.  If there are requests in the queue (more on this later), then the config app will determine what changes are needed and then apply those changes.
Notably, this process also happens on program startup as well.  The config app has a function which maps the global configuration (as an instance of the lwAFTR YANG schema) to an app graph.  On program startup (the first breath), the initial app graph is the empty set; so the config app compiles a list of requests to make of the engine (app instantiations, links, unlinks, removals, and updates; more on updates below) that will take the engine from the old state to the new, and then performs those requests.
In a multi-process setup (https://github.com/snabbco/snabb/pull/1021) the change requests can be shipped over a back-channel from the main process where the config app runs, to the worker processes
where the changes are applied.  In a single-process setup the changes can be applied directly.
Manipulating the app graph is necessary for when (for example) you decide to add an ingress filter but you don't have a pflua app instantiated and linked.  However there are also other changes you might want that involve a configuration change in an existing app.  In general, the process of taking app configuration A and transforming it to app configuration A* can compile to an unlinking of that app, tearing it down, reinstantiating it, and relinking it.
However in some cases we can avoid this somewhat heavyweight operation.  In particular for the lwAFTR, a change from config A to A* where the only difference is in the binding table can proceed by just calling a method on the lwaftr.  So we will provide, for certain app types, the ability to compile some changes as method calls on apps.
Current status
The lwAFTR and a number of other apps already have a function that maps a configuration to an app graph; in the lwaftr it's in program/lwaftr/setup.lua.  So everything that we have to implement is generic except the bit that specializes compilation of lwaftr configuration changes.
The Snabb engine already implements a form of state tracking and change compilation -- see compute_config_actions and apply_config_actions in core.app.  So we just have to make it so that instead of the actions being applied in the main process, we apply them in the worker (in a multi-process scenario); and we need to handle reconfig in a more fine-grained way.  Currently compute_config_actions does a full lib.equal comparison to see whether two configs are the same (and thus to know whether to reconfig or keep an existing app), and that's not a good strategy for the binding table.
The current state of the actions applier thing also has the problem that compute_config_actions makes a changeset that is O(n) in the size of the graph, instead of O(n) in the size of the change.  This is because the actions effectively copy the old graph to the new graph instead of mutating the graph in place.  And, apply_config_actions has a lot of internal logic as well.  I think probably this pair of routines should do more compilation, such that apply_config_actions has less work to do and never has to check whether a step in its work plan is valid or not.  (cc @eugeneia and @lukego on this point)
To update a binding table entry, the reconfig application should probably just recognize change requests that can happen by a direct update on part of a YANG configuration.  In that case the manager process updates its config appropriately and issues a similar change request to the worker processes.
Implementation plan

[x] Make a config app that takes a schema name, a configuration file name (or a configuration object), and a function from configuration->app-graph, and arrange for it to populate the app graph on first breath.  (Confusingly, the low-level app graph concept is called a "configuration" also in snabb :/)

[x] Optimize core Snabb reconfiguration to minimize change lists.

[ ] Merge in the multiprocess work from https://github.com/snabbco/snabb/pull/1021 and separate the worker from the config app.  Probably we need to make an additional app in the worker to listen for change lists and apply them.  How to communicate big changes though?  Probably this backchannel needs to be a ring buffer mapped into both address spaces like the apps.lwaftr.channel but it needs a side storage area for arbitrary-sized configurations like the binding table.

Maybe we can compile each app configuration to a binary file, using the yang binary compiler, and then store them named SHA256.conf in the group directory where SHA256 is the hash of the binary?  At least that way we can know when big configurations change, which would be nice.  We'd need a way to serialize generic configurations though.  Perhaps Snabb apps could indicate their YANG model (for the app) and if not given you could get a "generic yang model" which has Lua data in some string representation that can be eval'd.  Dunno, an open question here.

[x] Make "snabb config" app listen to a UNIX socket or something for change requests.  One command on this socket should be "reload your configuration".  Another could be "get your configuration" which would print out the program configuration using the YANG schema.  Obviously this is not appropriate for running in the data plane at this point -- we need to be in a multiprocess world here.

Thinking about it more, there are two cases.  One is where you are just told to reload your configuration.  There, the protocol in the core will work well enough for us, especially after the recent optimizations in SnabbCo#1067.  The other is when you are told to make a change -- like "set external-interface.ip = <IP_ADDRESS>" or something.  In that second case there are two sub-cases :)  One is the generic one: you update your configuration directly, rerun your setup function to get an app graph, compute configuration actions that will take worker state from old to new, then apply the changes.  As before.  The other one is where you are able to know, given the small change, a corresponding small change to make to specific apps in the data plane.  This is domain-specific knowledge.  An example of this second case would be "add this binding table entry".
To implement this we should add a routine to the YANG schema support module that takes a change and compiles it to a special action type somehow that will result on a method being called on the app,or something like that.

@wingo: I like the concept of a config/setup app as part of a graph. Works well with the single threaded nature and leaves it up to each app to do the "right" thing if required.
Looking at your current work with lwaftr, merging binding table and configuration into one file, do you see lwaftr knowing how to act on a configuration change that goes beyond a simple value change? Like e.g. changing the graph by adding / removing apps?
Right now I have some intelligence in the control app for snabb, which signals a binding table change to snabb or restarts snabb. With a single configuration file this becomes a bit more complex, but I can see snabb config app taking ownership of such a decision.

@mwiget Interestingly I think snabb should handle non-simple changes well.  The special case is where you can just change one value, and we're currently working on a good solution for that, but it's been possible for a while to reconfigure the graph as a whole.  See paragraph 3 under current status and https://github.com/snabbco/snabb/issues/1067 for the core code.

Done via #576.


Focused cell after cursor left the SimPanel

When the cursor left the SimPanel, the the cell which was focused at the moment stays focused.

The same thing happens when changing the window via Alt+Tab.


Estimate radius at <100% when under-utilizing space

Problem
When running trin before storage hits the allocation limit, we currently run at 100% radius. If the steady state radius will land at 3% for a particular storage allocation, then 97% of the stored content we receive early on (in the period before radius starts to shrink) will end up being deleted. During that period, 97% of the network transfer, cpu for verification, and storage I/O is wasted.
Proposal
Pre-shrink the radius
We can't know the size of the network with arbitrary precision ahead of time, but we can bake in a rough guess to the binary. This rough guess can massively shrink the wasted data.
Asymmetry in the guess
The effect of incorrectly estimating the radius is highly asymmetrical:

don't preshrink the radius enough, and ... we're still much better off than before
preshrink the radius too much, and you never store as much as you are willing to store (without us adding a more complicated feature, to bump up the radius)

So we probably want to take our radius estimate and double it. We still get a plenty large benefit, with only a small downside when our baked-in numbers get stale over time (assuming it's quite rare for a network total storage to shrink).
Real example
Our current fleet of history nodes at steady state show a ~2% radius with 35 GB allocated each, which implies a 1.75 TB total network size.
If you launch a fresh trin instance for history with 17.5 GB of space, we can estimate that the radius will end up at 1%. With a 2x buffer, we could preshrink the radius to 2%. That means that we waste ~1% of total resources compared to the status quo of ~99%. (During the period before the radius shrinks below 100%) 99x gains are good. The less storage you allocate, the more there is to gain with this change. Those are the folks who probably care the most about performance.
History approaches the correct radius relatively quickly in practice, but state takes a while, and is more resource-intensive. So state is probably where we will feel the benefit of this the most.

This is another one that will probably cause hive test failures.  A way around this would be to have this value be able to be configured at the CLI so that in hive tests it can effectively be disabled.  Seems potentially as simple as something like --history.total_network_data=600_000.  This would allow us to take the storage limit they provided (or the default) and extrapolate the appropriate starting radius.  For hive, we might also want to introduce something like --history.max_allowed_radius=auto as the default, which means it will extrapolate the value from the total network size.  In hive, we can then do --history.max_allowed_radius=100 to have it do the default thing it does now and start the radius at 100%.

Note that this approach suffers from the problem of  growth over time making it less accurate as time goes on but in practice my guess is that this will be a non-issue.  We could choose a more complex metric to account for this like --history.network_growth_rate but I would be surprised if the complexity of such an approach was worth it.  We'll be making releases regularly and we can update


[GSoC 2022 Proposal] Reworking the Rough Edges

Whereas applicants are typically excited about contributing more code through GSoC, my proposal is about addressing some of the existing technical debt. I am posting it here to get some feedback from the community.
Background
The probml/probml-notebooks repository has witnessed a lot of contributions, presumably because one of the prerequisites of applying to the JAX ML Textbook project is to commit at least one pull request. However, crowd-sourcing can result in notebooks of varying character and inconsistent style, and I argue that there is still a lot of work to be done to ensure the quality of these notebooks.
The Rough Edges
Unclear objective
I am not sure how the notebooks are supposed to be used. Typically notebooks are self-contained pieces with sufficient narration to walk the reader through the code, but the D2L notebooks barely contain any commentary. If we don't want to follow the path of literal programming, then why not use plain .py files which are much easier imported from? In addition, I suppose they are meant as companion notebooks for a textbook, but it is not clear to me which part of the narration should go into the book, and which should go into the notebooks.
JAX with a PyTorch accent
I believe most recent contributions come from https://github.com/probml/pyprobml/issues/686, which asks us to translate a lot of PyTorch notebooks to JAX. The problem is that JAX calls for a fundamentally different way of thinking (JAX: stateless, functional, JIT; PyTorch: stateful, object-oriented, backend written in C++), so our word-by-word translation is essentially "writing PyTorch in JAX". One can easily tell the notebooks are far from idiomatic by comparing them against the official examples. This is probably not a big issue for sophisticated programmers, but we should teach beginners how to write pure JAX, as opposed to JAX with a PyTorch accent.
Inconsistency
Inconsistency can manifest itself in many ways. First, the notebooks originally written for "Dive Into Deep Learning" may not suit the need for our textbook well, and using them without change introduces consistency issues between the textbook and the notebooks. Secondly, different contributors also have their own preferred style, which is often different from each other, and readers might be distracted by the different approaches to address the same problem. For instance, all notebooks should ideally use a similar training loop to lower the cognitive load.
Moreover, almost all notebooks contain identical data loading, prepossessing, and plotting code, and the JAX training loop in the notebooks also shares more duplicate code than necessary. These components could divert readers from the key points unless we extract them to separate scripts and import from them. Inconsistency across notebooks may raise if we are not careful in the process.
Minor issues
A number of notebooks contain leftovers from their PyTorch counterpart. Most notably, almost all notebooks contain necessary import statements. As another example, the Reshape class in notebooks-d2l/lenet_jax.ipynb becomes redundant when the code is translated to JAX. Most cross-reference URLs in the notebooks are also broken and require attention. Finally, this repository is not extremely well organized, with all notebooks living in a few flat namespaces. Switching to a finer directory structure would make things tidier and easier to navigate.
Proposal
I believe the best way to solve this problem is to rewrite all D2L notebooks from scratch, because as I stated above, there are so many issues to be fixed that fixing the existing ones won't be easier. That being said, the existing notebooks can be kept aside and used as a reference. An example of the notebooks I intend to deliever is the multi_gpu_training_jax.ipynb notebook I contributed a few days ago. Note that it barely shares anything with the original D2L notebook.
Reworking the notebooks is a lot of work, and the contributors are best restricted to a small, highly-coordinated group, or preferably an individual, to ensure consistency in style and programming paradigm. I am not sure if @murphyk has sufficient time and energy to embrace the grind, but I am willing to take it up. Along the path, I will consult the mentors for their preferences and create a style guide for future contributors (e.g. use of jax.xmap). Additionally, I will keep the need for our textbook in mind and make some adjustments accordingly during the renovation. For example, I could remove some nonessential code to make the notebooks more focused, add some type annotations and assert to make the code more concrete, or write more commentary to make them more self-contained.

Programs must be written for people to read, and only incidentally for machines to execute.
-- Harold Abelson

This is especially true for reference implementations and demos to appear in a textbook. I will do my best to produce correct, clear, and consistent notebooks for pedagogical use.
Questions
I have a few questions when preparing my proposal

First of all, are you interested in this project?
Apart from D2L, which part of the repository do you want me to rework first? (I'm a graduate student in Statistics from UChicago, which means I'm pretty comfortable with both coding and math!)
How are you going to use these notebooks? I need to know your goal to define the milestones and deliverables.

This sounds great. I can discuss details with you privately. Meanwhile
please apply to GSOC since deadline is approaching this Friday.
I am AFK but will look at your PRs later.
On Mon, Apr 11, 2022 at 9:44 AM Qingyao Sun @.***>
wrote:

Whereas applicants are typically excited about contributing more code
through GSoC, my proposal is about addressing some of the existing
technical debt. I am posting it here to get some feedback from the
community.
Background
The probml/probml-notebooks https://github.com/probml/probml-notebooks
repository has witnessed a lot of contributions
https://github.com/probml/probml-notebooks/graphs/commit-activity,
presumably because one of the prerequisites of applying to the JAX ML
Textbook project is to commit at least one pull request. However,
crowd-sourcing can result in notebooks of varying character and
inconsistent style, and I argue that there is still a lot of work to be
done to ensure the quality of these notebooks.
The Rough Edges Unclear objective
I am not sure how the notebooks are supposed to be used. Typically
notebooks are self-contained pieces with sufficient narration to walk the
reader through the code, but the D2L notebooks barely contain any
commentary. If we don't want to follow the path of literal programming,
then why not use plain .py files which are much easier imported from? In
addition, I suppose they are meant as companion notebooks for a textbook,
but it is not clear to me which part of the narration should go into the
book, and which should go into the notebooks.
JAX with a PyTorch accent
I believe most recent contributions come from probml/pyprobml#686
https://github.com/probml/pyprobml/issues/686, which asks us to
translate a lot of PyTorch notebooks to JAX. The problem is that JAX calls
for a fundamentally different way of thinking (JAX: stateless, functional,
JIT; PyTorch: stateful, object-oriented, backend written in C++), so our
word-by-word translation is essentially "writing PyTorch in JAX"
https://dl.acm.org/doi/pdf/10.1145/1039511.1039535. One can easily tell
the notebooks are far from idiomatic by comparing them against the official
examples https://github.com/google/jax/tree/main/examples. This is
probably not a big issue for sophisticated programmers, but we should teach
beginners how to write pure JAX, as opposed to JAX with a PyTorch accent.
Inconsistency
Inconsistency can manifest itself in many ways. First, the notebooks
originally written for "Dive Into Deep Learning" may not suit the need for
our textbook well, and using them without change introduces consistency
issues between the textbook and the notebooks. Secondly, different
contributors also have their own preferred style, which is often different
from each other, and readers might be distracted by the different
approaches to address the same problem. For instance, all notebooks should
ideally use a similar training loop to lower the cognitive load.
Moreover, almost all notebooks contain identical data loading,
prepossessing, and plotting code, and the JAX training loop in the
notebooks also shares more duplicate code than necessary. These components
could divert readers from the key points unless we extract them to separate
scripts and import from them. Inconsistency across notebooks may raise if
we are not careful in the process.
Minor issues
A number of notebooks contain leftovers from their PyTorch counterpart.
Most notably, almost all notebooks contain necessary import statements.
As another example, the Reshape class in notebooks-d2l/lenet_jax.ipynb
https://github.com/probml/probml-notebooks/blob/61d818c5e7475149b6e0384eaa655ee8598dea72/notebooks-d2l/lenet_torch.ipynb
becomes redundant when the code is translated to JAX. Most cross-reference
URLs in the notebooks are also broken and require attention. Finally, this
repository is not extremely well organized, with all notebooks living in a
few flat namespaces. Switching to a finer directory structure would make
things tidier and easier to navigate.
Proposal
I believe the best way to solve this problem is to rewrite all D2L
notebooks from scratch, because as I stated above, there are so many issues
to be fixed that fixing the existing ones won't be easier. That being said,
the existing notebooks can be kept aside and used as a reference. An
example of the notebooks I intend to deliever is the
multi_gpu_training_jax.ipynb notebook
https://github.com/probml/probml-notebooks/pull/77 I contributed a few
days ago. Note that it barely shares anything with the original D2L
notebook
https://github.com/probml/probml-notebooks/blob/main/notebooks-d2l/multi_gpu_training_torch.ipynb
.
Reworking the notebooks is a lot of work, and the contributors are best
restricted to a small, highly-coordinated group, or preferably an
individual, to ensure consistency in style and programming paradigm. I am
not sure if @murphyk https://github.com/murphyk has sufficient time and
energy to embrace the grind, but I am willing to take it up. Along the
path, I will consult the mentors for their preferences and create a style
guide for future contributors (e.g. use of jax.xmap). Additionally, I
will keep the need for our textbook in mind and make some adjustments
accordingly during the renovation. For example, I could remove some
nonessential code to make the notebooks more focused, or add more
commentary to make them more self-contained.
Programs must be written for people to read, and only incidentally for
machines to execute.
-- Harold Abelson
This is especially true for reference implementations and demos to appear
in a textbook. I will do my best to produce correct, clear, and consistent
notebooks for pedagogical use.
Questions
I have a few questions when preparing my proposal

First of all, are you interested in this project?
Apart from D2L, which part of the repository do you want me to
rework first? (I'm a graduate student in Statistics from UChicago, which
means I'm pretty comfortable with both coding and math!)
How are you going to use these notebooks? I need to know you goal
to make a plan.

—
Reply to this email directly, view it on GitHub
https://github.com/probml/probml-notebooks/issues/79, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDK6EFOZ357FONZNDBDPLDVERJG3ANCNFSM5TDZIQAQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Sent from Gmail Mobile


Create a guide for understanding the different ROS documentation sites

This document seeks to be a single source for the status and offerings of our ROS documentation.

Should we add anything here for Gazebo?

I changed the PR to use links between sections instead of links to the relevant sites when mentioning other sections. I think this makes it easier to jump around and understand the relationships between our various sites. Each section heading still has a link to the relevant site (e.g., ROS docs goes to docs.ros.org).

A couple of larger questions about this document:

it overlaps with some of the stuff in https://github.com/ros2/ros2_documentation/blob/rolling/source/Contact.rst .  I'm not quite sure whether we should combine the two of them.
I'm not quite sure if this qualifies as a "Guide".  It is not a problem -> solution type of document.  I think it more properly belongs as part of "Contact" or "Contributing", though I lean more towards Contact.  Thoughts?

Whoops, I didn't see this. I'll think about these points and make some changes.

it overlaps with some of the stuff in https://github.com/ros2/ros2_documentation/blob/rolling/source/Contact.rst .  I'm not quite sure whether we should combine the two of them.

I see two spots where there is some overlap.

We direct them to ROS answers for questions.
We mention that discussion is happening on Discourse.

Both of these seem quite natural on the contact page. I'm hesitant to combine them directly with this document, as I think much of the document I've made in this PR isn't related to getting in touch with us or our community, and I think it is nice to have a short focused document on how to get in touch, like the contact page.
I've created links to the contact page from the document that I made. I think this might be the best of both worlds: it keeps the focus of both documents and links from the minimal page I have created to more information about how to use ROS Answers and ROS Discourse. What do you think?

I'm not quite sure if this qualifies as a "Guide".  It is not a problem -> solution type of document.  I think it more properly belongs as part of "Contact" or "Contributing", though I lean more towards Contact.  Thoughts?

I'm not sure. My impression is that it should be its own page somewhere. I chose "guide," because I see it as a high-level guide on how our sites are organized. Looking at the existing guides, I am not sure if it is so different in spirit, but I'm open to other places to put it.
I lean towards not lumping this document with the "Contact" or "Contributions," because it has some information that is relevant to each (Discourse and Answers, for Contact; and ROS Docs, for Contributions) but I believe a good bit that is not as relevant. I also prefer not to split it up as then it is no longer a central spot for this information.

2\. I'm not quite sure if this qualifies as a "Guide".  It is not a problem -> solution type of document.  I think it more properly belongs as part of "Contact" or "Contributing", though I lean more towards Contact.  Thoughts?

While this isn't a problem -> solution type of document, it does prescribe to users how they should use the documentation resources. Also, I don't know if I'd look for this information in the Contact/Contributing sections. This isn't a strong opinion though because, again, it doesn't quite qualify as a guide.

@clalancette, I updated the location to the top level and placed it after installation, as per our conversation earlier today.


Node-Local API: listener

We want to listen for node management RPCs on every node, where each node terminates its own requests (no backhaul between nodes, the client has to dial the correct node they wish to introspect).
This should be a separate Proto/gRPC Service on a separate port. It should use the same PKI/CA as the curator listener.

APIs to implement:

[ ] https://github.com/monogon-dev/monogon/issues/113


feat: Initial version

Your new script is ready to be implemented :tada:
You can run it locally on your machine with
node cli.js

Once you are happy with it, push your changes to this pull request
git commit -a -m "feat: initial version"
git push

Before merging:

[x] Install the Octoherd GitHub App
[x] Add NPM_TOKEN to the repository secrets. Set it to an npm automation token

WIP, I have to go to sleep now but would it be something you would like to try for Octokit?
context: https://github.blog/changelog/2021-07-02-github-actions-setup-node-now-supports-dependency-caching/

Can you pass it through prettier before writing the file? That could solve the issue with indentation

Can you pass it through prettier before writing the file? That could solve the issue with indentation

Running npx prettier --parser yaml <file> --write does not convert sequences to arrays since they are both YAML valid and equivalent: https://stackoverflow.com/a/33136212/3237248
Tried using yaml library (the one used by prettier) and no luck neither. I'm not sure if there is a cool way to parse/dump yaml files and not lose data/format in the process. Posting a issue/discussion to js-yaml.

Do you think this could be interesting for Octokit and other orgs? @gr2m

Absolutely! I just talked about this exact thing with @travi