Applying dirichlet BC conditions in example 18

Hello all,
I am trying to apply dirichlet BC condiitons to the DG discretization of Euler equations using the basic template of ex18.
I modified example18 using the following process.
a) I changed the mesh from periodic to "inline-quad.mesh"
b) I populated the array attributes vector array<int> attr(pmesh.bdr_attributes.Max()); appropriately for each deg of freedom (4 or 5 depending on the no of spatial dimensions.)
c) I also populate a vector for each deg of freedom which contains attr and Coefficient* coeff (a function to evaluate the value on the boundary depending on the coordinate.)
d) 	Then I Get the essential dof from the finite element space containing all deg of freedom ( vfesin the example)
finiteElementSpace->GetEssentialTrueDofs(density_ess_attr_, density_ess_tdof_, 0);
finiteElementSpace->GetEssentialTrueDofs(momentumX_ess_attr_, momentumX_ess_tdof_, 1);
finiteElementSpace->GetEssentialTrueDofs(momentumY_ess_attr_, momentumY_ess_tdof_, 2);
finiteElementSpace->GetEssentialTrueDofs(energy_ess_attr_, energy_ess_tdof_, 3);

I am right now a bit confused as to where I should really provide these data structures in the finite element space in example18.
Any help on where I should supply these data structures would be much appreciated.

You could take a look at issue #1636, where boundary conditions for the Sod shock tube problem were discussed. Generally speaking, the boundary conditions in this example should be enforced weakly by modifying the boundary integrators, and not by enforcing "essential" boundary conditions.

Thank you very much Will for your response.  I am a new user of mfem and still a bit confused about the bdr_attr strategy.  I have two follow up questions.

I understand from reading issue #1636 that weak imposition of the boundary conditions can be done using face integrators.  The boundary markers (Array<int> &bdr_marker) specified in the AddBdFaceIntegrator method, are they specific to each deg of freedom ? I anticipate having different dirichlet BC on different boundary attributes for each deg of freedom.

I am asking this because I am trying to create a generic way to apply BC for different cases, sod shock would be one of them.

Strong imposition of boundary conditions should also be an option right, if that is the route I want to take, how should I do it? Even if this may induce some oscillations near the boundary.  I feel that this should also be an option, then I can compare weak and strong imposition of BC.

If there is an example for a codebase I can look at which does Dirichlet BC for Euler that would be great. Even if there is an example where DG method is used with multiple deg of freedom and Dirichlet BC are used, that would be great.

I understand from reading issue #1636 that weak imposition of the boundary conditions can be done using face integrators.  The boundary markers (Array<int> &bdr_marker) specified in the AddBdFaceIntegrator method, are they specific to each deg of freedom ? I anticipate having different dirichlet BC on different boundary attributes for each deg of freedom.

The bdr_marker array is used to specify the different parts of the boundary where you want to enforce different conditions. These are not per degree of freedom, but rather associated with different geometric subsets of the mesh boundary.
I'm not sure what you mean by having different boundary conditions for each degree of freedom. It would be helpful if you included more specific information about your problem.

Strong imposition of boundary conditions should also be an option right, if that is the route I want to take, how should I do it? Even if this may induce some oscillations near the boundary.  I feel that this should also be an option, then I can compare weak and strong imposition of BC.

You can of course use the list of DOFs returned by GetEssentialTrueDofs to set the grid function to strongly satisfy your given BCs, but I do not recommend doing this with DG discretizations.

If there is an example for a codebase I can look at which does Dirichlet BC for Euler that would be great. Even if there is an example where DG method is used with multiple deg of freedom and Dirichlet BC are used, that would be great.

I agree this would be useful (and such an example probably exists), but I don't have a reference off-hand.

Hello Will,  Thanks for your response.

The bdr_marker array is used to specify the different parts of the boundary where you want to enforce different conditions. These are not per degree of freedom, but rather associated with different geometric subsets of the mesh boundary.

I understand now that the whole flux vector is adjusted for a particular boundary condition when being weakly imposed, so I understand now what bdr_marker should contain, it contains the markers where the boundary integral needs to be computed.

I'm not sure what you mean by having different boundary conditions for each degree of freedom. It would be helpful if you included more specific information about your problem.

To help with the questions, I am trying to reproduce the implementation mentioned in issue #1636.  So the SOD shock problem with inline-quad mesh.  I am a bit confused on the implementation of the boundary integrator.   If I understand correctly I have to compute the flux corrected for the dirichlet boundary condition on the boundary.  I am confused because AssembleFaceVector which is the method I am overriding takes two elements as input, I am not sure which two elements are these if we are on the boundary? Maybe I am missing something here. If I understand correctly one needs to implement (F.n, w) on the boundary where F is the corrected for the Dirichlet BC using the Riemann solver, this requires only one boundary element.  I am not able to implement this as AssembleFaceVector takes two elements.  Also the decision to compute Ur or Ul probably depends on the normal right?
On the other had if I try to implement (F.n,[w]) like the interior face terms just by modifying interior face integrator, with corrected flux adjusted for Dirichlet BC, when I did this I run into asserts in AddElementVector with the following error
Assertion failed: (dofs.Size() == elemvect.Size()) is false:
 --> Size mismatch: length of dofs is 16, length of elemvect is 32
 ... in function: void mfem::Vector::AddElementVector(const mfem::Array<int>&, const mfem::Vector&)
 ... in file: /home/makrand/Packages/blastfemdg/external/mfem/linalg/vector.cpp:575

You can of course use the list of DOFs returned by GetEssentialTrueDofs to set the grid function to strongly satisfy your given BCs, but I do not recommend doing this with DG discretizations.

It would be great if I could still implement this, as I need to compare the two options.  Could you tell me a method I could use the list of DOFs returned by component wise GetEssentialTrueDofs.

I agree this would be useful (and such an example probably exists), but I don't have a reference off-hand.
It would be great If you could let me know about this or maybe refer to someone who know about such an example/codebase.
I really think an example would be very beneficial on applying Dirichlet BC in example 18 as there are many people asking these questions, but that is just a suggestion.

I appreciate your time and help.

To help with the questions, I am trying to reproduce the implementation mentioned in issue #1636. So the SOD shock problem with inline-quad mesh. I am a bit confused on the implementation of the boundary integrator. If I understand correctly I have to compute the flux corrected for the dirichlet boundary condition on the boundary. I am confused because AssembleFaceVector which is the method I am overriding takes two elements as input, I am not sure which two elements are these if we are on the boundary? Maybe I am missing something here. If I understand correctly one needs to implement (F.n, w) on the boundary where F is the corrected for the Dirichlet BC using the Riemann solver, this requires only one boundary element. I am not able to implement this as AssembleFaceVector takes two elements. Also the decision to compute Ur or Ul probably depends on the normal right?

You are correct, AssembleFaceVector takes in two elements. In the case of an interior face, these correspond to the two neighboring elements containing the given face. In the case of a boundary face, there is only one element containing the given face, and so the second element is just a dummy element pointing to the first one.
Typically for the Euler equations, AssembleFaceVector should compute an (approximate) solution to the Riemann problem involving the states on either side of the face. This can be done using e.g. the Lax-Friedrichs method (this is probably the simplest/easiest, and is implemented in ex18 in the RiemannSolver class, but you can also use Roe, or an exact Riemann solver, etc.
A typical way of handing the boundary conditions is to provide a fictitious external state to the Riemann solver. In other words, if the interior state is denoted uL, then you can specify an external state uR, and return the result Fhat(uL, uR, n), where Fhat corresponds to the Riemann solver. It is probably sufficient in your case to set uR to be the farfield state.

On the other had if I try to implement (F.n,[w]) like the interior face terms just by modifying interior face integrator, with corrected flux adjusted for Dirichlet BC, when I did this I run into asserts in AddElementVector with the following error
Assertion failed: (dofs.Size() == elemvect.Size()) is false:
 --> Size mismatch: length of dofs is 16, length of elemvect is 32
 ... in function: void mfem::Vector::AddElementVector(const mfem::Array<int>&, const mfem::Vector&)
 ... in file: /home/makrand/Packages/blastfemdg/external/mfem/linalg/vector.cpp:575

Are you sure this is on the boundary? On the boundary, elemvect should contain the residual corresponding to one element, and on an interior face, elemvect should contain the residual corresponding to the two neighboring elements. Since in this case your vector is twice as long as the number of DOFs, it looks like this is an interior face.

It would be great if I could still implement this, as I need to compare the two options. Maybe this is really straightforward but because I am new to mfem I am not sure which function needs to be used and on where to apply the BC after GetEssentialTrueDofs is computed.
Could you tell me a function I could use the list of DOFs returned by component wise GetEssentialTrueDofs?

Actually, I realize that this is a bit harder to implement in MFEM than my previous message indicated. Functions like GetEssentialTrueDofs and GridFunction::ProjectBdrCoefficient will not work properly with DG spaces. In particular, with DG you can choose to use the Gauss-Legendre basis, which does not have nodes lying on the boundary, so the projection becomes nontrivial. If you use Gauss-Lobatto basis, you can set the boundary DOFs, but this might require some manual effort.

I fixed the assert problem the size of elvect was incorrectly set.  I am adding code here so it may help other users looking at the discussion too.  Replicating and modifying the interior face integrator in the example. I implemented the Boundary integrator as follows:
void BoundaryFaceIntegrator::AssembleFaceVector(const FiniteElement &el1,
                                                const FiniteElement &el2,
                                                FaceElementTransformations &Tr,
                                                const Vector &elfun, Vector &elvect) {
	// Compute the term <F.n(u),[w]> on the interior faces.
	const int dof1 = el1.GetDof();
	const int dof2 = el2.GetDof();

	shape1.SetSize(dof1);
	shape2.SetSize(dof2);

	elvect.SetSize((dof1) * inputData_->num_equation);
	elvect = 0.0;

	DenseMatrix elfun1_mat(elfun.GetData(), dof1, inputData_->num_equation);
	DenseMatrix elfun2_mat(elfun.GetData() + dof1 * inputData_->num_equation, dof2,
	                       inputData_->num_equation);

	DenseMatrix elvect1_mat(elvect.GetData(), dof1, inputData_->num_equation);
	DenseMatrix elvect2_mat(elvect.GetData() + dof1 * inputData_->num_equation, dof2,
	                        inputData_->num_equation);

	// Integration order calculation from DGTraceIntegrator
	int intorder;
	if (Tr.Elem2No >= 0)
		intorder = (std::min(Tr.Elem1->OrderW(), Tr.Elem2->OrderW()) + 2 * std::max(el1.GetOrder(), el2.GetOrder()));
	else {
		intorder = Tr.Elem1->OrderW() + 2 * el1.GetOrder();
	}
	if (el1.Space() == FunctionSpace::Pk) {
		intorder++;
	}
	const IntegrationRule *ir = &IntRules.Get(Tr.GetGeometryType(), intorder);

	/// loop over integration points
	for (int i = 0; i < ir->GetNPoints(); i++) {
		const IntegrationPoint &ip = ir->IntPoint(i);

		Tr.SetAllIntPoints(&ip); // set face and element int. points

		// Calculate basis functions on both elements at the face
		el1.CalcShape(Tr.GetElement1IntPoint(), shape1);
		el2.CalcShape(Tr.GetElement2IntPoint(), shape2);

		// Interpolate elfun at the point
		elfun1_mat.MultTranspose(shape1, funval1);
		elfun2_mat.MultTranspose(shape2, funval2);

		// Get the normal vector and the flux on the face
		CalcOrtho(Tr.Jacobian(), nor);

                // Calculates funval2 as 2D - funval1, where D is the dirichlet state.
		this->calcDirichletState(funval1, funval2);

                /// Calculate the Fhat using Reimann solver
                const double mcs = rsolver.Eval(funval1, funval2, nor, fluxN);

		// Update max char speed
		if (mcs > inputData_->max_char_speed) { inputData_->max_char_speed = mcs; }

		fluxN *= ip.weight;
		for (int k = 0; k < inputData_->num_equation; k++) {
			for (int s = 0; s < dof1; s++) {
				elvect1_mat(s, k) -= fluxN(k) * shape1(s);
			}
			for (int s = 0; s < dof2; s++) {
				elvect2_mat(s, k) += fluxN(k) * shape2(s);
			}
		}
	}
}

This is still calculating (F.n, [w]) at the boundary.  Do you think this is correct.  I am not sure whether I should modify this to (F.n,w) on the boundary.
I call this in main like this, assuming only application of BC on the left boundary
Array<int> ess_bdr(pmesh.bdr_attributes.Max());
	ess_bdr = 0;
	ess_bdr[3] = 1;
	//ess_bdr[2] = 1;
	std::cout<< "ess_bdr is:  " << ess_bdr[0] << ", " << ess_bdr[1] << ", " << ess_bdr[2] << ", " << ess_bdr[3] <<
	std::endl;
	A.AddBdrFaceIntegrator( new BoundaryFaceIntegrator(rsolver, dim, &inputData, &bc), ess_bdr);

Is this correct ?? Here &inputData and &bc are references to custom objects which carry some information as I am trying to write a general framework to handle multiple cases.
I am still running into nans when I use this.

Actually, I realize that this is a bit harder to implement in MFEM than my previous message indicated. Functions like GetEssentialTrueDofs and GridFunction::ProjectBdrCoefficient will not work properly with DG spaces. In particular, with DG you can choose to use the Gauss-Legendre basis, which does not have nodes lying on the boundary, so the projection becomes nontrivial. If you use Gauss-Lobatto basis, you can set the boundary DOFs, but this might require some manual effort.

How do I change to Gauss Lobatto basis probably in the constructor on finiteElementspace? I did not know about this function GridFunction::ProjectBdrCoefficient, Is there an example I can look at for how to apply these BC for multiple dof and uses GridFunction::ProjectBdrCoefficient to apply specific values on the boundaries?
I am pursuing both options because I want to give users both options weak and strong imposition and I want to do stability analysis and convergence analysis with both weak and strong imposition.
Thank you very much for your help. I really appreciate it.

Hello Will @pazner,  I was able to implement a general class for handling various boundary conditions weakly using this guidance. Thanks for your help on this.
I would like to bring one issue to your notice which might be useful to other users.  The Array<int> boundaryMarkers is not owned by NonlinearForm class so this array has to stay in scope all the time as the boundary conditions may get applied multiple times (every timestep if the bc is time dependent).
When I was writing a class to abstract out the setup of weak boundary conditions I had to declare Array<Array<int>> boundaryMarkers; object in the main which I populated inside the method of this class as follows,
BoundaryConditions bc(&pmesh, &inputData, &A, &rsolver, verbose);
Array<Array<int>> boundaryMarkers;
bc.setupBoundaryConditions(boundaryMarkers);

as opposed to having a data member inside BoundaryCondtions as it would go out of scope.  It is cleaner if the boundaryMarkers is a member of BoundaryConditions class because then the main of the code is cleaner.
I am not sure why boundaryMarkers is not owned by NonlinearForm class.  Could one of you let me know the reason for this?
I just wanted to let the users know about this, as they might make the same mistake as me, and let it go out of scope producing undefined behaviour.
I am still not sure how to add the boundary conditions strongly but maybe that is a discussion for a different post.
Thanks Will @pazner again for your help on this.


10% of audio data is duplicated during preprocessing

https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/assets/66376113/ccc991bc-f614-4753-a533-ab0afdd017cf
If you check the preprocessed wavs folder before doing feature extract, you can notice that they are not properly aligned when put up next to each other back to back. This applies for about the last ~0.3s, so it's duplicating roughly 10% of the dataset unnecessarily, slightly worsening the quality of the model.

On top of this, SOVITS training used to require you to do this processing by hand, and the standard at the time was to use 10s portions instead of ~4 second portions (which is what RVC currently does.) I would recommend boosting this to 10s, since this may have potential gains for model training speed? Or maybe a low value is better. I wouldn't know. Maybe add an option to split files?

Using duplicate is right. The duration training audio-clip of 40k setting is 0.32s. If you don't set the duplicate part, the trainingset will be decrease, because the data across two parts can't be leanrt.
A longer segment duration may not necessarily achieve better results, but it will definitely increase the usage of graphics memory, which makes it difficult for many people using low-end graphics cards to train. The configuration of Sovits may not always be optimal.

Wouldn't splitting the portions based on detected silent gaps between words be a more ideal way to do that? If the point is to preserve continuity, why not just split based on when speech naturally ends rather than a solution that seems more 'rough' in approximating features (and likely introduces more duplicate data bias?)

At present, the data normalization strategy of "rvc" is to force slicing again after being sliced by "auto slicer", which will ultimately cut the dataset into multiple segments of about 3 seconds. Increasing the threshold of forced slicing will significantly increase the memory usage.


Sort by Multiple Columns

Hi,
Looking at the API docs, it seems that Airtable should be able to support sorting via multiple columns via the API:

A list of sort objects that specifies how the records will be ordered. Each sort object must have a field key specifying the name of the field to sort on, and an optional direction key that is either "asc" or "desc". The default direction is "asc".

But I'm noticing that the records method (among others) in lib/airtable/table.rb seems to hard-code sort at one parameter:
    def records(options={})
      options["sortField"], options["sortDirection"] = options.delete(:sort) if options[:sort]
      results = self.class.get(worksheet_url, query: options).parsed_response
      RecordSet.new(results)
    end

I've tried simply commenting out the first line of that method and passing my own sort object (ie sort: [{field: 'fieldName', direction: :asc}], but likely due to my unfamiliarity with HTTParty or the Airtable API, I can't figure out how to pass multiple sort objects through the records method.
Basically: I would have submitted a pull request to adjust this method to allow multiple sort, but I can't figure it out :/ Hopefully you know more than me on the topic.
Thanks!

I haven't looked in the ruby code, but it seems that sort was an array with field and direction.
The sort should be json object like this:
[{"field": "Item", "direction": "desc"}]

Thanks, yes that appears to be the format for sorting by a single field. What I'm trying to figure out is how to sort by multiple fields.
For example, I want to sort a list of students by dormitory, then name, via the API. The best I've been able to come up with is to use a view in Airtable and set that view with a default multi-column sort. However that approach that doesn't work well when the data gets out-of-order (ie comes in randomly from a form) because I have to manually sort the form in Airtable before accessing it via my app. So I'd love to be able to sort multi-column directly via the API, I just can't figure out how to do it :/

@dmerand can you try sortBy branch to see if that works for you?

I tried the sortBy branch in my code, and it's working great! Thanks for taking the effort to make this update. It's very cool to be able to work directly on the returned object, rather than convert it to an array + sort it, which was my previous approach.

Could the sortBy branch be merged and the gem be offered to everyone?  There's lots of us needing the multiple key sorting.

...I think I was able to install the gem from GitHub from sortBy branch, but I cannot figure out how to use multiple keys.  Have tried so far:
@rec = @tbl.records(:sort => [["key1", :asc], ["key2", :asc]])
Could you give an example on how to use the multiple key sorting properly?

@Etsija try sortByTake2 branch - it now handles :asc being a symbol.


feature: rework the Key struct in the placement code (or equivalent for PlacementResolver)

Feature Description
The use of the Key struct in the PlacementResolver to identify objects is adding a lot of GVK <-> GVR conversion overhead, which:

Complicates code
Creates instances of erroring where can be avoided
Introduces validation issues when these mappings go out of sync

Initially, the use of Key was for being as close as possible to the existing code (under some efficiency constraints), and a rework was planned for after initial merge.
During the review of #1681, @MikeSpreitzer repeatedly raised the point that things can be better if the resolver worked with resource instead of kind.
Proposed Solution
Rework placement's Key or introduce a new one for the new PlacementResolver.
Want to contribute?

[ ] I would like to work on this issue.

Additional Context
No response

More specifically, the issue is that centering the internal data structures on "kind" rather than "resource" is a mismatch with the most commonly used library fuctions --- the discovery and dynamic libraries are in terms of "resource".
However, centering the internal data structures on "resource" rather than "kind" will probably not completely remove the need for mapping between the two, because this is a fundamental pain point in Kubernetes. The representation of an API object uses "kind" rather than "resource".

The representation of an API object uses "kind" rather than "resource".

Yes. That was the motivation to use GVK as key for informers/listers. The idea was to do the GVR->GVK conversion only at informers creation time and then use the GVK (available in the object) when looking up the informer for the object. If the key is moved to be based on GVR, we need to convert GVK -> GVR for every update/create/delete of objects.


Fix tickInterval

This was broken at some point (perhaps since the advent of 3.1?) by faulty logic. This addresses that and fixes it.
Co-authored by: @nelsonsbrian

That makes sense -- I went ahead and changed it to still do some defensive
coding without redundantly setting it if it's already zero.
On Wed, Jun 16, 2021 at 7:56 PM Brian Nelson @.***>
wrote:

@.**** commented on this pull request.
In src/EffectList.ts
https://github.com/Ranvier-TS/core-ts/pull/20#discussion_r653128932:

  		) {

  			continue;
  		}
  		effect.state.lastTick = now;

		effect.state.ticks && effect.state.ticks++;

		if (!effect.state.ticks) {

I was just pointing out that most likely the state.ticks would be 0
(unless was set differently) from the Effect constructor. Didn't know if
you had seen that.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Ranvier-TS/core-ts/pull/20#discussion_r653128932, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AB6GX6JU2VZY3N2N2BED4W3TTE2ZRANCNFSM462KXDOQ
.


Hide Prices

Is there a way to create a quotation and have it publish only the total of the lines rather than publish the individual line prices?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9a588a9f-2dda-d33d-5ce9-1c25f25a5a06
Version Independent ID: c9547cc6-74a5-5822-db46-08fddf332ff1
Content: Create and edit sales quotations - Finance & Operations | Dynamics 365 | #MSDyn365FO
Content Source: articles/supply-chain/sales-marketing/tasks/create-edit-sales-quotations.md
Service: dynamics-ax-applications
GitHub Login: @omulvad
Microsoft Alias: omulvad

@SimonHoey, thanks for posting this! In the meantime, to get product help you can contact Support using one of the options described here: Support contact options. In particular, you might want to go to the Dynamics 365 for Finance and Operations community page to get help with your questions from the Dynamics community.
@omulvad, can you please review and answer the question.

Bug 323662 has been logged to track this issue.

@SimonHoey when you say publish are you referring to the price simulation or printing out a quote confirmation ? Either way we still have line level prices.


doc: add openSUSE as supported distro

SUSE supports cilium and bpf in  Leap 15/Tumbleweed
Signed-off-by: Nirmoy Das<EMAIL_ADDRESS>

This change is

Coverage increased (+0.03%) to 43.186% when pulling 92a1c2b58815e7b7575c1d3939dbaf0b986d107e on nirmoy:doc_opensuse into b60a98e2daf65552b5e99c078762fb83fb435b78 on cilium:master.

test-docs-please

test-docs-please


feat: Portkey integration with Llama_index

Description
This pull request introduces the Portkey integration to the Llama_index repository, enhancing its production capabilities. Portkey offers a comprehensive set of tools and features to optimize the performance and reliability of AI applications.
Key Features of Portkey's Integration with Llamaindex:

AI Gateway:

Automated Fallbacks & Retries: Ensure your application remains functional even if a primary service fails.
Load Balancing: Efficiently distribute incoming requests among multiple models.
Semantic Caching: Reduce costs and latency by intelligently caching results.

Observability:

Logging: Keep track of all requests for monitoring and debugging.
Requests Tracing: Understand the journey of each request for optimization.
Custom Tags: Segment and categorize requests for better insights.

Changes Made:
Incorporated a new LLM (Large Language Model) within the LLMs folder, which inherits from the existing CustomLLM class. This new LLM is responsible for integrating and utilizing Portkey's enhanced production capabilities while maintaining consistency with the repository's coding standards.
Dependencies:
This integration introduces one external library called "Rubeus," which is added to the list of dependencies. Rubeus is an integral part of Portkey and contributes to its internal functioning.
Fixes # (issue)
Type of Change

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] This change requires a documentation update

How Has This Been Tested?

[x] Added new notebook (that tests end-to-end)
[x] I stared at the code and made sure it makes sense

Suggested Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] New and existing unit tests pass locally with my changes

Updated the code with the required dependency import errors.
This seems to fail the workflows - Build Package and Unit Testing. I hope that's expected.
The error is the import failure in both of those cases.

Thanks again for the work on this @noble-varghese ! Going to merge now 🙌

Pleasure has been mine @logan-markewich . Looking forward to contribute more to Llama_index.


build: add lint instruction to makefile to alphabetize and normalize words

By alphabetizing the word list and normalizing each word by turning it to lowercase, we make it easier for contributors to add words to the library without worrying about duplicating entries.
If a contributor adds a new word and then runs make lint, they'll be able to clearly see in the diff if they added a duplicate since it will be right before or after the already existing word.

@web-mech Thanks for this library; it's been exciting to see the recent new releases. Would love to hear your thoughts on this PR.

LGTM. We'll integrate a CI/CD pipeline in the near future and make this a part of it


Add automatic broken link checker

For the convenience of checking broken link. adding CI seems to be useful.
Reference:
https://github.com/awesome-selfhosted/awesome-selfhosted/blob/0cdbe4c738e5136ea5d6573945735c8acec1c38a/Makefile#L23
https://github.com/becheran/mlc

Yes, we will consider to use the tool. thanks,

Thank you for considering,
I'm now trying to add those tool. But, shouldn't this issue be opened? I want to solve out this issue.
If it's done I will summit PR.


escaping in filters

Is escaping in filters supporting eg what if I want to filter by the literal text "+cross"
If not can this be added
If already can it be documented in https://www.feednotifier.com/support/keyword-filtering-guide/
Could there just be an advanced option for regex in text ?
Thanks

@fogleman how difficult would it be to provide an alternate search mechanism using regular expressions? https://docs.python.org/3/library/re.html


HDF5 array compression

Extend compression in HDF5 to numpy arrays, xarray DataArray and DataSet.
A very quick test on random runs gives me about 30% size reduction.

I assume the read overhead will be negligible for the vast majority of arrays.

I hope so, although gzip is relatively slow both on compression & decompression by modern standards. But hopefully the benefits of reading less data balance it out.
There is a potential drawback of enabling compression on arrays of unknown shape, however. Compression in HDF5 requires chunking, and if you don't specify a chunk shape, h5py will guess it, aiming to make it equally good for slicing along any axis. But that can easily mean equally bad, especially if the chunks in one slice exceed the chunk cache size (a paltry 1 MB by default), leaving you repeatedly decompressing the same data.
Where we know the nature of the data, it's usually easy to manually specify a chunk shape to make access efficient. But in this case we're trying to handle whatever the context file returns. :shrug:

My feeling is that a variable result is most likely a single entity one would want to load at once to use. And that's what our API does anyway at the moment, so yea 🤷
Do you think it can be so bad that we should not try to compress on arbitrary arrays? the 30% reduction is pretty appealing though.

I think that's probably reasonable enough, at least to start with and see how we go. Results have to be able to fit in memory when they're created, because of the way context files work.
It wouldn't entirely surprise me if someone decides to store large-ish results (10s of GB) this way and wants to load parts of them, either with an extension to our API or using the files directly. But we can cross that bridge when we come to it. :+1:

LGTM!


check the default branch

Can you elaborate?

we are checking the main branch of the repositories via GitHub API. It is not an issue anymore.


Server side compile in rails

Hey guys-
So I serve a JS site using rails (I love my asset pipeline), and am looking to switch over from eco, but this workflow is quite dependent on the https://github.com/sstephenson/ruby-eco gem — which provides the hooks necessary for sprockets, etc...for out of the box compilation.
Does an equivalent gem (or process) exist for ect?  Just curious if anyone else has done this process before I jump down a rabbit hole that may be deeper than I want it to be. :)

+1


LeaderfFile in project

vim or neovim?

[ ] vim
[x] neovim

vim --version or nvim --version:

NVIM v0.3.0-dev
Build type: RelWithDebInfo
Lua 5.1
Compilation: /usr/bin/x86_64-linux-gnu-gcc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wconversion -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -O2 -g -DMIN_LOG_LEVEL=3 -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wvla -fstack-protector-strong -fdiagnostics-color=auto -Wno-array-bounds -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -I/build/neovim-xCPrEf/neovim-0.2.2ubuntu2+git201804222147-ffb8904-f2f288d/build/config -I/build/neovim-xCPrEf/neovim-0.2.2ubuntu2+git201804222147-ffb8904-f2f288d/src -I/build/neovim-xCPrEf/neovim-0.2.2ubuntu2+git201804222147-ffb8904-f2f288d/.deps/usr/include -I/usr/include -I/build/neovim-xCPrEf/neovim-0.2.2ubuntu2+git201804222147-ffb8904-f2f288d/build/src/nvim/auto -I/build/neovim-xCPrEf/neovim-0.2.2ubuntu2+git201804222147-ffb8904-f2f288d/build/include
Compiled by buildd@lcy01-amd64-007

Output of :echo has("python"): 0
Output of :echo has("python3"): 1
Output of :echo &pythondll(only vim, not neovim):
Output of :echo &pythonthreedll(only vim, not neovim):
Output of :py print(sys.version):
Output of :py3 print(sys.version):
Operating system:

[x] Linux
[ ] Mac OS X
[ ] Windows
[ ] Etc.

Configurations related to LeaderF in vimrc:

" LeaderF
let g:Lf_ShortcutF = '<leader>ff'
let g:Lf_ShortcutB = '<leader>fb'
nnoremap <leader>fm :LeaderfMru<cr>
nnoremap <leader>fc :LeaderfFunction<cr>
nnoremap <leader>ft :LeaderfTag<cr>

let g:Lf_RootMarkers = g:root_markers
let g:Lf_WorkingDirectoryMode = 'Ac'
let g:Lf_WindowHeight = 0.35
let g:Lf_CacheDirectory = expand('~/.vim/cache')
let g:Lf_ShowRelativePath = 1
let g:Lf_StlColorscheme = 'default'
let g:Lf_StlSeparator = { 'left': '', 'right': '' }

let g:Lf_NormalMap = {
   \ "File":   [["<ESC>", ':exec g:Lf_py "fileExplManager.quit()"<CR>'],
   \            ["<F6>", ':exec g:Lf_py "fileExplManager.quit()"<CR>'] ],
   \ "Buffer": [["<ESC>", ':exec g:Lf_py "bufExplManager.quit()"<CR>'],
   \            ["<F6>", ':exec g:Lf_py "bufExplManager.quit()"<CR>'] ],
   \ "Mru":    [["<ESC>", ':exec g:Lf_py "mruExplManager.quit()"<CR>']],
   \ "Tag":    [["<ESC>", ':exec g:Lf_py "tagExplManager.quit()"<CR>']],
   \ "Function":    [["<ESC>", ':exec g:Lf_py "functionExplManager.quit()"<CR>']],
   \ "Colorscheme":    [["<ESC>", ':exec g:Lf_py "colorschemeExplManager.quit()"<CR>']],
   \ }

Steps to reproduce
project organization:
<project_root>
    .git
    src
        a.py
    ...

run LeaderfFile
Actual behaviour
leaderf try to open src/a.py instead of a.py, which is valid.
Expected behaviour
open a.py

Sorry, I can not quite understand your issue, could you describe in more detail?

@Yggdroot updated

This works as designed, because you have set let g:Lf_WorkingDirectoryMode = 'Ac'.
https://github.com/Yggdroot/LeaderF/blob/559b80aad4960d20f97a82a42d30abb3f6ef17ac/doc/leaderf.txt#L457-L474

So what does A mode design for?
It seems to be meaningful only when vim work in top of the project.

The desired workflow for me is below:

May work in sub-dir of the project
Switch between the files, which may be not a sub-dir of cur-dir

Anyway to implement it?

let g:Lf_WorkingDirectoryMode = 'Ac'  meets your request.

It does not.
project organize as follow
.
├── .git
└── src
    ├── a.py
    └── b.py

start from the root of the project

cd src
vim a.py
LeaderfFile, enter b.py<cr>
it will open src/src/b.py instead of src/b.py

I think this is the bug of neovim(https://github.com/neovim/neovim/issues/3636), does vim has the same behavior?

Tested on vim, it works well.

The patch for https://github.com/neovim/neovim/issues/3636 has been merged, you can update neovim and have a try.

I remove the neovim from pip and clone and install it by pip install ., still broken.

I think the neovim you installed this way does not have the patch, you can install through compiling the source code.

python-neovim 0.2.5 is used, where the patch has been applied.
The issue still occurs.

Perhaps this is a new bug of neovim, I'll try to find out what it is exactly and open an issue for neovim.

I have opened an issue for neovim: https://github.com/neovim/neovim/issues/8336.

I see from the doc vim.chdir may work. But it seems not recommended.

Fixed, please have a check.

It is already fixed, thanks.


Configure docker extension to use node.js path modules

Notes for Reviewers
This PR fixes:
There was error coming while building because the @layer5/sistent package or any project that is attempting to use the Node.js path module. Since Webpack 5 does not include polyfills for Node.js core modules by default, this causes the build to fail when trying to resolve path.
Now we can use sistent package easily.

Why used react script rewired:
react-app-rewired allows you to customize React App configurations (like Webpack) without ejecting.
❌Without react-app-rewired:
Customization requires ejecting, which exposes configuration files but makes maintenance harder.
✅With react-app-rewired:
You can modify configurations while staying within the managed CRA setup, keeping upgrades easier and your project simpler to maintain.

[!IMPORTANT]
This is do not merge because i have to fix styles of catalog card properly from sistent and release then we can move forward.

Signed commits

[x] Yes, I signed my commits.

Thanks @sudhanshutech . react-app-rewired makes sense ✅

@humblenginr please give a final look and let me know so we can merge and the filp card will be available in  next release. I have to little work in sistent for that . But here everything is working fine so we don't have to wait.


test: HACBS-1130/Create-and-Delete-namespaces-in-the-new-deployment-of-Pre-KCP

Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
Issue ticket number and link
https://issues.redhat.com/browse/HACBS-1130
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist:

[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] I have updated labels (if needed)

/assign @kasemAlem

Please remove all commented lines

Please remove all commented lines
Done

/assign @psturc

@psturc ,
we had some failures when creating the namespaces in e2e at beginning this why we wanted to make sure that this basic method does work as should ,
thus we started by the simple thin in e2e-test,
we are trying to build methods of CRDs changed due to ReleasePlan and ReleasePlanAdmissions


Please invite me to the GitHub Community Organization

Name
Sandip Pramanik
Discord Username (if applicable)
Inferno#1159
Additional Context
I found this organization from a tweet by @Pradumnasaraf yesterday while sharing GitHub profiles.
Also, a few days back, I was listening to a Twitter space titled "How to start Content Creation in Tech" where I get to know Eddie Jaoude @eddiejaoude and get interested in his work and contribution to the open-source community.
I am willing to learn about the open-source community and how it works. I want to get connected with a lot of other experienced developers, learn their way of thinking and collaborate in the creation of new open-source software. I'm also prepared to assist others to grow and contribute to the open-source community.

Name
Sandip Pramanik
Discord Username (if applicable)
Inferno#1159
Additional Context
I found this organization from a tweet by @Pradumnasaraf yesterday while sharing GitHub profiles.
Also, a few days back, I was listening to a Twitter space titled "How to start Content Creation in Tech" where I get to know Eddie Jaoude @eddiejaoude and get interested in his work and contribution to the open-source community.
I am willing to learn about the open-source community and how it works. I want to get connected with a lot of other experienced developers, learn their way of thinking and collaborate in the creation of new open-source software. I'm also prepared to assist others to grow and contribute to the open-source community.

Awesome, thank you joining 🎉

Welcome 🎉


scanner: Fall back to the package hash if the version is empty

Even for projects that do not use a package manager we want to be able to
cache scan result files by a unique key.

This change is

Review status: 0 of 1 files reviewed at latest revision, 1 unresolved discussion.

scanner/src/main/kotlin/Scanner.kt, line 71 at r1 (raw file):

        // TODO: Consider implementing this logic in the Package class itself when creating the identifier.
        // Also, think about what to use if we have neither a version nor a hash.
        val pkgRevision = pkg.version.let { if (it.isNullOrBlank()) pkg.hash.take(7) else it }

This should use "vcsRevision" instead of "hash".

Comments from Reviewable

Review status: 0 of 1 files reviewed at latest revision, 1 unresolved discussion.

scanner/src/main/kotlin/Scanner.kt, line 71 at r1 (raw file):
Previously, mnonnenmacher (Martin Nonnenmacher) wrote…
This should use "vcsRevision" instead of "hash".

Done.

Comments from Reviewable

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable


Feature Request: SASS Mixins

What I really loved about merbjedi's sprite gem was that it could output sass mixins (https://github.com/merbjedi/sprite/) which enabled you to add images via :before or :after pseudo-elements. you could just add your mixin to it with +sprite('icons', 'arrow'), no need for extra-markup.
would that be hard to do with sprite-factory?

I would like to see that feature as well.
Rake task is producing garbage, imo the work with sprites should be transparent and the same transparently as regular asset, including deploying project.


Specify conda dependencies at environment creation

This way conda can pick the appropriate python version e.g openmm doesn't necessarily support the latest.

Just to be explicit, this will mean that our installation instructions will have to be rewritten such that the user must conda install nanover-server before nanover-imd (currently the instructions are modular so that the user can install either package independently). I don't think this is a problem, but I know you had a couple of concerns @hjstroud

Just to be explicit, this will mean that our installation instructions will have to be rewritten such that the user must conda install nanover-server before nanover-imd (currently the instructions are modular so that the user can install either package independently). I don't think this is a problem, but I know you had a couple of concerns @hjstroud

I did, but I agree with you that it's not a problem.


Add to the documentation how to encode % sign

I have noticed that translating characters which contains "&" sign should be properly scaped
Example:
\"%word1%,word2%\"
to output
"%word1%,word2%
But maybe this information will be useful in the docs

HTML escaping is not in php-i18n's scope, but I will accept merge requests to add some documentation for this.


Parallel compilations appears to troll hash-for-dep cache

I noticed that rebuild time for a particular addon (with a lot of handlebars AST transforms) recently ballooned from a few seconds to 80-90 seconds.
It appears that something (I'm guessing the parallel compile) is causing hash-for-dep to be invoked as multiple modules, instead of a single module. Since the hash-for-dep cache is module scoped, invoking as multiple modules means that the cache is effectively useless.
cc @stefanpenner

thanks, I think I'll try and (tonight) disable the parallel support in this plugin, until we re-work it to not have this issue.

Specifically, the issue appears to be localized to the changes in babel-plugin-htmlbars-inline-precompile. Looking at the diff in v1.0.2 to v1.0.3, you can see that one of the dependency bumps includes babel-plugin-htmlbars-inline-precompile, where it reved version v0.2.3 to v0.2.5. v0.2.5 of that plugin contains fixes to parallel builds.
Pinning to v1.0.2 of this lib fixes the issue.

@scalvert the plugin in question, needs to reduce what hash-for-dep considers a dependency of it. I explained how to @asakusuma.
Previously, this plugin was not correctly considering it's cache key or opted out of parallel.
Both the plugin in question, and this plugin need some work. But if the above recommendation I gave asa is applied the problem will go away entirely, and it will perform better even when this library improves itself.

@stefanpenner since I forgot, and for future travelers, can you explain how to control what ember-cli-htmlbars-inline-precompile's hash-for-dep thinks is a dependency?

Improve hash-for-dep docs: https://github.com/stefanpenner/hash-for-dep/blob/master/README.md#what-does-hash-for-dep-consider-a-dependency
If something can be made clearer please open a PR/issue over there.
We will eventually add docs to the various consuming repos that have plugin authorship extensions (htmlbar/babel)


Add internal callback variant

What changed?
Added an internal callback variant for callbacks that are delivered internally within the system.
This variant is not settable in the API and will be rejected by the service with an INVALID_ARGUMENT error.
The only reason that this is exposed is because callbacks are replicated across clusters via the
WorkflowExecutionStarted event, which is defined in the public API.
Why?
The first use case for this will be for the scheduler as a state machine rewrite.

I do wish the internal things didn't require only public protos but instead wrapped them in internal protos that can have internal fields

We've been discussing this, I agree, long term we should have a better solution.


Unable to view the full diagnostic by clangd

The diagnostic by clangd is in red color. There is no horizontal scrollbar.

Yes, I mean code lens.

Btw, is it possible to turn off code lens?

Sure, it's a linter feature so you need to go to Tools -> Plugin Manager -> Select "Linter" -> click "Preferences" and then set enable_error_lens to false.
All plugins options are explained in ecode's README here. You have plenty of configuration there, but lacks some visibility, that needs to be improved.
For the moment I won't change its current behavior, it's intentional to be just a quick preview, I don't like changing the document view-port size over something that it's not part of the document, I could add a mouse hover over the lens zone and display the same error than the text zone, but I don't see a good reason to do it, I might add it though.


Enable google search box

I'm still waiting for the live site to be crawled but this hooks up the search box to a google search of the site.

Looks good, merged.


“go install” instructions don’t work

$ go install github.com/buildkite/terminal/cmd/terminal-to-html
can't load package: package github.com/buildkite/terminal/cmd/terminal-to-html: cannot find package "github.com/buildkite/terminal/cmd/terminal-to-html" in any of:
        /usr/local/Cellar/go/1.7.5/libexec/src/github.com/buildkite/terminal/cmd/terminal-to-html (from $GOROOT)
        /Users/ticky/go/src/github.com/buildkite/terminal/cmd/terminal-to-html (from $GOPATH)

Looking at the readme, this has already been fixed - go get works.

Oh, yep, fixed in bff759e6fbce9c24f211498c1c0bcc1d177b20b4!


Cayenne info shows up as France

Clicking on Cayenne shows the same info as France, including the title.

fixed. Thank you!


tvOS failed to verify bitcode in Alamofire

Using Alamofire v4.3.0 for tvOS, I'm failing to export an ipa from an archive, with this error:
2017-02-17 16:27:31 +0000 [MT] Presenting: Error Domain=IDEFoundationErrorDomain Code=1 "Failed to verify bitcode in Alamofire.framework/Alamofire:
error: Cannot extract bundle from /var/folders/tf/trlj_nmd34n35gv0bbr13skj9vb2f5/T/XcodeDistPipeline.8wM/Root/Payload/...

This isn't an Alamofire issue, we're just the first framework alphabetically.
In the future, you should open questions like this on Stack Overflow and tag alamofire.
Cheers. 🍻

From our Contribution Guidelines
Asking Questions
We don't use GitHub as a support forum. For any usage questions that are not specific to the project itself, please ask on Stack Overflow instead. By doing so, you'll be more likely to quickly solve your problem, and you'll allow anyone else with the same question to find the answer. This also allows maintainers to focus on improving the project for others.


[SwiftASTContext] Trim down the interface quite a bit.

All these functions are not used. No need to carry them around.

@swift-ci test and merge

Thanks!


[m0rehud] Add built in customizations

Describe the feature you're suggesting.
There's a lot of customizations in the m0rehud folder that aren't in the editor. They should be added to the schema.
Additional information
Reported by Sapo Mago on Discord.

Changes are being made in a separate branch here: https://github.com/CriticalFlaw/TF2HUD.Editor/tree/%23194

Development complete. A change to the editor was required, which is why this will be released in version 3.2 of the editor.


pscale: 0.85.0 -> 0.88.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/pscale/versions.
meta.description for pscale is: "The CLI for PlanetScale Database"
meta.homepage for pscale is: "https://www.planetscale.com/"
meta.changelog for pscale is: "https://github.com/planetscale/cli/releases/tag/v0.88.0"
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
/nix/store/al3v15120kb24ldswrpi7q5k8rxj5r0s-pscale-0.88.0/bin/pscale passed the binary check.
1 of 1 passed binary check by having a zero exit code.
0 of 1 passed binary check by having the new version present in output.
found 0.88.0 in filename of file in /nix/store/al3v15120kb24ldswrpi7q5k8rxj5r0s-pscale-0.88.0

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

2 package rebuild(s)

2 x86_64-linux rebuild(s)
2 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
docbookrx
pscale

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/al3v15120kb24ldswrpi7q5k8rxj5r0s-pscale-0.88.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A pscale https://github.com/r-ryantm/nixpkgs/archive/3b713d85d0387d28b9883b08334a708ffa6f897f.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/al3v15120kb24ldswrpi7q5k8rxj5r0s-pscale-0.88.0
ls -la /nix/store/al3v15120kb24ldswrpi7q5k8rxj5r0s-pscale-0.88.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

pscale

Maintainer pings
cc @pimeys @kalbasit for testing.

Tested. Logged in successfully and got a shell to the planetscale instance.


get cactus set up for blockchain-carbon-accounting working

Get @petermetz's cactus set up for blockchain-carbon-accounting working. See

https://github.com/hyperledger/cactus/issues/540
https://github.com/hyperledger/cactus/issues/665
https://github.com/hyperledger/cactus/tree/main/examples/carbon-accounting

@sichen1234 I can't self assign, could you please add me? Thank you in advance!

@Zzocker If you've gotten this to work then we can close this task.

Yes !!! I am able to run the carbon accounting example from cactus. But I did not used docker setup instead run on my local machine. @petermetz I had to make some changes to make it work, have created a pull request in your fork.

@Zzocker Awesome, said this on that PR as well: I recommend submitting against the upstream repo otherwise it's not actually getting merged back to Cactus


chat template for the function/messages

Hi, is there a jinja template somewhere that shows the chat template that should be used. I would like not to rely too much on vllm/fschat to do the formatting of the messages.
Something akin to chat_template in https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/main/tokenizer_config.json

Hi! thanks for the PR.
@khai-meetkai is working on the chat template. https://github.com/MeetKai/functionary/pull/50

@musabgultekin Thanks for the reply - cool!


Replicate NST fine-tuning on Alvenir machine to sniff out training or data issues

The loss curve has looked wrong for training on NST and we want to see whether that also happens on other compute.

Please let me know, when doing this is good to go, @saattrupdan - should I wait for #29 merge?
I should use finetune_model.py with the wav2vec2.yaml model config and the nst_da.yaml config? Or is there a place I can find the exact run command? :)

You should probably wait for #29 to be merged first, yeah.
As for the command, you can run:
$ python src/scripts/finetune_model.py model=wav2vec2 datasets=[nst_da,common_voice_da] dataset_probabilities=[0.95,0.05]

Note that NST-da doesn't have any evaluation split, so only the common voice evaluation split is used. You can also simply try to completely overfit the NST-da dataset, running the following, which disables all regularisation and trains solely on the NST-da dataset (without any evaluation set). If I replace nst_da with common_voice_da then I've made it work (training loss ~ 0), but I haven't succeeded for NST-da, which might either be (a) a fault with the dataset, or (b) simply because it's so large that it takes way longer to get to 0.
$ python src/scripts/finetune_model.py model=wav2vec2 datasets=nst_da model.activation_dropout=0 model.attention_dropout=0 model.hidden_dropout=0 model.feat_proj_dropout=0 model.final_dropout=0 model.mask_time_prob=0 model.mask_feature_prob=0 model.layerdrop=0 early_stopping=false

Currently running the NST overfitting training, will report back with progress :)

Initial progress after 1600 steps looks fine: Started with a training loss value around 25 and is currently below 0.4

Initial progress after 1600 steps looks fine: Started with a training loss value around 25 and is currently below 0.4

Nice! What is your hyperparameters? Default?

I ran it with the default, using your call:
$ python src/scripts/finetune_model.py model=wav2vec2 datasets=nst_da model.activation_dropout=0 model.attention_dropout=0 model.hidden_dropout=0 model.feat_proj_dropout=0 model.final_dropout=0 model.mask_time_prob=0 model.mask_feature_prob=0 model.layerdrop=0 early_stopping=false

Now, I get after 27K steps (~4 epochs) a sudden fall where it succesfully hits 0, so maybe there is no fault with the dataset and you are correct that it is:

simply because it's so large that it takes way longer to get to 0.

The curve looks very funky but I guess that might be fine for this overfitting task.
Should I try to run the other command to test that? (though the larger GPU is occupied the next couple of days)

Aha, that's good to hear. I only finetuned for 10k steps, so must've just been that.
And yep, please run the first command that I sent, to see if you can reproduce the results of your finetuned model (https://huggingface.co/chcaa/alvenir-wav2vec2-base-da-nst-cv9). You might have to tweak early_stopping_patience in case it plateaus for a long time, as your plot above. Currently it's set to 50 steps. Alternatively, you can just disable early stopping completely by setting early_stopping=false.

I ran
$ python src/scripts/finetune_model.py model=wav2vec2 datasets='[nst_da,common_voice_da]' dataset_probabilities.train='[0.95,0.05]' early_stopping=false

and sadly, this looks very strange :(

Furthermore, the WER is 100% during the entire training - something is certainly wrong.
I note that the sudden fall to 0-loss happens at the same time; after 27K steps. This time preceded by a spike in loss.
There is some debugging to do, I suppose.
I can look at it later this week; any tips of how to approach this?
The trainer state of the training: trainer_state.json. Let me know if I can share any more details that can help debugging.

I see in your logs that the 0-loss is NaN rather than 0, so there might be some overflow/underflow issue going on here. But it's super weird that the WER stays constant.
Also, the validation loss seems "stuck" after 4600 steps, which is also super strange. What happens if you crank up the gradient accumulation, which makes the effective batch size larger and hopefully more stable as well?
$ python src/scripts/finetune_model.py model=wav2vec2 datasets='[nst_da,common_voice_da]' dataset_probabilities.train='[0.95,0.05]' early_stopping=false model.gradient_accumulation=16

This won't fix the NaN validation loss issue though. Some things to try out:

Manually check what comes out of the data collator during both training and validation. We replace padding tokens with -100; is the entire batch converted to -100 for some reason, say?
Found a bug in the wav2vec2 module: word_delimiter_token is set to |, where this should now be   (i.e., a space).
Check whether self.processor.tokenizer.get_vocab actually fetches the correct vocab, in the wav2vec2 module.
Check whether do_eval is set to True in the wav2vec2 module. It also doesn't seem right that we're calling values() on self.cfg.datasets, since it's a list?

I see in your logs that the 0-loss is NaN rather than 0, so there might be some overflow/underflow issue going on here. But it's super weird that the WER stays constant.

Sure? Isn't it just the eval loss that turns into NaN after step 27440?
But yes, also seems suspiciously overflowy with the high jump.
Here is the resut plotted of running with higher grad accum. Behaviour looks similar.

I might have time to look at your debugging suggestions tomorrow, ended up not having time today :/

I see in your logs that the 0-loss is NaN rather than 0, so there might be some overflow/underflow issue going on here. But it's super weird that the WER stays constant.

Sure? Isn't it just the eval loss that turns into NaN after step 27440? But yes, also seems suspiciously overflowy with the high jump.
Here is the resut plotted of running with higher grad accum. Behaviour looks similar.
I might have time to look at your debugging suggestions tomorrow, ended up not having time today :/

Ah yeah, just the evaluation loss, my bad.
And fair enough. I'm guessing that the typo in point (2) will probably affect the WER stuff at least, since I'm guessing that's where it's getting its notion of "word" from, so that could be the first thing to check when you've got the time 🙂

Manually check what comes out of the data collator during both training and validation. We replace padding tokens with -100; is the entire batch converted to -100 for some reason, say?

From superficial inspection, it seems to work. Some examples do have a lot of padding - I see some examples with all padding. I have a version training now where I log the number of -100's in a batch to see whether the fateful step happens because of a batch with all examples with all padding. Maybe a pretty good explanation? We should thus maybe remove empty audio segments.

Interesting regarding the all -100's. Makes sense to remove them I guess, yep.

It does produce a nice looking vocab. It is certainly differently ordered compared to previous models (e.g. chcaa/alvenir-wav2vec2-base-da-nst-cv9). (Would it be nice if I just sorted the characters before dumping vocabulary?)

Sure, feel free to sort them before dumping them 🙂
Excited to see if the "|" replacement made any difference!

Maaan, this didn't help anything. Still get 0 train loss and NaN eval loss from step 27560. Getting WER of 1 all the way through. Didn't find a batch of where every example had all -100's :(
I guess more in depth debugging of what happens in the evaluation + the significance of this specific step is required. Did you also get the problem at this step?

Hmm that's really annoying. I think I'll have some time to look at it next week. Maybe try saving checkpoints so that we have the checkpoint and trainer state just before it gets to 0 loss? That would help with the debugging at least.

That sounds nice, I will try to get some debugging material later this week :)

Sorry, my debug run failed before hitting step 27K because I accidentally saved outputs to a disk that ran out of space. Setting it up again now to get checkpoints before/after step 27K but it probably takes a couple of days on the GPU that is free atm

Now I have some checkpoints around the fateful step: https://drive.google.com/drive/folders/1TO9kHtZrf27ZgNS3RpPJRXP2XoUtM3j2?usp=sharing (just request access)
Checkpoint-27400 is just before and checkpoint-27500 just after (see log excerpt below).
(And wtf, gzip seems to be able to compress the step 27500 model into almost nothing even though both checkpoint dirs originally were 3.6GB??? Model must be all zeros or something crazy)
{'loss': 3.4909, 'learning_rate': 2.8414528699927665e-08, 'epoch': 4.12}
{'loss': 3.4932, 'learning_rate': 2.7358640664088553e-08, 'epoch': 4.12}
{'loss': 3.5171, 'learning_rate': 2.6322726410502595e-08, 'epoch': 4.12}
{'loss': 3.481, 'learning_rate': 2.530678732109937e-08, 'epoch': 4.12}
{'loss': 3.5319, 'learning_rate': 2.431082475116142e-08, 'epoch': 4.12}
{'loss': 3.4564, 'learning_rate': 2.3334840029319293e-08, 'epoch': 4.12}
{'loss': 3.5391, 'learning_rate': 2.2378834457558173e-08, 'epoch': 4.12}
{'loss': 3.5177, 'learning_rate': 2.144280931120457e-08, 'epoch': 4.12}
{'loss': 3.4667, 'learning_rate': 2.052676583893298e-08, 'epoch': 4.12}
{'loss': 3.4484, 'learning_rate': 1.963070526276589e-08, 'epoch': 4.12}
***** Running Evaluation *****
  Num examples: Unknown
  Batch size = 8
{'eval_loss': 3.5697641372680664, 'eval_wer': 1.0, 'eval_runtime': 80.0147, 'eval_samples_per_second': 27.77, 'eval_steps_per_second': 3.474, 'epoch': 4.12}
Saving model checkpoint to /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400
Configuration saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/config.json
Model weights saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/pytorch_model.bin
tokenizer config file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/tokenizer_config.json
Special tokens file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/special_tokens_map.json
Deleting older checkpoint [/mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27000] due to args.save_total_limit
{'loss': 3.5564, 'learning_rate': 1.8754628778060446e-08, 'epoch': 4.12}
{'loss': 3.4582, 'learning_rate': 1.7898537553521775e-08, 'epoch': 4.12}
{'loss': 3.4657, 'learning_rate': 1.706243273118968e-08, 'epoch': 4.12}
{'loss': 140.4032, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
***** Running Evaluation *****
  Num examples: Unknown
  Batch size = 8
{'eval_loss': nan, 'eval_wer': 1.0, 'eval_runtime': 81.174, 'eval_samples_per_second': 27.373, 'eval_steps_per_second': 3.425, 'epoch': 4.12}
Saving model checkpoint to /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500
Configuration saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/config.json
Model weights saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/pytorch_model.bin
tokenizer config file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/tokenizer_config.json
Special tokens file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/special_tokens_map.json

Now I have some checkpoints around the fateful step: https://drive.google.com/drive/folders/1TO9kHtZrf27ZgNS3RpPJRXP2XoUtM3j2?usp=sharing (just request access) Checkpoint-27400 is just before and checkpoint-27500 just after (see log excerpt below).
(And wtf, gzip seems to be able to compress the step 27500 model into almost nothing even though both checkpoint dirs originally were 3.6GB??? Model must be all zeros or something crazy)
{'loss': 3.4909, 'learning_rate': 2.8414528699927665e-08, 'epoch': 4.12}
{'loss': 3.4932, 'learning_rate': 2.7358640664088553e-08, 'epoch': 4.12}
{'loss': 3.5171, 'learning_rate': 2.6322726410502595e-08, 'epoch': 4.12}
{'loss': 3.481, 'learning_rate': 2.530678732109937e-08, 'epoch': 4.12}
{'loss': 3.5319, 'learning_rate': 2.431082475116142e-08, 'epoch': 4.12}
{'loss': 3.4564, 'learning_rate': 2.3334840029319293e-08, 'epoch': 4.12}
{'loss': 3.5391, 'learning_rate': 2.2378834457558173e-08, 'epoch': 4.12}
{'loss': 3.5177, 'learning_rate': 2.144280931120457e-08, 'epoch': 4.12}
{'loss': 3.4667, 'learning_rate': 2.052676583893298e-08, 'epoch': 4.12}
{'loss': 3.4484, 'learning_rate': 1.963070526276589e-08, 'epoch': 4.12}
***** Running Evaluation *****
  Num examples: Unknown
  Batch size = 8
{'eval_loss': 3.5697641372680664, 'eval_wer': 1.0, 'eval_runtime': 80.0147, 'eval_samples_per_second': 27.77, 'eval_steps_per_second': 3.474, 'epoch': 4.12}
Saving model checkpoint to /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400
Configuration saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/config.json
Model weights saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/pytorch_model.bin
tokenizer config file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/tokenizer_config.json
Special tokens file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27400/special_tokens_map.json
Deleting older checkpoint [/mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27000] due to args.save_total_limit
{'loss': 3.5564, 'learning_rate': 1.8754628778060446e-08, 'epoch': 4.12}
{'loss': 3.4582, 'learning_rate': 1.7898537553521775e-08, 'epoch': 4.12}
{'loss': 3.4657, 'learning_rate': 1.706243273118968e-08, 'epoch': 4.12}
{'loss': 140.4032, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
{'loss': 0.0, 'learning_rate': 1.6979921544251764e-08, 'epoch': 4.12}
***** Running Evaluation *****
  Num examples: Unknown
  Batch size = 8
{'eval_loss': nan, 'eval_wer': 1.0, 'eval_runtime': 81.174, 'eval_samples_per_second': 27.373, 'eval_steps_per_second': 3.425, 'epoch': 4.12}
Saving model checkpoint to /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500
Configuration saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/config.json
Model weights saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/pytorch_model.bin
tokenizer config file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/tokenizer_config.json
Special tokens file saved in /mnt/data/asr-training/coral-models/models/wav2vec2-finetuned/checkpoint-27500/special_tokens_map.json

Really interesting with the high compression. It really sounds like some kind of under/overflow happening somewhere. I requested access now 🙂

Also did some more experiments related to overfitting. Tried disabling all regularisation again, and then then finetuned the model at varying amounts of dataset_probabilities. Here are the findings:

At 95% NST, the flatline + mysterious NaNs happen.
At 50% NST the same flatline happens, and I presume that the NaNs happen at some point as well, but just way later than 30k steps.
At 10% NST the loss starts going down, albeit very slowly (see plots below).
Changing all_exhausted to first_exhausted doesn't make any difference.
If finetuning on both NST and CV, but setting dataset probabilities to 100% CV, then the results are similar to only finetuning on CV - so there's nothing wrong with the dataset_probabilities argument, at least.

Points (1)-(3) seems to indicate that there is a real tension between NST and CV, since we can overfit them separately, but when we start interleaving them then the model is struggling to overfit.
Charts related to (3):

Another curious observation:
The Alvenir model used this fairseq config, which sets dataset.max_tokens=1_000_000. I just ran the associated tokeniser on all of NST-da, and the entire dataset has about 10.5M tokens. To only use 1M tokens is thus using less than 10% of the NST-da. Can you confirm that that's true? Because that changes the total number of training steps quite a bit, in our case.

And yet another one:
I can't seem to find the batch size used in Fairseq, but in the XLSR paper, which is what the Alvenir model seems to follow, it mentions that the effective batch size is 0.44 hours, roughly corresponding to a batch size of 256, which is of course way bigger than the current 32. This might stabilise training more, as well as seeing way more samples of course.

Also did some more experiments related to overfitting

Looks interesting, especially if the cause is inter-dataset heterogeneity/tension. Could it also be some error in the NST data?

dataset.max_tokens=1_000_000
batch size

I must admit that the trainings are before my time and I haven't worked much in Fairseq ASR  (I can try to get more info of Martin and Lasse) but my fairseq understanding tells me:

dataset.max_tokens is the number of tokens in a per device mini-batch as fairseq does not operate on examples but normalized length batches in terms of a number of either sentences or tokens. This approach means that the per device mini-batch size in terms of examples is varying. See a plot below of this value (called bsz in fairseq) which should be for a run with the same config. I sadly cannot figure out right now why this number is a max and when fairseq decides to do go lower than this.

update_freq then acts as accumulation steps.

So one parameter update would include data from at most the following number of tokens: max_tokens * update_freq * number of devices. Here I get 32M tokens for the Alvenir model. I must admit that I am a little confused whether this number is in terms of audio samples. This would give 33 minutes in max effective batch size at 16kHz.

Number of examples in each per device mini-batch:

Dangit, I also found this plot which contradicts my understanding a bit :((
I'll try to get these things answered tomorrow!

Dangit, I also found this plot which contradicts my understanding a bit :((
I'll try to get these things answered tomorrow!

I think this plot might be showing number of transcription tokens, since with ~50 transcription tokens per sample it would correspond to roughly the same batch size as your first plot.
Regarding max_tokens: I see! I never thought of each audio segment as a token, but it does seem like that's what's happening here, probably because fairseq is the same framework for both text models and audio models. And then the batch size makes sense too.
In any case, the previous effective batch size in this repo was 32, which is 10x smaller than the Alvenir config, so that might explain a bit. I've tried replicating the Alvenir config more correctly now, and started a run now. However, I've reduced the number of optimisation steps from 120k to 12k, since the steps they mention in the XLS-R paper is closer to that.

Hoping that larger batch size can make it work🤞
Otherwise, I can try to debug the NST data format next week and set up a debug training with our preprocessed NST if that makes sense.

Hoping that larger batch size can make it work🤞 Otherwise, I can try to debug the NST data format next week and set up a debug training with our preprocessed NST if that makes sense.

Sounds good. One thing I noticed is that there are a decent amount of samples in the NST dataset with no transcription (because nothing is being said). I've removed those now, and recompiled the dataset. I'm in the process of uploading the dataset to HF Hub now, so a new version should be up today (with a larger shard size as well).

Looks like the larger batch size fixed things - loss and WER curves are looking much better now:

They are clearly underfitting, so we should be able to push the WER down way more 🤞
Closing here for now.


.github: fix renovate config

The regex defined in the renovate config is not valid and it's causing renovate to not being able to read it.
Fixes: 9c47083781f7 ("renovate: ignore all gops updates")
Fixes https://github.com/cilium/cilium/issues/23230

Thanks for fixing this up.


Extensions are gone in side bar when syncing settings from multiple Windows accounts

Type: Bug
I have 2 Windows accounts, Administrator and standard user account.
I always sign into standard account and use VSCode there normally.
Recently I started VSCode as Administrator to be able to run scripts as Admin and I turned on settings sync while VSCode was run as Admin, thus settings sync is enabled for both Win. accounts.
What happened is that extensions are no longer shown in the action bar -> extensions when I run VSCode as normal user, but this happens only when I open a workspace folder.
What I have tried is:

Uninstall VSCode (was installed system wide)
Deleted %appdata%.vscode for both Win. accounts
Deleted Roaming\Code directory for boath accounts.
Installed VSCode again.

This worked until I run it as Administrator and turned on settings sync on both accounts, after which the problem come back.
I'm not sure what information I could give since I never had this problem and have no idea how to resolve it except to reinstall VSCode and never run it as Admin but that's not a solution because I need it with both Win. accounts.
VS Code version: Code 1.74.0 (5235c6bb189b60b01b1f49062f4ffa42384f8c91, 2022-12-05T16:38:16.075Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Sandboxed: Yes

System Info

Item
Value

CPUs
Intel(R) Core(TM) i3-8100 CPU @ 3.60GHz (4 x 3600)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
undefined

Memory (System)
15.95GB (10.72GB free)

Process Argv
C:\dev\GitHub\WindowsFirewallRuleset --crash-reporter-id f4ba6678-424f-4312-b3c7-78a7cb995f13

Screen Reader
no

VM
0%

Extensions (21)

Extension
Author (truncated)
Version

Bookmarks
ale
13.3.1

project-manager
ale
12.7.0

vscode-markdownlint
Dav
0.48.1

LogFileHighlighter
emi
2.16.0

filter-line
eve
2.0.1

todo-tree
Gru
0.0.220

rainbow-csv
mec
3.4.0

remote-ssh
ms-
0.92.0

remote-ssh-edit
ms-
0.84.0

powershell
ms-
2022.11.0

remote-explorer
ms-
0.0.3

fix-json
oli
0.1.2

vscode-auto-scroll
pej
1.1.0

vscode-xml
red
0.22.0

vscode-yaml
red
1.10.1

vscode-sort-json
ric
1.20.0

trailing-spaces
sha
0.4.1

code-spell-checker
str
2.12.0

highlight-bad-chars
wen
0.0.6

markdown-all-in-one
yzh
3.5.0

json
Zai
2.0.2

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
pythondataviewer:30285071
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
azure-dev_surveyone:30548225
vsccc:30610678
pyindex848:30577860
nodejswelcome1:30587005
282f8724:30602487
gswce1:30612156
iaj6b796:30613358
dbltrim-noruby:30604474
f6dab269:30613381
fim-prod:30623723
vscsb:30625888

EDIT:
screenshot when opened in one workspace folder:

sreenshot when opened in another workspace folder:

screenshot when no workspace is opened:

UPDATE
I would like to update my issue and report that this problem is related to VSCode 1.74, I uninstalled VSCode and installed previous version 1.73.1 and the problem is no longer present! extension are shown in action bar in both Windows account and regardless if workspace is opened or not.
However it's not completely resolved, one issue remains which is that if I disable certain extensions in one windows account (run as Administrator) the change is not reflected in another windows user account (normal run) even though I'm syncing settings for both accounts with same Microsoft account.
So this are now 2 related issues, one present in VSCode 1.74 and another present since before.
I think there is no need to provide steps to reproduce as it's self explanatory, but if you need any additional information such as logs let me know.

@metablaster Thanks for reporting the issue and lets focus on the issue in 1.74. Can you please confirm following

When extensions are not shown in UI, are the extensions running? Is it just extensions are not visible?
Does reloading the window fixes it?
Can you please share this file - F1 > Extensions: Open Extensions Folder > extensions.json

When extensions are not shown in UI, are the extensions running? Is it just extensions are not visible?

I see extensions are working as expected, for ex. when I open PS script the PowerShell extension fires up and provides it's services,
Same is true for other extensions which are language agnostic.
Output of the extension host is clear:
2022-12-09 13:51:54.689 [info] Extension host with pid 9036 started
2022-12-09 13:51:54.689 [info] Skipping acquiring lock for c:\Users\User\AppData\Roaming\Code\User\workspaceStorage\3ba295179874fb39a80228b5cd4f619c.
2022-12-09 13:51:54.824 [info] ExtensionService#_doActivateExtension emilast.LogFileHighlighter, startup: false, activationEvent: 'onLanguage:log'
2022-12-09 13:51:54.841 [info] ExtensionService#_doActivateExtension pejmannikram.vscode-auto-scroll, startup: false, activationEvent: 'onLanguage:log'
2022-12-09 13:51:54.844 [info] ExtensionService#_doActivateExtension vscode.microsoft-authentication, startup: false, activationEvent: 'onAuthenticationRequest:microsoft'
2022-12-09 13:51:54.875 [info] ExtensionService#_doActivateExtension vscode.git-base, startup: true, activationEvent: '*', root cause: vscode.git
2022-12-09 13:51:54.880 [info] ExtensionService#_doActivateExtension yzhang.markdown-all-in-one, startup: true, activationEvent: 'workspaceContains:README.md'
2022-12-09 13:51:55.023 [info] ExtensionService#_doActivateExtension vscode.markdown-math, startup: false, activationEvent: 'api', root cause: yzhang.markdown-all-in-one
2022-12-09 13:51:55.074 [info] ExtensionService#_doActivateExtension vscode.git, startup: true, activationEvent: '*'
2022-12-09 13:51:55.100 [info] ExtensionService#_doActivateExtension vscode.github, startup: true, activationEvent: '*'
2022-12-09 13:51:55.323 [info] Eager extensions activated
2022-12-09 13:51:55.368 [info] ExtensionService#_doActivateExtension vscode.debug-auto-launch, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.372 [info] ExtensionService#_doActivateExtension vscode.emmet, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.377 [info] ExtensionService#_doActivateExtension vscode.merge-conflict, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.395 [info] ExtensionService#_doActivateExtension ms-vscode-remote.remote-wsl-recommender, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.406 [info] ExtensionService#_doActivateExtension alefragnani.Bookmarks, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.421 [info] ExtensionService#_doActivateExtension alefragnani.project-manager, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.442 [info] ExtensionService#_doActivateExtension Gruntfuggly.todo-tree, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.482 [info] ExtensionService#_doActivateExtension shardulm94.trailing-spaces, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:55.540 [info] ExtensionService#_doActivateExtension streetsidesoftware.code-spell-checker, startup: false, activationEvent: 'onStartupFinished'
2022-12-09 13:51:57.352 [info] ExtensionService#_doActivateExtension vscode.github-authentication, startup: false, activationEvent: 'onAuthenticationRequest:github'
2022-12-09 13:51:58.447 [info] ExtensionService#_doActivateExtension vscode.configuration-editing, startup: false, activationEvent: 'onLanguage:jsonc'
2022-12-09 13:51:58.450 [info] ExtensionService#_doActivateExtension vscode.json-language-features, startup: false, activationEvent: 'onLanguage:jsonc'
2022-12-09 13:51:58.472 [info] ExtensionService#_doActivateExtension vscode.typescript-language-features, startup: false, activationEvent: 'onLanguage:jsonc'
2022-12-09 13:52:07.043 [info] ExtensionService#_doActivateExtension ms-vscode.powershell, startup: false, activationEvent: 'onLanguage:powershell'
2022-12-09 13:52:07.460 [info] ExtensionService#_doActivateExtension mechatroner.rainbow-csv, startup: false, activationEvent: 'onLanguage:plaintext'

Does reloading the window fixes it?

No it doesn't.
CTRL + SHIFT + P Developer:Reload Window does not resolve the problem.
Restarting VSCode also doesn't help.

Can you please share this file - F1 > Extensions: Open Extensions Folder > extensions.json

zippyshare.com -> extensions.zip

Enable log level to trace - F1 > Set Log Level... and Can you also please share following logs

F1 > Open View... > Shared
F1 > Open View... > Window
F1 > Open View... > Extension Host
Do you see any thing suspicious here - F1 > Toggle Developer Tools > Console/Network

I am not able to access the extensions.json file from above comment. Can you please upload it to GitHub?

I am not able to access the extensions.json file from above comment. Can you please upload it to GitHub?

Sorry I didn't know I can attach file here by simply dragging them here, that's definitely new to me, had to google this out...
Here are the logs for both user accounts you requested:
logs.zip

Following is what I found suspicious in developer tools:
TRACE Error while resolving configuration file 'file:///c%3A/dev/GitHub/WindowsFirewallRuleset/.vscode/tasks.json': Unable to read file 'c:\dev\GitHub\WindowsFirewallRuleset\.vscode\tasks.json' (Error: Unable to resolve nonexistent file 'c:\dev\GitHub\WindowsFirewallRuleset\.vscode\tasks.json')

That's file in my repo which I don't use at all and didn't change it for some 2 years, it was present in previous VSCode versions.
TMScopeRegistry.ts:47 Overwriting grammar scope name to file mapping for scope text.log.
Old grammar file: file:///c%3A/Program%20Files/Microsoft%20VS%20Code/resources/app/extensions/log/syntaxes/log.tmLanguage.json.

Old grammar file: file:///c%3A/Program%20Files/Microsoft%20VS%20Code/resources/app/extensions/yaml/syntaxes/yaml.tmLanguage.json.
New grammar file: file:///c%3A/Users/User/.vscode/extensions/redhat.vscode-yaml-1.10.1/syntaxes/yaml.tmLanguage.json

I don't know what this means, perhaps extension is registering it's services?
these are surely from YAML extension from red-hat and Log file highlighter extension.
WARN [redhat.vscode-yaml]: Cannot register 'redhat.telemetry.enabled'. This property is already registered.
log.ts:304
INFO Creating lazy extension host: contains 1 extension(s): wengerk.highlight-bad-chars.

This is likely not issue because Cannot register 'redhat.telemetry.enabled' was present in VSCode 1.73.1
TRACE EntryNotFound (FileSystemError): Error: ENOENT: no such file or directory, stat 'c:\dev\GitHub\WindowsFirewallRuleset\node_modules'
    at l (C:\Program Files\Microsoft VS Code\resources\app\out\vs\code\electron-main\main.js:34:4023)
    at i.Y (C:\Program Files\Microsoft VS Code\resources\app\out\vs\code\electron-main\main.js:39:36565)
    at i.stat (C:\Program Files\Microsoft VS Code\resources\app\out\vs\code\electron-main\main.js:39:29933)

This is likely due to extension which uses node.exe which is XML extension from red-hat
The files from the log above are here:
devtools.zip
I don't see anything wrong in the network tab.

Could be a duplicate of https://github.com/microsoft/vscode/issues/168703
May I know if the extensions views/sections are collapsed? Can you please do following and let me know if it worked?

Expand the extensions views/sections and reload the window?

Yes, when I expand extension views and then reload window all of the extensions appear in all views.
But those views which are collapsed after reloading the window don't show any.
I've tested in both Win. accounts and that works.
Also worth noting that if I leave the views expanded (after window reload) extensions appear normally on fresh start of VSCode in expanded views but not in collapsed ones.

May I know if syncing (with same MS account in VSCode) but on different OS users accounts should sync which extensions are installed and which extension views are expanded or not is by design?
For example if uninstall one extension in standard account that change is not reflected in Admin account, even though both are synced with same MS account.
In another user account I need to explicitly uninstall extension.

May I know if syncing (with same MS account in VSCode) but on different OS users accounts should sync which extensions are installed and which extension views are expanded or not is by design?

Extensions installation and uninstallation shall be synced automatically. Expansion state of views is not synced.

For example if uninstall one extension in standard account that change is not reflected in Admin account, even though both are synced with same MS account. In another user account I need to explicitly uninstall extension.

Please follow these steps and share logs

Set log level to trace - F1 > Set Log Level... > Trace
On standard account: Uninstall an extension
Wait for a moment for sync to kick in: Share the log - F1 > Open View... > Settings Sync
Go to admin account: Wait for a moment for sync to kick in: Share the log - F1 > Open View... > Settings Sync

Sadly I could no longer reproduce this problem the way I described initially but with additional testing I see what I was referring to, this is partially reproducible but only with disabled extensions with the following behavior:

Uninstall a disabled extension in standard account wait for sync to happen
Open VSCode as Admin and the extension is uninstalled as expected but extension view still shows the extension offered for install as shown in the image below (vscode-icons was uninstalled):

Then after "reload window" the extension view no longer shows or offers this same extension for install, it's gone from extensions view.

I tested the other way around as well (install extension) but the behavior is as expected as follows:

Install extension in user account and disable it
Open VSCode as Admin and the extension is installed and disabled as expected, as shown in the image below:

But I think I know why this happens, this same extension vscode-icons requires window reload after uninstall if it was enabled, but if it's disabled no reload is offered after uninstall, therefore to get the expected result one has to reload manually.
I've collected logs for both cases which indicate and sync works:
vscode-icons-trace.zip
Expected behavior:
If a disabled extension is uninstalled in one OS account, the extension view does not show it as uninstalled nor offer it for install in other OS accounts afterwards.
Alternatively if this is by design, then a user in other OS account should be suggested to reload window for the change to take effect on extensions view.

Open VSCode as Admin and the extension is uninstalled as expected but extension view still shows the extension offered for install as shown in the image below (vscode-icons was uninstalled):

Yes, this is true and same even if you uninstall manually. We do not remove entries automatically from the view. If you refresh the view, it will go away and it does not require reload.

Open VSCode as Admin and the extension is installed and disabled as expected, as shown in the image below:

In this case, we automatically adds extension to the view.


Nuclei classification can run without feature / annotation files

If the user doesn't provide any nuclei features or nuclei annotation files, the system generates both of these files based on default parameters.
Once that is generated it classifies the nuclei based on the model provided by the user.

- Issues faced

Number of nuclei and features are not the same since they are generated by two different code sets with different methodologies. ( mainly it affects the WSI, for small image patches they are the same )
Nuclei features sometimes create NaN entries in the feature file ( observed for WSI ).
automatic feature and annotation generation takes a long time if it's a large WSI.

- Temporary fix for these issues

The features that exceed the number of nuclei are removed.
Rows with NaN are dropped from the generated nuclei features.

@subinkitware Do you remember what this needs to be ready for review?

@subinkitware Do you remember what this needs to be ready for review?

Yeah, this is ready for review, I just rechecked the code. In a gist this update is resolving this issue:  In older code, user has to upload the classification model, annotations, and features to run a classification job. Otherwise, it won't let the user run it. With this update, if the user doesn't provide any features or nuclei segmentation, HistomicsTK calculates both annotations & features by invoking the segmentation & feature extraction code in the other cli's


Delete button placement in menubar items

Is your feature request related to a problem? Please describe.
I accidently deleted my container many times while trying restart it.
Describe the solution you'd like
Delete option for containers in menu bar should be red fonts and placed at the bottom of all the options instead of being with restart button.
If possible can we have a confirmation when deleting containers from menubar?
Additional context.

I don't think it's necessary, containers don't have value, just delete them and recreate and start them, shouldn't store anything of value inside the container


Feature request: initiate a test without entering the browser

I myself usually write code and deploy servers in pure command line, and I really wanted to be able to start my tests without entering the locust's localhost website.
the situation
the server which I run locust is behind NAT and I wasn't allowed to poke a hole in my company firewall just to be able to access the locust web UI, and I can't deploy locust on my own laptop because the development website could only be accessed through my test server, which is in my company behind NAT, that's when I wish locust could have this obvious feature.

you can use the --no-web parameter on the command line.  See the help listing for available command line args:  $ locust -h
--no-web    Disable the web interface, and instead start running
                    the test immediately. Requires -c and -r to be
                    specified.

I figured I could make a GET request towards <IP_ADDRESS>:8089/swarm with parameters, as I answered this question, that would solve my problem too, your solution's better @cgoldberg