Balanced cut tree method for Ward hierarchical clustering

Dear colleagues, the current version of the cut_tree function for the ward hierarchical clustering only accepts two cutting options, (a) by setting the tree height, or (b) by setting a specific number of resulting clusters.
By setting one of these parameters, probably you will end up having a few big clusters (where the number of data samples is high), and many small clusters (each containing very few data samples). Thus, the resulting clustering is unbalanced, i.e. it contains clusters of very variable size.
I recently developed a balanced cut tree method which addresses this specific issue.
The proposed method looks recursively along the hierarchical tree, from the root (single cluster gathering all the samples) to the leaves (i.e. the clusters with only one sample), retrieving the biggest possible clusters containing a number of samples lower than a given maximum. Since all output clusters contain no more than a given maximum number of samples, the resulting clustering is considered to be more balanced than the standard tree cut.
If you consider this method plausible, useful and interesting, I would be very happy to integrate it into the main ScyPy repo. I would be open to consider all proposed improvements and amendments, so that the function integrates smoothly within the ScyPy framework.
Many thanks in advance for any comments and suggestions.

Hi @v-reyes, thanks for proposing this. I have two questions for now:

Is your implementation based on a published paper?
You added a GPL license to your repo, which is incompatible with SciPy's license. Would you mind relicensing to BSD-3 so we could accept code based on it under SciPy's license?

Hi @rgommers, many thanks for comments and questions.

That is a very good point. In fact, I am not aware of any publication describing directly this method, nor any previous implementation of it (althought it is indeed a trivial variation of the standard cut tree method). I can take look next week for relevant citations in the literature. If I do not find anything, should I proceed and write myself a paper? Which journal do you think would be appropriate? Would be there any alternatives to classical journal papers?
Already done. Is it ok like that, or should I change something within the license? (I am very flexible on this point).
Again, many thanks for your support.

If I do not find anything, should I proceed and write myself a paper?

That does seem like a lot of overhead for a pull request, so I definitely wouldn't suggest that unless you're interested in writing a paper anyway (in that case, please do!). A paper is just helpful, because then we have some external validation that this algorithm is an improvement over what we already have. If it's a small variation though, we can apply common sense and thorough review:) Your implementation does seem to have an example that demonstrates the improvement. Maybe you can add that here, with some dendrograms to show the effect?

Is it ok like that, or should I change something within the license? (I am very flexible on this point).

that's excellent, thanks

Hi @rgommers, many thanks again for your very useful comments.

That does seem like a lot of overhead for a pull request, so I definitely wouldn't suggest that unless you're interested in writing a paper anyway.

Actually I was not planing to write a paper describing this method. In principle I would do that only if necessary for trying the pull request. Following your suggestion, I think that a thorough review of the method and the code would be very appropriate and beneficial. Anyway, I will look on the next days for proper references that might describe this idea. For instance, an algorithm included in the ELKI project follows a similar idea (although the algorithm is different, i.e. based on k-means instead of hierarchical clustering).

Your implementation does seem to have an example that demonstrates the improvement. Maybe you can add that here, with some dendrograms to show the effect?

That's a very sensible comment. As a first step, I updated the Readme in order to document better the intention of the new method and its improvement over the established tree cut method. Further, I will work on the following days on your suggestion, i.e. providing the corresponding dendrograms to visually illustrate the main idea.
Many thanks & best regards

Hi @rgommers, following your suggestions I updated the code and the Readme in order to:

Provide dendrograms to visually illustrate the intention of the method and the main results, and
Provide relevant links describing related work (which might be used as references for the method).

Do you think these enhancements are helpful? What would be the next steps in order to proceed with a PR?
Many thanks & best regards

Thanks @vreyespue. Yes, I can see the value of this method. Dendrograms are nice. The dynamic tree cut paper has >1000 citations, so there's clearly a need.
Looking at what's being done, this is a little different from tree_cut, so I'm not sure it makes sense to combine them in a single function. Your algorithm is more similar to dynamic tree cut. It would actually also be good to compare to https://github.com/kylessmith/dynamicTreeCut.
My current impression is that a new function would be good. Unfortunately tree_cut is poorly documented - for example a simple explanation of what an ultrametric tree is would be useful.
@kylessmith, as the author of dynamicTreeCut you may be interested in this. If you have an expert opinion on whether the proposed algorithm belongs in scipy.cluster as a separate new function or inside cut_tree as an option, that would be great to hear. If dynamicTreeCut was license-compatible, it seems like that could have fit in nicely as well.

What would be the next steps in order to proceed with a PR?

opening a PR would be good. for now I'd suggest a new function (cut_tree_balanced?)

Hi @rgommers, as always many thanks for your comments and thoughts.

Your algorithm is more similar to dynamic tree cut. It would actually also be good to compare to https://github.com/kylessmith/dynamicTreeCut.

Yes, this is a very sensible task to do. Obviously the dynamicTreeCut method is more complex and advanced than mine. I will work in the next days on this point, so that we can at least provide some results from both methods.

If dynamicTreeCut was license-compatible, it seems like that could have fit in nicely as well.

I also think so. Definitely the dynamicTreeCut method would be very valuable for the SciPy framework. It would be great if @kylessmith is open to collaborate on this point.

opening a PR would be good. for now I'd suggest a new function (cut_tree_balanced?)

I will first do a small comparison between both discussed methods, and then we can decide on how to proceed. I agree with you, probably it would be better to add the method as a separate function.
Many thanks and best regards

Dear @rgommers,
I performed a comparison between the results from my method, and the results from the dynamicTreeCut in Python. In order to do so, I used a small piece of code which generates the same input data as in my example script, and generates from these data a similar number of dynamic clusters.
from dynamicTreeCut import cutreeHybrid
from scipy.spatial.distance import pdist
from scipy.stats import gamma
import numpy as np
from scipy.cluster.hierarchy import linkage

# Initialize the random seed
np.random.seed(5)

# Create a input matrix containing 100 data samples with 4 dimensions. Note: random sample
# from gamma distribution in order to obtain an unbalanced clustering (see below)
X = gamma.rvs(0.1, size=400).reshape((100,4))

# Compute the distances and linkage matrix using the scipy methods
distances = pdist(X, "euclidean")
link = linkage(distances, "average")

# Perform the tree cut by usind the dynamicTreeCut library. Note: the parameters minClusterSize
# and cutHeight were set in order to obtain 20 clusters as result (see below)
clusters = cutreeHybrid(link, distances, minClusterSize=1, cutHeight=0.1,
    respectSmallClusters = True)

# Use only the labels from the returned dictionary as result
standard_cut_cluster_id = clusters["labels"]

# Print information about the resulting clusterings
print("Type of the standard clustering result: %s" % type(standard_cut_cluster_id))
print("Shape of the standard clustering result (one cluster id per data sample): %s" %
    str(standard_cut_cluster_id.shape))
print("First 10 rows of the standard clustering result (one cluster id per sample):")
print(str(standard_cut_cluster_id[0:10].reshape(10)) + " ...")
standard_cluster_values, standard_cluster_counts = np.unique(standard_cut_cluster_id,
    return_counts=True)
print("Total number of resulting clusters = %s" % standard_cluster_values.shape[0])
print("For each resulting cluster: Cluster ID")
print(standard_cluster_values)
print("For each resulting cluster: Count of data samples")
print(standard_cluster_counts)
print("Count of data samples per cluster: mean = %d, max = %d, min = %d, std = %.2f" %
    (np.mean(standard_cluster_counts), np.max(standard_cluster_counts),
    np.min(standard_cluster_counts), np.std(standard_cluster_counts)))
print('')

As shown below, the output is a numpy array of 100 elements, assigning one cluster ID to each input vector (of 4 dimensions). Note that the ID of the resulting clusters go from 0 to 19 in this case. The resulting clustering is unbalanced, i.e. containing two big clusters (where the number of data samples is 19 and 32, respectively), and many small clusters (each containing 5 or less data samples). As result, the range of cluster sizes goes from 1 to 32, showing a standard deviation of 7.20 data samples.
Type of the standard clustering result: <type 'numpy.ndarray'>
Shape of the standard clustering result (one cluster id per data sample): (100,)
First 10 rows of the standard clustering result (one cluster id per sample):
[ 3  7  1 12  1  0  5  0  1  0] ...
Total number of resulting clusters = 20
For each resulting cluster: Cluster ID
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]
For each resulting cluster: Count of data samples
[19 32  5  5  4  4  3  3  3  2  2  2  2  2  2  2  2  2  2  2]
Count of data samples per cluster: mean = 5, max = 32, min = 2, std = 7.20

Note that the resulting clustering from my method is more balanced (for an equal number of resulting clusters), since the range of cluster sizes goes there from 1 to 10, showing a standard deviation of 2.68 data samples. Please see the Readme in my repo for more information.
Further, please note that I did not include this comparison within my repo, since the dynamicTreeCut in Python is not part of the SciPy stablished framework, nor it is very well documented (as its R counterpart). Therefore, adding more unnecessary comparisons into my repo might be rather confusing, and not beneficial for the potential user.
All in all, this small experiment already shows that my method adds value to both the standard cut_tree and the dynamicTreeCut methods, and thus it might considered as independent contribution to the SciPy framework on its own (obviously only if you agree).
I have two small questions before proceeding with the arrangement of a suitable PR.

Should I include my cut_tree_balanced() function within the main 'hierarchy.py' file? If yes, just below the standard cut_tree() function?
Should I include the tests already in the first version of the PR? If so, should I include them within the the main 'test_hierarchy.py' file?
Are benchmarks strictly necessary?
I saw in the SciPy docu that the PEP8 and doctest standards should be followed. Are there any other standards I should take into consideration?

As always, many thanks for your patience and for your kind support.
Best regards :)

Hi @vreyespue, thanks for the update. Quick answers:

yes
yes
no, but they would be nice and are quite easy to write:)
no (just following similar code style and test arrangement as we have should be fine)

Hi @rgommers, as discussed I already prepared the corresponding PR: https://github.com/scipy/scipy/pull/10730
Please let me know if you need further information.
Many thanks & best regards

Three questions:

Is this available in the current SciPy version?
Can this be used with non-ward linkage?
Do you know of any dendrogram cutting algorithms like cuttree dynamic that are available in Python? I've had issues with the Python port and have been using rpy2 but the dependencies are always horrendous.

Hi @jolespin, many thanks for your interest. About your questions:

No, it isn't. It has been added as a PR which has been planned for the 1.6.0 milestone. However, please note that the PR is already more than one year old, so I doubt this will be ever added into the official SciPy (probably very low priority). But maybe @rgommers or @tylerjereddy can provide some more information about this.
Good question. Actually I think it should be possible (no warranty). The method ward_cut_tree_balanced() takes as input a linkage matrix of type ndarray (Z, see the documentation at the original repo). Note that Z can be created using the ward method, or any other linkage method.
I quoted some further methods within the documentation at the original repo (see Related Work section by the end). I think you might be interested on the translation of the dynamicTreeCut method to Python.

Best regards.

@vreyespue The reason why I asked if the other linkage metrics could be used is because I know ward linkage works specifically on Euclidean distances so I wasn't sure if this was a similar situation.
Regarding dynamicTreeCut in Python, I've had compatibility issues https://github.com/kylessmith/dynamicTreeCut/issues/9 and the author hasn't addressed them nor has there been any activity on the project for a few years so I feel that this method is abandoned.
What is needed to make the PR a reality in the coming versions?

@jolespin That's a very good point. The method just works with any other linkage method, but at this point I am not sure whether the result would be mathematically correct in case a distance different than Euclidean is used. Food for thought.
In essence the balanced method would take similar inputs as the standard cut_tree function. So in principle I would argue that anything that works for the standard cut_tree function should also work for the balanced version. But in any case, I would have to work on this issue to be sure.
About the PR, I can only say that the review is still in process, and that I am eager to apply any fixes and changes to the code necessary to include the function into the official SciPy repo. Maybe @rgommers or @tylerjereddy can provide some more information about this point.

That’s great to know that it’s somewhat active.  I use hierarchical clustering for so many projects and it’s such an intuitive analysis of diversity.  I’m not sure if you’ve written a paper on it but if this was easily accessible I would certainly use this and cite in papers.  Right now, my only other option really is dynamicTreeCut which has its own issues and the Python port is a dead github archive. This is literally the only other Python alternative so it would be awesome to have it in SciPy and citeable.

@jolespin Now I remember that @ljmartin was working on a balanced_cut version for Scikit (see link1, link2, link3), which was actually inspired by my own method. But I don't know whether that Scikit function is already available.
Certainly it would be great to get this method available for citations in papers. I was waiting for its inclusion into the official ScyPy, but it is taking a long time (@rgommers or @tylerjereddy might know how long it might take).
In the meanwhile, maybe we could write a small paper (in Arxiv or something like that) describing the method and citing our own repos. Please let me know if you would be interested in a co-authorship, or if you would know somebody who might be interested.

Hi @vreyespue , the history of that implementation was that I wanted to use your nice balanced cut approach on large dendrograms (i.e. more than 10,000 data points). In my case, I had about 300k items, so a 300k x 300k ndarray as gets created in the current implementation would be too large for memory. That one is merged in scikit-network now ( https://github.com/sknetwork-team/scikit-network/blob/e1bfad780b6781c8af32716a282dda6ebec78ea9/sknetwork/hierarchy/postprocess.py#L122 ) - it does the same thing but is 'online' in the sense that it iterates through the dendrogram once without creating big arrays.
I'd be happy to contribute something to a paper - it's been really useful thanks!

Hi @ljmartin, you are right, my implementation for SciPy does not show a good performance on large dendrograms. Actually its memory consumption should be similar to the standard cut_tree() function, if I remember correctly.
Anyhow, I think your implementation for scikit-network is very elegant, and it is great to hear that it shows a good performance. That's great news 😄 👍
I think the most important would be to provide something that can be quoted in papers. @jolespin do you think the implementation for scikit-network could be enough for that purpose? The implementation for SciPy should be eventually available, but I don't know when (might take some time, @rgommers could provide more info about this).
If that helps, I would be also eager to contribute something to a paper, but I am not sure about the format, the length, etc. In addition, I do not have at the moment any academic affiliation, which is necessary for publishing in many journals. If you know of any workaround for this, that would be very helpful. Please let me know what you think.
Best regards & many thanks!

Is it the same implementation you created for your project?
https://scikit-network.readthedocs.io/en/latest/reference/hierarchy.html#sknetwork.hierarchy.cut_balanced
If so, then yes that is awesome it's already available!

Hi @jolespin, as far as I understand, the implementation is not identical (since it is based in different data structures, i.e.  scikit-network dendrogram vs. scipy linkage matrix). But I think the idea behind is basically the same. Maybe @ljmartin can provide more info on this point.
About quoting this method, I don't know if it is possible to quote the scikit-network repo. As I understood, it is not part of the official scikit framework. Maybe @ljmartin can provide some more info on this point as well.
In any case, I think it is great to have two implementations of the same method available 😄 👍
Many thanks & best regards

the implementation in scikit-network just works on a dendrogram. They implement dendrograms in the same way scipy does, so it should be a complete drop-in replacement

I proceed to close this issue and the related PR due to its low priority. Many thanks for your consideration and best regards.


make Scan1 results available as numpy record array

This is a small addition to the Scan1() method to make the results of the scan available as a record array. The fields of the record array are created automatically from a concatenated list of [mod.scan_in]+mod.scan_out and populated with the numerical results from mod.scan_res. Existing functionality of mod.scan_res is unchanged. This  convenience function will make it very easy to plot selective scan results when exploring models interactively, e.g.
plt.plot(mod.scan.Vf1, mod.scan.J_R1)

It is also easy to do post-operations on scan results, e.g. calculate concentration ratios.
Comments?

Hi Johann
In principle this is a good idea the only minor drawback with the current proposal is that the scan results are stored twice ... which is not so good for very large scans. What do you think of the following type thing that uses a function to generate the new array.
def getScanResAsRecArray():

return
Then all a user would have to do is:
mod.Scan1(rng)
scan_rec_arr = mod.getScanResAsRecArray()
This way we only store the data in one place.
b.

Hi Brett
Your point is well taken, and indeed had occurred to me as well. While your solution gets rid of the problem, it is almost as "bad" in terms of required typing as what I've been using in scripts up to now:
scan_rec_arr = np.rec.fromrecords(mod.scan_res, names=[mod.scan_in]+mod.scan_out)
Your solution is still a lot of typing. I'm thinking of a use case where one is interactively exploring a model by scans and then wants to plot or otherwise work with the data. Most of the times these are smaller models (up to say 20 rxns) with scans of say 100 points, so the size of the scan_res array is not an issue in terms of memory usage.
I agree that it can become an issue for very large scans or a very large number of scan outputs. What about a flag you can change in mod.__settings__? We can discuss whether to create the recarrary by default or not :-) The point is that your proposed solution still creates a duplicate of the results when you call your getScanResAsRecArray() method. And I don't want to call that method after each scan.
Again, this stems from my personal use case. I agree that there needs to be a way to switch it off.
J.

Hi Johann
My editor has text completion, but point taken :-) The advantage of a get
method is that it returns a copy, but doesn't store it as part of the model
object itself, setting a flag is a good idea:
boolean create_scan_res_recarray (default=True)
Flag sets whether a duplicate recarray is created or not (current plan)
or
boolean scan_res_is_recarray (default=False)
Flag sets scan_res is either of type ndarray or recarray (and we update
PyscesModel to deal with both)
Cheers
Brett
On Wed, Nov 25, 2015 at 9:21 AM, Johann Rohwer<EMAIL_ADDRESS>wrote:

Hi Brett
Your point is well taken, and indeed had occurred to me as well. While
your solution gets rid of the problem, it is almost as "bad" in terms of
required typing as what I've been using in scripts up to now:
scan_rec_arr = np.rec.fromrecords(mod.scan_res,
names=[mod.scan_in]+mod.scan_out)
Your solution is still a lot of typing. I'm thinking of a use case where
one is interactively exploring a model by scans and then wants to plot or
otherwise work with the data. Most of the times these are smaller models
(up to say 20 rxns) with scans of say 100 points, so the size of the
scan_res array is not an issue in terms of memory usage.
I agree that it can become an issue for very large scans or a very large
number of scan outputs. What about a flag you can change in
mod.settings? We can discuss whether to create the recarrary by
default or not :-) The point is that your proposed solution still creates a
duplicate of the results when you call your getScanResAsRecArray()
method. And I don't want to call that method after each scan.
Again, this stems from my personal use case. I agree that there needs to
be a way to switch it off.
J.
—
Reply to this email directly or view it on GitHub
https://github.com/PySCeS/pysces/pull/16#issuecomment-159531867.

--
Brett G. Olivier

If you update PySCeSModel to deal with a recarray I can see no reason as to why one should not turn scan_res into a recarray finish an klaar. However, this will involve some work as Scan1() as well as everything that depends on scan_res will have to be re-worked. Dunno how much work this is and if it is worth the effort. In any event it is trivial to retrieve the underlying ndarray from a recarray using .view('<f8'), but it always creates a new object.

I have a suggestion:
A possible solution could be leave scan_res in order to preserve the API and then make scan a property of the model that is "lazy loaded". An implementation could look like this:
In the PysMod init add a self._scan = None
def __init__(*args):
    ....
    self._scan = None

and then set the property up like this:
@property
def scan(self):
    if not self._scan:
        self._scan = np.rec.fromrecords(self.scan_res,
                                        names = [self.scan_in] + self.scan_out)
    return self._scan

Now no extra setting will be needed and scan (actually _scan) will not be populated unless a user actually tries to access it. Even though scan is set up like a method it is accessed like a normal variable field because of the property decorator.
I use this pattern all the time in PySCeSToolbox in order to prevent CPU intensive calculations from taking place during object instantiation (they are only done when a user actually wants to access the data).

Interesting idea. Running a new scan would simply have to set self._scan =
None otherwise it wouldn't update.
On Wed, Nov 25, 2015 at 10:57 AM, Carl Christensen<EMAIL_ADDRESS>
wrote:

I have a suggestion:
A possible solution could be leave scan_res in order to preserve the API
and then make scan a property of the model that is "lazy loaded". An
implementation could look like this:
In the PysMod init add a self._scan = None
def init(*args):
....
self._scan = None
and then set the property up like this:
@propertydef scan(self):
if not self._scan:
self._scan = np.rec.fromrecords(self.scan_res,
names = [self.scan_in] + self.scan_out)
return self._scan
Now no extra setting will be needed and scan (actually _scan) will not be
populated unless a user actually tries to access it. Even though scan is
set up like a method it is accessed like a normal variable field because of
the property decorator.
I use this pattern all the time in PySCeSToolbox in order to prevent CPU
intensive calculations from taking place during object instantiation (they
are only done when a user actually wants to access the data).
—
Reply to this email directly or view it on GitHub
https://github.com/PySCeS/pysces/pull/16#issuecomment-159555596.

--
Brett G. Olivier

oops.. forgot about that, but yes, that's you would do it.

OK to sum up the discussion I see two options for the way forward:

Implement the scan recarray with lazy loading and stick with the current implementation of scan_res. This would leave us with the issue of duplicated data.
Implement scan_res as a recarray from scratch. This is my preferred solution, but could break existing scripts using scan_res in terms of backward compatibility. I'm not too sure how to handle this or how serious it is. One option is to do the recarray implementation but give the array a new name (e.g. scan_data), and then also use this in Scan1Plot(). One could then implement a lazy loading of scan_res that would create the (data duplicated) ndarray, but only when required by legacy scripts (which would be the minority cases). One could even issue deprecation warnings. This configuration is thus the "other way round" compared to the suggestion by Carl.

Ideas? Thoughts? I actually have some time on hand to do some coding, so can get stuck in once we agree :-)

this would be on-demand so I could live with it i.e. not generating both
arrays by default all the time
whatever we do we should try to keep backwards compatibility for at
least a few releases or
implement scan_res_r as the default holder of scan results (recarray,
used internally) and scan_res as a lazy load (duplicated on demand so ok)
ndarray

On Wed, Nov 25, 2015 at 12:50 PM, Johann Rohwer<EMAIL_ADDRESS>wrote:

OK to sum up the discussion I see two options for the way forward:

Implement the scan recarray with lazy loading and stick with the
current implementation of scan_res. This would leave us with the issue
of duplicated data.
Implement scan_res as a recarray from scratch. This is my preferred
solution, but could break existing scripts using scan_res in terms of
backward compatibility. I'm not too sure how to handle this or how serious
it is. One option is to do the recarray implementation but give the array a
new name (e.g. scan_data), and then also use this in Scan1Plot(). One
could then implement a lazy loading of scan_res that would create the
(data duplicated) ndarray, but only when required by legacy scripts (which
would be the minority cases). One could even issue deprecation warnings.
This configuration is thus the "other way round" compared to the suggestion
by Carl.

Ideas? Thoughts? I actually have some time on hand to do some coding, so
can get stuck in once we agree :-)
—
Reply to this email directly or view it on GitHub
https://github.com/PySCeS/pysces/pull/16#issuecomment-159584950.

--
Brett G. Olivier

The lazy loading of scan_res and a new scan_res_r (or scan_data) recarray as the default scan results container sounds like the best idea to me.
Sorry to bring up another option, but it should be possible to create a new object that behaves like a recarray and like a ndarray (allowing a user to access data using traditional indexing and column names). This would obviously solve both problems, however I don't know exactly how you would implement it.

Carl I am not sure if this is possible. An ndarray and recarray have different shapes. A recarray sees every record as an item. To convert a recarray to an ndarray you have to get a view on it and then reshape according to the original dimensions of the ndarray:
In [1]: a = np.arange(10.0)

In [2]: a = a.reshape((5,2))

In [3]: a
Out[3]:
array([[ 0.,  1.],
       [ 2.,  3.],
       [ 4.,  5.],
       [ 6.,  7.],
       [ 8.,  9.]])

In [4]: b = np.rec.fromrecords(a, names=['first','second'])

In [5]: b
Out[5]:
rec.array([(0.0, 1.0), (2.0, 3.0), (4.0, 5.0), (6.0, 7.0), (8.0, 9.0)],
      dtype=[('first', '<f8'), ('second', '<f8')])

In [6]: b.shape
Out[6]: (5,)

In [7]: a.shape
Out[7]: (5, 2)

In [8]: b.view()
Out[8]: rec.array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])

In [15]: b.view('<f8').reshape(5,2)
Out[15]:
array([[ 0.,  1.],
       [ 2.,  3.],
       [ 4.,  5.],
       [ 6.,  7.],

If you do come up with something, let me know. But I suspect that such an object would have additional computational overhead since it can behave in "both ways". Not sure this is something we desire...

What I have in mind will mostly behave like an ndarray but will also allow you to access results like you would in a recarray (rec.col_name or rec['col_name'])
Such an object would inherit from ndarray or (2) uses an array to store data internally. The only difference would be keep track of column names and the '_getitem_' and '_setitem_' methods (which allows you to access the data stored in the array with square brackets) would be overwritten. You would also create fields for each of the columns which would contain a view of that column. This is not a full implementation but here's an idea:
class TestArr(np.ndarray):
    def __init__(self,*args,**kwargs):
        super(TestArr, self).__init__(*args, **kwargs)
        self.column_names = ('a','b')
    def __getitem__(self,z):
        if type(z) == str:
            col = self.column_names.index(z)
            return self[:,col]
        else:
            return np.ndarray.__getitem__(self,z)

t = TestArr(shape=(2,2))

t[0,0] = 1.0
t[0,1] = 2.0
t[1,0] = 1.0
t[1,1] = 2.0

t
output: TestArr([[ 1.,  2.],
                [ 1.,  2.]])

t[:,0]
output : TestArr([ 1.,  1.])
t['a']
output : TestArr([ 1.,  1.])

Sorry but I don't like this idea at all. Subclassing stuff in this case is
overkill, never mind performance, serialization and maintenance. I think in
the end it's either lazy import of one or the other arrays or we just live
with the duplication until it becomes a problem ... then you can always
manually delete one of the objects.
After consideration most high density scans are going to probably be done
with the n-dimensional scanner, so, the use case for Scan1 here is more
like Johann's, let's just duplicate the results to "scan_res_r" for now.
On Wed, Nov 25, 2015 at 2:45 PM, Carl Christensen<EMAIL_ADDRESS>wrote:

What I have in mind will mostly behave like an ndarray but will also allow
you to access results like you would in a recarray (rec.col_name or
rec['col_name'])
Such an object would inherit from ndarray or (2) uses an array to store
data internally. The only difference would be keep track of column names
and the 'getitem' and 'setitem' methods (which allows you to access
the data stored in the array with square brackets) would be overwritten.
You would also create fields for each of the columns which would contain a
view of that column. This is not a full implementation but here's an idea:
class TestArr(np.ndarray):
def init(self,*args,**kwargs):
super(TestArr, self).init(*args, **kwargs)
self.column_names = ('a','b')
def getitem(self,z):
if type(z) == str:
col = self.column_names.index(z)
return self[:,col]
else:
return np.ndarray.getitem(self,z)
t = TestArr(shape=(2,2))
t[0,0] = 1.0
t[0,1] = 2.0
t[1,0] = 1.0
t[1,1] = 2.0
t
output: TestArr([[ 1.,  2.],
[ 1.,  2.]])
t[:,0]
output : TestArr([ 1.,  1.])
t['a']
output : TestArr([ 1.,  1.])
—
Reply to this email directly or view it on GitHub
https://github.com/PySCeS/pysces/pull/16#issuecomment-159611687.

--
Brett G. Olivier

OK I've added a commit that uses lazy loading for the record array.

I've opted to keep the ndarray for mod.scan_res as default (i.e. the current situation), and calculate the record array on demand using lazy loading. The reason is that the whole PyscesPlot2 framework (which is also used by Scan1Plot() is written with ndarrays in mind. While the individual plotting functions of Scan1Plot() might be simpler to implement using record arrays, this is not the case for the whole plotting framework, and we don't want to re-write that.
I've opted to keep the name m.scan for the record array in favour of m.scan_res_r or m.scan_data for the following reasons:

I want to keep typing as short as possible (as already mentioned above). The main use cases I see for this are to interactively plot flux or concentration ratios, for example, and then the extra typing makes a huge difference. Compare mod.scan.J_R1/mod.scan.J_R2 with mod.scan_res_r.J_R1/mod.scan_res_r.J_R2 for example.
There are no naming conflicts or clashes.
Backward compatibility is maintained.

I trust this is now ready for merging :-)

Looks good to me ... go for it!
On a related note. If there is any other stuff that needs to be fixed/added
now would also be a good time as we should roll out a new release soon. The
matplotlib/numpy compatibility fixes are needed for the latest Anaconda
(and probably Ubuntu) releases :-)
cheers
b,
On Thu, Nov 26, 2015 at 12:57 PM, Johann Rohwer<EMAIL_ADDRESS>wrote:

OK I've added a commit that uses lazy loading for the record array.

I've opted to keep the ndarray for mod.scan_res as default (i.e. the
current situation), and calculate the record array on demand using lazy
loading. The reason is that the whole PyscesPlot2 framework (which is
also used by Scan1Plot() is written with ndarrays in mind. While the
individual plotting functions of Scan1Plot() might be simpler to
implement using record arrays, this is not the case for the whole plotting
framework, and we don't want to re-write that.
I've opted to keep the name m.scan for the record array in favour of
m.scan_res_r or m.scan_data for the following reasons:

I want to keep typing as short as possible (as already mentioned
above). The main use cases I see for this are to interactively plot flux or
concentration ratios, for example, and then the extra typing makes a huge
difference. Compare mod.scan.J_R1/mod.scan.J_R2 with
mod.scan_res_r.J_R1/mod.scan_res_r.J_R2 for example.
There are no naming conflicts or clashes.
Backward compatibility is maintained.

I trust this is now ready for merging :-)
—
Reply to this email directly or view it on GitHub
https://github.com/PySCeS/pysces/pull/16#issuecomment-159894717.

--
Brett G. Olivier


Make compile emits error.

Observed behavior
Make install gives the error Recursive 'require' for feature 'org'.
M-x org-version gives 9.1.6-elpa
System information
I have also added the ("org" . "https://orgmode.org/elpa/") package-archive.

Click to expand
DOOM Doctor
Running Emacs v25.3.2, commit d465c6199249fb6e6e8b49474a04f94e7d5e3cdd

shell: /bin/zsh
Compiled with:
  XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
  NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
  TOOLKIT_SCROLL_BARS GTK3 X11 MODULES
uname -a:
  Linux zeronone-ubuntu 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec
  4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Attempt to load DOOM: success!

I'm getting the same error:
Importing package-keyring.gpg...

Reloading active Emacs session...

These issues have been fixed in develop. The 2.1 release is this weekend, so they will be in master soon.

Well, it certainly didn't come out that weekend! Haha.
It'll be up this weekend. 2.1 is already on develop as of two days ago, minus the new documentation. That's all there is left to do. Doom's more adventurous users are putting it through its rounds now.


Add confidence inteval (or sigma) to date model

"Some cite the uncertainty at 1σ and some at 2σ (both of which implicitly assume Gaussian statistics, as is the case with luminescence dating), and others give age ranges at the 68% or 95% confidence intervals (such as 14C dating, as the calibrated age distributions are often not normally distributed)"

currently I try to upload the 95% CI and the 2sigma values, but I make mistakes, so it would be cool to add this as part of the model!

[ ] Make sure the upload doesnt break
[ ] Add the "sigma" sign to view if its not %


start container then no display

when i start a container eg firefox or steam the there is no display in screen, just black
nvidia-smi
Fri Aug 30 16:21:04 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 D      Off |   00000000:02:00.0 Off |                  Off |
| 37%   57C    P2             60W /  425W |    7064MiB /  24564MiB |      5%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090 D      Off |   00000000:82:00.0 Off |                  Off |
| 33%   55C    P2             67W /  425W |    6277MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

nvidia-container-cli -V
cli-version: 1.16.1
lib-version: 1.16.1
build date: 2024-07-23T14:57+00:00
build revision: 4c2494f16573b585788a42e9c7bee76ecd48c73d
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

wolf logs
08:17:11.914621393 INFO  | RTP server started on port: 48200
08:17:11.914628023 INFO  | RTP server started on port: 48100
08:17:12.006325936 WARN  | Nvidia DRM is not loaded with the flag modeset=1
Please read the docs at https://games-on-whales.github.io/wolf/stable/user/quickstart.html
08:17:12.035325162 INFO  | NVIDIA_DRIVER_VOLUME_NAME not set, assuming nvidia driver toolkit is installed..
1:37:24.335055637     1 0x7f8740000e20 WARN        cudaconvertscale gstcudaconvertscale.c:1396:gst_cuda_base_convert_set_info:<cudaconvertscale7> Can't calculate borders
1:37:24.335195142     1 0x7f8740000e20 WARN           cudaconverter gstcudaconverter.c:2104:gst_cuda_converter_setup:<cudaconverter4> Couldn't compile to cubin, trying ptx
1:37:24.426632118     1 0x7f86f4d5b5e0 WARN                  appsrc gstappsrc.c:2469:gst_app_src_push_internal:<wolf_wayland_source> Dropping old item buffer: 0x7f8708569420, pts 0:00:00.066666664, dts 0:00:00.066666664, dur 0:00:00.016666666, size 3686400, offset none, offset_end none, flags 0x0
08:17:13.777711534 INFO  | [DOCKER] Starting container: /WolfSteam_16920338566807001700
[2024-08-30 09:22:55]
[2024-08-30 09:22:55] [ /etc/cont-init.d/10-setup_user.sh: executing... ]
[2024-08-30 09:22:55] **** Configure default user ****
[2024-08-30 09:22:55] Container running as root. Nothing to do.
[2024-08-30 09:22:55] DONE
[2024-08-30 09:22:55]
[2024-08-30 09:22:55] [ /etc/cont-init.d/15-setup_devices.sh: executing... ]
[2024-08-30 09:22:55] **** Configure devices ****
[2024-08-30 09:22:55] Exec device groups
[2024-08-30 09:22:56] Adding user 'root' to groups: gow-gid-107,root
[2024-08-30 09:22:56] DONE
[2024-08-30 09:22:56]
[2024-08-30 09:22:56] [ /etc/cont-init.d/30-nvidia.sh: executing... ]
[2024-08-30 09:22:56] Nvidia driver detected, assuming it's using the nvidia driver volume
/sbin/ldconfig.real: /usr/local/lib/liblibgstwaylanddisplay.so.0.3 is not a symbolic link

[2024-08-30 09:22:56]
[2024-08-30 09:22:56] [ /etc/cont-init.d/init-gamescope.sh: executing... ]
[2024-08-30 09:22:56] Launching the container's startup script as user 'root'
Stack trace (most recent call first):
#0  0x000056356aabde26 in shutdown_handler(int) at /wolf/src/moonlight-server/./exceptions/exceptions.h:69:5
#1  0x00007f87aa29331f at /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f874ea34185 at /lib/x86_64-linux-gnu/libnvidia-eglcore.so.550.107.02
#3  0x00007f87ad2530e0 at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#4  0x00007f87ad155ce7 at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#5  0x00007f87ad1a8102 at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#6  0x00007f87ad1969e7 at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#7  0x00007f87ad15ab47 at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#8  0x00007f87ad116ccf at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#9  0x00007f87ad11aa7d at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#10 0x00007f87ad31274a at /usr/local/lib/liblibgstwaylanddisplay.so.0.3
#11 0x00007f87aa2eaa93 at /lib/x86_64-linux-gnu/libc.so.6
#12 0x00007f87aa377a33 at /lib/x86_64-linux-gnu/libc.so.6
08:22:57.025274103 INFO  | Gstreamer version: 1.25.0-1
08:22:57.032167178 INFO  | Reading config file from: /etc/wolf/cfg/config.toml
0:00:00.144669361     1 0x55ce85e6c960 WARN               cudanvrtc gstcudanvrtc.cpp:177:gst_cuda_nvrtc_load_library_once: Failed to load 'nvrtcGetCUBINSize', 'nvrtcGetCUBINSize': /usr/local/nvidia/lib/libnvrtc.so: undefined symbol: nvrtcGetCUBINSize
08:22:59.907399265 INFO  | Using H264 encoder: nvcodec
08:22:59.908660590 INFO  | Using HEVC encoder: nvcodec
08:22:59.908995528 INFO  | Using AV1 encoder: nvcodec
08:22:59.937546167 INFO  | RTSP server started on port: 48010
08:22:59.937875952 INFO  | Control server started on port: 47999
XDG_RUNTIME_DIR (/tmp/sockets) is not owned by us (uid 0), but by uid 1000! (This could e.g. happen if you try to connect to a non-root PulseAudio as a root user, over the native protocol. Don't do that.)
08:22:59.938279884 WARN  | [PULSE] Unable to connect, Access denied
08:22:59.938575069 INFO  | Starting PulseAudio docker container
08:22:59.938592699 INFO  | HTTP server listening on port: 47989
08:22:59.940873289 WARN  | [DOCKER] Container WolfPulseAudio already present, removing first
08:22:59.941187763 INFO  | HTTPS server listening on port: 47984
08:23:00.440584795 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:00.605836484 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:03.794760309 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:03.891919760 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:06.971596885 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:07.083878481 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:10.159973575 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:10.258621145 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:13.341602051 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:13.442440591 WARN  | Unable to get mac address of ip address: <IP_ADDRESS>, you can override this by settings the env variables WOLF_INTERNAL_MAC or WOLF_INTERNAL_IP
08:23:15.728240687 INFO  | RTP server started on port: 48100
08:23:15.728267396 INFO  | RTP server started on port: 48200
08:23:15.813228845 WARN  | Nvidia DRM is not loaded with the flag modeset=1
Please read the docs at https://games-on-whales.github.io/wolf/stable/user/quickstart.html
08:23:15.832500000 INFO  | NVIDIA_DRIVER_VOLUME_NAME not set, assuming nvidia driver toolkit is installed..
0:00:19.466494009     1 0x7fe630000e20 WARN        cudaconvertscale gstcudaconvertscale.c:1396:gst_cuda_base_convert_set_info:<cudaconvertscale3> Can't calculate borders
0:00:19.466651140     1 0x7fe630000e20 WARN           cudaconverter gstcudaconverter.c:2104:gst_cuda_converter_setup:<cudaconverter0> Couldn't compile to cubin, trying ptx
08:23:16.521726577 INFO  | [DOCKER] Starting container: /WolfFirefox_16920338566807001700
0:00:19.624060671     1 0x7fe63401a4f0 WARN                  appsrc gstappsrc.c:2469:gst_app_src_push_internal:<wolf_wayland_source> Dropping old item buffer: 0x7fe648679d00, pts 0:00:00.066666664, dts 0:00:00.066666664, dur 0:00:00.016666666, size 3686400, offset none, offset_end none, flags 0x0

container log
[2024-08-30 09:23:16]
[2024-08-30 09:23:16] [ /etc/cont-init.d/10-setup_user.sh: executing... ]
[2024-08-30 09:23:16] **** Configure default user ****
[2024-08-30 09:23:16] Setting default user uid=1000(retro) gid=1000(retro)
userdel: ubuntu mail spool (/var/mail/ubuntu) not found
useradd: warning: the home directory /home/retro already exists.
useradd: Not copying any file from skel directory into it.
[2024-08-30 09:23:16] Setting umask to 000
[2024-08-30 09:23:16] Ensure retro home directory is writable
[2024-08-30 09:23:16] Ensure XDG_RUNTIME_DIR is writable
[2024-08-30 09:23:16] DONE
[2024-08-30 09:23:16]
[2024-08-30 09:23:16] [ /etc/cont-init.d/15-setup_devices.sh: executing... ]
[2024-08-30 09:23:16] **** Configure devices ****
[2024-08-30 09:23:16] Exec device groups
[2024-08-30 09:23:16] Path '/dev/input/*' is not present.
[2024-08-30 09:23:16] Adding user 'retro' to groups: gow-gid-109,video,root
[2024-08-30 09:23:16] DONE
[2024-08-30 09:23:16]
[2024-08-30 09:23:16] [ /etc/cont-init.d/30-nvidia.sh: executing... ]
[2024-08-30 09:23:16] Nvidia driver detected, assuming it's using the nvidia driver volume
[2024-08-30 09:23:16] Creating symlink to nvidia-drm_gbm.so
'/usr/lib/x86_64-linux-gnu/gbm/nvidia-drm_gbm.so' -> '../libnvidia-allocator.so.1'
[2024-08-30 09:23:16] Creating json 10_nvidia.json file
[2024-08-30 09:23:16] Creating json nvidia_icd.json file
[2024-08-30 09:23:16]
[2024-08-30 09:23:16] [ /etc/cont-init.d/init-gamescope.sh: executing... ]
[2024-08-30 09:23:16] Launching the container's startup script as user 'retro'
[2024-08-30 09:23:16] [Sway] - Starting: `/usr/bin/firefox`
00:00:00.003 [ERROR] [sway/main.c:62] !!! Proprietary Nvidia drivers are in use !!!
00:00:00.022 [ERROR] [wlr] [EGL] command: eglQueryDmaBufModifiersEXT, error: EGL_BAD_PARAMETER (0x300c), message: "EGL_BAD_PARAMETER error: In eglQueryDmaBufModifiersEXT: Invalid format
"
00:00:00.022 [ERROR] [wlr] [render/egl.c:856] Failed to query dmabuf number of modifiers
00:00:00.022 [ERROR] [wlr] [EGL] command: eglQueryDmaBufModifiersEXT, error: EGL_BAD_PARAMETER (0x300c), message: "EGL_BAD_PARAMETER error: In eglQueryDmaBufModifiersEXT: Invalid format
"
00:00:00.022 [ERROR] [wlr] [render/egl.c:856] Failed to query dmabuf number of modifiers
src/nv_gbm.c:288: GBM-DRV error (nv_gbm_create_device_native): nv_common_gbm_create_device failed (ret=-1)

KMS: DRM_IOCTL_MODE_CREATE_DUMB failed: Permission denied
00:00:00.305 [ERROR] [wlr] [render/allocator/gbm.c:117] gbm_bo_create failed
00:00:00.305 [ERROR] [wlr] [render/swapchain.c:110] Failed to allocate buffer
KMS: DRM_IOCTL_MODE_CREATE_DUMB failed: Permission denied
00:00:00.305 [ERROR] [wlr] [render/allocator/gbm.c:117] gbm_bo_create failed
00:00:00.305 [ERROR] [wlr] [render/swapchain.c:110] Failed to allocate buffer
00:00:00.305 [ERROR] [wlr] [types/output/swapchain.c:109] Swapchain for output 'WL-1' failed test
00:00:00.305 [ERROR] [sway/config/output.c:525] Failed to commit output WL-1
00:00:00.331 [ERROR] [sway/config.c:694] Error on line 'workspace main; exec /usr/bin/firefox && killall sway': Can't run this command while there's no outputs connected.
dbus-daemon[130]: [session uid=1000 pid=130] Activating service name='org.freedesktop.systemd1' requested by ':1.0' (uid=1000 pid=139 comm="systemctl --user import-environment DISPLAY WAYLAN" label="docker-default (enforce)")
dbus-daemon[130]: [session uid=1000 pid=130] Activated service 'org.freedesktop.systemd1' failed: Process org.freedesktop.systemd1 exited with status 1
Failed to import environment: Process org.freedesktop.systemd1 exited with status 1
dbus-daemon[130]: [session uid=1000 pid=130] Activating service name='org.a11y.Bus' requested by ':1.3' (uid=1000 pid=136 comm="waybar -b bar-0" label="docker-default (enforce)")
dbus-daemon[130]: [session uid=1000 pid=130] Successfully activated service 'org.a11y.Bus'
[2024-08-30 09:23:17.169] [info] Using configuration file /home/retro/.config/waybar/config.jsonc
dbus-daemon[130]: [session uid=1000 pid=130] Activating service name='org.freedesktop.portal.Desktop' requested by ':1.2' (uid=1000 pid=136 comm="waybar -b bar-0" label="docker-default (enforce)")
dbus-daemon[130]: [session uid=1000 pid=130] Activating service name='org.freedesktop.portal.Documents' requested by ':1.5' (uid=1000 pid=160 comm="/usr/libexec/xdg-desktop-portal" label="docker-default (enforce)")
dbus-daemon[130]: [session uid=1000 pid=130] Activating service name='org.freedesktop.impl.portal.PermissionStore' requested by ':1.6' (uid=1000 pid=166 comm="/usr/libexec/xdg-document-portal" label="docker-default (enforce)")
dbus-daemon[130]: [session uid=1000 pid=130] Successfully activated service 'org.freedesktop.impl.portal.PermissionStore'
dbus-daemon[130]: [session uid=1000 pid=130] Successfully activated service 'org.freedesktop.portal.Documents'
fusermount3: fuse device not found, try 'modprobe fuse' first
error: fuse init failed: Can't mount path /tmp/sockets/doc
dbus-daemon[130]: [session uid=1000 pid=130] Activating service name='org.freedesktop.impl.portal.desktop.gtk' requested by ':1.5' (uid=1000 pid=160 comm="/usr/libexec/xdg-desktop-portal" label="docker-default (enforce)")
dbus-daemon[158]: Activating service name='org.a11y.atspi.Registry' requested by ':1.1' (uid=1000 pid=181 comm="/usr/libexec/xdg-desktop-portal-gtk" label="docker-default (enforce)")

(xdg-desktop-portal-gtk:181): xdg-desktop-portal-gtk-WARNING **: 09:23:17.270: error: Could not connect: No such file or directory

dbus-daemon[158]: Successfully activated service 'org.a11y.atspi.Registry'
SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
dbus-daemon[130]: [session uid=1000 pid=130] Successfully activated service 'org.freedesktop.impl.portal.desktop.gtk'

(/usr/libexec/xdg-desktop-portal:160): xdg-desktop-portal-WARNING **: 09:23:17.284: Failed to create RealtimeKit proxy: Could not connect: No such file or directory
[W][20759.916071] pw.conf      | [          conf.c: 1031 try_load_conf()] can't load config client.conf: No such file or directory
[E][20759.916081] pw.conf      | [          conf.c: 1060 pw_conf_load_conf_for_context()] can't load config client.conf: No such file or directory

(/usr/libexec/xdg-desktop-portal:160): xdg-desktop-portal-WARNING **: 09:23:17.295: Failed connect to PipeWire: Couldn't create PipeWire context
dbus-daemon[130]: [session uid=1000 pid=130] Successfully activated service 'org.freedesktop.portal.Desktop'
[2024-08-30 09:23:17.303] [info] Using CSS file /home/retro/.config/waybar/style.css

Hi @lyj0309
Taking a look at the logs, you do not have the nvidia drm module loaded.
08:17:11.914621393 INFO  | RTP server started on port: 48200
08:17:11.914628023 INFO  | RTP server started on port: 48100
08:17:12.006325936 WARN  | Nvidia DRM is not loaded with the flag modeset=1

On the quickstart guide I noticed the instructions for the container toolkit don't mention the steps below but they are still needed (will get that updated) https://games-on-whales.github.io/wolf/stable/user/quickstart.html

Also while looking at the logs I also noticed these errors
fusermount3: fuse device not found, try 'modprobe fuse' first
error: fuse init failed: Can't mount path /tmp/sockets/doc

there are few things mounted from the host into the Wolf container, you may also want to check the docker containers are running as a user that has permissions to /tmp/sockets
    environment:
      - XDG_RUNTIME_DIR=/tmp/sockets
    volumes:
      - /tmp/sockets:/tmp/sockets:rw

after install  nvidia drm, it works, thanks.


DiContainer Inject tags not detected

Hi,
For this one it could be that my understanding of the system that is not complete is to blame but here is the situation :

Details on What I'm doing

I made an installer which takes a scriptable object as parameter. This scriptable object maintain a list of extensions.
An extension is simply a class that can holds some data and a function that hooks in some other class logic to add some functionality.
Since extensions can depend on other ones, I also have an Install method that binds the relevant data to the container.
So during the install phase, I have the Installer scan through this list and call the install method of each extension.
Naturally, for this to work, I need my extensions to be injected with the dependencies they need to hook themselves in the system.

Situation

So in my scene, I have a mono installer that holds a reference to a scriptable object containing my list of the extensions. Since all of the extensions are created and added to the list in the editor and not at runtime, Zenject should be able to pick them up for injection right ?
(Note that : Because I create them in the inspector, I kept the constructor of my extensions empty, and inject them using a [Inject] Construct() method instead, just like a monobehaviour. Also not that my scriptable object is not a ScriptableObjectInstaller.)

Problem

So when I validate the scene, zenject stays completely oblivious to these extensions and doe'snt Inject them.

Work around : Bad practice ?

It is not a big problem since I can easily work around it by simply adding a Start method in the mono installer which calls Inject on the container for each extensions. I was wondering though if it wasn't a bad practice to tinker directly with the Inject() method ? Anyway, working this way, I cannot use the scene validation tool which is a bit sad, so if you have an advice, it is welcome 🗡
Oh and to give a little context as to why I'm defining my behaviour like this, it's just that I want my designer (which is also me with another hat) to be able to simply add and remove logic to elements and test gameplay quickly without tinkering with the code.

Question

Also, if it is intended, then why doesn't Zenject pick up my [Inject] tags inside my extension classes?
Sorry if it's a little long, I wanted it to be as easy to read as possible, hope it's not too confusing !
Thanks for the supprot !

It sounds like the best solution for you would be to call Container.QueueForInject and pass in every scriptable object instance inside the MonoInstaller.  See the docs here for QueueForInject.
The problem is that Zenject only injects on the instances that it creates itself, so for these cases where the instances already exist, you need to use QueueForInject.  This is necessary because in some cases you don't want to inject an instance even if it is bound with FromInstance

Injecting all the scriptable objects during Start using Container.Inject is ok too, but there's advantages to using QueueForInject.  For example, if A is injected into B and B is injected into C then they will be injected in the correct order (A, B, C) to guarantee that for any [Inject] method that all the parameters given to it have already been themselves injected.
Also, like you say validation should play nicely with QueueForInject, and it will be injected at the same time as everything else during Awake which is probably nice as well

Oh I see, thank you for your answer. I understand a little bit more now.
I was just running into that issue of inject order, so your answer is gonna be pretty useful !

Also this is awesome !
I was implementing some nasty "ILateInject" interface to inject dependencies into some already existing stuff I had to make sure that the dependencies were already injected themselves.
I feel so happy right now to be able to throw that garbage :P Sounds like it's time for me to read the advance section of the manual a bit more carefully !

All of the docs are getting an update in a couple days so if you're going to re-read it I'd wait for that  :)

Awesome :D Can'T wait !


k8s.io blocked be F-Secure Freedome VPN

This is a...

[ ] Feature Request
[X] Bug Report

Problem:
The domain k8s.io is blocked by F-Secure VPN Freedome which means that follow in guides form the documentation doesn't work, for example running:
kubectl create -f http://k8s.io/docs/tasks/configure-pod-container/task-pv-volume.yaml

Proposed Solution:
Contact F-Secure and have them dropped the block and ask them why it was blocked in the first place. I think that the request should come from the domain owner thus I haven't contact F-Secure my self.

Is kubernetes.io blocked or just k8s.io?

Not an issue anymore. Can now access both domains via Freedome.


[feature] https out of the box?

Hi,
Pygmy proxies https requests out of the box. It seems like pygmy-go doesn't, unless I missed something?
Thanks

I'll open up port 443 by default to haproxy.
Seeing as the port is open on the original pygmy - even with virtually no support, it's not a big ask.
I would be interested in how you use https with pygmy - if you were prepared to share.
In the meantime I'll organise this.

Well, I've been directed here by the amazee team when I asked about how to get pygmy to set an extra subjectAltName field in the self signed certificate you get when using https, so you can trust it on the system and Chrome doesn't complain about it still missing subjectAltName, and you have to re-enable the exception every few days (after each Chrome update?). So if the haproxy version pygmy-go doesn't do that yet, that would be a great addition too!

The containers are like for like - so it wouldn't support anything new. There's a lot of interesting things that the haproxy container could be doing and I get the feeling it will be updated in the future. I've been rather curious about this mechanism myself so I might go back down the rabbit hole.
I'm just getting the tests to pass again and I should be good to submit a PR. Could probably swing a release as well...

Thanks! re subjectAltName, it turns out it's not generated on the fly like I thought, but stored in the amazee haproxy image:
https://github.com/amazeeio/docker-haproxy/blob/master/server.pem
So, to get that extra field and make it Chrome friendly, I guess the image should be updated.

I guess the image should be updated.

There's so much awesome that can come from updates and changes to that image, but because this project is made to replicate pygmy like for like - despite the infinite amount of complexity you can add onto it, official support needs to come from upstream (amazee). I'm guessing there will need to be new logic for architectural differences as the Apple M1 architecture has complicated possible support. This tool is designed for universal support - and having such differences can complicate things.
All of that said, you can however provide your own custom image to use instead of amazeeio/haproxy, you could load up the changes and just make use of that image with minimal effort. I could write something up quickly if that sounds more appealing to you.

Release v0.7.0 is now out, I hope it helps you somewhat :+1:

Using a custom image would be the last resort, but I appreciate the offer to add the option to specify such an image.
I'll check with Amazee, I don't see the addition of this cert field as a deal breaker, more keeping it up to date with standards, which seems to be arround since 2003 ;-) https://www.digicert.com/faq/subject-alternative-name-compatibility.htm

Everything that pygmy-go runs is all in configuration, it passes Docker-native API objects directly to Docker's API.
So, changing the image is literally baked into the support here - it's just changing the configuration is probably not what most people want to do. It's one of the founding reasons I made this project - I wanted it to be customisable.
Happy you're using it!
If you are curious, I wrote a whole heap of documentation that never got finished that is all still relevant:
https://github.com/fubarhouse/pygmy-go/blob/issue/203/docs/customisation/examples_simple.md#image-replacement

Thanks for releasing v0.7.0!
I just had a go and https now works!
I'm getting some weird stdout when launching it tho:
# pygmy-go up    Successfully started amazeeio-dnsmasq Successfully started amazeeio-haproxy Successfully started amazeeio-mailhog Successfully started amazeeio-ssh-agent Successfully connected amazeeio-haproxy to amazeeio-network Successfully connected amazeeio-mailhog to amazeeio-network Successfully connected amazeeio-ssh-agent to amazeeio-network exit status 1 exit status 1 /bin/sh -c cat /etc/resolver/ exit status 1 Identity added: /Users/xxxx/.ssh/id_rsa (xxxx@xxxxx)
! http://docker.amazee.io/stats (amazeeio-haproxy)
`

Glad it works! :)
I'll have to have a think about this one... it appears too much of the ssh-add logs are coming through.
I'll open a new ticket and work on that tomorrow night.


Resource reload failed when zipped

I get the "Resource Reload Failed" notif when I try and load my zipped resourcepack. Unzipping fixes this problem.
I have a single resourcepack that contains some custom sounds, several CITs, and a reskinned villager.
I have tries Zipping on MacOS and on Linux.

Cant help much without more information.
Please send logs and the pack.

Log: ATLauncher Log
Zip:Archive.zip

Not really sure how unzipping it would fix the issue but the logs clearly say what's wrong:
[31/12/2021 11:16:08 AM] Caused by: net.minecraft.class_151: Non [a-z0-9/._-] character in path of location: minecraft:optifine/cit/items/machine gun.properties

The space in the file name is breaking the path.
It looks like machine gun.properties is the exact same file as machinegun.properties, deleting the space one should fix the pack.

More info available here:
https://citresewn.readthedocs.io/usage/help/
https://citresewn.readthedocs.io/usage/config/

Thanks! I'll make sure to not do that. (I'm really bad at reading logs lol)


Install ring.com floodlight on brick wall stuck as 1) drill hole not going well 2) hole made with 3/16 in drill bit too tight

As per title.

What worked ok
With Ryobi handheld drill, and 3/16 inch's Ryobi's masonry drill bit (3-piece) (homedepot.com), I was able to make 1 drill hole after several minutes of drilling.
Problems detailed
Issue-1. drill hole not going well
Unlike the one hole, other drill holes don't go well. It'd take hours I'm afraid if I had continued. Also I'm afraid the motor would burn down if I continue drilling that long, with extensive force I exert by hands.
Issue-2. Hole may be too small
The masonry anchor barely fits into the only drill hole I managed to make. Afraid if I'd continue hammer it down the anchor may collapse.
Anchor says 6x30. Does this mean??:

6 mm diameter
30 mm length

If above is correct then I was wrong in using 3/16 in. drill bit, as 6mm is about 0.24 inch, which 1/4 inch is closer than 3/16.

Conclusion: Try again with 1/4 inch drill bit.

Asked diy.stackexchange.com#q238293.

Done. As I've updated diy.stackexchange.com#q238293, I ended up using the drill bit of the right size that came with the outside camera.
Drilling 2 holes on the brick took long, more than 30 minutes I think, using the automated drill. This is something to avoid in the future if possible.
I unscrewed something remained from previous home owner. Indeed there's masonry anchor used.

Base screwed in, but this turned out wrong. I noticed I did backside front. So I re-did (took no pic of the right config).


Feature/final corrections

Final corrections finished

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

199 of 206 (96.6%)  changed or added relevant lines in 15 files are covered.
10 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-1.6%) to 98.407%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

repository/IngSoft2-Model/AllyTankManager.class.st
33
36
91.67%

repository/IngSoft2-Model/Game.class.st
67
71
94.37%

Files with Coverage Reduction
New Missed Lines
%

repository/IngSoft2-Model/AllyTankManager.class.st
2
92.42%

repository/IngSoft2-Model/RedoCard.class.st
4
77.78%

repository/IngSoft2-Model/RepeatCard.class.st
4
82.61%

Totals

Change from base Build<PHONE_NUMBER>:
-1.6%

Covered Lines:
1050

Relevant Lines:
1067

💛  - Coveralls


feat(binder): bind Parenthesized SELECT subquery

e.g.
dev=> create materialized view mv as (select * from t);
ERROR:  QueryError: Feature is not yet implemented: Parenthesized SELECT subquery: (SELECT * FROM t)
You can try to remove the parentheses if they are optional, Tracking issue: https://github.com/singularity-data/risingwave/issues/3584.

fixed by https://github.com/risingwavelabs/risingwave/pull/6363


Killua from Hunter x Hunter [Theme Request]

Who/What do you want to be featured?

Killua from Hunter x Hunter
What do they look like?

What type of theme?

[x] Light Theme
[x] Dark Theme

Additional Information (Optional)

Killua is also super cool

Possible Stickers

Are the themes still being created?
I need this sooo much


Throws error on All parameterless method in base repo

"Invalid input ')': expected an identifier (line 1, column 10 (offset: 9))
"MATCH (e:)"

Nevermind, it's just the null condition on the label. Might be worth some error handing on that?


[SPARK-19678][FOLLOW-UP][SQL] Add behavior change test when table statistics are incorrect

What changes were proposed in this pull request?
Since Spark 2.2.0 (SPARK-19678), the below SQL changed from broadcast join to sort merge join:
-- small external table with incorrect statistics
CREATE EXTERNAL TABLE t1(c1 int)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
  'serialization.format' = '1'
)
STORED AS
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'file:///tmp/t1'
TBLPROPERTIES (
'rawDataSize'='-1', 'numFiles'='0', 'totalSize'='0', 'COLUMN_STATS_ACCURATE'='false', 'numRows'='-1'
);

-- big table
CREATE TABLE t2 (c1 int)
LOCATION 'file:///tmp/t2'
TBLPROPERTIES (
'rawDataSize'='23437737', 'numFiles'='12222', 'totalSize'='333442230', 'COLUMN_STATS_ACCURATE'='false', 'numRows'='443442223'
);

explain SELECT t1.c1 FROM t1 INNER JOIN t2 ON t1.c1 = t2.c1;

This pr add a test case for this behavior change.
How was this patch tested?
unit tests

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8588/
Test PASSed.

Test build #103131 has started for PR 24003 at commit 39c8c84.

Test build #103131 has finished for PR 24003 at commit 39c8c84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103131/
Test PASSed.

cc @cloud-fan

thanks, merging to master!


chore(experiments): Replaces default container runtime to containerd

What this PR does / why we need it: Replaces default container runtime to containerd
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Checklist:

[ ] Fixes #
[ ] PR messages has document related information
[ ] Labelled this PR & related issue with breaking-changes tag
[ ] PR messages has breaking changes related information
[ ] Labelled this PR & related issue with requires-upgrade tag
[ ] PR messages has upgrade related information
[ ] Commit has unit tests
[ ] Commit has integration tests
[ ] E2E run Required for the changes

Also removed LIB from test.yaml files


rename and refactor begin_block and end_block

Both the Component trait and App have methods called begin_block and end_block. In addition to this, the corresponding, deprecated ABCI structs are needlessly passed through as arguments to these. These methods are no longer present in ABCI as of CometBFT v0.38, so we should rename them and refactor such that we aren't passing objects which have nothing done with them.
┆Issue Number: ENG-1006

This issue is stale because it has been open 45 days with no activity. Remove stale label or this issue
be closed in 7 days.


Error when compiling

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2022.05.18 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Description
Linux Mint 20.3
Python 3.8.10
When running make on the current master (92922fe), I get this:
(see verbose)
The file AUTHORS does not exist...
Verbose log
make
/usr/bin/env python3 devscripts/make_contributing.py README.md CONTRIBUTING.md
/usr/bin/env python3 devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE/1_broken_site.yml
/usr/bin/env python3 devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE/2_site_support_request.yml
/usr/bin/env python3 devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE/3_site_feature_request.yml
/usr/bin/env python3 devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE/4_bug_report.yml
/usr/bin/env python3 devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml .github/ISSUE_TEMPLATE/5_feature_request.yml
/usr/bin/env python3 devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/6_question.yml .github/ISSUE_TEMPLATE/6_question.yml
/usr/bin/env python3 devscripts/make_supportedsites.py supportedsites.md
AUTHORS Changelog.md LICENSE README.md README.txt supportedsites \
completions yt-dlp.1 requirements.txt devscripts/* test/*
/bin/sh: 1: AUTHORS: not found
make: *** [Makefile:13: pypi-files] Fout 127

https://github.com/yt-dlp/yt-dlp/commit/b5899f4f19116bb4d98907413fa3fb84a952ef13


Reduce command run times for SQL CLI

This PR moves the import lines to the functions that use them to delay imports. It also changes the "logging" code that sets loggers to critical. Previously since "airflow" was loaded from imports, it sets the level for it as Critical, now we explicitly set the level just for Airflow
flow version:
Before: 2.156s
Now: 0.175s

flow init:
Before: 2.108s
Now: 0.409s

flow validate:
Before: 2.342s
Now: 1.121s

flow run:
No improvement here as it needs most of the libraries
Before: 1.953s
Now: 2.036s

Feel to update, remove, adapt any changes from the PR.

Hmm 🤔 not sure If I like it or not. Is it really worth it moving the imports for 1-2 seconds speed improvement? I am wondering if this a best practice for cli apps?

@uranusjr WDYT of this change?

If we decide to go this way, we could use something like I created in airflint to automatically move the imports into functions via a pre-commit hook. This would make it as fast as we can 😬.
But It also depends on which commands and how often you call the commands, because python will cache the imports right?

Yup, if a module is already imported, it will be cached.

Hmm 🤔 not sure If I like it or not. Is it really worth it moving the imports for 1-2 seconds speed improvement? I am wondering if this a best practice for cli apps?

Yeah I was not sure as well but generally it is annoying if it takes a long time to load the version 😄  -- but it is more of "annoyance" rather than "problem". In Airflow, we load the CLI commands lazily to avoid it:
https://github.com/apache/airflow/blob/7be777254192f5a7cdcdd9066917217980916f0e/airflow/cli/cli_parser.py#L46-L56
https://github.com/apache/airflow/blob/7be777254192f5a7cdcdd9066917217980916f0e/airflow/cli/cli_parser.py#L1002-L1008

If we decide to go this way, we could use something like I created in airflint to automatically move the imports into functions via a pre-commit hook. This would make it as fast as we can 😬.

Oohhh nice, personally I would be in favor of it

@tatiana WDYT? I think we can give this a try in the next release after 0.1? Together with a pre-commit hook which would move the imports automatically.

@feluelle is there such a pre-commit hook? I almost wonder if we can do this in a cleaner manner by moving the functions out of the main class? Either way I think this is fine for now and we can decide later if we want a more permanent solution.

@dimberman please see https://github.com/astronomer/astro-sdk/pull/1122#issuecomment-1286696737


Open API generated code for making remote calls

Currently we won't fix this as we have a solution for using apistar instead of the forked apistar


Tmux/Tmate/Vim navigation not working

Hello!
Thank you for this great package!
I love this plugin, but have recently noticed an issue when using tmate that my keybindings don't seem to work.
When I run :TmuxPaneCurrentCommand, I get this output:
protocol version mismatch (client 8, server 7)
This is pane-ful (get it?) when using tmate to pair with others remotely, as I've grown accustomed to navigating in vim normally while in tmux. Do you have any suggestions on how I can work around this? I am using Mac OSX Mavericks, homebrew, and iTerm.

I'm running Yosemite and tried to downgrade to tmux 1.8. When you start tmux it says something to the effect of: "warning this version is unsupported, please upgrade to 1.9a". I also use tmuxinator which uses some of the new -c stuff (I guess?) and so it didn't work either. Hopefully we can get tmate upgraded to 1.9a soon, but unfortunately it looks like the author hasn't had time for the last year to update it https://github.com/nviennot/tmate/issues/33


Submitting forms via GET does not send values

As reported on slack, forms submitted via GET do not serialize their values into the query string, effectively not sending the values at all.
Let's investigate if enforcing POST is feasible, as it is much easier and less prone to future bugs than manipulating the query string.

In my particular case, it was "necessary" to keep the QS parameters, so the option with enforcing POST would not solve my issue. It was some kind of search-url builder form.


Autocomplete: Fix suggestion event over counting

This PR fixes an issue that sometimes caused completion suggestion events to not fire correctly. The issue here was that we decided wether or not a completion is visible in the getInlineCompletions code that was now change to still yield completions.
The yielding is fine, we have relied on these results to be used for the last candidate cache. However, this would also mean that we would store a log id that was deemed as not visible but that never reconsidered this condition. This meant that such a completion could become visible later and we would be able to accept it, even though it was never logged as suggested.
To fix this, we move this logic into the VS Code specific part (which conceptually makes more sense anyways since it's full of VS Code specific logic) and also test the visibility after every cache retrieval.
Test plan
I've added three conditions to test invariants that should not be possible. To test locally, I've added debugger break points into either of them and repeatedly triggered a bazillion of completions. After these changes, I don't seem to be able to trigger them anymore. I’m leaving some logs in there, though, so we can see if they happen on production.

Thank you for the fix and sorry for the regression!


Canonical form of 'ETag' header is incorrect

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
1.7.4
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOOS="linux"
What did you do?
This was found by code inspection.
The 'ETag' header's canonical form has a capital 'T' (https://tools.ietf.org/html/rfc7232#section-2.3).
But the Go apis use lowercase 't' in 'Etag'  instead.
This is in

net/http/fs.go:300 - literal string
net/textproto/reader.go:562 - CanonicalMIMEHeaderKey doesn't handle this case.

Consequence
No adverse effects of this have been observed. It's impact will be limited to rare cross-compatibility problems with other systems that don't correctly handle canonicalisation and/or case-insensitivity of header keys.

We don't intend to change this. It would be too many special cases everywhere for something that's already working correctly and would only exist to encourage other's broken implementations.
And in http2, you can't send capital letters on the wire anyway, so this is even more irrelevant with modern HTTP.


nixos/rosetta: Support AOT caching

Description of changes
When AOT Caching is enabled on the MacOS host, Rosetta will mount a caching daemon rosettad which can be run to do AOT translation.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

For some reason rosettad ignores whatever config I give it and always listens on $CACHE_DIRECTORY/rosetad/uds/rosetta.sock whilst roesetta tries to connect to the configured /run/rosettad/rosetta.sock  .
I have been stracing the binary and checking around in gdb and i'm pretty convinced this is a bug on Apple's side.
As a workaround I had to:

Change the permissions of the socket so all users can read itsudo chmod ga+w /var/cache/rosettad/uds/rosetta.sock
Create a symlink sudo ln -s /var/cache/rosettad/uds/rosetta.sock /run/rosettad/rosetta.sock

I opened a Feedback number with Apple on the issue FB14580944

@arianvp I can confirm your findings. I am using the following module in my config with Parallels as the hypervisor:
https://gist.github.com/stv0g/f8d472d6d4df453057fe597cc78abf90
Note the different socket path which has the Parallels specific prefix (prl).

FWIW I also forwarded this to our contact at Apple but I haven't heard back from them yet. I'll follow up.


Could molotov exit with a non-zero value in case of a failure

Hey,
I'm still trying to reuse my molotov scripts to do an integration test -- as a mean to both test my sever and my molotov script.
But molotov always exits with a zero success value, even in case of a failure.
Is it easy to change?
Thanks!

Hi, did you take a look into the --fail CLI flag? I think that's pretty much it https://molotov.readthedocs.io/en/stable/cli/

Ah that's interesting, I guess I missed it indeed...
Thanks for the pointer!

I guess we should also exits with 1 when --fail is not used and at least one test fails


Reusability of common functionality for other cosmos crates

Hej frens, I'm currently bootstrapping a rust crate for Osmosis very much in the same fashion of this crate, done so for Terra over the last months but that might be less relevant now.  I'm finding myself recreating the infrastructure for proto generation, prost extensions and mapping of domain types to proto types.
Wanted to pose an open-ended question as to what could potentially extracted/shared from this repo to be used across app chain crates?

Msg trait
proto-build
proto_ext

Your comment over here https://github.com/informalsystems/tendermint-rs/issues/654#issuecomment-1127905073 is in line what I'm looking for.


Graph has an error.

Hi Team,
When ever I tried to onboard a project I get the onboard success
[2023-12-11 19:10:01,589: INFO/MainProcess] Task solidgpt.src.api.celery_tasks.celery_task_onboard_repo_graph[0e3c67f4-1e02-4fc0-b62a-c3d264ff44df] succeeded in<PHONE_NUMBER>9997951s: '---****-
However when I start chatting I this error
raise ValueError(f"Collection {collection_name} not found")
ValueError: Collection d2d1a8a9-1067-429f-9525-0e6090045e8e not found
Complete log
[2023-12-11 19:10:48,574: WARNING/MainProcess] Directory 'D:\SolidGPT-0.2.7\solidgpt../localstorage\workspace\out\d32ab2ce-1e97-4447-854c-3aa669a6d8f0\Query_code_0' created.
[2023-12-11 19:10:48,575: WARNING/MainProcess] Directory 'D:\SolidGPT-0.2.7\solidgpt../localstorage\workspace\out\d32ab2ce-1e97-4447-854c-3aa669a6d8f0\Repo_Chat_1' created.
[2023-12-11 19:10:48,575: INFO/MainProcess] Executing SolidGPT...
[2023-12-11 19:10:48,575: WARNING/MainProcess] Node begins Query code task...
[2023-12-11 19:10:48,577: INFO/MainProcess] Start to search best five code...
[2023-12-11 19:10:49,580: ERROR/MainProcess] Task solidgpt.src.api.celery_tasks.celery_task_repo_chat_graph[56417a7d-fb8c-499f-acd1-e79e375d97c6] raised unexpected: ValueError('Collection d2d1a8a9-1067-429f-9525-0e6090045e8e not found')
Traceback (most recent call last):
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\celery\app\trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\celery\app\trace.py", line 760, in protected_call
return self.run(*args, **kwargs)
File "D:\SolidGPT-0.2.7\solidgpt\src\api\celery_tasks.py", line 117, in celery_task_repo_chat_graph
g.execute()
File "D:\SolidGPT-0.2.7\solidgpt\src\workgraph\workgraph.py", line 115, in execute
self.__execute_node(node)
File "D:\SolidGPT-0.2.7\solidgpt\src\workgraph\workgraph.py", line 122, in __execute_node
node.skill.execute()
File "D:\SolidGPT-0.2.7\solidgpt\src\workskill\workskill.py", line 52, in execute
self.execution_impl()
File "D:\SolidGPT-0.2.7\solidgpt\src\workskill\skills\query_code_local.py", line 51, in execution_impl
top_five = self.__find_top_five()
File "D:\SolidGPT-0.2.7\solidgpt\src\workskill\skills\query_code_local.py", line 61, in __find_top_five
search = self.client.search(
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\qdrant_client\qdrant_client.py", line 315, in search
return self._client.search(
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\qdrant_client\local\qdrant_local.py", line 154, in search
collection = self._get_collection(collection_name)
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\qdrant_client\local\qdrant_local.py", line 113, in _get_collection
raise ValueError(f"Collection {collection_name} not found")
ValueError: Collection d2d1a8a9-1067-429f-9525-0e6090045e8e not found
When I checked the UI I can see the message  - "Graph has an error."
Let me know what may be the issue. I am trying to onboard JAVA code base.

Can you try re-onboarding. Seems like onboarding cache file missing

Hi,
Even after onboarding again I am still getting the same error.


Atom fails to open project root from cmd

Prerequisites

[ ] Can you reproduce the problem in safe mode?
[x ] Are you running the latest version of Atom?
[x ] Did you check the debugging guide?
[x ] Did you check the FAQs on Discuss?
[x ] Are you reporting to the correct repository?
[x ] Did you perform a cursory search to see if your bug or enhancement is already reported?

For more information on how to write a good bug report or enhancement request, see the CONTRIBUTING guide.
Description
If there is an existing instance of Atom running, I am unable to open any other files/directories from the command line. If there isn't an existing instance of Atom running, I'm able to open files/directories as normal with atom -a dir/ or atom filename.
Steps to Reproduce

Open Atom
Attempt to open a file/directory using atom -a dir/ or atom filename

Expected behavior:
Expect that when Atom is open, I can open a new file or directory using atom filename or atom -a dir/.
Actual behavior:
When I try to open up a file or directory, Atom doesn't do anything at all. It looks like Atom is opening on my system, but then the app quickly closes. Note, this is independent of the existing running instance of Atom.
Here is the stack trace from /var/log/system.log
Jul 11 13:46:52 SIA-MC-W01PG3QC loginwindow[91]: -[SFLListManager(ServiceReplyProtocol) notifyChanges:toListWithIdentifier:] Notified of item changes to list with identifier com.apple.LSSharedFileList.RecentApplications
Jul 11 13:46:52 SIA-MC-W01PG3QC iTerm2[42729]: Time to encode state for window <PseudoTerminal: 0x7fe60062d500 tabs=1 window=<PTYWindow: 0x7fe60072a0c0 frame=NSRect: {{-1080, 60}, {1074, 1881}} title=1. tmux alpha=1.000000 isMain=0 isKey=0 isVisible=1 delegate=0x7fe60062d500>>: 0.00199204683303833
Jul 11 13:46:52 SIA-MC-W01PG3QC Atom[6463]: App load time: 320ms
Jul 11 13:46:52 SIA-MC-W01PG3QC Atom[90817]: Unexpected end of input
Jul 11 13:46:52 SIA-MC-W01PG3QC Atom[90817]: SyntaxError: Unexpected end of input
            at Object.parse (native)
            at Socket.<anonymous> (/Users/nland/Applications/Atom.app/Contents/Resources/app.asar/src/browser/atom-application.js:244:28)
            at emitOne (events.js:77:13)
            at Socket.emit (events.js:169:7)
            at readableAddChunk (_stream_readable.js:146:16)
            at Socket.Readable.push (_stream_readable.js:110:10)
            at Pipe.onread (net.js:523:20)
Jul 11 13:46:52 SIA-MC-W01PG3QC Atom[90817]: Unexpected token r
Jul 11 13:46:52 SIA-MC-W01PG3QC Atom[90817]: SyntaxError: Unexpected token r
            at Object.parse (native)
            at Socket.<anonymous> (/Users/nland/Applications/Atom.app/Contents/Resources/app.asar/src/browser/atom-application.js:244:28)
            at emitOne (events.js:77:13)
            at Socket.emit (events.js:169:7)
            at readableAddChunk (_stream_readable.js:146:16)
            at Socket.Readable.push (_stream_readable.js:110:10)
            at Pipe.onread (net.js:523:20)

Versions
apm version 1.9.2
atom version 1.8.0
Running on osx 10.11.5
You can get this information from executing atom --version and apm --version at the command line. Also, please include the OS and what version of the OS you're running.

As I stated on the message board, I couldn't reproduce this despite running the same general configuration. I'm going to leave this open for a while to see if we can get more information.

Looks similar enough to #11712, so closing in favor of that issue.


⚠ Fortnite has degraded performance

In 5ed3212, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1578 ms

Resolved: Fortnite performance has improved in 26757a0.


docs: Update changelog and README for new release.

@microsoft-github-policy-service agree company="Microsoft"