For discussion: use dedicated shape objects

This PR offers an implementation that replaces the current std::array and std::vector shapes with a dedicated shape class tiny_array. The PR is mainly intended for discussion, although it might eventually be merged if people happen to like the proposed shape class 😃 . I successfully executed the full xtensor test suite using Visual Studio, but did not yet test much on other compilers. Changes come in three groups:
Preliminaries
The PR adds headers xconcepts.hpp (concept checking and type inference), xmathutil.hpp (additional mathematical functions) and xtags.hpp (simple keyword argument support). Concept checking can now be done via
template <class T,
          XTENSOR_REQUIRE<std::is_arithmetic<T>::value>>
T foo(T t);

which is arguably more readable than enable_if_t applied to the return type.
Shape Class
Files xtiny.hpp and test_xtiny.cpp implement and test the classes tiny_array (arrays that own their memory) and tiny_array_view (arrays that don't own their memory) along with a large set of arithmetic, logical, and algebraic functions. tiny_array<size_t, 3> corresponds to std::array<size_t, 3>, and tiny_array<size_t, runtime_size> corresponds to std::vector<size_t>. The shape classes are designed to resemble the behavior of built-in arithmetic types. In particular, operations usually return new objects, rather than modifying existing ones in-place. This is key to the unification of static and dynamic shapes. IMHO, this design decision won't cause performance problems, because shape calculations are rarely done in the inner loop.
It is open for debate if it would be prefarable to implement mathematical functions for tiny_array in terms of the existing xexpression machinery. A possible advantage of the present design is that the compiler will unroll loops automatically for static shapes. (As an aside, I had to add XTENSOR_REQUIRE<xexpression_concept<E>::value> to some functions in xmath.hpp because these templates matched too greedily and thus dominated the corresponding functions in xtiny.hpp.)
Changes to the Existing Code Base
In the original code, declarations of shape types are spread all over the place (I found >30 occurences in the headers and another ~60 in the tests). I first implemented a single point of reference for these declarations by defining aliases stat_shape and dyn_shape in xutils.hpp and replacing hard-coded declarations of std::array and std::vector as appropriate. I then added overloads of the required utility functions and metafunctions for tiny_array and finally replaced the types in stat_shape and dyn_shape with tiny_array. Enjoy!
A major advantage of the present design is that xarray and xtensor essentially become the same class -- all differences can potentially be handled by tiny_array.

I think it's a really great addition to xtensor! Thanks for contributing.
However, I wonder wether it would be better to remove some features for now, and add them at a later stage?
If we concentrate on the TinyArray being a shape type, do we really need all the math functions defined for it? E.g. the trigonometric stuff etc.
Also, it might be confusing to have another kind of view in xtensor which doesn't adhere to the same interface as the existing views.
In general I guess it's easier to add features than to remove them ...
For example, I am not sure if we really need all_greater and related functions ... with other xexpressions you'd generally do xt::all(a > b) to get the same effect. Johan and I were quickly discussing the idea to make the TinyArray into an xexpression itself by inheriting from xcontainer ... I think conceptually it would be cool, and we'd save all the declarations for math etc. but maybe it also adds some overhead.
What do you think?
I hope this doesn't come across as too negative. I think this is really important work and personally I really want to see it in xtensor.

Is it possible to make the shape an expression?
The shape itself is somehow needed within the expression system right?
I guess this  would give an infinite template recursion.
How would you bootstrap that?
Seems like a nice meta-programming challenge =).

Luckily we can define the shape type of the shape type to be something else ;)
So xshape-container shape could be a std::array<xt::index_t, 1> container! (Which is easy to assume as shapes will always be 1D). The question is if this carries any overhead (which I can't answer).

Each xexpression has a shape type. In order to make the shape an xexpression, we only need to give it a particular shape type that would not be an xexpression (that could be an std::array with a single element for instance).

Ok, but if the TinyArray is an expression, and the shape of that expression is again something very special, then whats the point of having the TinyArray shape.
So I guess shapes would need a different expression mechanism ^^

You need to perform mathematical operations on the shapes, not on the size of the shape (which would become the shape of the shape). Making the shape_type an xexpression is just a way to avoid code duplication and reuse the implementation of the math functions. The special type used as shape_type of the shape has no importance, it's only meant to stop the template recursion.

If we concentrate on the TinyArray being a shape type, do we really need all the math functions defined for it? E.g. the trigonometric stuff etc.

I was of the same opinion initially (a decade or so ago 😄), but functions make all the difference. The most important lesson I learned over the years is "Users love convenience functions." Just consider
dot(point, strides)  // memory address offset
prod(shape)          // number of array elements
norm(shape)          // length of the diagonal

Since I always forgot which functions were available and which weren't, I adopted the easy-to-remember rule "support everything from <cmath> plus some numpy-inspired array-specific stuff".

Also, it might be confusing to have another kind of view in xtensor which doesn't adhere to the same interface as the existing views.

True, but tiny_array_view is needed for the subarray function. For example
auto backstrides = stride_type::unit_vector(0);
backstrides.subarray<1, 3>() = cumprod(shape.subarray<0,2>());

(ok, this doesn't special-case singleton dimensions ATM).

For example, I am not sure if we really need all_greater and related functions ... with other xexpressions you'd generally do xt::all(a > b) to get the same effect.

all_greater and xt::all(a > b) are very different here: in order to conform to the std::vector-API, I implemented a > b as a lexicographic comparison returning bool. I also considered implementing it element-wise, but came to the conclusion that lexicographic was more useful (e.g., one can put tiny_arrays into a std::map) and more consistent (would any user expect a == b to be element-wise for a shape class?). OTOH, if you have good arguments, the design is easy to change.

Johan and I were quickly discussing the idea to make the TinyArray into an xexpression itself by inheriting from xcontainer.

I don't know the details of xexpression sufficiently to judge if this would be a good idea. It might avoid some code duplication, but would make the design harder to understand. In the particular case of a shape class, code duplication is not a very big concern because the arithmetic and algebraic functions are unlikely to ever change once implemented. I also had some trouble with circular dependencies when I tried to incorporate the contents of xmathutil.hpp into xmath.hpp (xtiny.hpp than included xmath.hpp which includes xutils.hpp which in turn was supposed to include xtiny.hpp). These problems are certainly solvable, but I'm not sure if the advantages are worthwhile.

I am definitly not arguing against having all the syntactic sugar, I think it's awesome. But I think it will be much easier to merge a bunch of small commits. And I think we need to be careful to integrate the ideas from xtiny into the overall framework. It might be tough to add a lot of functions now, only to decide to e.g. rename some of them later.
Couldn't we do a PR with only the tinyarray and the arithmetic functions first, and then do separate PR's for views, and more functionality? At least that's what I think would be easier to review and figure everything out.
But maybe it's also worth it to hold back a little, as Sylvain and Johan might have some other feedback.

I'm running into some weird compiler bug it seems:
[ 28%] Building CXX object test/CMakeFiles/test_xtensor.dir/test_xio.cpp.o
In file included from /shape/xtensor/include/xtensor/xview.hpp:24:0,
                 from /shape/xtensor/include/xtensor/xio.hpp:22,
                 from /shape/xtensor/test/test_xio.cpp:18:
/shape/xtensor/include/xtensor/xview_utils.hpp: In instantiation of ‘struct xt::detail::view_temporary_type_impl<double, xt::tiny_array<long unsigned int, -1>, (xt::layout_type)1, int, xt::xall<long unsigned int> >’:
/shape/xtensor/include/xtensor/xview_utils.hpp:98:76:   required from ‘struct xt::view_temporary_type<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >, int, xt::xall<long unsigned int> >’
/shape/xtensor/include/xtensor/xview_utils.hpp:102:79:   required by substitution of ‘template<class E, class ... SL> using view_temporary_type_t = typename xt::view_temporary_type::type [with E = std::decay<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&>::type; SL = {int, xt::xall<long unsigned int>}]’
/shape/xtensor/include/xtensor/xview.hpp:37:77:   required from ‘struct xt::xcontainer_inner_types<xt::xview<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&, int, xt::xall<long unsigned int> > >’
/shape/xtensor/include/xtensor/xsemantic.hpp:39:82:   required from ‘class xt::xsemantic_base<xt::xview<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&, int, xt::xall<long unsigned int> > >’
/shape/xtensor/include/xtensor/xsemantic.hpp:201:11:   required from ‘class xt::xview_semantic<xt::xview<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&, int, xt::xall<long unsigned int> > >’
/shape/xtensor/include/xtensor/xview.hpp:74:11:   required from ‘class xt::xview<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&, int, xt::xall<long unsigned int> >’
/shape/xtensor/include/xtensor/xview.hpp:744:13:   required from ‘auto xt::detail::make_view_impl(E&&, std::index_sequence<I ...>, S&& ...) [with E = xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&; long unsigned int ...I = {0ul, 1ul}; S = {int, xt::xall_tag}; std::index_sequence<I ...> = std::integer_sequence<long unsigned int, 0ul, 1ul>]’
/shape/xtensor/include/xtensor/xview.hpp:759:38:   required from ‘auto xt::view(E&&, S&& ...) [with E = xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::tiny_array<long unsigned int, -1> >&; S = {int, xt::xall_tag}]’
/shape/xtensor/test/test_xio.cpp:59:40:   required from here
/shape/xtensor/include/xtensor/xview_utils.hpp:88:38: internal compiler error: in tsubst, at cp/pt.c:12186
             using type = xarray<T, L>;
                                      ^

When I change the T to some hard coded value (e.g. double) it seems to work.

With clang 3.8 it builds if I remove the constexpr on line 1723 in xtiny.hpp as the return is not a LiteralType.
I also commented out the traits tests in test_xtiny as they failed to link.

GCC 7 seems to work with the exception of xadapt. I think I found the issue: The templates are using std::size_t N while the tiny_array uses an int and therefore no overload is found. I can try to debug this tomorrow.

I wouldn't mind to get rid of tiny_array_view entirely (especially as this would also eliminate the need for tiny_array_base). However, I would not like to loose the subarray() function on the way. If someone comes up with a better way of distinguishing arrays that own their memory from those that don't, I'd be all ears. One possibility is to define
template <class T, bool owns_memory, int ... N>
class tiny_array;

but this doesn't look terribly elegant either.

I can split the PR into several parts, but not before end of August (I'm heading off to vacation tomorrow 😀 ). And of course, once the tests succeed on all platforms, there are a ton of warnings waiting to be fixed. Moreover, if the PR gets accepted one might want to change some of the existing shape manipulation code to take advantage of the new syntactic sugar.

Another important point that needs to be discussed: right now, the shape's value_type is size_t. I'd strongly recommend to change this into a signed type such as std::ptrdiff_t, so that something like abs(shape1 - shape2) would work without unpleasent surprizes.

How about the following idea to eliminate tiny_array_view and tiny_array_base?
template <class T, bool owns_memory, int M=runtime_size, int ... N>
class tiny_array_impl;

template <class T, int M=runtime_size, int ... N>
using tiny_array = tiny_array_impl<T, true, M, N...>;

template <class T, int M=runtime_size, int ... N>
using tiny_array_adaptor = tiny_array_impl<T, false, M, N...>;

template <class T, int M, int N>
using tiny_matrix = tiny_array_impl<T, true, M, N>;

@ukoethe after reading your proposal, I have a better understanding of where you want to go.
I wanted to say that this is really sophisticated work. And it also shows that you guys really have a real mastery of the type of metaprogramming that is in play here.
I would like to propose a plan to move forward with these ideas. However, rather than integrating a massive change like this as a first PR, I would like to have a more progressive approach.
1) Static and dynamic shapes
One feature of both xtensor and in your work is that we both support dynamic and static dimensionality. There is a huge gain in performances from using statically-sized shapes and strides. The dynamic case is required for wrapping the containers from interpreted languages such as python (numpy) and julia, in xtensor-python and xtensor-julia.
A first way to reduce the gap between stack-allocated shapes and heap-allocated shapes would be to use a dynamically-sized container type that

would have a stack-allocated buffer of constant size, used until a certain dimension  N (e.g. N=4)
a heap allocated buffer, that would only be used for higher dimensions.

Such a container would have the same performances as std::array for D<=N, (the most common case) and the same performances as std::vector for higher dimensions.
A good place to start the implementation of this new container would be to start from our uvector container. uvector differs from std::vector in that it does not initialize its values, and is used as a storage type in our container expressions (xarray, xtensor). Then, everywhere we use std::vector as a shape type by default, such as in xarray, we would use that new xvector. This change only would reduce the gap between the performances of xtensor and xarray.
2) Update to the promote_shape mechanism
Currently in xtensor, if we do auto e = a + b / d * sin(d) where all the members of the expressions are xexpressions, e is an un-evaluated expression. The shape type for e is

std::array if all the members of the expression have shape type std::array
std::vector if any of the members of the expression have another shape type.

Instead of having std::array as the only shape type considered "static", we would

add a constexpr static boolean in our custom shape types specifying if it is static or dynamic.
use a traits mechanism to handle the containers from the STL.

Then, downstream could use their custom shape types that would be recognized as static.
3) arithmetic and extra operations on shapes

xtensor and xarray are just template aliases for specific shape types. We document how a use can create a custom expression (that would work with xtensor and xarray) with their own data and shapes containers...  I think that it is a very powerful thing, and the only requirement here is that these types implement the standard interface for an STL container. Requiring more would limit the ability to define your own expressions...

However, I get the need for arithmetic operations on shapes. A means to do this without duplicating all the math operations from xtensor would be for shape() to return an adaptor as an xexpression on the underlying shape type. That would remove a lot of code, since one would not need to re-implement all the arithmetic operations and features for the shape types, while they are already in xtensor.

4) Conclusion
Apologies for taking so much time to give you proper feedback. We should probably talk viva voce (over skype, hangout or appear.in) sometimes.
Let me know if you like this plan ! Items 1) and 2) and 3) can be done in independent PRs.

to item 1)
We implemented such thing within OpenGM
https://github.com/DerThorsten/opengm/blob/master/include/opengm/datastructures/fast_sequence.hxx

We should probably talk viva voce.

That would be good!

Rather than integrating a massive change like this as a first PR, I would like to have a more progressive approach.

I think it is beneficial for our discussion when you and others can look at the PR in its entirety. Actual integration - if agreed upon - can then proceed more gradually.

Static and dynamic shapes

The most important benefit of my design is that static and dynamic shapes have the same API and can be used interchangeably in generic code. In my experience, this leads to huge simplifications in functionality like promote_shape, dynamic ROI definition etc. Shape arithmetic in this PR is implemented such that static shapes are used whenever the resulting dimension can be determined at compile time, and dynamic shapes otherwise (this happens automatically, no additional template magic is needed).

heap allocated shapes

I'm not sure if allocation of the shape storage is really the bottleneck. According to my experience, the most critical operation is the transformation of index objects into memory addresses (i.e. the computation of the scalar product between index and strides), which regularly occurs in inner loops. The compiler can optimize this much much better if the dimension is known at compile time, but this optimization is independent of the allocation mechanism. If benchmarking experiments demonstrate that heap allocation is indeed too slow, I can implement the mixed allocation mode you proposed (stack for small N, heap for bigger N) or a custom allocator that calls new for batches of shape objects. Without such benchmarks, replacing heap allocation might be premature.

Then, downstream could use their custom shape types that would be recognized as static.

I'm proposing to replace std::vector and std::array in their role as shape types with tiny_array in the entire library, because tiny_array fully unifies static and dynamic shapes and provides a rich set of arithmetic and algebraic functions. I don't (yet) see the benefit of retaining std::vector and std::array and supporting additional custom shape types - this sounds unnecessarily complicated.

The dynamic case is (only) required for wrapping the containers from interpreted languages.

I found over the years that dynamic tensors are more generally useful. For example, functions which are separable over dimensions (e.g. Gaussian filters, distance transforms, data I/O) don't run faster if the dimensionality is known at compile time, whereas the benefits of dynamic dimensions are substantial: compilation times are greatly reduced and binaries get much smaller, because the compiler needs to instantiate these templates only once instead of repeatedly for every dimension.

A means to do this without duplicating all the math operations from xtensor...

In general, I agree that redundancy in code should be avoided. However, I'm not so sure about the present case:

Using xexpressions for shape computation makes the inner workings of tiny_array a lot harder to understand. This may be a problem because many programmers don't master advanced template techniques, which means that they have a hard time interpreting error messages for even trivial mistakes and are less likely to contribute back to the code base. 1200 extra lines (minus the required adaptor code) of easy arithmetic might be a preferable trade-off.
Redundancy is not that big a problem if the code is unlikely to ever change, as I expect to be true for shape arithmetic.
I'm not sure if the same arithmetic rules apply or should apply to tensors and shapes (e.g. shape1 < shape2 implements lexicographic comparison returning bool, to be consistent with std::vector).

We should probably talk viva voce.

That would be good!

I am coming back from JupyterCon on wednesday. Both I and Johan would be available pretty much anytime Thurday or Friday, or next week.

Rather than integrating a massive change like this as a first PR, I would like to have a more progressive approach.

I think it is beneficial for our discussion when you and others can look at the PR in its entirety. Actual integration - if agreed upon - can then proceed more gradually.

Awesome, I think that we are on the same page.

[...]

I'm not sure if allocation of the shape storage is really the bottleneck. According to my experience, the most critical operation is the transformation of index objects into memory addresses (i.e. the computation of the scalar product between index and strides), which regularly occurs in inner loops. The compiler can optimize this much much better if the dimension is known at compile time, but this optimization is independent of the allocation mechanism. If benchmarking experiments demonstrate that heap allocation is indeed too slow, I can implement the mixed allocation mode you proposed (stack for small N, heap for bigger N) or a custom allocator that calls new for batches of shape objects. Without such benchmarks, replacing heap allocation might be premature.

I agree with you about the cost of indexed-based indexing, but the iterators that we have in xtensor don't rely on such inner products. So in the case of iterator-based indexing, the heap allocation becomes dominant.

Then, downstream could use their custom shape types that would be recognized as static.

I'm proposing to replace std::vector and std::array in their role as shape types with tiny_array in the entire library, because tiny_array fully unifies static and dynamic shapes and provides a rich set of arithmetic and algebraic functions. I don't (yet) see the benefit of retaining std::vector and std::array and supporting additional custom shape types - this sounds unnecessarily complicated.

We have a very strong use-case for supporting user-provided shape types, for the adaptors from other data structures. (e.g. in xtensor-python, the inner shape type is a buffer adaptor). However, I hear your need for richer arithmetic on shapes, so I have a proposal which should address this usecase.

only require the inner shape type to have the API of a standard container.
have shape() return a in-place adaptor as an xexpression on those.

So that user-defined expressions adapting existing data structures don't need to recursively define expressions for the shapes etc...

The dynamic case is (only) required for wrapping the containers from interpreted languages.

I found over the years that dynamic tensors are more generally useful. For example, functions which are separable over dimensions (e.g. Gaussian filters, distance transforms, data I/O) don't run faster if the dimensionality is known at compile time, whereas the benefits of dynamic dimensions are substantial: compilation times are greatly reduced and binaries get much smaller, because the compiler needs to instantiate these templates only once instead of repeatedly for every dimension.

👍 I agree with that 100%

A means to do this without duplicating all the math operations from xtensor...

In general, I agree that redundancy in code should be avoided. However, I'm not so sure about the present case:
Using xexpressions for shape computation makes the inner workings of tiny_array a lot harder to understand. This may be a problem because many programmers don't master advanced template techniques, which means that they have a hard time interpreting error messages for even trivial mistakes and are less likely to contribute back to the code base. 1200 extra lines (minus the required adaptor code) of easy arithmetic might be a preferable trade-off.
Redundancy is not that big a problem if the code is unlikely to ever change, as I expect to be true for shape arithmetic.

I think that the adaptor approach proposed above bridges the gap by avoiding the duplication of the code, while proving a rich API for shape types and making shape type interoperable with expressions.

I'm not sure if the same arithmetic rules apply or should apply to tensors and shapes (e.g. shape1 < shape2 implements lexicographic comparison returning bool, to be consistent with std::vector).

Gotcha. I don't think that is a big issue.

Closing this one which, which was split into multiple discussions.


Error due to no consent

Hey Jason,
I have uploaded everything to my web server and managed to get it working up until getTokenFromAuthCode.
I am receiving a 400 error with the below message:
"error":"invalid_grant","error_description":"AADSTS65001: The user or administrator has not consented to use the application with ID...."
I have granted permission through the Azure AD panel, and I also granted permission on first login. I am not sure why it's still asking for consent?

I was also wondering, can I read a users calendar if their Office 365 is associated with another organisation? I am trying to make read staff calendars to use in an external website where people outside of the organisation can view calendars and request meetings. Whenever I try to login to the app I have created using a staff email, from another organisation, I get an error saying "The application could not be found under the organisation ...."
Edit; This works for apps made in Azure AD by switching Multi-Tenant on, but there is no option in Outlook Dev Portal.
Edit:
One other thing I noticed is that when I register an app using the Outlook Dev Portal and use the Password as the client secret, it returns and error stating the client secret is incorrect.
AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.
Is the password generated on the Outlook Dev Portal not the client secret?
Thanks in advance, sorry to bother you.
Closed:
Got the Javascript repo working

From your screenshot you granted Application Permissions, but the sample needs Delegated Permissions. The difference here is that Delegated Permissions are the sort where a user can allow the app to act on their behalf (which is what this sample does), and Application Permissions are where an organization administrator can allow the app to access information without acting as a user.
For this sample, you should be registering the app in Azure AD portal.
Yes, you can access users in other organizations if you turn on the Multi-tenant option on your app. They will have to go and log in like anyone else.

Thanks so much Jason, it works now.
I just want to make sure I am understanding this correctly. I can use the request token at any time to generate another token and load the users calendar? For example, I could store the tokens, then every day, or week, I can use the refresh token to get another token?


"Risky" is now confusing

I received some feedback from the field that the use of "Risky" here is now confusing with availability of CA policies based on Sign-in Risk (which requires P2 license). Later in the article we correctly state that CA only requires P1.

@KeithBrewer-MSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi @KeithBrewer-MSFT - This PR was opened in the public repository. PMs should work in the private repo, per the contributor guide. Would you open future PRs in the private repo? Thanks!

@MicrosoftGuyJFlo - Would you review the proposed update? Thanks.

Thanks. Resubmitting in -pr
Thanks,
[AAD_Logo2] Keith Brewer
From: John Flores<EMAIL_ADDRESS>Sent: Thursday, September 3, 2020 1:32 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Keith Brewer<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] "Risky" is now confusing (#61831)
@MicrosoftGuyJFlo requested changes on this pull request.
Risk is what it is called. Maybe adding a (Requires Identity Protection) note?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fpull%2F61831%23pullrequestreview-482057657&data=02|01|Keith.Brewer%40Microsoft.com|ecb893ec917045c0d5c208d8502f46ee|72f988bf86f141af91ab2d7cd011db47|1|0|637347511274610026&sdata=zhvZmV%2BeyGLd%2FKoYacjXFvQhcHQaOtERsxTCci%2BIs1g%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQX7JJNMR73DPRSCFYQJXPDSD7HJLANCNFSM4QRCIYVA&data=02|01|Keith.Brewer%40Microsoft.com|ecb893ec917045c0d5c208d8502f46ee|72f988bf86f141af91ab2d7cd011db47|1|0|637347511274619983&sdata=kwrQbIq8VRfF%2FKm1Lwmjx2ORWN%2BO%2F5hqzhJAtVtXsh0%3D&reserved=0.


Copy to clipboard button on Git auth flow

Closes https://github.com/weaveworks/weave-gitops-enterprise/issues/1713
Currently the user is asked to copy across a code for auth with GitHub. Triple clicking to select all results in an additional space (as shown in screenshot) and then doesn't paste into the field on GitHub.
If we could provide a copy to clipboard button next to the code, that would make things much simpler than having to highlight.

Closing it as we're moving it to WGE


ETE3_db_status_check trying to write to default database_directory with database_directory specified

Hi,
I was trying to run mob_recon on the lab server which I have limited access. My account doesn't have the permission to write to the default database_directory. I thought I could solve this problem by specifying the database_directory while running mob_recon. Here is the command that I used:
mob_recon -u --infile myseq.fasta --outdir myoutput/ --database_directory ~/.local/lib/python3.8/site-packages/mob_suite/databases --debug

However, I got the Error Messages as follows. It seems like it is still trying to create the Lock file under the defaut database_directory?
2021-03-11 00:49:27,904 root INFO: Creating Lock file /opt/anaconda3/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock [in /opt/anaconda3/lib/python3.8/site-packages/mob_suite/utils.py:438]
Traceback (most recent call last):
  File "/opt/anaconda3/bin/mob_recon", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/lib/python3.8/site-packages/mob_suite/mob_recon.py", line 1005, in main
    dbstatus = ETE3_db_status_check(1, ETE3_LOCK_FILE, ETE3DBTAXAFILE, logging)
  File "/opt/anaconda3/lib/python3.8/site-packages/mob_suite/utils.py", line 439, in ETE3_db_status_check
    open(file=lockfilepath, mode="w").close()
PermissionError: [Errno 13] Permission denied: '/opt/anaconda3/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock'

@shuzvvn Thank you for brining you user case. This is a know issue that we plan to address soon in next release. Currently mob-suite is not read-only file system friendly. I have the same issue on public Galaxy EU read-only file system. The database lock was put to avoid race condition upon eventual ete3 database update (that could sporadically occur). We will need to make lock file being written to your local copy of the database directory and also implement recovery on permission denied error during lock file write.  For now you can quickly fix this issue by putting a try and except directive command around the line 439. I will create issue so that we can address this use case scenario. We would ideally turn off ete3 library taxonomy updates to avoid the need of having this preventive lock file logic.

Worth noting ETE writing to ~/.etetoolkit during mob_init also causes similar issues when trying to deploy mob-suite in a containerised workflow.
Any idea if there a very rough horizon for an update to more read-only friendly behavior (totally acknowledging we are all neck-deep in other public health work right now)?

Indeed we are currently busy with multiple tasks linked both to COVID-19 work and food pathogen surveillance (Salmonella and E. coli). Yet we've started working on this issue today. The code will be posted at the v3.1.0 branch and I will provide updates here. Will be working slowly towards release v3.1.0 that will address this and other issues in parallel with other commitments. I agree that this custom database_directory is a very desirable feature to have especially in light of read-only Singularity containers and public Galaxy read-only file system configurations.  In addition, ete3 taxonomy database initialization will be performed as a post-install task avoid a racing condition. You will be able to pull intermediate code from this branch or build and install temporary Python package. Thank you for your patience.

Issue confirmed with the placement of the lock file after singularity run https://depot.galaxyproject.org/singularity/mob_suite:3.0.1--py_0 mob_recon -i test_file.fasta -o test
2021-04-26 13:39:32,868 root INFO: Creating Lock file /usr/local/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock [in /usr/local/lib/python3.8/site-packages/mob_suite/utils.py:438]
Traceback (most recent call last):
  File "/usr/local/bin/mob_recon", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/mob_suite/mob_recon.py", line 1095, in main
    dbstatus = ETE3_db_status_check(1, ETE3_LOCK_FILE, ETE3DBTAXAFILE, logging)
  File "/usr/local/lib/python3.8/site-packages/mob_suite/utils.py", line 439, in ETE3_db_status_check
    open(file=lockfilepath, mode="w").close()
OSError: [Errno 30] Read-only file system: '/usr/local/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock'

I am having this issue with the program installed as a module - my users don't have write permission for this lock file. Is there any word on when this will be fixed?

@cmfield and @fmaguire   Sorry for the delay with the read-only file systems compatibility implementation. We see this is an important issue to fix and I will start working on it starting this Friday and hope to have a fix by the end of the next week. This is also relevant for Singularity containers compatibility.

Had completed preliminary testing of the 3.0.2 version inside fresh ubuntu 21.04 container.
Please pull mob-3.0.2 branch and install from the source. Also updated README and added info in install from the source code (in case you are testing/building container).
git checkout mob-3.0.2
python setup.py install

Will continue testing tomorrow and then release it as a new version 3.0.2 which will automatically trigger Singularity and Conda resources.


Artists Specs

There should be some procedure for disambiguation.  Not only in search results but for cleaning up current artist pages with torrents from 2 or more artists with the same or similar names.  Also when artists have spelled their names differently or have had many aliases the system should be able to deal with those issues.

The term "artists" is very specific to music. Since this project is a framework, should this be more generic? Make taxonomies configurable?

@sneakysneakysneaky Yes it should be. "Content creator" was thrown around as an alternative. We should try to stay as generic as possible.

@ajax-wcd and @sneakysneakysneaky Perhaps the term contributor would be most generic. This means that for applications that are hosted the producing company would be considered the primary contributor whereas with music the contributor is specifically an artist. I want to start tackling this spec but I think it will be difficult until we actually have a solid decision about this because I think it affects the spec at a core level.

I think we need an even more generic meta-model in which you can create these relationships. Otherwise there's no way we're going to get this flexible enough.
Eg. books are similarly related to music, but movies or software not so much.
We need something like groups can contain torrents, or other groups?

I've created #137 for this.


Getting Started instructions for Bower: missing minified files

MDL version: 1.0.4
What steps will reproduce the problem:

Follow Getting Started: http://www.getmdl.io/started/#download
Click "Bower" option
Follow instructions

What is the expected result?
Minified final build files should be included in bower repo.
What happens instead of that?
bower_components/material-design-lite/material.min.css Failed to load resource: net::ERR_FILE_NOT_FOUND
bower_components/material-design-lite/material.min.js Failed to load resource: net::ERR_FILE_NOT_FOUND

Am I supposed to build the bower component with gulp first? If so, the instructions should mention that. However, I'd prefer if the bower repo had the minified final files included so I don't have to.

I believe the NPM instructions will need the same treatment.
Thanks for this project! This is EXACTLY what my team needs. So much easier than trying to shove Polymer into our already written Backbone codebase...

Is this a somewhat recent change? My bower install from a few days ago contains the minified files I need.
But when I saw this issue and try to re-install it, the files aren't there anymore.

+1

@surma Was this due to the mis-tag which has since been addressed? Ref: https://github.com/google/material-design-lite/pull/278#issuecomment-132951679

Yes, this is due to the mis-tag. Bower has been fixed. npm is being worked on :) Closing this.


Update AlphaWallet video on google play

@JamesSmartCell bump, I cannot do this myself as I don't have your access.


Translation added request

In order to improve the practicability of this plugin, l really hope a translation can be added. However ,since l am not a front engineer and l am not familiar with the js syntax and your code structure , it's hard for me to add the translation code. One of the logseq developers have made a lib for translation and here is the lib . Would you please integrate this to your plugin , and translate the English into Chinese? If you don't want to translate it, l can translate it too! Thank you very much!

Chinese translation will be added to the development plan, but it may take some time.

Sure, thanks!


Update s3, sdk-core, sqs to 2.26.26

About this PR
📦 Updates

software.amazon.awssdk:s3
software.amazon.awssdk:sdk-core
software.amazon.awssdk:sqs

from 2.26.4 to 2.26.26
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "software.amazon.awssdk" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #731.


How does an organization manage a digital service through its lifecycle?

Using Coco Pharmaceuticals persona to drive the scenario, provide Egeria capability with supporting resources to manage the governance aspects of each stage of evolution of a digital service.

This is work in progress

@mandy-chessell can you can assign this to me ?

Keep open

I think this should be kept open @wbittles ? Could be worth keeping a summary in here


First concept of ESM build

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 85.819%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
9535

Relevant Lines:
10704

💛  - Coveralls


android: pass correct constraints map to VideoCaptureController

Incorrect constraints map being passed to VideoCaptureController.

If you don't mind could you also take a look at https://github.com/jitsi/lib-jitsi-meet/issues/801? It's a related issue and I have a quick fix written in the issue description. Thanks!


Tech debt around duration config

A few things here:

Our config files have durations for various things, and they're largely defined as "integer number of seconds"
We have a bunch of ad-hoc duration constants with FIXMEs to make them configurable
We have a dedicated type for picking random durations in a range (util.TimeRange), but we're not using that outside of util.Watch.

It'd be good to resolve some of these, ideally

[ ] Using util.TimeRange in the configs; and
[ ] Determining which duration constants should be made configurable (and which should be left as-is)

See also: https://github.com/neondatabase/autoscaling/pull/223#discussion_r1185110442


Cannot edit an InteractionMessage

Summary
Cannot edit a message using the function .edit
Reproduction Steps
"Dropping existing records..\nApplying new records..." basically returns this:
Ignoring exception in command <nextcord.application_command.SlashApplicationSubcommand object at 0x0000022FF5DE32F0>:
Traceback (most recent call last):
File "C:\Users\ng_jo\AppData\Local\Programs\Python\Python312\Lib\site-packages\nextcord\application_command.py", line 918, in invoke_callback_with_hooks
await self(interaction, *args, **kwargs)
File "C:\Users\ng_jo\OneDrive\Backup\Desktop\joshuang\coding.GITHUBCLONE\Testing-Bot\cogs\tickets_init_.py", line 319, in register_config
await msg.edit(
File "C:\Users\ng_jo\AppData\Local\Programs\Python\Python312\Lib\site-packages\nextcord\interactions.py", line 1180, in edit
message = await self._state._interaction.edit_original_message(
^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ConnectionState' object has no attribute '_interaction'
The above exception was the direct cause of the following exception:
nextcord.errors.ApplicationInvokeError: Command raised an exception: AttributeError: 'ConnectionState' object has no attribute '_interaction'
Minimal Reproducible Code
@setup.subcommand(name="register")
    # @application_checks.is_owner()
    async def register_config(
        self,
        interaction,
        guild_id=str,
        category_id=str,
        staff_id=str,
        transcript_id=str,
    ):
        guild = self.bot.get_guild(int(guild_id))
        if self.bot.get_guild(int(guild_id)):
            if guild.get_channel(int(transcript_id)):
                if nextcord.utils.get(guild.categories, id=int(category_id)):
                    if not fetch(
                        "storage/tickets.sqlite",
                        "TicketConfig",
                        f"guild_id={guild_id}",
                    ):
                        insert(
                            "storage/tickets.sqlite",
                            "TicketConfig",
                            ("guild_id", "category_id", "staff_id", "transcript_id"),
                            (guild_id, category_id, staff_id, transcript_id),
                        )
                    else:
                        view = ConfirmView()
                        data = fetch("storage/tickets.sqlite", "TicketConfig")
                        for i in range(len(data)):
                            if data[i][0] == int(guild_id):
                                data = data[i]
                                break
                        msg = await interaction.response.send_message(
                            f"Guild already registered, overwrite?\n### CONFIGURATION FOR THIS GUILD\n- Guild ID: {interaction.guild.name} ({data[0]})\n- Category ID: <#{data[1]}> ({data[1]})\n- Staff ID: <@&{data[2]}> ({data[2]}\n- Transcript Channel: <#{data[3]}> ({data[3]})",
                            view=view,
                        )
                        await view.wait()
                        if view is None:
                            await msg.edit("Action cancelled.")
                        elif view.value:
                            msg = await msg.edit(
                                content="Overwriting the existing record...", view=None
                            )
                            drop(
                                "storage/tickets.sqlite",
                                "TicketConfig",
                                f"guild_id={guild_id}",
                            )
                            await msg.edit(
                                "Dropped existing records..\nApplying new records..."
                            )
                            insert(
                                "storage/tickets.sqlite",
                                "TicketConfig",
                                (
                                    "guild_id",
                                    "category_id",
                                    "staff_id",
                                    "transcript_id",
                                ),
                                (guild_id, category_id, staff_id, transcript_id),
                            )
                            # await msg.edit(
                            #    f"Registered guild.\n{fetch('storage/tickets.sqlite', 'TicketConfig', f'WHERE guild_id={guild_id}')}"
                            # )
                else:
                    await interaction.response.send_message(
                        "Invalid `category_id`. Perhaps the bot doesn't have the permission to view it?"
                    )
            else:
                await interaction.response.send_message(
                    "Invalid `transcript_id`. Maybe the bot doesn't have access to all the channels?"
                )
        else:
            await interaction.response.send_message(
                "Invalid `guild_id`, maybe the bot is not in the guild?"
            )

Expected Results
edit the message
Actual Results
threw me an error idk what it means
Intents
nextcord.Intents.all()
System Information
nextcord-ext-menus 1.5.7

Python v3.12.1-final
nextcord v2.6.0-final
aiohttp v3.9.1
system info: Windows 10 10.0.19045
i've got discord.py i don't think it would affect it tho...

Checklist

[X] I have searched the open issues for duplicates.
[X] I have shown the entire traceback, if possible.
[X] I have removed my token from display, if visible.

Additional Context
support thread here: https://ptb.discord.com/channels/881118111967883295/1212126972784807987

Can you make the minimal reproducible code actually minimal? It's hard to understand with multiple nested conditions.

@nextcord.slash_command(name="bug-test")
async def bugtest(self, interaction):
    msg = await interaction.response.send_message("abc1")
    msg = await msg.edit("abc2")
    msg = await msg.edit("abc3")```
At third edit:
![image](https://github.com/nextcord/nextcord/assets/77912444/c494398b-ec1b-4ffc-82bb-1151e374840d)
```Python
Ignoring exception in command <nextcord.application_command.SlashApplicationCommand object at 0x00000244D4B5D3D0>:
Traceback (most recent call last):
  File "C:\Users\ng_jo\AppData\Local\Programs\Python\Python312\Lib\site-packages\nextcord\application_command.py", line 918, in invoke_callback_with_hooks
    await self(interaction, *args, **kwargs)
  File "C:\Users\ng_jo\OneDrive\Backup\Desktop\joshuang\coding\.GITHUBCLONE\Testing-Bot\cogs\dev\__init__.py", line 123, in bugtest
    msg = await msg.edit("abc3")
          ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ng_jo\AppData\Local\Programs\Python\Python312\Lib\site-packages\nextcord\interactions.py", line 1180, in edit
    message = await self._state._interaction.edit_original_message(
                    ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ConnectionState' object has no attribute '_interaction'

The above exception was the direct cause of the following exception:

nextcord.errors.ApplicationInvokeError: Command raised an exception: AttributeError: 'ConnectionState' object has no attribute '_interaction'```

![image](https://github.com/nextcord/nextcord/assets/77912444/c9c51562-9fd7-4db2-a3ae-5848035b5eea)

Can reproduce.
As a temporary fix you can not collect the message until the end:
@nextcord.slash_command(name="bug-test")
async def bugtest(self, interaction):
    msg = await interaction.response.send_message("abc1")
    await msg.edit("abc2")
    msg = await msg.edit("abc3")
	print(msg.content)  # abc3


New Question presented after answering

GIVEN I am taking a code quiz
WHEN I answer a question
THEN I am presented with another question

This issue is now resolved from a previous push!


Added additional error info when testing, custom events, and "Do nothing" action

This PR adds 3 small changes:

Additional info (via the browser's console) when clicking "Test sensor"
Firing custom events when the sensor is triggered (useful for plugins like https://github.com/tjjfvi/OctoPrint-IFTTT that allow you to subscribe to custom events)
A 'Do Nothing' action option to perform when the sensor is triggered (useful for users who only want the navbar status & popup  notification or for users who will use the custom events mentioned above to create their own action/notification)

@valemaio2 I think I'm going to have to move my PR to your repo


Extension bundle table points to 3.4.0 (unreleased version of extension bundle)

The Extension Bundle table available here points to version 3.4.0 of extensions.json. However, the latest 3.X version of extension bundle released is 3.3.0 based on the release link here: Releases · Azure/azure-functions-extension-bundles (github.com)
This is causing confusion as the 3.X extensions.json file linked in this doc mentions that webpubsub bindings are available, but this version (3.4.0) is not available for use by Functions Core Tools as it is not released.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5fb82e02-5c41-239b-4903-aab80ff15215
Version Independent ID: 38f1c69b-f7af-5e13-1cdc-b6b7053d2f7b
Content: Register Azure Functions binding extensions
Content Source: articles/azure-functions/functions-bindings-register.md
Service: azure-functions
GitHub Login: @craigshoemaker
Microsoft Alias: *cshoe

@oneAnuragByte Thank you for your feedback! I'm not seeing any references to the 3.4.0 version on this doc and I see that you've closed this issue, let me know if the issue still exists and I'd be happy to help mitigate it.


Add end to end fraud scenarios

Some scenarios where contracts burn gas/memory with recursive operations

Testnet command sets a minimum block gas value

Note: scenarios use a custom hackatom contract that is not published, yet

No problem I have more from a Sanchez, cabrry,  users too


add code for user-service, jobs-service, frontend

add code for user-service jobs-service and frontend.

OK ... This way we can see what changed to make jobs-service and frontend work.


Main

Fix the macro derive "Enum" as it seems to be broken due to the access level of "not_null" limited to crate. Also replacing the use of Kind inside the macro from external libpq crate by the one re-exported by elephantry.

Thank you!
Can you just add the #[doc(hidden)] attribute on not_null?

Sure! I've already added it.

Merged :slightly_smiling_face:


Add query time aggregator to the View

Currently query time aggregation just uses the aggregation specified in the schema. It would be useful to be able to provide a different set of aggregation functions.
This issue should add a aggregator section to the View.
The HBase and Accumulo stores would need to be updated to use the new aggregator functions provided in the View.
As part of this ticket, the integration tests should be updated to include tests for query time aggregation

It is quite simple to make this change, it won't cause a breaking change, just simply an extra 'aggregator' field in the ViewElementDefinition class. So I'm happy for this to come after version 1.0.
We should wait until someone actually needs this feature before implementing it.

OK, I take my previous comment back. Seeing as thought it is quite simple to implement, it would be worth adding as it allows users to ask questions like this:
Assume you have some Edges in day buckets (achieved using a day timestamp groupBy property), so you can either query for Edges on a particular day or summarise over multiple days. If you have a property like 'count' on your Edges, then for a particular day the count is the sum of Edges that occurred in that day.
At query time if you were able to change the Sum aggregator to a Max aggregator for the count property you could then summarise the days together and calculate the maximum count that occurred in the provided day range.

It might be easier to do #1060 first.

This is what the updated documentation will look like:
Aggregation
The code for this example is Aggregation.
Aggregation is a key powerful feature in Gaffer and we will guide you through it in this walkthrough.
Aggregation is applied at ingest and query time.

Ingest aggregation permanently aggregates similar elements together. Elements will be aggregated if:

An entity has the same group, vertex, visibility and any specified groupBy property values are identical.
An edge has the same group, source, destination, directed and any specified groupBy property values are identical.

Query aggregation allows additional summarisation depending on a user's visibility permissions and any overridden groupBy properties provided at query time.

For this dev.walkthrough we have added a timestamp to our csv data:
M5,10,11,2000-05-01 07:00:00
M5,10,11,2000-05-01 08:00:00
M5,10,11,2000-05-02 09:00:00
M5,11,10,2000-05-03 09:00:00
M5,23,24,2000-05-04 07:00:00
M5,23,24,2000-05-04 07:00:00
M5,28,27,2000-05-04 07:00:00

The schema is similar to what we have seen before, but we have added a start date and a end date property.
These properties have been added to the groupBy field.
Properties in the groupBy will be used to determine whether elements should be aggregated together at ingest.
Property values in the groupBy are required to be identical for Gaffer to aggregate them at ingest.
Elements schema
{
  "edges": {
    "RoadUse": {
      "description": "A directed edge representing vehicles moving from junction A to junction B.",
      "source": "junction",
      "destination": "junction",
      "directed": "true",
      "properties": {
        "startDate": "date.earliest",
        "endDate": "date.latest",
        "count": "count.long"
      },
      "groupBy": [
        "startDate",
        "endDate"
      ]
    },
    "RoadHasJunction": {
      "description": "A directed edge from each road to all the junctions on that road.",
      "source": "road",
      "destination": "junction",
      "directed": "true"
    }
  }
}

Types schema
{
  "types": {
    "junction": {
      "description": "A road junction represented by a String.",
      "class": "java.lang.String"
    },
    "road": {
      "description": "A road represented by a String.",
      "class": "java.lang.String"
    },
    "count.long": {
      "description": "A long count that must be greater than or equal to 0.",
      "class": "java.lang.Long",
      "validateFunctions": [
        {
          "class": "uk.gov.gchq.koryphe.impl.predicate.IsMoreThan",
          "orEqualTo": true,
          "value": {
            "java.lang.Long": 0
          }
        }
      ],
      "aggregateFunction": {
        "class": "uk.gov.gchq.koryphe.impl.binaryoperator.Sum"
      }
    },
    "true": {
      "description": "A simple boolean that must always be true.",
      "class": "java.lang.Boolean",
      "validateFunctions": [
        {
          "class": "uk.gov.gchq.koryphe.impl.predicate.IsTrue"
        }
      ]
    },
    "date.earliest": {
      "description": "A Date that when aggregated together will be the earliest date.",
      "class": "java.util.Date",
      "validateFunctions": [
        {
          "class": "uk.gov.gchq.koryphe.impl.predicate.Exists"
        }
      ],
      "aggregateFunction": {
        "class": "uk.gov.gchq.koryphe.impl.binaryoperator.Min"
      }
    },
    "date.latest": {
      "description": "A Date that when aggregated together will be the latest date.",
      "class": "java.util.Date",
      "validateFunctions": [
        {
          "class": "uk.gov.gchq.koryphe.impl.predicate.Exists"
        }
      ],
      "aggregateFunction": {
        "class": "uk.gov.gchq.koryphe.impl.binaryoperator.Max"
      }
    }
  }
}

Once we have loaded the data into Gaffer, we can fetch all the edges using a GetAllElements operation.
Note this operation is not recommended for large Graphs as it will do a full scan of your database and could take a while to finish.
final GetAllElements getAllRoadUseEdges = new GetAllElements.Builder()
        .view(new View.Builder()
                .edge("RoadUse")
                .build())
        .build();

final CloseableIterable<? extends Element> roadUseElements = graph.execute(getAllRoadUseEdges, user);

All edges:
Edge[source=10,destination=11,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Mon May 01 23:59:59 BST 2000,count=<java.lang.Long>2,startDate=<java.util.Date>Mon May 01 00:00:00 BST 2000]]
Edge[source=10,destination=11,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Tue May 02 23:59:59 BST 2000,count=<java.lang.Long>1,startDate=<java.util.Date>Tue May 02 00:00:00 BST 2000]]
Edge[source=11,destination=10,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Wed May 03 23:59:59 BST 2000,count=<java.lang.Long>1,startDate=<java.util.Date>Wed May 03 00:00:00 BST 2000]]
Edge[source=23,destination=24,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Thu May 04 23:59:59 BST 2000,count=<java.lang.Long>2,startDate=<java.util.Date>Thu May 04 00:00:00 BST 2000]]
Edge[source=28,destination=27,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Thu May 04 23:59:59 BST 2000,count=<java.lang.Long>1,startDate=<java.util.Date>Thu May 04 00:00:00 BST 2000]]

You will see in the results, the timestamp has been converted into start and end dates - the midnight before the timestamp and a millisecond before the following midnight, i.e. the start and end of the day. This effectively creates a time bucket with day granularity.
So all edges with a timestamp on the same day have been aggregated together (ingest aggregation).
These aggregated edges have had all their properties aggregated together using the aggregate functions specified in the schema.
Next we will further demonstrate query time aggregation and get all the 'RoadUse' edges with the same source to the same destination to be aggregated together, regardless of the start and end dates.
This is achieved by overriding the groupBy field with empty array so no properties are grouped on. Here is the operation:
final GetAllElements edgesSummarisedOperation = new GetAllElements.Builder()
        .view(new View.Builder()
                .edge("RoadUse", new ViewElementDefinition.Builder()
                        .groupBy() // set the group by properties to 'none'
                        .build())
                .build())
        .build();

final CloseableIterable<? extends Element> edgesSummarised = graph.execute(edgesSummarisedOperation, user);

The summarised edges are as follows:
Edge[source=10,destination=11,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Tue May 02 23:59:59 BST 2000,count=<java.lang.Long>3,startDate=<java.util.Date>Mon May 01 00:00:00 BST 2000]]
Edge[source=11,destination=10,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Wed May 03 23:59:59 BST 2000,count=<java.lang.Long>1,startDate=<java.util.Date>Wed May 03 00:00:00 BST 2000]]
Edge[source=23,destination=24,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Thu May 04 23:59:59 BST 2000,count=<java.lang.Long>2,startDate=<java.util.Date>Thu May 04 00:00:00 BST 2000]]
Edge[source=28,destination=27,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Thu May 04 23:59:59 BST 2000,count=<java.lang.Long>1,startDate=<java.util.Date>Thu May 04 00:00:00 BST 2000]]

Now you can see the edges from 10 to 11 have been aggregated together and their counts have been summed together.
If we apply some pre-aggregation filtering, we can return a time window summary of the edges. The new operation looks like:
final GetAllElements edgesSummarisedInTimeWindowOperation = new GetAllElements.Builder()
        .view(new View.Builder()
                .edge("RoadUse", new ViewElementDefinition.Builder()
                        .preAggregationFilter(new ElementFilter.Builder()
                                .select("startDate")
                                .execute(new IsMoreThan(MAY_01_2000, true))
                                .select("endDate")
                                .execute(new IsLessThan(MAY_02_2000, false))
                                .build()
                        )
                        .groupBy() // set the group by properties to 'none'
                        .build())
                .build())
        .build();

final CloseableIterable<? extends Element> edgesSummarisedInTimeWindow = graph.execute(edgesSummarisedInTimeWindowOperation, user);

The time window summaries are:
Edge[source=10,destination=11,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Mon May 01 23:59:59 BST 2000,count=<java.lang.Long>2,startDate=<java.util.Date>Mon May 01 00:00:00 BST 2000]]

Now we have all the RoadUse edges that fall in the time window May 01 2000 to May 03 2000. This filtered out all edges apart from 2 occuring between junction 10 and 11. Therefore, the count is has been aggregated to just 2 (instead of 3 as seen previously).
Aggregation also works nicely alongside visibilities. Data at different visibility levels is stored separately then at query time, the query time aggregation will summarise just the data that a given user can see.
There is another more advanced feature to query time aggregation.
When executing a query you can override the logic for how Gaffer aggregates properties together.
So by default the count property is aggregated with Sum.
At query time we could ask for a change that, to force the count property is aggregated with the Min aggregator, therefore finding the minimum daily count.
This feature doesn't affect any of the persisted values and any store aggregation that has already occurred will not be modified.
So in this example the Edges has been summarised into daily time buckets and the counts have been aggregated with Sum.
Now at query time we are able to ask: What is the minimum daily count?
Here is the java code:
final GetAllElements edgesSummarisedInTimeWindowWithMinCountOperation = new GetAllElements.Builder()
        .view(new View.Builder()
                .edge("RoadUse", new ViewElementDefinition.Builder()
                        .preAggregationFilter(new ElementFilter.Builder()
                                .select("startDate")
                                .execute(new IsMoreThan(MAY_01_2000, true))
                                .select("endDate")
                                .execute(new IsLessThan(MAY_03_2000, false))
                                .build()
                        )
                        .groupBy() // set the group by properties to 'none'
                        .aggregator(new ElementAggregator.Builder()
                                .select("count")
                                .execute(new Min())
                                .build())
                        .build())
                .build())
        .build();

final CloseableIterable<? extends Element> edgesSummarisedInTimeWindowWithMinCount = graph.execute(edgesSummarisedInTimeWindowWithMinCountOperation, user);

we have increased the time window to 3 days just so there are multiple edges to demonstrate the query time aggregation.
The result is:
Edge[source=10,destination=11,directed=true,group=RoadUse,properties=Properties[endDate=<java.util.Date>Tue May 02 23:59:59 BST 2000,count=<java.lang.Long>1,startDate=<java.util.Date>Mon May 01 00:00:00 BST 2000]]

Merged into develop.


Allow new attributes to be defined in #update

right now model.update('status', true) is not working, as only the model.status field is updated, and not model.attrs.status (which is the field used in the save() method: this._schema.db[collection].update(this.attrs.id, this.attrs);).

#719

Thanks for kicking this off!
I don't think we need to modify the associations - they're taken care of, because associations must be defined on models. What we need to do is call _definePlainAttribute on attrs that are passed to update that the model wasn't aware of when it was instantiated.
I think we want to add a line here in the forEach that calls _definePlainAttribute for each attr. I believe _definePlainAttribute is idempotent (it can be called multiple times safely), but we should add a test for this.
Think you could update your PR?

I'm not sure how to use _definePlainAttribute, because I'm not sure what it does (define get / set on each attribute?, but then we still need to set the attribute later in the update hook, what's the difference with just writing this.attrs[attr] = this[attr] then??
The code on the belongsTo is just picked up from https://github.com/samselikoff/ember-cli-mirage/pull/648, just to insist this is really in need to be merged..

@samselikoff updated to use _definePlainAttribute, but we still need to add a line beforethis.attrs[attr] = attrs[attr] to make sure that this.attrs.hasOwnProperty(attr)) is true. Also added a test.
Removed the relationship part which was not part of this fix, but the PR in https://github.com/samselikoff/ember-cli-mirage/pull/568 seems very sensible to me (you intend to have a model returned by server.create but that's not the case for now and things need to work).
L

Also added the configurable option without which Object.defineProperty can not be called twice (and corresponding test which is failing without configurable).

Awesome, thank you for your work, I think we're almost there!

Starting to think we should have used a .attrs field on models. link.someNewProp = 'foo' is technically part of the api but this actually doesn't work:
link.someKnownProp = 'foo';
link.someNewProp = 'bar';

link.save();

Save won't persist the unknown prop. 😕
I guess the the best we can do for now is make update work with new attrs. Possible eventual paths forward are (1) tell users to interact with .attrs (2) use .set or (3) allow users to define a schema.

One last comment then we can merge this. Thanks for all your work!

🎊


chore(RHTAPWATCH-552): sync staging member cluster's for segment-bridge

The segment-bridge component had sync issues on some of its object. The cause seem to be the toolchain-host-operator setting, which moved into the segment-bridge directory and so got applied to all of its cluster's settings.
Applying the settings to the RH clusters in which
the toolchain-host-operator component does not exist, seem to be the cause for the unsync issue.
This change separates the toolchain-host-operator settings between the different clusters for the segment-bridge component. currently applied only for the staging environment, to verify it works before moving to production.

I maybe missing something here but how does the Application from the  ApplicationSet picks up the right folder of the component, e.g. components/segment-bridge/staging/stone-stg-m01?
kustomize build argo-cd-apps/overlays/staging | grep segment-bridge-member$ -A 29
  name: segment-bridge-member
  namespace: argocd-staging
spec:
  generators:
  - merge:
      generators:
      - clusters:
          selector:
            matchLabels:
              appstudio.redhat.com/member-cluster: "true"
          values:
            clusterDir: ""
            environment: staging
            sourceRoot: components/segment-bridge
      - list:
          elements: []
      mergeKeys:
      - nameNormalized
  template:
    metadata:
      name: segment-bridge-member-{{nameNormalized}}
    spec:
      destination:
        namespace: segment-bridge
        server: '{{server}}'
      project: default
      source:
        path: '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}'
        repoURL: https://github.com/redhat-appstudio/infra-deployments.git
        targetRevision: main

The path point to '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}' but clusterDir is not set.

I maybe missing something but how does the Application from the ApplicationSet picks up the right folder of the component, e.g. components/segment-bridge/staging/stone-stg-m01?
kustomize build argo-cd-apps/overlays/staging | grep segment-bridge-member$ -A 29
  name: segment-bridge-member
  namespace: argocd-staging
spec:
  generators:
  - merge:
      generators:
      - clusters:
          selector:
            matchLabels:
              appstudio.redhat.com/member-cluster: "true"
          values:
            clusterDir: ""
            environment: staging
            sourceRoot: components/segment-bridge
      - list:
          elements: []
      mergeKeys:
      - nameNormalized
  template:
    metadata:
      name: segment-bridge-member-{{nameNormalized}}
    spec:
      destination:
        namespace: segment-bridge
        server: '{{server}}'
      project: default
      source:
        path: '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}'
        repoURL: https://github.com/redhat-appstudio/infra-deployments.git
        targetRevision: main

The path point to '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}' but clusterDir is not set.

@Omeramsc, @hugares is right, you've reated directories with the cluster names but that does not provide a mechanism that would actually limit a resource to a particular kind of cluster. We have the deploy-to-host-cluster and deploy-to-member-cluster k-components to help us achieve that. You can find numerous examples of them being used in this repo.

I maybe missing something but how does the Application from the ApplicationSet picks up the right folder of the component, e.g. components/segment-bridge/staging/stone-stg-m01?
kustomize build argo-cd-apps/overlays/staging | grep segment-bridge-member$ -A 29
  name: segment-bridge-member
  namespace: argocd-staging
spec:
  generators:
  - merge:
      generators:
      - clusters:
          selector:
            matchLabels:
              appstudio.redhat.com/member-cluster: "true"
          values:
            clusterDir: ""
            environment: staging
            sourceRoot: components/segment-bridge
      - list:
          elements: []
      mergeKeys:
      - nameNormalized
  template:
    metadata:
      name: segment-bridge-member-{{nameNormalized}}
    spec:
      destination:
        namespace: segment-bridge
        server: '{{server}}'
      project: default
      source:
        path: '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}'
        repoURL: https://github.com/redhat-appstudio/infra-deployments.git
        targetRevision: main

The path point to '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}' but clusterDir is not set.

@Omeramsc, @hugares is right, you've reated directories with the cluster names but that does not provide a mechanism that would actually limit a resource to a particular kind of cluster. We have the deploy-to-host-cluster and deploy-to-member-cluster k-components to help us achieve that. You can find numerous examples of them being used in this repo.

I fixed the files and moved some things around, following other similar components; member clusters should not use the manifests for the host cluster's configurations.
as for the k-components,  deploy-to-member-cluster-merge-generator is already applied to segment-bridge's member cluster configurations; I did not applied the same for the host cluster configs as I couldn't find definitions for those components (or the difference between the "merge-generator" and non "merge-generator" components).

/lgtm

/retest


Please document performance

At the first glance, pure-Python seems slow, is the underlying crypto implemented in pure Python, or does it use something from stdlib, etc...
So, for someone who doesn't want to dig too deep, README.md could have a simple performance comparison, e.g. against pypi:scrypt with same settings.

The algorithm is in Pure-Python. It will be excruciatingly slow.
In PyPy it is snappy though.
The main values in a Pure-Python implementation

Ease of understanding for other learning about how scrypt works
Environments which do not allow modules

I have not had time to document performance, but it will not be good. :)


Improve description of budgets top ups

Make a flowchart/interactive tool to clarify process of requesting additional credits.

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."


sageWithDoc: fix documentation index

Description of changes
Since basically forever, Sage docbuilding on Nix generated harmless errors of the following form:
Extension error (matplotlib.sphinxext.plot_directive):
Handler <function _copy_css_file at 0x7fd51e53c8b0> for event 'build-finished' threw an exception (exception: [Errno 13] Permission denied: '<dir>/share/doc/sage/html/en/reference/valuations/_static/plot_directive.css')

This happens because matplotlib copies a css file from the Nix store using shutil.copy, resulting in a read-only file. For whatever reason[^1], the copying happens multiple times, and therefore we get a "Permission denied" error when overwriting the file.
Sage builds docs in two passes for parallelism: Subdirectories are built separately, and then a second pass stitches them together by looking at some state files (environment.pickle in particular) created by Sphinx in each of the documentation subdirectories. Sphinx deletes these state files if the build fails, but the "build-finished" callback is an interesting gray area: What happens if a "build-finished" callback fails? Well, before Sphinx 5 this was ignored, but Sphinx 5 considers this to be a build failure and deletes the relevant state files. Without them, Sage docbuilding can't create the documentation index.
This PR patches Sage's docbuilding to change plot_directive.css's permissions, because this is easier than patching matplotlib. Even though this is caused by copying a read-only file from the Nix store, I think this should be fixed upstream. I will file an issue as soon as I have time to investigate if the problem is with Sage or matplotlib (or even with our patch to use subprocesses for docbuilding), but I'd rather get this patch in soon since this fixes a user-reported problem.
Fixes #189607.
[^1]: I think _copy_css_file gets registered twice for some reason due to Sage's two-pass docbuilding setup, but weirdly both executions of the function happen during the second docbuilding pass?! Sage also tries to symlink the _static directory, but I think this happens later.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

The x86_64 failure is random (https://trac.sagemath.org/ticket/33751). Unfortunately, aarch64 is still completely borked and I have no clue how to debug it (there's more about it on https://trac.sagemath.org/ticket/34297).

Thanks @collares! Is there an easy way I can check this? I don't know anything about nix apart from simple nix-env invocations.

@aikrahguzar If you don't mind waiting for a while as Sage gets built by Nix, you can run nix-build -I nixpkgs=https://github.com/collares/nixpkgs/archive/382129c3a83f7e58b9fa57c18f516b471bbf6826.tar.gz -E 'with (import <nixpkgs> {}); sage.doc'. This will print a path as the very last line, as well as create a result symlink in your current directory. Both of these should point to the new docs.

t

@aikrahguzar If you don't mind waiting for a while as Sage gets built (on your machine) by Nix, you can run nix-build -I nixpkgs=https://github.com/collares/nixpkgs/archive/382129c3a83f7e58b9fa57c18f516b471bbf6826.tar.gz -E 'with (import <nixpkgs> {}); sage.doc'. This will print a path as the very last line, as well as create a result symlink in your current directory. Both of these should point to the new docs.

I tried this and I can confirm that I get a full index now. Thanks a lot!


sendFile and streamFile should be different

The first should just add to buffer, the latter streams straight to io


TypeError: Argument 'detection_confidence_threshold' has incorrect type (expected bool, got int)

On the Jetson Xavier (Jetpack 4.2), I got below TypeError error when trying to run Tutorial 6. I have ZED SDK 3.0.1 and ZED 2.0 device.  I built the python ZED sdk from source.  Any idea?
$ python3 object_detection.py
[ZED][Init] Depth mode: PERFORMANCE
[ZED][Init] Video mode: HD720@60
Object Detection: Loading Module...
Traceback (most recent call last):
File "object_detection.py", line 99, in
main()
File "object_detection.py", line 63, in main
obj_runtime_param.detection_confidence_threshold = 40
TypeError: Argument 'detection_confidence_threshold' has incorrect type (expected bool, got int)

It's a bug from the python wrapper, we'll commit a fix


Different return types for S3Path.stat() and WindowsUPath.stat()

Which operating system and Python version are you using?
Windows 11, Python 3.9.6
Which version of this project are you using?
0.1.3
What did you do?
I am attempting to use universal_pathlib in order to have a unified way of handling files whether they are local or in S3.
One of the things I need to do is get all the files in a folder (yes I know S3 doesn't have actual folders, but hopefully you understand what I mean) and get their sizes.
>>> from upath import UPath
>>> lpath = UPath('local_data/files/')
>>>spath = UPath('s3://test_bucket/files/')
>>> lfiles = lpath.iterdir()
>>> lf = next(lfiles)
>>> type(lf.stat())
<class 'os.stat_result'>
>>> sfiles = spath.iterdir()
>>> sf = next(sfiles)
>>> type(sf.stat())
<class 'dict'>
>>> lf.stat().st_size
1657
>>> sf.stat().st_size
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'st_size'

What did you expect to see?
I expected to be able to use same code to get file sizes whether they are in local directory or in S3
What did you see instead?
The return types of stat() are different for S3Path vs WindowsUPath and I can't get file size in the same way from each.
Would this difference in behavior be something you would consider reconciling?  Alternatively, do you have suggestions on another approach to achieving what I'm trying to do?

Hi @theogaraj
This is definitely an issue, and we should create a UPathStatResult class (name up for discussion) that provides an interface compatible with dict (fsspec) and os.stat_result
We need to ensure that the os.stat_result attribute names map to the equivalent type for each of the fsspec filesystems, too.
relevant fsspec issues:

https://github.com/fsspec/filesystem_spec/issues/526

For now I don't have a good recommendation other than
_stat = pth.stat()
size = _stat["size"] if isinstance(_stat, dict) else _stat.st_size

Cheers,
Andreas

I vaguely remembered that I implemented something related to this already...
getmtime on fsspec
https://github.com/mdshw5/pyfaidx/blob/cac82f24e9c4e334cf87a92e477b92d4615d260f/pyfaidx/__init__.py#L1318-L1345

@ap-- thank you for checking and responding so promptly.  Because of another problem I had with iterdir (logged as https://github.com/fsspec/universal_pathlib/issues/146) I ended up going with the slightly clunkier spath.fs.ls(str(spath)) and then accessing the ['size] attribute common to both local and S3.
So this is by no means a showstopper.  I'll track this issue and can update my code whenever someone is able to resolve these two issues.

Just ran across this. See https://github.com/apache/airflow/blob/main/airflow/io/store/stat.py for a stat compatible version.

I just ran across the same issue here and would be happy to fire a PR with a fix, though it looks like @ap-- may already be investigating?
@bolkedebruin make several updates to the airflow io provider while inheriting from UPath in https://github.com/apache/airflow/pull/35612 which are great prior art here.
I would propose that we just port the changes to support stat into UPath, which is now under:

stat: https://github.com/apache/airflow/blob/2.8.1/airflow/io/utils/stat.py
def stat: https://github.com/apache/airflow/blob/2.8.1/airflow/io/path.py#L180-L186

Collected Info
For links to info() dicts of a lot of FileSystem implementations check https://github.com/fsspec/filesystem_spec/issues/526#issuecomment-1936188996
All filesystems have "name", "size" and "type".

"name" is a string
"size" is an int or None
"type" is a string that is almost always "file" or "directory" but it can be "other", or (something else)

For translating to os.stat_result.st_* attributes, these are the keys that could be checked:

Attribute
Possible Info Keys

st_mode
mode, unix.mode, writable, isLink, nlink, permission, isexec

st_ino
ino, name, id, sha, hex, Digest?

st_dev

st_nlink
nlink, isLink

st_uid
uid, owner, uname, unix.owner

st_gid
gid, group, gname, unix.group

st_size
size

st_atime
time, last_accessed_on, accessTime

st_mtime
mtime, last_modified, last_modification_time_ms, timeModified, modify, modificationTime, LastModified, modified_at

st_ctime
created, creation_time, timeCreated, created_at

Additional info to be considered comes from specific filesystems:

https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#FileStatus
https://cloud.google.com/storage/docs/json_api/v1/objects#resource
https://developers.google.com/drive/api/reference/rest/v3/files#File
https://dropbox-sdk-python.readthedocs.io/en/latest/api/files.html#dropbox.files.FileMetadata

All data types need to be normalized to int


[Bug]: "Import Project" doesn't allow opening of a directory

Is there an existing issue for the same bug?

[X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
[X] I have checked the existing issues.

Describe the bug
I clicked on the "import project" button on the new UI.

When I clicked on this I expected that it would allow me to open a directory that I could use as the workspace, but in the mac file selection menu it doesn't allow me to open a directory (the "Open" button is inactive when I select a directory).

Current OpenHands version
0809d26f4d8a5514c0a255ed0a3bf3d999b1c29f

Installation and Configuration
make build; make run

Model and Agent
No response
Operating System
No response
Reproduction Steps
No response
Logs, Errors, Screenshots, and Additional Context
No response

@amanape has mentioned this only works with .zip files. Have to decide whether to make this more clear or to allow directories to be selected.

Oh, when this is running locally I just assumed it would set the workspace directory. It seems like a nicer solution than requiring a zip file

We'd probably have to zip the directory either way when doing so, seems a matter of extending functionality. There doesn't seem to be any downsides I can think of, I'm good with the idea.
@rbren Thoughts?

This seems to have reasonable support in desktop browsers: https://caniuse.com/input-file-directory
We will probably have to handle it as a multipart upload, if I had to guess...not great but doable

Or we could allow user to select a "folder" - and automatically zipping that folder for upload if we are using it on saas, otherwise just set that as workspace (in case it is launched locally?)


Fixing "N" sharp edges

"N" remains an attractive nuisance for supervisor-capable cores (for instance it is on the NOEL-V roadmap, and the Andes AX27 claims to implement it) due to the virtualization and storage-channel issues in the current spec.  While I remain skeptical of the value of "N" and support satp.MODE=Bare as an alternative, I now believe that it is possible to repair the actively harmful portions of "N" with minimally consequential changes:

Fixing the virtualization issue:

uret may be executed at any privilege level; regardless of privilege level, it sets pc to uepc and ustatus.uie to ustatus.upie.  Unlike other x ret instructions, uret never modifies the current privilege level.
Since uret does not read or modify any privileged state, including the privilege level, it has no effect on virtualization.

(Currently uret is an untrappable instruction which drops to U-mode, so code written for S-mode can detect classical virtualization by executing uret, taking a trap, then checking whether the trap appeared to originate from S or U mode.)

Fixing the storage channel:

If the N extension is implemented in an implementation that also provides supervisor mode, it must be possible to disable the N extension by setting misa.N to 0.  While N is disabled, any attempted access to uepc, ustatus, uscratch, utvec, utval, ucause, sideleg, sedeleg, uip, uie, or attempted execution of the uret instruction, must raise an illegal instruction exception.  Platform initialization code must set misa.N to 0 prior to the first entry to supervisor code, and misa.N must remain 0 until a specific request to enable it is received from the supervisor via a SBI call or other platform-defined mechanism.
Disabling access to U-mode trap CSRs prevents storage channels between user threads with a supervisor that is not aware of the N extension.  Disabling access to the S-mode sideleg and sedeleg prevents a storage channel between virtual machines managed by a hypervisor unaware of the N extension.

(This is essentially redundant with what the Unix platform spec says about "user-visible state", however the latter is extremely terse and frequently not read by hardware designers.  This is also largely redundant with what the functionality of setting NS=Off would be if NS were fully specified, but hardware designers do not seem to be waiting for us to specify NS.)

There is a broader issue here that IP vendors are advertising cores with "Linux support" that are prima facie incompatible with the Unix platform spec.  Andes manages to do this in a single sentence:

The 64-bit AX27 is a 5-stage processor that supports the latest RISC-V specification, including “G” (“IMAFD”) standard instructions, “C” 16-bit compression instructions, “P” Packed-SIMD/DSP instructions, “N” for user-level interrupts and Memory Management Unit (MMU) for Linux support.

I don't know what, if anything, should be done about the broader issue.

Since I received a private email requesting clarification on this: "NS" refers to the hypothetical future feature described in https://github.com/riscv/riscv-isa-manual/blob/d9ea1ca369d6f1f8e61644b0bb87416a4f2857d8/src/n.tex#L232-L235 .


Team 9 - Rough animations and sprite batches for unit troops (Soldiers)

Description
Task: Creating rough animations as well as sprites for the unit troops
Feature: This contributes to any units, especially Soldiers and effectively ensures they match the Enemy and Worker units as well (#8, #12, #5 )
Different animations will be implemented to every unique unit depending on their specialities and type of movements.
Examples
-Archer: The animations could include but are not limited to, aiming, shooting, and reloading the arrows.
-Hoplites: The animations could include but are not limited to, attacking, and blocking an attack.
Dependencies
This task is dependent on matching the animation with the opposing troops (Enemy) (see #12 ), as well as other friendly units (see #5)
Milestones
The list of troops needs to be animated.

[x] Archers
[x] Spearmen
[ ] Pikemen (possibly removed)
[x] Hoplites
[x] Swordsmen

The task includes:

[x] Creating sprites for the units
[x] Animation for every movement and direction
[x] Matching the design of the sprites with other teams

Documentation

Units Home
Animation

Member

Garryco Thandi (Harqua)
Yolanda Siruit (yolandawsiruit)

Oml why were none of these closed


BUG: Token action HUD for M&M not working after last update

When loading with the new Token Action HUD core, the game gives a UI notifications of "The installed Token Action HUD system module reuires Token Action HUD Core Module version 1.4. Install the required version to continue using Token Action HUD." and no token action HUD is working.
larkinabout said to "Roll back to Token Action HUD Core 1.4.22 until the Token Action HUD module for the system supports 1.5"

This should be fixed.
I haven't tested it in depth, so if there's a bug following the update, don't hesitate to let me know!


Add BatchTaskRunner to StressBench to run multiple task

What changes are proposed in this pull request?
According to the discussion in PR : Add write-type parameter to StressBench, add a tool BatchTaskFRunner to run multiple test for StressBench. Currently, only the plan for MasterStressBench is completed in this request.
**link : https://github.com/Alluxio/alluxio/pull/14842
Why are the changes needed?
The plan runner can help the operater to find out the overall benchmark of the system by executing multiple StressBench test.
Does this PR introduce any user facing changes?
The user can now use BatchTaskRunner to run multiple test.
**the introduction for BatchTaskRunner
The possible parameters are --clients, --threads, --warmup, --base, --stop-count，--create-file-size and the BaseParameter of StressBench.
The following is an example for the input in PlanRunner:
bin/alluxio runClass alluxio.stress.cli.BatchTaskRunner MasterComprehensiveFileBatchTask --stop-count 5000 --create-file-size 0 --threads 1 --base alluxio:///stress-master-base
The comprehensive plan will run tests in the following order:
CreateFile->ListDir->ListDirLocated->GetBlockLocations->GetFileStatus->OpenFile->DeleteFile

alluxio-bot, merge this please


chore: add terminal colors

[ ] TODO: verify ansi colors in fzf-lua, search uses terminal

see `:help terminal-config'
Configuration					*terminal-config*

Options:		'modified', 'scrollback'
Events:			|TermOpen|, |TermEnter|, |TermLeave|, |TermClose|
Highlight groups:	|hl-TermCursor|, |hl-TermCursorNC|

Terminal sets local defaults for some options, which may differ from your
global configuration.

- 'list' is disabled
- 'wrap' is disabled

You can change the defaults with a TermOpen autocommand: >vim
    au TermOpen * setlocal list

TERMINAL COLORS

The `{g,b}:terminal_color_x` variables control the terminal color palette,
where `x` is the color index between 0 and 255 inclusive.  The variables are
read during |TermOpen|. The value must be a color name or hexadecimal string.
Example: >vim
    let g:terminal_color_4 = '#ff0000'
    let g:terminal_color_5 = 'green'
Only works for RGB UIs (see 'termguicolors'); for 256-color terminals the
color index is just forwarded.

Editor highlighting (|syntax-highlighting|, |highlight-groups|, etc.) has
higher precedence: it is applied after terminal colors are resolved.

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


SA1137: Unexpected behaviour with switch statements

Suppose I have a method with a switch statement in it:
private string ToLongHand(int x)
{
    switch (x)
    {
        case 1:
            return "One";
        case 2:
        {
            // Do some complicated logic
            return "Two";
        }

        default:
            throw new ArgumentOutOfRangeException(nameof(x), "I cant count past 2...");
    }
}

SA1137 is raised for the brace immediately after case 2. Is that intended? Whilst personally I think this code is acceptable, I might be persuaded by a rule saying that all cases should have a brace or none of them should have.
Additionally, when I run the fix, it converts the code to the following:
private string ToLongHand(int x)
{
    switch (x)
    {
        case 1:
            return "One";
        case 2:
            {
            // Do some complicated logic
            return "Two";
        }

        default:
            throw new ArgumentOutOfRangeException(nameof(x), "I cant count past 2...");
    }
}

... which is definitely worse than before!
Theres also an issue if the first case statement has a brace and the second one doesnt - it highlights the second block and suggests that it should be aligned with the start of the "case" statement
I'm using StyleCop.Analyzers.1.1.0-beta004 on Visual Studio 2017 (v15.3.2)

Yes, this is a currently-unhandled edge case. I would expect the correct solution would involve comparing switch cases that contain exactly one block as a child to each other, and all other cases the way they are currently compared.


refactor incremental models

Sample output:
22:41:23 | Concurrency: 1 threads (target='dev')
22:41:23 |
22:41:24 | 1 of 1 START incremental model dbt_dbanin.users............. [RUN]
         +-> Building new table...
         +-> Dropping table: dbt_dbanin.users
         +-> Renaming `users__dbt_incremental_tmp` to `users`
22:41:26 | 1 of 1 OK created incremental model dbt_dbanin.users........ [SELECT in 2.66s]
22:41:27 |
22:41:27 | Finished running 1 incremental models in 2.93s.

I added a verbose var (default=False) which controls those in-model log lines. This branch generally cleans up the incremental model materialization. Additionally, if the --full-refresh flag is provided (and a couple other things are true), then the existing table will now be dropped after the new table is built.
Fixes https://github.com/fishtown-analytics/dbt/issues/525

@cmcarthur I was hoping to roll this into 090-a2 but I think this needs some more work. Let's aim for 0.9.1

👍


Unable to spawn anything in a server

As of Patch 6, I am unable to spawn anything in the game via the spawngun while in a server. It affects both vanilla items and NPCs as well as modded items. I can provide logs if need be.

Fusion does not work on patch 6, please wait for the next update


feat: add Dropbox as supported oauth provider

For testing follow these steps:

Login with a Dropbox account and create an app here https://www.dropbox.com/developers/apps/create
Select Scoped Access API, Full Dropbox Access, and the name of the app
In your app Settings, copy App key (Client ID) and App secret (Client Secret)
Create a .env file and add the credentials respectively NUXT_OAUTH_DROPBOX_CLIENT_ID and NUXT_OAUTH_DROPBOX_CLIENT_SECRET
In your app settings, set Redirect URIs to http://<IP_ADDRESS>:3000/auth/dropbox and click on Add
Scroll up and click on the Permissions tab
Scroll down to OpenID Scopes and select the openid and email scopes and click on Submit to save the changes
Go to http://<IP_ADDRESS>:3000/ and test Dropbox login

Thanks ❤


Crash in Publishes Menu while Repost is Pending

DESCRIPTION:
If the user Reposts someone else's post and then navigates to the Publishes menu while its still pending, the app will crash.

REPRO STEPS:

Launch LBRY 0.15.3 on Android 10
Open another user's video
Repost their video
Navigate to 'Menu > Publishes'
Wait a few seconds for screen to update

Results: Crash
Expected: Pending Publishes load

ENVIRONMENT:

Pixel 2 XL
Android 10
LBRY 0.15.3

MEDIA:

INTERNAL USE:

DEFINITION OF DONE:

[ ]  Tested against acceptance criteria
[ ] Tested against the assumptions of the user story
[ ] The project builds without errors
[ ] Unit tests are written and passing
[ ] Tests on devices/browsers listed in the issue have passed
[ ] QA performed & issues resolved
[ ] Refactoring completed
[ ] Any configuration or build changes documented
[ ] Documentation updated
[ ] Peer Code Review performed

LBRY ADDRESS:
bZSqrVdpxt5KJdn2eGMUzT9Jn3gEiRet7b
<EMAIL_ADDRESS>

Thank you for your report. Does this happen every single time?

It happened every time in 0.15.3, but the Publishes menu no longer shows Pending Reposts in 0.15.6, so the issue doesn't occur.  If it's intended design to not show Pending Reposts, then I think this can be closed, but if you do want Pending Reposts to display here then the Crash may reappear.

No longer occurs on 0.16.11.  Recommend closing.


Arguments to CompositeDisposable.add must have a .dispose() method

Fails to activate plugin in Atom 1.10.x

I saw this but didn have the time, I'll check this next week.
The other bug is the automated generated bug, so I marked this as duplicate.