Purchase QR disappears

The purchase QR code disappears if I don't pay first time round.

I'm not a fan of the unfunded purchases displaying in your Transactions section for a number of reasons:

The Purchases/Sales tabs imply finalized orders to me.
I'm not sure this matches the user's expectations. When they're checking out and abandon the checkout modal, the idea of that unfunded transaction living on forever seems odd to me.
I'm not sure anyone would think to check their transaction > purchases section to finalize an order they abandoned. If anything, they would be much more likely to go through the checkout process again on the vendor's page.
It's confusing to vendors. It will definitely be helpful to get a sense of how many users are abandoning orders, but that's more of a reporting thing that should eventually live somewhere else.

So imo the best way to fix this issue is by not even making it possible in the first place. When the user abandons the checkout modal, simply delete the transaction.

I don't think we have a way to delete the transaction. Until we do, we'll need to give the user a way to complete an order, especially since they may think they're done before they send any bitcoins.

Also, this was fixed a while ago.


feat(frequencies): add support function for frequencies conversion

Summary
Add a convenience method to convert Counts to QuasiDistributions in frequencies utils.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

7 of 7 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
593

Relevant Lines:
593

💛  - Coveralls


Add support for Multisig v2 LockBalance

Eventually LockBalance might need to be supported, but for now, just construction
Originally posted by @Schwartz10 in https://github.com/Zondax/filecoin-signing-tools/issues/327#issuecomment-726230634

Gah @jleni - just learned - we'll need to add support for LockBalance too. PL wants to be able to fund multisig wallets after their construction according to a vesting schedule, too.


placemarkFromCoordinates results different on IOS  and Android

placemarkFromCoordinates returning different results on IOS and android is this a bug or I am missing something? At android its working fine but on IOS few attributes like (thoroughfare, locality) are missing.

In this case the Geolocator plugin uses the native SDK to acquire placemark information which could result in slightly different data:

On iOS we use the CLGeocoder SDK.
On Android we use the Geocoder SDK

oh no that means there is no way. CLGeocoder is not supporting Pakistan according to this list
https://developer.apple.com/library/archive/technotes/tn2289/_index.html#//apple_ref/doc/uid/DTS40011305-CH1-TNTAG3


configurable: allow fixes for types from docblock

Allow auto-fixing from docblock if config like
<pluginClass class="Orklah\NotEmpty\Plugin">
	<fromDocblock value="true" />
</pluginClass>

Thanks!


[cloud sync] Cloud saved project disappeared, then reappeared.

This was on a low-end machine, struggling to be responsive. While signed in, user was editing a sprite's x and y position but the changes weren't being reflected in the simulator. To remediate, they thought they would try to reload the project from the home screen. The project was not listed.
After creating a new project then going back to the home screen, the previous project showed up.
This may be a UI refresh issue.

Closing as no repro. If this reproduces lets reopen


Cursor is Pointer instead of Caret after closing Settings

Steps to reproduce:

Open file to edit
Open settings (Click gear icon)
Close settings
Hover cursor on the text editor. It starts changing between Pointer and Caret really fast

OS: Windows 10

Same for me on macOS

Thanks for reporting this issue!
I can't reproduce it on Windows 11. Could you please check if this still exists?

I think #410 should have fixed this so I'm closing this ticket, but feel free to reopen if you experience this issue in the next release.


Remove dryrun logic

There's not really any need for the dryrun logic, it was a very niche implementation anyway.
One could consider separating the CouchbaseHelper class from the cluster initiation, then just passing a cluster instance to the class, and then offer a "dryrun" cluster which would basically do what the dryrun methods are doing today.

This was done as part of changes in #12, closing.


Linked app in Readme is not updated, request to switch to github ci which is free(unlimited minutes) for public projects

@prateekmedia I changed the link to a Playstore link


deleted any existing tlog-related (meta)data during delete vdisks

depends on:

#273;
functionality to delete tlogserver-dictated (meta)data;

open questions:

Should we also delete all aggregations?

And what to do if the aggregations are used by multiple chains (see: forks)?

@iwanbk how would we delete metadata and whatever else for a given vdisk, now that we use 0-stor? And what about the aggregations? I suppose we keep them there?
Or should this be deleted using a different zeroctl delete command rather than combining it with the already existing zeroctl delete vdisk command?

@iwanbk how would we delete metadata and whatever else for a given vdisk, now that we use 0-stor? And what about the aggregations? I suppose we keep them there?

We can simply delete the metadata because we can't forked it.
For the aggregation: keep if there is still another vdisk referencing it, otherwise delete it
Do we need this in alpha-8 milestone, because deleting 0-stor object is planned in beta-1 milestone

It's not in the milestone plans on zero-os/home, so I guess not. I'll move this issue to beta-1.
What about issue #280, is everything we need for that issue already available in 0-stor, or should it also be moved to beta-1?

What about issue #280, is everything we need for that issue already available in 0-stor, or should it also be moved to beta-1?

Please move it also

things to do:

remove metadata
remove reference list to the aggregation stored in 0-stor, and remove the aggregation if there is no other reference


Restore Alert Badges in App Nav

Lower right of user photo there used to be a badge indicating the number of outstanding alerts..
capture
In efforts to restore this we need to talk about/build a consensus around which alerts to display, in this location.
Totally open to a new system also possibly one that displays the count of multiple alerts of varying types.

Notifications vs Alerts
This discussion will detail the differences between the two, propose why and how both should be implemented and outline build steps moving forward.
Comparing
Notifications

informational, existence of alerts can generate a notification
exist as a single type
directly created by users through admin notifications center; indirectly created by system if an alert is set to precipitate a notification
loose scope, encompassing alert scope and extending to user groups or project members

Alerts

event based, a rule based conditional outcome generates an alert
exist on a gradient of types, (ok, warn, error)
indirectly created by system
focused scope, to single project or all projects

Implementation
The base premise will remain familiar to other jf elements, a list state from which CRUD functionality is accessed if you are a user with right, a details state displaying full information about a particular item
Creation
Notifications - Directly through a Notification Center(think present alert implementation, except not located under the Admin cluster of pages). This page will be visible to all for review of current and archived notifications, the caveat being admins and project managers are the only users able to add notifications (through the same mechanism used throughout jf); Indirectly by an alert if specified.
Alerts - Indirectly (exclusively) when a predefined condition is within or beyond set parameters. Conditions can be set on a single or multiple projects through a new Alert Center page
Visual Representation/Cues
Reconfigure user photo to have an icon bottom left (triangle with an exclamation point) and bottom right (a letter icon), each of these icons would have a badge consisting of the unread count of each.  Clicking either icon will reveal a popover of the outstanding items, selecting any one item will take you to the details state of that particular item. (Possible option to clear all? )
Build Steps
Some of this exists. Notifications (presently as alerts) can be reconfigure and implemented easily enough with minimal backend lifting (though this does not include targeted notifications, to members of a group or members of a project).  Alerts on the other hand will require a bit of lifting as presently none (to my knowledge) of this is in place.  This is something we need to discuss, but I would like to propose the following workflow:

The CRUD to submit alert conditions to the api, existing variables on which to establish conditions include budget, months remaining, service status, service health
api magic that sends a "everybody flip out" when a condition is met or exceeded
a service to regularly poll the api magic, to see if an "everybody flip out" has been transmitted, (in the case it has update the badge count)

I totally reserve the right to revise this as necessary.

SO @chriskacerguis @stackus  what I need is buyin or a nod to go forward in one direction or another.  maybe to begin, reconfigure alerts to be Notifications (as described above) modify the nav profile to support both counts (alerts and notifications) and setup a polling service to check for new notifications/ alerts?

Allen, we should discuss this if notifications are going to be a new thing to be added.


Depth data shows up as green in programs other than realsense-viewer

Camera Model: SR300
Firmware Version: <IP_ADDRESS>
Operating System & Version: Ubuntu 18.10
Kernel Version: 4.18.0
Platform: PC
SDK Version: 2.16.4
Issue Description
Trying to view data from the depth camera in other programs than realsense-viewer results in a green screen
The screenshots below are from howdy's test program(a face recognition program,similar to windows hello), and cheese webcam viewer,and terminal output of realsense-viewer

You should have two video devices when connecting the depth camera - one for depth information and one for color. If you select the second one, you should be able to use it as a regular web-cam. The depth device provides depth information that cannot be processed by regular cv algorithms. We hope that using librealsense more developers will start taking advantage of depth but RealSense also supports several other industry standards, such as OpenNi2 and Microsoft's new Video Capture API.

[Realsense Customer Engineering Team Comment]
Librealsense does provide the ability to stream the infrared stream of SR300.  Should be available in the API stack.  I believe it's outlined in the src/ivcam folder.

@PancakeSparkle - the reason it is problematic, is that for SR300 the infrared image is not exposed as a separate video device, but rather via special FOURCC formats of the depth device.


Can't select from dropdown

Target URL:
http://www.supremenewyork.com/shop/jackets/uaxjeqvro/fm9kozqa6
Target Element: #s
Problem: Can't select a value from the drop down. I've tried multiple things, only related question I could find on Stack Overflow was this,[https://stackoverflow.com/questions/45864516/how-to-select-an-option-from-dropdown-select] but none of these answers was my solution. Appreciate all help!

Solved on StackO - https://stackoverflow.com/a/51991895/9747324


windows: Generic improvements

Documentation:

timeout in seconds under the docstrings
Fix calc = find_window("name:Calculator") -> calc = windows.find_window("name:Calculator") in the main docs

Behaviour:

List user only processes (when matching a window based on PID) -> speed gain
windows.find() might be too verbose in the terminal output (printing the element tree multiple times when there are no matches)

@cmin764 The verbose errors may be tweaked in the config already.
See: robocorp.windows._config.Config.disabled_verbose_errors

@cmin764 The verbose errors may be tweaked in the config already.
See: robocorp.windows._config.Config.disabled_verbose_errors

Do you think it would be better to have these disabled by default and enabling them on request?

Well, initially it was like that, but I found myself wishing it was active when I made an error, which prompted the change (because then it gives you information to fix it), whereas the cases where I didn't want it were only those where I actually expected an error.

Finding a window inside the main window with .find() will work instead:
main_window = windows.find_window('control:WindowControl and class:Notepad')
page_window = main_window.find('"Page setup"')
page_window.send_keys("{Enter}")  # presses OK and closes the sub-window

This is on purpose... the problem with the approach on RPA.Windows is that it wouldn't scope the initial window and thus could end up going very deep on the depth first approach on wrong windows before finding the actual top-level window of interest, so, we now force the user to find the top-level window first.

@fabioz Do you see as good design to introduce a strict=True param to the finding functions so they won't raise when this is set to False? (to test the existence of an element/window)

I would not add this use case as a parameter to find
I think just having a clearly separate .exists() that returns a boolean would make the resulting code more readable and serve the use case.
So the use case stems from "detect in which state the app is" doing this by detecting is a specific element is present or not, and just having direct if-else paths etc.

OK, I did not understand the comment around the default in the previous messages.
This solves the use case better.

@fabioz We should pay attention on how we write the params description on the docstring, as any empty line break in between the argument description will break the bullet-point formatting. (example)


Add processing of images in sub-folders

Process images in 1 level deep subfolders in the specified root
If using --delete and --interval, no need to track duplicates

@marcbelmont resolved the 2 issues. Please review


fix: logging and pf improvements, skip namespaces & non-ClusterIP svcs

This PR contains a number of reliability and UX improvements for using
localizer in a larger cluster.

When creating a port-forward, we don't pass a stop channel anymore.
This was previously passed ctx.Done() which would silently stop the
port-forward on ^C. This caused invalid "use of closed connection"
logging that was confusing to the user but also possibly could end up
in a port-forward not being fully stopped.
When shutting down, pass a temporary context that has a timeout of 30
seconds. This ensures that /etc/host modifications, ip pool
cleanups, and other shutdown functions properly finish instead of
possibly being terminated midway through during normal shutdown.
Enabled delibird to remove tracing/metrics information being sent by
default in localizer binaries. This is a OSS friendly way of us
getting telemetry by logging to disk instead, which a user can opt to
send us when reporting bugs (more on that in future work!)
Upgraded all dependencies, including client-go. We still need a
fork, but thankfully our fork is a single line now so it's much more
likely we can upstream the change we made (the other change was
upstreamed a year ago by another user!)
Exposes stderr from the port-forwarder to the console with a colored
prefix. This makes it easier to tell when a port-forward was busted
but localizer didn't detect it (or if it did, why it failed!)
Skipped namespaces by default (kube-system) and enabled the user to
pass --skip-namespace to provide more (helps #212).
Skip non-clusterIP services. These aren't addressable in the cluster,
so we shouldn't create a tunnel for them. This helps with NodePort
and LoadBalancer service overhead that some larger clusters may
have.
Reduced logging to be easier to read, while keeping a lot of helpful
logs in the debug level. Defaults to info logging instead of
debug logging (pass --log-level debug for that!)

It does not seem like the --skip-namespace flag actually got hooked up to anything. is that on purpose?


Change how users access OpenCL kernels

Checklist

[x] Math issue #973

[x] Copyright holder: (fill in copyright holder information)
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

Rok Češnovar and Erik Štrumbelj (Faculty of Computer and Information Science, University of Ljubljana)
Steve Bronder

[x] the code base is stable

all unit tests pass
continuous integration passes

[x] the changes are maintainable

the code design is idiomatic C++ or there's a good reason it's not
please include appropriate documentation

[x] the changes are tested

[x] the changes adhere to the Math library's C++ standards

Summary
Easier Access To Kernels
Creates a kernel_cl_base singleton to manage the compiled kernels and a kernel_cl friend adapter class to give users access to kernels along with helper functions. Currently, kernels are compiled from within the opencl_context_base class. The GPU team is concerned that this class is becoming too bulky and so we created a new kernel_cl class that users can utilize to access kernels. Currently users run something like the following to compile and send arguments over to an OpenCL kernel.
cl::Kernel kernel = opencl_context.get_kernel("add");
opencl_context.set_kernel_args(kernel, C.buffer(), A.buffer(), B.buffer(),
                                   A.rows(), A.cols());

Users can now call the following
kernel_cl kernel("add");
kernel.set_args( C.buffer(), A.buffer(), B.buffer(),
                                   A.rows(), A.cols());

Remove Kernel Groups
This PR also removes the kernel groups such that each kernel is now compiled into it's own program. Originally the OpenCL context would compile kernels which were grouped together (IE 'cholesky', 'checks' ,etc.) into a single program with the idea being that if multiple kernels are used together it's smarter to compile all of them at once. While compiling multiple kernels into a single program is faster, as we add more kernels how we choose to label and group them would end up being difficult and too heuristic. When running some tests it did not seem like compiling the kernels separately produced a noticeable amount of overhead wrt the life of the executable.
Tests
The kernel_cl test checks whether the transpose kernel can be compiled. Code to run the test below
echo STAN_OPENCL=true>> make/local
echo OPENCL_PLATFORM_ID=0>> make/local
echo OPENCL_DEVICE_ID=0>> make/local
make test/unit/math/gpu/kernel_cl_test && ./test/unit/math/gpu/kernel_cl_test

Side Effects
None
Additional Notes
Leave additional notes for the code reviewer here.

Please create a new issue. It helps us manage this stuff; if there's a single issue per pull request, we can use "fixes #___" to automatically link and close issues.

whoa. I just read the description and that's great. what's with the naming? why kernel_cl_base?
If I'm understanding correctly, the constructor to kernel_cl looks up the compiled kernel by string?

We are following the same naming convention we setup in the opencl_context (opencl_context_base is the singleton, opencl_context is the friend class).
kernel_cl checks if a kernel with the provided name has been compiled. If yes, it sets the member that holds the compiled kernel. If it wasnt compiled yet, it compiles it first.
One side effect of this PR is also that it organizes how we treat the kernel parameters (like LOWER, UPPER,...) a lot better.

Along with what Rok mentioned above, I want to make a PR soon that changes all of the names from *_gpu_* to *_cl_*. I think it looks nicer and makes more sense. I'll file an issue soon describing the naming change in more detail

Can we also make it so that the cl files are not string literals but would actually be readable by an IDE? :D
Also I'm happy to review GPU stuff - going through a workshop with Rok and Erik now on it and it's really cool. Let me know if I can take this off your plate, @syclik

@seantalts, please and thank you! I've been buried trying to get the make stuff to work consistently. (It's working through Math and Stan now! Now onto CmdStan.)

Can we also make it so that the cl files are not string literals but would actually be readable by an IDE?

@seantalts I would absolutely love this but am having a hard time figuring out how to do it. When we tried reading them in with something like an ifstream we would get the instantiation fiasco error. Also wasn't sure how to specify the file locations since we are a header library.
The only other way I've seen to do it is via stringify like this stackoverflow Q. Though when I tried this it also threw up a bunch of errors, though maybe this is worth a second look
If you know a better way to do it then I'm 100% about it!

Also I'm happy to review GPU stuff

!! V appreciated !!

Did you try using FILE to get the correct directory and then read it in at runtime? (I haven't tried it but can play around with it if you haven't either)
https://stackoverflow.com/questions/20394183/how-to-get-current-source-path-in-c-linux

Another idea that seemed to work for me locally:
std::string kernel_code =
    #include "sassy.cl"

And
#define STRINGIFY(src) #src

STRINGIFY(
    void kernel matrix_mul(global float* C, const global float* A, const global float* B, const unsigned int M, const unsigned int N, const unsigned int K) {
      const unsigned int i = get_global_id(0);
      const unsigned int j = get_global_id(1);
      float sum = 0.0;
      for(unsigned int k = 0;k < N;k++){
        sum+=A[i*N+k]*B[k*K+j];
      }
      C[i*K+j] = sum;
    };
);

Seems slightly different from the one that is not supposed to work for GCC... I actually double checked and I think this is what the original poster ended up doing. Did this not work for our compilers?

@seantalts I tried this locally with the https://github.com/bstatcomp/math/blob/kernel_cl/stan/math/gpu/kernels/add_matrix_kernel.cl and it seems to work. Dealing with defines becomes a bit more trickier, but I can live with not having the "niceties" like A(i, j) if it means that the kernels are readable in a proper IDE.

I did not try FILE yet, I'll give that a shot tonight. I personally would really like to keep those #defines in there since they make the code a lot more legible

I don't have permission to view that link :P

Dealing with defines becomes a bit more trickier, but I can live with not having the "niceties" like A(i, j)

Not sure what that's referring to but I just started looking at this stuff today, sorry!

Oh, I took a loot at the matrix multiply kernel and I think I get it. The #define STRINGIFY(src) #src does not properly escape other pre-processor macros. I have some hopes for FILE! You'll likely need to use Boost's filesystem code to manipulate the path you get that way.

The error is something like this https://stackoverflow.com/questions/9723793/undefined-reference-to-boostsystemsystem-category-when-compiling and the only viable solution is to link with boost_system.
I will go and check what boost does and try and throw something together today.

I found the defines you were talking about: https://github.com/bstatcomp/math/blob/develop/stan/math/gpu/kernels/add_matrix_kernel.cl#L3
What if you just created functions marked inline? Wouldn't the compiler be smart enough to figure it out?

With the FILE approach, wouldn't it break the program if the user moves the program to another system? Idk if that's a big deal for us or not. Working with an MPI cluster of computers I think that could be bad?

Yes, it would break if transferred to a system that didn't have the kernels at the same absolute path. However, I think Sebastian mentioned that on the MPI systems he's worked on, they all share the same file system.

Also is it possible to pass a define to the compiler like -DA(i, j) A[j * rows + i]? If so then the STRINGIFY() solution does seem to give us what we need.

I think so, but I'd rather someone not have to go look through the makefiles to see how the kernel code is going to be executed. That seems like a nightmare for onboarding new people and my own self in the future, haha :P And according to this stackoverflow post OpenCL will inline all functions automatically, though I don't see a way to make functions that match the convenience of the macros (since C99 doesn't seem to allow nested functions).
There are some other creative solutions here. I kind of like the accepted answer - we could add a build step like that to make, though I'm not sure if xxd comes with RTools (ugh why is it so hard to find out what RTools includes?).
Sorry for the deluge of information / options here. Everything has negatives, haha.

@SteveBronder I tried this just now and it works. We could do this and write the available macros in a doc, so that people that will write kernels would know what is already available.
@seantalts those would not be defined in the makefiles, but in kernel_cl.hpp https://github.com/stan-dev/math/pull/966/commits/e5de8111b5e88db1fb3a8bf49281f4cf53240f95. Hope this time the link works. Sorry about the link earlier, I still dont understand how you dont have the rights since its a fork.

Nice! Yes we could include docs in each kernel / in kernel_cl to say, "Hey we have some helper things defined you can use these in a kernel."
@seantalts how do you feel about the above? Would that be too weird when onboarding new people? I also like the first answer from that stackoverflow post as well. Idt I'll have time tonight but tmrw I'll give that a shot.

I went with the stringify version because it's a clean mix of 'better than before' and 'level of effort'.
I started messing around with the an xxd version, but then we have to #include a file that doesn't exist yet and I didn't like how I was making the makefile.
For the helper macros, I made a separate file helpers.cl (that contains a string literal) which is included at the top of the kernel compilation and contains the helper macros. For documenting / making people aware of the helpers I included a function get_helpers() in kernel_cl to grab the string. Though the function mostly exists to tell users/developers, "Hey, here are some kernel macros that are good."

I think this is great!
It makes the kernel code readable without the need of having kernel sources mandatory to run the executable. The macros are still less readable but I think that we can live with that.
I would maybe just add a line in kernels that use macros. Something like This OpenCL kernel uses macros that are defined in helpers.cl.

I hadn't heard of scoped enums before this, but they look like the way to go for enums.

@rok-cesnovar Looks like we are clear to switch to scoped enums if either you or Steve want to. Not required for this PR.

Great, thanks!
I would prefer changing it here, as we are already refactoring the kernel stuff.
I will let @SteveBronder anwser the other review questions since he did most of the kernel_cl_base stuff.

We currently only allow GPUs (the definition is set here). I forgot why we decided on this instead of CL_DEVICE_TYPE_ALL, I can't really think of a reason.
You can change this locally and run the tests. There is no reason they should not work on CPUs.
However, the troubling thing here is that on a system without a GPU it should output the CL_DEVICE_NOT_FOUND error here while your error seems to suggests that it tried to compile and failed for some reason.
I know I tried this on a GPU-less AWS instance a while ago. I have to check this again. Will report back.
Thanks!

Okay, I'm going to mess around with it a bit, I think I might see an interesting way to organize this stuff. I have some questions though that might preclude that:

Any reason we're not doing many of these functions as operators on matrix_gpu?
It's less performant to have to access two static singletons, one to get a queue, and one to get the compiled kernels. Can these be the same, or possibly have e.g. kernel_cl have a pointer to the opencl_context?
What about a builder pattern here? Something like

kernel_cl::get(Kernels::copy)
  .setArgs(A, B, A.rows(), B.rows())
  .enqueue(cl::NullRange, cl::NDRange(A.rows(), A.cols()), cl::NullRange, NULL, NULL);

Do we always know the arguments to cmdQueue.enqueueNDRangeKernel(...); based on the arguments to the function? It seems like in all of the current kernels cmdQueue.enqueueNDRangeKernel always gets the same arguments.
Any reason we did this static opencl_context opencl_context; thing instead of just creating static methods on opencl_context or opencl_context_base?
Did anyone look at boost::compute?

I am guessing you guys talked about these before... if anyone has links handy that's great, otherwise feel free to say "yes we talked about boost::compute before and decided it was too future" or whatever :)

Any reason we're not doing many of these functions as operators on matrix_gpu?

Not really. At first we had these as separate functions and have not really discussed this. Should we move them? I am fine with reworking this, it would make the code where we use add/subtract/multiply/... more readable. Can we keep the operator overload in separate files (add.hpp, subtract.hpp) or do you prefer having them inside the class definition?

It's less performant to have to access two static singletons, one to get a queue, and one to get the compiled kernels. Can these be the same, or possibly have e.g. kernel_cl have a pointer to the opencl_context? What about a builder pattern here? Something like...

The build pattern looks good to me. Since we decided to split kernel_cl from the context to not make the opencl_context classes too bloated, I think the pointer to the context would be a better idea.

Do we always know the arguments to cmdQueue.enqueueNDRangeKernel(...); based on the arguments to the function? It seems like in all of the current kernels cmdQueue.enqueueNDRangeKernel always gets the same arguments.

It gets more tricky once we will start including more complex stuff, where we have to set the thread-block sizes for instance. We check for the maximum allowed number of threads in a block and decide based on that. See here.

Did anyone look at boost::compute?

We had a quick discussion in a pull request on our fork a while ago: https://github.com/bstatcomp/math/pull/1#issuecomment-366102980

Gotcha, that helps... Did you guys look at KernelFunctors at all?

 Curious what you guys think of this code for defining a kernel,
namespace opencl_kernels {

class unary_kernel {
 private:
  const char* name_;
  const char* source_;

 public:
  unary_kernel(const char* name, const char* source): name_(name), source_(source) {};
  auto make_functor() const {
    return cl::make_kernel<cl::Buffer, cl::Buffer, int, int>(
        kernel_cl::compile(name_, source_));
  };

  void operator()(cl::Buffer src, cl::Buffer dest, int rows, int cols) const {
    auto f = make_functor();
    f(cl::EnqueueArgs(opencl_context.queue(), cl::NDRange(rows, cols)),
      src, dest, rows, cols).wait();
  }
};

const unary_kernel copy("copy",
#include <stan/math/gpu/kernels/copy_matrix.cl>  // NOLINT
                         );

const unary_kernel transpose("transpose",
#include <stan/math/gpu/kernels/transpose_matrix.cl>  // NOLINT
                         );
...
}

...and using it,
inline void copy(matrix_gpu& dst, const matrix_gpu& src) {
  check_size_match("copy (GPU -> GPU)", "src.rows()", src.rows(), "dst.rows()",
                   dst.rows());
  check_size_match("copy (GPU -> GPU)", "src.cols()", src.cols(), "dst.cols()",
                   dst.cols());
  if (src.size() > 0) {
    try {
      /**
       * Copies the contents of the src buffer to the dst buffer
       * see the matrix_gpu(matrix_gpu&) constructor
       *  for explanation
       */
      opencl_kernels::copy(src.buffer(), dst.buffer(), dst.rows(), dst.cols());
    } catch (const cl::Error& e) {
      std::cout << e.err() << std::endl;
      check_opencl_error("copy GPU->GPU", e);
    }
  }
}

I went through a bunch of different designs today and this one seems the cleanest, though there's still more work to do and open questions:

Would like to somehow only access a singleton once (instead of once for the kernel and once for the command queue)
I haven't done binary kernels yet but I think they should be very similar. Don't think they need to share any code - will probably just copy and paste the unary_kernel class instead of trying to abstract that they both have name_, source_, and make_functor...
I haven't seen any places where it's useful to have these kernels all share some interface or concept, but the make_functor is in there to show one way that could work. It's only necessary if some users of the kernels will want to supply their own cl::EnqueueArgs, which @rok-cesnovar hinted at.
I think the configurable triangular ones might require their own templated _kernel classes, which is fine I think.

Should these tests work with a CPU OpenCL device?

Yep. We were/are focused on GPUs, though at one point in time did support CPUs. We can def support both (related, I also want to rename matrix_gpu to matrix_cl)

Any reason we're not doing many of these functions as operators on matrix_gpu?

We could definitely overload operators for the matrix_gpu. I wanted to have the arguments to a lot of these be the matrix_gpu instead of the buffers etc, but matrix_gpu imports kernel_cl so as it's currently written we would need to move the kernel_cl functions out to another header and do
kernel_cl::copy(matrix_gpu& A)

In a previous PR Dan didn't like this style because it's not used elsewhere in stan so I didn't do that. Though if we want to reconsider that I'm open to it. We could define them in each functions header ie kernel_cl::copy_triangular(...) would be above the definition of the copy_triangular() function.

It's less performant to have to access two static singletons, one to get a queue, and one to get the compiled kernels. Can these be the same, or possibly have e.g. kernel_cl have a pointer to the opencl_context?

I agree that I don't like the two singletons thing now, though I think I would rather move the kernel compilation into the opencl_context and still keep the kernel_cl class as an API for getting and setting kernels.

What about a builder pattern here? Something like
kernel_cl::get(Kernels::copy).setArgs(A, B, A.rows(), B.rows()).enqueue(cl::NullRange,  cl::NDRange(A.rows(), A.cols()), cl::NullRange, NULL, NULL);

If you and Rok both want a builder style I'm cool with it, though personally I like there to be a separation of the kernel and queue. I've been trying to read about design patterns that would help abstract away the job -> queue style OpenCL has. I haven't found anything yet but I'm open to playing around with some ideas.

Do we always know the arguments to cmdQueue.enqueueNDRangeKernel(...); based on the arguments to the function? It seems like in all of the current kernels cmdQueue.enqueueNDRangeKernel always gets the same arguments.

As Rok said above.

Any reason we did this static opencl_context opencl_context; thing instead of just creating static methods on opencl_context or opencl_context_base?

We did the static class because we wanted a single context. There's nothing inherently wrong with having multiple contexts, though I think when we start working on multiple devices it's going to be easier to manage if we force all the kernels into one queue and context. Would having a non-static class with static members still allow for only one context to exist?
We were also running into initialization fiascos and our solution to that at the time was having a singleton. Though I'm a bit hazy on where and why those popped up.

Did anyone look at boost::compute?

As Rok noted I found boost compute awhile ago, but was not a big fan of the style. Also the current build seems to be failing.

Gotcha, that helps... Did you guys look at KernelFunctors at all?

I'm looking at the header for OpenCL 1.2.8 and it does not look like KernelFunctor is defined?
https://github.com/stan-dev/math/blob/develop/lib/opencl_1.2.8/CL/cl.hpp#L4834
Though looking at how it works I really like it for our application.
https://stackoverflow.com/a/23993715/2269255
Mby we could just make that?

On Sat, Aug 18, 2018 at 19:28 Steve Bronder<EMAIL_ADDRESS>wrote:

Should these tests work with a CPU OpenCL device?
Yep. We were/are focused on GPUs, though at one point in time did support
CPUs. We can def support both (related, I also want to rename matrix_gpu to
matrix_cl)

Oh let’s not support both unnecessarily (CPU seems much worse than threads
by default) but was just having trouble running the tests on both. This
could be an issue in the future if someone writes code that breaks gpu and
wants to reproduce it locally :/ seems fine for now.

Any reason we're not doing many of these functions as operators on
matrix_gpu?

We could definitely overload operators for the matrix_gpu. I wanted to
have the arguments to a lot of these be the matrix_gpu instead of the
buffers etc, but matrix_gpu imports kernel_cl so as it's currently written
we would need to move the kernel_cl functions out to another header and do
kernel_cl::copy(matrix_gpu& A)
In a previous PR Dan didn't like this style because it's not used
elsewhere in stan so I didn't do that.

Which part of the style did he not like?
Though if we want to reconsider that I'm open to it. We could define them

in each functions header ie kernel_cl::copy_triangular(...) would be
above the definition of the copy_triangular() function.

It's less performant to have to access two static singletons, one
to get a queue, and one to get the compiled kernels. Can these be the same,
or possibly have e.g. kernel_cl have a pointer to the opencl_context?

I agree that I don't like the two singletons thing now, though I think I
would rather move the kernel compilation into the opencl_context and
still keep the kernel_cl class as an API for getting and setting kernels.

That’s a good idea.

What about a builder pattern here? Something like
kernel_cl::get(Kernels::copy).setArgs(A, B, A.rows(),
B.rows()).enqueue(cl::NullRange, cl::NDRange(A.rows(), A.cols()),
cl::NullRange, NULL, NULL);

If you and Rok both want a builder style I'm cool with it, though
personally I like there to be a separation of the kernel and queue. I've
been trying to read about design patterns that would help abstract away the
job -> queue style OpenCL has. I haven't found anything yet but I'm open to
playing around with some ideas.

What do you see the queue being useful for on its own to an end developer?
The design I posted has an entry point using kernel functions for someone
to override queue and work shape with cl::EnqueueArgs but the default I
assumed would not care about queue.

Do we always know the arguments to
cmdQueue.enqueueNDRangeKernel(...); based on the arguments to the function?
It seems like in all of the current kernels cmdQueue.enqueueNDRangeKernel
always gets the same arguments.

As Rok said above.

Any reason we did this static opencl_context opencl_context; thing
instead of just creating static methods on opencl_context or
opencl_context_base?

We did the static class because we wanted a single context. There's
nothing inherently wrong with having multiple contexts, though I think when
we start working on multiple devices it's going to be easier to manage if
we force all the kernels into one queue and context. Would having a
non-static class with static members still allow for only one context to
exist?

There is a context base and a context. You want only one context base I
think. The others are just helper functions and could be within a namespace
or static on a class, I believe...

We were also running into initialization fiascos and our solution to that
at the time was having a singleton. Though I'm a bit hazy on where and why
those popped up.

Did anyone look at boost::compute?

As Rok noted I found boost compute awhile ago, but was not a big fan of
the style. Also the current build seems to be failing.
Gotcha, that helps... Did you guys look at KernelFunctors at all?
I'm looking at the header for OpenCL 1.2.8 and it does not look like
KernelFunctor is defined?
https://github.com/stan-dev/math/blob/develop/lib/opencl_1.2.8/CL/cl.hpp#L4834

It’s not in 1.2 but you can use auto with make_kernel still.
Though looking at how it works I really like it for our application.

https://stackoverflow.com/a/23993715/2269255
Mby we could just make that?

Check out my post above, see what you think!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/pull/966#issuecomment-414073774, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAxJ7EiW2LzvLbSG0IdIOKcSWoQbf8Saks5uSE6rgaJpZM4VvFmt
.

--
Laboriously extracted via diminutive keyboard

Apologies Sean I'm just seeing your comment above starting with

I went a little crazy on my day off

I think that's really neat! Could you put that up on a branch so I could play around with it?

Which part of the style did he not like?
Pinging @syclik. I can't remember but I think it was just that we did not use that style elsewhere in stan

Oh I'm just wondering what you are referring to when you say "that style" - something about namespaces or function calls or ?
I will put it up on a branch but I have to warn you, it's not pretty. I went through so many styles and the code is a mess right now - I started focusing on just copy and transpose (and the transpose and kernel_cl tests) and got that working end-to-end first. The rest doesn't work right now.

Here's maybe the best way to view the diff between this PR and the branch online: https://github.com/bstatcomp/math/pull/7/files

If you and @rok-cesnovar think this is promising I'm happy to finish fleshing it out, but I wanted to 1) get it going end-to-end and make sure it could actually work and 2) get feedback first

I'm looking through it rn and like it! I would be for using this style, going to play around with what you posted and try to learn a bit more about variadic templating because I think we would want that for make_functor and I want to understand variadic templating more

I wrote a short example with notes on using variadic arguments which may be helpful.

Also just a note for those following this thread: some discussion has continued on the proposal above here: https://github.com/bstatcomp/math/pull/7

Without digging through notes, one of the things was the use of forward
declarations and implementation somewhere else. I think the only place
where we have it in the math library now is where it's absolutely
necessary; in the memory allocation area. For consistency, that was another
thing... some things were declare + defined and others were declared and
defined elsewhere, within the same class. That was a bit confusing. If it
wasn't detrimental to the code, I thought it was better for it to look like
the rest of the codebase unless there was a good reason otherwise.
For adding methods to matrix_gpu, there wasn't an objection to making the
class heavier.
On Sun, Aug 19, 2018 at 5:39 AM seantalts<EMAIL_ADDRESS>wrote:

Also just a note for those following this thread: some discussion has
continued on the proposal above here: bstatcomp#7
https://github.com/bstatcomp/math/pull/7
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/pull/966#issuecomment-414115866, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F9IzttQ3rhwBacP2WihZ_IeQo92vks5uSTI3gaJpZM4VvFmt
.

hmmm: http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Math Pipeline/detail/PR-966/23/pipeline

Thanks @seantalts for all the help and effort in helping us improve this! Really appreciated.

The GPU tests no longer run on a PR and they broke on develop I think because we all forgot about a GPU test that isn't in the GPU folder:
test/unit/math/prim/mat/fun/opencl_copy_test.cpp
Should we move that into the gpu test folder? or

I would move it. Can we do it in #1001 ?


StarRating can display incorrectly as a flex child with siblings

Notice that the StarRating in this CodeSandbox has rating={3} but it gets displayed as rating={2} or rating={5}. Thanks @toraora and @aislematt for the bug report:
https://codesandbox.io/s/thumbprint-571-ixmdi
cc: @tomgenoni

The StarRating component is set to display: inline-block and the rating functionality relies on the width of the component (~150px). The width of the highlighted stars is expressed in a width percentage, e.g. 2.5 stars becomes width="50%" or roughly 75px.
When this component is an immediate child of a flex flex-column parent the width of the star rating becomes 100%. This means that a width of 50% is now computed based on the outer container.

   <StarRating.../>

...other content...

I don't see a good fix in the component itself* so I'd recommend we show an example in the documentation to work around it.

Add example with the <StarRating/> in a
.

<StarRating.../>

...other content...

Add example with an additional items-start class.

    <StarRating.../>

...other content...

I'd recommend (1) as it likely has the fewest number of potential side-effects.

Note we could add this wrapping div to the component but it already has a lot of nodes and is only needed for this one situation AFAIK.

Chatted with Tom offline and we decided to simplify the StarRating component works so that it doesn't rely on the width or display of the parent. This will also simplify the outputted DOM which should help avoid the "excessive DOM" warnings in Ligthhouse.


[AIRFLOW-2545] Eliminate DeprecationWarning

JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title.

https://issues.apache.org/jira/browse/AIRFLOW-2545

Description

[ ] Here are some details about my PR, including screenshots of any UI changes:

Do not import BaseOperator as KubernetesOperator.
This eliminates confusing DeprecationWarnings when
setting up a default airflow install where additional
kubernetes modules are not installed.
Also, use LoggingMixin instead of logger module.
Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Code Quality

[ ] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

cc @NielsZeilemaker 👍


How to retrieve data from the fields?

The documentation shows how to "start" using it, but doesn't show how to complete the process. It seems that it solely relies on form submission to transfer the data... this would be helpful to be documented. I looked into the source code, and there doesn't seem to be any methods that "get" the data from the fields for an AJAX post.

Added get value functionality, add further examples and extend readme. As of v1.0.11 9489130

Thanks!


Allow for the —names flag when +opening things

🚀 Feature Proposal
Let us +open (openable) —-names and have the loot received display the name tags of the items received. Would be very useful for BSO boxes, but would also have increased utility when Fishy’s openables expansion is released.
From Benny: --text flag would also be good

Bump

Bump

Bump

can now do this


Tests for ockam_hub

Currently ockam_hub application runs a few workers and provides an HTTP API to create kafka services, but it doesn't have any unit tests.
Some test coverage for Ockam.Hub.Service.Alias, Ockam.Hub.Service.Echo, Ockam.Hub.KafkaStreamHandler and Ockam.Hub.Web would be a good start for the test coverage.

Addressed in #1456


CREATE TABLE with PARTITION BY is unsupported

Any attempt to create a table with a partition column[s] specified produces this error.
For example:
CREATE TABLE test_events_date (ID date) PARTITION BY ID
Result:

BigQuery error in query operation: Error processing job 'test:bqjob_r3d26e65599bb2335_000001867f166412_1': failed to analyze: INVALID_ARGUMENT: CREATE TABLE with PARTITION BY is
unsupported [at 1:41]

PS: used docker image ghcr.io/goccy/bigquery-emulator:0.2.13

It seems that any feature related to PARTITIONED TABLES has issues, such as the _PARTITIONTIME pseudo column.

Encounter something similar googleapi: Error 400: failed to analyze: INVALID_ARGUMENT: Unrecognized name: _PARTITIONTIME [at 44:23], jobInternalError

The same is true for me. Are there any updates on this issue ?

At the moment, I can create a partitioned table using the client libraries. At least, I won't get an error (what already helps in tests and local emulation!). The emulator, however, accepts queries that would not work with BigQuery for the schema.
Here's an example. Given the following table definition
{
    "type": "TABLE",
    "schema": {
      "fields": [
        {
          "mode": "REQUIRED",
          "name": "value",
          "type": "DATE"
        }
      ]
    },
    "timePartitioning": {
      "field": "value",
      "type": "DAY",
      "expirationMs":<PHONE_NUMBER>00,
      "requirePartitionFilter": true
    }
  }

BigQuery would reject a query that does not contain the value field within its' WHERE clause, where the emulator accepts such a query.
I guess it is a lot of work to support all the partitioning features of BigQuery, but I could imagine that it's easier to check if the query is valid according to requirePartitionFilter fields of the schema. This would at least help to ensure that queries are well formed. Would this be a first step into the direction of better partitioning support ?

Are there any updates on this topic ?

Hi, any updates around it?


Documentation could include more details

There is nothing about the spline itself in the documentation, like a model description. That should be added.

It's added in b08b67cc16f107980af3abd14af623f8dea0443d


Big refactor

Unfortunately the CPU offloading is failing with this error:
andromeda_chain-guidance-1  |   File "/guidance_server/utils/load_gptq.py", line 22, in load_gptq_model
andromeda_chain-guidance-1  |     model = load_quant_with_offload(model_path, checkpoint, wbits=gptq_settings.wbits, groupsize=gptq_settings.group_size, pre_layer=gptq_settings.pre_loaded_layers)
andromeda_chain-guidance-1  |   File "/usr/local/lib/python3.10/dist-packages/gptq_for_llama/llama_inference_offload.py", line 226, in load_quant
andromeda_chain-guidance-1  |     model.model.layers[i].to(DEV)
andromeda_chain-guidance-1  |   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 295, in __getitem__
andromeda_chain-guidance-1  |     return self._modules[self._get_abs_string_index(idx)]
andromeda_chain-guidance-1  |   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 285, in _get_abs_string_index
andromeda_chain-guidance-1  |     raise IndexError('index {} is out of range'.format(idx))
andromeda_chain-guidance-1  | IndexError: index 32 is out of range

Error was causing by passing too many pre loaded layers. By setting a lower value (like 20), it works fine. It is super slow though.


cdl: Disable buffering when logging to a file

Switch to fstream for consistency with the dump code.

CI CrashDiagnosticLayer build queued with queue ID 292152.

CI CrashDiagnosticLayer build # 301 running.

CI CrashDiagnosticLayer build # 301 passed.


Should Scroll to edited location

If I'm working near the bottom of a document, when MarkPad refreshes the preview pane it remains at the top.
It should scroll down in the preview to match where I'm editing, so I can see the new edits I've made (or the image I just inserted).  Currently, I keep having to scroll all the time to look at the results of my formatting.

:+1:

+1


-setup called twice after -init

-initWithFrame: method is Designated Initializer for UIView. UILabel, UITextView, UITextField are classes inheriting from UIView, so after code
[[SSDynamicLabel alloc] init];

first will be called method -init then -initWithFrame:, so method -setup is called twice.
To fix this, just remove -init override from all UIView inheriting classes.

Good catch @Grubas7!

@jhersh Could you update podfile also on cocoapods.org?
Thanks in advance!

@Grubas7 done and done!


Add .kql as extension for KQL (Kusto Query Language)

Description

Please, this was already considered in https://github.com/github/linguist/pull/4827 . I think currently .kql is widely used over .csl
I would like to open another PR to change the grammar (or I will commit in this PR if you help me please), I would like to add this grammar (although it is outdated)
https://github.com/rosshamish/kuskus/blob/7aa46fd48ebf50c9cbfb98c06782a5e9d86dc535/kusto-syntax-highlighting/syntaxes/kusto.tmLanguage
The only "problem" is that this repo has several extensions apart from syntax highlighting, please, do you know if as long as the engine finds the kusto.tmLanguage file is enough?
Checklist:

[x] I am associating a language with a new file extension.

[x] The new extension is used in hundreds of repositories on GitHub.com

Search results for each extension:

https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3Akql+where+NOT+nothack
https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3Akql+summarize+NOT+nothack
https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3Akql+extend+NOT+nothack
https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3Akql+project+NOT+nothack
https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3Akql+join+NOT+nothack

[x] I have included a real-world usage sample for all extensions added in this PR:

Sample source(s):

Application consent or assignment.kql

Sample license(s):

MIT License

[ ] I have included a change to the heuristics to distinguish my language from others using the same extension.

[ ] I am adding a new language.

[ ] The extension of the new language is used in hundreds of repositories on GitHub.com.

Search results for each extension:

https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3AFOOBAR+KEYWORDS+NOT+nothack

[ ] I have included a real-world usage sample for all extensions added in this PR:

Sample source(s):

[URL to each sample source, if applicable]

Sample license(s):

[ ] I have included a syntax highlighting grammar: [URL to grammar repo]
[ ] I have updated the heuristics to distinguish my language from others using the same extension.

[ ] I am fixing a misclassified language

[ ] I have included a new sample for the misclassified language:

Sample source(s):

[URL to each sample source, if applicable]

Sample license(s):

[ ] I have included a change to the heuristics to distinguish my language from others using the same extension.

[x] I am changing the source of a syntax highlighting grammar

Old: https://github.com/josin/kusto-syntax-highlighting
New: https://github.com/rosshamish/kuskus/tree/master/kusto-syntax-highlighting
New's license: Apache 2.0 License

[ ] I am updating a grammar submodule

[ ] I am adding new or changing current functionality

[ ] I have added or updated the tests for the new or changed functionality.

[ ] I am changing the color associated with a language

[ ] I have obtained agreement from the wider language community on this color change.

[URL to public discussion]
[Optional: URL to official branding guidelines for the language]

I would like to open another PR to change the grammar (or I will commit in this PR if you help me please), I would like to add this grammar (although it is outdated) https://github.com/rosshamish/kuskus/blob/7aa46fd48ebf50c9cbfb98c06782a5e9d86dc535/kusto-syntax-highlighting/syntaxes/kusto.tmLanguage

I'd prefer a new PR so it can easily be reverted independent of other changes if necessary.

The only "problem" is that this repo has several extensions apart from syntax highlighting, please, do you know if as long as the engine finds the kusto.tmLanguage file is enough?

Generally yes, however the location of the grammar files is important as the compiler only looks in certain locations depending on the expected location for the upstream editor that uses the grammars:
https://github.com/github/linguist/blob/249bbd1c2ffc631ca2ec628da26be5800eec3d48/tools/grammars/compiler/loader.go#L119-L128
I've taken a quick look at the repo and you should be ok. It's not go too much other cruft and the grammar should be easily found. Give it a try 😁.

Please, this was already considered in #4827 . I think currently .kql is widely used over .csl

Yup. Usage looks to have grown quite considerably since then. 👍

Thank you @lildude I have opened another PR for the grammar change (https://github.com/github/linguist/pull/5996)


Adding missing parameters. Correcting base-path double dash

What
Adding missing parameters. Correcting base-path double dash

@owans Thanks you !!!


connection failure causes session restart

Running xrdp 0.9.5 + xorgxrdp  both built from source.  Everything has been working fine for >6 months until the last week or so.
Sometimes I get disconnected from my session (typically a connection failure / spotty wifi / spotty vpn), when I reconnect I get a new session instead of resuming my session.  This is causing headaches like losing unsaved files in progress etc.  This seems to happen about 50% of the time.  Not sure if it's related to how long the disconnect is (?)
home folder .xorgxrdp.log:
http://paste.ubuntu.com/p/N4t9sPQVCC/
last 500 lines of /var/log/xrdp-sesman.log:
http://paste.ubuntu.com/p/vcmtNp5CXZ/
Any ideas on lines of investigation? Nothing in the logs jumps out at me.  I tried clearing stale sessions and rebooting the server, no improvement.  It's weird that it worked for a long time with no issues, now there are issues.

xorgxrdp log looks wrong.  This line shows us display number 42 used however as far as I look at xrdp-sesman.log, no one uses display number 42.
[2231419.229] rdpLoadLayout: keylayout 0x00000409 variant  display 42

Only xrdp-sesman.log is useful so far. As far as I look at your log, connection failure doesn't causes session restart but it looks session crash is causing disconnection to me.
Just for the tips in long time use, if you build from source and using for a long time, don't forget rebuild xorgxrdp after Xorg version change.

@metalefty   This is happening for multiple users -- not sure what is causing session crash.   What should I look for?  Suggestions on how to proceed / troubleshoot?
We haven't changed versions of anything built from source.  Are there any library/package dependencies that could have changed, separate from "built from source" stuff?

Anyway, show me the correct log.

@metalefty  Weird, the logfile rotation seems to be out of order...
Latest home folder xorgxrdp logfile
-rw-r--r--  1 johnar johnar     18126 Nov 26 19:15 .xorgxrdp.13.log
johnar@netdev1-westus2:~$ date
Mon Nov 26 19:19:25 UTC 2018
johnar@netdev1-westus2:~$ pastebinit .xorgxrdp.13.log
http://paste.ubuntu.com/p/tZTKJ8Pjtm/

I'm going to close this issue, as the problem went away when I upgraded to 0.9.8.


TypeError on launch in 0.6.1

Hi, GraphQL folks!
I just downloaded GraphiQL for the first time on this computer, and here's what I see on first run:

TypeError: Failed to fetch
    at TypeError (native)
    at e.graphQLFetcher (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:23:29032)
    at t.value (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:26:29129)
    at t.value (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:26:22695)
    at e.notifyAll (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:7:12508)
    at r.close (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:23:4786)
    at r.closeAll (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:7:15916)
    at r.perform (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:7:15405)
    at u (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:23:22899)
    at r.perform (file:///Applications/GraphiQL.app/Contents/Resources/app.asar/dist/bundle.js:7:15322)

$ uname -a
Darwin Zekes-MBP 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64

First time I installed with brew cask install graphiql, then tried re-installed from the DMG GitHub release. Same result.

Dupe of #82, closed by https://github.com/skevy/graphiql-app/pull/93

Thanks for the quick fix, @gjtorikian!

I still get these errors when installing from brew


Update location reference

Update solution so it now points to the relocated pipelines files.

Sorry about that, thanks for checking.


[Bug]:

What happened?
Error
What version of the tool are you using?
Power Platform Toolbox
What version of Visual Studio Code are you running?
1.88.1
Any relevant session id from VSCode?
3c310f78-ac77-4cd4-9afc-95ddd2ade8ed1714555240069
Relevant log output
VSCode version: 1.88.1.
Extension version: 2.2.0.
Error occured in connectDataverse command.
Error code: undefined.
Error message: Cannot read properties of undefined (reading 'label').
Error stack: TypeError: Cannot read properties of undefined (reading 'label')
    at t.DataverseHelper.<anonymous> (c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:719985)
    at Generator.next (<anonymous>)
    at c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:718699
    at new Promise (<anonymous>)
    at o (c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:718444)
    at t.DataverseHelper.connectToDataverse (c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:719916)
    at c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:705576
    at Generator.next (<anonymous>)
    at c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:703425
    at new Promise (<anonymous>)
    at o (c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:703170)
    at callback (c:\Users\pospino\.vscode\extensions\danish-naglekar.dataverse-devtools-2.2.0\dist\extension.js:2:705528)
    at d.h (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:152:186874)
    at d.$executeContributedCommand (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:152:187734)
    at u.S (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:149:5502)
    at u.Q (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:149:5268)
    at u.M (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:149:4361)
    at u.L (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:149:3440)
    at o.value (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:149:2227)
    at s.y (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:82:659)
    at s.fire (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:82:876)
    at r.fire (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:107:14173)
    at o.value (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:175:8018)
    at s.y (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:82:659)
    at s.fire (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:82:876)
    at r.fire (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:107:14173)
    at MessagePortMain.<anonymous> (c:\Users\pospino\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:175:6298)
    at MessagePortMain.emit (node:events:517:28)
    at Object.MessagePortMain._internalPort.emit (node:electron/js2c/utility_init:2:2285).

Need more details here. What did you try? What type of connection it was? Please provide more details.


Date Picker

aligned the date picker to the new spacings...
https://zalando.invisionapp.com/d/main#/console/10454474/223650900/preview

:+1:

Update... changed the fonts of the dates and the background color of the current day.
https://zalando.invisionapp.com/d/main#/console/10454474/227056253/preview


Retirement/ Repurpose or Cleanup  of OSC Cluster 1-

This issue is used to track activities towards the retirement of cluster 1 including migration of pipelines.

For starters, here are the notebooks that reference CL1.  These notebook references should be replaced by tracker issues as they are created:

[ ] aicoe-osc-demo (demo1, demo2, demo3): https://github.com/os-climate/aicoe-osc-demo/issues/169
[ ] ./data-platform-demo/notebooks/trino-iceberg-ingestion.ipynb:1
[ ] ./data-platform-demo/notebooks/itr-ingest-raw-emissions.ipynb:1
[ ] ./data-platform-demo/notebooks/odsc-west-2021-federations.ipynb:1
[ ] ./data-platform-demo/notebooks/Sankey.ipynb:2
[ ] ./data-platform-demo/notebooks/EPA_GHGRP-demos.ipynb:1
[ ] ./data-platform-demo/notebooks/trino-datavault-test.ipynb:3
[ ] ./data-platform-demo/notebooks/trino-iceberg-demo.ipynb:1
[ ] ./data-platform-demo/notebooks/itr-ingest-raw-company-data.ipynb:1
[ ] ./data-platform-demo/notebooks/itr-ingest-raw.ipynb:2
[ ] ./data-platform-demo/notebooks/deprecated/trino-access-control-demo.ipynb:2
[ ] ./data-platform-demo/notebooks/deprecated/test-trino-access.ipynb:1
[ ] ./data-platform-demo/notebooks/deprecated/trino-api-demo.ipynb:1
[ ] ./data-platform-demo/notebooks/deprecated/trino-sqlalchemy-demo.ipynb:1
[ ] ./data-platform-demo/notebooks/deprecated/trino-hive-iceberg-ingest.ipynb:1
[x] (This will list all available clusters, including CL1 until it is retired ./apps/docs/content/notebooks/onboarding_project.ipynb:1)
[x] (Of course a CL1 cluster configuration file will reference CL1: ./apps/kfdefs/overlays/osc/osc-cl1/superset/configmaps/superset_config.py:3)

I have created a branch iceberg in data-platform-demo that migrates various demos from CL1 to CL2.  Other Pull Requests welcome.
If you are aware of other migration issues or trackers, please feel free to edit above.

I suggest we delete/archive data-platform-demo/notebooks/itr-* as (1) they are very old, demonstrating usage of a very early version of osc-ingest-tools functionality, (2) the sample data is a public slice of a proprietary dataset that is not presently available in any meaningful way in the Data Commons.  @erikerlandson are you OK with that?

While majority of the active development should happen on osc-cl2. There is argument to be made that we may want to keep cluster one for new platform/tool testing in scale-down format. As part of it once all pipelines will be migrated we may want to upgrade and sync it with cluster 2

Migrate the relevant or needed workloads to cluster 2

Eric to fix ingest script to remove the 4 day latency .  OSC Ingest tools is the .
This is the main blocker to prevent us moving to Cluster 2

ingest script fixed.  hive catalog is up.  workflow runs; on-boarding new workflows.
ITR ones left to be moved and AICOE
Mikhail - Openshift Upgrade 4.8 to 4.9, synchronize pipelines and tools that are different from CL2, identify namespaces for Airbus, downsize cluster

ingest script fixed.  hive catalog is up.  workflow runs; on-boarding new workflows.
ITR ones left to be moved and AICOE
Mikhail - Openshift Upgrade 4.8 to 4.9, synchronize pipelines and tools that are different from CL2, identify namespaces for Airbus, downsize cluster

ITR will be removed by Michael T - as everything is moved to Cluster 2 that is needed. @MichaelTiemannOSC

Lets wait for Heather to confirm . Seem like work is being done.

WITNESS and SoS Trades are now fully operational on CL2, so no longer require CL1 resources (per tech check-in meeting today).

@redmikhail will create a new set of tasks/issues to perform specific work: (1) review what is left (2) sync CL1 with CL2 with version/config (3) reduce CL1 nodes

@redmikhail still pending - to be done by 30-sep

@redmikhail still pending - to be done by 30-sep

@redmikhail to complete by 10-Oct

Further steps to re-purpose  cluster cl1 :

[ ] Upgrade Openshift version from 4.8 to 4.9 to match with cluster 1

[ ] Check/remove versions of CRD/API that are not compatible with Kubernetes 1.22 (https://access.redhat.com/articles/6955985)
[ ] Upgrade cluster to version 4.9

[ ] Synchronize management of cluster resources with cluster 2

[ ] Add machinesets to github with proper labeling
[ ] Add taints to GPU enabled nodes
[ ] Add taints to non-GPU jupyterhub nodes

[ ] Synchronize setup and versions of OS Climate components: ODH, Trino , Kubeflow, Openmetadata
[ ] Review capacity of components above and downsize if possible (eg. do we need 3 workers for trino if there is no workload)
[ ] Document design of the environments (cluster 1 and cluster 2)

on hold until 21-nov (post COP27); @redmikhail will reach out to risk mgmt team

the upgrade to open shift to 4.9 is blocked by namespace postgres (physical risk) operator issue
reduced GPU enabled nodes to 1; namespaces can be scaled down as well
@redmikhail to let @HeatherAck know the current AWS spend

@redmikhail to send AWS spend today

AWS spend has been stable. 5 remaining accounts to be removed.  @MichaelTiemann to send @ryanaslett credentials for 2 accounts.  @ryanaslett to install kubecost to understand cost per workload (Heather to create a Jira)

@redmikhail to upgrade to new version of Open Shift

@HeatherAck it has been completed , I have marked tasks in the list above

YOU ARE AWESOME!  Thank you!  Thank you!


How to add labels on this Pie

Hi coolbutuseless,
ggthread is a great package, but how could I add labels on this pie or near pie? Like this:

Thanks a lot.
Bests,
Shisheng

I would also like it a lot if this feature can be implemented!


[FLOC 977] add elk download file

Fixes 977
An old issue which adds a downloadable elk file where the text suggests one should be. A few tweaks to the wording around this.
I've not manually tested this, as it was an obvious error (the contents of the elk file was always provided - this just adds a link for users to download the file)

I haven't actually tested that the code works, but it appears to be more likely to work than what was there initially.  There don't appear to be any acceptance tests for this code.  Please add an issue to create one.
OK to merge this.

thanks @jongiddy


allow users to limit output of models by package

Describe the feature
If i am a user of a package that contains models, i may want to exclude those models from DPE checks. Right now, doing so would require use of the exceptions seed, with entries for each combination of model-to-exlude + fct_model_name to make sure not to run any check on any external model (or use of wildcards when that PR is closed)
it would be easier to just declare what set of packages to consider, and have the logic in the models take care of the rest!
Describe alternatives you've considered
Seed only
Additional context
related thread
Who will this benefit?

package users
multi-project deployments in the future

Are you interested in contributing this feature?
of course i am, me

@b-per Rumor has it you've got a solution in mind for this?

I added comments to your PR, but I'd be keen tackling this and that with a common approach

I think we can close this as done - @dave-connors-3 can you confirm?

Yes, this is done!


Bug? block_diff

I had to make the change indicated to correctly show how far behind my bitcoind was when syncing the mainchain on my Raspibolt.
(I was doing this just for testing)
At the point where I had downloaded and verified only 7% of the blockchain, the output of the 3 commands in question were:

getblockcount = 260724
getblockchaininfo -> headers = 518291
getblockchaininfo -> blocks = 260580

Here are the full outputs ( as few seconds apart)
admin ~  ฿  bitcoin-cli -conf=/home/bitcoin/.bitcoin/bitcoin.conf getblockcount
260724
admin ~  ฿  bitcoin-cli -conf=/home/bitcoin/.bitcoin/bitcoin.conf getblockchaininfo
{
"chain": "main",
"blocks": 260580,
"headers": 518291,
....
"verificationprogress": 0.0768998938604351,
...
}

BTW - as a write this, the Bitcoin blockchain height = 518291


Making coordinate variables searchable

Is your feature request related to a problem? Please describe.
Currently, the ACCESS-NRI Intake Catalog doesn't allow for searching of coordinate variables: for example, searching for st_edges_ocean will return 0 datasets. This can make searching for coordinate variables difficult, with 2 main pain points:

If the coordinate variable is know to be stored in a netCDF file with specific data variables:

The user needs to know that these data & coordinate variables are found in the same files, and then search for the data variable in order to access the coordinate variables.
In some instances, the user can directly access the coordinate variables by searching the data variables. In others, they need to perform something like

        VARNAME = ...
        COORD = ...
        fname = cat.search(variable=VARNAME).df.loc[0,'file']
        ds = xr.open_dataset(fname)
        coord = ds[COORD]

Although this doesn't require the user to (semi) manually work out what file to open, it's still messy as it requires passing round file names.

In other instances, coordinate variables are stored completely separately. For example, ocean_grid.nc files only contain coordinate variables, and so cannot be found using the catalogue. The only way to currently access these files is to search the catalogue to get a handle on the directory structure - and then construct a file path and load it: eg:

        fname = cat.search(variable=VARNAME).df.loc[0,'file']
        dirname = Path(fname).parent
        grid_file, = [file for file in os.listdir(dirname) if file.endswith('_grid.nc')]
        ds = xr.open_dataset(grid_file)
        coord = ds[COORD]

This requires the user to start poking round in directory structures to try to work out where to load their data - which is the problem intake is trying to solve.
This has caused some pain points migrating COSIMA recipes from cosima_cookbook => intake.
I also think this might be the same issue as discussed in #63? @aidanheerdegen - seem to be some concerns about coordinates being listed as variables when they shouldn't be there?
Describe the feature you'd like
Searchable coordinates: in the same way that the catalog currently lets you perform searches over variables, it would be useful to be able to do the same on coordinates:
var = cat.search(variable=VARN).to_dask()
coord = cat.search(coord=COORD).to_dask()

Doing this is subject to a couple of constraints:

The catalog needs to know that coordinates & data variables aren't the same & need to be treated differently - xr.combine_by_coords will fail if passed a coordinate variable.
This requires an upstream patch of intake-esm - see issue 660.
Cannot cause serious performance regressions - see above issue.

Proposed Solution

Update intake-esm with @dougiesquire's proposed solution from issue 660.
Add separate coordinate coordinate variable fields to the ACCESS-NRI Intake Catalog, rather than just making the same change as in Intake-ESM (data_vars => variables), as this would then confuse coordinates & variables in the ACCESS-NRI Intake Catalog as well as causing concatenation issues. This is implemented on branch 660-coordinate-variables.

Additional Info

Due to the release cycle of Intake-ESM, this solution will probably require us to maintain a fork - at least for some time.
I've performance tested the proposed solution & changes in catalogue build times are small (typically ~5%), catalogue read times similar (typically 5-10%, sometimes faster), and the size of datastore.csv.gz files writted by builder.save() are typically approximately doubled.

I also think this might be the same issue as discussed in https://github.com/ACCESS-NRI/access-nri-intake-catalog/issues/63? @aidanheerdegen - seem to be some concerns about coordinates being listed as variables when they shouldn't be there?

That was specifically for a project where we were using the intake catalogue as a source for an "experiment explorer", to expose the variables saved in an experiment in timeline to assist users in understanding what variables are available at different times in an experiment. For this purpose we really only wanted diagnostic model variables that have a time-varying component.

Add separate coordinate variable fields to the ACCESS-NRI Intake Catalog

I'm confused. Does this mean

Have a "coordinate" flag (field)?
Move coordinates into a separate catalogue?
or neither?

BTW this is a somewhat related issue I think about encoding grid information:
https://github.com/ACCESS-NRI/access-nri-intake-catalog/issues/112

Yup, a coordinate flag, sorry for the confusion with terms.
As currently implemented, this would just add coordinates & coordinate relevant information to the catalogue -  I've tried to keep changes to a minimum. We could think about having a separate catalogue specifically for coordinates but that feels potentially a bit harder for users to me?
I've attached a screencap with a demo of how the change affects catalogue results (N.B both are run using the same kernel, so cat_without_coords.search(coords='st_edges_ocean') returns an empty search - running it on main would raise an error).

Also would be interested in knowing if this would cover your use cases @marc-white @anton-seaice

@charles-turner-1 my main concern to this point hasn't been how to search for the coordinates, but how to access them. In particular:

Currently, when you search for a variable and then do to_dask, you get back a Dask array of the requested variables, plus the coordinates that the variable relates to. Would this work in reverse? So, if I search for a coordinate, and then converted to an array, would the Dask array include the coordinate, and then the related variables, and then the other coordinates that those variables rely on?
There's also the issue of intake-esm crashing when it tries to pull out the coordinate data by stacking it on itself (in effect). How could we modify things to know to treat variables and coordinates differently in this regard?

@marc-white In response to question 1, currently searching for a coordinate will load the entire dataset - it works very much like searching other metadata, eg.
>>> ocean_files = cat_with_coords.search(start_date='2086.*',frequency='1mon',file_id='ocean',).to_dask()
>>>print(ocean_files)
<xarray.Dataset> Size: 847GB
Dimensions:                (time: 12, st_ocean: 75, yt_ocean: 2700,
                            xt_ocean: 3600, yu_ocean: 2700, xu_ocean: 3600,
                            sw_ocean: 75, potrho: 80, grid_yt_ocean: 2700,
                            grid_xu_ocean: 3600, grid_yu_ocean: 2700,
                            grid_xt_ocean: 3600, neutral: 80, nv: 2,
                            st_edges_ocean: 76, sw_edges_ocean: 76,
                            potrho_edges: 81, neutralrho_edges: 81)
Coordinates: (12/18)
  * xt_ocean               (xt_ocean) float64 29kB -279.9 -279.8 ... 79.85 79.95
  * yt_ocean               (yt_ocean) float64 22kB -81.11 -81.07 ... 89.94 89.98
  * st_ocean               (st_ocean) float64 600B 0.5413 1.681 ... 5.709e+03
  * st_edges_ocean         (st_edges_ocean) float64 608B 0.0 1.083 ... 5.809e+03
  * time                   (time) object 96B 2086-01-16 12:00:00 ... 2086-12-...
  * nv                     (nv) float64 16B 1.0 2.0
    ...                     ...
  * potrho                 (potrho) float64 640B 1.028e+03 ... 1.038e+03
  * potrho_edges           (potrho_edges) float64 648B 1.028e+03 ... 1.038e+03
  * grid_xt_ocean          (grid_xt_ocean) float64 29kB -279.9 -279.8 ... 79.95
  * grid_yu_ocean          (grid_yu_ocean) float64 22kB -81.09 -81.05 ... 90.0
  * neutral                (neutral) float64 640B 1.028e+03 ... 1.038e+03
  * neutralrho_edges       (neutralrho_edges) float64 648B 1.028e+03 ... 1.03...
Data variables: (12/28)
    temp                   (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    pot_temp               (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    salt                   (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    age_global             (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    u                      (time, st_ocean, yu_ocean, xu_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    v                      (time, st_ocean, yu_ocean, xu_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    ...                     ...
    bih_fric_v             (time, st_ocean, yu_ocean, xu_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    u_dot_grad_vert_pv     (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    average_T1             (time) datetime64[ns] 96B dask.array<chunksize=(3,), meta=np.ndarray>
    average_T2             (time) datetime64[ns] 96B dask.array<chunksize=(3,), meta=np.ndarray>
    average_DT             (time) timedelta64[ns] 96B dask.array<chunksize=(3,), meta=np.ndarray>
    time_bounds            (time, nv) timedelta64[ns] 192B dask.array<chunksize=(1, 2), meta=np.ndarray>
Attributes: (12/22)
    filename:                                 ocean.nc
    title:                                    ACCESS-OM2-01
    grid_type:                                mosaic
    grid_tile:                                1
    intake_esm_vars:                          ['temp', 'pot_temp', 'salt', 'a...
    intake_esm_attrs:filename:                ocean.nc
    ...                                       ...
    intake_esm_attrs:coord_calendar_types:    ['', '', '', '', 'NOLEAP', '', ...
    intake_esm_attrs:coord_bounds:            ['', '', '', '', 'time_bounds',...
    intake_esm_attrs:coord_units:             ['degrees_E', 'degrees_N', 'met...
    intake_esm_attrs:realm:                   ocean
    intake_esm_attrs:_data_format_:           netcdf
    intake_esm_dataset_key:                   ocean.1mon

I like the suggestion that we only load 1. variables that depend on the coordinate variable searched for, and 2. other coordinate variables those data variables depend on, though. I'll have a look into how straightforward that might be to implement.

With regards to the second, searching for a coordinate & obtaining a multi-file dataset works fine right now, so I think this approach has sidestepped the crashing issue:

>>> st_edges_search = cat_with_coords.search(coords='st_edges_ocean',frequency='1mon',file_id='ocean',start_date = '2086.*')
>>> print(st_edges_search.df.head(3))
filename file_id                                               path  \
0  ocean.nc   ocean  /g/data/ik11/outputs/access-om2-01/01deg_jra55...
1  ocean.nc   ocean  /g/data/ik11/outputs/access-om2-01/01deg_jra55...
2  ocean.nc   ocean  /g/data/ik11/outputs/access-om2-01/01deg_jra55...

  filename_timestamp frequency            start_date              end_date  \
0                NaN      1mon  2086-01-01, 00:00:00  2086-04-01, 00:00:00
1                NaN      1mon  2086-04-01, 00:00:00  2086-07-01, 00:00:00
2                NaN      1mon  2086-07-01, 00:00:00  2086-10-01, 00:00:00

>>> st_edges = st_edges_search.to_dask()
>>> print(st_edges)
<xarray.Dataset> Size: 847GB
Dimensions:                (time: 12, st_ocean: 75, yt_ocean: 2700,
                            xt_ocean: 3600, yu_ocean: 2700, xu_ocean: 3600,
                            sw_ocean: 75, potrho: 80, grid_yt_ocean: 2700,
                            grid_xu_ocean: 3600, grid_yu_ocean: 2700,
                            grid_xt_ocean: 3600, neutral: 80, nv: 2,
                            st_edges_ocean: 76, sw_edges_ocean: 76,
                            potrho_edges: 81, neutralrho_edges: 81)
Coordinates: (12/18)
  * xt_ocean               (xt_ocean) float64 29kB -279.9 -279.8 ... 79.85 79.95
  * yt_ocean               (yt_ocean) float64 22kB -81.11 -81.07 ... 89.94 89.98
  * st_ocean               (st_ocean) float64 600B 0.5413 1.681 ... 5.709e+03
  * st_edges_ocean         (st_edges_ocean) float64 608B 0.0 1.083 ... 5.809e+03
  * time                   (time) object 96B 2086-01-16 12:00:00 ... 2086-12-...
  * nv                     (nv) float64 16B 1.0 2.0
    ...                     ...
  * potrho                 (potrho) float64 640B 1.028e+03 ... 1.038e+03
  * potrho_edges           (potrho_edges) float64 648B 1.028e+03 ... 1.038e+03
  * grid_xt_ocean          (grid_xt_ocean) float64 29kB -279.9 -279.8 ... 79.95
  * grid_yu_ocean          (grid_yu_ocean) float64 22kB -81.09 -81.05 ... 90.0
  * neutral                (neutral) float64 640B 1.028e+03 ... 1.038e+03
  * neutralrho_edges       (neutralrho_edges) float64 648B 1.028e+03 ... 1.03...
Data variables: (12/28)
    temp                   (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    pot_temp               (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    salt                   (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    age_global             (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    u                      (time, st_ocean, yu_ocean, xu_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    v                      (time, st_ocean, yu_ocean, xu_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    ...                     ...
    bih_fric_v             (time, st_ocean, yu_ocean, xu_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    u_dot_grad_vert_pv     (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
    average_T1             (time) datetime64[ns] 96B dask.array<chunksize=(3,), meta=np.ndarray>
    average_T2             (time) datetime64[ns] 96B dask.array<chunksize=(3,), meta=np.ndarray>
    average_DT             (time) timedelta64[ns] 96B dask.array<chunksize=(3,), meta=np.ndarray>
    time_bounds            (time, nv) timedelta64[ns] 192B dask.array<chunksize=(1, 2), meta=np.ndarray>
Attributes: (12/22)
    filename:                                 ocean.nc
    title:                                    ACCESS-OM2-01
    grid_type:                                mosaic
    grid_tile:                                1
    intake_esm_vars:                          ['temp', 'pot_temp', 'salt', 'a...
    intake_esm_attrs:filename:                ocean.nc
    ...                                       ...
    intake_esm_attrs:coord_calendar_types:    ['', '', '', '', 'NOLEAP', '', ...
    intake_esm_attrs:coord_bounds:            ['', '', '', '', 'time_bounds',...
    intake_esm_attrs:coord_units:             ['degrees_E', 'degrees_N', 'met...
    intake_esm_attrs:realm:                   ocean
    intake_esm_attrs:_data_format_:           netcdf
    intake_esm_dataset_key:                   ocean.1mon

As before, searching for a variable will only load relevant data, eg:
>>> pot_temp_search = cat_with_coords.search(variable='pot_temp',frequency='1mon',file_id='ocean',start_date = '2086.*')
>>> print(pot_temp_search.df.head(3))
 filename file_id                                               path  \
0  ocean.nc   ocean  /g/data/ik11/outputs/access-om2-01/01deg_jra55...
1  ocean.nc   ocean  /g/data/ik11/outputs/access-om2-01/01deg_jra55...
2  ocean.nc   ocean  /g/data/ik11/outputs/access-om2-01/01deg_jra55...

  filename_timestamp frequency            start_date              end_date  \
0                NaN      1mon  2086-01-01, 00:00:00  2086-04-01, 00:00:00
1                NaN      1mon  2086-04-01, 00:00:00  2086-07-01, 00:00:00
2                NaN      1mon  2086-07-01, 00:00:00  2086-10-01, 00:00:00
...
>>>  pot_temp = pot_temp_search.to_dask()
>>> print(pot_temp)
<xarray.Dataset> Size: 35GB
Dimensions:   (time: 12, st_ocean: 75, yt_ocean: 2700, xt_ocean: 3600)
Coordinates:
  * xt_ocean  (xt_ocean) float64 29kB -279.9 -279.8 -279.7 ... 79.75 79.85 79.95
  * yt_ocean  (yt_ocean) float64 22kB -81.11 -81.07 -81.02 ... 89.89 89.94 89.98
  * st_ocean  (st_ocean) float64 600B 0.5413 1.681 2.94 ... 5.511e+03 5.709e+03
  * time      (time) object 96B 2086-01-16 12:00:00 ... 2086-12-16 12:00:00
Data variables:
    pot_temp  (time, st_ocean, yt_ocean, xt_ocean) float32 35GB dask.array<chunksize=(1, 7, 300, 400), meta=np.ndarray>
Attributes: (12/22)
    filename:                                 ocean.nc
    title:                                    ACCESS-OM2-01
    grid_type:                                mosaic
    grid_tile:                                1
    intake_esm_vars:                          ['pot_temp']
    intake_esm_attrs:filename:                ocean.nc
    ...                                       ...
    intake_esm_attrs:coord_calendar_types:    ['', '', '', '', 'NOLEAP', '', ...
    intake_esm_attrs:coord_bounds:            ['', '', '', '', 'time_bounds',...
    intake_esm_attrs:coord_units:             ['degrees_E', 'degrees_N', 'met...
    intake_esm_attrs:realm:                   ocean
    intake_esm_attrs:_data_format_:           netcdf
    intake_esm_dataset_key:                   ocean.1mon
    ```

Instead of this:

st_edges_search = cat_with_coords.search(coords='st_edges_ocean',frequency='1mon',file_id='ocean',start_date = '2086.*')
st_edges = st_edges_search.to_dask()

We should just be able to do:
cat_with_coords.search(coords='st_edges_ocean').to_dask() (possibly the file_id part is still needed)
'st_edges_ocean' doesn't change over time, so we should check that all files have the same values, but having to supply the start_date = '2086.*' is messy. And then calling to_dask, ideally would only return the coordinates (without the data_vars). e.g.

or

I've looked into it, & I think adding data_vars that depend on the coordinate vars we've searched is going to

Probably require modifications to ecgtools.
Potentially cause memory usage blowups. Subsetting down to just a few files by providing a year etc, eg.

st_edges_search = cat_with_coords.search(coords='st_edges_ocean',frequency='1mon',file_id='ocean',start_date = '2086.*')
st_edges = st_edges_search.to_dask()

is messy, I agree - in the search above a large ARE instance will crash with default chunking - seems to be the result of Dask trying to set one chunk per file. Subsetting down to just a few files (start_date = '2086.*') resolves this.
I'd rather not change default chunking when we perform coordinate searches - it seems messy and error prone - depending on the coordinate we might load most of the data variables in the file.
With the current implementation, we can easily search & load the dataset as follows:
>> st_edges_search = cat_with_coords.search(coords='st_edges_ocean',frequency='1mon',file_id='ocean')
>>> varnames = st_edges_search.df.loc[0,'variable']
>>> print(st_edges_search.to_dask(xarray_open_kwargs = {'drop_variables' : varnames}))<xarray.Dataset> Size: 229kB
Dimensions:           (xt_ocean: 3600, yt_ocean: 2700, st_ocean: 75,
                       st_edges_ocean: 76, time: 2760, nv: 2, xu_ocean: 3600,
                       yu_ocean: 2700, sw_ocean: 75, sw_edges_ocean: 76,
                       grid_xu_ocean: 3600, grid_yt_ocean: 2700, potrho: 80,
                       potrho_edges: 81, grid_xt_ocean: 3600,
                       grid_yu_ocean: 2700, neutral: 80, neutralrho_edges: 81)
Coordinates: (12/18)
  * xt_ocean          (xt_ocean) float64 29kB -279.9 -279.8 ... 79.85 79.95
  * yt_ocean          (yt_ocean) float64 22kB -81.11 -81.07 ... 89.94 89.98
  * st_ocean          (st_ocean) float64 600B 0.5413 1.681 ... 5.709e+03
  * st_edges_ocean    (st_edges_ocean) float64 608B 0.0 1.083 ... 5.809e+03
  * time              (time) object 22kB 1950-01-16 12:00:00 ... 2179-12-16 1...
  * nv                (nv) float64 16B 1.0 2.0
    ...                ...
  * potrho            (potrho) float64 640B 1.028e+03 1.028e+03 ... 1.038e+03
  * potrho_edges      (potrho_edges) float64 648B 1.028e+03 ... 1.038e+03
  * grid_xt_ocean     (grid_xt_ocean) float64 29kB -279.9 -279.8 ... 79.85 79.95
  * grid_yu_ocean     (grid_yu_ocean) float64 22kB -81.09 -81.05 ... 89.96 90.0
  * neutral           (neutral) float64 640B 1.028e+03 1.028e+03 ... 1.038e+03
  * neutralrho_edges  (neutralrho_edges) float64 648B 1.028e+03 ... 1.038e+03
Data variables:
    *empty*
Attributes: (12/16)
    filename:                               ocean.nc
    title:                                  ACCESS-OM2-01
    grid_type:                              mosaic
    grid_tile:                              1
    intake_esm_attrs:filename:              ocean.nc
    intake_esm_attrs:file_id:               ocean
    ...                                     ...
    intake_esm_attrs:coord_calendar_types:  ['', '', '', '', 'NOLEAP', '', ''...
    intake_esm_attrs:coord_bounds:          ['', '', '', '', 'time_bounds', '...
    intake_esm_attrs:coord_units:           ['degrees_E', 'degrees_N', 'meter...
    intake_esm_attrs:realm:                 ocean
    intake_esm_attrs:_data_format_:         netcdf
    intake_esm_dataset_key:                 ocean.1mon
    ```
I think the most straightforward solution here is going to be to get searches on coordinates to drop all data variables by default.

I think the most straightforward solution here is going to be to get searches on coordinates to drop all data variables by default.

This makes sense to me, generally you are searching for a coordinate when trying to load them from a different file than where the variable is anyway, so you'll need a different search & .to_dask operation. e.g., you might want SST and area_t
and this is where I'm not sure they should have their own field:
e.g.
cat.search(variable='SST', coords='area_t')
would return 0 datasets ?
And we would have to do this
cat.search(variable='SST')
cat.search(coords='area_t')
to return two datasets then then need to be combined ?
but a different implementation would allow cat.search(variable=['area_t','SST]) to return the two datasets we want?

I spent a good chunk of time thinking about this yesterday afternoon. I've got a couple of thoughts:

I agree it's counterintuitive that cat.search(variable='SST', coords='area_t') returns nothing.

In a search like this:

cat.search(variable='SST')
cat.search(coords='area_t')
to return two datasets then then need to be combined ?
but a different implementation would allow cat.search(variable=['area_t','SST]) to return the two datasets we want?

we're going to obtain two datasets from the search anyway, right? And so downstream of the search we're going necessarily have something like:
sst, area = datasets.get('SST'), datasets.get('area_t')

which the user will have to merge anyway. So perhaps the file structure enforces a bit of unavoidable clunkiness in these situations?

The major difficulty (that I'm thinking about right now) is that adding coordinate variables to the variables we request leads to intake-esm telling xarray to request that variable when it loads the dataset. I'm looking into whether we can add a step to check whether variables are data or coordinate variables before intake-esm actually opens and begins to concatenate the datasets.
If not, I'm not sure that there's a straightforward solution - bundling coordinate variables in with data variables effectively loses the information that the two are different and need to be treated differently when we open files until we actually open the files.
TLDR; I'm gonna keep prodding - I agree that it would nice to be able to search for coordinates or data variables, rather than coordinates and.

we're going to obtain two datasets from the search anyway, right? And so downstream of the search we're going necessarily have something like:

Ah yes - I think my point was just ... do we need an extra field, can we just put the coordinates in the variable field?

I'm looking into whether we can add a step to check whether variables are data or coordinate variables before intake-esm actually opens and begins to concatenate the datasets.

I think we need this for #117 anyway right ?

I see your point - I think if we can put the coordinates into the variable field then that would be preferable - it doesn't add any user complexity.
And yeah, look like #117 is the same issue. In fact, it looks like the cosima_cookbook n=1 argument @dougiesquire mentions there is the workaround that the cosima_cookbook used - which makes me suspect this problem is hiding in a few other places too.