Add option to specify "filePath" for dts generateTypes

Clear and concise description of the problem
In order to address the generated types of a certain manifest.json file behind a proxy, it is necessary to parse and overwrite the manifest.json fields metaData.types.zip and metaData.types.api because the default paths of the generated types are not reachable for our use-cases.
Currently, we use a custom plugin that is overwriting the variables afterBuild to have a valid pointer when calling the manifest.json which is very inconvenient.
Suggested solution
The request is to provide more control over the file generation by adding a filePath field in the generatedTypes object.
...
new ModuleFederationPlugin({
  ...moduleFederationConfig,
  manifest: {
    filePath: 'my/custom/path'
  },
  dts: {
    generatedTypes: {
      typesFolder: 'mf-custom-types',
      // new requested field to give more control over the behavior of module federation file orchestration
      filePath; 'my/custom/path/'
    }
  }
})
...

Alternative
No response
Additional context
No response
Validations

[X] Read the Contributing Guidelines.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Do you have an idea in the source code about where you would implement this update?
Is this something you want to send a PR for or would like us to look into?

@YanPes is this not something that publicPath: auto resolves?

@ScriptedAlchemy I just found the publicPath === 'auto' in DTSManager.ts. We will update the MF version and test the new feature. Thanks for the hint - I will give an update soon!

Yeah we recently added it since users were not thrilled about needing to know the url upfront at build haha.

To load the remotes via traefik proxy we need Modern.js to serve the files inside the static folder.
Our custom plugin moves the .zip and d.ts files from the dist root to the assets folder so it can be accessed by other applications. At the same time the content of the manifest.json is adapted to point to the correct folder of the .zip and d.ts's.
This then allowed us to load the zip from another container.
function afterFederation(module: string, publicPath: string) {
  const assetFolder = `modules/${module}/static/js/`;
  const typesFolder = `mf-${module}-types`;
  const manifestFile = path.join(
    './',
    '/dist/' + assetFolder + 'mf-manifest.json',
  );
  updateManifestJson(manifestFile, assetFolder, typesFolder, publicPath);
  moveTypes(assetFolder, typesFolder, '.zip');
  moveTypes(assetFolder, typesFolder, '.d.ts');
}

It would be great to have a native integration solving this issue. As requested above the easiest solution would be to have a manual override of the dist path of the generatedTypes defined inside the DTSmanager.ts.
generatedTypes: {
      typesFolder: 'mf-custom-types',
      // new requested field to give more control over the behavior of module federation file orchestration
      filePath; 'my/custom/path/'
    }

@YanPes merged

@ScriptedAlchemy Tested on our side - Works as expected! Thanks to all contributors for your fast action!


Adapt SendGrid integration to use the updated libraries.

Description

SendGrid changed their API about three years ago to use API tokens instead of username/password combos. The sendgrid library adapted, but Luigi's code that depends on SendGrid did not.
Motivation and Context

This re-instates SendGrid integration for sending out emails when Luigi Tasks fail. It solves #2714 and #1956 (since they are duplicate issues).
Have you tested this? If so, how?

I ran a simple Task that force-failed by calling sys.exit(), together with a config file using an apikey value in [sendgrid] and a method=sendgrid on [email]. I received the "Task failed" emails, even with multiple email addresses set up.

Please take a look at failed tests. Looks like you'll need to fix the flake8 and update the test cfg for apikey

jeez, that took way too long for me to understand what was going on.
tests should all pass now. sorry for the back-and-forth - i'm not super familiar with travis/tox and unit testing in general. does this look good or should any more changes be done?


Added ProgressEvent support for HTML5 target

This change adds ProgressEvent functionality to the Loader/LoaderInfo classes in HTML5.
The underlying functionality is slightly different, so to opt-out of the new functionality (if for example, it's not behaving correctly in some browser), remove the ProgressEvent listener, which doesn't work in older versions of openfl anyway.
It depends on this Lime commit:
https://github.com/ImaginationSydney/lime/commit/a6bf5edf959d14672782aede9dbc0629030fac66
Which has a pull request to Lime:
https://github.com/openfl/lime/pull/769

We added asynchronous load calls under the hood, and need to wire them up to openfl.display.Loader
I'm going to close this PR, but we'll track the enhancement here:
https://github.com/openfl/openfl/issues/1435
Thank you :smile:


Encryption / Decryption

Love the browser.  Downright user friendly, intuitive and all the rest.
My clients encrypt their data prior to it being sent to backblaze b2.
Using the Rclone Browser I notice I can see all files in their native unencrypted state which is wonderful.
I am not an expert at the encryption / decryption process and frankly I do not see any other way this could be done but I must ask the question to be certain.

When viewing the data from the cloud service (backblaze b2 in my case) it appears in its 'decrypted' form.  Is this data being downloaded as 'encrypted data' and then decrypted within the Rclone browser as part of its wonderfulness or is this data in some way being downloaded in an unencrypted format ?

Could you please verify this process.
Thanks.
Jay

RcloneBrowser does not encrypt or decrypt anything on its own. It uses rclone command-line tool to do all the work. So your questions are pretty much about how rclone works.
When you are requesting to get data from encrypted remote, the rclone first downloads encrypted data from remote to local machine, decrypts it and then saves in unencrypted format.
If you are asking about filenames - then this is the same - rclone first gets encrypted file names from remote, decrypts them locally, and shows then to you unencrypted.
If you want to know how rclone works in more details, please consult the rcloen documentation at https://rclone.org/docs/ and visit rclone forum: https://forum.rclone.org/


Proposal: Convert this repo to a Go module

Please add a 👍 reaction if you support this proposal. If you do not, please leave a comment describing why.
Now that Go modules have solidified and are on by default in Go 1.16 we are considering how we migrate this repo to officially support them (this is for our track 1 content; track 2 already supports modules).
In keeping with our current release strategy and version, the natural option would be to place a go.mod file in the root of the repo.  However, given that we've already had dozens of major releases (v52.3.0 at the time of this writing), it would mean that the next major version would introduce semantic import versioning.
In go.mod:
module github.com/Azure/azure-sdk-for-go/v53

This would require consumers to update their import statements starting with v53 and beyond.
import github.com/Azure/azure-sdk-for-go/v53/services/foo/bar

While this "plays by the rules" it introduces a burden on consumers and we'd like feedback on the impact.

well at some point go may force the change, so not easy on anybody. i for 1 have an issue azure-sdk-for-go is required by terratest but where i work we use artifactory and as there is no go.mod it makes downloading hard as artifactory assumes .mod.

I tend to agree.  It's a bitter pill but is likely inevitable.  In addition, I recall there was talk about tooling to simplify updating a major module version.
CC @tombuildsstuff

Regarding upgrade automation, I am aware of at least https://github.com/golang/go/issues/32816 and https://github.com/golang/go/issues/32014, as well as a couple of existing tools (https://github.com/icholy/gomajor and https://github.com/marwan-at-work/mod, but others may exist too).

I'm also planning to address module aliasing and renaming (golang/go#26904) for Go 1.18 (due out EoY this year), and it's possible that that will include a way for an existing module to leave a “fowarding address”.
(More design work is needed, but migrating from one module path to another — including from an unversioned path to a versioned one — is an important use-case that we're actively thinking about.)

Given that /services has moved to maintenance mode and is superseded by our new, module-based SDKs, I don't think it's worth making this change to the legacy SDK.


Update docs for pipelines

Description
This updates the docs for the pipelines given the transition to miniwdl/swipe.
Test plan
previewed the doc on github

I'll leave it for @phoenixAja to update the docs for the pangolin job.


USHIFT-632: Update MicroShift description in RPM

$ rpm -qi _output/rpmbuild/RPMS/x86_64/microshift-4.12.0_20221206093532_239202f3_dirty-1.el8.x86_64.rpm
Name        : microshift
Version     : 4.12.0_20221206093532_239202f3_dirty
Release     : 1.el8
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 117975457
License     : ASL 2.0
Signature   : (none)
Source RPM  : microshift-4.12.0_20221206093532_239202f3_dirty-1.el8.src.rpm
Build Date  : Tue 06 Dec 2022 02:58:18 PM UTC
Build Host  : microshift-dev.local
Relocations : (not relocatable)
URL         : https://github.com/openshift/microshift
Summary     : MicroShift binary
Description :
MicroShift is a project that optimizes OpenShift Kubernetes for small
form factor and edge computing.

Edge devices deployed out in the field pose very different operational,
environmental, and business challenges from those of cloud computing.
These motivate different engineering trade-offs for Kubernetes at the
far edge than for cloud or near-edge scenarios.

MicroShift design goals cater to this:
* make frugal use of system resources (CPU, memory, network, storage, etc.)
* tolerate severe networking constraints
* update securely, safely, speedily, and seamlessly (without disrupting workloads)
* build on and integrate cleanly with edge-optimized operating systems like RHEL for Edge
* provide a consistent development and management experience with standard OpenShift

These properties should also make MicroShift a great tool for other use cases
such as Kubernetes applications development on resource-constrained systems,
scale testing, and provisioning of lightweight Kubernetes control planes.

Closes USHIFT-632

/assign @fzdarsky

The update to the README reads really well, thank you!
For the package description, I feel it is not appropriate to simply copy the text from the README, because the README's goal is to answer "What is MicroShift, why does it exist, and what are its use cases?", whereas a package description should answer "What's in this specific package and when should I install it?"
So I'd suggest updating the summaries and descriptions of all 3 packages with something very concise, along the lines of:

This package provides MicroShift, a small form factor OpenShift Kubernetes optimised for edge computing.
This package provides SELinux policies required by MicroShift.
This package provides the host networking dependencies for MicroShift's default CNI provider.

/lgtm


hangs and unstable cycle counts

Running some test code. Sometimes the programs (which work perfectly under simulators; including GNU make, a version of the OCaml runtime…) seem to hang (control-C exits though).
Also, when they don't hang, they seem to be randomly affected by some delays leading to high cycle counts.
Could there be that sometimes accesses to the SD card fail and hang programs?

On Thu, Jan 14, 2021 at 12:36:56PM -0800, David Monniaux wrote:

Could there be that sometimes accesses to the SD card fail and hang
programs?

Yes. SD card support (mainly the Linux litex_mmc driver, but also to
an extent the LiteSDCard gateware itself) is still a work in progress.
At the moment, at least when using 64-bit Linux and LiteX + Rocket,
there are data and DMA timeouts, and only single-block transfers are
enabled in the Linux driver.
I'm in the process of tracking down the source of those bottlenecks,
so things should improve, hopefully soon.

On Thu, Jan 14, 2021 at 12:36:56PM -0800, David Monniaux wrote:

Could there be that sometimes accesses to the SD card fail and hang
programs?

Yes. SD card support (mainly the Linux litex_mmc driver, but also to
an extent the LiteSDCard gateware itself) is still a work in progress.
At the moment, at least when using 64-bit Linux and LiteX + Rocket,
there are data and DMA timeouts, and only single-block transfers are
enabled in the Linux driver.
I'm in the process of tracking down the source of those bottlenecks,
so things should improve, hopefully soon.

Thanks @monniaux, would you mind sharing your test? We could use it as an additionnal stress test while looking at the other SDCard issues.

Thanks @monniaux, would you mind sharing your test? We could use it as an additionnal stress test while looking at the other SDCard issues.

see also https://github.com/enjoy-digital/litesdcard/issues/22

LiteSDCard is now working fine, so I'm going to close this.
@monniaux -- please feel free to (re-)open this or a new issue if you're still experiencing any trouble.


Registration form. w/o validation&firebase

#cm_17

also, set your git preferences so we can see who have commited.
currently, it looks like this:
Registration form. w/o validation&firebase
unknown authored 2 hours ago


Finer adjustments to scroll zoom

I was wondering if there could be some modifications made to the scrollToZoom so that a simple mouse wheel doesn't zoom, but wheel + pressing a key (like shift or Ctrl) does zooming? Thanks!

@msalsbery Could MouseTracker be used on the entire viewer to intercept and prevent this event unless there's a modifier key pressed?

@ShashanKV98
The easiest way to do this is to add a canvas-scroll handler to the viewer.
For legacy reasons, the event object passed to that handler only passes the shiftKey property from the originating DOM event (as 'shift'), but you could look at the originalEvent.ctrlKey and/or originalEvent.altKey properties as well.
If you wish to prevent the default wheel zoom behavior in your canvas-scroll handler, you can set the preventDefaultAction to true on the passed event object.
(Note: there's also a preventDefault property you could set to false to let the browser do its default wheel event handling, but that would probably never be useful on an OpenSeadragon viewer)
Example:
    viewer.addHandler('canvas-scroll', (event) => {
        //
        // Only zoom if user presses ctrl key while using wheel
        //
        if (!event.originalEvent.ctrlKey) {
            event.preventDefaultAction = true;
        }
    });

@msalsbery Could MouseTracker be used on the entire viewer to intercept and prevent this event unless there's a modifier key pressed?

Definitely could, but since there's already two  MouseTrackers on the full viewer elements, and most if not all gesture handlers are available for override at the Viewer level, it's generally only necessary in extreme cases. The Viewer 'canvas-xxx' events are a friendly user level place to do these simpler overrides

@msalsbery Could MouseTracker be used on the entire viewer to intercept and prevent this event unless there's a modifier key pressed?

Definitely could, but since there's already two MouseTrackers on the full viewer elements, and most if not all gesture handlers are available for override at the Viewer level, it's generally only necessary in extreme cases.
The Viewer 'canvas-xxx' events are a friendly user level place to do these simpler overrides

Ah, that makes sense, thanks!

@ShashanKV98 @pearcetm
I woke in the middle of the night thinking wait, that example won't work, but there's a plugin for that. I'll be back with a better solution...

@ShashanKV98
Never mind, the original example will work. Apparently I was dreaming at a different level of the event chain  🤦

Thanks @msalsbery @pearcetm ! The addHandler approach worked for me. I used preventDefault() though, since I wanted the browser scrolling too.


Queue.dequeue retains the most recently returned value

First I must say that I love to see different takes on how to implement schedulers in OCaml!
Out of curiosity I took a look at the queue implementation and noticed a few issues.

As a side note, if possible, I would recommend e.g. using a queue implementation (e.g. MPSC queue or Michael-Scott queue) from the Saturn library (formerly "lockfree"). Implementing lock-free algorithms is super hard. I should know, I first started tinkering with lock-free algorithms about a decade ago and I still occasionally spot subtle issues in my code I didn't foresee.

But, take a look at the dequeue implementation with a note I added:
let dequeue t =
  let rec go () =
    let p = Atomic.get t.head in
    match Atomic.get p.next with
    | None -> raise Empty
    | Some next ->
        if Atomic.compare_and_set t.head p next then
          let[@warning "-8"] (Some node) = Atomic.get p.next in
          node.value  (* NOTE: node == next *)
        else go ()
  in
  go ()

This is a very typical issue/bug in an implementation of the Michael-Scott queue. I recently noticed the same issue in the lockfree implementation.
So, the problem is that t.head is updated to point to a node whose node.value is returned and the node is logically removed from the queue. However, that node is not physically removed from the queue and it still points to the value. So, that last value dequeued from the queue will still be pointed to by the queue and it might be retained for a long time (indefinitely if nothing goes through the queue after) — causing a space leak.
A simple fix is to overwrite the value:
 let dequeue t =
   let rec go () =
     let p = Atomic.get t.head in
     match Atomic.get p.next with
     | None -> raise Empty
     | Some next ->
         if Atomic.compare_and_set t.head p next then
-          let[@warning "-8"] (Some node) = Atomic.get p.next in
-          node.value
+          let value = next.value in
+          next.value <- Obj.magic (); (* Safe as `next.value` is never accessed again *)
+          value
         else go ()
   in
   go ()

The length function is not part of the public interface of Queue, but looking at it made me realize an interesting aspect of it.  Consider the implementation with a note I added:
let iter ~f t =
  let rec go p =
    match Atomic.get p.next with
    | None -> ()
    | Some next -> f next.value; go next
  in
  go (Atomic.get t.head)

let length t =
  let v = Atomic.make 0 in  (* NOTE: This need not be an Atomic - a ref would be fine *)
  iter ~f:(fun _ -> Atomic.incr v) t;
  Atomic.get v

The interesting issue isn't related to the comment.  It is that, if there are concurrent consumers and producers, then length may actually return a value that is greater than the what the length of the queue has been at any point in time.  That is because nodes added by concurrent producers would be counted while nodes removed by concurrent consumers would not be counted (negatively).
Implementing an efficient and correct length operation for a lock-free queue is not impossible.  One way to do it is to have a count field in the nodes:
-and 'a node = { mutable value: 'a; next: 'a node option Atomic.t }
+and 'a node = { mutable value: 'a; next: 'a node option Atomic.t; mutable count: int }

You then make it so that when you add a node to the queue, the count of the new node is + 1 of the count of the predecessor (previous tail) node (read the count from predecessor node and update the count of the new node before CAS).  This way you can then read the length as follows:
let rec length t =
  let head = Atomic.get t.head in
  let tail = Atomic.get t.tail in
  match Atomic.get tail.next with
  | Some node ->
      (* Tail had fallen begind, so update it *)
      Atomic.compare_and_set t.tail tail node |> ignore;
      length t
  | None ->
      (* Check that we saw a consistent snapshot of the queue *)
      if Atomic.get t.head != head then length t else tail.count - head.count

This way the returned length is at least a true length that the queue had at a point in time.

Another subtle thing is that the transfer operation, which is part of the public interface, uses dequeue internally.  This of course means that concurrent updates of the queue interfere with the transfer operation and so the operation is not atomic.  It is possible to implement an atomic version of the transfer operation.  One way to do it is to introduce a snapshot concept that merely points to two nodes in the queue at a point in time:
type 'a snapshot = 'a node * 'a node

To take a snapshot you can use the same kind of loop as with the length in a previous comment:
let rec snapshot t : 'a snapshot =
  let head = Atomic.get t.head in
  let tail = Atomic.get t.tail in
  match Atomic.get tail.next with
  | Some node ->
      Atomic.compare_and_set t.tail tail node |> ignore;
      snapshot t
  | None -> if Atomic.get t.head != head then snapshot t else (head, tail)

In fact, you could reimplement length using snapshot:
let length t =
  let head, tail = snapshot t in
  tail.count - head.count

But towards an atomic version of transfer one could then introduce a dequeue_all operation that takes a snapshot of the queue and leaves the queue empty (which could of course change after taking the snapshot):
let rec dequeue_all t =
  let ((head, tail) as snapshot) = snapshot t in
  if Atomic.compare_and_set t.head head tail then snapshot else dequeue_all t

You could then replace the last node of the snapshot (in O(n) time as you need to traverse through the snapshot) and create a new queue or you could have some operations, like iter, for snapshots:
let iter ~f (head, tail) =
  let rec go prev =
    if prev != tail then
      match Atomic.get prev.next with
      | None -> ()
      | Some next -> f next.value; go next
  in
  go head

One subtle issue here is that a snapshot also retains all the nodes added to the queue after the snapshot was taken.  So, if you need to keep the snapshot for a long time, you should either replace the last node (as mentioned before) or convert the snapshot to some other form after taking it to avoid the potential space leak.

I don't see how I could comment directly on git.robur.io so I'll comment here.
See here:
let p = Atomic.get t.tail in
if Atomic.compare_and_set p.next None (Some q) then (
q.count <- p.count + 1;

The q.count update must be done before the compare_and_set:
let p = Atomic.get t.tail in
q.count <- p.count + 1;
if Atomic.compare_and_set p.next None (Some q) then (

That is because the compare_and_set publishes the node to other threads. So, if you update q.count after the compare_and_set, then it is possible for those other threads to read the unupdated q.count.

Thanks, yes, GitHub is good enough to talk bout the PR. I will do the update and merge, Thanks again for this great review.

Thinking about the snapshot approach I introduced a bit further yesterday, I realized it has a familiar (not unfixable) problem... 😅
Can you see it?

Well, the problem is the same one I pointed out as the first issue.
Namely that it can cause a space leak just like dequeue, because the .value field of the tail node is still reachable.  So, while the snapshotting works, the .value field of the tail node must be cleared after you are done with the snapshot.
Looking at the changes, the snapshot is not exposed, so it should be easy to fix the leak.  Just clear the .value field of the tail node of the snapshot, i.e. tail.value <- Obj.magic (), after done with the snapshot in the drop function.

Also, one thing I noticed is that the to_list function gives the list in LIFO order, which can be a bit surprising.

I finally made this PR which proposes your review: https://git.robur.coop/robur/miou/pulls/3 :+1:.

LGTM


Bettering your code...

I really liked your module but it contained repetitions and many _if_s and for loops. I tried to improve it, so I edited it, and ended with a shorter, cleaner, faster version that I called Toucho.js.
I wish you'd take a look at it (and why not update your code if you like what I did)
Peace <3

Hi,
thanks for writing. This module is really old and I continue to postpone its rewrite for too long time.
I'm currently on holidays but once I'm back I'll see your code and will think about improvements ;)


Lazy loading the low quality version

On a project I'm working on, we're using Lighthouse for performance testing, which is pretty much mobile 3G first. This means that every http request counts!

We are using the LQIP technique: <img> tags have a low quality thumbnail as src, and the full resolution image as data-src, which is lazy loaded by javascript.

Right now we stand to gain over 2.5 seconds in load time if we were to reduce the number of HTTP requests for the thumbnails on page load.

I was wondering if there is any interest for adding a lazy-lazyload feature to lozad, where instead of having a src that loads instantly and a data-src that is lazyloaded, we have data-lq-src or similar and data-src, which are both lazyloaded at the same time when they approach the viewport.

Thank you for your time!!

If this is your problem you should only use LQIP for your hero image(s)/first 3-4 images and leave. Also consider that load time doesn't mean much. It's about first impression. When does the page look ready for the user. The user doesn't know that there are images loading below the fold and he doesn't care.
Maybe check: https://www.webpagetest.org

See also this plugin I wrote in another project. Note you will need to write some CSS. It was done to create a blurup animation using a low quality image that is lazyloaded:
const regEmptySpace = /\s/g;

const matchingMedia = (source)=> {
    const media = source.getAttribute('media');
    return !media || matchingMedia(media).matches;
};

const createBlurup = (insertElement, img, src)=>{
    let timer, blurImg;
    const removeImg = ()=>{
        requestAnimationFrame(()=>{
            blurImg.remove();
        });
    };
    const onload = ()=> {
        clearTimeout(timer);
        img.removeEventListener('load', onload);

        if(blurImg){
            blurImg.classList.add('is_loaded');
            setTimeout(removeImg, 999);
        }
    };

    img.addEventListener('load', onload);

    timer = setTimeout(()=>{
        if(!img.complete){
            requestAnimationFrame(()=>{
                if(!img.complete) {
                    blurImg = document.createElement('img');
                    blurImg.className = 'mediabox_blurupmedia ' +
                        (img.className.split(regEmptySpace).filter(cName => cName.startsWith('var_')).join(' '));
                    blurImg.src = src;
                    insertElement.parentNode.insertBefore(blurImg, insertElement.nextSibling);
                }
            });
        }
    }, 33);
};

document.addEventListener('lazybeforeunveil', function(e){
    const img = e.target;
    const picture = img.closest('picture');
    const sources = picture ?
        Array.form(picture.querySelectorAll('source, img')) :
        [img]
    ;
    const source = sources.find(matchingMedia);
    const src = source.getAttribute('data-lowsrc');

    if(src){
        createBlurup(picture || img, img, src);

    }
});

Please see also https://github.com/aFarkas/lazysizes/issues/497 in case you are interested in to see this plugin.


generic webhooks + markdown, and docs

if I receive a "generic" webhook without jstransform stuff, would it be a plain m.text without formatting? it says that markdown is rendered after jstransform if you use them.

If the body contains a text key, then that key will be used as a message body in Matrix.
result will be automatically transformed by a Markdown parser.

I would find it very useful to send json with a formatted_text key which would be used literally as formatted_body of a org.matrix.custom.html message. is markdown applied to text?

If the body also contains a username key, then the message will be prepended by the given username.

so, like a relaybot bridge? would be nice to have "puppeting" this way, as matrix-appservice-webhooks used to

@Half-Shot thoughts about my feature request?

send json with a formatted_text key which would be used literally as formatted_body of a org.matrix.custom.html message

i.e. send html preformatted to the webhook instead of markdown

Totally missed that, sounds reasonable to me! Can you create a separate issue for the feature request?


Simplify README

Since the docs have finally been setup, the README should be pared back to just a brief intro to the package. All the details currently there should be moved to the docs for further in depth reading.

Intro
Requirements
Installation
Getting Started

Small example with just the basics of the API
One NavItem and one NavGroup with no url and one NavItem within it
No permissions or extra_context
Basic template, no styling

Examples
Documentation
License


[WIP] 1021/http outputs endpoint

Current coverage is 96.45% (diff: 98.46%)

Merging #1066 into master will decrease coverage by 0.30%

@@             master      #1066   diff @@
==========================================
  Files            48         49     +1
  Lines          2128       2173    +45
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           2059       2096    +37
- Misses           69         77     +8
  Partials          0          0

Powered by Codecov. Last update 3344fdf...b2eac4d


Add PList PlutusType wrapper for PBuiltinLists

Summary: Add PList type that wraps PBuiltinLists containing PData. A couple of helper functions for working with lists. I've hopefully struck a good balance between ergonomics and performance, as the produced seems to mirror what I've seen when using just the PBuiltinList type. Should this exist as a separate library? Bool/Integer has a bunch of helper functions too, so I assume it's okay for now.
TODO:

[ ] Figure out an efficient way to deal with wrapping and unwrapping list elements. (Am I missing something? Is this impossible to get around? i.e. without explicit fusion we will always wrap/unwrap from PData?)
[ ] Ergonomic lists from Haskell types after #53. I've just added integerList in examples for now.

You could add the constructors to PBuiltinList

Yeah, this works, although it's a bit of a pain to get around the cyclic dependency (without using orphans.)

and make all of the functions work for all lists, not just lists of data AFAICT.

Are you proposing I scott-encode PBuiltinList? I don't see how to put non-data in a PBuiltinList otherwise, even without the wrapper:
> printScript <$> eval (pconsBuiltin # (fromInteger @(Term _ PInteger) 0xdeadbeef) # pemptyListBuiltin)
*** Exception: HUnitFailure (Just (SrcLoc {srcLocPackage = "interactive", srcLocModule = "Ghci2", srcLocFile = "<interactive>", srcLocStartLine = 12, srcLocStartCol = 17, srcLocEndLine = 12, srcLocEndCol = 102})) "Script evaluation failed: EvaluationError [] \"(CekEvaluationFailure,Just (Apply () (Apply () (Force () (Builtin () MkCons)) (Constant () (Some (ValueOf integer<PHONE_NUMBER>)))) (Constant () (Some (ValueOf list (data) [])))))\""

> printScript <$> eval (pconsBuiltin # (pdata $ fromInteger @(Term _ PInteger) 0xdeadbeef) # pemptyListBuiltin)
"(program 1.0.0 [#1adeadbeef])"

I agree it would be regrettable to reimplement all of these functions for both "types" of lists, though. At worst, we can implement it once for scott-encoding and have a conversion operation incurring the cost a single time. Maybe this scott/data dichotomy can be turned into a class, but that's out of scope.
ptoScottList :: PIsData a => PBuiltinList a -> PScottList a
pfromScottList :: PIsData a => PScottList a -> PBuiltinList a

Or maybe we need to generalize a little more and enable implementing lists more abstractly?
class PIsList (list :: (k -> Type) -> k -> Type) where
  type PElemConstraint list :: (k -> Type) -> Constraint
  pelimList :: PElemConstraint list a => Term s (a :--> list a :--> r) -> Term s (PDelayed r) -> Term s (list a :--> r)
  pconsList :: PElemConstraint list a => Term s (a :--> list a :--> list a)
  pnilList :: PElemConstraint list a => Term s (list a)

Then PList becomes a proper Scott-encoded list, and we can create instances for both. With instance PIsList PBuiltinList where type PElemConstraint PBuiltinList = PIsData

Okay, here's what it looks like with a ListLike typeclass. I think now with #62 both of the TODOs are completed.

One thing we have to think about is what to do with constraints and type classes in Plutarch.
E.g. https://github.com/Plutonomicon/plutarch/pull/63/files#diff-3e4793aafaeb7a3e085513e7fbe66023660ce529c5ba5f4de98ffc4553fec9b0R111
This is actually monomorphised, meaning that a separate function is generated for each PListLike. I'm not sure this is optimal. We have to think more about the schism between the two encodings.


WinAppDriver always fetches empty string value for the price Text field  on "Store Purchase App" page

In my payment Win10 UI test(with a sandbox user for test payments) during purchasing procedure from the Store I check the product price text which should be "$0.00" to confirm that the purchase is  a test one. See attachments.
Run with: WinAppDriver <IP_ADDRESS> , appium 1.6.5 beta.
There is issue with the text(element properties) caching on the driver side. When the test fetches the text element from "PriceContainer" group it is always "" (empty). But visually the text "$0.00"  exists. See screenshot. Winappdriver`s findElement command returns the real text only if I investigate manually the field with 'inspect' tool from Windows 10 SDK. Suppose this action impacts on an internal driver cache update.
Name:	"https://www.microsoft.com/webblend/getClientView?clientType=DeviceStoreClient&version=2"
ControlType:	UIA_PaneControlTypeId (0xC371)
LocalizedControlType:	"pane"

I tried to fetch the text element with different selector type by id and xpath without success.
log.info("Windows Store: tap 'Get' button");
WebElement getBtn = new WebDriverWait(ad(), 10, 500).until(ExpectedConditions.elementToBeClickable(MobileBy.AccessibilityId("confirmButton")));
String productText = ad().findElement(MobileBy.AccessibilityId("ProductContainer")).findElement(MobileBy.tagName("Text")).getText();
String priceText2 = ad().findElement(MobileBy.AccessibilityId("PriceContainer")).findElement(MobileBy.tagName("Text")).getText();
String priceText = ad().findElement(MobileBy.xpath("//*[@AutomationId='PriceContainer']/*[@LocalizedControlType='text']")).getText();
log.info(String.format("Windows Store: product=%1$s, price=%2$s", productText, priceText));
Objects.requireNonNull(productText);
getBtn.click();

//new WebDriverWait(ad(), 10, 1000).until(ExpectedConditions.invisibilityOfElementLocated(MobileBy.name("Store Purchase App")));
new WebDriverWait(ad(), 10, 1000).until(ExpectedConditions.invisibilityOfElementLocated(MobileBy.AccessibilityId("confirmButton")));

[

](url)

after inspect the element with inspector:

canbe relates with #210


Fixed #23718 -- Fixed DB alias TEST['MIRROR'] setting.

Ticket #23718 -> https://code.djangoproject.com/ticket/23718

I have also refactored the tests so I hope it is fine now.

Hi @IlianIliev.
Thanks for the effort here. As it stands this is OK now. I think it's testing what it's meant to be.
My concern is with the mocking. It seems unnecessary here. If we just define a replica alias then we can do without it.
In settings:
    'replica': {
        'ENGINE': 'django.db.backends.sqlite3',
        'TEST': {
            'MIRROR': 'default',
        }
    },

The test in test_runner just becomes:
from django import db

self.assertEqual(db.connections['default'], db.connections['replica'])

This could be a single case in settings_tests.tests.TestDatabaseMirror (just to keep it all together).

The issue with this approach is that it would require updating test settings files (for Jenkins and devs running the test suite) with other DBs.
@timgraham: what's your thought here? Which way would you prefer?

@carltongibson I like the idea, it shouldn't affect other tests performance and will make it much simple and ease the creation of future tests using that feature. I'll wait for Tim to speak up before changing it but if he agrees it should be a quick change on my side.
I am not super familiar with Jenkins so probably there is a reason for that, but why isn't the Jenkins configuration part of this repo?

@timgraham The test in settings_tests is actually to verify that the ORM is behaving correctly if you have two connections that actually share the same connection object. If the settings fix works correctly the connections object should behave in this way, but I understand what you mean and yes the settings_tests is not the correct place for this test.
Do you think that queries is a better place to move this test?
As for how the tests are testing the mirror - as I explained above the one in settings_tests verifies that the behaviour is correct if there are two keys in the connection dictionary pointing to the same connection object.
Then the test in test_runner actually verifies that if the runner is instantiated with the mirror setting it will result in having the "two keys to the same object" behaviour described above.
I understand that it looks a bit confusing and messy but this is because I don't see other way than mocking to modify the connections without messing the connections object for all tests that follow.
Going with the separate replica setting like @carltongibson propose will allows us to avoid the whole mocking part and have only the test in settings_tests to verify the mirror behaviour.

...they pass without the fix...

Hmmm. Well, yes, because we're mocking the connections dict, when the very issue here is that that dict is incorrect. I'm not at all convinced by the mocking: it doesn't seem like we're actually testing what we're claiming to fix.
(Whilst it's a bit of a pain updating the other configs I think adding the replica alias would be better — and the tests would fail without the fix.)

...settings_tests is not the correct place for this test...

This is an issue on TEST['MIRROR'] setting. I'd imagined a single case (in settings_tests) with the following checks:

Optional: create in default and other, fetch single item from default.
Create in default and replica, fetch two items from default.
assert that default and replica have the same connection.

(i.e. the tests here but just all in one place.)
I know at the first two are querying tests, but they're how we know the setting is working as expected.
Happy to accept another approach but that was my thought.

I pushed (rough) edits of the version of this with a new replica alias here.
Without the patch this version fails the new test:

 django.db.utils.OperationalError: database table is locked

Two connections trying to use the same DB table in the same transaction.
In order to go with this we'd need to:

Update the role files for each config in the django-box VM project, rebuild and publish.
Same for Jenkins.
Update the docs on Using another settings file to say "you need three aliases" (etc.)
Possibly, add a (temporary?) check in runtests.py to make sure the alias was defined, and give a helpful message if not.
... anything else...?

I'm happy to do those bits if this is the way to go.
Alternatively we can just move the tests and use the mocking approach, as per Tim's previous comment. I'm not sure it's testing the bug, but maybe it's good enough.

I think https://github.com/django/django/pull/9940 might be sufficient. The ORM tests proposed in settings_tests are testing other areas of code that are probably tested elsewhere. At least, the test in the other PR seems to cover the change in behavior for this fix.

Then this is a documentation issue, or it looks like one:
This passes without any changes:
class TestDatabaseMirror(TransactionTestCase):
    available_apps = ['settings_tests']

    def test_replica_writes_to_default_db(self):
        with transaction.atomic():
            Simple.objects.using('default').create()
        with transaction.atomic():
            Simple.objects.using('replica').create()
        self.assertEqual(Simple.objects.count(), 2)

So a Note saying that you have to do it this way to get the expected results should be enough no?

What @charettes says makes sense to me and in this case @carltongibson is right that this is a documentation issue. We just need to add a note with more details for the behaviour.
But in this case what do you think should be the behaviour of fixtures and any DB interaction during the test setUp?
I'll verify the current behaviour on Monday but I was wondering whether they are/should be executed as an atomic transaction so all data created during the fixtures/setUp is available both in the master DB and the replica.


http.Server: http: Accept error: request has been rate-limited

Output of the info page (if this is a bug)
# agent status
Getting the status from the agent.

===============
Agent (v6.14.1)
===============

  Status date: 2019-11-25 08:04:57.021342 UTC
  Agent start: 2019-11-05 04:05:44.527832 UTC
  Pid: 1
  Go Version: go1.12.9
  Python Version: 2.7.16
  Check Runners: 4
  Log Level: WARN

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -53µs
    System UTC time: 2019-11-25 08:04:57.021342 UTC

  Host Info
  =========
    bootTime: 2019-11-05 03:11:53.000000 UTC
    kernelVersion: 4.15.0-1034-gke
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: 10.1
    procs: 60
    uptime: 53m53s

  Hostnames
  =========
    host_aliases: [gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf.yyy gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf-xxx-test-app-0]
    hostname: gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf.c.yyy.internal
    socket-fqdn: datadog-f49ddd458-6zk92
    socket-hostname: datadog-f49ddd458-6zk92
    host tags:
      gke-xxx-test-app-0-4060f4d1-node
      prd-app
      zone:asia-northeast1-b
      instance-type:n1-standard-64
      internal-hostname:gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf.c.yyy.internal
      instance-id:5444726203643418295
      project:yyy
      numeric_project_id:824996079768
      cluster-location:asia-northeast1
      gci-ensure-gke-docker:true
      cluster-name:xxx-test-app-0
      gci-update-strategy:update_disabled
      kube-labels:beta.kubernetes.io/fluentd-ds-ready=true,cloud.google.com/gke-nodepool=app-red-0,cloud.google.com/gke-os-distribution=ubuntu,server_group=red
      created-by:projects/824996079768/zones/asia-northeast1-b/instanceGroupManagers/gke-xxx-test-app-0-app-red-0-2c62eb0b-grp
      enable-oslogin:false
      cluster-uid:4060f4d13b73b0d6375b30d37b5c05676cd2eaae63ce5cdcc73ab215d3fa3584
      disable-legacy-endpoints:true
      instance-template:projects/824996079768/global/instanceTemplates/gke-xxx-test-app-0-app-red-0-2c62eb0b
    hostname provider: gce
    unused hostname providers:
      configuration/environment: hostname is empty

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 116,156
      Metric Samples: Last Run: 6, Total: 696,930
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    disk (2.5.0)
    ------------
      Instance ID: disk:89011cceb1f16288 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.yaml
      Total Runs: 116,157
      Metric Samples: Last Run: 200, Total: 1 M
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 95ms

    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 116,157
      Metric Samples: Last Run: 578, Total: 1 M
      Events: Last Run: 0, Total: 1,909
      Service Checks: Last Run: 1, Total: 116,157
      Average Execution Time : 135ms

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 116,156
      Metric Samples: Last Run: 5, Total: 580,780
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 116,157
      Metric Samples: Last Run: 65, Total: 1 M
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    kubelet (3.3.2)
    ---------------
      Instance ID: kubelet:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 116,156
      Metric Samples: Last Run: 628, Total: 1 M
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 464,624
      Average Execution Time : 375ms

    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 116,157
      Metric Samples: Last Run: 6, Total: 696,942
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 116,156
      Metric Samples: Last Run: 17, Total: 1 M
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    network (1.11.4)
    ----------------
      Instance ID: network:e0204ad63d43c949 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 116,157
      Metric Samples: Last Run: 79, Total: 1 M
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 2ms

    ntp
    ---
      Instance ID: ntp:7e1f812862c38a66 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.yaml
      Total Runs: 116,157
      Metric Samples: Last Run: 1, Total: 116,157
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 116,157
      Average Execution Time : 0s

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 116,157
      Metric Samples: Last Run: 1, Total: 116,157
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

========
JMXFetch
========

  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 116,156
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 10,447
    Metadata: 0
    Requeued: 1
    Retried: 1
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    SketchSeries: 0
    Success: 242,759
    TimeseriesV1: 116,156

  Transaction Errors
  ==================
    Total number: 1
    Errors By Type:

  API Keys status
  ===============
    API key ending with 2c40c: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - 2c40c

==========
Logs Agent
==========

  Logs Agent is not running

=========
Aggregator
=========
  Checks Metric Sample: 176.4 M
  Dogstatsd Metric Sample: 21.7 M
  Event: 1,910
  Events Flushed: 1,910
  Number Of Flushes: 116,156
  Series Flushed: 168.1 M
  Service Check: 1.8 M
  Service Checks Flushed: 1.9 M

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 21.7 M
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 1.9 G
  Udp Packet Reading Errors: 0
  Udp Packets: 21.7 M
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0

Describe what happened:
datadog agent showed http: Accept error: request has been rate-limited,
and then we can not see any apm events.
Describe what you expected:
We want to know the root cause and how to solve it.
Steps to reproduce the issue:
Additional environment details (Operating System, Cloud provider, etc):
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:07:52.624988201Z 2019-11-25 06:07:52 UTC | TRACE | WARN | (pkg/trace/info/stats.go:111 in LogStats) | [lang:php lang_version:7.2.24 interpreter:fpm-fcgi tracer_version:0.33.0] -> traces_dropped(decoding_error:55). Enable debug logging for more details.
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:07:52.625030217Z 2019-11-25 06:07:52 UTC | TRACE | WARN | (pkg/trace/api/api.go:518 in watchdog) | CPU threshold exceeded (apm_config.max_cpu_percent: 50): 1
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:02.624100679Z 2019-11-25 06:08:02 UTC | TRACE | WARN | (pkg/trace/api/api.go:518 in watchdog) | CPU threshold exceeded (apm_config.max_cpu_percent: 50): 1
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.297445136Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 5ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.302707184Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 10ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.313077695Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 20ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.333479258Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 40ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.373653815Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 80ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.454073332Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 160ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.614451933Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 320ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.934798176Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 640ms
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:08.575108959Z 2019-11-25 06:08:08 UTC | TRACE | ERROR | (pkg/trace/agent/log.go:65 in ReceiveMessage) | Too many messages to log, skipping for a bit...
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:12.578466285Z 2019-11-25 06:08:12 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 1s
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:13.579612403Z 2019-11-25 06:08:13 UTC | TRACE | ERROR | (pkg/trace/api/api.go:372 in handleTraces) | Cannot decode v0.4 traces payload: unexpected EOF
datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:13.579646122Z 2019-11-25 06:08:13 UTC | TRACE | ERROR | (pkg/trace/api/api.go:372 in handleTraces) | Cannot decode v0.4 traces payload: unexpected EOF

@axot was this datadog limiting you?  We are running into something similar now.

https://docs.datadoghq.com/tracing/troubleshooting/?tab=java#max-connection-limit This is something that I found that might help you out.


How the incremental changes to a notebook work with the solution ?

I am not sure if this is a limitation of the solution or a bug.
In the sample given notebook, we have exampleInputA and exampleInputB producing the output exampleoutput1. If I replace the exampleInputB with a new input exampleInputC to produce the same output - I get 3 inputs to the notebook in the lineage map where it should have been 2 only ( A&C).  So my question:  does the solution only append and don't delete any lineage mapping?
val exampleC = (
spark.read.format("csv")
.schema(exampleCSchema)
.option("header", true)
.load(adlsRootPath+"/examples/data/csv/exampleInputC/exampleInputC.csv")
)
val outputDf = exampleA.join(exampleC, exampleA("id") === exampleC("id"), "inner").drop(exampleC("id"))
outputDf.repartition(1).write.mode("overwrite").format("csv").save(adlsRootPath+"/examples/data/csv/exampleOutput/")
Mapping in the purview ( I was expecting exampleInputB to be removed automatically)

Hi, @Sam-Panda , You can click into the advanced properties of a given notebook and observe individual databricks processes. Each process represents a run of the notebook that points to a unique combination of inputs and outputs.  To see a given "state" of lineage, please go to the notebook or notebook task and click on the Properties tab and select one of the processes.  Alternatively, you could look at exampleInputC's lineage view (e.g. switch to the exampleInputC asset) and you should only see A+C = Output.
The purpose behind this is to capture historical lineage. For example, if this notebook were parameterized, we'd want to have all history of the inputs and outputs being used (because they were generated / influenced by a given parameter).
Even for notebook that are not parameterized, if you had an object used as an input / written out as an input, you'd still want to note its historical lineage because you want to know how that object was used.
I hope that helps explain why you are seeing this and how you can see your desired view of lineage.

Closing out since we've answered the question but happy to re-open if necessary.


Learning: Grading Learners

The "sequel" to #496.
After the feedback from @lucasmoura and @norvig, I changed the previously introduced function grade_learner. It now returns the ratio of correct answers.
I chose not to use the test function since it is tailored to the evaluation of a learner with the fold validation method (bulk error-finding in the folds). I believe it would be best if we also had a simple function to indicate loosely and quickly how good a learner is.
So I wrote the simpler function grade_learner to check a learner against a set of examples (in the form of tuples).
Also, I renamed test to err_ratio. Unfortunately pytest mistakes the name for a function it needs to test and throws an error. err_ratio does not have that issue.
Finally, I used the new function to test the Neural Network and the Perceptron.

@MrDupin I think I have not explained my idea properly, and I am sorry for that. I was thinking about using the test function after training the model and using to evaluate the learner. Currently, you are using modified test cases to evaluate the learner right ? I think that what you can do is to evaluate the whole trainining process, and not just a few examples. And for that, use the test function.
But again, maybe this is not the best approach :)

Your suggestion is definitely one way to go about it and I don't think there is right and wrong in this case. In my opinion a simple function to run a learner through some tests is useful to have around. The test function is mostly for the bulk error-checking used in operations like the fold validation and for a more robust evaluation of a learner (with the option to print out as it goes along).
I think I will add the test in test_learning though, since we need to be checking that one too. I will have the learner evaluated with the function on the whole iris dataset.
Having both is the best way to go, in my opinion.


why semicolon is after 2 lines?

https://github.com/expressjs/express/blob/f3c5f7ee29e9b935e95e628e002ee9eaebda176a/lib/express.js#L87

FYI: https://github.com/expressjs/express/issues/3558#issuecomment-364785025


Clarification request for identifiers published date, draft date, edition date.

For drafts we have date:
IEEE Draft Std P802.1AX-REV/D4.0, March 2014
But also we have date for identifiers with ISO stage, like:
IEEE Unapproved Draft Std P12207/FDIS, Nov 2007
Sometimes without stage or draft but it have date:
IEEE/IEC P60076-57-1202, July 2014
Sometimes there are just year:
IEEE Std 1491-2012
Also we have edition date I didn't mentioned:
IEC 61691-7 Edition 1.0 2009-12
Is the draft date and identifier with ISO stage (now considered as draft) are the same?
When identifier without draft or stage have date is it publication date? Publication date and draft date are not the same, right? And all these are not related to edition date?
Currently for IEEE we have:

draft date
publication date
edition date

IEEE/IEC P60076-57-1202/D{draft id}, July 2014

@ronaldtse Does it means that when we have date in identifier, like ", July 2014" at the end, it is draft, right?
But when we have only year as in identifier IEEE Std 16085-2020 that doesn't mean it's draft, is it correct?

Also we have edition date I didn't mentioned: IEC 61691-7 Edition 1.0 2009-12

Edition is "1.0". Publication date is "2009-12".

Should we reformat it as IEC 61691-7 Edition 1.0, December 2009?

Is the draft date and identifier with ISO stage (now considered as draft) are the same?

What do you mean? I don't understand this question.

Identifiers with ISO stage usually also have publication date, should it be considered as "draft date"?

Actually, every document only has 1 date, which is the date this document is "offered".
published document => has publication date, which is also edition date. draft document => has draft date.

Should we use only "publication date" for any types of dates like edition date, draft date, publication date?
Now edition date, publication date and draft date are different attributes.

I'm coming in to this with little context, but, to confirm:

draft date is indeed date of publication (or circulation pending endorsement) of a draft of a document.
edition date is date of publication of a specific edition of a document.
There is a point in storing both pieces of information in a bibliographic record, because that gives the history of a document. But for identifiers, don't you only ever publish one date at a time? Does this have formatting implications?

But also we have date for identifiers with ISO stage, like:
IEEE Unapproved Draft Std P12207/FDIS, Nov 2007

Yes, that is a draft date, since it is the date of publication of a draft of a document, and the identifier is explicitly saying that.
It is usual for the date of publication of a document to be year, and the date of draft to be month or even month and day; that's because you can have many drafts of a document issued in a year, but I don't think I've seen a case where there are two different publications/editions of a document in a year.
That said, BSI we have cases where some classes of document include the month in  a publication date, and some do not.  I really think you should only infer draft vs publication status of a document from whether the document is said to be a date. Yes, I would be surprised if a month is given for a publication date, but it is not impossible: BSI does it.

Should we use only "publication date" for any types of dates like edition date, draft date, publication date?
Now edition date, publication date and draft date are different attributes.

Like I said: there's a reason bibliographic entries differentiate them, but I would be very surprised if any standards identifier uses more than one date. Drafts don't have  a publication date. Published documents don't care in their identifier about draft dates. Edition dates usually override first publication dates. The only case where we routinely see two dates is in amendments and corrigenda, where you have a date of publication of the original document, vs a date of publication of the amendment.
Only model these as different dates if you have an identifier that uses two of them, or if the formatting requirements are different. The month status of drafts, I think, is inherited from the document being a  draft, not from the date being differently formatted.

Do you think we can just use 1 "date" to handle? Unless we have multiple dates in one PubID?

@ronaldtse My question is more about, did I make a mistake using 3 different dates for draft, edition, and publication?
I'm fine with all them using separate attributes, but maybe separate attributes for draft date and publication date are too much?

@mico its not a mistake per se but it’s difficult to determine the type of date accurately in the identifier, and hence difficult to use. Better stick with one date.

Does this ticket need to stay open, @mico?

Unless we have multiple dates in one PubID?

IEC/IEEE 62582-1:2011 Edition 1.0 2011-08
This should be converted to IEEE format, then it will be IEC/IEEE Std 62582-1-2011 Edition 1.0 2011-08
Twice mentioned the same year.

Year 2011 was the part of identifier but later on it was removed. I'm not sure how to approach this case.
Could anything from these work?

IEC/IEEE Std 62582-1-2011
IEC/IEEE Std 62582-1-2011 Edition 1.0
IEC/IEEE Std 62582-1, August 2011 Edition 1.0

So in IEC/IEEE Std 62582-1-2011 Edition 1.0 2011-08 (wherever the hell that came from), there is a year date for the standard, and a year + month date for the edition. They are not in fact different years, they are different granularities of the same year, and if the next edition were published in May 2024, it would turn into IEC/IEEE Std 62582-1-2024 Edition 2.0 2024-05. You are not going to get IEC/IEEE Std 62582-1-2011 Edition 2.0 2024-05, that is not how those dates work. The dates in fact indicate the edition in ISO practice (which IEC inherits), and the edition date is merely an elaboration of that.
IEC/IEEE Std 62582-1-2011 Edition 1.0 2011-08 seems to me to be preferable to the last two alternatives, but it is your job to investigate IEEE possible identifier formats: I have no idea what IEEE expect to see, and I don't have access to any more information than you do via Google. I am not willing to make stuff up that a client will turn around and tells us is wrong.


Support returning double and boolean from evaluate

The return value of duktape.evaluate returns null when not a string:
duktape.evaluate("2 + 3"); // returns null

This is a bit tricky if you're not aware of it.
It would be great when evaluate would also allow returning primitives like double and boolean, maybe like this:
Double result = duktape.evaluate("2 + 3", Double.class);

We’ll implement this in a type safe way as a part of https://github.com/square/duktape-android/issues/43

Thanks!

#43 is implemented.  Also, Duktape.evaluate now coerces the result to string, so

duktape.evaluate("2 + 3");

returns the String "5".

Great news, thanks!

how are you returning a Double instead of a String? I can't find anything in the doc


fix(docs): note that -C options can be prefixed with skbuild.

https://github.com/scikit-build/scikit-build-core/pull/556 added the ability for -C options to be prefixed with skbuild., but this was not noted in the configuration documentation. Add a note for this.

Thanks! This is mostly for allowing user options in the future, which we haven't added yet.

I noticed this when working on https://github.com/rapidsai/rapids-build-backend/pull/37. In general I think namespacing config options is a good idea - we plan to do it for rapids-build-backend.


fixed resetDefaultDate(): let currDate is the first of the month

issue:  bug: nextMonth's button #156

Coverage remained the same at 99.663% when pulling 6bf8eb9cc32ab3f7d73471aa618f5ec56a78ad5d on Kyleloh:master into<PHONE_NUMBER>82eea49b7af37985e132c7f2c039bc on charliekassel:master.

Good spot! Nice work adding the test.

Available from v0.7.13

ok, thanks~


No email sent to channel

Hey,
I configured the policy-reporter to send summary reports via email. This works perfectly fine for the 'global' summary report, which gets sent to the email specified under emailReports.summary.to.
I used the same email in a channel, where I include only specific namespaces. For some reason, there is no email sent, although there are multiple PolicyReports in the specified namespaces with failed results.
The helm values look like this:
  emailReports:
    clusterName: "hcp-int" # (optional) - displayed in the email report if configured
    titlePrefix: "Policies Report for HCP int" # title prefix in the email subject
    smtp:
      secret: "smtp-configuration" # (optional) secret name to provide the complete or partial SMTP configuration

    # basic summary report
    summary:
      enabled: true
      schedule: "*/5 * * * *" # CronJob schedule defines when the report will be send
      activeDeadlineSeconds: 300 # timeout in seconds
      backoffLimit: 3 # retry counter
      ttlSecondsAfterFinished: 0
      restartPolicy: Never # pod restart policy

      to<EMAIL_ADDRESS># list of receiver email addresses
      filter: {} # optional filters
      #  disableClusterReports: false # remove ClusterPolicyResults from Reports
      #  namespaces:
      #    include: []
      #    exclude: []
      #  sources:
      #    include: []
      #    exclude: []
      channels:
      - to<EMAIL_ADDRESS>        filter:
          disableClusterReports: true
          namespaces:
            include: ["utilities", "trivy-system", "common"]
          sources:
            include: ["Trivy*"]

Any idea what might be going wrong?

Hey @sudoleg, currently the sources filter doesn't support wildcards.

It works now, thank you very much @fjogeleit
One more suggestion: As wildcards are not supported, I would adjust the documentation:

https://github.com/kyverno/policy-reporter-docs/blob/main/content/02. core/email-reports.md?plain=1#L191
https://github.com/kyverno/policy-reporter-docs/blob/main/content/02. core/email-reports.md?plain=1#L191
https://github.com/kyverno/policy-reporter-docs/blob/main/content/02. core/email-reports.md?plain=1#L191 (if wildcards aren't supported for namespaces)

I find it quite confusing. If you agree, I would adjust the docs :)

No, I didn't try wildcards with namespaces, as I don't have a usecase for that. I will leave the docs for namespaces as it is and adjust it for sources


tests: create test cases for Room.py

Partially resolves #3, but still need tests for methods developed in the future


schema/url: raise ResolveError if the URL is invalid

This is similar to #3364.
When using lbrynet get URL, if the URL is not a valid URL the function url.URL.parse will raise a ValueError exception which will produce a whole backtrace.
For example, this is the case if we provide a channel name with a forward slash but without a stream name.
lbrynet get @Non-existing/

Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1106: Error downloading @Non-existing/: Invalid LBRY URL

Now we raise ResolveError which can be trapped in the upper functions that use url.URL.parse such as FileManager.download_from_uri. The traceback won't be shown.
WARNING  lbry.file.file_manager:249: Failed to download @Non-existing/: Failed to resolve 'Invalid LBRY URL'.
WARNING  lbry.extras.daemon.daemon:1106: Error downloading @Non-existing/: Failed to resolve 'Invalid LBRY URL'.

The FileManager class raises ResolveError in many situations but I don't know if this is the best exception to throw. Maybe a new exception should be defined like NotDownloadableError derived from WalletError in lbry/error.

ResolveError would be misleading to use for this case. Parsing a URL is a ValueError, meaning that the URL passed in is not an actual URL (an error of value), whereas ResolveError should be used when resolution fails (attempting to resolve a valid URL fails for some reason).


Fix Node.CompareDocumentPosition

Node.CompareDocumentPosition didn't handle nested elements correctly. Example:

	1

	2

Both child1.CompareDocumentPosition(child2) and child2.CompareDocumentPosition(child1) would result in DocumentPositions.Following.
I added a few test cases as well though they're probably not comprehensive.

Hi Livven looks like a great change, but I don't accept PR to master any more. Can you please PR into an appropriate feature branch?
It is all outlined in the Wiki. Thanks!

Thanks, I created a new pull request (#168) on another branch.
BTW I don't think Github lets you assign yourself to issues unless you're a collaborator, as suggested in the development workflow:

If no issue already exists for the work you'll be doing, create one to document the problem(s) being solved and self-assign.

Yes that is true, but is completely unrelated. Assigning somebody to an issue and really working / committing are two different things.
Until this point in time no one asked / wanted to be a collaborator (even though many people post issues / PR on a more regular basis). So if it would be more convenient for you with the "collaborator" status I can add you to the team.


[Help Wanted] AttributeError: module 'spconv' has no attribute 'SubMConv3d'

Hi everyone!
I am trying to run this project, and I'm locked. I try to run the project after installing all the necessary modules, which I will indicate later, and I only receive this error. I have tried to change the version of cuda, (11.1, 11.4, 11.5), install drivers, run from ubuntu, and I always get this error:

C:\Users\v\Desktop\Cilindro\Cylinder3D>py demo_folder.py --demo-folder . --save-folder .
Namespace(config_path='config/semantickitti.yaml', demo_folder='.', demo_label_folder='', save_folder='.')
[480 360  32]
Traceback (most recent call last):
File "demo_folder.py", line 151, in
main(args)
File "demo_folder.py", line 83, in main
my_model = model_builder.build(model_config)
File "C:\Users\v\Desktop\Cilindro\Cylinder3D\builder\model_builder.py", line 19, in build
cylinder_3d_spconv_seg = Asymm_3d_spconv(
File "C:\Users\v\Desktop\Cilindro\Cylinder3D\network\segmentator_3d_asymm_spconv.py", line 265, in init
self.downCntx = ResContextBlock(num_input_features, init_size, indice_key="pre")
File "C:\Users\v\Desktop\Cilindro\Cylinder3D\network\segmentator_3d_asymm_spconv.py", line 49, in init
self.conv1 = conv1x3(in_filters, out_filters, indice_key=indice_key + "bef")
File "C:\Users\v\Desktop\Cilindro\Cylinder3D\network\segmentator_3d_asymm_spconv.py", line 17, in conv1x3
return spconv.SubMConv3d(in_planes, out_planes, kernel_size=(1, 3, 3), stride=stride,
AttributeError: module 'spconv' has no attribute 'SubMConv3d'`

I am using python 3.8.0, and a virtual environment
Requirements;
Yaml 6.0
Cython 0.29.24
Pytorch 1.8.1+cu111
Torch-scatter 2.0.9
Spconv 2.0.2-cu114
nuscenes-devkit 1.1.9
I greatly appreciate if someone can contact me and help me with this error.

I do not test with other version of spconv. Can you try with spconv==1.2.1?

I do not test with other version of spconv. Can you try with spconv==1.2.1?

Yeah, running 1.2.1 solve the problem.
Thank you

The problem is that in new versions of spconv, some modules are moved from the spconv. path to spconv.pytorch.conv., such as SubMConv3d, SparseConv3d, etc.
So, a new solution:
Change spconv.SubMConv3d to spconv.pytorch.conv.SubMConv3d, SparseConv3d and other modules are similar.
I tested it successfully in version 2.1.2.

Sorry to  bother @Ryyyc . I encountered an assertion error(assert my_model_dict[k].shape == value.shape') when I use 2.1.2.  I tried to print the shape info as print("model shape:{}, pre shape:{}".format(str(my_model_dict[k].shape), str(value.shape)))
The result is:
model shape:torch.Size([32, 1, 3, 3, 32]), pre shape:torch.Size([1, 3, 3, 32, 32])
I tried to modified as 'my_model_dict[k] = my_model_dict[k].transpose(0, 1).transpose(1, 2).transpose(2, 3).transpose(3, 4)'
however, some sample still produce error as:
model shape:torch.Size([3, 3, 3, 512, 256]), pre shape:torch.Size([3, 3, 3, 512, 512])
I want to know whether it is caused by the spconv version or the Cylinder. looking forward to your reply, thanks again

Sorry to bother @Ryyyc . I encountered an assertion error(assert my_model_dict[k].shape == value.shape') when I use 2.1.2. I tried to print the shape info as print("model shape:{}, pre shape:{}".format(str(my_model_dict[k].shape), str(value.shape)))
The result is: model shape:torch.Size([32, 1, 3, 3, 32]), pre shape:torch.Size([1, 3, 3, 32, 32])
I tried to modified as 'my_model_dict[k] = my_model_dict[k].transpose(0, 1).transpose(1, 2).transpose(2, 3).transpose(3, 4)' however, some sample still produce error as: model shape:torch.Size([3, 3, 3, 512, 256]), pre shape:torch.Size([3, 3, 3, 512, 512])
I want to know whether it is caused by the spconv version or the Cylinder. looking forward to your reply, thanks again

@zhouyuchen01
From your description, I'm guessing you're migrating model weights from spconv1.x to spconv2.x. I think you can refer to the change in spconv weight layout at
https://github.com/traveller59/spconv/blob/master/docs/SPCONV_2_BREAKING_CHANGEs.md

weight layout has been changed to RSKC (native algorithm) or KRSC (implicit gemm), no longer RSCK (spconv 1.x). RS is kernel size, C is input channel, K is output channel.

Of course, first you should make sure that your model hyperparameter settings are the same the pre-trained model, and secondly, you should only change the weights of the spconv module, and should not include other modules.

Thanks for your suggestion as well as your kind help.

@ZhouYC-X sorry to bother. Have you successfully reproduce on spconv2.x, using the given weight?


Losing Page in my Cucumber.js Tests over Node

Here's my code.  I lose the page as it ends up undefined in my first Cucumber.js step_definition and I"m not sure how to resolve this.  I don't know if I am just doing this all wrong with Phantomjs-node or what.  I've tried all sorts of things such as move the phantom.create to within the node exports function and also tried moving the end bracket (end of function) for the create to have it wrap all my Given functions for cucumber with no luck.
//var assert = require('chai').assert;
var phantom = require('phantom'), _ph, _page;
phantom.create("--web-security=no", "--ignore-ssl-errors=yes", { port: 12345 }, function (ph) {
_ph = ph;

_ph.createPage(function(page) {
    _page = page;
    console.log("_page1:  " + _page); // here I DO get a valid page instance so all is good so far
    //_ph.exit();
});

});
module.exports = function() {
"use strict";
this.Given(/^I visit the customers display page$/, function (callback) {

    console.log("_page:  " + _page); // problem is now _page instance above is lost, and is undefined here
    _page.open("/", function(status){
        console.log("status:  " + status);

        status.should.equal("success");
        _ph.exit();
        callback();
    });
});

this.Then(/^I should not see any customers listed$/, function (callback) {
    // not implemented
    callback.pending();
});

this.Given(/^there are some customers to view$/, function (callback) {
    // not implemented
    callback.pending();
});

this.When(/^I go to the customers display page$/, function (callback) {
    // not implemented
    callback.pending();
});

this.Then(/^I should see a list of customers grouped by Topic$/, function (callback) {
    // not implemented
    callback.pending();
});

};

From the readme, you need to probably do  this.World = require("../support/world.js").World; and expose something like
// features/support/world.js
var webdriver = require("selenium-webdriver");

var World = function World(callback) {
  this.driver = new webdriver.Builder()
    .withCapabilities(webdriver.Capabilities.chrome())
    .build();
  callback();
}

module.exports = World;

I don't know cucumber.js that well. So hopefully this helps.

Also, I don't see the reason for using cucumber. http://blog.8thlight.com/kevin-liddle/2013/09/18/a-case-against-cucumber.html.

Nice!  I'm a huge follower of 8th Light.  Let me check that out, I've seen that post before.

I had a recent conversation about that post, and blogged about it: http://code-zest.blogspot.com/2015/04/do-you-need-cucumber-or-even-bdd.html

I went to write a comment on the post, and it didn't work. So here is my take...
I have never worked in an environment that PM or non technical people in the team wrote acceptance criteria. We had very intention for that to happen. But it just didn't happen. It go to a point that the cucumber tests got complex and only the engineers maintained. If you work in an environment that you can get them to write these tests then I think it makes sense. But if you can't get non-technical people to commit to these test cases, then I completely disagree that it could be more valuable, faster, or better in way to write code that needs to get parsed. It is not easier to read. It is more code to look at. And with your assumption of it actually being maintained by a PM, I don't think there is any other valid reason.
I had wrote all this comment on the blog but it didn't work.

I think PMs can do it, and they are where he's at.  They do it in the
stories, there is no coding.  The language Gherkin syntax works perfectly
inside a story!  You type stories together as a team with the business
anyway, this is just a nice template to do it in.  And so at his
organization nobody is bitching, they love it.
I think the argument comes when people think PMs are to code too.  There's
no coding with Gherkin.  And if you do it and make it a standard language
in your iteration meetings and story application (agile app) I don't see
why it couldn't be something that works great.  All you do is paste the
Given, etc. wordage in and type in the requirement.  Simple.
So not sure if you're talking about the same argument...
On Thu, Apr 2, 2015 at 1:55 PM Amir Raminfar<EMAIL_ADDRESS>wrote:

I went to write a comment on the post, and it didn't work. So here is my
take...
I have never worked in an environment that PM or non technical people in
the team wrote acceptance criteria. We had very intention for that to
happen. But it just didn't happen. It go to a point that the cucumber tests
got complex and only the engineers maintained. If you work in an
environment that you can get them to write these tests then I think it
makes sense. But if you can't get non-technical people to commit to these
test cases, then I completely disagree that it could be more valuable,
faster, or better in way to write code that needs to get parsed. It is not
easier to read. It is more code to look at. And with your assumption of it
actually being maintained by a PM, I don't think there is any other valid
reason.
I had wrote all this comment on the blog but it didn't work.
—
Reply to this email directly or view it on GitHub
https://github.com/sgentle/phantomjs-node/issues/262#issuecomment-89008491
.

So another thing can't I just use the PhantomJS-Node page instance and do I really need web driver if all I'm doing is opening a page and not really interacting with it yet?  I mean maybe later I'll find I need web driver but I figured I can just use page.open() for now to get my BDD tests to past in the minimal amount of config and PhantomJS needed.


Installation error

When installing the package, 247 dependencies are pre-compiled. 246 with no issues but one throws a pre-compile error: ImplicitDifferentiation -> ImplicitDifferentiationForwardDiffExt. I have tried on different recent versions of Julia, and on both Win 11 and Ubuntu with the same issue. This appears to make the package unusable.

Thanks for reporting this.
I just checked with MacroModelling 0.1.37, Julia 1.11.2 on Mac in an otherwise empty environment and I also get the precompile error. Nonetheless, I can go on and use MacroModelling irrespective of the precompilation error. Could you check whether it works for you?
FYI: the next release will not throw this precompilation error

Hi, thanks for the reply.
Yes, actually the package does seem to be useable despite the error. However, every time one runs "Using MacroModelling" Julia attempts to pre-compile this one dependency and the same error messages appear. I also get the same pre-compile error on the same function when getting to the estimation stage and adding the Zygote package.

glad to hear it works. I'll close the issue then


Add an environment variable to force the use of a particular gRPC adapter

This was implemented in #541.


Add golang runtime metrics e2e test

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[ ] Update the CHANGES log.

Add golang runtime metrics e2e test

Codecov Report

Merging #171 (90d0e66) into master (cbdde90) will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #171      +/-   ##
==========================================
- Coverage   61.88%   61.85%   -0.04%
==========================================
  Files          23       23
  Lines        1220     1219       -1
==========================================
- Hits          755      754       -1
  Misses        405      405
  Partials       60       60

Impacted Files
Coverage Δ

reporter/grpc.go
51.20% <ø> (-0.17%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

miss e2e.yaml

miss e2e.yaml

Sorry, I forgot it, I will update it now


Add authentication support

app.UseEasyRpc("RpcApi", api =>
{
  c.Expose<FloatMathService>().As("FloatMath").Authorize();
  c.Expose<IntMathService>().As("IntMath").Authorize(policy: "PolicyName");
  c.Expose<DoubleMathService>().As("DoubleMath").Authorize(role: "SomeRole");
});

Should be pretty easy to build on top of AspNetCore principle and claim pipeline

This has been implemented


Make task worker pool size adjustable without service restart

What changed?

Make task worker pool size adjustable without service restart
Enforce task processor and scheduler interface contract where if no error is returned for Submit() or TrySubmit(), Ack() or Nack() will eventually be called on the submitted tasks.
Move in memory dynamicconfig client to separate file so that it can be exported and used by other packages in tests.

Why?
Update worker pool size should not require service restart, also a first step towards automatically adjust worker pool size based on loads.

How did you test it?
Local test.

Potential risks
In worst case, task may be lost. Need to reload corresponding shard to resolve.

Coverage increased (+0.03%) to 68.027% when pulling 3dcd86e24fa9c8393da121dba6a59d64eb9042e6 on yycptt:task-processor-contract into 1f0b344cb8099e6aa884f0cf0ef7a33a8aa4ddb0 on uber:master.


Suggestion on font colors in Dashboard

I realized texts like Engagement texts (Green T, Parallel and HeadOn) and getting AS or AS+ is red, wonder if you guys can change it to green so it suits the engagement and Air superiority achieved.
Edit : Somewhat its green now :v
Edit 2 : Looks like the red fonts only shown in PvP, I cant pvp right now since I done all of it, perhaps any of you want to test it? Pretty sure only PvP one only have the red text.

If you know some HTML/CSS and how to inspect element in chrome then you will know why xD
@dragonjet I currently have already fixed this issue in my Battle Mode branch for both horizontal.js and vertical.js but if you want to fix it immediately then just go ahead :sparkles:

Oh LOL. I totally didnt think to check HTML and CSS before hand facewall. Thanks for the quick fix! And no worry, you have all the time o3o/

@bamboo3250 ah I'll just wait for that so I can copy :relieved:


Add error cause details to the log message

Description:
Current log API only supports printing error messages and stack traces of the parent error. it does not consider the cause of the error. Give a way to print full error details with causes in a readable manner.
For more info check ballerina-platform/module-ballerina-http#1193

In the current support, when the user add error logs with error like below.
type ErrorDetails record {|
    int id;
    string user;
|};

type ErrorWithDetails distinct error<ErrorDetails>;

public function main() {
    error err1 = getInternalError();
    error err2 = error("This is an error", err1);
    ErrorWithDetails err = error("This is the main error", err2, id = 123, user = "John");

    log:printError("Error message with causes.", 'error = err);
}

isolated function getInternalError() returns error {
    return error("This is an internal error");
}

log module only prints the log output like below,
time=2022-08-29T15:52:36.482+05:30 level=ERROR module=danesh/temp3_log_error message="Error message with causes." error="This is the main error"

But the expectation is to get full error details with cause like,
time=2022-08-29T15:52:36.482+05:30 level=ERROR module=danesh/temp3_log_error message="Error message with causes." error= "This is the main error {\"id\":123,\"user\":\"John\"}"
stackTrace = ["callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 14"]
cause = [
"This is an error"
at ["callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 13"],
"This is an internal error"
at ["callableName: getInternalError moduleName: user.sample_1.0 fileName: main.bal lineNumber: 20",
"callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 12"]
]

But this error log spans to multiple lines depending on the error causes. Some log processors have problems dealing with this multiline stack traces, because each line is treated like a separated log entry.

Suggestion 01: Introduce new log level config to print full error stack trace.
[ballerina.log]
printErrorDetails = true

In order to maintain backward compatibility, this flag is disabled by default.
when user enable error details,

LogFMT format will print the log event like below in multiple lines.

time=2022-08-29T15:52:36.482+05:30 level=ERROR module=danesh/temp3_log_error message="Error message with causes." error="This is the main error {\"id\":123,\"user\":\"John\"}
        at [callableName: getInternalError moduleName: user.sample_1.0 fileName: main.bal lineNumber: 20,]"
cause="This is an error
        at [callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 13]"
"This is an internal error
        at [callableName: getInternalError moduleName: user.sample_1.0 fileName: main.bal lineNumber: 20],
        [callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 12]"

JSON format will print the log event like below,

{
  "time": "2022-08-29T15:52:36.482+05:30",
  "level": "ERROR",
  "module": "danesh/temp3_log_error",
  "message": "Error message with causes.",
  "error": "This is the main error {\"id\":123,\"user\":\"John\"}\"\n\tat [callableName: getInternalError moduleName: user.sample_1.0 fileName: main.bal lineNumber: 20,]\n",
  "cause": [
    "This is an error\n\tat [callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 13]\n",
    "This is an internal error\n\tat [callableName: getInternalError moduleName: user.sample_1.0 fileName: main.bal lineNumber: 20, callableName: $get$error moduleName: user.sample_1.0.$anonType$_1 fileName: main.bal lineNumber: 12]\n"
  ]
}

@jclark @sameerajayasoma Could you please share your thoughts on the proposed design.

As regards the JSON format, why is the error field a string rather than something with structure?
As regards the text format,

I wouldn't want  a single  log message to be spread over multiple lines
it looks more like a compiler error message than a log message (I don't think these are the same kind of thing)
my log may have thousands of entries; when I want stack traces, I don't think I would necessarily want all entries to have stack traces

+1 for the structured format for the error field.
Following is the structured format I came up with for the full error details,
type StackFrame record {|
    string callableName;
    string? moduleName;
    string fileName;
    int lineNumber;
|};

type Error record {|
    string message;
    error:Detail detail;
    StackFrame[] stackTrace;
|};

type FullErrorDetails record {|
    *Error;
    Error[] causes;
|};

The log output looks like as follows,

JSON format will print the log event like below,

{
  "time": "2023-06-27T16:41:58.641+05:30",
  "level": "ERROR",
  "module": "danesh/temp3_log_error",
  "message": "Error message with causes.",
  "err": {
    "message": "This is the main error",
    "detail": {
      "id": "123",
      "user": "John"
    },
    "stackTrace": [
      {
        "callableName": "main",
        "moduleName": "danesh.temp3_log_error.0",
        "fileName": "main.bal",
        "lineNumber": "14"
      }
    ],
    "causes": [
      {
        "message": "This is an error",
        "detail": {},
        "stackTrace": [
          {
            "callableName": "main",
            "moduleName": "danesh.temp3_log_error.0",
            "fileName": "main.bal",
            "lineNumber": "13"
          }
        ]
      },
      {
        "message": "This is an internal error",
        "detail": {},
        "stackTrace": [
          {
            "callableName": "getInternalError",
            "moduleName": "danesh.temp3_log_error.0",
            "fileName": "main.bal",
            "lineNumber": "21"
          },
          {
            "callableName": "main",
            "moduleName": "danesh.temp3_log_error.0",
            "fileName": "main.bal",
            "lineNumber": "12"
          }
        ]
      }
    ]
  }
}

LogFMT format will print the log event like below in a single line.

time = 2023-06-28T09:23:31.642+05:30 level = ERROR module = danesh/temp3_log_error message = "Error message with causes." error = "{\"message\":\"This is the main error\",\"detail\":{\"id\":123,\"user\":\"John\"},\"stackTrace\":[{\"callableName\":\"main\",\"moduleName\":\"danesh.temp3_log_error.0\",\"fileName\":\"main.bal\",\"lineNumber\":14}],\"causes\":[{\"message\":\"This is an error\",\"detail\":{},\"stackTrace\":[{\"callableName\":\"main\",\"moduleName\":\"danesh.temp3_log_error.0\",\"fileName\":\"main.bal\",\"lineNumber\":13}]},{\"message\":\"This is an internal error\",\"detail\":{},\"stackTrace\":[{\"callableName\":\"getInternalError\",\"moduleName\":\"danesh.temp3_log_error.0\",\"fileName\":\"main.bal\",\"lineNumber\":21},{\"callableName\":\"main\",\"moduleName\":\"danesh.temp3_log_error.0\",\"fileName\":\"main.bal\",\"lineNumber\":12}]}]}"

Other options available for providing this support instead of the configuration params are,
Option 01: Provide a new API to log errors with details as follows,
log:printErrorWithDetails("Error message with causes.", 'error = err);

This API will print the full error details with causes when we pass the error object as shown above. All existing APIs remain the same.
Question: Do we need to support printing error details with other log levels like INFO, DEBUG, etc? If so we need to come up with similar API for other levels as well.
Option 02: Define another special key for printing errors with details as we have for error and stackframes[1].
API will look like below,
public isolated function printError(string msg, error? 'error = (), error:StackFrame[]? stackTrace = (), error? errorWithDetails, *KeyValues keyValues) {
}

So users can pass an error object if they want to print an error with details as follow,
log:printError("Error message with causes.", errorWithDetails = err);

https://github.com/ballerina-platform/module-ballerina-log/blob/master/ballerina/natives.bal#L110

This will only break if the users already have errorWithDetails as a key in their log statements.
Option 03: Break the current behavior and print the error with details when the user passes an error object for error key instead of just the error message as follows,
log:printError("Error message with causes.", `error = err);

This will break the way we print the log output when an error object passes to the error key. If the user only needs the error message to print, he/she needs to change the code to something like below,
log:printError("Error message with causes.", err = err.message());

Among these three options, I prefer going with option 02. As it does not break the user experience. but there are two param keys that accept error objects.

@jclark @sameerajayasoma Could you please share your thoughts?

I prefer option #3 as that is the most familiar approach. Do you think it'll break existing users?

Yes, that is the most familiar approach. Since we are not supporting it in the current version, changing the way we print the error may cause to change in the user code. Otherwise, I am +1 for the option 03

While doing the change, observed we are adding extra spaces in between the key and value in the logFMT log message. This violates the logFMT format. for example,
The previous log message:
time = 2023-07-17T16:29:09.909+05:30 level = INFO module = "" message = "hello" foo = "bar"

The correct log message:
time=2023-07-17T16:29:09.909+05:30 level=INFO module="" message="hello" foo="bar"

We are going to fix this along with the full error details fix


CSS for the signin-banner needs amendment to correctly display a banner

Currently:

a monospace font is not used
the banner read from file is displayed as one long string when it should be treated as pre-formatted text
the width of the form displaying the banner needs to be a little bigger to fit in 80 chars

I've worked around this by creating a /etc/web/authentic-theme/styles.css file and in it having the following content:
html.session_login, body.session_login, form.session_login, .container.session_login {
    font-family: monospace !important;
    white-space: pre;
    background-color: #f0f0f0;
}

.session_login .form-signin-banner, .session_login .form-signin {
    position: relative;
    max-width: 800px;
    margin: 80px auto 0;
    padding: 30px 30px 40px 30px;
    text-align: center;
    color: #444;
    border: 1px solid #ddd;
    -webkit-border-radius: 4px;
    -moz-border-radius: 4px;
    border-radius: 4px;
    background-color: #fff;
    -webkit-box-shadow: 1px 1px 10px rgba(0,0,0,.15);
    box-shadow: 1px 1px 10px rgba(0,0,0,.15);
}

Alright, thank you for your contribution. I'll take a look and add it to the distro for 18.00.
Why having min-width: 800px? Could you send me on the email your banner file so I could have clearer idea what are you trying to do?

Attached the /etc/issue banner file I was trying to display. (issue.std)

Here is the what the signing banner looks like before applying the
styles.css file:

Here is what it looks like after applying the styles.css file:

I changed "max-width: 800px" to allow the text to fit. The original value
of 320px caused lines to be cut off. hmm, I just noticed I lost the red
colouring of text and the warning triangle but that is ok.

On 29 April 2016 at 13:38, Ilia Rostovtsev<EMAIL_ADDRESS>wrote:

Alright, thank you for your contribution. I'll take a look and add it to
the distro for 18.00.
Why having min-width: 800px? Could you send me on the email your banner
file so I could have clearer idea what are you trying to do?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/qooob/authentic-theme/issues/451#issuecomment-215700068

--
Jinesh ChoksiDevOps Engineer
+44 (0)<PHONE_NUMBER>
ALGOMI LIMITEDOne America Square, 17 Crosswall, London EC3N 2LB. Service
Desk (UK): +44 (0)<PHONE_NUMBER>
[image: Algomi Limited] http://www.algomi.com/
*This email and its contents, including all accompanying communications and
attachments, is strictly confidential and only for the intended
recipient(s). If you have received this email by mistake, please notify the
sender immediately and do not disclose its contents. Any retention, use or
disclosure of this email or its contents not expressly authorised by Algomi
Limited is prohibited. **Algomi Limited is incorporated in England and
Wales under company number. 08111389 with its registered office at One
America Square, 17 Crosswall, London EC3N 2LB, United Kingdom. Algomi
Limited and its affiliates are not regulated, nor are they registered as
broker-dealers or investment advisors in any jurisdiction. **Algomi
Corporation is incorporated in the State of Delaware, USA under file number
5425455. **Algomi Asia Pacific Limited is incorporated in Hong Kong with
company number 2200830. *Please visit http://www.algomi.com/contact-us
http://www.algomi.com/contact-us for the contact information of
Algomi’s offices worldwide.

Hi, again! ;)
I took a closer look. Well even though I like your banner, I will not be able to merge all of your css rules. Having monospace font, is good, I will do, the rest is very specific and this is why we have extensions for - to make everyone happy.