Add webhook destination subnet filtering

Starts to add internal filtering of webhook destinations such that private IPs are blocked by default.
This is still a draft. Still to do is to

Improve error handling
Cleanup and split worker into separate files
Make the allowed private IP subnets configurable
Add unit testing

Replaced with #704


Specification for direct-to-device messages

Based on Mark's doc at https://docs.google.com/document/d/1v8lxwKi1y074ggYYAjZTHN2EJYBFPtnot2nM3vVWTR0.
One particular question is over what these things should be called. At once point the idea was to call them direct-to-device messages, but then that sounded a bit like they would skip the HS altogether. I've gone with "store-and-forward" here, but will take suggestions.

Fixes https://matrix.org/jira/browse/SPEC-138.

@ara4n: PTAL?

lgtm, sorry for sitting on it


resolve warnings

npm: Remove unnecessary react-helmet plugin
gatsby-node: Update query
pages/index: Update query

Same issue as https://github.com/parcel-bundler/watcher/issues/152


Improve UI

The user interface will be improved updating RedKite jquery plugins, rewriting javascript code, implementing better validation for forms and more

In addition the interface has been improved introducing knockoutjs framework


Enable video after start only audio call

I'm sorry can not find in the documentation
how to request a new SDP with the included video and send re-invite, without completing the current only audio call
I start a call like this
var options = {
        media: {
            constraints: {
                audio: {
                    optional:[
                        {googEchoCancellation:true},
                        {googAutoGainControl:true},
                        {googNoiseSuppression:true},
                        {googHighpassFilter:true},
                        {googAudioMirroring:false},
                        {googNoiseSuppression2:true},
                        {googEchoCancellation2:true},
                        {googAutoGainControl2:true},
                        {googDucking:false}
                    ]
                },
                video: false
            },
            render: {
                remote: this.getRemoteVideo(),
                local: this.getLocalVideo()
            }
        }
    };

var session = ua.invite('sip:'+number+'@'+host, options);

@Slawutich you can add a MediaStream to the PeerConnection and then call session.sendReinvite. You can get at the PeerConnection with session.mediaHandler.peerConnection.

Thank you very much
but it seems that for the new video stream are no candidates in the SDP
I'll look again, I can somewhere made a mistake

I think I'm facing the same problem as you:  the re-INVITE is being sent before the video candidates are collected.
Before I go digging deep into the code, did you manage to find the problem?

no
I was very busy, so it was not possible to solve the problem

Thanks @Slawutich I'll try to get time to delve into the code.

onIceCompleted does not wait for the selection of candidates and immediately called
var mediaHandler = session.mediaHandler;

mediaHandler.mediaStreamManager.acquire({media: {
	constraints: {
		audio: false,
		video: true
	}
}})
.then(function acquireSucceeded(streams) {
	/**
	 * add to local media, after the end of call stream must be deleted
	 */
	mediaHandler.mediaStreamManager.release(mediaHandler.localMedia);
	mediaHandler.peerConnection.removeStream(mediaHandler.localMedia[0]);
	mediaHandler.localMedia = streams;

	mediaHandler.addStreams(streams)
	.then(function streamAdditionSucceeded(streams){
		/**
		 * reset onIceCompleted, after setLocalDescription he immediately called
		 * without waiting for the collection of candidates
		 */
		mediaHandler.onIceCompleted = SIP.Utils.defer();
		mediaHandler.onIceCompleted.promise.then(function(pc) {
			mediaHandler.emit('iceGatheringComplete', pc);
			if (mediaHandler.iceCheckingTimer) {
				SIP.Timers.clearTimeout(mediaHandler.iceCheckingTimer);
				mediaHandler.iceCheckingTimer = null;
			}
		});

		session.sendReinvite();
	},
	function streamAdditionSucceededError(err){
		console.log('streamAdditionSucceededError: ', err)
	});

	return streams;

}, function acquireFailed(err) {
	console.log('acquireFailed: ', err);
})

but remote video still does not appear, only the local video

Any news about this issue? PR looked promissing, especially in terms of api ( session.sendReinvite({media: { constraints: { video: true } }))
renegotiation and multiple streams work good in both Chrome and Firefox (although they do not interop well due to Plan B being used in Chrome and Unified plan in Firefox). I do not know for sure which one is used in FS/Asterisk, so may be this was the problem of PR not working in Chrome?
This is a very desired functionality nowadays - to be able to enable/disable video without audio stream interrupt.

I asked on a question on the WebRTC discussion forum: https://groups.google.com/forum/#!topic/discuss-webrtc/tcLBoTOpxtI
It looks like I need to listen on the stream's onaddtrack() event.  Hopefully, I can do that today.

New media may be added in one of the following ways:

PC.addStream(new_stream), PC.removeStream(old_stream) - new stream (with new tracks) is added and old one is then removed. This is supported currently in Chrome, but will be removed in the near future because these methods are not present in the spec any more. This is also a bad approach because audio strem gets interruped for some time, so it may lead to a minor sound lag. PC.onaddstream should be listened then.
PC.addTrack() - current spec, suported in Firefox and in Chrome 49 (hopefully: https://code.google.com/p/chromium/issues/detail?id=465349). Here you should listen to PS.ontrack
Stream.addTrack() - supported in Chrome currently (Firefox already implemented in v44: https://developer.mozilla.org/en-US/Firefox/Releases/44). I did not test this approach ever.

So there is not single way which will work in both Chrome and Firefox today (: I think you should look into implementing both 2 and 3, but both need testing with FS. May be some sdp conversions will be required on client side depending on the current browser. You could look into this project for some useful information: https://github.com/jitsi/sdp-interop

@soulhunter1987 I'm only interested in the remote stream which gets automatically added to the stream by the PeerConnection which makes things easier!  All I have to do is call the MediaHandler render() function whenever a onaddstream or onaddtrack event is received.

It is if new track has been added by the other leg, and what if you want to add track (enable video) by yourself? You should anyway call PC.addTrack/Stream.addTrack() then.

SIP.js already handled the addition of the local track.  I must admit I don't know how it does it.  But, it does work!

And what is the syntax for doing that? I did not see it anywhere in the docs...

I don't know how it does it - it just does!!!!
The docs could do with a bit of an update in the area of the re-INVITE.

I don't know how it does it - it just does!!!!

It is not possible ) I searched through sip.js source code and only found calls to PC.addStream(), so it may (possibly) send several streams (not tracks) at one time. One of them may contain only audio track and the second one (if two streams are send) only video track. That's OK, but you would have to render both of them in separate media elements (and they will not be synchronized properly), because only one stream may be renderered by audio/video element at a time. So this is not so easy to solve currently. I suggest you to try using Stream.addTrack()/removeTrack() and make tests with current Chrome and FF 44 Beta (which adds support for stream.addTrack/stream.removeTrack). But all my advices are related to raw WebRTC only (broswer to browser direct call), there may be other issues related to WebRTC multistream/multitrack support in FS/Asterisk.

Hm, I see that you made new changes to PR - and you say it really works now? If it does - it seems that new aquired stream is added locally with PC.addStream(), but when some media it added by the other leg - FS sends the SAME stream with just new track in the SDP. That's some king of a big mess... But I admit that it might work according to current browser support.

Can you retest this with our new renegotiation guide and  0.8.0? We think that should resolve this issue. If we do not hear back by 11/1/17 we will close this issue. Thanks.

As stated in previous post, closing due to inactivity and proposed solution in 0.8.X. If this turns out to still be an issue, please re-open.


fix: fix the tray icon not showing after package

BTY, maybe we need add a exit button.

BTY, maybe we need add a exit button.

🎉 thanks for this


Performance problem: -cmatch prevents automatic caching of on-demand-compiled regexes

Note: This isn't a bug per se, but a performance issue.
Note that -creplace and switch -regex -casesensitive appear NOT to be affected.
Steps to reproduce
$strArray = , 'foo' * 1e5  # an array with N elements containing 'foo'
# Contrast the performance of -match vs. -cmatch
$scriptBlocks =
 { foreach ($str in $strArray) { $null = $str -match  'f?(o)' } },
 { foreach ($str in $strArray) { $null = $str -cmatch 'f?(o)' } }

$scriptBlocks | % { (Measure-Command $_).TotalSeconds }

Expected behavior
Roughly the same execution time in both cases - -cmatch should actually be slightly faster.
Actual behavior
Sample result from my macOS 10.14.3 machine:
0.6651756
1.3135713

That is, -cmatch unexpectedly took about twice as long.
That this may be related to unexpectedly bypassing  .NET's  automatic caching of on-demand-compiled regexes is evidenced by the fact that if you force caching as follows, the problem goes away:
# Forces caching of the regex; running the above code again then performs as expected.
$null = switch -casesensitive -regex ($null) { 'f?(o)' { } }

Environment data
PowerShell Core v6.2.0-preview.4 on macOS 10.14.2
PowerShell Core v6.2.0-preview.4 on Ubuntu 18.04.1 LTS
PowerShell Core v6.2.0-preview.4 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.471)
Windows PowerShell v5.1.17134.407 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.471)

It's funny :-)


Mouse and keyboard lag sometimes

Hello. About my problem: sometimes my mouse and keyboard are lagging - it means that for example I'm playing in something and there is 3-5 sec lag and I had 'W' and 'A' pressed. When this lag appear, everything I'm clicking isn't "recorded" by system, but still I "have" (physically I don't) pressed 'W' and 'A'. The same goes to mouse. I've been searching a lot. Finally i got this. First I thought that AIMP is the problem giver. But after turning off this plugin - there is no lags.
So... This plugin makes my computer lag and this is very annoying.
About specs: Intel i7 2600, 8GB RAM, GTX 460, 850W - so this is not my hardware problem.
Waiting for response.
PS Also I scanned my system for viruses - the system is clear.
Sorry for my English, but I think you can easily understand what I wrote.
Regards.

Hi,
There was similar problem in the past. Unfortunately it was not solved that time. As I can't reproduce the problem on my machines, I need all information available to have a chance to solve it. Please response to this request.
What Windows version do you use?
Thanks,
Alexey

Windows 7 SP1 x64.
Now I've enabled debug, will see later what's happening.
I have a lot of playlists - about 30/40?
For now when I had lag I've looked into logs and there is nothing special, becouse timing is < 0.1sec in most cases.

So... There is nothing interesting in logs. Every callback is executed in less than 1.5 sec.

Same issue here, i have a lost of playlist and it happens sometimes exactly the same way

@a0ivanov so? Found anything?

No. I had no time for the plugin from long May weekend.
Did you check how does it work on empty playlist? I'm interested if it's reproduced when tens playlists are opened or always.

Ok. I've deleted all my playlists and I'm using only 2.
For now I don't have lags. If any lag would be - I will edit this post.

You got something? I like to have many playlists, but then I have these lags (seems to randomly).

33k songs in one playlist, It happens to me almost everytime

Sorry, I had no time to work on the project, and it seems nothing will
change in near future.
16 дек. 2015 г. 19:03 пользователь "Damianoseczek"<EMAIL_ADDRESS>написал:

You got something? I like to have many playlists, but then I have these
lags (seems to randomly).
—
Reply to this email directly or view it on GitHub
https://github.com/a0ivanov/aimp-control-plugin/issues/4#issuecomment-165155816
.

You know maybe what could cause this?

It could be some unexpected too often refresh of track info, which
overloads system hard drive queue which cause OS level lags.
16 дек. 2015 г. 19:47 пользователь "Damianoseczek"<EMAIL_ADDRESS>написал:

You know maybe what could cause this?
—
Reply to this email directly or view it on GitHub
https://github.com/a0ivanov/aimp-control-plugin/issues/4#issuecomment-165171339
.


Change 100-continue behavior to send when Body has been polled

Use of the 100 (Continue) Status (https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3):

The purpose of the 100 (Continue) status (see section 10.1.1) is to allow a client that is sending a request message with a request body to determine if the origin server is willing to accept the request (based on the request headers) before the client sends the request body. In some cases, it might either be inappropriate or highly inefficient for the client to send the body if the server will reject the message without looking at the body.

In my case, for example, I can upload file or reject already uploaded file in one atomic request.
Current state:

0.9.x - handler can make desigion and allow/reject client request by additional check_continue method. This method is not supported by https://github.com/iron/iron and https://github.com/nickel-org/nickel.rs. Also this method require duplicate logic (it not called on HTTP/1.0) for routing and check permissions;
master - looks like this feature is forgotten.

I think, much better to send "100 Continue" status on first Body::read() call (similar logic implemented by https://github.com/frewsxcv/tiny-http project).
In this case no additional logic/callback needed for make desigion about "100 Continue":

If you call Body::read() method, then you needs to request body and hyper send to client "100 Continue";
If you have anought information to send final status without body, then hyper send status without "100 Continue".

I try to change logic on 0.9.x branch by send "100 Continue" status of first read call, but I failed :(

master - looks like this feature is forgotten.

Not so much forgotten, I just wanted to come up with a better design. What you outline here seems like it should work on master:
fn on_request(&mut self, req: Request) -> Next {
    match request.headers().get() {
        Some(&ContentLength(len)) if len > 10_000_000 => {
            self.status = StatusCode::PayloadTooLarge;
            Next::write()
        },
        None => {
            self.status = StatusCode::LengthRequired;
            Next::write()
        },
        Some(&ContentLength(len)) => {
            self.length = len;
            // hyper looks for Expect-100, and sends 100 status because of this read()
            Next::read()
        }
    }
}

Yes, about this behavior and I had in mind.
I'm sorry if offended. Loss (sometimes temporary) features on full code redesign is a normal and I did not mean anything bad. On the contrary, from my point of view this is a very good time to remind about corner cases :)

Not offended at all! Thanks for helping think this issue through.
On Mon, Jun 20, 2016, 11:42 PM Artem V. Navrotskiy<EMAIL_ADDRESS>wrote:

Yes, about this behavior and I had in mind.
I'm sorry if offended. Loss (sometimes temporary) features on full code
redesign is a normal and I did not mean anything bad. On the contrary, from
my point of view this is a very good time to remind about corner cases :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/hyperium/hyper/issues/838#issuecomment-227353726, or mute
the thread
https://github.com/notifications/unsubscribe/AADJF2kg53rTezjlZGbnHTCzmF2ekBXDks5qN4ffgaJpZM4I5p6B
.

Hey all,
I'm looking at implementing this on master with the tokio integration.
I've literally only been using rust a few days so it can be hard to follow through the code with inferred type assignments and generics mixed together.
Here is what I have so far.
diff --git a/src/http/conn.rs b/src/http/conn.rs
index 7954fadb..3e90c8ba 100644
--- a/src/http/conn.rs
+++ b/src/http/conn.rs
@@ -116,6 +116,7 @@ impl<I: Io, T: Http1Transaction, K: KeepAlive> Conn<I, T, K> {
                     }
                 };
                 self.state.busy();
+                let wants_continue = head.expecting_continue();
                 let wants_keep_alive = head.should_keep_alive();
                 self.state.keep_alive &= wants_keep_alive;
                 let (body, reading) = if decoder.is_eof() {
@@ -124,6 +125,10 @@ impl<I: Io, T: Http1Transaction, K: KeepAlive> Conn<I, T, K> {
                     (true, Reading::Body(decoder))
                 };
                 self.state.reading = reading;
+                if wants_continue {
+                    self.state.reading = Reading::Init;
+                }
+
                 return Ok(Async::Ready(Some(Frame::Message { message: head, body: body })));
             },
             _ => {
@@ -674,6 +679,8 @@ mod tests {
             Ok(())
         }).wait();
     }
+
+
     #[test]
     fn test_conn_closed_write() {
         let io = AsyncIo::new_buf(vec![], 0);
diff --git a/src/http/mod.rs b/src/http/mod.rs
index 13c50119..a0ea5a27 100644
--- a/src/http/mod.rs
+++ b/src/http/mod.rs
@@ -2,7 +2,7 @@
 use std::borrow::Cow;
 use std::fmt;

-use header::{Connection, ConnectionOption};
+use header::{Connection, ConnectionOption, Expect};
 use header::Headers;
 use method::Method;
 use status::StatusCode;
@@ -68,6 +68,10 @@ impl<S> MessageHead<S> {
     pub fn should_keep_alive(&self) -> bool {
         should_keep_alive(self.version, &self.headers)
     }
+
+    pub fn expecting_continue(&self) -> bool {
+        expecting_continue(self.version, &self.headers)
+    }
 }

 /// The raw status code and reason-phrase.
@@ -115,6 +119,16 @@ pub fn should_keep_alive(version: HttpVersion, headers: &Headers) -> bool {
     ret
 }

+#[inline]
+pub fn expecting_continue(version: HttpVersion, headers: &Headers) -> bool {
+    let ret = match (version, headers.get::<Expect>()) {
+        (Http11, Some(expect)) if expect == &Expect::Continue => true,
+        _ => false
+    };
+    trace!("expecting_continue(version={:?}, header={:?}) = {:?}", version, headers.get::<Expect>(), ret);
+    ret
+}
+
 #[derive(Debug)]
 pub enum ServerTransaction {}

The function I added to http/mod.rs seems correct and with RUST_LOG=trace I get
TRACE:hyper::http                               : expecting_continue(version=Http11, header=Some(Continue)) = true

Ignoring the changes I made to conn.rs just for testing.
Could you give me some direction?
I can gather that we would be using Sink and not a Stream for this but I am not sure about what would be the best way to go about integrating this into the code base.
I'm am assuming this is a good time for api breakages.

@martell You've only been using Rust a couple days? That makes this work even more impressive! I'd suggest opening a Pull Request with your work, it makes it easier to discuss changes.
As for your current diff, it definitely looks like the right direction. I imagine we would want to add some new states, of which I'm not entirely certain the best combination at the moment. I'll think more, but I can describe the expected behavior.
We would need to save somehow that the request would like a 100-continue response. We would also need a way to know whether we should respond with it, and also to know that we already have. A way we could decide to write 100-continue is if the user decides that they wish to keep reading from the body, we should flush that response. If the user decides they don't want to read from the body, they'll just return a Response, and we can just write that.
if !req.headers().has::<ContentLength>()
    return Response::new().with_status(StatusCode::LengthRequired);
} else {
    req.body().for_each(|chunk| something(chunk))
}

In the above example, if the request didn't have a Content-Length header, the server wants to reject it immediately. In this case, hyper should not have responded with 100-continue. We can do this because Conn::poll() will be called again if the user tries to poll on the request body (the req.body().for_each).
So, we can record that an expectation exists, and then in Conn::poll(), if were in this expecting state, we can write (and it needs to be flushed immediately) the 100-continue response (record that we've since written that), and then try to keep reading the body. If Conn::start_send() is called to write a response, we can just remove the expectation state, since it no longer matters.
Does this all make sense?

Most of that makes sense to me.
I have setup a WIP PR for this
We have to be compliant with the rfc spec to only send the 100-continue when we get the Content-Length.
The use case for this is typically for a REST API server and we only really want to send the body after Auth has happened based on the header data, probably an oauth2 bearer token etc.
So when the user requests the body is an ideal time to send the 100-continue
I see the Conn::poll() in conn.rs
I'm just not sure what states yet either but can play around with that to achieve the goal and refactor, from initial reading I see
Reading::Init
Reading::Body(..) in the function `can_read_body`
Reading::KeepAlive
Reading::Close

I think we want a state like Reading::Head because we are beyond the init stage but are not onto reading the body yet.
My biggest problem is I can't find where I can where to write to the SINK, this is probably a combination of my lack of rust knowledge and the fact that this is not my own code base :)

I've gone back and forth whether it should be Reading::Expect or Writing::Expect. It kind of affects both. Whichever is chosen, we'd want to update read_body to check that we've written the 100-continue, and write_head would want to update the state that that there is no need to send a 100-continue, since some other response is already being sent.
The content is written via self.io.buffer(bytes) and then flushed with self.io.flush().

I have a PR open to implement this: #1232.

@sfackler Added support for immediately replying with 100-continue
This is a good short term fix for getting support in tree.
Ideally we would only send this on the attempted read of the body.
Discussions on IRC lead to some work being done in tokio which needed a new future Sink to support this.
https://github.com/alexcrichton/futures-rs/pull/414
It is still up in the air about how we are going to best solve this.
I wanted to get some context into this thread so we can proceed.
@seanmonstar did mention he might have an idea about how to help this in hyper?

This should no longer be blocked on any dependencies, on master. We have our own dispatcher, and use a custom channel for the body.
I think it wouldn't be too hard to tie want into the body channel, to know when and if the user has polled for data on the body.

@AlexDvorak no, that's adding the 100 Continue response to the write buffer immediately, right after parsing. This issue is describing that it shouldn't add it immediately, but instead only if the user polls the Body. That way, if the user rejects the request for some reason, the client doesn't see a weird HTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 400 Bad Request.

So, would modification to the Body struct like so be fine?
pub struct Body {
    kind: Kind,
    /// Keep the extra bits in an `Option<Box<Extra>>`, so that
    /// Body stays small in the common case (no extras needed).
    extra: Option<Box<Extra>>,
    UserHasPolled: bool,
}

Or should the modification be happenning elsewhere?

That wouldn't work, since the Conn no longer has the Body side. When there's a request body, a channel is made (Body::channel). The channel implementation could be where this information is stored. The client::dispatcher already has a concept of knowing when the receiver side has been polled, by making use of want.

what are you reffering to by channel implementation?


[bug][docs] my_ds -> my_df

Looks like it is supposed to be my_df from the example code further down the page

Can one of the admins verify this patch?
Admins can comment ok to test to allow this one PR to run or add to allowlist to allow all future PRs from the same author to run.

ok to test


Bounce back has strange behavior on chrome 45

When doing a strang flicker movement with my mouse on chrome desktop, the bounce does a weird jump and then does the bounce animation.
I can also get this behavior in the demo:
http://lab.cubiq.org/iscroll5/demos/simple/
It's like it does the bounce animation from the wrong direction.
Video of this behavior:
http://www.youtube.com/watch?v=Tv_-IZ1xF7w
My code:
        scroller = new IScroll(element[0], {
          mouseWheel:            mouseWheel,
          scrollbars:            needsScrollbar ? 'custom' : false,
          scrollX:               horizontal,
          scrollY:               !horizontal,
          shrinkScrollbars:      'clip',
          interactiveScrollbars: true,
          eventPassthrough:      true,
          bounceEasing:          'back'
        });

The video would be more understandable if there were different images
But I get what you mean, with the pace of iScroll development, your best bet is probably to manually modify/improve the bounce routine
Is this on a low performance device by the way? (+ which OS?) - generally, low performance causes issues like these, where slowing due to low performance sometimes reflect as anomalies / speed bursts etc.

It's a high powered device:
Desktop windows 10, i7 CPU.
I'll see if I can redo the video with higher framerate  and different images

I experience the issue on http://lab.cubiq.org/iscroll5/demos/simple/ too - using OSX/Latest Chrome, there is a weird pause and then a bounce
I wonder if this: https://github.com/cubiq/iscroll/pull/973 could fix the issue

Here is a script with the above fix and more applied, so that you can use for testing: https://www.geobird.com/js/iscroll/iscroll-5.1.3.c18.js?v=5

Yes that fixed it.
Hopefully that pull-request will make it in soon :D

Interesting to know, thanks for the info
Don't hold your breath for the PR, manually apply your fixes to iScroll :)
(iScroll seems to have major releases, so I'm guessing the PR wouldn't apply to the next release anyway)

I tried the pull request but I get the same behavior ..

I think iScroll is not supporting since last year (latest commit)


Paper Key

I have a Keybase account, it was created in 2019. I have full email, password, github link. I'm still logged in on the website, how can I log into the app if I lose my paper key?
I still use the old device when I first signed up for the app, it has software corrupt and I have to restore it.

 Once you've logged into keybase with your password once, you will need to use a key to log in afterwards (paper, device or other).

 Once you've logged into keybase with your password once, you will need to use a key to log in afterwards (paper, device or other).


test: adjustments for Windows

Adjust the invocation on Windows to improve the Index test coverage.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

@swift-ci please smoke test and merge


v0.17.0 abnormal command line break

v0.17.0 abnormal command line break:
I upgrade openpai from v0.15.0 to 0.17.0 and resubmit a task from previous version.
It seems that the commands in the new version do not break line.
Here is an example:
My config file is :
protocolVersion: 2
name: just-test_3f34372a
type: job
jobRetryCount: 0
prerequisites:
  - type: dockerimage
    uri: openpai/tensorflow-py36-cu90
    name: docker_image_0
taskRoles:
  taskrole:
    instances: 1
    completion:
      minFailedInstances: 1
      minSucceededInstances: -1
    taskRetryCount: 0
    dockerImage: docker_image_0
    resourcePerInstance:
      gpu: 1
      cpu: 4
      memoryMB: 8192
    commands:
      - pip install hdfs
      - touch ~/.hdfscli.cfg
      - 'echo ''[dev.alias]'' >> ~/.hdfscli.cfg'
      - 'echo ''url = http://<IP_ADDRESS>:50070'' >> ~/.hdfscli.cfg'
      - mkdir /data
      - hdfscli download --alias=dev -t 20 /datasets/kitti_odometry.tar.gz /data
      - nvidia-smi
defaults:
  virtualCluster: default

and I get error of
errorLogs:
  user: |

    Usage:
    pip install [options] <requirement specifier> [package-index-options] ...
    pip install [options] -r <requirements file> [package-index-options] ...
    pip install [options] [-e] <vcs project url> ...
    pip install [options] [-e] <local project path> ...
    pip install [options] <archive url/path> ...

    no such option: --alias

OpenPAI Environment:

OpenPAI version: v0.17.0
OS: Ubuntu 16.04.6 LTS
Kernel: Linux ubuntu 4.4.0-142-generic
Hardware:
machine-sku: GENERIC:   mem: 250   gpu:     type: 2080ti     count: 8   cpu:     vcore: 64   os: ubuntu16.04

@shuangjiexu  Can you provide us the full logs, includes init.log.
You can find use browse log folder to get all logs

Thanks for your replay @Binyang2014
I do not have Browse log folder button but have Go to yarn tracking page button,
I put all logs found into a github project:
https://github.com/shuangjiexu/pai-logs

Thanks @shuangjiexu, it's a regression bug. Now PAI support pure k8s version and yarn version. When developing k8s version, seems we break some features in yarn version...
For workaround, you can use && to contact all your commands, such as:
    commands:
      - >-
        pip install hdfs && touch ~/.hdfscli.cfg && echo '[dev.alias]' >>
        ~/.hdfscli.cfg && echo 'url = http://<IP_ADDRESS>:50070' >> ~/.hdfscli.cfg
        && mkdir /data && hdfscli download --alias=dev -t 20
        /datasets/kitti_odometry.tar.gz /data && nvidia-smi

We will fix it ASAP

Thanks @Binyang2014. use && can skip this issue.
I wonder if there is a way to config a pure k8s version when deploying? Could you please give me some documentation or guidelines? Thanks!

We use kubespray to deploy k8s first, then deploy pai-service. Here is a doc about this: https://github.com/microsoft/pai/tree/master/contrib/kubespray

I got it.
Thanks again for your reply.

I got it.
Thanks again for your reply.

Closing the issue as the question seems been addressed, feel free to reopen if it is not.


Add an option to upload resources at a namespace level

Description
Changes proposed in this pull request:

Add an option to upload resources at a namespace level
...
...

Related issue(s)

Resolves #1131

/retest


[FEATURE] Move the tests to hosted machines

Description

Right now, we have the tests on a DSVM that is running 24/7. These tests run sequentially.
We want to change this and run the test in hosted VMs and in parallel. We should have a matrix that tests python 3.6 and 3.7
Expected behavior with the suggested feature

Other Comments

@laserprec added you as owner

Non-gpu tests are moved to Github-hosted machines with PR #1494


Timit_asr dataset cannot be previewed recently

Dataset viewer issue for 'timit_asr'
Link: https://huggingface.co/datasets/timit_asr
Issue: The timit-asr dataset cannot be previewed recently.
Am I the one who added this dataset ? Yes-No
No

Thanks for reporting. The bug has already been detected, and we hope to fix it soon.

TIMIT is now a dataset that requires manual download, see #4145
Therefore it might take a bit more time to fix it

Thank you for your quickly response. Exactly, I also found the manual download issue in the morning. But when I used list_datasets() to check the available datasets, 'timit_asr' is still in the list. So I am a little bit confused. If 'timit_asr' need to be manually downloaded, does that mean we can not automatically downloaded it any more in the future?

Yes exactly. If you try to load the dataset it will ask you to download it manually first, and to pass the downloaded and extracted data like load_dataset("timir_asr", data_dir="path/to/extracted/data")
The URL we were using was coming from a host that doesn't have the permission to redistribute the data, and the dataset owners (LDC) notified us about it.

I downloaded the timit_asr data and unzipped. But I can't run my code. Could you resolve this problem for me? Thanks


Refactor HeaderView to access parentReportAction from props

Fixed Issues
Part of https://github.com/Expensify/App/issues/27262
Tests

Go into any chat and start a thread
Post a message in the thread
Click the three-dot menu in the chat header
Select Leave
Go back into the thread
Verify there is a Join button in the chat header

[ ] Verify that no errors appear in the JS console

Offline tests
Same as tests
QA Steps
Same as tests

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome NOTE: Unable to test due to broken emulator
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

I'm pulling this out of draft, but it looks like iOS builds are currently broken. We are discussing it here in Slack. Feel free to review this however.

I think @jjcoffee should be assigned to this for the C+ review?

Sounds good, thank you for clarifying! I'll leave that up to @tgolen for what he wants for the PR. Let me know if you want an Expensify engineer to review this or if you want a C+ review first Tim 🙂

@aimane-chnaif that would be great if you could please review and test this.

Tim can you complete the author checklist so we have fully green checks? Then we're good to merge.

Yeah, sorry about that. I will need to see if I am able to build the app yet. I haven't checked yet to see if there was a solution for the error with Flipper that was causing the build issue. I'll update this later today with the result.

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/blimpich in version: 1.4.52-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/Beamanator in version: 1.4.52-6 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Question : About getId

Hi,
I have a doubt going through the implementation getId of google
https://github.com/thephpleague/oauth2-google/blob/1c6e59c801cbc0ba7942a7cd84f859bdb8ba786e/src/Provider/GoogleResourceOwner.php#L58
github
https://github.com/thephpleague/oauth2-github/blob/5f48034ee21bb025b4710f9e2e587a9a094bc9ff/src/Provider/GithubResourceOwner.php#L53
and that of the oauth2 client https://github.com/thephpleague/oauth2-client/blob/6f837e21c1777e619234b0382ce8e6ca3d1d72bf/src/Provider/GenericResourceOwner.php#L49
Isn't that should be $this->resourceOwnerId than $this->response[$this->resourceOwnerId]; .
Also it may be good if we can unify the resourceOwnerId and id of getId .
This came across when I was looking at the toArray interface https://github.com/thephpleague/oauth2-client/blob/2950326f8701cee971da47affa96c2100037a651/src/Provider/ResourceOwnerInterface.php#L34  . I am also interested to hear if there is some unified key names that will be across the providers ? For one in google have https://github.com/thephpleague/oauth2-google/blob/1c6e59c801cbc0ba7942a7cd84f859bdb8ba786e/src/Provider/GoogleResourceOwner.php#L116 , but the others basically seems sending the response array itself.
Thank you

Regarding GenericResourceOwner the value of $this->resourceOwnerId is the key, not the value, so the current code is accurate.
The code for the Github owner is actually incorrect. I've opened thephpleague/oauth2-github#1 to address that issue.
In regards to the toArray method, there are no assumptions made about what the return value will be, only that it will return all of the details available about the resource owner. That said, the Google provider should probably be modified to do less manipulation of key names.

Thank you for making it clear @shadowhand . The use case of toArray key was basically for https://github.com/harikt/Aura.Auth/blob/usability/src/Adapter/LeagueOAuth2Adapter.php#L53 , so that it guarantees there will be things like email, name etc. I know it also depends on the providers. But even if there is nothing provided by provider some sort of functionality will be nice to have.
Thank you.

so that it guarantees there will be things like email, name etc.

There are absolutely no guarantees about any information being available from the provider. Most providers will include some kind of identifier, which is what we expose via the getId method. Nothing else is assumed to be available with every provider.

All 9 of the providers that I maintain have been updated per these discussions. Thanks for surfacing!


Explicit way to mark attribute as optional

I think it would be very helpful to mark a resource atributte as optional explicitly, don't you?

If you refer to the new MSON syntax as promoted in #135, marking attribute as optional is already in the spec and as far as I know, it is already supported also by parser.

I've reading the docs for a while and I didn't know about MSON, thanks!
So the way should be to write a MSON for the object and explicit its required attributes (as optional is the default), is it?

So the way to go should be to write a MSON for the object and explicit its required attributes (as optional is the default), is it?

Correct! But you can describe just one of the attributes. Generally the MSON / attributes description is about what you MAY get and if you get it what it means not what you WILL get (unless it is required obviously)

Great, thank you!


Add alloc free override functions for Windows

This will allow users of the library to pool allocations

Thanks for looking at the code and thinking about how to improve it.
I don't see how this would work as there are often multiple users of the SymCrypt module in the same address space. If one caller overrides the memory allocations to get a benefit, then it will very often lead to problems when other callers in the same address space also use SymCrypt and their allocations also get re-routed. I've seen a library that had the allocation parameters on Thread-Local-Storage, but that fails if you have worker threads where the same thread does work for multiple high-level modules in the address space, and the low-level code that actually calls SymCrypt doesn't know about any of that.
SymCrypt already works hard to minimize the allocations needed. We only use allocations for asymmetric operations, which are relatively expensive already making the overhead small. We also limit ourselves to one allocation per operation; an RSA signature will do a single allocation for the whole signature operation. I don't see any significant benefit possible with making the allocations configurable.
Note that the SymCrypt static-link library does use callback functions allowing the caller to implement the memory allocation.

@NielsFerguson On my i9 machine 10% of the time is spent is allocating the 18kb scratch space that is needed. So this is reasonably large improvement. It goes from 10.5 micro seconds to about 9.1 microseconds with no heap fragmentation occuring.
I'd also like to propose we make SymCrypt faster for validating signatures that are found commonly on the internet, namely SHA256 signatures seen in things like AAD tokens.
For example, take a look at this code I put out there that does stack allocation of the 18kb scratch space, does not need allocate memory to fix up the padding and is able to do it given we know the 224 bytes out of the 256.
https://gist.github.com/mjsabby/b87b629fa902cd641b955892b1ab9604#file-verifyrs256jwtsignature-cs-L179
What are your thoughts?
I'm trying to make SymCrypt stack allocation only and not have to pay for PKCS1 padding removal/addition.
I only care about token validation.

We had an internal discussion about this back in April, and the conclusion we came to among the SymCrypt team was that we do not intend to change this. We were also unable to reproduce the profiling results mentioned above. I'm going to close this for now, but @mjsabby feel free to resume the internal email thread if you'd like to continue discussing this.


Make compatible with older versions of Node

mockyeah's use of latest ECMAScript features make incompatible with older versions of node. We should consider increasing its compatibility so more people can benefit from its use.
Done when:

A minimum Node version is determined.
CI automation is implemented to assert compatibility with multiple Node versions (e.g. https://github.com/victorbjelkholm/autochecker)
mockyeah is updated to support the lowest agreed upon version of Node.

Could transpile with Webpack and use babel-preset-env.

With pr https://github.com/ryanricard/mockyeah/pull/53, Travis CI configuration has been updated to execute automation against the latest version of the following respective versions of Node: 8, 6, and 4. This should accomplish this issue for now. Feel free add to this thread if you believe there is more that should be done to support older versions of Node.


Inconsistency in Image naming between New-NavContainer and New-BCImage

Describe the issue
This is not an issue, probably more like a feature request.
New-NavContainer together with -ImageName creates an image which includes the Version in the auto-generated Tag name.
New-BCImage however is just adding :latest as the tag when you use the -ImageName parameter and leave out the tag.
It would be fantastic if New-BCImage would follow the same principle as New-NavContainer in this regard. So you can pull the latest version of the Artifacts, let it create an image which automagically has the correct version in the Tag :-)
Example:
New-NavContainer `
 -accept_eula `
 -containerName test `
 -artifactUrl $artifactUrl `
 -Credential $credential `
 -auth UserPassword `
 -updateHosts `
 -imagename businesscentral/insider

When this is finished, we end up with an image named like
businesscentral/insider:sandbox-17.0.15311.0-de
New-BCImage `
 -artifactUrl $artifacturl `
 -imageName businesscentral/insider`

When this is finished, we end up with an image named like
businesscentral/insider:latest
NavContainerHelper is version <IP_ADDRESS>
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Version 1809 Build 17763.1339
Docker Client Version is 19.03.5
Docker Server Version is 19.03.5
...

Additional context

does it happen all the time?
Reproducable every time.

did it use to work?
No

That is by design.
New-BcImage creates the image you decide.
New-BcContainer uses New-BcImage to "cache" an image and adds a naming algorithm to ensure that it selects the right image.

As I wrote, not a Bug, more a feature request :-) It would be very convenient to have the same behaviour for the -ImageName parameter when utilizing your cmdlets to populate a private repository.
Don't get me wrong, I am very happy with what we have right now, it already saves a lot of time.

After writing my previous response, I did reconsider and I agree that it indeed would be nice to have New-BcImage make the same defaulting of the tag.
I will fix this.

Awesome, Thank you so much for all the hard work you put into this 👍

Note, that this fix will go into the BcContainerHelper module, NavContainerHelper will only get bug fixes.
I will write a blog post about switching from NavContainerHelper to BcContainerHelper very soon, sorry for the inconvenience.

I don't see any inconvenience ;-) I'm glad you are doing this. Thanks for pointing that out, because I really thought they are the same (just different name), since there are BC Aliases for NAV cmdlets in the NavContainerHelper. I will switch ASAP then. Thanks for the heads up!

Shipped in BcContainerHelper 1.0.1


Update/remove bio python library

The version of bio referenced in requirements.txt is no longer available. See https://pypi.org/project/bio/#history.
bio~=0.2.3

A quick look through the code, I don't see the bio utility being used directly. I do see references to biopython, which happens to be a dependency for bio. So, possible that bio could be removed and biopython added to requirements.txt.

Thanks Ben! We are indeed working on updating the requirements, so we should have a commit soon fixing this.

@samozm @YukeWu

Fixed with commit.


improve handling for only

add bindings for top level zoraOnly bindings, needed to correctly run only tests
update README.md to further document this functionality

Thanks! I’ll try to cut a release later today.

Has been published to NPM. Thanks again!


Remove WinRAR

Nobody pay for use WinRAR

Imagine that the next WinRAR update could ship with some kind of virus.


[PARK-23481][WebUI]lastStageAttempt should fail when a stage doesn't exist

What changes were proposed in this pull request?
The issue here is AppStatusStore.lastStageAttempt will return the next available stage in the store when a stage doesn't exist.
This PR adds last(stageId) to ensure it returns a correct StageData
How was this patch tested?
The new unit test.

cc @vanzin

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/993/
Test PASSed.

Test build #87595 has started for PR 20654 at commit eeeea98.

LGTM pending tests; you have a typo in the PR title though.

@marmbrus Thanks! Fixed the title.

Test build #87595 has finished for PR 20654 at commit eeeea98.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87595/
Test PASSed.

Thanks! Merging to master and 2.3.


openWithNewTab doesn't close on IE 11

Description
In IE 11 openWithNewTab and potentially openWithPopup never resolve.
Expected outcome
The page closes and the old page recognizes the login.
Actual outcome
The new tab stays open and the old tab never gets the authentication information.
Live Demo
https://salte-auth-166.glitch.me Edit
Steps to reproduce

Login with IE 11
Window doesn't close after redirect

Browsers Affected

[ ] Chrome
[ ] Firefox
[ ] Safari 10
[ ] Safari 9
[ ] Safari 8
[ ] Safari 7
[ ] Edge
[x] IE 11
[ ] IE 10

This is caused by IE 11 not accepting name changes to windows in conjunction
with window.opener getting lost on when we navigate away from the initial page.


Zuul default route

In my project, I use Zuul to route the incoming request to correct service component, all of my service components are registered to consul, and the api-gateway (zuul) configuration is as below:
...
zuul:
  routes:
    auth:
       path: /auth/**
       serviceId: auth
...

And then, I have a new service to register to the consul, namely user service,and this user service has succeeded in registering itself in consul, the instance name is user, but I forget to add the route configurations just as showed in code blocks above.
But, surprisedly， I found I can get the right endpoint response back by the address: http://localhost:8888/user/info (Here, 8888 is the service port of api-gateway service), and here I want to request the /info Actuator endpoint provided by spring cloud.
And my question is : How can this request go through the api-gateway(zuul), and find the correct user service ?
Thanks a lot
Sunrry

Zuul  will create default route for every service：
path: //**
serviceId:
When your user service registering，zuul create a route： path is /user/**，serviceId is user
If you do not want user service export，you can set ：
zuul:
  ignoredServices: user

I'm afraid I don't quite understand. By default, our zuul code adds routes for all services in discovery. To ignore them, use the tip from @dyc87112 above.

@dyc87112 So you mean the default route will be set by Zuul once our service is registered to consul, and this default route does not need any configuration like:
zuul:
    routhes:
        user:
            path: /user/**
            serviceId: user

Because I alway think that all of routes of Zuul need to be declared by configurations, so there are many routes in my project like:
zuul:
    routhes:
        auth:
            path: /auth/**
            serviceId: auth
         expense:
            path: /expense/**
            serviceId: expense
        monitor:
            path: /monitor/**
            serviceId: monitor

It seems I do a lot of needless work.
Thanks

@Sunrry yeah.....

Thank you all so much.


Spotio doesnt open

Hi, I cant seem to open Spotio, its telling me its damaged. Can you help? Thank you

@amrthameen is this on OS X ?

@sandervankasteel  OS Sierra

@amrthameen you need to resign your binary or disable GateKeeper on MacOS. Although disabling GateKeeper isn't a recommended method, it is the easiest to do.
A while back I managed to resign the Spotio / Spotify application with my own keys; more info on this can be found here: https://github.com/devinhalladay/spotio/issues/23

@amrthameen did my comment resolve your issue ?

I'm gonna go ahead and close this issue since it's inactive but if the issue persists just tag me and I can reopen it


Chris branch

Updated markdown text; removed an outdated version of the D notebook

Also, it seems there are some conflicts here - you'll need to update your branch.


Add more verbose output about kafka schema compatibility

Hello.
Currently we are facing with the issue in CI/CD that terrafrorm produce following log:
Error: kafka schema is not compatible with version :4

  on .terraform/modules/kafka_data/modules/service_kafka_data/main.tf line 36, in resource "aiven_kafka_schema" "this":
  36: resource "aiven_kafka_schema" "this" {

Error: kafka schema is not compatible with version :4

  on .terraform/modules/kafka_data/modules/service_kafka_data/main.tf line 36, in resource "aiven_kafka_schema" "this":
  36: resource "aiven_kafka_schema" "this" {

We cannot fastly determine which schema is not compatible with Aiven schema registry. Can you add more verbose output to this error log with an information about schema subject?

Hello,
It looks like there is no error in your case coming from the Aiven API which may help to understand a compatibility issue - more can be found here https://github.com/aiven/aiven-go-client/blob/master/kafka_schemas.go#L227
If we do have an error message coming from Aiven API response on an attempt to add a new Kafka Schema TF users should see that, but there is none in your case.
I will have a look if we have another API endpoint or any way to provide more information regarding schema compatibility issues.

After testing Aiven API Kafka Schema validation endpoint, it looks it sends back an error with explanation only in the situation when a user tries to send a broken Avro format and responds with 500 HTTP. But when a user tries to send an incompatible schema, it returns only a boolean compatibility flag, and in case it is false TF users see that generic error message Kafka schema is not compatible with version: %n.
I created a ticket to improve this situation in our internal issue tracker, unfortunately, cannot give you a link. No changes in TF provider itself are necessary, and when this feature will out, all users will immediately benefit from it.

Thank you, Ivan. I am looking forward for news regarding this enhancement.


fullwidth left parenthesis (U+FF08)

I can't find where the various alternative parentheses are defined?

In the "Punctuation characters" section of the notebook, I say that the various brackets and parentheses all seem to normalise to their ASCII equivalents, so you don't need to treat them specially. The table in that section only lists characters that don't normalise to anything in the "categories already understood by Numbas JME" defined at the top of the notebook (at the moment, that's '".{}?/\n:&;|^>=<-+*#!(),[]

Ok, we're not getting them to normalise!
I'm planning to add some from here: https://www.fileformat.info/info/unicode/category/Ps/list.htm

Is that because you can't do normalisation, or you just choose not to? I'm happier relying on a standard algorithm.
If it'd help, I can add a bit to the notebook to make it produce lists of characters that normalise to each of the ASCII ones in the list I gave above.

Well, we've tried normalisation and it doesn't work.  In particular https://www.php.net/manual/en/class.normalizer.php with Normalizer::FORM_KC transforms x² to x*2, and doesn't solve the brackets issue!  The other options for PHP don't seem to work for me either.

I don't think you should normalise an entire expression string before parsing it: as you've shown, you lose important information like the subscriptness of ².
(To be clear, "x²" normalises to "x2", and I guess that you then insert the 'missing' asterisk as part of your existing filters)
The Numbas JME tokeniser first matches a prefix of the string which matches a regular expression corresponding to a particular token type, and then might normalise just that substring. So we have a separate regex for subscripts, which will insert ^ and parentheses as needed, and apply the superscript replacements from this project.
... and now that I look at it, the list of parentheses is still hardcoded, but not in this project! The JME parser has the following strings, which are used in the regex to match punctuation tokens:
    /** Characters representing a left parenthesis.
     *
     * @type {Array.<string>}
     */
    left_parentheses: "(❨❪⟮﹙（﴾⦅｟",

    /** Characters representing a right parenthesis.
     *
     * @type {Array.<string>}
     */
    right_parentheses: ")﹚）❩❫﴿⟯⦆｠",

once those are matched, they're normalised using NFKD.  So you're right that some important information was missing here!

In 29b5e8a I've added a file brackets.json mapping the bracketing characters that don't already normalise under NFKD.
So you can detect bracket characters by looking for character classes Ps and Po, normalise with NFKD, then apply the mapping from brackets.json if necessary.


Ideas for E2E tests and peripherals

A report which shows what happened in the E2E test - the steps that happened, any actions, any verifications and which actions succeeded and which failed, which verifications passed and which failed, and any reason for failure, so that we can see quickly what's the problem! :) Instead of going through the full logs!! JSON report please!!

Automated Golang program to pull GitHub Actions run data - which passed, which failed, logs of failed, ability to grep the logs and get some data etc to understand similarity etc, and get data like number of the test, name of the test (AWS, Azure etc) and the URL to the workflow, the jobs, the steps etc

When writing tests as native golang testing library tests, one big benefit we get is - we can choose which tests to run by choosing tests with regexp

When using credentials in E2E tests, the E2E test code should probably do a check on the credentials as the first thing - check if it's defined (as environment variable), not empty (empty string), and valid (not expired) and has appropriate permissions - enough permissions to run the E2E test, for example, create VMs, create networks, create disks, and similar so that it has enough permissions to create a cluster and destroy it too as part of the E2E test

When downloading tar ball for installation of tanzu from TCE or TF, we could show progress 😁
https://duckduckgo.com/?t=ffab&q=show+progress+for+download+in+golang&ia=web
https://gist.github.com/albulescu/e61979cc852e4ee8f49c
We could use dots . to show progress, probably 10 or 20 dots to show 100 % progress. If it's 10 dots, each dot represents 10% being done, if it's 20 dots, each dot shows 5% progress. Why dots? So that it's CI/CD friendly and easy to print the dots in a line to show progress. Something like -
Progress:..........

Checking out existing E2E test framework in TCE and upstream - done by Shwetha and Vladimir and others
- Take inspiration from it
- Understand it
- Understand pros and cons
https://github.com/vmware-tanzu/community-edition/tree/03201d8852c20c34a39086d4a113053bc990cdfd/test/e2e
https://github.com/kubernetes-sigs/e2e-framework

Think about the requirements to satisfy

Providing libraries and functions to run clusters

For E2E tests, what we need to do is - show proper error messages. This is true for any software program actually. When something fails, try to show why exactly it fails and let it be as intelligent as possible and also give suggestions / advice on good action items on what to do next so that the user can take some actions to fix / get rid of the error

The vSphere E2E test can check if vSphere control plane endpoint(s) are already accessible before the test starts so that we can error out and say that there is an IP clash and that static virtual IP(s) that aren't in use only has to be provided as input. Note that tanzu CLI does this check too as part of it's validation and pre-requisites but we can do this way earlier as part of test input sanitization and fail early, unless we are testing this feature of tanzu CLI in which case we don't have to do this check in the test code and instead just check if tanzu CLI fails when using an IP that's already in use for a control plane endpoint IP

Test input for E2E tests - formats -

CLI Flags and CLI flag values
CLI arguments
Environment variables
Test config file

Have a Makefile for convenience and maybe use https://earthly.dev as the supported build system 😁

Apart from having a summary / report of what are the steps / actions that happened, we need to create logs for each of those steps and be able to look at the logs of a given step quickly. Note - the logs should mask all the test secret inputs. For example, if a log contains an error showing that the client secret passed is wrong / invalid and logs the client ID, we should mask it with dummy characters or a placeholder like ""
Basically - mask all the test secret inputs / data with their corresponding names in a placeholder like fashion so that it's clear what was there but not clear about what's the exact value

Some sample logs for Azure test -
tce-e2e-test $ go test -v ./...
=== RUN   TestAzureManagementAndWorkloadCluster
2022/04/09 15:13:05 Checking required environment variables...
2022/04/09 15:13:05 Extracting Azure test secrets from environment variables...
2022/04/09 15:13:05 Logging into Azure...
2022/04/09 15:13:05 Getting marketplace terms for Azure VM image...
2022/04/09 15:13:11 Azure VM image agreement terms are already accepted
--- PASS: TestAzureManagementAndWorkloadCluster (5.68s)
PASS
ok  	github.com/karuppiah7890/tce-e2e-test	6.083s
tce-e2e-test $ go test -v ./...
=== RUN   TestAzureManagementAndWorkloadCluster
2022/04/09 15:13:21 Checking required environment variables...
2022/04/09 15:13:21 Extracting Azure test secrets from environment variables...
2022/04/09 15:13:21 Logging into Azure...
2022/04/09 15:13:21 Getting marketplace terms for Azure VM image...
2022/04/09 15:13:26 Azure VM image agreement terms are already accepted
--- PASS: TestAzureManagementAndWorkloadCluster (5.34s)
PASS
ok  	github.com/karuppiah7890/tce-e2e-test	5.665s
tce-e2e-test $ go test -v ./...
=== RUN   TestAzureManagementAndWorkloadCluster
2022/04/09 15:13:33 Checking required environment variables...
2022/04/09 15:13:33 Extracting Azure test secrets from environment variables...
2022/04/09 15:13:33 Logging into Azure...
2022/04/09 15:13:33 Getting marketplace terms for Azure VM image...
2022/04/09 15:13:39 Azure VM image agreement terms is not already accepted. Signing and accepting the Azure VM image agreement terms now...
2022/04/09 15:13:44 Error while signing and accepting the agreement terms for Azure VM image: POST https://management.azure.com/subscriptions/87fbb054-796e-47cc-a777-c7a052901de1/providers/Microsoft.MarketplaceOrdering/agreements/vmware-inc/offers/tkg-capi/plans/k8s-1dot22dot8-ubuntu-2004/sign
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: BadRequest
--------------------------------------------------------------------------------
{
  "error": {
    "code": "BadRequest",
    "message": "You must accept the legal terms through the Azure portal (http://go.microsoft.com/fwlink/?LinkId=534873). Correlation Id: '479de9c1-f4e8-4879-a573-82d6a8399699'."
  }
}
--------------------------------------------------------------------------------
FAIL	github.com/karuppiah7890/tce-e2e-test	10.769s
FAIL
tce-e2e-test $

https://docs.microsoft.com/en-us/rest/api/marketplaceordering/marketplace-agreements/create
https://docs.microsoft.com/en-us/rest/api/marketplaceordering/marketplace-agreements/sign

signResponse, err := client.Sign(ctx, azureVmImagePublisher,
	azureVmImageOffer,
	azureVmImageBillingPlanSku,
	nil)
if err != nil {
	log.Fatalf("Error while signing and accepting the agreement terms for Azure VM image: %+v", err)
}

signedAgreementTerms := signResponse.AgreementTerms

if signedAgreementTerms.Properties == nil {
	log.Fatalf("Error while signing and accepting the agreement terms for Azure VM image: Azure VM image agreement terms Properties field is not available")
}

if signedAgreementTerms.Properties.Accepted == nil {
	log.Fatalf("Error while signing and accepting the agreement terms for Azure VM image: Azure VM image agreement terms Properties Accepted field is not available")
}

if isTermsSignedAndAccepted := *signedAgreementTerms.Properties.Accepted; !isTermsSignedAndAccepted {
	log.Fatalf("Error while signing and accepting the agreement terms for Azure VM image: Azure VM image agreement terms was not signed and accepted")
}

tce-e2e-test $ go test -v ./...
=== RUN   TestAzureManagementAndWorkloadCluster
2022/04/09 16:26:37 Checking required environment variables...
2022/04/09 16:26:37 Extracting Azure test secrets from environment variables...
2022/04/09 16:26:37 Logging into Azure...
2022/04/09 16:26:37 Getting marketplace terms for Azure VM image...
2022/04/09 16:26:42 Azure VM image agreement terms is not already accepted. Signing and accepting the Azure VM image agreement terms now...
2022/04/09 16:26:46 Error while signing and accepting the agreement terms for Azure VM image: POST https://management.azure.com/subscriptions/87fbb054-796e-47cc-a777-c7a052901de1/providers/Microsoft.MarketplaceOrdering/agreements/vmware-inc/offers/tkg-capi/plans/k8s-1dot22dot8-ubuntu-2004/sign
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: BadRequest
--------------------------------------------------------------------------------
{
  "error": {
    "code": "BadRequest",
    "message": "You must accept the legal terms through the Azure portal (http://go.microsoft.com/fwlink/?LinkId=534873). Correlation Id: 'ccd645a2-389a-4157-8624-43d597d59743'."
  }
}
--------------------------------------------------------------------------------
FAIL	github.com/karuppiah7890/tce-e2e-test	9.753s
FAIL
tce-e2e-test $

https://duckduckgo.com/?t=ffab&q=You+must+accept+the+legal+terms+through+the+Azure+portal&ia=web
https://www.nblabs.net/2017/12/20/accepting-azure-marketplace-terms-powershell/
https://carlos.mendible.com/2018/05/21/accepting-azure-marketplace-terms-with-azure-cli/
https://duckduckgo.com/?t=ffab&q=azure+cli&ia=web
https://github.com/Azure/azure-cli
https://github.com/Azure/azure-cli/search?q=accept-terms
https://github.com/Azure/azure-cli/blob/9bae6080ec1fc258ffb52f7218915971f6e8051f/src/azure-cli/azure/cli/command_modules/vm/commands.py
https://github.com/Azure/azure-cli/blob/9bae6080ec1fc258ffb52f7218915971f6e8051f/src/azure-cli/azure/cli/command_modules/vm/commands.py#L390
https://github.com/Azure/azure-cli/blob/9bae6080ec1fc258ffb52f7218915971f6e8051f/src/azure-cli/azure/cli/command_modules/vm/commands.py#L394
https://github.com/Azure/azure-cli/blob/9bae6080ec1fc258ffb52f7218915971f6e8051f/src/azure-cli/azure/cli/command_modules/vm/commands.py#L111-L114
azure.mgmt.marketplaceordering.operations#MarketplaceAgreementsOperations

https://docs.microsoft.com/en-us/answers/questions/52637/cannot-sign-azure-marketplace-vm-image-licence-thr.html

tce-e2e-test $ go clean -testcache
tce-e2e-test $ go test -v ./...
=== RUN   TestAzureManagementAndWorkloadCluster
2022/04/09 17:06:36 Checking required environment variables...
2022/04/09 17:06:36 Extracting Azure test secrets from environment variables...
2022/04/09 17:06:36 Logging into Azure...
2022/04/09 17:06:36 Getting marketplace terms for Azure VM image...
2022/04/09 17:06:42 Azure VM image agreement terms is not already accepted. Accepting the Azure VM image agreement terms now...
2022/04/09 17:06:47 Accepted the Azure VM image agreement terms!
--- PASS: TestAzureManagementAndWorkloadCluster (10.93s)
PASS
ok  	github.com/karuppiah7890/tce-e2e-test	11.305s
tce-e2e-test $ az vm image terms show --publisher "vmware-inc" --offer "tkg-capi"         --plan "k8s-1dot22dot8-ubuntu-2004" --subscription "${AZURE_SUBSCRIPTION_ID}" --debug
{
  "accepted": true,
  "id": "/subscriptions/<subscription-id>/providers/Microsoft.MarketplaceOrdering/offerTypes/VirtualMachine/publishers/vmware-inc/offers/tkg-capi/plans/k8s-1dot22dot8-ubuntu-2004/agreements/current",
  "licenseTextLink": "https://mpcprodsa.blob.core.windows.net/legalterms/3E5ED_legalterms_VMWARE%253a2DINC%253a24TKG%253a2DCAPI%253a24K8S%253a2D1DOT22DOT8%253a2DUBUNTU%253a2D2004%253a24KQJXKK2NIUVNWR2B7X4J2ZWNZGFLGIQPBB46SSI4WO32VUTVGS6WFNL5JR3J6WPKJZK5YCCZV7WEHQGADEWGPC7PEJHUIA2S4ONKRFY.txt",
  "marketplaceTermsLink": "https://mpcprodsa.blob.core.windows.net/marketplaceterms/3EDEF_marketplaceterms_VIRTUALMACHINE%253a24AAK2OAIZEAWW5H4MSP5KSTVB6NDKKRTUBAU23BRFTWN4YC2MQLJUB5ZEYUOUJBVF3YK34CIVPZL2HWYASPGDUY5O2FWEGRBYOXWZE5Y.txt",
  "name": "k8s-1dot22dot8-ubuntu-2004",
  "plan": "k8s-1dot22dot8-ubuntu-2004",
  "privacyPolicyLink": "https://www.vmware.com/help/privacy/products-and-services-notice.html",
  "product": "tkg-capi",
  "publisher": "vmware-inc",
  "retrieveDatetime": "2022-04-09T11:37:01.5262201Z",
  "signature": "QN2LE7QN5TSE57PKJIAYN326FVPAWUS4SRBYUANY4W7ETANMMDORKWXSQ543VP7YKQTI2EAI2ESHH6IAPE2R34WDWTUZTCAUVDG5I5Q",
  "systemData": {
    "createdAt": "2022-04-09T11:37:01.598348+00:00",
    "createdBy": "<subscription-id>",
    "createdByType": "ManagedIdentity",
    "lastModifiedAt": "2022-04-09T11:37:01.598348+00:00",
    "lastModifiedBy": "<subscription-id>",
    "lastModifiedByType": "ManagedIdentity"
  },
  "type": "Microsoft.MarketplaceOrdering/offertypes"
}

https://mpcprodsa.blob.core.windows.net/legalterms/3E5ED_legalterms_VMWARE%253a2DINC%253a24TKG%253a2DCAPI%253a24K8S%253a2D1DOT22DOT8%253a2DUBUNTU%253a2D2004%253a24KQJXKK2NIUVNWR2B7X4J2ZWNZGFLGIQPBB46SSI4WO32VUTVGS6WFNL5JR3J6WPKJZK5YCCZV7WEHQGADEWGPC7PEJHUIA2S4ONKRFY.txt
https://mpcprodsa.blob.core.windows.net/marketplaceterms/3EDEF_marketplaceterms_VIRTUALMACHINE%253a24AAK2OAIZEAWW5H4MSP5KSTVB6NDKKRTUBAU23BRFTWN4YC2MQLJUB5ZEYUOUJBVF3YK34CIVPZL2HWYASPGDUY5O2FWEGRBYOXWZE5Y.txt
Examples of .env files - https://terraspace.cloud/docs/config/dotenv/

Before running tests against management and workload cluster or any plugin, we can check if the plugin exists and is accessible like this -
$ tanzu management-cluster version
v0.11.2

$ tanzu cluster version
v0.11.2

$ tanzu unmanaged-cluster version
v0.11.0

We can parse the version of the plugins and show warning in case the E2E tests have not been tried and tested with newer versions. We can hard code the version we have tried and tested it with and anything (any version) newer - it's possible to have breaking changes etc, which may result in the test not working correctly to test the newer version

The E2E tests can fail early if any of the required plugins are not present for doing the current test. For example, if any of management and workload cluster plugins are not there, they cannot run management and workload cluster plugin test

The E2E tests can fail early by checking if Docker daemon is running and accessible. This way it's faster for it to fail. But yeah, Tanzu CLI already does this and fails, but it's way later in the process of the E2E test but not too late I guess as it's the first step in Tanzu CLI, to check pre-requisites

Examples for listing workload clusters -
$ tanzu cluster list -o json
[
  {
    "name": "workload-cluster-26-08-2021-1",
    "namespace": "default",
    "status": "deleting",
    "plan": "dev",
    "controlplane": "0/1",
    "workers": "",
    "kubernetes": "v1.21.2+vmware.1",
    "roles": [],
    "labels": {
      "tanzuKubernetesRelease": "v1.21.2---vmware.1-tkg.1-zshippable",
      "tkg.tanzu.vmware.com/cluster-name": "workload-cluster-26-08-2021-1"
    }
  }
]

$ tanzu cluster list
  NAME                           NAMESPACE  STATUS    CONTROLPLANE  WORKERS  KUBERNETES        ROLES   PLAN
  workload-cluster-26-08-2021-1  default    deleting  0/1                    v1.21.2+vmware.1  <none>  dev

$ tanzu cluster list
  NAME  NAMESPACE  STATUS  CONTROLPLANE  WORKERS  KUBERNETES  ROLES  PLAN

https://pkg.go.dev/bytes - for bytes.Buffer
https://pkg.go.dev/encoding/json - for JSON decoding

Store all command logs (standard output, standard err, combined - just for easy use) in log files and ensure that the secrets are masked in these log files. Note - let's not post process these files to mask the secrets, instead, when we write the log to the files itself the secrets should be masked

                      Secrets
                        ||
                        ||
                        \/
output stream => secrets masker => write masked output to file

The test should know it's timeout time and accordingly ensure that the test is over within that time. Or else, the test can start cleaning up things. For example, if the timeout time is 30 minutes, the test can have a max time of 20 minutes and then trigger off a cancellation / stop / cleanup of all the things it started off so that they will be done in those 10 minutes. Or, we can do the cleanup separately but we need to know what is half way done / half baked to do the cleanup, or know what actions started and happened to revert them

A logger library which we can use to log at different levels including custom levels (summary etc) and can log to different destinations - standard output, standard error, log files, summary files etc and also log in structured format (json etc) and unstructured format (simple log line)
We can consider third part log libraries. I have heard Uber Zap to be popular - https://github.com/uber-go/zap/

At any point in time, we should be able to understand the progress of the E2E test. Use progress bars / dots etc, use logs etc to show that there is some progress and how much progress is there - some sort of percentage apart from indefinite progress notification
We should be able to find out what the E2E test is doing at any point in time and be able to tell if it's stuck or if it's just taking time to finish it's job

Idea for Credentials - Login to the cloud account exactly once and reuse that among the test functions. Either pass the credentials all over, or use a package which stores an inaccessible variable and provides it to globally as a singleton. Logic? If not already available in the variable, instantiate and store it in the single variable and return. If already available, just return
Idea for log - same as credentials, provide it globally as a singleton through a package. Logic? If not already available in the variable, instantiate and store it in the single variable and return. If already available, just return

We gotta check required quota and quota available before running E2E tests to avoid errors like the below -
controller/azuremachine "msg"="Reconciler error" "error"="failed to reconcile AzureMachine: failed to create virtual machine: failed to create VM test-wkld-1649820052-control-plane-gsfj8 in resource group test-wkld-1649820052: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code=\"OperationNotAllowed\" Message=\"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: australiaeast, Current Limit: 10, Current Usage: 8, Additional Required: 4, (Minimum) New Limit Required: 12. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22<subscription-id>%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22australiaeast%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22cores%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:12,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22cores%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests\"" "name"="test-wkld-1649820052-control-plane-gsfj8" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AzureMachine"


How to provide environment variables for tests?

I'm wondering how I can provide env variables for nodejs tests? The only idea I have is to manually copy the .env file before the build to the output directory, but I would prefer to do it in some better way. For example, a preTest event in the build options or some property that will automatically pass the object as env variables. Or make a property that will remove the automatic launch of npm test and let it run manually with whatever environment variables I want

You should be able to set any environment variables for the test in the post build hook, which runs after build and before the tests. These should then be inherited when it runs the node.js process for the tests:
await build({
  // ...
  postBuild() {
    // set your env variables here or load from the .env file (ex<EMAIL_ADDRESS>  },
});

Oh, I hadn't noticed that preBuild was running before the tests. Thank you very much!


Use maven-compiler-plugin for annotation processing

...instead of the unsupported apt-maven-plugin.
The main advantage of this change is that Intelij IDEA would automatically pick up the annotation processor and run it when building inside IDE. Thus eliminating the need to manually run mvn generate-sources or mvn compile to get the generated classes every time after changing the branch. maven-compiler-plugin also properly cleans up the generated sources whenever a model is removed, unlike the apt-maven-plugin, where user has to manually go and remove the no-longer-needed Q-class from the build folder as otherwise compilation will fail.
Closes #2091
Creating the PR as a draft just to demonstrate the idea and get feedback. If you think that this is the right approach, I can extend it to all querydsl modules and update documentation.

Originally tested this in my own project, which is still on 4.3.1, but looks like with 5.0.0 jakarta.annotation-api is not needed at all.

Don't forget to update the examples as well:

https://github.com/querydsl/querydsl/blob/99e3ca936f46471bc61d9af6e1a3dbb16f3f12b3/querydsl-docs/src/main/docbook/en-US/content/tutorials/jpa.xml#L51-L66
https://github.com/querydsl/querydsl/blob/99e3ca936f46471bc61d9af6e1a3dbb16f3f12b3/querydsl-docs/src/main/docbook/en-US/content/tutorials/jdo.xml#L47-L62
https://github.com/querydsl/querydsl/blob/99e3ca936f46471bc61d9af6e1a3dbb16f3f12b3/querydsl-docs/src/main/docbook/en-US/content/general/codegen.xml#L228-L247
https://github.com/querydsl/querydsl/blob/99e3ca936f46471bc61d9af6e1a3dbb16f3f12b3/querydsl-docs/src/main/docbook/en-US/content/tutorials/collections.xml#L167-L182


Link underline colour

Link to Ticket
Description of Changes Made

Update all links to have an underline of  rgba(255, 255, 255, 0.5) or #989b9b
Don't apply this to navigation links in the dropdowns as the new colour does not pass colour contrast - stick with the colour of the text
Update some instances where the link-styles() mixin was not used

How to Test

Browse the site and make sure all link underlines look consistent
Might be easier to test components on the styleguide page in the pattern library

Screenshots
There's too many components that this has changed to provide screenshots.
MR Checklist

[x] Add a description of your pull request and instructions for the reviewer to verify your work.
[x] If your pull request is for a specific ticket, link to it in the description.
[x] Stay on point and keep it small so the merge request can be easily reviewed.
[x] Tests and linting passes.

Unit tests

[ ] Added
[x] Not required

Documentation

[ ] Updated build docs
[ ] Updated editor guidelines (See https://intranet.torchbox.com/torchbox-com-project-docs for the private link, for Torchbox employees only)
[ ] Updated field spec (See https://intranet.torchbox.com/torchbox-com-project-docs for the private link, for Torchbox employees only)
[x] Not required

Browser testing

[ ] I have tested in the following browsers and environments (edit the list as required)

Latest version of Chrome on mac
Latest version of Firefox on mac
Latest version of Safari on mac
Safari on last two versions of iOS
Chrome on last two versions of Android

[ ] Not required

Data protection

[ ] Not relevant
[ ] This adds new sources of PII and documents it and modifies Birdbath processors accordingly

Accessibility

[ ] Automated WCAG 2.1 tests pass
[ ] HTML validation passes
[ ] Manual WCAG 2.1 tests completed
[ ] I have tested in a screen reader
[ ] I have tested in high-contrast mode
[ ] Any animations removed for prefers-reduced-motion
[ ] Not required

Sustainability

[ ] Images are optimised and lazy-loading used where appropriate
[ ] SVGs have been optimised
[ ] Perfomance and transfer of data considered
[ ] If JavaScript is needed alternatives have been considered
[ ] Not required

Pattern library

[ ] The pattern library component for this template displays correctly, and does not break parent templates
[ ] The styleguide is updated if relevant
[ ] Changes are not relevant the pattern library

@chris-lawton I think this MR in particular would be good to pull into the light mode integration branch, so that we can test the changes in light mode as well - and because it involves adding a new colour, and editing some of the same files. It's a non-urgent one, so doesn't matter if it gets a bit delayed waiting for light mode to be merged. Would you be OK to rebase against the integration branch and then do a new pull request to that branch?