Help: How to access all Rules before and after they are validated, specifically the CustomState and ErrorCode properties

System Details

FluentValidation version: 8.4.0
Web Framework version Core 2.1 (eg ASP.NET Core 2.1, MVC 5, WebApi 2. Delete if not applicable):

Issue Description
New to FluentValidation. FluentValidation home page. I am using .NET Core with C#.
Please see my plea for Help on Stackoverflow
Need to access all of the Rules on a Validator, after it has been built but BEFORE Validation, because all rules must be displayed on Uix.
The CustomState property is assigned an instance of UixEntityProperty that holds metadata explaining the rule, help, property name, error code etc. Need to display a list of all these for the user, so that they can view all rules, once Validated, all Rules must be displayed again showing which failed and passed.
Our system has very large configurable forms with hundreds of fields. Thus we need more elaborate info for each rule and field(s) (property). The users must be able to view ALL Rules before and after validation, not just those which failed.
IValidationRule does not contain a reference to to CustomState. Casting to PropertyRule did provide more properties but I suspect there are issues here with threading and contexts?
The XUnit test class below defines a couple of tests that fail at accessing the the CustomState.

CustomState is contextual per validation run and the callback for generating it is stored against the individual validators, not against the rule instance. You would need to loop over each validator on the Rule instance and then you can access the state from the validator’s Options property. However this won’t be useful - it will only give you a reference to the delegate. The value is only available once validation has occurred, and is made available only on the final ValidationResult.

Jeremy,
Thanks for the super quick response much appreciated.
My understanding is that CustomState can be distinct per Rule rather than common to the Validator?
I am guess that the internal logic means that there is no access to this CustomState per Rule?

The value is only available once validation has occurred, and is made available only on the final ValidationResult.

Is there a way to access all Rules, whether failed or not, after Validation and thus retrieve the Custom state instance?
Failing that is there another way to simply store an object instance per Rule and access this before and after Validation, I understand that CustomState is meant for this purpose, just trying to find a workaround.

My understanding is that CustomState can be distinct per Rule rather than common to the Validator

They are distinct per individual property validator. So a call to individual validators such as NotNull, Length etc can all have their own custom state. A rule (created by calling RuleFor) is a collection of property validators. It doesn't have custom state itself, but each of its property validators does.

Is there a way to access all Rules, whether failed or not, after Validation and thus retrieve the Custom state instance?

As I said before, you can only get a reference to the delegate that generates the custom state:
var validator = new PersonValidator();

foreach (var rule in validator.OfType<PropertyRule>()) {
  foreach (var propertyValidator in rule.Validators) {
    Func<PropertyValidatorContext, object> stateProvider = propertyValidator.Options.CustomStateProvider;
  }
}

This gives you a reference to the delegate for each property validator, but it's only intended to be invoked during validation when an individual property validator fails (it requires a validation context to be executed).
To reiterate - custom state is associated with validation results. It is only generated/invoked for failures. This isn't intended for holding general metadata about rules.

To do what you want, I’d take a different approach:

have an external class with a static concurrent dictionary that will be used to hold metadata
add a custom FluentValidation extension method on your rule chain (called WithMetadata or something like that) which receives your metadata instance
this method would build a key based on the property and the parent type on which the rule is defined, and then used this key to store the metadata in the external dictionary/cache
you can then use this to retrieve the metadata for display purposes

I’m out at the moment and don’t have my laptop with me, but if you’re struggling with how to implement this let me know and I can put together an example later today.

Jeremy,
Sorry for the delay, a long weekend for a family wedding.
I have put together a very small Project on GitHub following your guidelines for extension methods as per last message.
Project: [FluentValidationUnitTests] on (https://github.com/raxuser100/dev_community_demos.git)
The issue I have is, with my limited knowledge of FV, how do we create a unique key per instance of a Rule?
public static string RuleMetaDataKeyBuild(this PropertyRule rule)
{
	MemberInfo memberInfo = rule.Member;
	return $"{memberInfo.ReflectedType.FullName}.{memberInfo.Name}";
}

The above will fail for multiple instances of the same `Validator `or a `Rule `based on the same property

I'm not sure I understand. Why are you trying to make them distinct? Why not just use a grouping of Type -> Property name -> metadata with a data structure such as ConcurrentDictionary<Type, List<KeyValuePair<string, UixEntityProperty>>>?
This way you can say "For a particular type, give me all the properties, and for each property give me all the metadata", that way there's no need to know which specific rule instance was originally involved.

Maybe a wrapper around the rule that also adds a property for the metadata would be the way to do this then. Rather than using the regular RuleFor, you could use a custom method (eg RuleForWithMetadata) that creates your custom rule instance rather than the regular one. Quick example:
public class PropertyRuleWithMetadata : PropertyRule {
	public PropertyRuleWithMetadata(MemberInfo member, Func<object, object> propertyFunc, LambdaExpression expression, Func<CascadeMode> cascadeModeThunk, Type typeToValidate, Type containerType)
		: base(member, propertyFunc, expression, cascadeModeThunk, typeToValidate, containerType) {
	}

	public new static PropertyRuleWithMetadata Create<T, TProperty>(Expression<Func<T, TProperty>> expression, Func<CascadeMode> cascadeModeThunk, bool bypassCache = false) {
		var member = expression.GetMember();
		var compiled = AccessorCache<T>.GetCachedAccessor(member, expression, bypassCache);
		return new PropertyRuleWithMetadata(member, FluentValidation.Internal.Extensions.CoerceToNonGeneric(compiled), expression, cascadeModeThunk, typeof(TProperty), typeof(T));
	}

	public UixEntityProperty Metadata { get; set; }
}

// Intermedite base class
public abstract class MyBaseValidator<T> : AbstractValidator<T> {
	public IRuleBuilderInitial<T, TProperty> RuleForWithMetadata<TProperty>(Expression<Func<T, TProperty>> expression, UixEntityProperty metadata) {
		var rule = PropertyRuleWithMetadata.Create(expression, () => CascadeMode);
		rule.Metadata = metadata;
		AddRule(rule);
		var ruleBuilder = new RuleBuilder<T, TProperty>(rule, this);
		return ruleBuilder;
	}
}

Your validator classes would then inherit from MyBaseValidator instead of AbstractValidator directly, and would then call RuleForWithMetadata(x => x.Foo, metadata).NotNull()...etc


Differentiate between "On", "Off" and "DIM" or other commands

When Alexa is asked to Turn a light on or off, the msg.on variable is true or false and can be used to identify the state.
If Alexa is asked to dim the light to say 25%, msg.on is also true and you have no way of telling, that it was a dim command. My physical light needs to know if it was on, off or dim.
Is that possible. msg.payload is rendunant to msg.on, so that could be used?

put a debug node behind the alexa node and set it from msg.payload to full msg. then you can see, there is also a msg.bri. you can seperate this value with a function node and send it to the bri objekt in iobroker

I know, but it is still not possible. Take this scenario:
"Alexa, turn on the Light".  "Alexa, dim the Light to 25%".
The user now changes the dim manually to 50% in the room. And later turns the light off manually.
"Alexa, turn on the Light" - this will now give 25% Light.
If instead your node would differentiate between ON, OFF and DIM, it would be possible to only send the ON command and thereby keep the Light at the 50% as it was intedend.

ok I understand.  Does the manual dimmer give any feedback to ioBroker?  Then you could save that value and send it in the next time, turn on the light.  What kind of hardware setup do you have?

you could also try what happens if you just leave the msg.bri away.  So filter out the value via a function node.  I will also try it

The physical switches/dimmers... in my house must remain intact (working) even if network, MQTT server ... do not work. So that is why they must be able to work separately. When I use the physical switch and the digital system (Alexa, remote or what ever) in a mix all the time, only the physical dimmer is allowed to remember the last dim setting.
And there is no feedback from for example a LIVOLO component - only a 433Mhz control signal TO the switch/dimmer.

I had a similar requirements so I modify the code as follows, which is along the same lines you're asking. The modification is simple and entails passing the variable body, which contains the Alexa response, to the msg.payload. The original handler is:
    function payloadHandler(hubNode, deviceId) {

      var msg = getDeviceAttributes(deviceId, hubNode.context());
      msg.rgb = colorSHB2RGB(msg.hue, msg.sat, 254);
      msg.payload = msg.on ? "on" : "off";
      msg.deviceid = deviceId;
      msg.topic = "";

      hubNode.send(msg);
    }

I've modified it as follows:
    function payloadHandler(hubNode, deviceId, body) {

      var msg = getDeviceAttributes(deviceId, hubNode.context().flow);
      msg.rgb = colorSHB2RGB(msg.hue, msg.sat, 254);
      msg.payload = body
      msg.deviceid = deviceId;
      msg.topic = "";

      hubNode.send(msg);
    }

Then in the portion of the code that handles the Alexa put, modify the call to the handler from:
app.put('/api/:username/lights/:id/state', function (req, res) {

        setDeviceAttributes(req.params.id, req.body, hubNode.context());

        var template = fs.readFileSync(__dirname + '/api/hue/templates/lights/se

        var data = getDeviceAttributes(req.params.id, hubNode.context());

        var output = Mustache.render(template, data);
        output = JSON.parse(output);

        res.json(output);

        payloadHandler(hubNode, req.params.id);
      });

to:
      app.put('/api/:username/lights/:id/state', function (req, res) {

        //hubNode.warn(req);
        setDeviceAttributes(req.params.id, req.body, hubNode.context().flow);

        var template = fs.readFileSync(__dirname + '/api/hue/templates/lights/se

        var data = getDeviceAttributes(req.params.id, hubNode.context().flow);

        var output = Mustache.render(template, data);
        output = JSON.parse(output);

        res.json(output);

        payloadHandler(hubNode, req.params.id, req.body);
      });

As you can see, req.body contains the Alexa payload that is put to the node red server. This variable is passed along to the function payloadHandler in a a new variable called body and is assigned to the payload  (msg.payload = body).
You can just make these modification to the index.js code in your installation. I've created my own branch with this and other modifications here, BTW.

Yes verjus !
That was exactly, what I need. Thank you.

thank you verjus for your kind contribute!
Can you please tell me, is it just necessary to stop node red, overwtite the  file /home/pi/.node-red/node_modules/node-red-contrib-amazon-echo/index.js with yours, and start again the node?
Thank you

If you don't want to make the changes above yourself, you can use the version in my fork above. The only files I've modified from the original are index.js and index.html. So you would need to simply replace these two files in the directory  /home/pi/.node-red/node_modules/node-red-contrib-amazon-echo/ and restart node-red. Be advised that I made 2 other changes to the original. The first change is that I require each amazon-echo-device node id (nodeid) to be given a friendly name in the configuration step. This friendly name (FriendlyName) can be anything and just needs to be unique. For example, Hue_6 or KitchenLight. I use zigbee2mqtt and set the same name in node-red as in zigbee2mqtt for consistency and simplicity. The reason I made this modification is that I can now send an input message to the echo-hub node of the form: Payload:{"on":true, "bri":11, "nodeid": "Hue_5"}. If I sent the topic to Brightness, this will set the light brightness. If topic is left blank, than the state is simply updated internally, without changing the bulb state. This is the second modification I made to the code.  If you don't use the input node, ignore all this and just set friendly names once for each node.I have tested these modification extensively and they work well for me and my setup.
Let me know if you have more questions.

Thank you for your kind reply! As you tested and introduced some new features I hope Datech will include your ready code to his next release! :) I see he's back here )))

One question yes, now this friendly id name is saved in the backup (export of flow) too? Because another limitation of this module is that it lets node red assign an ID, and every time it will be different. Probably because of this if you restore a backup you have to delete all devices stored in Alexa and recognize them again..
Have you tried it? Thanks

I checked and it appears to be saved in the backup, since the friendly name is part of the configuration of each node. I have not tried but if I had to guess, I would say it should work.
On Monday, September 23, 2019, 11:06:45 PM PDT, Barabba11<EMAIL_ADDRESS>wrote:

One question yes, now this friendly id name is saved in the backup (export of flow) too? Because another limitation of this module is that it lets node red assign an ID, and every time it will be different. Probably because of this if you restore a backup you have to delete all devices stored in Alexa and recognize them again..
Have you tried it? Thanks
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

thanks for your answer. Unfortunately to be sure it's needed to delete the flow and import it again, but doing it you risk to rediscover all devices of Echo again.
Maybe a solution is to backup the actual flow file, in case the restore will not work you may overwrite it with the old one. Unfortunately I can't make these tests as the Raspberry is located alsewhere ((

@verjus I have a problem here,
I stopped node red, copied (overwrite) index.html and index.js, the origina updated author's ones, started but I don't see any difference downloaded from the previous version, still no difference in the packets output from the node when I say "Alexa switch on.. or alexa change intensity to.. ", maybe the author still not update them yet?
I've tried yours and I have no output at all!! I checked inside modules, there is possible to change the node name and the topic, in the hub only th port and enable input.  I put back the recent original files and again I have output.. What is going wrong here?? amazing
Thanks for help

With the author's one I got also strange behaviour. The forse command I told Alexa is to regulate intensity, I had the vale on:false (with the proper bri value), I tried again, again false (but proper bri value), I told switch on and it went on:true, I told to change bri, it did with on:true. I told to swithc off, it gave on:false, I told to change intensity and this time on went true and bri was correct as always..
Is the problem may come from Alexa? Strange

@verjus with your script, from HUB
object
on: true
bri: 128
hue: 0
sat: 254
ct: 199
colormode: "ct"
rgb: array[3]
payload: object
deviceid: "8fa94a2de60c4"
topic: ""
_msgid: "a12d78b9.557078
Topic always "" with a on/off command or BRI
Author HUB output:
on: false
bri: 128
hue: 0
sat: 254
ct: 199
colormode: "ct"
rgb: array[3]
percentage: 50
payload: "off"
deviceid: "8fa94a2de60c4"
topic: ""
_msgid: "b06742ab.d5a81"
After starting node red, Echo was not restarted, told only command to change intensity, on is false (??)
object
on: true
bri: 128
hue: 0
sat: 254
ct: 199
colormode: "ct"
rgb: array[3]
percentage: 50
payload: "on"
deviceid: "8fa94a2de60c4"
topic: ""
_msgid: "d0f2c280.d773"
With command switch it on
object
on: true
bri: 153
hue: 0
sat: 254
ct: 199
colormode: "ct"
rgb: array[3]
percentage: 60
payload: "on"
deviceid: "8fa94a2de60c4"
topic: ""
_msgid: "6f720478.c0d0dc"
told to change intensity to 60%

I checked again, there is a bug in the script, if after restart is told to change intensity it will keep the on to false. After switching it on, off and give intensity again it goes to on true.

Hi Barabba11,
If you're using my version, you need to set the friendly name in the node configuration. Here is what I get in output after asking Alexa to turn on the light:
{"on":true,"bri":254,"hue":0,"sat":254,"ct":199,"colormode":"ct","rgb":[255,0,0],
"payload":{"on":true},
"deviceid":"Osram",
"topic":"zigbee2mqtt/Osram/set","_msgid":"66c8301c.d9e5e"}
Here is the output after a change of brightness:
{"on":true,"bri":52,"hue":0,"sat":254,"ct":199,"colormode":"ct","rgb":[255,0,0],
"payload":{"bri":52},
"deviceid":"Osram",
"topic":"zigbee2mqtt/Osram/set","_msgid":"fa2cb487.dd0f08"}
Note the payload. The first time it contains {"on":true}, which is what was asked from Alexa and the second one, "payload":{"bri":52}, which corresponds to setting the brightness to 20%.
Also, note that the deviceID contains the friendly name that I assigned in the node configuration (Osram).
The output you show is not consistent with my version. Maybe you are not getting any output because you haven't set the friendly name and the node doesn't get a match on names.

Thank you a lot for your kind quick answer, how can I set the friendly name? In the node there isn't such option, only the name of the node and the msg.topic that it will return as output. Should I delete all nodes and drag to flow new nodes? Only in that way the friendly name will appear in the options?
Another thing, which I suppose I haven't catch right, after set the friendly name somehow, will the Node Red export it as backup and an eventually restore of the flow will not mean to reset alexa and configure it again from reset?
Thank you!

I don't think you need to delete the nodes. After replacing the original index.js and index.html and restarting Node Red, you need to refresh the browser so that it reloads the modified index.html. Force reload or close and re-open the browser. You should than see the option to enter a friendly name in the configuration of each Echo Device Nodes. Set it to anything you want, as long as it's unique. Also, try a few times sending a request via Alexa so that the context is set for each node (you can check in Node Red under Context Data on the left panel with a arrow pointing down). BTW, this name is independent from the name of the node, which is used by Alexa. In my setup, I give it the same friendly name that I use with Zigbee2mqtt. This provides me with easy integration. From what I understand, the original author chose a different approach, using the name of the name as an alternative.
Hope this helps!

thank you, it helps, I'll try as soon as I can!
So seems the Alexa will get the ID of node, so export/import isn't possible, the only way to backup seems copy the flow .js file

@MosesHill, @verjus, @Barabba11
Please, let me know if the changes introduced in #64 will work for you.

Hello Datech,
Your implementation is conceptually different from what I tried. The way I approached this was to simply pass along the body of the request from Alexa (req.body) to the msg.payload.
If I understand correctly, in your implementation you update the current state with the request from Alexa as before, but you than look for differences between the updated state and previous state. The differences are reported in the changed attribute of the output msg.
I am not sure about this (I haven't tested it) but doesn't this introduce a dependency on the previous state? If the previous state was changed by the input node, for example on:false, and Alexa is asked to turn off that lamp, nothing will be reported in changed, right?
Wouldn't it be simpler to just pass along the req.body from Alexa?
Or keep the "changed" attribute but also pass along the request from Alexa (req.body) in msg.payload as above?

Agree. If I may suggest, it would be great to pass on an object all possible parameters that Alexa sending, and let us filter them to catch what we may find can be useful.. I don't know if for example there can be also the IP source address..
Thank you!

On the questions:
"If the previous state was changed by the input node, for example on:false, and Alexa is asked to turn off that lamp, nothing will be reported in changed, right?"
Yes, the changed attribute will be empty, but do we really need to do something with the payload if the devices state is not updated?
"Wouldn't it be simpler to just pass along the req.body from Alexa?
Or keep the "changed" attribute but also pass along the request from Alexa (req.body) in msg.payload as above?"
Having changed values could help you if you want to know how the state is modified.
For example:

the color is changed from green to red
the brightness is changed with more than 20%
the device color mode is changed from ct to hs

The Alexa input or the input payload, if you send it directly to the hub, is already visible in the output, because the state is modified based on the requested values

I agree, but for me, however, it is more intuitive to just pass along the request from Alexa (with the current state). That way, if I say turn off the light, I get the new state and the command that was used to change the state in the payload. The command is more important to me than the state because the state might not always be in synch with the physical state. It is also more intuitive to have the this command in the msg.payload. It is also more consistent with other systems, such as Openhab, that require a command in input. I have switched to Zigbee2mqtt and it can do both.
I think both work and are more or less equivalent. I just find it more intuitive to receive the new state along with Alexa's request in the message payload.
Finally, I wanted to thank you again for publishing and maintaining this great software that I enjoy using every day!

Datech can you please provide us the command? It will really simplify also my flow, thanks. Moreover I don't need to fail the first time after reload because I don't have a previous value to compare.
About the first value, have you checked the bug returning an on:false if the command asked Alexa was a bri only?

Definitely, it is more natural to think about the input instead of the changed attributes.
I will include the request also.

@vejus I still have no results, sorry, I overwritten the files, started again NR and yes with a refresh of the page I?m able to see freindly name. I put inside lijasdkljasdk as unique name, I told different requests to Alexa, I restarted again, no outputs coming.
By the way I haven't understand what you meant with "Also, try a few times sending a request via Alexa so that the context is set for each node (you can check in Node Red under Context Data on the left panel with a arrow pointing down)", what is this and what is this for? I don't think it's crucial to have an output..

Not sure why. Do you see the expected info when you look at  <IP_address>:80/api/pi ?
In my case, I see:
{"lights":{"Hue_1":{"state":{"on":false,"bri":254,"hue":0,"sat":254,"effect":"none","xy":[0,0],"ct":199,"alert":"none","colormode":"ct","mode":"homeautomation","reachable":true},"swupdate":{"state":"noupdates","lastinstall":"2019-10-04T16:11:36"},"type":"Extended color light", "name":"Kitchen Light",  "modelid":"LCT007","manufacturername":"Philips","productname":"Hue color lamp","capabilities": (...),
"Hue_1" is the Friendly Name and "Kitchen Light" is the Alexa Name.
Also, when you put a debug node at the exit of the Alexa Hub, do you see any message coming through? You should see something like:
 {"on":false,"bri":254,"hue":0,"sat":254,"ct":199,"colormode":"ct","rgb":[255,0,0],"payload":{"on":false},"deviceid":"Hue_1","topic":"","_msgid":"xxxxxxxxxxx"}
The deviceid should match the friendly name ("Hue_1" in my case).
The state information is kept in a flow context (see here for more info). You can see that context by looking under "Context Data" in the Node Red panel (under the arrow pointing down all the way to the right. You should see under Flow something like:
Hue_1 | { on: false, bri: 254, hue: 0, sat: 254, ct: 199 … }
My last thought is that if you don't have any output at all, maybe the code is not running properly. It might be that when you download the file, you introduced some funny characters or formatting and the code doesn't run because of an error. Make sure you downloaded the code in raw mode from github and maybe conpare the downloaded version with the original or make the edits yourself starting from the original code. Start by just changing the function payloadHandler(hubNode, deviceId, body) as described above.

Thanks for your help, I know what you mean, I can assure you the files are draggedextracted fom the zip file (winrar) and dropped to MobaXterm_Personal_11.0, in the same way I extract the Datech files and it become work again, so no dubts about that.
You're telling me that putting a dubug node on HUB will give friendly name, how HUB knows it? I thought it was a parameter of the device node, but maybe the script telling HUB to use it. Because I'm thinking if this friendly name you should specify too in Alexa app, so Echo sends it and that's why it exit from HUB..
You're suggesting me to place a debug node after hub, I did with Datech in past, I will with yours too.
As well I'll access the <IP_address>:80/api/pi ? and check
And context data ad you suggested, I?ve found it now.
On monday, I can't before ((( I hope to have success, but it's very stange.
I hope that Datech may realease his script with command too.. thanks for that )

@datech any chances to see your module ready to send commands sent by echo? Thank you!

#67 Meta attribute is included in payload output
Released in v0.1.9


What information Anchorwave require to be able to parse the gff file correctly?

Hi,
First, congratulations on the development of Anchorwave, it seems a great addition to the Comparative Genomics toolkit!
I have several questions, the majority linked to the "Empty CDS file from gff2seq #36" issue raised by nwespe (https://github.com/baoxingsong/AnchorWave/issues/36), so this issue is about what information Anchorwave require to be able to parse the gff file correctly?
As example, I will be using the information on the gff file of "Zea_mays.AGPv4.34.gff3" (information on the first gene is reproduced bellow) that you have used on your paper.
1	wareLab	chromosome	1	307041717	.	.	.	ID=chromosome:1

1	gramene	gene	44289	49837	.	+	.	ID=gene:Zm00001d027230;biotype=protein_coding;gene_id=Zm00001d027230;logic_name=maker_gene
1	gramene	mRNA	44289	49837	.	+	.	ID=transcript:Zm00001d027230_T001;Parent=gene:Zm00001d027230;biotype=protein_coding;transcript_id=Zm00001d027230_T001
1	gramene	five_prime_UTR	44289	44350	.	+	.	Parent=transcript:Zm00001d027230_T001
1	gramene	exon	44289	44947	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon1;constitutive=1;ensembl_end_phase=0;ensembl_phase=-1;exon_id=IRS1_030385-RA.exon1;rank=1
1	gramene	CDS	44351	44947	.	+	0	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	exon	45666	45803	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon2;constitutive=1;ensembl_end_phase=0;ensembl_phase=0;exon_id=IRS1_030385-RA.exon2;rank=2
1	gramene	CDS	45666	45803	.	+	0	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	exon	45888	46133	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon3;constitutive=1;ensembl_end_phase=0;ensembl_phase=0;exon_id=IRS1_030385-RA.exon3;rank=3
1	gramene	CDS	45888	46133	.	+	0	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	exon	46229	46342	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon4;constitutive=1;ensembl_end_phase=0;ensembl_phase=0;exon_id=IRS1_030385-RA.exon4;rank=4
1	gramene	CDS	46229	46342	.	+	0	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	exon	46451	46633	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon5;constitutive=1;ensembl_end_phase=0;ensembl_phase=0;exon_id=IRS1_030385-RA.exon5;rank=5
1	gramene	CDS	46451	46633	.	+	0	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	exon	47045	47262	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon6;constitutive=1;ensembl_end_phase=2;ensembl_phase=0;exon_id=IRS1_030385-RA.exon6;rank=6
1	gramene	CDS	47045	47262	.	+	0	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	CDS	47650	47995	.	+	1	ID=CDS:Zm00001d027230_T001;Parent=transcript:Zm00001d027230_T001;protein_id=Zm00001d027230_T001
1	gramene	exon	47650	48111	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon7;constitutive=1;ensembl_end_phase=-1;ensembl_phase=2;exon_id=IRS1_030385-RA.exon7;rank=7
1	gramene	three_prime_UTR	47996	48111	.	+	.	Parent=transcript:Zm00001d027230_T001
1	gramene	exon	48200	49247	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon8;constitutive=1;ensembl_end_phase=-1;ensembl_phase=-1;exon_id=IRS1_030385-RA.exon8;rank=8
1	gramene	three_prime_UTR	48200	49247	.	+	.	Parent=transcript:Zm00001d027230_T001
1	gramene	exon	49330	49837	.	+	.	Parent=transcript:Zm00001d027230_T001;Name=IRS1_030385-RA.exon9;constitutive=1;ensembl_end_phase=-1;ensembl_phase=-1;exon_id=IRS1_030385-RA.exon9;rank=9
1	gramene	three_prime_UTR	49330	49837	.	+	.	Parent=transcript:Zm00001d027230_T001
1	gramene	gene	50877	55716	.	-	.	ID=gene:Zm00001d027231;biotype=protein_coding;gene_id=Zm00001d027231;logic_name=maker_gene
First question:
Assuming that the "CDS row" is necessary, is it also required the information comprised in the "gene row", "exon row" or "mRNA row"?
Second question:
For Anchorwave to be able to parse the gff file, does it require that the ID of CDS is of type "ID=CDS:Zm00001d027230_T001", with the "CDS:" prefix?
Third question:
In case the "gene row" is also required, will Anchorwave require the ID of gene is of type "ID=gene:Zm00001d027230", with the "gene:" prefix?
Fourth question:
I am applying your software to a number of strains belonging to a specific bacterial species whose genome sequence that have several rearrangements, inversions, duplications, insertions, etc.
Bacterial genes do not have introns, is there some impediment using a gff file where genes are equivalent to CDSs, it is foreseeable that Anchorwave will raise some error at any point of the multistage pipeline?
Best regards,
Paulo Dias
PS: in attachment, I send a gff file automatically generated by Prokka, the most used software for genome annotation of Bacteria, that also was unable to generate the cds.fa file as pointed out in the "Empty CDS file from gff2seq #36" issue.
input_compatible_2.zip

Thanks for giving AnchorWave a try. I will set a specific parameter for bacteria in the future version.
But currently, we are using
"^(\S*)\t([\s\S])\tCDS\t(\S)\t(\S*)\t(\S*)\t(\S*)\t(\S*)\t([\s\S]*)Parent=([abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.:_-]+)"
to parse the CDS lines.
And using "ID=(\S+?);.*Parent=(\S+?);" to parse  mRNA records.
Could you also share the genome fasta file please? So that I could test the updating code.

Dear Baoxing,

I think Anchorwave performs very well at the bacteria level (and potentially at the yeast and fungi level), and have lots of potential for performing Comparative Genomics between this type of organisms because their genome sequences frequently present inversions, duplications, translocations, etc.

At the level of higher Eukaryotes, I have try to parse the files you have made available regarding the case Zea_mays/Sorghum_bicolor, and the feed-back that I got is that Anchorwave becomes slow at that level.
Since you warned that
"you could increase the value of -t to run the command using multiple threads. By increasing each thread, ~50Gb more memory would be used. The memory cost is associated with sequence diversity. The above dataset is roughly represent the dataset that cost largest memory."
maybe you could try to optimize Anchorwave so it assigns less memory per thread (see suggestions bellow!).
In fact, I believe that your software might be very useful in the fields of Pangenome graphs since it could provide an alternative alignment software, sensible to genome rearrangenoments.
Dont know if you know that this new field, if you dont you can check it at
https://github.com/pangenome/pggb
https://github.com/ComparativeGenomicsToolkit/cactus
https://github.com/vgteam/vg
https://github.com/pangenome/odgi
https://github.com/ekg/seqwish

Regarding the gff file, if you intend to make it available for bacteria, you should take into consideration two points:

First, frequently researchers are sequencing many related microorganisms and dont have information about the mRNA, where the transcript starts and where it ends, because many of these are not at the level of model-organisms, they might be sequenced from a perspective of biodiversity, taxonomy, metagenomics, etc, and researchers will not take the time to characterize in detail the transcriptome, will just apply a general genome annotation tool such as Prokka to identify where genes start and where they end (for instance, the gff file will only exhibit the set of CDSs, and there will be no rows with "gene", "mRNA" and "exon" information)
Second, bacteria dont have introns, so you should allow researchers to submit to Anchorwave when using the option "bacteria" as input a simpler gff file, only comprising information about the CDSs (or genes, if you prefer)
Because previously I had the necessity to parse gff files and be able to adapt them, or convert them into a tablature format, I have a series of scripts that allowed me to convert a PROKKA gff file comprising only CDSs rows (file "input.gff" in attachment) to a gff file that Anchorwave can use as input (file "gff3_output_2" in attachment).
So, I already have bypassed the problem that it is reported in this issue!
What is the main lessons that I have learned to parse the "input.gff" into "gff3_output_2"?
Bellow is the first gene of "gff3_output_2" to exemplify the parsing:
Lactococcus_lactis_AI06_CP009472.1	IST	gene	587	1954	.	+	.	ID=gene:JBNCDKON_00001;gene_id=JBNCDKON_00001
Lactococcus_lactis_AI06_CP009472.1	IST	mRNA	587	1954	.	+	0	ID=transcript:JBNCDKON_00001;Parent=gene:JBNCDKON_00001;transcript_id=JBNCDKON_00001
Lactococcus_lactis_AI06_CP009472.1	IST	exon	587	1954	.	+	0	Parent=transcript:JBNCDKON_00001
Lactococcus_lactis_AI06_CP009472.1	IST	CDS	587	1954	.	+	0	ID=CDS:JBNCDKON_00001;Parent=transcript:JBNCDKON_00001;protein_id=JBNCDKON_00001
First, I altered my script to add the "gene", "mRNA" and "exon" rows in addition to the "CDS" row.
In fact, the CDS row is not necessary, Anchorwave will accept a gff file comprising only "gene", "mRNA" and "exon" and will parse it sucessfully!
Second, the key point for Anchorwave to be able to parse the gff file, is that the "Parent=gene:JBNCDKON_00001" information has to be present in the "mRNA" row, so that the software can map the transcript into the corresponding gene.
Third, in case of bacteria, since there are no alternative transcripts, I have also changed the script that I was using to parse the gff file to include the suffix "_T001", "_T002", ... , to handle multiple transcripts in parallel solution to what I have observed in the Zea_mays/Sorghum_bicolor example.
However, this change to the script code and change of mRNA/exon/CDS name was not necessary, since the software was able to successfully parse the gff file when the gene name, the transcript name, the exon name and the CDS name were the same!

Suggestions to improve Anchorwave:

A) First suggestion is the one mentioned above, optimization of memory per thread. I'm not good at hardware optimization, but maybe you can follow the example of seqwish (https://github.com/ekg/seqwish/issues/16), where instead of RAM, part of the computation is stored as files in the hard drive, so it can become lighter regarding memory at the cost of disk space?
B) Regarding the gff files, in order to make the life of researchers more easy when testing the Anchorwave software, maybe you should include detailed information in the github page describing what rows are necessary to be inside the gff file so that the software can parse the gff file successfully.
In complement, you should include in the software the ability for users to specify at the level of the "anchorwave gff2seq" command the option of using  different types of gff files as input, in particular those that are frequently generated and that we can obtain either from reference databases (Genbank, RefSeq, etc), or from highly used genome annotation software (Prokka, Funannotate or Maker, etc).
C) My final suggestion concerns the possibility of giving users the ability of getting the output in different formats other than the maf format! For instance, getting the output in the sam format or even in the paf format (so it can be used to feed the seqwish software and construct the corresponding pangenome graph, see https://github.com/ekg/seqwish) would be very useful!
I will be waiting for the next Anchorwave versions, but I have to congratulate you on the development of this software, it is very useful and powerful as it is right now!
Kind regards,
Paulo Dias
two_gff_files.zip

Hi @baoxingsong,
I'm attempting to try your code on my study system and am running into this same issue.
In the above, you provided the regex pattern for CDS records, but there appears to be some issue(s) with the regex pattern:
^(\S*)\t([\s\S])\tCDS\t(\S)\t(\S*)\t(\S*)\t(\S*)\t(\S*)\t([\s\S]*)Parent=([abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.:_-]+)
               ^          ^
               Missing asterisks

Presumably, since you want to extract the chromosome and positional information from GFF3 files, the following would be a more-specific pattern (requiring those fields be defined):
^(\S+)\t([\s\S]*)\tCDS\t(\d+)\t(\d+)\t(\S+)\t([+\-\.\?]*)\t([012\.])\t(.*)Parent=([^;=&,]+)

The above also allows a more flexible character set for the Parent attribute value, but still parses the key=value pair correctly; since ;=&, are reserved characters (https://github.com/The-Sequence-Ontology/Specifications/blob/master/gff3.md) in the GFF3 attributes field, allow matching everything else.
Likewise for mRNA records:
^(\S+)\t([\s\S]*)\t(mRNA|.*transcript)\t(\d+)\t(\d+)\t(\S+)\t([+\-\.\?]*)\t(\S+)\t.*ID=([^;=&,]+);.*Parent=([^;=&,]+)

The above pattern allows matching mRNA and (primary_)transcript records, requires strand to be defined, and is more permissive of the characters allowed in the ID and Parent attribute values (your current pattern requiring ; at the end of the Parent attribute value fails to parse GFF3 files where the Parent attribute is the last key=value pair).
I hope this helps.
-Jessen

I put my version of gff file here and hope it be helpful for those working with bacterium.
The minimum information required in gff is as bellow:
NC_000962.3	RefSeq	gene	1	1524	.	+	.	ID=gene0
NC_000962.3	RefSeq	mRNA	1	1524	.	+	0	ID=gene0;Parent=gene0
NC_000962.3	RefSeq	CDS	1	1524	.	+	0	ID=cds0;Parent=gene0

The mRNA row is needed and added using simple awk command. CDS is needed as there is no exon here.


Refactor person settings (EXPOSUREAPP-11989)

Migrate to DataStore in PersonSettings
Due to regrouping and Dcc Reissuance , a new way to save settings per person is needed independently from how certificate are saved. HERE you go
Provided a way to show / dismiss Dcc Reissuance badge
Provided a way to show / dismiss Dcc Booster badge (Should be used as part of regrouping migration)

TODO

[x] Unit tests

Ticket
https://jira-ibs.wbs.net.sap/browse/EXPOSUREAPP-11989

@mtwalli Will the properties in VaccinatedPersonData that are now in PersonSettings will be removed in a different PR?

@mtwalli Will the properties in VaccinatedPersonData that are now in PersonSettings will be removed in a different PR?

I think it should be part of the re-grouping migration,
@jurajkusnier ?


Solidity test fail on default template

Execution failed on default template with the following error.
I suspect that the problem lies when truffle try to run solidity unit tests like TestMetacoin.sol

Truffle version: v3.2.4
Node version: v6.10.3
npm version: 3.10.10

test@codevm:~/solcover$ node runCoveredTests.js
instrumenting  originalContracts/ConvertLib.sol
instrumenting  originalContracts/MetaCoin.sol
instrumenting  originalContracts/Migrations.sol
Compiling ./contracts/ConvertLib.sol...
Compiling ./contracts/MetaCoin.sol...
Compiling ./contracts/Migrations.sol...
Compiling ./test/TestMetacoin.sol...
Compiling truffle/Assert.sol...
Compiling truffle/DeployedAddresses.sol...

  TestMetacoin
    ✓ testInitialBalanceUsingDeployedContract (107ms)
    1) testInitialBalanceWithNewMetaCoin
    > No events were emitted

  Contract: MetaCoin
    ✓ should put 10000 MetaCoin in the first account (97ms)
    ✓ should call a function that depends on a linked library (216ms)
    ✓ should send coin correctly (414ms)

  4 passing (2s)
  1 failing

  1) TestMetacoin testInitialBalanceWithNewMetaCoin:
     Error: VM Exception while processing transaction: invalid opcode
      at Object.InvalidResponse (/usr/local/lib/node_modules/truffle/node_modules/web3/lib/web3/errors.js:35:16)
      at /usr/local/lib/node_modules/truffle/node_modules/web3/lib/web3/requestmanager.js:86:36
      at XMLHttpRequest.request.onreadystatechange (/usr/local/lib/node_modules/truffle/node_modules/web3/lib/web3/httpprovider.js:118:13)
      at XMLHttpRequestEventTarget.dispatchEvent (/usr/local/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:64:18)
      at XMLHttpRequest._setReadyState (/usr/local/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:354:12)
      at XMLHttpRequest._onHttpResponseEnd (/usr/local/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:509:12)
      at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:469:24)
      at endReadableNT (_stream_readable.js:974:12)

+1 I'm seeing the same problem with my unit tests... Running truffle test and everything works, running via this plugin I get invalid opcode.

Closing this now that we're over at solidity-coverage. If you still encounter this problem with the version installed from npm, via npm install solidity-coverage, please let us know over there.


Mutex cannot return a mutable pointer

Currently Mutex implementation in cortex-m returns a normal references instead of mutable one. FWIW bare_metal (used by rtic for example) returns mutable reference and is more full-featured (it may be worthwhile to use it to provide interoperability).

Seems like a dupe of #224

I'm triaging Cortex-M issues today, this seems like a duplicate of #224 so I'm going to close it; feel free to re-open if I missed anything!


Fix Vulnerability: User enumeration via login response time

I want to merge this change because...
I have identified and fixed an user enumeration vulnerability in the authentication process. This vulnerability allowed attackers to determine if an email was registered in the system via response time.

Impact

[ ] New migrations
[ ] New/Updated API fields or mutations
[ ] Deprecated API fields or mutations
[ ] Removed API types, fields, or mutations

Docs

[ ] Link to documentation: https://docs.google.com/document/d/137G3aHZ-aMdV7JFDzgkWsygjUHoL2NTy/edit?usp=sharing&ouid=101128278685376508073&rtpof=true&sd=true

Pull Request Checklist

[ ] Privileged queries and mutations are either absent or guarded by proper permission checks
[ ] Database queries are optimized and the number of queries is constant
[ ] Database migrations are either absent or optimized for zero downtime
[ ] The changes are covered by test cases
[ ] All new fields/inputs/mutations have proper labels added (ADDED_IN_X, PREVIEW_FEATURE, etc.)
[ ] All migrations have proper dependencies
[ ] All indexes are added concurrently in migrations
[ ] All RunSql and RunPython migrations have revert option defined

Hello! First of all thank you for the contribution!
Regarding the PR, I will close this one, as we are doing the changes for the response time in other place. Please also don't make target branch as other feature branch unless for specific reason, as it is hard to distinguish actual changes made here.
Regarding the solutions from your changes, unfortunately getting all users from the database is not the way to go for this, and this should not be done in throttling logic as well.

Hello! First of all thank you for the contribution! Regarding the PR, I will close this one, as we are doing the changes for the response time in other place. Please also don't make target branch as other feature branch unless for specific reason, as it is hard to distinguish actual changes made here.
Regarding the solutions from your changes, unfortunately getting all users from the database is not the way to go for this, and this should not be done in throttling logic as well.

Thanks for the feedback. I would like to know where is the place that I can make my contribution.


Add TortoiseORM examples for related objects and computed fields

Pull Request Checklist

[x] Have you followed the guidelines in the Starlite Contribution Guidelines?
[ ] Have you got 100% test coverage on new code?
[x] Have you updated the prose documentation?
[ ] Have you updated the reference documentation?

By submitting this issue, you agree to:

follow Starlite's Code of Conduct
follow Starlite's Contribution Guidelines

Description
Added examples for computed fields and creating related objects in TortoiseORM plugin
Close Issue(s)
Closes:  #1351
Remaining Questions:
Maybe this is for another issue, but if I wanted to return a call with nested objects (Say a tournament and its events), I would then have to create my own DTO, I'm assuming? Might be a nice feature to start including this in the plugin ala https://www.django-rest-framework.org/api-guide/relations/#nested-relationships

@all-contributors please add @Galdanwing for docs


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/43615

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/44165

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/54371

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/74870

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/100853

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/108543

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/110310


Modify UdpSocket::connect() to take SocketAddr by ref (#667)

Change argument for UdpSocket::connect() to take &SocketAddr. This is more consistent with other UdpSocket methods, plus the TcpSocket::connect() method. Closes issue #667.

Thanks @bobbo! This looks good to me :+1:

This is unfortunately a breaking change :(
So, this can't be merged until an 0.7 release is planned. or UdpSocket::connect is deprecated in favor of UdpSocket::connect2 (and it is switched back or 0.7).

If you are up for it, could you rebase this against the v0.7.x branch? I am using that as staging for v0.7.x

Hi @carllerche, I've rebased on top of v0.7.x and changed PR target branch.


DAG - dbt_example.py error:Expecting value: line 1 column 1 (char 0)

Hi,
Thank you for this awesome toolkit, really appreciate your hard work.
I am trying to run "Toolkit #1: Local Desktop Kubernetes Airflow Deployment" on my machine and facing the below error (in bold):
Changes done:

changed dbt version to 0.18.1
have tried running "dbt_bigquery_example" on CLI and it's working fine, looks like this is some airflow issue
my local python virtual env is 3.7.9 (if it helps)

Could you please help ?
Error logs:
[2020-11-02 13:13:03,349] {logging_mixin.py:112} INFO - [2020-11-02 13:13:03,349] {pod_launcher.py:125} INFO - b"Cloning into 'composer-template'...\n"
[2020-11-02 13:13:07,742] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,742] {pod_launcher.py:125} INFO - b'+ cd composer-template/dbt_bigquery_example\n'
[2020-11-02 13:13:07,743] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,743] {pod_launcher.py:125} INFO - b'+ export PROJECT_ID=xxxxx\n'
[2020-11-02 13:13:07,743] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,743] {pod_launcher.py:125} INFO - b'+ PROJECT_ID=xxxxx\n'
[2020-11-02 13:13:07,745] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,745] {pod_launcher.py:125} INFO - b'++ pwd\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ export DBT_PROFILES_DIR=/dbt/composer-template/dbt_bigquery_example\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ DBT_PROFILES_DIR=/dbt/composer-template/dbt_bigquery_example\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ export DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json\n'
[2020-11-02 13:13:07,747] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ dbt run --target service_account_runs\n'
[2020-11-02 13:13:08,502] {logging_mixin.py:112} INFO - [2020-11-02 13:13:08,501] {pod_launcher.py:125} INFO - b'Running with dbt=0.18.1\n'
[2020-11-02 13:13:10,056] {logging_mixin.py:112} INFO - [2020-11-02 13:13:10,056] {pod_launcher.py:125} INFO - b'Found 8 models, 21 tests, 1 snapshot, 0 analyses, 155 macros, 0 operations, 3 seed files, 3 sources\n'
[2020-11-02 13:13:10,061] {logging_mixin.py:112} INFO - [2020-11-02 13:13:10,061] {pod_launcher.py:125} INFO - b'\n'
[2020-11-02 13:13:11,040] {logging_mixin.py:112} INFO - [2020-11-02 13:13:11,040] {pod_launcher.py:125} INFO - b'Encountered an error:\n'
[2020-11-02 13:13:11,041] {logging_mixin.py:112} INFO - [2020-11-02 13:13:11,041] {pod_launcher.py:125} INFO - b'Expecting value: line 1 column 1 (char 0)\n'
[2020-11-02 13:13:12,059] {logging_mixin.py:112} INFO - [2020-11-02 13:13:12,059] {pod_launcher.py:142} INFO - Event: dbt-run-8cec9cee had an event of type Running
[2020-11-02 13:13:12,060] {logging_mixin.py:112} INFO - [2020-11-02 13:13:12,060] {pod_launcher.py:135} INFO - Pod dbt-run-8cec9cee has state running
[2020-11-02 13:13:14,068] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,067] {pod_launcher.py:142} INFO - Event: dbt-run-8cec9cee had an event of type Failed
[2020-11-02 13:13:14,068] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,068] {pod_launcher.py:234} INFO - Event with job id dbt-run-8cec9cee Failed
[2020-11-02 13:13:14,072] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,072] {pod_launcher.py:142} INFO - Event: dbt-run-8cec9cee had an event of type Failed
[2020-11-02 13:13:14,072] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,072] {pod_launcher.py:234} INFO - Event with job id dbt-run-8cec9cee Failed

Welcome and thanks for using the toolkit!
Problem: dbt run isn't working as expected and errors out with a vague error message.
Non-Issues:

your local environment shouldn't matter as the dbt docker image is used
dbt running on your local desktop CLI will inherently have a different setup to how dbt is run within a docker container within airflow

This may be the issue
[2020-11-02 13:13:07,743] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,743] {pod_launcher.py:125} INFO - b'+ export PROJECT_ID=xxxxx\n'
[2020-11-02 13:13:07,743] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,743] {pod_launcher.py:125} INFO - b'+ PROJECT_ID=xxxxx\n'

I noticed the PROJECT_ID is being set to xxxxx which will override the profiles.yml dbt config.
Did you run the below instructions and replace with your relevant values?
#!/bin/bash
# follow terminal prompt after entering below command
# leave this terminal open to sustain airflow webserver
# Set of environment variables
export ENV="dev"
export PROJECT_ID="wam-bam-258119"
export DOCKER_DBT_IMG="gcr.io/$PROJECT_ID/dbt_docker:$ENV-latest"

source deploy_local_desktop_airflow.sh

After running the above and replacing PROJECT_ID with an actual project name, it should work.
If it doesn't, can you detail exact steps to reproduce the above error logs?
You changed directory structures too: [2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ export DBT_PROFILES_DIR=/dbt/composer-template/dbt_bigquery_example\n'
This may be a compounding factor to the issue you're seeing above, but I won't know until I see exact steps to reproduce the problem.

Thanks for the response and apologies for not making myself clear.
I had masked PROJECT_NAME in logs due to privacy issues and yes I had already ran all export statements above post making changes to PROJECT_ID
I tried debugging more on the error and found something interesting:
a. In dbt_example.dag , "dbt-debug" task fails internally with the same error message "Expecting value: line 1 column 1 (char 0)" though the status shown is green. Please find below the logs (masked project & database name below) :
[2020-11-03 13:17:32,392] {logging_mixin.py:112} INFO - [2020-11-03 13:17:32,391] {pod_launcher.py:125} INFO - b"+ export 'GIT_SSH_COMMAND=ssh -i .ssh/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'\n"
[2020-11-03 13:17:32,392] {logging_mixin.py:112} INFO - [2020-11-03 13:17:32,392] {pod_launcher.py:125} INFO - b"+ GIT_SSH_COMMAND='ssh -i .ssh/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'\n"
[2020-11-03 13:17:32,393] {logging_mixin.py:112} INFO - [2020-11-03 13:17:32,393] {pod_launcher.py:125} INFO - b'+ git clone -b master git@github.com:ggupta36/airflow-toolkit.git\n'
[2020-11-03 13:17:32,394] {logging_mixin.py:112} INFO - [2020-11-03 13:17:32,394] {pod_launcher.py:125} INFO - b"Cloning into 'airflow-toolkit'...\n"
[2020-11-03 13:17:36,334] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,333] {pod_launcher.py:125} INFO - b'+ cd airflow-toolkit/dbt_bigquery_example\n'
[2020-11-03 13:17:36,334] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,334] {pod_launcher.py:125} INFO - b'+ export PROJECT_ID=xxxxx\n'
[2020-11-03 13:17:36,334] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,334] {pod_launcher.py:125} INFO - b'+ PROJECT_ID=xxxxx\n'
[2020-11-03 13:17:36,339] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,338] {pod_launcher.py:125} INFO - b'++ pwd\n'
[2020-11-03 13:17:36,340] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,340] {pod_launcher.py:125} INFO - b'+ export DBT_PROFILES_DIR=/dbt/airflow-toolkit/dbt_bigquery_example\n'
[2020-11-03 13:17:36,341] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,340] {pod_launcher.py:125} INFO - b'+ DBT_PROFILES_DIR=/dbt/airflow-toolkit/dbt_bigquery_example\n'
[2020-11-03 13:17:36,341] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,341] {pod_launcher.py:125} INFO - b'+ export DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json\n'
[2020-11-03 13:17:36,341] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,341] {pod_launcher.py:125} INFO - b'+ DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json\n'
[2020-11-03 13:17:36,343] {logging_mixin.py:112} INFO - [2020-11-03 13:17:36,343] {pod_launcher.py:125} INFO - b'+ dbt debug --target service_account_runs\n'
[2020-11-03 13:17:37,055] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,055] {pod_launcher.py:125} INFO - b'Running with dbt=0.18.1\n'
[2020-11-03 13:17:37,065] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,063] {pod_launcher.py:125} INFO - b'dbt version: 0.18.1\n'
[2020-11-03 13:17:37,065] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,065] {pod_launcher.py:125} INFO - b'python version: 3.7.9\n'
[2020-11-03 13:17:37,066] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,066] {pod_launcher.py:125} INFO - b'python path: /usr/local/bin/python\n'
[2020-11-03 13:17:37,066] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,066] {pod_launcher.py:125} INFO - b'os info: Linux-4.19.76-linuxkit-x86_64-with-debian-10.6\n'
[2020-11-03 13:17:37,067] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,067] {pod_launcher.py:125} INFO - b'Using profiles.yml file at /dbt/airflow-toolkit/dbt_bigquery_example/profiles.yml\n'
[2020-11-03 13:17:37,071] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,069] {pod_launcher.py:125} INFO - b'Using dbt_project.yml file at /dbt/airflow-toolkit/dbt_bigquery_example/dbt_project.yml\n'
[2020-11-03 13:17:37,072] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,072] {pod_launcher.py:125} INFO - b'\n'
[2020-11-03 13:17:37,287] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,287] {pod_launcher.py:125} INFO - b'Configuration:\n'
[2020-11-03 13:17:37,288] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,288] {pod_launcher.py:125} INFO - b'  profiles.yml file [OK found and valid]\n'
[2020-11-03 13:17:37,288] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,288] {pod_launcher.py:125} INFO - b'  dbt_project.yml file [OK found and valid]\n'
[2020-11-03 13:17:37,288] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,288] {pod_launcher.py:125} INFO - b'\n'
[2020-11-03 13:17:37,288] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,288] {pod_launcher.py:125} INFO - b'Required dependencies:\n'
[2020-11-03 13:17:37,295] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,295] {pod_launcher.py:125} INFO - b' - git [OK found]\n'
[2020-11-03 13:17:37,295] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,295] {pod_launcher.py:125} INFO - b'\n'
[2020-11-03 13:17:37,298] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,297] {pod_launcher.py:125} INFO - b'Connection:\n'
[2020-11-03 13:17:37,298] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,298] {pod_launcher.py:125} INFO - b'  method: service-account\n'
[2020-11-03 13:17:37,298] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,298] {pod_launcher.py:125} INFO - b'  database: xxxxx\n'
[2020-11-03 13:17:37,298] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,298] {pod_launcher.py:125} INFO - b'  schema: dbt_bq_example\n'
[2020-11-03 13:17:37,299] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,299] {pod_launcher.py:125} INFO - b'  location: US\n'
[2020-11-03 13:17:37,299] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,299] {pod_launcher.py:125} INFO - b'  priority: interactive\n'
[2020-11-03 13:17:37,299] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,299] {pod_launcher.py:125} INFO - b'  timeout_seconds: 300\n'
[2020-11-03 13:17:37,299] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,299] {pod_launcher.py:125} INFO - b'  maximum_bytes_billed: None\n'
**[2020-11-03 13:17:37,308] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,308] {pod_launcher.py:125} INFO - b'  Connection test: ERROR\n'
[2020-11-03 13:17:37,308] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,308] {pod_launcher.py:125} INFO - b'\n'
[2020-11-03 13:17:37,309] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,308] {pod_launcher.py:125} INFO - b'dbt was unable to connect to the specified database.\n'
[2020-11-03 13:17:37,309] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,309] {pod_launcher.py:125} INFO - b'The database returned the following error:\n'**
[2020-11-03 13:17:37,309] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,309] {pod_launcher.py:125} INFO - b'\n'
**[2020-11-03 13:17:37,309] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,309] {pod_launcher.py:125} INFO - b'  >Expecting value: line 1 column 1 (char 0)\n'**
[2020-11-03 13:17:37,310] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,309] {pod_launcher.py:125} INFO - b'\n'
[2020-11-03 13:17:37,310] {logging_mixin.py:112} INFO - [2020-11-03 13:17:37,310] {pod_launcher.py:125} INFO - b'Check your database credentials and try again. For more information, visit:\n'

This signifies that dbt somehow is not able to receive service account creds from env variable DBT_GOOGLE_BIGQUERY_KEYFILE to connect to DB which justifies that weird error message which usually gets thrown in case the json being read (cred file in our case) is empty
Please find below steps to reproduce:

Cloned the repo
Followed all steps mentioned in "One Time Setup for All Toolkits" & "Toolkit #1: Local Desktop Kubernetes Airflow Deployment" with minor change - Instead of using cloud source repo, I used github and made following changes in airflowutils.py file: (masked PROJECT_ID below)

GIT_REPO = "git@github.com:ggupta36/airflow-toolkit.git"
PROJECT_ID = "xxxxx"
DBT_IMAGE = f"gcr.io/{PROJECT_ID}/dbt_docker:dev-latest"
FOLDER_NAME = GIT_REPO.split("/")[1].split(".")[0]

dbt_setup_cmds = f"""
    {git_clone_cmds} &&
    cd {FOLDER_NAME}/dbt_bigquery_example &&
    export PROJECT_ID={PROJECT_ID} &&
    export DBT_PROFILES_DIR=$(pwd) &&
    export DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json """

I updated DBT_SERVICE_ACCOUNT in kube_secrets.py to volume type and it worked but later realized I was missing invoking entrypoint.sh in dbt_setup_cmds above. It's working with both env and volume type.
Thanks for your help

@ggupta36 Yeah. I reproduced and resolved your issue this morning by simply adding the /entrypoint.sh command within the git_clone_cmds object in airflow_utils.py.
committed changes in test branch: HERE
Before:
git_clone_cmds = f"""
    export GIT_SSH_COMMAND='ssh -i .ssh/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' &&
    git clone -b {GIT_BRANCH} {GIT_REPO}"""

After:
git_clone_cmds = f"""
    /entrypoint.sh &&
    export GIT_SSH_COMMAND='ssh -i .ssh/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' &&
    git clone -b {GIT_BRANCH} {GIT_REPO}"""

Making it a volume should work fine for you too! I specifically designed it where /entrypoint.sh is invoked so there are no implicit commands being run within a kubernetes pod.
Your approach to using a volume is easiest in this situation. I go through the extra steps in case the docker image is used for isolated development.

I'm going to close this as you resolved the issue yourself!


Bug: Percentage should be after the number and not before

before:

after:

why the name itself is not in the right order? what if we get coupon with correct order?

yep , im checking the stripe api  , maybe i'll fix from backend


Feat/add Coin98 wallet

Description

PLEASE NOTE- Checklist must be complete prior to review.
Checklist

[x] Increment the version field in package.json of the package you have made changes in following semantic versioning and using alpha release tagging
[x] Check the box that allows repo maintainers to update this PR
[x] Test locally to make sure this feature/fix works
[x] Run yarn check-all to confirm there are not any associated errors
[ ] Confirm this PR passes Circle CI checks
[x] Add or update relevant information in the documentation

Docs Checklist

[x] Include a screenshot of any changes (see docs README on running locally)
[x] Add/update the appropriate package README (if applicable)
[x] Add/update the related module in the docs demo (if applicable)
[x] Add/update the related package in the docs/package.json file (if applicable)

If this PR includes changes to add an injected wallet or SDK wallet module:
Please complete the following using the internal demo package.
To run this demo use the command yarn && yarn dev to get the project running at http://localhost:8080/
Tests with demo app (injected)

[ ] send transaction
[ ] switch chains
[ ] sign message
[ ] sign typed message
[ ] disconnect

Tests with demo app (SDK)

[ ] send transaction
[ ] switch chains
[ ] sign message
[ ] sign typed message
[ ] disconnect

@khiemsoft this is looking good- can you give me access to the repo so I can test locally?

@khiemsoft this is looking good- can you give me access to the repo so I can test locally?

It's done. Could you please check? Thanks for your review.

@khiemsoft You need to update the version on the package.json for  packages/injected/package.json to 2.10.3-alpha.2
and then updated the "@web3-onboard/injected-wallets" version in packages/demo/package.json to match in order to be able to follow the instructions for starting the demo for testing- I was unable to connect to coin98 wallet on desktop or mobile. Please run through all tests listed in the checklist above (send transaction, sign message, etc) and let me know when you have completed this. Thanks!

@khiemsoft You need to update the version on the package.json for packages/injected/package.json to 2.10.3-alpha.2 and then updated the "@web3-onboard/injected-wallets" version in packages/demo/package.json to match in order to be able to follow the instructions for starting the demo for testing- I was unable to connect to coin98 wallet on desktop or mobile. Please run through all tests listed in the checklist above (send transaction, sign message, etc) and let me know when you have completed this. Thanks!

Thanks for your review. All changes are updated and have already been tested. Could you please check?

@khiemsoft Thanks for updating! I tested locally and was able to connect, disconnect, and sign a message.
I could not sign a typed message or switch chains.

@khiemsoft Thanks for updating! I tested locally and was able to connect, disconnect, and sign a message. I could not sign a typed message or switch chains.

Hi, this is a bug in the custom network in the Coin98 Wallet Extension. We have just updated to version 7.2.7. Could you please update it and test it again? Thanks for your review.

@khiemsoft I can now successfully connect, disconnect, sign a message, sign a typed message, and switch chains.
I cannot send a transaction (attempted on Goerli)

@khiemsoft I can now successfully connect, disconnect, sign a message, sign a typed message, and switch chains. I cannot send a transaction (attempted on Goerli)

@leightkt  Hi, could you please check again? My transaction is still fine. Here is my example video
https://github.com/blocknative/web3-onboard/assets/87562184/3332961b-58bc-4131-966a-e97580f5dede

@khiemsoft I'm still getting the same error. I wonder if it has something to do with how I've set up my wallet? I've tried to transfer Goerli ETH into my coin98 from my metamask wallet twice - both transactions were successful on etherscan, but my coin98 wallet is still telling me I have a 0 balance:

@khiemsoft I'm still getting the same error. I wonder if it has something to do with how I've set up my wallet? I've tried to transfer Goerli ETH into my coin98 from my metamask wallet twice - both transactions were successful on etherscan, but my coin98 wallet is still telling me I have a 0 balance:

The balance of the testnet main token will not be displayed on this main display screen. You need to scroll down, and you will see a custom token like this. @leightkt

@khiemsoft Ok thanks- I'm still getting the same error as above when trying to send a tx

@khiemsoft Ok thanks- I'm still getting the same error as above when trying to send a tx

Hi, I just found that your key when added to the Infura Goerli custom network is an error.
Could you please add it from the Coin98 Wallet custom network or try another key in Infura? Thanks for your review.

@khiemsoft I tried a different rpcURL and I tried adding from the wallet directly and I still get the same error

@khiemsoft I tried a different rpcURL and I tried adding from the wallet directly and I still get the same error

Hi, this is our re-test video. This is how to reproduce your bug and how to change this key. Thanks for your review. @leightkt
https://github.com/blocknative/web3-onboard/assets/87562184/cd5dac50-19cc-460f-b3a8-430f8ef8e10b

@khiemsoft I had to go through a few extra steps to reactivate a goerli wallet after removing the network but I got there and was able to send a transaction.
Please pull in the most recent changes from develop into your branch and resolve any conflicts and this will be good to go.

Conflicts fixed. Thanks for your review. @leightkt

@khiemsoft left a comment to update the injected package version
please also update package/demo/package.json "@web3-onboard/injected-wallets" to "^2.10.5-alpha.1"

@khiemsoft left a comment to update the injected package version please also update package/demo/package.json "@web3-onboard/injected-wallets" to "^2.10.5-alpha.1"

Package version updated Thanks for your review.

@khiemsoft thanks so much for getting this across the finish line! It will go out in our next release!


Demo GraphQL Relay client

NB: doesn't use GitHub GraphQL API because was down this weekend, and was nice opportunity to experiment with graph.cool.

[ ] improve UX
[ ] test coverage
[ ] doc

Coverage decreased (-1.5%) to 97.5% when pulling 706db1ba7a020129ed771db92b816ce45a646075 on sedubois:gql into 2ba50b406cff9d70aa8de06f02d7775c635804c8 on mxstbr:dev.

Coverage decreased (-1.5%) to 97.5% when pulling f374b71a99897e0658b2461534f65f0776b1a490 on sedubois:gql into 2ba50b406cff9d70aa8de06f02d7775c635804c8 on mxstbr:dev.

Coverage decreased (-8.3%) to 90.698% when pulling 065c478ef209a5195dc9ce60cc19002da13c6cb8 on sedubois:gql into 68eef925e9c1a6593c6aead84e128f1cc8182fe6 on mxstbr:dev.

Feedback would be great, to know if this should be readied for merge 😊
My main concern left right now is whether to choose Relay or Apollo. But they have similar functionalities and they will evolve in next weeks/months anyway (especially when Relay 2 is out). So I'd say going with Relay is good enough in the scope of this PR.

Apollo seems the better choice in relation to Redux concerns and features. Why would we want to use Relay instead of Apollo?

Unfortunately Relay 1 is quite large within the final bundle:

@sedubois How does that compare to Apollo?
I'm looking at both Apollo and relay atm and must say that Apollo is light years easier to grok coming from a react, redux, router background.
I really think they struck the right abstraction level and everything feel familiar, the way you use it as a redux middleware, can observe requests in redux devtools, their HOC etc. Relay OTH feels like magic. Just my two opinionated cents :)

@GGAlanSmithee I haven't put RBP + Apollo in a working state (work so far was here), feel free to check and report 😄

@sedubois ah ok, I have apollo working client side in a small project, if I get the time I'll see if there's anything I can contribute to your example.

@sedubois I think you should pass the immutable prop to ApolloProvider
<ApolloProvider immutable store{...} client={...}>

and change reduxRootSelector to
reduxRootSelector: (state) => state.get('apollo')


[FEAT]: backup main to dev and delete those experiments part

Description
Don't include unfinished experiment parts.
Additional Information
No response

solved by #604


Child padding top

Can I set default padding top on a child element?
Thanks

Do you mean the distance between the top of the stacks and the top of the device?

Hello,
No, I mean the distance between the top of the view and it’s contents.
Seems that each stack view has an empty header
Thanks

Il giorno 20 feb 2020, alle ore 08:25, AliYigitBireroglu<EMAIL_ADDRESS><EMAIL_ADDRESS>ha scritto:
Do you mean the distance between the top of the stacks and the top of the device?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/aliyigitbireroglu/flutter-cupertino-stackview/issues/2?email_source=notifications&email_token=ACZWAQ3NY4MOVWLUC524HG3RDYV4ZA5CNFSM4KVDBN62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMLF5UA#issuecomment-588668624, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZWAQ7TPR5RZGTDAD5Z26TRDYV4ZANCNFSM4KVDBN6Q.

I'm sorry, I don't understand what you mean. Can you share an image?

I mean this

At the moment, a customisable value is not a part of this package but you can play with the values in "misc.dart" to change that.


change "pending" behavior ("this.skip()"); closes #2286 [DO NOT MERGE]

This is a proposal for how this.skip() should work.  Related: #2286 and #2148.
When this.skip() is called in a:

"before all" hook: Skip all tests and all "each" hooks.  Continue to run "after all" hooks.
"before each" hook: Skip subsequent tests.  Continue to run all remaining hooks.
test:  Skip just this test.  Continue to run all remaining hooks.
"after each" hook: Skip subsequent tests.  Continue to run all remaining hooks.
"after all" hook: Skip this just hook.  Continue to run all remaining hooks.

I feel like the intent of this.skip() is more of a "permanent" directive; if you need to temporarily disable things, you can use it.skip(), describe.skip(), etc.  This is why we should tear down if we've set up.  It becomes the user's responsibility to be explicit about which hooks should be skipped, if there are multiple of the same type.
My tests here are a bit crude (assertions in an "after all" hook?!), and possibly should live elsewhere.  Also, we need assertions against "nested" suites, but I was about at my limit for abstracting these tests.

"before each" hook: Skip subsequent tests. Continue to run all remaining hooks.

...

"after each" hook: Skip subsequent tests. Continue to run all remaining hooks.

You sure about this? While I can't think of any obvious use cases either way, my gut reaction is I'd expect this.skip() during setup/before each to skip just this test, and this.skip() during teardown/after each to skip just this hook. After all, before all doesn't skip all subsequent suites after the current suite.

It seems right to me; speaking for myself here's what I'm thinking:
beforeEach should do (must do? is there even a way for it to find out which is the current test in order to behave otherwise?) the same thing for all tests, so if it calls skip once, it should always call it, right?
afterEach that reasoning doesn't apply -- the tests would have already run before it -- but on the other hand, if all it skipped was that hook, it'd basically be equivalent to return. On the other hand, under the proposed logic, if you call it unconditionally in afterEach you basically skip all but the first test... But then again, it might be appropriate to call conditionally: if the cleanup process discovers something that renders the remaining tests moot.

@ScottFreeCode: Well, consider the following hypothetical use case that is based on my real use case.
My real use case is a test for whether my code interacts correctly with browser behavior relating to focusing a textarea. Unfortunately, the browser behavior I'm testing against only happens when the browser window is focused, so if I start my tests and Cmd-Tab away, I'd like for that test to be marked neither passed nor failed, just skipped.
Now, I currently only have the one test with this very corner case requirement, but it's easy to imagine having this requirement for multiple such tests, like to test my code against different aspects of this browser behavior. It'd sure be nice, if I Cmd-Tabbed away and then back, to only skip just the tests run while the browser window was unfocused. If this.skip() skips all subsequent tests, then to do so I would need to put if (!document.hasFocus()) this.skip(); in every single test; with my proposed behavior, that could be put in just one setup/beforeEach clause.
I'd also like to propose forbidding this.skip() in "after" hooks, I really don't see any possible use case and we shouldn't add features we see no use case for.
I think this results in something easier to document and understand:

"before all" hook: Skip "each" hooks and tests in this suite. Run remaining non-"each" hooks.
"before each" hook or test: Skip this test. Run remaining hooks.
"after each" or "after all" hook: Forbidden. ☠

There is a use case for afterEach that's distinct from return, though, if you don't start from the limitation that it can only affect the current test: skipping the rest of the tests if some problem is discovered after any of them complete. I'm not sure I have a real example of this, but I expect it would mostly come up with sanity checks... or something like that?
after, on the other hand, I can't think of any reason to use this.skip instead of return, except maybe as a cheat to break out of some kind of non-trivial, non-easily-short-circuitable callback structure that should probably be considered a serious code smell anyway (using exceptions specifically to evade actual structuring is rather goto-like), or except perhaps blind/meaningless consistency with tests and other hooks. I suppose the latter might be a reason to be graceful or at least helpful when people use it anyway... On the other hand, if there's no "real" reason to use it, then it's possible the only reason most people would use it is a false expectation that it will do something it in fact does not, in which case we'd actually want to warn them about that. Recommendation: this.skip() in an after hook is an error, but with a clear message that it doesn't do anything return wouldn't rather than simply leaving the skip property on the context object undefined and getting a cryptic JS "not a function" error.
I'm of mixed minds on beforeEach.
On the one hand, I can't help but suspect there's a cleaner way to handle the browser focus example. What good is it to run the tests if they may or may not be ignored based on unrelated actions being taken at the same time? If it's running on CI that will either never be an issue or always be an issue; but running locally and opening the console... shouldn't the tests be paused for debugging somehow, or something like that, instead of changing the scope of the current test run? And if I had a more general case of needing to run something locally but needing to be able to Cmd+Tab away from the test window, personally I'd rather find a way to do it that would make those tests run anyway (open a dedicated browser in the background and ensure that tests work if the browser never has focus to begin with, or perhaps trigger the tests locally but run them on a dedicated server or service such as Sauce) instead of having those local runs not actually test stuff because to do so I have to actually sit and watch the browser while it happens. (Maybe I'm just not fully understanding the broader user behavior context and how the test run fits into it? I also could be biased because in my day job we're currently having problems due to other developers choosing to build websites in a way that they stop functioning correctly if they don't retain the browser focus, and I worry that it's some kind of trend since we recently went from only running into it once in a blue moon to running into three instances in the same week.)
On the other hand, the only disadvantages to running subsequent tests (along with their Each hooks) after a test is skipped in beforeEach are the inefficiency of running the hooks and throwing the Pending over and over even if beforeEach is going to this.skip() every time and the inconsistency with afterEach if afterEach is allowed for the one use case of stopping subsequent tests.
I guess I'll let you both know if I have any new thoughts or change my mind; I do think this isn't necessarily as trivial as it seems one way or the other.

Thanks for the input @laughinghan and @ScottFreeCode.

I guess I'll let you both know if I have any new thoughts or change my mind; I do think this isn't necessarily as trivial as it seems one way or the other.

"Skip" and "only" are basically the bane of my existence rn.  Happily, I haven't heard too much grousing about "grep" for awhile since v3.0.
I'll agree it needs more thought and input from others.

@laughinghan I have a few comments.

I'd also like to propose forbidding this.skip() in "after" hooks, I really don't see any possible use case and we shouldn't add features we see no use case for.

The cat is already out of the bag.

I'd expect this.skip() during setup/before each to skip just this test

Given what you're using this.skip() for, this makes total sense.  Yet, I can't help but think it's not the right tool for your job.  Why not do something like:
describe('my finicky browser', function () {
  beforeEach(function (done) {
    if (!document.hasFocus()) {
      this.timeout(0);
      const t = setInterval(() => {
        if (document.hasFocus()) {
          clearInterval(t);
          done();
        }
      }, 500);
    }
    done();
  });

  it('should do something when the browser is ready', function () {
    // ...
  });
});

@boneskull: Thanks for commenting.

The cat is already out of the bag.

Fair enough

Why not do something like: ...

If I'm understanding correctly you're suggesting pausing my tests while my tab is backgrounded? That's not what I want, my whole test suite takes like 30s to run, which is fast enough to run locally in full (rather than only in pieces locally, and only in full on a server), but slow enough I don't want to twiddle my thumbs staring at it while it runs, I want to background it and check my email or Slack or something, and have it still run to completion, skipping the focus tests that it can't run.

@ScottFreeCode: Thanks for replying, sorry it took so long for me to get back to this.
TL;DR: just merge it, I want it fixed more than I care about these details

Recommendation: this.skip() in an after hook is an error, but with a clear message that it doesn't do anything return wouldn't rather than simply leaving the skip property on the context object undefined and getting a cryptic JS "not a function" error.

:+1:

What good is it to run the tests if they may or may not be ignored based on unrelated actions being taken at the same time? If it's running on CI that will either never be an issue or always be an issue
...
Maybe I'm just not fully understanding the broader user behavior context and how the test run fits into it?

No you sound like you understand it

personally I'd rather find a way to do it that would make those tests run anyway (open a dedicated browser in the background and ensure that tests work if the browser never has focus to begin with, ...)

Sure that would be The Right Thing but that's just so much more work than if (!document.hasFocus()) this.skip(); and leaving my browser window focused every once in a while, right? (Or, until this bug fixed, if (!document.hasFocus()) { doTeardownStuff(); this.skip(); }.) I think I mentioned that this has only come up with this one test, not with multiple tests yet, and this only showed up years into a project and it's been months since then and I haven't had to add any more. So I think even if it were you in my shoes, you wouldn't think this is worth investing in either.

or perhaps trigger the tests locally but run them on a dedicated server or service such as Sauce

I actually do run them on Sauce, and I could probably find some way to always disable this test locally, but, isn't it better to do "feature testing" and run this test whenever possible, rather than having to manually enable it locally?

developers choosing to build websites in a way that they stop functioning correctly if they don't retain the browser focus, and I worry that it's some kind of trend since we recently went from only running into it once in a blue moon to running into three instances in the same week

Nah, this is my blue moon, I just want to make sure my blue moon has a test case.
Also to be clear, it's not weird for rare random things to cluster, 'cos like, what are the chances they're evenly spaced, right? Did y'all track down the culprit in each of those cases and find out what technique they're using is brittle in this way, or? I'm pretty curious now, it def didn't happen like that for me, I'm still pretty weirded out that it's a problem for me at all.

I do think this isn't necessarily as trivial as it seems one way or the other.

Really? I think this is pretty trivial. I would way rather this be merged as-is, or with your modifications and zero of my suggestions accepted, than spend weeks and weeks more debating this. As-is (or with your modifications), I wouldn't be able to do beforeEach(() => { if (!document.hasFocus()) this.skip(); }); like I suggested would be nice, but I could easily do:
function testThatRequiresFocus(name, fn) {
  test(name, function(done) {
    if (!document.hasFocus()) this.skip();
    return fn(done);
  });
}

instead, it's really not that inconvenient.

@laughinghan It's not in the project's best interests to move fast and break things.
It's trivial in the sense that the changes are few.  It's non-trivial in that these changes will impact many users.  While it's a "bug fix", it's also going to break some tests that rely on the buggy behavior.  And we're going to get more issues complaining about the broken tests, challenging why it was changed, how it was changed, and when it was changed.
Unfortunately there's not much participation in PRs like this before they are published, for many reasons; the most obvious being that it'd require users to watch this entire repo.
For all the people that this change will impact, how many have I heard from?  I can answer that, but I can't answer how many users will be affected by this change.  Even if it's a "bug fix"--and some are sure to argue it isn't--we need to be crystal-clear with the answers to why, how, and when.
@ScottFreeCode seems to think the change is a good idea, and you are saying it's better than nothing. I'm inclined to think your proposal (or a portion there of) is a better one--yet your use case is unusual at best, and a misuse of the feature at worst.  Where's the story that we can all nod our heads in agreement on?
It's better to break tests once and be confident with the direction--instead of breaking them, realizing we made a mistake, then potentially breaking them again.
But we can pull the "beta" card.  😈   I'm going to modify this PR to exhibit the behavior you suggest--and re-target to a different branch we'll prerelease, tagged mocha@beta (this means closing the PR)--unless @mochajs/core is opposed.

It's not in the project's best interests to move fast and break things. ... It's non-trivial in that these changes will impact many users.

I don't think we should move fast and break things either! But it's been almost 9 months since I first reported this issue. Since then there's been one related ticket and no comments by additional users on either ticket. I don't think this is particularly fast, nor likely to break things for many users at all :)

a "bug fix"--and some are sure to argue it isn't

Are they sure to? I think my bug report, at least, that this.skip() inside a test skips teardown() even though setup() was run, isn't likely to be controversial.

your use case is unusual at best, and a misuse of the feature at worst

:+1:

But we can pull the "beta" card.

🎉 🎊 🎈 💥 😈

I'm quite time-constrained this weekend, sorry. I'm created a task to check the discussion on Monday. 🙇🏻

It'd sure be nice, if I Cmd-Tabbed away and then back, to only skip just the tests run while the browser window was unfocused. If this.skip() skips all subsequent tests, then to do so I would need to put if (!document.hasFocus()) this.skip(); in every single test; with my proposed behavior, that could be put in just one setup/beforeEach clause

That's interesting. I didn't imagine your case since the condition is not constant. (If you assume the condition is constant, skipping all remaining tests makes sense)

after, on the other hand, I can't think of any reason to use this.skip instead of return

If skip was able to log a "reason", that could be a use for it. At the same time, you could raise an Error I guess? But I don't recall what would be the effect on test/hook execution.

What good is it to run the tests if they may or may not be ignored based on unrelated actions being taken at the same time?

Good point.

On the other hand, the only disadvantages to running subsequent tests (along with their Each hooks) after a test is skipped in beforeEach are the inefficiency of running the hooks and throwing the Pending over and over even if beforeEach is going to this.skip() every time and the inconsistency with afterEach if afterEach is allowed for the one use case of stopping subsequent tests.

Maybe you can stop subsequent tests by throwing an Error in the beforeEach? I can't recall the behavior.

Okay, and now my opinion on the behavior. Trying to think what would be the most simple and logic behavior.

before. Runs once before all the tests and their each hooks. I imagine the context (this) to be the suite. Therefore, all the changes made through the context (this.timeout(), this.tag()) affect the suite. this.skip() skips the suite. (after hooks are ran)

beforeEach. Runs once before each test. I imagine the context (this) to be the test. Therefore, all the changes made through the context (this.timeout(), this.tag()) affect the test. this.skip() skips the test. (afterEach/after hooks are ran)

afterEach. Runs once after each test. I imagine the context (this) to be the test. Therefore, all the changes made through the context (this.timeout(), this.tag()) affect the test. this.skip() skips the test. (remaining afterEach/after hooks are ran, since their before/beforeEach counterpart has ran)

after. Runs once after all the tests and their each hooks. I imagine the context (this) to be the suite. Therefore, all the changes made through the context (this.timeout(), this.tag()) affect the suite. this.skip() skips the suite.

You skip a test/suite to avoid running it AND to mark it as pending.
Skipping a test/suite after running it is a bit counter logical. The only purpose would be marking it as pending. Which is not very clean because you've already emitted pass/fail events for such test. Not sure there's value in changing the test state/outcome post-mortem. (<- feedback pls)
Therefore I think afterEach/after hooks shouldn't be able to skip their corresponding test/suite, and somehow mocha should let the user know that skip is not meant to be used in afterEach/after hooks, that whatever is being done should be done in before/beforeEach instead.

Any opinions on deprecating skip in after/afterEach?

Soft deprecation (docs only) until 4.x
Hard deprecation (print msg to STDERR) until 5.x
Removed in 6.x

I agree that there's probably no real use case for a skip in these hooks.

I wonder how could we get more opinions on the point. It's a shame we don't have a twitter account with followers. Do we even have a twitter account? Feels like a good platforms to get opinions, make announcements, polls ...
Deprecating looks good to me, just hoping we don't miss any valid use cases.
@boneskull what do you think about my proposed skip behavior in beforeEach? (skipping only the current test)

yes, that's what this doing now re beforeEach, right? the new PR anyway

@boneskull Sorry, I didn't notice a new PR was created. 👌

late to the party but, imho, skip in beforeEach should only skip current test and not all subsequent tests. And a skip in beforeEach should either skip only the currentTest, or skip all subsequent [before|after]Each and currentTest. I do agree skip in after/afterEach doesn't make much sense.
I have a use case where we need an automated way of maintaining a list of failing tests and skipping them during test execution. Instead of updating the specs with it.skip() or xit(), it's easier to maintain this information in a single separate file. We can then implement a single beforeEach() in root suite which checks list of failing tests and if this.currentTest.title is on that list, the test is skipped.

I don't think this is actually fixed for the it() case:

test: Skip just this test. Continue to run all remaining hooks.

When I skip tests in an it() block, I do not see the afterEach() hooks run. Here's a screenshot of my output:

As you can see, afterEach is not being logged after the skipped test, but it being logged after the run test.

To clarify, I'm skipping a test by calling this.skip() within the it() block. I've tracked down where it happens in the code and if I comment out these lines everything works as expected:
https://github.com/mochajs/mocha/blob/ade8b90a46c641345b67c7979f855bcfbda900d5/lib/runner.js#L671-L673


Update issue

Tried to update the open-nti using "make update" hitting following error
docker pull juniper/open-nti-input-snmp:latest Error response from daemon: repository juniper/open-nti-input-snmp not found: does not exist or no pull access Makefile:91: recipe for target 'update' failed make: *** [update] Error 1

Hi @panks21
Did you clone the project recently? or do you have an old copy of it?
Regards

It is more than a month old copy. Tried to update today. Fails at SNMP

Have the same problem.  Cloned last week.
OpenNTI - Update the files from Github
git pull
Already up-to-date.
OpenNTI - Update the containers from Docker Hub
docker pull juniper/open-nti:latest
latest: Pulling from juniper/open-nti
Digest: sha256:1859f2205d491797f4f8a81acbf86aecea1cfc3405972cfd3696c34b292af18b
Status: Image is up to date for juniper/open-nti:latest
docker pull juniper/open-nti-input-jti:latest
latest: Pulling from juniper/open-nti-input-jti
Digest: sha256:56116f66565e085b0af6727d0272f4ca2ed2dd1ccab5d3587f982c92ac5d728d
Status: Image is up to date for juniper/open-nti-input-jti:latest
docker pull juniper/open-nti-input-syslog:latest
latest: Pulling from juniper/open-nti-input-syslog
Digest: sha256:95c7bec19a9bb1140d9ea4c23c4539a02ed2c5f88561b9adb27fc6bbb3b9141c
Status: Image is up to date for juniper/open-nti-input-syslog:latest
docker pull juniper/open-nti-input-snmp:latest
Error response from daemon: repository juniper/open-nti-input-snmp not found: does not exist or no pull access
Makefile:91: recipe for target 'update' failed
make: *** [update] Error 1

I understand I am already up to date but the pull should not fail on verification, no?

Agreed. Also it only fails at snmp and not in others!!!

Hello,
I'm seeing the same problem. I am updating for the first time. I deleted the open-nti directory and ran the command to clone it again.
After running make update, many files are downloaded and installed. However, I also get stuck at SNMP.
make update OpenNTI - Update the files from Github git pull Already up-to-date. OpenNTI - Update the containers from Docker Hub docker pull juniper/open-nti:latest latest: Pulling from juniper/open-nti Digest: sha256:cd64f4569adedf0cf87f6d75e98942777f7fa8876072788eacb4569332a2b1a5 Status: Image is up to date for juniper/open-nti:latest docker pull juniper/open-nti-input-jti:latest latest: Pulling from juniper/open-nti-input-jti Digest: sha256:e7eaebed54d8f428e05a1b14428bc41d200ae36abe3633fe34b6358243412f25 Status: Image is up to date for juniper/open-nti-input-jti:latest docker pull juniper/open-nti-input-syslog:latest latest: Pulling from juniper/open-nti-input-syslog Digest: sha256:568c8416b5a95eaa53b5b059c1b3537bd5cd86ad1ce7719ae4cd46df55d8a45d Status: Image is up to date for juniper/open-nti-input-syslog:latest docker pull juniper/open-nti-input-snmp:latest Pulling repository docker.io/juniper/open-nti-input-snmp Error: image juniper/open-nti-input-snmp:latest not found make: *** [update] Error 1

Has this been resolved?  I still see this issue after performing a make update.
latest: Pulling from juniper/open-nti-input-syslog
e110a4a17941: Already exists
2b3730f08b98: Already exists
01bbd7d4a2f9: Already exists
a1949c91ad97: Already exists
d1dc0c8bb461: Already exists
c4d6853a23f8: Already exists
2222a88b9be3: Already exists
2cfd79515b53: Already exists
ca758ca02dcb: Pull complete
a9a1d63bab69: Pull complete
2325adcdf4df: Pull complete
cdb611fa8c8d: Pull complete
c00c7bfe4f2f: Pull complete
20ee0625456b: Pull complete
00c2bc529dc7: Pull complete
7f356ad6cdf1: Pull complete
b3460dbd0e5c: Pull complete
Digest: sha256:66fbcd56619de0cfdec80476b1ecffe55678dd736f48b874a40363c6a1d7b64b
Status: Downloaded newer image for juniper/open-nti-input-syslog:latest
docker pull juniper/open-nti-input-snmp:latest
Error response from daemon: pull access denied for juniper/open-nti-input-snmp, repository does not exist or may require 'docker login'
Makefile:95: recipe for target 'update' failed
make: *** [update] Error 1

Hi @klrobertson
I think the best way to fix this 'make update' issue is to remove those container images that does not exist on docker hub (in fact they never existed).
IMHO, the best way to update the project is to clone/pull the latest source and build it.
Regards

I just update the Makefile
https://github.com/Juniper/open-nti/commit/ad6faec21ea87f5cce7d2cfd687e1e42ab8953a4