All containers stopping first during stack upgrade, regardless of batch-size

Rancher Version:
1.0.1
Docker Version:
1.10.2
OS and where are the hosts located? (cloud, bare metal, etc):
Cloud, BM
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
HA (external DB)
Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:
Deploy any stack and run an upgrade with rancher-compose up --upgrade without start_first: true in the rancher-compose.yml.
Results:
All existing containers will stop first, then the new ones will deploy, regardless of batch-size.
Expected:
With using data containers, creating a duplicate stack doesn't solve the issue. I am also unable to use the start_first flag due to a limited number of hosts being able to run these containers and explicit port mapping (so multiple of the same service cannot run on the same host). Ideally, there would be an option (if not the default) that when upgrading a service, containers (as per the batch-size) would stop, then the same number of new containers would start, wait for the interval period, then continue on with the next batch instead of stopping all the old, starting all the new (causing downtime).

I am unable to reproduce. Using v1.1.0-dev3 and rancher-compose v0.8.2. I started a service with 6 containers and did an upgrade. It upgraded using the default batch-size of 2.
Can you share your docker-compose.yml and rancher-compose.yml?
What was the scale of your service?

Working in a local environment trying to replicate. We are definitely seeing it consistently in our live stack, but am having an issue replicating locally.

Can't seem to replicate, hoping we'll see it resolved in 1.1.0. Will open a new ticket if I end up being able to replicate it.


Alt+"Right Arrow" doesn't switch me on right hand side tab

It switches me between window tabs on opposite direction.

I think the standard keybindings are these:

So if you have 3 tabs open, then click on the rightmost, then the centre, then the leftmost, then type Alt+Left the focus will move rightwards by one tab. This is as designed; you are moving back and forward through your navigation history.


AT-X, BS11,and the channels in Tokyo are down

Not only AT-X and BS11, but also the channels in the following picture are down. They are perhaps HD channels. Please fix them.
Thank you for maintaining the channel.

 Not only AT-X and BS11, but also the channels in the following picture are down. They are perhaps HD channels. Please fix them. Thank you for maintaining the channel.

I think those channels are geo-ip blocked. Still working in Vietnam.

Okay, that makes sense.
I'll try using ip in Vietnam.

fixed


bower ECMDERR Failed to execute

bower install http://someserver.com/git/kendo.git
bower not-cached    http://someserver.com/git/kendo/.git#*
bower resolve       http://someserver.com/git/kendo/.git#*
bower not-cached    http://someserver.com/git/kendo.git#*
bower resolve       http://someserver.com/git/kendo.git#*
Username for 'http://someserver.com': admin
Password for<EMAIL_ADDRESS>bower checkout      kendo#master
Username for 'http://someserver.com': admin
Password for<EMAIL_ADDRESS>bower progress      kendo#* remote: Compressing objects:   1% (1/83)
...
bower progress      kendo#* remote: Compressing objects: 100% (83/83)
bower progress      kendo#* remote: Compressing objects: 100% (83/83), done.
bower resolved      http://someserver.com/git/kendo.git#ceea7c2f98
bower ECMDERR       Failed to execute "git ls-remote --tags --heads http://someserver.com/git/kendo/.git", exit code of # 128

FYI, bower install http://someserver.com/git/kendo/.git, http://someserver.com/git/kendo/
httpS://someserver.com/git/kendo/
git://someserver.com/git/kendo/
do not work.
ssh is not an option.
Why does the last line try to access kendo/.git if the git files are in kendo.git/ ??
git server: bonobo

Please post steps / publicly available repository to reproduce, guys :(


make bigInt more applicability in adaptive-expressions

Fixes #2211
Currently, JS sdk uses Number to handler all number input, and with BigInt to handler ticks().
Here SDK provides such a solution to reduce the gap between these two languages:

Keep the range gap of the normal number.-2^63 ~ 2^63 -1 for C# and -2^53+1 ~ 2^53 for JS. (JS would provide inaccurate calculation of numbers beyond this range)
Provide a solution to convert bigint object into a normal number in js. int() function can accept a bigint input, and output a fuzzy number output.
For example: int(ticks(utcNow())) - 1 would be valid and the output would be a number. But the accuracy of the output is not guaranteed

Pull Request Test Coverage Report for Build 492679107

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 84.502%

Files with Coverage Reduction
New Missed Lines
%

libraries/botbuilder-lg/src/templateExtensions.ts
1
84.48%

Totals

Change from base Build 486704763:
0.2%

Covered Lines:
17818

Relevant Lines:
20146

💛  - Coveralls

Pull Request Test Coverage Report for Build 492679107

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 84.502%

Files with Coverage Reduction
New Missed Lines
%

libraries/botbuilder-lg/src/templateExtensions.ts
1
84.48%

Totals

Change from base Build 486704763:
0.2%

Covered Lines:
17818

Relevant Lines:
20146

💛  - Coveralls


🎄🎄🎄 Add integration with AoC leaderboard API

So, AoC leaderboard has an API:

You can also get the [JSON] for this private leaderboard. Please don't make frequent automated requests to this service - avoid sending requests more often than once every 15 minutes (900 seconds). If you do this from a script, you'll have to provide your session cookie in the request; a fresh session cookie lasts for about a month. Timestamps use Unix time.

Endpoint Link
That means, we can add a simple but cool integration. We can ask the Nayn to put some notifications about successful selves of daily problem. It helps to increase the stress level of other participants and will add a reason for pride for those who have already solved the problem.

[x] find the way to communicate with AoC API (@C-Pro private leaderboard), looks like we need some credentials

[x] add credentials into dev \ prod environment

[x] add periodic task to sync leaderboard board with Nayn persistent store (according to the API doc, we shouldn't hit the API often that once per 15 minutes)
[x] (optional | discussable) check the diff between local store and board state and send a notification in the Chat, if someone bit the next task

[x] remove notification after a while

[ ] @C-Pro: Environment variable AOC_SESSION should be set to valid value of cookie session from AOC 🎄
Notification would look like:

Nyan: Wow! @getjump just solves Day 4 problem in 0:34:12! Gut Gemacht! 🔥🔥🔥

To turn this mode ON we need to put some cred's into the Production ENVs.

Environment variable AOC_SESSION should be set to valid value of cookie session from AOC 🎄

@C-Pro could you check this out please?

Done


Support deploying driver binary via daemonset (#38)

This deploys the oci flexvolume driver binary onto every node (including master) through a deploy script and the binary being bundled into a docker image ran as a daemonset.
Probably can't merge this until the wcr.io/oracle/oci-flexvolume-driver repository has been created.
The build-release->release workflow will also need 'push' adding in the middle (and maybe tweaking copying through the rest of dist so it's still available for the 'release' stage.

We can use a Kubernetes Secret to provide the credentials file by mounting it in the appropriate place.

@owainlewis depending on how you deploy the controller managers you don't need two daemonset's. Since our KCM is deployed as a deployment, we use a hostvolume for hte plugin itself, then point the config directory to the secret mounted in. You could do the same with manifest based control plane as well.

Tests now use kubernetes daemonset too, and all ansible has been removed.


British voice

Hi Noel -
This PR shouldn't be merged as I'm waiting on my PR against the google-tts-api module, but here's a preview if you're interested that adds an option to use the british voice from google translate instead of the default american one.
I'm actually not british btw I've just been playing with your module to do spoken medicine reminders for my mom, and I was getting tired of hearing the american voice all the time!
Originally I'd hoped maybe there was an option for a male voice or something but googling it seemed this still-female-but-british-accent-voice is the only other option possible. But it was easy to do and I thought - what the heck better than no other option at all...
Hope you're interested.
I've done it very simply imitating your "device" function but adding another function called "accent" that takes "us" or "uk" that they can call to switch the voice.  It defaults to the american voice if they don't call it (i.e. so it stays the same as now).
For my medicine reminder I'm using a variation of your example.js with IFTTT.com - I had a thought of maybe letting them stick like an optional prefix i.e. "uk:Hello Google Home" on the actual spoken text so that it would be super easy to switch right inside the IFTTT applet configuration.
But I wasn't 100% sure others would care about that (i.e. I thought they might just be happy to set it once using this "accent" call so they're choosing their voice for everything not each time they speak something).  So I didn't do that after all I kept it simple.
If you want to try it this should work for you btw because I've pointed package.json at my branch of google-tts-api.  If they merge my PR there I will update package.json here and update my comments here in case you wish to merge this for real.

fyi I found out from the google-tts-api guys that simply sending e.g. "en_GB" for the "lang" will let you pick the british voice (or 'en_AU' for the australian voice, etc).
So I will send an  update changing what I did to simply provide a way to override the "lang" since you'd always assumed 'en' right (so maybe instead of the "accent" function it could have simply been a "lang" function allowing them to set e.g. 'en_GB' instead of the default 'en').
If you see my PR though please let me know if you're open to the change if you think it's something you're ok with adding...
Thanks,
Darren

Hi, my PR exactly for this purpose. You can init your device with a language like this:
var googlehome = require('./google-home-notifier');
googlehome.device('XYZ', 'fr-CA'); // Change to your Google Home name


improve speed, thanks to the following improvements:

loop unrolling and eleminating extra calculations
eleminating instance level variable access
quicker equals check, checking long by long chunks instead of bytes
update GraalVM version to the latest

Check List:

[x] Tests pass (./test.sh <username> shows no differences between expected and actual outputs)
[x] All formatting changes by the build are committed
[x] Your launch script is named calculate_average_<username>.sh (make sure to match casing of your GH user name) and is executable
[x] Output matches that of calculate_average_baseline.sh
[x] For new entries, or after substantial changes: When implementing custom hash structures, please point to where you deal with hash collisions (lines 175 and 198)

Execution time: ~2.2s
Execution time of reference implementation: ~3m.22s

This one produces incorrect results for the 10K keyset (see create_measurements_3.sh). Could you take a look? Thx!

Hi @gunnarmorling
There was a bug in the equals method but now it's fixed. Could you please have another look? Many thanks.

It's actually a bit slower than before:
Benchmark 1: timeout -v 300 ./calculate_average_yavuztas.sh 2>&1
  Time (mean ± σ):      3.811 s ±  0.089 s    [User: 27.167 s, System: 0.707 s]
  Range (min … max):    3.761 s …  3.970 s    5 runs

Summary
  yavuztas: trimmed mean 3.7753897134400005, raw times 3.78230971344,3.76071144844,3.76312621244,3.97048912444,3.78073321444

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:03.775 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_yavuztas.java)| 21.0.2-graal | [Yavuz Tas](https://github.com/yavuztas) | GraalVM native binary, uses Unsafe |

Hmm, that's strange. Normally, it should be faster. The results I get in my machine compared with top 3:
Summary
  royvanrijn: trimmed mean 2.13166147388, raw times 2.1277600158800003,2.14154143188,2.0986444738800003,2.15448226588,2.12568297388
  thomaswue: trimmed mean 2.111515374326667, raw times 2.09593734666,2.1239312216600004,2.1093937626600003,2.11407201366,2.11108034666
  artsiomkorzun: trimmed mean 2.0592471423866665, raw times 2.0852324207199997,2.05694900372,2.05830246172,2.06248996172,2.05160812772
  yavuztas: trimmed mean 2.12233382756, raw times 2.11259103556,2.13310224456,2.14045282756,2.11848041056,2.11541882756

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
| 1 | 00:02.059 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)| 21.0.2-graal | [Artsiom Korzun](https://github.com/artsiomkorzun) | GraalVM native binary, uses Unsafe |
| 2 | 00:02.111 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_thomaswue.java)| 21.0.2-graal | [Thomas Wuerthinger](https://github.com/thomaswue) | GraalVM native binary, uses Unsafe |
| 3 | 00:02.122 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_yavuztas.java)| 21.0.2-graal | [Yavuz Tas](https://github.com/yavuztas) | GraalVM native binary, uses Unsafe |
|   | 00:02.131 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_royvanrijn.java)| 21.0.2-graal | [Roy van Rijn](https://github.com/royvanrijn) | GraalVM native binary, uses Unsafe |

Is that really the difference for different hardware? I haven't tested on Linux/AMD only on my Mac with m1. @royvanrijn , @thomaswue did you have any similar experience like your solution is much slower on Linux/AMD while it's actually faster on Mac? If so how did you overcome this?

Yes, folks definitely observe different performance characteristics between M1 and the eval machine, due to the different ISAs, e.g. in terms of SIMD register widths. Testing and tuning on a similar hardware is the most promising approach beyond a certain threshold.

Hey @yavuztas, how would you like to proceed here?

Hi @gunnarmorling, I'm working on a complete rewrite, optimized for the target machine's hardware -- hopefully :)
I will push it soon.

Hi again @gunnarmorling, I think I'm done with it. I'm out of ideas yet how to improve it more. Hopefully, a better solution. Can you please have a look?

It's faster for the standard keyset indeed, but it doesn't produce any output at all for the 10K key set :(

Really? It works for 10K on me and the output also perfectly matches. Any errors or it just hangs?

No errors, just no output whatsoever. When I run it in JVM mode, the output indeed is correct. But I have no idea how to approach that native issue, unfortunately.

Sorry, accidentally pushed it to the main branch :(
I will revert the last two commits

Yepp, looking good now, also passing with 10K keys.
Benchmark 1: timeout -v 300 ./calculate_average_yavuztas.sh 2>&1
  Time (mean ± σ):      2.328 s ±  0.050 s    [User: 0.003 s, System: 0.003 s]
  Range (min … max):    2.277 s …  2.407 s    5 runs

Summary
  yavuztas: trimmed mean 2.31906419842, raw times 2.3034457774200003,2.27678964042,2.30782189442,2.40697815042,2.34592492342

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:02.319 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_yavuztas.java)| 21.0.2-graal | [Yavuz Tas](https://github.com/yavuztas) | GraalVM native binary, uses Unsafe |

Hey @yavuztas!
Congrats again on being in the Top 20 of the One Billion Row Challenge!
To celebrate this amazing achievement, I would like to send you a 1BRC t-shirt and coffee mug. To claim your prize, fill out this form by Feb 18. After submitting the form, please provide a comment with the random value you've specified in the form, so that I know it is you who submitted it.
All data entered will solely be used in relation to processing this shipment. Shipments can be sent to any country listed here or here (I'll use whichever one is cheaper for me to ship to your location). A big thank you to Decodable for sponsoring these prizes!
Thanks a lot for participating in 1BRC,
--Gunnar

Hi @gunnarmorling
Thanks for the prizes, and also to Decodable for sponsoring.
Here is the secret: 23049 31739


Update netlify.toml - add GO_VERSION environment

What type of PR is this?
/kind documentation
What this PR does / why we need it:

Update netlify.toml

set GO_VERSION environment

Which issue(s) this PR fixes:

Fixes #2650

Does this PR introduce a user-facing change?:
NONE

✅requesting go 1.21
https://app.netlify.com/sites/kubernetes-sigs-gateway-api/deploys/65708bf890b49b0008b35d24

passed. GO_VERSION must exact match.
https://app.netlify.com/sites/kubernetes-sigs-gateway-api/deploys/6570fc1c7468750008f7861f
7:56:59 AM: Started restoring cached go cache
7:57:01 AM: Finished restoring cached go cache
7:57:01 AM: Installing Go version 1.21.4 (requested 1.21.4)
7:57:04 AM: go version go1.21.4 linux/amd64


Added emptyStringsReplace option

Added empty Strings Replace option.
This is useful in some scenarios where empty strings can't just be deleted, but replaced with some other value.

@nunofgs can you merge it?

Thanks @belavenuto, but unfortunately this is one that I'd prefer not to include. Personally I view clean-deep as a module that omits certain properties. Replacing values feels like it's out of scope for this module and could be taken in a myriad of different ways.
I hope that's understandable but I do appreciate the effort and the contribution!


REQUEST: listchannels / listpeers to include peer "Alias" Info

Background
When issuing a listchannels command, it is not immediately apparent which remote_pubkey name belongs to which alias.
This information would be useful to display.
Your environment
   "version": "0.4.2-beta commit=bca926d6af69b117eb6afa0e50ec2dc32d2899d4"
Steps to reproduce

lncli listchannels
Notice node aliases are not displayed

Expected behaviour
Would be useful to display node alias in listchannels / listpeers

You can easily loo up the alias of the node using getnodeinfo. As a result, there's no need for us to add this information in listpeers or listchannels. Aliases are also not to be relied upon. They're completely unreliable an non-unique.

Aliases are also not to be relied upon. They're completely unreliable an non-unique.

I know they are unreliable and non-unique, but you're effectively forcing two calls to get some, otherwise, interesting/human-readable information.
Instead of seeing the alias info in one call, we are forced to read in the entire list of channels, and then do a getnodeinfo for each pub_key.
I mean, it's something simple enough, that I'm thinking of just learning enough GoLang to patch things myself.
Would a simple patch like that be rejected ?

Won't be out right rejected, but it won't have as high priority for review/testing compared to the other PR's.


Add duplication logic in Nessus parser to handle Nessus Professional compliance imports

Issue Template
Issue Type

[ ] Bug
[x] Enhancement

Description
In nesssus/parser.py the duplication logic assumes unique plugin IDs for each finding.  This seems to work for Security Center imports of compliance findings where each finding has a unique plugin ID.  However, Nessus Professional exports compliance results with the same plugin ID for all findings  for that plugin.  For example, Unix Compliance is plugin ID 21157 so all its findings are dupes - you get one finding in dojo for potentially hundreds of audit findings - regardless of status.
In the case of the NessusXMLParser class, the enhancement would use the additional "cm" related tags to create titles and therefore dupe keys based on the severity, plugin ID, and something like plugin name from the compliance namespace.  The Nessus v2 file format for reference:
https://static.tenable.com/documentation/nessus_v2_file_format.pdf
Using the "plugin-ID" tag might work but its a hash - so there may never be real dupes - and this would not work for title.

I'm willing to contribute this as a pull request but I wanted to self-screen for feedback.  Also, this kinda smelled like something that may warrant a configuration property/setting.

@borbidgea Sounds like a solid approach. I've assigned this to you if you still want to tackle it. I've been thinking about a setting that enables dedupe on the parser importers. From what you explain though it seems like it's a bug and for those using it they would benefit from the patch. Also, would you happen to have a sample compliance file we can add?
Thanks!

I'll give it a shot.  And I will try to add the compliance sample as well - either need approval for my original or I'll cobble something together from the Nessus v2 spec and use that in test.


Add costs and supplier to known resources

Adds costs and supplier to available resources.

 Frontend Costs Forms within Services

After merging in, do an npm publish. After that, https://www.npmjs.com/package/gapi-js should show you the correct URL


Ensure best practices are followed and all models are updated in the repo with latest changes

Proposed change
Ensure best practices are being followed in the repo
Why this is important
To improve maintainability and extensibility of the CMF models repo
Additional context
Starting with a basic run through of all the models and checking if they are following best practices like: DTO declaration, code formatting

@abhishektvz , in the curve_finance.py model where slug = "curve-fi.gauge-yield" , there is error in code.

and the error for this is:


Build Pipeline

[x] Code coverage
[x] Code quality checks
[x] Build status
[ ] go-releaser

Changes to fix this issue #10

And GoReleaser in #11


Change fixpoint

[x] Remove magic fixpoint code
[x] Switch fixpoint representations to something with more fractional bits (S8.23 or S6.25 for now S12.19)

with nengo.Network() as network:
    step = nengo.Node([1.0, -0.4])
    probe = nengo.Probe(step, synapse=nengo.LinearFilter([1.0], [0.001, 0.11, 1.0]))

Before:

After:

@tcstewar - does this change to the representational range seem reasonable to you? It's not as savage as I'd wanted but small ensembles have LARGE decoder components. I think a future plan is to assess where some scaling could be applied to improve the numerics, but that might be for a later/post-thesis-written day.

Hmm, I'm getting a lot of saturated values in a cconv network I'm testing, so this may still be too big a change to the fixed point format.

does this change to the representational range seem reasonable to you? It's not as savage as I'd wanted but small ensembles have LARGE decoder components. I think a future plan is to assess where some scaling could be applied to improve the numerics, but that might be for a later/post-thesis-written day.

Definitely seems like a step in the right direction, but I'm wondering whether different parts of the system might need different ranges.  Or even maybe a small_ensemble that uses 15,16 and a large_ensemble that uses 5.26.  Or maybe having an ensemble shift everything down by round(log2(n_neurons)).  Not sure what the easy things are to try here....

I was hoping to avoid different ranges, but yeah, might be necessary :/


Unknown column in BREAD read

Laravel Version: 5.6
Voyager Version: 1.0
PHP Version: PHP 7.1.8-1ubuntu1
Database Driver & Version: MySQL 5.7.21-0ubuntu<IP_ADDRESS>

Description:
Added widgets table, no camel case, and error after adding BREAD. "SQLSTATE[42S22]: Column not found: 1054 Unknown column 'created_at' in 'order clause' (SQL: select * from widgets order by created_at desc)"
Steps To Reproduce:
Create new table widgets.

Field
Type
Null
Key
Default
Extra

id
integer
NO
PRI

auto_increment

ModelNumber
integer
YES

ModelName
text
YES

Price
float
YES

Description
text
YES

Create BREAD.
Browse BREAD.
Get Error Message.

https://laravel.com/docs/5.6/eloquent#eloquent-model-conventions

I can see these things as issues, but if this is a general purpose admin area then creating the timestamps might be a default thing for any table being manipulated.  At the very least a "hey you need to add this" on the submit button might be nice.  Just a suggestion.


optimization with optim package

Hi ,great job.
It seems that the optimization part is done manually , i wonder whether can we combine optim package (or some other automatic way)with it or not, if yes, how to do that, i am a new comer to torch, thank you in advance.

This is going to be a little annoying to do because the optim function expects as one of its inputs, a function that calculates the loss and returns the derivative (as a single, one-dim tensor). However, we have three tensors of derivatives (encoder, decoder, and linear+softmax--what i call the generator).
Hope this helps!

@yoonkim   Thank you for your reply.
another issue: what is the meaning of function of setReuse, why set the gradInput equal to the the  output, in my opinion, they are not the same thing:
    function nn.Module:setReuse()
        if self.reuse then
         self.gradInput = self.output
      end
   end

That's just a hack to share memory between gradinput/output for certain layers.
This reduces memory footprint somewhat.


Recommendation for importing/exporting Stamps?

Open-ended question here, but would this be a reasonable pattern for Stamp-based modules?
api.js
export default const Api = stampit({
  // some API logic
});

usersApi.js
import Api from 'api'

export default const usersApi = stampit({
  // some user-specific API logic
}).compose(Api);

Also: Are Api() and Api.create() the same?
Cheers.

Yes, it is reasonable. You don't even need to assign the stamp to a variable.
export default stampit({})...

And yes, they are the same.

Thanks for the prompt reply.  Are there any recommendations for using () vs. .create()?  Personally I like the simplicity of Api() but it seems too easily misinterpreted as a class.  Using Api.create() makes it pretty clear you are dealing with a factory.
Just as an aside, after re-reading the DbConnection() example I should probably have done this instead:
import Api from 'api'
export default const usersApi = Api({ ... });

since usersApi would be the final static object, it should probably not be a Stamp.

It's personal preference. Api is a factory function, there should be no misinterpretation. Classes get instantiated with the reserved word new.

Classes get instantiated with the reserved word new

Totally fair point.  I guess I always have this in my mind as something people currently do to avoid/protect against omitting new for standard function-based classes.
function Foo() {
   if (!(this instanceof Foo)) return new Foo();
   // ...
}

You shouldn't account for other people's shortcuts. Stick with the language's specifications. :)

You shouldn't account for other people's shortcuts. Stick with the language's specifications. :)

Touché

function Foo() {
   if (!(this instanceof Foo)) return new Foo();
   // ...
}

This effectively turns Foo() into a factory, as well, albeit with some weird side-effects related to this and instanceof.


Converted wayfinding task example for VORC cora model and marina world

Changed the starting point of cora model to match that of perception_task.
Adjusted one of the waypoints from the VRX version to fit the marina world.
Other changes reflected in pull request #17

Code looks good. Thanks!
Could you add some steps for how to test this? I can see the error changing as I move the boat but I don't know which direction is the goal looking at the lat long. Is the red marker supposed to be on the vehicle at startup? Seems different from the station-keeping task where the red marker marks the goal. I can see that is the case in VRX too but just wondering.
Maybe @caguero could test this out faster since he's more familiar with the task. This has already been merged to master but needs testing.

Reopening for visibility so that we know it needs to be tested

Hmm now it shows conflicts. I'll leave it closed so we aren't confused.


Merge next into master

As titled.

CI is failing

For some reason npm keeps installing the alpha.1 release of Fastify and I just don't have the patience to figure it out today. I'll come back later.

The only way I could figure out to fix the problem was to mark the mislabeled<EMAIL_ADDRESS>version as deprecated.


Brute force approach

Ok this one uses the fact that we can safely call create and start on the node even if is is already created and/or started. It's more of a brute force approach where we call create and start on the node for every background or foreground event we get no matter what. Additionally, if the event is a background event, we do our background task thing and let the node run for 20 seconds, but that can get interrupted by another foreground event.
@andrewxhill let's see how this works on that old phone of yours.

If this doesn't work, I'm out of ideas.

now we're talking


Problems with SPARK_Mode

I see a lot of problems using SPARK GPL 2015. For example, this (concocted from stm32f4-adc.ads and stm32f42xxx.ads) occurs frequently (there are others).
with System.Storage_Elements;
use System;
use type System.Storage_Elements.Storage_Offset;

package Access_To_Volatile with Spark_Mode is

   type Analog_To_Digital_Converter is limited record
      Dummy : Integer;
   end record
   with Volatile;

   type Analog_Input_Channel is mod 19;

   type ADC_Point is record
      ADC     : access Analog_To_Digital_Converter;
      Channel : Analog_Input_Channel;
   end record;

   VBat_Channel : constant Analog_Input_Channel := 18;

   Peripheral_Base : constant Address := System'To_Address (16#4000_0000#);
   APB2_Peripheral_Base : constant Address := Peripheral_Base + 16#0001_0000#;
   ADC1_Base   : constant Address := APB2_Peripheral_Base + 16#2000#;

   ADC_1 : aliased Analog_To_Digital_Converter
     with Volatile, Address => ADC1_Base;
   VBat : constant ADC_Point :=
     (ADC_1'Access, Channel => VBat_Channel);

end Access_To_Volatile;

results in
$ gnatprove -P sparking.gpr
Phase 1 of 2: generation of Global contracts ...
access_to_volatile.ads:28:07: volatile object cannot appear in this context (SPARK RM 7.1.3(13))
gnatprove: error during generation of Global contracts

I see you have some Codepeer constructs in bodies; is it that you don’t use SPARK, or is it that the GPL 2015 version has problems in this area? (I note that the error message is wrong, there is no (13) in the SPARK RM released with GPL 2015!)

Hello Simon,
This issue is not related to the driver repository.
Regards,

So there is no point in opening the other issue I had in mind, which is that compiling with -gnaty results in many many style warnings?
You seem to be saying that this code isn’t intended to be compatible with  formal methods?

Discussion about coding style for this repository is not a problem. I'm actually in favor of -gnaty. Let's see what the others think about it.
About formal methods, it is not a goal for this project at the moment. If you can make the drivers compatible without losing features and usability then please go ahead.

FWIW I'm also in favor of -gnaty.

FWIW, I'm also in favor of -gnaty.

I am in favor of -gnaty except for the line-length restriction.

I stick with the line-length restriction because I like to have two windows (Emacs frames) side-by-side. But that’s just my 2p.


Rui/docker support

Purpose

As mentioned in this issue #53 , this PR creates an option for the developer to create a docker container as dev environment, which eliminates the platform-related bugs during the setting up phase of the project.
Changes

adding new npm run start:dev command and start-dev-container.sh script, the command will check for docker environment and setting up a development container for the user with hot module reload support. Dependencies are installed inside the container.
New Requirements

docker environment is needed

@XxWirusonxX @Xu369 Could you two pull this branch and test the changes out please to see if it resolves your issues?

Sure, im not familiar with Docker, is there any Installation Guide ? :) thanks @bbaayezi

Sure, im not familiar with Docker, is there any Installation Guide ? :) thanks @bbaayezi

You can check the official site here.

Was running out of space to install Docker yesterday lmao, im going to test it.. or should i wait until next Push if Louis has done that already?

@XxWirusonxX @Xu369 Could you two pull this branch and test the changes out please to see if it resolves your issues?

yeah, sure, though my issues have been resolved, I will test it to see if it works

Going to close this PR for now. We can reopen it once we return to working on this.

Step 4/8 : COPY package*.json .
When using COPY with more than one source file, the destination must be a directory and end with a /
ERROR: Service 'web' failed to build : Build failed

Step 4 fails for me. I think this is due to changes in later versions of Docker. My versions are:

Docker 20.10.8
Docker-compose 1.29.2

@louislefevre can you test if change COPY package*.json . to COPY package*.json ./ solves the issue?

@louislefevre can you test if change COPY package*.json . to COPY package*.json ./ solves the issue?

Yup, that seems to fix it 👍

Also, where are the files created for the Docker environment stored? So the virtual environment, .node-red directory, etc. Where can that be found?

Also, where are the files created for the Docker environment stored? So the virtual environment, .node-red directory, etc. Where can that be found?

@louislefevre This line in docker-compose.yml creates a docker volume and bind it with the .node-red directiory:
https://github.com/node-red-quantum/node-red-contrib-quantum/blob/6cfa28ba3b2d2cb85b797e70365d1932e0afd83b/docker-compose.yml#L17
so when you run docker volume ls, there should be a volume named quantum_flows, that's where the flow configs are stored. The content inside a docker volume can be transferred to the host machine or to other containers, but currently, there is no explicit tutorial in README showing how to achieve that, maybe we can add it in the future PR.


[W6.4h][W11-B1]WANG ZI

@alexawangzi
Some comments added. Please ack & close the PR after reading comments.


Groom workflow

@manthey any idea how to install swcc as a dependency for tox so that the test will succeed?
Please test and let me know if any unexpected behavior occurs. I think I fixed our problem from this morning, but there may be a better way.

Have you restarted celery since switching branches?

It turned out that my issue was that I messed up the project sheet path in my python REPL and didn't notice the error -- so I had the dataset but without any projects.  When trying to groom, we use the project ID, which didn't exist.
This isn't a problem for the MVP (but the error message could certainly be improved), and I could see wanting to create a project anew if there isn't one.

Ok, I filed #169 to track this as a future change.

Do you know how to get swcc installed as a dependency for the tox tests?

It might be as simple as adding ./swcc in the deps = chunk of the [testenv:test] section.  I'll have to verify tomorrow.


yasa mice branch

This issue contains brief details of what I changed to adapt staging.py to work with the recordings I had from mice.
The most significant change is the use of epoch_sec in get_features(), fit(), and sliding_window().
I don't remember why I kept this min() call. My epoch_sec was 2.5 seconds, so I didn't test what happens when epoch_sec is different.
win_sec = min(5, epoch_sec)  # = 2 / freq_broad[0]

I removed the temporal axis because mice don't sleep in one lump. I think this might also help with classifying human napping data. I just commented it out, but it wouldn't be difficult to put a conditional statement there or have a better solution.
        # Add temporal features
        # for mice, relying in "time since start of the night"
        # is a bad idea
        # if we remove this, we can't use the default classifier
        #features['time_hour'] = times / 3600
        #features['time_norm'] = times / times[-1]

I changed the units, which I think has been superseded in  #59
        # Get data and convert to microVolts
        data = raw_pick.get_data() #* 1e6

Another minor thing is the naming of features, that hardcodes the "min" into the variable name. I would consider using "epoch" instead of "min".
In the future, I also plan to change this, because I expect to be able to run yasa in real-time.
        # Extract duration of recording in minutes
        duration_minutes = data.shape[1] / sf / 60
        assert duration_minutes >= 5, 'At least 5 minutes of data is required.'

I think  these lines create problems for people used to mice data because it's usually the case that they don't use all these ratios. For my classifier, I used them and I think they contain value, but it would be nice to check whether things are present before calculating ratios.
            # Add power ratios for EEG
            # TODO: when some bands are not included,
            # this results in key error
            if c == 'eeg':
                delta = feat['sdelta'] + feat['fdelta']
                feat['dt'] = delta / feat['theta']
                feat['ds'] = delta / feat['sigma']
                feat['db'] = delta / feat['beta']
                feat['at'] = feat['alpha'] / feat['theta']

Below everything you can find the full file.
"""Automatic sleep staging of polysomnography data."""
import os
import mne
import glob
import joblib
import logging
import numpy as np
import pandas as pd
import antropy as ant
import scipy.signal as sp_sig
import scipy.stats as sp_stats
import matplotlib.pyplot as plt
from mne.filter import filter_data
from sklearn.preprocessing import robust_scale

from yasa.others import sliding_window
from yasa.spectral import bandpower_from_psd_ndarray

logger = logging.getLogger('yasa')

class SleepStaging:
    """
    Automatic sleep staging of polysomnography data.

    To run the automatic sleep staging, you must install the
    `LightGBM <https://lightgbm.readthedocs.io/>`_ and
    `antropy <https://github.com/raphaelvallat/antropy>`_ packages.

    .. versionadded:: 0.4.0

    Parameters
    ----------
    raw : :py:class:`mne.io.BaseRaw`
        An MNE Raw instance.
    eeg_name : str
        The name of the EEG channel in ``raw``. Preferentially a central
        electrode referenced either to the mastoids (C4-M1, C3-M2) or to the
        Fpz electrode (C4-Fpz). Data are assumed to be in Volts (MNE default)
        and will be converted to uV.
    eog_name : str or None
        The name of the EOG channel in ``raw``. Preferentially,
        the left LOC channel referenced either to the mastoid (e.g. E1-M2)
        or Fpz. Can also be None.
    emg_name : str or None
        The name of the EMG channel in ``raw``. Preferentially a chin
        electrode. Can also be None.
    metadata : dict or None
        A dictionary of metadata (optional). Currently supported keys are:

        * ``'age'``: age of the participant, in years.
        * ``'male'``: sex of the participant (1 or True = male, 0 or
          False = female)

    Notes
    -----

    If you use the SleepStaging module in a publication, please cite the following publication:

    * Vallat, R., & Walker, M. P. (2021). An open-source, high-performance tool for automated
      sleep staging. Elife, 10. doi: https://doi.org/10.7554/eLife.70092

    We provide below some key points on the algorithm and its validation. For more details,
    we refer the reader to the peer-reviewed publication. If you have any questions,
    make sure to first check the
    `FAQ section <https://raphaelvallat.com/yasa/build/html/faq.html>`_ of the documentation.
    If you did not find the answer to your question, please feel free to open an issue on GitHub.

    **1. Features extraction**

    For each 30-seconds epoch and each channel, the following features are calculated:

    * Standard deviation
    * Interquartile range
    * Skewness and kurtosis
    * Number of zero crossings
    * Hjorth mobility and complexity
    * Absolute total power in the 0.4-30 Hz band.
    * Relative power in the main frequency bands (for EEG and EOG only)
    * Power ratios (e.g. delta / beta)
    * Permutation entropy
    * Higuchi and Petrosian fractal dimension

    In addition, the algorithm also calculates a smoothed and normalized version of these features.
     The resulting smoothed features are then normalized using a robust
    z-score.

    .. important:: The PSG data should be in micro-Volts. Do NOT transform (e.g. z-score) or filter
        the signal before running the sleep staging algorithm.

    The data are automatically downsampled to 100 Hz for faster computation.

    **2. Sleep stages prediction**

    YASA comes with a default set of pre-trained classifiers, which were trained and validated
    on ~3000 nights from the `National Sleep Research Resource <https://sleepdata.org/>`_.
    These nights involved participants from a wide age range, of different ethnicities, gender,
    and health status. The default classifiers should therefore works reasonably well on most data.

    The code that was used to train the classifiers can be found on GitHub at:
    https://github.com/raphaelvallat/yasa_classifier

    In addition with the predicted sleep stages, YASA can also return the predicted probabilities
    of each sleep stage at each epoch. This can be used to derive a confidence score at each epoch.

    .. important:: The predictions should ALWAYS be double-check by a trained
        visual scorer, especially for epochs with low confidence. A full
        inspection should be performed in the following cases:

        * Nap data, because the classifiers were exclusively trained on full-night recordings.
        * Participants with sleep disorders.
        * Sub-optimal PSG system and/or referencing

    .. warning:: N1 sleep is the sleep stage with the lowest detection accuracy. This is expected
        because N1 is also the stage with the lowest human inter-rater agreement. Be very
        careful for potential misclassification of N1 sleep (e.g. scored as Wake or N2) when
        inspecting the predicted sleep stages.

    References
    ----------
    If you use YASA's default classifiers, these are the main references for
    the `National Sleep Research Resource <https://sleepdata.org/>`_:

    * Dean, Dennis A., et al. "Scaling up scientific discovery in sleep
      medicine: the National Sleep Research Resource." Sleep 39.5 (2016):
      1151-1164.

    * Zhang, Guo-Qiang, et al. "The National Sleep Research Resource: towards
      a sleep data commons." Journal of the American Medical Informatics
      Association 25.10 (2018): 1351-1358.

    Examples
    --------
    For a concrete example, please refer to the example Jupyter notebook:
    https://github.com/raphaelvallat/yasa/blob/master/notebooks/14_automatic_sleep_staging.ipynb

    >>> import mne
    >>> import yasa
    >>> # Load an EDF file using MNE
    >>> raw = mne.io.read_raw_edf("myfile.edf", preload=True)
    >>> # Initialize the sleep staging instance
    >>> sls = yasa.SleepStaging(raw, eeg_name="C4-M1", eog_name="LOC-M2",
    ...                         emg_name="EMG1-EMG2",
    ...                         metadata=dict(age=29, male=True))
    >>> # Get the predicted sleep stages
    >>> hypno = sls.predict()
    >>> # Get the predicted probabilities
    >>> proba = sls.predict_proba()
    >>> # Get the confidence
    >>> confidence = proba.max(axis=1)
    >>> # Plot the predicted probabilities
    >>> sls.plot_predict_proba()

    The sleep scores can then be manually edited in an external graphical user interface
    (e.g. EDFBrowser), as described in the
    `FAQ <https://raphaelvallat.com/yasa/build/html/faq.html>`_.
    """

    def __init__(self, raw, eeg_name, *, eog_name=None, emg_name=None, metadata=None):
        # Type check
        assert isinstance(eeg_name, str)
        assert isinstance(eog_name, (str, type(None)))
        assert isinstance(emg_name, (str, type(None)))
        assert isinstance(metadata, (dict, type(None)))

        # Validate metadata
        if isinstance(metadata, dict):
            if 'age' in metadata.keys():
                assert 0 < metadata['age'] < 120, 'age must be between 0 and 120.'
            if 'male' in metadata.keys():
                metadata['male'] = int(metadata['male'])
                assert metadata['male'] in [0, 1], 'male must be 0 or 1.'

        # Validate Raw instance and load data
        assert isinstance(raw, mne.io.BaseRaw), 'raw must be a MNE Raw object.'
        sf = raw.info['sfreq']
        ch_names = np.array([eeg_name, eog_name, emg_name])
        ch_types = np.array(['eeg', 'eog', 'emg'])
        keep_chan = []
        for c in ch_names:
            if c is not None:
                assert c in raw.ch_names, '%s does not exist' % c
                keep_chan.append(True)
            else:
                keep_chan.append(False)
        # Subset
        ch_names = ch_names[keep_chan].tolist()
        ch_types = ch_types[keep_chan].tolist()
        # Keep only selected channels (creating a copy of Raw)
        raw_pick = raw.copy().pick_channels(ch_names, ordered=True)

        # Downsample if sf != 100
        assert sf > 80, 'Sampling frequency must be at least 80 Hz.'
        if sf != 100:
            raw_pick.resample(100, npad="auto")
            sf = raw_pick.info['sfreq']

        # Get data and convert to microVolts
        data = raw_pick.get_data() #* 1e6

        # Extract duration of recording in minutes
        duration_minutes = data.shape[1] / sf / 60
        assert duration_minutes >= 5, 'At least 5 minutes of data is required.'

        # Add to self
        self.sf = sf
        self.ch_names = ch_names
        self.ch_types = ch_types
        self.data = data
        self.metadata = metadata

    def fit(self, epoch_sec=30, bands=None):
        """Extract features from data.
        Returns
        -------
        self : returns an instance of self.
        epoch_sec: Time window in seconds to be used for feature extraction. Defaults to 30 seconds.
        """
        #######################################################################
        # MAIN PARAMETERS
        #######################################################################

        # Bandpass filter
        freq_broad = (0.4, 30)
        # FFT & bandpower parameters
        win_sec = min(5, epoch_sec)  # = 2 / freq_broad[0]
        sf = self.sf
        win = int(win_sec * sf)
        kwargs_welch = dict(window='hamming', nperseg=win, average='median')
        if bands is None:
          bands = [
              (0.4, 1, 'sdelta'), (1, 4, 'fdelta'), (4, 8, 'theta'),
              (8, 12, 'alpha'), (12, 16, 'sigma'), (16, 30, 'beta')
          ]

        #######################################################################
        # CALCULATE FEATURES
        #######################################################################

        features = []

        for i, c in enumerate(self.ch_types):
            # Preprocessing
            # - Filter the data
            dt_filt = filter_data(
                self.data[i, :], sf, l_freq=freq_broad[0], h_freq=freq_broad[1], verbose=False)
            # - Extract epochs. Data is now of shape (n_epochs, n_samples).
            times, epochs = sliding_window(dt_filt, sf=sf, window=epoch_sec)

            # Calculate standard descriptive statistics
            hmob, hcomp = ant.hjorth_params(epochs, axis=1)

            feat = {
                'std': np.std(epochs, ddof=1, axis=1),
                'iqr': sp_stats.iqr(epochs, rng=(25, 75), axis=1),
                'skew': sp_stats.skew(epochs, axis=1),
                'kurt': sp_stats.kurtosis(epochs, axis=1),
                'nzc': ant.num_zerocross(epochs, axis=1),
                'hmob': hmob,
                'hcomp': hcomp
            }

            # Calculate spectral power features (for EEG + EOG)
            freqs, psd = sp_sig.welch(epochs, sf, **kwargs_welch)
            if c != 'emg':
                bp = bandpower_from_psd_ndarray(psd, freqs, bands=bands)
                for j, (_, _, b) in enumerate(bands):
                    feat[b] = bp[j]

            # Add power ratios for EEG
            # TODO: when some bands are not included,
            # this results in key error
            if c == 'eeg':
                delta = feat['sdelta'] + feat['fdelta']
                feat['dt'] = delta / feat['theta']
                feat['ds'] = delta / feat['sigma']
                feat['db'] = delta / feat['beta']
                feat['at'] = feat['alpha'] / feat['theta']

            # Add total power
            idx_broad = np.logical_and(freqs >= freq_broad[0], freqs <= freq_broad[1])
            dx = freqs[1] - freqs[0]
            feat['abspow'] = np.trapz(psd[:, idx_broad], dx=dx)

            # Calculate entropy and fractal dimension features
            feat['perm'] = np.apply_along_axis(
                ant.perm_entropy, axis=1, arr=epochs, normalize=True)
            feat['higuchi'] = np.apply_along_axis(
                ant.higuchi_fd, axis=1, arr=epochs)
            feat['petrosian'] = ant.petrosian_fd(epochs, axis=1)

            # Convert to dataframe
            feat = pd.DataFrame(feat).add_prefix(c + '_')
            features.append(feat)

        #######################################################################
        # SMOOTHING & NORMALIZATION
        #######################################################################

        # Save features to dataframe
        features = pd.concat(features, axis=1)
        features.index.name = 'epoch'

        # TODO: change here, rolling windows are hardcoded
        # and assume epoch = 30 sec
        # this will change when epochs change
        # I would consider changing this to '_c15epoch_norm'
        # Apply centered rolling average (15 epochs = 7 min 30)
        # Triang: [0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.,
        #          0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125]
        rollc = features.rolling(
            window=15, center=True, min_periods=1, win_type='triang').mean()
        rollc[rollc.columns] = robust_scale(rollc, quantile_range=(5, 95))
        rollc = rollc.add_suffix('_c7min_norm')

        # Now look at the past 2 minutes
        rollp = features.rolling(window=4, min_periods=1).mean()
        rollp[rollp.columns] = robust_scale(rollp, quantile_range=(5, 95))
        rollp = rollp.add_suffix('_p2min_norm')

        # Add to current set of features
        features = features.join(rollc).join(rollp)

        #######################################################################
        # TEMPORAL + METADATA FEATURES AND EXPORT
        #######################################################################

        # Add temporal features
        # for mice, relying in "time since start of the night"
        # is a bad idea
        # if we remove this, we can't use the default classifier
        #features['time_hour'] = times / 3600
        #features['time_norm'] = times / times[-1]

        # Add metadata if present
        if self.metadata is not None:
            for c in self.metadata.keys():
                features[c] = self.metadata[c]

        # Downcast float64 to float32 (to reduce size of training datasets)
        cols_float = features.select_dtypes(np.float64).columns.tolist()
        features[cols_float] = features[cols_float].astype(np.float32)
        # Make sure that age and sex are encoded as int
        if 'age' in features.columns:
            features['age'] = features['age'].astype(int)
        if 'male' in features.columns:
            features['male'] = features['male'].astype(int)

        # Sort the column names here (same behavior as lightGBM)
        features.sort_index(axis=1, inplace=True)

        # Add to self
        self._features = features
        self.feature_name_ = self._features.columns.tolist()

    def get_features(self, epoch_sec=30, bands=None):
        """Extract features from data and return a copy of the dataframe.

        Returns
        -------
        features : :py:class:`pandas.DataFrame`
            Feature dataframe.
        """
        if not hasattr(self, '_features'):
            self.fit(epoch_sec, bands)
        return self._features.copy()

    def _validate_predict(self, clf):
        """Validate classifier."""
        # Check that we're using exactly the same features
        # Note that clf.feature_name_ is only available in lightgbm>=3.0
        f_diff = np.setdiff1d(clf.feature_name_, self.feature_name_)
        if len(f_diff):
            raise ValueError("The following features are present in the "
                             "classifier but not in the current features set:", f_diff)
        f_diff = np.setdiff1d(self.feature_name_, clf.feature_name_, )
        if len(f_diff):
            raise ValueError("The following features are present in the "
                             "current feature set but not in the classifier:", f_diff)

    def _load_model(self, path_to_model):
        """Load the relevant trained classifier."""
        if path_to_model == "auto":
            from pathlib import Path
            clf_dir = os.path.join(str(Path(__file__).parent), 'classifiers/')
            name = 'clf_eeg'
            name = name + '+eog' if 'eog' in self.ch_types else name
            name = name + '+emg' if 'emg' in self.ch_types else name
            name = name + '+demo' if self.metadata is not None else name
            # e.g. clf_eeg+eog+emg+demo_lgb_0.4.0.joblib
            all_matching_files = glob.glob(clf_dir + name + "*.joblib")
            # Find the latest file
            path_to_model = np.sort(all_matching_files)[-1]
        # Check that file exists
        assert os.path.isfile(path_to_model), "File does not exist."
        logger.info("Using pre-trained classifier: %s" % path_to_model)
        # Load using Joblib
        clf = joblib.load(path_to_model)
        # Validate features
        self._validate_predict(clf)
        return clf

    def predict(self, path_to_model="auto", epoch_sec=30, bands=None):
        """
        Return the predicted sleep stage for each 30-sec epoch of data.

        Currently, only classifiers that were trained using a
        `LGBMClassifier <https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html>`_
        are supported.

        Parameters
        ----------
        path_to_model : str or "auto"
            Full path to a trained LGBMClassifier, exported as a joblib file. Can be "auto" to
            use YASA's default classifier.

        Returns
        -------
        pred : :py:class:`numpy.ndarray`
            The predicted sleep stages.
        """
        if not hasattr(self, '_features'):
            self.fit(epoch_sec, bands)
        # Load and validate pre-trained classifier
        clf = self._load_model(path_to_model)
        # Now we make sure that the features are aligned
        X = self._features.copy()[clf.feature_name_]
        # Predict the sleep stages and probabilities
        self._predicted = clf.predict(X)
        proba = pd.DataFrame(clf.predict_proba(X), columns=clf.classes_)
        proba.index.name = 'epoch'
        self._proba = proba
        return self._predicted.copy()

    def predict_proba(self, path_to_model="auto"):
        """
        Return the predicted probability for each sleep stage for each 30-sec epoch of data.

        Currently, only classifiers that were trained using a
        `LGBMClassifier <https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html>`_
        are supported.

        Parameters
        ----------
        path_to_model : str or "auto"
            Full path to a trained LGBMClassifier, exported as a joblib file. Can be "auto" to
            use YASA's default classifier.

        Returns
        -------
        proba : :py:class:`pandas.DataFrame`
            The predicted probability for each sleep stage for each 30-sec epoch of data.
        """
        if not hasattr(self, '_proba'):
            self.predict(path_to_model)
        return self._proba.copy()

    def plot_predict_proba(self, proba=None, majority_only=False,
                           palette=['#99d7f1', '#009DDC', 'xkcd:twilight blue',
                                    'xkcd:rich purple', 'xkcd:sunflower']):
        """
        Plot the predicted probability for each sleep stage for each 30-sec epoch of data.

        Parameters
        ----------
        proba : self or DataFrame
            A dataframe with the probability of each sleep stage for each 30-sec epoch of data.
        majority_only : boolean
            If True, probabilities of the non-majority classes will be set to 0.
        """
        if proba is None and not hasattr(self, '_features'):
            raise ValueError("Must call .predict_proba before this function")
        if proba is None:
            proba = self._proba.copy()
        else:
            assert isinstance(proba, pd.DataFrame), 'proba must be a dataframe'
        if majority_only:
            cond = proba.apply(lambda x: x == x.max(), axis=1)
            proba = proba.where(cond, other=0)
        ax = proba.plot(kind='area', color=palette, figsize=(10, 5), alpha=.8, stacked=True, lw=0)
        # Add confidence
        # confidence = proba.max(1)
        # ax.plot(confidence, lw=1, color='k', ls='-', alpha=0.5,
        #         label='Confidence')
        ax.set_xlim(0, proba.shape[0])
        ax.set_ylim(0, 1)
        ax.set_ylabel("Probability")
        ax.set_xlabel("Time (30-sec epoch)")
        plt.legend(frameon=False, bbox_to_anchor=(1, 1))
        return ax

This is great @matiasandina!! I'm sure this will be super useful to others. Can I ask: did you re-train a new classifier using these updated features? If so, can you describe the training set as well? And more importantly, would you be willing to share the training set and/or the trained classifier (lightgbm tree paths)?

The dataset was collected from OSF. It contains mouse EEG/EMG recordings (sampling rate: 512 Hz) and sleep stage labels (epoch length: 2.5 sec).
Training was performed using a portion of the open source dataset and testing was performed using another portion (partition is not random but training and testing sets are separate). Training was performed using extracted features from 24h recordings.
I am happy to share everything, what's the best way to go about it?
As of now, the python code with procedure is split in 3 .qmd files (https://quarto.org/). I can make a repo contribution but the dataset needs to be independently downloaded and formatted into folders as expected by the code.
I also have the generated classifier with epoch = 2.5 seconds.
This classifier might use a smoothing of fast transitions that I didn't get to do, but there are a few very fast transitions that are unlikely (a 2.5 sec bout of NREM in the middle of wake).

Thanks @matiasandina! I think that if we are to create a new branch (like "yasa_mice"), the minimum that we need is:

Update the features in staging.py
The trained classifier (*.joblib format) in yasa/classifiers/

And then, we can create a separate repo (see https://github.com/raphaelvallat/yasa_classifier as an example) with the code to reproduce the trained classifier, i.e. model training, data partitioning, performance evaluation, etc.
A few questions:

What is the performance of the algorithm on the test set?
Is there a single EEG channel in this dataset? Do you think that the choice of the electrode will have a strong impact for mice sleep?

This is what I can do for now, please let me know if you can reproduce it.
https://github.com/matiasandina/yasa_classifier

[ ] @matiasandina  will later update the features in staging.py. I can do this inside a yasa_mice branch on my repo if you want to evaluate later whether to merge or what. I think it's not going to be a problem to merge into the original yasa because I didn't change much and the defaults run as the original yasa. But I'm also happy to keep it separate.
[ ]  The trained classifier (*.joblib format) in yasa/classifiers/. This is currently hosted in https://github.com/matiasandina/yasa_classifier/tree/main/output/classifiers

Performance is quite high, above 90 for accuracy and a bit lower for cohen's kappa. I think it can be smoothed out to gain even more performance but I'm OK so far. You can find performance values in this folder, which includes accuracy, cohen's kappa, and classification matrices for each of the 50 4h recordings that I used for testing.
The dataset contains one EEG and one EMG. I believe the choice of electrode will for sure affect the algorithm. Again, data in mice is not as standard. Even for labs that have detailed information, they see that the brain itself enters stages at slightly different moments. Below a result from LFP electrodes by Soltani 2019

I am collecting with a "high throughput" (9 EEG + 2 EMG), so I will have more to say about this in the future.

Most critical thing is, it is extremely dependent on the nature of the data itself. Mice data is not standard (in the same way human data is more standard). I expect all labs to need to re-train with data they generated themselves before getting good results with this. I have not done so because I haven't collected enough data yet, but I don't see this as being an issue in the future (provided me and a few others in lab can label our own data!).

This is really great work @matiasandina, thank you!

Mice data is not standard

Could you say more? What if we only include features that are normalized to the mean and unit variance of the recording, i.e. we normalize data amplitude across recordings?

I can do this inside a yasa_mice branch on my repo if you want to evaluate later whether to merge or what.

I am still undecided on what is the best way to go, i.e. a separate mice branch in YASA (which would however require to fork the repo on GitHub and install the develop version of YASA), or a whole new repository pip install yasa_mice. In both cases, we will need to keep the fork/repo up to date with the main YASA repo. A third option could be to add a species="human" parameter to the yasa.SleepStaging class, which would automatically switch to the relevant features and classifier. This will add some burden to the code however.

or each of the 50 4h recordings that I used for testing.

Do you think that the length of the input data will change the output? Can you just pass 24-hour recordings or even multi-day recording to the sleep staging function?
If there are mice researchers following this thread, please feel free to chime in with your ideas and preferences :)

@matiasandina thanks for the detailed explanation! Another naive question: is mice sleep similar to rat sleep? Do you think your classifier would work well on rats data?
Maybe another option is to have some sort of configurable file that determine the features, e.g. which feature to compute, the length of the epoch, the length of the smoothing window, etc. That way, you just need the config file (*.json or *.yaml) and the updated classifier to run YASA on another species.

Sorry for the late reply. Provided re-training, I think this is flexible to work multi-species. I like the idea of a config file that determines the features! It's been a bit difficult to find some time to work on this since coming back from vacation and trying to get my PhD in motion again, but it's on the list!


Flatten styles to avoid crash

Details
Flatten styles
Before:

After:

Fixed Issues

$ https://github.com/Expensify/App/issues/13508
PROPOSAL: GH_LINK_ISSUE(COMMENT)
Tests

Access staging.new.expensify.com
Sign into any valid account
Tap on Profile > Workspace > Manage members
Invite any member
Turn off the internet connection
Remove a user from the workspace while offline
Verify that the user gets removed and the App doesn't crash

[x] Verify that no errors appear in the JS console

Offline tests

QA Steps

Access staging.new.expensify.com
Sign into any valid account
Tap on Profile > Workspace > Manage members
Invite any member
Turn off the internet connection
Remove a user from the workspace while offline
Verify that the user gets removed and the App doesn't crash

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] iOS / native
[x] Android / native
[x] iOS / Safari
[x] Android / Chrome
[x] MacOS / Chrome
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product was added in all src/languages/* files
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/87341702/207145346-bb329a85-b87f-46f1-bcb0-c39fb0d70251.mov

Mobile Web - Chrome
https://user-images.githubusercontent.com/87341702/207145122-fdfd1c7e-b4cb-46a0-8eb8-ada9e8641d99.mov

Mobile Web - Safari
https://user-images.githubusercontent.com/87341702/207145151-16276bd3-0bc3-422f-9b4c-954b30e60e82.mov

Desktop
https://user-images.githubusercontent.com/87341702/207145657-8a781a15-90f0-4c36-b9a6-ee479b2968f8.mov

iOS
https://user-images.githubusercontent.com/87341702/207144574-f4751bf5-75a7-424f-b741-59f6976cf0ec.mov

Android
https://user-images.githubusercontent.com/87341702/207144402-05888990-9ba1-42ba-87f9-bc25c169b738.mov

Made it ready for review to speed up things, I'll be testing all platforms now

Tested in all platforms.

Asked about StyleUtils.combineStyles and StyleSheet.flatten here, and I think in this case it should have been StyleUtils.combineStyle because later one of these styles is getting spread into an array: https://github.com/Expensify/App/blob/a464a5ce6498a62dcc93bb6a28f9209c265d5505/src/components/DisplayNames/index.js#L86

This is ready for review again

Reviewer Checklist

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I included screenshots or videos for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] iOS / native
[ ] Android / native
[ ] iOS / Safari
[ ] Android / Chrome
[ ] MacOS / Chrome
[ ] MacOS / Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product was added in all src/languages/* files
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

🚀 Cherry-picked to staging by @chiragsalian in version: 1.2.38-6 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@Expensify/applauseleads please QA this PR and check it off on the deploy checklist if it passes.

🚀 Deployed to production by @chiragsalian in version: 1.2.38-6 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Cherry-picked to staging by https://github.com/AndrewGable in version: 1.3.28-2 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@Expensify/applauseleads please QA this PR and check it off on the deploy checklist if it passes.

🚀 Deployed to production by https://github.com/AndrewGable in version: 1.3.28-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Allow Heading increase from size 0

Using Lower limit of heading 0 lets me decrease from # to nothing. I'd like to be able to go from nothing to # and etc. This way I can easily use Increase Headings and Decrease Headings to cycle through all including no heading at all.

Do you mean you want to be able to use “Increase Headings” instead of “Apply Heading X” to change the heading from 0 to 1?
If so, “Increase Headings” function only targets rows that already have a Heading when selecting multiple rows, so it is difficult to target rows that do not have a Heading (0).
If this is misplaced or you have other ideas, please let us know.

Apologies if I was unclear. Yes I want to be able to go from no heading at all to a heading by running "Increase Headings". Functionally I want this so I can use the same keyboard shortcuts to shift up / down headings as well as heading / no heading. I assume you're saying this is currently not possible based on how the plugin works?

Essentially "If there's no heading under the current text and I run increase heading then apply heading 1"

Yes. Not currently possible, but there are several ways to achieve this.

create an option “Include non-heading rows in Increase"
create another command “Increase Headings(forced)” and make it work the same as when the option ↑ is set to true
if only one row is selected, make the line without headings also be the target of “Increase”.

In the case of 1 and 3, it is troublesome to determine whether lines with list (e.g., -, 1) or other notations should be included, so it is easier to create 2, which is unconditionally included.
By the way, which key do you currently use for “Increase Headings”, “Tab” or another key?

I use Control + Command + - to decrease and Control + Command + = to increase. I think it works as ctrl + alt on Windows.

Hi @k4a-l I have noticed a very minor thing in the usability of this feature.
When I run the shortcut my cursor stays to the left of the inserted ##s. It would be great if your cursor stuck to the right with a space after the ##s so you can just start typing in the heading if I go from 0 headings to anything up.

fixed


AlertDialog: action buttons' callbacks are not trigged in native alerts

Currently, the recommended way to pass callbacks to AlertDialog is to use the Press events on the Button elements nested in AlertDialog.Action. However, those do not get called when the AlertDialog is rendered natively (i.e. <AlertDialog native>). The only way to pass a callback on native is to use onOpenChange but there is only one for the entire dialog.

Need a more clear example + repro, but the native alertdialog is limited, what do you want to do?

We'd like to attach a callback to each action (e.g. onClickOk and onClickCancel). Using the  example from the AlertDialog, the callbacks onCancel and onAccept will be called on web, but not on native.
export function AlertDialogDemo() {
  return (
              ...
              <AlertDialog.Cancel asChild>
                <Button onPress={onCancel}>Cancel</Button>
              </AlertDialog.Cancel>
              <AlertDialog.Action asChild>
                <Button theme="active" onPress={onAccept}>Accept</Button>
              </AlertDialog.Action>
            ...
  )
}

It should be possible to attach callbacks since Alert in react-native allows passing an onPress prop to each action.


mac-os 13 runner need intermediates certificates update

Description
I'm using fastlane to build my apps with github actions
I think the macos-13 runner need an update for the intermediates apple certificates because i didnt change anything and my builds started to fail 3 months ago, around the expire day of the WWDRCAG7 certificate
I have this error:
Error: Revoke certificate: Your account already has an Apple Development signing certificate for this machine, but its private key is not installed in your keychain. Xcode can create a new one after revoking your existing certificate.
I saw people solved it locally before with installing the new WWDR certificates (AppleWWDRCAG8) but i cant find a way to fixed it, i tried this but it didnt worked:
- name: install intermediates certificates         run: wget https://www.apple.com/certificateauthority/AppleWWDRCAG8.cer && sudo           security add-trusted-cert -d -r unspecified -k
The previous certificates expired in november:
Worldwide Developer Relations - G7 (Expiring 11/17/2023 20:40:52 UTC)
Platforms affected

[ ] Azure DevOps
[X] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[X] macOS 13
[ ] macOS 13 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
runner: macos-13
https://github.com/Rythme-Tech/rythme-ios/actions/runs/7530996744
Is it regression?
no
Expected behavior
Should not fail to build testflight build
Actual behavior
It failed with this error:
Revoke certificate: Your account already has an Apple Development signing certificate for this machine, but its private key is not installed in your keychain. Xcode can create a new one after revoking your existing certificate.
Repro steps
.

Hey! We do not install WWDRCAG7 in runner, only WWDRCAG3 which expires somewhere in 2030, where is this cert coming from in your builds, might be we missed something?

Oh okay so the issue might be on fastlane side then i'll try to search more
i'm not downloading manually the certificates i just give the credentials to fastlane
- name: Setup Env         env:           FASTLANE_USER: ${{ secrets.FASTLANE_USER }}           FASTLANE_PASSWORD: ${{ secrets.FASTLANE_PASSWORD }}           FIREBASE_AUTH_KEY: ${{ secrets.FIREBASE_AUTH_KEY }}           APPSTORE_PRIVATE_KEY_64: ${{ secrets.APPSTORE_PRIVATE_KEY_64 }}         run: |           echo $APPSTORE_PRIVATE_KEY_64 | base64 --decode > AuthKey.p8           echo $FIREBASE_AUTH_KEY > Firebase.json           echo FASTLANE_PASSWORD=$FASTLANE_PASSWORD >> $GITHUB_ENV           echo FASTLANE_USER=$FASTLANE_USER >> $GITHUB_ENV

@clementmorissard fastlane fetches G1-G6 and not G7, we only install G3, what's landing into your keychain on fastlane execution?

Found the issue. It's not on the runner issues fault.
Fastlane is recreating a new certificate everytime i build for appstore, and when i reach the certificates limit, i get the error saying i need to revoke certificate :/
I'll try to see if i can change the way i use fastlane then
thanks !


Change Blocked Example Port

The example port (6000) is blocked on modern browsers to prevent cross-protocol scripting attacks to X11 which runs on the same port. Changed to a non-restrictive port (5000) so that people following the deployment guide will not end up with issues such as #126
See these articles for more info:
http://www-archive.mozilla.org/projects/netlib/PortBanning.html
code.google.com/archive/p/browsersec/wikis/Part2.wiki#Port_access_restrictions

From https://github.com/GitbookIO/nuts/pull/132

This is great as it matches whats in .example.env


Gradients break Component#replaceText()

Code:
// MiniMessage NOT required to reproduce but used for readability
final Component component = MiniMessage.miniMessage().deserialize("<gradient:red:blue>This is a test, {player}").replaceText(b -> b.matchLiteral("{player}").replacement("Demo"));

System.out.println("Demo: " + PlainTextComponentSerializer.plainText().serialize(component));

The way this is implemented (by the way, MiniMessage is NOT required to reproduce, simply chain letters with different colors to do so) the Component turns into {"extra":[{"color":"red","text":"T"},{"color":"#F8555C","text":"h"},{"color":"#F05564","text":"i"},{"color":"#E9556B","text":"s"},{"color":"#E15573","text":" "},{"color":"#DA557A","text":"i"},{"color":"#D35581","text":"s"},{"color":"#CB5589","text":" "},{"color":"#C45590","text":"a"},{"color":"#BC5598","text":" "},{"color":"#B5559F","text":"t"},{"color":"#AE55A6","text":"e"},{"color":"#A655AE","text":"s"},{"color":"#9F55B5","text":"t"},{"color":"#9855BC","text":","},{"color":"#9055C4","text":" "},{"color":"#8955CB","text":"{"},{"color":"#8155D3","text":"p"},{"color":"#7A55DA","text":"l"},{"color":"#7355E1","text":"a"},{"color":"#6B55E9","text":"y"},{"color":"#6455F0","text":"e"},{"color":"#5C55F8","text":"r"},{"color":"blue","text":"}"}],"text":""} and the replaceText() method is unable to find the letters.
The only way we "fixed" it so far is to replace variables on the raw message before parsing it.
Thanks
Matej

Unfortunately this is a limitation of replaceText, it doesn't look for matches across component boundaries. It's possible to redesign the replacement logic to not have this limitation, but we haven't figured out a way to do it without significantly increasing the complexity of the API.


'Run All Tests' button does not invoke Auto Save

Environment data

VS Code version: 1.38.1
Extension version (available under the Extensions sidebar): 2019.10.37964-dev
OS and version: MS Windows 10 Pro 1903 18362.356 64bits
Python version (& distribution if applicable, e.g. Anaconda): Python 3.7 64-bit
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
Relevant/affected Python packages and their versions: pytest 5.1.3
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): Jedi
Settings has Auto Save enabled with onFocusChange option selected
pytest was selected as test framework

Expected behaviour
When a file is dirty, clicking on the button 'Run All Tests' or 'Run Failed Tests' in the Test Explorer saves the dirty file before the tests are run. Tests are run with the updated code.
Similar for the 'Discover Tests' button.
Actual behaviour
When a file is dirty, clicking on the button 'Run All Tests' or 'Run Failed Tests' in the Test Explorer does not save the dirty file before the tests are run. Tests are run with the outdated code.
Similar for the 'Discover Tests' button.
The dirty file is saved when clicking clicking the 'Show Test Output' or 'Collapse All' buttons.
Steps to reproduce:

IMHO obvious from above description, but I'll provide them if you cannot reproduce.

Logs
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
> ~\venvs\rentomatic-env\Scripts\python.exe -m pytest --junitxml=C:\Users\roger\AppData\Local\Temp\tmp-95569HLLaMJLU5V3.xml -s
cwd: c:\Users\roger\projects\py\rentomatic

Normally, the linter is invoked when focus shifts:
> ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
cwd: c:\Users\roger\projects\py\rentomatic
> ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
cwd: c:\Users\roger\projects\py\rentomatic
##########Linting Output - pylint##########

--------------------------------------------------------------------

Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)
[Extension Host] Info Python Extension: 2019-09-24 22:15:05: Cached data exists ActivatedEnvironmentVariables, c:\Users\roger\projects\py\rentomatic
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:05: getActivatedEnvironmentVariables, Class name = _, completed in 1ms, Arg 1: <Uri:c:\Users\roger\projects\py\rentomatic>, Arg 2: undefined, Arg 3: undefined
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:05: > ~\venvs\rentomatic-env\Scripts\python.exe -m pytest --junitxml=C:\Users\roger\AppData\Local\Temp\tmp-95562XVhzmAhK5Nb.xml -s
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:05: cwd: c:\Users\roger\projects\py\rentomatic

And only after focus shifts:
[Extension Host] Info Python Extension: 2019-09-24 22:15:26: Cached data exists getEnvironmentVariables, c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: > ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: cwd: c:\Users\roger\projects\py\rentomatic
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: > ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: cwd: c:\Users\roger\projects\py\rentomatic

Thanks for reporting this issue. Changing to enhancement as this is by design.
Extension does not save files on behalf of the user.
Saving would require a design decisiom, decision ls sick as displaying promoted to warm users, then considering backwards compatibility (users didn't expect files to be saved in the past).

Hi Don,
thanks for picking up this one.
I'm not sure I fully understand the (second part of the) design decision: some buttons in Test Explorer lead to saving the file, other buttons in Test Explorer don't.
As I see it, it is the responsibility of the Editor to save a file when it looses focus (given my chosen option onFocusChange), be that either to an extenstion's panel, the Terminal panel, or even a browser or MS Word.
Anyway, thanks for this awesome piece of work by you and your team!

As I see it, it is the responsibility of the Editor to save a file when it looses focus

Thats right. However the extension doesn't save the files (at least not to my knowledge).
Hence the need for the decision (when should we - extensions do this, and when should the editor - VS Code do this).

Clicking on "debug tests" saves the file. I agree that it would be expected for the file to be saved when clicking on "Run All Tests" as well.

As far as I can tell the issue is still in effect. I've found this ticket after looking how to save the current test file when running a single test from the test mark in the file, although that probably would be a separate issue.
I'm unsure what the [needs proposal] label means and couldn't find anything in the wiki, but here is are my two cents:
I'd propose to save all files on test execution in general with a settings flag to enable this behavior. The default should be false.
That solves the issue initially described, my use case and is compatible with previous experience.

Hello! We have just finished our testing rewrite and are beginning the roll out to users. I have tested this issue with the re-write and I am no longer reproducing the bug! If you would like to try it yourself, you need to add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Let me know if it doesn’t work for you and we can re-open this issue. Thanks!


net: lwm2m: Publicize firmware block context

I apologize for the delay.  My work schedule has been very busy and I was away on vacation for the last few weeks.  I'm reviewing this now.

I recommended squashing the 2nd patch back into the first, but overall, I wonder if this isn't a situation that the LwM2M subsys should just handle.  Why leave it up to the client at all?

@mike-scott What should we do with this, merge or not to merge?

@jukkar don't merge.  There is still the open question concerning the 2nd patch.

@yaomon can you move the brace change back into the first patch (so there is only 1 patch)?

Yea, I'll try to get on it ASAP.

Ok, the commits have been squashed. Let me know what else needs to be changed. Thanks for taking the time to respond :)

stale, closing. Reopen when there are updates...


Fixed the intersectionXYZ bug generating duplicate intersections

Holding this open in case you want to implement Earlin's test idea, and his fix for the Arc3d constructor.


unbound: 1.21.0 -> 1.21.1

Changes: https://www.nlnetlabs.nl/news/2024/Oct/03/unbound-1.21.1-released/

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Fixes CVE-2024-8508.
https://nlnetlabs.nl/downloads/unbound/CVE-2024-8508.txt

I don't know.  Maybe it's worth updating in staging-24.05 as well (from 1.20.0).

For 24.05 the patch was applied https://github.com/NixOS/nixpkgs/pull/347156


Investigate performance optimizations for Automatic Parse Tree Creation

immutable frozen empty arrays as place holders?

Out of scope for 1.0.0 as it would probably not change the API


Feature Request: Adding timestamps back to console/terminal

I like seeing it on the terminal xd

https://github.com/Nicklason/tf2-automatic/commit/b375674f29d5a91f073f103613ca35806686bf63