Crunchyroll URLs not being recognized by crunchyroll extractor

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.10.04 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
United States
Provide a description that is worded well enough to be understood
Around 2 weeks ago, I ran into the same issue with Crunchyroll that many people were having, where every attempted download was throwing a 403: forbidden error. I found the workaround in issue https://github.com/yt-dlp/yt-dlp/pull/5294: updating from Python version 3.8 to 3.10 indeed made this issue go away.
However, now every single video I attempt to download from Crunchyroll results in yt_dlp defaulting to the generic extractor, and not recognizing Crunchyroll at all. Downloads worked on all videos on there prior to the transition out of beta that many people experienced. I have tried several attempts, with no arguments at all and with cookies, and nothing has worked. I have not made any changes to my instance of yt_dlp or anything else, aside from upgrading to Python 3.10. Additionally, I just tried this on 3.8 and I'm now having the same issue, of it defaulting to the generic extractor.
Edit: Just realized this seems to be a duplicate of https://github.com/yt-dlp/yt-dlp/issues/5292. Please ignore.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['--verbose', 'https://www.crunchyroll.com/watch/G9DUEJ002/meowys-whereabouts']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f] (pip)
[debug] Python 3.10.8 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-108640-g9e8a327e68-20221013 (setts), ffprobe N-108640-g9e8a327e68-20221013
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, secretstorage-3.3.3, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] [generic] Extracting URL: https://www.crunchyroll.com/watch/G9DUEJ002/meowys-whereabouts
[generic] meowys-whereabouts: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] meowys-whereabouts: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
ERROR: Unsupported URL: https://www.crunchyroll.com/watch/G9DUEJ002/meowys-whereabouts
Traceback (most recent call last):
  File "C:\Users\***********\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1477, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\***********\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1553, in __extract_info
    ie_result = ie.extract(url)
  File "C:\Users\***********\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\common.py", line 672, in extract
    ie_result = self._real_extract(url)
  File "C:\Users\***********\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\generic.py", line 3062, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.crunchyroll.com/watch/G9DUEJ002/meowys-whereabouts

duplicate of #5292


User can't change the password : reset password token is invalid

I' am trying use the password reset function of devise, i set up an account on mandrill for sending emails.
So when i go on /users/password/new and click on send instructions i receive an email with a link "Change my password" , here is the link ref :
http://mandrillapp.com/track/click.php?u=30099686&id=bc6ced2b3ac34c9f93f7cd95b1ea1106&url=http%3A%2F%2Flocalhost%3A3000%2Fusers%2Fpassword%2Fedit%3Freset_password_token%3Dd8c12b50e66069b5d8b22e9b054af3441ced62e6ab3648392960823373db1524&url_id=c142395039d4f32dd19fa3b6a6f18291971049fc
when i click on it, it  sends me to /users/password/edit?reset_password_token=XXX.
But if i change the password the following flash message appears in the page: Reset password token invalid.
I am not overriding any devise controllers.
Anyone can help me?

I have the same problem as @cderche using the following:
Rails 4.2.0
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]
activeadmin-1.0.0.pre1
Thanks ahead for any pointers!

I am using,
Rails 3.2.13
Devise 3.1.0
also changed @resource.*_token to @token but still it says '... token is invalid'

I was able to resolve this through overriding authentication method of devise. below is my override function.
`class User
...
def self.find_first_by_auth_conditions warden_conditions
conditions = warden_conditions.dup
if (email = conditions.delete(:email)).present?
where(email: email.downcase).first
elsif conditions.has_key?(:reset_password_token)
where(reset_password_token: conditions[:reset_password_token]).first
end
end
end
`
It worked then after.

I was able to resolve this through overriding authentication method of devise. below is my override function.
class User ... def self.find_first_by_auth_conditions warden_conditions conditions = warden_conditions.dup if (email = conditions.delete(:email)).present? where(email: email.downcase).first elsif conditions.has_key?(:reset_password_token) where(reset_password_token: conditions[:reset_password_token]).first end end end
It worked then after.

I am using devise 4.2
Rails 4.2.8
Ruby 2.3.4
I don't know how to setup devise forgot password with my Gmail. Can anyone help me?


PYNONYMIZER_STRATEGY dropped without deprecation

Describe the bug
A working v1.x pynonymizer config that has env var PYNONYMIZER_STRATEGY set is failing with v2  :
Missing values for required arguments:
Missing INPUT
Missing STRATEGYFILE

There's no mention in the CHANGELOG of PYNONYMIZER_STRATEGY being deprecated vs PYNONYMIZER_STRATEGYFILE - I'm wondering if it's been unintentionally dropped, as at the moment it's a breaking change without a prior deprecation?
To Reproduce
as above
Expected behavior
either PYNONYMIZER_STRATEGY continues to work, or a deprecation message
Additional context

likewise, PYNONYMIZER_START_AT (and presumably PYNONYMIZER_STOP_AT) have also been dropped without doc changes or a deprecation warning?

Hi, thanks for reporting this. It looks like this was accidentally changed because of the differences in argument names in the new CLI library I'm using. This is absolutely a bug and I will restore the previous behaviour. I've added this to v2.2.1, stay tuned for updates.

PYNONYMIZER_START_AT has been accidentally renamed to PYNONYMIZER_START_AT_STEP. I'll make sure this behaviour is reverted.

I've changed my pipelines to use the new variable names, other folks have probably done the same - might be nice to deprecate the new names rather than just a revert?

Good suggestion, thanks! Both variations will be supported in 2.2.1. Closing this, but let's keep talking if it's not fixed.


Issues with .build

Hi
I have a 'build method doesn't exist' issue. Can you update your doc ?
I tried to use:
"
var mail = mailcomposer({from: '...', ...});
mail.build(function(err, message){
process.stdout.write(message);
});
"
Thanks for your help :)

build method behaves like documented. This is how you'd use it:
var mailcomposer = require('mailcomposer');
var mail = mailcomposer(data);
mail.build(function(err, message){
     console.log(err || message);
});

My bad, I based my code on http://www.dominikschmidt.net/2015/08/sending-emails-with-attachement-for-new-s3-objects/ and he used "var MailComposer = require('mailcomposer').MailComposer"...
It works now!
One more issue for me (probably not a bug): I am trying to send an email with an mp3 attachment coming from S3.
My code:
s3.getObject(params, function(err, data) {
  if (err) {
  } else {
    var mailOptions = {
      from: 'xxxx',
      to: 'vxxxx',
      subject: 'xxxx',
      body: 's3://' + bucket + '/' + key,
      attachments: [{
        filename: 'voicemail.mp3',
        contents: data.Body,
        contentType: 'audio/mpeg',
        encoding: 'binary',
        //cid: key,
        }]
    };

    var mailcomposer = MailComposer(mailOptions);
    mailcomposer.build(function(err, messageSource) {
      if (err) {
        } else {
            ses.sendRawEmail({RawMessage: {Data: messageSource}}, function(err, data) {
              if(err) {
              } else {
                  context.done(null, data);
              }
            });
        }
    });
  }
});

data.Body is a Buffer but it doesn't seems to get transformed well
I get the email fine, with the proper attachment name and type but with a Zero byte file....
Thanks again for the great piece of software and the quick support!


[keyvault] Make Managed HSM tests only weekly

API change check
API changes are not detected in this pull request.

/azp run js - keyvault-keys - tests

/azp run js - keyvault-admin - tests

/azp run js - keyvault-admin - tests


sw-precache doesn't add service-worker-registration.js

Currently, sw-precache doesn't add service-worker-registration.js file to project. Users need download https://github.com/googlechrome/sw-precache/blob/master/demo/app/js/service-worker-registration.js and add it manually every time.

Right, that's expected. That service-worker-registration.js file is isn't meant to be automatically injected into your codebase. You could adapt it and incorporate it as you see fit, but it's up to you to decide which pages should trigger SW registration, where you serve you service-worker.js file from, whether you want to take addition post-registration steps like ask for push notification permissions, etc.
There isn't a one-size-fits-all solution.


Help

After installing and going through the video and written instructions and running the application it gave this error. The items on figma are carefully labeled. What could be the problem?
Installation
py -m venv env
.\env\Scripts\activate
where python
git clone https://github.com/ParthJadhav/Tkinter-Designer
cd Tkinter-Designer
pip3 install -r requirements.txt
py tkinter_designer.py

Error Message
window_elements = fig_window["children"]
KeyError: 'children'

Link: Figma link

Yes @Johnny-Five1982 everything is right except you missed one thing - you didn't create a frame.
Press F and create a frame around all elements. It's like Tkinter Window.

Thanks! Got it to work now. Your doing a great job with this! Keep it up!

Nice to hear that , Thanks @Johnny-Five1982  !


Support for Typescript files

Seems like hackmud rules don't get applied to TypeScript files

I thought this would have been a simple fix, but typescript-eslint has a bug that means you can't use the no-closure-sibling rule.
https://github.com/typescript-eslint/typescript-eslint/issues/2439


Example for Config not setting title on client side

(I made a couple of forced pushes to your branch; let me know if you prefer having me create my own branch instead.)

(I made a couple of forced pushes to your branch; let me know if you prefer having me create my own branch instead.)

No, it's all good, thank you for looking at it.


Use jsmpeg in worker (disable document references from jsmpeg.js)

Hi,
I have a use case where I use jsmpeg.js in a worker. In order to make this possible, I had to disable many modules which I do not use in the output file, and remove the references to the document in the src/jsmpeg.js file.
I wonder if could be possible to make it possible to make this configurable. Editing a custom build.sh file is pretty easy, so the main issue would be to disable/enable document references from src/jsmpeg.js.

I'm afraid the bigger problem would be the use of WebGL in a WebWorker. Last time I checked only Mozilla offered this option.
See https://github.com/phoboslab/jsmpeg/issues/157

As far as I understand WebGL is only used for rendering. In my use case I'm using the worker only to perform decoding, while rendering is performed in the main thread, so I don't mind if using WebGL from the worker is possible or not.

@saste @samsonradu
Any progress here? Would be possible to use web workers with transferable objects to do the decoding and transfer the result back for rendering.


[WIP][KYUUBI #2499] Add Iceberg and minimal DSv2 commands to Authz plugin

Why are the changes needed?

Current Authz plugin cannot handle DSv2 and Iceberg related commands. Add them to the module.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

I'm writing unit tests, but I'm stuck to add Iceberg extensions for testing. For testing Iceberg commands, I need to add IcebergSparkSessionExtensions when SparkSession.builder()
  protected val extension: SparkSessionExtensions => Unit = _ => Unit
  protected lazy val spark: SparkSession = {
  ...
    SparkSession.builder()
      .master("local")
      .config("spark.ui.enabled", "false")
      .config("javax.jdo.option.ConnectionURL", s"jdbc:derby:;databaseName=$metastore;create=true")
      .config("spark.sql.catalogImplementation", catalogImpl)
      .config(
        "spark.sql.warehouse.dir",
        Utils.createTempDir(namePrefix = "spark-warehouse").toString)
      .withExtensions(extension)
      .getOrCreate()
  }

My scala understanding is not good, so I'm lookging for help.

Codecov Report

Merging #2564 (9f219cc) into master (93f13ef) will decrease coverage by 0.04%.
The diff coverage is 9.32%.

@@             Coverage Diff              @@
##             master    #2564      +/-   ##
============================================
- Coverage     63.58%   63.54%   -0.05%
- Complexity       69       73       +4
============================================
  Files           371      376       +5
  Lines         17710    18053     +343
  Branches       2358     2445      +87
============================================
+ Hits          11261    11471     +210
- Misses         5404     5480      +76
- Partials       1045     1102      +57

Impacted Files
Coverage Δ

.../kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
60.68% <7.92%> (-19.25%)
:arrow_down:

...ache/kyuubi/plugin/spark/authz/OperationType.scala
60.21% <17.64%> (-9.79%)
:arrow_down:

...yuubi/engine/hive/operation/ExecuteStatement.scala
92.30% <0.00%> (-7.70%)
:arrow_down:

.../org/apache/kyuubi/engine/hive/HiveSQLEngine.scala
69.56% <0.00%> (-5.04%)
:arrow_down:

...org/apache/kyuubi/operation/OperationManager.scala
90.62% <0.00%> (-3.13%)
:arrow_down:

...rg/apache/kyuubi/operation/AbstractOperation.scala
82.27% <0.00%> (-1.55%)
:arrow_down:

...ache/kyuubi/engine/trino/schema/SchemaHelper.scala
95.23% <0.00%> (-1.49%)
:arrow_down:

...pache/kyuubi/engine/KyuubiApplicationManager.scala
82.97% <0.00%> (-1.12%)
:arrow_down:

...uubi/engine/spark/operation/ExecuteStatement.scala
81.69% <0.00%> (-0.81%)
:arrow_down:

...he/kyuubi/engine/hive/events/HiveEngineEvent.scala
88.88% <0.00%> (-0.40%)
:arrow_down:

... and 34 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93f13ef...9f219cc. Read the comment docs.

hi @minyk  have you seen the test class IcebergSuiteMixin in kyuubi-common module ? it should help

hi @minyk have you seen the test class IcebergSuiteMixin in kyuubi-common module ? it should help

@ulysses-you thank you, I missed that.

I'm still working on unit test.

PR #2675 contains some V2Commands, so I would like to drop V2 related code in this PR, then mainly focused on Iceberg commands.
How about this? @yaooqinn

that seems right. I will close this PR and make an issue for v2 commands first.

yea, let's walk through builtin DSv2 statements first


chore: merge 1.5.2 to main

Description
Merges the v1.5.2 release changes into main

Github has put warning on console.log lines in diff view, you might want to review them.

Github has put warning on console.log lines in diff view, you might want to review them.
@kjonesffdc
I know about them I need to decide whether to add an exception or do it a different way
]


Insert  experimental design for 2016 Sorghum trials at MAC into TERRAREF BETYdb

Detailed Description
The Experimental design for Sorghum trials is defined in this google spreadsheet
Next steps

Define intended plot layouts and plant locations on the gantry reference system (meters offset from [0,0]. We need:

[x] plot and alley dimensions
[x] row spacing
[x] locations of the corner of the first plot and first plant in that plot

we can start with the experimental design (as opposed to what we can later observe as the actual locations) and update the layout as necessary
if we have information that may not be accurate that is okay - we can update it later. Best if we have an estimate of the scale of uncertainty (e.g. within X cm)? For plot boundaries a few cm should be sufficiently precise.
we can compare and update the information with observed positions of the plants and outputs of the precision planter

later we can convert the plot layouts into geospatial data. This requires lat/lon coordinates for [0,0]
Import the genotypes information into the TERRAREF/BETYdb cultivars table

Genotype information is stored in the "Source", "Common Name", and "Pedigree fields" of the 'Treatments'
[x] Identify the unique identifier / 'primary key' for these. Pedigree information can be later imported to BMS, and the primary key can link from BETYdb to BMS.

Import traits from Treatments table (Which traits are useful?)

BETYdb isn't currently set up to ingest categorical traits, this feature is requested in PecanProject/bety#409 and can be implemented if there is need
BMS does store categorical traits.
Which categorical variables can be converted to numeric values with date and location of observation?

for example, It would be preferable to store these as numeric values if possible. For example, instead of 'maturity group' we could store the phenological observations as boolean (0/1) This could be combined with date of planting or emergence to compute the number of days to any particular phenological stage and then the cutoffs could be defined (and altered if desired).

Where, when, and under what conditions was height measured?

import information from Experiments worksheet on google drive into experiments table in BETYdb

[x] Create 'experiments' table (PecanProject/bety#410)

@dlebauer - please touch base with Scott about this.

remaining issues:

[ ] link cultivars to trait records
[ ] associate plots with experiments

Experiments
Season 1:
relationships are first two columns of https://docs.google.com/spreadsheets/d/1iHmSyFeCO7np_lxYyvdiXQQcAwBJpIKpeAOyMXFd-OY/edit#gid=901711246
for each unique expt name add all of the associated sites where name like 'UA MAC Season 1 Plot [col B]%
For For season 2:

add a experiment named "Stay-green RILs F10"
add all sites where name like 'Season 2%'

@dlebauer To be clear, should all of the season 1 experiments should have the same start and end dates?  Looking at all of the managements in https://terraref.ncsa.illinois.edu/bety/managements, the latest harvest date is 2016-07-14.  There are planting dates 2016-04-19, 2016-04-20, and 2016-08-03.  I assume the latter is for season 2.  There are no planting dates in May.  So should the duration be "2016-04-19 to 2016-07-14" for all season 1 experiments?

Yes.  duration is "2016-04-19 to 2016-07-14" for all season 1 experiments
On Fri, Jan 13, 2017 at 11:48 AM Scott Rohde<EMAIL_ADDRESS>wrote:

@dlebauer https://github.com/dlebauer To be clear, should all of the
season 1 experiments should have the same start and end dates? Looking at
all of the managements in
https://terraref.ncsa.illinois.edu/bety/managements, the latest harvest
date is 2016-07-14. There are planting dates 2016-04-19, 2016-04-20, and
2016-08-03. I assume the latter is for season 2. There are no planting
dates in May. So should the duration be "2016-04-19 to 2016-07-14" for all
season 1 experiments?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/terraref/computing-pipeline/issues/97#issuecomment-272501300,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcX5xyxQmKu4VFquoEbfIeLG8pE92KBks5rR7jvgaJpZM4IXn6E
.

@dlebauer I did season 1.  I used insert statements of the form
INSERT INTO experiments_sites (experiment_id, site_id) SELECT e.id, s.id FROM experiments e, sites s WHERE e.name = 'BAP' AND sitename ~ '^MAC Field Scanner Season 1 Field Plot 705( [WE])?$';

I have two questions about season 2:
Site names for season 2 come in two forms: they either match the regular expression
'^MAC Field Scanner Season 2 Range \d+ Pass \d+$'

or match
'^MAC Field Scanner Field Plot \d+ Season 2$'

Should I associate the "Season 2 Stay-green RILs F10" experiment with any site that contains the string 'Season 2'?
Secondly, there is no management corresponding to the season 2 harvest, so I don't know what the precise end date should be.

Yes associate with all season 2 sites
End date is Dec 2 2016
On Fri, Jan 13, 2017 at 4:05 PM Scott Rohde<EMAIL_ADDRESS>wrote:

@dlebauer https://github.com/dlebauer I did season 1. I used insert
statements of the form
INSERT INTO experiments_sites (experiment_id, site_id) SELECT e.id, s.id FROM experiments e, sites s WHERE e.name = 'BAP' AND sitename ~ '^MAC Field Scanner Season 1 Field Plot 705( [WE])?$';
I have two questions about season 2:
Site names for season 2 come in two forms: they either match the regular
expression
'^MAC Field Scanner Season 2 Range \d+ Pass \d+$'
or match
'^MAC Field Scanner Field Plot \d+ Season 2$'
Should I associate the "Season 2 Stay-green RILs F10" experiment with any
site that contains the string 'Season 2'?
Secondly, there is no management corresponding to the season 2 harvest, so
I don't know what the precise end date should be.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/terraref/computing-pipeline/issues/97#issuecomment-272560637,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcX5yI5J7_xOO0Gi1ahFwtQf8MpJrEvks5rR_UcgaJpZM4IXn6E
.

@dlebauer OK, season 2 is now done.  But it occurred to me that maybe id numbers for the experiments and experiments_sites are supposed to be in the 6 billion range.  If so, I'm not sure what mechanism @robkooper uses to make sure each machine starts autonumbering at the correct place.

@dlebauer I re-did the experiments and experiments_sites inserts with the correct id range restriction in effect.  Did a hotfix release to set the starting id numbers correctly.

@dlebauer Can this be closed?


PSR-15 support

Following #150, PSR-15 has now been accepted.
I understood that v4 will support PSR-15.
Is there a roadmap / timeline or a way to contribute PSR-15 support?

Here's a quick class I use myself to use this route package as a middleware in a PSR-15 stack:
<?php

namespace App\Infrastructure\Middleware;

use League\Route\RouteCollection;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;
use Psr\Http\Server\MiddlewareInterface;
use Psr\Http\Server\RequestHandlerInterface;
use Zend\Diactoros\Response;

class LeagueRouterMiddleware implements MiddlewareInterface
{
    /**
     * @var RouteCollection
     */
    private $router;

    /**
     * @param RouteCollection $router
     */
    public function __construct(RouteCollection $router)
    {
        $this->router = $router;
    }

    /**
     * @param ServerRequestInterface $request
     *
     * @return ResponseInterface
     */
    public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
    {
        return $this->router->dispatch($request, new Response());
    }
}

Have fun!

It's not about using the router as middleware, its about adding support for PSR-15 to \League\Route\Middleware\StackAwareTrait::middleware(). Though, the other way around might also work.
@shirshir what do you think?

@willemstuursma Right! Thanks for pointing that out for me 🙈
Yeah allowing PSR-15 middleware shouldn't be too hard. Maybe even without a new major version by allowing old and new style (or with a different method name).

Most of the next major version is complete, I'm hoping to get PSR-15 added and a release drafted over easter weekend

@philipobenito Did you get anywhere with the PSR-15 integration?

It's coming, I've just been slowed down a little, life happens lol, hopefully next couple of weeks though

Any news on this?

Been on holiday but there will be a major version release this week

yes https://github.com/thephpleague/route/pull/178 just writing docs, will be released in the next couple of days


Rosetta notebook testing updates

If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Closes #203. Closes #180. Closes #181. Closes #166.
Update the rosetta notebooks to reflect the toffy pipeline ordering, which indicates image extraction will have been completed before compensating the images.
How did you implement your changes

Create a new copy_image_files() in rosetta.py

keeps the existing directory structure from before, but without manual user copying

implement in notebooks 4a and rosetta fine-tuning, allowing for removal of bin extraction code

also updated 4a to run rosetta on list of runs instead of just one, similar to fine-tuning

update the documentation
change the notebook default channel to be only Noodle
make sure all outputted images (i.e. normalized, rosetta compensated, and stitched) are float32 dtype

Remaining issues
(1) I think it would be a good idea to create a normalize_test_images() helper function in rosetta.py for the code below, since it's used in both notebooks.
for fov in fovs:
    fov_dir = os.path.join(img_out_dir, fov)
    sub_dir = os.path.join(fov_dir, 'normalized')
    os.makedirs(sub_dir)
    chans = list_files(fov_dir)
    for chan in chans:
        img = io.imread(os.path.join(fov_dir, chan))
        img = (img / 100).astype('float32')
        io.imsave(os.path.join(sub_dir, chan), img, check_contrast=False)

This is the code that originally caused the float64 issue that affected the stitched images, so in rosetta_test.py we could verify the image data type moving forward as well.

Yes, move to helper file and update normalized to rescaled in rosetta notebook and normalization notebook.

(2) For the fine-tuning notebook I added code to copy the matrix file from toffy to the rosetta testing directory so the user could tweak it as needed. However, I have the 4a notebook just using the file path, but not actually copying it to the new folder. Is that something we would want them to have in the rosetta_testing directory for reference? I imagine if they want to use their own matrix file they can put it in the folder and specify that path.

Combining the notebooks solves this issue and requires file copy over.

(3) The documentation in the fine-tuning notebook gives example multipliers of 0.5, 1, and 2, but the default in the notebook is set to multipliers = [0.25, 1, 4]. Should these match?

Use 0.5, 1, and 2 as defaults.

I'll take a look at the notebooks then address your questions above

It looks like with the new streamlined version, there is a lot of duplicate code between the fine-tuning notebook and the default one.
What do you think about adding in the finetuning cells to the middle of the rosetta notebook, with the option to run them if the user wants?

It looks like with the new streamlined version, there is a lot of duplicate code between the fine-tuning notebook and the default one.
What do you think about adding in the finetuning cells to the middle of the rosetta notebook, with the option to run them if the user wants?

I think that's a good idea. I can extract the main code chunk from the fine-tuning notebook into a helper function and just supplement the documentation with info about the new matrix files/stitched dirs that get created in a matrix testing subdir.
Would you want this part to become a step 2b essentially? I think they should probably run the compensation with the default matrix (or one they provide) first and then decide if they want to tweak it by looking at the images produced.

I think we can kill two birds with one stone. Instead of the first output being a stitched image with only a single rosetta coefficient, the first output can be a stitched image with the range of outputs. If they like the middle one, no need to change anything, they can process the whole cohort. If they want to change things, here's how to do it, etc.

@HPiyadasa

I left on a comment where I had a different opinion from Ben/Hadeesha, mostly in agreement with all of their changes


PROPOSAL: alternative syntax and separate name space for Definitions

Proposal: Alternative Syntax for Definitions
Definitions are currently marked using the :: syntax. This document proposes an alternative syntax. The purpose is to separate the name space of definitions from that of regular fields.
Background
CUE currently uses :: to mark a field as a definition. The field name is a regular identifier, meaning that definitions and regular fields occupy the same name space. This can pose a problem in mapping certain languages to CUE, where similar constructs live in a separate namespace.
The most notable case causing issues is JSON Schema. JSON Schema allows a definitions (or $defs) section to introduce definitions. Logically, such definitions would have to be mapped at the same level as the top-level fields in CUE. This does not pose a problem along as a convention is followed where fields are camel case and definitions are upper case. Unfortunately, this is often not the case.
A similar issue exists, theoretically, for OpenAPI, Protocol Buffers, and Go. It is just that conventions are more strictly followed for these languages and that this issue is not a problem in practice.
Several secondary issues, like simplification of export rules, are addressed by this proposal.
Overview
In this document we propose lexically distinguishing identifiers for definitions from regular fields by requiring a # prefix that is part of the identifier of such definition, thereby implicitly creating a separate namespace for definitions. This would replace the :: notation.
Before:
Definition :: {
    foo: string
    bar: Map

    Map: [string]: int
}

a: Definition
b: a.Map
c: Definition.Map

After:
#Definition: {
    foo: string
    bar: #Map

    #Map: [string]: int
}

a: #Definition
b: a.#Map
c: #Definition.#Map

Regular fields can still have a name starting with a # when the name is
enclosed in quotes "#foo": value. The proposed mechanism is thus analogous to
the now legacy, but still supported construct of hidden fields (_foo).
In fact, with this proposal the reintroduction of hidden fields becomes an option.
Details
Use of # in regular fields
Regular fields that need to start with a # can use aliases:
// These are NOT definitions, but regular fields. The X= is an alias to that it can be referred to.
X="#Regular": {
    foo: string
    bar: Y

    Y="#Map": [string]: int
}

a: X
b: X.foo
c: X["#Map"]
c: X."#Map" // alternative as per Query Proposal
// Note X.Y is not possible, as Y is not visible in this scope.

Export rules
The spec currently defines exporting rules for fields. These are not implemented. Part of the difficulty is exactly that definitions and regular fields occupy the same namespace: the spec has different rules for the two cases, but since they share the same namespace, it is not always obvious which rules should apply where.
With the two distinct namespaces, the following exporting rules would be feasible:

All identifiers of regular fields (those not starting with a #) are exported.
A definition identifier is exported if the first character following #  is a Unicode uppercase letter (Unicode class "Lu").
Any other definition is not visible outside the package and resides  in a separate namespace than namesake identifiers of other packages.

package mypackage

foo: string  // visible outside mypackage

#Foo: {      // visible outside mypackage
    a: 1     // visible outside mypackage
    B: 2     // visible outside mypackage

    #C: {    // visible outside mypackage
        d: 4 // visible outside mypackage
    }
    #e: foo  // not visible outside mypackage
}

This still relies on casing, which may still not generally work when CUE is automatically translated from other languages. Given the conventions of existing languages, it generally seems to give a desirable outcome, however. A more aggressive exporting policy, for instance exporting all identifiers starting with non-lowercase letters or even exporting any definition not
starting with #_, may be in order.
Interaction with hidden fields
The notation of the proposed change is analogous to that of hidden fields (no longer part of the spec, but still implemented in search of a good guideline for alternatives). The implementation of hidden fields is somewhat complicated. It is also a syntactically very different construct for something that is almost identical to definitions. It was therefore decided that we need
to phase out hidden fields.
With the current proposal, hidden fields become merely a slight variant of definitions: with #Foo: { … } the struct will be closed whereas for _Foo: {…} it won’t. In both cases the field will not be part of data output.
API
The current CUE API already distinguishes between regular fields and definitions. With proposed changes, bringing hidden fields in line with definitions, the same API could now be used to look up hidden fields. Other than that, the lookup API would not have to change.
The AST API could be simplified by removing the token type. There would be a long transition period, however, to support the old representation.
Discoverability
With the proposed change, it becomes easier to explain all different field types within a single table:

foo: x: regular field
$foo: x: also a regular field, often interpreted as some meta field by the user. The $ has no meaning to CUE itself.
#foo: x: Definition: not part of the output when converted to data. Structs are implicitly closed. Can be used to define a complete definition of a type.
_foo: x: Hidden field: like definitions, but structs are not implicitly closed. Can be used to define partial values that are not complete types. (TBD)
"foo": x: using double quotes any valid JSON string can be a field name for a regular field, including including "#foo" and "_foo".

All of these are just identifiers. Today, a._foo and a.$foo are valid references. The advantage of this syntax is that the _ and $ signal to the users what kind of value is referenced. With #-style identifiers this benefit is extended to definitions as well. For orthogonality, a."foo" should be allowed as a valid reference (see The Query Proposal).
Transition
Firstly, the proposal justifies the reintroduction of hidden fields. So any pain caused by the introduction of #-style definitions could be offset by no longer needing to transition off of hidden fields.
A transition phase could allow both the ::-style and #-style identifiers to coexist and mean the same thing. A transition period could consist of the following steps:

Compiler automatically rewrites ::-style definitions to # form, including all references.
cue fmt to rewrite old style definitions to new (where possible).
Parser stops supporting old style in regular mode.
Parser removes support for ::.
AST removes support for token.ISA.

In order to move definitions to their own namespace early on, it is important for parsed CUE to be rewritten to #-style identifiers before each compile. This means that representations of references will exhibit the #-style identifiers even for code that uses the then legacy double colons.
In the strictest implementation, definitions may only have #-style identifiers and cannot have free-form strings like regular fields can. This introduces the following incompatibilities:

Definitions of the form "\(expr)" :: value or [expr] :: value are no longer possible.
The -l command line flag would no longer accept fields of the form "\(foo)"::.

A workaround for the first limitation is to move generated definition inside a map of a static definition (everything can be solved with another level of indirection). For instance:
foo: "y"
x: {
    "\(foo)" :: value
    "\(bar)" :: value
}
a: x.y

can be rewritten as
foo: "y"
x: {
    #m: "\(foo)": value
    #m: "\(bar)": value
}
x.#m.y

The flag issue can be solved by allowing some annotation to indicate a field is a definition specific to this flag (perhaps even supporting the then legacy ::). See also extensions.
Extensions
The transition section discusses several limitations imposed by the proposal. If need be, the language could be extended to allow for "dynamic" definitions, for instance of the form #(expr): value, where expr needs to evaluate to a valid identifier. See The Query Proposal for more details.
Discussion
Comparison
Precedence
The use of #foo for definition has some analogy in JSON Schema, where the same notation is allowed for anchors to refer to schema in the $ref field.
Use of double colon
The use of double colon was derived from Haskell. It also has parallels with Jsonnet, where it is used to mean almost the same thing. The "almost" can also lead to confusion here. Note that Jsonnet also has a :::. This proposal will remove any pressure for CUE to follow suit.
Alternatives
An alternative way to deal with separate namespaces, keeping :: is to have another selector operator specifically for definitions. For instance,
Foo :: {
    Bar :: {}
}

a: Foo.:Bar // or, for instance, Foo->Bar

As CUE is lexically scoped, there is no ambiguity which of the two namespaces is meant and there is no need for any special marker using the first reference. The special operator is only needed for the selector, Bar.
In other words, introducing separate namespaces without distinguishing identifiers lexically introduces a similar amount of clutter, but leads to less clarity. The users will have to learn a new symbol (.:) and will be more confronted between the difference between a reference and selector. Distinguishing fields from definitions lexically results in more symmetry between the two (#Foo.#Bar) which in turn seems to lead to a more intuitive reading.
Feedback wanted

Does this look like a reasonable change? Like/ not like?
Anybody relying on "dynamic" definitions (e.g. "\(foo)" :: bar or [foo] :: bar)?
Has anybody else ran into issues from having regular fields and definitions be in the same namespace?

In general I approve of this. I've been working with data where I wanted composition on hidden fields and the closedness of definitions was a hassle to deal with, so I am happy to see them stay in the language
As for (regular) hidden fields, it might be a good idea to choose another sigil. A lot of C language families allow _ to start identifiers and I can see it being confusing to someone just picking up the language that the fields they added are not being emitted.
is used as a comment marker for a lot of languages (notably yaml). It might be a good idea to explicitly detect if people use one for more friendlier errors

I generally like this idea as well.
One thing I like about Golang is the ability to see visibility of constructs by just looking at the first character. This would be a good measure (the ease with determining scope/visibility, open/closed) for the choice of denotation or syntax. I don't have any strong opinions on the exact format, just the ease with which I can comprehend code.

I like the '#' character for referring to definitions. It looks odd, though, using the same character for establishing (defining?) these definitions. Have you considered using a separate token such as def for the latter case? That would look nice for top-level definitions, but it doesn't look as nice when they're nested within other structs (as in your Transitions example's "m" definition).

@verdverm Yes the proposed approach was, perhaps not surprisingly, inspired by exactly this property of Go. It is harder to use the casing approach of CUE, just because it is not always feasible to have the strict casing guidelines as in Go. Hence the choice for something like #.
@seh: yes I definitely considered using a separate token. However, this will add some significant complexity to the language. Syntactically, one would need a separate production for the def ident construct, as well as a new production and token for the specialized selector (.#). With the current proposal, these are just identifiers, requiring no addition and actually allowing for a removal of a token and production. This doesn't sound like much, but it actually is quite impactful.
Whatever the choice is, though, given the nested nature of CUE, the rule would have to be the same for top-level as nested fields. Personally I don't think having nested def foo: x fields is all that bad.
Anyway, you hit on exactly the biggest issue of this proposal: the ugliness of #foo: value. So
cons (compared to def foo):

#foo: bar is ugly

pros (compared to def foo):

#foo: bar and its counterpart a: #foo are intuitively clear when it comes to resolution, at least, when reading (analogous to $foo: bar and a: $foo and _foo: bar and a: _foo). One does not separately have to learn a new construct like .#.
the language change is significantly simpler then introducing a pair of constructs for def foo: bar and a: #foo.

So the question is whether people think these benefits weigh up to the ugliness.
I think a big mitigating factor is that #foo is used for anchors in other languages. For instance, in JSON Schema, when using anchors, a definition ID may be #address which is then referred to as #address (# is required for anchors).

Feedback wanted
* Does this look like a reasonable change? Like/ not like?

This sounds reasonable :+1:

* Anybody relying on "dynamic" definitions (e.g. `"\(foo)" :: bar` or `[foo] :: bar`)?

No (I didn't realize this was supported)

* Has anybody else ran into issues from having regular fields and definitions be in the same namespace?

Yes. It's usually something I can work around though.
The best benefit I can think of is the #MyDefinition syntax would probably make it immediate what something is. Right now, unless you're looking at the the declaration of the definition, it isn't immediate clear if something is a definition or a field so for that reason alone I really like this.

One thought on how to make the syntax a little nicer and unified:
How about # as the syntax for decoration as well as retrieval?  e.g.
Foo: {
  #MyDef: string
  MyField: number
}

Foo#MyDef // => string
Foo.MyField // => number

@rudolph9: interesting thought. This would require extra syntax in the language, unfortunately. The Foo.#Bar notation is more orthogonal and requires no additional syntax rules. In fact, this proposal as is simplifies the spec!, albeit a tiny bit.
But it is definitely a neat thought. Comments from others welcome in this regard.

i like of this proposal
but # is comment marker for  yaml
e.g.
c: #Definition.#Map
difficult understand

I like the proposal as is, especially if it is simpler in both code and effort for the maintainers.
Definitions definitely need to stand out more, the : vs. :: has been a source of pain. It's also nice that the variations are all very close syntactically. I like keeping hidden fields, something that enables module and library authors to hide implementation will be important.
I am ok with # as a define because of #define in C/C++ , and that it will stand out. Nesting of def:  seems more complicated and I worry less about the comment confusion because there are many different comment styles across languages.

Done

This issue has been migrated to https://github.com/cue-lang/cue/issues/307.
For more details about CUE's migration to a new home, please see https://github.com/cue-lang/cue/issues/1078.


[Fix #9448] Fix an error for Style/SoleNestedConditional

Fixes #9448.
This PR fixes an error for Style/SoleNestedConditional when using nested unless modifier single as in single expression.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

This PR fixes an error for Style/SoleNestedConditional when using nested unless modifier single conditional.

single in which sense? As in single-line or as in single expression?

This PR fixes an error for Style/SoleNestedConditional when using nested unless modifier single conditional.

single in which sense? As in single-line or as in single expression?

Thanks for the suggestion. I updated this PR.

Thanks for the suggestion. I updated this PR.


Round function from doctrine-extensions not registered

Summary
The oro/doctrine-extensions package provides a round function.  However the OroEntityBundle does not register it and it can't be used.
Steps to reproduce
Try to call "ROUND()" in a DQL query (such as that generated from a datagrid) results in an error indicasting  that "ROUND()" is an unknown function (Syntax error).
Actual Result
Syntax error.
Expected Result
Query is executed and value is rounded as expected.
Details about your environment

OroPlatform version: 4.1.13
PHP version: N/A
Database (MySQL, PostgreSQL) version: MySQL

Additional information
I can see at https://github.com/oroinc/platform/blob/master/src/Oro/Bundle/EntityBundle/Resources/config/oro/app.yml#L24 a lot of functions provided by this bundle are registered, but round is not.  I don't know if there's an issue with the round function specifically, or whether it not being registered is an oversight.
I have registered this function from an app.yml in my own bundle and used it.  It does seem to work, so I dopn't understand why platform doesn't register it for you?

Hi, @jimohalloran
Thank you for your response.
Doctrine extensions is a separate project that provides a set of useful functions that allow transparent usage of MySQL or PostgreSQL in your application. At the moment platform has no need in the  ROUND  DQL function, so it was not registered. Registering a new DQL function with app.yml when needed is expected behavior.

Hi @x86demon ,
Thanks for that info.  It seemed odd that most of the functions from that package were registered, but not ROUND().  If this is expected, I assume there are no issues with adding an app.yml into one of my own bundles and registering it?  Having done so already, it seems to work fine and I can't see that it could/would break anything.


pyup config

Description
Disable pyup checks for test-requirements

Codecov Report

Merging #64 into dev will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               dev       #64   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            1         1
  Lines          167       165    -2
=========================================
- Hits           167       165    -2

Impacted Files
Coverage Δ

named_enum/__init__.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d821c19...4e03510. Read the comment docs.


Service-Stop not handled gracefully

[This is an outsource of an additional - a bit offtopic - discussion in #15.]
A small summary:
I had noticed, that not everything is synced properly. On every insert I had a "small" number of missing documents between mongoDB and ES, which is not synced. I observed differences between 70 and 900 documents.
I could saw which documents where missing. When inserting new ones, the old ones - which were missing before - were synced then. That brought me to the idea, that maybe the last collection of new documents, which is collected for the next bulk-insert, is perhaps not transferred/committed properly to ES.
The solution delivered by @sliwinski-milosz was to activate autoCommitInterval.
Due to the changed config-file I had to restart the connector, but then, the documents which weren't committed at this point, where lost. I had to do a resync.
In @sliwinski-milosz opinion, this wasn't necessary when I would have gracefully stopped the connector. ("If you killed it, then of course you have to resync.")
The problem is solved, everything is as desired now, my documents are where they need to be. So there is no real issue anymore.
Only one (a bit trivial) question, which I want to mention when I already open a new issue: Was it really necessary to do the (work-intensive) resync? Or in other words: Worked the logic, which commits docs from bulk while stopping the connector, properly?
I installed the connector as a service, so I just restarted this service after making the changes in the config. So, firstly, I need to now, if this is a "gracefully stop"? Or how does he work, if not?

How did you do the restart? Was it: sudo service mongo-connector restart?
If yes.. resync should not be necessary. In that case, either there is a bug in stopping elastic2_doc_manager, or there is a bug in restarting service using above commend, which could not stop the service gracefully.
If you have stopped service other way, then that could not be the proper way :)

Yes, I did it with sudo service mongo-connector restart.
So, it really seems to be a bug.
I took a look into the service-script. Can't find anything that looks like an intelligent logic that gives the doc-manager a chance to commit docs from the bulk.
You mean, that this doc-manager-function needs to be called when stopping, right?

nope... I think you are right. Mongo-connector catches only KeyboardInterrupt which catches Ctrl+C (and maybe SIGINT signal if they are the same).
In service-script it calls: killproc which sends SIGTERM signal, which is not handled. So it looks like mongo-connector needs to be updated to catch all signals (bug in mongo-connector).
I will take a closer look on that later and will send and update.

Okay, got it.
But changing the killproc param from -9 to -2 wouldn't be enough?! SIGINT needs to be handled from the connector, too.
I think with something like that needs to be in the connector.py:
import signal

# the other code

def Exit_gracefully(signal, frame):
    dm.stop() #letting doc-manager commit remaining docs
    sys.exit(0)

if __name__ == '__main__':
    signal.signal(signal.SIGINT, Exit_gracefully)
    main()

(Source)
I'm not that familar with python, but I could test it on my system tomorrow and put this small piece of code into a PR. But I can't write a corresponding test for this, there I need definitily help respectively somebody has to do this.

The shell sends SIGINT to the foreground process on Ctrl+C so they are effectively the same; sending SIGINT to a Python process will cause KeyboardInterrupt to be raised.
It is a bug that mongo-connector does not gracefully handle SIGTERM. This reminded me that there's already an issue for catching SIGTERM but https://github.com/mongodb-labs/mongo-connector/issues/461. Thanks for pointing this out, I'll make the signal changes to master this week.

We should probably also increase the time that killproc waits until sending SIGKILL:
 -t<sec>
	      The  number <sec> specifies the seconds to wait between the sent signal SIGTERM and
	      the subsequentially signal SIGKILL if the first SIGTERM does not	show  any  result
	      within the first few milli seconds.  This defaults to 5 seconds.

http://www.unix.com/man-page/suse/8/killproc/

Thanks @BlueRayONE and @sliwinski-milosz for your help debugging this issue.


Amend user-related forms

Trello: https://trello.com/c/aQiPdsiP
New links near top of home page

Create new user form

Change existing user form

Supersedes #1242.

We still want to test this on integration to make sure the integration with Zendesk works as expected. However, I'm going to mark this as "Ready for review" so I can get feedback on it as soon as possible.

Based on what I know and @floehopper taking me on a tour of the code, this looks good to me. We're going to work together to make sure this works with Zendesk in integration, so I'll hold off approving until then.

@floehopper  & I have just deployed the changes in this PR to integration temporarily and submitted each of the two new forms to create two Zendesk tickets:

https://govuk.zendesk.com/agent/tickets/5603775
https://govuk.zendesk.com/agent/tickets/5603781

Our team is having a look to see if these tickets match what is expected, but otherwise we think this is ready for deployment.


Able to access files in bot's computer

Users of the robot are able to access and make the robot send files anywhere on the bot's computer by saying @BOT_NAME 说@fil@[FILE_PATH] and replacing [FILE_PATH] with the path of the file.

If the file does not exist, nothing will happen.

@BOT_NAME 说@img@[IMG_PATH] also works if the file is an image.

Fix by regex?

Found a fix! send_raw_msg() function will only send out the raw message.


Ensure proper use of include files under different project config scenarios

Originally filed here: https://github.com/lfe-rebar3/compile/issues/28

a project with an lfe include and no additional settings in rebar.confg
a project with no include but which uses the include from a dep and has no additional settings in rebar.confg
both of those first two options, but with {lfe_first_files, ["src/.."]}. set in the rebar.confg
both of those first two options, but with {lfe_opts, [{i,"..."}]}. set in the rebar.confg
both of those first two options, but with lfe_first_files and lfe_opts set in the rebar.confg

I think I've started running into this a fair amount, especially in release builds. We may need to bump this up to the current milestone ...


generator-langium: extended yeoman generator extension to offer parsing, linking, and validation test stubs (#1282)

providing additional automated tests for the cases of 'stubsOnly', 'vitest', and 'jest'

@sailingKieler
Running npm run test currently fails for me with:
 FAIL  packages/generator-langium/test/yeoman-generator.test.ts > Check yeoman generator works > 3 Should produce files for Vitest
Error: EACCES: permission denied, mkdir '/__yo-gen-testing'

Could it be that there are too many ..?
Dir name in my case is: __dirname: /workspace/langium/packages/generator-langium/
Test tries to write to: Generating into directory: /__yo-gen-testing

Hi Dennis,

@sailingKieler Running npm run test currently fails for me with:
 FAIL  packages/generator-langium/test/yeoman-generator.test.ts > Check yeoman generator works > 3 Should produce files for Vitest
Error: EACCES: permission denied, mkdir '/__yo-gen-testing'

Could it be that there are too many ..?
Dir name in my case is: __dirname: /workspace/langium/packages/generator-langium/ Test tries to write to: Generating into directory: /__yo-gen-testing

The test is supposed to write __yo-gen-testing next to langium, i.e. to /workspace/__yo-gen-testing in order to not see the langium root node_modules. Did you test it on Gitpod?

@sailingKieler
Yes, I tested it with GitPod. I think the tests should not create files outside the workspace in any case. However it seems that there is an additional ..  in '../../../../__yo-gen-testing'  that can me omitted. Can you confirm?

@dhuebner you're right, and I guess that's due to the introduction of https://github.com/eclipse-langium/langium/blob/02b3c92388bf6740cf1dc879f22c2b4ae5cd73f6/packages/generator-langium/test/yeoman-generator.test.ts#L13 I had to rebase on. I'll fix that! Thanks a lot for testing! 🙏

@dhuebner feel free to give another try

okay I dropped the options, and the Jest support, apparently I'm the only lonely wolf using it (in a commercial project).
Should it then be a question, at all? Maybe not.

@spoenemann the second problem I will fix with a separate PR. The minor updates of monaco-vscode-api are pretty major and therefore this error surfaces here. There is https://github.com/CodinGame/monaco-vscode-api/issues/291

⬆ Fix is here: https://github.com/eclipse-langium/langium/pull/1317
@sailingKieler consider upgrading to vite v5 / vitest v1 before merging this PR.

Wouldn't that be out of scope for this PR?

Only for the generated output, not the whole project

Only for the generated output, not the whole project

Of course 🤦🏽♂ 😅

consider upgrading to vite v5 / vitest v1 before merging this PR.

Tested upgrading to vitest ~1.0.0 but that requires Node.js v18, see
https://github.com/vitest-dev/vitest/blob/66933c3a590fa5cff35e252a5f107b0f41d9248b/packages/vitest/package.json#L103-L105 and
https://github.com/vitest-dev/vitest/blob/66933c3a590fa5cff35e252a5f107b0f41d9248b/packages/vitest/package.json#L111-L113
The latter causes a conflict during npm i:
npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/@types/node
npm ERR!   dev<EMAIL_ADDRESS>from the root project
npm ERR!   peerOptional @types/node@">= 14" from<EMAIL_ADDRESS>npm ERR!   node_modules/vite
npm ERR!     dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peerOptional<EMAIL_ADDRESS>|| >=20.0.0" from<EMAIL_ADDRESS>npm ERR! node_modules/vitest
npm ERR!   dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Conflicting peer dependency<EMAIL_ADDRESS>npm ERR! node_modules/@types/node
npm ERR!   peerOptional<EMAIL_ADDRESS>|| >=20.0.0" from<EMAIL_ADDRESS>npm ERR!   node_modules/vitest
npm ERR!     dev<EMAIL_ADDRESS>from the root project

Do you wanna switch to v18 as minimally required node.js version?

Do you wanna switch to v18 as minimally required node.js version?

The question is why not. node 18 was the last LTS version. 20 is the current LTS version. 16 is outdated.

Do you wanna switch to v18 as minimally required node.js version?

The question is why not. node 18 was the last LTS version. 20 is the current LTS version. 16 is outdated.

Filed dedicated PR https://github.com/eclipse-langium/langium/pull/1319

@msujew @kaisalmen I upgraded the versions of node.js & vitest in the generated package.json, and fixed some further tiny bugs.
Can you guys have a final look?


Try to create a common event with somebody - acceptance screen is broken - unreadable text

@vitaly-zdanevich: Could you provide a screenshot?


Script Error

Everytime I open the login for the dashboard and log in, I get a script error, and then I can't get past the screen that asks which subscription plan you want. Can't click on Free or any of the other buttons. I've removed the URL in case it contains information relevant to me. If you need the screenshot with the full URL let me know.

Can you please email nate @ mobileminerapp.com with your registered email address if the problem is still occurring?


[WIP] [Persistence] TreeStore Module Tests

Description
Adds tests for the TreeStore module

Summary generated by Reviewpad on 21 Jun 23 23:39 UTC
This pull request includes changes that affect multiple packages in the persistence module.

A new file named transaction.go has been added to the package that defines a Coordinator struct and a Tx struct.
Changes to the existing indexer package include the addition of atomic transaction methods to the TxIndexer interface.
Changes to the treeStore struct and its methods include the addition of a new field and renaming of a method.

In addition, changes have been made to individual files that add new methods, implement unit tests, and remove unused variables.

Issue
Relates to #756
Type of change
Please mark the relevant option(s):

[ ] New feature, functionality or library
[ ] Bug fix
[x] Code health or cleanup
[ ] Major breaking change
[ ] Documentation
[ ] Other

List of changes

Adds a WithTxIndexer option function for configuring the tree module
Adds a test for creating a TreeStoreModule
Adds tests for the Update function
Lays the ground work for further detailed state hash tests

Testing

[ ] make develop_test; if any code changes were made
[ ] make test_e2e on k8s LocalNet; if any code changes were made
[ ] e2e-devnet-test passes tests on DevNet; if any code was changed
[ ] Docker Compose LocalNet; if any major functionality was changed or introduced
[ ] k8s LocalNet; if any infrastructure or configuration changes were made

Required Checklist

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added, or updated, godoc format comments on touched members (see: tip.golang.org/doc/comment)
[ ] I have tested my changes using the available tooling
[ ] I have updated the corresponding CHANGELOG

If Applicable Checklist

[ ] I have updated the corresponding README(s); local and/or global
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added, or updated, mermaid.js diagrams in the corresponding README(s)
[ ] I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

Superseded by #861


Function like macro as builder alternative

First off: Amazing project. We've been looking something like this for a while.
I have a small question and a possible usage enhancement idea. I tried to use some code constants to create the api docs like so:
const SERVER_URL: &str = "https://example.com";

#[derive(OpenApi)]
#[openapi(
    servers((url = SERVER_URL))
    // ...
)]
struct ApiDoc;

This doesn't work due to expected string literal.
I know I could manually use the whole Builder syntax, or just modify the specific parts dynamically with Modify. But having it all in one place in a more readable format would obviously be nicer.
Perhaps, we could use a function like macro in place of the builder pattern like so:
const SERVER_URL: &str = "https://example.com";

fn generate_api() {
        let docs = openapi!(
            servers((url = SERVER_URL)
            // ...
        ));
        // ... do some other things with the docs
}

I'm wondering, would it be a lot of work to make this work?

Hey @Plebshot ,

I'm wondering, would it be a lot of work to make this work?

In general having support for function macro should not be a very large contribution if the same parsing logic can be used as for the derive macro.  Otherwise it is entirely a seprate functionality. In such a case the effort would be a lot bigger. Nonetheless adding support for the derive macro to handle expressions as literal strings would invalidate the reason for the function macro (at least that reason). Extending the OpenApi and other derive macros to handle the case above should be doable with quite small effort and can be added in the future.

const SERVER_URL: &str = "https://example.com";

#[derive(OpenApi)]
#[openapi(
   servers((url = SERVER_URL))
   // ...
)]
struct ApiDoc;


Python API's I2c.writeReg() Does Not Produce Expected Results

When using the Python I2c module, I don't get the results I expect with writeReg().
My /proc/version reads
Linux version 3.10.17-poky-edison+ (sys_dswci@tlsndgbuild004) (gcc version 4.8.2 (GCC) ) #1 SMP PREEMPT Fri Jan 30 14:16:35 CET 2015`

Not only does most of the mraa.I2c module seemingly do nothing, it produces no meaningful error messages either, making it nigh impossible to debug. The writeReg() function returns the integer 5 (?).
#!/usr/bin/python

import mraa

for i in range(0, 8):
    bus = mraa.I2c(i)
    bus.address(0x40)
    bus.writeReg(0xfc, 0xff)

Here's a little program to loop over all of the possible busses and write to the address I want... Perhaps someone here can enlighten me: What am I doing wrong?
FYI, the chip in question is a PCA9685. It's address is 0x40.

And I can see the change in an oscilloscope, so I know that those programs work.

Tried upgrading per instructions in readme which went fine...
Upgrading libmraa0 on root from 0.5.2-r0 to 0.6.1+git0+805d22f0b1-r0...
Downloading http://repo.opkg.net/edison/repo/core2-32/libmraa0_0.6.1+git0+805d22f0b1-r0_core2-32.ipk.
Removing obsolete file /usr/lib/libmraa.so.0.5.2.
Configuring libmraa0.

Proof-
>>> mraa.getVersion()
'v0.6.1'

But still no dice

You don't mention what breakout board you're using but I guess you're needing i2c-1, mraa respects the krenel i2c bus numbering so just ask for I2c(1). I suspect you haven't looked at the debugging page http://iotdk.intel.com/docs/master/mraa/debugging.html which explains how to use journalctl to get debug messages from syslog. If you use printError(5) you'll get the error message which is invalid handle - likely because mraa 0.5.2 does not support anything but the arduino breakout. I suspect on 0.6.1 the error is different at least
What does the following give you?
import mraa
mraa.getVersion()
x = m.I2c(1)
type(x)
x.address(0x40)
mraa.printError(x.writeReg(0xfc, 0xff))

It gives me MRAA: Invalid Handle.
It's a Sparkfun DEV-13042 board

As for syslog... Looks like that code doesn't output anything there. Alls I see is right after boot:
[   14.223182] wl_bss_connect_done succeeded with 58:6d:8f:56:83:a8
[   14.293219] wl_bss_connect_done succeeded with 58:6d:8f:56:83:a8
[   15.772075] ip (330) used greatest stack depth: 5344 bytes left

Sorry for the delay, I've had a quick look - it seems like the kernel i2c driver is refusing to write to 0x40 for some reason. I'm thinking the edison arduino breakout must use the same module or something and the kernel is getting confused. It'll need a further look.

The breakout board has a few places to bridge the chip to change it's address. I'll give that a shot and post results.

Disabled the all call address (0x70), moved the chip to 0x40 and rebooted the Edison with the same result as before. I'll take a look at the debugging document and see what information I can collect.

Very weird - btw you're not looking at syslog use journalctl -f whilst running your calls you should at least get one message from mraa on initialisation.

D'oh!
Ok, the relevant lines from journalctl -f are...
libmraa[382]: Above i2c bus count for 0 <= bus number <= 9
Between 0 and 8 I get libmraa[382]: Invalid i2c bus, moving to default i2c bus except for 6, which appears to run successfully.
After that, no matter what bus number I choose, I get libmraa[382]: i2c: Failed to write when writing to an address.

@pnovotnak, is this still actual?
Here's a working program (though in C) for this expansion board: https://communities.intel.com/thread/63552
It uses I2C bus #1 and address 0x40 and looks like it worked back in April - so should work with any recent mraa version.

@arfoll, I propose to close this one as no response for a while and a reported solution provided.

I am seeing this problem as well. No matter what I do, I get Jan 04 00:26:44 edison libmraa[583]: i2c: Failed to write in the journalctl log, when doing it in either c++ or python.

I'm using the Sparkfun DEV-13042 (PWM block) attached to the Sparkfun DEV-13097 (Edison Arduino breakout)

Please don't use old bugs this bug is more than likely a failure to use bus 1 or 6.
In your case what mraa version and can you share your code? Make sure your arguments to writebyte are the correct way round. Open a new bug with a code snippet if you still have an issue.


Histogram widgets are collapsed when adding another widget

Context
Histogram widgets are collapsing when adding a new widget
Steps to Reproduce

Create a map with a histogram widget
Add another widget

Current Result

Expected result
The histogram widget keeps the same state.

Shipped! 🚀


SN-80 Added gyro PSD plot to LogInspector

@aabogdanov Notice in their plots their units are dB dps/rtHz verses Hz.  What units are most conventional here?

This is the PR for SN-80.


Implement Guard::from_inner

As discussed in #25, implement Guard::from_inner. I'm keeping the refactorings, because they actually let me use Guard::from_inner in the implementation of load.

Thanks. I'll have to do a more thorough review. I'm not sure the changes are equivalent (the manually-drop doesn't happen in one of the if branches now). I'll find the time soon.

If you are thinking of the former call to unprotected from into_inner, it doesn't matter because the unprotected guard was forgotten anyway. Now or skips extracting the inner object and forgetting the container.

I'm referring to the call to T::from_ptr in lock_fallible. But reading the code and docs more thoroughly it seems your version is actually the correct one and the previous code could break an invariant promised to the implementer of the RefCnt trait. That would matter only if someone from outside the crate was implementing that, which I doubt a bit.
Do you want to add a code example to the new method, or should I just go ahead, merge and release it?

Sure I can add an example (tomorrow). I would just reuse the clone example.

Thank you. I'm going to do a release once all the tests run through.

Released as 0.4.5.


It just opens to a blank black window

This is very weird. Sometimes it will load fine and other times it will just be a blank black rectangle.  I've tried re-downloading and putting in a new directory. No luck.  This is on Windows 10. I'd be happy to share log files. thanks

When this happens disconnect your VPN then reconnect. Problem will be solved. Happens here every few days.

When this happens disconnect your VPN then reconnect. Problem will be solved. Happens here every few days.

.. thanks, I'm not connected via vpn. I actually forgot it was installed in just a couple days ago it started working again. When you say VPN, do you mean cable modem if that's what I use?

You could try disconnecting your cable modem for a minute or so then reconnecting. Not sure if your setup uses a static or dynamic IP address. When I disconnect my VPN and reconnect I get a new IP address. It appears CPW is using a service that limits the number of daily calls for prices. After a while you use up that number tied to that IP address. A new IP address lets the price updates begin again. Hope that helps.

You could try disconnecting your cable modem for a minute or so then reconnecting. Not sure if your setup uses a static or dynamic IP address. When I disconnect my VPN and reconnect I get a new IP address. It appears CPW is using a service that limits the number of daily calls for prices. After a while you use up that number tied to that IP address. A new IP address lets the price updates begin again. Hope that helps.

Ahhhh. Makes sense. Inside my home network I use dynamic but my external facing IP is static. Next time it happens I'll connect with the VPN and see what happens. Thank you

Done


hax-types: version for serde-brief

You can not publish a crate with a wildecard as version number.
I published all the crates. You need to move all the tags to this commit when merged.
I published with
cargo release -p hax-frontend-exporter-options -p hax-adt-into -p hax-frontend-exporter -p hax-types -p hax-subcommands -p cargo-hax -p hax-export-json-schemas -p hax-pretty-print-diagnostics -p hax-lib-macros-types -p hax-lib-macros -p hax-lib -p hax-driver --no-tag --execute

Ah, thanks, makes sense, I missed that!
This list of crate is supposed to be somewhere in the repo though. I will investigate why it's not picked up by cargo release.

This list of crate is supposed to be somewhere in the repo though.

I don't think we ever merged a config.


[Bug] Splash radius of Depth charges are incorrect

Describe the bug
The value shown in the shipbuilder is using "splashCubeSize * 30 / 2" but this parameter is irrelevant to the splash radius against submarines.
To Reproduce
Steps to reproduce the behavior:

Go to a ship
Check ASW Depth Charges
See an error

Expected behavior
The parameter should be "depthSplashRadius * 30" instead of the aforementioned value.
Program version. You can find it in the Info tab of the setting window
App Version: 1.4.5
Data version. You can find it in the Info tab of the setting window
Data Version: 0.11.3#2

Apparently, the ASW airstrike section shows the correct parameter while the ship-mounted DCs part is incorrect.

Fixed through a DataConverter fix


There are unwanted page review popups

As described here, sometimes concrete5 shows page review popups, and as @JohntheFish explained on the forum the bug appears when the core of concrete5 breaks. And then we have this message, bug clicking on the "Review" button shows an empty "Compare Versions" modal and the popup can't be removed :

I don't know why the core breaks sometimes but I managed to replicate the bug by adding an exit(); on line 236 of the concrete/src/Workflow/Progress/Progress.php file before the $r = $wf->start($this); to trigger an error.

Yes, it's going to happen if you add an exit – that's because we create a workflow request and then immediately approve the request. If you add an exit the request won't be approved. We'd be happy to fix the cases that cause the core to break and create these popups, if the reason for it could be deduced.

That is a flawed approach. Find every bug that could occur when publishing a page, fix those bugs, and the orphaned workflow will never happen. I agree that finding as many bugs as possible and fixing them is desirable, but you will never find all. Furthermore, anyone developing blocks could inadvertently introduce further bugs that cause an orphaned workflow. The core needs to be robust against such.
It needs a more defensive solution that can cope whatever the circumstance. Such as:

Modify the publication/workflow process to make it atomic, so it either happens or does not, rather than leaving an orphan should it break on the way.
Somewhere in the dashboard, a button for cancelling broken workflows.
The server code behind the popup checking if the workflow is broken and cleaning up where necessary, perhaps with a 1-time red popup to advise the user of the fault cleanup.
Any other ways to achieve this gracefully.

@JohntheFish I understand your point, my goal when opening this issue was to let the core team know that there were these popups showing sometimes, and maybe another concrete5 user would know why and give more informations.
The part where I suggest to use an exit() was just to inform that the core might breaks just before but I agree that it's not very helpful to correct the bug(s) that could occur when publishing a page.
I think your first proposal, to make the publication/workflow process atomic, would be the most suitable to correct this bug.
What do you think @aembler ?

If we could make the entire process atomic, that would be best; I agree.
The challenge lies in that we have a full workflow system that can be modified by different types of workflows – so we'd have to make sure this atomic approach works for the built-in workflow requests and extensions to it. But let's give it a try.


Minor: Improve docstrings on WriterPropertiesBuilder

Which issue does this PR close?
N/A
Rationale for this change
Even though the docs did mention the fact that the limits were a function of the write batch size, I think both @Ted-Jiang and I missed the subtlety initially. See conversation on  https://github.com/apache/arrow-datafusion/pull/4131#discussion_r1016554574
What changes are included in this PR?
Add some additional commentary on write batch size docs as well as backlinks from other limits
Are there any user-facing changes?
Only docstrings

@alamb Thanks 👍


Initial WASM/Cross build factory code

This is able to successfully cross compile Python on WASM, but fails on the make pythoninfo stage of the build, there need to be some fixes to CPython before this will work fully.
I figured that a general cross-compiler factory would probably be useful in future, so I tried to make the code a bit more generic.
Things I still want to do:

[x] Fix CPython so make pythoninfo works https://github.com/python/cpython/pull/91931
[x] Fix CPython so make buildbottest works https://github.com/python/cpython/pull/91930
[x] Allow disabling those for crossbuilds (in case one wants to test just the cross compilation itself maybe?)
[x] consider better directory names instead of build/host? These are based on the names autoconf uses...
Check that uploading results works?

How many tests are skipped? Does the test suite pass?

What is the goal of this work? Do you plan to add a buidbot worker? Which WASM target would it use? A browser? Node? Something else?

Hi Victor!
I am working with Brett and Christian (and several others) to make WebAssembly a supported platform for CPython eventually. Currently we run a nightly build and test job here: https://github.com/ethanhs/python-wasm/runs/6151353583?check_suite_focus=true
Of course, to be supported we need to add a buildbot, so these changes are to set up a nodejs based worker.
The current summary of the tests (just running node python.js -m test) is:
== Tests result: SUCCESS ==

342 tests OK.

92 tests skipped:
    test__xxsubinterpreters test_asyncgen test_asynchat test_asyncio
    test_asyncore test_check_c_globals test_clinic test_cmd_line
    test_concurrent_futures test_contextlib_async test_ctypes
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
    test_docxmlrpc test_embed test_epoll test_faulthandler test_fcntl
    test_file_eintr test_fork1 test_ftplib test_gdb test_grp
    test_httplib test_httpservers test_idle test_imaplib
    test_interpreters test_ioctl test_kqueue test_launcher test_lzma
    test_mmap test_msilib test_multiprocessing_fork
    test_multiprocessing_forkserver test_multiprocessing_main_handling
    test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
    test_pdb test_poll test_poplib test_pty test_pwd test_readline
    test_regrtest test_repl test_select test_selectors test_smtplib
    test_smtpnet test_socket test_socketserver test_spwd test_sqlite3
    test_ssl test_stable_abi_ctypes test_startfile test_subprocess
    test_sys_settrace test_syslog test_tcl test_telnetlib test_tix
    test_tk test_tools test_ttk_guionly test_ttk_textonly test_turtle
    test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
    test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
    test_winreg test_winsound test_wsgiref test_xmlrpc test_xmlrpc_net
    test_xxlimited test_zipfile64 test_zipimport_support test_zoneinfo

1 test run no tests:
    test_dtrace

Total duration: 9 min 24 sec
Tests result: SUCCESS

I believe that Microsoft has offered to donate a buildbot worker (if not I can likely donate one), and we will likely start with a nodejs based worker, since running the test suite in the browser would be difficult.
I also factored out the cross build code because I figured it could come in handy in future for other buildbots.
As I mention in my earlier comment, the WebAssembly buildbot will require a few tweaks to CPython itself. For example, some instances of calling python ... from the Makefile will need to be replaced by calling node python.js ....

At first we are going to use Node.JS with NODERAWFS for testing to ensure that Python compiles and works to wasm32-emscripten. Browser tests may follow in the future. The entire test suite was passing on Node.JS. Last time I tried one test_decimal test case was flaky. The test suite skips tests that don't work on Emscripten, e.g. socket tests or missing extension modules.

Please make sure that test.pythoninfo collects all versions useful to debug, like the Node.js version. The bare minimum would be to log the newly added sys._emscripten_info attribute: https://github.com/python/cpython/pull/91781
Is the ./configure --with-emscripten-target=TARGET value logged somewhere? I guess that it will be logged in the buildbot configure step.

Node.JS with NODERAWFS

Is it possible to log Node.js options somehow in test.pythoninfo? It seems like there are many implementation for the filesystem: NODERAWFS, MEMFS, BrowserFS, IDB, etc. (depending on the target)
Also please write a documentation somewhere explaining how to replicate the buildbot worker configuration, to reproduce the issue locally. For example, I expect that the Node.js version will matter, and how Node.js is run (options).

The Node.JS version is not that useful. The relevant version is emsdk version. Since we are using node from emsdk, the emsdk version encodes all other relevant bits and pieces.

The Node.JS version is not that useful. The relevant version is emsdk version. Since we are using node from emsdk, the emsdk version encodes all other relevant bits and pieces.

So please log it into test.pythoninfo ;-)

Also please write a documentation somewhere explaining how to replicate the buildbot worker configuration, to reproduce the issue locally. For example, I expect that the Node.js version will matter, and how Node.js is run (options).

This is already documented in the Tools/wasm directory. You only need the same emsdk, that's it.

Ping me when this is ready to review @ethanhs @tiran 🎉
Very excited about this ❤

make pythoninfo now works for emscripten-node and wasi targets.

Hey @pablogsal I think this is ready for review, huge thanks to @tiran for making some needed fixes to push this forward :)

This change adds bcannon-wasm worker. Is it owned by Brett Cannon? Do you have a password for this worker?

This change adds bcannon-wasm worker. Is it owned by Brett Cannon? Do you have a password for this worker?

@brettcannon has acquired a buildbot host for WASM.

https://github.com/python/cpython/pull/93844 adds documentation how to set up the SDKs and dependencies locally without a container.

This change adds bcannon-wasm worker. Is it owned by Brett Cannon? Do you have a password for this worker?

No password yet as I haven't asked for one. I've been waiting to be given the go-head from @tiran and @ethanhs that everything is in-place, then I'm going to acquire the appropriate Azure credits to set up a buildbot for this.

No password yet as I haven't asked for one. I've been waiting to be given the go-head from @tiran and @ethanhs that everything is in-place, then I'm going to acquire the appropriate Azure credits to set up a buildbot for this.

You are good to go! I was able to run WASI and Emscripten Node tests with a local buildbot instance inside my EMSDK container. https://github.com/python/cpython/pull/93844 adds documentation how to set up EMSDK, WASI-SDK and other dependencies on an Ubuntu VM for your buildbot.

I'm going to be given a chunk of credits to use every month, so I've been asked to budget what we need. Is 8 GB RAM enough? 16 GB? How many CPUs to keep up with merges? @pablogsal @vstinner any ideas/suggestions?
People can play with https://azure.microsoft.com/en-us/pricing/calculator/ if you're curious or have specific input to provide since I will be getting credits which means we can use whatever Azure services we want, I just need to give them an estimate.

The instance will be testing Emscripten Node, Emscripten Browser (build-only, no tests yet), and WASI for 3.11 and main branch. I would go for at least 4 vCPUs. 8 vCPUs would be better, though. RAM wise, we are effectively testing 32bit programs. The extra memory consumption of NodeJS and wasmtime should not make a bit difference. So 8 GB RAM might be enough, 16 GB would not hurt, though.
IMHO the smallest useful instance is B4ms (4 vCPU, 16 GB RAM) + 64 GiB SSD for persistent data ($150 USD/month or ~$1000 USD/year upfront). This should be good enough until we have more branches with WASM support. I would not object F8sv2 instances (8 vCPU, 16 GB RAM, optimized for computational load). It's twice as expensive as the B4ms instance.

A B4ms instance runs the Emscripten Node tests with make buildbottest TESTOPTS=-j2 in 13:30 and WASI tests in 6 minutes. Good enough :)

I'm going to be given a chunk of credits to use every month, so I've been asked to budget what we need. Is 8 GB RAM enough? 16 GB? How many CPUs to keep up with merges? @pablogsal @vstinner any ideas/suggestions?
People can play with https://azure.microsoft.com/en-us/pricing/calculator/ if you're curious or have specific input to provide since I will be getting credits which means we can use whatever Azure services we want, I just need to give them an estimate.

8GB is enough but if we plan to have a bunch of builders it uses parallel tests (we should because otherwise the worker will be quite behind on tests on busy days) I would recommend 2 or 4 cores if that's possible.

@brettcannon:

No password yet as I haven't asked for one. I've been waiting to be given the go-head from @tiran and @ethanhs that everything is in-place, then I'm going to acquire the appropriate Azure credits to set up a buildbot for this.

I sent you a password in private.
About the hardware, I would say that https://devguide.python.org/buildworker/#required-resources is outdated: a worker should have at least 1 GB. 4 GB should be better. More is good to have, but I don't think that it's required.
In my experience, 50 GB of disk is short if you want to have multiple configs (like regular, LTO, PGO, clang, etc.). 100 GB of disk should be more comfortable. A single build takes around 600 MB of disk (on sub-directory in /home/buildbot/buildarea).
Example with a RHEL8 worker:
$ lsblk
NAME                                            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                                             252:0    0   50G  0 disk
├─vda1                                          252:1    0  500M  0 part /boot
└─vda2                                          252:2    0 49,5G  0 part
  ├─vg_root_python--builder--rhel8.osci.io-root 253:0    0 13,8G  0 lvm  /
  └─vg_root_python--builder--rhel8.osci.io-home 253:1    0 35,7G  0 lvm  /home

$ LANG= df -h / /home/buildbot/
Filesystem                                               Size  Used Avail Use% Mounted on
/dev/mapper/vg_root_python--builder--rhel8.osci.io-root   14G  4.8G  8.2G  37% /
/dev/mapper/vg_root_python--builder--rhel8.osci.io-home   36G   22G   12G  66% /home

On this RHEL8 machine, /tmp is part of /: it's mounted as ext4, not tmpfs.

Thanks for the suggestions, everyone! I will play with the proposed amounts and see what I can get funding for.

I have test instance running on an Azure VM using some free credits that will expire in a couple of weeks. Could you please merge the PR so we can give it a try?
/opt/buildbot/buildbot-worker/bin/pip install --upgrade buildbot-worker
/opt/buildbot/buildbot-worker/bin/buildbot-worker create-worker /opt/buildbot/bcannon-wasm buildbot.python.org bcannon-wasm SECRET_PASSWORD

Merged, thanks.
@tiran: @brettcannon sent you the buildbot worker password?

Yes, Brett sent me the password


The cuda version is too high

cuda 12.1
Can it be reduced to version 11.7?

I think cuda 11.7 is fine.

my using cuda 11.7, is not ok.

in addition, also need torchvision=0.17.1 editor

Can you provide the error message. The enviroment of the code is similar to LAVIS. I think installing torch and torchvision that matches the version of coda 11.7 can be fine.

您能否提供错误消息。代码的环境类似于 LAVIS。我认为安装与 coda 11.7 版本匹配的 torch 和 torchvision 就可以了。

我已经安装11.7版本（自身是11.5，11.5版本也会出现这个错误），出现错误
Building wheel for DCNv4 (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.


Show warning if PR submitter tries to uncheck Allow edits from maintainers

I often get PRs where the user has unchecked Allow edits from maintainers. when creating a PR. This is super annoying as it means I cannot easily do changes to the PR myself. There's no reason for users to do this. Would be nice to show a warning if a user tries to uncheck it that explains why unchecking it would be annoying to the maintainers.

For anyone that wants this, please also send this feature request to GitHub<EMAIL_ADDRESS>

@bfred-it since there is no element to attach a warning message to, could triggering the pop-up from the learn more button work as a solution? (with maybe a modified message)

It needs to be a visible warning. The code should probably be similar to this:
https://github.com/sindresorhus/refined-github/pull/1758#issuecomment-462204170

It needs to be a visible warning. The style and code should probably be similar to this:
#1758 (comment)

Thanks @bfred-it , that's exactly what I needed. I am open to suggestions on the final wording that could be used, but this is what I have so far.


Bug fix prop name

Issue #21

I would try and not get too hung up on the whole generateNextPropertyName thing; if it's easier just don't use 0-9 in your alphabet - it's not really that useful to fiddle with this before we can measure end-to-end minification performance (gzipped result from minifying e.g. Angular with this). IMHO it's more interesting to spend some time on the externs/internal symbols/etc topic (and don't forget your talk!).

LGTM, @mprobst I asked Daria to simply add numerics and '$', '_', assuming it shouldn't be too complicated. Lets merge this and move on.
Daria, if more bugs appear in the generation, feel free to reduce the alphabet if easier that fixing the bug.


SSD Samsung PM981

Is this SSD supported? I want to try hackintosh, but i don't know if exists kext for ssd Samsung PM981

As far as I know SSD Samsung PM81has still some issue with mackintosh in general: the fact is not due to this particular laptop.
For reference you can see: https://www.reddit.com/r/hackintosh/comments/evkljr/samsung_pm981_nvme_hackintosh_reboot_loop/
Some efforts for its support can be found here: https://github.com/acidanthera/NVMeFix/blob/master/Changelog.md

Thanks for your effors @profzei! How can i put this kext in Clover? Copying it in CLOVER > kexts > others?
I have another question, how is the performance in Catalina? Best than Windows?

Hey @sgutirodri don't waste your time trying to install it on pm981
I tried the past 5 days with absolutely no luck. Sometimes I got through the first stage to continue the install from my ssd but every time the ssd raises a KP. It's not worth the hustle.
Tried last night with a transcend usb nvme and installed catalina in less than 30min on my USB, works so far without problems.

@sgutirodri: May be there is a last chance...
I found a "detailed guide" (you need to read all the discussion...) at https://github.com/tylernguyen/x1c6-hackintosh/issues/43
You can give it a try...

Thanks for your information @juicypop and @profzei , I think that I will buy a new ssd.

@sgutirodri I ordered a Kingston a2000 1tb, looked like the best value/gain in my opinion and is known to be working with osx. I will let you know later this evening or tomorrow if it's true or not.
@profzei I read this thread and it seems to be a bit too much work for a ssd that may still throw KP, i ended up in installing on usb and move later to my a2000

Just to let you know, Kingston a2000 1TB works flawlessly :+1:

@sgutirodri I ordered a Kingston a2000 1tb, looked like the best value/gain in my opinion and is known to be working with osx. I will let you know later this evening or tomorrow if it's true or not.
@profzei I read this thread and it seems to be a bit too much work for a ssd that may still throw KP, i ended up in installing on usb and move later to my a2000

So PM981 works perfectly after you get macOS installed. I have stress tested the drive and have dual boot running(Not a single crash due to the drive). No KP is observed. People who observed KP are the ones who couldn't follow the instructions.

@sgutirodri I ordered a Kingston a2000 1tb, looked like the best value/gain in my opinion and is known to be working with osx. I will let you know later this evening or tomorrow if it's true or not.

@profzei I read this thread and it seems to be a bit too much work for a ssd that may still throw KP, i ended up in installing on usb and move later to my a2000

So PM981 works perfectly after you get macOS installed. I have stress tested the drive and have dual boot running(Not a single crash due to the drive). No KP is observed. People who observed KP are the ones who couldn't follow the instructions.

How works, clearly?


fix(validate): allow for optionality among potential standards in a component definition

Description
Component definitions are comprised of 1-> components of 1 -> N Control Implementations of 1 -> N Implemented Requirements
The intent of validate should support a default behavior of assessing all control implementations (containing common control-id's) and producing a result per targeted source of controls.
In doing so - Lula can then support validate/evaluate of specific control-implementations via a specified flag instead of needing to execute against non-applicable standards/controls. This will also be an expected behavior for generation of an assessment-plan where we will need to know which standard/framework to choose.
Core Functionality Updates

Introduces a framework and target Lula oscal prop

framework will be used in the component definition and allows for the combination of multiple standards/control-implementations into a single result
target is a generated prop that is used by Lula to conduct evaluate in a targeted manner vs the filtering and evaluation of all unique "targets"

Adds a -t / --target flag to validate and evaluate
The assessment-result is now generated from a slice of results.
Each unique control-implementation identified will create a unique result

When the framework prop is utilized an additional result will be created for the framework
This allows targeting a single standard/framework OR creating results for all standards/frameworks by default

Constraints

Proposed changes remove the connection between an implemented-requirement/control-implementation and it's subsequent component (which wasn't thoroughly leveraged before. I'd like to consider implications and determine if adding this back in the future is helpful.
This update is not entirely backwards compatible. Existing thresholds will need to be updated after determining what the new execution means for impact (if using multiple standards).

Local Testing

Unit tests and E2E tests were updated to fix removed/modified functions
see src/test/unit/common/oscal/valid-multi-component.yaml as a target for performing various permutations of validate/evaluate and noting the amount of results created

Release Note
TODO: insert release not here for how to adopt these changes in an existing validate/evaluate workflow with an existing threshold.
Related Issue
Closes #327
Closes #499
Closes #500
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Other (security config, docs update, etc)

Checklist before merging

[x] Test, docs, adr added or updated as needed
[x] Contributor Guide Steps followed

A couple other questions/comments as I'm reviewing:

Looks like observations aren't all appended to related-observations when two different components are calling the same control (perhaps we need a test case for this scenario)
This one is more personal preference - but the "Findings" section in the validate output is taking up a lot of space in the terminal, especially just with line breaks, would it be possible to condense those to a single line -> Control ID: %s, Finding UUID: %s, Status: %s? Maybe there's a reason they should be that expansive though so happy to hear the counterargument there.
How does framework get added? It appears to be manual right now, is that something that ideally will happen via lula generate?
When I run lula validate against the valid-multi-component.yaml I get 4 results - I thought there'd be two for just each of the "frameworks" specified, but seems like the catalog source also gets added as a target... can you ELI5?

I noticed the 4 findings but I think it's a good thing.
You'll get one per framework and one per catalog/source.

I noticed the 4 findings but I think it's a good thing.
You'll get one per framework and one per catalog/source.

Would there be a situation where source and framework would be different or like independent? They just seemed to kind of indicate the same thing but different values, basically. But there might be more nuance that the example doesn't really cover

A couple other questions/comments as I'm reviewing:

Looks like observations aren't all appended to related-observations when two different components are calling the same control (perhaps we need a test case for this scenario)

This is a real bug - should be fixed by checking for existing related observations instead of overwriting when generating findings.

This one is more personal preference - but the "Findings" section in the validate output is taking up a lot of space in the terminal, especially just with line breaks, would it be possible to condense those to a single line -> Control ID: %s, Finding UUID: %s, Status: %s? Maybe there's a reason they should be that expansive though so happy to hear the counterargument there.

Agreed - I re-used the table functionality from message and consolidated the output to Control ID / Status. Open to feedback on if we really need the finding UUID (which are really just unique control ID's to my knowledge)

How does framework get added? It appears to be manual right now, is that something that ideally will happen via lula generate?

Manually or as part of generate in another PR: #574


Export DensityState type

Sorry for making all these tiny PRs, is it better for you to have PRs aggregated before or do you prefer to keep features apart also in PRs?
Best,
Gio

Sorry for making all these tiny PRs, is it better for you to have PRs aggregated before or do you prefer to keep features apart also in PRs?
Best, Gio

It is best OSS practice to have many small PRs, so that maintainers can better pick and choose what to accept, and be able to give focussed feedback on any requested changes