CamelCasePlugin Support

Is your feature request related to a problem? Please describe.
Hi! First of all amazing plugin! Great work, we wanted to do the same and now are just using yours :)
Atm CamelCasePlugin does not correctly work. The returned items are still snake_case. I think it would be easily achievable to check if CamelCase Plugin is active and if yes, manually rewrap the result in the transformResult() function.
Normally the engine/driver would do it, but as thus this plugin is only used as query builder, it won't be done automatically.
Thank you and greetings from Germany!
Paul

Hi! First of all amazing plugin! Great work, we wanted to do the same and now are just using yours :)

Danke schön! I'm delighted to know the plugin was useful for you. :)

Atm CamelCasePlugin does not correctly work. The returned items are still snake_case. I think it would be easily achievable to check if CamelCase Plugin is active and if yes, manually rewrap the result in the transformResult() function.
Normally the engine/driver would do it, but as thus this plugin is only used as query builder, it won't be done automatically.

This should be pretty straightforward to implement! I've taken a quick look at Kysely's plugin API, and unfortunately, we can't get a list of plugins from the Kysely object after it's been constructed, though we can add more with .withPlugin().
To me, the simplest workaround would be to pass the list of plugins to the kyselyExtension() function as well, so it can call transformResult() as appropriate.
For example:
const prisma = new PrismaClient().$extends(
  kyselyExtension({
    kysely: new Kysely<DB>({
      dialect: { /* ... */ },
    }),
    /* prisma-extension-kysely will invoke transformResult() from these plugins: */
    plugins: [
      new CamelCasePlugin(),
    ],
  }),
);

Under the hood, these plugins will be added to the prisma.$kysely object using .withPlugin(), and their transformResult() functions will be invoked on all query results.
Does this seem like the best solution for your use case? If so, I should be able to get it coded up, documented and released swiftly!

Would love this feature as well!

This turned out to be a little more complex than I thought it would be at first glance, but it's pretty much ready to release as part of v2.0.0 of the extension! By hooking into Kysely's driver class, I've managed to enable support for all Kysely plugins, not just CamelCasePlugin. Keep an eye out for PR #37 being merged! Feel free to test it out in the meantime too.


[CBR-366] Adding test for partial getters

Description
Adding test for partial getters
Linked issue
https://iohk.myjetbrains.com/youtrack/issue/CBR-366
Type of change

[ ] 🐞 Bug fix (non-breaking change which fixes an issue)
[ ] 🛠 New feature (non-breaking change which adds functionality)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)
[ ] 🏭 Refactoring that does not change existing functionality but does improve things like code readability, structure etc
[x] 🔨 New or improved tests for existing code
[ ] ⛑ git-flow chore (backport, hotfix, etc)

Developer checklist

[x] I have read the style guide document, and my code follows the code style of this project.
[x] If my code deals with exceptions, it follows the guidelines.
[x] I have updated any documentation accordingly, if needed. Documentation changes can be reflected in opening a PR on cardanodocs.com, amending the inline Haddock comments, any relevant README file or one of the document listed in the docs directory.
[x] CHANGELOG entry has been added and is linked to the correct PR on GitHub.

Testing checklist

[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

QA Steps
Unit tests:
stack test --ta "-m GetAccountBalance" cardano-sl-wallet-new:wallet-unit-tests
stack test --ta "-m GetAccountAddresses" cardano-sl-wallet-new:wallet-unit-tests
Integration tests:
nix-build release.nix -A tests.walletIntegration
Screenshots (if available)

yes, in the integration test added the following is taken place:

Genesis wallet and random wallet with 5 accounts are created
Balances as obtained by getAccounts and by getAccountBalances are compared (here zero balances)
Money is transferred from genesis wallet to random wallet's accounts
delayThread is adopted to confirm transactions in the blockchain
Point 2 is repeated, this time for accounts with nonzero balances

Money is transferred from genesis wallet to random wallet's accounts

You mean in wallet-new/integration/AccountSpecs.hs right?

yes
                    { pmtSource =  PaymentSource
                        { psWalletId = walId genesis
                        , psAccountIndex = accIndex fromAcct
                        }
                    , pmtDestinations = pure PaymentDistribution
                        { pdAddress = addrId toAddr
                        , pdAmount = V1 (Core.mkCoin amount)
                        }
                    , pmtGroupingPolicy = Nothing
                    , pmtSpendingPassword = Nothing
                    }
            amounts <- generate $ shuffle [1..5]
            let addrAndAmount = zip (map (\(addr : _) -> addr) $ map accAddresses accs) amounts
            forM_  addrAndAmount $ \(addr, amount) ->
                postTransaction wc (payment amount addr)```

Ok, sounds good. Ideally we'd modify the wallet tests themselves to be able to do something similar, perhaps hook it up to our blockchain infrastructure, but for now, this will do just fine :) Thanks!


Rule to enforce default only import from JSON module

I'm trying to find a way to enforce default only imports from files with a .json extension, I wasn't able to find a rule that fits.
The TC39 proposal for JSON Modules is at stage 3, and Webpack 5 emits a warning when using a non-default import from a JSON file:
// config.json
{
  "path": "/somewhere"
}

// some-file.js
import { path } from './config.json'

// Webpack compiler warning:
// ⚠ Should not import the named export 'path' (imported as 'path') from default-exporting module (only default export is available soon)

From the Webpack 5 release announcement (Oct 10, 2020):

JSON modules now align with the proposal and emit a warning when a non-default export is used. JSON modules no longer have named exports when importing from a strict ECMAScript module.
MIGRATION: Use the default export.

The rule might look something like:
rules:
  no-named-import: [2, { json: true }]
  # or
  only-default-import: [2, { json: true }]

import { path } from './config.json' ❌

import config from './config.json' ✅

Absolutely open to suggestions if there's anything I should try, thanks in advance!

I’d expect the “named” rule to do this.

Thanks — the documentation on the rule doesn't mention anything about restricting it to a specific file type, is that configurable?

What i mean is, zero json files have a named export, so I’d expect the rule to warn on them with no configuration required. Have you tried? (the Airbnb config enables this for you)

Doesn't change anything for me. I have the Airbnb Typescript config enabled, and I just manually configured the rule — tried with named imports in TS and JS files.

There is no actual Airbnb typescript config - the one you’re using is unofficial - but I’m sure that doesn’t change things here.
If the named rule doesn’t cover it, then that’s surprising. Is your TS config set to resolveJsonModule: true?

Right, yes, it's the eslint-config-airbnb-typescript plugin.
Yes, TS config is set to resolve JSON modules. As far as I was aware that was required for TS to import JSON. I realize that it does also generate a static type for the JSON module as well, would that affect how the module is resolved?

Yes, it would. the TS resolver for this plugin uses the type.
If the type of a JSON module doesn't indicate that it only has a default export, then that type is wrong. If it's TS generating that type, I'd file a bug on TS.

Ok, thank you for taking the time to help sort this out!


Adding native_arch to the Architecture enum

Hi,
I added the function native_arch to the Architecture enum, which returns the build target architecture.
(Like the native_object function from my reverus pull request source but for the Architecture enum )
Usage:
You can use the function like followed:
let arch = Architecture::native_arch();

Bye

Sorry, but I don't want to add this. It is an untestable maintenance burden, and not appropriate for this crate. There isn't a one to one correspondence between the target_arch and the Architecture variants, and some of the target_arch names you are using are not the same as what rustc uses. Instead, please do this in your own crate, where you only need to add support for the architectures that you have implemented and tested. native_object was already borderline, but I accepted it because it was small and unlikely to change much, but even the implementation you gave was already incomplete.

Ok


[MM-17546] Support plugin pagination in the Plugin Marketplace

Summary
This PR adds plugin pagination in the Plugin Marketplace.
Ticket Link
Fixes https://github.com/mattermost/mattermost-server/issues/13269
MM-17546
Screenshots

Creating a new SpinWick test server using Mattermost Cloud.

Mattermost test server created! :tada:
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Account Type
Username
Password

Admin
sysadmin
Sys@dmin123

User
user-1
User-1@123

The plugin count is not centered in the footer, it is centered in the container it is in.
So we need to center that with respect to the whole footer.

Furthermore, instead of giving it a text color of #777.
We should give it a color of :
centerChannelTextColor with an opacity of 0.6.
And a font-size of 14px.

New commit detected. SpinWick will upgrade if the updated docker image is available.

@asaadmahmood thanks for the review. I've just uploaded a new commit. Hope it will be correct.

Mattermost test server updated with git commit dcf2032a5b1d57d9e0d4ecc2a449ad15d221a984.
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

@kosgrz Looks better, however, the font-size is still not 14px.

New commit detected.  Now it has font size of 14px.

Mattermost test server updated with git commit 74e8badb7de37dcd7f80ba9e1bb935cb03dbd6bc.
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Thanks @kosgrz! Heads up that tests appear to fail

@kosgrz Looks great to me now, and no worries :) Thanks for the quick changes.

New commit detected. SpinWick will upgrade if the updated docker image is available.

Nit: Not sure if this is from this PR or if this was there before...but when you click the "previous" button there is a slight "jitter" to the page.  Watch the "Plugin Marketplace" title shift a bit....
@asaadmahmood any idea why this might happen?

Looks good @kosgrz Thanks for your help!

@aaronrothschild It's because the browser scroll bar gets added due to change in height.

@deanwhillier, by chance do you know of a way to avoid this "pop" using CSS?

@lieut-data, possibly but I'll have to take a closer look. Will try to do so today!

@lieut-data, I do have a possible solution. Will check out the PR and submit for review.

@kosgrz, you could do something like this to prevent the horizontal shift due to the browser scroll bar.
FYI for everyone regarding this tweak, it will disable horizontal scrolling on the Fullscreen Modal, but that shouldn't be a problem as content should adjust/flex to support smaller widths ... apart from the Marketplace modal, which isn't supposed to support smaller screens.

@deanwhillier thanks. I will try to test it tomorrow.

@marianunez, 0/5 but I suspect we could still accept this PR as-is, and pivot to server-side pagination later, especially if we decide to tweak the user experience in the interim.

@marianunez, 0/5 but I suspect we could still accept this PR as-is, and pivot to server-side pagination later, especially if we decide to tweak the user experience in the interim.

@lieut-data Sounds good to me. Could we get this first approach of pagination into v5.20?

New commit detected. SpinWick will upgrade if the updated docker image is available.

Mattermost test server updated with git commit 8ebafdcb95f00404ddc06537e0edc705a1b41a2d.
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Test server destroyed

This issue has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!
/cc @jasonblais @hanzei

@ali-farooq0 friendly reminder to help with review :)

/update-branch

This issue has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!
/cc @jasonblais @hanzei

@jasonblais sorry for that delay. I will update the PR in about 10 hours :)

/update-branch

/update-branch

Hey @prapti, @kosgrz had to rework some of the implementation wondering if you can give this a quick test to make sure all is working well before we merge it in.

Creating a new SpinWick test server using Mattermost Cloud.

Mattermost test server created! :tada:
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Account Type
Username
Password

Admin
sysadmin
Sys@dmin123

User
user-1
User-1@123

Test server destroyed

Creating a new SpinWick test server using Mattermost Cloud.

Mattermost test server created! :tada:
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Account Type
Username
Password

Admin
sysadmin
Sys@dmin123

User
user-1
User-1@123

Test server destroyed

Creating a new SpinWick test server using Mattermost Cloud.

Mattermost test server created! :tada:
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Account Type
Username
Password

Admin
sysadmin
Sys@dmin123

User
user-1
User-1@123

Hi @kosgrz! One thing I'd like to note here is that the pagination starts when there are 15 plugins on the list with the Next button, which leads to an empty page. And when you click on the Next button, the page says 16 - 15 plugins of 15 total. (Please see the screenshots below) Can this be fixed?

New commit detected. SpinWick will upgrade if the updated docker image is available.

@prapti thanks for the review! And sorry for this bug. I hope now it's ok.

Mattermost test server updated with git commit a24b2742ace31132156f2d760ce3e28b9c32ab0d.
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Test server destroyed

Creating a new SpinWick test server using Mattermost Cloud.

Mattermost test server created! :tada:
Access here: https://mattermost-webapp-pr-4567.test.mattermost.cloud

Account Type
Username
Password

Admin
sysadmin
Sys@dmin123

User
user-1
User-1@123

Test server destroyed

/update-branch

Thanks @kosgrz and @prapti! 🎉

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.

Trying to auto merge this PR.

Pull Request successfully merged
SHA: 268774b7d0be1473226c7bf4654e7960f6ee828c


BAU: Set include manifest to true

Change description
Follow-up to: https://github.com/communitiesuk/funding-service-design-authenticator/pull/158 hypothesis is this didn't work because the manifest env vars weren't being set as this was set to false by default
How to test
Can test post-merge with test deployment.
Screenshots of UI changes (if applicable)

Closing as this caused a regression.


Filename is blank when importing cost data .csv file via Management API

This was originally raised as an issue against Google Analytics itself:
https://issuetracker.google.com/issues/35353754
It was found there that the issue is with the api client stripping the filename from the POST body, so I'm raising this as an issue here, potentially to get a better response.

@gregroberts Do you by chance have a short sample? Were you using MediaFileUpload?
  media = MediaFileUpload('cow.png', mimetype='image/png',
    chunksize=1024*1024, resumable=True)
  farm.animals().insert(
      id='cow',
      name='cow.png',
      media_body=media).execute()

Some more information from the issue tracker issue:

Method: https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/uploads/uploadData

There's a short sample here, if you have not already seen it.
# Note: This code assumes you have an authorized Analytics service object.
# See the Data Import Developer Guide for details.

# This request uploads a file custom_data.csv to a particular customDataSource.
# Note that this example makes use of the MediaFileUpload Object from the
# apiclient.http module.
from apiclient.http import MediaFileUpload
try:
  media = MediaFileUpload('custom_data.csv',
                          mimetype='application/octet-stream',
                          resumable=False)
  daily_upload = analytics.management().uploads().uploadData(
      accountId='123456',
      webPropertyId='UA-123456-1',
      customDataSourceId='9876654321',
      media_body=media).execute()

except TypeError, error:
  # Handle errors in constructing a query.
  print 'There was an error in constructing your query : %s' % error

except HttpError, error:
  # Handle API errors.
  print ('There was an API error : %s : %s' %
         (error.resp.status, error.resp.reason))

Currently working on reproducing.

Was able to reproduce 'Unknown Filename' result with following sample:
from __future__ import print_function
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
from httplib2 import Http
from oauth2client import file, client, tools

# If modifying these scopes, delete the file token.json.
SCOPES = ["https://www.googleapis.com/auth/analytics"]

def main():
    """Upload custom data to Google Analytics.
    """
    # The file token.json stores the user's access and refresh tokens, and is
    # created automatically when the authorization flow completes for the first
    # time.
    store = file.Storage("token.json")
    creds = store.get()
    if not creds or creds.invalid:
        flow = client.flow_from_clientsecrets("credentials.json", SCOPES)
        creds = tools.run_flow(flow, store)
    service = build("analytics", "v3", http=creds.authorize(Http()))

    # Call the Analytics API
    media = MediaFileUpload(
        "data.csv", mimetype="application/octet-stream", resumable=False
    )
    upload = (
        service.management()
        .uploads()
        .uploadData(
            accountId="144017141",
            webPropertyId="UA-144017141-1",
            customDataSourceId="AbuL8x04Qg2L4flXypb1ag",
            media_body=media,
        )
        .execute()
    )

    print(upload)

if __name__ == "__main__":
    main()

from future import print_function
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
from httplib2 import Http
from oauth2client import file, client, tools
If modifying these scopes, delete the file token.json.
SCOPES = ["https://www.googleapis.com/auth/analytics"]
def main():
"""Upload custom data to Google Analytics.
"""
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
store = file.Storage("token.json")
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets("credentials.json", SCOPES)
creds = tools.run_flow(flow, store)
service = build("analytics", "v3", http=creds.authorize(Http()))
# Call the Analytics API
media = MediaFileUpload(
    "data.csv", mimetype="application/octet-stream", resumable=False
)
upload = (
    service.management()
    .uploads()
    .uploadData(
        accountId="144017141",
        webPropertyId="UA-144017141-1",
        customDataSourceId="AbuL8x04Qg2L4flXypb1ag",
        media_body=media,
    )
    .execute()
)

print(upload)

if name == "main":
main()

This is a known issue in all of the Google client libraries as far as I know.  Management API Cost Data Upload .csv Doesn't Have Filename
Its a consequence of the client libraries stripping the filename from POST body information

No longer have access to the codebase, but the code above is exactly what I was doing.

This is still reproducible. Any workaround available?

@alvarogomezuria You can avoid this by talking to the REST API directly. See https://b.corp.google.com/issues/35353754#comment7

Thanks for the heads up!
Rather not to talk to the REST API directly. I'll keep an eye here! Hopefully it will be resolved soon

Sorry to insist, but i'd like to know if there is any updates/anyone working on this.
Thanks in advance

Hi @alvarogomezuria. If you have a support contract with Google, please reach out that way for faster resolution. I opened another issue internally to ask the Analytics team about their discovery document. I don't think it is something we can fix on the client library side, unfortunately.
Internal issue 138960059

A year passed, and the mistake remained.
Is it so difficult to test the code that Google develops

Closing this as there is no action that can be taken on the library side until the API surface itself is updated. Please reference https://issuetracker.google.com/issues/35353754 for status.


Doesn't work in Windows

I'm trying to run the project on Windows 10 64-bit but so far when it tries to initialize an IPC pipe, it fails with a {ZMQError}Protocol not supported error...
I tried using tcp instead of ipc but it resulted in a {ZMQError}Invalid argument error...
Here's where the error happens : tensorpack/dataflow/prefetch.py:157 => self.socket.bind(self.pipename)
If I use inproc instead of ipc, this error doesn't happen but when it calls self.start_processes() we'll get another error with this traceback:
  File "PATH_TO_PACKAGE\Tensorpack\tensorpack\dataflow\prefetch.py", line 159, in __init__
    self.start_processes()
  File "PATH_TO_PACKAGE\Tensorpack\tensorpack\dataflow\prefetch.py", line 166, in start_processes
    start_proc_mask_signal(self.procs)
  File "PATH_TO_PACKAGE\Tensorpack\tensorpack\utils\concurrency.py", line 152, in start_proc_mask_signal
    p.start()
  File "C:\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Anaconda3\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Anaconda3\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'AugmentImageComponents.__init__.<locals>.func'
Exception ignored in: <bound method PrefetchDataZMQ.__del__ of <tensorpack.dataflow.prefetch.PrefetchDataZMQ object at 0x000001C397C81978>>
Traceback (most recent call last):
  File "PATH_TO_PACKAGE\Tensorpack\tensorpack\dataflow\prefetch.py", line 192, in __del__
    x.terminate()
  File "C:\Anaconda3\lib\multiprocessing\process.py", line 113, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

has anyone been able to run this project on Windows?

Windows doesn't support ipc communication: http://stackoverflow.com/questions/15386121/does-zeromq-support-ipc-as-a-transport-channel-on-windows.
To make it work you'll have to use tcp. The invalid argument error is probably because you didn't write the address correctly. It should be something like tcp://<IP_ADDRESS>:1234.
Using tcp, it probably will run as slow as the python multiprocessing module. So maybe you can just use PrefetchData instead of PrefetchDataZMQ.

@ppwwyyxx
Thanks for the quick response... Changing the protocol to TCP (or inproc) avoids the first error but the second error (which I explained in the post above) still remains... I do believe its main problem resides here:
  File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'AugmentImageComponents.__init__.<locals>.func'

which eventually results in this error:
  File "PATH_TO_PACKAGE\Tensorpack\tensorpack\dataflow\prefetch.py", line 193, in __del__
    x.terminate()
  File "C:\Anaconda3\lib\multiprocessing\process.py", line 113, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

inproc wouldn't work at all. Regarding the error, are you using any of the examples or have you modified the data? It shouldn't try to pickle the augmentor. Do you have an stack trace of the first error?
Btw, if you are running Image2Image.py, maybe just use PrefetchData(ds, 100, 1), or just remove prefetch completely. Dataflow is not the bottleneck here.

Ok... I commented this line ds = PrefetchDataZMQ(ds, 1) and had to change 2 parts of the code...
the first part was to use config.dataflow instead of config.dataset in tensorpack-master/examples/GAN/GAN.py:18 and the second part was to change labels to targets in sigmoid_cross_entropy_with_logits... I'm using Tensorflow 0.12.

Yes it's alright. The first one was fixed 6d67faf93a7 today. The second one is because tensorflow changes their API recently on master.

Thank you for your time... it's working right now and is pretty much faster than what yenchenlin has implemented... I only need time to get around the architecture...
But somehow, this project reminds me of Keras: abstracting some layers and providing easier to work interfaces... am I wrong?


Pictthumb.dll locks files

Windows 10  [Version 10.0.16299.431]
I've noticed that recently we've started to experience troubles with version control software(perforce, git) and graphics software like photoshop. It started to give an error that some files are locked for write. I've discovered that the process Com Surrogate (dllhost.exe) was started by Pictus image viewer via pictthumb.dll. I think its some background process for generating previews.
Please, please, please fix!

Not a solution but perhaps as a workaround you could use Pictus portably instead, as it excludes the thumbnail generation part.

@poppeman
Is Pictus abandoned for good?

I have stopped developing Pictus. If you can manage without the explorer thumbnails, either run portably as @Fironet said or uncheck the shell extension while installing.

OH NOOO!!! It's the best-in-class viewer for Windows!!!


Register Jupyter as a language identifier by default

I would like to activate the extension json-color-token in .ipynb files to change colors in a handy way when plotting. However, it needs a language identifier.
I didn't find jupyter in the know identifiers list at https://code.visualstudio.com/docs/languages/identifiers#_known-language-identifiers.
Could you please register Jupyter as a language identifier by default? Or is there any instruction that helps me to handle this by myself ?

@ivaquero Can you activate using the "onNotebook:jupyter-notebook" activation event? Jupyter not being in the language list is intentional as a Jupyter notebook isn't a language itself, just a container that contain cell of Python, Julia, R, or whatever else. We don't put it in the known languages list as that leads to some oddities like the ability to create a "Jupyter" language cell in a Jupyter document.

You could optionally just ship your extension with a custom language for *.ipynb files. that way when ipynb files are opened as plain text (i.e. as JSON), then your extension will get activated.
Hopefully that would be sufficient.


Consider Token Categories in Lookahead Functions Calculation

Token Categories should be taken into account when performing lookahead calculation.
For example:
            const A = createToken({ name: "A" })

            const B = createToken({
                                   name: "B",
                                   // `B` is a typeof `A`
                                   categories: A
                             })
            const C = createToken({ name: "C" })
            const D = createToken({ name: "D" })

            class CategoriesLookaheadBugParser extends Parser {
                constructor() {
                    super([A, B, C, D])
                    this.performSelfAnalysis()
                }

                public main = this.RULE("main", () => {
                    this.OR([
                        // 'alt1' is `BC`
                       // 'alt2' is `AD`
                      // However `B` can also match `A` due to the categories definition.
                     // So we need to look **two** tokens ahead
                    //  to decide between these two alternatives.
                        { ALT: () => this.SUBRULE(this.alt1) },
                        { ALT: () => this.SUBRULE(this.alt2) }
                    ])
                })

                public alt1 = this.RULE("alt1", () => {
                    this.CONSUME(B)
                    this.CONSUME(C)
                })

                public alt2 = this.RULE("alt2", () => {
                    this.CONSUME(A)
                    this.CONSUME(D)
                })
            }

This new capability will also allow to detect ambiguities that involve token categories.
e.g:

https://sap.github.io/chevrotain/docs/guide/resolving_grammar_errors.html#COMMON_PREFIX


WARNING: Your certificate has expired!

How to recreate server certificate?

I-'ve recreated everything. But I can't retrieve new keys:
docker-compose run --rm openvpn ovpn_getclient $CLIENTNAME > $CLIENTNAME.ovpn
The result of this command is empty file...

It says here that you can renew the certificate and then it should work again.
Any idea what commands that should be in the context of the docker container?
https://forums.openvpn.net/viewtopic.php?t=34389

Ok, managed to find the commands here:
https://github.com/angristan/openvpn-install/issues/1002
They need to be adapted to work on the docker container.
Something along the lines of:
mv /etc/openvpn//pki/reqs/your-host-name.com.req your-host-name.com.req.backup
mv /etc/openvpn//pki/private/your-host-name.com.key your-host-name.com.key.backup
mv /etc/openvpn//pki/issued/your-host-name.com.crt your-host-name.com.crt.backup
mv /etc/openvpn/your-host-name.com.crt your-host-name.com.crt.backup
mv /etc/openvpn/your-host-name.com.key your-host-name.com.key.backup

cd /etc/openvpn
easyrsa build-server-full your-host-name.com nopass
cp /etc/openvpn/pki/issued/your-host-name.com.crt /etc/openvpn
cp /etc/openvpn//pki/private/your-host-name.com.key /etc/openvpn

it worked for me
thanks


Add SQL Identity support

Issue:
EF chose to use 'sequence' to generate PK values by default which is not supported in SQL Azure. This was causing templates to break functionally when published to Azure. It is being tracked for beta5 by the EF team aspnet/EntityFramework#1960
Fix:
As a workaround for beta4, we are adding the workaround in IdentityDbContext which should fix the issue for templates and music store. Apps that are started from empty template need to adds this in their DBContext class manually.
@divega @HaoK

/cc triage @Eilon @danroth27 @DamianEdwards

:shipit:

signed off by triage as well


[SPARK-9893] User guide with Java test suite for VectorSlicer

Add user guide for VectorSlicer, with Java test suite and Python version VectorSlicer.
Note that Python version does not support selecting by names now.

Merged build triggered.

Merged build started.

Test build #41099 has started for   PR 8267 at commit 61092cc.

Test build #41099 has finished for   PR 8267 at commit 61092cc.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41099/
Test FAILed.

Build triggered.

Build started.

Test build #41110 has started for   PR 8267 at commit 3ff4b5d.

Test build #41110 has finished for   PR 8267 at commit 3ff4b5d.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41110/
Test FAILed.

Build finished. Test FAILed.

Build triggered.

Build started.

Test build #41113 has started for   PR 8267 at commit d5cdabd.

Test build #41113 has finished for   PR 8267 at commit d5cdabd.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41113/
Test FAILed.

@yinxusen Sorry, I think there's some merge conflicts. Do you mind rebasing master?

Could you also modify the title to include java tests and python API?

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41237/
Test FAILed.

@yinxusen The documentation looks good. Only things left:

Can you update the title of the PR to include python api and java tests?
Do you mind resolving merge conflicts?

Thanks!

Merged build triggered.

Merged build started.

@feynmanliang Conflicts fixed.

Test build #41305 has started for   PR 8267 at commit 6d066c7.

Test build #41305 has finished for   PR 8267 at commit 6d066c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41305/
Test PASSed.

@yinxusen Could you split this PR into two? One for the user guide with example code and Java test suite, which can be merged into 1.5, and one implements the Python API. We count Python API as new features, and try not to merge them during QA.

Merged build triggered.

Merged build started.

Test build #41345 has started for   PR 8267 at commit f8f0d61.

Test build #41345 timed out     for PR 8267 at commit f8f0d61     after a configured wait of 175m.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41345/
Test FAILed.

jenkins test this please

LGTM pending tests

Merged build triggered.

Merged build started.

Test build #41384 has started for   PR 8267 at commit f8f0d61.

Test build #41384 has finished for   PR 8267 at commit f8f0d61.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaVectorSlicerSuite

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41384/
Test PASSed.

Merged into master and branch-1.5. Thanks!


Maybe add two-player mode (either cooperative or competitive - or both)

Not really an issue, more of a feature I think would be really cool

Good idea, we have considered it, but given the size of the projection we will not be including a two-player mode, at least for this first release.


the autohiding tray feature in the horizontal layout positioned to the left,  makes the menu appear only in the top or bottom instead of left

this his what am using in the configuration file for a left tray (autohide="true" halign="left" valign="center" layout="horinzontal") i suggest that the (autohide feature should have  instead of "true" his "left" "right" "top" "bottom" so we can chose wich side off the screen the menu appears back again!

This should be available in the latest snapshot.  "true" and "false" are no longer supported since I'm breaking config compatibility anyway.  Now it's "left", "right", "top", "bottom", or "off".


[BUG] Modus Table - Truncated cell ellipsis should display only when the full width is false and this should be applied on all table functionalities

Prerequisites

[X] I have searched for duplicate or closed issues
[X] I have read the contributing guidelines

Describe the issue
Table Component - Truncated cell ellipsis should display only when the full width is false and this should be applied on all table functionalities
Tested on PR: https://github.com/trimble-oss/modus-web-components/pull/1463
Expected Result: Truncated cell ellipsis should display only when the full width is false and when full width is true, truncated cell ellipsis should not be displayed
Actual Result:  Truncated cell ellipsis is set to be displayed when full width is false and true(both)
Reduced test cases
No response
What operating system(s) are you seeing the problem on?
Windows
What browser(s) are you seeing the problem on?
Chrome, Firefox, Microsoft Edge
What is the issue regarding ?
@trimble-oss/modus-web-components
What version of npm package are you using ?
No response
Priority
Medium
What product/project are you using Modus Components for ?
QA
What is your team/division name ?
QA
Are you willing to contribute ?
None
Are you using Modus Web Components in production ?
No response

Issue is fixed and is working fine.
Verified on PR#: 1485, 1463,  and 1501


linuxPackages*.turbostat: init

Motivation for this change
Fixes #68847.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

when we can expect turbostat to get merge into the main tree?

Today's your lucky day! ;)


Multiple windows for multiple repos?

Open repos in separate windows?

Personally I'm against this, though I don't necessarily have strong reasoning as to why. I think I just like the workflow of one repo at atime

I'm also fine with the single-window view. If you want multiple windows, I suppose you can just run multiple instances of the app?

I think having separate repos in separate windows would be useful and less clunky than that of running the app twice. That said, since you can run the app twice, this would be a low priority fix.

Also consider tabs like in sourcetree?


How to express invasionStage for plants?

@qgroom @SoVDH @timadriaens what invasion stage information do we want to extract from the manual of alien plants?
This is the information we have:

D/N is information you have already mapped (filter on "plants")
presence_value comes from the presence columns Fl, Wa, and Br, which can have x for present and ? for unknown (I assume).

In it's most simple form, we only express that a taxon is:

present = x for that region
unknown = ? for that region

But we could also take some of the information from the D/N column, to make more nuanced statements, such as established, introduced, to be determined, unknown. Note that even for values in D/N without a question mark, we do have question marks in the presence_value column (and vice versa), so we need to decide which one takes preference.

And finally, a lot of taxa have empty presences for regions. We could either express this as absent from that region, or take the more cautious approach (maybe there was no monitoring for that region) and not say anything about the occurrence of that species in that region.

Could you give some input regarding this, so we know how to basically map the above table to occurrenceStatus (that's the field to use right @qgroom)?

@qgroom I'm probably mixing a number of DwC terms, so I'm definitely need your input here. 😅

I'm referring to #60  for a detailed discussion on how to map invasion stage combined with presence information.
With respect to the third option, this is fully discussed in https://github.com/trias-project/alien-plants-belgium/issues/45#issuecomment-351099040 (starting from there).
So I'm closing this issue


Reactive DiscoveryClient Implementation Using Kubernetes Java Client

@ryanjbaxter one minor question, is the general approach to move to kubernetes java client instead of fabric8?

@wind57 Yes we would like to get there, unless the community tells us otherwise

@wind57 Yes we would like to get there, unless the community tells us otherwise

thank you.
This also means that spring-cloud-kubernetes-fabric8-autoconfig will be replaced with spring-cloud-kubernetes-client-autoconfig in spring-cloud-starter-kubernetes, eventually, right?
Thing is I did some PRs against spring-cloud-kubernetes-fabric8-autoconfig and it seems they can be closed since this code will not be maintained in the long run and if I would like to do something it should happen in spring-cloud-kubernetes-client-autoconfig. Did I understand correctly?

Did I understand correctly?

@wind57 No that is not entirely correct.
I saw the PRs but have not had a chance to look at them yet, hopefully tomorrow.
Right now we are maintaining both, so if you are fixing issues in the Fabric8 modules that is more than welcome.  If you are adding functionality then we would want that functionality to be present in both.
There are separate starters for the kubernetes java client and fabric8 client modules now.

I see, I was more interested into understanding how things are rather than triggering you into looking the PRs, they are pretty minor anyway. I am mainly trying to understand the overall code and that is my way of doing it. thank you again for the answers.


Regex rules for copyright check

Is your feature request related to a problem? Please describe.
Required copyright statement format is fixed and not flexible. It requires to use standard c++ comments '//'.
And here we have a few problems:

this could not work for other types of files, e.g. pure C
doxygen cannot be used for copyrigth statement (@copyrigth)
the lines of the statement cannot be flexible - for instance contain different year depending when the file was created

Describe the solution you'd like
Besides current approach also allow to use regex statement description in scargo.toml file, e.g.,:
[check.copyright]
description = '''
/\*\*
 \* @copyright Copyright \(C\) \d* SpyroSoft Solutions. All rights reserved.
 \*'''
exclude = []

and this would fit for instance this doxygen comment:
/**
 * @copyright Copyright (C) 2023 SpyroSoft Solutions. All rights reserved.
 */

Sample implementation in check.py:
    def __get_copyright_lines_regex(self) -> List[str]:
        return [f"{el}" for el in self.copyright_desc.split("\n")]

    def check_file(self, file_path: Path) -> CheckResult:
        copyright_lines = self.__get_copyright_lines()

        with open(file_path, encoding="utf-8") as file:
            file_lines = file.readlines()

            for line_no, line in enumerate(file_lines[: -(len(copyright_lines) - 1)]):
                if line == copyright_lines[0]:
                    if all(
                        line_from_file == line_from_copyrights
                        for line_from_file, line_from_copyrights in zip(
                            file_lines[line_no:], copyright_lines
                        )
                    ):
                        return CheckResult(problems_found=0)

        copyright_lines = self.__get_copyright_lines_regex()

        with open(file_path, encoding="utf-8") as file:
            file_lines = file.readlines()

            for line_no, line in enumerate(file_lines[: -(len(copyright_lines) - 1)]):
                if re.match(copyright_lines[0], line):
                    if all(
                        re.match(line_from_copyrights, line_from_file) != None
                        for line_from_file, line_from_copyrights in zip(
                            file_lines[line_no:], copyright_lines
                        )
                    ):
                        return CheckResult(problems_found=0)

        logger.warning("Missing copyright line in %s.", file_path)
        return CheckResult(problems_found=1)

Just keep in mind that there can be any custom Copyright  set, not necessarily Spyro one

Done in release 2.1.1


Fix fix qualifier problem that affects delete stmt in another catalog #32853

cherry-pick #32853 to branch 2.0

run buildall

run buildall


Remove reference to epydoc and supporting Python 2 only

The README.rst file currently says:

Currently Implemented

Python (2 only -- Epydoc doesn't support Python 3)
.NET
Go
Javascript

Note epydoc is a no longer maintained Python module which has not been updated for nearly 10 years:
https://pypi.python.org/pypi/epydoc
http://epydoc.sourceforge.net/
Searching the code this is the only mention of epydoc - so I believe the README entry is out of date.
Reading the code, I think you use the Python standard library modules ast and tokenize instead, so this ought to work on Python 3 too. Indeed the .travis.yml file explicitly tests under Python 3.

We've had a number of discussions around this, we want to avoid maintaining another AST traversal mechanism. Supporting python 2 and python 3 would require maintaining two separate ast traversal patterns, and quite frankly, even one is plenty of work -- hence epydoc.
Currently, we are using pydocstyle to parse the python code, so the documentation is out of date.

pdoc (http://pdoc.burntsushi.net/pdoc) is something that could be useful to include in this discussion.

https://github.com/BurntSushi/pdoc/issues/82

Thanks - the README file is much clearer now :)


Design Polish

Resolves #998
Resolves #895

Pull Request Test Coverage Report for Build 1010

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 52.073%

Totals

Change from base Build 1000:
0.0%

Covered Lines:
4014

Relevant Lines:
6928

💛  - Coveralls


Azure Terraform Outputs.tf - Fails and halts during terraform destroy

What's not working? Or What's happening that shouldn't be?
If the issue is not covered by either of the above questions - please don't log an issue.
ISSUE:
Terraform destroy - fails as the outputs cannot be evaluated
Error: Error applying plan:

1 error(s) occurred:

* local.hashistackdemo: local.hashistackdemo: Resource 'data.azurerm_public_ip.web_front_end' does not have attribute 'ip_address' for variable 'data.azurerm_public_ip.web_front_end.ip_address'

Need to review how the data variables are evaluated -

https://github.com/hashicorp/terraform/issues/17881
https://github.com/hashicorp/terraform/blob/master/CHANGELOG.md#0111-november-30-2017
https://github.com/hashicorp/terraform/issues/18026

SOLUTION:

Will be rebuilding tf code in 2020 for all major clouds


Controlled UnitaryGate not displayed correctly.

Information

Qiskit Terra version:
Python version:
Operating system:

What is the current behavior?
This issue was brought up in the Qiskit Slack workspace.
https://qiskit.slack.com/archives/C7SJ0PJ5A/p1591208099333800?thread_ts=1591204734.332900&cid=C7SJ0PJ5A
Steps to reproduce the problem
from qiskit import *
from qiskit.extensions import UnitaryGate
import numpy as np
c=QuantumCircuit(3)
op = np.matrix([[1,0],[0,-1]])
r=UnitaryGate(op, label='rt')
re=r.control(2, ctrl_state='11')
c.append(re, [0,1,2])
c.draw('mpl')

What is the expected behavior?
Suggested solutions

There are actually several issues here. The first most obvious is that the mpl drawer does not draw it properly. The text drawer, though, has this output.
q_0: ───────■────────
     ┌──────┴───────┐
q_1: ┤0             ├
     │  circuit1_dg │
q_2: ┤1             ├
     └──────────────┘

The name is the base_gate name that is created by the QuantumCircuit call. The op name is c-unitary. I assume that should be used for the display, or the label.
The bigger problem is that the call to ControlledGate in the control() method for UnitaryGate does not pass the num_ctrl_bits or ctrl_state, so you're not getting the gate you thought you were. I'm going to open a separate issue to fix this, and leave this one for fixing the displays.

@1ucian0 This issue already exists #4549

I agree with @enavarro51 that there are many issues together. I will close this one as duplicate following @ewinston advice once we break this issue.


build(main): release 0.4.3

:robot: I have created a release beep boop
0.4.3 (2024-12-06)
Code Refactoring

text for events and messages (5218402)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v0.4.3
:sunflower:


[AMORO-2802] Change the folder name of optimizer module

Why are the changes needed?

Close #2802 .
Brief change log

Add prefix amoro
Keep the same with module and folder name

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

lgtm

Hi,
We used to say Flink optimizer/Spark optimizer. So do you think amoro-flink-optimizer/amoro-spark-optimizer is a proper name for both the artifact id and the folder name?

amoro-flink-optimizer/amoro-spark-optimizer

amoro-flink-optimizer/amoro-spark-optimizer is better.

Hi,
We used to say Flink optimizer/Spark optimizer. So do you think amoro-flink-optimizer/amoro-spark-optimizer is a proper name for both the artifact id and the folder name?

@zhoujinsong @xleoken
Yes, we used to say Flink/Spark optimizer, in order to maintain the unity of the module format, I looked at the flink connector.
Like the structure of flink connector, the module name does not affect us calling it flink kafka connector[1].
[1] : https://github.com/apache/flink-connector-kafka

hi @zhongqishang @zhoujinsong
We can keep the style as amoro-mixed-format module, so amoro-optimizer-flink and amoro-optimizer-spark is ok.
amoro-mixed-format

amoro-mixed-format-flink
amoro-mixed-format-hive
amoro-mixed-format-spark
amoro-mixed-format-trino


Keystone Data Import to HubSpot

Import all active family accounts, site visitor accounts and active social worker accounts into HubSpot.
Fields to import:
Families:

[x] email address (login)
[x] contact 1 first name
[x] contact 1 last name
[x] contact 1 phone (use home phone number)
[x] contact 1 date of birth
[x] contact 2 email address
[x] contact 2 first name
[x] contact 2 last name
[x] contact 2 mobile phone number
[x] contact 2 date of birth
[ ] street 1
[ ] street 2
[ ] city
[ ] state
[ ] zip code
[ ] region
[ ] keystone record URL (created)

Site Visitors

[ ] email (login)
[ ] first name
[ ] last name
[ ] home phone number
[ ] street 1
[ ] street 2
[ ] city
[ ] state
[ ] zip code

Social Workers

[ ] email address (login)
[ ] first name
[ ] last name
[ ] work phone number
[ ] keystone record URL (created)

Investigate assigning users to lists dynamically

Investigate assigning users types by account types

Will export to excel first, then imports created into HubSpot

Review Tasks

[ ] separate contact 1 and contact 2 into their own rows

[ ] contact 2 row should only be created if an email address exists

[ ] combine street 1 and street 2 into a single column named 'street address'


[SPARK-29914][ML][FOLLOWUP] fix SQLTransformer & VectorSizeHint toString method

What changes were proposed in this pull request?
1,modify the toString in SQLTransformer & VectorSizeHint
2,add toString in RegexTokenizer
Why are the changes needed?
in SQLTransformer & VectorSizeHint , toString methods directly call getter of param without default values.
This will cause java.util.NoSuchElementException in REPL:
scala> val vs = new VectorSizeHint()
java.util.NoSuchElementException: Failed to find a default value for size
  at org.apache.spark.ml.param.Params.$anonfun$getOrDefault$2(params.scala:780)
  at scala.Option.getOrElse(Option.scala:189)

Does this PR introduce any user-facing change?
No
How was this patch tested?
existing testsuites

I manually scan other impls, and do not find other similar places.

Test build #115722 has started for PR 26999 at commit b274237.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20516/
Test PASSed.

Test build #115722 has finished for PR 26999 at commit b274237.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115722/
Test PASSed.


Invalid string length

Cause our angular project has a big size module,we can't package it success.
It will throw an error "var string = JSON.stringify(message) + '\n'; " and "Invalid string length".
When I reduce some component of this module,I can package it success.
So, I sure it  because of the body volume.
Version: "1.2.6"

@dds-du need reproducible test repo, without this i can't help, sorry

@evilebottnawi https://github.com/trivago/parallel-webpack/issues/54
You can see this issue.it's a same problem with our project.

Sorry, without minimum reproducible test repo i can't solve your problem, also you can increase nodejs memory usage (it should be help). In some cases 512mb is not enough to uglify bug projects. Feel free create new issue with minimum reproducible test repo.


fix: clean up optional components of the schema

This PR cleans up different elements of the schema that were a mismatch between expectations and validation mostly around the use of None.

didn't look inside yet but tests (besides dandi-cli) seems to be not happy.

@yarikoptic - should be ok now

you would want this to be released @satra (do not see a tag) or we better absorb more changes before the release?

i wanted to wait to see if there were other changes to be made to the schema before tagging for release. i want to slowly start consolidating schema changes as much as possible.

ok, then let's proceed with a merge without release


devApp of the plugin scaffolder don't work

📜 Description
I did a test top of the backstage master branch (commit 2c2f705fe169 2024-03-18 Patrik Oldsberg (HEAD -> master, upstream/master) Merge pull request #23663 from backstage/rugvip/watch-fix ), build the project and launch using yarn start the DevApp of the plugin scaffolder to test it without success.
The browser's console reports the following issue:
react-dom.development.js:86 Warning: Cannot update a component (`AlertDisplay`) while rendering a different component (`TemplateGroups`). To locate the bad setState() call inside `TemplateGroups`, follow the stack trace as described in https://reactjs.org/link/setstate-in-render
    at TemplateGroups (webpack-internal:///../scaffolder-react/src/next/components/TemplateGroups/TemplateGroups.tsx:47:120)
    at div
    at Grid (webpack-internal:///../../node_modules/@material-ui/core/esm/Grid/Grid.js:237:35)
    at WithStyles (webpack-internal:///../../node_modules/@material-ui/styles/esm/withStyles/withStyles.js:65:31)
    at Content (webpack-internal:///../catalog-react/src/components/CatalogFilterLayout/CatalogFilterLayout.tsx:128:25)
    at div
    at Grid (webpack-internal:///../../node_modules/@material-ui/core/esm/Grid/Grid.js:237:35)
    at WithStyles (webpack-internal:///../../node_modules/@material-ui/styles/esm/withStyles/withStyles.js:65:31)
    at CatalogFilterLayout (webpack-internal:///../catalog-react/src/components/CatalogFilterLayout/CatalogFilterLayout.tsx:142:25)
    at article
    at Content (webpack-internal:///../../packages/core-components/src/layout/Content/Content.tsx:65:13)
    at main
    at ThemeProvider (webpack-internal:///../../node_modules/@material-ui/styles/esm/ThemeProvider/ThemeProvider.js:45:24)
    at Page (webpack-internal:///../../packages/core-components/src/layout/Page/Page.tsx:54:13)
    at EntityListProvider (webpack-internal:///../catalog-react/src/hooks/useEntityListProvider.tsx:60:96)
    at TemplateListPage (webpack-internal:///./src/next/TemplateListPage/TemplateListPage.tsx:76:106)
    at RenderedRoute (webpack-internal:///../../node_modules/react-router/dist/index.js:577:5)
    at Routes (webpack-internal:///../../node_modules/react-router/dist/index.js:1267:5)
    at Router (webpack-internal:///./src/components/Router/Router.tsx:58:25)
    at RoutableExtensionWrapper (webpack-internal:///../../packages/core-plugin-api/src/extensions/extensions.tsx:63:99)
    at AnalyticsContext (webpack-internal:///../../packages/core-plugin-api/src/analytics/AnalyticsContext.tsx:71:13)
    at PluginErrorBoundary (webpack-internal:///../../packages/core-plugin-api/src/extensions/PluginErrorBoundary.tsx:54:9)
    at Suspense
    at Result (webpack-internal:///../../packages/core-plugin-api/src/extensions/extensions.tsx:182:73)
    at RenderedRoute (webpack-internal:///../../node_modules/react-router/dist/index.js:577:5)
    at FlatRoutes (webpack-internal:///../../packages/core-app-api/src/routing/FlatRoutes.tsx:50:83)
    at div
    at StyledComponent (webpack-internal:///../../node_modules/@material-ui/styles/esm/styled/styled.js:96:28)
    at SidebarPinStateProvider (webpack-internal:///../../packages/core-components/src/layout/Sidebar/SidebarPinStateContext.tsx:51:13)
    at SidebarPage (webpack-internal:///../../packages/core-components/src/layout/Sidebar/Page.tsx:74:84)
    at Router (webpack-internal:///../../node_modules/react-router/dist/index.js:1201:15)
    at BrowserRouter (webpack-internal:///../../node_modules/react-router-dom/dist/index.js:694:5)
    at AppRouter (webpack-internal:///../../packages/core-app-api/src/app/AppRouter.tsx:105:140)
    at Suspense
    at RoutingProvider (webpack-internal:///../../packages/core-app-api/src/routing/RoutingProvider.tsx:34:28)
    at ThemeProvider (webpack-internal:///../../node_modules/@material-ui/styles/esm/ThemeProvider/ThemeProvider.js:45:24)
    at StylesProvider (webpack-internal:///../../node_modules/@material-ui/styles/esm/StylesProvider/StylesProvider.js:50:24)
    at ThemeProvider (webpack-internal:///../../node_modules/@mui/private-theming/ThemeProvider/ThemeProvider.js:45:5)
    at ThemeProvider (webpack-internal:///../../node_modules/@mui/system/esm/ThemeProvider/ThemeProvider.js:57:5)
    at ThemeProvider (webpack-internal:///../../node_modules/@mui/material/styles/ThemeProvider.js:26:14)
    at StyledEngineProvider (webpack-internal:///../../node_modules/@mui/styled-engine/StyledEngineProvider/StyledEngineProvider.js:31:5)
    at UnifiedThemeProvider (webpack-internal:///../../packages/theme/src/unified/UnifiedThemeProvider.tsx:50:13)
    at Provider (webpack-internal:///../../packages/app-defaults/src/defaults/themes.tsx:45:22)
    at AppThemeProvider (webpack-internal:///../../packages/core-app-api/src/app/AppThemeProvider.tsx:73:29)
    at AppContextProvider (webpack-internal:///../../packages/core-app-api/src/app/AppContext.tsx:32:31)
    at ApiProvider (webpack-internal:///../../packages/core-app-api/src/apis/system/ApiProvider.tsx:46:13)
    at Provider (webpack-internal:///../../packages/core-app-api/src/app/AppManager.tsx:198:29)
    at AppRoot
```

### 👍 Expected behavior

We should be able to see some templates to scaffold under link `/create`

### 👎 Actual Behavior with Screenshots

![Screenshot 2024-03-18 at 15 11 07](https://github.com/backstage/backstage/assets/463790/d06b6546-d6f1-49f4-9503-54057d4bd129)
![Screenshot 2024-03-18 at 15 20 35](https://github.com/backstage/backstage/assets/463790/009f5a9f-51f3-4ac2-899c-eb100079133c)

### 👟 Reproduction steps

git clone https://github.com/backstage/backstage
cd backstage
yarn install
yarn tsc
yarn build:all
cd plugins/scaffolder
yarn start

### 📃 Provide the context for the Bug.

n/a

### 🖥 Your Environment

See : https://gist.github.com/cmoulliard/28723488a2ce13835936c22d0ca7b5a0

### 👀 Have you spent some time to check if this bug has been raised before?

- [X] I checked and didn't find similar issue

### 🏢 Have you read the Code of Conduct?

- [X] I have read the [Code of Conduct](https://github.com/backstage/backstage/blob/master/CODE_OF_CONDUCT.md)

### Are you willing to submit PR?

No, but I'm happy to collaborate on a PR with someone else

I think that you'll also need to start the backend in general for this frontend to work. It's going to need the catalog + scaffolder in order to run. Maybe we can make this a little clearer and easier to setup somehow?

Maybe we can make this a little clearer and easier to setup somehow?

Yes, please as this not evident/logic for a new developer to figure out that the backend should be started in another terminal.
Important: That fails too if you launch separately the backend
// Terminal 1
cd backstage/plugins/scaffolder
yarn start

// Terminal 2
cd backstage/packages/backend-next
yarn start

@cmoulliard what's the actual failure for the network requests? Maybe it's something with backend-next and the new auth work that's ongoing, does it work with packages/backend instead?

Maybe it's something with backend-next and the new auth work that's ongoing, does it work with packages/backend instead?

No luck too using packages/backend instead of packages/backend-next

what's the actual failure for the network requests?

Nothing is logged as errors within the terminal of front packages/plugins/scaffolder => yarn start or backend packages/backend => yarn start

@cmoulliard sorry I mean in the chrome dev tools, what's the network request failures say when it's trying to communicate with the backend?

what's the network request failures

I dont see network's errors

I dont find errors if I filter using -status-code:200  within DevTools - network tab

Here is the HTTP Archive json output: https://gist.github.com/cmoulliard/beb7042dc2be376b5e6584c44d63ef2b

Any idea ?

The existing code needs some refactoring :-) I was able to use this index.tsx file (of our quarkus plugin) with the backstage packages/backend-next
import { catalogPlugin } from '@backstage/plugin-catalog';
import { createDevApp } from '@backstage/dev-utils';
import { scmIntegrationsApiRef } from '@backstage/integration-react';
import React from 'react';
import { ScaffolderClient, ScaffolderPage } from '@backstage/plugin-scaffolder';
import { scaffolderApiRef, ScaffolderFieldExtensions } from '@backstage/plugin-scaffolder-react';
import {
  discoveryApiRef,
  fetchApiRef,
  identityApiRef,
} from '@backstage/core-plugin-api';
import { CatalogEntityPage } from '@backstage/plugin-catalog';
import {QuarkusExtensionListField, QuarkusVersionListField} from "../src";

createDevApp()
  .registerPlugin(catalogPlugin)
  .registerApi({
    api: scaffolderApiRef,
    deps: {
      discoveryApi: discoveryApiRef,
      fetchApi: fetchApiRef,
      scmIntegrationsApi: scmIntegrationsApiRef,
      identityApi: identityApiRef,
    },

    factory: ({ discoveryApi, fetchApi, scmIntegrationsApi, identityApi }) =>
      new ScaffolderClient({
        discoveryApi,
        fetchApi,
        // @ts-ignore
        scmIntegrationsApi,
        identityApi,
      }),
  })
  .addPage({
    path: '/create',
    title: 'Create',
    element: <ScaffolderPage />,
    children: (
        <ScaffolderFieldExtensions>
            <QuarkusExtensionListField />
            <QuarkusVersionListField />
        </ScaffolderFieldExtensions>
    )
  })
  .addPage({
    element: <CatalogEntityPage />,
  })
  .render();

Ticket can be closed as PR fixed the issue


Utilize Modularity for Performance, Scalability and Reliability

Problem
cilium-agent is excessively using node and K8s Control Plane resources (CPU, Memory, Network Throughput). It hinders its and the systems performance and scalability.
Why does the problem occur?
cilium-agent by default runs a lot of processes for the features that aren’t enabled or used.
K8s watchers are initialized despite not being used, or the features that require them may not be used at a time.
Proposal
Utilize modularization (the hive system) to ensure that only required processes are running.
Modularization
Every feature is a self contained cell, decoupled from other feature cells as much as possible.
Every feature is implemented as a cell that can be started and stopped independently of other feature cells.
Feature Enablement Framework
Enablement
A feature cell is started when a feature is enabled and stopped when a feature is disabled.
A feature cell can be restarted any number of times without side effects.
K8s Resource Tracking
Track which feature cells are running and the required K8s API resources for those cells.
Watchers for K8s API resources that aren’t required by any features will be disabled.
Dynamic Feature Enablement
Allow cilium-agent and cilium-operator to read configuration changes while running, and not to require restart.
High Level Design
A new ConfigMap cilium-dynamic-config will be created.
cilium-agent watches the new ConfigMap and dynamically starts and stops cells.
cilium-operator watches the base ConfigMap cilium-config and other K8s resources, determines which features are enabled and writes it to the cilium-dynamic-config.
Alternative
Use mounted volumes. Read from files. Delay of changes to ConfigMap will be 1 minute. Can it be reduced?
No advantages, since kubelet anyway creates a watcher for each mounted ConfigMap.
Lazy Feature Initialization
A feature cell is initialized only after it’s used, despite the feature being enabled.
(e.g. network policy enforcement system is only initialized after a network policy is created in the cluster)
End goal example

Network policy feature is disabled based on a flag in cilium-config.
Network policy cell isn’t started. Cilium Endpoint, Cilium Identity and Cilium Endpoint Slice aren’t managed (created, updated or 3. deleted).
The watchers for the same K8s API resources aren’t started.
Network policy feature is enabled when the flag in cilium-config is switched on.
cilium-operator reads the changed cilium-config, but doesn’t make any actions because there are no network policies in the cluster.
A network policy resource is created in the cluster.
cilium-operator updates the dynamic ConfigMap.
cilium-agent reads it and starts the network policy cell.
cilium-agent starts the required watchers.
Network policy enforcement system is enabled.

Dependencies
Dependency for network policy modularization: Modularization: pkg/identity https://github.com/cilium/cilium/issues/23767
List of Features
Additional features that are suitable to use the described optimizations will be added to the list over time.

cc @ovidiutirla @sypakine @joestringer @marseel

Some quick thoughts:
Modularization:
on high-level of enabling/disabling cells definitely makes sense. I think we already do it, at least for new features, for example l2announcement: https://github.com/cilium/cilium/blob/4219e5c3292612b7ecd221d39a683de1996ce71a/pkg/l2announcer/l2announcer.go#L122
so it's the matter of modularizing "old" features.
There might be gaps when a feature is disabled, like for example we create CRD in the operator - but that's something to improve on. I don't think we start watchers for disabled features / unused CRDs: https://github.com/cilium/cilium/blob/main/pkg/k8s/resource/resource.go#L462
Dependant features:
Theoretically we have graph dependency in the hive, so for example, if someone wants to enable EGW policy, we could determine that CEP and CID is also required and turn that on too, but it sounds a bit tricky to automatically ensure that all dependant features are on too.
For dynamic enablement:
I'm not sure about it. The usual way of enabling a feature is through rolling upgrade, which is definitely much safer operation than turning the feature at the same time in all agents - which would essentially break things cluster-level while in rolling upgrade it would potentially stop on the first agent failing.
There might be cases in datapath as well, where enabling particular feature needs recreation of ebp programs/maps and I think that currently we perform it safely only on startup - but I am far from the expert in this area, so would appreciate someone else input.
So for dynamic enablement sounds like we would need some orchestrator that enables it per agent, which would essentially be similar to the rolling upgrade done by k8s.
For NetworkPolicy example:
Synchronized starting of all watchers sounds like a significant spike load. Also, assuming that CEPs are not created, creating them with the first NetworkPolicy would be significant load too on control-plane.
Alternatively, enabling NetworkPolicies through regular config-map and performing rolling upgrade sounds like more predictable load on control plane.

Modularization makes sense, already tracked by this issue: https://github.com/cilium/cilium/issues/23425 .
Dynamic enablement is conceptually interesting. As Marcel points out, it would be helpful to have a model of understanding about how invasive the change is and how we appropriately stage those changes to minimize impact. Not all changes can be safely transitioned, for instance the spike load case. Also worth mentioning that sometimes certain changes are implemented in Helm at the moment with changes to other Cilium resources, not just the ConfigMap. Depending on the functionality, maybe we need to modify other settings, change capabilities, or otherwise configure multiple resources. Dynamically handling ConfigMap changes is just one piece of the puzzle. Related, I think it would help to have a few really good use cases to demonstrate the current pain points and how this proposal would improve those situations from a user perspective.
The network policy example doesn't make sense to me at all. I feel like we'd need to take a step back to consider the high level architecture of Cilium, the functionality it provides, and the personas who we are trying to design this system for.

Thanks for the feedback.
I understand that network policy is one of the main and most useful features of Cilium, and I agree that ensuring network security is very valuable. Therefore, perhaps network policies are not a good example that I included in the description. However, I can definitely see how Cilium can be used for a wide variety of its amazing features, without enabling all / most of the features. Designing the system in a way that will allow supporting maximum scalability and minimum footprint for any specific configuration brings great benefit.

Hi, following up on this.
I wanted to provide more information behind the motivation for trying to move in this direction.
We want to modularize network policy enforcement system because we have users on GKE that require very high scale (# of nodes and pod churn rate) which currently isn’t within the scalability limits of Cilium network policies. We want network policies to be enabled eventually, by making them more scalable with the other proposed improvements. Right now, we have a choice to either disable network policies in Cilium, or to disable Cilium entirely if we cannot only disable network policies. I would prefer to disable only network policies temporarily until the required scalability is achieved.
Besides strictly requiring disabling features, the way I see we can achieve this for all users, in the most user friendly way, is to modularize and lazily initialize parts of the system, so that Cilium can scale up to it’s maximum for any feature set.

Thank you @learnitall for the detailed explanation.
I agree with you. Your insights help me better understand the direction we want to take, one that fits in and benefits different aspects both from developer and user sides.
Perhaps before moving forward with the Lazy Initialization and related discussions, I want to see how I can contribute to modularization. I've assigned the Modularization: pkg/identity issue to myself and I'll start from there. https://github.com/cilium/cilium/issues/23767
I’ve taken a look into Audit Policy (flag policy-audit-mode) and as far as I see it doesn’t disable the network policy enforcement system, specifically Cilium Endpoints and Cilium Identities, which presents the scalability bottleneck.
We’ve been able to run Cilium on GKE on clusters with up to 5k nodes, 200k pods, 100/s pod churn rate, and there is a similar ongoing effort here https://github.com/cilium/cilium/issues/33778.
Another thing we want to do is to provide data about the issues that Cilium faces at such and higher scales, so that we can together start tackling these scalability obstacles, and plan for having network policies on even higher scales. The two most relevant scalability dimensions are number of nodes in the cluster and pod churn rate, and there are users that want to stretch both of these dimensions.

@dlapcevic In case you haven't seen already, does the feature set provided by CiliumNodeConfig align with what you're looking for with cilium-dynamic-config ?


Difficulties using component-defined linker script with regions file or non-preprocessor linker script include

Background
We are a vendor implementing a DFP for a family of devices. There are 8 distinct parts, each with the same 2 cores (and these 2 are different from one another). In other words, we have:

DEVICE_1:CORE_1
DEVICE_1:CORE_2
DEVICE_2:CORE_1
...
DEVICE_8:CORE_1
DEVICE_8:CORE_2

as available targets.
Some features are defined by the core (clock, local TCM, first part of the vector table), while others are defined by the device (bulk/external memory, peripherals, second part of the vector table). We had no problems specifying these in the startup component of the CMSIS pack: it selects some files based on the core and some based on the device. This approach is much easier and more logical than maintaining all 16 distinct combinations (especially if we add more cores and devices going forward).
Describe The Problem To Be Solved
[problem A] The first problem we ran into is with using two part-linker script. Because of unusual layout of the device, our startup component provides linker script in two files (both with config attribute). Device part specifies regions: their base addresses and sizes, and is then included in the core part that serves as an actual script used for linking. This is using standard INCLUDE directive in LD and doesn't rely on C preprocessor. This unfortunately doesn't work because link stage (with GCC at least) is not given any include paths (library lookup paths serve as include paths for LD) and the included script path needs to be referenced from the build folder, e.g.:
INCLUDE ..\..\..\RTE\Device\DEVICE1_CORE1\mem_regions.ld
Since the core linker script is generic for a given core on all devices, it won't know this path and hence this approach is impossible implement (at least without additional directives in the cdefault/cproject file).
[problem B] Seeing as csolution already generates a convenient (and configurable) regions header automatically, we attempted to use this approach, especially given its clearly defined in the spec. However this also proved difficult to implement, because preprocessing of the linker script is only enabled if both, the filename suffix is .src AND cdefault/cproject specifically lists the regions file. Given the name of our auto-generated regions header will be different for every device:core combination, we can't make it work out of the box with a generic project configuration.
[problem C] Defining our own generic regions.h file included in our startup component doesn't prevent csolution from auto-generating one, even if no memory regions are specified in the .pdsc.
Suggest A Solution

For problem A: Changing the default .cmake templates to add a config destination path (i.e.: RTE/Device/DEVICE_CORE) as an include/library lookup path for link stage should be enough. I can do this change myself but we'd like to see this working out of the box. This change is unlikely to break any compatibility.
For problem B: Auto-include the generated regions header in the selected linker script if the extension matches the rule, without requiring any additional input in the cproject file.
For problem C: Allow .pdsc to define file with type "regions" to let vendors provide their own regions header and prevent one from being auto-generated. This is probably the best enhancement to tackle this issue.

Ideally A & B or A & C could be implemented to allow for easier regions management.

@KarolSaja-AlifSemi thanks for this input that comes very timely. Is your DFP accessible somewhere?  Having an example make things easier.

@ReinhardKeil thank you for picking this up so quickly. This issue relates to an experimental pack file we're building internally (.pdsc attached), where we're trying different layouts to determine which one is the most logical and simplifies our maintenance going forward as more device variants (with different memories, peripherals & etc.) are introduced. Right now we're just focusing on startup code.
AlifSemiconductorDEV.Ensemble.pdsc.txt
For reference, our official "DFP" is already submitted to the Keil database here: https://www.keil.arm.com/packs/ensemble-alifsemiconductor/devices/ so you can see what kind of device/core combinations there are.
In the PDSC you can see that the startup component includes two linker scripts: one based on the processor (core) name and one based on the device name. Device linker script has its category set to "other" to avoid confusing csolution/cbuild with multiple linker scripts, as its sole intention is to be included in the processor linker script.
Our intention is to make this work without passing any special settings to the project configuration (especially if they're device/processor specific, it will make our basic project template a nightmare to maintain), by simply including our startup component. We also want to avoid having to create (and maintain) a unique linker script for each device/processor combination, as some of our devices have up to 4 cores, all of which we'd like to support with our DFP.

@KarolSaja-AlifSemi
When a project specifies the linker: node with regions: and script: there should be no automatic generation of any script or regions files.
When My_M55_HP.cproject.yml contains the below nodes it should work.
  device: :M55_HP    # select core
  :
  linker:
    regions: .\RTE\Device\AE101F4071542\mem_config.h
    script: .\RTE\Device\AE101F4071542\linker_gnu.ld.src

mem_config.h could include another header file.  Is this how you have composed it?

@ReinhardKeil thanks for pointing this out, could be useful. For now mem_config.h/mem_regions.h will be a single header, without making things too complicated. However with this solution, there is still a problem of having to explicitly specify the name of the device/processor, as it makes up part of the path to where "config" attribute files are copied (like linker script, regions & etc.). This prevents us from having one generic cproject/cdefault template that suits all of our devices.
That path is already automatically included for the compilation step, can we also include it by default when linker script is preprocessed? Even better if we could add all the include paths that are defined in the project: scope (e.g. with components). Looks like a simple change to the .cmake templates included in the /etc. With this change we could simply write:
  device: :M55_HP    # select core
  :
  linker:
    regions: mem_config.h
    script: linker_gnu.ld.src

Thanks for the feedback.  I created this ticket #1719 and we will discuss the options today in the Open-CMSIS-Pack technical meeting (I will post the recording).  Feel free to join this meeting.
We should also take a look to your proposal regarding project templates.  There are already provisions to bring in templates in the IDE.  Could we work together on this?  Can you provide an example template that reflects your project structure?

@ReinhardKeil I pushed a draft of the template to my repo: https://github.com/KarolSaja-AlifSemi/alif_vscode-minimal
The proposal in the #1719 is fine, as long as we have a way to either know the path to the configs, or have a variable for the name of the selected device and processor.
We'd like to make this a generic starting point, where the only selection needed is the device/processor. Currently I hardcoded this for our top-end device. The goal is to eventually include these in the  section of the PDSC.
For the IDE we're using VS Code with Arm csolution plugin and Arm Environment manager, but currently there's no ability to change the selected device once the project is created (without modifying the .yaml directly).

Small update, regarding the following comment:

or have a variable for the name of the selected device and processor.

I forgot that these already exist (https://github.com/ReinhardKeil/cmsis-toolbox/blob/main/docs/YML-Input-Format.md#access-sequences), giving us a generic way to include device/processor specific regions header (or anything else in the config folder, for that matter):
  linker:
    - regions: "RTE/Device/$Dname$_$Pname$/regions_$Dname$_$Pname$.h"

documentation can be updated independently from tools.


Set up a location for elasticsearch 5 backups in s3

This will be used by the env sync script in puppet.

Trello card

Going to do this with snapshots instead, which means we need a more annoying S3 set-up.


Reorganization and name change

I'm currently using this script to build FlyEM's package stack, and I'd like to start recommending it to others when appropriate.  What do you think of the following proposal:

Reorganize the files into a proper python package with setup.py, entry_points, a conda recipe, docs, etc.  Heck, maybe even some tests...
Remove project-specific configuration files like ilastik-recipe-specs.yaml, etc.
Rename the package to something more generic.  Any opinions here?  Maybe something like, I don't know, build-recipes, or publish-recipes, or publish-stack, etc.

When renaming it, we should keep in mind what the actual use-case for this is.

Okay, how about publish-conda-stack?

we build everything in order ... Do we want to keep it that way ... Or somehow solve the dependency graph for all the packages specified and build them in order

I think changing this behavior is beyond the scope of what I have time for in the near term.  In the long term, I'm still a little wary of this.  Of course, Carsten and I originally considered performing an automatic toposort as you propose, but there are some potential pitfalls:

Circular dependencies are a potential issue.  OK, in practice, there is usually only one circular dependency in our stack, and it is handled in conda by this if statement I wrote a long time ago, so if we use their toposort() function, it should be okay.

If I'm not mistaken, then performing a provably-correct toposort requires obtaining the ENTIRE dependency graph for all packages in your stack -- even ones that aren't listed in your recipe-specs.yaml file.  Running conda search on so many packages takes time.  Maybe there is a better way to fetch the dependencies for lots of packages -- after all, the conda solver manages it -- but it would take some investigation.
It is tempting to just download the dependency info for the particular packages you're building, but that might not be sufficient in some cases.  Perhaps such cases are a bit contrived, but it's at least theoretically possible.  If a stack has dependencies A <- B <- C, but you only rebuild A and C, toposort() might emit A and C in an arbitrary order (unless you include B in the input).
OK, I think I've come up with a concrete example, if you care to indulge me.  :-) Suppose my stack consisted of dvid <-- pydvid, <-- ilastik.  And suppose dvid has a REST API to return the server name as JSON: {"server": "foobar"}, and ilastik reads the message via this line:
server_name = pydvid.fetch_server_info()["server"]

So far, so good.  But now suppose DVID changes its API.  Now it returns {"server-NAME": "foobar"}.  Since pydvid was just a pass-through, we wouldn't rebuild pydvid.  But both dvid and ilastik would need to be rebuilt.  Hence, this is the A/C scenario I mentioned above.  Such cases would be rare, but potentially confusing if they do occur.

Maybe a "best-of-both-worlds" solution would be to perform a check based on the packages the user is building, and emit an error if they appear NOT to be toposorted in the yaml file.  So, users are still required to toposort the inputs manually, but we'll help them spot mistakes.

Thank you for those very concise thoughts.

Maybe a "best-of-both-worlds" solution would be to perform a check based on the packages the user is building, and emit an error if they appear NOT to be toposorted in the yaml file. So, users are still required to toposort the inputs manually, but we'll help them spot mistakes.

Maybe this is the way to go! It's great that this functionality exists, already. And failure with some nice hint on where to spot it is already very nice to have.

OK, I think I've come up with a concrete example, if you care to indulge me. :-) Suppose my stack consisted of dvid <-- pydvid, <-- ilastik. And suppose dvid has a REST API to return the server name as JSON: {"server": "foobar"}, and ilastik reads the message via this line:

But, then, where does the pydvid package come from? Isn't this kind of pathological, already? I know, pydvid might be noarch and sourced from a different channel, but depends on some host specific package that one rebuilds for some different OS or something?
I just wanted to raise the point, that one might end up with a stack that doesn't necessarily reflect the intentions, like, building vigra against a specific version of hdf5 and not having the order right. This would be unnoticed (build might still pass, hdf5 is available somewhere else)...

Thank you for those very concise thoughts.

LOL, sorry.  You know by now that brevity difficult concept for me.   :-P

Maybe this is the way to go!

OK, cool.  Even so, I think I won't tackle this just yet.

But, then, where does the pydvid package come from?

I'm speculating that it's one of our own packages, but that it required no rebuild (none of its source code changed), so the user typed ./build-recipes.py ilastik-recipe-specs.py ilastik dvid

Isn't this kind of pathological, already?

Yes, perhaps I'm worrying too much over unrealistic scenarios.

Okay, so we'll keep it "linear", as is, in the beginning and just make the current version proper. Any idea on where to put this then? I guess the question is, whether to have it here in the ilastik org or at flyem? I'm fine either way. Do you have any time frame in mind, do you want to do it alone, or wouldn't you mind if I pitched in? I'm sure up for it

Just a progress update: I didn't have as much time to invest as I hoped and I also had to work on a cookiecutter first, to go with the project ;).
In any case, I've pushed the current status here: https://github.com/ilastik/ilastik-publish-packages/tree/rework
I mainly reorganized everything into a package with a proper main and a basic conda recipe. There is still a lot to do. However, when installed, one can already use the script after installing:
Example invocation:
publish-conda-stack ilastik-recipe-specs

I uploaded the current package to ilastik-forge. Install:
conda install -c ilastik-forge -c conda-forge publish-conda-stack


Define role names and use them consistently

e.g. "administrator" and "collaborators".  I have started this in the exercise setups (clearly define that leader is administrator, others are collaborators), but could be done further.  Each step could clearly specify with "done by collaborator" or "done by administrator".
I'm not sure if these are the best names.

I changed to "maintainer" and "collaborators". Also not sure these are best names.


Add PqJx9Value>>elementCount and test.

Fixed #testInjectJsonKeyValueExecuteGet. Added an assertion on the returned JSON array's element count.

errors in osx are not a real error (is something with the binaries)


Fix typos in docs\ide

@GitHubPang : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


added  "ctrl+enter" generate hotkey

hey!
I really missed this convenience hotkey and added it! what do you think?
that said, there are some things I was wondering, but wasn't sure about:

should it be restricted to only trigger when on the generate tab? if so, where do you track that?
is there any other conditions that this should take into account where it should not trigger? if yes, which would that be?

if so, please let me know and I'll add that. :+1:

Please match the formatting to the surrounding code (so don't move the else onto the same line as the }
you'd want isVisible(getRequiredElementById('main_image_area')) for the check
you should test specifically what happens if you hit CTRL+Enter while the prompt box is selected, as this already triggers a generation so that might lead it triggering twice at once if it's not properly blocked

Went ahead and applied those myself