Refactor of workflow submission modal

The V2 workflows seem to be very optimized for continuing workflows from where one has stopped. But another typical behavior is that a user reruns the whole workflow from the beginning.
That gets confusing, because there can already be an existing image list and existing filters that are set. For example, if a user runs a workflow that goes Convert OME-Zarr, MIP.
Then they want to rerun it, it fails. Because the is_3D filter was set to False and rerunning Convert OME-Zarr does not change the filter.
I haven't fully figured out what we'll need to handle this better. Here are a few points:

We'll need to expose what filters are already active when a workflow gets submitted.
I think we may need different concepts for rerunning a workflow from the beginning vs. continuing an existing workflow

Rerunning a workflow:

What happens with the existing image list?
What happens with the existing filters?

2 issues to address some of it by making filters more visible & editable:
https://github.com/fractal-analytics-platform/fractal-web/issues/441
https://github.com/fractal-analytics-platform/fractal-web/issues/442
Let's start with that and brainstorm how we handle full reruns of workflows based on that

brainstorming notes
Another part to consider here:
Does this depend on whether the workflow is run from the first task or an intermediary task?
Maybe we have different submissions in the web interface for that:

Initial run: Select dataset and until which task you want to run
Rerun workflow: Resets filters & image list, always starts from the beginning until the task that is set
Continue workflow (or better word): Start from a user selected task with the current filters [show them] until a given task. Could be from the task after the last successful task, or from an arbitrary task

I'm mostly thinking about this as a way the web portal offers different submission strategies (and calls one or multiple APIs depending on the selection), not as new backend features.

I thought about this some more and tested additional workflows. It's something we need to address. Here's a proposal on how we address it now (part 1) & where we could go in the future (part 2).

Part 1: What we should do now
We should split the submission into different categories, as outlined in the comment above:
i) No workflow has been run:
We do basically what we've done so far. But let's set the first task of the workflow as default "First task". (depending on what we do in iii, we could also make the "First task" something the user can't change anymore, it's just shown that the workflow will start there, as it's the first task in the workflow).

ii) Workflow has been run & I want to continue
This is how the submission currently behaves when a workflow has already been run. Let's rename the button to Continue Workflow if a workflow has already been run.

This just submits the workflow as we're doing so far.

Best-case, we would make the First Task default to
a) The task that failed if any task failed
b) the next task after the last one that finished successfully
c) The last task if there are no unfinished tasks (?)
The user can then change which task is chosen there, e.g. continue the workflow by rerunning a task that has already ran before.
This likely depends on https://github.com/fractal-analytics-platform/fractal-server/issues/1083 and may not be trivial. In that case, the defaults for "First task" can also be moved to a future goal (part 2)
ii) Workflow has been run & I want to start over from the beginning
This is currently not supported. One way we can support this:
Add a "Rerun workflow from the beginning" button. It has the following behavior:

It deletes the image list
It removes all filters
It submits the workflow from the first task

It should come with a big warning saying something like:

! Warning !
Rerunning a workflow from the beginning will remove all items from the image list and reset all filters. It will not remove any of the actual image data though.

Also, to avoid users using this option for the wrong thing (e.g. my task 7 failed, let me rerun the workflow), I'd say the "First task" should be selected as the first task of the workflow and be unchangeable (=> it just shows that it will start in the beginning with no option for the user to change that).
I'm open to different suggestions for naming the button, but quite convinced that we should have these 3 distinct modes.

Part 2: Goals we may achieve later
Potentially defaults for Continue workflow will go into this category
Also, continue workflow would benefit a lot from showing something like:
This workflow will continue with the following image list, these filters are set.
But we can add that feature later.

I think this is mostly something we can solve on the front-end, with 2 areas that would involve new server functionality:

Finding the right "default start task" for the Continue option
Easy way to delete the whole image list & remove all filters?

Minor comment:

It deletes the image list

It removes all filters

It submits the workflow from the first task

This list should also include the dataset history, right?
If not, it will appear as if some of the workflow tasks already ran, and they will have a done/failed status.

Another remark - cc @jluethi
In the scenario 1.iii (that is, where I want to restart from the beginning with an empty dataset), is there any value in emptying an existing dataset rather than just creating a new one?
If this is made to simplify the user flow (that is, to avoid that they need to go back to the dataset page), this feature can be embedded within the restart-workflow-from-the-beginning modal. The zarr-dir could be prefilled from the original one, and the dataset-name could be the same as before with a suffix. TBD

tl;dr: I think the dataset approach should work. Let me know what you think.
I'll sketch it in more detail tomorrow morning :)

I think the dataset approach can work like this:
If a user clicks "Rerun workflow from the beginning", they see a warning message that says:

! Warning !
Rerunning a workflow from the beginning will create a new dataset and optionally delete the old dataset with its image list, filters & history.

By default, a checkbox is clicked to have the old dataset replaced (e.g. delete the old dataset, create a new one with the same name as the deleted one & give that new dataset the same zarr_dir).

If the user unticks the Replace existing dataset box, a new entry shows below the dataset selector for "New dataset name", which defaults to the existing dataset name + suffix (e.g. _1, _2, _n+1).
If Replace existing dataset is unchecked, then the old dataset does not get deleted, but a new dataset with the same zarr_dir is still created

Here is our plan with @zonia3000.
Preliminary changes
These changes would make the following easier:

When no dataset is selected, and the "submit" button is clicked, show an error message like "you should first select a dataset".
Given 1, then we can make the dataset selection in the submission modal non-editable. Note: we still always display it (for any of the submission scenarios).

cc @jluethi are these two minor changes OK?
Has any job run?
Given the previous section, we know workflow_id and dataset_id.
We then call
GET /api/v2/project/{project_id}/status/?dataset_it={dataset_id}&workflow_id={workflow_id}

and if the response is {"status": {}} then we set HasAnyJobRun=false, otherwise it is true.
Note: if we hit any edge case here, we could switch to using GET /api/v2/project/{id}/workflow/{id}/job/
If HasAnyJobRun=false
We show a "submit" button, which opens a modal similar to the current one.
Differences:

The dataset is non-editable (see above)
The first task is set to match with the first task in the workflow (editable, for now)

If HasAnyJobRun=true
We show two buttons: "continue" or "restart"
If "continue" is clicked
We show a submission modal similar to the current one, possible with a pre-selected first task.
Let's postpone the definition of this first task to a different issue (ref XXX).
If "restart" is clicked
First, we show a big warning with text

WARNING: Rerunning a workflow from the beginning will create a new dataset and optionally delete the old dataset with its image list, filters & history.

Then the user can uncheck a "replace existing dataset" (checked by default)
If "replace dataset" is checked
Show original dataset name (non editable).
Store current dataset name and zarr_dir.
Replace dataset with new one. For the moment this should go through
DELETE original dataset
POST new dataset with same name and zarr_dir (and no other property).
(but maybe we'll introduce a specific PATCH in the future)
Update the currently-selected dataset ID in workflow-page and submission modal (same variable).
If "replace dataset" is unchecked
Show original dataset name (non editable).
Show "new dataset name" editable prompt, with a default (see below)
POST new dataset with new name and old zarr_dir (and no other property).
How to set the default for new dataset name?
Prefix is the current name.
Existing names are obtained via GET /project/{id}/dataset/.
We iterate over {prefix}_1, {prefix}_2, .. until we find one that doesn't exist yet.

Looks great overall!
A few minor comments:

When no dataset is selected, and the "submit" button is clicked, show an error message like "you should first select a dataset".
Given 1, then we can make the dataset selection in the submission modal non-editable. Note: we still always display it (for any of the submission scenarios).

Why don't we make the run button greyed out / unclickable then when no dataset is selected? With the work in https://github.com/fractal-analytics-platform/fractal-web/issues/445, no dataset selection will eventually anyway become rarer and we'd avoid having to have an error message dialogue
+1 on the making the dataset something the user can't change, but is still shown.

Flow for HasAnyJobRun=false: Sounds good! On the naming of the button: "Run workflow" seems clearer than "submit" to me.
Flow for continue: Sounds good, fine with postponing the first task selection choice. Let's call it "Continue Workflow"
On restart:
Let's call it "Restart workflow"
If "replace dataset" is checked => fine with that flow!
If "replace dataset" is unchecked:
Any good reason for using prefixes instead of suffixes? I like the logic for choosing them. Would have intuitively gone with suffixes, as a user could have e.g.
another_test_dataset
my_core_dataset
some_subset
In that case, adding prefixes groups them by prefix when sorted alphabetically, which isn't very meaningful. Suffixes would have them sorted by the core name

Why don't we make the run button greyed out / unclickable then when no dataset is selected?

That is also fine, but if we only make the button unclickable then we are not telling the user how to proceed. In that case do we need to add a warning message that is always displayed when no dataset is selected?

As for the renaming of different components -> sure, thanks. Our names were meant to be placeholders.

Any good reason for using prefixes instead of suffixes? I like the logic for choosing them. Would have intuitively gone with suffixes, as a user could have e.g.

Then I may have misuderstood the previous comment:

"New dataset name", which defaults to the existing dataset name + suffix (e.g. _1, _2, _n+1).

Let's make this more explicit:
If there exists a single dataset and it named "MyDataset", what is the default for the new name? My guess is that we want to set it to "MyDataset_1", but if you prefer "1_MyDataset" there is no problem.

That is also fine, but if we only make the button unclickable then we are not telling the user how to proceed. In that case do we need to add a warning message that is always displayed when no dataset is selected?

Given that in the near future, there will by default always be a dataset selected, I think that's not necessary. And the selector is right next to the run button.

If there exists a single dataset and it is named "MyDataset", what is the default for the new name?

My bad, I misunderstood. Yes, the dataset name is the prefix, the number is the suffix.

My guess is that we want to set it to "MyDataset_1", but if you prefer "1_MyDataset" there is no problem.

Yes, I'd want "MyDataset_1". Didn't read your message above carefully enough

Good idea. The warning message could be the one drafted above extended:

WARNING: Restarting a workflow will create a new dataset and optionally delete the old dataset with its image list, filters & history. It does not remove the existing Zarr data though. Either remove it yourself before rerunning or set the overwrite option to True in the corresponding tasks.

(The way users usually get around this when rerunning workflows in V1 is setting overwrite to True and that works robustly)

And the issue that rerunning a workflow without overwrite = True fails is something we already have and where we provide informative error messages that guide users to setting overwrite=True in many of our tasks

Run workflow button disabled if no dataset is selected:

If a previous job was run the 2 buttons "Continue workflow" and "Restart workflow" are shown:

Clicking on "Continue workflow":

Restart workflow with "Replace existing dataset" checked:

Restart workflow with "Replace existing dataset" unchecked:

Nice!
Very quick comment: maybe the "restart workflow modal should include a pre-selected first task, set to the first task in the workflow?


"last" file always parsed even if it was already done in incremental mode

Hello,
in incremental mode, when running again the same command on same file, the last one is always parsed again, even if no modifications were done :

Checking the script, the issue sounds related to this code : $more_lines++;  on line 4685
when the log file is not modified, the last timestamp equal the saved one in LAST_PARSED, therefore the test on line 4678 on 		if ($saved_date gt $prefix_vars{'t_timestamp'}) is nok which result in a final more_lines variable wrongly set to 1.
I can understand the fact we could have some new lines on the same timestamp but like the file size is exactly same than the saved one, we shouldn't consider here.
regards

Commit a95cc19 might solve this issue.


Fixes #83: Added Raspbian 8 testcases

I've tried adding a raspbian testcase. Let me know if I forgot anything

Hi Martijn,
this is wonderful!
If os-release is the only distro information file it has (i.e. no lsb_release command, no single-line distro release file), the tests are perfect. Thanks a lot!

These are the only files that exist in Raspbian 8


Feature Request for Druid Source Connector

Hi,
I'm looking for the functionality of Druid Source Connector, which will get the data from Druid and stream it in kafka topic.

@austindsouza We can look at it. First we have other connectors to finish. We can change the priority but we normally only do this on commercial terms. Send me a direct mail if you want to discuss this.

@andrewstevenson I am new to Confluient,  If i wanted to write druid source connector using Apache Calcite's jdbc driver what is the approach or what all things do I need?
Can you guide me?

Not on our radar so closing for now


[CHORE] revert the v3.0.0 that was triggered by error

python-semantic-release  has/had an error (besides ##285),
that triggered a release v3.0.0 when it should have triggered v2.7.1 ...
v2.7.1 was triggered manually
the remains of v3.0.0 need to be removed,
as an actual v3 will be released eventually.

[ ] remove the v3.0,0 tag
[ ] remove 3.0.0 from https://github.com/CycloneDX/cyclonedx-python-lib/blob/main/CHANGELOG.md
[ ] remove 3.0.0 from GH releases: https://github.com/CycloneDX/cyclonedx-python-lib/releases/tag/v3.0.0
[x] revert 3.0.0 from PyPI: https://pypi.org/project/cyclonedx-python-lib/

@madpah  FYI

Thanks @jkowalleck - had not realised all this other stuff had happened too!

Caused by #285


Allow comments & whitespace in gitignore files & fix issue where global ignore rules may not be written.

As outlined in my commits, I typically don't install Magento to htdocs relative to this project, I will install it directly to the same directory as Composer.
I usually have a well structured gitignore file that works well for me. It has comments and various things in it.
I refactored the GitIgnore class to allow for whitespace, comments etc without breaking the formatting.
I also fixed an issue whereby if you had directories that were ignored entirely that differed from the state of the gitignore file, it won't have been updated. Only a change in file entries could have previously triggered a rewrite of the gitignore file, not just ignore rules.

Hey, the third issue was intentional, was that causing you an issue?

OK I'm going to go ahead and merge this, thanks!


Display Configuration Profiles

Description of the new feature / enhancement
A feature allowing the user to save the current display settings as a profile, in a list of profiles.
The user can load any profile from the list.
Basically like "Monitor Profile Switcher", or "Display Fusion", but fancy and not as a stand-alone tool.
Scenario when this would be used?
I use my notebook on different Thunderbolt Docks around the place.
On some, i want

specific settings
my notebooks display to be active
my notebooks display to be disabled, despite the lid not being closed
...

Supporting information
No response

/dup #1052


swri_profiler: 0.2.1-1 in 'melodic/distribution.yaml' [bloom]

Increasing version of package(s) in repository swri_profiler to 0.2.1-1:

upstream repository: https://github.com/swri-robotics/swri_profiler.git
release repository: https://github.com/swri-robotics-gbp/swri_profiler-release.git
distro file: melodic/distribution.yaml
bloom version: 0.9.0
previous version for package: 0.2.0-1

swri_profiler
* Add repository url to package.xml
* Correct license tag in package.xml.
* Clean up CMakeLists
* Contributors: Isaac I.Y. Saito, Matthew Bries, P. J. Reed

swri_profiler_msgs
* Add repository url to package.xml
* Correct license tag in package.xml.
* Contributors: Isaac I.Y. Saito, Matthew Bries, P. J. Reed

swri_profiler_tools
* Clean up CMakeLists
* Add repository url to package.xml
* Contributors: Matthew Bries, P. J. Reed

Going to close this request, I think it needs another tweak first...


fix encoding problem in windows

Just a small fix in windows. Without it, the following error will throw on my computer when using "python setup.py install":
Traceback (most recent call last):
File "setup.py", line 27, in
license=open('LICENSE', 'r').read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0x93 in position 25: illegal multibyte sequence

Merged, thank you! 🎉


TC-BC-2.1 Binding Cluster - binding attribute showing wrong number of entries

Actual Behavior - TH writes two binding entries in which the response is SUCCESS but during read, the response is only one entry (first entry)
Expected Behavior - When TH writes two binding entries, the response should be SUCCESS and during read, the response should consist of two binding entries
App used - allclusters app
Platform - Chip-tool - RPI-4, 8GB RAM
DUT         - RPI-4, 8GB RAM
Network - Ble-wifi
Commit id - 425f1fb916ca154deeb75cd82cd5196778aba152
Steps to reproduce :
Write binding entry : ./chip-tool binding write binding '[{"node" : 1  , "cluster" : "0x0006" , "endpoint" : 1 }] , [{ "group" : "0x0001" }]' 1 0
Read binding entry : ./chip-tool binding read binding 1 0
DUT_Logs.txt
TH_Logs.txt

Write binding entry : ./chip-tool binding write binding '[{"node" : 1 , "cluster" : "0x0006" , "endpoint" : 1 }] , [{ "group" : "0x0001" }]' 1 0

This command is wrong.  It's trying to write two lists (which I would expect to end up in an error in the parser; will look into why that does not happen).  But in any case, the first list is what gets written.  Fixing the command like so:
 Write binding entry : ./chip-tool binding write binding '[{"node" : 1 , "cluster" : "0x0006" , "endpoint" : 1 }, { "group" : "0x0001" }]' 1 0

gives the right behavior, as expected.

@sethunk Did that command come from the verification spreadsheet?


Task 1 discussion

Who is doing task 1 in this group?

@JustinJaniszewski will do that

can confirm.

I will be doing task 2

Oh okay. @JustinJaniszewski  have you started ?

No I have not

@Mosaddeq107 Who is doing task 2. I am expecting some assistance from you guys?

@JustinJaniszewski have you done task 1? which file format did you choose?

I have started it. I'm doing a jupyter notebook file if thats what you mean.

The task 1 by g1 can be seen under task1 folder in the main branch of group 1 main repo.
feedbacks are appreciated.


feature: example code block copy without prompt

Is your feature request related to a problem? Please describe.

https://mkdocstrings.github.io/python/usage/configuration/docstrings/?h=example#show_docstring_examples
The example code shows here is:
>>> print("hello")
hello

When I copy the code by the click button, I copy all the code in it(with >>> ).
Describe the solution you'd like

After copy, we get print("hello") only, without the prompt >>> and the output, just as sklearn docs in here
Describe alternatives you've considered

Additional context

Hi @shenxiangzhuang, thanks for the request :slightly_smiling_face: We already track it here: https://github.com/mkdocstrings/mkdocstrings/issues/668. Closing!


Add CI

Lint succeeds, test will fail due to outdated Cargo.lock (decided not to fix it right away to demo how a failure would look like in Github UI)
You may need to allow or activate Github Action in the project to let it run :)

Ah thank you! I'll be making a v0.2.8 with the updated Cargo.lock shortly.


Adds NPM package file.

So the project can be installed using npm install github:tkrkt/reveal.js-elapsed-time-bar.

Thanks for the PR!
I have merged this.


Architecture parameter initialization

https://github.com/quark0/darts/blob/f276dd346a09ae3160f8e3aca5c7b193fda1da37/cnn/model_search.py#L123
Hi, quark.
I noticed that the architecture parameters are multiplied by 1e-3 when being initialized. Could you please explain the reason for this initialization manner?
Thanks!

I've also noticed this point. Probably it is just a random pick. You may try with other initialization such as Xavier etc.
Let me know if you try with different initialization.


Tray: bump version renamed to Red Hat OpenShift Local

This needs an updated version file in the project.


chore(release): 2.22.0

Version bump in package.json and package-lock.json for release 2.22.0

:tada: This PR is included in version 2.22.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


ARM64 R_AARCH64_CALL26 relocation breaks on rebase.

Version and Platform (required):
3.5
Bug Description:
When you rebase this binary, main()'s call to myAdd() is lost.

Steps To Reproduce:
Open this file:  test.o.zip
Observe main() is calling myAdd():
00000048  eeffff97   bl      myAdd

Note this call is targetted by a relocation:

Symbol index 6 holds the struct for MyAdd(). Now rebase the binary to 0x4000 and the call is destroyed:

Looking at the hex view, those bytes that comprised the bl got destroyed:

Expected Behavior:
The relocation should work correctly. The call should survive rebasing.

Fixed in 3.5.4489-dev


[CdkTable] expose _insertRow to derived classes

Bug, feature request, or proposal: proposal
What is the expected behavior?
_insertRow is visible to inheriting classes
What is the current behavior?
_insertRow is private
What is the use-case or motivation for changing an existing behavior?
In _insertRow the context is created (RowContext<T>).
The row context is very useful as a metadata storage for the row, storing information about the row that helps when we want to create features that require state.
For example, flagging if the row is currently being edited, which cell is edited. If the row is being dragged or not etc...
This is also true for a specific cell, metadata for it can be saved on the row context.
Is there anything else we should know?
Because _insertRow calls _renderRow, which is also private, it will require exposing _renderRow as well, which is dont think is such a bad thing.
But, if we want to avoid that, maybe it will be better to expose a new protected method, _createRowContext(data: T): RowContext<T>  that people can extend

@andrewseguin can this get re-evaluated? causing a lot of pain when trying to extend CdkTable to something more powerful.


bug: MongoDB Adapter does not use plural collection name by default

I encountered an issue with the Mongoose adapter where it does not automatically use the plural form of the collection name. When I try to access the Account model using await Account.findOne(), I receive an empty array
const accountSchema = new mongoose.Schema({
  id: { type: String, required: true, unique: true },
  userId: { type: String, required: true, ref: "User" },
  accountId: { type: String, required: true },
  providerId: { type: String, required: true },
  accessToken: { type: String },
  refreshToken: { type: String },
  expiresAt: { type: Date },
  password: { type: String },
});

export const Account = mongoose.model("Account", accountSchema);

Attempt to fetch an account:
const account = await Account.findOne();

I receive an empty array when accessing await Account.findOne()
When using the following configuration with BetterAuth:
const client = mongoose.connection

export const auth = betterAuth({
  database: mongodbAdapter(client),
  emailAndPassword: {
    enabled: true,
  },
  user: {
    modelName: "users",
  },
  session: {
    modelName: "sessions",
  },
  account: {
    modelName: "accounts",
  },
  verification: {
    modelName: "verifications",
  },
});

The problem is resolved, but it should ideally work without the need to specify the model name in plural form
I believe the adapter should default to using the plural form of the model name without requiring explicit configuration for the collection name

in mongoose yes: The first argument is the singular name of the collection your model is for. Mongoose automatically looks for the plural, lowercased version of your model name. Thus, for the example above, the model Tank is for the tanks collection in the database..
I personally did not use mongo directly, only through mongoose

Prisma Adapter is supported for MongoDB??


Add 2 new organizations to existing network

I'm working on a multi-channel setup where some organizations need to be added to multiple channels. Following is the high-level network setup:
network1 - org1, org2, orderer1 (1 peer each) - channel1
network2 - org3 (1 peer)
network3 - org4 (1 peer)

All networks are set up using netup as I don't want to install chaincode at the time of network init (even if I install something, I'll need to install more chaincodes once the network is running). However, I ran channelcreate,channeljoin,anchorupdate on network1 which has channel1. Also, orgjoin is used to connect network2 and network3 channel1 one after another. All steps run without any errors and I see channelquery including all 4 organizations.
The problem comes when I try to install chaincode. I'm unable to commit as I can't collect the 3 endorsements (4 organizations) only from 2 organizations in network 1. Following is the specific error and it seems this is related to issue #199
  The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'Layouts'

  The error appears to be in '/home/playbooks/common/peerselection.yaml': line 25, column 5, but may
  be elsewhere in the file depending on the exact syntax problem.

  The offending line appears to be:

    - name: Select a random endorser group
      ^ here

I suspect network3 was not completely/correctly added to channel1 as I never did a channelsign from org3. Hence, I also tried 3. New organization to application channel (Fast Way) steps mentioned in AddNewOrganization.md. However, I can't figure out how to do the step where it says "The signed file will have to be passed from one org to the other so that all the endorsements are gathered when channel update command runs." While I copied channel2_update_envelope.pb from network1 to network 2 I still can't sign it because channelsign try to rebuild it.
Also, note that if I only connect org3 to network1 (i.e., org1, org2) I can install, initialize, and invoke chaincode. The problem just comes when I add org4. Also, adding more peers for org1 and org2 isn't desirable as we are going to use this on K8s for a PoC. Further, I prefer not to downgrade endorsement policy from MAJORITY to ANY
Appreciate any tips on how to resolve this issue.

Looking into the problem further my vars/discover/channel1/simple_endorsers.json file is empty on all networks. I suspect this lead to the above error. Also, I wonder whether not calling anchorupdate after adding other 2 networks lead this issue. So I tried that. But now getting the following error:
  non-zero return code
  2021-08-23 10:00:35.183 UTC [common.tools.configtxgen] main -> INFO 001 Loading configuration
  2021-08-23 10:00:35.196 UTC [common.tools.configtxgen.localconfig] Load -> INFO 002 Loaded configuration: /vars/configtx.yaml
  2021-08-23 10:00:35.196 UTC [common.tools.configtxgen] doOutputAnchorPeersUpdate -> INFO 003 Generating anchor peer update
  2021-08-23 10:00:35.198 UTC [common.tools.configtxgen] doOutputAnchorPeersUpdate -> INFO 004 Writing anchor peer update
  2021-08-23 10:00:35.238 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
  Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'channel1': error authorizing update: error validating ReadSet: proposed update requires that key [Group]  /Channel/Application be at version 1, but it is currently at version 2```

Found the issue with anchorupdate. Apparently anchorupdate.j2 uses configtxgen -outputAnchorPeersUpdate which is deprecated. Also, it only works for the first update of the default channel creation. Fixed anchorupdate.j2 to use configtxlator instead. I'll do a pull request for this. Other issues to still unresolved.


Scalability: number of services in a cluster

Community Note

Please vote on this issue by adding a 👍 reaction
to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description
The total number of services in a cluster is 10k (as long as IP ranges allow).
References

gke-scalability-components-matrix

sum (kube_service_info{cluster="cluster-waw",location="europe-central2",project_id="gke-policy-demo-711"})


Not able to configure Azure B2C tenant as external identity provider in Identity server 4

Hi Team,
We are trying to configure Azure B2C tenant as external identity provider in identity server 4 developed using dotnet core 2.1. We added below code to the start up file. Also added button on login page and enabled external login from account controller.
.AddOpenIdConnect("AzureB2C", "Azure AD B2C", options =>
{
options.ClientId = "appId registered at Azure B2C app registration";
options.Authority = "https://b2ctenantname.b2clogin.com/tfp/directoryname/signinpolicy";
options.ClientSecret = "client secrete";
options.ResponseType = "id_token";
options.CallbackPath = "/signin-aad";
options.SignedOutCallbackPath = "/signout-callback-aad";
options.RemoteSignOutPath = "/signout-aad";
options.TokenValidationParameters = new TokenValidationParameters
{
NameClaimType = "name",
RoleClaimType = "role"
};
});
We also tried different patterns of the authority url for B2C tenant like belows.
https://b2ctenantname.b2clogin.com/tfp/directoryname/signinpolicy
https://b2clogin.com/tfp/tenantid/policy
https://b2clogin.com/tenantid/signinpolicy
https://login.microsoftonline.com/tfp/tenantId/signinpolicy/v2.0
https://login.microsoft.com/tfp/tenantId/signinpolicy/v2.0
Its not redirecting to the login page for the B2C.
Please share the how we can use B2C as external identity provider.

We can close this issue. Successfully configured Azure B2C directory user flows using below authority url
https://b2cdirectoryname.b2clogin.com/b2cdirectoryname.onmicrosoft.com/userpolicyname/v2.0


Cannot find stash file

When opening TQVault AE (latest version), I get this error for both the stash file and relic file. When the vault loads, both those tabs are missing. I am playing the mod Soulvizier and have enabled the custom mods and selected the correct mod.
My path names are all correct as it shows my character and items except those from the stash and relic storage. There also seems to be the issue that the items do not fit inside the storage boxes, what is causing this issue?
https://ibb.co/1K26BFq

Here is the error again. Also other errors I get when hovering over certain items.

Note the items being outside the boxes and being cut off also.

Here is the error again. Also other errors I get when hovering over certain items.
Note the items being outside the boxes and being cut off also.

I'm experiencing the same issue.  Downloaded the EE Test3 version.  Running a mod in the /custommaps folder.  Equipment and Storage Area work, Transfer and Relic disabled.  Yes, I have items stored in both tabs.

This looks like a relic of the original release of the game.  The on release TQ Save data structure in your saved games folder is like this (dashes [-] denote folder levels):
CustomMaps
  - Mod Folders = Each one contains the mod data for each mod you have installed.
SaveData
  - Main = Contains player character saves for the main story quest.
  - Sys
    - miscsys.dxb = Relic vault for main story
    - miscsys.dxg = Also Relic Vault for main story
    - winsys.dxb  = Transfer stash for main story
    - winsys.dxg  = Also Transfer stash for main story
    - Folders with names matching the folders in the CustomMaps folder.
      - miscsys.dxb = Relic vault for the specific mod that it resides in.
      - miscsys.dxg = Also Relic Vault for the specific mod that it resides in.
      - winsys.dxb  = Transfer stash for the specific mod that it resides in.
      - winsys.dxg  = Also Transfer stash for the specific mod that it resides in.     -
  - User = Contains player character saves for modded games

TQVault assumes this structure when creating the path to locate the transfer stash and relic vault for the mod.  If the folder structure is different, the lookup will fail.
It looks like the custom map code for this does not work even if the folder structure is correct.  For example the code to locate the transfer stash file:
public string TransferStashFile
		{
			get
			{
				if (IsCustom)
					return Path.Combine(MapName, "SaveData", "Sys", Path.GetFileName(MapName), "winsys.dxb");

				return Path.Combine(ImmortalThroneSaveFolder, "SaveData", "Sys", "winsys.dxb");
			}
		}

For a custom map named MyMod in the custom maps folder the path would resolve to:
...\Saved Games\Titan Quest - Immortal Throne\CustomMaps\MyMod\SaveData\Sys\MyMod\winsys.dxb
This path should not contain the transfer stash.
Are the transfer stash and relic vault working at all with mods in TQVault?

No they don't work with any mods I've used. Must be some way to activate them?

As I suspected.  It looks like the code to construct the path to the Transfer stash and Relic vault file locations is broken for mods.  Odd that no one complained before.  It's a fairly simple to fix, but it would require a fix in the code.
As a workaround, you could copy your Transfer Stash and Relic vault files to the path that TQVault is looking.  Keep in mind this will make 2 copies of the files and they would need to be copied back to the original location for the game to recognize the changes.

All good I will wait for the update, thanks for looking into it!

Same issue on android saves: it splits stash into 3 files:
0winsys.dxb
0winsys.dxg
1winsys.dxb
1winsys.dxg
2winsys.dxb
2winsys.dxg
TQVault can load only character with equipement but not stash.

@eccodolf This is mentioned in #380 and basically the mobile version has a different file format for the stashes.
On top of that, it also has a different naming convention for the files and possibly an internal system of keeping backup copies of the stash files.


ModuleNotFoundError: No module named 'pyngrok'

Hello,
after a few hours using fast-SD on google colab I got the following error:
ModuleNotFoundError                       Traceback (most recent call last)
 in <cell line: 6>()
4 import sys
5 import fileinput
----> 6 from pyngrok import ngrok, conf
7
8 Use_Cloudflare_Tunnel = False #@param {type:"boolean"}
ModuleNotFoundError: No module named 'pyngrok'
And now the error comes back every 30 minutes or so. Afterwards I need to start from scratch.
Any ideas how to fix it? Many thanks!

each time you start a fresh session, you need to run the cells from the top

each time you start a fresh session, you need to run the cells from the top

I got this error after I completed the code and then ran .
ModuleNotFoundError                       Traceback (most recent call last)
 in <cell line: 6>()
4 import sys
5 import fileinput
----> 6 from pyngrok import ngrok, conf
7
8 Use_Cloudflare_Tunnel = True #@param {type:"boolean"}
ModuleNotFoundError: No module named 'pyngrok'

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.

Copy the requirements cell here

Copy the requirements cell here

#@markdown # Requirements
print('Installing requirements...')
with capture.capture_output() as cap:
%cd /content/
!wget -q -i https://raw.githubusercontent.com/TheLastBen/fast-stable-diffusion/main/Dependencies/A1111.txt
!dpkg -i *.deb
if not os.path.exists('/content/gdrive/'+mainpth+'/sd/stablediffusion'):
!tar -C /content/gdrive/$mainpth --zstd -xf sd_rep.tar.zst
!tar -C / --zstd -xf gcolabdeps.tar.zst
!rm *.deb | rm *.zst | rm *.txt
%env LD_PRELOAD=libtcmalloc.so
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ['PYTHONWARNINGS'] = 'ignore'
if Use_Latest_Working_Commit:
!pip install gradio==3.16.2 --no-deps -qq
else:
!pip install gradio==3.23 --no-deps -qq
clear_output()
inf('\u2714 Done','success', '50px')

you're still getting the error ?

you're still getting the error ?

Not yet
If more comes up I'll post it.

I am still getting the error every 30 minutes or so. And I am starting from scratch every time, so I run all the cells from the top. Any idea why is it happening? Many thanks!

are you restarting the runtime ?

nope, I am not restarting anything, just the error above comes and doesn't go away until I restart everything completely from scratch

does the error happen during a running session ? you should know that everytime you start a new session, you'll need to run all the cells from the top

Dear Friend. Thank you for your work. But my program does not work, every 30 minutes it closes with an error. I have a paid version of Colab, but that doesn't help either. I have to delete everything from Google Disc each time and install again. Help me please.

ModuleNotFoundError                       Traceback (most recent call last)
 in <cell line: 6>()
4 import sys
5 import fileinput
----> 6 from pyngrok import ngrok, conf
7
8 Use_Cloudflare_Tunnel = False #@param {type:"boolean"}
ModuleNotFoundError: No module named 'pyngrok'

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.

does the error happen during a running session ? you should know that everytime you start a new session, you'll need to run all the cells from the top

I generate pictures, all is well and after 30 minutes a new generation comes out an error that the session is complete, I open Colab and there is the error I wrote above

a new generation comes out an error that the session is complete

what do you mean by "the session is complete" ?

a new generation comes out an error that the session is complete

what do you mean by "the session is complete" ?

possibly colab stopped the session ?

possibly colab stopped the session ?

yes and text
ModuleNotFoundError Traceback (most recent call last)
in <cell line: 6>()
4 import sys
5 import fileinput
----> 6 from pyngrok import ngrok, conf
7
8 Use_Cloudflare_Tunnel = False #@param {type:"boolean"}
ModuleNotFoundError: No module named 'pyngrok'

probably colab now is kicking free users after 30 minutes of SD

Try using nkgrok and see if the issue persists

I'm getting this error now too.  Paid colab, happens whether or not I add an ngrok token.  First time it's happened to me.

I'm in the same bucket as @buckwheaton and @Jamztradamus . I'm a paid colab user and this error just now appeared. I was successfully running the colab earlier today, deleted my runtime, came back about an hour or so, connected to a new runtime, ran all the cells from the top and once the colab got to the "Start Stable-Diffusion" cell:
ModuleNotFoundError                       Traceback (most recent call last)
 in <cell line: 6>()
4 import sys
5 import fileinput
----> 6 from pyngrok import ngrok, conf

@rayvinscreative fixed now, update your notebook

Awesome thank you!

it was the best way to increase the quality of the photo

Sr The Last Ben I have the same error a week ago and I can't find how to solve it, I already reinstalled the notebook more than three times, run it from another account and I get the same error. I saw that you told another user to install pyngrok (After searching a bit on the internet I found how to install it with CMD, but I'm not sure if this has something to do with the notebook)
If it's not too much trouble, could you explain the steps and where to install pyngrok?
Sorry for my English I'm using a translator

try using the latest notebook from the official link and let me know if the issue persists

I am using the google colab version
https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast_stable_diffusion_AUTOMATIC1111.ipynb
I can't find a newer version of that notebook
Thank you very much for answering

the link you gave is correct, try reproducing the error then do a screenshot

The first time I run the notebook it works perfect, but after generating 4 or 5 it stops without giving any error.
When I want to run the notebook again I get this error

In the part of the code the error is indicated there

that error happens because SD doesn't work with free colab anymore

Thank you very much for answering and for the advice

so if i buy colab pro will it work?

if i buy the colab pro will it work normally?

I acquired colab pro and the error continues :/

you need to run all the cells from the top

a new generation comes out an error that the session is complete

what do you mean by "the session is complete" ?

I received this message too... the link was generated and i could open de SD UI but a time after that ( 5-10min maximum) a notification like this appeared and a message like the one belowe appears when I try to restart SD... Could they be blocking Colab integration?
ModuleNotFoundError                       Traceback (most recent call last)
 in <cell line: 6>()
4 import sys
5 import fileinput
----> 6 from pyngrok import ngrok, conf
7
8 Use_Cloudflare_Tunnel = False #@param {type:"boolean"}
ModuleNotFoundError: No module named 'pyngrok'

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.

Thank you very much @TheLastBen, I managed to make some videos

that error happens because SD doesn't work with free colab anymore

Where can I pay to use it then?
Should I pay here for "Colab Pro"?
https://colab.research.google.com/signup/pricing?utm_source=resource_tab&utm_medium=link&utm_campaign=payg_learn_more

@Asrael88 yes, once paid, set the runtime to High-RAM in colab

@Asrael88 yes, once paid, set the runtime to High-RAM in colab

Now I need to understand how can I do that. But thank you anyway!

@Asrael88 in colab menu, go to runtime then runtime settings


tests: Limit the segments to be merged under tsan to suppress deadlock

What problem does this PR solve?
Issue Number: close https://github.com/pingcap/tiflash/issues/9257
Problem Summary:
The TSAN will failed when one thread hold locks larger than 64. StoreIngestTest.ConcurrentIngestAndWrite will try to do "segmentMerge" on many segments (more than 500), causing this problem.

The code in question is in lib/sanitizer_common/sanitizer_deadlock_detector.h
It limits the number of simultaneously held locks in a given thread to an arbitrary large number 64.
If you hold 65 locks in one thread at once, this will fail.

What is changed and how it works?

Limit the segments to be merged under tsan
Suppress a data race on xxh3

Check List
Tests

[x] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

/test pull-unit-test


iTorch Not Exiting on Ubuntu 14.04.2

I successfully installed iTorch on Ubuntu 14.04.2 (IPython 4.0.1) running on ec2 server, but however I can't exit itorch console after I run it (using itorch command)
The program just hangs. Nothing returns, not even an error log (wished there was one so I could understand what's happening)
The only way I exit itorch console is by killing the screen where it runs.
Can anyone help? I included an image of what's happening below

@tralpha To exit, you can either use Ctrl + D, or Ctrl + \


Add sorting support to internal links and snippets

This PR extracts a lot of functionality from the media content type (especially the sorting) into a new type in Husky, and adapts the internal links and snippets to this new type. Enables the sorting for these content types.
Depends on this Husky PR: https://github.com/massiveart/husky/pull/449
Tasks:

[x] test coverage
[ ] gather feedback for my changes
[ ] submit changes to the documentation
[x] added changelog line
[x] fix internal links storage (array of references instead of JSON string)
[x] fix snippet storage (array of references instead of JSON string)
[x] write migration script for internal link and snippet content type
[x] check https://github.com/sulu-cmf/sulu/issues/423
[x] check if preview is still working with sorting
[x] check automatic load on empty media
[x] add internal links without reloading the list
[x] check if multiple media assignments work in the same templates
[x] check #554

Informations:

Q
A

Fixed tickets
fixes #780 , fixes #423, fixes #554, fixes https://github.com/sulu-cmf/sulu-standard/issues/254

BC Breaks
none

Doc
none

I would like to update the way the internal links are stored. Currently the internal links also have a display option, which is actually not available at all. I would prefer a simple array instead, which would also give us some advantages, e.g. correct reference handling in PHPCR. But I am afraid that we would have to update the current installations, maybe an migration script should be added?
/cc @chirimoya

+1 for storage optimization, but only if a migration scripts will be provided which work fine ;-)

Of course it will work fine ;-P

@dantleech I have seen that you have been saving the snippets without any additional data as references anyway, so it really doesn't make sense to build this array with the ids-key in the code. The harder part to keep BC is the data in the content repository.
And I have got another question: I would have liked to save the internal links the same way, but it looks like the page is broken when you link a page to itself. We would have to prohibit that, but I would like to do that later... Would it be easy to change existing data from an simple array to an array of references?

@danrot ACK when new release of jackalope is ready!


Draft: FTL-7826 Return valid credentials

Return valid creds are returned if there is one corrupted for some users.

LGTM! small suggestion about tests only


Extract keymap to file.

Changing the key mappings requires recompilation. Extracting them to a file would make the system way more flexible.
Example:
00, Select, ENTER,
01, Up, ARROW_UP
02, Down, ARROW_DOWN
03, Left, ARROW_LEFT
04, Right, ARROW_RIGHT
0A, Options, C
0D, Exit, BACKSPACE
20, 0, 0
21, 1, 1
22, 2, 2
23, 3, 3
24, 4, 4
25, 5, 5
26, 6, 6
27, 7, 7
28, 8, 8
29, 9, 9
35, Display Information, I
44, Play, P
45, Stop, X
46, Pause, SPACE
48, Rewind, R
49, Fast Forward, F
51, Subtitle, L

Thoughts?

Interesting suggestion.
How would you envisage this working in:

development
installation

Note, this is an embedded system and there is no filesystem right now, so reading a file during runtime would require integrating more stuff.

Note, this is an embedded system and there is no filesystem right now, so reading a file during runtime would require integrating more stuff.

My reference before this was a full RPi with Raspbian which has a Linux kernel, filesystems, Kodi and an integrated Pulse-Eight compatible(?) interface.
 Users write the configuration, including mapping, on it and the system reads it. This would take this project from the developer realm to the consumer realm and could make it a commercial product. (Making it fully Pulse-Eight compatible would also be a giant step in the that direction.)
But for now take a look at littlefs or something similar. I have used jffs(2) on my Asus router in the past, but that runs a beefy kernel and has a lot of flash.

@dennizzzz Thank you for the considered comment, appreciated.
OK, I think I understand your motivation, allow me to offer some thoughts.

full RPi with Raspbian

Comparing to such a setup is misleading (but not incorrect). My 'other' HTPC is exactly this and indeed was my reference for what kind of 'experience' to replicate.

fully Pulse-Eight compatible

The Pulse-Eight HDMI CEC hardware cost is what drove me to create this project. One should note however that it presents as a special USB device requiring driver installation to function. pico-cec was deliberately created to emulate an HID keyboard so it would be "plug'n'play", I would like to retain that.
Furthermore, I wonder about possible legal issues with mirroring a commercial product.

the flexibility of configuration could even make it better

OK, I believe this is the real core of the issue.
Yes, it is inconvenient for an end user to install a full development environment because they want to map, say, 'red' button to the 'backspace'.
As you note, the useful configuration is just the mapping table, and we want to expose that. There are several options:

present as a USB mass storage device, offer a virtual filesystem and a file defining the mapping

this is possible, we can create a virtual FAT filesystem and file and allow the user to edit said file
this solution is challenging, it opens up a lot of vectors for instability due to the 'free' nature:

user tries to format the device
user misconfigures the file
user uploads garbage

with no UI, we have no way to help the user correct the issue

not totally true, we could blink morse code on the LEDs

present a serial console, offer a basic CLI to show and edit mapping

this seems the simplest, the user can open a serial terminal to pico-cec and is restricted by what they can do
the CLI doesn't need to be complex, all it needs is:

show - display the mappings
edit - edit/add a mapping entry

config can be validated

we have text UI now, we can report errors
this could even provide the ability to debug the CEC traffic or other aspects

use a PicoW

everything is wifi enable these days
we could migrate to PicoW (or ESP32) and present a simple web UI
this costs more and requires wifi configuration (eg SSID, key, etc)

My order of preference and ease of implementation:

USB serial interface
Pico W + web UI
USB mass storage + configuration file

Apologies for the long comment, it's a really interesting discussion.
Please let me know if I have misinterpreted anything.

@dennizzzz Thank you for the considered comment, appreciated. OK, I think I understand your motivation, allow me to offer some thoughts.

full RPi with Raspbian

Comparing to such a setup is misleading (but not incorrect). My 'other' HTPC is exactly this and indeed was my reference for what kind of 'experience' to replicate.

How can MY reference be misleading? To flatten it: a box that work very well with HDMI-CEC.

fully Pulse-Eight compatible

The Pulse-Eight HDMI CEC hardware cost is what drove me to create this project.

:+1: :+1: :+1: The P8 is becoming rare and so the price is going up even more.

One should note however that it presents as a special USB device requiring driver installation to function. pico-cec was deliberately created to emulate an HID keyboard so it would be "plug'n'play", I would like to retain that. Furthermore, I wonder about possible legal issues with mirroring a commercial product.

The device could be configured the HID-keyboard way (better for Windows) or the P8-driver-compatible way (better for Linux). If the P8 does not have a specific patent then there are no legal grounds for this.

the flexibility of configuration could even make it better

OK, I believe this is the real core of the issue. Yes, it is inconvenient for an end user to install a full development environment because they want to map, say, 'red' button to the 'backspace'. As you note, the useful configuration is just the mapping table, and we want to expose that.

:+1:

There are several options:
* present as a USB mass storage device, offer a virtual filesystem and a file defining the mapping

  * this is possible, we can create a virtual FAT filesystem and file and allow the user to edit said file
  * this solution is challenging, it opens up a lot of vectors for instability due to the 'free' nature:

    * user tries to format the device
    * user misconfigures the file
    * user uploads garbage
  * with no UI, we have no way to help the user correct the issue

    * not totally true, we could blink morse code on the LEDs

Blink twice ... :smile: .

* present a serial console, offer a basic CLI to show and edit mapping

  * this seems the simplest, the user can open a serial terminal to `pico-cec` and is restricted by what they can do
  * the CLI doesn't need to be complex, all it needs is:

    * show - display the mappings
    * edit - edit/add a mapping entry

      * config can be validated
  * we have text UI now, we can report errors
  * this could even provide the ability to debug the CEC traffic or other aspects

* use a PicoW

  * everything is wifi enable these days
  * we could migrate to PicoW (or ESP32) and present a simple web UI
  * this costs more and requires wifi configuration (eg SSID, key, etc)

My order of preference and ease of implementation:
* USB serial interface

Yes, this could work very well with an ncurses interface. (You duplicated 'serial' here :smile: )

* Pico W + web UI

Different hardware, big :-1:,  and you would need to configure something to be able to configure something else. But once that is out of the way, writing complex web UI's is part of my day job.

* USB mass storage + configuration file

This would be easiest for the user, not the developer.
I would like to propose two more options:

A dedicated configuration application ("pico-cec-configurator" with a web UI?) that takes a compiled version of pico-cec, lets you configure everything and then flashes the entire thing.
Pico W + bluetooth + phone app.

Apologies for the long comment, it's a really interesting discussion. Please let me know if I have misinterpreted anything.

None needed, yes it is, only the misleading part.

Hi all, loving this project and the ideas coming out of it.  If ok, I'd like to also throw in my ideas?
From looking at the code, currently it looks like the HID USB keyboard output only responds to known general CEC codes - If the CEC code isn't known would it be possible to output the CEC code string (even just as ascii characters) to the USB HID i/o?  This would be good (for devs at least) to be able to test CEC compatible remotes and buttons to implement support for various 'unused' (by this project, so far) buttons like the red, green, blue yellow etc. - potentially an addition I can work on (still wating for my Seeed RP2040s to arrive)
This potentially then may be used down the track (with something like dedicated configuration application mentioned above) to assist end users generate their own config.txt file with CEC commands to USB HID keyboard / kodi compliant commands, which is then ''baked into'' the rp2040 firmware which is uploaded to the device.

@gizmomelb Currently if you build the firmware from source you get a debug firmware file, if you flash that you get a serial interface instead of the HID output. if you open a serial connection and start pressing buttons you will see all the unmapped keycodes.
If we do get a HID + Serial firmware in the future it might also be nice to implement certain features like software resetting to bootloader mode through the serial interface to get around bad access to the button on the current case design.
I'd also argue against trying to make it only pulse-eight compatible (possibly a seperate firmware file or serial command setting) since those adapters don't let you input into your bios or multi-boot select screen.

@dennizzzz Long comments and replies just grow, so I'll just summarise the major thrust here quickly.
None of the suggestions are necessarily mutually exclusive or permanent.
Unlike commercial activities, we can make mistakes, try again and not get fired.
I am light on free time so obviously prefer the 'developer easy' paths.
If we avoid changing hardware, then wifi and bluetooth are out (for now).
Our options come down to:

a terminal interface
a mass storage device interface

These can coexist and thus should just be part of some roadmap.

I would like to propose two more options:

A dedicated configuration application ("pico-cec-configurator" with a web UI?) that takes a compiled version of pico-cec, > lets you configure everything and then flashes the entire thing.

I looked into Pico flashing from a web UI, it exists quite well for ESP32, but isn't there for Pico.
Ultimately, yes, this is the best way if achievable, maybe in future when easier/simpler ways to flash are created.

Pico W + bluetooth + phone app.

This would make sense if we did go PicoW route, let's keep it as a possible future.

@gizmomelb Currently if you build the firmware from source you get a debug firmware file, if you flash that you get a serial interface instead of the HID output. if you open a serial connection and start pressing buttons you will see all the unmapped keycodes.

Just be aware that this debug mode is prone to hang. I think the serial output from multiple tasks isn't properly mutex protected.
This would be fixed if the serial UI we are discussing is implemented properly.

If we do get a HID + Serial firmware in the future it might also be nice to implement certain features like software resetting to bootloader mode through the serial interface to get around bad access to the button on the current case design.

Agreed, I can imagine useful things like:

show and edit mappings
reset mappings to default
reboot
reboot to bootloader
dump/analyse CEC/HID traffic

I'd also argue against trying to make it only pulse-eight compatible (possibly a seperate firmware file or serial command setting) since those adapters don't let you input into your bios or multi-boot select screen.

I don't know enough about pulse-eight to make an informed comment, I defer to others.

I just enabled discussions for this project, which may or may not be a place for further discussions. I'm not fussed.

@gizmomelb Currently if you build the firmware from source you get a debug firmware file, if you flash that you get a serial interface instead of the HID output. if you open a serial connection and start pressing buttons you will see all the unmapped keycodes.

Just to add more, I believe both unmapped CEC messages and CEC user control messages are printed to serial.
NB. The output is not well formatted, it down right sucks, but I didn't have enough motivation to prettify it.

I just enabled discussions for this project, which may or may not be a place for further discussions. I'm not fussed.

I would advise you to disable that again for now and just be agnostic about the differences between a real issue, a discussion posted as an issue, an issue that turns into a discussion and a discussion that leads to an issue. (Just add a 'Discussion' tag (x)or have me prepend the title with [DISCUSSION]) You probably don't want entropy of information where you eventually have to look everywhere for information. This means you would also rather extend the README.md than creating a Wiki or a dedicated website until that just isn't feasible anymore.

I'd also argue against trying to make it only pulse-eight compatible (possibly a separate firmware file or serial command setting) since those adapters don't let you input into your bios or multi-boot select screen.

No, not pulse-eight compatible mode only but configurable of course.

If P8 compatibility would be supported, what would be the default mode? (P8, HID)

@gkoh Okay, it looks like we are landing on a terminal interface. What are the limitations/possibilities of that? Some more decisions to make ...

Real telnet-like interface or custom a client-server (PC-Pico) interface?
If  we go for a real telnet-like interface, what would be the UI type? (CLI or ncurses ...)
If  we go for a client-server interface what would be the client UI type? (CLI, ncurses, Web, ...)
If  we go for a client-server interface what would be the fatness / thinness of each side? (I would suggest thin server in this case.)
If  we go for a client-server interface what would the protocol type be like? (Binary <=> JSON, custom <=> existing, ... )

Note that this is YOUR project (for now) and that you should make decisions (/lean towards something) based on your experience, interests and your planned investment of time. "When you point in a direction and move in that direction, you are on a path to something happening." - Me.

@gizmomelb

... If the CEC code isn't known ...  buttons like the red, green, blue yellow etc. ...

AFAIK the meat of a CEC message is just a single byte containing the standard(-ish?) code of the button pressed. The entire discussion in this issue is HOW to configure the mappings of these codes to keystrokes (or series of keystrokes?), currently these mappings are statically compiled from code and we want to at least move away from that.
I could envisage some interactive/live learning mode where you are asked to press a button (Or highlight a row associated with a button pressed) and then select the corresponding mapping or create a new mapping.

@dennizzzz

If P8 compatibility would be supported, what would be the default mode? (P8, HID)

It would probably have to be a separate firmware file if it gets implemented at all from looking at the libcec library and the way it communicates over serial with compatible adapters like the pulse-eight adapter or the raspberry pi built-in adapter and the way it handles message queuing.

Real telnet-like interface or custom a client-server (PC-Pico) interface?

You mean a serial interface? It will probably just be ascii command > response messages and you can design how you want to interact with that however you want.
Look at how other projects use serial to set settings like tasmota.

I could envisage some interactive/live learning mode where you are asked to press a button (Or highlight a row associated with a button pressed) and then select the corresponding mapping or create a new mapping.

This should be relatively easy to do once the HID + Serial is up and running.
@gkoh Since it seems like this project is getting some traction it might be worth it to look into adding a code of conduct.

AFAIK the meat of a CEC message is just a single byte containing the standard(-ish?) code of the button pressed.

It depends on the CEC message.  ie: if we were wanting to display some feedback from the dongle (playback device 1) to the display device (display device 0) then we could send a CEC frame' like 40:64:00:74:65:73:74   and it would display ''test" using the display device's OSD  (created using this site - https://www.cec-o-matic.com/  ).
I feel this could be used as at the very least as a test diagnostic to display what different button codes have been pressed on the remote, so that the frame/button code can then be noted and be used to remap buttons.
Any potential 'unique' codes (ie: SONY being a great example of doing things in a non-standard fashion) could then be discovered and help the end user to map the codes to KODI functions.  My LG TV remote doesn't have a subtitle / text button on it, so mapping that for subtitle enable / disable would be nice - I can see in your code submission you're using CEC frame command 51 to enable subtitles - that is PIP (Picture in Picture), which my remote also doesn't have.. so it won't work for all devices.

The entire discussion in this issue is HOW to configure the mappings of these codes to keystrokes (or series of keystrokes?),
currently these mappings are statically compiled from code and we want to at least move away from that.

Agreed.  But since all remotes don't have all functions we need to be able to first display the CEC code for each button that may be remapped on screen, so at the very worst the end user can write down the button code number, then use a separate app to generate the mapping config file which the dongle can then interpret.

I could envisage some interactive/live learning mode where you are asked to press a button (Or highlight a row associated with
a button pressed) and then select the corresponding mapping or create a new mapping.

See above.
I've finally received my Seeed rp2040s so I'm hoping to spend some time tinkering this coming weekend.

@gizmomelb About the CEC messages, yes they can be longer, but I was talking about the messages related to button presses. And furthermore, they might all be possibly great ideas but @gkoh first needs to make some decisions about where to go next.

See #13 for the most basic foundation to support the features discussed here.
Now we can interact with the system, I will slowly start to add in the pieces.
For this issue (configurable user keymap):

CEC user control messages are limited to a single byte, so we can easily hardcode the entire 256 entry map to NVS
to make it useful, the next steps are:

wrap the keymap in a structure with version header and checksum
add ability to initialise/read/write the keymap to NVS
add CLI support to display the keymap
add CLI support to initialise/modify the keymap

@gizmomelb Hope you were able to get your rp2040s going.
I'm going to split out the vendor support to a new issue. Unlike the user control keymap, vendor commands don't appear to be strictly defined.

with serial output enabled I've seen a youtube video where the user wrote data to a (MacOS) desktop, so data could be exported from the rp2040 to an external desktop app like that.. or reading and writing keymap settings to a fat32 file system with SDcard could be added like as shown here - https://youtu.be/JrYT7aJnP_I?t=610

@gizmomelb What problem are you trying to solve here? AKAICT @gkoh is working on full serial communication with a CLI. Why would we need to read/write from/to an SD card? One of my suggestions was to emulate a USB stick. Is this related perhaps?

@gizmomelb What problem are you trying to solve here? AFAICT @gkoh is working on full serial communication with a CLI. Why would we need to read/write from/to an SD card? One of my suggestions was to emulate a USB stick. Is this related perhaps?

Yes, sorry I was thinking aloud for an easy solution to write/read remote control and keyboard bind configs for end user.   Potentially using serial communication may not be easiest for end users (also I am thinking now about binding key configs to different keyboard keys for Kodi / Jellyfin client / VLC / MPC-HC support).  all outside the scope of this project at the moment.
I am also thining about network / Wake on LAN support to suspend and wake up end user PC - again, all outside the scope of this project.

@gizmomelb No apologies necessary. I just wondering where you were going with that.
I was thinking about how a Kodi plugin for configuring the pico-cec might work. Since the up/down/left/right/enter buttons are standard (correct?) you could configure the rest from within Kodi. (If Kodi is allowed to open a serial connection of course.)

hey all.. ok I've re-read this thread and here are my observations and thoughts.
GKoh's initial vision was for a plug and play device which would mimic a usb keyboard to output key pressess in response to specific HDMI CEC commands.  This also allows the dongle to control other media player apps besides KODI.
Therefore we need to be able to store the (eventually) user defined key presses somehow on the RP2040 without it being encoded into the compiled uf2 file - so the file needs to either be able to be stored in the 2MB FLASH on the RP2040, OR needs to be accessible another way on boot (without the RP2040 being connected to a PC) if the RP2040 loses power by being unplugged or the host device being switch off etc.
FUTURE STUFF: As an aside - the VCC and GND pins on the underside of the Seeed XAIO RP2040 can be connected to HDMI pins 17 (GND) and 18 (+5V) to continuously power the RP2040 even when it is not drawing power from USB.  This could be useful with some extra investigation I am doing with powering down the host pc and using WAKE ON LAN to start it back up - useful for the HDMI CEC power on / power off along with the TV and AVR etc.)
If we can read the config file from RP2040 FLASH, then serial over USB is fine for storing the config file on the RP2040.  However if we cannot easily store the config file in the flash, then may I suggest using SPI and an SD CARD as a way for the RP2040 to read the config file at boot time?
The data format could be like dennizzzz suggested in the first post and essesntially be a lookup table (referenced in the HDMI-CEC.C code loop) where the CEC command is looked up and then the second data lot is output to the USB keyboard.  We may also need a value (or value pattern such as &FFFF) to distinguish between values in the 'list' / signify end of data.
eg:first value is CEC code value, followed by the HID_KEY command/string
x00,&FFFF,HID_KEY_ENTER command,&FFFF
x01,&FFFF,HID_KEY_ARROW_UP,&FFFF
x02,&FFFF,"text string to send via HID keyvoard",&FFFF
FUTURE STUFF: With development this may potentially allow ''macros'' to be run where a HDMI CEC command could be configured by the user and the configured keyboard output is text string which could be used to launch a program, shutdown a pc etc.
So to me the main thing seems we need to define the data format for the key configuration and how it is implemented
So my newbie code loop would be something like:
Initialisation:  (system is fresh booted)
look for config file (in FLASH, or on SD CARD after initialising ports)
if config exists readfile
flash LEDS RED to indicate to user no config file found
STOP
readfile: open file and read contents to RAM, maybe flash the LED GREEN so user knows successful data read
close file system (if appropriate) and set MOSI free so it can be used in main loop for CEC
main CEC loop then runs
sorry, gotta go.. just some thoughts and trying to get back on track.

ok found this article about writing to the flash - https://www.makermatrix.com/blog/read-and-write-data-with-the-pi-pico-onboard-flash/

ok found this article about writing to the flash - https://www.makermatrix.com/blog/read-and-write-data-with-the-pi-pico-onboard-flash/

Thanks.
I have an offline branch with basic flash access, so this will certainly be the way forward.
RP2040 flash access requires interrupts to be disabled before access to prevent corruption.
To make it simple and robust I'm thinking to:

store configuration in flash
load configuration from flash into RAM during boot
to reconfigure:

create a copy of the running configuration
allow user to modify said copy
when user saves changes, suspend everything, write copy to flash and require a reboot

the CEC task is completely IRQ driven and needs to be stopped before accessing flash

FUTURE STUFF: As an aside - the VCC and GND pins on the underside of the Seeed XAIO RP2040 can be connected to HDMI pins 17 (GND) and 18 (+5V) to continuously power the RP2040 even when it is not drawing power from USB. This could be useful with some extra investigation I am doing with powering down the host pc and using WAKE ON LAN to start it back up - useful for the HDMI CEC power on / power off along with the TV and AVR etc.)

I did originally look into dual-power (VIN + USB-C) and it was unclear if the power protection was sufficient to prevent smoke. It will interesting to see what you find.
Responding to CEC 'power on' actually adds a third category of command mapping, so the three are now:

User Control buttons (eg. navigation, numbers, etc)
Vendor Command (#14)
HDMI CEC 'primary' messages, eg:

on CEC 'Standby' send keystroke to suspend PC

also having even a basic checksum when saving / loading the config will help avoid any corruptions.  I'm not an experienced programmer but I'm willing to have a go, especially the stuff outside of the main loop.  That way I hopefully can't stuff up too easily :)

@gizmomelb

Therefore we need to be able to store the (eventually) user defined key presses somehow on the RP2040 without it being encoded into the compiled uf2 file - so the file needs to either be able to be stored in the 2MB FLASH on the RP2040, OR needs to be accessible another way on boot (without the RP2040 being connected to a PC) if the RP2040 loses power by being unplugged or the host device being switch off etc.

There still has to be a statically compiled default/base configuration.
Reading about wear and tear of flash memory, I suggest we split up what we write into:

A, an address block of 256B and the last block of flash memory. This will be written on first initialization and then only when C gets corrupted.
C, a configuration block of 8KB, let's assume the complete configuration will fit in that. It will be located just before A or any previously corrupted configuration blocks.

Upon booting pico-cec reads A to get the address of C. A should start with a 4 byte magic number to identify it as an address block. If A does not have the magic number we do not read C and we revert to the statically compiled configuration. If A does have the magic number we parse the configuration from C. When we change the configuration it gets prepared for writing to C, a CRC is calculated and put at the end. When the configuration is written to C, we read back C but NOT as configuration, recalculate the CRC, and match with the previously written CRC. If the CRC does not match then C is corrupt, A will be changed to point to an 8KB lower address to a new C and the prepared configuration block will be written to C and rechecked with CRC. Two(?) consecutive write fails indicates flash failure. How to deal with that?

If we can read the config file from RP2040 FLASH, then serial over USB is fine for storing the config file on the RP2040. However if we cannot easily store the config file in the flash, then may I suggest using SPI and an SD CARD as a way for the RP2040 to read the config file at boot time?

No. The pico-cec needs to work stand-alone. So we don't read/write a file, we read/write blocks of flash. The way flash works does impact the flow of pico-cec. We can't record a button and write it to flash immediately. We need to record all the buttons to RAM and then write the configuration as a single block. Other than that your suggested loop looks reasonable.
I also think that there should be something like the Konami code to do a "factory reset" with the remote control.
Ah, I see now some of our comments have crossed each other, we appear to be in sync on most of this.


Define GatewayClient when starting JLD

Description
How can I define?
-GatewayClient.url
-GatewayClient.headers

When launching JL desktop.
Reproduce
N/A

Expected behavior
N/A
Context
I want to connect external kernels using: https://github.com/GoogleCloudPlatform/notebook-kernels-mixer
Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the terminal inside of JupyterLab-Desktop here.
You may want to sanitize the paths in the output.

Browser Output (required for interface issues after loading)

Paste the output from the DevTools JavaScript console here, if applicable.

Logs (required for issues with loading, server and startup)

Paste the logs from the `main.log` file here (optionally also from `renderer.log`).

@gogasca customizing server launch arguments is not supported yet but I plan to add it soon.


Top bar layout broken

On mobile device, CSR logo is centered and Run button is absent.

This is a duplicate of #6 .


Update Steam.php

openid is now https

yes, since steam switched their openid to https, validating auth will fail, so users won't be able to sign in via steam

It works for me. If you want, you can make the preg_match https? so it still works if for some odd reason valve decides to switch again.

Nah, I'd find it highly unlikely that Valve would revert that type of change. Thanks for the PR. :heart:


Update the     README.md file to describe the     repo.

Another good practice is to have a descriptive title and description to your prs. So right now it says  Update the README.md file to describe the repo but that's not what the pr is doing.

Also it's always a good idea for front end changes to include a screenshot of the change you're making to the front end so it's more visible to the reviewer