model/views: Add settings support for "playground" feature

This PR adds the database model for configuring realm playground settings and the API endpoints for POST and DELETE.
Testing Plan:
Backend tests added and manual testing done.
GIFs or Screenshots:

TODO
(Things mentioned in #16432 and more)

[ ] Build a configuration UI (like the Linkifiers UI).
[ ] Figure out how to handle the default playgrounds for an organization.
[ ] Investigate what services like repl.it exist and contact a few for permission when thinking about what defaults to support.
[ ] Document disabling the feature for organizations who don't want a button that sends code to a third party.
[ ] Document the feature on /features, /for/open-source, /help/format-...-markdown, etc.

Fixes: #16432

@timabbott Opened up this WIP PR for some early feedback :)

@timabbott Pushed a bunch of changes here. It's ready for another round of review.
A couple of points before reviewing:

I'm not 100% sure of the import_realm changes above and you may want to review that in more detail.
Do let me know if there is a better way to organize what change goes into which commit/split into further commits.

OK, I just posted a batch of comments that cover I think everything that'll need to change for the backend for this feature to be merged.  A few other notes:

Can you open issues with zulip-terminal and zulip-mobile for this feature, with the "api migrations" label?  That'll give them a heads-up that this is a feature they may want to integrate.  (Probably not a high near-term priority for either).

Sorry for the slow review on this!  I think we probably only need one more round trip on this PR.  (Probably worth just doing a new PR for the frontend, since this is a pretty complete self-contained backend PR).

It looks like CI is failing in the OpenAPI tests.
As a sidenote, I think some of the OpenAPI documentation changes in the last commit belong in the previous couple commits (the ones adding new API endpoints); also the final commit should update changelog.md with a few API_FEATURE_LEVEL etc.

@timabbott Thanks for giving this a detailed review!  I have pushed an update that addresses each of the comments you left. Find a changelog of this version compared to the previous, below:

Renumbered the migration
Added the API documentation for both the endpoints (along with python & curl examples); I caught a small typo when reading the "Documenting REST API" guide and fixed it in the first commit :)
The RealmPlayground model name is now a TextField name: str = models.TextField(db_index=True)
Added a comment to the do_add_realm_playground function on how we handle input validation.
I removed the no-coverage comment. We in fact didn't have any test which covered the ValidationError codepath in the view, so I added a test case for this (see RealmPlaygroundTests.test_create_already_existing_playground ).
check_string is now check_capped_string which uses RealmPlayground.MAX_PYGMENTS_LANGUAGE_LENGTH
Added a comment about not restricting the language to be from the list of Pygments languages.
Moved the json_success after the ValidationError
The "Add a playground" codepath now passes the parameters as a kwargs to its do_* function.
Thanks for catching the security pitfall! I have changed delete playground workflow to first do an access_playground_by_id check and fetche the realm playground object, later passing it to the do_remove_* function.
I removed hardcoding id=1 and instead using last_id in the events tests, based on what you suggested.
Lastly, I updated the changelog and bumped the API_FEATURE_LEVEL.

(btw I'll opening up issues in the terminal and mobile repos as you mentioned and will link to this from them)

Merged, with the following changes:

Added an id field to the RealmPlayground dictionaries -- we need to include that field so that clients have the ID to pass to the DELETE endpoint.  (And made appropriate documentation changes).
A few small edits to the API documentation.  We should make a note to add links from the API documentation to the Help Center to explain what this feature is once we have the UI finished and thus Help Center docs for it.

Thanks @sumanthvrao!  It'll be nice to have this API change finished.  I think the next step is hooking things up so that playgrounds configured using this system are actually used on the frontend, and then the final step will be adding the "organization settings" UI for configuring them, which can hopefully crib a bit off the linkifiers page (though that one is certainly not a paragon of quality; see https://github.com/zulip/zulip/pull/17045 and https://github.com/zulip/zulip/pull/17596).
We'll need to be a bit careful in doing the frontend work to avoid breaking the hardcoded ones we have; I think a good theory for v1 is to just use the hardcoded set if the realm_playground dict is empty, and then plan to remove that code in the commit that makes actual UI for creating these.


TODO: Create a second JS file explicitly for the page with the search form/landing page. [I think this is done with the start.js file already, just needs to be linked]

Done, moved all files into the suggested folder structure (Assets folder) and linked the files in each


ENH: Allow to_tree to render a literal.

So that input literals which are not bound to a data resource, can be sent to
the server. e.g. a frozenset which is the _keys of an IsIn statement.

Coverage decreased (-0.2%) to 87.188% when pulling 4e1f767c8a7632a68880bfe02ceec6afb6803ce2 on quantopian:to-tree-supports-literal-inputs into f9c83148a906ac1d751c713b16cce28c7f55d999 on blaze:master.

Coverage decreased (-0.2%) to 87.188% when pulling 4e1f767c8a7632a68880bfe02ceec6afb6803ce2 on quantopian:to-tree-supports-literal-inputs into f9c83148a906ac1d751c713b16cce28c7f55d999 on blaze:master.

Coverage decreased (-0.002%) to 87.377% when pulling db0b6fbe3aac110fc8ce5aba2b0e04d195bd33ab on quantopian:to-tree-supports-literal-inputs into f9c83148a906ac1d751c713b16cce28c7f55d999 on blaze:master.


Long Dockerfile command needs to be splitted

Docker file install needs to be splitted in multiple commands. As it is now it is hard to read.
https://github.com/kubenow/KubeNow/blob/master/Dockerfile#L26-L61

Yes but otherwise it creates more layers and get substantially larger... In the future Travis might support the new (currently experimental) command docker --squash

Docker squash is now in Travis: https://github.com/travis-ci/travis-ci/issues/7380

Unfortunately it is still part of the "experimental" features not enabled on travis - I did try to enable it with env-vars but never got it to work..

Is there any thread we can follow, so we kwon when it will become available?

Well, if you want to have your stuff readable and in one RUN, simply follow the dockerfile best practices. There is no need to split it and squash it.

Pack the arguments into a new line (and if the arguments are a set, sort it!)
I usually add true as the first and last command. This allows me to insert commands at the beginning/end without touching the previous first/last command and not polluting the git history

I'm submitting a PR. Give me some time pls.

That's a good remark 🙂

So, I also reviewed this part of the dockerfile (But I'm not touching these with the PR):

Also, there should be an export DEBIAN_FRONTEND=noninteractive in every RUN statement involving apt. (Do not export it in an ENV).
The multiple ENVs should also be grouped together.
If you install apt-transport-https, I think you should also add the repository with https instead of http.
As you're downloading from https, it might be also good to make sure, that ca-certificates package is installed and up to date.

@mcapuccini Thanks for merging #312. But KIM the other points, I also have noted in my previous comment here.

Yep, reopening the issue and renaming it 🙂


fix: added addendum for participant amount selector

Description :sparkles:
Added an addendum for participant selector to remind users to click update button when increasing or decreasing participant count.
Issues :bug:
Closes :no_good_woman:
LINK-2211):
Related :handshake:
Testing :alembic:
Automated tests :gear:
Manual testing :construction_worker_man:
Screenshots :camera_flash:
Additional notes :spiral_notepad:

LINKEDREGISTRATIONS-UI branch is deployed to platta: https://linkedregistrations-pr199-ui.dev.hel.ninja :rocket::rocket::rocket:

e2e tests result is success for https://linkedregistrations-pr199-ui.dev.hel.ninja :laughing::tada::tada::tada:


Make parrots talk like parrots

About the PR

In the rare cases that parrot become controlled by players, they now sound a bit more parrot-y.

Added the parrot squawk sound as a speech sound, so they no longer speak silently.
Gave parrots a speech verb set, so they "squawk", "tweet", and "chirp" their messages, instead of just "say"-ing them.
Parrots can "scream" now (which just plays the squawk sound).
Parrots have their own accent that gets applied to everything they say! It adds some random squawks to the beginnings and ends of messages. Sometimes they repeat the longest word in a message. RAAWK! Sometimes!

Why / Balance

There are a lot of animals, but very little that sets them apart from each other. This makes parrots a little more interesting (though there's plenty more to be done).
And no, I've never seen a player-controlled parrot in the game either. But it could happen! And when it does, now it'll be cooler.
Technical details

Added a few components to the parrot entity prototype, and made EmoteSounds, SpeechSounds, and SpeechVerb for parrots.
Added ParrotAccentComponent/System, which handle the accent.
Media

https://github.com/space-wizards/space-station-14/assets/85356/a863f384-8244-4d03-b9bb-2c0394b44488
Sorry for the lack of audio; blame Apple. There are squawking noises whenever the parrot chats or screams.

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

:cl:

add: Parrots now sound more like parrots when they talk. RAWWK!

parrot could use a movement sprite set and regular (stationary) one, so it isnt constantly flying.


[Torch] Cannot perform shape inference for cuda only operations (such as cudnn)

Doing something as follows crashes.
    nn = """
    (function()
        mod = cudnn.SpatialConvolution(%d, %d, %d, %d, %d, %d, %d, %d, 1)
        mod = mod:cuda()
        return mod
    end
    )()"""%(nInputPlane, nOutputPlane, kernel[0], kernel[1], stride[0], stride[1], pad[0], pad[1])
    return mx.sym.TorchModule(data_0=data, lua_string=nn, num_data=1, num_params=2, num_outputs=1, name=name)

torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:33: Only Cuda supported duh!

It is an issue with shape inference not knowing what device to run the operation on.

shape inference is only run on cpu.
An easy solution is to add a flag in TorchModeleParam to specify if you want shape inference to run on gpu, but this own allow selecting which gpu to run on.

I was thinking something similar. I will look into this when I get a few moments free.

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!


Expected one of: av1, hevc for compression but received undefined of type undefined

Hi
In version 2.2.1 i get an error when loading an image:
[2024-04-11 23:30:20.440] error: Expected one of: av1, hevc for compression but received undefined of type undefined
Error: Expected one of: av1, hevc for compression but received undefined of type undefined
    at Object.invalidParameterError (/home/l/dev/landing/service/cms/node_modules/.pnpm/sharp@0.33.3/node_modules/sharp/lib/is.js:135:10)
    at Sharp.heif (/home/l/dev/landing/service/cms/node_modules/.pnpm/sharp@0.33.3/node_modules/sharp/lib/output.js:1069:16)
    at sharpAddFormatSettings (/home/l/dev/landing/service/cms/node_modules/.pnpm/strapi-plugin-image-optimizer@2.2.1_@strapi+core@0.0.0_@strapi+strapi@4.22.1_@strapi+utils@4.22.1_yup@1.4.0/node_modules/strapi-plugin-image-optimizer/dist/server/services/image-optimizer-service.js:82:10)
    at resizeFileTo (/home/l/dev/landing/service/cms/node_modules/.pnpm/strapi-plugin-image-optimizer@2.2.1_@strapi+core@0.0.0_@strapi+strapi@4.22.1_@strapi+utils@4.22.1_yup@1.4.0/node_modules/strapi-plugin-image-optimizer/dist/server/services/image-optimizer-service.js:52:21)
    at generateImage (/home/l/dev/landing/service/cms/node_modules/.pnpm/strapi-plugin-image-optimizer@2.2.1_@strapi+core@0.0.0_@strapi+strapi@4.22.1_@strapi+utils@4.22.1_yup@1.4.0/node_modules/strapi-plugin-image-optimizer/dist/server/services/image-optimizer-service.js:44:21)
    at /home/l/dev/landing/service/cms/node_modules/.pnpm/strapi-plugin-image-optimizer@2.2.1_@strapi+core@0.0.0_@strapi+strapi@4.22.1_@strapi+utils@4.22.1_yup@1.4.0/node_modules/strapi-plugin-image-optimizer/dist/server/services/image-optimizer-service.js:28:38
    at Array.forEach (<anonymous>)
    at /home/l/dev/landing/service/cms/node_modules/.pnpm/strapi-plugin-image-optimizer@2.2.1_@strapi+core@0.0.0_@strapi+strapi@4.22.1_@strapi+utils@4.22.1_yup@1.4.0/node_modules/strapi-plugin-image-optimizer/dist/server/services/image-optimizer-service.js:27:15
    at Array.forEach (<anonymous>)
    at optimizeImage (/home/l/dev/landing/service/cms/node_modules/.pnpm/strapi-plugin-image-optimizer@2.2.1_@strapi+core@0.0.0_@strapi+strapi@4.22.1_@strapi+utils@4.22.1_yup@1.4.0/node_modules/strapi-plugin-image-optimizer/dist/server/services/image-optimizer-service.js:26:13)

I think this is due to the fact that when sharpInstance is initialized, the compression parameter is not passed for heif option: https://github.com/marlokessler/strapi-plugin-image-optimizer/blob/26a3212c70d4714859261574cb4c696b8cedcee0/server/services/image-optimizer-service.ts#L128-L138
Sharp documentation: https://sharp.pixelplumbing.com/api-output#heif
I use the following plugin configuration:
'image-optimizer': {
  enabled: true,
  config: {
    include: ["jpeg", "jpg", "png"],
    exclude: ["gif"],
    formats: ["original", "webp"],
    sizes: [
      {
        name: "xs",
        width: 300,
      },
      {
        name: "sm",
        width: 768,
      },
      {
        name: "md",
        width: 1280,
      },
      {
        name: "lg",
        width: 1920,
      },
      {
        name: "xl",
        width: 2840,
        withoutEnlargement: true,
      },
      {
        name: "original",
      },
    ],
    additionalResolutions: [1.5, 3],
    quality: 70,
  } satisfies ImageOptimizerConfig
}

Same issue here.


Tab onclosing event

I would like to let user to confirm before the tab is really closed.
Is there any event supporting this?

https://github.com/ButchersBoy/Dragablz/blob/master/Dragablz/TabablzControl.cs#L433


Support for Jooq ActiveRecords

Does the Vertx Jooq Reactive module support the ActiveRecord feature of Jooq. We would like to do CRUD with UpdatableRecord with optimistic locking. If we use the UpdatableRecord methods (store, refresh etc) will it internally use the reactive pg api?

Unfortunately no.
Am Mi., 5. Dez. 2018, 08:17 hat sowmyark<EMAIL_ADDRESS>geschrieben:

Does the Vertx Jooq Reactive module support the ActiveRecord feature of
Jooq. We would like to do CRUD with UpdatableRecord with optimistic
locking. If we use the UpdatableRecord methods (store, refresh etc) will it
internally use the reactive pg api?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jklingsporn/vertx-jooq/issues/82, or mute the thread
https://github.com/notifications/unsubscribe-auth/AGC83v25CYRTnqKhUCDJv9FjrWAF5qCYks5u13L0gaJpZM4ZCI6R
.

Thanks for you quick reply.
Do you have any plans of supporting this in the near future?
Also we would like to understand the complexity involved in implementing this with the reactive pg api so that we can give it a try. Could you give us some starting pointers for this.

I think it is nearly impossible. For example when you call store on an ActiveRecord it automatically calls the blocking jdbc method equivalent.
To really implement non-blocking CRUD for Records you'd have to overwrite the generateRecord method in the JavaGenerator/VertxGenerator and add an async API for records (like I did with the VertxDao).


Update meson build files

build: Update version numbers to 2.1.1

This was forgotten when doing the release.

meson: Fix warning about extract_all_objects usage

We use this because of a meson bug that was fixed in 0.52:

https://mesonbuild.com/Release-notes-for-0-52-0.html#improved-support-for-static-libraries
Bump the requirement and remove the extract_all_objects workaround.
This gets rid of a meson warning:

WARNING: extract_all_objects called without setting recursive
keyword argument. Meson currently defaults to
non-recursive to maintain backward compatibility but
the default will be changed in the future.

Commit 2 is once again an issue which would be solved by #3351

Maybe this one will be simpler to review and merge for the maintainers! :)

Should not master be updated to whatever the next version will be called? Probably as a separate commit though.
I guess next version also changes ABI so .so version for shared libs should be bumped?

That is traditionally something done during the release process itself.
Some projects will as a matter of personal preference do a post-release version bump to a *-dev or *.dev1 suffixed version number, but that's very much again, a matter of personal preference. Does this project happen to have such a policy?

Version number has been updated in master now but not the external_all_objects meson rule

can you help resolve the conflicts?

Done!


 Can you merge it?

Looks good enough to me, thanks for updating it!


Support FORCE RISC-V for verification

https://github.com/openhwgroup/force-riscv

Yeah I saw this too looks cool
On Tue, Dec 8, 2020 at 1:20 AM Dan Petrisko<EMAIL_ADDRESS>wrote:

https://github.com/openhwgroup/force-riscv
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/black-parrot/black-parrot/issues/702, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEFG5AAGRATKUWQNWVZZJLLSTXVUTANCNFSM4URVYGTA
.


[WIP] First attempt at smarter reconcile

TODO:

[ ] Implement reconcileChanges
[ ] setStaticChildren should always return an array

E.g:

  for foo in bar

 foo

This currently compiles to setStaticContent(array of
), but we need it to be setStaticContent([array of
]) so that it's consistent.

The last issue means that it will currently only work correctly if you do this:

  for foo in bar

 foo

I can fix the inconsistency in the compiler - it drops the array of there is only one expression/child in tag. Makes no sense. Will look at the changes later today. Nice work!

What do you think about this approach?
def Imba.markStatic array, id
    array:imbaStatic = id
    return array

# Generate tags like this:

t$('div').setChildren(Imba.markStatic(["Hello", other_tag], 1))

Then we avoid a custom method for setStaticChildren and it's more consistent with multiple "levels" of staticness. setChildren would look something like this:
def setChildren nodes
  if nodes isa Array
    if nodes:imbaStatic
      # Handle it as a static array (aka. setStaticChildren)
    else
      # Handle it as a dynamic array (aka. for-loops)

As for the difference between setChildren and setContent there's this approach:
def body # on tag
    self

<foo>
  <bar>
# will generate:
t$('foo').body().setChildren(…)

<self>
  <bar>
# will generate:
t$('foo').setChildren(…)

If you want to implement a custom wrapper/container you can just override body:
tag foo
  def body

  def render
    <self>

 "Menu"
      body

I really like the proposed change body().setChildren. There's one issue I'm not sure how to solve though. <foo> <bar> must return <foo>. We could cache foo before calling body like `((cached = foo),foo.body.setChildren(...),cached) but we would need to do this for every single node in the tree, since we don't know whether they have a custom implementation of body or not.

I've now implemented your proposed way of marking static arrays by sending them through Imba.static. Much simpler and more robust than the previous approach :) Opens up for further simplification of reconciliation.


Release Google.Cloud.ServiceHealth.V1 version 1.0.0-beta02

Changes in this release:
Documentation improvements

Update proto comments (commit bef44ef)

Created release for Google.Cloud.ServiceHealth.V1-1.0.0-beta02


Fixed the lock problem with AggregationHandleSum::mergeStates()

This PR is a one-line fix to the lock problem with AggregationHandleSum::mergeStates(). The fix is that we create a named variable for SpinMutexLock. The original line of code is somehow a typo that the lock is non-named and would be released immediately after creation, and causes non-deterministic results for TPC-H query 06.

LGTM. Merging.


Newest version only allows VTT 9

With no way to install anything older.
0.8.9 is still release with 0.9.0 still in testing, can we please have a way to install for the release version?

2.5.13 should be last version for FVTT 0.8, but the download should be fixed now.


[BUG] prompty-quickstart.ipynb run evaluation not working due to RuntimeError

Describe the bug
Run evaluation with PFClient throws RuntimeError
How To Reproduce the bug
Steps to reproduce the behavior, how frequent can you experience the bug:

follow the exact instructions in prompty-quickstart.ipynb up to pf = PFClient()
Run the following code:

flow = "./basic.prompty"  # path to the prompty file
data = "./data.jsonl"  # path to the data file

# create run with the flow and data
base_run = pf.run(
    flow=flow,
    data=data,
    column_mapping={
        "first_name": "${data.first_name}",
        "last_name": "${data.last_name}",
        "question": "${data.question}",
    },
    stream=True,
)

Happens consistently.
Expected behavior
execute pf.run()
Screenshots

Running Information(please complete the following information):

Promptflow Package Version using pf -v: 1.10.1
Operating System: Windows 11
Python Version using python --version: python== 3.9.13

Hi @imk4novazoon , does this issue occur when executing the notebook or a python script?
I did not reproduce this issue when executing it in notebook.
If you execute this code in a python script, which is by design, you need to move pf.run to name=="main"

Thank you @lalala123123 . That was it.


Automatically detect AWS EC2 AMI costs

A few users have requested this and our answer has been to use the usage file so far:
https://github.com/infracost/infracost/issues/26
https://github.com/infracost/infracost/issues/272
https://github.com/infracost/infracost/issues/348
https://twitter.com/gortron/status/1348209997583437824
@PatMyron pointed out an AWS announcement that can help us. I did a quick test for a RHEL AMI and if we find the UsageOperation from aws ec2 describe-images --image-ids ami-091967a4c3dbe5eac and pass it as a filter here (and remove the operatingSystem filter) then it returns the correct price. However, we should check if we currently have all EC2 Operating System Marketplace prices in the Cloud Pricing API.
The aim of this ticket is update the aws_instance, aws_autoscaling_group and aws_eks_node_group (they all use the same "Instance usage" cost component logic) such that:

if usage file's operating_system is present, use that as that's the user telling us what they want to see the cost estimate for.
else if Infracost is being run with a non-plan-json path (e.g. a Terraform directory), try to make the previously mentioned AWS API call to find the correct price.
else default to Linux (as we do just now).

We should also update this FAQ to mention that we sometimes use the cloud creds to find metadata that impacts price, e.g. AWS AMIs.

@alikhajeh1, I want to work on it!


Bug fix for targets with & without tags

This code has been defective if a compound scan has multiple targets, even if they differ just based on the number of tags. This is the case with the current and historical versions of observation/interferometric_pointing.py but this only becomes apparent when the number of measured points are reduced. See also PR #417 .
The change is to test for equivalence of targets EXCLUDING tags, and then to select the scans for all of those targets. Previously the code would typically have omitted the first scan, which is the one where the target has more tags than the rest of the scans in the compound scan.

@spassmoor please review this.

@ludwigschwardt  could you merge this if you are happy

For the record, @adriaanph, since I missed the "merge window":

You are welcome to assign reviewers to your PR to get a speedy response (myself or Sean will probably do in most cases).
Targets can be selected by name, description, object OR index. So once you have target_indices, you are good to go for selection.
A PEP8 tip: variable names should not contain uppercase letters as a rule.


Exchange Online PowerShell - Remoting must be enabled

The included requirements fail to mention that PSRemoting must be enabled. Otherwise, the associated winrm commands fail with a message similar to the following:
WSManFault
    Message = The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig".
Error number:  -2144108526 0x80338012

The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig".

To alleviate this, the guide should mention the PowerShell Remoting requirements:
Enable-PSRemoting

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 39fbe5a2-c138-b566-c094-82ffc3d1b734
Version Independent ID: 4d69ee5d-5a6d-a203-69fe-bf5441a8b5eb
Content: Connect to Exchange Online PowerShell
Content Source: exchange/docs-conceptual/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell.md
Service: exchange-online
GitHub Login: @chrisda
Microsoft Alias: chrisda

@officedocsbot assign @yogkumgit

Hi @jakezatecky, thank you for your feedback.
I think the author decided to not include id because WinRM is enabled by default and you should disable it with Disable-PSRemoting.
@chrisda could you kindly help here? Do you think we can add it? Thanks!

Hello @dariomws, I do not believe Windows Remote Management is enabled by default. I ran the following command on a newer machine, and it failed with the same error as with the previous machine:
winrm get winrm/config/client/auth

My anecdotal evidence suggests that Windows 10 does not enable this service by default. However, it may very well be that Windows Server instances (as opposed to client instances) have Remote Management enabled by default.

As an aside, the same article mentions that Basic Authentication must be enabled and provides a command to enable if it is disabled. I can verify that Basic Authentication for WinRM is enabled by default.
Since Basic Authentication is enabled by default, whereas anecdotal evidence suggests WinRM is not by default, it stands to reason that a brief mention of enabling of the WinRM service would fit in with the article contents
Thanks!

I have verified and it´s true, WinRM is not enabled by default on Windows client. I will add a note. Thanks!

@dariomws  Thank you very much for the contribution and updating the documentation with PR. @jakezatecky Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


automatically test .py files in examples

unless a test already exists which runs some tests.

Also useful over in gpkit-models: https://github.com/hoburg/gpkit-models/issues/6

Favoring explicitly listed tests, now.

This could be convenient, but I don't think it adds much.


New release with heartbeat API

Current JavaScript Exceptionless client is calling hearbeat api that doesn't exist in release 3.3.2327.
Is it possible to publish new version of Exceptionless that is compatible with JavaScript client?

@baalazamon yes, we will be pushing a new version probably next week. We are working on getting a beta out of the .net client first

In the mean time you could pull down master and run the build script or compile the solution and deploy the artifacts. We'll work on getting a release out there. Please feel free to join our slack channel.

https://github.com/exceptionless/Exceptionless/releases/tag/v3.4.0 has been released


Error in Refresh token

Hello all,
I'm trying to make a refresh token call using
oauth2Client.refreshAccessToken(function(err, tokens) {
// your access_token is now refreshed and stored in oauth2Client
// store these new tokens in a safe place (e.g. database)
});
But I'm receiving the following error
{ error: 'deleted_client',
error_description: 'The OAuth client was deleted.' } },
code: '401' }
Does anybody how to handle this error?
Thank you

What token are you refreshing, I don't understand. Is this directly related to gcsfs, or something else.
Maybe related, the client identity used by gcsfs for its own auth (e.g., when using browser credentials) does expire every six months, so updating your version of gcsfs may fix this.

Hi, @martindurant ...
I'm trying to refresh the refresh token as it expires in 1 hour so I'm trying to refresh it on every call.
The complete code bellow:
const { google } = require('googleapis');
var oauth2Client = google.auth.OAuth2;
var CLIENT_ID = 'xxxxxx;
var CLIENT_SECRET = 'xxxxxx';
var access_token = 'xxxxxx'
var refresh_token ='xxxxx'
var BASE_URL = 'http://localhost:4000';
var REDIRECT_URL = 'http://localhost:3000/oauth2Callback';
var oauth2Client = new google.auth.OAuth2(CLIENT_ID, CLIENT_SECRET,REDIRECT_URL);
const credentials = {access_token: access_token, refresh_token: refresh_token};
oauth2Client.setCredentials(credentials);
oauth2Client.refreshAccessToken(function(err, tokens) {
//console.log(err);
// your access_token is now refreshed and stored in oauth2Client
// store these new tokens in a safe place (e.g. database)
});

I'm sorry, I still don't understand. gcsfs does not require you to refresh google tokens it does that itself. Your code doesn't mention gcsfs: perhaps you wanted to post this in another repo?


BadIdentityTokenInvalid (0x80200000) using securityPolicy:Basic256 and securityMode:Sign

Klaus,
another strange thing. If I test the iiot connector with securityPolicy:Basic256 and securityMode:SIGN I get the same error as using securityPolicy:None and securityMode:NONE.

Now my questions:

Why will the first connection always close?
Why are the certificate and private key fields displayed on Policy:NONE Security:None?
Why is the default policyId: Username/Password and tokenType set to USERNAME if the username checkbox is not checked?

Please find attached the detailed log output:
2018-03-06T15:51:09.576Z opcuaIIoT:core /data/red/node_modules/node-opcua-client 2018-03-06T15:51:09.579Z opcuaIIoT:connector:details config: 2018-03-06T15:51:09.579Z opcuaIIoT:connector:details default key: /data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem 2018-03-06T15:51:09.580Z opcuaIIoT:connector:details config: 2018-03-06T15:51:09.580Z opcuaIIoT:connector:details default key: /data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem 2018-03-06T15:51:09.612Z opcuaIIoT:inject injecting once at start delay timeout 5000 msec. 2018-03-06T15:51:09.621Z opcuaIIoT:inject injecting once at start delay timeout 5000 msec. 6 Mar 15:51:09 - [info] Started flows 2018-03-06T15:51:11.593Z opcuaIIoT:connector Connecting On opc.tcp://<IP_ADDRESS>:4840/ 2018-03-06T15:51:11.594Z opcuaIIoT:connector:details Options {"securityPolicy":"Basic256","securityMode":"SIGN","defaultSecureTokenLifetime":60000,"keepSessionAlive":true,"certificateFile":"/data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem","privateKeyFile":"/data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem","endpoint_must_exist":false} 2018-03-06T15:51:11.914Z opcuaIIoT:connector Connected On opc.tcp://<IP_ADDRESS>:4840/ Options {"securityPolicy":"Basic256","securityMode":"SIGN","defaultSecureTokenLifetime":60000,"keepSessionAlive":true,"certificateFile":"/data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem","privateKeyFile":"/data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem","endpoint_must_exist":false} 2018-03-06T15:51:11.927Z opcuaIIoT:connector endpoint: opc.tcp://<IP_ADDRESS>:4840 2018-03-06T15:51:11.927Z opcuaIIoT:connector urn:<IP_ADDRESS> 2018-03-06T15:51:11.928Z opcuaIIoT:connector:details SIGNANDENCRYPT 2018-03-06T15:51:11.928Z opcuaIIoT:connector:details securityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#Basic256 2018-03-06T15:51:11.935Z opcuaIIoT:connector:details endpoint: opc.tcp://<IP_ADDRESS>:4840 2018-03-06T15:51:11.935Z opcuaIIoT:connector:details Application URI: urn:<IP_ADDRESS> 2018-03-06T15:51:11.935Z opcuaIIoT:connector:details Product URI: Loytec OPC UA Server 2018-03-06T15:51:11.936Z opcuaIIoT:connector:details Application Name: Loytec OPC UA Server 2018-03-06T15:51:11.936Z opcuaIIoT:connector:details Type: SERVER 2018-03-06T15:51:11.937Z opcuaIIoT:connector:details SIGNANDENCRYPT 2018-03-06T15:51:11.937Z opcuaIIoT:connector:details securityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#Basic256 2018-03-06T15:51:11.938Z opcuaIIoT:connector:details discoveryUrls: opc.tcp://<IP_ADDRESS>:4840 2018-03-06T15:51:11.957Z opcuaIIoT:connector:details serverCertificate: <obfuscated> 2018-03-06T15:51:11.957Z opcuaIIoT:core /data/red/node_modules/node-opcua-client 2018-03-06T15:51:11.958Z opcuaIIoT:connector:details /data/red/node_modules/node-opcua-client/certificates/PKI/server_certificate0.pem 2018-03-06T15:51:11.963Z opcuaIIoT:connector:details endpoint: opc.tcp://<IP_ADDRESS>:4840 2018-03-06T15:51:11.963Z opcuaIIoT:connector:details Application URI: urn:<IP_ADDRESS> 2018-03-06T15:51:11.964Z opcuaIIoT:connector:details Product URI: Loytec OPC UA Server 2018-03-06T15:51:11.964Z opcuaIIoT:connector:details Application Name: Loytec OPC UA Server 2018-03-06T15:51:11.965Z opcuaIIoT:connector:details Type: SERVER 2018-03-06T15:51:11.965Z opcuaIIoT:connector:details SIGN 2018-03-06T15:51:11.966Z opcuaIIoT:connector:details securityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#Basic256 2018-03-06T15:51:11.966Z opcuaIIoT:connector:details discoveryUrls: opc.tcp://<IP_ADDRESS>:4840 2018-03-06T15:51:11.974Z opcuaIIoT:connector:details serverCertificate: <obfuscated> 2018-03-06T15:51:11.975Z opcuaIIoT:core /data/red/node_modules/node-opcua-client 2018-03-06T15:51:11.975Z opcuaIIoT:connector:details /data/red/node_modules/node-opcua-client/certificates/PKI/server_certificate1.pem 2018-03-06T15:51:11.978Z opcuaIIoT:connector:details Identify Token for : Security Mode=SIGNANDENCRYPT Policy=http://opcfoundation.org/UA/SecurityPolicy#Basic256 2018-03-06T15:51:11.979Z opcuaIIoT:connector:details policyId: Username/Password 2018-03-06T15:51:11.981Z opcuaIIoT:connector:details tokenType: USERNAME 2018-03-06T15:51:11.981Z opcuaIIoT:connector:details issuedTokenType: null 2018-03-06T15:51:11.982Z opcuaIIoT:connector:details issuerEndpointUrl: null 2018-03-06T15:51:11.983Z opcuaIIoT:connector:details securityPolicyUri: null 2018-03-06T15:51:11.984Z opcuaIIoT:connector:details Identify Token for : Security Mode=SIGN Policy=http://opcfoundation.org/UA/SecurityPolicy#Basic256 2018-03-06T15:51:11.985Z opcuaIIoT:connector:details policyId: Username/Password 2018-03-06T15:51:11.985Z opcuaIIoT:connector:details tokenType: USERNAME 2018-03-06T15:51:11.986Z opcuaIIoT:connector:details issuedTokenType: null 2018-03-06T15:51:11.986Z opcuaIIoT:connector:details issuerEndpointUrl: null 2018-03-06T15:51:11.987Z opcuaIIoT:connector:details securityPolicyUri: null 2018-03-06T15:51:11.990Z opcuaIIoT:connector Setup Certified Options For opc.tcp://<IP_ADDRESS>:4840/ 2018-03-06T15:51:11.993Z opcuaIIoT:connector:details Options {"securityPolicy":"Basic256","securityMode":"SIGN","defaultSecureTokenLifetime":60000,"keepSessionAlive":true,"certificateFile":"/data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem","privateKeyFile":"/data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem","endpoint_must_exist":false,"serverCertificate":{"type":"Buffer","data":<obfuscated> 2018-03-06T15:51:12.013Z opcuaIIoT:connector !!!!!!!!!!!!!!!!!!!!!!!!  CONNECTION CLOSED !!!!!!!!!!!!!!!!!!! 2018-03-06T15:51:12.019Z opcuaIIoT:connector Disconnected From opc.tcp://<IP_ADDRESS>:4840/ 2018-03-06T15:51:12.021Z opcuaIIoT:connector:details Options {"securityPolicy":"Basic256","securityMode":"SIGN","defaultSecureTokenLifetime":60000,"certificateFile":"/data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem","privateKeyFile":"/data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem","endpoint_must_exist":false,"serverCertificate":{"type":"Buffer","data":<obfuscated> 2018-03-06T15:51:12.109Z opcuaIIoT:connector:details The certificate file /data/red/node_modules/node-opcua-client/certificates/PKI/server_certificate0.pem has been saved! 2018-03-06T15:51:12.118Z opcuaIIoT:connector:details The certificate file /data/red/node_modules/node-opcua-client/certificates/PKI/server_certificate1.pem has been saved! 2018-03-06T15:51:12.859Z opcuaIIoT:connector Secured Connected On opc.tcp://<IP_ADDRESS>:4840/ 2018-03-06T15:51:12.862Z opcuaIIoT:connector:details Options {"securityPolicy":"Basic256","securityMode":"SIGN","defaultSecureTokenLifetime":60000,"certificateFile":"/data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem","privateKeyFile":"/data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem","endpoint_must_exist":false,"serverCertificate":{"type":"Buffer","data":<obfuscated> 2018-03-06T15:51:12.864Z opcuaIIoT:listener starting OPC UA session with delay of 1000 2018-03-06T15:51:12.865Z opcuaIIoT:listener starting OPC UA session with delay of 1000 2018-03-06T15:51:13.868Z opcuaIIoT:listener Listener Start OPC UA Session 2018-03-06T15:51:13.871Z opcuaIIoT:connector Request For New Session From Listener Node 2018-03-06T15:51:13.934Z opcuaIIoT:listener Listener Start OPC UA Session 2018-03-06T15:51:13.936Z opcuaIIoT:connector Request For New Session From Listener Node 2018-03-06T15:51:14.340Z opcuaIIoT:connector Session Start Error Error:  ServiceResult is BadIdentityTokenInvalid (0x80200000) 2018-03-06T15:51:14.342Z opcuaIIoT:listener Handle Session Error Error:  ServiceResult is BadIdentityTokenInvalid (0x80200000) 2018-03-06T15:51:14.348Z opcuaIIoT:connector Session Start Error Error:  ServiceResult is BadIdentityTokenInvalid (0x80200000) 2018-03-06T15:51:14.349Z opcuaIIoT:listener Handle Session Error Error:  ServiceResult is BadIdentityTokenInvalid (0x80200000)

Please, try to build a very new flow - does it happen with a whole new flow and connector too?
Check the configuration to delete unused connectors in the background!

happens with none too

Sorry problem persist using NONE or using Basic256 after deleted every thing.
I cannot understand why I need certificates if the mode is NONE and user anonymous, why is trying to use the default none. Or can you tell me how to eliminate the default keys if I am trying to use NONE. Please advise. ATM I am running out of ideas.

In none Mode there is no key in use
Sent with GitHawk

I updatet the prosys server to 2.3 and it works well - 2.0 of prosys Server doesn't work
Sent with GitHawk

I am using a real OPC/UA server from Loytec Gateway 952
https://www.loytec.com/de/produkte/gateways/lgate-95x
BTW: I found that you use the certificates even in NONE mode, I guess for NONE there might not be any
certificates right?
This behavior worked with 1.3.3

let nodeOPCUAClientPath = coreConnector.core.getNodeOPCUAClientPath()
coreConnector.detailDebugLog('config: ' + node.publicCertificateFile)
if (node.publicCertificateFile === null || node.publicCertificateFile === '') {
node.publicCertificateFile = path.join(nodeOPCUAClientPath, '/certificates/client_selfsigned_cert_1024.pem')
coreConnector.detailDebugLog('default key: ' + node.publicCertificateFile)
}
coreConnector.detailDebugLog('config: ' + node.privateKeyFile)
if (node.privateKeyFile === null || node.privateKeyFile === '') {
node.privateKeyFile = path.join(nodeOPCUAClientPath, '/certificates/PKI/own/private/private_key.pem')
coreConnector.detailDebugLog('default key: ' + node.privateKeyFile)
}
node.opcuaClientOptions = {
securityPolicy: node.securityPolicy || 'None',
securityMode: node.messageSecurityMode || 'NONE',
defaultSecureTokenLifetime: node.defaultSecureTokenLifetime,
keepSessionAlive: true,
certificateFile: node.publicCertificateFile,
privateKeyFile: node.privateKeyFile,
endpoint_must_exist: node.endpointMusExist
}

      publicCertificateFile: {value: ''},
      privateKeyFile: {value: ''},

The keys are null - the UI shows just placeholders ;)

                <input type="text" id="node-config-input-publicCertificateFile" placeholder="./certificates/client_selfsigned_cert_1024.pem" style="min-width:480px">

                <input type="text" id="node-config-input-privateKeyFile" placeholder="./certificates/PKI/own/private/private_key.pem" style="min-width:480px">

please test v2.1.1

Sorry Klaus,
even if certificates are null, same error
first connect fails, second connect try to start a session ending with the same error:
BadIdentityTokenInvalid (0x80200000)

10 Mar 00:57:26 - [info] Starting flows
2018-03-10T00:57:26.935Z opcuaIIoT:core /data/red/node_modules/node-opcua-client
2018-03-10T00:57:26.936Z opcuaIIoT:connector:details config:
2018-03-10T00:57:26.937Z opcuaIIoT:connector:details config:
2018-03-10T00:57:26.937Z opcuaIIoT:connector:details securedCommunication: false
10 Mar 00:57:26 - [info] Started flows
2018-03-10T00:57:28.940Z opcuaIIoT:connector Connecting On opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:28.940Z opcuaIIoT:connector:details Options {"securityPolicy":"None","securityMode":"NONE","defaultSecureTokenLifetime":"60000","keepSessionAlive":true,"certificateFile":null,"privateKeyFile":null,"endpoint_must_exist":false}
2018-03-10T00:57:28.988Z opcuaIIoT:connector Connected On opc.tcp://<IP_ADDRESS>:4840 Options {"securityPolicy":"None","securityMode":"NONE","defaultSecureTokenLifetime":"60000","keepSessionAlive":true,"certificateFile":null,"privateKeyFile":null,"endpoint_must_exist":false}
2018-03-10T00:57:29.001Z opcuaIIoT:connector endpoint: opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:29.001Z opcuaIIoT:connector urn:<IP_ADDRESS>
2018-03-10T00:57:29.002Z opcuaIIoT:connector:details NONE
2018-03-10T00:57:29.002Z opcuaIIoT:connector:details securityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#None
2018-03-10T00:57:29.002Z opcuaIIoT:connector:details endpoint: opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:29.003Z opcuaIIoT:connector:details Application URI: urn:<IP_ADDRESS>
2018-03-10T00:57:29.003Z opcuaIIoT:connector:details Product URI: Loytec OPC UA Server
2018-03-10T00:57:29.003Z opcuaIIoT:connector:details Application Name: Loytec OPC UA Server
2018-03-10T00:57:29.004Z opcuaIIoT:connector:details Type: SERVER
2018-03-10T00:57:29.004Z opcuaIIoT:connector:details NONE
2018-03-10T00:57:29.005Z opcuaIIoT:connector:details securityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#None
2018-03-10T00:57:29.005Z opcuaIIoT:connector:details discoveryUrls: opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:29.005Z opcuaIIoT:connector:details serverCertificate: None
2018-03-10T00:57:29.006Z opcuaIIoT:connector:details Identify Token for : Security Mode=NONE Policy=http://opcfoundation.org/UA/SecurityPolicy#None
2018-03-10T00:57:29.006Z opcuaIIoT:connector:details policyId: NoIdentityTokens
2018-03-10T00:57:29.006Z opcuaIIoT:connector:details tokenType: ANONYMOUS
2018-03-10T00:57:29.007Z opcuaIIoT:connector:details issuedTokenType: null
2018-03-10T00:57:29.007Z opcuaIIoT:connector:details issuerEndpointUrl: null
2018-03-10T00:57:29.007Z opcuaIIoT:connector:details securityPolicyUri: null
2018-03-10T00:57:29.009Z opcuaIIoT:connector Setup Certified Options For opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:29.009Z opcuaIIoT:connector:details Options {"securityPolicy":"None","securityMode":"NONE","defaultSecureTokenLifetime":"60000","keepSessionAlive":true,"certificateFile":null,"privateKeyFile":null,"endpoint_must_exist":false}
2018-03-10T00:57:29.019Z opcuaIIoT:connector !!!!!!!!!!!!!!!!!!!!!!!!  CONNECTION CLOSED !!!!!!!!!!!!!!!!!!!
2018-03-10T00:57:29.022Z opcuaIIoT:connector Disconnected From opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:29.023Z opcuaIIoT:connector:details Options {"securityPolicy":"None","securityMode":"NONE","defaultSecureTokenLifetime":"60000","certificateFile":null,"privateKeyFile":null,"endpoint_must_exist":false}
2018-03-10T00:57:29.070Z opcuaIIoT:connector Secured Connected On opc.tcp://<IP_ADDRESS>:4840
2018-03-10T00:57:29.071Z opcuaIIoT:connector:details Options {"securityPolicy":"None","securityMode":"NONE","defaultSecureTokenLifetime":"60000","certificateFile":"/data/red/node_modules/node-opcua-client/certificates/client_selfsigned_cert_1024.pem","privateKeyFile":"/data/red/node_modules/node-opcua-client/certificates/PKI/own/private/private_key.pem","endpoint_must_exist":false}
2018-03-10T00:57:29.072Z opcuaIIoT:client:read starting OPC UA session with delay of 1000
2018-03-10T00:57:30.073Z opcuaIIoT:client:read Read Start OPC UA Session
2018-03-10T00:57:30.074Z opcuaIIoT:connector Request For New Session From Read Node
2018-03-10T00:57:30.122Z opcuaIIoT:connector Session Start Error Error:  ServiceResult is BadIdentityTokenInvalid (0x80200000)
2018-03-10T00:57:30.123Z opcuaIIoT:client:read Handle Session Error Error:  ServiceResult is BadIdentityTokenInvalid (0x80200000)

a) Did you test with the opcua-commander? Does it work?
b) Did you try to set up a whole new flow or is it a modified example? Try with whole new flow, please!
c) How did you update via npm? (update/install) Did you try to uninstall and install again?
d) Please, search for issues in node-opcua repo!
e) Is it possible to connect online from my Node-RED?

Ckeck for that your keys are trusted in the server. Does it work with keys?

some hint: also some servers need to restart to load trusted certificates

Want to test first without certificates in anonymous mode, before going to add one. The debug log  shoes a second connection with certificates,then the session ends with badIdentityInvalidToken.
Again no certificates, no user. Tested with ProSys Client it works with None.

a) Did you test with the opcua-commander? Does it work?
b) Did you try to set up a whole new flow or is it a modified example? Try with whole new flow, please!
c) How did you update via npm? (update/install) Did you try to uninstall and install again?
d) Please, search for issues in node-opcua repo!
e) Is it possible to connect online from my Node-RED?

test with v2.2.x

For the v2.2.x test - build and wire a whole new flow, please!

do a complete uninstall before, check that Node-RED is missing the nodes and than do a fresh install, please

Hi Klaus,
fixed!
The connection works now for securityPolicy:NONE and securityMode:None


Bugfix on Windows 10: command not working

This pull requests addresses a bug I ran into on Windows 10.
Every audit of a package fails with the error spawn npm ENOENT.
This is due to (at least my) Windows requiring npm.cmd instead of npm to function.
This pr replaces the hardcoded npm with a conditional string based on the OS.

Thanks for your response @AdrieanKhisbe.
My apologies for the lack of tests: this is my first PR so it might be a bit rocky.
Let me first say that this is not an enhancement, but a fix: the package as it is now does not work at all on my Windows machine. The proposed npm.cmd is not a different package: it is the exact name (with file extension) of the one and only npm package. Therefore every Windows computer that has npm installed has npm.cmd installed.
I agree with you though that an OS check is a bit of an ugly solution to include in that particular file. Therefore I propose a different approach: by providing execFile with the option {shell: true}, the command is run (in the background) through the system shell. When fired like this npm resolves correctly on Windows and the command works perfectly fine without the need for conditionals or additional tests. Seeing that all the checks (on Linux) have passed as well I think it is safe to implement this fix and make the package function cross-platform.

Thanks @soimon,
if it's just adding the shell option flag to node execFile, there is no real need to add cross platform test indeed.
(I was cautious as was thinking this maybe was not a generalised problem on windows, and so wanted to ensure no regression would be introduce on that env)
and yes, that was definitely an issue introduced in #33 by removing shelljs :shell:

@soimon, I was about to merge, but I thought to check if there was another place that would need the flag.
And indeed, there was one : https://github.com/AdrieanKhisbe/bundle-phobia-cli/blob/master/src/install.js#L114
Would you mind adding the {shell: true}? or allow me to push on your branch? 🙂
thanks in advance :)

Good one, I hadn't used that feature so I didn't run into the problem yet.
I pushed the final change, so if the checks complete then it's ready for launch 🚀
Thanks for your assistance 👍

Released as v0.14.11 [f37949a]

I just published it @soimon,
you can give a try and tell me if it works for you on windows 🙂

The bundle-phobia command now works without a problem 😄
There is a problem with bundle-phobia-install though that seems unrelated to this change.
As I do not use that feature myself I will just file a new issue with the details.


include twamm volume in the hourly volume rollup table

eventually, low prio

our price data is often off because of the unbalanced EKUBO dca sale, which causes the occasional large swap against a pool
this doesn't fix that, since we use a separate query in the api to compute price data, but it would help us understand how to fix the price query


Jobs Stuck In 'Enqueued' State

I am getting an issue where multiple jobs will be in the Enqueued state, but never make it out of there. When I click on the queue in the dashboard, it shows no jobs to be found, even though there is a number of them showing. In Atlas, when I query for {StateName:"Enqueued"}, it shows those n jobs that are stuck. The jobs are enqueued every minute from a recurring job setup, which is set up to loop through a number of objects, and enqueue subsequent job for each one, like such:
public void ProcessCallbacks() {
  var objects = repo.GetCallbacks();
  foreach(var object in objects) {
    BackgroundJob.Schedule(() => Processor(object.ID), DateTime.Now);
  }
}

This happens randomly, there doesn't seem to be any specific date or time that causes it to get stuck like that. Most of the time it runs fine, but there are usually ~5-10 each night that cause this to happen. Could this be a worker issue not being able to pick up a job because all workers are in use, which causes it to get stuck?
Any help into this would be greatly appreciated.

@nickmistretta9 ,
Would it be possible to attach some info about the stuck job? I especially interested in
JobQueueDto's still in the queue.
Below is from memory so it might be faulty ¯_(ツ)_/¯
{ "$and": [{ "_t": "JobQueueDto"},  { "FetchedAt", {"$ne": null} }] }
Also try to query JobQueueDto's by the stuck jobs "JobId", see if its been fetched.
Also all writes are logged to hangfire's logging system as DEBUG, you could attach a logger:
http://docs.hangfire.io/en/latest/configuration/configuring-logging.html
if the data is not too sensitive, I would love to take a look at it.
Meanwhile looking your snippet, it should be possible to reproduce locally, I will try to reproduce locally.

@gottscj Thanks for your response! When I query for "JobQueueDto" I am getting no results from the hangfire.jobGraph DB. I filtered by {StateName:"Enqueued"} to get the list of jobs that are "stuck" in the enqueued state and here is an example of one of the response objects:
{
    "_id":objectId(5c8c3e24ee0a523becca8123),
    "_t": Array [
        0:"BaseJobDto",
        1:"ExpiringJobDto",
        2:"JobDto"
    ],
    "ExpireAt":null,
    "StateName":"Enqueued",
    "InvocationData":"{"Type":"SPPC_Jobs.Controllers.SCPController, SPPC_Jobs, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null","Method":"ProcessIncomingCallbacks","ParameterTypes":"[]","Arguments":"[]"}",
    "Arguments":"[]",
    "Parameters":Object {
        RecurringJobId:""SCPController.ProcessIncomingCallbacks""
        CurrentCulture:""en-US""
        CurrentUICulture:""en-US""
    },
    "StateHistory":Array [
        0:Object {
            Name:"Enqueued",
            Reason:"Triggered by recurring job scheduler",
            CreatedAt:2019-03-16T00:07:00.834+00:00,
            Data:Object {
                EnqueuedAt:"2019-03-16T00:07:00.7724447Z",
                Queue:"scp"
            }
        }
    ],
    "CreatedAt":2019-03-16T00:07:00.772+00:00
}

I am going to attach a logger and see what I can get out of it, but let me know if this is helpful in the meantime!

@nickmistretta9 ,
Can you query the JobQueueDto object by the JobId you showed above? Im wondering if the job is actually run, but the state is not updated.
Looking at the Job you show above it seems to be a recurrent job, but the loop in your first post its a scheduled job.
Do you see a pattern in the stuck jobs in regards with how its scheduled?
Thanks!

@nickmistretta9,
I am able to reproduce locally. It seems to be related to a lock timing out. if you wait 90s the lock is droppet and tried again. I am looking into why its not able to acquire the lock

@gottscj Hey sorry for the delay in my response! So that first loop is called by another recurring method, like so:
var scpController = new SCPController();
RecurringJob.AddOrUpdate(() => scpController.ProcessIncomingCallbacks(), Cron.Minutely());

Which database would the JobQueueDto appear in? When I query hangfire.jobGraph, the only object that shows up for that JobID is the one that I linked above.

@nickmistretta9,
The JobQueueDto lives in the jobgraph. But only while the job is being processed.
I figured it out. There is not a bug per se. and your job will be processed. Our wait times is be bit out of date, compared to SqlServer implementation. If the distributed lock fails to acquire a lock it will wait the timeout given by Hangfire.Core / 10.
Hangfire.Core gives a timeout of 900000 ms. divided by 10 that gives 90000ms eg. 90 seconds before the job is processed. Looking at the Hangfire.SqlServer implementation. The timeout is divided by 1000 which gives 900 ms.
So. All your jobs will be processed with v0.5.15.
You will just have to wait 90 seconds if the job is re-queued or the lock is somehow not aquired in first attempt.
for 0.6.0, I will change the timeout to resemble whats done in the sql server implementation.
and ultimately use tailable cursors to signal when the lock can be released.

@gottscj Ok great thank you for that information! So even if the job is processed after 90 seconds, will this new implementation remove them from the database upon completion? We are trying to also ensure that should these jobs timeout or not be picked up by the lock that in the dashboard/database we won't have a "queue" with however many jobs timed out.

@nickmistretta9,
I apologize I was a bit unclear.
The actual job is not "stuck" in 90s. Its executed as fast as possible. Its the job data Update which is stuck. eg. changing the state from "Enqueued" to "Processing" and then to "Succeeded" (or "Failed").
The enqueued jobs are not executed under lock, only updating the state data. So the information in the dashboard is not showing the correct information while the lock waits for access.

Should be resolved in 0.6.0, please let me know if it doesnt work as expected.

I'm continuing to see this issue in 0.6.4
i'm not sure what details you need, but we are queuing up around 900 jobs and once they are finished they queue themselves up again. It seems random, but the jobs get stuck and don't process, or only process 4-5 at a time instead of the usual 35.

@KyleCraviotto,
Sorry, for late reply.
logs are always helpful. The job is enqueuing itself after a succesful run? is it scheduled by a cron or as fast as possible?
How do you see this information? in the dashboard? can you confirm the job is not actually run or its the monitoring api which is lagging?
thanks

@gottscj
Here's the gist of how we are using hangfire.
 This job also makes sure we are not queuing up duplicate jobs and we are deleting any duplicate jobs that may have been created as well.
Step 2) Each background job runs after 3 minutes, does it's processing and then kicks off another background job of itself.
This continues usually for anywhere from 2-24 hours and then we notice that the number of queues jobs goes up and up, and usually stacks up to 700ish, while the processing jobs goes down and down usually in the 8 or so range.
I see this both in the hangfire dashboard and it the database itself.

@Kronos11,
Thanks for the update, I will try to reproduce.

We continued to have this issue so I switched to redis to see if that would resolve the issue and unfortunately it did not. I would guess this is some underlying hangfire issue, or a problem with our code.

@KyleCraviotto,
Sorry I didnt try this yet. Its been a little hectic at my end. I will still try to reproduce it, to see if I can figure out what the problem is. If its a hangfire issue, we can add a ticket in that project

Hello guys! I get similar bug. My recurring jobs stuck in "Enqueued" state and dont run. They are executed only after restart application, but new jobs stuck again in "Enqueued" state.
My recurring job

Stuck jobs

Servers

Job in queue

Stack:
ASP.NET Core 3.1, MySql
Configuration:

Can anyone help with this problem?


FAQ Question mentions emuNAND rather than redNAND

Nitpicky thing but in the FAQ it states:
Q: Can I do this without a computer (e.g. an Android phone)?
A: You may be able to complete up to part 3 of the guide (getting emuNAND running). However, part 4 as of now requires a computer to inject FBI into Health & Safety.
emuNAND should be replaced with redNAND since part 3 strictly talks about redNAND now.

EmuNAND and RedNAND refer to slightly different variations of the same thing and can almost always be used interchangeably.


Introducing leapfrog script into N

This is a first pass, to make sure an OSA leapfrog can happen
in the gates.
(backport of master to N)
Connected https://github.com/rcbops/u-suk-dev/issues/1745

recheck_cit_all

recheck_cit_upgrade
recheck_cit_swift
recheck_cit_ceph
recheck_cit_artifact-trusty-Git

recheck_cit_upgrade
recheck_cit_artifact_trusty_git

recheck_cit_artifact_trusty_container

recheck_cit_artifact_trusty_python

recheck_cit_upgrade

recheck_cit_artifact-xenial.*

recheck_cit_upgrade
recheck_cit_artifact_xenial_container
recheck_cit_artifact_xenial_git
recheck_cit_artifact_xenial_python


Fix version in release

We're using semantic release, which means we have no version in package.json, but then we try to read the version from the package.json before semantic-release pre runs.

#173


automate Thaumaturge regalia through GrantItem

Selecting "Regalia" as a thaumaturge implement grants a "Regalia" item. This item grants the regalia benefits that can be automated when wielded, checking the flags set by the choices built into the class features to upgrade the abilities appropriately.


checkbox for replace group

Make it possible to check the checkbox for groups of plugins. That is, for example, I want text to be fasted in one group and only pictures in the other.

I assumed you mean this feature, #482, #474 and #176. Please stop making duplicate issues.

I assumed you mean this feature, #482, #474 and #176. Please stop making duplicate issues.

yes but will it be added?

I assumed you mean this feature, #482, #474 and #176. Please stop making duplicate issues.

yes but will it be added?

Yes, it is under the todo tag, see #176.


chore: test PR for merge queue

This is an empty commit, which I will use for testing the newly enabled merge queue.

We have successfully tested the merge queue with real commits, so I’ll close this one.


Magic Image Maps doesn't work

Hello, I've got a problem with the /mmap load command. If I try to load a image from the internet by using /mmap load [url] it doesn't work. I just get a blank map. How can i fix this?

I just tried it on my dev server and it appeared to work.
Can you please check your server logs for errors? The image is downloaded server-side, so if you have a firewall there blocking outgoing connections (unusual, but possible) that might get in the way.
It's also possible it's something with the specific image you're trying, so if you want to let me know the URL I can try it myself.

I had a weird issue yesterday where I map I had previously loaded looked blank. Later, after a server restart maybe, it came back.
Just noting that here. Did you ever get it working?


How to add assets to ide

submitting your network information. Afterward, we will review and merge it. Below is an example of the network's metadata:
{
"rpc": "https://testnet-rpc.orai.io",
"rest": "https://testnet-lcd.orai.io",
"chainId": "Oraichain-testnet",
"chainName": "Oraichain Foobar",
"stakeCurrency": {
"coinDenom": "ORAI",
"coinMinimalDenom": "orai",
"coinDecimals": 6,
"coinGeckoId": "oraichain-token",
"coinImageUrl": "http://localhost:3000/public/assets/tokens/orai.png"
},
"bip44": {
"coinType": 118
},
"bech32Config": {
"bech32PrefixAccAddr": "orai",
"bech32PrefixAccPub": "oraipub",
"bech32PrefixConsAddr": "oraivalcons",
"bech32PrefixConsPub": "oraivalconspub",
"bech32PrefixValAddr": "oraivaloper",
"bech32PrefixValPub": "oraivaloperpub"
},
"currencies": [
{
"coinDenom": "ORAI",
"coinMinimalDenom": "orai",
"coinDecimals": 6,
"coinGeckoId": "oraichain-token",
"coinImageUrl": "http://localhost:3000/public/assets/tokens/orai.png"
}
],
"feeCurrencies": [
{
"coinDenom": "ORAI",
"coinMinimalDenom": "orai",
"coinDecimals": 6,
"coinGeckoId": "oraichain-token",
"coinImageUrl": "http://localhost:3000/public/assets/tokens/orai.png"
}
],
"gasPriceStep": {
"low": 0,
"average": 0.0025,
"high": 0.004
},
"features": [
"stargate",
"ibc-transfer",
"cosmwasm"
],
"cosmwasmVersion": "0.13.2" // the cosmwasm std your network is using.
}
It follows the same format as that of Keplr so that the webview can work with the extension. We also add a few attributes, which include comments above.
Please append the network information into the config file if you prefer to let your network be included in the IDE permanently. Please note that this requires reviewing & a pull request. The IDE also provides a feature that helps you add and remove custom networks & store them locally.

PR on it :)


Evaluate using ARM templates to deploy Azure Policy Definition/Assignments

Why ARM template: Easy to deploy locally, with Enterprise-Scale or other DevOps pipelines than GitHub Actions.
Examples:

ARM template for Policy Definitions - here
ARM template for Policy Assignments - here

Merged to main. Closing issue.


How wappalyzer detects database?

Is it added manually?

Hi, yes. Have a look at https://github.com/AliasIO/wappalyzer/blob/master/README.md.

The word 'database' is not found

Databases are usually identified indirectly. For instance when Magento is detected, MySQL is implied:
https://github.com/AliasIO/wappalyzer/blob/master/src/technologies.json#L7906


Change google deploy default context name to gcp

Sorry for the flow response on this @gpkc, this LGTM. Just waiting for CI to pass and then I'll merge.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closing this PR, as it's obsolete


on-the-fly rendering of link diagram

Instead of including the rendered version of the link creation diagram, I've added a link to render the image from the latest source on github master.
Thanks to @maxandersen for pointing that out.

@kwk snapshot fabric8-wit image is available for testing.  docker pull docker.io/fabric8/fabric8-wit:SNAPSHOT-PR-1944-1

Codecov Report

Merging #1944 into master will increase coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1944      +/-   ##
==========================================
+ Coverage   63.76%   63.82%   +0.05%
==========================================
  Files         144      144
  Lines       13589    13589
==========================================
+ Hits         8665     8673       +8
+ Misses       4089     4081       -8
  Partials      835      835

Impacted Files
Coverage Δ

remoteworkitem/scheduler.go
60.97% <0%> (+7.31%)
:arrow_up:

remoteworkitem/jira.go
100% <0%> (+25%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51b124c...296e0ef. Read the comment docs.


Documentation incorrectly states that you can enable MongoDB endpoint

I have attempted to run the CosmosDB emulator, from both WSL and Windows, using the following command:
docker run
-p 8081:8081
-p 10251-10254:10251-10254
--memory 3g --cpus=2.0
-e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
-e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
-e AZURE_COSMOS_EMULATOR_ENABLE_MONGODB_ENDPOINT=4.0
-it
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
I have also attempted to launch it from a docker-compose file:
cosmosdb:
image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
tty: true
stdin_open: true
restart: always
cpu_count: 2
volumes:
- type: bind
source: c:/CosmosDBData
target: /data/db
mem_limit: 3g
environment:
- AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
- AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
- AZURE_COSMOS_EMULATOR_ENABLE_MONGODB_ENDPOINT=4.0
ports:
- 8081:8081
- 10251:10251
- 10252:10252
- 10253:10253
- 10254:10254
- 10255:10255
I am quite certain these are equivalent, but I end up with the following error in any case:
cosmosdb_1                   | This is an evaluation version.  There are [145] days left in the evaluation period.
cosmosdb_1                   | Invalid endpoint type enabled; only SQL endpoint type is supported.
I believe I can safely ignore the first message as there is no actual time lock on the image. However the second error is fatal. If I leave off the ENABLE_MONGODB_ENDPOINT=4.0 environment variable, CosmosDB starts, but the mongodb URL provided at https://localhost:8081/_explorer/index.html doesn't work. Please advise.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0a59d2ab-cacd-5545-bddb-b9304f204605
Version Independent ID: bbcf9128-f9c3-fa84-b085-ba566f45d56f
Content: Run the Azure Cosmos DB Emulator on Docker for Linux
Content Source: articles/cosmos-db/linux-emulator.md
Service: cosmos-db
GitHub Login: @StefArroyo
Microsoft Alias: esarroyo

@cubikca Thank you for your feedback. We reviewing this and will back shorlty.

#assign:milismsft
@milismsft, can you take a look at this issue? I didn't see anything obvious in the troubleshooting guide for the Linux Docker emulator.

The solution is to use mongodb as the container image tag. IOW, run this:
docker run \
    --publish 8081:8081 \
    --publish 10251-10254:10251-10254 \
    --env AZURE_COSMOS_EMULATOR_ENABLE_MONGODB_ENDPOINT=4.0 \
    --interactive \
    --tty \
    mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:mongodb

The solution is to use mongodb as the container image tag. IOW, run this:
docker run \
    --publish 8081:8081 \
    --publish 10251-10254:10251-10254 \
    --env AZURE_COSMOS_EMULATOR_ENABLE_MONGODB_ENDPOINT=4.0 \
    --interactive \
    --tty \
    mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:mongodb

I'm trying this command, but I get this issue:
This program has encountered a fatal error and cannot continue running at Tue Sep 13 09:09:52 2022
The following diagnostic information is available:

         Reason: OS Hard Error (0x00000002)
    Stack Trace:
                 file://package1/windows/system32/sqlpal.dll+0x0000000000298F1C
                 file://package1/windows/system32/sqlpal.dll+0x000000000029802A
                 file://package1/windows/system32/sqlpal.dll+0x0000000000265CD9
                 file://package1/windows/system32/sqlpal.dll+0x00000000002ACD9B
                 file://package1/windows/system32/sqlpal.dll+0x0000000000394658
        Process: 19 - cosmosdb-emulator
         Thread: 125 (application thread 0x1c0)
    Instance Id: 1d149154-eeea-4737-bebd-167b22243b9f
       Crash Id: 57f0f055-2169-4a42-ac43-db3337480df1
    Build stamp: (null)
   Distribution: Ubuntu 20.04.4 LTS
     Processors: 8
   Total Memory:<PHONE_NUMBER> bytes
      Timestamp: Tue Sep 13 09:09:52 2022
     Last errno: -34938881
Last errno text: Unknown error -34938881
*********** PAL PANIC CORE DUMP GENERATION FAILED **********
Unable to locate handle-crash.sh. Error: File: signals.cpp:483 [Status: 0xC0000034 Object name not found errno = 0x2(2) No such file or directory]
Executing: /usr/local/bin/cosmos/handle-crash.sh with parameters
     handle-crash.sh
     /usr/local/bin/cosmos/cosmosdb-emulator
     19
     /usr/local/bin/cosmos
     /tmp/cosmos/appdata/log/

     1d149154-eeea-4737-bebd-167b22243b9f
     57f0f055-2169-4a42-ac43-db3337480df1

*********** PANIC CORE DUMP GENERATION FAILED **********
Attempt to launch handle-crash.sh failed.
This program has encountered a fatal error and cannot continue running at Tue Sep 13 09:09:52 2022
The following diagnostic information is available:

         Reason: Host Extension RTL_ASSERT (0x00000003)
         Status: STATUS_WAIT_2 (0x00000002)
        Message: !killTheTarget
          Stack:
                 SP               IP               Function
                 ---------------- ---------------- ----------------
                 00007fae4a6b9df0<PHONE_NUMBER>cbfbba <unknown>
                 00007fae4a6bae50<PHONE_NUMBER>cbf58f <unknown>
                 00007fae4a6bb0b0<PHONE_NUMBER>c22b26 <unknown>
                 00007fae4a6bb0e0<PHONE_NUMBER>cc30a4 <unknown>
                 00007fae4a6bc310<PHONE_NUMBER>cc2769 <unknown>
                 00007fae4a6bc440 00007fae4c814609 start_thread+0xd9
                 00007fae4a6bc500 00007fae4c295133 clone+0x43

        Process: 17 - cosmosdb-emulator
         Thread: 18
    Instance Id: 1d149154-eeea-4737-bebd-167b22243b9f
       Crash Id: 57f0f055-2169-4a42-ac43-db3337480df1
    Build stamp: (null)
   Distribution: Ubuntu 20.04.4 LTS
     Processors: 8
   Total Memory:<PHONE_NUMBER> bytes
      Timestamp: Tue Sep 13 09:09:52 2022
     Last errno: 2
Last errno text: No such file or directory
Aborted

Not sure where to report it.

After some investigation, this seems to be a bug with the product as the :mongodb tag incorrectly does not expose the mongodb endpoint. Unfortunately, we can't deal with product bugs here as this is a documentation repository and the docs articles are using the correct tag.
I looked at Microsoft Q&A and there doesn't seem to be a question about this issue. I recommend that you post your question over there so the custom support team can escalate this issue as it's blocking your work.
I'm sorry that I was unable to resolve this on the documentation site. Thanks for your patience and I hope this can be resolved quickly for you!
#please-close


Verilator test failure: caliptra-fmc::test_rtalias test_pcr_log

Test executed against 046881797e05d0e956422ab0a85b3960acd9c7fd
2023-09-19T09:31:53.9936422Z         FAIL [31168.002s] caliptra-fmc::test_rtalias test_pcr_log
2023-09-19T09:31:54.0070321Z
2023-09-19T09:31:54.0075821Z --- STDOUT:              caliptra-fmc::test_rtalias test_pcr_log ---
2023-09-19T09:31:54.0076272Z
2023-09-19T09:31:54.0399856Z running 1 test
2023-09-19T09:31:54.0400871Z test test_pcr_log has been running for over 60 seconds
2023-09-19T09:31:54.0402139Z       6,155 ready_for_fuses is high
2023-09-19T09:31:54.0403251Z       6,333 writing to cptra_bootfsm_go
2023-09-19T09:31:54.0404036Z     195,201 UART:
2023-09-19T09:31:54.0404851Z     195,256 UART: Running Caliptra ROM ...
2023-09-19T09:31:54.0840782Z     196,662 UART:
2023-09-19T09:31:54.0841688Z     202,819 UART: [state] CFI Enabled
2023-09-19T09:31:54.0842681Z     205,066 UART: [state] LifecycleState = Unprovisioned
2023-09-19T09:31:54.0843597Z     207,326 UART: [state] DebugLocked = No
2023-09-19T09:31:54.0847708Z     209,730 UART: [state] Watchdog Timer is not started because the device is not locked for debugging
2023-09-19T09:31:54.0849304Z     357,594 UART: ROM Digest: BBB6BA0BFBA2F1E2103A1060EB333FF6613A1F43E352AA823F62EF22650FF80F
2023-09-19T09:31:54.0850290Z     368,834 UART: [kat] ++
2023-09-19T09:31:54.0850869Z     369,346 UART: [kat] sha1
2023-09-19T09:31:54.0902805Z     378,506 UART: [kat] SHA2-256
2023-09-19T09:31:54.0903955Z     381,378 UART: [kat] SHA2-384
2023-09-19T09:31:54.0904570Z     385,643 UART: [kat] SHA2-384-ACC
2023-09-19T09:31:54.0905148Z     387,940 UART: [kat] ECC-384
2023-09-19T09:31:54.0905696Z   3,958,294 UART: [kat] HMAC-384
2023-09-19T09:31:54.0906192Z   3,969,346 UART: [kat] LMS
2023-09-19T09:31:54.0906695Z   5,071,148 UART: [kat] --
2023-09-19T09:31:54.1360306Z   5,074,555 UART: [cold-reset] ++
2023-09-19T09:31:54.1360933Z   5,076,995 UART: [idev] ++
2023-09-19T09:31:54.1361467Z   5,077,574 UART: [idev] CDI.KEYID = 6
2023-09-19T09:31:54.1362113Z   5,078,824 UART: [idev] SUBJECT.KEYID = 7
2023-09-19T09:31:54.1362636Z   5,080,296 UART: [idev] UDS.KEYID = 0
2023-09-19T09:31:54.1363203Z   5,096,725 UART: [idev] Erasing UDS.KEYID = 0
2023-09-19T09:31:54.1363787Z   8,363,501 UART: [idev] Using Sha1 for KeyId Algorithm
2023-09-19T09:31:54.1364459Z   8,386,797 UART: [idev] --
2023-09-19T09:31:54.1365343Z   8,391,014 UART: [ldev] ++
2023-09-19T09:31:54.1365958Z   8,391,607 UART: [ldev] CDI.KEYID = 6
2023-09-19T09:31:54.1366475Z   8,392,849 UART: [ldev] SUBJECT.KEYID = 5
2023-09-19T09:31:54.1367077Z   8,394,321 UART: [ldev] AUTHORITY.KEYID = 7
2023-09-19T09:31:54.1367633Z   8,395,903 UART: [ldev] FE.KEYID = 1
2023-09-19T09:31:54.1368166Z   8,405,116 UART: [ldev] Erasing FE.KEYID = 1
2023-09-19T09:31:54.1368794Z  11,690,261 UART: [ldev] Signing Cert with AUTHORITY.KEYID = 7
2023-09-19T09:31:54.1369775Z  15,286,382 UART: [ldev] PUB.X = 504D38CA45D997901F48BA333A149A2FB2668B973AAC64D3B79ECC09A663F02ED2FABA133F5FA499677AEB7687C99B25
2023-09-19T09:31:54.1371068Z  15,302,080 UART: [ldev] PUB.Y = A5CD31125698B3322C086E4398A591946BF20A3ECEA8EB7D7C23410D49FA877E7ECFBE47BD883BD53D7DA865AC217F1D
2023-09-19T09:31:54.1372504Z  15,318,150 UART: [ldev] SIG.R = 14B5D494DF875AC7CEE9FE3E9A6168A6B64BFB01171F5ECC5216C8BB945FAF957CED1BE6C7CFBB1DDB2B7F4BFD76A492
2023-09-19T09:31:54.1373738Z  15,334,062 UART: [ldev] SIG.S = 821D8CC92F40AFF934771D7652C9BC3C59D47E5635E110A8486B76F7433B02B3C6B0FD8B1B6548B5994C481547111FC5
2023-09-19T09:31:54.1374643Z  15,357,005 UART: [ldev] --
2023-09-19T09:31:54.1375115Z  15,363,503 ready_for_fw is high
2023-09-19T09:31:54.1375798Z  15,363,505 <<< Executing mbox cmd 0x46574c44 (23288 bytes) from SoC
2023-09-19T09:31:54.1376442Z  15,363,564 UART: [fwproc] Waiting for Commands...
2023-09-19T09:31:54.1377038Z  15,377,210 UART: [fwproc] Received command 0x46574c44
2023-09-19T09:31:54.1377643Z  15,379,824 UART: [fwproc] Received Image of size 23288 bytes
2023-09-19T09:31:54.1380378Z  18,071,236 UART: [fwproc] Image verified using Vendor ECC Key Index 0
2023-09-19T09:31:54.1381122Z  18,155,053 UART: [fwproc] Loading FMC at address 0x40000000 len 12964
2023-09-19T09:31:54.1381799Z  18,236,459 UART: [fwproc] Loading Runtime at address 0x40004000 len 4440
2023-09-19T09:31:54.1680777Z  18,267,431 >>> mbox cmd response: success
2023-09-19T09:31:54.1681626Z  18,267,436 <<< Executing mbox cmd 0x10000001 (0 bytes) from SoC
2023-09-19T09:31:54.1682273Z  18,271,374 UART: [afmc] ++
2023-09-19T09:31:54.1682854Z  18,271,953 UART: [afmc] CDI.KEYID = 6
2023-09-19T09:31:54.1683404Z  18,273,203 UART: [afmc] SUBJECT.KEYID = 7
2023-09-19T09:31:54.1683955Z  18,274,649 UART: [afmc] AUTHORITY.KEYID = 5
2023-09-19T09:31:54.1684758Z  21,590,894 UART: [afmc] Signing Cert with AUTHORITY.KEYID = 5
2023-09-19T09:31:54.1685601Z  23,886,489 UART: [afmc] Erasing AUTHORITY.KEYID = 5
2023-09-19T09:31:54.1686944Z  25,195,583 UART: [afmc] PUB.X = 74A2DA8F43E3CF7B034EB91F9FEBBBEBD3F2BB8B9B15CCF6B2944AE9C548D4ACDD20742598EA46075F5EE3D570A3C884
2023-09-19T09:31:54.1691443Z  25,211,371 UART: [afmc] PUB.Y = F0B6D79EFAF624F13C60ACE8287337FA10B411125D5F76A85844E286F5E554EBD2D1B41B1DFB30A078203A651EFE8C34
2023-09-19T09:31:54.1692877Z  25,227,279 UART: [afmc] SIG.R = 8C94183EECD46495965C1F58565A801C567721535D1E46E8FBD8FE28C4361A17C85DDA0427F22D62A34CAFBA60DE91CF
2023-09-19T09:31:54.1694134Z  25,242,943 UART: [afmc] SIG.S = 57083EFAA8C09D86B408ABDEB8120F3A0A87C9C1DAD6EA4940C67CDFEED457B052E07185C310A798191C6F16BBB24B74
2023-09-19T09:31:54.1695724Z  25,269,252 UART: [afmc] --
2023-09-19T09:31:54.1696478Z  25,273,598 UART: [cold-reset] --
2023-09-19T09:31:54.1697093Z  25,288,461 UART: [fht] Storing FHT @ 0x50003400
2023-09-19T09:31:54.1697980Z  25,306,701 UART: [state] Watchdog Timer is not stopped because the device is not locked for debugging
2023-09-19T09:31:54.1698740Z  25,312,423 UART: [state] Locking Datavault
2023-09-19T09:31:54.1699368Z  25,317,301 UART: [state] Locking PCR0, PCR1 and PCR31
2023-09-19T09:31:54.1699945Z  25,319,377 UART: [state] Locking ICCM
2023-09-19T09:31:54.1700534Z  25,322,445 UART: [exit] Launching FMC @ 0x40000134
2023-09-19T09:31:54.1701085Z  25,331,255 UART:
2023-09-19T09:31:54.1701567Z  25,331,288 UART: Running Caliptra FMC ...
2023-09-19T09:31:54.1702036Z  25,332,124 UART:
2023-09-19T09:31:54.1702559Z  25,335,115 UART: [alias rt] Extend RT PCRs
2023-09-19T09:31:54.1703154Z  25,368,189 UART: [alias rt] Extend RT PCRs Done
2023-09-19T09:31:54.1703705Z  25,369,221 UART: [alias rt] Lock RT PCRs
2023-09-19T09:31:54.1704273Z  25,370,024 UART: [alias rt] Lock RT PCRs Done
2023-09-19T09:31:54.1704844Z  25,370,995 UART: [alias rt] Populate DV
2023-09-19T09:31:54.1705388Z  25,371,887 UART: [alias rt] Populate DV Done
2023-09-19T09:31:54.1705971Z  25,372,859 UART: Handoff : FMC CDI: 6
2023-09-19T09:31:54.1706696Z  25,373,622 UART: FMC Alias Private Key: 7
2023-09-19T09:31:54.1707310Z  25,375,525 UART: [alias rt] Derive CDI
2023-09-19T09:31:54.1707914Z  25,376,260 UART: [alias rt] Store in in slot 0x4
2023-09-19T09:31:54.1708518Z  25,409,991 UART: [alias rt] Derive Key Pair
2023-09-19T09:31:54.1709147Z  25,410,891 UART: [alias rt] Store priv key in slot 0x5
2023-09-19T09:31:54.1710111Z  28,650,087 UART: [alias rt] Derive Key Pair - Done
2023-09-19T09:31:54.1710761Z  28,661,092 UART: [alias rt] Signing Cert with AUTHO
2023-09-19T09:31:54.1711353Z  28,662,248 UART:             RITY.KEYID = 7
2023-09-19T09:31:54.1711976Z  30,940,366 UART: [alias rt] Erasing AUTHORITY.KEYID = 7
2023-09-19T09:31:54.1712966Z  30,941,865 UART: [alias rt] PUB.X = 52392ECE98D74317C65B97161A94C14FB020DCFBA7CF9EF95910E52A1E178EF526875037598894E2CFBA6C2D50DB6E3D
2023-09-19T09:31:54.1714153Z  30,949,283 UART: [alias rt] PUB.Y = FB503E4E8EF4F4EE3FF055F9879086684945D3F6FC7FB290589CD56D44D4B86800583BEC106EECB66E5E2EC694907FED
2023-09-19T09:31:54.1715394Z  30,956,852 UART: [alias rt] SIG.R = 27D5193BAA4A56402451FEF882EA18BF12FE9C8A4F613A02949AC348858A13E5FA980468ADAED803B06F40CBC7EED927
2023-09-19T09:31:54.1717754Z  30,964,252 UART: [alias rt] SIG.S = 42F11AFA12D187FCF55EC3915BBD157C62450CCB1571420F3C17739FF6C37D098D3392E3D1321CB7C38FED42492779C1
2023-09-19T09:31:54.1718607Z  32,267,768 UART:
2023-09-19T09:31:54.1719134Z  32,267,801 UART:    _____                 __     __________   __
2023-09-19T09:31:54.1719747Z  32,269,484 UART:   /     \   ____   ____ |  | __ \______   \_/  |_
2023-09-19T09:31:54.1720383Z  32,271,167 UART:  /  \ /  \ /  _ \_/ ___\|  |/ /  |       _/\   __\
2023-09-19T09:31:54.1721052Z  32,272,850 UART: /    Y    (  <_> )  \___|    <   |    |   \ |  |
2023-09-19T09:31:54.1721664Z  32,274,533 UART: \____|__  /\____/ \___  >__|_ \  |____|_  / |__|
2023-09-19T09:31:54.1722254Z  32,276,216 UART:         \/            \/     \/         \/
2023-09-19T09:31:54.1722734Z  32,277,875 UART:
2023-09-19T09:31:54.1723224Z  32,278,049 UART: Waiting for mailbox commands...
2023-09-19T09:31:54.1724216Z  32,279,143 UART: [fmc-test-harness] Received command: 0x10000001
2023-09-19T09:31:54.1724931Z  32,292,084 >>> mbox cmd response data (2048 bytes)
2023-09-19T09:31:54.1725665Z  32,293,234 <<< Executing mbox cmd 0x10000000 (0 bytes) from SoC
2023-09-19T09:31:54.1727594Z  32,293,298 UART: [fmc-test-harness] Received command: 0x10000000
2023-09-19T09:31:54.1728327Z  32,297,173 >>> mbox cmd response data (336 bytes)
2023-09-19T09:31:54.1728998Z  32,297,369 <<< Executing mbox cmd 0x10000002 (0 bytes) from SoC
2023-09-19T09:31:54.1729922Z  32,297,431 UART: [fmc-test-harness] Received command: 0x10000002
2023-09-19T09:31:54.1730994Z  32,318,903 >>> mbox cmd response data (1536 bytes)
2023-09-19T09:31:54.1731634Z  32,319,769 <<< Executing mbox cmd 0x46574c44 (23288 bytes) from SoC
2023-09-19T09:31:54.1732498Z  32,332,768 UART: [fmc-test-harness] Received command: 0x46574C44
2023-09-19T09:31:54.1733120Z  32,334,419 UART: RT Panic!!
2023-09-19T09:31:54.1733625Z  32,334,800 UART: RT Error: 0x0000DEAD
2023-09-19T09:31:54.1734223Z  32,335,537 >>> mbox cmd response: failed
2023-09-19T09:31:54.1734826Z test test_pcr_log ... FAILED
2023-09-19T09:31:54.1735165Z
2023-09-19T09:31:54.1735326Z failures:
2023-09-19T09:31:54.1735588Z
2023-09-19T09:31:54.1735739Z failures:
2023-09-19T09:31:54.1736266Z     test_pcr_log
2023-09-19T09:31:54.1736548Z
2023-09-19T09:31:54.1737020Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out; finished in 31167.99s

Probably the same root cause as https://github.com/chipsalliance/caliptra-sw/issues/786. The test is not performing hitless updates as it should.


Update readme.md

Kudos for putting this elaborate documentation together. I love Next.js.
I noticed that preset stage-0 is not a hard requirement for next.js anymore?
I think this preset can be removed from the main documentation as the language suggests next.js has a hard dependency on it.
Please correct me if I am mistaken.
Thanks!

The babelrc is illustrating an example. It should be like that 👍


Added cancellableTaskValidation feature, tests, and documentation

Proposed Changes
I added support for CancellableTaskValidation with relevant documentation and unit tests. I also fixed/updated some miscellaneous documentation
Types of changes
What types of changes does your code introduce to FsToolkit.ErrorHandling?
Put an x in the boxes that apply and remove ones that don't apply

[ ] Bugfix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[X] Build and tests pass locally
[X] I have added tests that prove my fix is effective or that my feature works (if appropriate)
[X] I have added necessary documentation (if appropriate)

Further comments
As a 'quick-fix' I brought over some of the code which was in the FsToolkit.ErrorHandling.Tests project: Expect.fs, SampleDomain.fs, and TestData.fs for easier unit test implementation (copied and modified from AsyncValidation). I didn't know you wanted to create a shared test library for reduced code or if you wanted to keep things siloed from each other. I am open to either implementation

❤ This is so awesome.

As a 'quick-fix' I brought over some of the code which was in the FsToolkit.ErrorHandling.Tests project: Expect.fs, SampleDomain.fs, and TestData.fs for easier unit test implementation (copied and modified from AsyncValidation). I didn't know you wanted to create a shared test library for reduced code or if you wanted to keep things siloed from each other. I am open to either implementation

Yeah I've gone back and forth on this. Do whatever you feel is easier at this point. It's not much work to clean up later.

I'm fine leaving it for now, and we can always decide later when looking at the job validation since that would technically need similar code


ESP32-C3 libstdc++ is missing some cmath C++11 functions (IDFGH-5009)

Environment

Development Kit:     ESP32-C3-DevKitM 1
Kit version (for WroverKit/PicoKit/DevKitC): [v1]
Module or chip used:  ESP32-C3
IDF version v4.4-dev-4-g73db14240
Build System:        idf.py
Compiler version riscv32-esp-elf-gcc.exe (crosstool-NG <IP_ADDRESS>_64eb9ff) 8.4.0
Operating System:     windows
(Windows only) environment type: [PowerShell
Using an IDE?: No, idf.py
Power Supply:        USB

Problem Description
Build TFLite Hello world  in EP32-C3 return
../components/tfmicro/tensorflow/lite/kernels/internal/cppmath.h:37:41: error: 'round' is not a member of 'std'
DECLARE_STD_GLOBAL_SWITCH1(TfLiteRound, round);
//Detailed problem description goes here.
When I try to build  this example https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/micro/examples/hello_world#deploy-to-esp32
in an ESP-32-C3, I get an error
../components/tfmicro/tensorflow/lite/kernels/internal/cppmath.h:37:41: error: 'round' is not a member of 'std'
DECLARE_STD_GLOBAL_SWITCH1(TfLiteRound, round);
If the target is ESP-32 there isn´t any problem
Expected Behavior
Get a binary file
Actual Behavior
Get an error
Steps to reproduce
I follow the instructions to deploy from the example. For ESP32-c3  I use  idf.py set-target esp32-c3
For ESP32 idf.py set-target esp32
After idf.py build
// If possible, attach a picture of your setup/wiring here.
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/micro/examples/hello_world#deploy-to-esp32
Debug Logs
FAILED: esp-idf/tfmicro/CMakeFiles/__idf_tfmicro.dir/tensorflow/lite/micro/kernels/activations.cc.obj
ccache C:\Users\roberto.espressif\tools\riscv32-esp-elf\<IP_ADDRESS>_64eb9ff-8.4.0\riscv32-esp-elf\bin\riscv32-esp-elf-g++.exe   -Iconfig -I../components/tfmicro -I../components/tfmicro/third_party/gemmlowp -I../components/tfmicro/third_party/flatbuffers/include -I../components/tfmicro/third_party/ruy -I../../../../components/newlib/platform_include -I../../../../components/freertos/include -I../../../../components/freertos/port/riscv/include -I../../../../components/esp_hw_support/include -I../../../../components/esp_hw_support/port/esp32c3/. -I../../../../components/esp_hw_support/port/esp32c3/private_include -I../../../../components/heap/include -I../../../../components/log/include -I../../../../components/lwip/include/apps -I../../../../components/lwip/include/apps/sntp -I../../../../components/lwip/lwip/src/include -I../../../../components/lwip/port/esp32/include -I../../../../components/lwip/port/esp32/include/arch -I../../../../components/soc/include -I../../../../components/soc/esp32c3/.

Hi @Robertou2,
Thanks for reporting this and providing a clear description of the error. Although we don't support third party projects, I can reproduce this issue using a simple C++11 test case so I've renamed the title.
#include <cmath>

int round_test(void)
{
    return std::round(3.33);
}

This seems like it will need to be fixed by updating the libstdc++ that is bundled with the ESP32-C3 toolchain. Will update once this is available.

Hi @projectgus,
thanks for your fast answer.

Thanks for letting us know about the workaround, @Robertou2 . This is still a bug in ESP-IDF (math headers should "just work"), so happy to keep it until we solve the root cause.


Explore: Hierarchical Namespaces for Orgs and Spaces

While investigating the efficiently listing a resource story we noticed a tool that might help with our Orgs and Spaces design. The tool is Hierarchical Namespaces and as the name suggests it lets you create a hierarchical relationship between namespaces, which can be used to represent orgs and spaces. The tool itself works by propagating Roles and RoleBindings (by default; it can also be used to propagate any resource), which sounds like it can be a direct replacement to the Org and Space CRDs that we spiked out previously. It can also give users permissions to create subnamespaces without actually having the permission to create cluster-wide namespaces.
Lets re-evaluate our spike assuming we choose to use this tool.
NOTE: The project states that it is not yet GA, however it looks like the v1.0 is likely to be released by the end of the year.

Notes
With the Hierarchical Namespaces controller (HNC), we can completely replace our previously suggested implementation of orgs, spaces and users. Below we briefly describe the HNC functionality, and how we can apply that to the org / space and permissions domain in CF.
Hierarchical Namespaces Overview
The tool allows construction of a namespace tree, and by default propagates Roles and RoleBindings created in a namespace to all descendant namespaces in the tree.
Any namespace can be included in a namespace tree by defining a parent/child relationship, but an interesting feature of HNC is that of sub-namespaces. It is possible to give permission to a user to create a sub-namespace from a particular namespace. This uses the namespaced SubNamespaceAnchor resource. HNC will use this to create a k8s namespace, labelled as a sub-namespace for HNC. Thus a user without namespace creation permissions can actually create sub-namespaces of a particular namespace or sub-namespace.
HNC comes with a kubectl plugin that can be used to visualise and manage the namespace trees. For example:
❯ kubectl hns tree cf
cf
├── [s] org1
│   ├── [s] space1
│   └── [s] space2
└── [s] org2
    ├── [s] space3
    └── [s] space4

[s] indicates subnamespaces

Applying HNC to CF Orgs, Spaces and User Roles
Let us suppose we bootstrap the system to have a root namespace called cf. This is a normal namespace, but the name is important. We will use it to bind permission for admin users.
cf create-org
Consider the command cf create-org ORG. This is available only to users with the global admin role. To implement this, we can create a sub-namespace of cf with the org name. The kubectl plugin command to achieve this would be kubectl hns create -n cf ORG.
We can restrict this to the admin role by creating a role binding for the admin user in the cf namespace to a role that allows the create verb on the hnc.x-k8s.io/subnamespaceanchors resource.
cf create-space
Now users with the admin or org-manager role can create spaces in an org. So we could implement the command cf create-space -o ORG SPACE with the equivalent of kubectl hns create -n ORG SPACE, creating SPACE as a sub-namespace of ORG. Note that some naming work is required to ensure the uniqueness of SPACE.
The user must have create permissions on the hnc.x-k8s.io/subnamespaceanchors resource in the ORG namespace. Now the admin user will already have this, because the RoleBinding giving that permission on the cf namespace will have been propagated by HNC to the ORG namespace automatically. For an org-manager user to have permission, we must ensure we have a RoleBinding in the ORG namespace to a role allowing the create verb on the hnc.x-k8s.io/subnamespaceanchors resource.
Restricting Tree Depth
With the setup described above, an admin user, or an org-manager user in the ORG organisation, would have the ability to create a sub-namespace of SPACE. This doesn't make sense in the CF model. So we would like to restrict the propagation of roles to disallow this.
For the org-manager RoleBinding, this is possible by adding the annotation propagate.hnc.x-k8s.io/none: "true".
For the admin RoleBinding, instead we can use a label selector annotation such as propagate.hnc.x-k8s.io/select: "cloudfoundry.org/org=true". We must then label all the org-level namespaces with the label cloudfoundry.org/org: "true". Then the admin namespace creation permission will be propagated to all orgs, but no lower.
cf delete-space
Space deletion permissions can also be controlled using the hnc.x-k8s.io/subnamespaceanchors resource. So the equivalent of kubectl delete hnc.x-k8s.io/subnamespaceanchors -n ORG SPACE is sufficient to make HNC delete the namespace for the space. This will remove all apps in the space, and remove all role bindings, hence space users.
cf delete-org
It is also possible for a admin user to delete an org. By default, HNC will prohibit namespace deletion when the namespace has child spaces. But it is possible to set an attribute on the namespace to allow cascading deletion. E.g. kubectl hns set --allowCascadingDeletion. Then removing the hnc.x-k8s.io/subnamespaceanchors resource for the ORG will trigger a cascading deletion of the org and its spaces.
User Management
Just as there is no need for org or space custom resources when using HNC, we can also do without any user custom resources. They can be completely implemented with RoleBindings.
To create an admin user, we need to create two RoleBindings in the cf namespace:

one granting all the admin permissions on the CF custom resources
one granting the SubNamespaceAnchor management permissions, with the propagate.hnc.x-k8s.io/select: "cloudfoundry.org/org=true" annotation.

To create an org-manager user, we need to create two RoleBindings in the ORG namespace, where ORG is the organisation associated with the org-manager:

one granting all the org-manager permissions on the CF custom resources
one granting the SubNamespaceAnchor management permissions, with the propagate.hnc.x-k8s.io/none: "true" annotation.

All other users are created as RoleBindings at the appropriate level:

global roles (admin, admin-read-only, global-auditor) are created in cf

they are automatically propagated to child org and space sub-namespaces, both those that exist at the time of creation and those subsequently created

org level roles are created in their ORG namespace

they are automatically propagated to space sub-namespaces

space level roles are created in their SPACE namespace, and apply only there.

We've updated the Org support proposal with the results of this explore. This explore however does not cover the multicluster support that the proposal doc has. We think that it could be done with kubefed, although this would be difficult to do. We are also not sure how balancing resources across clusters would work, as there is no "out of the box" solution.


KV store refactor

In order to fix #22

Test passed on DO.
Drone currently have performance issues and timeout on running the test.