[Language Text] Remove FHIR and Auto Language Detection Feature

Packages impacted by this PR
ai-language-text
Issues associated with this PR
Remove FHIR in healthcare task and auto language detection feature
Describe the problem that is addressed by this PR
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Are there test cases added in this PR? (If not, why?)
Provide a list of related PRs (if any)
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[ ] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-ai-language-text


Add unlock function to provide consistency in locks manipulation

This modification was asked by Ariel and i had forgotten to make it.

This modification was asked by Ariel and i had forgotten to make it.

Im sorry, i ve made a mess with the code formatter. I add a function to unlock the locks using this function instead of asing them a 0 directly


Applying force/torque to RigidBody3D does not reset its sleep timer

when using apply_force() on a RigidBody3D in the _physics_process, every 30 frames the body falls asleep and wake up again, causing a jittery movement.
setting can_sleep to false fixes the lag.

Please provide a minimal reproduction project that showcases the issue.

Oh, nevermind, I managed to reproduce it pretty easily.
That is indeed a bit strange. I would have thought that waking up the body as part of applying the force would reset the sleep timer of the body, but I guess that's not in fact happening?
I'll look into it.

It seems this was just me making too many assumptions about the Jolt API, which explicitly does an early out if the body is already active. There is a separate ResetSleepTimer method instead.
@jrouwe Obviously changing this would be a breaking change, but is there some performance reason for this? It strikes me as a little bit odd. If someone is explicitly calling ActivateBody/ActivateBodies, wouldn't the expectation be for it to reset the sleep timer if the body is already active?

That part I can understand, which is why I explicitly wake the body up at any applied force/torque already. If I don't, the body ends up being locked in-place until the user explicitly wakes it up (or some collision response happens I guess), which is not how Godot Physics behaves.
Interestingly it seems that Godot Physics is still able to somewhat handle the case you mentioned, where if I apply some very small force, then let the body settle, it'll go to sleep, but if I then amp up the force applied to it then it'll still react to it appropriately, without needing to be woken up explicitly.
So I guess the discrepancy at play here is less about me needing to reset the sleep timer every time a force is applied, but instead the fact that the sleep timer is in fact reset when activating a body with Jolt.
In fact, this comment seems to tell the story pretty well. Basically, Godot only resets the sleep timer if the sleep test fails, which happens at the end of the physics step, so a body can have a force applied to it, wake up and then fall asleep again, all within the same step.
I'm assuming I can't replicate that behavior without modifying Jolt, if it's even worth replicating to begin with, so perhaps just resetting the sleep timer myself is the better option.

What I was referring to in my previous comment was calling ActivateBody on an already active body, and not having its sleep timer reset.

Ah, sorry, I misunderstood. I think you have a valid point here, but I will need to do some investigations to see what side effects this would have. I'm also thinking about adding a parameter BodyInterface::AddForce(..., EActivation inActivationMode) to give the caller control if the body should be activated or not.

Basically, Godot only resets the sleep timer if the sleep test fails, which happens at the end of the physics step, so a body can have a force applied to it, wake up and then fall asleep again, all within the same step, hence why it looks like it's sleeping despite having forces applied to it.

This would be an option, but quite an expensive one because effectively you're waking up the body every frame (=overhead), simulate it, make it sleep again (=overhead) meaning that effectively the body is only reporting that it is sleeping while actually it is simulated and costing even more than a normal simulated body.

I'm assuming I can't replicate that behavior without modifying Jolt, if it's even worth replicating to begin with, so perhaps just resetting the sleep timer myself is the better option.

Indeed, BodyManager::ActivateBodies calls ResetSleepTimer so it can never go to sleep in the same frame.

Yeah, I caught it. :)
Thank you, much appreciated!

The fix for this can now be found in the latest pre-release, 0.13.0-beta1, which you'll find here.
Note that this version targets the just recently released Godot 4.3-beta1, so won't work with Godot 4.2. Also note that this version is not available through the Godot Asset Library, so you'll need to download it through GitHub.
A proper release will be out some time around Godot 4.3-stable.

I figured I'd give a minor update here, as 0.13.0-stable has just been released, so if you've been relying on 0.13.0-beta1 for this functionality until now I would highly recommend updating, as this most recent release includes some important bug fixes.
Note that the release on the Godot Asset Library is still pending approval as of typing this, so you might need to download it from the link above.


Don't change the indentation of secrets

Thanks for the interesting tool!
I want to store yaml and json data exactly as they are without changes to their indentation:
{
  "foo": "bar"
}

Indentation in json is not very important, but can become ugly when the json data is larger.
version: v1alpha1
metadata:
  name: secret-data
spec:
  data:
    - hello world

Expected
The expected outcome is that I get the same data back, without change in indentations. Especially yaml should stay exactly the same because the syntax depends on indentation.
Actual result
{
"foo": "baar"
}

version: v1alpha1
metadata:
name: secret-data
spec:
data:
- hello world

It might be that the fault lies in the underlying encryption package.

Hi @Xcalizorz, thanks for filing the issue.
I looked into it and it seems as though the textarea in which we display the decrypted secret incorrectly implements the white-space CSS property.
It should indeed be pre or pre-wrap.
Myself or @rickymarcon will look into fixing this shortly!

Thanks for picking this up @Xcalizorz! We'll rectify this! 🚀

@Xcalizorz this is now resolved. As @slavovojacek mentioned, the root cause was CSS in the browser. Closing for now. Let us know if you run into any other issues and thanks for the post!


MF_NON_INTERACTIVE does not seem to work at all

Currently we manually set MF_NON_INTERACTIVE to true under CI, and try to detect it otherwise. But I'm pretty sure the detection is not working at all. After looking into it, I'm not even sure it's possible to detect an interactive shell under make. None of the suggestions I have seen hold up to actual testing (e.g. https://stackoverflow.com/questions/4251559/how-can-i-tell-if-a-makefile-is-being-run-from-an-interactive-shell).
Unless @jmalloc you have any ideas about how to actually detect non-interactivity under make, perhaps we should either:

Stop passing "non-interactive" options to commands completely, and let the individual commands handle it; OR
Always pass "non-interactive" options to commands; OR
Always pass "non-interactive" options to commands under CI, but never outside of CI

The following Makefile:
DEFAULT_GOAL := test

export EXAMPLE_A := $(shell [ -t 0 ] || echo true)
export EXAMPLE_B := $(shell [[ $$- == *i* ]] && echo true)
export EXAMPLE_C := $(shell echo $$-)

test:
	echo "EXAMPLE_A=$(EXAMPLE_A)"
	echo "EXAMPLE_B=$(EXAMPLE_B)"
	echo "EXAMPLE_C=$(EXAMPLE_C)"

Gives the following output under various interactive and non-interactive circumstances:
$ make
echo "EXAMPLE_A="
EXAMPLE_A=
echo "EXAMPLE_B="
EXAMPLE_B=
echo "EXAMPLE_C=hBc"
EXAMPLE_C=hBc

$ bash -c make
echo "EXAMPLE_A="
EXAMPLE_A=
echo "EXAMPLE_B="
EXAMPLE_B=
echo "EXAMPLE_C=hBc"
EXAMPLE_C=hBc

$ make > /dev/stdout
echo "EXAMPLE_A="
EXAMPLE_A=
echo "EXAMPLE_B="
EXAMPLE_B=
echo "EXAMPLE_C=hBc"
EXAMPLE_C=hBc

Based on our discussion this morning it seems like our logic might be lacking there.
Here are some example expression that checks both STDIN and STDOUT are TTYs and hence likely to be interactive:
xxx:
	[[ -t 0 && -t 1 ]] && echo YES

Both TTYs:
$ make
[[ -t 0 && -t 1 ]] && echo YES
YES

STDIN not a TTY:
$ echo foo | make
[[ -t 0 && -t 1 ]] && echo YES
make: *** [xxx] Error 1

STDOUT not a TTY:
$ make | cat
[[ -t 0 && -t 1 ]] && echo YES
make: *** [xxx] Error 1

It might be the case that performing this test directly in the recipe like this behaves differently to the way another sub-shell would detect it's STDIN/STDOUT too.  If we can't get this working I would opt for:

Always pass "non-interactive" options to commands;

It might be the case that performing this test directly in the recipe like this behaves differently to the way another sub-shell would detect it's STDIN/STDOUT too.

Yeah, I think that was part of it too. We do the detection inside a $(shell) call currently, which could change the results. I'll give it a test.

Confirmed that $(shell) changes the results. Given the following Makefile:
RESULT := $(shell [[ -t 0 && -t 1 ]] && echo interactive || echo non-interactive)

test:
	@echo Direct:
	@[[ -t 0 && -t 1 ]] && echo interactive || echo non-interactive
	@echo Shell call:
	@echo $(RESULT)

I get the following results:
$ make # both TTYs
Direct:
interactive
Shell call:
non-interactive

$ echo | make # STDIN not a TTY
Direct:
non-interactive
Shell call:
non-interactive

$ make | cat # STDOUT not a TTY
Direct:
non-interactive
Shell call:
non-interactive


TimeGenerated date/time format

My customer reported that date/time format for TimeGenerated (exported in csv) was changed as below at some point. And I would like to make sure if it's changed on purpose and will not change in the future.
I know this is not a feedback for content, but I couldn't find TimeGenerate specific content in other pages.
OLD: "2020-10-15T00:00:00.247Z"
NEW: "2020/10/16 0:00:01.403"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bceaf516-a1b0-eb1e-e106-62c31db3e69c
Version Independent ID: abbbd2d6-e0fc-d370-f4a7-ac669fc4acd6
Content: Monitoring Azure Functions with Azure Monitor Logs
Content Source: articles/azure-functions/functions-monitor-log-analytics.md
Service: azure-functions
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

@kazshi Thanks for the question! We are investigating and will update you shortly.

Hello,
Yes, this was a change intended to match the UI and exported format, to let users see date time values aligned.
This isn't something that we plan to change further.
Thanks,
Yossi

Thank you @yossi-y for the response, this is very helpful.
@kazshi, I will now proceed to close this issue out. Please tag me in you reply if there are any additional concerns, I will be glad to continue with our discussion. Thank you!


Dashboard should display all posts from all collections

Right now, navigating to the dashboard (http://localhost:8081/admin/#/) will display only posts from the first collection in the list of collections. We should either redirect there or show all posts on this page.

I don't think we should display everything, the dashboard should show a summary of what you have and can do. Showing all posts could be very unfriendly if you have lots and lots of articles.

Yes, I agree dashboard should not display everything. But in the current implementation, it is even more confusing since it is displaying post from the first category and very is no visual clue to that so one can think he's looking at all posts.

We will make the dashboard an independent area. Right now, it will contain only the Editorial Workflow card board, but the idea is to grow in the future.


Update benchmarks.md

fix wrong spelling in Chinese translation benchmarks.md

What's next step? Is that all? I have no idea about PR in Github.-_-@waynerv

Yeah, you just need to wait until the creator merges it.

I see this PR is closed, not sure if there was something else needed here, but I imagine it's no longer needed in some way. Anyway, thanks for the effort! 🍰

Sorry for the long delay! 🙈  I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.


support gzipped data file on 3d-tiles branch?

Hi,
I just noticed you are now supporting to serve gzipped data file on the 3d-tiles-samples server, and I tried it out and found it is a really cool fix. I am wondering if it is an easy fix for your 3d-tile branch here to support gzipped data file as well. Thanks!

I should have started the conversation here instead of 3d-tiles-samples. See @mramato's comment: https://github.com/AnalyticalGraphicsInc/3d-tiles-samples/pull/11

I'm not sure why we would add support for it in Cesium, since we don't include any gzipped tilesets or data, right?
It sounds like they are using the NYC sample data we have up on cesiumjs.org, in my opinion we should change that sample to not include gzipped data.
That being said, if we just want to make our users happy and avoid any potential headaches, I'm not again adding it "just because", but I do have some minor performance concerns, since it means every files gets read twice.
Perhaps we can punt for now and revisit when we handle the http2 PR. The server is long overdue for an update either way.


[Backport 2.x] Introducing ZStd compression codec plugin (#9658)

Backports 76f1b52647cf19bd19e3ce9d0759953b5d376593 from #9658 to 2.x

@reta This is ready to go. Thanks!


Issue with Kusama sent from Bifrost wallet to my Ledger

Hi guys, really hope someone can help me out here. I've sent my Kusama from my Bifrost wallet directly to my Ledger Nano address showing in the Kusama substrate portal. Now they don't show up in my Ledger account. Looking at the transaction in Subscan I can see that they have been sent to a completely different address that is somehow connected to my Ledger address when I look it up on the Sub ID website. Does anyone have experience with this and know how I can access my Kusama? :-(
Thanks!

:link: zboto Link

Follow this tutorial on Kusama app, and you should be able to recover your funds.
Feel free to reopen if this doesn't solve your issue!
Thanks for your contribution!


Event Request - COD90.OnBeforeCheckExternalDocumentNumber, OnBeforeInvoiceRoundingAmount, OnAfterFillInvoicePostBuffer

Please add below events to Codeunit 90
Event signatures:
OnBeforeCheckExternalDocumentNumber@12047959(VAR VendLedgEntry@12047959 : Record 25;VAR PurchaseHeader@12047958 : Record 38;VAR Handled@12047960 : Boolean);

OnBeforeInvoiceRoundingAmount@12047961(PurchHeader@12047961 : Record 38;TotalAmount@12047960 : Decimal;UseTempData@12047959 : Boolean;VAR InvoiceRoundingAmount@12047958 : Decimal);

OnAfterFillInvoicePostBuffer@12047960(PurchLine@12047959 : Record 39;VAR InvoicePostBuffer@12047958 : Record 49);

Call the events from:
FillInvoicePostBuffer:
InvoicePostBuffer."Deferral Code" := PurchLine."Deferral Code";
OnAfterFillInvoicePostBuffer(PurchLine,InvoicePostBuffer);
UpdateInvoicePostBuffer(TempInvoicePostBuffer,InvoicePostBuffer);
FillDeferralPostingBuffer(PurchHeader,PurchLine,InvoicePostBuffer,AmtToDefer,AmtToDeferACY,DeferralAccount,PurchAccount);

InvoiceRounding:
LOCAL InvoiceRounding(PurchHeader : Record "Purchase Header";VAR PurchLine : Record "Purchase Line";UseTempData : Boolean;BiggestLineNo : Integer)
Currency.TESTFIELD("Invoice Rounding Precision");
InvoiceRoundingAmount :=
  -ROUND(
    TotalPurchLine."Amount Including VAT" -
    ROUND(
      TotalPurchLine."Amount Including VAT",
      Currency."Invoice Rounding Precision",
      Currency.InvoiceRoundingDirection),
    Currency."Amount Rounding Precision");
OnBeforeInvoiceRoundingAmount(PurchHeader,TotalPurchLine."Amount Including VAT",UseTempData,InvoiceRoundingAmount);
IF InvoiceRoundingAmount <> 0 THEN BEGIN

CheckExternalDocumentNumber:
LOCAL CheckExternalDocumentNumber(VAR VendLedgEntry : Record "Vendor Ledger Entry";VAR PurchaseHeader : Record "Purchase Header")
OnBeforeCheckExternalDocumentNumber(VendLedgEntry,PurchaseHeader,Handled);
IF Handled THEN
  EXIT;
VendLedgEntry.RESET;

Delta file built on CTP16 W1:
COD90.txt

Hello,
for the OnAfterFillInvoicePostBuffer event, it would be better to also add the TempInvoicePostBuffer parameter like:
OnAfterFillInvoicePostBuffer(PurchLine,InvoicePostBuffer,TempInvoicePostBuffer);
Thanks.


JS/Python API for scripting

We have discussed an API to promote scripting with DeepForge. This issue is meant to track design discussion, etc, around this API.
So far, I have been thinking using gRPC as this would allow the API to be largely generated from a protobuf file. However, one shortcoming would be that gRPC is not available in the browser without a gateway proxy (grpc-web).
Tasks that would be useful to be able to script:

Importing/uploading artifacts
Executing pipelines
Downloading artifacts
Creating pipelines?
Creating operations?
Creating utilities?

Yeah this looks like a great feature to have.
But does grpc-web proxy has to reside in client machine or its something that needs to handled from the server side?

They use envoy (similar to nginx) which runs on the server side: https://www.envoyproxy.io/

Another useful feature would be mounting projects on the filesystem so they can be edited as text. On linux, this could be achieved by creating a custom FUSE adapter though this wouldn't address all the challenges. For example, in order to add a reference to an operation, the operation interface editor is currently required. This could be changed by adding support for type annotations to differentiate btwn different types of references or attributes.


WIP Integrations API Proposal (+ whichkey and mapper integrations)

Hi LionC, this is an preliminary implementation and proposal for an integrations API.  Right now this works but I'm sure it has a lot of bugs that need to be ironed out and plenty of room for improvement.  I'm looking for feedback on this architecture to make sure that I'm heading in the right direction.  I want to make sure that we land on a solid, easy to use API on release so everything is open to criticism and feedback :)
Proposed API
nest.enable(require('nest.integrations.mapper'))
nest.enable(require('nest.integrations.whichkey'))

nest.applyKeymaps { '<leader>', {
        { 'm', '<cmd>Telescope mapper<cr>', 'Mapper'},
        { 'p', require'telescope.builtin'.planets, 'Planets' },
        -- Prefix every nested keymap with f (meaning actually <leader>f here)
        { 'f', name = '+File', {
            { 'f', '<cmd>Telescope find_files<cr>', 'Find Files' },
            -- This will actually map <leader>fl
            { 'l', '<cmd>Telescope live_grep<cr>', 'Search Files', },
            -- Prefix every nested keymap with g (meaning actually <leader>fg here)
            { 'g', name = '+Git', {
                { 'b', '<cmd>Telescope git_branches<cr>', 'Branches' },
                -- This will actually map <leader>fgc
                { 'c', '<cmd>Telescope git_commits<cr>', 'Commits' },
                { 's', '<cmd>Telescope git_status<cr>', 'Status' },
            }},
        }},

Integration API
-- This is an example integration to help future users implement their own nest.nvim integration
-- This integration will log your nest config for you by pretty printing your applyKeymaps config
--
--- @type NestIntegration
local module = {}
module.name = 'example';

-- Utility length function
local len = function(table)
  local length = 0;
  for _, _ in ipairs(table) do
    length = length +1
  end
  return length
end

--- This is run before the keymaps are bound
--- Use it to setup data structures before the applyKeymaps config is traversed
--- @param config table<number, NestNode>
module.on_init = function(config)
  print('nest.nvim starting keymap binding');
end

--- This will be called for every element in the applyKeymaps config.
--- Here you can pass the nest.nvim config to whatever library you're integrating with
--- or build a different representation of the data to be applied in the on_complete function.
--- @param node NestIntegrationNode
--- @param node_settings NestSettings
module.handler = function (node, node_settings)

  -- If node.rhs is a table, this is a group of keymaps, if it is a string then it is a keymap
  local is_keymap_group = type(node.rhs) == 'table'

  if is_keymap_group then
    print('nest.nvim: ' .. node.lhs .. ' is group of ' .. len(node.rhs) .. ' keymaps.')
  else
    print('nest.nvim: ' .. node.lhs .. ' is keymap binding to ' .. node.rhs .. '.')
  end

  -- You can get different keymap settings from the node_settings object
  local msg = '    buffer: ' .. string.format('%s', node_settings.buffer) .. ' | mode: ' .. node_settings.mode .. ' | '
  if (node_settings.options.noremap) then
    msg = msg .. 'noremap,'
  end
  if node_settings.options.silent then
    msg = msg .. 'silent,'
  end
  if node_settings.options.expr then
    msg = msg .. 'expr'
  end
  print(msg)
end

--- This is run before the keymaps are bound
--- Use it cleanup or apply data that was created in the handler/on_init functions
module.on_complete = function()
  print('nest.nvim: finished keymap binding.');
end

return module;

Change list:

Allows specifying a name for a keymap using either the third element in the table or the name property
Allows specifying a description for a keymap using either the fourth element in the table or the description property.
Added enable function to nest module allowing users to pass in integrations with different plugins.
Split applyKeymaps into applyKeymaps and traverse (generic function for traversing over they keymap config)

Needed to facilitate the on_init/on_complete functions on integrations
Also handy having traverse seperate as it allows the user to run or avoid running a specific integration if they need to for a one off.

Added a (very basic and untested) whichkey integration
Added a (very basic and untested) nvim-mapper integration

Points of concern

I will probably do another pass on the names for the api when I've had more sleep, but if you have any feedback that'd be much appreciated.
I can remove the Makefile etc for release, is just helpful for development.

How to test

Checkout my fork in a folder that also has plenary.nvim, whichkey, telescope and nvim mapper in it.
Cd into nest.nvim
Run make test

Hey @LionC, any chance you can check this out on the weekend?

Looking at it now - sorry for the delays, might take a bit longer in total

Hey again @LionC, no worries!  It's a very busy time of year.
I've done some refactoring on top of my previous changes, and over-all I'm pretty happy with everything now.  I've tested the integrations and it looks like everything is working ok including buffer keybindings which was my main point of concern.  I've also improved the PR to make the integrations api.
I'll describe the changes of the whole PR to help the review process:

Added "integrations"

An integration takes the nest.nvim API and does something.
Has a on_init (before traversing nest config) on_complete (after traversing nest config) and handler (handles a keymap or keymap group) function.
Check lua/nest/integrations/example.lua for an example

Added an enable function that allows user to enable an integration.

Example: nest.enable(require('nest.integrations.example'))
This must be run before applyKeymaps to take effect.

Moved the default behaviour of binding keymaps into its own integration that is automatically enabled.
Split applyKeymaps into two functions

applyKeymaps runs the on_init of all integrations, then traverses the config, then runs on_complete of all integrations.
traverse navigates over all nodes of the nest config and passes the data to the integrations.

Removed duplicated functionality

copy and merge functions are duplicates of existing nvim api.

Documented integrations in including example config README.md

Also it'd be really cool to have  debug integration that performs more checks on the provided nest config to catch any errors, like right now I don't think nest.nvim can handle nils in the rhs (occurs when referencing a lua function that doesn't exist).
Let me know what you think! 😄

First of all: thank you a lot for the amount of work and thought put into this. I really appreciate the architectural ideas and the written explanations to go with it.
I think I like the overall approach, it goes in a very similar direction to where I was thinking but a lot more thought out and actually done^^ I think integrations / modules / whatever will need to be able to apply public APIs as well, that is at least how I started drafting a solution for #24 and I think it fits this design well enough.
It will take some more days for me to find the time to review this line by line. The only concern I could maybe see with that architecture is that the core "key binds only" functionality (which will stay the most popular use) is affected by this performance wise, but if that is insignificant, it should be fine.

I'm trying the whichkey integration, and I really like it !
Thank you !

Hey, thanks for reviewing the PR.  I really appreciate it.
Yep that's right, I'll annotate some examples with the categories.
{ 't', name = 'telescope', {
    { 'f', name = 'find', {
        {'f', 'Telescope find_files'}, -- (category = find)
        {'g', 'Telescope live_grep'} -- (category = find)
    }},
    { 'a', 'foo' }, -- (category = telescope)
    { 'b', 'bar' } -- (category = telescope)
}}

However, right now if the name for the old find keymap group is nil the category will be unknown.
{ 't', name = 'telescope', {
    { 'f', {
        {'f', 'Telescope find_files'}, -- (category = unknown)
        {'g', 'Telescope live_grep'} -- (category = unknown)
    }},
    { 'a', 'foo' }, -- (category = telescope)
    { 'b', 'bar' } -- (category = telescope)
}}

Maybe this is something I should fix using the same merging/overwriting behaviour as the keymap settings (noremap/silent etc).


Initial docs

MDS Provider deserves some docs.
This PR creates a docs directory, where you can place .md or .rst files to generate documentation. You can then serve it on the web using github pages.
The structure is as follows.
├── Makefile
├── conf.py
├── index.rst
└── validate.md

Use make build to build html docs. conf.py defines some settings.
Additionally, this PR adds a Pipfile with just dev dependencies.

@hunterowens this is really cool! I was finally able to get make to work on Windows, and build and view these docs as HTML.
I've setup GitHub pages for static HTML and Jekyll sites, but not for something like this with custom build commands. I found this write-up of a possible flow but it doesn't get in to automation.
Do you have experience with that? Would CircleCI or Travis or something be able to help here?
@allejo maybe you've setup a python project with GH pages docs that build on commit/PR, or something similar?

I see that Sphinx also includes an extension githubpages but I'm not sure if/how this helps.

@thekaveman read the docs is your friend. I'll add a link in the README

Also re: make on windows, you can install a ubuntu VM from the windows store now, which is how I use ssh, make and other tools on my work computer.

rebased on master

Rebased on the forthcoming release in mds-0.3.
Added a small Docker setup for building and serving docs locally in a known environment.


New options for cloud-init: Azure datasource and reporting/logging

Added support for configuring the Azure datasource.  Only options currently needed are defined.  More can be added later. Each defined datasource has to be listed in a top-level array in the config file.  This is added automatically.
Added support for configuring reporting handlers.  It's likely that the names of the handlers can be arbitrary strings, but I couldn't confirm this, so I added the known names.

Setting to draft for tests.

Added tests and verified the contest of the files in the tree from the result of running cloud-init/b.json:
❯ ssh -q localvm 'head -n-0 .osbuild/refs/3438b827bc2ea813ecc0666e48436933b6aa207fec57913a766eaceac3fcd72c/etc/cloud/cloud.cfg.d/{00-default_user,10-azure-kfp,91-azure_datasource}.cfg'
==> .osbuild/refs/3438b827bc2ea813ecc0666e48436933b6aa207fec57913a766eaceac3fcd72c/etc/cloud/cloud.cfg.d/00-default_user.cfg <==
system_info:
  default_user:
    name: ec2-user

==> .osbuild/refs/3438b827bc2ea813ecc0666e48436933b6aa207fec57913a766eaceac3fcd72c/etc/cloud/cloud.cfg.d/10-azure-kfp.cfg <==
reporting:
  logging:
    type: log
  telemetry:
    type: hyperv

==> .osbuild/refs/3438b827bc2ea813ecc0666e48436933b6aa207fec57913a766eaceac3fcd72c/etc/cloud/cloud.cfg.d/91-azure_datasource.cfg <==
datasource:
  Azure:
    apply_network_config: false
datasource_list:
- Azure

NB: I think we should use default_flow_style=False for dump.


Remove HTTP round trip for determining blob size and get blob sizes from layer descriptors

Is your feature request related to a problem? Please describe.
When the snapshotter resolves a blob/layer, it needs to know the blob size. Currently the snapshotter does an HTTP round trip call just to get the blob size. This HTTP round trip is not needed as we can/should get the blob size from the image manifest.
Describe the solution you'd like

Remove the http round trip for getting blob size: https://github.com/awslabs/soci-snapshotter/blob/main/fs/remote/resolver.go#L262-L263
Get the blob size from manifest instead

Describe alternatives you've considered
None.

The change is merged. Resolving.


RCU CPU stall warning in a multi-core system simulation

Execute semu with multi-core system simulation
make check SMP=4

The RCU CPU stall warning, as discussed in #49 , is accompanied by an increase in timer interrupts and clock_gettime system calls to produce a real-time timer. This causes the CPU to wait longer than a typical grace period, which is usually 21 seconds.
[   33.044071] rcu: INFO: rcu_sched self-detected stall on CPU
[   33.048071] rcu: 	2-....: (5247 ticks this GP) idle=da1c/1/0x40000002 softirq=689/689 fqs=1847
[   33.056071] 	(t=5253 jiffies g=-887 q=11 ncpus=4)
[   33.064071] CPU: 2 PID: 8 Comm: kworker/u8:0 Not tainted 6.1.94 #12
[   33.068071] Hardware name: semu (DT)
[   33.072071] Workqueue: events_unbound async_run_entry_fn
[   33.080071] epc : eat+0x34/0x54
[   33.088071]  ra : do_reset+0x38/0x6c
[   33.096071] epc : c03811fc ra : c038137c sp : c0871e40
[   33.104071]  gp : c04de858 tp : c0852f40 t0 : c0aabc70
[   33.112071]  t1 : 00000008 t2 : 67e0cb47 s0 : c0871e50
[   33.120071]  s1 : c03a48b8 a0 : 00000001 a1 : 00000000
[   33.128071]  a2 : 00000000 a3 : 0053be47 a4 : 00000000
[   33.136071]  a5 : c03a48b8 a6 : 00000000 a7 : 0000000c
[   33.144071]  s2 : 004725a7 s3 : 0038da58 s4 : 00000000
[   33.152071]  s5 : 00000030 s6 : 00000007 s7 : c03a496c
[   33.156071]  s8 : c03a48b8 s9 : c038112c s10: c03a4968
[   33.160071]  s11: c0381538 t3 : c09e4140 t4 : 00000003
[   33.168071]  t5 : ec37d331 t6 : c0871e08
[   33.172071] status: 00000120 badaddr: c03a48c0 cause: 80000005
[   33.180071] [<c03811fc>] eat+0x34/0x54
[   33.184071] [<c038137c>] do_reset+0x38/0x6c
[   33.192071] [<c0381514>] write_buffer+0x40/0x64
[   33.200071] [<c0381d20>] unpack_to_rootfs+0x1c4/0x300
[   33.212072] [<c03824ec>] do_populate_rootfs+0x70/0xd8
[   33.220072] [<c002bd58>] async_run_entry_fn+0x30/0xd0
[   33.232072] [<c002195c>] process_one_work+0x198/0x224
[   33.240072] [<c0021ee4>] worker_thread+0x238/0x294
[   33.248072] [<c002898c>] kthread+0xd4/0xd8
[   33.252072] [<c0002508>] ret_from_exception+0x0/0x1c

Implementing multi-thread support for semu, as discussed in PR #49, might improve the performance of the booting process.

Although implementing multi-threaded system emulation can significantly alleviate this issue, I think it does not mean the problem will no longer occur after multi-threaded system emulation is complete. Perhaps we can start by optimizing the timer.
Currently, the function semu_timer_get relies on clock_gettime, and semu_timer_get is primarily called during reads and writes to mtime. In other words, every time mtime is accessed, clock_gettime is invoked. This leads to an excessive number of timer interrupts, causing performance issues in the emulator.
In issue #49, it was suggested that lowering the frequency set in semu_timer_init could mitigate this problem. However, this would mean the Linux kernel perceives the clock as no longer being "real-time" but rather a "slowed down" or "imprecise" clock, which goes against the intent of that pull request.
To strike a balance between these two extremes, I think we can maintain a dedicated emulator timer and updating it at the start of each emulation cycle. This way, clock_gettime would only be called once per emulation cycle, while tick updates would be maintained separately. The logic for reading mtime would also be simplified.
I made some modifications to test this approach, and it did result in a slight performance improvement for the emulator. On my machine, with SMP=6, the RCU CPU Stall warning no longer appears. However, with SMP=8, the warning still occurs.

I completely agree that your implementation can help avoid RCU CPU stall warnings. However, as the number of simulated cores increases, the warning is likely to occur again unless the emulation period per cycle is also increased proportionally to the number of cores. Is my understanding correct?
I rebuilt the Linux kernel with CONFIG_PREEMPT=y to enable kernel preemption. This configuration suppresses the warning when simulating with SMP=32. The system boots successfully, as shown below:
[  351.875391] virtio_blk virtio1: [vda] 4800 512-byte logical blocks (2.46 MB/2.34 MiB)
[  356.664664] virtio_net virtio0: Assigned random MAC address 6e:8d:9a:9d:3f:86
[  360.199865] clk: Disabling unused clocks
[ 1133.368994] Freeing initrd memory: 8188K
[ 1135.085089] Freeing unused kernel image (initmem) memory: 180K
[ 1135.149092] Kernel memory protection not selected by kernel config.
[ 1135.201095] Run /init as init process
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: OK

Welcome to Buildroot
buildroot login:

But preemption may increase the frequency of context switches, causing overall CPU usage to increase.
For high-throughput tasks, overall performance may be reduced.

However, as the number of simulated cores increases, the warning is likely to occur again unless the emulation period per cycle is also increased proportionally to the number of cores. Is my understanding correct?

yes, the improvement is limited, as the number of cores grow up, the warning would occur again.

The accuracy of the timer primarily affects user programs. I think that during the boot process, it is unnecessary to use such a high-precision timer. Instead, a less precise timer, or even one that simply increments in a straightforward manner, could be used. After the boot process is complete, the system can switch back to a more precise timer.
Here is a simple example. I added a flag boot_complete to detect whether the boot process has finished. This can be determined by the first transition to U mode. Before the boot process is complete, I didn't use clock_gettime. Instead, I simply increment tv_nsec and combine it with the lower frequency mentioned in PR #49 to eliminate the CPU stall warning.
bool boot_complete = false;
static struct timespec host_time;

// ...

void semu_timer_init(semu_timer_t *timer, uint64_t freq)
{
    timer->freq = freq;
    clock_gettime(CLOCKID, &host_time);
    semu_timer_rebase(timer, 0);
}

static uint64_t semu_timer_clocksource(uint64_t freq)
{
#if defined(HAVE_POSIX_TIMER)
    if (boot_complete) {
        clock_gettime(CLOCKID, &host_time);
        return host_time.tv_sec * freq +
               mult_frac(host_time.tv_nsec, freq, 1e9);
    } else {
        return host_time.tv_sec * freq +
               mult_frac(host_time.tv_nsec++, freq / 1000, 1e9);
    }

// ...
#endif
}

// ...

This is the sample output:
mes@DESKTOP-HLQ9F6A:~/semu$ make check SMP=32
  CC    riscv.o
  CC    ram.o
  CC    utils.o
  CC    plic.o
  CC    uart.o
  CC    main.o
  CC    aclint.o
  CC    virtio-blk.o
  CC    virtio-net.o
  CC    netdev.o
  LD    semu
 DTC    minimal.dtb
Ready to launch Linux kernel. Please be patient.
failed to allocate TAP device: Operation not permitted
[    0.000000] Linux version 6.1.99 (jserv@node1) (riscv32-buildroot-linux-gnu-gcc.br_real (Buildroot 2024.02.4) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Thu Jul 18 13:04:10 CST 2024
[    0.000000] Machine model: semu
[    0.000000] earlycon: ns16550 at MMIO 0xf4000000 (options '')
[    0.000000] printk: bootconsole [ns16550] enabled
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] SBI specification v2.0 detected
[    0.000000] SBI implementation ID=0x999 Version=0x1
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI HSM extension detected
[    0.000000] riscv: base ISA extensions aim
[    0.000000] riscv: ELF capabilities aim
[    0.000000] percpu: Embedded 10 pages/cpu s11604 r8192 d21164 u40960
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 130048
[    0.000000] Kernel command line: earlycon console=ttyS0
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 126976 bytes
[    0.000000] printk: log_buf_len min size: 65536 bytes
[    0.000000] printk: log_buf_len: 262144 bytes
[    0.000000] printk: early log buf free: 63952(97%)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 504100K/524288K available (3578K kernel code, 345K rwdata, 873K rodata, 185K init, 140K bss, 20188K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=32, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 32 local interrupts mapped
[    0.000000] plic: interrupt-controller@0: mapped 31 interrupts with 32 handlers for 32 contexts.
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] riscv-timer: riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0xefdb196da, max_idle_ns:<PHONE_NUMBER>67 ns
[    0.000000] sched_clock: 64 bits at 65MHz, resolution 15ns, wraps every<PHONE_NUMBER>550ns
[    0.000007] Console: colour dummy device 80x25
[    0.000008] Calibrating delay loop (skipped), value calculated using timer frequency.. 130.00 BogoMIPS (lpj=260000)
[    0.000009] pid_max: default: 32768 minimum: 301
[    0.000014] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.000015] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.000038] ASID allocator disabled (0 bits)
[    0.000040] rcu: Hierarchical SRCU implementation.
[    0.000040] rcu:     Max phase no-delay instances is 1000.
[    0.000057] smp: Bringing up secondary CPUs ...
[    0.000295] smp: Brought up 1 node, 32 CPUs
[    0.000320] devtmpfs: initialized
[    0.000381] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns:<PHONE_NUMBER>100000 ns
[    0.000382] futex hash table entries: 8192 (order: 7, 524288 bytes, linear)
[    0.000406] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.000486] platform soc@F0000000: Fixed dependency cycle(s) with /soc@F0000000/interrupt-controller@0
[    0.000662] clocksource: Switched to clocksource riscv_clocksource
[    0.000806] NET: Registered PF_INET protocol family
[    0.000811] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000858] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.000859] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000860] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.000862] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.000865] TCP: Hash tables configured (established 4096 bind 4096)
[    0.000868] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.000869] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.000874] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.000883] Unpacking initramfs...
[    0.005874] Freeing initrd memory: 8188K
[    0.005907] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    0.006895] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.006917] printk: console [ttyS0] disabled
[    0.006918] f4000000.serial: ttyS0 at MMIO 0xf4000000 (irq = 1, base_baud = 312500) is a 16550
[    0.006919] printk: console [ttyS0] enabled
[    0.006919] printk: console [ttyS0] enabled
[    0.006920] printk: bootconsole [ns16550] disabled
[    0.006920] printk: bootconsole [ns16550] disabled
[    0.006952] virtio_blk virtio1: 1/0/0 default/read/poll queues
[    0.006977] virtio_blk virtio1: [vda] 4800 512-byte logical blocks (2.46 MB/2.34 MiB)
[    0.006993] virtio_net virtio0: Assigned random MAC address a6:8b:4b:ad:09:18
[    0.007019] clk: Disabling unused clocks
[    0.007029] Freeing unused kernel image (initmem) memory: 180K
[    0.007030] Kernel memory protection not selected by kernel config.
[    0.007031] Run /init as init process

However, when it switch back to the high-precision timer, the RCU stall warning will occur again, but I think after the multi-threaded emulation is implemented, the situation would be significantly alleviated.
[  157.431102] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  157.451102] rcu:     1-....: (4 ticks this GP) idle=eb6c/0/0x1 softirq=1/1 fqs=0
[  157.481102] rcu:     2-....: (4 ticks this GP) idle=d3b4/0/0x1 softirq=1/2 fqs=0
[  157.501102] rcu:     6-....: (4 ticks this GP) idle=6cfc/0/0x1 softirq=1/2 fqs=0
[  157.531102] rcu:     7-....: (4 ticks this GP) idle=532c/0/0x1 softirq=1/2 fqs=0
[  157.551102] rcu:     9-....: (4 ticks this GP) idle=1bf4/1/0x40000002 softirq=1/2 fqs=1
[  157.581103] rcu:     10-....: (4 ticks this GP) idle=0144/0/0x1 softirq=1/2 fqs=1
[  157.601103] rcu:     13-....: (4 ticks this GP) idle=af04/0/0x1 softirq=1/2 fqs=1
[  157.631103] rcu:     14-....: (4 ticks this GP) idle=91a4/0/0x1 softirq=1/2 fqs=1
[  157.661103] rcu:     15-....: (4 ticks this GP) idle=8cac/1/0x40000002 softirq=1/2 fqs=1
[  157.691103]  (detected by 3, t=39421 jiffies, g=-1199, q=390 ncpus=32)
[  157.711103] Task dump for CPU 0:
[  157.721103] task:kworker/0:1     state:R stack:0     pid:175   ppid:2      flags:0x00000008
[  157.751104] Workqueue: events deferred_probe_timeout_work_func
[  157.771104] Call Trace:
[  157.781104] [<c037aac0>] __schedule+0x218/0x3a8
[  157.801104] [<c037acac>] schedule+0x5c/0xa4
[  157.821104] [<c037f85c>] __do_softirq+0x1c/0x2c
[  157.851104] Task dump for CPU 1:
[  157.861104] task:swapper/1       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  157.891104] Call Trace:
[  157.901104] [<c037aac0>] __schedule+0x218/0x3a8
[  157.911104] hrtimer: interrupt took 50000277 ns
[  157.921104] [<c037f12c>] default_idle_call+0x48/0x4c
[  157.931105] [<c003d4c0>] do_idle+0xd4/0xe8
[  157.961105] [<c003d640>] cpu_startup_entry+0x30/0x34
[  157.981105] [<c0004994>] riscv_set_ipi_ops+0x0/0x20
[  158.001105] Task dump for CPU 2:
[  158.011105] task:swapper/2       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  158.051105] Call Trace:
[  158.061105] [<c037aac0>] __schedule+0x218/0x3a8
[  158.081105] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.1.99 #1
[  158.101105] Hardware name: semu (DT)
[  158.111106] epc : do_idle+0xa8/0xe8
[  158.131106]  ra : do_idle+0xd4/0xe8
[  158.151106] epc : c003d494 ra : c003d4c0 sp : c08a7f20
[  158.171106]  gp : c04de840 tp : c087de80 t0 : c0929cd0
[  158.181106]  t1 : 00000004 t2 : 00000000 s0 : c08a7f40
[  158.201106]  s1 : c04df13c a0 : 0062ba49 a1 : 00000003
[  158.221106]  a2 : 00000001 a3 : 0000000c a4 : c03ac0a4
[  158.231106]  a5 : 00000002 a6 : 00000008 a7 : ffffffff
[  158.251106]  s2 : 00000003 s3 : c04df278 s4 : 00000000
[  158.271106]  s5 : 00000000 s6 : 00000000 s7 : 00000000
[  158.291107]  s8 : 00000000 s9 : 00000000 s10: 00000000
[  158.301107]  s11: 00000000 t3 : 00000004 t4 : 00000000
[  158.321107]  t5 : 00000000 t6 : c0929d60
[  158.331107] status: 00000120 badaddr: c003d4c0 cause: 80000005
[  158.351107] [<c003d494>] do_idle+0xa8/0xe8
[  158.371107] [<c003d640>] cpu_startup_entry+0x30/0x34
[  158.401107] [<c0004994>] riscv_set_ipi_ops+0x0/0x20
[  158.421107] Task dump for CPU 4:
[  158.431107] task:swapper/4       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  158.461108] Call Trace:
[  158.471108] [<c037aac0>] __schedule+0x218/0x3a8
[  158.491108] [<c037f11c>] default_idle_call+0x38/0x4c
[  158.511108] Task dump for CPU 5:
[  158.521108] task:swapper/5       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  158.561108] Call Trace:
[  158.571108] [<c037aac0>] __schedule+0x218/0x3a8
[  158.591108] Task dump for CPU 6:
[  158.601108] task:swapper/6       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  158.631108] Call Trace:
[  158.641109] [<c037aac0>] __schedule+0x218/0x3a8
[  158.661109] Task dump for CPU 7:
[  158.671109] task:swapper/7       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  158.701109] Call Trace:
[  158.711109] [<c037aac0>] __schedule+0x218/0x3a8
[  158.731109] [<c037f12c>] default_idle_call+0x48/0x4c
[  158.751109] [<c003d4c0>] do_idle+0xd4/0xe8
[  158.771109] [<c003d640>] cpu_startup_entry+0x30/0x34
[  158.801109] [<c0004994>] riscv_set_ipi_ops+0x0/0x20
[  158.821110] Task dump for CPU 8:
[  158.831110] task:swapper/8       state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  158.861110] Call Trace:
[  158.871110] [<c037aac0>] __schedule+0x218/0x3a8
[  158.891110] Task dump for CPU 9:
[  158.901110] task:init            state:R  running task     stack:0     pid:1     ppid:0      flags:0x00000008
[  158.941110] Call Trace:
[  158.951110] [<c037aac0>] __schedule+0x218/0x3a8
[  158.971110] Task dump for CPU 10:
[  158.981110] task:ksoftirqd/10    state:R stack:0     pid:61    ppid:2      flags:0x00000000
[  159.011111] Call Trace:
[  159.021111] [<c037aac0>] __schedule+0x218/0x3a8
[  159.031111] [<c037acac>] schedule+0x5c/0xa4
[  159.051111] [<c002c4e0>] smpboot_thread_fn+0x118/0x158
[  159.071111] [<c00289c0>] kthread+0xd4/0xd8
[  159.091111] [<c0002514>] ret_from_exception+0x0/0x1c
[  159.111111] Task dump for CPU 11:
[  159.121111] task:swapper/11      state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  159.161111] Call Trace:
[  159.171112] [<c037aac0>] __schedule+0x218/0x3a8
[  159.191112] Task dump for CPU 12:
[  159.201112] task:swapper/12      state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  159.231112] Call Trace:
[  159.241112] [<c037aac0>] __schedule+0x218/0x3a8
[  159.261112] [<c037f12c>] default_idle_call+0x48/0x4c
[  159.281112] Task dump for CPU 13:
[  159.291112] task:swapper/13      state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  159.321112] Call Trace:
[  159.331112] [<c037aac0>] __schedule+0x218/0x3a8
[  159.351113] [<c037f12c>] default_idle_call+0x48/0x4c
[  159.371113] [<c003d4c0>] do_idle+0xd4/0xe8
[  159.391113] [<c003d640>] cpu_startup_entry+0x30/0x34
[  159.421113] [<c0004994>] riscv_set_ipi_ops+0x0/0x20
[  159.441113] Task dump for CPU 14:
[  159.451113] task:swapper/14      state:R  running task     stack:0     pid:0     ppid:1      flags:0x00000000
[  159.481113] Call Trace:
[  159.491113] [<c037aac0>] __schedule+0x218/0x3a8
[  159.511113] [<c037af78>] schedule_idle+0x24/0x48
[  159.531114] [<c003d448>] do_idle+0x5c/0xe8
[  159.551114] [<c003d640>] cpu_startup_entry+0x30/0x34
[  159.581114] [<c0004994>] riscv_set_ipi_ops+0x0/0x20
[  159.601114] Task dump for CPU 15:
[  159.611114] task:ksoftirqd/15    state:R stack:0     pid:86    ppid:2      flags:0x00000000
[  159.641114] Call Trace:
[  159.641114] [<c037aac0>] __schedule+0x218/0x3a8
[  159.661114] [<c037acac>] schedule+0x5c/0xa4
[  159.681114] [<c002c4e0>] smpboot_thread_fn+0x118/0x158
[  159.701115] [<c00289c0>] kthread+0xd4/0xd8
[  159.721115] [<c0002514>] ret_from_exception+0x0/0x1c
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: OK

Welcome to Buildroot
buildroot login: root
# ls
# cat /etc/pro
profile     profile.d/  protocols
# cat /proc/cpuinfo
processor       : 0
hart            : 0
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 1
hart            : 1
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 2
hart            : 2
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 3
hart            : 3
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 4
hart            : 4
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 5
hart            : 5
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 6
hart            : 6
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 7
hart            : 7
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 8
hart            : 8
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 9
hart            : 9
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 10
hart            : 10
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 11
hart            : 11
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 12
hart            : 12
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 13
hart            : 13
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 14
hart            : 14
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 15
hart            : 15
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 16
hart            : 16
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 17
hart            : 17
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 18
hart            : 18
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 19
hart            : 19
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 20
hart            : 20
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 21
hart            : 21
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 22
hart            : 22
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 23
hart            : 23
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 24
hart            : 24
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 25
hart            : 25
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 26
hart            : 26
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 27
hart            : 27
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 28
hart            : 28
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 29
hart            : 29
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 30
hart            : 30
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

processor       : 31
hart            : 31
isa             : rv32ima
mmu             : sv32
mvendorid       : 0x12345678
marchid         : 0x80000001
mimpid          : 0x1

I agree that your changes can quickly and easily resolve the RCU stall warning issue! However, your log cannot represent the actual boot time in this situation ([   17.029125]). It should  take much more time, am I correct?
I try to reproduce your work, the RCU stall warning is resolved when simulate SMP=32
[    0.007006] Run /init as init process
[    0.026121] hrtimer: interrupt took 12000723 ns
Starting syslogd: OK

another message hrtimer: interrupt took XXXX ns is shown (also in your log)
A quick research that this warning is produced by hrtimer_interrupt() (high resolution timer) because our kernel has config CONFIG_HIGH_RES_TIMERS=y
I'm not sure is any side effect here

However, your log cannot represent the actual boot time in this situation ([   17.029125]). It should take much more time, am I correct?

Yes, the primary reason for incrementing tv_nsec is to allow the boot process time to progress, ensuring the boot process can proceed smoothly. Therefore, during the boot process, the timer does not serve as an actual timing mechanism.
Once the system switches to U mode for the first time, the boot_complete flag is set, and the time reference is reset. As a result, the [17.029125] here represents 17.029125 seconds since the first switch to U mode.
As for the HRT warning, I think it was triggered due to a sudden jump in the system clock right after the boot process completed. I'm not sure is any side effect here too.

A quick research that this warning is produced by hrtimer_interrupt() (high resolution timer) because our kernel has config CONFIG_HIGH_RES_TIMERS=y

This option was set for the sake of benchmarking purpose. RT-Tests relies on HRT features.

However, your log cannot represent the actual boot time in this situation ([ 17.029125]). It should take much more time, am I correct?

The timer increments should align with the frequency defined in the device tree. We could use an approach similar to BogoMips to make the necessary adjustments.

After multiple attempts, I realize that independently maintaining nsec is not a good idea. This results in time intervals between reads being uniform, whereas in reality, we call clock_gettime at varying time intervals.
In contrast, I believe maintaining a frequency scaling factor is a better solution. As mentioned in #49, this achieves the purpose of slowing down time during the boot process. Since it’s merely a scaling factor, we can still derive real-time values from it.
As for the hrtimer: interrupt took XXXX ns issue, after a few days of testing, I found that this doesn’t seem to be caused by time drift. In fact, during these tests, I re-based the timer after the boot process using the recorded time to ensure no drift, but the hrtimer message still appeared. The timing of the message is not consistent—it sometimes appears after the "Welcome to Buildroot" message, and sometimes right after switching to U-mode. But always after switch to U-mode.
Interestingly, on my machine, the warning doesn't appear at all with SMP=16, regardless of how the frequency is adjusted. Moreover, the current implementation only affects the timer during the boot process; after switching to U-mode, the timer behaves exactly as before. Therefore, I believe this warning is due to the current sequential emulation approach. After the multi-threaded emulation is implemented, I think the situation would be mitigated a lot.
Here’s a diagram of the overall flow:

Below is a simple example:
static uint64_t semu_timer_clocksource(uint64_t freq)
{
#if defined(HAVE_POSIX_TIMER)
    struct timespec t;
    clock_gettime(CLOCKID, &t);

    if (boot_complete)
        return t.tv_sec * freq + mult_frac(t.tv_nsec, freq, 1e9);
    else
        return t.tv_sec * (freq / 100) + mult_frac(t.tv_nsec, (freq / 100), 1e9);
#elif defined(HAVE_MACH_TIMER)
    static mach_timebase_info_data_t t;
    if (t.denom == 0)
        (void) mach_timebase_info(&t);
    return mult_frac(mult_frac(mach_absolute_time(), t.numer, t.denom), freq,
                     1e9);
#else
    return time(0) * freq;
#endif
}

uint64_t semu_timer_get(semu_timer_t *timer)
{
    static bool first = true;
    if (first && boot_complete) {
        first = false;
        semu_timer_rebase(timer, 0);
        printf("\033[1;33m[SEMU LOG]: Switch to real time\033[0m\n");
    }

    return semu_timer_clocksource(timer->freq) - timer->begin;
}

I think this approach is better than maintaining two separate timers during the boot process. Dividing the frequency by 100 means the boot process operates at one one-hundredth of real-time, allowing us to easily derive the actual boot time. This scaling factor can also be configured in the Makefile. I attempted to dynamically measure the cost of clock_gettime to set this value, but realized that using clock_gettime as a benchmark was not accurate. Therefore, in this example, a scale of 100 was used for simplicity.
Here is the output of the test:
mes@DESKTOP-HLQ9F6A:~/MesRv32emu/semu$ make check SMP=32
 DTC    minimal.dtb
Ready to launch Linux kernel. Please be patient.
failed to allocate TAP device: Operation not permitted
[    0.000000] Linux version 6.1.99 (jserv@node1) (riscv32-buildroot-linux-gnu-gcc.br_real (Buildroot 2024.02.4) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Thu Jul 18 13:04:10 CST 2024
[    0.000000] Machine model: semu
[    0.000000] earlycon: ns16550 at MMIO 0xf4000000 (options '')
[    0.000000] printk: bootconsole [ns16550] enabled
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] SBI specification v2.0 detected
[    0.000000] SBI implementation ID=0x999 Version=0x1
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI HSM extension detected
[    0.000000] riscv: base ISA extensions aim
[    0.000000] riscv: ELF capabilities aim
[    0.000000] percpu: Embedded 10 pages/cpu s11604 r8192 d21164 u40960
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 130048
[    0.000000] Kernel command line: earlycon console=ttyS0
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 126976 bytes
[    0.000000] printk: log_buf_len min size: 65536 bytes
[    0.000000] printk: log_buf_len: 262144 bytes
[    0.000000] printk: early log buf free: 63952(97%)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 504100K/524288K available (3578K kernel code, 345K rwdata, 873K rodata, 185K init, 140K bss, 20188K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=32, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 32 local interrupts mapped
[    0.000000] plic: interrupt-controller@0: mapped 31 interrupts with 32 handlers for 32 contexts.
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] riscv-timer: riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0xefdb196da, max_idle_ns:<PHONE_NUMBER>67 ns
[    0.000000] sched_clock: 64 bits at 65MHz, resolution 15ns, wraps every<PHONE_NUMBER>550ns
[    0.000694] Console: colour dummy device 80x25
[    0.000893] Calibrating delay loop (skipped), value calculated using timer frequency.. 130.00 BogoMIPS (lpj=260000)
[    0.000992] pid_max: default: 32768 minimum: 301
[    0.001489] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.001588] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.004070] ASID allocator disabled (0 bits)
[    0.004269] rcu: Hierarchical SRCU implementation.
[    0.004368] rcu:     Max phase no-delay instances is 1000.
[    0.006056] smp: Bringing up secondary CPUs ...
[    0.049211] smp: Brought up 1 node, 32 CPUs
[    0.055530] devtmpfs: initialized
[    0.071056] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns:<PHONE_NUMBER>100000 ns
[    0.071360] futex hash table entries: 8192 (order: 7, 524288 bytes, linear)
[    0.077127] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.097399] platform soc@F0000000: Fixed dependency cycle(s) with /soc@F0000000/interrupt-controller@0
[    0.141221] clocksource: Switched to clocksource riscv_clocksource
[    0.176651] NET: Registered PF_INET protocol family
[    0.177847] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.189801] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.190000] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.190199] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.190697] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.191394] TCP: Hash tables configured (established 4096 bind 4096)
[    0.192190] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.192390] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.193684] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.196073] Unpacking initramfs...
[    0.217861] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    0.463628] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.468923] printk: console [ttyS0] disabled
[    0.469123] f4000000.serial: ttyS0 at MMIO 0xf4000000 (irq = 1, base_baud = 312500) is a 16550
[    0.469422] printk: console [ttyS0] enabled
[    0.469422] printk: console [ttyS0] enabled
[    0.469622] printk: bootconsole [ns16550] disabled
[    0.469622] printk: bootconsole [ns16550] disabled
[    0.477610] virtio_blk virtio1: 1/0/0 default/read/poll queues
[    0.483698] virtio_blk virtio1: [vda] 4800 512-byte logical blocks (2.46 MB/2.34 MiB)
[    0.487487] virtio_net virtio0: Assigned random MAC address 1e:07:10:f3:00:37
[    0.494367] clk: Disabling unused clocks
[    1.486134] Freeing initrd memory: 8188K
[    1.488423] Freeing unused kernel image (initmem) memory: 180K
[    1.488622] Kernel memory protection not selected by kernel config.
[    1.488821] Run /init as init process
[SEMU LOG]: Switch to real time
[    7.946974] hrtimer: interrupt took 69481784 ns
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: OK

Welcome to Buildroot
buildroot login:

Another output for the factor set to 50:
mes@DESKTOP-HLQ9F6A:~/MesRv32emu/semu$ make check SMP=32
  CC    riscv.o
  CC    ram.o
  CC    utils.o
  CC    plic.o
  CC    uart.o
  CC    main.o
  CC    aclint.o
  CC    virtio-blk.o
  CC    virtio-net.o
  CC    netdev.o
  LD    semu
 DTC    minimal.dtb
Ready to launch Linux kernel. Please be patient.
failed to allocate TAP device: Operation not permitted
[    0.000000] Linux version 6.1.99 (jserv@node1) (riscv32-buildroot-linux-gnu-gcc.br_real (Buildroot 2024.02.4) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Thu Jul 18 13:04:10 CST 2024
[    0.000000] Machine model: semu
[    0.000000] earlycon: ns16550 at MMIO 0xf4000000 (options '')
[    0.000000] printk: bootconsole [ns16550] enabled
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] SBI specification v2.0 detected
[    0.000000] SBI implementation ID=0x999 Version=0x1
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI HSM extension detected
[    0.000000] riscv: base ISA extensions aim
[    0.000000] riscv: ELF capabilities aim
[    0.000000] percpu: Embedded 10 pages/cpu s11604 r8192 d21164 u40960
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 130048
[    0.000000] Kernel command line: earlycon console=ttyS0
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 126976 bytes
[    0.000000] printk: log_buf_len min size: 65536 bytes
[    0.000000] printk: log_buf_len: 262144 bytes
[    0.000000] printk: early log buf free: 63952(97%)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 504100K/524288K available (3578K kernel code, 345K rwdata, 873K rodata, 185K init, 140K bss, 20188K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=32, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 32 local interrupts mapped
[    0.000000] plic: interrupt-controller@0: mapped 31 interrupts with 32 handlers for 32 contexts.
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] riscv-timer: riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0xefdb196da, max_idle_ns:<PHONE_NUMBER>67 ns
[    0.000000] sched_clock: 64 bits at 65MHz, resolution 15ns, wraps every<PHONE_NUMBER>550ns
[    0.001390] Console: colour dummy device 80x25
[    0.001589] Calibrating delay loop (skipped), value calculated using timer frequency.. 130.00 BogoMIPS (lpj=260000)
[    0.001788] pid_max: default: 32768 minimum: 301
[    0.002781] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.002980] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.007549] ASID allocator disabled (0 bits)
[    0.007947] rcu: Hierarchical SRCU implementation.
[    0.007947] rcu:     Max phase no-delay instances is 1000.
[    0.011523] smp: Bringing up secondary CPUs ...
[    0.095725] smp: Brought up 1 node, 32 CPUs
[    0.108034] devtmpfs: initialized
[    0.138606] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns:<PHONE_NUMBER>100000 ns
[    0.139202] futex hash table entries: 8192 (order: 7, 524288 bytes, linear)
[    0.150715] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.190212] platform soc@F0000000: Fixed dependency cycle(s) with /soc@F0000000/interrupt-controller@0
[    0.279523] clocksource: Switched to clocksource riscv_clocksource
[    0.350564] NET: Registered PF_INET protocol family
[    0.352978] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.375270] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.375872] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.376273] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.377275] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.378478] TCP: Hash tables configured (established 4096 bind 4096)
[    0.380083] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.380484] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.382890] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.387903] Unpacking initramfs...
[    0.415743] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    0.910292] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.921461] printk: console [ttyS0] disabled
[    0.921867] f4000000.serial: ttyS0 at MMIO 0xf4000000 (irq = 1, base_baud = 312500) is a 16550
[    0.922476] printk: console [ttyS0] enabled
[    0.922476] printk: console [ttyS0] enabled
[    0.922881] printk: bootconsole [ns16550] disabled
[    0.922881] printk: bootconsole [ns16550] disabled
[    0.939299] virtio_blk virtio1: 1/0/0 default/read/poll queues
[    0.951841] virtio_blk virtio1: [vda] 4800 512-byte logical blocks (2.46 MB/2.34 MiB)
[    0.959919] virtio_net virtio0: Assigned random MAC address 8e:5e:2e:78:c2:e5
[    0.974247] clk: Disabling unused clocks
[    2.984914] Freeing initrd memory: 8188K
[    2.989718] Freeing unused kernel image (initmem) memory: 180K
[    2.989918] Kernel memory protection not selected by kernel config.
[    2.990318] Run /init as init process
[SEMU LOG]: Switch to real time
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting network: OK

Welcome to Buildroot
buildroot login: [   88.934001] hrtimer: interrupt took 60891661 ns

In my environment, even with varying scale factors, the hrtimer warning consistently appeared at approximately 60000000 ns. I think this observation supports my hypothesis.


Can not build SDK2.0 from source on MacOS Mojave

Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

Consider checking out SDK examples.
Have you looked in our documentations?
Is you question a frequently asked one?
Try searching our GitHub Issues (open and closed) for a similar issue.

All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info

Camera Model
{ R200 / F200 / SR300 / ZR300 / D400 }

Firmware Version
(Open RealSense Viewer --> Click info)

Operating System & Version
MacOS

Kernel Version (Linux Only)
(e.g. 4.14.13)

Platform
PC/Raspberry Pi/ NVIDIA Jetson / etc..

SDK Version
2.16.5

Language
{C/C#/labview/nodejs/opencv/pcl/python/unity }

Segment
{Robot/Smartphone/VR/AR/others }

Use Xcode 10 to build, however there appears error:
ld: library not found for -lusb-1.0
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<Describe your issue / question / feature request / etc..>

Hi @darrenjiayinxie , AFAIK there is not a complete version that supports MacOS yet.
From Intel's documentation updated May 2018, it says:

How did you compile your code with? What camera are you using?

I follow the tutorial
https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_osx.md
I am using D415
Eugene<EMAIL_ADDRESS>于2018年11月22日周四 下午7:13写道：

Hi @darrenjiayinxie https://github.com/darrenjiayinxie , AFAIK there is
not a complete version that supports MacOS yet.
From Intel's documentation updated May 2018
https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-SDK2-Github-Guide.pdf,
it says:
[image: image]
https://user-images.githubusercontent.com/21957042/48924833-7e69dd00-eef7-11e8-8cdd-7c068ba6f820.png
How did you compile your code with? What camera are you using?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/IntelRealSense/librealsense/issues/2764#issuecomment-441136319,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APaST2C67o8UjDKrnqOvUxx_hVNC2Dniks5uxz2egaJpZM4Yv9qz
.

We can use pre-build version until the team solve the issue.
(we can install the latest version)
$ brew install librealsense

We can use viewer etc, but unfortunately not fully functional.

Thanks! Now I switch to the ubuntu 16.04 and it works well. Hope the Mac
version can work soon.
On Mon, Nov 26, 2018 at 10:48 PM Koji<EMAIL_ADDRESS>wrote:

We can use pre-build version until the team solve the issue.
(we can install the latest version)
$ brew install librealsense
We can use viewer etc, but unfortunately not fully functional.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/IntelRealSense/librealsense/issues/2764#issuecomment-441892850,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APaST33vBSkvTQbpFVwac53LnGuTwbhEks5uzLYjgaJpZM4Yv9qz
.

@dorodnic Hi, I'm running into the same ld: library not found for -lusb-1.0 bug when trying to run on XCode, and I attempted to fall-back to 2.15.0 based on your previous comment, but am still seeing the same issue. What were your precise steps of falling back? Did you just clone the 2.15.0 release branch and then follow the steps outlined here: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_osx.md?

bump for @thomasliu623 question

I am encountering this same error: ld: library not found for -lusb-1.0 when trying to compile with Xcode.
Taking @sleepy-maker advice and Installing with brew allowed me to open the examples, but I get black screens with no indication the D415 is recognized or active.

I have this same error when trying to build with Xcode: ld: library not found for -lusb-1.0
@sleepy-maker suggestion works for testing the camera.   Starting by running /usr/local/bin/rs-sensor-control, I can see the device and then start to use the other examples. I am trying to run in Unity, and found a .dll but could really use a fix!

I have the same issue building from source on macOS 10.14.2
ld: library not found for -lusb-1.0
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Has anyone had any luck with this?  I can verify that the library is here:
/usr/local/Cellar/libusb/1.0.22/lib/libusb-1.0.dylib

The library path has /usr/local/Cellar/libusb/1.0.22/lib/
I can run rs-sensor-control just fine.  Has anyone solved this?

@sharner when you use make you may have to specify the Library where libusb is stored.
make LIBRARY_PATH=/usr/local/lib

I had trouble building from source on mac and here is what I did to get it to work:
https://github.com/GitwellAnyohub/Real_Sensible_Moseq/blob/master/bash_and_install/install_realsense_sdk_mac_os.sh

@GitwellAnyohub thanks, your script worked for me.  I was never able to get Xcode working, as in the instructions.

Glad it worked!

[Realsense Customer Engineering Team Comment]
Submit issue as solved.


fish shell documentation is installed to share/doc/index.html, creating conflicts

Describe the bug
The fish derivation installs its HTML documentation directly into /share/doc, cluttering the shared namespace. This can cause file conflicts with packages that have the same bug.
To Reproduce
Steps to reproduce the behavior:

Install fish
/nix/store/m1bid6zqjjd8kszsn4ya2h1693i6z4mz-fish-3.1.2/share/doc/index.html exists (or similar).

Expected behavior
The documentation should be installed in a fish subdirectory.
Notify maintainers

@ocharles @cole-h
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

system: "x86_64-linux"
host os: Linux 5.6.14-arch1-1, Arch Linux, noversion
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.2.2
channels(root): "nixpkgs-20.03pre203598.f97746ba272"
channels(jonas): "home-manager, nixpkgs-20.09pre229671.9d0c3ffe678"
nixpkgs: /home/jonas/.nix-defexpr/channels/nixpkgs

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
- fish
# a list of nixos modules affected by the problem
module:
- programs.fish

Should be fixed when https://github.com/NixOS/nixpkgs/pull/90513 gets merged. Thanks for bringing this to my attention!

It's been merged into master! If you're using unstable, a channel with the commit should be available within a day (barring any blockers appearing). If you're on 20.03, you'll want to get fish from unstable once it's available. There are various guides around the internet you can use for reference (but since I run my system from unstable, I don't have any experience with doing this).
Thanks once again for the report!

Great, thanks for the quick fix!


Use fingerprints for empty source handling

Instead of actually walking the file system.
Fixes #9980

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.


quantum_info "on this page" does not show methods properly

URL to the relevant documentation
If we look at the "on this page" sidebar for quantum_info page, I see References and Examples, instead of the actual methods like hellinger_fidelity
Select all that apply

[ ] typo
[ ] code bug
[ ] out-of-date content
[ ] broken link
[x] other

Describe the fix.

Go to https://docs.quantum-computing.ibm.com/api/qiskit/quantum_info#references-1
Try finding hellinger_fidelity
Look at the right side "on this page"

I think this is likely the same problem described by this. I was just looking at QuantumCircuit docs https://docs.quantum.ibm.com/api/qiskit/qiskit.circuit.QuantumCircuit as I wanted to provide a link in an answer to compose. So I was looking in the right-hand sidebar for compose method and got confused when starting to look down the list under Methods that I come across an Examples heading at the same level.

I realized after a bit that what was under it was still the methods though. Looking down the list there are other headings that seem to break up the list like Notes and Example These seem to be inline examples and notes for the methods, where the presentation in the sidebar makes it appear not the case.
I searched to see if the issue I was seeing was already noted. I think this is probably the same thing or at least related. The issue here describes the method being looked for as not found. In my case the method, compose, I was looking for was there it was just these "extra" headings that added confusion.

This has now been fixed by @frankharkins in https://github.com/Qiskit/documentation/pull/683. The change is deployed for current docs and @arnaucasau will regenerate all the historical API docs with the change. Thank you, Frank, and thank you both for the feedback!
quantum_info:

QuantumCircuit:


SSL validation issues

I'm running into issues connecting to my Confluence server because it has a cert that is not in the normal cert chain.  I don't want to turn all SSL validation off (ssl._create_default_https_context = ssl._create_unverified_context).
Would you accept a PR that would allow a context to be passed into xmlrpclib.ServerProxy?
From the docs:
context may be ssl.SSLContext and configures the SSL settings of the underlying HTTPS connection.

I went ahead and did this work.  If it's of interest, it can found in #25.

Thanks, @devkmsg ! I've merged your contribution, closing this issue.


Improve efficiency of sparse symmetric ERI

Problem:
This blog post  describes how to implement sparse matrix multiplication in Jax as:
@jax.partial(jax.jit, static_argnums=(2))
def sp_matmul(A, B, shape):
    """
    Arguments:
        A: (N, M) sparse matrix represented as a tuple (indexes, values)
        B: (M,K) dense matrix
        shape: value of N
    Returns:
        (N, K) dense matrix
    """
    assert B.ndim == 2
    indexes, values = A
    rows, cols = indexes
    in_ = B.take(cols, axis=0)
    prod = in_*values[:, None]
    res = jax.ops.segment_sum(prod, rows, shape)
    return res

This has been done for ERI in sparse_symmetric_ERI.py in sequentialized_iter(...)
Alex generated a profile for it, and thinks it is not as efficient as it should

Solution:
Identify the source of inefficiency in sequentialized_iter(...) and find a work around.

FYI Alex identified that the problem is actually in eigh not in sequentialized_iter(...)


Update project name and instructions

@Essmaw please wait.

@Essmaw this PR is ready for review and merge.
Do not forget to run git pull in your local repo after merge.


[Feature]: stationary groups

Describe the feature
I love the concept that adibags did and that you're continuing doing, however, the group constantly moving locations during selling or equiping/using makes it annoying to rehunt down the item. Is there anyway to set up the window to be a certain size and then have the groups stay in the locations while bag is open, or even constant would be nice. appreciate your time :).
Describe the implementation
just an option for static group location in the bags.

Right now, groups and items do not shift if you sell them, but shift if you buy or do a 1:1 swap. There's no real way to keep sections and windows static physically, because as items are put in the bag, they have to go somewhere -- this means I need to make room to draw them.
You should not be seeing items shift when selling items. Can you confirm this behavior?

you are correct about the items not shifting during selling, i apologize, and the items needing a place to go makes sense, is there a way to prioritize in which order the sections appear than? as well as a way to remove the "new items" section?
ty again for your time :)

That's actually coming soon! It's already planned work. Stay tuned for a future release :)


Need the API document

The list of api could be found here http://<IP_ADDRESS>:5000/api/v0.1

The document from someone's blog https://dmsimard.com/2014/01/01/documentation-for-ceph-rest-api/


[NNPA] Fix to run numerical tests for NNPA

I forgot this line in https://github.com/onnx/onnx-mlir/pull/2283 to enable numerical tests for NNPA in check-onnx-backend-numerical-nnpa

Jenkins Linux s390x Build #11525 [push] Fix to enable numerical ... started at 12:54

Jenkins Linux amd64 Build #11517 [push] Fix to enable numerical ... started at 11:55

Jenkins Linux amd64 Build #11517 [push] Fix to enable numerical ... passed after 1 hr 14 min

Jenkins Linux s390x Build #11525 [push] Fix to enable numerical ... failed after 1 hr 43 min

Jenkins Linux ppc64le Build #10575 [push] Fix to enable numerical ... passed after 2 hr 1 min


Sending more than 20 bytes for characteristic

Is it possible to write more than 20 bytes for a characteristic with the library? Using the writeDevice method it seems like it is not. What would be the best way of achieving this?

not support more than 20 bytes

split into multiple packets, send data multiple times


cryptsetup: update project homepage

The project's homepage was moved from http://code.google.com/p/cryptsetup/ to https://gitlab.com/cryptsetup/cryptsetup/.

Pushed to master (77d9966d931a0b74616f7d782060bdf33713bcda). Thanks!


License?

Thank you for such a nice library? May you clarify, please, under which License is it distributed?

Hi @SemyonSinchenko , thanks for the note!
I have added the licence to the repo

Thank you!


Plot the displacement field of B-Spline transformation

There was already a similar issue (#932) but it regarded the Jacobian. I like to plot something similar, but directly the displacement field of the transformation.
I wrote this function here to visualize the field:
def get_deformation_field(image, transform):
    displ = sitk.TransformToDisplacementField(transform,
                                  sitk.sitkVectorFloat64,
                                  image.GetSize(),
                                  image.GetOrigin(),
                                  image.GetSpacing(),
                                  image.GetDirection())
    det = sitk.GetArrayFromImage(sitk.DisplacementFieldJacobianDeterminant(displ))
    vectors = sitk.GetArrayFromImage(displ)

    x, y = det.shape
    X, Y = np.meshgrid(np.arange(0, x, 1), np.arange(0, y, 1))
    U = vectors[:,:,0]
    V = vectors[:,:,1]

    print(det.shape)
    print(vectors.shape)
    print(U.shape)
    print(V.shape)

    plt.imshow(det, cmap='jet')
    plt.colorbar()
    plt.quiver(X, Y, U, V)

which should work for 2D images.
I assume that the return value of TransformToDisplacementField is a vector field, containing vectors with the same dimension as the input transformation - right?
Assuming that I have an image with size (40, 50) and a spacing of 1, the code should produce the correct result, but instead I get this:

Output of the shapes:
(40, 50)
(40, 50, 2)
(40, 50)
(40, 50)

Instead of using the functions, I also applied the transformation to a set of points and calculated the difference, which lead to this:
    x, y = image.GetSize()
    X, Y = np.meshgrid(np.arange(0, x, 1), np.arange(0, y, 1))

    U = np.zeros(X.shape)
    V = np.zeros(Y.shape)
    for index, value in np.ndenumerate(X):
        px, py = transform.TransformPoint((float(value), float(Y[index])))
        U[index] = px - value
        V[index] = py - Y[index]
    plt.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)

which looks like this:

and would make more sense to me.
Should the vector field which is the result of TransformToDisplacementField not look the same?
Why are my coordinates flipped around in the output?
Thanks in Adavance!

Hello,
The SimpleITK ( and ITK ) images are indexed and size as image[X, Y, Z] with the x axis being the fastest. While as Numpy arrays are index array[z, y, x] and the x-axis is still the fastest. This is the conventions for accessing the same buffer in these two libraries. It is unfortunate that they are different.
So the problem is the interfacing between Numpy and SimpleITK coordinates.
The np.meshgrid does have a long note about this issue.
HTH,
Brad

So plotting plt.quiver(Y, X, V, U) instead of plt.quiver(X, Y, U, V) should resolve the issue? The image obviously fits now, but the result still does not look like the displacement field:

Or is this correct?! Comparing with the Jacobian plot in the background, my solution using the vector transformation looks more reasonable - at least to me :D

Yet another question emerged from that: The parameters from the BSplineTransformation, how are they ordered in the list? I found a single example which removed the Z displacement of the control points by setting the first len(params)//3 values to zero - so I assume the ordering of the Values is z_0, z_1, ..., z_n, y_0, y_1, ..., y_m, x_0, x_1, ..., x_p?

Hello @reox,
The standard ordering in ITK/SimpleITK is x-y-z, [x_0...x_N,y_0...y_N,z_0...z_N] and not z-y-x which is the standard in numpy.

@zivy thanks for the reply, but then this makes no sense:
https://github.com/faustomilletari/VNet/blob/584c1502f369f2547e7ed384f9cc25beb09f65e1/utilities.py#L85-L93
So is this code wrong in removing the z-displacements as it actually removes the x-displacements?
Or is this yet another notation mix-up, as the original image probably was zyx hence the x coordinates in ITK looked like the z coordinates?
I actually tried this now by creating a displacement field manually like [1,2,3,4, ..., 0, 0, 0, 0, ...], hence it should only affect the x coordinate - which it does.

Hello @reox,
Looks like a bug in that code, or it could be that the documentation there is wrong. While we all trust the documentation it is always best to create a simple test (pun intended). For projects that have been going on for a long while it is not uncommon to have discrepancies between code and comments. Code is updated but we forget to update the documentation.
Bottom line, only trust your tests. I will create an issue to improve the BSpline documentation (either add an example or discuss in the relevant Jupyter notebook).

thanks for the clarification! I think I understand the ordering now.
Yes, I should have done these simple experiments already yesterday ;)


sqlsrv_next_result() returns incorrect result with varchar(max) field when Always Encrypted is enabled

To reproduce this, connect with Always Encrypted enabled, create a table with only one varchar(max) column and then fetch the result by running two selects in a row. This is an example (the problem is gone when the column type is varchar(1024)):
$tableName = 'TextTable';
$sql = "create table $tableName (col1 varchar(max) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (ENCRYPTION_TYPE = deterministic, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256', COLUMN_ENCRYPTION_KEY = AEColumnKey) )";

$stmt = sqlsrv_query($conn, $sql);

$sql = "insert into $tableName (col1) VALUES (?)";
$params = array("This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test This is a test ");

$stmt = sqlsrv_prepare($conn, $sql, $params);
if ($stmt) {
    $r = sqlsrv_execute($stmt);
    if (!$r) {
        print_r(sqlsrv_errors());
    }
}

$stmt = sqlsrv_prepare($conn, "SELECT * FROM [$tableName]; SELECT * FROM [$tableName]");
if ($stmt) {
    $r = sqlsrv_execute($stmt);
    if (!$r) {
        print_r(sqlsrv_errors());
    }
}

echo "1. next result? ";
$next = sqlsrv_next_result($stmt);
var_dump($next);

$row = sqlsrv_fetch($stmt);
if ($row) {
    $fld = sqlsrv_get_field($stmt, 0, SQLSRV_PHPTYPE_STRING(SQLSRV_ENC_CHAR));
    echo(substr($fld, 0, 15)) . PHP_EOL;
}

echo "2. next result? ";
$next = sqlsrv_next_result($stmt);
var_dump($next);

echo "3. next result? ";
$next = sqlsrv_next_result($stmt);
var_dump($next);

The expected output is
1. next result? bool(true)
This is a test
2. next result? NULL
3. next result? bool(false)

The actual output is
1. next result? bool(true)
This is a test
2. next result? bool(true)
3. next result? NULL

Fixed in 5.1.2-preview


Regularized Regression: Change cross validation stats

JIRA:MADLIB-1169
Cross Validation seems to be supported by Elastic Net, SVM, and
Decision Trees. If a module is run with cross validation optimization
params, the output table corresponding to it displays mean and
std of the negative loss error for each permutation of the CV
params.

This commit changes column names: mean->mean_neg_loss and
std->std_neg_loss.
CV now uses negative Root Mean Squared Error, instead
of the negative Mean Squared Error.
Update Elastic Net user docs to reflect these changes.

Closes #210

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/296/


[possible BUG] IP address shown in PD pod log is http://<IP_ADDRESS>:port_number

Testing 1.8.0 of PD with podman 4.9.3,
going through the steps described here.
When looking up the local URL (line 21 of the guide), I found this line in pod log:
Running on local URL:  http://<IP_ADDRESS>:7860
When connecting to this address, it returns "Unable to connect" error. When I manually switch it to localhost address, it displays the local app as expected.
My question is whether this is the expected behaviour?

Yes that is a log from inside the container you should use localhost to connect to that port


Add option for custom script type for injected js.

It would be nice to have the ability to have a custom type for the scriptType for the injected script.
My use case for needing this is so that it would play nicely with Google AMP, which requires all scripts to be of type application/ld+json.
PR: #188

Merged.


fix or disallow 'break' and 'continue' in 'finally'

break (and likely continue) in finally can create confusing definite initialization scenarios that aren't properly handled by the type checker.
It may make sense to disallow them (and return too, for good measure?)
Here's an example where break is used to "skip past" a return statement, which confuses the type checker:
shared void run() {
    String s;
    while (true) {
        try {
            if (1==2) {
                s = "hello";
                break; // if the type checker were smarter, we could comment out this line
            } else {
                return;
            }
        } finally {
            break; // skips over the "return" in "else" above
        }
    }
    print(s); // prints "<null>"!  (or on the JVM, a backend error)
}

By the way, I haven't tried it, but I remember someone saying that C# disallows this.

Yeah, I think they made the right decision with C#. These things are too confusing to possibly be useful.

This one (not involving finally) also compiles:
shared void run() {
    String s;
    while (true) {
        try {
            throw;
        } catch (e) {
            break;
        }
    }
    print(s);
}

@jvasileff so I've done two things:

I've fixed the definite assignment checking for the scenario of break in finally, so at least the two cases you provided are correctly detected as errors by the typechecker.
I've added warnings for any sort of control directive within a finally block. This seems reasonable to me: finally blocks definitely aren't for doing break/return/continue. However, I'm not certain about throw. I've certainly seen people throw from finally, and, though I don't like it, perhaps you need to sometimes?

WDYT? Is this a good solution?

I believe throw should always be allowed without warning (and, it’s unrelated to the initialization issue, and exceptions can occur in any code anyway), but otherwise that sounds reasonable to me.