Liqo route restarting due to Kubernetes API IO timeout

What happened:
Some of the Liqo Route pods is getting restarted due to IO timeout from Kuberentes API server while querying for discovery.liqo.io CRD.
Is it possible to disable discovery because we don't auto peer the new cluster.
Error,
k logs liqo-route-zhdtg -n liqo --previous
E0907 14:54:37.886389       1 mapper.go:109] Get "https://<IP_ADDRESS>:443/apis/discovery.liqo.io/v1alpha1?timeout=32s": dial tcp <IP_ADDRESS>:443: i/o timeout
E0907 14:54:37.886524       1 mapper.go:53] Get "https://<IP_ADDRESS>:443/apis/discovery.liqo.io/v1alpha1?timeout=32s": dial tcp <IP_ADDRESS>:443: i/o timeout
E0907 14:54:37.886555       1 cluster.go:202] "Failed to get API Group-Resources" err="Get \"https://<IP_ADDRESS>:443/apis/discovery.liqo.io/v1alpha1?timeout=32s\": dial tcp <IP_ADDRESS>:443: i/o timeout"
E0907 14:54:37.886568       1 route-operator.go:116] unable to get manager: Get "https://<IP_ADDRESS>:443/apis/discovery.liqo.io/v1alpha1?timeout=32s": dial tcp <IP_ADDRESS>:443: i/o timeout

What you expected to happen:
No restarts due to discovery CRD
How to reproduce it (as minimally and precisely as possible):
It's difficult to reproduce as it depends on Kubernetes API response
Anything else we need to know?:
Environment:

Liqo version: v0.9.1
Liqoctl version: v0.9.1
Kubernetes version (use kubectl version): 1.21
Cloud provider or hardware configuration: AKS
Node image:
Network plugin and version:
Install tools:
Others:

Liqo route DaemonSet does not exist anymore, starting from Liqo v1.0.0-rc.1


 System Linux 3.19.0-25-generic
gyp ERR! command "/usr/bin/nodejs" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/reply/Desktop/node_modules/v4l2camera
gyp ERR! node -v v5.5.0
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
npm WARN ENOENT ENOENT: no such file or directory, open '/home/reply/Desktop/package.json'
npm WARN ENOENT ENOENT: no such file or directory, open '/home/reply/Desktop/node_modules/build/package.json'
npm WARN ENOENT ENOENT: no such file or directory, open '/home/reply/Desktop/node_modules/frames/package.json'
npm WARN EPACKAGEJSON Desktop No description
npm WARN EPACKAGEJSON Desktop No repository field.
npm WARN EPACKAGEJSON Desktop No README data
npm WARN EPACKAGEJSON Desktop No license field.
npm ERR! Linux 3.19.0-25-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "v4l2camera"
npm ERR! node v5.5.0
npm ERR! npm  v3.3.12
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the v4l2camera package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls v4l2camera
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /home/reply/Desktop/node_modules/npm-debug.log

solved with the prerequisites installation
The error is due to gcc version


DELETE calls should not require a body

Describe the bug
Calling a delete rest endpoint with out causes an error and prevents call from being made.
To Reproduce
Steps to reproduce the behavior:

Call a delete rest endpoint with no body.
e.g. [DELETE] https://mygreatapi/resource/1234

Expected behavior
Delete calls in general usually don't have bodies, it's not forbidden but most apis don't use it and some go so far as to forbid it.  In Either case including a body on a delete to a resource is usually not applicable in a fully restful api.
Desktop (please complete the following information):

iOS

Additional context
The issue is in the HttpRequestHandler.swift file where 'isHttpMutate' set for multiple calls, and if that's set it explicitly checks for a body in the request, if it doesn't find one it rejects the call.  Mutate with body make sense for 'PUT', 'POST', 'PATCH', but 'DELETE' should not be included in that logic as majority of calls will not have a body.
Supporting information
https://stackoverflow.com/questions/299628/is-an-entity-body-allowed-for-an-http-delete-request
Proposed fix tested in fork
`
// Change this
let isHttpMutate = method == "DELETE" ||
method == "PATCH" ||
method == "POST" ||
method == "PUT";
// To This
let isHttpMutate =
method == "PATCH" ||
method == "POST" ||
method == "PUT";
`

Closing since this is merged in #196


Ensure the iam group is cleaned up

In one test run, the group was left behind, which made future test runs fail earlier, and it would never recover automatically.
Signed-off-by: Justin Kulikauskas<EMAIL_ADDRESS>

/retest

@JustinKuli A rebase on this one should get the CI to pass since the Gatekeeper tests were removed, assuming the PR is still relevant.


No repository description.

Proposed one:

Android application that pulls in news & inspiration from Designer News, Dribbble & Product Hunt.

Thanks for the prompt, I added a description.


gzip support.

Hi guys there is anyway to use gzip with adonis ?
Regards.

I believe you can use any gzip library to create the response

Hi Vik, i'm trying to create a Middleware to compress the responses but the brower is displaying
Content Encoding Error
'use strict'
const zlib = require('zlib');
class GzipCompressor {

handle (request, response, next) {
// here goes your middleware logic
// yield next to pass the request to next middleware or controller
response.header('Content-Encoding','gzip');
const output = zlib.createGzip();
output.pipe(response.response);
yield next
}

}
module.exports = GzipCompressor

I have been trying to send the response with Gzip so i installed an NGINX server for a reverse proxy with the
gzip              on;
gzip_http_version 1.0;
gzip_proxied      any;
gzip_min_length   500;
gzip_disable      "MSIE [1-6].";
gzip_types        text/plain text/xml text/css
text/comma-separated-values
text/javascript
application/x-javascript
application/atom+xml;
and the site is responding with gzip but the assets (js, css, images) are not.
When the page renders in the browser the scripts and links tags try to resolve the files and the adonis process is delivering those files to the browser not the NGINX process.
Do you know any ways this can work ?
Thanks a lot in advance!

Hey @pikitgb ! 👋
You probably need to change your NGINX configuration by adding a new location to ~/public that will overwrite your default proxy_pass location.

Closing since no response from the issue reporter.


Create Custom Resources to track Storage Classes metadata

Goal
Create Custom Resource to track all the Storage Class metadata such as description, name, enablement...
Dependency issue
No dependencies
Itemized goals
Create a Custom Resource for the dashboard that allows:

Link the CR to an existing Storage Class
Add Display Name
Enablement of the resource
Select if the resource is the default one
Date of last modification

Aspects continued elsewhere

https://github.com/opendatahub-io/odh-dashboard/issues/1363

Mocks

https://www.sketch.com/s/f6fa37fc-0c25-49e3-8b24-ecaf981433a8/a/8VkRW3O

Migrated to:  https://issues.redhat.com/browse/RHOAIENG-1104


Update GetDIDs to accept a deleted query parameter

Overview
This PR introduces a deleted query parameter to the GetDIDs API call.
WIP - some sections not completed yet, more work to be done to this PR
Description
When calling GetDIDs, the presence of a query param deleted="true" should return only DIDs that have been soft deleted (has isSoftDeleted flag), while calling GetDIDs without the presence of the query param should return only DIDs that have not been soft deleted.
How Has This Been Tested?
Describe the tests that you ran to verify your changes. Provide instructions for verification.

[ ] Test A (e.g. Test A - New test that does ... run in ...)
[ ] Test B

Checklist
Before submitting this PR, please make sure:

[ ] I have read the CONTRIBUTING document.
[ ] My code is consistent with the rest of the project
[ ] I have tagged the relevant reviewers and/or interested parties
[ ] I have updated the READMEs and other documentation of affected packages

References
Please list relevant documentation (e.g. tech specs, articles, follow up or related work) relevant to this change, and note if the documentation has been updated.

Codecov Report

Merging #402 (23a4b4d) into main (d673467) will increase coverage by 0.02%.
The diff coverage is 21.42%.

@@            Coverage Diff             @@
##             main     #402      +/-   ##
==========================================
+ Coverage   20.43%   20.46%   +0.02%
==========================================
  Files          48       48
  Lines        5422     5430       +8
==========================================
+ Hits         1108     1111       +3
- Misses       4139     4144       +5
  Partials      175      175

Impacted Files
Coverage Δ

pkg/server/router/did.go
4.59% <0.00%> (-0.26%)
:arrow_down:

pkg/service/did/handler.go
0.00% <ø> (ø)

pkg/service/did/ion.go
49.74% <0.00%> (+0.25%)
:arrow_up:

pkg/service/did/key.go
0.00% <0.00%> (ø)

pkg/service/did/service.go
0.00% <0.00%> (ø)

pkg/service/did/web.go
0.00% <0.00%> (ø)

pkg/service/did/storage.go
58.53% <100.00%> (+1.97%)
:arrow_up:


Update orion references to opentransit-collector

The git repository was renamed from orion to opentransit-collector. The code (e.g. environment variables and config files) and documentation should be updated accordingly.

Fixed in #2


Convert offline/snapshots and some utils to ts

Some work related to https://github.com/rnmapbox/maps/pull/2529#issuecomment-1458137599

@mattijsf thanks much! Merged as #2667


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_1d9hy860df/clone/recipe/meta.yaml:16: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_1d9hy860df/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_1d9hy860df/clone/recipe/meta.yaml:28: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_1d9hy860df/clone/recipe/meta.yaml:35: missing_dev_url: The recipe is missing a dev_url
WARNING: /tmp/abs_1d9hy860df/clone/recipe/meta.yaml:35: missing_description: The recipe is missing a description
ERROR: /tmp/abs_1d9hy860df/clone/recipe/meta.yaml:35: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


Remove '--service' usage in templates

'--service' has been deprecated. Pass the service as-is.
Fixes: https://github.com/Azure/azure-dev-pr/issues/1497

Repoman Generation Results
Repoman pushed changes to remotes for the following projects:
Project: todo-csharp-cosmos-sql
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-cosmos-sql -b pr/1846

View Changes | Compare Changes

Project: todo-csharp-sql-swa-func
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-sql-swa-func -b pr/1846

View Changes | Compare Changes

Project: todo-csharp-sql
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-sql -b pr/1846

View Changes | Compare Changes

Project: todo-java-mongo-aca
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-java-mongo-aca -b pr/1846

View Changes | Compare Changes

Project: todo-java-mongo
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-java-mongo -b pr/1846

View Changes | Compare Changes

Project: todo-nodejs-mongo-aca
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-aca -b pr/1846

View Changes | Compare Changes

Project: todo-nodejs-mongo-aks
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-aks -b pr/1846

View Changes | Compare Changes

Project: todo-nodejs-mongo-swa-func
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-swa-func -b pr/1846

View Changes | Compare Changes

Project: todo-nodejs-mongo
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo -b pr/1846

View Changes | Compare Changes

Project: todo-nodejs-mongo-terraform
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-terraform -b pr/1846

View Changes | Compare Changes

Project: todo-python-mongo-aca
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-aca -b pr/1846

View Changes | Compare Changes

Project: todo-python-mongo-swa-func
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-swa-func -b pr/1846

View Changes | Compare Changes

Project: todo-python-mongo
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo -b pr/1846

View Changes | Compare Changes

Project: todo-python-mongo-terraform
Remote: azure-samples-staging
Branch: pr/1846
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-terraform -b pr/1846

View Changes | Compare Changes

Flaky tests, will override

/check-enforcer override


Sampler creation on one 12 second file takes 5-8seconds on iphone 8 plus.

I'm on v5 develop.
I have 1 AVAudioFile of about 5 seconds in length.
The code to load the file from bundle to avaudiofile takes 0.0007 seconds.
        let fullpath = Bundle.main.path(forResource: path, ofType: "")
        let file = try AVAudioFile(forReading: URL(fileURLWithPath: fullpath!))

Then, I call Sampler(descriptor, file).
For a 4 second file, Sampler(descriptor, file) takes 0.63 to 0.8 seconds.
for a 12 second file, Sampler(descriptor, file) takes 5.1 to 8.3 seconds
This is on a physical device - iphone 8 plus, on ios14.
func initDescriptor(note: MIDINoteNumber) -> SampleDescriptor {
    return SampleDescriptor(noteNumber: Int32(note),
                          noteFrequency: Float(PolyphonicNode.tuningTable.frequency(forNoteNumber: note)),
                          minimumNoteNumber: -1,
                          maximumNoteNumber: -1,
                          minimumVelocity: -1,
                          maximumVelocity: -1,
                          isLooping: false,
                          loopStartPoint: 0,
                          loopEndPoint: 1,
                          startPoint: 0,
                          endPoint: 0)
}

    let a = Date()
    let file = Audio.loadAudioFile("12secondsample.wav")!
    let b = Date()
    sampler = Sampler(sampleDescriptor: initDescriptor(note: 60), file: file)
    print("create sampler time : \(Date().timeIntervalSince(b)), file duration: \(file.duration), file load: \(b.timeIntervalSince(a))")

For a file of duration 0.1 seconds, the load and creation happen in < 0.001 seconds.
The only part that scales with the size of the sample is the loadAudioFile.
This worked very fast in v4. (the loadAudioFiles() part -- i had one sampler that'd unload and reload samples on the fly when the instrument setting changed and things worked very well).
Please help Aure sir!

I tried with 12 files of 3 seconds each. Sampler creation took 9 seconds.
These messages are spaced in that interval:
AVAudioFile+Utilities.swift:toAVAudioPCMBuffer():39:Created buffer with format <AVAudioFormat 0x6000038437a0:  2 ch,  48000 Hz, Float32, non-inter> (AVAudioFile+Utilities.swift:toAVAudioPCMBuffer():39

For the same files, AppleSampler takes about 0.02 seconds to load.

Please provide a project that demonstrates this, really output thing load times as its output.

SamplerComparison.zip
5 Files of nearly 20 second samples, 48KHz.
Here's a project. Apple Sampler create/load time: 0.2 seconds.
AudioKit sampler create/load time: 12.35 seconds! ~62 times slower...
(On device - 7 plus and simulator - iphone SE 1 simulator, takes roughly the same time).
 Use Apple's "Instruments" app to get more detail on where that time is being spent.

I'll try and learn about instruments. I'm taking an intro to swift programming course, but this is covered in the next semester i think. To be honest, i need to demo my code to get a passing grade :-( I'm trying with apple sampler for now because the deadline is in 3 weeks.
I'll try and learn about instruments etc as soon as the final project is done. then we have a 2 week break in mid november for diwali.
Thank you for all your help and for audiokit - i hope my end project wows the prof :)

So, the author of the AudioKit Sampler, @getdunne and I took a look at this this afternoon. When this was first written, it relied on AKAudioFile type to get the floatChannelData. Since, then we moved onto AVAudioFile. This work, but it seems like a lot of time is spent grabbing the floatChannelData out in Swift and then processing it for the Sampler's loadAudioFile function. Not sure what can be done about this.

I'm going to reopen and get to the bottom of this.

With the latest AudioKit I'm now getting
av load time : 0.002042055130004883, apple sampler create time : 0.08357799053192139, akSampler create time: 2.977813959121704
Is that acceptable? Reopen if not.


Warper local storage

When you add a config file, add an option to have warpers be local and remote storage. I like to change it to demand only after I build everything.

Closed with https://github.com/Pasukaru/DSP-Mods/pull/3 .


Please use new import path

When building with golang1.4, it requires "golang.org/x/crypto/blowfish" but not "github.com/golang/crypto/blowfish".
Here: https://github.com/golang/crypto/blob/master/blowfish/cipher.go#L6
Can you fix this?
Thanks!

wtf,本来我提交 pr #63是为了让下载/编译时不要再用hg 去clone googlecode的那份代码
1.4的编译机制有啥区别呢

Great, thank you very much!


ci: deprecate old chart repo syncs

This PR stops syncing the previous helm repo at https://github.com/odigos-io/odigos-charts
The crds there are not relevant for long time, and the new helm repo is available for 5 month to migrate. see this blog post.
This PR completes the migration

Let's also update our pinned repo's 😃


[CORE] Rename CoalesceExecTransformer to ColumnarCoalesceExec

What changes were proposed in this pull request?
This pr renames CoalesceExecTransformer to ColumnarCoalesceExec.

xxxTransformer is used for operator which support offload to native
ColumnarXXX is used for operator which support columnar but not native

How was this patch tested?
Pass CI

cc @zhztheplayer @PHILO-HE  thank you


Conditional formatting - how does it work?

Hi,
firstly let me say I love this project. Absolutely fantastic.
I am having a problem implementing conditional formatting though. I can find no examples of how this is done in the documentation and am flailing around blindly. The closest I have got to something that compiles is:

let out_sheet = out_book.new_sheet("listDataTable").unwrap();

let mut new_cond = ConditionalFormatting::default();
let mut ran = Range::default();
ran.set_range("B2:B4");

let mut add = Address::default();
add.set_range(ran);
add.set_sheet_name("listDataTable");

let mut form = Formula::default();
form.set_address(add);
form.set_string_value(">0");

let mut cond = ConditionalFormattingRule::default();
cond.set_formula(form);

println!("cond coll: {:?}", out_sheet.get_conditional_formatting_collection());

new_cond.set_conditional_collection(vec![cond]);

out_sheet.set_conditional_formatting_collection(vec![new_cond]);

however this produces a mangled sheet that excel has to run a repair on.
I'm thrashing about and have tried various combos of code, but it's not googleable as not that many are doing talking/about it.
A simple/solid example of how to conditionally colour a range of cells if their value is over a certain threshold would be appreciated!
Thanks in advance!

@jimmer74
Thank you for contacting us.
ConditionalFormatting has not been a feature that has been talked about much since it was implemented.
Therefore, I believe it still has many glitches.
We will proceed to investigate the defects first.
I will respond as soon as the investigation is complete.

@MathNya thanks for the fast response.
Just looking for a simple example of the feature really. Just something so I get started (and test it and see how reliable it is for my use case).
TIA, Jim

@jimmer74
An example implementation is shown below.
    let mut out_book = new_file();
    let out_sheet = out_book.new_sheet("listDataTable").unwrap();

    let mut color = Color::default();
    color.set_argb("FF0000");  // Background color when conditions are met
    let mut pattern_fill = PatternFill::default();
    pattern_fill.set_background_color(color);
    let mut fill = Fill::default();
    fill.set_pattern_fill(pattern_fill);
    let mut style = Style::default();
    style.set_fill(fill);

    let mut form = Formula::default();
    form.set_string_value("20"); // threshold

    let mut cond = ConditionalFormattingRule::default();
    cond.set_type(ConditionalFormatValues::CellIs)
        .set_operator(ConditionalFormattingOperatorValues::GreaterThan)  // Judgment condition with threshold value
        .set_priority(1)
        .set_style(style)
        .set_formula(form);

    let mut seq = SequenceOfReferences::default();
    seq.set_sqref("B2:B4");  // scope

    let mut new_cond = ConditionalFormatting::default();
    new_cond.set_sequence_of_references(seq);

    new_cond.set_conditional_collection(vec![cond]);
    out_sheet.set_conditional_formatting_collection(vec![new_cond]);

    let path = std::path::Path::new("./tests/result_files/issue_218.xlsx");
    let _ = writer::xlsx::write(&out_book, path);


[Docs] Direct Line app service extension | endpoint for ASE given as "azurewebsite.net"

Document mentions the bot endpoint to be "azurewebsite.net" where in if the same is behind ASE it will be https://.appserviceenvironment.net/api/messages

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6c72c259-a864-9e1c-b585-562d6bca80ff
Version Independent ID: 30a3fce8-d236-5c7b-88a0-003c971f2819
Content: .NET bot with direct line app service extension - Bot Service
Content Source: articles/bot-service-channel-directline-extension-net-bot.md
Service: bot-service
GitHub Login: @Kaiqb
Microsoft Alias: kamrani

@MeenakshiBalekar, thanks for the feedback. The current document is specific to enabling the direct line app service extension in the Azure App Service resource where the bot is hosted and not for the App Service Environment scenario. For more details about using the ASE, please refer to https://docs.microsoft.com/en-us/azure/app-service/environment/using-an-ase#app-access


how can i jump to another IP port from Plugin ?

as the path to the rewrite must start with '/'
https://github.com/apache/apisix-java-plugin-runner/issues/54
that is to say now we can not jump to another IP port from Plugin ? Except rewrite path , has any other ways jump to another IP port?

no way in java runner.

The selection of IP and port is implemented in APISIX. I don't know exactly what you mean by jumping.

The selection of IP and port is implemented in APISIX. I don't know exactly what you mean by jumping.

rewrite host:port/path in java runner.
now i only can rewrite path，I don't know how to rewrite host in java runner .

rewrite host:port/path in java runner.

not support rewrite host: port in java runner.

I don't know how to rewrite host in java runner .

you can try  request.setHeader( "Host", "test.com"); to rewrite the Host header.

rewrite host:port/path in java runner.

not support rewrite host: port in java runner.

I don't know how to rewrite host in java runner .

you can try  request.setHeader( "Host", "test.com"); to rewrite the Host header.

config like this:
{
"uri": "/graphql",
"name": "graphql-ext-plugin-pre-req",
"plugins": {
"ext-plugin-pre-req": {
"conf": [{
"name": "AnalysisRequestFilter",
"value": "{}"
}]
}
},
"upstream": {
"nodes": {
"<IP_ADDRESS>:12:14561": 1
},
"type": "roundrobin"
}
}
when exec AnalysisRequestFilter I want to rewite upstream nodes host,  i don't want to use <IP_ADDRESS>:12:14561.

when exec AnalysisRequestFilter I want to rewite upstream nodes host, i don't want to use <IP_ADDRESS>:12:14561.

As I already said, this is not supported in java runner.

when exec AnalysisRequestFilter I want to rewite upstream nodes host, i don't want to use <IP_ADDRESS>:12:14561.

As I already said, this is not supported in java runner.

has other ways ? i set a new route like
"plugins": {
"proxy-rewrite": {
"host": "<IP_ADDRESS>:12:14561"
}
}
and i want use ext-plugin-pre-req AnalysisRequestFilter result in the new route rule to do some filter.

and i want use ext-plugin-pre-req AnalysisRequestFilter result in the new route rule to do some filter.

not support yet

and i want use ext-plugin-pre-req AnalysisRequestFilter result in the new route rule to do some filter.

not support yet

another question :
doc:  https://apisix.apache.org/zh/docs/apisix/admin-api/#body-请求参数
如果不同路由包含相同 uri，根据属性 priority 确定哪个 route 被优先匹配，值越大优先级越高，默认值为 0。
Q:  如果同时匹配到多个route的时候，只会按照优先级最高的route配置执行 还是？

Q: 如果同时匹配到多个route的时候，只会按照优先级最高的route配置执行 还是？

ref: https://github.com/apache/apisix/blob/master/docs/en/latest/router-radixtree.md
longest prefix match


Descriptions with new lines are improperly formatted in SCSS (or formats that use 'short' commentStyle).

We use the scss/variables format and attempting to migrate to this package from token-transformer. One issue i've noticed is that our descriptions contain new lines in the Figma plugin, and are being improperly formatted in the generated .scss file.
This doesn't seem to happen if we use css/variables or javascript/module formats.
Our build.js script
const { registerTransforms } = require('@tokens-studio/sd-transforms');
const StyleDictionary = require('style-dictionary');

registerTransforms(StyleDictionary);

const sd = StyleDictionary.extend({
  source: ['tokens/source/*.json'],
  platforms: {
    scss: {
      transformGroup: 'tokens-studio',
      prefix: 'viu',
      buildPath: 'src/scss/',
      files: [
        {
          destination: '_tokens.scss',
          format: 'scss/variables',
        },
      ],
    },
  },
});

sd.cleanAllPlatforms();
sd.buildAllPlatforms();

Screenshot of the token in the Token Studio plugin:

Screenshot of generated SCSS file:

I tried to follow the code in both this package and style-dictionary, and it seems certain formats are configured in style-dictionary with commentStyle: 'short' which expects a single-line comment string.
I forked the repo to try and come up with a solution, but I don't know the internals of style-dictionary well-enough to know if there's a way to override this configuration in the mapDescriptionToComment file. The only thing I could think to do was to escape the new lines in the string.
I would try to dig deeper, but I'm swamped with higher-priority tasks at the moment. For now, we'll continue to use the token-transformer package as we're also facing other issues with this package.
All of that being said, I'd just like to thank you for all of the hard work y'all have put into these tools. Token studio has been a great addition to our project in helping us streamline our dev/design handoff workflow.

Thanks for raising this, I will create a fix for this today as I believe it's an easy one.
Any other issues preventing you to move away from token-transformer, let me know!

Reproduction below. Switch the config css to scss to see the issue, this shows that with css/variables it wraps the comment into /* */, but because commentStyle is 'short' for  sass, less, scss, a bug happens with new lines, because short means "single line comments".
https://www.style-dictionary-play.dev/#project=rVLLTsMwEPyVytyqqCkXJHoEqVy54wptEyd1mtiR16lAVf8dP9rEDpBwYBXF9u7MeB8+k0zWUqV7QLbS8sgEriqUgmzImYrFgvo4JZuFO1uPxQYO6yoVfEYu66x5edAjr/WfoO6cACV3z84oSWJIJpuGCe1BWykpFeZ7ApPIALwk4W0Ny3nXzFz36Ox3kRzUcUbi3lkkcdsOWpQoln9rSCiz3a6NDTIht1SMiUn2em35Adtv3GJ+F5KYFoqCl6NhouxU5kTebtNcLtNlOHkvuks8oa1BF1I1GL2ADHH0ALQCgRb4omTX+iwdKiirVazgHz6GeRTad7zOX0EffNQdU8NPI1TBa4ZB8tZGk8oZai5Acym81PsJFIe9Ia7ifK6SJmXQfb5pj/5xwLtrm5NbI6q/9KHC+Vqrfyh1qLSaLrSCE2CmeKtThg+TlfYP6vIF
In my opinion, even for CSS with short commentStyle is looks really bad... I would expect something like:
/**
 * Do not edit directly
 * Generated on Tue, 21 Mar 2023 06:49:34 GMT
 */

:root {
  /**
   * Foo
   *
   * Bar
   */
  --sd-color-base-gray-light: #cccccc;
  --sd-color-base-gray-medium: #999999;
  --sd-color-base-gray-dark: #111111;
  --sd-color-base-red: #ff0000;
  --sd-color-base-green: #00ff00;
}

I could map descriptions to comments and trim the newlines, so you'd end up with:
$viuNeutral400: #8295ad; // Formerly named "Fog" Accessibility: 3:1 AA (used w/ white) For use with form field strokes and non-interactive elements only

This is the "fix" I was thinking of, but the problem isn't really in the transformer, the root cause of the problem is in Style-Dictionary createPropertyFormatter. The description is mapped to comment prop, it's up to a format to then deal with it appropriately. If we trim newlines in the transformer, we disable users from using those newlines in their formats to make good looking code comments, which is why I won't be doing this "fix".
I'll raise an issue in Style-Dictionary repo, I'm happy to raise a PR to fix it as well if they are okay with it.


Difference between num_hits and num_matches

Hi,
I am using,
type: frequency
num_events: 20
filter:
- query:
    query_string:
         query: 'some query'

When the alert is triggered I could see
num_hits: 0
num_matches: 1

I am confused with num_hits and num_matches. If there are 0 hits how can there be 1 match?

This looks like a bug I introduced recently with a separate fix. Should be fixed in #1769. Thanks for reporting


No route to host after scripts of job

Describe the bug
After the scripts in the job have finished, the runner throws an error.
To Reproduce
Sometimes I get a runner failure.
This is the error I'm getting at the end of the job. The scripts of the job have succeeded, but the job wasn't able to process the success result.
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4d1ca121 ...
WARNING: Failed to pull image with policy "always": error during connect: Post "https://<IP_ADDRESS>:2376/v1.41/images/create?fromImage=registry.gitlab.com%2Fgitlab-org%2Fgitlab-runner%2Fgitlab-runner-helper&tag=x86_64-4d1ca121": dial tcp <IP_ADDRESS>:2376: connect: no route to host (manager.go:237:3s)

Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4d1ca121 ...
WARNING: Failed to pull image with policy "always": Cannot connect to the Docker daemon at tcp://<IP_ADDRESS>:2376. Is the docker daemon running? (manager.go:237:0s)
ERROR: Job failed (system failure): Cannot connect to the Docker daemon at tcp://<IP_ADDRESS>:2376. Is the docker daemon running?

Expected behavior
The success result of the job is passed to GitLab
Additional context
I am using the latest version of the provider using the public network example.

This article seems to be interesting: https://forum.gitlab.com/t/debug-credentials-from-root-docker-config-json-vs-credentials-from-payload-with-gitlab-runner-and-container-registry/87757
Could you share some details of your .gitlab-ci.yml and some more lines of the job output?

There is a section at the bottom of this page describing the errors you encountered. Did you check this?

I have added the docker:dind service and the DOCKER_HOST variable in the terraform code for the runner worker. I'll send a new comment when I know more after some testing.

Below is the output of a job. This is just a sample, but more jobs are failing.
Job:
stages:
  - preparation
  - build
  - test
  - scan
  - publish

variables:
  WEB_DIR: "$CI_PROJECT_DIR/web"
  PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic"
  REACT_COMPONENTS_PROJECT_ID: "xxx"

image:
  name: public.ecr.aws/docker/library/node:lts-alpine

before_script:
  - cd $WEB_DIR

cache:
  key: web-npm
  paths:
    - $WEB_DIR/node_modules

web:npm:
  stage: preparation
  script:
    - npm config set ${CI_API_V4_URL#http*:}/projects/${REACT_COMPONENTS_PROJECT_ID}/packages/npm/:_authToken=${NPM_TOKEN}
    - npm config set @${CI_PROJECT_ROOT_NAMESPACE}:registry=${CI_API_V4_URL}/projects/${REACT_COMPONENTS_PROJECT_ID}/packages/npm/
    - npm ci

Complete log:
Running with gitlab-runner 15.8.2 (4d1ca121)
  on glrunner s[2](https://gitlab.com/<repo>/-/jobs/<project>#L2)LsxhJV, system ID: s_cd[3](https://gitlab.com/<repo>/-/jobs/<project>#L3)ed69122d5
Resolving secrets
00:00
Preparing the "docker+machine" executor
00:03
Using Docker executor with image public.ecr.aws/docker/library/node:lts-alpine ...
Pulling docker image public.ecr.aws/docker/library/node:lts-alpine ...
Using docker image sha256:3a617f[4](https://gitlab.com/<repo>/-/jobs/<project>#L4)39a70c99c88b4f6179a9c23bdc9ab21df2ad73[5](https://gitlab.com/<repo>/-/jobs/<project>#L5)4aae079c52caa5e3a7 for public.ecr.aws/docker/library/node:lts-alpine with digest public.ecr.aws/docker/library/node@sha25[6](https://gitlab.com/<repo>/-/jobs/<project>#L6):8e015de364a2eb2ed[7](https://gitlab.com/<repo>/-/jobs/<project>#L7)c52a558e9f716dcb615560ffd132234087c10ccc1f2c63 ...
Preparing environment
00:01
Running on runner-s2lsxhjv-project-41923800-concurrent-0 via runner-s2lsxhjv-glrunner-1700051085-f5e5e77b...
Getting source from Git repository
00:04
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/<repo>/.git/
Checking out 2279becd as refs/merge-requests/977/head...
Removing web/node_modules/
Skipping Git submodules setup
Restoring cache
00:10
Checking cache for web-npm-10-non_protected...
Downloading cache.zip from https://glrunner-301136870199-gitlab-runner-cache.s3.dualstack.eu-central-1.amazonaws.com/runner/s2LsxhJV/project/41923800/web-npm-10-non_protected
Successfully extracted cache
Executing "step_script" stage of the job script
03:30
Using docker image sha256:3a617f439a70c99c88b4f6179a9c23bdc9ab21df2ad7354aae079c52caa5e3a7 for public.ecr.aws/docker/library/node:lts-alpine with digest public.ecr.aws/docker/library/node@sha256:8e015de364a2eb2ed7c52a558e9f716dcb615560ffd132234087c10ccc1f2c63 ...
$ cd $WEB_DIR
$ npm config set ${CI_API_V4_URL#http*:}/projects/${REACT_COMPONENTS_PROJECT_ID}/packages/npm/:_authToken=${NPM_TOKEN}
$ npm config set @${CI_PROJECT_ROOT_NAMESPACE}:registry=${CI_API_V4_URL}/projects/${REACT_COMPONENTS_PROJECT_ID}/packages/npm/
$ npm ci
npm WARN deprecated subscriptions-transport-ws@0.9.1[8](https://gitlab.com/<repo>/-/jobs/<project>#L8): The `subscriptions-transport-ws` package is no longer maintained. We recommend you use `graphql-ws` instead. For help migrating Apollo software to `graphql-ws`, see https://www.apollographql.com/docs/apollo-server/data/subscriptions/#switching-from-subscriptions-transport-ws    For general help using `graphql-ws`, see https://github.com/enisdenjo/graphql-ws/blob/master/README.md
added 1136 packages, and audited 1137 packages in 30s
258 packages are looking for funding
  run `npm fund` for details
found 0 vulnerabilities
$ npm run lint -- --no-fix
><EMAIL_ADDRESS>lint
> eslint . --no-fix
Cleaning up project directory and file based variables
00:13
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4d1ca121 ...
WARNING: Failed to pull image with policy "always": error during connect: Post "https://<IP_ADDRESS>:2376/v1.41/images/create?fromImage=registry.gitlab.com%2Fgitlab-org%2Fgitlab-runner%2Fgitlab-runner-helper&tag=x86_64-4d1ca121": dial tcp <IP_ADDRESS>:2376: connect: no route to host (manager.go:237:3s)
ERROR: Job failed (system failure): error during connect: Post "https://<IP_ADDRESS>:2376/v1.41/containers/7a4cd71b8df1ae05271d55f1f0f86bab8[9](https://gitlab.com/<repo>/-/jobs/<project>#L9)b6ed60c39ae4920a7[10](https://gitlab.com/<repo>/-/jobs/<project>#L10)506a38e9d4d/wait?condition=not-running": dial tcp <IP_ADDRESS>[20](https://gitlab.com/<repo>/-/jobs/<project>#L20):[23](https://gitlab.com/<repo>/-/jobs/<project>#L23)76: connect: no route to host

Where should I set the DOCKER_HOST environment variable? It looks like it doesn't work using the runner_worker section.
  runner_worker = {
    environment_variables = [
      "DOCKER_HOST=tcp://docker:2375"
    ]
  }

I don't want to specify this for every pipeline, but globally for every pipeline at once.


Add tests for interactions with database

Look into https://pypi.org/project/pytest-postgresql/

[like]  Crabtree, Alastair reacted to your message:

From: Michele Facchinelli @.>
Sent: Friday, August 2, 2024 2:26:53 PM
To: ImperialCollegeLondon/imap-pipeline-core @.>
Cc: Subscribed @.***>
Subject: [ImperialCollegeLondon/imap-pipeline-core] Add tests for interactions with database (Issue #14)
Look into https://pypi.org/project/pytest-postgresql/
—
Reply to this email directly, view it on GitHubhttps://github.com/ImperialCollegeLondon/imap-pipeline-core/issues/14, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABP3TFMFCEAUHYAZX3P7EU3ZPOJK3AVCNFSM6AAAAABL4U4HE2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2DKMBXG44DSNA.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Allow subclassing from missing dependency

Problem
So far generalimport does not allow users to create subclasses of missing dependencies. So this code:
generalimport("my_module")
import my_module

class SubClass(my_module.BaseClass):
    def __init__(self):
        ...

fails with:
Traceback (most recent call last):
  File "/work/generalimport/example.py", line 6, in <module>
    class SubClass(test_module.BaseClass):
  File "/work/generalimport/generalimport/fake_module.py", line 21, in error_func
    raise MissingOptionalDependency(f"Optional dependency {name} was used but it isn't installed.")
generalimport.exception.MissingOptionalDependency: Optional dependency 'test_module' was used but it isn't installed.

This is quite limiting in OOP and makes using generalimport with libraries like sqlalchemy close to impossible.
Proposed solution
I propose leveraging __mro_entries__ to make this usecase work. Classes based on non-existing modules should fail only when initialized.
PR coming soon :slightly_smiling_face:

This is a really good idea, I haven't thought of this! I noticed that __init_subclass__ = error_func is the only method preventing subclassing currently.
I've never heard of __mro_entries__  before, never found it when I did my initial dunder research!
Found PEP 560 referencing it
Like I mentioned in the PR - I'd love if we could return a FakeModule object when it's subclasses


Macro: Spacing increases between peptide chains in snake mode when connected to RNA/DNA chain

Steps to Reproduce

Switch to Macro
Open file rna-chain-connected-to-peptide-chain.zip
Press 'Snake-mode'

Actual behavior
Spacing increases between peptide chains in snake mode when connected to RNA/DNA chain
Expected behavior
Spacing should be as before between peptide chains and RNA/DNA chain (see snapshot)
Screenshots

Desktop (please complete the following information):

OS: Windows 11
Browser Chrome
Version 112.0.5615.138 (Official Build) (64-bit)

Ketcher version
[Version 2.19.0-rc.2]

Test automated.

Moved to Refined Backlog since no fix over 2 versions


undefined name 'tags' in docs/conf.py

flake8 testing of https://github.com/zulip/zulip on Python 3.6.3
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./docs/conf.py:62:5: F821 undefined name 'tags'
    tags.add('unreleased')
    ^
1     F821 undefined name 'tags'
1

This is not a bug -- it's part of how Sphinx/ReadTheDocs works (they have this global, tags, that comes from elsewhere).  We have exclude rules with our linters (tools/lint) for handling cases like this.


Remove quelling blade from sell list

Makes PA not able to buy BF.
match code for reference:
7978520552

This isn't the root cause. Why would pa buy a bkb before buying a bf?

This fix will cause issues. You need to pair the items for selling. Items the selling list is paired with another item: when gets the first item -> sells the second item

Let me know if the above comment makes sense to you. I think without this code fix the PA should still be able to compete the BF. You can try to play a few more games and pick a PA for bots.
I'm going to close this issue in a few days since this won't fix any issue but can cause more issues.

I just did this as a workaround, whenever they sell the quelling they wont be able to complete the BF. I'm not sure whats the correct fix.

This workaround worked too, because I tried having PA in my next game and it was able to buy BF normally.

my point is bots can complete BF even without your fix. we need to know what is the actual root casue first before being able to fix it. And it seems we both dont know what the actual issue is here

it was because of
"item_power_treads",
"item_quelling_blade",

such that PA sells qb onces she gets pt
I've fixed it in commit: https://github.com/forest0xia/dota2bot-OpenHyperAI/commit/e5a7891c5a1db49d416a366026cd3f4446eecc35

Thanks for reposting this issue

a reminder, don't change the code the way you did, even if you saw some issues got fixed, you introduced issues such that other heroes may not be able to sell items properly and potentially other issues .

But dont get me wrong. I like to see more changes from you, just not the same buggy code without understanding the larger picture. Hope what I said can be heard

Oh no problem. Was just experimenting base on what I understand on the code. Thanks for fixing it,


How to attach to scroll controller

Is there any kind of way to attach to scroll controller. I'm trying to use swiper along with pull to refresh and when I set to loop false it uses the scroll physics and seems to absorb the scroll container above in pull to refresh. Any direction would bee appreciated.

Any news on this ? I can't use pull to refresh when using swiper

No news on this?


borderColor on Box uses given string value in the CSS, instead of the associated color

Describe the bug
When I set borderColor="border.default" on my Box component like the examples show, the generated CSS is border-color: border.default; instead of the expected border-color: #d0d7de, based on the theme reference.
To Reproduce
Create a Box with the following properties:
    <Box
      mt={4}
      p={4}
      textAlign="center"
      borderTopWidth={1}
      borderColor="border.default"
      borderTopStyle="solid"
    >

View your React app and see the following CSS was generated for the Box:
margin-top: 24px;
padding: 24px;
text-align: center;
border-top-width: 1px;
border-color: border.default;
border-top-style: solid;

Expected behavior
I expected border-color: #d0d7de to be the generated CSS for my component.
Screenshots

Desktop (please complete the following information):

OS: macOS
Browser: Safari
Version: @primer/components 28.5.0

Additional context
My React app component: https://github.com/cheshire137/nash-grave-plot/blob/63b7dc4148a27accbff50f5818a5d3a4e99e5190/src/components/Footer.js#L6-L20

I'll try this now, thank you!

Yay, the border color is now coming through, thanks!

I upgraded to @primer/components version 30.0.0.


Unwanted behaviour when keys are pressed too quickly

example:
direction=LEFT
if 'UP(or DOWN) & RIGHT' pressed too quickly

trigger gameover() (when snake.length > 2)
reverse snake direction.  (when snake.length <= 2)

Reason : It has to come from the fact that the eventListener of the keypress is not subject of the timing event of the displayFrame.
Possible mitigation : synchronize the timing event between keypress & displayFrame.

Solution : Define a boolean (canPress) if key can be pressed or not.
Set to true once per frame.
Set to false once a key is pressed.

Result : it feels like some directional inputs are not registered.
idk  what to do to get a better feeling


Improvement of select dropdown. Make arrow icon clickable

Status Quo: The arrow icon on top of the HTML5 select element cannot be clicked by the user. There is also no hover effect.
The arrow icon is created using an react-icon svg which is moved on top of the dropdown.
Solution:
https://tailwindcss.com/docs/pointer-events

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[Bug]

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
1.14
What's Wrong?
Datax Doris Write 以Json格式导入数据到Doris中，如果Json字符串中包含"%"，会报500错误
What You Expected?
解决方法：
com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver
-->
private Map<String, Object> put(String loadUrl, String label, byte[] data) throws IOException {
-->
httpPut.setHeader("Content-Type", "application/x-www-form-urlencoded");
需要修改成
if (Keys.StreamLoadFormat.JSON.equals(options.getStreamLoadFormat())) {
httpPut.setHeader("Content-Type", ContentType.APPLICATION_JSON.getMimeType());
} else {
httpPut.setHeader("Content-Type", "application/x-www-form-urlencoded");
}
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

There is no problem with the program test, and the code you mentioned is not found, please update the latest code


fp not declared problems

I was trying to make c2mir bootstrap on my windows x64 device today. After resolving all compatibility issues within standard library headers, I'm meeting undeclared func reg fp.
The problem is due to c2mir built-ins, here especially when va_arg returns a struct type, but fp is not set up for built-ins.
#include <stdlib.h>
#include <stdarg.h>
typedef struct T {
    int a;
} T;
static void* new_insn (void* ctx, void* code, int nops, va_list argp) {
  T *insn_ops = malloc (sizeof (T) * nops);

  for (int i = 0; i < nops; i++) insn_ops[i] = va_arg (argp, T);
  va_end (argp);
  return insn_ops;
}

(The script also reproduces the problem on my ubuntu machine, and I'm wondering why mir.c bootstraps on my ubuntu machine.)
#0  0x00007ffb4dd8a7d0 in msvcrt!exit () from C:\WINDOWS\System32\msvcrt.dll
#1  0x00000000004066af in default_error (error_type=MIR_undeclared_func_reg_error,
    format=0x4d1b87 <insn_descs+9863> "undeclared func reg %s") at mir.c:97
#2  0x000000000040e6a6 in get_func_rd_by_name (ctx=0x8b5c80, reg_name=0x4e343b <err_node+14971> "fp", func=0x9f2bc40)
    at mir.c:2025
#3  0x000000000040e711 in MIR_reg (ctx=0x8b5c80, reg_name=0x4e343b <err_node+14971> "fp", func=0x9f2bc40)
    at mir.c:2037
#4  0x00000000004a7792 in target_add_call_res_op (c2m_ctx=0x7db150, ret_type=0x85b8f90, arg_info=0x7a2228,
    call_arg_area_offset=0) at c2mir/x86_64/cx86_64-ABI-code.c:221
#5  0x00000000004b509c in gen (c2m_ctx=0x7db150, r=0x68d50f0, true_label=0x0, false_label=0x0, val_p=0,
    desirable_dest=0x7a3ab0) at c2mir/c2mir.c:12195

(gdb) p ((struct type*)0x85b8f90)->mode
$14 = TM_STRUCT
(gdb) p ((struct node*)0x68d50f0)->u.ops.head->u.s
$20 = {s = 0x866190 "__builtin_va_arg", len = 17}

Another problem is with empty structs.
typedef struct Foo{
} Foo;
Foo f() {
    return ((struct Foo){});
}
int main(void) {
    f();
    return 0;
}

This returns a structure, which makes c2mir go into the second branch in target_add_call_res_op which refers to the frame pointer. However, the previous detection code thinks that the stub does not use stack allocation, so fp is not defined in the context, causing an error.

A small refinement: the built-in case is due to fp not set up for caller, not built-in itself.

I think that the original mir bootstraps on linux because alloca macro expands to some explicit stack-using code, but on windows it does not.

After resolving all compatibility issues within standard library headers

your header set provides wrong type for va_list. since impl of varargs cannot be expressed with standard C, a compiler specific builtin has to be used.

After resolving all compatibility issues within standard library headers

your header set provides wrong type for va_list. since impl of varargs cannot be expressed with standard C, a compiler specific builtin has to be used.

I'm pretty sure that the MIR compiler injects the built-ins correctly, because it does not give errors if I include #error in my header file. Also, I am using the standard C headers for the reproducer on my ubuntu machine.

I've fixed it by https://github.com/vnmakarov/mir/commit/929113513818b42183d7c43b2495b8a3b4c89d6c

Cool, but it seems that the fix does not touch the core of the problem. By enlarging the size of struct T the problem comes up again.
#include <stdlib.h>
#include <stdarg.h>
typedef struct T {
    int a[100];
} T;
static void* new_insn (void* ctx, void* code, int nops, va_list argp) {
  T *insn_ops = malloc (sizeof (T) * nops);

  for (int i = 0; i < nops; i++) insn_ops[i] = va_arg (argp, T);
  va_end (argp);
  return insn_ops;
}

Here I would suggest setting up the frame pointer for all functions, but I do not know the performance implications of the operation.

Here I would suggest setting up the frame pointer for all functions, but I do not know the performance implications of the operation.

Performance would hurt for many important cases.  The problem is that FP can be potentially used in machine-dependent code of c2m  but fortunately only for va_arg calls.  So I will generate frame pointer for functions with va_arg calls.
I fixed the new test case by https://github.com/vnmakarov/mir/commit/237d058f3c38afc2e83ce6373bff38e6a2e57aab


Segfaults under Ubuntu when the ViewModel is null for a View

(This is more of an FYI for others who might encounter the same issue. )
Under Ubuntu-16.04, if I omit to provide an instance of the ViewModel to a View, I get segfaults or illegal instructions errors and the application abruptly stops. This does not happen in my development Windows 10 environment.
It is really simple to reproduce.
Controller:
[AllowAnonymous]
[HttpGet]
public IActionResult Login(string returnUrl = null)
{
	ViewData["ReturnUrl"] = returnUrl;
	return View();
}

View (basic Login form):
@model LoginViewModel

<form ....>
    <input asp-for="Username" />
    <input asp-for="Password" />
    ...
</form>

Result:
Segmentation fault (core dumped)

Or sometimes
Illegal instructions (core dumped)

This happens either after the very first request to that action, or after a few refreshes of the page.
To resolve this, simply pass a new instance of the ViewModel to the view:
[AllowAnonymous]
[HttpGet]
public IActionResult Login(string returnUrl = null)
{
	ViewData["ReturnUrl"] = returnUrl;
	return View(new CreateViewModel());
}

This makes sense, however I was expecting a NullReferenceException instead of segfaults. Under Windows, I noticed this first chance exception in the logs.
dotnet --info
.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

Hmm this definitely doesn't sound good. The process shouldn't crash because of something fairly trivial.
@kichalla - can you try to repro and we can file a new bug on https://github.com/dotnet/coreclr ?

I have tried to reproduce it many times but I am unable to. @BenjiZombie I suppose you created the application starting with something like dotnet new --auth Individual?

@kichalla I created the project on my Windows 10 dev env using VS 2017, and have since then upgrade to 2.0 and VS 15.3. The project is quite large now; I will try and create a smaller application to reproduce the issue and report back here.
Thanks

Thanks @BenjiZombie I will close this issue for now. Feel free to re-open it with more information. Thanks


ukvm on ARM

ukvm (and Solo5) is currently x86 specific.  It would be great if they ran on ARM as well.  I believe most of the mirageOS components already work with ARM.  A couple of discussion points:

how architecture independent can we make the solo5 code (e.g., by pushing hw-specific setup out of the kernel and into ukvm)
how should the ukvm code be arranged to cleanly support multiple platforms (e.g., KVM/Linux, Hypervisor.framework/MacOSx, vmmapi.h/FreeBSD?) and multiple architectures (e.g., x86, ARM) at the same time?

Hi Daniel,
My colleague Kaly had contacted you before. We're trying to port ukvm and Solo5 to ARM. But we are in  the very first stage. We have done some investigation and set up a development environment on ARM server. Currently, we can compile the sample apps in mirage-skeleton, but of course, they could not run.
In our consider, we can refer to Linux about the architecture independent design. We can create
an "arch" folder to store different architecture files. For example, use "arch/arm64" to store arm64
architecture files like cpu.S, cpu_vectors.S and etc. And we can split the system APIs that are used in kernel.c into two parts, use intr_init as an example:

Abstract interface:
void intr_init(void)
{
...
arch_intr_init();
...
}
Architecture implementation: arch_intr_init in arch/arm64/intr.c

About the ukvm code, I think it has combined with the x86_64 and KVM/Linux. I am sorry, I am not
very familiar with Hypervisor.framework/MacOSx, vmmapi.h/FreeBSD. The first plan could be using #ifdef CONFIG_ARM64 to add ARM64 support.
Above are the initiatory considers from us,  we'd like to discuss with you to push the multiple architectures/platforms supports of solo5 and ukvm :) .
Regards.

Ideally, all architecture-dependent code for Solo5 should be in one of the backends: kernel/virtio/ or kernel/ukvm/.  Clearly this is not the case now, because of things like kernel/cpu.S that you point out.
It should be possible to move all architecture-dependent code from kernel/ to kernel/<backend>, by trivially duplicating the code in either backend.
So, assuming kernel/ has no architecture-dependent code left, we can consider each backend separately.  I believe kernel/virtio will need an architecture subdir because it expects to run on fully emulated hardware.
Importantly, though, I'm not sure that kernel/ukvm needs ANY architecture-dependent code (with the exception of a hypercall.S or equivalent which contains the trapping instructions for the hypercalls (e.g., outl)).  For example, kernel/ukvm doesn't need the cpu_sse_enable routine in cpu.S because that setup can be done in the monitor (e.g., ukvm/ukvm-core.c) at start time and the Solo5 will never touch the control registers again anyway (we can set the VMCS up to trap all CR accesses).  I don't know the ARM architecture, but I would expect a similar situation.  Same for interrupts, descriptor tables, page tables, etc.
At that point Solo5 wouldn't need any work to port to other architectures, only ukvm would.  We need to talk to @mato to brainstorm how to organize the ukvm code for both multiple architectures and multiple "platforms".
So, to summarize, my preference is to try to avoid arch/arm64 in the kernel/ukvm subtree and try to push architecture-dependency down to ukvm/ in that case.
Does that make sense?

Hi Daniel,
Thanks for your summary. I have a little concerns about it :)

It should be possible to move all architecture-dependent code from kernel/ to kernel/, by trivially duplicating the code in either backend.

If we duplicate the arch-dependent files (Eg: intr.c, cpu_vector.S and cpu.S ) to each backend. Would they
be a little hard to maintenance in the future?

Importantly, though, I'm not sure that kernel/ukvm needs ANY architecture-dependent code.

There may be a little code need arch-dependent in kernel/ukvm , for example, the "cpu_halt" is using in the kernel/ukvm :)

Solo5 will never touch the control registers again anyway (we can set the VMCS up to trap all CR accesses). I don't know the ARM architecture, but I would expect a similar situation. Same for interrupts, descriptor tables, page tables, etc.

Yes, ARM has a similar situation. But I think the ukvm/ukvm-core should need arch-dependent code,
because ARM and x86 have different registers, and the KVM ioctls for these two platforms have a little
different behaviors.

At that point Solo5 wouldn't need any work to port to other architectures, only ukvm would. We need to talk to @mato to brainstorm how to organize the ukvm code for both multiple architectures and multiple "platforms".

Yeah, a brainstorm can help us to make more clear about multiple architectures support for ukvm :)
Regards,

If we duplicate the arch-dependent files (Eg: intr.c, cpu_vector.S and cpu.S ) to each backend. Would they be a little hard to maintenance in the future? ... There may be a little code need arch-dependent in kernel/ukvm

I believe all architecture-dependent code should be related to exits, which is the one architecture-dependent part needed in ukvm: defining the hypercall interface.  Things like rdtsc and hlt, which are now part of the interface (implicitly) could either be made to be explicit hypercalls (e.g., outl(PORT_RDTSC)) or also part of that single architecture-dependent file.  Any other architecture-dependent parts should be able to be completely removed from the kernel/ukvm subtree, so only one copy would exist in the kernel, specifically in the kernel/virtio subtree, avoiding a maintenance problem.
I agree that the monitor, ukvm, is massively architecture dependent.   Since we will have multiple backends in 2 dimensions (platform and arch), I think we will need to have all combinations:
E.g., a directory structure of:
            monitors
          /      |    \
    common    linux      macOSx ___
    /  |  \    | \ \__     |  \    \
noarch x86 arm |  \   \ noarch x86 arm
            noarch x86 arm

The monitors directory and multiple "platforms" are introduced in https://github.com/Solo5/solo5/pull/150, which still needs discussion.  Some examples of files (from that PR) that share commonality in different ways:
noarch/linux: monitors/ukvm/ukvm-net.c
noarch/macosx: monitors/uhvf/uhvf-net.c
noarch/common: monitors/blk.c
x86/linux: monitors/ukvm/ukvm-core.c
x86/macosx: monitors/uhvf/uhvf-core.c
x86/common: monitors/core.c

Any thoughts?

On Friday, 20.01.2017 at 06:29, Dan Williams wrote:

Importantly, though, I'm not sure that kernel/ukvm needs ANY architecture-dependent code (with the exception of a hypercall.S or equivalent which contains the trapping instructions for the hypercalls (e.g., outl)).  For example, kernel/ukvm doesn't need the cpu_sse_enable routine in cpu.S because that setup can be done in the monitor (e.g., ukvm/ukvm-core.c) at start time and the Solo5 will never touch the control registers again anyway (we can set the VMCS up to trap all CR accesses).  I don't know the ARM architecture, but I would expect a similar situation.  Same for interrupts, descriptor tables, page tables, etc.

On x86_64, while VMX does allow for very fine-grained control over what
constitutes a VMEXIT (including things like CR writes or exception
handling), none of this is exported via the KVM API to userspace (i.e. the
monitor). I'd imagine that on ARM the situation is more or less the same.
This brings up two important questions:

Should a guest be allowed to perform it's own exception handling?

Should a guest be allowed to use virtual memory (i.e. manipulate page
tables)?

If our answer to all of the above is "no", then we may as well run the
guest entirely in ring 3 (on x86_64, I believe the ARM equivalent is
"user mode"). This may have other consequences which would need to be
examined, at least TLS (*) and SMP (**) come to mind.
If our answer to either of the above is "maybe" or "we'd like to encourage
not doing that", then we should continue to use ring 0 for guests and move
the relevant setup and guest-side handler code(!) to the monitor, but not
enforce the situation.
If our answer to either of the above is "yes", then the code should stay
where it is now, and in that case I'd prefer it not be duplicated across
platforms (virtio and ukvm).
(*) On x86_64, TLS support relies on the FS.base and GS.base hidden
descriptors which are wired up to MSRs, so guests will need (some form of)
access to these.
(**) In my opinion SMP is not worth considering for unikernels as the
complexity involved is not worth the cost. My standard answer to people
asking for this is ("just spin up N unikernels"). I realise this is a
contentious point so I'm including it here and I've not thought about the
consequences of the above for supporting SMP guests.

So, to summarize, my preference is to try to avoid arch/arm64 in the kernel/ukvm subtree and try to push architecture-dependency down to ukvm/ in that case.

I agree that we should push as much of the architecture dependencies as is
practical to the monitor side. However, as I write above, I'm not sure that
this is 100% possible or desirable.
Thoughts?

Should a guest be allowed to perform it's own exception handling?
Should a guest be allowed to use virtual memory (i.e. manipulate page tables)?

I think for both of these, a default answer of "no" is appropriate, as there are a number of unikernel applications that do not require dynamic manipulation of these properties.
In the event that dynamic memory hotplug is required, this could be an additional library API that (optionally) uses ring-switching, but could also do something simple like sbrk if the guest only needs to extend a fixed mapping (a common scenario for a GC'ed language with a contiguous heap).

I also think that the answer to both questions is "no".
Wanted to add one more option: ukvm could do binary translation on the guest elf (at launch time) and do things like replace instructions with exits. It's a crazy option, but wanted to throw it there at least.

For clarity, I want to make sure we're all on the same page that we are not talking about the virtio backend: that needs to have some architecture-dependent parts.
For the ukvm backend, I also would hope the answer is "no" to both of your questions @mato .  I think of unikernels more like applications or processes tailored for the cloud, rather than stripped down systems.
The main issue with the KVM granularity from a ukvm point of view is things that we want to trap that don't or that get handled invisibly by the KVM module.  I was aware of a few of these: rdtsc for example, doesn't trap.  I had thought that wholesale trapping of CR's would be possible through KVM, but that might be wishful thinking.  Part of the Hypervisor.framework exercise was to understand the difference a bit better, but I think I just think of things in terms of VMCS now instead of KVM's abstractions :)  I still don't understand the intricacies of the MSRs.
If we can't get the things we want to trap but we still want an architecture-independent unikernel, we have a few options:

using ring 3
changing the KVM module
doing some BT

I think architecture-independence is possible (except for the hypercall interface) and that we should try to get there, even if it means doing some of the above points.   The first steps toward that goal would be:

push down as much arch-dependent code as possible to ukvm
start booting ukvm in ring 3 (we will also need to decide on what a "hypercall" is in this case)
re-evaluate whether KVM is giving us enough monitor-control

FWIW I also don't think SMP is something we should support, but I am curious as to what is going on with TLS...

E.g., a directory structure of:

        monitors
      /      |    \
common    linux      macOSx ___
/  |  \    | \ \__     |  \    \
...

This directory structure make sense.

The main issue with the KVM granularity from a ukvm point of view is things that we want to trap that don't or that get handled invisibly by the KVM module. I was aware of a few of these: rdtsc for example, doesn't trap. I had thought that wholesale trapping of CR's would be possible through KVM, but that might be wishful thinking.

Currently, I think both ARM and x86 KVM modules could not handle all traps invisibly :)

If we can't get the things we want to trap but we still want an architecture-independent unikernel, we have a few options:
using ring 3
changing the KVM module
doing some BT

I prefer to move the unikernel to user mode (ARM is EL0, x86 is ring3). But I am not sure this option can eliminate all traps to KVM module. For example, in ARM64, if we want to modify the PC register (equivalent is IP in x86) directly in user mode, we have to trap to kernel mode. But we don't have any code in kernel mode, we should have to configure the control register to trap such exception to KVM module...

Responding to both Dan's and Wei's recent comments:
@djwillia:

For clarity, I want to make sure we're all on the same page that we are not talking about the virtio backend: that needs to have some architecture-dependent parts.

Yes, all of this discussion is related to the ukvm backend. FWIW I don't
think it's worth the effort to port the virtio backend to ARM, however
anything we do in the process of reorganising the code for multi-arch and
multi-platform should be done in a way that doesn't preclude anyone
interested from doing so.

For the ukvm backend, I also would hope the answer is "no" to both of your questions @mato .  I think of unikernels more like applications or processes tailored for the cloud, rather than stripped down systems.

I agree with the above definition, however there are valid use cases
for applications that want to manage virtual memory themselves. Whether or
not we want to/can support these is something to think about further.

The main issue with the KVM granularity from a ukvm point of view is things that we want to trap that don't or that get handled invisibly by the KVM module.  I was aware of a few of these: rdtsc for example, doesn't trap.  I had thought that wholesale trapping of CR's would be possible through KVM, but that might be wishful thinking.  Part of the Hypervisor.framework exercise was to understand the difference a bit better, but I think I just think of things in terms of VMCS now instead of KVM's abstractions :)

Why do you want to trap on RDTSC?

I still don't understand the intricacies of the MSRs.

My simplified understanding of TLS in the x86_64 ABI:
If you compile the following C code on Linux with -O0:
__thread int x = 1;

int main()
{
    return x;
}

the compiler will generate the following assembly:
mov    %fs:0x0,%eax

This will translate to the address loaded into the FSBASE MSR + 0x0. This
means that on every (?) thread context switch the scheduler must use WRMSR
to set FSBASE correctly. For example, in rumprun this is done here:
https://github.com/rumpkernel/rumprun/blob/master/platform/hw/arch/amd64/machdep.c#L157
See below for the implications of this.

If we can't get the things we want to trap but we still want an architecture-independent unikernel, we have a few options:

using ring 3
changing the KVM module
doing some BT

Option 3) feels very complex and we should avoid it if possible. Option 2)
less so however requiring a custom kernel module or contributing patches
to KVM means that we lose a large majority of potential users.
Option 1) is reasonable, if we can make it work. Exceptions, TLS and
efficient VMEXITs (hypercalls) from ring 3 look like the biggest
showstoppers on KVM.
Some preliminary thoughts on how Option 1) could work with KVM:
For exceptions: Move the exception handling code (i.e.  cpu_vectors.S and
friends) out of Solo5 and into the monitor, and run it in ring 0 on the
guest side.
For TLS: WRMSR is a privileged instruction, so it cannot be executed from
ring 3. Some very new (my Sandy Bridge i5-M does not, my Broadwell i7-M
does) Intel processors support WRFSBASE which is unprivileged. For older
processors we would have to implement ring 0 code provided by the monitor
and an interface to it from ring 3 (using SYSCALL or SYSENTER).
For efficient VMEXITs directly from ring 3, we could set RFLAGS.IOPL to 3
and continue using the existing port-based mechanism with its limitations
(no access to memory >4GB). VMCALL would be ideal (and is unprivileged) but
KVM both steals it for its own purposes and apparently (due to stealing
it) disallows its use from ring 3.

I think architecture-independence is possible (except for the hypercall interface) and that we should try to get there, even if it means doing some of the above points.   The first steps toward that goal would be:

push down as much arch-dependent code as possible to ukvm
start booting ukvm in ring 3 (we will also need to decide on what a "hypercall" is in this case)
re-evaluate whether KVM is giving us enough monitor-control

We need to start somewhere, so we should definitely continue on the path of
step 1). I'll take a look at your proposed changes in #150 to get those
unblocked and we'll see where it takes us.
For step 2), the full transition from ring 0 to ring 3 I'd say the jury is
still out on that.

I prefer to move the unikernel to user mode (ARM is EL0, x86 is ring3). But I am not sure this option can eliminate all traps to KVM module. For example, in ARM64, if we want to modify the PC register (equivalent is IP in x86) directly in user mode, we have to trap to kernel mode. But we don't have any code in kernel mode, we should have to configure the control register to trap such exception to KVM module...

@Weichen81:
Could you use a similar approach as what I describe above with exception
handling/TLS? I.e have ukvm provide code in EL1(?) to handle the "kernel-mode"
parts?
Further, if we punt on the issue of running the guest entirely in ring
3/EL0 and continue to run it in (virtualised) ring 0/EL1(?), does that make
your job easier to start with?

Hi Mato,

Could you use a similar approach as what I describe above with exception
handling/TLS? I.e have ukvm provide code in EL1(?) to handle the "kernel-mode"
parts?

Yes, I think it's possible. ARM provide some registers for system software to configure the target
that the exceptions will trap to. We can configure all EL0 exceptions trap to EL2 (A hypervisor mode of ARM, Linux ARM KVM module are running on EL2). And then the EL2 code will handle this exception in EL2 directly or use UKVM EL1 code to handle it.

Further, if we punt on the issue of running the guest entirely in ring
3/EL0 and continue to run it in (virtualised) ring 0/EL1(?), does that make
your job easier to start with?

Yeah, through the discussion of running the guest entirely in ring3/EL0 I understand
the relations among unikernel, backend and monitor much better. It make me easier to
start my job :)
BTW: I am on vacation until 6th Feb. I can still receive the emails, but maybe I could not
respond quickly. Sorry about that in advance :)

Hi Guys,
Some updates about the Solo5 for ARM64. Currently, we can run the test_hello.ukvm application on
an ARM64 server. But we encountered an IO issue. Because on ARM64 platforms, the IO port is not
mandatory. So the KVM_EXIT_IO is not supported on ARM64. (In our test, we configured the 0x499 port
of console as a normal memory, and all printf infomation has been write to this memory).
In our consider, we want to use the KVM_EXIT_MMIO to replace the KVM_EXIT_IO on ARM64. This means
all IO modules like ukvm-net, ukvm-blk need to do some modification to add KVM_EXIT_MMIO support.
Regards,
Wei Chen

@Weichen81 The lack of an "I/O port space" on ARM does not surprise me. Are there any other KVM_EXIT_XXX mechanisms available on ARM (apart from MMIO) which we could use? All we need is a fast path to "exit the guest and hand a (at least) 32-bit value to the VMM". The best mechanism for this on x86 would actually be VMCALL however KVM steals that for its own purposes, hence the use of OUT.

Hi Martin,
Yes, there is a KVM_EXIT_MMIO on ARM that we can use. Now, we are using it to "exit guest and do IO emulation in user space".  I also found that the x86 use CPU
Halt instruction in guest to exit guest with
KVM_EXIT_HLT. Unfortunately, ARM halt would not trigger such Kind exit reason.
ARM is using HVC to trigger KVM_EXIT_SYSTEM_EVENT and then use PSCI interface to shutdown the VM. But it will add more architecture dependent code to unikernel. So I suggest to add another ioport/mmio address to handle the guest shutdown. So both ARM and x86 can use this new IOPORT to implement CPU halt function in unikernel. How do you think about it?
Regards,
在 2017年2月22日，21:08，Martin Lucina <notifications<EMAIL_ADDRESS>写道：
@Weichen81https://github.com/Weichen81 The lack of an "I/O port space" on ARM does not surprise me. Are there any other KVM_EXIT_XXX mechanisms available on ARM (apart from MMIO) which we could use? All we need is a fast path to "exit the guest and hand a (at least) 32-bit value to the VMM". The best mechanism for this on x86 would actually be VMCALL however KVM steals that for its own purposes, hence the use of OUT.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Solo5/solo5/issues/151#issuecomment-281664209, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYDF64LYjOsGLQ1h7mbmpcFtahJbEFz3ks5rfDNEgaJpZM4Llucz.

Hi Wei,
On Wednesday, 22.02.2017 at 06:41, Weichen81 wrote:

Hi Martin,
Yes, there is a KVM_EXIT_MMIO on ARM that we can use. Now, we are using it to "exit guest and do IO emulation in user space".  I also found that the x86 use CPU
Halt instruction in guest to exit guest with
KVM_EXIT_HLT. Unfortunately, ARM halt would not trigger such Kind exit reason.
ARM is using HVC to trigger KVM_EXIT_SYSTEM_EVENT and then use PSCI interface to shutdown the VM. But it will add more architecture dependent code to unikernel. So I suggest to add another ioport/mmio address to handle the guest shutdown. So both ARM and x86 can use this new IOPORT to implement CPU halt function in unikernel. How do you think about it?

Yes, we will probably need an explicit "shutdown" hypercall. What I don't
know is what are the options for the ukvm hypercall mechanism on ARM. Is
it only KVM_EXIT_MMIO? Can you point me to a manual or the relevant KVM
code so that I can get a better idea of what the possibilities are?
Martin

@mato On ARM platform, when guest try to access I/O memory (its address is not in the memory region
that has been registered as standard RAM by user space). The guest will trap to VMM. If the KVM
code in Linux kernel could not handle this I/O request, it will exit to user space with KVM_EXIT_MMIO.
You can find the relevant KVM code in following URL (see io_mem_abort function for more details):
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/arm/kvm/mmio.c
When the guest trap to userspace, we will get a data structure.
struct {
__u64 phys_addr; ==> ioport number, eg, 0x499 for platform_puts
__u8  data[8];       ==> It's enough save a iov (eg, ukvm_puts) address
__u32 len;             ==> store the iov size
__u8  is_write;       ==> similar as the direction in KVM_EXIT_IO
} mmio;
For more detail you can find in the section of KVM_EXIT_MMIO in following document:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/virtual/kvm/api.txt
Regards,

Hi Guys,
Now, we can run test_hello on ARM64 server now!

root@ubuntu:~/solo5# ./bin hello
[load_code. 100000
[setup_cpuid] empty function
|      |
|  _ \  |  _ \ __
_ \ (   | | (   |  ) |
____/_/ _|_/____/
Solo5: Memory map: 512 MB addressable:
Solo5:     unused @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x105fff)
Solo5:     rodata @ (0x106000 - 0x106fff)
Solo5:       data @ (0x107000 - 0x108fff)
Solo5:       heap >= 0x109000 < stack < 0x20000000
**** Solo5 standalone test_hello ****
Hello, World
Command line is: ''
Solo5: solo5_app_main() returned with 0

And we also had been approved by ARM legal team, this means we can upload our code to public :)
Next setp, I think we can discuss how to organize the ARM code and x86 code (a branch or another git project?) .
But before that I want to ask a memory cache problem about solo5. While we was implementing the
IO emulation in userspace, we encountered a cache problem about the IO data. We have to flush the
cache in guest and invalidate the cache in host to sync the IO data between guest and host, or totaly disable the memory cache for the guest. But I can't find any cache maintain code in x86-solo5. Does it mean that, when x86 is running with physical memory address the mmu and cache are disable by default? If yes, without cache, how can we  guarantee the performance?
Regars,
Wei Chen

Hi Wei,
On Monday, 27.02.2017 at 00:02, Weichen81 wrote:

Hi Guys,
Now, we can run test_hello on ARM64 server now!
[...]
**** Solo5 standalone test_hello ****
Hello, World

That's great news!

And we also had been approved by ARM legal team, this means we can upload our code to public :)
Next setp, I think we can discuss how to organize the ARM code and x86 code (a branch or another git project?) .

Excellent. To get us started I think it'd be best if your code goes on a
branch (say, 'arm') of solo5/solo5 for now. Then we can look at how to best
integrate it. Seeing the code will help speed up design decisions around
the details. Please ensure your branch is based off the latest Solo5
'master'.

But before that I want to ask a memory cache problem about solo5. While we was implementing the
IO emulation in userspace, we encountered a cache problem about the IO data. We have to flush the
cache in guest and invalidate the cache in host to sync the IO data between guest and host, or totaly disable the memory cache for the guest. But I can't find any cache maintain code in x86-solo5. Does it mean that, when x86 is running with physical memory address the mmu and cache are disable by default? If yes, without cache, how can we  guarantee the performance?

To clarify, you're referring to synchronizing the hypercall mechanism
itself (which you've implementing using MMIO), to ensure that the data from
the MMIO write by the guest is seen correctly by the ukvm process. Correct?
Remember, on x86 we use PIO exclusively for the hypercalls. PIO is entirely
synchronous, so the MMU is not involved. Therefore caching does not come
into play. This is why I've been asking what other alternatives there are
on ARM for implementing the hypercall mechanism.
I'm not an expert on the finer details of memory caching, but if MMIO is
the only feasible way to implement the hypercall mechanism on ARM then we
would probably have to reserve a region of guest physical memory for this
purpose and disable caching on those pages only. @djwillia or @ricarkol may
have other ideas.
Martin

Hi Mato,
Regards,
在 2017年2月27日，17:41，Martin Lucina <notifications<EMAIL_ADDRESS>写道：
Hi Wei,
On Monday, 27.02.2017 at 00:02, Weichen81 wrote:

Hi Guys,
Now, we can run test_hello on ARM64 server now!
[...]
**** Solo5 standalone test_hello ****
Hello, World

That's great news!

And we also had been approved by ARM legal team, this means we can upload our code to public :)
Next setp, I think we can discuss how to organize the ARM code and x86 code (a branch or another git project?) .

Excellent. To get us started I think it'd be best if your code goes on a
branch (say, 'arm') of solo5/solo5 for now. Then we can look at how to best
integrate it. Seeing the code will help speed up design decisions around
the details. Please ensure your branch is based off the latest Solo5
'master'.
Ok, we are refining the code to patches. We will rebase the patches to the latest master upstream and then upload them to ARM branch as soon as possible.

But before that I want to ask a memory cache problem about solo5. While we was implementing the
IO emulation in userspace, we encountered a cache problem about the IO data. We have to flush the
cache in guest and invalidate the cache in host to sync the IO data between guest and host, or totaly disable the memory cache for the guest. But I can't find any cache maintain code in x86-solo5. Does it mean that, when x86 is running with physical memory address the mmu and cache are disable by default? If yes, without cache, how can we guarantee the performance?

To clarify, you're referring to synchronizing the hypercall mechanism
itself (which you've implementing using MMIO), to ensure that the data from
the MMIO write by the guest is seen correctly by the ukvm process. Correct?
No, the data from the MMIO written by guest is seen correctly by the Ukvm process. The synchronous issue is occurred while we are accessing the indirect data. The data pointer and length of guest console message are saved in ukvm_puts. Guest write ukvm_puts to MMIO, and ukvm process can get ukvm_puts correctly. But when we try to get console message through ukvm_puts, we found we always get the first message we print in guest.  Until we flush the cache in guest or disable the data cache of guest.
Remember, on x86 we use PIO exclusively for the hypercalls. PIO is entirely
synchronous, so the MMU is not involved. Therefore caching does not come
into play. This is why I've been asking what other alternatives there are
on ARM for implementing the hypercall mechanism.
But, on x86, only ukvm_puts is written through PIO, the guest console message is saved in a normal memory that is allocated by guest and shared by ukvm process. I feel very confused about that if the normal memory on x86 guest is cacheable too, why x86 have not encountered the same issue as ARM? So I want to know that when x86 disable the mmu, whether the memory cache is disabled either?
I'm not an expert on the finer details of memory caching, but if MMIO is
the only feasible way to implement the hypercall mechanism on ARM then we
would probably have to reserve a region of guest physical memory for this
purpose and disable caching on those
pages only. @djwillia or @ricarkol may
have other ideas.
That is what we are investigating now :)
Martin
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Solo5/solo5/issues/151#issuecomment-282672324, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYDF65sfdgYq4EUbnelfrccA84vZGqHHks5rgppggaJpZM4Llucz.

@mato We have refined the patches to two parts:

Initialize the multi-arch support of solo5.
64-bit ARM support for solo5
Both of these two parts are almost done, we'll upload them in these last few days.
Can I create a branch and upload patches directly to solo5 git? Or should I need some authorization?

Regards,
Wei Chen

@Weichen81 Please publish the changes on a branch (not master) of your own Github fork of Solo5/solo5. We can then figure out how to merge them via a normal pull request. Note that I am also working on a refactoring of Solo5/ukvm for both multi-arch and multi-backend support, so we will have some reconciliation of the two approaches to do. I hope to publish this work within a week -- probably earlier for the first cut which will not have 100% final interfaces yet.

@mato I had tried to upload the patches to a new branch, but I got following message:
root@ubuntu:~/mul-arch/upstream/solo5# git push -u origin arm64
Username for 'https://github.com': xxxx
Password for<EMAIL_ADDRESS>remote: Permission to Solo5/solo5.git denied to Weichen81.
fatal: unable to access 'https://github.com/Solo5/solo5.git/': The requested URL returned error: 403

@Weichen81 Please upload your changes to a non-master branch on your own Github fork of Solo5/Solo5.

@mato I have uploaded my changes to my fork Solo5:
https://github.com/Weichen81/solo5/tree/arm64

root@ubuntu:~/mul-arch/solo5/tests# ./run-tests.sh
Starting tests at Wed Mar 22 07:59:42 UTC 2017
test_hello.ukvm                 : PASSED
test_globals.ukvm               : PASSED
test_exception.ukvm             : ERROR
**** Solo5 standalone test_exception ****
..
ukvm-bin: Invalid guest port access: port=0x400000000
test_fpu.ukvm                   : PASSED
test_blk.ukvm                   : PASSED
test_ping_serve.ukvm            : PASSED

We have passed ukvm test cases except test_exception.ukvm. That is because we don't enable the MMU in guest, all addresses that are not in the RAM range will be treated as IO abort.
Enable MMU in guest will also help us to solve the cache issue. We had discussed with Marc Zyngier (KVM/ARM maintainer) about this issue. He suggest us to enable MMU in guest to override the cache
attribute of the memory. Because on ARM64, if the guest set the memory to be non-cacheable then the memory will become non-cacheable, even though the host configure this memory to cacheable.
The MMU enable work is still in progress, we temporarily patch the Linux kernel to make the guest memory  non-cacheable to do above tests. But this patch will be dropped as soon as the MMU done.
Regards,
Wei Chen

Hi,
We have enabled the MMU for ARM64 guest to enable memory cache.
https://github.com/Weichen81/solo5/commit/9d1f576fb41886a7f533375e9d3be7494c3cd7e8
With this commit, we can solve the data inconsistent for IO emulation without patching the Linux kernel.
BTW:
We have sent an abstract about unikernels on 64-bit ARM to Open Source summit Japan and has been accepted: http://events.linuxfoundation.org/events/open-source-summit-japan
Here is the abstract:

Unikernels are specialised machine images that are generated using Library Operating Systems, which have smaller footprints, no operating systems and accessing baremetal hardware directly. These desirable properties make Unikernels small, fast and secure. Most Unikernels are running on backends. This backend simplifies the difficulty of hardware support needed by Library Operating Systems, as well as provides established resource isolation. A small modular monitor based on KVM called uKVM is such a backend whose functionality and interfaces are customized to the Unikernels. The 64-bit ARM Unikernels running on ARM64 needs uKVM support.

I want to introduce the current status of 64-bit ARM Unikernels on uKVM. Introduce the simpler I/O interfaces and performance boosts. Compare the interfaces of uKVM, Container and Virtual Machine. Discuss the problems need to be resolved.
If some guys of you would attend this conference I excepted to see you there :)
In next step I will prepare a presentation  and try to do a demo with solo5. I hope you can give me some
suggestions :)
Regards,
Wei Chen

@Weichen81 #192 is my follow-up to #171 with the multi-arch refactoring, please take a look. Once that is merged we can start upstreaming (merging) your ARM64 port.

(At LF OSS Summit Japan, May 31-June 2)
I want to introduce the current status of 64-bit ARM Unikernels on uKVM. Introduce the simpler I/O interfaces and performance boosts. Compare the interfaces of uKVM, Container and Virtual Machine. Discuss the problems need to be resolved.

That's good news. Hopefully ARM64 support will be fully upstreamed by then :-)

@Weichen81:
Given that #192 and #171 are now both merged to master, now would be a good time to refactor your work in progress to the new order of things.
I realise that you have a long-lived branch off a (now rather old) master with lots of intermediate history of your work on it, however given that master has changed so much in the mean time I don't think that attempting to rebase onto it (thus keeping your history) is worth the effort.
So, I would recommend that you start with a new branch off master and refactor your changes anew.
Let's continue to discuss the steps leading up to a PR for your ARM64 port in this issue.

@mato

Given that #192 and #171 are now both merged to master, now would be a good time to refactor your work in progress to the new order of things.

That's great!

I realise that you have a long-lived branch off a (now rather old) master with lots of intermediate history of your work on it, however given that master has changed so much in the mean time I don't think that attempting to rebase onto it (thus keeping your history) is worth the effort.

So, I would recommend that you start with a new branch off master and refactor your changes anew.

Yes, it's hard to rebase the changes to latest master. I will create a new branch to refactor ARM64 changes.

Let's continue to discuss the steps leading up to a PR for your ARM64 port in this issue.

Ok, thank you :)

Hi @Weichen81, how are you getting on with refactoring the ARM port? Any progress?

Hi mato,
I am still reworking the changes of arm64,
The slides of oss 2017 costs me sometime,
I think I would push the reworked patches to
GitHub within this week.
Regards,
Wei Chen
Regards,
在 2017年5月23日，17:45，Martin Lucina <notifications<EMAIL_ADDRESS>写道：
Hi @Weichen81https://github.com/weichen81, how are you getting on with refactoring the ARM port? Any progress?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Solo5/solo5/issues/151#issuecomment-303347592, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYDF6yrgFCpveWiVwRwAu0X-gyzF2ckgks5r8qrDgaJpZM4Llucz.

@mato I have pushed a new branch which is based on your refactoring on:
https://github.com/Weichen81/solo5/tree/arm64-refactor
Regards,
Wei Chen

@mato Sorry, I forgot to do "git rebase" after I merged the latest master to my fork.
Now, the branch has been updated and my changes are based on latest commit:
8dc4934f4ce6f28f1dec0f297281e7100d71031f
Merge pull request #192 from mato/multiarch
And I am not very familiar with github, if I have something wrong in the merge progress,
please remind me :)
Regards,
Wei Chen

@Weichen81 Ok, this looks like it's correctly rebased. Can you open a PR and we'll do the review there? Thanks!

hi mato，
The pull request has been created.
Regards,
在 2017年5月26日，18:29，Martin Lucina <notifications<EMAIL_ADDRESS>写道：
@Weichen81https://github.com/weichen81 Ok, this looks like it's correctly rebased. Can you open a PR and we'll do the review there? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Solo5/solo5/issues/151#issuecomment-304248131, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYDF63PGuDKmC-qoltzI66B-I5I6yCjpks5r9qmJgaJpZM4Llucz.

Work in progress branches of Solo5 and MirageOS mid-layers on ARM64:
https://github.com/mato/solo5/tree/arm64
https://github.com/mato/ocaml-freestanding/tree/arm64
https://github.com/mato/mirage-solo5/tree/arm64
@Weichen81 The solo5 arm64 branch is identical to #193  except for an opam metadata change and the mmio_read32() change, feel free to pick these into your branch when you next update.
Currently these also require an opam pin add base https://github.com/janestreet/base.git and OCaml 4.04.1 in order to work around https://github.com/janestreet/base/issues/15.
MirageOS unikernels do not (yet) link successfully with these changes due to missing GCC math builtins, unclear where to get these from.
/cc @avsm

在 2017年6月14日，22:37，Martin Lucina <notifications<EMAIL_ADDRESS>写道：
Work in progress branches of Solo5 and MirageOS mid-layers on ARM64:
https://github.com/mato/solo5/tree/arm64
https://github.com/mato/ocaml-freestanding/tree/arm64
https://github.com/mato/mirage-solo5/tree/arm64
@Weichen81https://github.com/weichen81 The solo5 arm64 branch is identical to #193https://github.com/Solo5/solo5/pull/193 except for an opam metadata change and the mmio_read32() change, feel free to pick these into your branch when you next update.
Currently these also require an opam pin add base https://github.com/janestreet/base.git and OCaml 4.04.1 in order to work around janestreet/base#15https://github.com/janestreet/base/issues/15.
MirageOS unikernels do not (yet) link successfully with these changes due to missing GCC math builtins, unclear where to get these from.
@mato
While I was using the mirage conduit_server sample application as my demo in OSS. I have encountered the math library issue. That may be because of the 128-bit float used in ocaml-freestanding vsprintf.c
I treated the long double in this file as double to solve this issue. And the mirage unikernels can be linked successfully.
I have made a temporary patch but haven't had time to refine and push it to github.
/cc @avsmhttps://github.com/avsm
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Solo5/solo5/issues/151#issuecomment-308451602, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYDF69F3QM8pcuhWKO_ywLvp3JZGQX3Uks5sD_AigaJpZM4Llucz.

@Weichen81 When replying to Github notifications by email, could you please trim the quoted text to a minimum? Otherwise your comments are extremely hard to read (and include eg. duplicate @mentions).

While I was using the mirage conduit_server sample application as my demo in OSS. I have encountered the math library issue. That may be because of the 128-bit float used in ocaml-freestanding vsprintf.c I treated the long double in this file as double to solve this issue. And the mirage unikernels can be linked successfully.

I think this is a bug that needs fixing in openlibm -- it needs an implementation of frexpl() for aarch64.
The math builtins (eg. __addtf3, ...) can be had in libgcc.a, though I'm not sure that's the best place to get them from. @talex5, do you know what the story is for Mirage/xen on ARM?

@mato on arm(32/xen), mirage injects libgcc.a into the linker, see https://github.com/mirage/mirage/blob/v3.0.3/lib/mirage.ml#L1814-L1827  I suspect we'll need sth similar for solo5+arm.

@hannesm Right, so we will need to link libgcc.a (and equivalent for clang, later). Also, as I mentioned above, we'll either need to patch the printf core in ocaml-freestanding (yuck) or add an implementation of frexpl() for aarch64 to openlibm.

@mato I have updated the PR# https://github.com/Solo5/solo5/pull/193
The major change is using elf symbol to replace memory_regions.
If you have time please help me to review this update :)

I now have aarch64 PRs ready for the MirageOS downstream bits at mirage/ocaml-freestanding#25 and mirage/mirage-solo5#23.
@hannesm I've implemented the libgcc.a dependency directly in ocaml-freestanding, so the mirage tool does not need to know about it for freestanding builds on aarch64.

On 19/07/2017 17:04, Martin Lucina wrote:

@hannesm I've implemented the libgcc.a dependency directly in ocaml-freestanding, so the mirage tool does not need to know about it for freestanding builds on aarch64.

excellent choice! :)

The ARM64 support PR has been changed to https://github.com/Solo5/solo5/pull/209

#209 is now merged. Further items to complete ARM64 support in Solo5 and Mirage downstream:

https://github.com/mirage/ocaml-freestanding/pull/25. I am waiting for JuliaLang/openlibm#158 to be merged so that I can update my PR with the upstream fix rather than a "local" change.
https://github.com/mirage/mirage-solo5/pull/23

There will probably be other mirage downstreams that need porting (nocrypto and gmp-freestanding come to mind), I will file separate issues for these.

@mato @djwillia @avsm @ricarkol
Thank you!
And could you please invite me to Slack #Solo5 channel? It's great if we can chat on that too :)
I remember I also modified following two packages for AArch64 to run mirage applications:
packages/base/base.v0.9.2
packages/mirage-entropy/mirage-entropy.0.4.0

This is essentially complete from the Solo5 side, and on the MirageOS side the stack builds and runs as far as the device-usage/network example from mirage-skeleton. Therefore, I'm closing this, any remaining downstream MirageOS issues can be discussed separately.


Clean include files

Checklist

[X] Used a personal fork of the feedstock to propose changes
[X] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

I noticed some warnings when compiling TangoTest:
ClobberWarning: Conda was asked to clobber an existing path.
source path: C:\Users\xxxxx\mambaforge\pkgs\omniorb-libs-4.2.5-hc7e8586_1\Library\include\openssl\ui.h
target path: C:\Users\xxxxx\mambaforge\conda-bld\tango-test_1648068062307_test_env\Library\include\openssl\ui.h
The omniorb-libs output includes files from other libs under "include".
By default conda-build only includes new files when creating a package but that doesn't seem to work
when using several outputs.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender


Minify generated schema.json

I added --minify option to ts-json-schema-generator because some of my schemas have gotten bigger.
I've just enabled it here too :)

We should really treat the schema file as generated output. So actually, can you do the following?

Delete public/schema.json
Add the file to the .gitignroe
Add schema task in the turbo.json pipeline

Now we can minify and also just have single file diffs

Just adding the schema task to turbo.json will not run it automatically. Will you run it separately?


feat: implement PUC-Rio tonumber

Implements tonumber according to the Lua 5.4 manual https://www.lua.org/manual/5.4/manual.html#pdf-tonumber.

Looks good now, thank you!