Node.js  Outreach Meeting 2020-10-13

Time
UTC Tue 13-Oct-2020 16:00 (04:00 PM):

Timezone
Date/Time

US / Pacific
Tue 13-Oct-2020 09:00 (09:00 AM)

US / Mountain
Tue 13-Oct-2020 10:00 (10:00 AM)

US / Central
Tue 13-Oct-2020 11:00 (11:00 AM)

US / Eastern
Tue 13-Oct-2020 12:00 (12:00 PM)

EU / Western
Tue 13-Oct-2020 17:00 (05:00 PM)

EU / Central
Tue 13-Oct-2020 18:00 (06:00 PM)

EU / Eastern
Tue 13-Oct-2020 19:00 (07:00 PM)

Moscow
Tue 13-Oct-2020 19:00 (07:00 PM)

Chennai
Tue 13-Oct-2020 21:30 (09:30 PM)

Hangzhou
Wed 14-Oct-2020 00:00 (12:00 AM)

Tokyo
Wed 14-Oct-2020 01:00 (01:00 AM)

Sydney
Wed 14-Oct-2020 03:00 (03:00 AM)

Or in your local time:

https://www.timeanddate.com/worldclock/fixedtime.html?msg=Node.js+Foundation+Outreach+Meeting+2020-10-13&iso=20201013T1600
or https://www.wolframalpha.com/input/?i=04PM+UTC%2C+Oct+13%2C+2020+in+local+time

Links

Minutes Google Doc: https://docs.google.com/document/d/1kgVQdulOgmXcQsXXvObYqtWMDamkhwYGW1vAsmzdtN4/edit

Agenda
Extracted from outreach-agenda labelled issues and pull requests from the nodejs org prior to the meeting.
Invited

Outreach Members: @nodejs/outreach

Observers/Guests
Feel free to follow along on the YouTube live stream, or attend meeting as a guest
by calling in to Zoom, using the links below. If you will be attending as a guest,
please comment on this issue to let us know you'll be joining.
Notes
The agenda comes from issues labelled with outreach-agenda across all of the repositories in the nodejs org. Please label any additional issues that should be on the agenda before the meeting starts.
Joining the meeting

Link for participants: https://zoom.us/j/526318636
For those who just want to watch: https://www.youtube.com/nodejs-foundation

Let's discuss the meet in a box and hacktoberfest issues this time.


make django-helpdesk more customizable + bug fixes

look at settings.py for all new options regarding customization.

settings can be accessed inside the templates via the new
templatetag 'load_helpdesk_settings'

allow editing of personal followups, but only if followup does not
contain any ticketchanges - otherwise this information is lost after
the editing.

add 'delete' link to attachments

link to list of closed tickets in queue overview

add 'closed & resolved' section to dashboard

hide 'pre-set reply' box if no pre-set replies are found.

use 'SelectDateWidget' for custom DateField

add possibility to translate ticket comments using jquery & google translate

fix how we update followups so that attachments don't get deleted

fix bug where resolution emails contained the solution 'None'

fix stats crashing bug

fix locale bug

this closes bugs #86, #82, #81, #80, #63 and partially #74

It's been about a month since this pull request.  Looks like there is a lot of good stuff in here.  Should I pull from kotowicz's fork and submit my pull request to that, or are you planning to merge this soon, rossp?  Also, I use mercurial, so would it be best if I forked and updated my github repo with diffs from my mercurial repo?

I had to manually merge a few files due to conflicts with the previous merge request, but it was nothing too painful. Thanks for this submission!


Messaging via MQTT

Is your feature request related to a problem? Please describe.
I would like to receive a push notification from my home assistant app when loading is finished etc. Currently, I don't see how that is possible.
Describe the solution you'd like
I like the possibilities of the messaging feature and would love to get these messages via MQTT. Then I could easily forward them to home assistant.
Therefore, it would be nice to have MQTT as a source in the messaging configuration.

Activate mqtt in evcc, then set up a helper in Homeassistant for the status of your loadpoint. Automations change the status of your helper when the mqtt topic evcc/loadpoints/1/mode changes. Then you can set up another automation which sends you a notification when the status of your helper changes.


Requests failing with Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.

azure-keyvault-administration tests are failing unexpectedly with some very odd behavior.
When the failing tests are run individually, this behavior does not repro. Otherwise, it only repros when all tests are run for the package. (like in CI)
I personally think that there is something going a bit wonky with the keyvault-administration client. Caching or the like? Sending wrong content-length? But we need to dig in and find out.
In the meantime going to dash out a PR that adjust MinRequestBodyDataRate.
Stacktrace:
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder\u00601.StateMachineBox\u00601.System.Threading.Tasks.Sources.IValueTaskSource\u003CTResult\u003E.GetResult(Int16 token)
at System.IO.Pipelines.PipeReader.CopyToAsyncCore[TStream](TStream destination, Func\u00604 writeAsync, CancellationToken cancellationToken)
at Azure.Sdk.Tools.TestProxy.RecordingHandler.ReadAllBytes(Stream stream) in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs:line 1081
at Azure.Sdk.Tools.TestProxy.RecordingHandler.CreateEntryAsync(HttpRequest request) in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs:line 547
at Azure.Sdk.Tools.TestProxy.RecordingHandler.HandlePlaybackRequest(String recordingId, HttpRequest incomingRequest, HttpResponse outgoingResponse) in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs:line 438
at Azure.Sdk.Tools.TestProxy.Playback.HandleRequest() in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Playback.cs:line 88
at lambda_method45(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Awaited|12_0(ControllerActionInvoker invoker, ValueTask\u00601 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask

This turned out to be un-necessary. The failure was because the request the python tests were sending had a Content-Length, but never actually sent a body.
Reason was that the test was caching an empty credential body erroneously. Ensuring the cache was properly cleared means that the body would be sent with actual value, which resolved the problem.


add new permission 'publiclink.create' and assign it to the 'Admin', …

Added a new permission 'PublicLink.Write' to check if a user can create or update public links.

:boom: Acceptance test localApiTests-apiAsyncUpload-ocis failed. Further test are cancelled...

Sonarcloud is failing because of the duplication. In this case this is necessary so I'll force merge.


Design keyboard UI

UI should be themeable. User can select colors and whether the keys are flat or colored in.
Follow standard Google keyboard design patterns for best user experience.
Keys should popup briefly on tap, and more options should pop up on long press (select by dragging).

Progress update:

Progress in 53ab927b
resizing_keyboard.webm


Chore: Update repo structure to use Turbo / PNPM & new versions of next & wagmi

Updating wagmi is causing issues with wagmi connector not passed through app to package:
https://github.com/wagmi-dev/wagmi/pull/408
looking at other packages (ie rainbowkit, family connect) using wagmi as a dep.


refactor: eliminate endpoint constant classes

Description
We mostly migrated over to using the openapi.json files (by importing them) but there still remains a few endpoint constant files that are holding redundant information about operation IDs and HTTP paths/verbs.
We need to get rid of these and migrate the endpoint class implementations to importing the relevant openapi.json files instead.
examples/cactus-example-supply-chain-business-logic-plugin/src/main/typescript/business-logic-plugin/web-services/insert-bamboo-harvest-endpoint-constants.ts
Acceptance Criteria

The entire project is covered, e.g. there are no more x-endpoint-constants.ts files anywhere in the code.
The openapi.json files are being imported instead so that there's a single source of truth for endpoint metadata (path, verb, operation ID)

cc: @ty-lazar @Jeff-Ushry @twiggins120 @awadhana @Leeyoungone

I will do this! :)


chore(launch): More detailed logging in runners

Fixes WB-12345 (nice)
Description
Add extra logging statements to make debugging easier in the future.
Testing
N/A

mypy currently failing due to an unrelated issue in wandb_artifacts.py.


"chunk section" -> "section"

Right now we have a mix of both "chunk section" and just "section" to refer to 16x16x16 regions. I think we should shorten this to "section" everywhere, since "section" is a common enough term to refer to them. A few vanilla strings also refer to them as just "sections".

I see it as the phrase "chunk section" which means 16x16x16 regions, and "section" on its own as meaningless, and too vague, if you don't already happen to know what it means. "section" is arguably fine in parameter names where "chunk" can be inferred from other context.

Now that mojang has tweaked height and sections may become the basic unit of loading (this is already the case for entities). Will we reconsider naming this section?

After thinking, I agree that we should treat this thing as "chunk section". We do have "item stack" and frequently call it stack, yet "stack" on its own does not mean an item stack. Similar logic should go here.
This has been open for more than 1 year. The vote as of this comment is 6×:-1:. If there is no extra comments or votes in 24 hours, the current vote will be taken as the result.

If we are to rename it, I'd rename it to subchunk, which is widely used in the niche tech community and I think is a better name than chunk section, but chunk section is the more widely accepted name for it.

Seeing that mojang is already using them a lot since the expansion of height. I guess they may have a colloquial name for it one day in the release notes, and we can go and see by that point. (though mojang colloquial names aren't always very good either, like "chat component" is just... you know)

So the vote is clear. We won't name this section until mojang does some big mechanism changes, in case we shall revisit again.


FLUID-6209: Documentation for Remote Model and DataSource components

https://issues.fluidproject.org/browse/FLUID-6209
Includes updates that are included in https://github.com/fluid-project/infusion/pull/933

@amb26 would you be able to review this PR?

@amb26 I believe I've addressed all of your comments. Ready for more review.

@amb26 would you be able to review this DOCs PR?

@amb26 could you please take another look at this PR?

@amb26 I've made the changes you recommended and merged from main. Ready for more review.

@amb26 ready for more review.


Latest TS type definitions break the whole project with unclear errors when migrating from 0.3.2 to 0.4.1-rc.0

After migrating from 0.3.2 to 0.4.1-rc.0 a list of type errors popped up when building the project.
The first kind of errors:
Argument of type 'Entity<MY_COMPONENT>' is not assignable to parameter of type 'Entity<undefined>'.
  Type 'Entity<MY_COMPONENT>' is not assignable to type '{ __jecs_value: undefined; }'.
    Types of property '__jecs_value' are incompatible.
      Type 'MY_COMPONENT' is not assignable to type 'undefined'.

The second kind of errors:
Argument of type 'unknown' is not assignable to parameter of type 'T'.
  'T' could be instantiated with an arbitrary type which could be unrelated to 'unknown'.
  world.set(entity, component, componentData)

The latest types seem to be inconsistent with the "flecs" terminology "jecs" relies on.

Closing because of this rationale.


Crash in collect_types prep_args

When collecting types on Ubuntu 16.04 with Python 3.5 on the partially annotated Mission Pinball Framework codebase (https://github.com/missionpinball/mpf/), I get this error:
  File "/usr/local/lib/python3.5/dist-packages/pyannotate_runtime/collect_types.py", line 719, in _trace_dispatch
    resolved_types = prep_args(arg_info)
  File "/usr/local/lib/python3.5/dist-packages/pyannotate_runtime/collect_types.py", line 474, in prep_args
    resolved_type = resolve_type(arg_info.locals[arg])
KeyError: 'source'

Changing if not isinstance(arg, (list, dict)): to if not isinstance(arg, (list, dict)) and arg in arg_info.locals: fixes this for me. Don't know it that is a proper solution.

Can you please submit a PR with the fix so I can review and test it?

Closing as duplicate of #21 (because the latter shows a small example).


Monster ball deletes and does not capture villagers

Minecraft version & Mod Version: Minecraft 1.12.2, Cyclic 1.10.9
Single player or Server: Server
Describe problem (what you were doing / what happened): When I use a Monster Ball to catch a villager, the villager simply "disappears" as if it was caught, but the ball lands on the ground empty.
Crash log (if any): None.
(If this is a suggestion and not a bug report, delete all this and just write your suggestion!)

Any update on this?


Create a 'Signature Event' page to capture collective data/info about signature events

Along the lines of Partner & Promote ( #122)
Will then inform what is added to #32
At present this will be pulling together The Awakening (already published) and #133

Data prep in #150

Headlines

Total events
Total attendances
Total tickets sold

Graphs

Aggregated events and attendances graphs (by date)
Aggregated events and attendances hexmaps (by ward)
Tickets sold by ward of residence
Tickets sold by date of event

Initial version added to https://data.leeds2023.co.uk/metrics/events/signature/
May wish to convert graphs to cumulative event count. Will share with client for feedback and content

Shared updated pages with client on 9th August
Nudge sent on 15th August, no feedback on the pages as yet.
Follow up sent today


Support running tests with Rails 5.2

At #63 @zealot128 asks me to add tests to my PR. So I started with trying to run tests in the project by following the readme file.
The Gemfile in the project defaults to the latest Rails 5 version (5.2 as for now). But the actual tests code doesn't work with Rails 5.2. Here I added changes with which the tests work again with the latest Rails as defined in the Gemfile.
Changes required by Rails 5:

Puma is now the default webserver so must be in the Gemfile
ActiveRecord::Migration must be referenced by Rails version
use_transactional_fixtures is now use_transactional_tests

A change required by Rails 5.2:

Requires bootsnap to be in the Gemfile

Please note that this improved build results on travis.
Here are the results for the master branch and here are the results for my PR. I fixed most of the scenarios comparing to master except of 6 scenarios for Rails 4.2 which fail for master the same way.

Awesome! Thanks!


Experiment with collapsible logs

This experiment has taught me the following. Only "boring" things should be collapsed, e.g., build logs. Things like lint warnings should be visible without having to click.


invalid request to private npm repository

Related issues #943
The author gived the method

credentials << {
"type" => "npm_registry",
"registry" => ENV["DEPENDENCY_REGISTRY"],
"token" => ENV["DEPENDENCY_TOKEN"]
}

I have a question about how to deal with the registry protocol is http? Are there relevant parameters to pass？

In https://github.com/dependabot/dependabot-core/pull/5928, the protocol passed in is now honored, so you should now be able to pass in http.
I'm going to close as resolved, but please try it and if for some reason it's not working comment and we can re-open.


CI tests fail due to missing requirements

The ci is failing when running tests due to pip packages not being installed. The workflow should be modified to install the required packages first

the python-packages.yml workflow allows for the use of a requirements.txt
add a requirements.txt file

yup, on the todo list!

thanks @lkurz - didn't know you were watching the HSDS live issue stream...


Make Janus a PyPI package

We would like to make JANUS a PyPI package to ease its integration into PROTEUS.

For JANUS to be made a PyPI package, it needs to be able to automatically compile SOCRATES.
For AGNI, which also uses SOCRATES, I've tried to make this a bit easier by adding a configure script. See files in this folder: https://github.com/nichollsh/AGNI/tree/main/socrates. The configure script generates a Mk_cmd file automatically, by locating the NetCDF libraries and adjusting things for your platform.
Maybe this is useful for JANUS.


Swift 4.2

Update to 4.2 please.

If you are using the latest Xcode version, just follow the hint on authorize and then Cmd + Shift + K to clean the Build Folder and try again. I have done and it run properly with Swift 4.2. Good luck!

Unfortunately, this library has not been updated for a long time and does not work after swift 4.2.
If you get an error and cannot use this library, you can visit it.
For the active repo and many new features,
https://github.com/ugurethemaydin/SwiftConfettiView

For anyone on a different version of Swift than SAConfettiView, you can do something like this in CocoaPods:
target 'myapp' do

  # CocoaPods isn't up-to-date, so use the latest commit
  pod "SAConfettiView", :git => 'https://github.com/sudeepag/SAConfettiView.git', :commit => '4c78e355def75a37b707af63b393ba09895405a9'

end

# Force SAConfettiView to build with Swift 4
post_install do |installer|
  installer.pods_project.targets.each do |target|
    if ['SAConfettiView'].include? target.name
      target.build_configurations.each do |config|
        config.build_settings['SWIFT_VERSION'] = '4.0'
      end
    end
  end
end

Thanks for this @bendytree! I'm now using it for other pods that haven't updated in a while either.


Implement all the client-specific context methods

Background
Several client-specific validation and execution methods remained unimplemented, including:

delete_update_meta
delete_consensus_state
consensus_state_heights

Additionally, there is room for improving the efficiency of existing methods to enhance maintainability and leverage reusable code from shared methods.
This effort is a preliminary step towards enabling the CW contract execute entrypoint to function properly.

Accomplished by https://github.com/informalsystems/sovereign-ibc/commit/6daa44705f6d385b39825951ac13b1a12c22a482


ARROW-9922: [Rust] Add StructArray::TryFrom and deprecate StructBuilder (+40%)

(This PR removes a lot of code ^_^)
(This PR is built on top of #8117)
The core problem that this PR addresses is the construction of a StructArray, whose spec can be found here.
The current API to build a StructArray of 4 entries of fixed type is (part of a test):
let string_builder = StringBuilder::new(4);
let int_builder = Int32Builder::new(4);

let mut fields = Vec::new();
let mut field_builders = Vec::new();
fields.push(Field::new("f1", DataType::Utf8, false));
field_builders.push(Box::new(string_builder) as Box<ArrayBuilder>);
fields.push(Field::new("f2", DataType::Int32, false));
field_builders.push(Box::new(int_builder) as Box<ArrayBuilder>);

let mut builder = StructBuilder::new(fields, field_builders);
assert_eq!(2, builder.num_fields());

let string_builder = builder
    .field_builder::<StringBuilder>(0)
    .expect("builder at field 0 should be string builder");
string_builder.append_value("joe").unwrap();
string_builder.append_null().unwrap();
string_builder.append_null().unwrap();
string_builder.append_value("mark").unwrap();

let int_builder = builder
    .field_builder::<Int32Builder>(1)
    .expect("builder at field 1 should be int builder");
int_builder.append_value(1).unwrap();
int_builder.append_value(2).unwrap();
int_builder.append_null().unwrap();
int_builder.append_value(4).unwrap();

builder.append(true).unwrap();
builder.append(true).unwrap();
builder.append_null().unwrap();
builder.append(true).unwrap();

let arr = builder.finish();

This PR's proposal for the same array:
let strings: ArrayRef = Arc::new(StringArray::from(vec![
    Some("joe"),
    None,
    None,
    Some("mark"),
]));
let ints: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), Some(2), None, Some(4)]));

let arr = StructArray::try_from(vec![("f1", strings.clone()), ("f2", ints.clone())]).unwrap();

Note that:

There is no Field, only name: the attributes (type and nullability) are obtained from the ArrayData's itself, and thus there a guarantee that the field's attributes are aligned with the Data.
The implementation is dynamically typed: the type is obtained from Array::data_type, instead of having to match Field's datatype to each field' builders
Option is used to specify whether the quantity is null or not

The construction uses an OR on the entry's null bitmaps to decide whether the struct null bitmap is null at a given index. I.e. the third index of the example in the spec is obtained by checking if all fields are null at that index.
There is an edge case, that this constructor is unable to build (and the user needs to use the other From): a struct with a 0 at position X and all field's bitmap at position X to be 1:
# array of 1 entry:
bitmap struct = [0]
bitmap field1 = [1]
bitmap field2 = [1]

this is because, in this TryFrom, the bitmap of the struct is computed from a bitwise or of the field's entries.
IMO this is a non-issue because a null in the struct already implies an unspecified value on every field and thus that field's value is already assumed to be undefined. However, this is important to mention as a round-trip with this case will fail: in the example above, bitmap struct will have a 1.
Finally, this has a performance improvement of 40%.

Benchmark results
git checkout HEAD^ && cargo bench --bench array_from_vec -- struct_array_from_vec && git checkout no_builder1 && cargo bench --bench array_from_vec -- struct_array_from_vec

struct_array_from_vec 128
                        time:   [8.6816 us 8.8225 us 9.0221 us]
                        change: [-39.407% -38.457% -37.537%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

struct_array_from_vec 256
                        time:   [11.218 us 11.448 us 11.707 us]
                        change: [-44.091% -42.210% -40.540%] (p = 0.00 < 0.05)
                        Performance has improved.

struct_array_from_vec 512
                        time:   [16.402 us 17.445 us 18.638 us]
                        change: [-49.129% -46.775% -44.109%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe

struct_array_from_vec 1024
                        time:   [25.527 us 26.019 us 26.648 us]
                        change: [-46.410% -42.028% -37.256%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  5 (5.00%) high mild
  5 (5.00%) high severe

Final note:
The general direction that I am heading with this is to minimize the usage of builders. My issue with builders is that they are statically typed and perform incremental changes, but almost all our operations are dynamically typed and in bulk: batch read, batch write, etc. As such, it is often faster (and much simpler from UX's perspective) to create a Vec<Option<_>> and use it to create an Arrow Array.
FYI @nevi-me @andygrove @alamb

Maybe I am misunderstanding, but I think there may be a flaw with this approach and we're not comparing apples with apples when looking at the benchmarks.
The original code is dynamically building a struct using the builder. The new code starts with a vec! where everything is known at compile time. In theory, the builders should be more efficient than building a Vec and then converting it.

Maybe I am misunderstanding, but I think there may be a flaw with this approach and we're not comparing apples with apples when looking at the benchmarks.
The original code is dynamically building a struct using the builder. The new code starts with a vec! where everything is known at compile time. In theory, the builders should be more efficient than building a Vec and then converting it.

I though that criterion::black_box() would block the compiler from optimizing the code on it, so that the benchmark would not be tainted by compiler optimizations. I use these in both the Builder and From.
Regardless, the reason I used this approach was because I looked through the code on where we use Builders, and I found two main inputs:

a vector:

constructed from reading batches of rows (e.g. StringRecord in CSV, &[Value] in json)
constructed in memory from some external source (e.g. MemoryScan)

an Arrow Array, in most in-memory calculations (e.g. RecordBatch and ArrayRef, in compute and DataFusion)

In all cases, we use the builders to append rows row-by-row:

see here for CSV
see here for JSON
in parquet we do not use Array builders
see here for an example in compute

Based on this analysis, I though that:

this benchmark was a good representation of our use-cases
we can use [Try]From to build our results instead of a builder. The from is essentially builder.append_data().finish(), with a significantly simpler API

Sorry, I should have been clearer with my comments. I was referring to the code samples in the PR description. The first example is using the builders to push data into the contiguous buffers that will eventually become the arrays. The second example is building an intermediate data structure (the Vec) and then copying from the Vec into buffers. so there is 2x the memory usage and additional memory allocations and copies. Maybe I'm reading too much into these examples though.
Aside from that, I can't see how we can remove the StructBuilder though. The builder has semantics that I don't think we can cover by creating a struct from individual ArrayRefs representing fields. For example, how would we append a null struct versus appending a struct with null fields?

Aside from that, I can't see how we can remove the StructBuilder though. The builder has semantics that I don't think we can cover by creating a struct from individual ArrayRefs representing fields. For example, how would we append a null struct versus appending a struct with null fields?

I will roll back the builder and associated tests.
Just for my understanding, since I do not know the background: is it fair to say that the primary purpose of builders is to enable users to create Arrow Arrays without having to fiddle with Buffers, ArrayData? I.e. they are like helpers.

Yes, that's correct. These are the main high-level APIs for users to create
Arrow structures. The convenience methods that convert from Vec are great
for examples and tests but I would guess that they are not widely used for
code that has high performance requirements.
On Sun, Sep 6, 2020 at 3:34 PM Jorge Leitao<EMAIL_ADDRESS>wrote:

Aside from that, I can't see how we can remove the StructBuilder though.
The builder has semantics that I don't think we can cover by creating a
struct from individual ArrayRefs representing fields. For example, how
would we append a null struct versus appending a struct with null fields?
I will roll back the builder and associated tests.
Just for my understanding, since I do not know the background: is it fair
to say that the primary purpose of builders is to enable users to create
Arrow Arrays without having to fiddle with Buffers, ArrayData? I.e. they
are like helpers.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow/pull/8118#issuecomment-687907669, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHEBRD56FPE45RQ4ILMOQ3SEP56DANCNFSM4Q4DPXMQ
.

but I would guess that they are not widely used for code that has high performance requirements.

That is interesting. So, in terms of performance, is it fair to rank then as:

Buffers / ArrayData
From
Builder*

@nevi-me and @andygrove , I reverted the change wrt to the builder, so that this is an additive PR.
@andygrove, wrt to the dynamically building the array, note that a StructArray is almost only composed by child data: the struct itself is a null bitmap and some pointers. Therefore, the cost of building a Struct will always be driven by the allocation of those buffers.
With that said, you are right that during the creation of the fields, the benchmark clones the arrays, while a builder will build them on the fly and thus reduce memory footprint.
IMO that issue is separated from the creation of the struct itself (but related to the build of its childs): it is how we efficiently build non-struct arrays without first allocating vectors, that the builders aimed at solving. I am outlying some of this on #8211, which allows to build primitive Arrays from an iterator without exposing a unsafe API to users and would avoid the double allocation that you refer to.


Write timeout on InfluxDb influxdb-<IP_ADDRESS>-1

I was trying to do performance testing on influxdb-<IP_ADDRESS>-1 using wrk2 tool. I wrote a lua script and using below wrk command to send write request.
test.lua
=========
wrk.method = "POST"
wrk.body   = "cpu_load_short,host=server01,region=us-west value=0.64<PHONE_NUMBER>000000000"
wrk.headers["Content-Type"] = "application/x-www-form-urlencoded"

wrk command
============
./wrk -s /tmp/test.lua -t2 -c40 -d30s -R2000 --timeout 15 http://localhost:8086/write?db=demo_db

My observation is that I am receiving 500 error resp (write timeout) if I increase number of connections (-c). Default write timeout in config file is 5s. I have increased write timeout to 15s. Then I was able to survive for sometime, but if I increase number of connections then write time also getting increased on DB and getting more 500 error resp.
Any advice how to get rid of this error. I want to reach 2k to 3k req/sec.

#4308 is in the latest nightly and should address this issue.  The tsm1 engine is not enabled by default though so you need to enable it in your config. See: https://influxdb.com/docs/v0.9/introduction/tsm_installation.html


Get info of other vehicles collisions

Hi,
I was trying to use highway_env with random Obstacles object on the road. In this scenario some IDM Vehicles seems to be colliding with these obstacles. Is there any way I can get the count of these collisions in addition to the collisions of ego vehicle.
on looking through the code env.road.vehicles[0].crashed will give if the 1st IDM vehicle collided, I was trying to get the info for total collisions of IDM vehicles while training. I was looking for something like
other_vehicle_collisions =0  at the beginning and
other_vehicle_collisions = 10 (as 10 collisions happened in that episode)
Is it possible to obtain this?
I tried to find how the step action is being executed for this env so that I can increment the value I need at the end.

Hi!
It is not provided directly, but you can can compute the total number of collided vehicles with something like
´´´
sum(vehicle.crashed for vehicle in env.road.vehicles)
´´´
e.g. just after a call to env.step.

Thanku for the reply, this really helped
Using the above code takes into consideration of the ego-vehicle crash. looping after tweaking that list got me what I wanted.


Changed ball touches ground detection

This is a much improved ball on ground detection that brings it closer to the behaviour of rocket league
this allows for touch detection on top of the goal in throwback stadium, and for the ball stop on ground.

Work has been halted due to rocket league update,
Will resume after RLBot is working again


Updating demo page by merging master to gh-pages

The link to the demo in the file README.md leads to a github 404 page.


toco convert quantized mobilenet_v1_ppn failed

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Y
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Arch
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:None of them
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): v1.9.0-rc2-623-geb04124bb7' 1.9.0-rc0
Python version: 3.6
Bazel version (if compiling from source): 0.15
GCC/Compiler version (if compiling from source): gcc-7.1
CUDA/cuDNN version: cuda-9.2,cudnn-0.71
GPU model and memory: 16G
Exact command to reproduce:

I trained model by  models/research/object_detection/model_main.py with backbone samples/configs/ssd_mobilenet_v1_ppn_shared_box_predictor_300x300_coco14_sync.config.The difference from official is that I added quantization code in this config file like
graph_rewriter {   quantization {     delay: 48000     activation_bits: 8     weight_bits: 8   } }
Everything works fine on PC(both train and eval)  then I tried to export it to tf-lite.

export_tflite_ssd_graph.py seems ok for I got tflite_graph.pb and tflite_graph.pbtxt.
bazel run -c opt tensorflow/contrib/lite/toco:toco -- \ --input_file=$OUTPUT_DIR/tflite_graph.pb \ --output_file=$OUTPUT_DIR/detect.tflite \ --input_shapes=1,300,300,3 \ --input_arrays=normalized_input_image_tensor \ --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3'  \ --inference_type=QUANTIZED_UINT8 \ --mean_values=128 \ --std_values=128 \ --change_concat_input_ranges=false \ --allow_custom_ops
failed with "tensorflow/contrib/lite/toco/graph_transformations/propagate_fixed_sizes.cc:116] Check failed: dim_x == dim_y (512 vs. 24)Dimensions must match"

Any idea?

@MaeThird  The PPN model is in float and not quantized.
You will need to follow the instructions here for conversion:
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md

@achowdhery you mean it's not enough to add
graph_rewriter { quantization { delay: 48000 activation_bits: 8 weight_bits: 8 } } in config file and retrain the  PPN model ?

@MaeThird Thanks. The quantized model config is correct. The transformation step in toco needs debugging. We will note this and address it based on our priorities. Is there a specific reason you need PPN over SSD? It has similar accuracy and higher latency in float version we released.

Hi,@achowdhery. I tried to use quantized PPN on SSD,because I want the running time is as short as possible on my low frequency embedded arm board. Anyway,waiting for your good news.

@MaeThird SSD with Mobilenet head is fastest in our benchmarks on Android phones.

@achowdhery  By printf the shape,I found some clues.
when import tensorflow model the tensorflow::Status ConvertCOnvOperator function will ReorderAxes from AxesOrder::kHWIO to AxesOrder::kOHWI. In my case :
BoxPredictor/weights/read with shape: [ 1, 1, 512, 24 ] reorder to shape: [ 24, 1, 1, 512 ]
At this moment BoxPredictor/weights_quant/FakeQuantWithMinMaxVars  has no shape
After some passes  the BoxPredictor/weights_quant/FakeQuantWithMinMaxVars  has shape: [ 24, 1, 1, 512 ] This shape is calculated correctly but the appending reorder changed it to [512,24,1,1].
I mean the later-happened reorder ops  overrode  the previously calculated shape on BoxPredictor/weights_quant/FakeQuantWithMinMaxVars
I skipped the reorder on BoxPredictor/weights_quant/FakeQuantWithMinMaxVars  and strode out.

@MaeThird Thanks. Is there a specific bug ? Can you attach the frozen graph in pbtxt format?

@achowdhery
tflite_pb.tar.gz Thanks.

@MaeThird : Thanks

@achowdhery  Hi, I use mobilenetv1_ppn, the train and eval is right. I convert it to frozen_inference_graph.pb, it can work well on pc and the results are right. When i convert it to tflite_graph.pb, then convert it to detect.tflite. No warning and errors occur. But when it runs on the mobile, it shows totally wrong results. By the way, before this，I used mobilenetV2_ssd, all thing works fine. Could you please help me find it out?

I encountered the same problem while trying to convert MobileNet PPN (quantize-aware training) from .pb to .tflite. @achowdhery is there an update on this issue? @MaeThird how did you skip the reorder?


A "Design FAQ" document

 Issue by bstrie
Tuesday Nov 27, 2012 at 20:34 GMT
For earlier discussion, see https://github.com/rust-lang/rust/issues/4047
This issue was labelled with: A-docs, I-wishlist in the Rust repository

In the vein of the Project FAQ and the Language FAQ.
In addition to being really interesting, this would save a lot of time and divert a lot of potential questions in the long run. Some topics to discuss:

Why was typestate removed?
Why are some types nominal instead of structural?
Why do I have to declare impls in the same crate as a type or trait?
Why is trait inheritance the only means of inheritance?
Why doesn't Rust do tail-call optimization?
(see e.g. rust-dev post.)

Explanations should be technical and include code examples wherever possible.
Some of the questions of this nature overlap with the latter half of the Language FAQ, but not all, e.g. "Can Rust code call C code?" and "What is the difference between a managed pointer and an owned pointer?" do not belong. The document should focus on large-scale design decisions and tradeoffs; think "Why split stacks?" rather than "How do split stacks work?".
If you think of good questions, add them in the comments here. There are probably a lot of ripe areas for explanation, especially with regard to mutability and regions.

 Comment by brson
Tuesday Nov 27, 2012 at 21:24 GMT

The manual has a related (and empty) section: http://dl.rust-lang.org/doc/0.4/rust.html#appendix-rationales-and-design-tradeoffs

 Comment by bstrie
Wednesday Dec 05, 2012 at 15:56 GMT

Why are early returns disallowed within closures?

 Comment by bstrie
Thursday Nov 29, 2012 at 18:53 GMT

Why are &str, @str, ~str, &[], @[], and ~[] treated differently from &T, @T, and ~T?

 Comment by bstrie
Friday Dec 07, 2012 at 18:19 GMT

Why are separate do and for constructs required?

 Comment by bstrie
Tuesday Jan 01, 2013 at 18:07 GMT

Why must imports be qualified relative to the crate?

 Comment by graydon
Wednesday Jan 02, 2013 at 03:38 GMT

These are great, please keep them coming

 Comment by catamorphism
Wednesday Jan 02, 2013 at 21:59 GMT

"What does const mean (and why)?" should be in there too, though I know we're thinking about renaming it (at least).

 Comment by bstrie
Tuesday Dec 11, 2012 at 12:45 GMT

Why do traits in Rust depart from the design in the original traits paper?

 Comment by bstrie
Thursday Jan 17, 2013 at 13:47 GMT

Why feature conditions rather than exceptions?

 Comment by jruderman
Tuesday Jan 29, 2013 at 22:50 GMT

Why so many pointer types?  Why not rely on escape analysis instead?
Answered at:
http://www.reddit.com/r/programming/comments/17hqg4/a_little_bit_rusty_practical_approach_on_rust/c85r1nc

 Comment by bstrie
Thursday Jan 31, 2013 at 13:26 GMT

Why is the .times() method no longer defined on signed integers? (discussion on the specifics of integer literal inference)

 Comment by fread2281
Sunday Mar 24, 2013 at 21:40 GMT

Why are there no exceptions?

 Comment by bstrie
Tuesday Apr 02, 2013 at 19:12 GMT

Why is name resolution such an intractably hard problem?
<@pcwalton> I know of no other static language with declarative imports
            that allows (a) reexports; (b) glob import; (c) mutual
            recursion everywhere; (d) nested modules; (e) two namespaces
<@pcwalton> java doesn't have reexports, Go doesn't have nested modules
            or reexports, C# doesn't have reexports I don't think, C++
            doesn't have reexports or mutual recursion everywhere, maybe
            D...
<@pcwalton> ML and Haskell don't have mutual recursion everywhere and
            don't have reexports
<@pcwalton> mutually recursive modules too
<@pcwalton> most languages do not have mutually recursive modules
< bstrie> pcwalton: so what you're saying is that rust has once again
          failed to be unoriginal :)
<@pcwalton> well, as you said, Python
<@pcwalton> but Rust is trying to achieve Python's module system's
            expressive power in a static system
  jld tries to remember how the various Scheme module systems work
<@pcwalton> turns out that's pretty hard :)
<@pcwalton> jld: no mutual recursion
<@pcwalton> btw, ES6 hit the exact same problems we did
<@pcwalton> and had to pare down their module system as a result
<@pcwalton> anyway none of this is an excuse, we should make it easier to
            use
<@pcwalton> but I'm just saying it's not that easy...

 Comment by bstrie
Thursday Feb 21, 2013 at 15:06 GMT

Why do borrow checks on @mut pointers happen at runtime?

 Comment by iampiyush
Thursday Jun 13, 2013 at 13:57 GMT

document how to make non-utf8 byte vectors with bytes!(…)
e.g.
use of bytes!("hell", 0xc0, 0xca, "o!") instead of bytes!("hell\xc0\xca\o1!")

 Comment by brson
Tuesday Apr 30, 2013 at 22:04 GMT

Here's a summary about dynamically sized types https://mail.mozilla.org/pipermail/rust-dev/2013-April/003933.html

 Comment by catamorphism
Tuesday Apr 30, 2013 at 22:43 GMT

Summary of 'why no purity-in-the-type-system, i.e., why no effect system': https://mail.mozilla.org/pipermail/rust-dev/2013-April/003926.html

 Comment by brson
Tuesday Apr 30, 2013 at 22:39 GMT

Summary of 'why no exceptions' https://mail.mozilla.org/pipermail/rust-dev/2013-April/003815.html

 Comment by kud1ing
Monday Aug 05, 2013 at 10:06 GMT

Why have we mostly moved from internal iterators to external iterators?
https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364.html

 Comment by brson
Tuesday Jul 30, 2013 at 16:39 GMT

-> in function syntax: https://mail.mozilla.org/pipermail/rust-dev/2013-July/005042.html

 Comment by kud1ing
Thursday Aug 08, 2013 at 07:23 GMT

Why was the runtime ported from C++ to Rust?

 Comment by brson
Wednesday Dec 04, 2013 at 20:11 GMT

Why is rustc slow? https://mail.mozilla.org/pipermail/rust-dev/2013-November/006975.html

Import rate limiting made this fail :( re-opening the original


Fixing problems with remove_copy algorithm tests

This fixes the occasional test failures we were seeing for the remove_copy_if tests. This also applies similar fly-by changes to the remove_copy test.

LGTM

This causes problems in buildbot, reverting...

This causes problems in buildbot

Any details on which problems does this PR cause? Buildbot seems to shows the same distribution of green, yellow and red after (bcb6705e22b9e2ee9cbfee1cfb8cb57ca27b7940) and before (3a803894123f3c51869d75e40e066e55feba37eb) it.

Nevermind, after looking at buildbot for some time I can see those two have been tested together, that's why there's no apparent difference in breakage.

Nod, and most of the breakage we see is still caused by file system issues on Hermione :/


i  upadte system careless so i have to update my atomsphere but poker doesn't work

Are you sure you have sys-botbase installed properly?
Please check your atmosphere/contents folder.
If the "430000000000000B" folder is missing, Please reinstall your sys-botbase.
You can get a new one Here.

i am sure  that sys-botbase installe is it aobut atomphere >
or i should change deepsea


Pmag GUI tutorial missing criteria file

Running [Save MagIc tables] in Thellier GUI adds IE-SPEC (etc.) to the site and sample tables.  However, validations require that a value in sites.criteria should be a value in the criteria table.
If a user follows our Pmag GUI tutorial, they do not currently generate a criteria table.  Because of this, they will have a validation error.
Should we change the tutorial directions to include selecting [Analysis] => [Acceptance Criteria] => [Change Acceptance Criteria] and putting in some acceptance criteria?  This could be done in both Thellier GUI and Demag GUI.  Or, does that add unnecessary complexity?
@ltauxe?

Well, there is a line in the tutorial that implies you should set acceptance criteria:

You can select bounds for the statistics under Analysis =>Acceptance criteria => Change acceptance criteria.

I'm just going to word it slightly more strongly, and leave it at that.


Karavan Operator should deploy karavan-app to an AWS Kubernetes cluster

Karavan Operator should deploy karavan-app to an AWS Kubernetes cluster

Please assign this Issue to me.

Karavan operator is deprecated in favour of installer


⚠ Bofrost.de has degraded performance

In 40f35bc, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1843 ms

Resolved: Bofrost.de performance has improved in 046badf after 27 minutes.


Complaint assesment link broken

Describe the bug
We conducted an exam with artemis and resolved all complaints. Some of the complaints received "Feedback" with additional points (https://imgur.com/a/b3Kx1Gq).

Such complaint responses and feedback are visible when accessed via: "Exercise dashboard" page (complaint assessment dashboard)
In this case this button links to  ...programming-exercises/11120/submissions/6680161/assessment?correction-round=1
This case works as expected.

Later we wanted to access the complaint from another place: "Scores" pasge
In that case the button links to ...programming-exercises/11120/submissions/6680161/assessments/7065444
Behind that link, the our complaint response is still visible, but no feedback points are shown (even though they clearly exist when accessed via the other link). This behaviour is not what we expect, since it differs from (1).

To Reproduce
The following does not work

Go to exam
Go to exercise groups
Go to an exercise
Go to "scores" of that exercise
Find a score row that has a "Show the complaint" button
Klick that button and find that no manually given points are shown at the "Feedback" section.

Expected behavior
The following works:

Go to exam
Go to exercise groups
Go to an exercise
Go to "exercise dashboard" of that exercise
Klick the "Show the complaint" button and find that manually given points are shown at the "Feedback" section.

Screenshots
No response
Which version of Artemis are you seeing the problem on?
6.4.0
What browsers are you seeing the problem on?
Firefox
Additional context
No response
Relevant log output
No response

The difference between these two links is the ?correction-round=1 parameter, it is missing on the scores page. I assume the logic for computing the correct correction round was not transferred correctly when combining the submissions and scores pages.


amp: git rev 2c88e8 (2019-06-09) -> 0.6.1

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

could someone be a dear and merge this cc @qknight @adisbladis


Fix invalid buffer id in Neo-tree test

This fixes a bug discovered in a Neo-tree test:
Fail    ||      Filesystem reveal command should toggle the reveal-state of the floating window
            .../neo-tree/sources/filesystem/filesystem_command_spec.lua:9: Vim(lua):E5108: Error executing lua ...m/site/pack/packer/start/nui.nvim/lua/nui/utils/init.lua:
121: Invalid buffer id: 11
            stack traceback:
                [C]: in function 'nvim_buf_clear_namespace'
                ...m/site/pack/packer/start/nui.nvim/lua/nui/utils/init.lua:121: in function 'clear_namespace'
                ...m/site/pack/packer/start/nui.nvim/lua/nui/popup/init.lua:247: in function '_buf_destory'
                ...m/site/pack/packer/start/nui.nvim/lua/nui/popup/init.lua:272: in function 'unmount'
                ...eickel/repos/neo-tree.nvim//lua/neo-tree/ui/renderer.lua:141: in function 'close_floating_window'
                /home/cseickel/repos/neo-tree.nvim//lua/neo-tree.lua:82: in function 'float'
                [string ":lua"]:1: in main chunk
                [C]: in function 'cmd'
                .../neo-tree/sources/filesystem/filesystem_command_spec.lua:9: in function 'run_focus_command'
                .../neo-tree/sources/filesystem/filesystem_command_spec.lua:122: in function <.../neo-tree/sources/filesystem/filesystem_command_spec.lua:105>
                [C]: in function 'xpcall'
                ...
                ...te/pack/packer/start/plenary.nvim/lua/plenary/busted.lua:132: in function 'describe'
                .../neo-tree/sources/filesystem/filesystem_command_spec.lua:70: in function <.../neo-tree/sources/filesystem/filesystem_command_spec.lua:40>
                [C]: in function 'xpcall'
                ...te/pack/packer/start/plenary.nvim/lua/plenary/busted.lua:74: in function 'call_inner'
                ...te/pack/packer/start/plenary.nvim/lua/plenary/busted.lua:120: in function 'describe'
                .../neo-tree/sources/filesystem/filesystem_command_spec.lua:40: in main chunk
                [builtin#25]: at 0x5a0ceefabb70
                [C]: in function 'pcall'
                ...te/pack/packer/start/plenary.nvim/lua/plenary/busted.lua:221: in function 'run'
                [string ":lua"]:1: in main chunk

            stack traceback:
                .../neo-tree/sources/filesystem/filesystem_command_spec.lua:9: in function 'run_focus_command'
                .../neo-tree/sources/filesystem/filesystem_command_spec.lua:122: in function <.../neo-tree/sources/filesystem/filesystem_command_spec.lua:105>

I believe the issue is that I am trying to umnount a window that has already been closed. I think such an action should still work and clean up whatever might have been left behind, which this change will facilitate.
This closes https://github.com/nvim-neo-tree/neo-tree.nvim/issues/496

Thanks! 😃


Ensure text mode file BIOs are buffered on Windows

Commit 8300a8742b2abc487594a09b5e6ee726dbd30771 disabled file buffering for text mode file BIOs on Windows to work around a bug in UCRT ftell. This is surprising to applications that expect BIO read/writes to be buffered, such as when reading cert.pem.
This wraps text mode file BIOs with a buffered BIO, so the overall result is buffered (just once).

I do not think this is a good idea. This is going to surprisingly return a BIO chain instead of a single BIO on one platform and that means to free it you'd have to use BIO_free_all() instead of BIO_free(). This is an API change.

@t8m ah sorry, I thought BIO_free behaved like BIO_free_all.
Would it make sense to instead create macros to wrap BIO_new_file and BIO_free like:
#if defined(OPENSSL_SYS_WINDOWS)
#define BIO_NEW_TEXT_FILE(f,m) BIO_push(BIO_new(BIO_f_buffer()), BIO_new_file((f), (m))
#define BIO_FREE_TEXT_FILE(b) BIO_free_all(b)
#else
#define BIO_NEW_TEXT_FILE(f,m) BIO_new_file((f), (m))
#define BIO_FREE_TEXT_FILE(b) BIO_free(b)
#endif

And then update the places where openssl creates file BIOs in text mode?
$ git grep 'BIO_new_file(.*"r")' crypto providers
crypto/conf/conf_def.c:    in = BIO_new_file(name, "r");
crypto/conf/conf_def.c:                next = BIO_new_file(include_path, "r");
crypto/conf/conf_def.c:    next = BIO_new_file(include, "r");
crypto/conf/conf_def.c:            bio = BIO_new_file(newpath, "r");
crypto/conf/conf_lib.c:    in = BIO_new_file(file, "r");
crypto/engine/eng_openssl.c:    in = BIO_new_file(key_id, "r");
crypto/ts/ts_conf.c:    if ((cert = BIO_new_file(file, "r")) == NULL)
crypto/ts/ts_conf.c:    if ((certs = BIO_new_file(file, "r")) == NULL)
crypto/ts/ts_conf.c:    if ((key = BIO_new_file(file, "r")) == NULL)
crypto/x509/by_file.c:    in = BIO_new_file(file, "r");
crypto/x509/v3_pci.c:            BIO *b = BIO_new_file(valp, "r");

For example in by_file.c:
-   in = BIO_new_file(file, "r");
+   in = BIO_NEW_TEXT_FILE(file, "r");
    if (in == NULL) {
        ERR_raise(ERR_LIB_X509, ERR_R_BIO_LIB);
        return 0;
    }
    inf = PEM_X509_INFO_read_bio_ex(in, NULL, NULL, "", libctx, propq);
-   BIO_free(in);
+   BIO_FREE_TEXT_FILE(in);

I would like to know what problem is this trying to solve. Does it matter if reading certificates/keys is not buffered? These data files aren't large.

Sure, the problem is our application takes more than 10% longer to start up using openssl 3.0 due to 116k calls to ReadFile, each one of which reads 2 bytes from cert.pem. Details are in https://github.com/puppetlabs/puppet-runtime/issues/786

I might suggest that OpenSSL consider just open everything internally in binary mode. The PEM parser already handles CRLF itself and this helps avoid platform differences in processing the same bytes. The ftell bug is then moot in those cases.
That doesn't account for the cases where OpenSSL takes a FILE* from the caller. But in those cases, the caller presumably is just expecting OpenSSL to call the libc functions on that FILE, platform warts and all. I.e. I think you can just assume the caller either doesn't care about ftell (most likely) or has applied a suitable workaround on their end.
Put those together and I would suggest reverting the ftell workaround and leaving it to the call sites to apply, seeing as the workaround has significant perf costs.

I might suggest that OpenSSL consider just open those cases internally in binary mode. The PEM parser already handles CRLF itself and this helps avoid platform differences in processing the same bytes. The ftell bug is then moot in those cases.

Yeah, that would make sense.

This PR has the label 'hold: cla required' and is stale: it has not been updated in 30 days. Note that this PR may be automatically closed in the future if no CLA is provided.  For CLA help see https://www.openssl.org/policies/cla.html

This PR has the label 'hold: cla required' and is stale: it has not been updated in 61 days. Note that this PR may be automatically closed in the future if no CLA is provided.  For CLA help see https://www.openssl.org/policies/cla.html

@joshcooper are you able.to sign a cla on this so it can be reviewed and merged?

Thanks for the ping. I signed the CLA and wanted to close and reopen the PR but I don't seem to have permission for the latter. Since this PR needs work, I'll just submit a new one later.

Don't quite understand whats going on, but your change seems to have broken several windows tests

@joshcooper Also you're saying that you have the problem with openssl-3.0. Could you please try openssl-3.3? There are several other improvements that could help fixing the performance issue for you.

@t8m are you referring to the various mitigations/workarounds for the provider system? Those seem unrelated to this PR, which is about a performance regression in file I/O caused by 8300a8742b2abc487594a09b5e6ee726dbd30771.
Also see https://github.com/openssl/openssl/pull/24249#issuecomment-2077482502. I don't think making Windows BIOs even more weird is the right solution to a regression caused by making Windows BIOs weird and not actually return the same BIO type. Rather, OpenSSL should:

Open PEM files, etc., in binary mode, not text mode. The PEM parser, for instance, handles CRLF just fine.
Revert 8300a8742b2abc487594a09b5e6ee726dbd30771 as the cure is worse than the disease. In the very rare instances where the application wants ftell and text mode, the application can apply the workaround themselves. After all, that is what it would do if it were just using FILE directly. In the vast, vast majority of cases where the application isn't using ftell, it does not make sense to completely change the performance properties of a file BIO.

Indeed, this PR itself breaks BIO_C_FILE_TELL anyway! It causes file BIOs to secretly be buffer BIOs and buffer BIOs  do not internally adjust BIO_C_FILE_TELL by the buffer amount.

@davidben I am just saying some of the performance penalties that they can see with 3.0 could be gained back by performance improvements in other areas in the 3.3 version.
Otherwise yeah, I agree with your proposal to fix this particular regression.

@joshcooper Will you open a new PR implementing what @davidben suggests above?

@t8m sure, I will submit a PR with the changes proposed by @davidben Feel free to close this one.

as requested, closing in favor of a new PR


NA OFI: Bad address error with psm2 during dma

Describe the bug
We are facing issues with using ofi+psm2 with GekkoFS when running multiple servers and clients (and client processes), stochastically producing the following error on some of the client processes:
z0263.mogonii.39693Unexpected error in writev(): Bad address (errno=14) (fd=19,iovec=0x2ba199c370e0,len=3) (err=23)

Experimental setup
We have seen this behavior first in GekkoFS, but we could reproduce it in a smaller Margo testing program (which I am happy to share if required). For example, we use eight servers (one per node). The client is an MPI program that sends a number of RPCs including bulk transfers (64 MiB each RPC). However, this also happens with smaller transfer sizes, e.g., 1 MiB. Clients are run on a separate set of nodes than the servers. The client pseudo-randomly decides for each RPC which server is the recipient. The more client processes are running on each node the more likely the error occurs.
Detailed observations
The error originates from the opa-psm2 library and only during bulk_transfers (see source here) during a writev() to a TCP socket.
Regarding Mercury, this error first appeared with commit https://github.com/mercury-hpc/mercury/commit/5b1de1b7432bcde16bedd99e1fbd5c5561504d2f and is occurring since then.
Additional context

The error happens consistently throughout multiple opa-psm2 and libfabric versions. I tested with opa-psm2 (10.3.58) and libfabric (1.6.2) and with the more recent opa-psm2 (11.2.91) and libfabric (1.9.1-rc2).
Waiting a number of milliseconds between each RPC on the client to sequentialize the sending of RPCs of all ranks makes the error significantly less likely to occur.
The error does not occur with a single client.
The error occurs immediately when sending but after all servers have been looked up by each client.
The client is started with the following mpirun arguments: --mca mtl ^psm2,ofi -x FI_PSM2_DISCONNECT=1 -x PSM2_MULTI_EP=1. PSM2_MULTI_EP=1 is also set on the server.
Compiler version: GCC 6.4.0
MPI version: OpenMPI 3.0.1

Thanks!

@marcvef thanks for letting me know about it. I have also seen it today with DAOS actually over PSM2. Yes if you have a smaller margo example that reproduces it that would be very helpful, thanks!

@soumagne Thanks for the quick reply. Please find the testing program attached: rpctester.tar.gz.
I've included a readme which explains how to start the servers and clients and their arguments.

I had issues running your reproducer, C++ and then some bad regex stuff when parsing the hostfile. Anyway after running mercury tests more extensively, I think I have an idea of what the problem is. There are multiple data races coming from psm2, which is obviously not thread safe at all. So can you please try to do that on both client and server:
struct hg_init_info init_info = HG_INIT_INFO_INITIALIZER;

init_info.na_init_info.progress_mode = NA_NO_BLOCK;

and pass that to HG_Init_opt() or margo_init_opt(). Let me know if that helps.

Thanks a lot! I've tried it and the issue no longer occurs even with larger process numbers (I tested up to 512 processes with 32 processes per node). I have yet to run with very large node numbers. Could you please elaborate a bit on what NA_NO_BLOCK does and if they are any side effects when using this mode?
Also are there other environment variables or Mercury configurations that you suggest to set when using ofi+psm2?

I think PSM2 is understood not to be thread safe on it's own, but I was assuming that libfabric would serialize as needed.
See FI_PSM2_LOCK_LEVEL in https://ofiwg.github.io/libfabric/master/man/fi_psm2.7.html for example.  This makes it sounds like it should be using a conservative locking model by default, but I haven't looked at the code to see what this means exactly.
From skimming that man page I see that they also discourage FI_PROGRESS_AUTO (at least in part because it exacerbates the thread safety problem?).  Mercury is using FI_PROGRESS_AUTO though (see https://github.com/mercury-hpc/mercury/blob/master/src/na/na_ofi.c#L146).  Maybe we should try toggling that to FI_PROGRESS_MANUAL to see if it helps matters?

Right that is also what I was assuming and it is most likely attempting that level of locking but obviously it is not doing that and it needs more fixes, given the number of data races that the thread sanitizer produces...
Yes that's just always the same problem, signaling is not done correctly when FI_PROGRESS_MANUAL is used, which is why we had to set it to FI_PROGRESS_AUTO by default. FI_PROGRESS_MANUAL is used when NA_NO_BLOCK is used, which is why the problem disappears, but that forces internal busy spinning since we no longer have the ability to signal completion in that case. @marcvef setting that progress mode will consume more CPU resources as there is no more blocking progress and it will keep checking completion queues in a busy loop without sleeping.
Unfortunately since PSM2 seems to be only minimally supported, I really don't see that being fixed in the near future.

What exactly is the signalling problem when FI_PROGRESS_MANUAL is used with the psm2 provider?  (if nothing else, to document for posterity in this github issue).  I assume it must be something psm2-specific; we are using MANUAL mode with verbs in Mercury.

To give some feedback: We have been running longer tests with up to 64 nodes and 1024 processes in total for three hours. We didn't see this particular issue appearing again, albeit as you expected with a much higher CPU utilization.

Thanks for the update @marcvef .  @soumagne , what can/should we do in Mercury to accommodate?  Can we make this the default setting somehow if the psm2 provider is used?  That's too bad about CPU utilization but it might be the safest option for now.

@carns Yes you're right, I should definitely do that.


Proposal - Reverse submission work-flow

Instead of creating a bottleneck with moderators/approvers, what if when an asset was submitted it was automatically available.
At that point it could show as "community" supported, and then the testing support can be added once a moderator/approver tests out the asset.
You could possibly ping github to validate/verify the repo as required (some kind of spam protection?).

@akien-mga I'm thinking... what if we expose a "testing API", which would allow everyone to get "bleeding edge" versions by autoapplying all edits on demand (and disabling SHA protections). Wouldn't this solve issues such as this one and #94?

Possibly. I guess I was questioning do you really want to be in the business of having to review every single submission and edit. Is that maintainable if this really takes off?
The reverse could be allowing users to feel safe by filtering/downloading "curated" (moderated) assets, while still being able to download "uncurated" which havent been reviewed/tested yet.
Or allow the community to do this via ratings or "as a user, i verify this worked for me".

Well it's not about checking that assets work, it's about security. We don't want Godot to be a major security vulnerability, where malicious content could be installed on the machines of our users.
With the way the assetlib works (allowing you to download the contents of any public git repo), it's trivial to ship a malicious binary and have it executed automatically by Godot using os.execute().

I'm not passionate enough about this topic to argue one way or another but would ask the question, isn't it as possible to download an npm module that does the same? Does someone review all npm modules submitted? Any shared code can potentially harmful code.
Does unity review all source code of submitted user assets (maybe they do)?

I think it's possible to do a "caveat user", wash our hands a bit and do the reverse approach: banning malicious content. However, we need to make sure the submission process can't be too automated to avoid bots filling the asset lib with bad content. If account and asset creation are manual and we enable GitHub hooks to just auto-update the assets, I believe it is manageable. (I assume the other providers have a similar function).
At one point there might be too much stuff to manually moderate, making the process itself less reliable anyway.
Just my 2 cents.

I would suggest: let's continue with the manual-moderation approach but let the community have access to the listings through a REST API: { name, version, url }

Just used the reverse-submission work flow to manually moderate a few nasty sounding (curated) bugs that had been promoted to high status. That should buy me at least another 24 hour life-cycle. Thanks to all the coding contributors! You rock on!

I think your time would be better spent elsewhere, working on Godot, rather than looking at asset submissions - sure, ban reported malicious stuff, but GitHub doesn't screen public repos, neither does any package repository I know of. (pip? npm?)
Developers have to do due diligence at some point when including a third party dependency anway:
Is it working? Is it secure? Is it maintained? Is it maintainable?
That should not be your job. You can recommend (curate) assets, but you should not aim to be the gatekeeper of every available asset.
Now, if, at some point, assets are available for money on the Asset Library and you get a cut of the profit, sure, moderate away. :)

@mhilbrunner

Developers have to do due diligence when including a third party dependency anway...That should not be your job. You can recommend (curate) assets, but you should not aim to be the gatekeeper of every available asset.

Preach it man~ ;-)

What about automatically approving assets from users who already have at least one asset approved?

@Calinou At a bare minimum, I'd think that should work.

Developers have to do due diligence when including a third party dependency anway:

I think it's worth keeping in mind that--in the context of Godot--"developers" includes e.g. less technical/experienced people such as designers and kids/teens coding their first game and therefore less able to evaluate code.
Another factor is the stated desire to keep the Godot core minimal and have standard practice be to require some functionality to be installed separately.
In addition, with any sort of package hosting system IMO it's pretty much inevitable that a malicious/damaging package submission will appear or be attempted--whether that's due to mistake, purposefully or due to a security breach. And if it's an open source project there's probably an issue just like this one that gets pointed to as a result. :)
I agree that it would be nice if the core developers were freed from doing review work but would there be enough other community members to step up to form a review team?
If such an event is "inevitable" I think it's worthwhile to consider these questions in relation to the topic:

What process/precautions can be put in place to be prevent such an occurrence?
How will damage be reversed/fixed/prevented should such an event occur?
What will be the cost/impact of such an event on the project?


problem: error after wallet creation

Codecov Report

Merging #1254 (904906d) into master (04c4fc6) will decrease coverage by 46.59%.
The diff coverage is n/a.

:exclamation: Current head 904906d differs from pull request most recent head 224bb37. Consider uploading reports for the commit 224bb37 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories.  Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Remove syn dependency from macros

Excellent!

btw this pr introduced a little bug about dependency:
https://github.com/fzyzcjy/flutter_rust_bridge/runs/4697344168?check_suite_focus=true
https://github.com/fzyzcjy/flutter_rust_bridge/commit/3e20f269e3c210dea736924b11e9084c2666502e

@fzyzcjy @Desdaemon just curious, what's the reason to avoid syn? As I see it:

it's already a complete and well-tested implementation of Rust syntax parsing, which saves effort otherwise spent on building custom parsers
it stays up to date whenever Rust starts supporting new syntactical structures
it's well-documented and widely adopted by Rust codegen tools, so external contributors may be able to help more easily

Also curious about the rationale behind leaving codegen out of procedural macros:

flutter_rust_bridge can just be a dependency in Cargo.toml without requiring a binary installation
codegen is handled in cargo build with no extra steps required
great support for error reporting, i.e. you can have custom error messages that point to problematic spans of code the same way normal compilation errors from cargo are reported
no need for users to check in an intermediate bridge_generated.rs file into version control
still super easy to inspect generated code using cargo-expand

Having built a few procedural macros myself, this crate seems like it could benefit a lot from both, and I'd be happy to help support that effort.

@antonok-edm Hi thanks for the suggestions!
This pr avoids syn in the proc macro, since that proc macro does nothing except for used as a marker. So we do not want to waste time for the users.
As for the frb_codegen, it uses syn and does the heavy job.

flutter_rust_bridge can just be a dependency in Cargo.toml without requiring a binary installation

I guess you mean flutter_rust_bridge_codegen?
I agree, your suggestion seem also promising, and thanks for your interest in helping.
It generates not only .rs files but also .dart files which the dart/flutter code requires to compile. So maybe a manual triggering is needed?
@Desdaemon what do you think?

I think there are two primary advantages to using proc-macros: error reporting with span information, and keeping the bridge module in sync with the API file. Outside of these use-cases, this library is more focused in generating non-Rust files and therefore a codegen step is always required, not unlike wasm_bindgen (which already uses proc-macros.)
What is more interesting is whether we report errors from flutter_rust_bridge_codegen, or we let rustc emit compiler errors. Both choices have their pros and cons. What do you think @antonok-edm?

As for the frb_codegen, it uses syn and does the heavy job.

Ah, I see it now. What confused me is that many of the functions in frb_codegen/src/parser.rs operate on string representations of code rather than syn types.

I guess you mean flutter_rust_bridge_codegen?
It generates not only .rs files but also .dart files which the dart/flutter code requires to compile. So maybe a manual triggering is needed?

Yeah, Dart code and C headers can't be generated by a procedural macro but it should definitely be possible to do this with a build script. flutter_rust_bridge can expose a build dependency library crate for that, then output paths can be configured in a build.rs file. The build dependency library should be able to read over the source files and work with signatures from functions with the relevant procedural macro attribute.

What is more interesting is whether we report errors from flutter_rust_bridge_codegen, or we let rustc emit compiler errors. Both choices have their pros and cons.

IMO rustc has really awesome error reporting infrastructure already, so the more it's possible to lean on that, the better.

What confused me is that many of the functions in frb_codegen/src/parser.rs operate on string representations of code rather than syn types.

@antonok-edm I agree it is a bit hacky (you know, originally I do not know this repo will be so popular!). You are welcomed to use syn to actually parse it and that would be great and much more robust!

@antonok-edm The suggestion about not using string manipulation is now in #340. Feel free to create other issues/PRs!


Investigate limitations of elasticsearch indexing

@yurishkuro mentioned that the elasticsearch implementation doesn't consider clauses that match in different spans in the same trace (ex across multiple documents).
We should more formally capture what the problem is, ideally via a test case so that we can ensure the problem is fixable and also that it doesn't also affect other data stores.

Assuming following trace:

So the idea of matching different spans in the same trace would be for example to match this trace by specifying tags http.method=GET and http.path=/_template/zipkin-autocomplete_templateas these tags appear in different spans on this same trace. Indeed, searching for this returns nothing.
Paging @adriancole then if this should remain open then or not. It's a bit counterintuitive I guess to want to search this way, but i agree if you have uniquely identifying tags (message ids for example) and a large volume of traces and you want to search for an occurence where message X was handled in the beginning and later on message Y in the same trace, it would be surprising not to find it.
[
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "b9f8b74521f41269",
    "kind": "CLIENT",
    "name": "update-template",
    "timestamp":<PHONE_NUMBER>649000,
    "duration": 43093,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>650104,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>691875,
        "value": "wr"
      }
    ],
    "tags": {
      "http.method": "PUT",
      "http.path": "/_template/zipkin-dependency_template"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "bb917cdebeafa010",
    "kind": "CLIENT",
    "name": "update-template",
    "timestamp":<PHONE_NUMBER>698000,
    "duration": 48074,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>699142,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>745805,
        "value": "wr"
      }
    ],
    "tags": {
      "http.method": "PUT",
      "http.path": "/_template/zipkin-autocomplete_template"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "dd68f18ee5f7be31",
    "name": "get-service-names",
    "timestamp":<PHONE_NUMBER>770977,
    "duration": 47994,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "d70114e4ab3a3f4b",
    "kind": "CLIENT",
    "name": "get-template",
    "timestamp":<PHONE_NUMBER>553000,
    "duration": 12744,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>554000,
        "value": "connection-acquire.start"
      },
      {
        "timestamp":<PHONE_NUMBER>554000,
        "value": "dns-resolve.start"
      },
      {
        "timestamp":<PHONE_NUMBER>554946,
        "value": "dns-resolve.end"
      },
      {
        "timestamp":<PHONE_NUMBER>555000,
        "value": "socket-connect.start"
      },
      {
        "timestamp":<PHONE_NUMBER>556532,
        "value": "socket-connect.end"
      },
      {
        "timestamp":<PHONE_NUMBER>557037,
        "value": "connection-acquire.end"
      },
      {
        "timestamp":<PHONE_NUMBER>557966,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>565240,
        "value": "wr"
      }
    ],
    "tags": {
      "error": "404",
      "http.method": "GET",
      "http.path": "/_template/zipkin-span_template",
      "http.status_code": "404"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "b90fe9ce5ca30dfc",
    "kind": "CLIENT",
    "name": "get-template",
    "timestamp":<PHONE_NUMBER>644000,
    "duration": 4186,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>645210,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>647949,
        "value": "wr"
      }
    ],
    "tags": {
      "error": "404",
      "http.method": "GET",
      "http.path": "/_template/zipkin-dependency_template",
      "http.status_code": "404"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "dd68f18ee5f7be31",
    "id": "ffed503a9012c610",
    "kind": "CLIENT",
    "name": "aggregation",
    "timestamp":<PHONE_NUMBER>772000,
    "duration": 38713,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>773661,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>810454,
        "value": "wr"
      }
    ],
    "tags": {
      "http.method": "POST",
      "http.path": "/zipkin*span-2020-05-31,zipkin*span-2020-06-01/_search"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "bd00b7918128f861",
    "kind": "CLIENT",
    "name": "get-template",
    "timestamp":<PHONE_NUMBER>693000,
    "duration": 3856,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>694019,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>696650,
        "value": "wr"
      }
    ],
    "tags": {
      "error": "404",
      "http.method": "GET",
      "http.path": "/_template/zipkin-autocomplete_template",
      "http.status_code": "404"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "id": "9f7ef86c2a57e024",
    "kind": "SERVER",
    "name": "get /zipkin/api/v2/services",
    "timestamp":<PHONE_NUMBER>983000,
    "duration": 842669,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "ipv4": "<IP_ADDRESS>",
      "port": 52802
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>983021,
        "value": "wr"
      },
      {
        "timestamp":<PHONE_NUMBER>825041,
        "value": "ws"
      }
    ],
    "tags": {
      "http.method": "GET",
      "http.path": "/zipkin/api/v2/services"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "95c4f85b79d23deb",
    "kind": "CLIENT",
    "name": "get-node",
    "timestamp":<PHONE_NUMBER>071000,
    "duration": 405061,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>089000,
        "value": "connection-acquire.start"
      },
      {
        "timestamp":<PHONE_NUMBER>089000,
        "value": "dns-resolve.start"
      },
      {
        "timestamp":<PHONE_NUMBER>184037,
        "value": "dns-resolve.end"
      },
      {
        "timestamp":<PHONE_NUMBER>197000,
        "value": "socket-connect.start"
      },
      {
        "timestamp":<PHONE_NUMBER>426867,
        "value": "socket-connect.end"
      },
      {
        "timestamp":<PHONE_NUMBER>426968,
        "value": "connection-acquire.end"
      },
      {
        "timestamp":<PHONE_NUMBER>431561,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>455885,
        "value": "wr"
      }
    ],
    "tags": {
      "http.method": "GET",
      "http.path": "/"
    }
  },
  {
    "traceId": "9f7ef86c2a57e024",
    "parentId": "9f7ef86c2a57e024",
    "id": "e4649fc8ba8ef782",
    "kind": "CLIENT",
    "name": "update-template",
    "timestamp":<PHONE_NUMBER>573000,
    "duration": 65066,
    "localEndpoint": {
      "serviceName": "zipkin-server"
    },
    "remoteEndpoint": {
      "serviceName": "elasticsearch",
      "ipv4": "<IP_ADDRESS>",
      "port": 9200
    },
    "annotations": [
      {
        "timestamp":<PHONE_NUMBER>574279,
        "value": "ws"
      },
      {
        "timestamp":<PHONE_NUMBER>637793,
        "value": "wr"
      }
    ],
    "tags": {
      "http.method": "PUT",
      "http.path": "/_template/zipkin-span_template"
    }
  }
]

I'm closing this mainly because it is very old and we would have heard from other people if it was a priority. Increasingly we are encouraging "local root" tags which allow more tags coupled but without inflating spans so much. we can revisit if the topic comes up but right now let's close.


Wind Instrument "Laylines" label incorrect and port/starboard colors inverted

Fixes #261

Thanks for the quick fix!
The lines are still based on True wind, not Apparent. Shall I raise a separate issue?

From you reference:

A given point of sail (beating, close reach, beam reach, broad reach, and running downwind) is defined in reference to the true wind—the wind felt by a stationary observer.


Handle malformed CosmosDB key

Harmless proposal to handle this edge case: https://github.com/a8m/documentdb-go/issues/4

Thanks @snobu


docker treats example.com:443 and example.com as different registries

Docker has always treated example.com/user/repo and example.com:443/user/repo as separate repos. This has been a source of errors and confusion with DTR users. This is a problem with Notary because Notary signs the repo name including the domain. There is no way to transform signed Notary data from one format to the other. This means that if you sign an image with the wrong one, you have to forever make sure no one switches their client to use the other format.
This issue also comes up with docker login because the credentials for these two registries are treated separately.
I propose that we standardize on which one of these is the correct name. I don't care which one it is, it just needs to be consistent.
One way to do this is to internally treat example.com/user/repo as example.com:443/user/repo. Let me know if you guys have other ideas for how to resolve this.

My guess is that the thought behind this is that multiple registries can run on the same domain, but using different ports (example.com:5000, example.com:5001).

Yeah, and that's okay. The issue is that port 443 is special because you can have two different tags in docker that are actually the same registry, namespace, repo, and tag, just with or without port 443 being explicit.

But without a protocol, would it be example.com:80 or example.com:443?

Hmm... That's a good question. I guess it's both right now. That's very confusing. In DTR's case we never allow non-HTTPS connections so it's always 443, but in general --insecure-registry falls back to both. This is just really deeply baked into the design of docker image names and it'll continue to cause problems in unexpected places :( Ideally, I don't think that domain name should be part of the image name at all, but that's a different discussion. I don't know what we can do here.

ping @stevvooe @dmcgowan any suggestions?

We are working on a proposal for always sending up the hostname component of the repository name to the registry. It's design is primarily for proxies but also discussing other uses for it on the backend. From a naming perspective they are different images and will remain different images, and I agree it would be ideal for the registry to recognize them as such. For login, I don't think it would be unreasonable to treat the two names the same, this would only effect how to the client determines the auth config.
When I have a proposal created I will link it here.

The simple fact here is that example.com and example.com:443 are different hosts. The former will switch ports based on protocol (http or https, or other future additions) while the latter will always use port 443, even when http or other protocol is specified.
This can be confirmed by running a plaintext server on port 443. The following works, because we are port bound:
$ curl -v http://localhost:443

Even though we are running on 443, because the host is port bound, this works because the scheme http will be used. With the same plaintext server, the following doesn't work:
$ curl -v https://localhost/
*   Trying ::1...
* Connected to localhost (::1) port 443 (#0)
* Unknown SSL protocol error in connection to localhost:-9847
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to localhost:-9847

Even though we are running on port 443, since the server is plaintext and the scheme is https, TLS will be used. The plaintext server obviously fails here. Another way of putting this is that the scheme is what selects TLS, not the port.
Any attempt to canonicalize these names would end up binding the naming too tightly to http/https. I also suspect that specifying the port as 443 may open a client up to potential security problems, since the assumption is that this port is secure, when it is clear that if the host is port bound, a plaintext connection can be made over this port. Departing from the conventions that these are different hosts in http and tls may open us up to a number of other issues.
The correct solution here is to remove the impetus to include the port in image names by decoupling the registry location from the name of the image. The best practice is to use example.com over example.com:443.

I agree completely. The port should never have been a part of the image name. I think the sanest way to manage the locations of registries is to follow the same model as git remotes. This would be fundamental change to how custom registries are handled, but I think it's necessary to reduce confusion and all the obscure bugs that stem from the image naming convention. Is there any existing work in this direction?

I have one registry but it exposes different ports, like this registry:8000, registry:8001, registry:8002 and so on. I do this to keep multiple credentials of the registry on  the docker client auth file (otherwise it keeps only on user credentials). my --insecure-registry flag is "registry/0"
I have this working for long time. I donot see any issues with that.

@Vish0007 Why are you running the registry on several different ports?
In general, it's best to access a registry through a single host name.

i want to store different more than one user credential on docker client side, so i work around it with several ports for same registry.
Btw, why its best to access registry with only one hostname (:port)? Are there any synchronization issues when we use different ports to access the registry? for example, executing docker push in parallel from two different docker clients: docker push registry:8000 and docker push 8001.


Rejuvenate command/verb should reset cooldowns #20261 (WIP)

About the PR

TODO
This PR is meant to fix #20261. At time of writing it's merely a draft, still scavenging for details. At the current moment, the changes in this PR only contain a suspected illustration of what the change might look like, and which functions may need changing.
Why / Balance

Refer to #20261
Technical details

TODO
Media

[TODO ] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

This PR, when completed, should not cause breaking changes.
Changelog

:cl:

add: Rejuvenate now resets cooldows

id imagine using TimeSpan.Zero for start and IGameTiming.CurTime for end would work

Current Status: @ElectroJr has added clarifying comments to the corresponding issue ticket. This PR just needs to implement the change in accordance with the commentary. I'm busy so I can't work on this right now. Anyone else is welcome to add the change and merge the PR. Otherwise I will come back to this when I have time.

closing due to https://github.com/space-wizards/space-station-14/pull/21057


Items are included in the estimation list which are not "Requirement" type items (Tasks, etc.) Makes a mess

When you start from an iteration or from a query?


Homebrew distribution

🐞 Whats wrong
The installation process is too complicated.
🌈 Desired state
brew install mobile-toolkit
📏 Challenges
I have not done anything like this yet.
📂 Sources
https://kylebebak.github.io/post/distribute-program-via-homebrew
www.google.com

brew create https://github.com/IntergalacticPenguin/mobile-toolkit/archive/1.1.0.tar.gz

class MobileToolkit < Formula
  desc "📱 Shell scripts for Android and iOS device management"
  homepage "https://github.com/IntergalacticPenguin/mobile-toolkit"
  url "https://github.com/IntergalacticPenguin/mobile-toolkit/archive/1.1.0.tar.gz"
  sha256 "e8d48513e3fb393072eeed2b850146b9ebbf9ebbd2c3f5d4c069a4c4645f2eaa"
  license "MIT"

  def install
    bin.install "ios/icheckdevice"
    bin.install "ios/icrashlogs"
    bin.install "ios/igif"
    bin.install "ios/iinstall"
    bin.install "ios/ilog"
    bin.install "ios/ireboot"
    bin.install "ios/irecord"
    bin.install "ios/iscreenshot"
    bin.install "ios/isimulator"
    bin.install "ios/iuninstall"

    bin.install "android/aanimationspeed"
    bin.install "android/aappinfo"
    bin.install "android/abounds"
    bin.install "android/abuildproject"
    bin.install "android/acamera"
    bin.install "android/acheckdevice"
    bin.install "android/acontrol"
    bin.install "android/aemulator"
    bin.install "android/aerase"
    bin.install "android/afontscale"
    bin.install "android/agoogleplay"
    bin.install "android/ainstall"
    bin.install "android/akill"
    bin.install "android/alaunch"
    bin.install "android/alog"
    bin.install "android/aoptions"
    bin.install "android/apaste"
    bin.install "android/apermissionreset"
    bin.install "android/areboot"
    bin.install "android/arecord"
    bin.install "android/ascreenshot"
    bin.install "android/aservices"
    bin.install "android/auninstall"
    bin.install "android/aurl"
    bin.install "android/awipe"
    bin.install "android/awireless"
  end
end

Optionally, you can add depends_on part (for example for libimobiledevice)
brew audit --new mobile-toolkit

Solve all issues found via audit (for example, add test do part)
brew install mobile-toolkit

⚠ But there is a problem with relative path to common_tools. The best option is to create distribute file or put all files into same folder (but prefer dist file).
Then just submit a pull request to https://github.com/Homebrew/homebrew-core 🎉

👀  https://neurobin.org/projects/softwares/unix/shc/


Choosing hour of irrigation

Hi,
I read some scientist documentation recommending to irrigate at evening in place of morning.

What do you think?
How to choose that in the Smart Irrigation plugin?

Thank you !

The choice is yours as there are multiple opinions on this. The way to achieve it would be to give a negative number in sunriseoffset: so a -60 in the setting will result in irrigation being concluded by 1 hour post sunrise. Calculate for your evening in terms of the number of minutes post sunrise.

Thank you.
Is it possible to define a timing related to sunset in place of sunrise?

Sorry.. that's not possible at the moment or planned 🙂


For recursive components, make sure to provide the "name" option.

[Vue warn]: Unknown custom element: <trix-vue> - did you register the component correctly? For recursive components, make sure to provide the "name" option.

Hello, I am getting this warning.
I did this
npm install --save trix-vue2
...
import { TrixVue } from 'trix-vue2'
...
components: { TrixVue },
...
<trix-vue></trix-vue>

@Matoo125 maybe you can try this ?
npm install --save trix-vue2
...
import { TrixVue } from 'trix-vue2'
...
components: {
  trixVue: TrixVue
}
...
<trix-vue></trix-vue>

no, that did not helped

@Matoo125 The module seems to be bundled incorrectly. If you are using webpack, try the following:
Add a new resolve alias to the webpack config:
//...
  resolve: {
    alias: {
      'trix-vue2': path.resolve(path.join(__dirname, 'node_modules', 'trix-vue2', 'src', 'index.js'))
    }
  }
// ...

This will load the source file instead of the dist version for any import from 'trix-vue2'.
If you don't use .vue files, you'll need two additional loaders for webpack: css-loader and vue-loader. npm i --save-dev them and add the following to the webpack config:
//...
  module:  {
    loaders: [
      {
        test:   /\.vue$/,
        loader: 'vue-loader'
      },
      {
        test:   /\.css$/,
        loader: 'css-loader'
      }
    ]
  }
// ...

You can include the component as intended now.


Define different function for concrete playback (no user impact)

Description of changes:
Split the definitions of functions that have specific logic for concrete playback to avoid generating code for unused logic.
Background: While upgrading the toolchain to nightly-2023-04-16, I noticed a few new warnings regarding caller location and the size of the model was much bigger in some case. The MIR file was going from a couple hundreds of lines to 20k+ lines.
It looks like the if cfg!() is no longer being propagated and the concrete playback code is increasing the logic that the compiler detects as reachable.
Resolved issues:
Related to #2383 and PR #2406
Related RFC:

Optional #ISSUE-NUMBER.
Call-outs:
We might want to split the code like we've done with the macro one to make it a bit easier to maintain but for now, I'm jut unblocking the toolchain update.
Testing:

How is this change tested? Current tests

Is this a refactor change? Yes

Checklist

[ ] Each commit message has a non-empty body, explaining why the change was made
[ ] Methods or procedures are documented
[ ] Regression or unit tests are included, or existing tests cover the modified code
[ ] My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Notes on the alleged performance regression: The resulting GOTO binaries are (as one should expect from the changes in this PR) the same, with one exception: the numbers in Kani-generated s2n_quic_core:::alloc[[:digit:]]+::init differ, which in turn means that their invocation order from __CPROVER_initialize will differ. This, in turn, results in SAT formulae that are equisatisfiable but not exactly the same (they will have the same number of variables and clauses, but there is a renaming between the variables). Such differences mean that the SAT solver will happen to make different decisions. Take a look at the following data (using CaDiCaL to further reduce possible noise from the system):

model-checking:main
celinval:issue-2383-playback

Runtime Symex: 6.64313s
Runtime Symex: 6.41786s

size of program expression: 177676 steps
size of program expression: 177676 steps

slicing removed 101675 assignments
slicing removed 101675 assignments

Generated 10049 VCC(s), 7355 remaining after simplification
Generated 10049 VCC(s), 7355 remaining after simplification

Runtime Postprocess Equation: 0.24202s
Runtime Postprocess Equation: 0.224713s

Passing problem to propositional reduction
Passing problem to propositional reduction

converting SSA
converting SSA

Runtime Convert SSA: 2.57943s
Runtime Convert SSA: 2.55088s

Running propositional reduction
Running propositional reduction

Post-processing
Post-processing

Runtime Post-process: 7.78618s
Runtime Post-process: 7.82671s

Solving with CaDiCaL sc2021
Solving with CaDiCaL sc2021

1846920 variables, 24318135 clauses
1846920 variables, 24318135 clauses

SAT checker: instance is SATISFIABLE
SAT checker: instance is SATISFIABLE

Runtime Solver: 8.27705s
Runtime Solver: 8.33516s

Runtime decision procedure: 10.8814s
Runtime decision procedure: 10.9116s

Running propositional reduction
Running propositional reduction

Solving with CaDiCaL sc2021
Solving with CaDiCaL sc2021

1846920 variables, 24318136 clauses
1846920 variables, 24318136 clauses

SAT checker: instance is SATISFIABLE
SAT checker: instance is SATISFIABLE

Runtime Solver: 3.83114s
Runtime Solver: 3.94184s

Runtime decision procedure: 3.85865s
Runtime decision procedure: 3.97045s

Running propositional reduction
Running propositional reduction

Solving with CaDiCaL sc2021
Solving with CaDiCaL sc2021

1846920 variables, 24318137 clauses
1846920 variables, 24318137 clauses

SAT checker: instance is SATISFIABLE
SAT checker: instance is SATISFIABLE

Runtime Solver: 53.2559s
Runtime Solver: 8.55057s

Runtime decision procedure: 53.3403s
Runtime decision procedure: 8.6407s

Running propositional reduction
Running propositional reduction

Solving with CaDiCaL sc2021
Solving with CaDiCaL sc2021

1846920 variables, 24318138 clauses
1846920 variables, 24318138 clauses

SAT checker: instance is SATISFIABLE
SAT checker: instance is SATISFIABLE

Runtime Solver: 3.421s
Runtime Solver: 62.0962s

Runtime decision procedure: 3.43368s
Runtime decision procedure: 62.1679s

Running propositional reduction
Running propositional reduction

Solving with CaDiCaL sc2021
Solving with CaDiCaL sc2021

1846920 variables, 24318139 clauses
1846920 variables, 24318139 clauses

SAT checker: instance is UNSATISFIABLE
SAT checker: instance is UNSATISFIABLE

Runtime Solver: 0.281313s
Runtime Solver: 0.17613s

Runtime decision procedure: 0.29297s
Runtime decision procedure: 0.179646s

I have highlighted the (only significant) differences. So unless we find a way to make Kani's choice of names deterministic there might not be much we can do to also have deterministic SAT solver performance.

Thanks @tautschnig for your thorough analysis. That's exactly what I was afraid of. Not sure what's the best course of action. Any suggestions?

Thanks @tautschnig for your thorough analysis. That's exactly what I was afraid of. Not sure what's the best course of action. Any suggestions?

As per the discussion that happened out of band in the meantime: let's keep collecting data for the time being, no obvious reason that this specific PR ought to be blocked.