Add boldfaced "Key" title in the table footer when %key is used

According to ISO's guidelines given in https://github.com/metanorma/mn-samples-iso/issues/96#issuecomment-2067627510 regarding the key element in the table footer, the key section should be preceded by a “Key” title in bold, as shown in the example below.

Currently, Metanorma generates the key section as below

when the following markup is used.
NOTE: Ce tableau est basé sur l'<<ISO7301,table=1>>.

NOTE: Certains contrats commerciaux peuvent exiger des complémentaires à celles fournies dans le présent tableau.

NOTE: Seul le riz (cargo) rouge entièrement décortiqué est pris en considération dans le présent tableau.

[%key]
N/A:: N'est pas applicable.

To move table key around properly in Word postprocessing, which is not currently happening, need to wrap it in div[@class = 'figdl'], treating  it the same as figure keys are already being treated.

I'm marking up this key title as the title of a definition list, which is ListTitle; but ListTitle is centered, and this Key is not. Ugh. Will override in postprocessing.

Need to keep definition list, preceding title paragraph and following notes together in a div in rendering in HTML and DOC output, since they are going to be treated as a unit.

Will finish tomorrow


Publish version 6.5.0 on pypi

Looking at release history of ivy on Pypi version 6.4.1 is shown as the latest version. Yet this repository's tags list includes 6.5.0

Thanks, updated.

It seems that "ivy" has been renamed to "ark" and there are separate projects present on pypi for each (see ivy and ark). IMHO the relation, i.e. ark supersedes ivy, could be made more explicit on pypi and also in this project's README, helping users to find their way to the latest version of this project.
FWIW: The pypi release history still does not show the 6.5.0 tag; maybe the 6.5.0 refers to the first ark release?
Hope this helps 🙂


Provide correct len to getsockopt(TCP_INFO).

The alignment of grpc_core::tcp_info is 8-bytes and socketlen_t is
4-bytes. So, there is one 4-byte padding at the end of tcp_info:
grpc_core::tcp_info {
  ...
  uint64_t tcpi_pacing_rate;
  ...
  socklen_t len;
  // There is a 4-byte hole here.
}

So, the length we calculate here is actually including len (we are
giving kernel 4 bytes more than we should):
info->length = sizeof(*info) - sizeof(socklen_t);
Kernel copies the length first, then copies the content. Hence it
overwrites the field by tcpi_rcv_ooopack. In cases, where the
tcpi_rcv_ooopack is less than the size of grpc_core::tcp_info we
won't get an asan error but otherwise we will correctly get an
asan complaint.
AFAICT, there is no real bufferoverflow though.

@drfloob

Thanks! Enabled auto-merge.


Fix signal handler

The variable “caught” should use the data type “sig_atomic_t”.

This code is somehow naive anyway. For instance, if you send the second signal immediately after the first one, that first signal can be ignored due to races.
I'm not sure it is worth to be fixed since that is used for tests only.

On my machine, the problem can be reproduced like this. First, create a script to be run:
#!/usr/bin/env perl

use strict;
use warnings;

$SIG{INT} = sub { print "INT\n" };
$SIG{TERM} = sub { print "TERM\n" };

sleep 1 while 1;

Then run it with timedrun:
./timedrun 100 ./script.pl

Now send two different signals
perl -e '$x = shift; kill("TERM", $x) && kill("INT", $x)' 12345

If timedrun is not nice enough, it works fine:
$ ./timedrun 100 ./script.pl
TERM
TERM

But we can get a different result by simply lowing priority:
$ nice -n 10 ./timedrun 100 ./script.pl
INT

Yeah the report is pedantic; this project gets a lot of static analyzer noise. I've marked this as "help wanted" since there's no real harm in fixing it, but anyone can do it :)

The field settings.sig_hup should also be similarly using sig_atomic_t.

please open new issues/pr's.


Mutacao-New-Sheet

Submission Checklist
Required

[X] The pull request title clearly contains the name of the sheet I am editing.
[X] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[ ] The pull request makes changes to files in only one sub-folder.
[X] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

New Sheet Details

The name of this game is: <  Mutacao >  (i.e. Dungeons & Dragons 5th Edition, The Dresden Files RPG)

The publisher of this game is: <  Daemon Editora > (i.e. Wizards of the Coast, Evil Hat)

The name of this game system/family is: <  Daemon > (i.e. Dungeons & Dragons, FATE)

[X] I have followed the Character Sheets Standards when building this sheet.

[ ] This sheet has been made on behalf of, or by, the game's publisher.
[ ] This game is not a traditionally published game, but a copy of the game rules can be purchased/downloaded/found at: <   >
[X] This sheet is for an unofficial fan game, modification to an existing game, or a homebrew system.

Changes / Description

Character Sheet Info Roll20 Internal Use only.


[release-1.23] waypoint: Respect port level DR settings

Manual cherrypick required.
#52747 failed to apply on top of branch "release-1.23":
Applying: Respect port level DR settings
Applying: tmp: fix auto alloc for waypoint
Using index info to reconstruct a base tree...
M	pilot/pkg/model/service.go
Falling back to patching base and 3-way merge...
Auto-merging pilot/pkg/model/service.go
CONFLICT (content): Merge conflict in pilot/pkg/model/service.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 tmp: fix auto alloc for waypoint
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

@howardjohn I hope this can be merged to release-1.23 soon. I very much appreciate your work.

@laiminhtrung1997 fix has been merged in #52818 and will be included in the next 1.23.x patch release


Task/cdd 2220 update alt text for tend line none

Description
This PR includes an update to plots_text for when the simplified chart colour is TREND_LINE_NONE
Fixes #CDD-2220

Type of change
Please select the options that are relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Tech debt item (this is focused solely on addressing any relevant technical debt)

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] I have added tests at the right levels to prove my change is effective
[ ] I have added screenshots or screen grabs where appropriate
[ ] I have added docstrings in the correct style (google)


Mux viewer count

This is a new element that shows the viewer count for the specified video.

shame we didn't merge this before. the shared-polyfills pkg is removed.
it should be a relative ../polyfills import now with a NPM script for copying it over

LGTM!


[BUG] Missing request body for delete_put, cat.pit_segments, action group, etc.

What is the bug?
Request body description missing for "delete_pit",
Request body description missing for "cat.pit_segments",
and also several other security client APIs have body and path param descriptions missing
"security.change_password", "create_action_group", "patch_action_group"("action_group"path param description is also missing),  "patch_action_groups", ....

I opened a meta issue for descriptions, there's lots more.
https://github.com/opensearch-project/opensearch-api-specification/issues/403


how to use carbon clickhouse with distributed tables ?

how to configure carbon-clikhouse with clickhouse distributed tables ?

I am not sure what you mean. I've used distributed tables for inserts there

you use partitionned tables, not distributed table in the readme.
clickhouse distributed table:
https://clickhouse.com/docs/en/sql-reference/distributed-ddl

with distributed table you can distribute data again clickhouse node shard.... and scale linearie with the number of node (depend on the eficiency of the distribution key)....

You use a single table, not "on cluster" clause.
See https://clickhouse.com/docs/en/engines/table-engines/special/distributed/

is the documentation what i'm searching !
the idea is to store not in single node but in cluster with multiple shard.... to scale...
the readme create table instruction, are for single node... or i've missed somephing ...

You should create regular tables on each node in cluster.
After that you can write into any of nodes.
For reading from all nodes in one request, use Distributed table.
When creating Distributed, you can set sharding_key. That allows you to write "to distribution table" -- this means that all incoming data will be routed by sharding_key.
Note here:
When you use rollup-conf = "auto" in graphite-clickhouse, you should set rollup-auto-table = "" pointed to regular table.
Here is examples of configs from my prod:
Tables:
CREATE TABLE IF NOT EXISTS graphite_repl ON CLUSTER datalayer (
    `Path`      String  CODEC(ZSTD(3)),
    `Value`     Float64 CODEC(Gorilla, LZ4),
    `Time`      UInt32  CODEC(DoubleDelta, LZ4),
    `Date`      Date    CODEC(DoubleDelta, LZ4),
    `Timestamp` UInt32  CODEC(DoubleDelta, LZ4)
)
ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/{shard}/graphite_repl', '{replica}', 'graphite_rollup')
PARTITION BY toYYYYMMDD(Date)
ORDER BY (Path, Time)
TTL
    Date + INTERVAL 1 WEEK TO VOLUME 'cold_volume',
    Date + INTERVAL 4 MONTH DELETE
SETTINGS
    index_granularity = 512;

CREATE TABLE IF NOT EXISTS graphite_dist ON CLUSTER datalayer AS graphite_repl
ENGINE = Distributed(datalayer, ..., graphite_repl);

carbon-clickhouse:
...
[upload.graphite]
type = "points"
table = "graphite_repl"
...

graphite-clickhouse:
...
[[data-table]]
 table = "graphite_dist"
 rollup-conf = "auto"
 rollup-auto-table = "graphite_repl"
...

i while go to test that !!

can be usefull to use chproxy in front to cache request (https://www.chproxy.org/) ?

If you use carbonapi - it also can cache requests. So that depends on what is your use case.
I would overall suggest to start with simple setup and then add extra pieces once you encounter a bottleneck

can be usefull to use chproxy in front to cache request (https://www.chproxy.org/) ?

No. chproxy can't cache requests with external data (used in points table queries).
Graphite-clickhouse can cache finder queries (in render requests).
Carbonapi can cache other (render, find, tags autocomplete).
So, no reason use chproxy for caching. But usefull as bouncer/connection pool limiter.


Getting this error Peer certificate cannot be authenticated with given CA certificate

Hi , I am trying to see how Service Discovery works in PCF Dev. I logged in to pcf dev using cf login -a https://api.local.pcfdev.io --skip-ssl-validation and published both my client and service applications. when i call the client url , i am getting HTTP 500 error, when i saw the logs it says -
2017-04-28T17:26:01.50-0500 [APP/PROC/WEB/0] OUT fail: Steeltoe.Discovery.Eureka.Transport.EurekaHttpClient[0]
2017-04-28T17:26:01.50-0500 [APP/PROC/WEB/0] OUT DoGetApplicationsAsync Exception: System.Net.Http.HttpRequestExce
ption: An error occurred while sending the request. ---> System.Net.Http.CurlException: Peer certificate cannot be authe
nticated with given CA certificates
please help me on this error:
As I was reading through : http://steeltoe.io/docs/steeltoe-discovery/#1-2-2-1-settings-to-discover
this they are saying i may run into ssl certificate issues, is this error related with this ??

Yes, you  will need to disable certificate validation when running on PCF Dev.

I did it using this  --skip-ssl-validation while logging in to pcf dev. Still my client is unable to discover the registered service.
Here is my class :+1:
public class ServiceDiscoveryService : IServiceDiscoveryService
{
DiscoveryHttpClientHandler _handler;
private ILogger _logger;
    private const string RANDOM_FORTUNE_URL = "http://ServiceDiscoveryService/api/values";

    public ServiceDiscoveryService(IDiscoveryClient client, ILogger<ServiceDiscoveryService> logger)
    {
        _handler = new DiscoveryHttpClientHandler(client);

        _logger = logger;
    }

    public  string RandomFortuneAsync()
    {
        _logger?.LogInformation("Before Calling GetClient");
        var client = GetClient();
        _logger?.LogInformation("After Calling GetClient");
        return  (client.GetAsync(RANDOM_FORTUNE_URL).Result).Content.ReadAsStringAsync().Result;
    }

    private HttpClient GetClient()
    {
        var client = new HttpClient(_handler, false);
        return client;
    }
}

In your appsettings.json files(s), add eureka:client:validate_certificates => false,   the default is true if not specified.

@dtillman , I will try this settings and will let you know either way.Thank you.

@dtillman, It Worked, my client application is able to call service and fetch data. Thank you.

@dtillman , Quick question, What about if a service has multiple versions , how should they get registered in to eureka ?? If spring-application-name is same for all versions then how do we make sure client application is consuming the appropriate version of service ??


deps: Upgrade react-native-vector-icons to 9.1.0, the latest

This release reportedly fixes the issue that made the "@" icon look
weird (the issue that was keeping us below 7.1.0):
https://github.com/oblador/react-native-vector-icons/pull/1311#issuecomment-1038258615
I've checked on iOS and Android, and the icon looks right at this
version.
See changelog:
https://github.com/oblador/react-native-vector-icons/releases
A few breaking changes were announced, but none that would affect
us:

Icon.TabBarItemIOS convenience component removed as it was removed
from React Native core.
Breaking change: Drop support for
react-native-web/react-native-desktop aliasing.

I didn't notice anything wrong with the icons on a quick tour of the
app at this version, on iOS or Android.

Thanks! Looks good; merging.


Flexible scale sets skipped when retrieving VMs Azure scale sets

The Azure API doesn't support retrieving VMs from Flexible Scale Sets (only Uniform Scale Sets have support for this).  This fix checks the OrchestrationMode of the Scale set, and skips over any that have OrchestrationModeFlexible.  This also bumps the version of the compute section of azure-sdk-for-go to 2021-12-01, in order to pick up the ability to read the scale set orchestration mode.
Resolves https://github.com/prometheus/prometheus/issues/11370
Signed-off-by: Derek Cheong<EMAIL_ADDRESS>

I don't like mixing 2021-12-01/compute and 2018-10-01/network. I think someone should tackle https://github.com/prometheus/prometheus/issues/10942 instead

We have looked at this during our bug scrub and we have decided that the proper fix for this is to update the Azure dependency as in #10942, hence we are closing this issue. Thanks!


@opentelemetry/otlp-transformer throw error when run in old Android webview, throw error "Cannot redefine property: name"

What happened?
@opentelemetry/otlp-transformer:0.52.1
When running the code that imports the current package in a low version webview（for example: Android 5.1.1; vivo X7L Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/<IP_ADDRESS>）, the error message is as follows:
Uncaught TypeError: Cannot redefine property: name
I traced the code, found the code in<EMAIL_ADDRESS>Object.defineProperty(TraceService.prototype["export"] = function export_(request, callback) {                             return this.rpcCall(export_, $root.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest, $root.opentelemetry.proto.collector.trace.v1.ExportTraceServiceResponse, request, callback);                         }, "name", { value: "Export" });
 Object.defineProperty(MetricsService.prototype["export"] = function export_(request, callback) {                             return this.rpcCall(export_, $root.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest, $root.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceResponse, request, callback);                         }, "name", { value: "Export" });
Object.defineProperty(LogsService.prototype["export"] = function export_(request, callback) {                             return this.rpcCall(export_, $root.opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest, $root.opentelemetry.proto.collector.logs.v1.ExportLogsServiceResponse, request, callback);                         }, "name", { value: "Export" });
When executing the above 3 code segments, Android low-version webview will report an error "Cannot redefine property: name" because these webviews will throw an error when the name attribute of the function is modified.
I don't know what the three lines of code do, and the code runs normally without any errors when I comment out those lines. Hi, friends, if I want to fix this problem, what should I do? Thank you very much.
Steps to Reproduce
Expected Result
Actual Result
Additional Details
OpenTelemetry Setup Code
No response
package.json
No response
Relevant log output
No response

The error happens in code that is generated by the protobuf dependency required to serialize output. Unfortunately I don't know if we will be able to support such an old version of android webview as we don't have control over the protobuf code generation. Which exporter are you using?
I'm going to remove the bug label since this is an old unsupported version of android. I'll leave the issue open for now so we can hopefully find a workaround for you.

The error happens in code that is generated by the protobuf dependency required to serialize output. Unfortunately I don't know if we will be able to support such an old version of android webview as we don't have control over the protobuf code generation. Which exporter are you using?
I'm going to remove the bug label since this is an old unsupported version of android. I'll leave the issue open for now so we can hopefully find a workaround for you.

Thank you for your reply and look forward to a workaround, thank you very much！

@fengyangbj-kanyun are you using @opentelemetry/otlp-transformer:0.52.1 directly or do you use it via an exporter, if yes: which one?

we are using @opentelemetry/otlp-transformer:0.52.1. not directly, we are using @opentelemetry/exporter-logs-otlp-http which depends on it.

Hmm, I see - so it's actually happening even though the code is unused (the http exporter is actually http/json), I think the solution to this might be the same as the solution to #4987.
Are you using tree-shaking when creating your bundle? It should actually be tree-shaken out since it's not used 🤔

Thank you very much, yes, finally, we use
tree-shaking to remove these codes，At first，the rollup does not work as expected.
----------Reply to Message----------
On Wed, Nov 27, 2024 17:19 PM Marc @.***> wrote:
Hmm, I see - so it's actually happening even though the code is unused (the http exporter is actually http/json), I think the solution to this might be the same as the solution to #4987.
Are you using tree-shaking when creating your bundle? It should actually be tree-shaken out since it's not used 🤔
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>


Log api key even if authentication for method is turned off.

Let's say that i have a method show and it's authentication is set to false
'show' => [
            'keyAuthentication' => false
        ],

And we have a user that is "logged in" (has api key in the header). This fix allows API to log api_key_id along with request for this method in data base. If user does not have api key in the header, data api_key_id will be null.

Hi @cernicc we'll test this out. Thanks!


Updated Vagrantfile to support Jenkins automation

This PR contains the updated Vagrantfile script required for Jenkins FIT automation.
The 'template' section is the server template for Jenkins runs.
The 'dev' section is for local workstation-based Vagrant runs.
Vagrant scripts are idiosyncratic and I had to do some weird things to make it work.
@hohene @johren @jimturnquist @diontje @larry-dean

Closing PR will revise later.


Use orjson instead of json for JSON serialization

Description
The authors of orjson describe it like this:

orjson is a fast, correct JSON library for Python. It benchmarks as the fastest Python library for JSON and is more correct than the standard json library or other third-party libraries.

Based on the benchmarks the authors have shared, orjson is 5 to 50 times faster than the standard json library.
Expected behaviour
Add an option to use orjson instead of json.
Actual behaviour
json is the only option for JSON serialization.

Comparing the standard Jsonifier used by Connexion and orjson in an example API call in my application, I see the difference in performance can be extreme, 570 times faster:
FlaskApi._serialize_data():
# 2434078 bytes, total time 8ms
body = orjson.dumps(data, option=orjson.OPT_NAIVE_UTC | orjson.OPT_NON_STR_KEYS)

# 3899087 bytes, total time 4565ms
body = cls.jsonifier.dumps(data)

To validate the output, I used json.loads() and verified that it creates identical objects. It does.

Please also include the option to use orjson for deserialisation.

Isn't this possible by custom json encoder?
In connexion 2.x
connexion_app.app.json_encoder = CustomJSONEncoder

and by jsonifier in connexion=3.x
jsonifier=Jsonifier(kw.json)

Jsonifier unfortunately always adds JSONEncoder to dumps calls:
self.dumps_args.setdefault("cls", JSONEncoder)
which is not allowed for orjson.
You can work around this by subclassing Jsonifier:
import orjson

class OrJsonifier(Jsonifier):
    """A Jsonifier that uses orjson for serialization and deserialization."""

    def dumps(self, data, **_kwargs):
        return orjson.dumps(data).decode() + "\n"

connexion_app = AsyncApp(name, jsonifier=OrJsonifier())


Add logic to ignore MATERIALIZED or ALIAS columns in insert statements.

If the table schema contains MATERIALIZED or ALIAS columns , they should be ignored.
    `_version` UInt64 MATERIALIZED 1,
    `_deleted` UInt8 MATERIALIZED 0,
    `_updated` DateTime MATERIALIZED now()

Hi!
any news about this?

Hello!
is there any news on this issue? We encountered it in our data transfer :(

Hello! is there any news on this issue? We encountered it in our data transfer :(

Hi @tellienne , working on it, will have a fix early next week.

@subkanthi Any update on this?
waiting for a fix to exclude the Materialized columns using column.exclude.list:


Update Get-NetIPInterface.md

Changed base reachable time default value to 30000
Fixes https://github.com/MicrosoftDocs/windows-powershell-docs/issues/3222

@JohanFreelancer9 Please copy review

@tiburd hope all is well. Could you please help us merge this PR? Ty!


Error when running prisma generate inside a docker container when using pnpm workspaces

Bug description
pretty straight forward setup where I'm inside a docker container and attempting to run pnpx prisma generate after a basic install.
This error presents itself:
 > [builder 11/12] RUN pnpx prisma generate: #19 2.088 .../.local/share/pnpm/store/v3/tmp/dlx-1 | Progress: resolved 1, reused 0, downloaded 0, added 0 #19 3.095 .../.local/share/pnpm/store/v3/tmp/dlx-1 | Progress: resolved 1, reused 0, downloaded 1, added 0 #19 4.567 .../.local/share/pnpm/store/v3/tmp/dlx-1 | Progress: resolved 2, reused 0, downloaded 1, added 0 #19 4.591 .../.local/share/pnpm/store/v3/tmp/dlx-1 |   +2 + #19 5.099 .../node_modules/@prisma/engines postinstall$ node scripts/postinstall.js #19 5.573 .../.local/share/pnpm/store/v3/tmp/dlx-1 | Progress: resolved 2, reused 0, downloaded 2, added 2, done #19 8.115 .../node_modules/@prisma/engines postinstall: Done #19 8.383<EMAIL_ADDRESS>preinstall$ node scripts/preinstall-entry.js #19 8.432<EMAIL_ADDRESS>preinstall: Done #19 8.432<EMAIL_ADDRESS>install$ node scripts/install-entry.js #19 8.477<EMAIL_ADDRESS>install: Done #19 9.851 Environment variables loaded from .env #19 9.855 Prisma schema loaded from prisma/schema.prisma #19 15.73 npm ERR! code EUNSUPPORTEDPROTOCOL #19 15.73 npm ERR! Unsupported URL Type "workspace:": workspace:*
How to reproduce

Create boilerplate pnpm workspace repo.
this dockerfile:

`
FROM node:16-alpine as builder
WORKDIR /usr/src/app
COPY package.json ./
COPY pnpm-lock.yaml ./
COPY pnpm-workspace.yaml ./
RUN npm install -g pnpm
RUN pnpm install
COPY ./apps/test ./apps/test
COPY ./packages/tsconfig ./packages/tsconfig
WORKDIR /usr/src/app/apps/test
RUN pnpx prisma generate
RUN pnpm run build
FROM alpine as production
RUN apk --no-cache add nodejs ca-certificates
WORKDIR /root/
COPY --from=builder /usr/src/app ./
CMD [ "node", "dist/index.js" ]`
Expected behavior
Expect the client to be generated
Prisma information

generator client {
  provider = "prisma-client-js"
  output = "./__generated__/"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Environment & setup

OS: Mac
Database: postgres
Node.js version: 16

Prisma Version
prisma                  : 4.11.0
@prisma/client          : 4.11.0
Current platform        : darwin-arm64
Query Engine (Node-API) : libquery-engine 8fde8fef4033376662cad983758335009d522acb (at ../../../../Library/pnpm/store/v3/tmp/dlx-14236/node_modules/.pnpm/@prisma+engines<EMAIL_ADDRESS>Migration Engine        : migration-engine-cli 8fde8fef4033376662cad983758335009d522acb (at ../../../../Library/pnpm/store/v3/tmp/dlx-14236/node_modules/.pnpm/@prisma+engines@4.11.0/node_modules/@prisma/engines/migration-engine-darwin-arm64)
Format Wasm             : @prisma/prisma-fmt-wasm 4.11.0-57.8fde8fef4033376662cad983758335009d522acb
Default Engines Hash    : 8fde8fef4033376662cad983758335009d522acb
Studio                  : 0.483.0

Planned to be addressed already.


fix browser crash

Listener events must be removed on page change.

This is still happening. Browser is crashing if you are runnning many carousels at time and you resize the window.


do you have any plan to support wav2vec?

hello, do you have any plan to support wav2vec?

Hi, not on the short term, what feature are you interested in? Deploy, GPU quantization, something else?

deploy. I want to speed up the inference. Thanks for your reply and look forward to the new feature!

any progress?

Not yet, we are working on seq2seq right now :-)

or any advice? I think I can contribute, but at the moment I don't have the relevant direction for this.
and an other question, would you consider provide some C++ deployment toolkit in the future? because if the performance really matters, C++ or Rust would be a natural choice.

I don't know if C++ matters that much if you deploy on Triton (C++).
Triton itself is in charge of the inference, the Python code is just here to call the tokenizer (Rust) and request the inference engine.
The only exception is generative model, because the greedy / beam search part is a bit slow (Pytorch code mainly, but AFAIK you can do better in pure C++). However, search part is tiny compared to the decoder part, so it's not really a priority for us.
Moreover, there are more interesting opportunities like using a tokenizer running on GPU (Nvidia has released one).
Regarding contribution, I would gladly help you.
Can you implement something abstracting the inference part, just a kind of wrapper following HF conventions that can reuse some ONNX model for instance? like in https://github.com/ELS-RD/transformer-deploy/blob/main/src/transformer_deploy/utils/generative_model.py#L44 ?
If you do that and provide an example on ho to use it, I can rework it and support this family of models in the library.


Ignored issue is still shown as an issue on the feature editor

http://preview.ideditor.com/master/#background=finds.jp_KBN_2500&disable_features=boundaries&id=w688368653&map=21.00/35.68932/139.78641

although the "warnings:unsquare_way" flag is on.

Yes, iD still will show any issues for each feature, even if they are not issues caused by your edits...  I think the message even says "Your edits look fine but there are issues with 47 other features".
I think the warnings:unsquare_way changeset tag just means that in that changeset there was an issue detected on one of your buildings.. However it's not necessarily this one, or if it was, you ignored the warning (which is totally ok to do).
I did notice that if I tried editing the building by dragging a node, it did not count that building as one of my edits.  This is a tricky problem, but maybe we can improve it.

I was trying the same thing last week, because I was curious if there are added changeset-tags for warnings when I hit the ignore selection for some issues. So far I remember the ignored issues were also flagged in the changeset tags. And maybe the idea should be for those actively ignored cases that they shouldn't monitored. But then I was happy to found the way for not tracking all issue when I uncheck the "disable all" box, before saving/uploading :-)

But then I was happy to found the way for not tracking all issue when I uncheck the "disable all" box, before saving/uploading :-)

Yes - unchecking the rules will let you opt out of the validator warnings.
(iD is still running the rules in the background, it just won't show you the results or warn you about them)

Yes, I had app. 100 issues about not squared buildings, while the validation was "Enable All" of course. I solved app. 95 of them and the remainng 5 were some kind of "false positives" for these 5 I used the ignored status. Because I doesn't wanted to be monitored and flaged for these 5 as changeset tag warming I switched off the validation only while & before saving/uploading the data to the osm-database. So I wonder if this is really wanted by intension. Maybe the ignored issue should also not count for the changeset warning tags...

@sun-geo I appreciate mappers working to address all validation errors! However, you needn't worry about the ignored warning counts in the changeset tags. It's imperfect derived data based on your public changes. We can't remove ignored warnings from the counts since then they would be unreliable—poor mappers could just manually ignore all the warnings and go unnoticed. In fact, I removed the workaround where disable rules wouldn't affect the counts.
A better solution is to reduce false positives in iD's validator. Let us know when there are ways we could improve it!

Closing this as stale. iD has been including ignored warnings in the changeset tags for months now without anymore pushback. I still think it's better to do than not do.
Once we do #6459 we can improve the warning tags to only include the issues actually created by the user, not all issues for edited features. For example, if you changed the tags of a nonsquare building it wouldn't flag a warning in the changeset since you didn't make the building unsquare yourself.


*BSD support

What is required to support keyring on a given unix system? FreeBSD presumably has something like gnome keyring in ports that could be used for this, or is it possible to point to a system like hashicorp vault as a generic secrets store?

Sorry for the delay in replying! I would love to support BSD, but am not sure what it would take. I’m happy to look into it when I have time, and am open to suggestions and pull requests.

thanks Walther,
Can you point out what library you require on linux at least? I can then see what's available
in FreeBSD for comparison.

You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/hwchen/keyring-rs/issues/36?email_source=notifications&email_token=AACFNUAMNEF4AJ6EMRR6Q5TREUSW5A5CNFSM4KYM2262YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM4EQRI#issuecomment-590891077, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACFNUHYVDZLBW7JT5FF7O3REUSW5ANCNFSM4KYM226Q.

I use secret-service, which is a protocol over dbus to interface with gnome keyring, kwallet, and other keyrings that implement the api.
https://github.com/hwchen/secret-service-rs
https://specifications.freedesktop.org/secret-service/latest/

Hi @dch, we have updated to the latest secret-service, and I believe it runs on BSD.  Can you try out the latest 0.10.4 release on BSD and let us know what you find?  Be sure to install lib-dbus on your BSD system if it's not already installed with the OS.

Hi @dch, I have been able to confirm that our recent 1.0 release runs on Free BSD if you use the Gnome desktop.  Closing this issue.

I believe *BSD support has regressed, on FreeBSD
$ cargo build
   Compiling keyring v2.0.0-alpha.3 (/root/code/keyring-rs)
error[E0433]: failed to resolve: use of undeclared crate or module `default`
   --> src/lib.rs:158:19
    |
158 |     let default = default::default_credential_builder();
    |                   ^^^^^^^ use of undeclared crate or module `default`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `keyring` due to previous error

Slightly change the code to use secret-service on FreeBSD
diff --git a/Cargo.toml b/Cargo.toml
index dfdab29..d72755d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -28,6 +28,9 @@ security-framework = "2.6"
 secret-service = "2.0"
 linux-keyutils = { version = "0.2", features = ["std"] }

+[target.'cfg(target_os = "freebsd")'.dependencies]
+secret-service = "2.0"
+
 [target.'cfg(target_os = "windows")'.dependencies]
 byteorder = "1.2"
 winapi = { version =  "0.3", features = ["wincred", "winerror", "errhandlingapi", "minwindef"] }
diff --git a/src/lib.rs b/src/lib.rs
index 03f2030..5805ebe 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -118,6 +118,9 @@ use windows as default;
 #[cfg(all(target_os = "linux", feature = "linux-secret-service"))]
 use crate::secret_service as default;

+#[cfg(target_os = "freebsd"]
+use crate::secret_service as default;
+
 pub mod credential;
 pub mod error;
 pub mod mock;
@@ -129,7 +132,7 @@ pub mod ios;
 pub mod keyutils;
 #[cfg(target_os = "macos")]
 pub mod macos;
-#[cfg(target_os = "linux")]
+#[cfg(any(target_os = "linux", target_os = "freebsd"))]
 pub mod secret_service;
 #[cfg(target_os = "windows")]
 pub mod windows;

$ cargo build
   Compiling zbus v1.9.3
error[E0432]: unresolved import `nix::sys::socket::sockopt::PeerCredentials`
   --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/zbus-1.9.3/src/connection.rs:163:44
    |
163 |         use nix::sys::socket::{getsockopt, sockopt::PeerCredentials};
    |                                            ^^^^^^^^^^^^^^^^^^^^^^^^ no `PeerCredentials` in `sys::socket::sockopt`

error[E0432]: unresolved import `nix::sys::socket::sockopt::PeerCredentials`
   --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/zbus-1.9.3/src/azync/connection.rs:180:44
    |
180 |         use nix::sys::socket::{getsockopt, sockopt::PeerCredentials};
    |                                            ^^^^^^^^^^^^^^^^^^^^^^^^ no `PeerCredentials` in `sys::socket::sockopt`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `zbus` due to 2 previous errors

keyring 1.2.0 also don't compile on FreeBSD
error[E0583]: file not found for module `platform`
  --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/lib.rs:79:1
   |
79 | mod platform;
   | ^^^^^^^^^^^^^
   |
   = help: to create the module `platform`, create file "/.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/platform.rs" or "/.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/platform/mod.rs"
error[E0412]: cannot find type `Error` in module `crate::platform`
  --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/error.rs:15:38
   |
15 |     PlatformFailure(crate::platform::Error),
   |                                      ^^^^^ not found in `crate::platform`
   |
help: consider importing one of these items
   |
7  | use core::error::Error;
   |
7  | use core::fmt::Error;
   |
7  | use crate::Error;
   |
7  | use std::error::Error;
   |
     and 2 other candidates
help: if you import `Error`, refer to it directly
   |
15 -     PlatformFailure(crate::platform::Error),
15 +     PlatformFailure(Error),
   |
error[E0412]: cannot find type `Error` in module `crate::platform`
  --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/error.rs:21:38
   |
21 |     NoStorageAccess(crate::platform::Error),
   |                                      ^^^^^ not found in `crate::platform`
   |
help: consider importing one of these items
   |
7  | use core::error::Error;
   |
7  | use core::fmt::Error;
   |
7  | use crate::Error;
   |
7  | use std::error::Error;
   |
     and 2 other candidates
help: if you import `Error`, refer to it directly
   |
21 -     NoStorageAccess(crate::platform::Error),
21 +     NoStorageAccess(Error),
   |
error[E0425]: cannot find function `platform` in module `platform`
  --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/lib.rs:71:15
   |
71 |     platform::platform()
   |               ^^^^^^^^ not found in `platform`
error[E0425]: cannot find function `set_password` in module `platform`
   --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/lib.rs:140:19
    |
140 |         platform::set_password(&self.target, password)
    |                   ^^^^^^^^^^^^ not found in `platform`
error[E0425]: cannot find function `get_password` in module `platform`
   --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/lib.rs:148:19
    |
148 |         platform::get_password(&mut map)
    |                   ^^^^^^^^^^^^ not found in `platform`
error[E0425]: cannot find function `get_password` in module `platform`
   --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/lib.rs:158:34
    |
158 |         let password = platform::get_password(&mut map)?;
    |                                  ^^^^^^^^^^^^ not found in `platform`
error[E0425]: cannot find function `delete_password` in module `platform`
   --> /.cargo/registry/src/github.com-1ecc6299db9ec823/keyring-1.2.0/src/lib.rs:166:19
    |
166 |         platform::delete_password(&self.target)
    |                   ^^^^^^^^^^^^^^^ not found in `platform`
Some errors have detailed explanations: E0412, E0425, E0583.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `keyring` due to 8 previous errors

Hi @messense, with the 1.0 release, platform support for keyring-rs became explicit, and FreeBSD has not been included in the supported platforms (because none of the maintainers use it, and GitHub Actions CI doesn't support it).  If you would like to submit a pull request that makes things run on freebsd, I am willing to consider including freebsd support as experimental.
If you do want to work on this, I recommend you target the current head (2.0.0-alpha) rather than the 1.2 release, as platform support has changed quite a bit with the introduction of traits.  It should actually be a lot easier to include FreeBSD support in the new model than it would have been in the old one.

secret-service 3.0 was just published, and I suspect it will fix several of the errors originating from zbus on FreeBSD.
If/when you decide to bump secret-service from 2.0 up to 3.0, I'd be willing to test this on FreeBSD.

Hi @ryanavella, thanks for looking at this.  In fact, I have already done the port of keyring to use secret-service v3, but I couldn't release it to crates.io because the service hadn't been released.  (It uses the block interface that secret-service v3 has provided for sync clients such as keyrong).  If you would like to do some testing right away feel free to download and build this published pre-release of keyring. I anticipate that I will release a keyring 2.0 later this week to crates.io that uses the secret-service v3.

Ah great, I was looking through the commit tags yesterday but somehow completely missed v2.0.0-alpha.4. I can confirm that after installing the gnome-keyring package and making a few small tweaks to Cargo.toml and lib.rs (similar to @messense's suggestion) it can compile on FreeBSD now.
The tests in error.rs and mock.rs all pass, but the tests in secret_service.rs all fail with NoStorageAccess(NoResult). I suspect this could be due to some of my system's hardening settings, I'll play around some more and report my findings.

Hi @ryanavella, I've just released v2.0.0-rc.1 (based on secret-service v3) to crates.io.  Please try the mods and see if this works on FreeBSD.

Hi @brotskydotcom, I tried both v2.0.0-rc.1 and v2.0.0-rc.3, and with the same few small tweaks I can get it to compile on FreeBSD.
I also realized that I should be testing through the linux_test.sh script rather than through cargo test, and now I'm getting new errors. See the collapsed test results below.
You'll notice that all of the failed tests are looking for an object at the path /org/freedesktop/secrets/collection/login. FreeBSD typically doesn't have an /org top-level directory, and as far as I can tell this path doesn't correspond to any XDG directories, environment variables, or config settings, so I am not sure where it is coming from other than dbus or gnome-keyring itself.
I realize this is likely user-error on my part, so I'll keep experimenting to see if I can get a working gnome-keyring setup. In the meantime, if you want me to submit a PR for FreeBSD support I'd be willing to do that, only a few small changes are needed in lib.rs and Cargo.toml
Test results

Here is what the patch would look like, @messense would you be willing to try on your end as well? If you have configured gnome-keyring on your system, I'd be curious to know what the output of ./linux_test.sh looks like for you.
diff --git a/Cargo.toml b/Cargo.toml
index 9531b02..c5cc777 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -34,6 +34,9 @@ security-framework = "2.6"
 secret-service = { version = "3", optional = true }
 linux-keyutils = { version = "0.2", features = ["std"] }

+[target.'cfg(target_os = "freebsd")'.dependencies]
+secret-service = { version = "3", optional = true }
+
 [target.'cfg(target_os = "windows")'.dependencies]
 byteorder = "1.2"
 winapi = { version =  "0.3", features = ["wincred", "winerror", "errhandlingapi", "minwindef"] }
diff --git a/src/lib.rs b/src/lib.rs
index 2952e92..87282dc 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -122,6 +122,11 @@ use crate::secret_service as default;
 #[cfg(all(target_os = "linux", feature = "linux-default-keyutils"))]
 use keyutils as default;

+#[cfg(target_os = "freebsd")]
+pub mod secret_service;
+#[cfg(target_os = "freebsd")]
+use crate::secret_service as default;
+
 #[cfg(target_os = "windows")]
 pub mod windows;
 #[cfg(target_os = "windows")]

Hi @ryanavella and hooray!  Thanks so much for trying with RC3, because maybe we can even get this into the v2 final release!  Please do submit a PR!
To help clarify what's happening in your tests: the "object path" used by secret-service has nothing to do with directory paths.  Your errors suggest that the linux-test.sh script, which is trying to create and unlock a default collection for secret service to use (canonically called "login"), is not actually getting that collection created on FreeBSD.  This makes sense because FreeBSD security is way better than linux security, so it probably won't allow access to the default collection headlessly at all.
As you will see from the docs, secret-service is not really suitable for use on headless boxes.  That test script really only exists so I can at least get some level of testing done on the secret-service code in an automated fashion on GitHub's Ubuntu build machines.  I will probably never add that kind of testing on FreeBSD, because AFAIK GitHub doesn't have any FreeBSD machines.
As long as you can get the full test suite to pass running headed as a developer, including the ignored tests (cargo test -- --include-ignored), I'm happy to have this code in the release.

Hey @ryanavella don't worry about the PR.  I'm merging your patch.


Provide examples or high level interfaces for large file uploads

Hello!
I'm currently using put_object with a Vec<u8> to upload, which is working fine for small files, but since I have to load everything into ram I'm looking for a way to read from a file instead. It seems StreamingBody is meant to support that, but I'm having trouble getting something to compile (I have an std::fs::File, but I need something that implements certain tokio types).
An example or some high level helper functions would be very appreciated.
Thanks!

I have a tokio::fs::File, and I wonder how to turn this into a ByteStream/StreamingBody. Any ideas?

@kpcyrd  maybe this also helps you https://users.rust-lang.org/t/turning-a-file-into-futures-stream/33480

@kpcyrd Please take a look at https://stackoverflow.com/questions/57810173/streamed-upload-to-s3-with-rusoto


MicroROS Build Error

Dear @pablogs9 ,
I found an issue when I have to rebuild microROS from start again. I use foxy with ubuntu20.04 without using docker. I followed all the steps and use with stm32f429-disco. The issue happen when I choose micro-ROS Ping Pong in menu config. The issue is the following:
I tried with usbnsh and with drivebase which I copy drivebase to stm32f429-disco, but issue is still happen.
Please suggest me the issue related with.

I guess that you are using Nuttx, is this right? I'm sorry but right now micro-ROS for stm32f429-disco is only supported for Zephyr. Have you tried to test the Zephyr port?


Handle sortable in nested collections recipe

Hi, I am using your forms for nested collections  :+1: . Can you please/handle sortable and drag drop in nested collections, if you want I can post a couple of samples.
thanks

A link to some code http://j.mp/1nraN6T

Hi I am interested in this feature as well :+1: can I ask you guys how do you persist this in the database table? what structure do you follow.

Are you guys talking about drag/drop for the existing icollection stuff on
the demo site (which handles moveup/movedown)
On 24 Jan 2016 19:00, "manualshifter"<EMAIL_ADDRESS>wrote:

Hi I am interested in this feature as well [image: :+1:] can I ask you
guys how do you persist this in the database table? what structure do you
follow.
—
Reply to this email directly or view it on GitHub
https://github.com/mcintyre321/FormFactory/issues/19#issuecomment-174330668
.

In my application, I am using this and need it for nested lists and sortable:

Within the same list
Across two lists

First one is better rubuxa is better
1  http://j.mp/1OQNh9G
2  http://j.mp/1NtXKWS

I'm unlikely to work on this any time soon, but I do think it would be a good feature to have, and would happily accept a pull request*.
*with certain coding standard to be met e.g.

example on docs page
works with nested collections
need good justification to use an external library, as we already depend on jquery ui
probably some other things :P

I would love to, I would need some help on how to wire it up. I am not as proficient as you   :))
The good thing is I clearly understand what would make a good user experience. So I would need to be guided on where all I would need to make changes.
Rather than update the current collection item you have, I was planning on adding support for a new one.
Just know that I might fail (pretty poorly) and you/someone here might need to help me. I see its getting popular recently.
thanks
drive baby drive :)

I tried myself as well, could not get it working so I gave up after sometime. I need to know how to wire it up at the different levels. UI, Domain, View Engine etc.

OK so I've enabled in-list drag and drop using the HTML5 draggable API, which unfortunately doesn't support touch screen, but mouse dragging works.
https://github.com/mcintyre321/FormFactory/commit/f391a4c8920f6dae9b38fcfdee1d301b75379d9b

There are libraries (as linked above) which support touch events, but they don't work with event delegation as far as I can tell. If anyone wants to improve this solution, please feel free to do so.

Hmm. This might only work in chrome :(

This is awesome :+1:
I want to help extend it. Not sure why it closed, lots of community participation
thanks again

I tested it works, would like to add an icon to the div besides the title
// snippet for highlight
 $(".droppable").droppable({
 accept: '.draggable',
 over: function(event, ui) {
   $(this).addClass('tempHighlight');
 },
 out: function(event, ui) {
   $(this).removeClass('tempHighlight');
 },
 drop: function() {
    //do some stuff
 }
 });

// snippet for drag handle
 $(row).draggable({
handle: '#handle1, #handle2',
 });

This was added -  draggable="true" id="listitem-@id" , wondering where is the order being serialized and saved/POSTED
Here are the changes, I cant find it though
https://github.com/mcintyre321/FormFactory/commit/f391a4c8920f6dae9b38fcfdee1d301b75379d9b
Wondering

Are you referring to this? I am trying to wire it up to an update/POST will let you know soon.
       $(document).on("drop", function (e) {
           e.preventDefault();
           e.stopPropagation();
           var $src = $(document.getElementById(e.originalEvent.dataTransfer.getData("text")))
               .closest("li[draggable=true]");
           var $target = $(e.target).closest("li[draggable=true]");
           if ($target.length && $target[0] !== $src[0]
                  && $target[0].parentElement === $src[0].parentElement) {
               var beforeOrAfter = $src.index() > $target.index();
               $src.remove();
               if (beforeOrAfter) {
                   $src.insertBefore($target);
               } else {
                   $src.insertAfter($target);
               }
               console.log("dropped");
           }
       });

This is how I can get the serialize the list
var droppedSortedList = $( ".selector" ).sortable( "serialize", { key: "sort" } );
for e.g. here it would be
we need to wire this to the Save/Update Handler   $src.sortable( "serialize", { key: "sort" } );
can someone tell me where the update function is, so I can wire it up there.

@mcintyre321 & @patagoniahiker thank you so much :100: for both...
can you please show me how to do the model update/save, after drag complete.

So all you should need to do to save/post it is put it inside a form and send it to the server.
FormFactory and the collections stuff should be compatible with http://www.hanselman.com/blog/ASPNETWireFormatForModelBindingToArraysListsCollectionsDictionaries.aspx (or at least it was about 3 years ago, last time i checked!)
If you want to use ajax, or something, you can serialize the form using form2js https://github.com/lookfirst/form2js

Please make any changes on a fork, and send a PR when you want me to review them. I'm not going to integrate snippets.

I am new to git hub, let me try to get it working and then make the update


new: test: #312: add cache watcher to notify tests when async requests are cached

Instead of waiting a fixed amount of time & hoping for async-downloaded files to be cached, this implementation will monitor the cache directory and notify the test when the requested files have been fully cached.
CacheWatcher will make sure that the expected files have been created and are the expected size.

Codecov Report

Merging #313 into master will decrease coverage by 0.29%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master     #313     +/-   ##
=========================================
- Coverage   51.17%   50.87%   -0.3%
=========================================
  Files          60       60
  Lines        3029     3035      +6
  Branches      334      335      +1
=========================================
- Hits         1550     1544      -6
- Misses       1388     1403     +15
+ Partials       91       88      -3

Impacted Files
Coverage Δ

...n/java/com/qubole/rubix/bookkeeper/BookKeeper.java
69.04% <0%> (-1.56%)
:arrow_down:

.../qubole/rubix/bookkeeper/RemoteFetchProcessor.java
76.78% <0%> (-5.36%)
:arrow_down:

.../qubole/rubix/hadoop2/Hadoop2ClusterManagerV2.java
49.12% <0%> (-3.51%)
:arrow_down:


Dev

Codecov Report

Merging #14 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #14   +/-   ##
=======================================
  Coverage   78.03%   78.03%
=======================================
  Files          91       91
  Lines        3405     3405
  Branches      371      371
=======================================
  Hits         2657     2657
  Misses        602      602
  Partials      146      146

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1f8d96...c87e716. Read the comment docs.


Added ccitt group 3 compression

Added ccitt group 3 compression

not include the packages

PDF with CCITTFAX3 compression:
http://u8885.netangels.ru/test.pdf

@gheeres Why didn't you merge this pull request? What's wrong?

Hi, I've got a modern fork github.com/Open-Inventions/PDFSharp.Extensions with ImageSharp and .NET 6 if you are interested.


add rhui and devtools rpms for fips go-toolset

Which issue this PR addresses:
Fixes https://msazure.visualstudio.com/AzureRedHatOpenShift/_workitems/edit/10354848/
What this PR does / why we need it:
downloads and sets cloud-user to use the fips compliant go-toolset
Test plan for issue:
tested by spinning up azure vm, and running scripts to confirm results are as expected. would like to deploy to CI vm's as well to confirm changes.
Is there any documentation that needs to be updated for this PR?
no

Adding hold label until I can deploy this PR to INT environment to validate changes.

still hitting a blocker with testing this. It turns out the devtools-rpms is not available in RHUI rpms, meaning it is locked behind a subscription which would require a BYOS in current model to fix this. Looking into RHEL8 and other solutions that my not require FIPS on the host if entire CI build process is moved into containers.

closing in favor of #1774; thanks @bennerv


flash ctrl register boundary

The flash controller region protection registers should be on a separate 4KB boundary from the remaining flash registers.  This is to allow separate control of the core registers and the regions.

i don't really think this is valid anymore and I don't believe there is a usecase for it.


[Feature Request] Copy code button on code blocks

I think copy code button on code blocks would be a nice addition to this already a great theme.

1 for this idea, adding this might involve js https://aaronluna.dev/blog/add-copy-button-to-code-blocks-hugo-chroma/

@dhananjaylatkar @danieltomasz
Hi there! It is merged from #76. Try it out :D and I would appreciate any ideas!
Thanks and best regards!

@monkeyWzr Thanks for quick implementation!!!
Side note -
It looks kind of out of place for now. You can fix colors when you get time or I'll give it a go if you don't mind. :)


Updates needed for recent trio versions (0.17.0 current as of this writing)

There have been a number of deprecations/removals in trio since trio-amqp was last updated:

open_cancel_scope() deprecated
Event.clear() deprecated
Python 3.5 support removed
trio.hazmat renamed to trio.lowlevel (also as #11)

I will submit a PR shortly to address these changes.
There are also some existing similar updates in master (at least the removal of trio.Queue) that have yet to be released. As a result, the current release of trio-amqp is currently only compatible with trio<=0.8.0 (0.17.0 is current). Please make a new release as soon as possible. There's been no movement on releasing https://github.com/M-o-a-T/asyncamqp as mentioned in #8, and it sounds like you intend to instead merge that back into trio-amqp as discussed in #9, but the situation right now is that there is no working, released AMQP client library for recent versions of trio.

Since the recent reset to match https://github.com/M-o-a-T/asyncamqp, which is anyio-based, means that this code no longer uses trio interfaces directly, this issue is effectively addressed.
@smurfix Which of the two repos would you say is the one I should dep going forward? Also, are you planning a PyPI release soon so that I don't have to use a git URL dep? :)

Since the recent reset to match https://github.com/M-o-a-T/asyncamqp, which is anyio-based, means that this code no longer uses trio interfaces directly, this issue is effectively addressed.
@smurfix Which of the two repos would you say is the one I should dep going forward? Also, are you planning a PyPI release soon so that I don't have to use a git URL dep? :)


Improved operator==() and operator<() efficiency

Refactored operator==() and operator<() methods to use data members directly instead of using unixtime() method. Greatly improved efficiency.

The warning “if one or both DateTime objects are invalid, logic errors can occur” (emphases mine) sounds both vague and scary to me. What could these “logic errors” mean? Could the program just do absolutely anything, like ignoring its control logic or resetting itself? This is reminiscent of the spectre of undefined behavior, complete with its proverbial nasal demons.
Truth is, nothing bad will happen. The operators will just return either true or false, without breaking the innards of the program. I would replace the sentence with something less scary, like:

If one or both DateTime objects are invalid, the returned value is meaningless.

But then, it sounds like stating the obvious: how could it be meaningful to ask whether a given date is before or after an invalid date? Maybe we could remove the warning, and instead add a note in the documentation of isValid(). Something along the lines:

isValid() is the only method that returns a meaningful value when called on an invalid object.

Agreed, "logic errors" is a bit vague. I think adding the note on isValid() is better than a note on every comparison operator. If your object is invalid then it's not going to do anything useful.
See the build details for clang's formatting suggestion.
I tested the changes on an Uno and the speedup is significant if you need to eke out every microsecond.

If we removed the warning for the comparison operators it's likely someone not familiar with this library may not realise the importance of using isValid() or even the possibility of a DateTime object being invalid. They may just use a comparison operator, complain that the library is buggy and get frustrated. With the warning, they'll realise the importance of validating the date/time components without needing prior knowledge of isValid()..


Upgrade raven-js to version 3.9

raven-js is a Bower package, see https://github.com/getsentry/raven-js/releases
Closes #70

Hi @chuchuva, thanks for PR.
Unfortunately we are going to remove the add-on raven bower dependency in favor of npm in a future release (see #66). For that reason, there is no need to update the bower file anymore, and I will not merge this PR.


Add try_stream equivalent

Proposed API:
fn try_stream<F, Ok, Error, Fut>(func: F) -> impl TryStream<Ok = Ok, Error = Error>
where
    F: FnOnce(TryYielder<Ok, Error>) -> Fut,
    Fut: TryFuture<Ok = Ok, Error = Error>;

impl<Ok, Error> TryYielder<Ok, Error> {
    async fn yield_item(&mut self, item: Result<Ok, Error>);
    async fn yield_ok(&mut self, item: Ok);
    async fn yield_error(&mut self, item: Error);
}

Makes it easier to construct fallible streams. I just need to figure out how to not duplicate too much of the internals because I really don't want to keep track of two very similar APIs.

(found this while trying to dog-food the crate in Kitsune)


User/aarjavjain101/standard frame gps imu

Only changes in the latest commit inside the link_telemetry.py file need to be pulled in. Just made changes to the log_upload function and the main to fix some small errors when trying to read the SD logger.

Changes on the old branch that was already merged in. It shows that there are changes to everything (again) which isn't the case.
If the changes were minimal, can you make a new branch off main and port the changes to that branch and create a PR for that. I'll close this one for now.


[C++][Parquet] Add --print-key-value-metadata option to parquet_reader tool

Added --print-key-value-metadata option to parquet_reader tool
https://github.com/apache/parquet-cpp/pull/450

Reporter: Jacek Pliszka / @JacekPliszka
Assignee: Jacek Pliszka](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jacek.pliszka) / @JacekPliszka
PRs and other links:

GitHub Pull Request #450

Note: This issue was originally created as PARQUET-1256. Please see the migration documentation for further details.

Wes McKinney / @wesm:
Issue resolved by pull request 450
https://github.com/apache/parquet-cpp/pull/450


Add fastsimcoal

Here is what I have so far for the dadi/fastsimcoal pipeline. I broke up the existing dadi pipeline to follow the format of the n_t pipeline, but other than breaking apart the simulation/sfs steps and changing the directory structure, I have left the dadi rules largely as-is. To that, I added rules to make an fsc-style SFS from the dadi SFS, and then set up and run a fsc analysis.
I've been able to get this pipeline to run and produce output for dadi and fsc, but still have to deal with a number of issues (also summarized in the readme):

dadi doesnt use the [0,0] entry of the 2D SFS, which fsc requires. The dadi sfs masks this entry, but unmasking it just produces a value of '0', so may need to go back into the ts_to_dadi_sfs function and  compute this value from the tree sequences.

The IM model that fsc is running is more or less just the example IM model from fsc, so some time needs to be spent making sure this is as identical as possible to the IM model run by dadi. May also want to add other models for fsc to run.

Figure out the best way to run fastsimcoal in the pipeline when downloading different versions for running on Mac/Windows/Linux (see here: http://cmpg.unibe.ch/software/fastsimcoal2/)

Add a plotting step to summarize and compare the parameter estimates from fsc and dadi.

Set up the snakemake pipeline to run with a config file.

QC, testing, and documentation. So far I've run this just with the human model but it would be good to run it with the drosophila two pop model.

Finally, its worth mentioning that I havent used fastsimcoal or dadi before working on this so there may be things I've done incorrectly or overlooked. When the time comes to QC this pipeline, there are some people in the Lohmueller lab with more experience with this stuff that I can ask to look over it. I'll be able work on this during most of next week, but will actually be out for Memorial day weekend starting tomorrow without computer access.

awesome @ckyriazis. i'm going to pink @jgallowa07 for help on the review and eventual merge

Just a quick update here - I think I've solved the issue with #1 above, I'll push an update early next week.

Pulling in @tjstruck for comments, since he wrote the dadi code

re: "As of now, this snakemake pipeline only works with the Mac version of fastsimcoal2."
is this still true?

Only in that the path to fastsimcoal in the shell command looks for the Mac
directory name - it would be easy to change that but I wasn’t sure the best
way to make that work with whatever version the user downloads.
On Thu, May 23, 2019 at 4:21 PM Jared Galloway<EMAIL_ADDRESS>wrote:

re: "As of now, this snakemake pipeline only works with the Mac version of
fastsimcoal2."
is this still true?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/popgensims/analysis/pull/29?email_source=notifications&email_token=ACZWM5KMHVEFCO5BDH2HPLTPW4RGNA5CNFSM4HOXAH72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDX2NY#issuecomment-495418679,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACZWM5N6P5Q7TLAE47WZRHLPW4RGNANCNFSM4HOXAH7Q
.

Only in that the path to fastsimcoal in the shell command looks for the Mac directory name - it would be easy to change that but I wasn’t sure the best way to make that work with whatever version the user downloads.

Gotcha (My vote is linux if we have to hard-code/choose a default executable)
On a similar note, I'm having trouble running the whole pipeline, it seems to be failing in rule run_fsc First, I had to do a chmod +x to make fsc26_linux64/fsc26 executable, and now it seems to be unable to open IM.tpl and IM.est:
fastsimcoal was invoked with the following command line arguments:
fsc26_linux64/fsc26 fsc26 -t IM.tpl -n 100000 -d -e IM.est -M -L 40 -q
Unable to open input file (IM.tpl). Exiting program.
Unable to open input file (IM.est)```

any thoughts?

As for the pipeline overall, the dependencies and structure look great!

Figure out the best way to run fastsimcoal in the pipeline when downloading different versions for running on Mac/Windows/Linux (see here: http://cmpg.unibe.ch/software/fastsimcoal2/)

I personally think we should just stick with linux - If you really want both, you can tell the user to download the appropriate executable and simply rename the fsc26_<dist> directory to something like fsc so that the Snakefile doesn't need to be altered.

Add a plotting step to summarize and compare the parameter estimates from fsc and dadi.

I like the Idea of boxplots as a comparison.

Set up the snakemake pipeline to run with a config file.

This is super easy to do and definitely helps keep the repo clean since you're not constantly changing the Snakefile.

QC, testing, and documentation. So far I've run this just with the human model but it would be good to run it with the drosophila two pop model.

As far as documentation, I would be slightly more clear on what needs to be done for fsc to be run.

Are the IM.* files being copied into the directories with the SFSs? For fsc to run, you need the .tpl, .est, and .obs (which is the SFS) file all in the same place.

I just sent some updates to the add_fsc branch that I think should resolve the things @jgallowa07  caught above. They should also resolve the issue with the [0,0] entry in the SFS (#1 above) and improving the documentation for downloading and installing fastsimcoal. I tried to make the instructions for installation general for any platform - would be grateful if others could test this to see if they have any further issues.
Right now I'm focusing on trying to make sure the IM model implementation in fsc and dadi is the same, and then plan to work on the plotting step to compare the results. Will keep you posted on any updates.

@ckyriazis just to let you know, dadi has been updated so that it can work with scipy 1.3.0.
The readme instructions worked for getting fastsimcoal running on a mac, linux, and windows. However, I've only tested the Snakefile on mac and linux.
I can run everything on Mac, but I did have to modify the path for it to locate the fsc26 folder (not sure if we want to do anything more generalized for fsc26's path).
On the linux machine (which is the HPC our lab uses) it's running into an issue with the sed command:
(base) -bash-4.1$ sed -i '' 's/SAMPLE_SIZE/20/g' homo_sapiens/1845187043/fsc_analysis/IM.tpl
sed: can't read s/SAMPLE_SIZE/20/g: No such file or directory

Thanks for the feedback @tjstruck. I'll change the readme regarding the issue with scipy 1.3. Also, I did have the instruction in the readme to change the name of the fsc directory (mv fsc26_* fsc26), but I'm sure thats easy for a user to miss. Any thoughts on a better way to do this?
I tried making the sed command portable on mac and linux following these guidelines: https://stackoverflow.com/questions/16745988/sed-command-with-i-option-in-place-editing-works-fine-on-ubuntu-but-not-mac
So the new command would be: sed -i.bak 's/SAMPLE_SIZE/{num_sampled_genomes_per_replicate}/' {output_dir}/{wildcards.seeds}/fsc_analysis/IM.tpl && rm {output_dir}/{wildcards.seeds}/fsc_analysis/IM.tpl.bak"
I'll push an update to my branch with that soon but in the meantime feel free to paste that in and see if it works.
Finally, on a different topic, would you be able to help code up a dadi IM model with no exponential growth and two migration parameters (so essentially a hybrid between the IM and split_mig model that are already included)? The fastsimcoal IM model has two migration parameters and as far as I know wont work with just one, and it would be ideal to have the models be the same. Let me know if this makes sense.

@ckyriazis, it looks like the new sed command works on both mac and linux.
I can update dadi's Demographics2D.py with a new model I'll add as IM_fsc, but just to make sure I'm understanding what we want:
The new model would be a modification of the split_mig model with a migration rate for each population (m12 and m21 like in dadi's IM model)?
The IM model includes a parameter "s" that represents the initial size of population 1 as a percentage of the ancestral population (with 1-s as the initial size of population 2). If I'm interpreting the fsc_files correctly, there isn't something similar to the "s" parameter in the IM model fsc uses. It just has the final population sizes, right?

@tjstruck, yes thats correct - the same as the split_mig model but just with one additional m parameter, and no s parameter as in the dadi IM model. Thanks!

@ckyriazis, IM_fsc has been added to Demographics2D.py and pushed to the dadi repo. Let me know if you find an error.

Thanks @tjstruck it appears to work! Just one question - what is the direction of the respective m parameters (i.e. pop 1 -> 2 vs. pop 2 -> 1)?

@ckyriazis
m12 is migration from pop 2 -> 1
m21 is migration from pop 1 -> 2

Would you mind squashing the commits for us, @ckyriazis?

Hey @jgallowa07 , I just tried to squash using this as a guide: https://medium.com/@vanshbadkul/squashing-commits-in-git-a-step-by-step-guide-884581224adf
but all of the commits are still there. Any idea what I might be doing wrong?

Also @tjstruck - could I ask you to code up an additional dadi model for use in the pipeline? What I'm trying to do now is fit a two population Gutenkunst model to the simulations for Africa and Europe to see how well it recovers the true parameters under the correct model. So the parameters would be:
Nanc - size of Africa population before increase to modern size
Naf - size of africa population
Taf - time of increase from Nanc to Naf
Nb - size of bottleneck out of africa population
Tb - timing of out of africa split
Neu0 - initial size of Europe population
Teu - timing of onset of exponential growth
Neu_final - final size of Europe population after exponential growth
m_af-b - symmetric migration rate between african and bottleneck population
m_af-eu - symmetric migration rate between african and europe population
So basically just the Gutenkust model ignoring any parameters having to do with the Asian population. Let me know if this makes sense.

Hey @jgallowa07 , I just tried to squash using this as a guide: https://medium.com/@vanshbadkul/squashing-commits-in-git-a-step-by-step-guide-884581224adf
but all of the commits are still there. Any idea what I might be doing wrong?

I would follow Jerome's instructions found here: https://github.com/popgensims/stdpopsim/blob/master/docs/development.rst

@ckyriazis, we played with something similar before as part of starting the two population analysis and getting useful code in the Snakefile, however ignoring the Asian population will make the parameters non-recoverable.
The more appropriate comparison would be marginalize the spectrum in the Gutenkunst model to be a 2D spectrum for the African and European populations and fitting that to the simulated date. However, that would be very slow for fitting.

@ckyriazis @tjstruck If the goal is to test the fitting process, then the easiest way is to change the msprime simulations temporarily to remove migration from the Asian population into Europe and Africa. Then one could fit a two-population Africa and Europe model and expect to recover the parameters quickly.

These are all good points, and I think it would be helpful to discuss and clarify what our goals are here. But my understanding at the moment is that we arent necessarily trying to test the fitting process (it seems to be doing well with the simpler IM models) as much produce a rough comparison of how fsc and dadi perform in inferring the parameters of a model that is a better fit than the IM model, even if it doesnt get the parameters 100% correct. Also, @petrelharp mentioned on our last call that he may be able to help with thinking about what the expected parameter values should be under the models that we're trying to fit?

Also @jgallowa07 - I cleaned up the commits using the hard reset described in that document, since things were getting pretty messy. Let me know if everything looks OK.

Okay... If the goal is just to increase complexity, then the model you proposed makes sense. @tjstruck, please do add it.

@ckyriazis do you explicitly want a bottleneck population/parameter rather than just having Europeans split from Africans? Currently in the Gutenkunst model the bottleneck population splits from the African population, and then the bottleneck population splits into European and Asian populations. So I'm not sure it is worth using for fitting.

@tjstruck, I think maybe leave it in so we can compare the inferred Nb and Neu0 parameters to the true ones? Unless you think it will make fitting the model substantially more difficult.

@ckyriazis I think I was overthinking somethings. Since the model is pretty specific to what the project, let's have it be in dadi_utils.py instead of the in the dadi repo.
See attached file for the function OoA_2D_Af_Eu_func.
OoA_2D_Af_Eu.txt

@tjstruck, I just tried a test run and got a "Dimensionality of phi and lengths of ns and xxs do not all agree." error. Heres how I set the function and parameters after pasting the function into dadi_utils.py:
if demo_model == 'Gute2pop': 		func = OoA_2D_Af_Eu_func 		params = [1.5, 1.5, 1.5, 1.5, 0.5, 0.5, 0.5, 0.5, 0.5] 		lower_bound = [0, 0, 0, 0, 0, 0, 0, 0, 0] 		upper_bound = [50, 50, 50, 50, 10, 10, 10, 10, 10]
Any idea what might be going wrong?

Sorry about that @ckyriazis, minor mistake at the end of the function when making the spectrum. This one should work, but let me know if you run into any other issues:
OoA_2D_Af_Eu.txt

No worries! And thanks 👍

hey @ckyriazis- just checking on the status here as I've lost track. Is this fully implemented now? should we do a review?

Hey @andrewkern - I'm still working on a few things, including testing the inference under a two population Gutenkunst model, running the pipeline with the two population Drosophila model and IM inference (I didnt anticipate how long these simulations would take..), and then once I finish with that I plan to work on other stuff like adding the config file, cleaning up the code in some places, etc. We also need to think a bit more about what the ground truth parameters would be when doing inference with these different models so I can add those to the plotting functions.
So overall, still a fair amount to be done but I think we're getting close to having a finished pipeline for review. Will be helpful to talk tomorrow.

@ckyriazis Just a heads up, I've actually been trying to get things working for Drosophila for awhile. I have some msprime simulations that took 5 to 6 days with 2 samples per-population. I opened an issue in stdpopsim to check if I'm doing anything wrong.
The main thing that slows things down is the recombination map, so if you don't have much better luck with running simulations, it might be best to have the framework for Drosophila without the recombination map?

@tjstruck thanks for this input - I must have missed the issue you opened. I had been running a simulation for 20 samples and one chromosome under the two population model for over a week until I killed it today. @andrewkern seemed to think that there might be an issue with the recombination map, and said they'd try to look into that.
Another thing I noticed when looking through stuff again just now is that the simulations.py module that is used both by the n_t and two population pipeline imports only homo_sapiens from stdpopsim and not the other species. Could this be causing any problems?

I posted this reply to the issue raised by @tjstruck, but I'll post it here too.
"Andy and I have been looking into this, and we think this is standard behavior and not a bug. I had previously used msprime to simulate the same dmel recombination maps under equilibrium for a different project, and I was experiencing run times of ~1hr for simulating a single chromosome (2R) with Ne =2.5e5. The human models take significantly longer to run as well if you bump Ne up > 100,000. It might be worth considering abandoning recombination maps for models with very large Ne, if we will need to do many simulation reps."

Thanks @jradrion. Is everyone fine with running this without a recombination map?

@ckyriazis @tjstruck
 ts=msp.simulate(              sample_size=4,              Ne=2.5e5,              mutation_rate=2.8e-9,              recombination_map=recomb_map)
This example for Dmel chromosome 2R just took 85 mins to run on my machine. I'm going to test one rep of the full analysis pipeline on our cluster to see what sort of runtimes we might be looking at.

that's @jradrion. can we get similar runtime benchmarks for the demographic models in drosophila?

And how is the scaling with sample size? Four is pretty small.

I'm currently running a test to simulate chr2R under the Li & Stephan model for samples sizes 2,4,8,16,and 32. I will update with runtimes when the jobs finish.

Hey all - I just pushed my (hopefully) finalized changes for the two population pipeline for IM inference. I set it up to run with a config file and updated the readme to mirror the one in the n_t pipeline. It should work with both the human and drosophila two population models, but the drosophila one remains untested until we figure out what to do with the simulations.
There are a few remaining loose ends including:

The ground truth values to be included for the plots when inferring from the Gutenkunst and Li & Stephan models under an IM model. Hoping @petrelharp can provide some insight here.

I had some trouble automating the number of fsc runs so currently it is hard coded to do 10 dadi and fsc runs for each simulation replicate. Ideally I would like this to be something that can be changed in the config file, although it seems to converge fine for the IM model with just 10 runs.

Let me know what you all think and if there are any changes that need to be made. I wont have much time to work on this over the next couple weeks, but will be able to focus on it again starting on July 8. Tagging @dortegadelv here so you can start working on adding SMC++ to this pipeline.

We've got a working method for this over at https://github.com/tskit-dev/msprime/pull/779 once this is merged.
I'm still working through how to plug this in to this code, @ckyriazis, but here's the basic idea: to compute mean coalescence time for two samples from today in population 0 of the Gutenkunst model:
import msprime
from stdpopsim import homo_sapiens

model = homo_sapiens.GutenkunstThreePopOutOfAfrica()
ddb = msprime.DemographyDebugger(**model.asdict())
m00 = ddb.mean_coalescence_time([0,0,2], rtol=.01)
print(m00)
>> 18094.08790235354

Divided by 2, this is a natural thing to compare to as the "true" Ne.

Ok, we've (finally) got mean_coalescence_time() merged into msprime:
https://github.com/tskit-dev/msprime/commit/fc8ad0b5c8ff9c02211fb06bb1402e4fc98c5e49

Thanks @petrelharp! I'll play around with this when I have time in a week or so.

@ckyriazis this PR might now be broken with the changes to generation_time that I just merged. would you mind double checking that?

@andrewkern I just checked and I dont think the two population pipeline (as it currently is) needs the generation time specified anywhere.

@petrelharp I'm a bit unclear on how to go from these mean coalescence times to a point estimate of the 'truth' to compare to the inferred values under an IM model. If I understand correctly, this method prints out mean coalescence time for a population at any given generation (though how do you specify the generation?) but we would need some sort of average over the duration of each epoch in the IM model we're trying to fit, right?

Hey @petrelharp just following up here - could you help clarify how you think this method should be added to the pipeline?

Sorry, missed this.
And, I guess we hadn't thought this through sufficiently. If you have a method to infer Ne of a single population, then the natural thing to expect that it will infer is the mean time since coalescence between two genomes from the sample (so, we don't need to average over time or anything). But, of course, an IM model does something more complicated than this. Given all the mean pairwise coalescence times, we could work out the best-fitting IM model, although this is complicated and maybe nonunique and starts to be an inference method.
So, here's my proposal. We'd like to compare the inferred values to something because, well, that's the point? And when we're fitting a model that we know doesn't actually match the truth, what do we compare to? This is the situation we also have in real life, where we don't really believe the models; so we should compare the results to whatever it is about the real world that we'd like the methods to describe. And, maybe there's a few of these:
0. Census size, today. Why not?

Census size, averaged somehow over time.  But, it's an easy thing to get, so a good comparison. An alternative would be a harmonic mean over time, weighted by probability of not having yet coalesced.
Mean coalescence time. Also, why not? It's a useful point of reference.

I think it's worth thinking ahead to how we will write up the results here. What exactly will we want to say about the results?
tldr; I'm proposing just adding the mean coalescence time within each pop to the plot, as well as with other census-size-derived points of reference.

Census size, today. Why not?

Sure. Could easily do this for N(t) too

1. Census size, averaged somehow over time.  But, it's an easy thing to get, so a good comparison. An alternative would be a harmonic mean over time, weighted by probability of not having yet coalesced.

Is there an exponential weighting to population size going back in time that would be the logical average to take here? Sort of like when people do exponential moving averages on times series data, but fixed for our single observation at time=0?

It's not a very sophisticated metric, but I would also include the simulated versus inferred divergence time. In my experience, that tends to be the "headline" number from these IM model inferences. (Even when there is ongoing gene flow that muddies the definition.)

Is there an exponential weighting to population size going back in time that would be the logical average to take here?

You could weight by exp(-t/2Ne), but that'd just be a proxy for P(hasn't coalesced), which we can easily compute now...

also include the simulated versus inferred divergence time. In my experience, that tends to be the "headline" number

Oh, good point. Maybe we should really focus on this, where the comparison is clear.

Hey @ckyriazis just to ask a quick question, how does your config.json file looks like? I want to reproduce your results while adding the smc++ results. I particularly want to know which "model" you are using in your config.json file.
{
"seed" : 12345,
"num_sampled_genomes_per_replicate" : 20,
"replicates" : 10,
"species" : "homo_sapiens",
"model" : "GutenkunstThreePopOutOfAfrica",
"genetic_map" : "HapmapII_GRCh37",
"chrm_list" : "chr22",
"generation_time" : 30
}

Hey @dortegadelv, here is an example of one that I ran for testing:
{
"seed" : 12345,
"num_sampled_genomes_per_replicate" : 20,
"replicates" : 2,
"species" : "homo_sapiens",
"model" : "GutenkunstThreePopOutOfAfrica",
"genetic_map" : "HapmapII_GRCh37",
"chrm_list" : "chr22",
}
So yours looks essentially the same, except for that I didnt need to specify a generation time for either fsc or dadi.

Hey @petrelharp, I'm finally getting around to adding the mean_coalescence_time to the pipeline but am getting a:
AttributeError: 'DemographyDebugger' object has no attribute 'mean_coalescence_time'
even after updating msprime to version 0.7.1 using conda. Should I be installing directly from the source?

Hm, that's right: the function was added in this commit, after the 0.7.1 release. You can get it by installing from source; maybe we'll need to push another release of msprime?

We can push out 0.7.2 easily enough: there's been a low-level bugfix for some extremely rare events, so arguably we should make this available ASAP anyway. Just let me know.

Thanks @jeromekelleher, that would be helpful.

Msprime 0.7.2 has been released on PyPI @ckyriazis --- conda packages should follow in about a day.

We can close this branch now. #39  now supercedes this branch. The pipeline from #39 now runs fastsimcoal, smc++ and dadi taking into account any demographic history and number of chromosomes.

thanks @dortegadelv. I will close this PR.


beta(deps): bump @readme/variable peer version

🧰 Changes

[x] bump @readme/variable peer dep to v16.0.0

just gonna merge this since it's behind the beta flag and all tests are passing!

This PR was released!
🚀 Changes included in v6.75.0-beta.23


Unify timestamps

It says these changes can be "automatically merged", but of course I'd like you to review them.
These fulfill the recommendation "stick to YYYYMMDD_hhmmss_subsec format and use UTC. Everywhere from now on."

Great.  I also selected "delete branch", just to keep the clutter down.  Some might recommend otherwise.  I also delete my local branch, just so I don't confuse myself by making new changes in the wrong branch (or making a branch from an inappropriate branch, etc.)  The less choices I have, the less likely I'll have a misstep.


PR Request: Change output for multi-value headers in SocketClient

PR request from Alan:

Should multi-value headers be sent as
headername: headervalue1,headervalue2

or
headername: headervalue1
headername: headervalue2

?
Right, now this is doing the first case.

PR review can be found here.
Note: the constructor for HttpUrlConnectionResponse handles headers with multiple values. It might be worth checking if the same can be implemented in SocketClient.

Note: OkHttp does it the latter way:
sdk/core/azure-core-http-okhttp/src/main/java/com/azure/core/http/okhttp/OkHttpAsyncHttpClient.java#L134-L138

buildPatchRequest() changed to allow for multi-value headers similar to the OkHttp example. Thanks Craig


Internal blogpost

Hey team! Please add your planning poker estimate with Zenhub @BeierKevin @FloBaumi @roman533


Enable winopts for all commands with positioning relative to the cursor

Info

Operating System: Gentoo Base System release 2.9 (~amd64)

Shell: zsh 5.9

Terminal: wezterm 20221016-210733-84de038d

nvim --version:  NVIM v0.9.0-dev-115-g042eb74

fzf --version: 0.34.0 (04d0b02)

[ ] The issue is reproducible with mini.sh

fzf-lua configuration

local fzf_lua = require('fzf-lua')

local bottom_row = {
  height = 0.4,
  width = 1,
  row = 1,
  col = 0,
  preview = {
    layout = 'horizontal',
    horizontal = 'right:55%',
  }
}
local right_popup = {
  height = 0.97,
  width = 0.2,
  row = 0.3,
  col = 1
}
local cursor_popup = {
  relative = 'cursor',
  height = 0.2,
  width = 0.3
}
local right_column = {
  height = 1,
  width = 0.45,
  row = 0,
  col = 1,
  preview = {
    layout = 'vertical',
    vertical = 'down:65%'
  }
}

fzf_lua.register_ui_select({
  winopts = cursor_popup
})

fzf_lua.setup({
  global_resume = true,
  winopts = bottom_row,
  builtin = {
    winopts = right_column
  },
  colorschemes = {
    winopts = right_popup
  },
  diagnostics = {
    winopts = right_column
  },
  files = {
    prompt = 'Files❯ ',
  },
  git = {
    branches = {
      winopts = right_column
    },
    bcommits = {
      winopts = right_column
    },
    commits = {
      winopts = right_column
    },
    status = {
      winopts = right_column
    },
  },
  grep = {
    cmd = 'ugrep -RIjnkzs --hidden --ignore-files --exclude-dir=".git"',
    winopts = right_column
  },
  highlights = {
    winopts = right_column
  },
  lsp = {
    ui_select = true,
    code_actions = {
      winopts = cursor_popup
    }
  },
  spell_suggest = {
    winopts = cursor_popup
  }
})

Description
Is it possible to make winopts available for all commands?
After switching from Telescope I tried to mimic the get_cursor theme which is really useful for spell suggestions or code actions with a popup relative to the cursor. You can see my attempts to achieve that above but until now I didn't succeed (due to a lack of deeper understanding how things work ;))

Is it possible to make winopts available for all commands?

The winopts directly under setup is used as the default one for all commands (I.e. what you set to bottom_row), if you specify another winopts under a provider it will be merged onto the global one overriding values, e.g. the one under grep (right_column) will take precedence when running grep/live grep commands.winopts can also be specified as part of the command direct call which will then be merged with both the global and the provider one, I.e. :lua require'fzf-lua'.files({ winopts = … }).

After switching from Telescope I tried to mimic the get_cursor theme which is really useful for spell suggestions or code actions with a popup relative to the cursor.

The current window is hard coded as relative = editor, if you wish to change that to cursor I can have another relative field under winopts which will be relayed to nvim_open_win, let me know if this is what you need?

Thanks a lot for your detailed explanation 👍 So I understood the configuration options correct at least ;)
I'm wondering though why the winopts setting for spell_suggest in my config is not recognized? And the keybinding I'm using only works if I use a function:
vim.keymap.set('n', '<Leader>fs', function() require('fzf-lua').spell_suggest({ winopts = cursor_popup }) end, { desc = 'Fzf: spell suggest' }

A direct call uses the global winopt settings and opens a floating window at the bottom:
vim.keymap.set('n', '<Leader>fs', require('fzf-lua').spell_suggest, { desc = 'Fzf: spell suggest' })

That's still not quite clear for me. But at least I got a working configuration now. Many thanks for that!
And a relative field under winopts would be mightily appreciated! :D

Thanks a lot for your detailed explanation 👍 So I understood the configuration options correct at least ;)
I'm wondering though why the winopts setting for spell_suggest in my config is not recognized? And the keybinding I'm using only works if I use a function:

That’s because for some odd reason I decided to define some providers under a neovim section, this doesn’t make sense anymore so I think I’ll change it, for now you can just use setup({ nvim = { spell_suggest = { … }  })

And a relative field under winopts would be mightily appreciated! :D

Will add it and keep this thread updated.

https://github.com/ibhagwan/fzf-lua/commit/c90a88b354ae12736760f623000d91b438b04234 - update to this and spell_suggest should work directly under setup.

And a relative field under winopts would be mightily appreciated! :D

This turns out to be already working without any changes :-)
https://github.com/ibhagwan/fzf-lua/blob/c90a88b354ae12736760f623000d91b438b04234/lua/fzf-lua/win.lua#L624
For spell suggest you can use:
:lua require'fzf-lua'.spell_suggest({ winopts = {relative="cursor"} })

Note that using this breaks preview/scrollbar layout so use this only without preview.

Now that looks better already! :)
The last thing nagging is the row position of the popup: it's still on the bottom and not on the cursor position.
My current configuration
    use({
      'ibhagwan/fzf-lua',
      requires = {
        { 'junegunn/fzf', run = './install --all --xdg' }
      },
      config = function()
        local fzf_lua = require('fzf-lua')

        local bottom_row = {
          height = 0.4,
          width = 1,
          row = 1,
          col = 0,
          preview = {
            layout = 'horizontal',
            horizontal = 'right:55%',
          }
        }
        local right_popup = {
          height = 0.97,
          width = 0.2,
          row = 0.3,
          col = 1
        }
        local cursor_popup = {
          relative = 'cursor',
          height = 0.2,
          width = 0.3
        }
        local right_column = {
          height = 1,
          width = 0.45,
          row = 0,
          col = 1,
          preview = {
            layout = 'vertical',
            vertical = 'down:65%'
          }
        }

        local function show_notifications()
          local opts = {}
          opts.prompt = 'Notifications> '
          opts.fzf_opts = {
            ['--no-multi'] = ''
          }

          local entries = require('notify').history()
          local notifications = {}
          for i = 1, #entries do
            local all_messages = ''
            local formatted_messages = {}

            local function diag_level_code(diag)
              local level = entries[i].level
              if level == "ERROR" then
                return fzf_lua.utils.ansi_codes.red(diag)
              elseif level == "WARN" then
                return fzf_lua.utils.ansi_codes.yellow(diag)
              elseif level == "DEBUG" then
                return fzf_lua.utils.ansi_codes.yellow(diag)
              elseif level == "INFO" then
                return fzf_lua.utils.ansi_codes.green(diag)
              elseif level == "HINT" then
                return fzf_lua.utils.ansi_codes.blue(diag)
              end
            end

            local function format_msg(...)
              for _, entry in ipairs(...) do
                table.insert(formatted_messages, '\n' .. entry)
              end
              return formatted_messages
            end

            local messages = entries[i].message
            if #messages > 1 then
              format_msg(messages)
              all_messages = table.concat(formatted_messages)
            else
              all_messages = messages[1]
            end

            table.insert(notifications,
              string.format('%-5s  %s %16s %s %s',
                fzf_lua.utils.ansi_codes.blue(vim.fn.strftime('%F %H:%M', entries[i].time)),
                diag_level_code(entries[i].icon),
                diag_level_code(entries[i].level),
                fzf_lua.utils.ansi_codes.cyan('« ' .. entries[i].title[1] .. ' »'),
                all_messages
              )
            )
          end

          if vim.tbl_isempty(notifications) then return end
          fzf_lua.fzf_exec(notifications, opts)
        end

        vim.keymap.set('n', '<Leader>F', require('fzf-lua').builtin, { desc = 'Fzf: builtin' })
        vim.keymap.set('n', '<Leader>b', require('fzf-lua').buffers, { desc = 'Fzf: buffers' })
        vim.keymap.set('n', '<Leader>c', require('fzf-lua').colorschemes, { desc = 'Fzf: colorschemes' })
        vim.keymap.set('n', '<Leader>f', require('fzf-lua').files, { desc = 'Fzf: files' })
        vim.keymap.set('n', '<Leader>o', require('fzf-lua').oldfiles, { desc = 'Fzf: oldfiles' })
        vim.keymap.set('n', '<Leader>fg', require('fzf-lua').lgrep_curbuf, { desc = 'Fzf: grep current file' })
        vim.keymap.set('n', '<Leader>fG', require('fzf-lua').live_grep_native, { desc = 'Fzf: grep all files' })
        vim.keymap.set('n', '<Leader>h', require('fzf-lua').help_tags, { desc = 'Fzf: help' })
        vim.keymap.set('n', '<Leader>M', require('fzf-lua').man_pages, { desc = 'Fzf: man' })
        vim.keymap.set('n', '<Leader>m', require('fzf-lua').marks, { desc = 'Fzf: marks' })
        vim.keymap.set('n', '<Leader>r', require('fzf-lua').registers, { desc = 'Fzf: registers' })
        vim.keymap.set('n', '<Leader>fgb', require('fzf-lua').git_bcommits, { desc = 'Fzf: git buffer commits' })
        vim.keymap.set('n', '<Leader>fgc', require('fzf-lua').git_commits, { desc = 'Fzf: git commits' })
        vim.keymap.set('n', '<Leader>fgf', require('fzf-lua').git_files, { desc = 'Fzf: git files' })
        vim.keymap.set('n', '<Leader>fgs', require('fzf-lua').git_status, { desc = 'Fzf: git status' })
        vim.keymap.set('n', '<Leader>fc', require('fzf-lua').command_history, { desc = 'Fzf: command history' })
        vim.keymap.set('n', '<Leader>fm', require('fzf-lua').keymaps, { desc = 'Fzf: keymaps' })
        vim.keymap.set('n', '<Leader>fq', require('fzf-lua').quickfix, { desc = 'Fzf: quickfix' })
        vim.keymap.set('n', '<Leader>fs', require('fzf-lua').spell_suggest, { desc = 'Fzf: spell suggest' })
        vim.keymap.set('n', '<Leader>fw', require('fzf-lua').grep_cword, { desc = 'Fzf: grep string' })
        vim.keymap.set('n', '<Leader>fn', function() show_notifications() end, { desc = 'Fzf: notifications' })

        fzf_lua.register_ui_select({
          winopts = cursor_popup
        })

        fzf_lua.setup({
          global_resume = true,
          winopts = bottom_row,
          builtin = {
            winopts = right_column
          },
          colorschemes = {
            winopts = right_popup
          },
          diagnostics = {
            winopts = right_column
          },
          files = {
            prompt = 'Files❯ ',
          },
          git = {
            branches = {
              winopts = right_column
            },
            bcommits = {
              winopts = right_column
            },
            commits = {
              winopts = right_column
            },
            status = {
              winopts = right_column
            },
          },
          grep = {
            cmd = 'ugrep -RIjnkzs --hidden --ignore-files --exclude-dir=".git"',
            winopts = right_column
          },
          highlights = {
            winopts = right_column
          },
          lsp = {
            code_actions = {
              winopts = cursor_popup
            }
          },
          spell_suggest = {
            winopts = cursor_popup
          }
        })
      end
    })

The last thing nagging is the row position of the popup: it's still on the bottom and not on the cursor position.

Unfortunately that’s a bigger change I won’t be able to complete until I refactor the window class which is mentioned in #462, note that this type of refactor although planned is not on my immediate timeline.

Have you tried this? It pops up right at the cursor position
:lua require'fzf-lua'.spell_suggest({ winopts = {relative="cursor",row=0, col=0, height=0.2, width=0.5} }
)

This is simply brilliant 😃 ❤
It does precisely what I was imagining but couldn't realize myself :)
Thank you so much Sir! 🙏