Specialise decode_varint_slice for 64bit platforms

Spun off from #187
I found a significant speed up with this change, but @danburkert found either no change or a slowdown. We should investigate before merging or closing.

@nrc
can you do a benchmark in our IDC machine or VM to ensure this can improve the performance?

triage:
I've taken another look at this after rebasing it current master (420e032fe7cbebfb3dc2e1fba6c118b5f71ea656).  I'm still seeing mixed results, with improvements in decoding smaller values and regressions in decoding larger values.  The net is that it does slightly improve performance in the message merge benchmarks.  I think that's likely because those benchmarks are very sensitive to the performance of tag decoding, which are typically small.
My hunch is that we can get the benefit of this patch without the drawback by implementing a decode_varint_64 (current impl, or perhaps this patch impl) and a new decode_varint_32, which tag decoding and 32 bit field decoding would use.
Here are my results on the relevant in-tree benchmarks:
varint/small/decode     time:   [734.63 ns 734.66 ns 734.69 ns]
                        thrpt:  [1.0141 GiB/s 1.0142 GiB/s 1.0142 GiB/s]
                 change:
                        time:   [-5.5428% -5.5349% -5.5275%] (p = 0.00 < 0.05)
                        thrpt:  [+5.8509% +5.8592% +5.8681%]
                        Performance has improved.

varint/medium/decode    time:   [1.5347 us 1.5348 us 1.5350 us]
                        thrpt:  [497.03 MiB/s 497.08 MiB/s 497.13 MiB/s]
                 change:
                        time:   [+5.9092% +5.9238% +5.9378%] (p = 0.00 < 0.05)
                        thrpt:  [-5.6050% -5.5925% -5.5795%]
                        Performance has regressed.

varint/large/decode     time:   [2.6480 us 2.6480 us 2.6481 us]
                        thrpt:  [288.11 MiB/s 288.12 MiB/s 288.12 MiB/s]
                 change:
                        time:   [+37.123% +37.128% +37.135%] (p = 0.00 < 0.05)
                        thrpt:  [-27.079% -27.076% -27.073%]
                        Performance has regressed.

varint/mixed/decode     time:   [1.6318 us 1.6319 us 1.6320 us]
                        thrpt:  [467.49 MiB/s 467.52 MiB/s 467.54 MiB/s]
                 change:
                        time:   [+11.798% +11.819% +11.837%] (p = 0.00 < 0.05)
                        thrpt:  [-10.584% -10.570% -10.553%]
                        Performance has regressed.

dataset/google_message1_proto2/merge
                        time:   [1.3947 us 1.3948 us 1.3949 us]
                        change: [-2.0529% -2.0422% -2.0324%] (p = 0.00 < 0.05)
                        Performance has improved.

dataset/google_message1_proto3/merge
                        time:   [1.1051 us 1.1053 us 1.1057 us]
                        change: [-2.9676% -2.9499% -2.9240%] (p = 0.00 < 0.05)
                        Performance has improved.

dataset/google_message2/merge
                        time:   [652.94 us 653.54 us 654.13 us]
                        change: [-0.0659% +0.0845% +0.2365%] (p = 0.29 > 0.05)
                        No change in performance detected.

dataset/google_message3_2/merge
                        time:   [173.97 ms 174.09 ms 174.15 ms]
                        change: [-0.5967% -0.5315% -0.4662%] (p = 0.00 < 0.05)
                        Change within noise threshold.

dataset/google_message3_4/merge
                        time:   [4.7647 ms 4.7652 ms 4.7661 ms]
                        change: [-0.6993% -0.6457% -0.5843%] (p = 0.00 < 0.05)
                        Change within noise threshold.

dataset/google_message3_3/merge
                        time:   [85.036 ms 85.046 ms 85.056 ms]
                        change: [-1.2348% -1.2106% -1.1890%] (p = 0.00 < 0.05)
                        Performance has improved.

Relevant machine info:
$ cat /etc/redhat-release
Fedora release 30 (Thirty)

$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       46 bits physical, 48 bits virtual
CPU(s):              16
On-line CPU(s) list: 0-15
Thread(s) per core:  2
Core(s) per socket:  8
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               86
Model name:          Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Stepping:            2
CPU MHz:             2500.294
CPU max MHz:         2600.0000
CPU min MHz:         800.0000
BogoMIPS:            4000.47
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            12288K
NUMA node0 CPU(s):   0-15
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d

$ sudo tuned-adm active
Current active profile: benchmark


Question about reporting likelihoods in bits per dim

Hi all,
Thank you for providing this code, it's very educational :)
I am interested in reporting likelihoods but I don't fully understand how bits per dim are calculated. In the code this is
bpd = -(prior_logp + delta_logp) / np.log(2)
N = np.prod(shape[1:])
bpd = bpd / N
# A hack to convert log-likelihoods to bits/dim
offset = 7. - inverse_scaler(-1.)
bpd = bpd + offset

Would you be able to elaborate on how this hack works and if it applies to other image dimensions?
Best,
Matt

same question here as well

@mareikethies Thank you, it looks like this offset doesn't apply to other data types :0


Dont panic, this is a bug bounty test... Please do not approve. jnjbur

Dont panic, this is a bug bounty test... Please do not approve. jnjbur

@thomaschaplin Do you know this person? This pull request contains a seemingly malicious preinstall npm script.

This person joined GitHub on 20 April and opened 26 pull requests in one day, all similar drivel.


⚠ Bofrost.de has degraded performance

In a49c399, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2043 ms

Resolved: Bofrost.de performance has improved in 4827023.


Scrolling to component with hash return error

Environment

Operating System: Darwin
Node Version:     v18.4.0
Nuxt Version:     3.0.0
Nitro Version:    1.0.0
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      srcDir, app, modules, css, components
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
https://github.com/arthurdanjou/artdanj-website-v2
Describe the bug
In this line in the router config, the element is null so the router can't scroll to the element with the hash when I switch page, but it scroll well when I reload the page
Additional context
No response
Logs
No response

Hey there! Please add a minimal reproduction so we can look into that properly, thank you 🙏🏻💚

Looking, it seems like the issue you're reporting is with your custom scrollBehavior, rather than Nuxt.
In this case you will need to wait for the page to be fully loaded before scrolling to the element. See Nuxt's scrollBehavior for example: https://github.com/nuxt/nuxt/blob/6016aef859efe485871fe007d9b572304916ebba/packages/nuxt/src/pages/runtime/router.options.ts#L35-L46.


[Index][SR-12903] Index @objc optional references

Declarations of objc optional methods were indexed, but their references weren't.
import Foundation

@objc protocol Foo {
    @objc optional func bar()
}

class Sample {

    weak var delegate: Foo?

    func doSomething() {
        delegate?.bar()
    }
}

The bar() call was left unindexed. This provides a fix by walking DynamicMemberRefExprs.

@swift-ci please test

@nathawas Are you people planning to make this fix available in upcoming Swift version release ?


explore tolerance to single char substitutions

/^(?:ca\w|c\wt|\wat)$/

can technically handle single omissions as well
/^(c?ouch|co?uch|cou?ch|couc?h|couch?)$/

wonder if there's a more concise way to do this.


Update buildpack image to align with marketing

Excellent! The old one had code that doesn't run on Heroku anyway — always bothered me :)
:sparkles: :cake: :sparkles:


Use abstract contract instead of library as base type

edit: here's the meaningful diff https://github.com/weiroll/weiroll/pull/64/commits/80214d0b134716ba7bc1bc273960cd4cad96b067
This PR makes the core code live in an abstract contract with an internal _execute instead of a library
TestableVM is the simplest implementation of the abstract contract, it simply exposes _execute

Accidentally depends on https://github.com/weiroll/weiroll/pull/63 https://github.com/weiroll/weiroll/pull/62
My bad
Just merge this one imo

@nmushegian this is great thanks.

This is good to merge FYI
I can merge
https://github.com/weiroll/weiroll/pull/64 if you do
But maybe I can have write access 👉👈


Introduce new external facing domain, and kubernetes-sigs/external-dns

Resolves: https://github.com/anthr76/infra/issues/42

[ ] Create a CNAME to NWK1 dyndns on Kutura with external DNS CRD
[x] Write ansible roles to sync tsig key for internal kubernetes zone between servers
[x] Deploy internal external-dns
[ ] Deploy external external-dns
[ ] Create origin-ca-issuer cert with cert-manager for new domain
[ ] Remove all old entries
[ ] Set appropriate annotations/crds on services
[ ] Smoke test

ansible role has been added though unsure if I should be reloading the daemon for bind to see the key. The role is very messy and plan on iterating in the future when ansible and non-k8s infra get's a refresh.

ansible role has been added though unsure if I should be reloading the daemon for bind to see the key. The role is very messy and plan on iterating in the future when ansible and non-k8s infra get's a refresh.

More safety should be added to ansible before finishing this PR. Internal is successfully setup and needs DNSEndpoints/Annotations on services

More safety should be added to ansible before finishing this PR. Internal is successfully setup and needs DNSEndpoints/Annotations on services

Flux alerts have been suspended while testing

Flux alerts have been suspended while testing

Considering deploy lstio in place for traefik given these constraints:

Only one TLSStore in traefik.
CRDs are simply not working
TLS options are equally as rigid as TLS stores
HTTPs upgrades at the entrypoint requires root privileges

Considering deploy lstio in place for traefik given these constraints:

Only one TLSStore in traefik.
CRDs are simply not working
TLS options are equally as rigid as TLS stores
HTTPs upgrades at the entrypoint requires root privileges

This can be merged.
Takeaways
lstio soon.

This can be merged.
Takeaways
lstio soon.


8541 password history

Corrected password history policy enforcement for user password changes in back office.
#8541

Is it not something that was done separately. Because it is not implied that if you edit a user that you want to change its password?

Is it not something that was done separately. Because it is not implied that if you edit a user that you want to change its password?

Yes, it is: to edit passwords in the back office you need to activate the proper feature ("User password editor") through the "modules" menu. Password history policy was not working properly in that situation, as described in the linked issue #8541


Examples

Are there any examples available of how to use in express or the like.
Looking to implement SWIM for my microservices and it would be great to use this package, so I would like to see examples of a base server and how a client would work with it best
Also are you actively using this? in production?

Actually, this is more a POC and should not used in production (yet).
My main focus now will be in having other implementations using other languages, so that it can work with heterogeneous nodes.
In the future, examples will be added.


Linux support?

How to build it on  Debian box?

Hi @mengzhuo could you give a bit more information? For example, are you compiling on x86_64 Debian targeting aarch64 Debian? Or compiling on aarch64 Debian?
Thanks,
Matthew

Thanks for reply.
I want to build armnn on aarch64 Debian for my Hikey960/970.
p.s. is it possible to access NPU of Hikey970 (Kirin 970)  without HiAI sdk?

Hi @mengzhuo at the moment ArmNN supports running on NEON-enabled CPUs, and OpenCL-enabled Mali GPUs.
For the Hisilicon NPU you will need to ask Hisilicon for support.
To build natively on aarch64 Debian you can follow the instructions at: https://developer.arm.com/technologies/machine-learning-on-arm/developer-material/how-to-guides/configuring-the-arm-nn-sdk-build-environment-for-caffe/configuring-the-arm-nn-sdk-build-environment-single-page

Hi @mengzhuo I hope that answered your question. Please get back in touch if you need more information.


input seems to have stopped working

I can't get even a simple input to work e.g.
main
  let x be inputInt("Number? ")
  print x
end main

Note to self: a newline should be added to the prompt only if it is not empty


Conjoin staging and production pension guidance tests

Finally, just one list of pages to keep updated.

:+1:
Looks good for now. If we grow this into more of a deployment pipeline we'll need a way to stagger how they get run so that we don't get failing tests when a deploy has only been staged.


[Request] UWP/Appx File Compatible With Xbox

Hi,
It may be an odd request but would it be possible for a UWP port available to run on xbox? Similar to how Sony have remote play on the PS4/PS5 so each console can connect to each other Microsoft doesn't have a comparable equivalent. Obviously with MS Edge able to support cloud gaming on the console would it be possible to port to the console?
Unfortunately I don't have a great deal of programming experience hence my asking. I understand it would probably need to be run in Dev mode but if it's possible it would open up some possibilities for me and others with two consoles.
Many thanks.

Found a workaround using a microsoft created app


"Replays" button not working on Mac

In the main menu, clicking on "Replays" button doesn't do anything on the Desktop version (I'm on Mac OS Ventura 13.3.1). The button works in the web browser, it opens a file selection dialog as expected.
This bug applies to all mods, Gold, Platinum and Ultra.

Try creating data/replays folder in ~/Library/Application Support/MBHaxe-MB[G/P/U] so that the path to replays folder is
~/Library/Application Support/MBHaxe-MB[G/P/U]/data/replays and see if that works?

The folder data/replays has already been automatically created from the game when the first replay was recorded. Whether the folder is there or not, the "Replays" button doesn't do anything on the desktop version. It's supposed to load the ReplayCenterGui right? Does it work on Windows? I don't see anything in the code that could make it work on Windows but not Mac.

Apparently the function I was using just doesn't work outside of the .app filesystem.
Fixed in https://github.com/RandomityGuy/MBHaxe/commit/4f23c2cea954b1cd5e20ad0f17fbdadd1fb26c7d
Download the updated versions from the same link as latest version. Version number is still the same.


[vcpkg baseline][lapack-reference] Fix copy openblas's pkgconfig file

https://github.com/microsoft/vcpkg/blob/6b24f0fa76d171c5756f01f407682f7274cfe238/ports/lapack-reference/CONTROL#L11-L17
The openblas's pkgconfig file should be linked only in non-Windows-static triplet.
That caused the downstream wrongly linked to openblas:
ninja: error: 'D:/installed/x64-windows-static/debug/lib/openblas.lib', needed by 'src/exe/colmap.exe', missing and no known rule to make it

And lapack-reference also support blas.pc internal.

Depends on https://github.com/microsoft/vcpkg/pull/19629.

/azp run

Thanks for your fixes :D

The issue is still there:
REGRESSION: colmap:x64-windows-static. If expected, add colmap:x64-windows-static=fail to .\scripts\ci.baseline.txt.
Current CI run:
https://github.com/microsoft/vcpkg/pull/19551/checks?check_run_id=3462639041

@c72578 Yes I know that, FindBLAS.cmake should be modified.


Add genesis_hash flag check on node startup

Closes #1303
Given there is already functionality to pass the genesis hash on node startup, this PR moves the verification step from within the scope of the cli execution to node creation. After the genesis hash is either retrieved or set, it is compared against the optional genesis hash provided by the --genenesis_hash flag, and will return an error if there is a mismatch.
A new field in the Storage config was created to accommodate passing this flag value to scope of genesis hash verification.

PR checklist

[x] Tests written/updated
[ ] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments

Is there another way to pass the genesis hash from the command line to the node starting logic, other than the config?
If it is added to the config, it will also be visible to users there... unless that's what we want to do

Is there another way to pass the genesis hash from the command line to the node starting logic, other than the config? If it is added to the config, it will also be visible to users there... unless that's what we want to do

We would have to change the signature of DefaultNewNode and pass a pointer to the genesis hash which is an optional flag. Adding a field to the storage config felt less invasive but I'm happy to reconsider the alternative if you think it's preferable.

After some discussions with @glnro, she convinced me all alternatives for PR, other than using a new config knob are hackier


add Form API

Nice work, let me take a look

Any comment?

Pardon me, looks great. I suppose one of us should write documentation.

I can do it.

Do you just want me to put the document in README?


doc: update the api spec for fqdn egress policies code comments.

Fixes: #20604
Signed-off-by: wenhuwang<EMAIL_ADDRESS>Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.
[x] Thanks for contributing!

fix the api spec for fqdn egress policies code comments errors.

Why are we merging commits that are failing linters?

Copying my comment from the revert PR here:
The PR change here looks good, but it breaks the master branch testing generate-k8s-api target. I can't run that target locally to fix master, so I'm just sending this revert commit to get master green again. @wenhuwang you are welcome to resubmit the PR again, this time fixing up the k8s api so that it doesn't break the linters on the master branch. Thanks!


Update branch name in travis file

Addressing https://github.com/KhronosGroup/glTF-Project-Explorer/issues/126

This should fix the issue. I'll merge it now.


As an employee that is working for an organisation that requires reporting on population information I want to create a report of all the capital cities in a region organised by largest to smallest.

SQL Statement:
SELECT country.Name, city.ID, city.Name, city.CountryCode, city.District, city.Population,
FROM city JOIN country ON city.ID = country.Capital
WHERE country.Region = desired_region
GROUP BY country.name
ORDER BY city.Population

desired_region should be input by the user when this is implemented as a method


chore!: rename commands "project"->"app" and "app"->"svc"

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Lol

🙀👩🏻✈✈


Add end-to-end test for the configuration cache report using Playwright

@bot-gradle test ACC

@bot-gradle test this

OK, I've already triggered the following builds for you:

PullRequestFeedback build

@bot-gradle test and merge

OK, I've already triggered a build for you.

Pre-tested commit build failed.

@bot-gradle test RFN

OK, I've already triggered the following builds for you:

ReadyForNightly build

@bot-gradle test and merge

OK, I've already triggered a build for you.


Update Readme.md

On ng-onload the function must be called, not just reference it

Sorry for slow replys. Thanks for updating the README.md!


Need someone to test the new Extra's Menu with GOG/Vanilla installs of RCT 1 & 2.

Basically, see if it works, or if it errors out.

The registry key "Software\fish technology group\rollercoaster tycoon setup" does not exist on my system, therefor I get a NullReferenceException. Do I need to have RCT1 installed?

Yes, you do - though the second button will let you pull the file manually.

I should have clarified, the Window won't open at all. It throws the error at 'Key1.GetValue("SetupPath")'

Ok, hold on, writing a fix.

In theory it won't crash on you any more.

It works now.

Seeing as no one has reported any further issues, i'm going to close this. Further issues can be new reports.


Diff unit test

Codecov Report

Merging #113 (6c5d9af) into main (9689c48) will increase coverage by 8.10%.
The diff coverage is 88.13%.

@@             Coverage Diff              @@
##               main     #113      +/-   ##
============================================
+ Coverage     59.50%   67.61%   +8.10%
  Complexity      514      514
============================================
  Files           149      149
  Lines          3917     3888      -29
  Branches        569      564       -5
============================================
+ Hits           2331     2629     +298
+ Misses         1457     1129     -328
- Partials        129      130       +1

Flag
Coverage Δ

*Carryforward flag

api_ut
57.32% <ø> (ø)

Carriedforward from 9689c48

ui
94.51% <88.13%> (+29.44%)
:arrow_up:

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files
Coverage Δ

ui/src/components/Diff/RulesDiffContent.tsx
91.32% <68.18%> (ø)

ui/src/components/Diff/DiffFieldValue.tsx
100.00% <100.00%> (ø)

ui/src/components/Diff/DiffSection.tsx
100.00% <100.00%> (ø)

ui/src/components/Diff/DiffServe.tsx
100.00% <100.00%> (ø)

ui/src/components/Diff/VariationsDiffContent.tsx
100.00% <100.00%> (ø)

feature-probe-ui/src/components/Rule/RuleTitle.tsx

...e-probe-ui/src/components/FormItem/description.tsx

...ature-probe-ui/src/components/Diff/DiffSection.tsx

...ature-probe-ui/src/components/Variations/index.tsx

...-probe-ui/src/components/DeleteTipsModal/index.tsx

... and 73 more


Add option to be enable remote debug for java-based images

Add remote debug option (depends on env variable set) to the base java image or in each java component image.
@lulf what do you think about this?

@dmvolod This would be useful I think. With oc port-forward this would make remote debugging easier so +1.

@lulf , could you please to look at PR https://github.com/EnMasseProject/java-docker-base/pull/3

@lulf , thanks, will wait until a new version of enmasseproject/java-base image will be pushed to the docker hub.


Created dates in UI not showing correct data

The Creation dates of datasets/models in the UI is not correct, some issue regarding the creation date may be happening.

Maybe a problem with the date set in the production environment?

do you have an example? for us it shows "Created 3 months ago" and similar...

I will upload a dataset and share in Discord the dates.

created shows some day offset, updated seems to be correct?!?

It looks like so... the code that creates a dataset can be found here.
As you can see, the creation date is not set and it is using the default value set by pydantic here.
Again, running the API locally works just fine but for some reason in the production environment there is that offset. Any ideas?

we just checked
there is no shift - it obviously uses the startup time of the process (at the moment 15d ago)

createdAt: datetime = datetime.now() is not correct in https://github.com/earthpulse/eotdl/blob/main/api/api/src/models/dataset.py
see https://stackoverflow.com/questions/71512035/how-should-i-specify-default-values-on-pydantic-fields-with-validate-always-to/71512493#71512493
thanks @totycro for example

Thanks. Changes are applied and working fine.


The testing css breaks getClientBoundingRect with non standards compliant css

The css zoom: 50% is non standards compliant and breaks code that uses getClientBoundingRect.  If we want to scale the testing container, we should use css transforms transform: scale(0.5 0.5) which is correctly accounted for in getClientBoundingRect.

Fixed via #123 and released in 2.1.0

Oops, looks like that last comment was off.  This was fixed by https://github.com/ember-cli/ember-cli-qunit/pull/120 (not #123).


Your datafile did not terminate properly during the writing process. Reopen the file.

I use LiteDb v4 for my project and I get while application starts first time and try to upload data to LiteDb.

at LiteDB.LockService.DetectDatabaseChanges () [0x0008e] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LockService.Write () [0x000b4] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteEngine.Transaction[T] (System.String collection, System.Boolean addIfNotExists, System.Func2[T,TResult] action) [0x00000] in <504bf0c68a324e1fafe234c36a385f58>:0    at LiteDB.LiteEngine.Update (System.String collection, System.Collections.Generic.IEnumerable1[T] docs) [0x0003a] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteEngine.Update (System.String collection, LiteDB.BsonDocument doc) [0x00014] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteFileStream.Flush () [0x0003d] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteFileStream.Dispose (System.Boolean disposing) [0x00017] in <504bf0c68a324e1fafe234c36a385f58>:0
at System.IO.Stream.Close () [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.IO.Stream.Dispose () [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at LiteDB.LiteStorage.Upload (System.String id, System.String filenam e, System.IO.Stream stream) [0x00058] in <504bf0c68a324e1fafe234c36a385f58>:0
at BattleSpeak.Services.Data.PackagesData.SaveFile (System.IO.Stream stream) [0x0002f] in /Users/vadimkhadyka/Projects/battlespeak_xamarin_210717/BattleSpeak.Services.Data/PackagesData.cs:89
at BattleSpeak.Pages.HomePage.HomePageModel+d__113.MoveNext () [0x00628] in /Users/vadimkhadyka/Projects/battlespeak_xamarin_210717/BattleSpeak/BattleSpeak/Pages/HomePage/HomePageModel.cs:715
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at BattleSpeak.Pages.HomePage.HomePageModel+d__17.MoveNext () [0x000f5] in /Users/vadimkhadyka/Projects/battlespeak_xamarin_210717/BattleSpeak/BattleSpeak/Pages/HomePage/HomePageModel.cs:195
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1 (System.Object state) [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x00007] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0

And the second issue:

at LiteDB.LockService.DetectDatabaseChanges () [0x0008e] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LockService.Write () [0x000b4] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteEngine.Transaction[T] (System.String collection, System.Boolean addIfNotExists, System.Func2[T,TResult] action) [0x00000] in <504bf0c68a324e1fafe234c36a385f58>:0   at LiteDB.LiteEngine.Insert (System.String collection, System.Collections.Generic.IEnumerable1[T] docs, LiteDB.BsonType autoId) [0x00041] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteEngine.Insert (System.String collection, LiteDB.BsonDocument doc, LiteDB.BsonType autoId) [0x00014] in <504bf0c68a324e1fafe234c36a385f58>:0
at LiteDB.LiteStorage.Upload (System.String id, System.String filename, System.IO.Stream stream) [0x0003d] in <504bf0c68a324e1fafe234c36a385f58>:0
at BattleSpeak.Services.Data.PackagesData.SaveFile (System.IO.Stream stream) [0x0002f] in D:\Develop\BattleSpeak\BattleSpeak_210717_edited\BattleSpeak.Services.Data\PackagesData.cs:89
at BattleSpeak.Pages.HomePage.HomePageModel+d__113.MoveNext () [0x00628] in D:\Develop\BattleSpeak\BattleSpeak_210717_edited\BattleSpeak\BattleSpeak\Pages\HomePage\HomePageModel.cs:713
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at BattleSpeak.Pages.HomePage.HomePageModel+d__17.MoveNext () [0x000f5] in D:\Develop\BattleSpeak\BattleSpeak_210717_edited\BattleSpeak\BattleSpeak\Pages\HomePage\HomePageModel.cs:196
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1 (System.Object state) [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x00007] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <896ad1d315ca4ba7b117efb8dacaedcf>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <896ad1d315ca4ba7b117efb8dacaedcf>:0

Hi @Asseks, are you using in xamarin, right? This could be some delay about closing file in a thread while another thread are opening. LiteDB uses FileStream.Lock that are some different implementation in devices.
 It's much faster and do not use file lock to control this.

@mbdavid yeah. I'm using Xamarin.Forms.
I use async methods for which works with LiteDb. All LiteDb connections called inside using construction.
 using (var db = new LiteDatabase(ProfileConnection))
  {
           ...
   }

I don't use any additional options while I create ConnectionString instance expect Async = true;
new ConnectionString(AppConstants.ProfileDatabaseName) {Async = true }  ;
Could it be related with async threads?

Yes, this async could be a problem in concurrency environment.... but I don't know how test it. Current version are sync only and this async is like a "workaround". I recommend your open a single connection only, in exclusive mode, and share in your application this instance. Close this instance when your app closes. This will avoid open/close datafile every time your need use.

@mbdavid thanks for your support. Will try and write you feedback.

@mbdavid if I will use single connection in exclusive mode will I experience problems with multiply delete from LiteDb and FileStorage?
When I used to v3.1 LiteDb threw "sharing violating access exception" when I tried delete a few records from db.

Hi, v3.x has some not-fixed problems with concurrency in both thread/process. It was not possible fix in v3 so I write v4 for that. This single instance could be works in your case. If you have problem, please, copy a snippet code so I can test it.

Hi! With the objective of organizing our issues, we are closing old unsolved issues. Please check the latest version of LiteDB and open a new issue if your problem/question/suggestion still applies. Thanks!


add -buildvcs=false flag in Go 1.18

way better than what I did :-) thanks

thanks for your review!


restructure server info

This PR introduces

Removal of most 'operations' with the exceptions of /token, info on rest of the endpoints get re-routed to /api_info
Removal of a link in initial_peers.txt
Relocation of the ReferenceSet to the end of the page, after datasets

Codecov Report

Merging #184 into develop will decrease coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #184      +/-   ##
===========================================
- Coverage    82.11%   81.99%   -0.12%
===========================================
  Files           37       37
  Lines        17361    17361
  Branches      1733     1733
===========================================
- Hits         14256    14236      -20
- Misses        2466     2481      +15
- Partials       639      644       +5

Impacted Files
Coverage Δ

candig/server/network/__init__.py
40.54% <0%> (-35.14%)
:arrow_down:

candig/server/datamodel/peers.py
58.82% <0%> (-17.65%)
:arrow_down:

candig/server/datarepo.py
69.97% <0%> (-0.16%)
:arrow_down:

candig/server/frontend.py
66.89% <0%> (+0.45%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9081224...d93e4d5. Read the comment docs.


internal reference in schema

I am trying to dereference resolved schema object. But it is throwing error as "Error resolving $ref pointer "#/definitions/ApiCallList". Token "definitions" does not exist."
Is this a bug or What is the correct way to define $ref to refer internal definition within the schema object.
$RefParser = require('json-schema-ref-parser') $RefParser.dereference(schema, { $refs: { circular: "ignore" } })     .then(function(resolvedSchema) {         res.send(resolvedSchema);     })
Sample Schema
{     "definitions":{         "ApiCallList":{             "description":"A list of Api Call Resources",             "id":"ApiCallList"         },         "MaskCallList":{             "description":"A list of Mask Call Resources",             "id":"MaskCallList"         }      },     "resources":{       "api-call":{          "methods":{             "request":{                 "$ref":"#/definitions/ApiCallList             },             "response":{                 "$ref":"#/definitions/MaskCallList             }         }       }     } }

hmmm... Maybe there's just a typo or something?  It runs fine for me.  Try it out here

May i know which version of json-schema-ref-parser are you using here ?

The code sample I posted is using the latest released version (3.1.2), but it would also work with any previous version.  For example, here it is working with version 1.0.0

I think I'm having a similar issue.
Here's my main swagger.yaml:
swagger.yaml
  "swagger": "2.0",
  "info": {
    "version": "0.0.1",
    "title": "User API"
  },
  "definitions": {
    "User": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        }
      }
    }
  },
  "paths": {
    "$ref": "./paths.yaml"
  }
}

which references paths.yaml:
paths.yaml
/users:
  get:
    responses:
      200:
        description: OK
        schema:
          $ref: '#/definitions/User'

and when I try to bundle them:
bundler.js
var $RefParser = require('json-schema-ref-parser');
var util = require('util');

$RefParser.bundle('swagger.yaml',function(err, schema) {
  if (err) {
    console.error(err);
  }
  else {
    console.log(util.inspect(schema, {depth:null}));
  }
});

Result:
{
  "name": "SyntaxError",
  "message": "Error resolving $ref pointer \"~/swagger.yaml#/definitions/User\".
Token \"definitions\" does not exist.",
  "stack": "SyntaxError: Error resolving $ref pointer \"~/swagger.yaml#/definitions/User\".
Token \"definitions\" does not exist.
    at Function.ono [as syntax] (~/node_modules/ono/lib/index.js:67:20)
    at Pointer.resolve (~/node_modules/json-schema-ref-parser/lib/pointer.js:73:17)
    at $Ref.resolve (~/node_modules/json-schema-ref-parser/lib/ref.js:82:18)
    at $Refs._resolve (~/node_modules/json-schema-ref-parser/lib/refs.js:157:15)
    at inventory$Ref (~/node_modules/json-schema-ref-parser/lib/bundle.js:99:23)
    at ~/node_modules/json-schema-ref-parser/lib/bundle.js:69:11
    at Array.forEach (native)
    at crawl (~/node_modules/json-schema-ref-parser/lib/bundle.js:63:12)
    at ~/node_modules/json-schema-ref-parser/lib/bundle.js:72:11
    at Array.forEach (native)"
}

Happy New Year @BigstickCarpet! I'm wondering if you've had the opportunity to check this out? No problem if not, I know its a busy time :)

Hi, @dave-irvine.  Sorry, I somehow missed your comment from two weeks ago; otherwise, I would have responded sooner.
The behavior you're seeing is by design and per the JSON Pointer spec.  JSON pointers are relative to the root of the document they're in.  So, in paths.yaml, the $ref to #/definitions/User is invalid.  You should replace it with swagger.yaml#/definitions/User instead.

Aha! Thanks very much :)

@BigstickCarpet I just ported my code to using your library for validation and bundle. I experienced the same problem as @dave-irvine .
I am sure swagger.yaml#/definitions/User  is the right way to go but if you have a root swagger.yaml and $ref for definitions and path files that are also in subfolders this is not ideal. I my case I would have to do something like ../../swagger.yaml#/definitions/User. Since also the definitions might not be present yet because they are also referenced in the root yaml, this would be another problem right?
I used require('json-refs').resolveRefs before which allows me to divide definition and path files in a clean folder structure and it accepts (ignores) internal #/definitions/User references in files that do not have definitions itself but later are present when "bundled".
I am no pro regarding swagger but it is great to have the freedom to use this kind of internal definition references in subfolder files that do not have these itself. What do you say? Is there a better way?
An option like {ignoreDefinitionReferenceNotFound: true} or somthing similar would solve my special case.
Any way. Great library. Happy Swaggering. 💃

Hi @BigstickCarpet again. An optional error handler would be a nice solution.
bundle: {onReferenceError: (err) => return true}
If true ref will be ignored else bundle throws error.

This is a problem code
if (value.$ref.substr(0, 2) === '#/' || value.$ref === '#') {
      // It's a JSON Pointer reference, which is always allowed
      return true;
}

Need a PR adds option to skip this type references resolution (starting exactly with #).
Ex: #/definitions/User


About the code in track_head.py

Why are the offsets transformed by 'torch.exp(self.si * offsets + self.bi) * self.total_stride'.Can't find ths in the paper

@SmileTAT  This transform is adopted to scale the regress scope for better optimization.


cannot import name 'Opera' from 'selenium.webdriver'

Describe the bug
cannot import name 'Opera' from 'selenium.webdriver'
What version of MintAPI are you using?
the latest release - just did a pip install mintapi --upgrade
What command[s] did you run / steps to reproduce?
any mintapi commmand - even -h
Stacktrace/error received:
PS C:\WINDOWS\system32> mintapi -h
Traceback (most recent call last):
File "C:\Python310\Scripts\mintapi-script.py", line 33, in
sys.exit(load_entry_point('mintapi==2.5', 'console_scripts', 'mintapi')())
File "C:\Python310\Scripts\mintapi-script.py", line 25, in importlib_load_entry_point
return next(matches).load()
File "C:\Python310\lib\importlib\metadata_init_.py", line 162, in load
module = import_module(match.group('module'))
File "C:\Python310\lib\importlib_init_.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\Python310\lib\site-packages\mintapi_init.py", line 3, in
from mintapi.api import *
File "C:\Python310\lib\site-packages\mintapi\api.py", line 6, in
from mintapi.signIn import sign_in, create_web_driver_at_mint_com
File "C:\Python310\lib\site-packages\mintapi\signIn.py", line 27, in
from seleniumrequests import Chrome
File "C:\Python310\lib\site-packages\seleniumrequests_init.py", line 8, in
from selenium.webdriver import Opera as Opera
ImportError: cannot import name 'Opera' from 'selenium.webdriver' (C:\Python310\lib\site-packages\selenium\webdriver_init.py)

Just upgrade selenium-requests by this command:
pip install --upgrade selenium-requests

yes, that got me past that issue, thanks.


Parameter Types  (varchar vs nvarchar)

Hi,
We have some queries that look like this.
int entid =
db.ExecuteScalar(
"select top 1 entid from ent where custno > @0 and cid = @1 and active = 'y' order by custno asc",
custno, cid);
This generates this which is sent to the SQL server
exec sp_executesql N'select top 1 entid from ent where custno > @0 and cid = @1 and active = ''y'' order by custno',N'@0 nvarchar(4000),@1 nvarchar(4000)',@0=N'ABC1',@1=N'DEMO'
The issue is the nvarchar(4000) parameter types.  That query returns the wrong value in our data, but changing the nvarchar(4000) to varchar(4000) returns the correct data.
Using ansistring() changes the query to use varchar.
int entid =
db.ExecuteScalar(
"select top 1 entid from ent where custno > @0 and cid = @1 and active = 'y' order by custno asc",
new AnsiString(custno), new AnsiString(cid));
Is there a way to force npoco to always use varchar instead of nvarchar without having to use the AnsiString on each query?
Thanks

Not really if you are writing queries manually.
However if you are using the LINQ provider then specifying the ColumnType as (typeof(AnsiString)) will automatically use varchar instead of nvarchar


Cross-module inlining, incremental extraction, use_extracted_interfaces, and already_cached

Aiming to solve a few problems with build and dependency analysis

An F* project should be able to build itself; another project that
depends on it should not have to try to build it again (and get it
wrong)

Concurrent builds should be possible, so long as they avoid racing
to write the same .checked file; mechanisms to ensure that a build
can fail if a .checked file isn't present would be useful.

We should be able to build a project given only .checked files for
its dependences.

With use_extracted_interfaces, we can shrink the dependences of a
project to only the dependences of the interfaces it depends on.

Points 1 and 2 are already possible with the current support
available, but it can be a bit cumbersome and may require recursive
invocations of make.
Proposal: A new F* option --already_cached <namespace selector>
where <namespace selector> is a string like FStar MiTLS.LowParse
or * -MiTLS etc., which can be used to select or deselect one or
more namespaces / module names.
Dependence analysis

fstar --dep full --already_cached ns ROOTS will scan starting from
ROOTS, as usual.

When it finds a dependence on module M that matches ns, rather
than searching for M.fst(i) it searches for M.fst(i).checked in
the include path. Say it finds it at path
<path_to_M>/M.fst.checked

Having found the M.fst.checked file, it computes the immediate
dependeces of M by just reading it from the list of immediate
dependences of recorded in the interface section of the .checked
file.
-- If any of those immediate dependences does not match the
namespace selector ns, the dependence analysis fails with an
error requesting to add that module to the namespace selector.
-- While .checked files have an interface section, they do not yet contain interface dependences separately from module dependences. This would also have to be added.

It adds those immediate dependences to the set of pending modules
to be scanned and continues.

The final output of the dependence analysis includes Makefile
dependence rules only for those modules whose names do not match
. For example, it will contain:
A.fst.checked: A.fst <path_to_M>/M.fst.checked
But it will not contain such a rule for M.fst.checked.

The dependence analysis also defines three variables:

ALL_FST_FILES: This will contain all .fst and .fsti files
encountered in the scan (i.e., excluding all the .checked files)

ALL_ML_FILES: This will contain .ml file names for all modules
encountered in the scan, including .fst and .checked versions,
subject to the "--extract " feature. For a
checked file, say, <path_to_M>/M.fst.checked, the dependence
analysis will search for M.ml in the include path and emit
<path_to_M.ml>/M.ml.

ALL_KRML_FILES: This behaves the same as for .ml files, except
using .krml files. Additionally, since KreMLin is a full-program
analysis, we include all .krml files, disregarding the --extract <namespace selector>.

Verification runs of F*

fstar --already_cached ns ROOTS will run the dependence analysis
as in the previous section.

F* will then load a set of .checked files as dictated by the
dependence analysis, and for any .checked file that is in , F*
will load the .checked file (it must exist, since otherwise the
dependence analysis would have already failed).
a. When loading M.fst.checked, if M.fst is found in the include
path, F* will check the digest stored in M.fst.checked for
consistency with M.fst (and its dependences).
b. If M.fst is not found, F* will load the checked file without
checking a digest.

Feedback from some of @aseemr @wintersteiger @s-zanella @fournet @protz @tahina-pro, would be great.

I'm not sure about the namespace selection, but I guess it can be helpful if someone ever wants to use some .checked files, but not other ones (e.g. FStar.*).
Re 5: Is path_to_M the directory in which M.fst (no .checked) lives? I suppose a natural choice would be to point to whatever --cache_dir points to. By default perhaps in the current directory?

I think I'm not understanding the proposal :/ Can you clarify why that setting is needed? My intuition would be that if a checked file is available it should be read, with no need for extra flags. Outdated .checked files (detected by checksum-ing the .fst or .fsti if available) should cause a warning (and F* should load the fst or fsti instead).

@cpitclaudel
What if when trying to run fstar A.fst a checked file for one of its dependences B.fst was not found. What should we do then?
Try to verify B.fst on the fly?
-- That can be expensive and unpredictable, since verifying B.fst may require some special options of its own and the author of A may know nothing about how B should be verified. (related to problem 1 in my list)
-- Even if it does succeed in verifying it, should it produce a .checked file for B and if so, where should it put it? We have to be careful about that since concurrent builds may then race to write the same file. (related to point 2 in my list)
So, this new setting is a way of side stepping these issues by expressing that a project's dependences must already be built before trying to build the project itself.

@wintersteiger

Re 5: Is path_to_M the directory in which M.fst (no .checked) lives? I suppose a natural choice would be to point to whatever --cache_dir points to. By default perhaps in the current directory?

Sorry, that's a typo: I meant to write <path_to_M_fst_checked>/M.fst.checked Revising.

@wintersteiger

Re 5: Is path_to_M the directory in which M.fst (no .checked) lives? I suppose a natural choice would be to point to whatever --cache_dir points to. By default perhaps in the current directory?

Sorry, this was not clear.
Earlier in the proposal I say:

When it finds a dependence on module M that matches ns, rather
than searching for M.fst(i) it searches for M.fst(i).checked in
the include path. Say it finds it at path
<path_to_M>/M.fst.checked

So <path_to_M> is really the path to M.fst.checked.

What if when trying to run fstar A.fst a checked file for one of its dependences B.fst was not found. What should we do then? Try to verify B.fst on the fly?

Yup, that'd be my intuition :) But we could also just bail out entirely (that's what Coq does: it never looks for .v files as dependencies).
Proof General has a lot of code to on the fly compilation of dependencies, and it's not pretty.  Conversely, Python generates pyc files on the fly.

That can be expensive and unpredictable, since verifying B.fst may require some special options of its own and the author of A may know nothing about how B should be verified. (related to problem 1 in my list)

What special options are we concretely thinking of? I think they should be in the file itself, or in a local fstar.conf project file.

Even if it does succeed in verifying it, should it produce a .checked file for B and if so, where should it put it? We have to be careful about that since concurrent builds may then race to write the same file. (related to point 2 in my list)

Yup, saving the checked file sounds good.  The location should be the one we usually use, which ATM seems to be in the same folder?
I'm not sure I understand the point about concurrency.  Of course, two fstar processes should not be writing the same checked file at the same time, but that's easy to avoid (both Windows and POSIX have ways to rename a file atomically, so the usual approach is to write to a temp file first, and then atomically rename to the intended destination).
Or are you worrying about something else?

But we could also just bail out entirely (that's what Coq does: it never looks for .v files as dependencies).

The option is to provide this behavior. It's exactly what we had in mind in discussions with @tahina-pro

This flag would be very useful. Looking at the log of a recent Everest build, it seems like we're re-verifying files in FStar/ulib up to 7 times! Most in --lax mode, but stil...

Searching for .checked/.checked.lax files in the include path but writing new files to --cache_dir, together would already be a big improvement over the current behavior.

@aseemr and I had a long discussion about this and came up with the following implementation plan. We are starting to tackle it now and expect it to take the two of us about a week to complete (except for item 8, which will take some more time and item 9, which requires more thought).
An implementation plan for the above proposal, also including
-- Context pruning
-- Cross-module inlining (CMI)

Some notes in less detail about
-- Friend modules
-- Removing --expose_interfaces
-- Memoizing extraction
-- Support for extracted interfaces in fstar-mode.el
-- A way to recover a bit of "separate extraction" even in the
presence of CMI

Fix delta depths: Maintain them with sigelts in the
environment rather than on each fvar.
(will simplify tasks later in the list)

Need to reload the desugaring environment with the extracted
interface so that subsequent modules are desugared with the
appropriate qualifiers etc.

--already_cached (1444); mainly involves the dependence analysis
--already_cached ... implies only 1 file on the command line
Revising the format of .checked files to include interface
dependences separately, in support of context pruning
A.fst(i).checked:
 A.impl={
      Header={
         immediate-deps=
            [self(A.fst, A.fst.hash);
             dep(B.fst.checked, B.fst.checked.hash);
             ...
             dep(...)]
      }
 },

 //missing for modules with explicit interfaces
 //or for .fstis
 optional A.iface= {
       Header={
           immediate-deps=
             [self(A.fst, A.fst.hash);
              dep(B.fst.checked, B.fst.checked.hash);
              ...
              dep(...)]
       }
 }

Cross-module inlining for extraction
A .checked file contains the module implementation;
and separately the module interface
fstar A.fst --codgen OCaml --extract A
Note:  --codegen _ implies --already_cached *

(A depends on B)

Both A.fst.checked and B.fst.checked should be present

We load B's implementation in to the context

We load A's implementation into the context
We extract A

This should remove the need for --expose_interfaces for extraction
purposes.

Verification runs
As is. i.e., we load the interface section of a .checked file, if
present

Friend modules for verification
This should remove the need for --expose_interfaces for revealing
representations during verification.

Memoizing extraction
This would add another section to .checked files
And every run of F* would actually do a bit of extraction for
populating this part of the .checked file, without actually
emitting any code

Extracted interfaces and interactive mode?
The "ideal" behavior would be to maintain 2 type-checking/SMT
environments in parallel as we progress through a buffer.
One environment for checking the module implementation (as currently)
And a second "shadow" environment in which we incrementally extract
and check the interface.
This sounds difficult to do in a single F* process, at least not
without some significant redesign of our code, though not
impossible, since a lot of work has already been done on making the
typechecker's state persistent.
Another, possibility may be for fstar-mode.el to maintain 2 running
F* processes.
An easier goal would be to run this check at the "end" of a module,
with a command triggered by fstar-mode.el
This is still open.

A variant we considered, but decided to leave for later, since it adds
even more complexity.
Cross-module inlining (CMI) is not an outright win. While it enables
more code specialization, it also gives up a bit of separate
extraction. That is, if A.fst changed although A.fsti did not, with
cross-module inlining we will have to re-extract all modules that
depend on A again.
To compensate for this, we considered automatically detecting when a
module requires cross-module inlining.
Cross-module inlining is enabled for a module B whenever
a. B's interface contains a symbol marked 'inline_for_extraction'
b. B's interface contains a symbol whose definition is an
non-informative type; in such cases we should insist that t
(below) is also marked inline_for_extraction.
  e.g,

     module B (.fsti)
       val t : Type0 //requires to be marked inline_for_extraction
       val f : (unit -> t) -> int

     module B (.fst)
       let t = unit
       let f x = 0                extracted as  let f (x:unit) = 0

This also has an impact on the format of .checked files, where we'd
add a header bit to record whether or not a module requires CMI.
The dependence analysis would also have to be aware of this:
The dependence analysis should be aware of the cross-module
inlining criterion below.
In case A depends on B, then
if there is no explicit B.fsti or if B requires cross-module
inlining then we should emit
  A.ml: A.fst.checked B.fst.checked

otherwise
  A.ml: A.fst.checked B.fsti.checked

@nikswamy, with this scheme, we should also remove the step of adding erase_for_extraction attribute when extracting interfaces. It won't be needed anymore, since CMI is the default.

With @protz, we discussed a lightweight variant of --already_cached.
The invocation fstar --already_cached A B.fst fails unless a valid A.fst(i).checked is found on the include path.
Furthermore, the --cache_dir is set to the lowest priority in the include path. So, typically, invocations of F* would prefer to use .checked files from ulib rather than from local cache directories.
This is implemented in nik_dep and awaiting a CI build from Everest before merging to master

The remaining items in this issue are related to use_extracted_interfaces, which is long gone. Closing this issue.


Support for Bigints (on the y-axis)

Feature Proposal
I am dealing with 64-bit unsigned Integers and would like to plot them on a line chart.
As far as I have understood, Chartjs Points can only deal with numbers which is limited to 2*53 without precision loss.
Do you have any plans to support bigints in the future?
Possible Implementation
No response

I don't have a direct answer for you.
But you can always look at the source of the scale and use it to implement your own custom scale that does support big ints:
https://www.chartjs.org/docs/4.4.6/developers/axes.html


The camera should keep moving after we swipe the screen

Currently, when we swipe the screen with the mouse, the camera just moves to the cursor where we released the mouse.
It should keep going in the direction of the swipe and slowly decelerate like in most games.

This is called "inertia" I think


Installing in FreeBSD

I am trying to install the emulator on a FreeBSD system with the pip command, but I need more information than is provided. This is on a server that has never had any PolicyEngine software on it before.

The readme tells me to "Open terminal in the parent directory", but doesn't say what directory that is. I tried making the parent "/home/feenberg/policyengine-taxsim" and ran:

feenberg@taxsimssh:~ % pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///usr/home/feenberg
ERROR: file:///usr/home/feenberg does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

I also tried spelling out "policyengine-taxsim" instead of ".". No help.

Next I tried the direct method:

root@taxsimssh:/etc/ssh # pip install git+https://github.com/PolicyEngine/policyengine-taxsim.git
Collecting git+https://github.com/PolicyEngine/policyengine-taxsim.git
  Cloning https://github.com/PolicyEngine/policyengine-taxsim.git to /tmp/pip-req-build-7um8pc8h
  ERROR: Error [Errno 2] No such file or directory: 'git' while executing command git version
ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?

Does the pip install also require git?

Moving back to Linux with a newly installed OS, I was able to clone the repository, but the pip step failed. See the last four lines below for the error message.:
[taxsim@txmlinux ~]$ git clone "https://github.com/PolicyEngine/policyengine-taxsim"
Cloning into 'policyengine-taxsim'...
remote: Enumerating objects: 350, done.
remote: Counting objects: 100% (216/216), done.
remote: Compressing objects: 100% (115/115), done.
remote: Total 350 (delta 104), reused 177 (delta 92), pack-reused 134 (from 1)
Receiving objects: 100% (350/350), 2.24 MiB | 7.71 MiB/s, done.
Resolving deltas: 100% (135/135), done.
[taxsim@txmlinux ~]$ ls -ltr
total 2328
-rwxr-xr-x. 1 taxsim taxsim 2376376 Nov  6 16:46 linux
drwxr-xr-x. 7 taxsim taxsim    4096 Nov 10 11:17 policyengine-taxsim
[taxsim@txmlinux ~]$ cd policyengine-taxsim
[taxsim@txmlinux policyengine-taxsim]$ git pull
Already up to date.
[taxsim@txmlinux policyengine-taxsim]$ pip install -e .
Defaulting to user installation because normal site-packages is not writeable
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /home/taxsim/policyengine-taxsim
(A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.)

However, there is a cli.py program so I tried it in spite of the message above:
[taxsim@txmlinux tests]$ python ../policyengine_taxsim/cli.py test_e2e.py
Traceback (most recent call last):
  File "/home/taxsim/policyengine-taxsim/tests/../policyengine_taxsim/cli.py", line 1, in <module>
    import click
ModuleNotFoundError: No module named 'click'
[taxsim@txmlinux tests]$ pwd
/home/taxsim/policyengine-taxsim/tests

Next step?

Realized that Redhat linux comes with Python3.9. I was able to install 3.12 got a different result:
cli.py  linux  policyengine-taxsim  test_e2e.py  test_mappers.py
[taxsim@txmlinux ~]$ python3.12 cli.py test_e2e.py
Error processing input: Error tokenizing data. C error: Expected 1 fields in line 25, saw 2

Traceback (most recent call last):
  File "/home/taxsim/cli.py", line 48, in <module>
    main()
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/taxsim/cli.py", line 26, in main
    df = pd.read_csv(input_file)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/pandas/io/parsers/readers.py", line 626, in _read
    return parser.read(nrows)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/pandas/io/parsers/readers.py", line 1923, in read
    ) = self._engine.read(  # type: ignore[attr-defined]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read
    chunks = self._reader.read_low_memory(nrows)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory
  File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows
  File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
  File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
  File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 25, saw 2

[taxsim@txmlinux ~]$


Add codeowners processor

Hey, I just made a Pull Request!
This is partially an experiment + RFC for support populating the entity spec.owner from a CODEOWNERS file. This takes inspiration from the ApiDefinitionAtLocationProcessor.
Suppose we had the following which omits the owner:
# https://github.com/spotify/backstage/catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: backstage
  annotations:
    github.com/project-slug: spotify/backstage
    backstage.io/codeowners-at-location: github:https://github.com/spotify/backstage/blob/master/.github/CODEOWNERS
spec:
  type: library
  # owner: will be populated
  lifecycle: experimental

The CodeOwnersProcessor will attempt to fill in owner by fetching the .github/CODEOWNERS file and parsing it to find the primary owner. This POC currently works, but I some additional thoughts / concerns:

Assumes the first * owner is the primary owner
May not properly support monorepo's or when different parts of a repo have different codeowners
Requires an additional annotation which could potentially be inferred by the github.com/project-slug annotation given the codeowners file is a well known location (at least for github).
Does another fetch to the location. I'd also like to populate the metadata.description from the repo description as well (if it is omitted). If each entity field requires a separate processor, this would be quite inefficient. Perhaps the idea around a context might help in this regard. For instance, I've experimented using the GH GraphQL API to query a bunch of fields + raw codeowners file in a single query. This could in theory be a step where the data is passed down to other processors.

Some ideas on making this more flexible are:

The location could be inferred by just providing the location type (e.g. backstage.io/codeowners-ref: github:) and then using the github.com/project-slug to construct the codeowners file location.
For monorepos, the owner entry path could be assumed based on the relative location of the component.yaml in the monorepo subtree.

:heavy_check_mark: Checklist

[ ] All tests are passing yarn test
[ ] Screenshots attached (for UI changes)
[ ] Relevant documentation updated
[ ] Prettier run on changed files
[ ] Tests added for new functionality
[ ] Regression tests added for bug fixes

Whoa, awesome that you are doing this! Can't review fully at the moment but a few quick thoughts based on the description:
The last matching entry takes precedence, we should take care to follow the logic closely so that we do not get any false attributions/owners. Apologies if this already is what you do.
The location of a yaml file is already known at processing time - as given to the processor function; should not be necessary to latch on to a separate annotation.
Is it not the case that codeowners always exist in very fixed well known locations? Given a yaml file location, is it not enough to try to fetch a file at exactly /.github/CODEOWNERS in the same repo?

The last matching entry takes precedence, we should take care to follow the logic closely so that we do not get any false attributions/owners. Apologies if this already is what you do.

Yes, definitely. I've actually borrowed just the parsing logic from codeowner-utils to keep this experiment small, but may make sense to re-use the package. Would need to confirm logic conforms to the spec fully.

The location of a yaml file is already known at processing time - as given to the processor function; should not be necessary to latch on to a separate annotation.

Ideally, I see the annotation being a flag to indicate ... Yes, I want the codeowers file to be the source of truth. I'm not sure we would want the processor to just assume codeowners is the default owner if it exists would we?

Is it not the case that codeowners always exist in very fixed well known locations? Given a yaml file location, is it not enough to try to fetch a file at exactly /.github/CODEOWNERS in the same repo?

For github yes. I'm not entirely sure about gitlab, and bitbucket, but I would assume they also have a well known location (but not .github clearly). Maybe something like this?
metadata:
  annotations:
    backstage.io/owner-managed-by-codeowners: github # or gitlab

And possible the relative codeowner owned path could be optional like:
metadata:
  name: backstage-techdocs
  annotations:
    backstage.io/owner-managed-by-codeowners: github:/plugins/techdocs # resolves to `@spotify/techdocs-core` as a contrived example

However, this could be added in later only if needed. Basically, I'm hoping to start with something simple, but that is also extendable with additive / non breaking changes to the annotation later.

I'd think that if you don't specify an owner in your file, then that's your signal that you want it supplied from the outside. And if the company in question chose to install the codeowners processor, then it could signal a desired ways of working. I would at least hope that that would be the default behavior without extra typing in the yaml.
The reason I'm thinking this way is that it's a chore to write these files, and one wants to aggressively pursue sensible defaults and a minimum of fuss, as part of the very social/personal challenge of getting excellent adoption across a company and retaining high data accuracy because it's trivial to keep fresh.
If you are forced to always remember the syntax of the annotation and adding It, with more effort than just writing the owner in the first place, it will not aid developer experience as much as I would have hoped. And if the value of the annotation is always the name of the VCS that you exclusively use at the company, or you have to manually point it at the actual codeowners file, then it becomes more of a chore than smooth automation.

This rationale makes perfect sense to me.
I guess the one thing missing would be to make the list of processors more easily composable. Right now, if I wanted to inject the optional processor, I think I'd have to redefine the entire list to ensure it's placed correctly before the policy check?

@andrewthauer Yeah entire list needs to be reconstructed right now. I could image us providing some utility that lets you inject your own processors at a couple of know points, like pre/main/post or smth like that. Separate stuff though.
Regarding the codeowners file, it should always exist at either .github/CODEOWNERS (or .gitlab), docs/CODEOWNERS, or just ./CODEOWNERS. Whether it's GitHub or GitHub is something we likely want to be able to derive from the location processor itself, since we can't really derive it from the URL in case of internal deployments. It's possible that's just a general need though, and that we don't actually need to know if it's GitHub or GitLab for codeowners.
Regarding monorepo support, I think it's safe to assume for all repos that the owner of the component is the owner of the component-info.yaml file itself.

Agreed, a convenience class for adding overrides or injecting more functionality (instead of replacing the entire tree of things by hand) has been at the back of my mind for quite some time. It could be added when the interfaces and hierarchies solidify a bit.
For now I wouldn't mind if we just naively try all possible codeowners file locations until we find one that exists. If that proves too chatty, we could later add a configuration to the processor itself that defines the common place to look.

Would suggest using the GitHub v4 API to quickly find both the CODEOWNERS file and the contents. I think GitLab has something similar that could be used too.
{
  repository(owner: "spotify", name: "backstage") {
    root: object(expression: "HEAD:CODEOWNERS") {
      ... on Blob { text }
    }

    docs: object(expression: "HEAD:docs/CODEOWNERS") {
      ... on Blob { text }
    }

    dotfolder: object(expression: "HEAD:.github/CODEOWNERS") {
      ... on Blob { text }
    }
  }
}

Drawback being, then we have to specify auth again.

There's probably something to be said about centralizing this a bit somehow.
Or perhaps give the read pipe multi file support, and having it read multiple files in parallel using the graphql api for efficiency.

@freben - I'm thinking it could just read the same auth token the GithubReaderProcessor uses?
@Rugvip - I've also experimented with the GitHub v4 GQL API. I also was thinking that querying other fields like the following would also be very useful at the same time:
{
  repository(owner: "spotify", name: "backstage") {
    primaryLanguage {
      name
    }
    homepageUrl
    description
    isArchived
    isFork
    root: object(expression: "HEAD:CODEOWNERS") {
      ... on Blob { text }
    }
    """ ...
}

Which leads me to think this could either be a more generalized processor, or somehow this data could be passed down through the processor chain and any relevant processors can act on the data if it's available.

Well it's a bit unfortunate to make processors dependent on each other, and there's the complexity of deciding WHICH of the secrets to use where there are serveral (you may have both public github and ghe configured for example) - and then multiply that by at least the three provider types that have codeowners support... That's why I mentioned that maybe it would be best if we could lift this config out a level so that there's a "global" setup for auth and how to read from providers. Then both processors can make use of that central facility to do their work.
Regarding other fields, definitely sounds worth exploring but probably start out small with the codeowners first.

So what I'd suggest initially is, maybe use the naive read that is already in place (and works with auth and for all providers transparently), and then we can change around the interfaces and optimize and add support for more fields separately.

@freben - Just to clarify, do you mean using the const result = await read({ type, target }); approach to a well known location in the first pass?
The reason, I'm bringing up the additional fields, is it will be a natural progression so trying to determine the best course forward. But, I agree, I think having isolated processors is ideal in long run. I can also see it being useful to have a general processor that has a customization hook. For instance, maybe a GitHubRepoProcessor that takes a callback which can be used to modify the entity based on the fetched data. Or possibly pass a custom graphql query to it as well so you just need to define the input and output and not the internal fetching logic, etc.

I'm also thinking about how to indicate the owners is managed by the codeowners file. For instance, it would be nice to have the catalog UI have an edit button that points to the codeowners file. Wondering if appending a backstage.io/owner-managed-by-codeowners like the managed-by-location annotation would be a good approach for this. This would be automatically appended. However, I'm not sure if these could be removed if for instance, you stopped using the codeowners. Actually that's a good question, is the entity passed to the location processors include data from the db or just what's been appended in the chain itself?

@freben - Just to clarify, do you mean using the const result = await read({ type, target }); approach to a well known location in the first pass?

Yep!

The reason, I'm bringing up the additional fields, is it will be a natural progression so trying to determine the best course forward.

Sure, but then with a new name to the processor I guess. We could have an ideas gathering phase (RFC?) to see what people want/need to import from their repos. Not everything has to be in the catalog, it'll risk turning into a glorified cache - but for truly useful data for either human or machine consumption, it could be considered.
Maybe it's useful to consider that we are planning to put a graphql layer in place that glues together things and exposes their relationships. Some of the data we are talking about might be possible to put in a Repository type that the entity links out to.

I'm also thinking about how to indicate the owners is managed by the codeowners file. For instance, it would be nice to have the catalog UI have an edit button that points to the codeowners file.
Wondering if appending a backstage.io/owner-managed-by-codeowners like the managed-by-location annotation would be a good approach for this. This would be automatically appended.

Hm, I'll have to think about that one. Sounds reasonable at face value, but I'm thinking long term if we are to use the annotations as that kind of a signaling solution or if it's part of the upcoming "state" idea or part of the "fragments" idea or... :) essentially, maybe it's in the "nice to have" bucket and can wait a bit? I'm thinking that the purple managing their yaml files probably intentionally choose to make use of this feature and know to quickly edit that file anyway.

However, I'm not sure if these could be removed if for instance, you stopped using the codeowners processor. Actually that's a good question, is the entity passed to the location processors include data from the db or just what's been appended in the chain itself?

The entity is always recreated entirely from source data each time (without reading from the db).
HOWEVER, when it gets written to the database, the new and the old annotations get merged. So no, it won't remove old annotations

Another edge case, might be if there are some repo's you don't want to use the codeowners processor. This could be potentially handled with an optional opt out annotation: backstage.io/ignore-codeowners.

Like if there's another processor using a different strategy that ALSO populates unpopulated owner? Hm, at this point I'm not worried about that happening

Love your ideas and attention to detail, keep it up :)

Hm, I'll have to think about that one. Sounds reasonable at face value, but I'm thinking long term if we are to use the annotations as that kind of a signaling solution or if it's part of the upcoming "state" idea or part of the "fragments" idea or... :) essentially, maybe it's in the "nice to have" bucket and can wait a bit? I'm thinking that the people managing their yaml files probably intentionally choose to make use of this feature and know to quickly edit that file anyway.

Yep, don't think this is needed for now :)

The entity is always recreated entirely from source data each time (without reading from the db).
HOWEVER, when it gets written to the database, the new and the old annotations get merged. So no, it won't remove old annotations

Not a problem if we don't add the annotation

Like if there's another processor using a different strategy that ALSO populates unpopulated owner? Hm, at this point I'm not worried about that happening

The current implementation will not update when the owner is already set.

Here's an updated version with tests. Please note the following:

The CodeOwnersProcessor is currently added to LocationReaders defaults. I can remove this if we don't think it should be included by default. It will ignore any entities that do not have owner set.
I removed the restriction on yaml files for all the git provider readers (github, gitlab, bitbucket, and azure) to make this work
I haven't tested anything but github but it's using git-parse-url which should work with all of them. The azure one will fail as the url builder function needs to be fixed. Someone with more knowledge can do that :)
For this naive approach, I thought using Promise.any is perfect. However, I needed to set ESNext.Promise in the tsconfig and also include core-js. Open to alternatives for this.

@andrewthauer could you fix up the conflict here? 👍


[BUG] Azure.Storage.Blobs ignoring ClientSideEncryptionOptions when using ConnectionString in ctor

Library name and version
Azure.Storage.Blobs 12.10.0
Describe the bug
BlobContainerClient has constructor overloads. Using the constructor which expects a ConnectionString does not set the ClientSideEncryptionOptions.
See https://github.com/Azure/azure-sdk-for-net/blob/d80f28049966b80d97634e9c3c83f2ea6daf6c62/sdk/storage/Azure.Storage.Blobs/src/BlobContainerClient.cs#L187-L207
It looks to me like this line of code is missing there.
 _clientSideEncryption = options._clientSideEncryptionOptions?.Clone();
When using the other constructors (for instance with credentials) it is working as expected.
Expected behavior
No matter which ctor is used, the ClientSideEncryptionOptions should be set correctly.
Actual behavior
When using the constructor with the connectionstring as a parameter, there is no ClientSideEncryption configured.
Reproduction Steps

Create BlobClientOptions with configured ClientSideEncryption
Initialize the BlobContainerClient passing the connectionstring, containername and the options

Environment
No response

@sebhaub Thanks for reporting this.  I could see the same and I am redirecting this to services team to provide details if this is by design or if they need to provide a fix for this.
Thanks
Saurabh

@jaschrep-msft Could you chime in on this?


Application-provided "uncaughtException" handler ignored

I initially reported it here but was told to the problem applied to this project. My test case:
process.on('uncaughtException', function(error) {
  console.log("uncaught");
});
throw new Error("Argh");

With Node:
# node test.js
uncaught

With 6to5-node:
# 6to5-node test.js
/tmp/test.js:6
throw new Error("Argh");
      ^
Error: Argh
[...]

@atombender I'm trying to get this fixed in babel

Does this not work?
require('source-map-support').install({
  handleUncaughtExceptions: false
});

@atombender What is the status on this issue?

It's working for me now. I believe @xjamundx's patch fixed it.

@evanw Maybe this issue could be close?

I am closing this issue, feel free to reopen it if you feel your problem has not been resolved.


Global conditioning

There is a unit test in test_model.py.
This code doesn't provider an AudioReader does not send back the speaker id, so it's not ready for use quite yet.
There are other global conditioning implementations in flight. Related discussions here.  Let's try to find the most expeditious way of getting something merged from the various implementations. We're past-due for it IMO.

How about that. We get a different numerical result running under python 3.5 than python 2.7.
Guess I'm going to have to figure out how to to run all this locally under python 3.5.

I didn't hear anything from our friends alexbeloi or Zeta36 regarding their versions of global conditioning so I've carried on, when I've had time.
So it's at the stage where I've got it to work. I trained on the full corpus.
When I told it to generate for speaker id 311, I  got this.
When I told it to generate for speaker id 245, I got  this.
When I told it to generate for speaker id 233, I got this.
So to me subjectively, if you listen to samples of those speaker ids in the vctk corpus, the generated samples sound about right. The quality of the samples is no better than before when I trained on just a single person, which is disappointing. I find when I train longer the loss goes down but the samples start sounding worse. So I imagine there's overfitting going on. So I think we need to separate the data into training and test sets, so we can watch for the test loss starting to rise, which I expect to see even while training loss is falling. I think I'd like to work on that next. Or other ideas?
At this point I think it's ready for review. My main reservations are: 1) I someone else needs to independently test this. Perhaps follow the instructions I put in the README.md and see if it works, and check if we've broken things for the non-globally-conditioned case. I think it should still train and generate from existing models. 2) The test_model.py code has gotten uglier with if global_conditioning: blah blah else blah blah, more so than it was when @lemonzi was suggesting we refactor that. I think my preference might be to do that in a separate PR, but then it's also bad practice to allow the accumulation of technical debt. I guess I need a nudge in one direction or the other.  3) Code review please?
So call for volunteers from amongst those who would like to see this merged. @ibab ? @lemonzi ? Bueller? Anyone?

@jyegerlehner, you are doing a great job with the global conditioning. I don't have too much time right now to spend in this project but I'm glad to see you are going forward. I think you could open a issue for people testing your advances (I don't have a good GPU).
Keep it up!!

I will take part in reviewing this pull request. Will give a feedback in a couple of days.

Just saw this PR, thanks for getting this done! I'm training a model with it right now and will get back to you once I have results.
Those generated samples sound good, I'm also thinking a more quantitative way to rate results would be to compare the fundamental frequency of a generated sample with that of the speaker's training samples. I don't have a very good ear for tone, so this may eliminate the "sounds about right" analysis. I'll see what I can do when this model's done.
I think it'd be fine to refactor test_model.py in another PR, but that's just my opinion. I can also give another set of eyes to a code review if needed.

Thanks for the comments so far.

a more quantitative way to rate results would be to compare the fundamental frequency of a generated sample with that of the speaker's training samples. I don't have a very good ear for tone, so this may eliminate the "sounds about right" analysis.

Hah hah, that sounds good. As far as "sounds about right", my observations were merely that 233 is female, and 311 and 245 are male. And 311 has a deeper voice than 245. All of which was preserved in the generated samples. Let us know what your fundamental frequency analysis shows.

Hi @jyegerlehner , I am still training right now. According to my calculation, our data has 100 persons, with the default setting, we have sample_size=100000 with sample_rate=16000, so 1 sample_size ~ 6s data. Each person has 150MB data which I expect 1s ~ 100kB, so we have 1500s for each person, which means our data contains around 150,000s (including silence before and after the sound in a file). So it will take around 25000 iterations to go through the whole corpus. Am I right? How many steps have you run to get your result? (based on these information, we can get a little bit about which causes overfitting).

How many steps have you run to get your result?

@fehiepsi I'm sorry to say I don't remember but think it was somewhere in the 20K-30K range. I've since thrown that model away and have been trying various things around scalar_input=True.
We had some bugs related to scalar input that I've fixed (locally), but still haven't been able to get it to work well yet. It's frustrating because ibab mentioned he got email from DeepMind author confirming that's how they did it, and was something I thought is the "right" way to do it. But as yet not much success.

I use Python 3.5, and after around 28k steps with gc_channels=16, all other settings are default; I get this. I cant figure out what is the reason at the moment.

use Python 3.5, and after around 28k steps with gc_channels=16, all other settings are default; I get this. I

I used gc_channels=32. Maybe train less (because you are overfitting?), or train more (because you haven't trained enough?). Perhaps that's the opposite of a helpful reply; sorry, I don't have a ready answer why your results are different.

About the scalar input problem, I would like to help. What is the current status?

I've got a model training that has a promising loss profile. I will try generating from it after loss gets a bit lower.
The current ibab master needs a couple fixes: the last two commits to this branch, I think.
The latest and greatest I'm training with is this branch. It has a couple other changes in it. If you grab that branch and run sh traingc.sh it should duplicate what I'm running at the moment. In case you wanted to do that.
I tried generating from a model a couple days ago that had reasonable loss and just got static. I think I ruled out a gross code error with the following experiment: in test/test_model.py I trained a globally-conditioned net with scalar input to generate one of three sine waves where "speaker id" was 0, 1 or 2 to select which sine wave frequency, and it passed the test. I modified the test to save that model to a checkpoint, and then generated from that checkpoint using generate.py. I was able to generate each of the desired sine wave frequencies (as specifed by --gc_id param). So I think that would seem to rule out an error in the generation code.
Oh, and fast generation is broken for scalar input. I only use slow generation. That fast generation code is a pain, since it means we need to maintain two duplicate implementations in parallel.
That's where I'm at, at the moment.

Thank you for your explanation! After making some last tests to see if the global condition will work with two different sine wave and to get understanding about why my model fails, I will start to work with scalar input.

@fehiepsi  I found that I have the logs for the samples above. The samples above weren't generated 20K-30K steps in, it was about 146K steps into the training (that was the number of on the checkpoint file right before the wav files were generated).

I see. Now it is reasonable. My model seems underfitting. Your model is running through 3 epochs. I will test on just two persons instead. :)

I have made the test of global condition on different sine waves which I fake them to be different persons. The overfitting model is doing a real global conditioning. Everything looks good to me.
@jyegerlehner : I will start working on scalar input from now on.

@fehiepsi  On scalar input, I did manage to generate "normal" sounding sample from the branch I mentioned above. That was a first.

Everything looks good to me.

Thanks for testing it.
Last call for comments. Anyone object to merging this?

My only remaining concern on this is the way we use sample_size to feed data if batch_size > 1. Does it cause any proplem if samples have different sizes when we reshape at various points? In case we use batch_size = 1, if the sample is small then the optimizer will give wrong direction to converge due to 0 padding at the beginning of dilations. If this behavior happens at a large portion of data then the result will be not good.

My only remaining concern on this is the way we use sample_size to feed data if batch_size > 1. Does it cause any proplem if samples have different sizes when we reshape at various points?

TestNetWithGlobalConditioning trains the net with a batch size=3, and the test passes. The test covers code in WaveNetModel, but not in train.py. It's possible there's a defect regarding batch size in the train.py part, but 1) I don't know of it, and 2)  batch size = 1 in all normal use cases.

In case we use batch_size = 1, if the sample is small then the optimizer will give wrong direction to converge due to 0 padding at the beginning of dilations. If this behavior happens at a large portion of data then the result will be not good.

I'm not sure that's true. Training the net to learn how the time sequence begins (i.e. transitions from silence to speech at the outset of speech) is a valid portion of the training data. We had a PR to change this and after some discussion closed it without merging.

This is another behavior, not about global conditioning. So I suggest to reverse to the original behavior, then merge, and work with this later.

What are you referring to that we should change back to original behavior? I'm not disagreeing, just saying I don't know what change you are suggesting.

I agree that we might just use batch_size=1 so I dont want to dive into the reshape things. About second issue, I have opened an issue where the model is overfiting but still not generate sine wave if we start with silence. By referring to original behavior, I mean that we use len(buffer) >= sample_size not len(buffer) > 0. Or maybe len(buffer) >= sample_size / 10 because most audio files in the vctk corpus not have such sample_size length.
All of these have nothing to do with global conditioning, so no objection to merge. :) I will read your suggested PR tomorrow, it seems interesting.

By referring to original behavior, I mean that we use len(buffer) >= sample_size not len(buffer) > 0

I discussed the rationale for len(buffer) > 0 above. I think len(buffer) >= sample_size is wrong because it appends > 1 wave file into buffer before enqueueing it for training, which trains the model to learn discontinuities.

I have opened an issue where the model is overfiting but still not generate sine wave if we start with silence.

Hmm that's odd. test/test_model.py trains to learn 3 different sine waves, conditioned by id 0, 1, 2 for each of three frequencies. There's code in there to save the generated sine waves to .wav files and I've listened to them. It generates fine, and does not use any priming.  Maybe one thing that is different about it is that each training batch starts with a random duration of silence before the beginning of the sine wave. I don't recall if that was necessary to get it to learn or not.

I mean that we use len(buffer) >= sample_size not len(buffer) > 0.

All of these have nothing to do with global conditioning

The change to len(buffer) > 0 is related to global conditioning. If we do it the other way, we enqueue a portion of the buffer that is a mix of different wav files from different speakers. Whereas there needs to be a single global id for any audio clip we train on.

I think len(buffer) >= sample_size is wrong because it appends > 1 wave file into buffer before enqueueing it for training, which trains the model to learn discontinuities.

Thank you, I get it. Now I understand what you mean by training the data to learn discontinuity. Though I dont agree with this approach (especially when it is coupled with trimming function), I think that I have to test more before giving further thoughts. :)

@jyegerlehner : I have worked on your branch fix_scalar_input for a while and see that with your fixes and setting initial_filter_width=2 (fast generation still not works with filter length > 2), the output of slow and fast generation with scalar_input and global_condition are the same. I have tested on the corpus of two persons (with setting initial_sample = [0] and use argmax for the prediction of next sample). Is there any other bug which I am missing?

@fehiepsi

fast generation still not works with filter length > 2

That's not particular to this branch.

Is there any other bug which I am missing?

I'm not aware of any bugs.

So everything is on a right tract now. (I have no intention to consider filter length > 2, just mentioned to emphasize the change of initial filter length for scalar input.) :+1:  for merge your fix-scalar-input branch, change initial filter length (or remove it, because I don't see any reason to keep it distinguish from 'filter length'), and commit. You have been doing a great job! Thank you.

@fehiepsi  As far as I know only lemonzi and ibab have commit privileges. I'm hoping someone who is willing to review PRs would volunteer to referee and ibab might make him a collaborator. In the mean time we wait.
I have a few more PRs I'd like to submit such as the scalar input fixes, separate test and training losses, but they all are on top of global conditioning and don't want to submit them until that's merged.
I've got pretty good results with scalar input and global conditioning now.

Hey guys, it has been a while since last updates to master. After reading the discussion it looks like this branch is ready-to-use and only awaits merging, am I right?

@akademi4eg  Yes. I've been running it. Along with other changes over in my repo for which I haven't submitted PRs.

Hi all -- no time for in-depth reviews for the currently open PR's, but since it seems several of you have been using this branch for a while now I'll go ahead and merge it. Thanks a lot @jyegerlehner!


Multiple radio groups on the same page have duplicate ids

When you have multiple options groups on the same page, they end up with duplicate ids. This means clicking the first item on the last group on the page with focus your cursor/select the first item of the first group on the page.
I'll PR a fix when I have time.

Thanks I also take a look later today

I've managed to solve the issue for the time being by setting the name prop, which becomes part of the id attribute.
Maybe automatically generating a random name when one isn't provided?


Troubleshooting the build

WIP changes to get the build working.

Build is looking good! I'm going to publish this one. 👍
Thanks for your help @sreejita-biswas!