[BUG] AWSV4SignerAuth doesn't freeze credentials before signing, leading to a race condition

What is the bug?
AWSV4SignerAuth passes the raw credentials object to SigV4Auth, instead of checking for a get_frozen_credentials method and calling that. This leads to the access key for a request differing from the secret key used to sign it.
How can one reproduce the bug?

Sign requests like it's going out of business
When you cross the refresh threshold, you'll get a request indicating it was signed with a different key than it actually was.

What is the expected behavior?
All requests get signed with matching access keys & secrets.
Do you have any additional context?
See this Python docstring in botocore for a more thorough explanation of the problem.

The problem here is that AWSV4SignerAuth passes raw credentials to SigV4Auth instead of frozen/static credentials. This leads to a potential race condition when signing requests across the credential update boundary.
The behavior is that all requests must be signed with a consistent set of credentials - the same access key identifier and a pair of secret keys. However, with the credentials being updated, if requests are made via an update, the access key identifier in the header may not match the secret key used in the signature.
As explained in the botocore docs here:
https://botocore.readthedocs.io/en/latest/reference/credentials.html#token-race-condition
To avoid this race condition, you should use frozen credentials instead of raw credential objects. The frozen credentials are a snapshot of the credentials at the time of signing the request.
The fix would be to update AWSV4SignerAuth to check for the get_frozen_credentials() method in the credentials object, call it first, and then pass the frozen credentials to SigV4Auth to generate the signature. This ensures that the header and signature use a consistent set of access key identifier and secret key.

That’s… what I said?

Hello @ReinGrad, please feel free to take up this issue and raise a PR with your proposed solution. Thanks :)

@saimedhi PR opened https://github.com/opensearch-project/opensearch-py/pull/469


Standalone test suite

💁  These changes facilitate running the test suites for StateMachine and StateMachineBundle as a standalone package, i.e. without being run inside the context of a Symfony application.
📝  Still to do:

[ ] Clean up interface errors

Argument 1 passed to StateMachine\StateMachine\PersistentManager::__construct() must be an instance of Doctrine\Common\Persistence\ObjectManager, instance of Mock_EntityManager_7f57dd13 given, called in ./src/StateMachine/Tests/PersistentManagerTest.php on line 18 and defined

./src/StateMachine/StateMachine/PersistentManager.php:19
./src/StateMachine/Tests/PersistentManagerTest.php:18

[ ] Clean up fatal errors

PHP Fatal error:  Call to undefined method Mock_RegistryInterface_1993ad6f::getManagerForClass() in ./src/StateMachineBundle/History/PersistentHistoryManager.php on line 63

Fatal error: Call to undefined method Mock_RegistryInterface_1993ad6f::getManagerForClass() in ./src/StateMachineBundle/History/PersistentHistoryManager.php on line 63

Superseded by @xdrive's work in #28.


 I tried to build MICore  and ran into such error:
Could not copy the file "obj\Debug\Microsoft.MICore.dll" because it was not found.
No such problem when build with VS 2017.
Thanks  for help !

Yes, it should work. What solution configuration are you using? Do you get any errors/warnings before that?

@digittigerw Please reopen/comment if it is still not working.


Use HTTP status to convey status of a call

Is your feature request related to a problem? Please describe
backtest and client use HTTP protocol, thus we should use http status to indicate call success/failure of command, rather than current mechanism:
{
    status:
    msg:
    data:
}

Describe the solution you'd like
use http status code and reason to denote the status of a call, for example, 409, f"when creating account, account name {name} already exists"
Additional context
use this method, the client handling logic could be simpler and concise. However, it take some effort to choose right/connected https status code

implemented in 0.3.1


Project still maintained / looking for maintainers?

Hi @guyonroche,
we've sent a couple of PRs your way that pass the test suite and are battle-tested on our production system. Our fork is growing more apart and we'd rather see the changes merged upstream, so that everyone can benefit.
On top of those open PRs that you haven't replied to we'd also like to send some more PRs your way - but are you still maintaining this project? We could also offer to help out as additional maintainers, if you're fine with that. The streaming excel reader is quite important for us and we'd like to give back to this library, if you let us ;)

@alubbe sorry, I've been really busy (still). I assumed you would all have the rights to merge. Is this not so?
Which PRs were you interested in?

As for release checks, just the usual suspects...

I try to add some kind of test for new features and especially for bug fixes.
Travis should be green
README.md should be updated - the New Features! section is copied to the Release History section (with appropriate version) and any changes documented
I try to follow semver so new version will depend on those changes.

yeah none of us have the rights to merge - would you mind granting us that?
regarding the PRs, I think it would be great if you could give your opinion on https://github.com/exceljs/exceljs/pull/702 and https://github.com/exceljs/exceljs/pull/722 as well as https://github.com/exceljs/exceljs/pull/736/files
then, I'd suggest to merge https://github.com/exceljs/exceljs/pull/727 - I will volunteer to update all remaining PRs by hand to avoid this causing any merge conflicts (it's really just checking out the branches and re-running prettier, no biggie).


Charts directory collision when using multiple repositories.

Hi
Two chart with the same name but in different repositories seems to use the same workspace chart directory.
$ helm fetch redmine
---> Fetched chart into workspace /home/migmartri/.helm/workspace/charts/redmine
---> Done

$ ls ~/.helm/workspace/charts/redmine/manifests/
redmine-rc.yaml  redmine-svc.yaml

$ helm repo add bitnami https://github.com/bitnami/charts
---> Cloning into '/home/migmartri/.helm/cache/bitnami'...
---> Hooray! Successfully added the repo.

$ helm fetch bitnami/redmine
---> Fetched chart into workspace /home/migmartri/.helm/workspace/charts/redmine
---> Done

$ ls ~/.helm/workspace/charts/redmine/manifests/
redmine-rc.yaml  redmine-svc.yaml redmine-controller.yaml redmine-service.yaml

Both charts have been fetched in the same directory. Causing some unexpected behavior:

If the k8s files have the same name, they will get overridden.

If the k8s files have different name, they will be placed next to the previous ones, adding unexpected resources to the deployment or causing installation errors.
 helm install bitnami/redmine
---> Running `kubectl create -f` ...
Error from server: error when creating "STDIN": services "redmine" already exists

One proposed solution would be that the workspace directory is namespaced with the repository name for personal repos.
This will result in the following directory structure:
~/.helm/workspace/charts/redmine/
~/.helm/workspace/charts/bitnami/redmine/

Thoughts?

The namespacing/uninstall issue in https://github.com/helm/helm/issues/392 might serve as a template for addressing this?

Similar to when you clone a git repository and may be wishing to avoid a naming conflict, you can specify your own name for a chart in your workspace at the time you fetch it.
Example:
$ helm fetch redmine
$ helm fetch bitnami/redmine bitnami-redmine

@krancour that works if you're aware of what the default behaviour is, for me this wasn't completely obvious and it wasn't immediately clear what had happened. I think being consistent with the charts repo namespace across all commands avoids a lot of confusion.

@prydonius I should be clear... I never meant to imply improvements aren't possible in this regard.  I was only suggesting an expedient workaround for anyone facing this issue at the moment.


v3(?) plans

Based on my conversations with a number of consumers, and trying stuff out myself, I now believe that I bit off more than was necessary with the previous v3 plans. the proposed splits would have just made stuff harder to get started, and the rewrite to typescript would have reduced iteration speed and made the codebase less familiar to newcomers (and myself!). Further, it was too big a change without being informed by usecases, and would have weakened the ecosystem. And even further, other stuff has happened in the css-in-js world, and we would have been in a weaker place to accommodate it.
I want to be clear, I am immensely grateful to the work from @ChristopherBiscardi, @otbe, and @donaldpipowitch, and I take full responsibility for leading them down the path they went. They've been phenomenal, and I'm utterly indebted to them for holding ship for the past few months. glamor would have likely died without them.
So I'm pulling back on the large scale plans, and instead, glamor will now favour incremental improvements over large scale ones, and be usecase-driven in adopting/creating new features.
In the short term, I want to clear the backlog of issues (already wiped out 50 over the last couple of days). I want to tackle #238, #201, and possibly tackle issues around the literal syntax(#188, #187, #140, etc), and maybe some others.
Long term plans -

documentation - the documentation story around glamor is weak, where peeps aren't even sure of what it can and can't do. The composition model is pretty superior and unique, and I'd like to surface that up to consumers. Recipes and patterns, performance tips... all that. I've gotten by with just markdown files so far, but I guess it's time to grow up and make a real website. I'd also like to make sure that peeps will be able to contribute to the website/codebase easily.

ecosystem - glamor has a healthy ecosystem of direct consumers and libraries, and I'd like to service them better. Glamorous, gatsby come to mind, but there are more. Further, there's an opportunity to service libraries like emotion with a subset of glamor's features/code.

perf/benchmark formalisation - it's all very well and good to make claims on twitter based on benchmarks that I've run privately, but I'd like to formalise that entire effort, across a variety of vectors. Further, I'd like to measure it across releases to make sure I don't regress. To be clear, This isn't to compare with other libraries, but to make sure glamor is fast enough for my purposes.

features - Nothing too drastic; I'd like to figure out support for iframes, web components, leverage babel-macros for the literal syntax, flow typings, help the reasonml folks with first class support, formalise/absorb glamor-inline, prep for streaming html/react v16. I might also deprecate stuff like glamor/ous (the skeleton port) considering no one really uses it. Further, I'd like to explore codemods for when I do inevitably break the api. (To be clear, I'd like these, but no guarantees about any)

bundle size - work on reducing the bundle size will always continue. figuring out a good plan that can leverage webpack's scope hoisting, dead code elimination, etc, besides lightening up our dependencies a bit.

None of this is 'final'/set in stone, and will depend on how much time I can spend on it, juggling a real life and new job in a strange country, all that. While I'm open to feedback, please understand that I'll be selfish and consider my own health and sanity above all else.

cc @kentcdodds @KyleAMathews @thk44

As a user of Glamor, this is fantastic news. 👍

have made issues for some of the new features., closing this.


Generic InvalidCast Error - How to Debug Further?

Been learning WinUI 3 for a little while now and everything has been coming along nicely until yesterday when my program just out of the blue has stopped working.
The error I get seems to be rather generic;
Exception thrown: 'System.InvalidCastException' in WinRT.Runtime.dll
WinRT information: No such interface supported

Which just points me to some generated code. I'm still fairly new to debugging 'properly' in Visual Studio, so this isn't at all helpful for me. And I have no idea how to further try to debug this.
Regardless, by spending a few hours today breakpointing my stuff, I think I've narrowed it down a bit.
I have 4 JSON files which I deserialize into 4 lists of object ResourceBase when the app is started. These files are deserialized properly and I have verified that all properties are valid.
After this is all done, I bind a GridView to display these objects, which then creates a custom UserControl for each item. This is where the error starts happening.
I have narrowed it down to a specific item, but that's about as far as I got. Which is this item;
{
  "Name": "Kashyyykian Domesticated Milk",
  "Category": "Organic/Creature/Resources/Creature Food/Milk/Domesticated Milk",
  "Icon": "Milk.png",
  "IsRecycled": false,
  "Attributes": [
    {
      "Name": "Decay Resistance",
      "ShortName": "DR",
      "Min": 1,
      "Max": 1000
    },
    {
      "Name": "Flavor",
      "ShortName": "FL",
      "Min": 1,
      "Max": 700
    },
    {
      "Name": "Overall Quality",
      "ShortName": "OQ",
      "Min": 1,
      "Max": 1000
    },
    {
      "Name": "Potential Energy",
      "ShortName": "PE",
      "Min": 300,
      "Max": 1000
    }
  ]
},

Specifically, when the UserControl attempts to bind to the list of attributes - it throws the aforementioned exception. The bindings are done like such;
<DataTemplate x:Key="ResourceAttributeTemplate" x:DataType="models:ResourceAttribute">
    <Grid BorderBrush="{StaticResource AccentControlElevationBorderBrush}" BorderThickness="1" Width="128" Height="auto">
        <Grid.RowDefinitions>
            <RowDefinition/>
            <RowDefinition/>
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition/>
            <ColumnDefinition Width="auto"/>
            <ColumnDefinition/>
        </Grid.ColumnDefinitions>
        <TextBlock Text="{x:Bind Name, Converter={StaticResource BottomCornerRadiusFilterConverter}}" Style="{StaticResource BaseTextBlockStyle}" Grid.Row="0" Grid.ColumnSpan="3" TextAlignment="Center"/>
        <TextBlock Text="{x:Bind Min, Converter={StaticResource StringFormatConverter}, ConverterParameter='{}{0}'}" Grid.Row="1" Grid.Column="0" TextAlignment="Center"/>
        <TextBlock Text="-" Grid.Row="1" Grid.Column="1" TextAlignment="Center"/>
        <TextBlock Text="{x:Bind Max, Converter={StaticResource StringFormatConverter}, ConverterParameter='{}{0}'}" Grid.Row="1" Grid.Column="2" TextAlignment="Center"/>
    </Grid>
</DataTemplate>

And the ResourceAttribute class is defined as such;
public class ResourceAttribute
{
    public string Name
    {
        get; set;
    }
    public string ShortName
    {
        get; set;
    }
    public int Min
    {
        get; set;
    }
    public int Max
    {
        get; set;
    }
}

This is the debug info I get before it crashes;
[ResourceDataService]: Decay Resistance(DR) | Min: 1(Int32) Max: 1000(Int32)
[ResourceDataService]: Flavor(FL) | Min: 1(Int32) Max: 700(Int32)
[ResourceDataService]: Overall Quality(OQ) | Min: 1(Int32) Max: 1000(Int32)
[ResourceDataService]: Potential Energy(PE) | Min: 300(Int32) Max: 1000(Int32)

The properties are deserialized as expected.
Any help how to further debug what's going on here would be greatly appreciated...

Spotted the error just after I posted this as tends to be the case...
For some reason Converter={StaticResource BottomCornerRadiusFilterConverter} had made its way to where I bind the Name property. Not entirely sure how.
While the issue can be closed now, I still would love to know if there's a better way to debug this error to find specifically where the cast fails...?

Closing this issue per the original poster's comment and length of time since last activity.


【求助】关于ESP-DL (AEGHB-197)

DL的官方安装和使用例子是基于linux的：https://github.com/espressif/esp-dl/tree/master/examples/human_face_detect
有没有基于windows的，还是只能在linux下使用？
谢谢！

DL的官方安装和使用例子是基于linux的：https://github.com/espressif/esp-dl/tree/master/examples/human_face_detect 有没有基于windows的，还是只能在linux下使用？ 谢谢！
The provided example should be compatible with both Linux and Windows operating systems, However If you could specify the required commands, I would be able to provide you with more details.

hi, Ali:
Thanks for your answer!  i prefer to use it in windows, since my IDF was run in windows. could you give some guide?

it can works in my windows VSCode! thanks for your reminder!


Unified account - Fix some UI bugs

Describe the task
Fix some UI bugs

[x] Account Name is not updated after edit account name with Ethereum account

Actual: Still show the old account name on the screen: Owned by (NFT details), History list, Earning details, earning actions...
Expect: Show the new account name

[x] Update component for Select account field
Actual :

Expect :

[x] Update UX when suggestion have > 3 items on Swap screen
Actual :

Expect : Show below

Version test : https://drive.google.com/file/d/1XOTQtxmqID62sEgrSeFmSFSSxn02T0Jo/view?usp=drive_link

[ ] Update UX when suggestion have > 3 items on Swap screen
Actual :

Expect : Show below

[ ] Update UI
Actual :

Expect :

Version test : https://drive.google.com/file/d/1Rk-B6vJNVzs1WcDwL7TpP7DfIVQJ_hC1/view?usp=sharing

[ ] [All account and Specific account mode] Update padding on Transfer screen in case suggestion have many items

Expect : Update padding uniformly on all 3 screens: Transfer token, transfer NFT, swap

Version test : https://drive.google.com/file/d/1kk-g9I8ASbXK6G37m0Z2PMTODYVKBBNS/view?usp=drive_link

[ ] Error page when perform Swap on All account/Specific mode

Version test : https://drive.google.com/file/d/1LQV5_2gLAT5uz6ICOOemXkl8nAHM_nG0/view?usp=sharing
Validation passed in case :

Install/Update version

Account Name is updated after edit account name

Update component for Select account field

Update UX when suggestion have > 3 items


Replace SemanticVersion

SemanticVersion is a data type for representing version numbers.
It's used in the compiler to implement the compare_versions macro, which in turn is used to adapt C bindings depending on the library version.
The implementation of SemanticVersion and compare_versions is solid. But the design is problematic. It's only targeting version numbers conforming to semver (major.minor.patch-pre). semver is a popular versioning system, but far from the only one. The reality shows a great variety of software version schemes employed in the wild.
For the implementation of SemanticVersion, a great number of current software versions can be considered compatible to semver, even if it doesn't actually follow semver, but has a similar format. But as soon as the format doesn't match up, SemanticVersion and compare_versions can't handle a version (such as 1.0 for example). For a general-purpose implementation, this is unacceptable. It should support a wide variety of version number formats.
SemanticVersion is therefore pretty much unusable for anything, unless all versions it needs to handle are guaranteed to conform to semver. In any real-world application, with versions coming from different vendors, this is very unlikely. shards for example uses a custom implementation for comparing version numbers that is more flexible.
Ruby has Gem::Version which is very versatile. I have a Crystal version of that type with a similar interface but a different internal implementation which should be more efficient and could even be further improved for comparison performance.
I think it could be used as a drop-in replacement of SemanticVersion and renamed to SoftwareVersion. It's available here: https://gist.github.com/straight-shoota/d8eec4a346018b0a51c1730b2a5c2bf8
I could also consider publishing it as a shard, but I think it would also fit in the stdlib and would improve compare_versions as well.

👍 From me.
About the implementation, I wonder why it's not storing an Array(String | Int32) as the internal representation. Having to parse the version on each comparison seems... strange (and probably not very efficient). That said, maybe comparing versions isn't something that's too frequent so it can be acceptable. And as long as we agree on an API we can always change the internal implementation later on.

Yeah, it's a trade-off between instantiation and comparison. Splitting the version string in an array is by itself costly. Comparing integers is obviously very fast, but the custom comparison method should be pretty efficient.
Performance obviously depends on usage characteristics, especially the ratio between instantiation and operations like a comparison. I haven't performed a detailed benchmark. I'm mostly interested in functionality and at least  decent performance which is both provided.
But refactoring is always possible. Gem::Version uses an array as internal representation and this could also be implemented with the same interface.

@RX14 in https://github.com/crystal-lang/crystal/pull/7642#issuecomment-483359601

I think things like Gem::Version prove that comparing version numbers isn't nearly as easy as it looks, there's a bunch of edgcases and tradeoffs. Semver fixed that, by being a rigid spec with rigid semantics. That means it's simple and unopinionated to make a SemanticVersion class.
I'd prefer this more generic class to be in a shard, where there are no backwards-compatibility guarantees past 1.0 so there won't be problems with locking in a bad behaviour. The shard should still have a way to validate that a string is exactly a semver string.
As for what happens to SemanticVersion i'd be fine with moving it into the compiler sources and making it private.

I'm answering here, because I think we should discuss the general design in the issue and only implementation in the PR.
I proposed publishing it as a shard as a possible alternative. The downside of this approach would be that the compare_versions macro can only handle (somewhat) semver-compatible version schemes. This strictly limits its usability for real world applications.

Following my question, SemanticVersion doesn't seem to be as used as I thought. Half of the search comes from Cloudfoundry's depwatcher.
compare_versions being mainly used to compare OpenSSL versions, LibCrypto::OPENSSL_VERSION >= 0x10100000 can be used instead, like in this commit. That was @ysbaddaden suggested too in the first place.
I even think the language can perfectly live without SemanticVersion/SoftwareVersion, being moved in separate shard. If compare_version is really needed, a little helper, without having a whole class, can do the job fine.

I haven't need more that semver so far. Along the way one might need to re-learn that M.m.p-x < M.m.p I tend to forget it :-P.
Change compare_versions to handle a superset of semver seems fine. But I would not get rid to SemanticVersion, at least for now.

Thoughts on also adding methods to handle altering the version?
sem_ver = SoftwareVersion.parse "1.1.8"

sem_ver.increment(:minor).to_s # =>"1.2.0"

@Blacksmoke16 I don't think a dedicated API would be a good thing. But you should be able to do something like version.copy_with(major: version.major + 1) which is also much more versatile.

I would be ok with that too.  Anything to really just make working with them a bit easier as the current state is dedicated to just parsing and comparing versus being able to manipulate.


README.md update

Updating README.md with the prefix "j-" and changing index.scss to jeet.scss

@jakecleary Looks good. Merge please.
@lucaslmuller Would you mind updating the gh-pages branch with these as well?

Seems like this has already been fixed by other contributors! Thank you for your contributions.


Cannot run properly

Hi,
I tried to run your code. I succeed to get nnet3.so file. Then, I tried to run examples/nnet3_online.py but I get the following error:
Traceback (most recent call last):
File "examples/nnet3_online.py", line 32, in
from kaldisimple.nnet3 import KaldiNNet3OnlineDecoder
ImportError: No module named kaldisimple.nnet3
Do you have any idea about this?

have you tried setting your PYTHONPATH env variable?

Hi,
Thanks for your response.
I think that there is no problem with PYTHONPATH env variable.
When I tried to debug using ipython for each line. This error is showed up:
In [2]: from kaldisimple.nnet3 import KaldiNNet3OnlineDecoder
ImportError                               Traceback (most recent call last)
 in ()
----> 1 from kaldisimple.nnet3 import KaldiNNet3OnlineDecoder
ImportError: kaldisimple/nnet3.so: undefined symbol: _ZTVN3fst11SymbolTableE
I think that something wrong with my Kaldi or GCC version. I use gcc version 5.4.0 in my Ubuntu 16.04 LTS.
How about you?
Thanks,
Alim

yeah, that sure looks like a compiler or linker issue. I am using gcc 4.8.5 on CentOS 7 here.

Hi,
I tried to downgrade my GCC compiler but it does not help. I don't know the problem.
Maybe it is related with FST library.
Thanks,
Alim

ah! that kind-of rings a bell :o)
at least here, kaldi uses it's own openfst library:
[bofh@hal py-kaldi-simple]$ ldd kaldisimple/nnet3.so
[...]
libfst.so.1 => /apps/kaldi/tools/openfst-1.3.4/lib/libfst.so.1 (0x00007f229b9e8000)
maybe you have a different openfst library installed somewhere else on your system that nnet3.so accidentially links to?
using ldd, you could also check (for comparison) what kaldi's nnet3 binaries link to, e.g.:
[bofh@hal py-kaldi-simple]$ ldd /apps/kaldi/src/nnet3bin/nnet3-latgen-faster
[...]
libfst.so.1 => /apps/kaldi/tools/openfst/lib/libfst.so.1 (0x00007f9c0ac65000)
maybe setting your LD_LIBRARY_PATH could help in that case?

Hi,
I think I'm facing the very same issue.
I've built kaldi and the shared libs successfully and invoking make to build your project creates the nnet3.so. But if I try to run your simple sample code, execution stops and complains about the mentioned missing symbol.
The thing is - my nnet3.so doesn't link against any libfst.so.
This is the output generated by ldd:

linux-vdso.so.1 =>  (0x00007fff0d765000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd5fcd2b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd5fca22000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd5fc80b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd5fc5ed000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd5fc226000)
/lib64/ld-linux-x86-64.so.2 (0x0000555c7c1c2000)

Do you have any advice if there is still something wrong with my kaldi build? Is there any other hint on how we may fix this?
May we have a look at the output ldd generates for your kaldisimple/nnet3.so?
Any help is much appreciated. Thanks in advance!

your nnet3.so doesn't seem to be linked against kaldi at all?!
here is mine:
[guenter@dagobert py-kaldi-simple]$ ldd ./kaldisimple/nnet3.so
	linux-vdso.so.1 =>  (0x00007ffcea9e3000)
	libfst.so.1 => /apps/kaldi/tools/openfst-1.3.4/lib/libfst.so.1 (0x00007f85af295000)
	libsatlas.so.3 => /usr/lib64/atlas/libsatlas.so.3 (0x00007f85ae4fa000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f85ae2de000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f85ae0da000)
	libkaldi-decoder.so => /apps/kaldi/src/lib/libkaldi-decoder.so (0x00007f85adb37000)
	libkaldi-lat.so => /apps/kaldi/src/lib/libkaldi-lat.so (0x00007f85ad4d2000)
	libkaldi-fstext.so => /apps/kaldi/src/lib/libkaldi-fstext.so (0x00007f85ad277000)
	libkaldi-hmm.so => /apps/kaldi/src/lib/libkaldi-hmm.so (0x00007f85ace57000)
	libkaldi-feat.so => /apps/kaldi/src/lib/libkaldi-feat.so (0x00007f85acbc4000)
	libkaldi-transform.so => /apps/kaldi/src/lib/libkaldi-transform.so (0x00007f85ac914000)
	libkaldi-gmm.so => /apps/kaldi/src/lib/libkaldi-gmm.so (0x00007f85ac682000)
	libkaldi-tree.so => /apps/kaldi/src/lib/libkaldi-tree.so (0x00007f85ac36a000)
	libkaldi-util.so => /apps/kaldi/src/lib/libkaldi-util.so (0x00007f85ac0fd000)
	libkaldi-thread.so => /apps/kaldi/src/lib/libkaldi-thread.so (0x00007f85abef6000)
	libkaldi-matrix.so => /apps/kaldi/src/lib/libkaldi-matrix.so (0x00007f85abbdc000)
	libkaldi-base.so => /apps/kaldi/src/lib/libkaldi-base.so (0x00007f85ab9d3000)
	libkaldi-nnet3.so => /apps/kaldi/src/lib/libkaldi-nnet3.so (0x00007f85ab2e7000)
	libkaldi-online2.so => /apps/kaldi/src/lib/libkaldi-online2.so (0x00007f85aafdb000)
	libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f85aac10000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f85aa906000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f85aa604000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f85aa3ee000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f85aa02c000)
	libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f85a9d09000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f85af8da000)
	libkaldi-chain.so => /apps/kaldi/src/lib/libkaldi-chain.so (0x00007f85a9745000)
	libkaldi-cudamatrix.so => /apps/kaldi/src/lib/libkaldi-cudamatrix.so (0x00007f85a94d7000)
	libkaldi-ivector.so => /apps/kaldi/src/lib/libkaldi-ivector.so (0x00007f85a928a000)
	libkaldi-nnet2.so => /apps/kaldi/src/lib/libkaldi-nnet2.so (0x00007f85a8d9c000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f85a8b99000)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f85a895c000)

Yeah, I kind of feared that there has to be some issue .. ;-)
Any specific steps to build kaldisimple/nnet3.so? If not I guess this might be kaldi related and has nothing to do with your project ...

there is not much to building nnet3.so other than setting the kaldi root path in the Makefile and run make.
could you do a
make clean all
to check the compiler/linker commandline?

thanks for your help so far ...
Here is my linker / compiler command line. Please note that I had to change compiling standard to gnu++98 in order get compilation done.

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -I/code/kaldi/src -std=gnu++98 -msse -msse2 -Wall -I.. -pthread -D
KALDI_DOUBLEPRECISION=0 -Wno-sign-compare -Wno-unused-local-typedefs -Winit-self -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_ATLAS -I/code/kaldi/tools/ATLAS/include -I/
home/zorc/code/kaldi/tools/openfst/include -g -fPIC -I/home/zorc/.local/lib/python2.7/site-packages/numpy/core/include -I/code/kaldi/src -I/code/kaldi/tools/
openfst/include -I/code/kaldi/tools/ATLAS/include -I/usr/include/python2.7 -c kaldisimple/nnet3.cpp -o build/temp.linux-x86_64-2.7/kaldisimple/nnet3.o

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -I/code/kaldi/src -std=gnu++98 -msse -msse2 -Wall -I.. -pthread -D
KALDI_DOUBLEPRECISION=0 -Wno-sign-compare -Wno-unused-local-typedefs -Winit-self -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_ATLAS -I/code/kaldi/tools/ATLAS/include -I/
home/zorc/code/kaldi/tools/openfst/include -g -fPIC -I/home/zorc/.local/lib/python2.7/site-packages/numpy/core/include -I/code/kaldi/src -I/code/kaldi/tools/
openfst/include -I/code/kaldi/tools/ATLAS/include -I/usr/include/python2.7 -c kaldisimple/nnet3_wrappers.cpp -o build/temp.linux-x86_64-2.7/kaldisimple/nnet3_wrappers.o
c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D
_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-lMBuS3/python2.7-2.7.12=. -fstack-protector-strong -Wformat -Werror=format-security -L/code/kaldi/tools/openfs
t/lib -L/code/kaldi/src/lib -lfst -lm -lpthread -ldl -lkaldi-decoder -lkaldi-lat -lkaldi-fstext -lkaldi-hmm -lkaldi-feat -lkaldi-transform -lkaldi-gmm -lkaldi-tree -lk
aldi-util -lkaldi-thread -lkaldi-matrix -lkaldi-base -lkaldi-nnet3 -lkaldi-online2 -I/code/kaldi/src -std=gnu++98 -msse -msse2 -Wall -I.. -pthread -DKALDI_DOUBLEPRECIS
ION=0 -Wno-sign-compare -Wno-unused-local-typedefs -Winit-self -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_ATLAS -I/code/kaldi/tools/ATLAS/include -I/home/zorc/code/kal
di/tools/openfst/include -g build/temp.linux-x86_64-2.7/kaldisimple/nnet3.o build/temp.linux-x86_64-2.7/kaldisimple/nnet3_wrappers.o -o /home/zorc/code/py-kaldi-simple/kaldisimp
le/nnet3.so
I know this is lots of stuff but maybe you see some hints? It should link against kaldi from what I can tell. Shared libs are all there...
Well thanks again!

you seem to have two different paths to your kaldi installation in there: sometimes it's /home/zorc/code/kaldi, other times simply /code/kaldi.
Just a wild guess: maybe you specified a relative path somewhere where the build mechanism expects an absolute one?

oh sorry, no that's actually symlinked. I removed the link for testing purposes but it's still the same issue.
It's very strange that there is no linking against the kaldi libs at all ...
By the way: I had to change the atlas-path too. Does ring any bells?

sorry, not really, no ... I do know kaldi needs its own version of openfst (AFAIK that is realted to c++ standards and being discussed right now), but other than that I did not have to do anything special to get kaldi and/or my python wrapper compiled.
just for reference, here is my linker invocation:
g++ -pthread -shared -Wl,-z,relro -rdynamic -L/apps/kaldi/tools/openfst/lib -L/apps/kaldi/src/lib -lfst /usr/lib64/atlas/libsatlas.so -lm -lpthread -ldl -lkaldi-decoder -lkaldi-lat -lkaldi-fstext -lkaldi-hmm -lkaldi-feat -lkaldi-transform -lkaldi-gmm -lkaldi-tree -lkaldi-util -lkaldi-thread -lkaldi-matrix -lkaldi-base -lkaldi-nnet3 -lkaldi-online2 -I/apps/kaldi/src -msse -msse2 -Wall -I.. -pthread -DKALDI_DOUBLEPRECISION=0 -Wno-sign-compare -Wno-unused-local-typedefs -Winit-self -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_ATLAS -I/apps/kaldi/tools/ATLAS/include -I/apps/kaldi/tools/openfst/include -g build/temp.linux-x86_64-2.7/kaldisimple/nnet3.o build/temp.linux-x86_64-2.7/kaldisimple/nnet3_wrappers.o -L/usr/lib64 -lpython2.7 -o /home/guenter/projects/hal9000/mainframe/py-kaldi-simple/kaldisimple/nnet3.so
[guenter@dagobert py-kaldi-simple]$ ldd /home/guenter/projects/hal9000/mainframe/py-kaldi-simple/kaldisimple/nnet3.so
	linux-vdso.so.1 =>  (0x00007ffd85d9c000)
	libfst.so.1 => /apps/kaldi/tools/openfst-1.3.4/lib/libfst.so.1 (0x00007f0b1121d000)
	libsatlas.so.3 => /usr/lib64/atlas/libsatlas.so.3 (0x00007f0b10482000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0b10266000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f0b10062000)
	libkaldi-decoder.so => /apps/kaldi/src/lib/libkaldi-decoder.so (0x00007f0b0fabf000)
	libkaldi-lat.so => /apps/kaldi/src/lib/libkaldi-lat.so (0x00007f0b0f45a000)
	libkaldi-fstext.so => /apps/kaldi/src/lib/libkaldi-fstext.so (0x00007f0b0f1ff000)
	libkaldi-hmm.so => /apps/kaldi/src/lib/libkaldi-hmm.so (0x00007f0b0eddf000)
	libkaldi-feat.so => /apps/kaldi/src/lib/libkaldi-feat.so (0x00007f0b0eb4c000)
	libkaldi-transform.so => /apps/kaldi/src/lib/libkaldi-transform.so (0x00007f0b0e89c000)
	libkaldi-gmm.so => /apps/kaldi/src/lib/libkaldi-gmm.so (0x00007f0b0e60a000)
	libkaldi-tree.so => /apps/kaldi/src/lib/libkaldi-tree.so (0x00007f0b0e2f2000)
	libkaldi-util.so => /apps/kaldi/src/lib/libkaldi-util.so (0x00007f0b0e085000)
	libkaldi-thread.so => /apps/kaldi/src/lib/libkaldi-thread.so (0x00007f0b0de7e000)
	libkaldi-matrix.so => /apps/kaldi/src/lib/libkaldi-matrix.so (0x00007f0b0db64000)
	libkaldi-base.so => /apps/kaldi/src/lib/libkaldi-base.so (0x00007f0b0d95b000)
	libkaldi-nnet3.so => /apps/kaldi/src/lib/libkaldi-nnet3.so (0x00007f0b0d26f000)
	libkaldi-online2.so => /apps/kaldi/src/lib/libkaldi-online2.so (0x00007f0b0cf63000)
	libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f0b0cb98000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f0b0c88e000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f0b0c58c000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0b0c376000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f0b0bfb4000)
	libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f0b0bc91000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0b11862000)
	libkaldi-chain.so => /apps/kaldi/src/lib/libkaldi-chain.so (0x00007f0b0b6cd000)
	libkaldi-cudamatrix.so => /apps/kaldi/src/lib/libkaldi-cudamatrix.so (0x00007f0b0b45f000)
	libkaldi-ivector.so => /apps/kaldi/src/lib/libkaldi-ivector.so (0x00007f0b0b212000)
	libkaldi-nnet2.so => /apps/kaldi/src/lib/libkaldi-nnet2.so (0x00007f0b0ad24000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f0b0ab21000)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f0b0a8e4000)

I guess we may have different compilers here - I am using
[guenter@dagobert py-kaldi-simple]$ g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

from CentOS 7 here.

Thanks so far! I'll be home at the end of the week and then I'll check on that!

Just for the record and maybe as an additional information to everyone reading this:
In order to get everything to work I had to do the steps listed below:

Build kaldi with shared libs
Change kaldi root in Makefile as described
I had to add -std=c++11 to CLFAGS since I decided to use OpenBLAS instead of ATLAS
Change path from libatlas.so to libopenblas.so (Makefile)
Add include paths for openfst, OpenBLAS and kaldi to setup.py.
Add -std=c++11 to setup.py
And maybe the most important step: Using a newer version of gcc/g++ as 4.8 changes the behaviour of ld and you have to add -Wl,-no-as-needed to LDFLAGS to link against supplied libraries even if not referenced at compile time.

But once this is done you code works like a charm. I juist did a quick test using your voxforge models - it rocks!
Thanks again!

very cool - thanks for the feedback!

Hi,
I follow the instruction from chief7. It can work now. The important thing is that Kaldi need to build using shared library (in kaldi/src --> ./configure --shared)
Thanks  a lot.

Hi @chief7 ,
When you say Add include paths for openfst, OpenBLAS and kaldi to setup.py., can you show me how exactly?
Also, when you say you have to add -Wl,-no-as-needed to LDFLAGS, where exactly do you add it in the existing LDFLAGS lines?
Thanks for your help!
I'd love to get this working. Very exciting!

Hi, sure ... I've just created a fork and will upload my changes tomorrow so you may have a look. It's easier to show than to explain.

Sounds good! Thank you for the help!

@chief7: I am also looking forward to see what you have done. maybe we can merge back your changes?

@migueljette Please see my fork here:
https://github.com/chief7/py-kaldi-simple
@gooofy Sure, I'd love to contribute! But I guess it may be more useful to commit a configure shell script or sth like that. I may have time during the next days and could come up with a first idea

Maybe my setup is too different:
g++ --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

I am never able to link properly:
ldd ./kaldisimple/nnet3.so
       	linux-vdso.so.1 =>  (0x00007ffdc398b000)
       	libfst.so.1 => not found
       	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4c5a597000)
       	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4c5a392000)
       	libkaldi-decoder.so => not found
       	libkaldi-lat.so => not found
       	libkaldi-fstext.so => not found
       	libkaldi-hmm.so => not found
       	libkaldi-feat.so => not found
       	libkaldi-transform.so => not found
       	libkaldi-gmm.so => not found
       	libkaldi-tree.so => not found
       	libkaldi-util.so => not found
       	libkaldi-thread.so => not found
       	libkaldi-matrix.so => not found
       	libkaldi-base.so => not found
       	libkaldi-nnet3.so => not found
       	libkaldi-online2.so => not found
       	libpython2.7.so.1.0 => /home/rave/anaconda2/lib/libpython2.7.so.1.0 (0x00007f4c59f91000)
       	libstdc++.so.6 => /home/rave/anaconda2/lib/libstdc++.so.6 (0x00007f4c59c7b000)
       	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4c59972000)
       	libgcc_s.so.1 => /home/rave/anaconda2/lib/libgcc_s.so.1 (0x00007f4c5975b000)
       	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4c59392000)
       	/lib64/ld-linux-x86-64.so.2 (0x0000555d283c9000)
       	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f4c5918f000)

Well, it seems as if your nnet3.so is linked against your shared kaldi libraries but your system can't find them. What you have to do is to add them to your ldconfig.
Usually you have a directory /etc/ld.so.conf.d which contains files with include paths for ld. So you should add a file to that directory, e.g. kaldi.conf which contains the path to your kaldi *.so files. Then check back with ldd if it works.

Hi, thanks for the support!
What does the kaldi.conf file look like?
A list of the *so files?
Like this?
/home/<USER>/kaldi/src/lib/libkaldi-base.so
/home/<USER>/kaldi/src/lib/libkaldi-chain.so
...

It didn't work. But maybe I need to re-read your instructions above.
I am trying to use ATLAS  instead of openBLAS... any reasons I should switch to the same as you?

Hi, its more like
/home/<USER>/kaldi/src/lib
It's basically just telling your system where to look for shared libraries not exactly what files to use.
It should work with ATLAS since gooofy and everyone else I've seen using kaldi is using it, no need to change.

Did you build ATLAS yourself or did you use the one that comes with kaldi?

I'm using the one from Kaldi.
It is looking better, but now i have this:
ldd ./kaldisimple/nnet3.so
./kaldisimple/nnet3.so: /home/<USER>/anaconda2/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/<USER>/kaldi/src/lib/libkaldi-decoder.so)
       	linux-vdso.so.1 =>  (0x00007fff32dd7000)
       	libfst.so.1 => not found
       	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fec75836000)
       	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fec75631000)
       	libkaldi-decoder.so => /home/<USER>/kaldi/src/lib/libkaldi-decoder.so (0x00007fec75067000)

Should I use python3?

Did you built kaldi from source?

Yes. With the --shared flag.

Ok, i fixed the error version GLIBCXX_3.4.21' not found` by updating libgcc in anaconda:
conda install libgcc

I can see it in here:
strings /home/<USER>/anaconda2/lib/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_DEBUG_MESSAGE_LENGTH
...
GLIBCXX_3.4.20
GLIBCXX_3.4.21

But my "fst" package still isn't linked.
I will continue to poke around and see the differences with your setup.py.

Ok, I don't have any experience with anaconda but good to hear that you figured that out.
You nnet3.so was linked against libfst.so.1 earlier. So maybe check that your libfst.so is known to the system. Kaldi keeps it's copy within kaldi/tools/openfst/lib or someting like that.

Ok, it is linked. I had to add the openfst path to /etc/ld.so.conf.d/kaldi.conf.
But my first line is still bad:
ldd ./kaldisimple/nnet3.so
       	linux-vdso.so.1 =>  (0x00007fff5e2cd000)

I'll continue to poke around. :)

Wow. After all of this. :)
My issue showed up as:
python examples/nnet3_online.py
Traceback (most recent call last):
  File "examples/nnet3_online.py", line 32, in <module>
    from kaldisimple.nnet3 import KaldiNNet3OnlineDecoder
ImportError: No module named kaldisimple.nnet3

I did have a bunch of issues with the way I linked to kaldi. So thanks to @chief7 for all your help!
In the end though, after checking that everything was linked properly, I was still getting the same issue... so i decided to debug the python call, as it was suggested above... line by line.
The problem was that I was calling the script like this:
python examples/nnet3_online.py

When you move the script to the same folder, then you get the expected result:
python nnet3_online.py

Maybe I need to add something to my PATH?
Very weird, but it works now!
I'm off to trying to use my own model now.
Thank you for all the support!!!

Glad to see that it's working now!

very cool to see you got this issue sorted! :) - If you have any patches/suggestions, please let me know.

hi chief7,
i am trying to work on this sample and ran into lots of issues(too many to post here).
Then i saw this issue and started following so that i can use C++11 standard as well.
I have a question. I took the following command from your git repo
CFLAGS = -I$(KALDI_ROOT)/src -std=c++11 -msse -msse2 -Wall -I.. -pthread -DKALDI_DOUBLEPRECISION=0 -Wno-sign-compare
-Wno-unused-local-typedefs -Winit-self -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_OPENBLAS -DHAVE_OPENFST_GE_10400
-I/home/zorc/code/OpenBLAS/lib/include -I$(KALDI_ROOT)/tools/openfst/include -g
So in it you use OpenBLAS and say -DHAVE_OPENBLAS
But when you compile kaldi, i am assuming you did not specify the OpenBLAS options, correct? If that is the case then I believe kaldi would have used ATLAS by default. Is that true?
Or will kaldi use the flags from the above mentioned command that has -DHAVE_OPENBLAS in the flags?
Thanks in advance

hi chief7
never mind. i was able to go past all my errors but i am still having one error which has been reported in issue #4 already
In file included from kaldisimple/nnet3.cpp:246:0:
kaldisimple/nnet3_wrappers.h:63:9: error: 'OnlineNnet3DecodingConfig' does not name a type
OnlineNnet3DecodingConfig                 nnet3_decoding_config;
I am awaiting response from maelp on that issue.
I have one question though. You have used -DHAVE_OPENFST_GE_10400
is it because you are using a dfferent openfst than the kaldi one? i did not use the above option though.
thanks

@migueljette
Hi, I was trying with pertained model in Mac:
got same error.
Shakhawats-MBP:goofy_zamia shakhawat_hossain_turag$ python kaldi_decode_wav.py -v demo?.wav
Traceback (most recent call last):
File "kaldi_decode_wav.py", line 36, in
from kaldiasr.nnet3 import KaldiNNet3OnlineModel, KaldiNNet3OnlineDecoder
ImportError: No module named kaldiasr.nnet3
Would you please let me know. How did you solve that?

From my comment above, it looks like I moved a script to a different folder. Read my Feb 8th comment to see. Good luck!

Thanks for your reply. I have seen that but a bit confused where the folder is actually located. And I couldn't install py-kaldi-asr with pip install. It might be problem with that. Btw have u installed py-kaldi-asr in python before?

Unfortunately, I have not used this repo in 3 years, so i am sure it has changed a lot since I used it.
Probably not having installed via pip is a good sign that something is wrong. Maybe kaldi is not installed properly as well.
Good luck!


Vulkano guide out of date with example

Template

Version of vulkano: 0.19.0
OS: Windows
GPU (the selected PhysicalDevice): GeForce RTX 2080
GPU Driver: Unknown

Issue
When running the Compute pipelines example from the vulkano.rs tutorial, an error occurs about a missing extension khr_storage_buffer_storage_class. The example in the code base was fixed in #1275, but it seems that the tutorial was not updated to include it

For me the solution was to add required DeviceExtensions:
    let required_device_extensions = DeviceExtensions {
        khr_storage_buffer_storage_class: true,
        .. DeviceExtensions::none()
    };

    let (device, mut queues) = {
        Device::new(
            physical,
            &Features::none(),
            RawDeviceExtensions::from(&required_device_extensions),
            [(queue_family, 0.5)].iter().cloned()
        ).expect("failed to create device")
    };

For me the solution was to add required DeviceExtensions:
    let required_device_extensions = DeviceExtensions {
        khr_storage_buffer_storage_class: true,
        .. DeviceExtensions::none()
    };

    let (device, mut queues) = {
        Device::new(
            physical,
            &Features::none(),
            RawDeviceExtensions::from(&required_device_extensions),
            [(queue_family, 0.5)].iter().cloned()
        ).expect("failed to create device")
    };

The Vulkano website should now be up to date and maintained regularly.


too many redirects

What version of Traefik are you using (traefik version)?
v1.2.2-alpine
What is your environment & configuration (arguments, toml...)?
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
compress = true
[entryPoints.https.tls]
What did you do?
enter a https address
What did you expect to see?
a valid response
What did you see instead?
a recursive 302 redirect. In chrome this is prevented with too many redirects.
If applicable, please paste the log output in debug mode (--debug switch)
using v1.2.1-alpine with exact same config works.
My toml should only redirect http not https.

Ducplicate if #1429

Ok thx. Sorry for duplicate, my search was too specific ;)
On Apr 13, 2017 16:32, "Emile Vauge"<EMAIL_ADDRESS>wrote:
Ducplicate if #1429 https://github.com/containous/traefik/issues/1429
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/containous/traefik/issues/1432#issuecomment-293912422,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANeBq55_rbz5i4FtMMLWCA2h5Svtfbzks5rvjHDgaJpZM4M8xgF
.


Use Pull Requests instead of pushing directly to the master

We should really work with Pull Requests here and not push directly to the master anymore. Protect it and only allow PR to be merged.

Doubled.


build of pkgsCross.aarch64-multiplatform.openalSoft with pipewire fails

https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2459
openal-soft-aarch64-unknown-linux-gnu> In file included from /nix/store/pkdf5x54hyhf0fga65bs5qy6ihg3r2i3-pipewire-aarch64-unknown-linux-gnu-0.3.56-dev/include/spa-0.2/spa/support/plugin.h:32,
openal-soft-aarch64-unknown-linux-gnu>                  from /nix/store/pkdf5x54hyhf0fga65bs5qy6ihg3r2i3-pipewire-aarch64-unknown-linux-gnu-0.3.56-dev/include/pipewire-0.3/pipewire/pipewire.h:32,
openal-soft-aarch64-unknown-linux-gnu>                  from /build/source/alc/backends/pipewire.cpp:68:
openal-soft-aarch64-unknown-linux-gnu> /nix/store/pkdf5x54hyhf0fga65bs5qy6ihg3r2i3-pipewire-aarch64-unknown-linux-gnu-0.3.56-dev/include/spa-0.2/spa/utils/json.h: In function 'char* spa_json_format_float(char*, int, float)':
openal-soft-aarch64-unknown-linux-gnu> /nix/store/pkdf5x54hyhf0fga65bs5qy6ihg3r2i3-pipewire-aarch64-unknown-linux-gnu-0.3.56-dev/include/spa-0.2/spa/utils/json.h:265:20: error: 'isnormal' was not declared in this scope; did you mean 'std::isnormal'?
openal-soft-aarch64-unknown-linux-gnu>   265 |  if (SPA_UNLIKELY(!isnormal(val))) {
openal-soft-aarch64-unknown-linux-gnu>       |                    ^~~~~~~~
openal-soft-aarch64-unknown-linux-gnu> /nix/store/pkdf5x54hyhf0fga65bs5qy6ihg3r2i3-pipewire-aarch64-unknown-linux-gnu-0.3.56-dev/include/spa-0.2/spa/utils/defs.h:244:46: note: in definition of macro 'SPA_UNLIKELY'
openal-soft-aarch64-unknown-linux-gnu>   244 | #define SPA_UNLIKELY(x) (__builtin_expect(!!(x),0))
openal-soft-aarch64-unknown-linux-gnu>       |                                              ^
openal-soft-aarch64-unknown-linux-gnu> In file included from /build/source/common/alnumeric.h:5,
openal-soft-aarch64-unknown-linux-gnu>                  from /build/source/alc/backends/pipewire.cpp:41:
openal-soft-aarch64-unknown-linux-gnu> /nix/store/fdr1d3dipncyg8piql2mrdadf54171z2-aarch64-unknown-linux-gnu-stage-final-gcc-debug-9.5.0/aarch64-unknown-linux-gnu/include/c++/9.5.0/cmath:654:5: note: 'std::isnormal' declared here
openal-soft-aarch64-unknown-linux-gnu>   654 |     isnormal(_Tp __x)
openal-soft-aarch64-unknown-linux-gnu>       |     ^~~~~~~~

they say its a toolchain issue
applying the following patch makes it work
diff --git a/pkgs/development/libraries/pipewire/default.nix b/pkgs/development/libraries/pipewire/default.nix
index 991ac1ad594..7e1faa50eb0 100644
--- a/pkgs/development/libraries/pipewire/default.nix
+++ b/pkgs/development/libraries/pipewire/default.nix
@@ -186,6 +186,10 @@ let
     '';

     postInstall = ''
+      sed '1i#include <cmath>' -i $dev/include/spa-0.2/spa/utils/json.h
+      substituteInPlace $dev/include/spa-0.2/spa/utils/json.h \
+        --replace "isnormal" "std::isnormal"
+
       mkdir $out/nix-support
       ${if (stdenv.hostPlatform == stdenv.buildPlatform) then ''
         pushd $lib/share/pipewire

lib.optional (stdenv.buildPlatform != stdenv.hostPlatform) stdenv.cc.libc
is suspicious
https://github.com/NixOS/nixpkgs/blob/a016165b6bec90d364e76e9bd0070fa4b58499b5/pkgs/development/libraries/openal-soft/default.nix#L32
added in
https://github.com/NixOS/nixpkgs/commit/1674e25fb15f3dd83a7338f4b252a6789ded5fc0
if it is made unconditional the build fails with
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cmath:45:15: fatal error: math.h: No such file or directory
openal-soft>    45 | #include_next <math.h>
openal-soft>       |               ^~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:76: CMakeFiles/common.dir/common/alcomplex.cpp.o] Error 1
openal-soft> make[2]: *** Waiting for unfinished jobs....
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/stl_algo.h:59,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/algorithm:62,
openal-soft>                  from /build/source/common/almalloc.h:4,
openal-soft>                  from /build/source/common/almalloc.cpp:4:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:104: CMakeFiles/common.dir/common/almalloc.cpp.o] Error 1
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/stl_algo.h:59,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/algorithm:62,
openal-soft>                  from /build/source/common/polyphase_resampler.cpp:4:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:146: CMakeFiles/common.dir/common/polyphase_resampler.cpp.o] Error 1
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/ext/string_conversions.h:41,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/basic_string.h:6608,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/string:55,
openal-soft>                  from /build/source/common/strutils.h:4,
openal-soft>                  from /build/source/common/dynload.cpp:6:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:132: CMakeFiles/common.dir/common/dynload.cpp.o] Error 1
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/ext/string_conversions.h:41,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/basic_string.h:6608,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/string:55,
openal-soft>                  from /build/source/common/strutils.h:4,
openal-soft>                  from /build/source/common/strutils.cpp:4:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/ext/string_conversions.h:41,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/basic_string.h:6608,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/string:55,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/locale_classes.h:40,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/ios_base.h:41,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/ios:42,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/istream:38,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/fstream:38,
openal-soft>                  from /build/source/common/alfstream.h:63,
openal-soft>                  from /build/source/common/alfstream.cpp:4:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:174: CMakeFiles/common.dir/common/strutils.cpp.o] Error 1
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:90: CMakeFiles/common.dir/common/alfstream.cpp.o] Error 1
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/ext/string_conversions.h:41,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/basic_string.h:6608,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/string:55,
openal-soft>                  from /build/source/common/alstring.cpp:7:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:118: CMakeFiles/common.dir/common/alstring.cpp.o] Error 1
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/ext/string_conversions.h:41,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/basic_string.h:6608,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/string:55,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/stdexcept:39,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/system_error:41,
openal-soft>                  from /build/source/common/threads.cpp:26:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:188: CMakeFiles/common.dir/common/threads.cpp.o] Error 1
openal-soft> In file included from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/bits/stl_algo.h:59,
openal-soft>                  from /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/algorithm:62,
openal-soft>                  from /build/source/common/almalloc.h:4,
openal-soft>                  from /build/source/common/ringbuffer.h:10,
openal-soft>                  from /build/source/common/ringbuffer.cpp:23:
openal-soft> /nix/store/xr6s02lgjmqcjfidslqzm63ns29j7lzg-gcc-11.3.0/include/c++/11.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
openal-soft>    75 | #include_next <stdlib.h>
openal-soft>       |               ^~~~~~~~~~
openal-soft> compilation terminated.
openal-soft> make[2]: *** [CMakeFiles/common.dir/build.make:160: CMakeFiles/common.dir/common/ringbuffer.cpp.o] Error 1
openal-soft> make[1]: *** [CMakeFiles/Makefile2:108: CMakeFiles/common.dir/all] Error 2
openal-soft> make: *** [Makefile:136: all] Error 2

Wouldn't surprise me if the libc inclusion is related to this issue

Yes, libc headers should not be included directly via -I flags. gcc wraps libc headers and filters out extras that pollute namespace. I would say it should never be included. It's already present via other means.

Looking at the build failure (if I remove libc from buildInputs) I think the problem here is that it tries to pull in build's headers, not host's headers:
$ nix build --no-link -f. pkgsCross.aarch64-multiplatform.openalSoft -L --cores 1
...
openal-soft-aarch64-unknown-linux-gnu> /nix/store/xy6ngc2gxqcypczxbahrqz31xr97m6gg-glibc-2.35-163-dev/include/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
openal-soft-aarch64-unknown-linux-gnu>     7 | # include <gnu/stubs-32.h>
openal-soft-aarch64-unknown-linux-gnu>       |           ^~~~~~~~~~~~~~~~

$ cat cat /nix/store/xy6ngc2gxqcypczxbahrqz31xr97m6gg-glibc-2.35-163-dev/include/gnu/stubs.h
/* This file is automatically generated.
   This file selects the right generated file of `__stub_FUNCTION' macros
   based on the architecture being compiled for.  */

#if !defined __x86_64__
# include <gnu/stubs-32.h>
#endif
#if defined __x86_64__ && defined __LP64__
# include <gnu/stubs-64.h>
#endif
#if defined __x86_64__ && defined __ILP32__
# include <gnu/stubs-x32.h>
#endif

These unexpected headers are injected from OSS headers: openal-soft-aarch64-unknown-linux-gnu> -- Found OSS: /nix/store/xy6ngc2gxqcypczxbahrqz31xr97m6gg-glibc-2.35-163-dev/include.

The header path detection comes from cmake/FindOSS.cmake:
$ cmake/FindOSS.cmake
find_path(OSS_INCLUDE_DIR
          NAMES sys/soundcard.h
          DOC "The OSS include directory"
)

Apparently glibc headers do provide sys/soundcard.h and confuse cmake.

The following seems to be enough to make it compile:
--- a/pkgs/development/libraries/openal-soft/default.nix
+++ b/pkgs/development/libraries/openal-soft/default.nix
@@ -29,7 +29,7 @@ stdenv.mkDerivation rec {

   nativeBuildInputs = [ cmake pkg-config ];

-  buildInputs = lib.optional (stdenv.buildPlatform != stdenv.hostPlatform) stdenv.cc.libc
+  buildInputs = []
     ++ lib.optional alsaSupport alsa-lib
     ++ lib.optional dbusSupport dbus
     ++ lib.optional pipewireSupport pipewire
@@ -40,6 +40,8 @@ stdenv.mkDerivation rec {
     # Automatically links dependencies without having to rely on dlopen, thus
     # removes the need for NIX_LDFLAGS.
     "-DALSOFT_DLOPEN=OFF"
+  ] ++ lib.optionals stdenv.hostPlatform.isLinux [
+    "-DOSS_INCLUDE_DIR=OFF"
   ];

   meta = with lib; {

Not sure if it's a reasonable way to workaround cmake's script deficiency.

we should create a issue upstream about using a better FindOSS.cmake
found these 2 which seem okay
https://github.com/gnuradio/gnuradio/blob/main/cmake/Modules/FindOSS.cmake
https://github.com/obsproject/obs-studio/blob/master/cmake/Modules/FindOSS.cmake

this works and finds oss
diff --git a/pkgs/development/libraries/openal-soft/default.nix b/pkgs/development/libraries/openal-soft/default.nix
index 368b1b911eb..fa1d5af4804 100644
--- a/pkgs/development/libraries/openal-soft/default.nix
+++ b/pkgs/development/libraries/openal-soft/default.nix
@@ -29,8 +29,7 @@ stdenv.mkDerivation rec {

   nativeBuildInputs = [ cmake pkg-config ];

-  buildInputs = lib.optional (stdenv.buildPlatform != stdenv.hostPlatform) stdenv.cc.libc
-    ++ lib.optional alsaSupport alsa-lib
+  buildInputs = lib.optional alsaSupport alsa-lib
     ++ lib.optional dbusSupport dbus
     ++ lib.optional pipewireSupport pipewire
     ++ lib.optional pulseSupport libpulseaudio
@@ -40,6 +39,9 @@ stdenv.mkDerivation rec {
     # Automatically links dependencies without having to rely on dlopen, thus
     # removes the need for NIX_LDFLAGS.
     "-DALSOFT_DLOPEN=OFF"
+  ] ++ lib.optionals stdenv.hostPlatform.isLinux [
+    # https://github.com/NixOS/nixpkgs/issues/183774
+    "-DOSS_INCLUDE_DIR=${stdenv.cc.libc}/include"
   ];

   meta = with lib; {

It looks like those variables aren't even used much, and they should just be using https://cmake.org/cmake/help/latest/module/CheckIncludeFile.html instead?

As far as I understand the workaround is enough to solve this for now.


Add a Rational Number type

Since some Matroska elements would be better as rational numbers we need a way to store them. I think a type would be better than using two EBML Signed Integers. The bits in the VDATA would be divided in half. The first part (signed) for the numerator, the second part (signed?) for the denominator.

Caveat: I didn’t check where this would be useful.
Generally speaking, in my personal experience, it’s however easier to implement the rational number type by two regular integers.

How do you understand the boundary between the signed integers? Advantage to store num and den as a single value?

It might use less space, although it depends on the length of the ID and the value itself.
when reading the value the parser doesn't have to keep the first value in a state before using the next value
if a rational element can be found more than one (a list of values) it's not possible to do it with 2 elements, or you need to wrap them in a master element

you can't have one value and not the other. You have both or none.

Is the rational type only for unsigned numbers or do we need a signed rational type too?

The first part (signed) for the numerator, the second part (signed?) for the denominator.

Signed (which includes unsigned apart from very large values)

If we really need such a field[1], I propose the following:

Starts with the regular EBML ID & element size fields; let's call this size field element_data_size (in bytes)
Following is a second size field encoded the same way the element size field is encoded; let's call this field numerator_size (in bytes) and the size of this field itself numerator_size_size (in bytes)
Following is an unsigned integer number of length numerator_size bytes; this is the numerator
Following is a signed integer number of length element_data_size - numerator_size_size - numerator_size; this is the denominator

Restrictions:

the value of the denominator MUST NOT be 0
the value of numerator_size_size + numerator_size MUST be less than element_data_size

Rationale:

Why make the denominator signed, not the numerator? Wild guess here: the denominator will stay constant (or almost constant) for most well-known applications such as video timestamps (e.g. the denominator staying at exactly 30 * <IP_ADDRESS>/1001 or something like that to represent 29.975 frames per second). The numerator usually grows, so let's give it that one bit more to grow into.
If we need a rational type, using a packed structure does indeed save a lot of space. The only other possible method of implementation would be a master element with two integer children, one signed, one unsigned. That's quite a lot of useless overhead: two more IDs (at least two bytes), one useless size field (the one of the master element). And it introduces a number of corner cases such as what if an element is missing? How to specify default values for nested elements? Etc.

[1] Not sure if we really need this data type given the smaple-accurate calculation method you specified the other day.

I'd rather have [num_size][num_value][den_size][den_value]. In the case the rationale value is equal to an integer we just use [num_size][num_value] and assume the denominator is 1.
We do need a way to specify the default attribute values, like we have a format for floats. I suppose a string with "numerator/denominator" would work. Maybe make the denominator optional if we allow it, in which case such a default value would be "numerator".

We always have the size field for the full element. There's really no need for three size fields as we can always calculate the third size from the other two. It would be a waste of space, a significant one if we used a rational element in something like a hypothetical BlockV3 structure.
I agree about default values. Leaving out /denominator simply means that denominator is 1.

OK my bad, I thought what you proposed was [num_size][den_size][num_value][den_value].
I agree with [num_size][num_value][den_value], the [den_value] part being optional.

Great!

I'm also fine what @mbunkus proposed.
But one thing I would change: that is the size-field for the numerator [num_size].
This size-field could be a "normal" size integer not a VINT, which makes parsing faster.
Reason:
The size of the numerator(and also the size of denominator) is limited to 8, that means the length of the size-field is always 1.
[EBML-ID] [element size-field as VINT] [numerator: "normal" size-field as unsigend integer] [numerator value] [denomiator value]

Yes, given the limitation of the width of the numerator an regular unsigned 1-octet integer is easier for everyone.

BTW we will probably need signed rational values. One way to encode the negative value (we should only allow one of the two values) could be to use the same encoding as the EBML-lacing in Matroska, see https://github.com/ietf-wg-cellar/matroska-specification/issues/726#issuecomment-1445148868

In my original proposal the denominator is a signed integer already. Not sure what you're missing there.
Taking @hubblec4's suggestion into account, I revise my original proposal to:

Starts with the regular EBML ID & element size fields; let's call this size field element_data_size (in bytes)
Following is a one-byte long second size field, an unsigned integer specifying the length of the following numerator field; let's call this field numerator_size (in bytes)
Following is a numerator_size bytes long unsigned integer number in big-endian byte order; this is the numerator
If numerator_size is less than element_data_size: following is a element_data_size - numerator_size bytes long signed integer number; this is the denominator. The signed integer is stored in two's complement notation with the leftmost bit being the sign bit. If numerator_size equals element_data_size, there is no denominator field and the denominator's value is 1.

Restrictions:

the value of the denominator MUST NOT be 0
the value of numerator_size MUST be less than or equal to element_data_size

Rationale:

Why make the denominator signed, not the numerator? Wild guess here: the denominator will stay constant (or almost constant) for most well-known applications such as video timestamps (e.g. the denominator staying at exactly 30 * <IP_ADDRESS>/1001 or something like that to represent 29.975 frames per second). The numerator usually grows, so let's give it that one bit more to grow into.
As a rational type might be used in each block, having a small encoding is a big plus — hence a packed structure instead of an EBML Master element with two children, one signed, one unsigned. That's quite a lot of useless overhead: two more IDs (at least two bytes), one useless size field (the one of the master element). And it introduces a number of corner cases such as what if an element is missing? How to specify default values for nested elements? Etc.

In my original proposal the denominator is a signed integer already.

@mbunkus Out of curiosity: Is there a reason why you chose the denominator rather than the numerator?

I actually explain my reasoning in the first bullet point of "Rationale".

@mbunkus Indeed, I saw it. I was asking, because I’m not convinced. I actually did the contrary (back in the 1980s and 1990s) when I implemented a few compilers. My reasoning, back then (!), was about rounding errors and processor cycles.

The rationals in media containers are somewhat unusual in that for most use cases their denominator remains constant.
I don't know whether having one more bit to play with in the numerator actually makes any noticeable difference wrt. file sizes. On the other hand, I'm even less convinced that making the numerator signed instead of the denominator has any measurable difference wrt. to processing time, given that a) reading an unsigned integer requires fewer instructions & b) that this is a media format — any potential miniscule gains while parsing the container are dwarfed by the time required for decoding the content anyway.

I don't know whether having one more bit to play with in the numerator actually makes any noticeable difference wrt.

Still good to get IMO. The parser is in charge of rounding errors and actually could internally move the sign at the numerator if it wishes.

The rationals in media containers are somewhat unusual in that for most use cases their denominator remains constant.

That could be a reason, yes. Thank you!

a) reading an unsigned integer requires fewer instructions

For me is that the best argument that the numerator should be an unsigned integer.
And that's the reason why I want to speak about the third way for a signed rational number.
In mathematics it is possible/usual  to set the sign before the rational number in the height of the fraction bar.
That means both numbers(numerator and denominator) are unsigned values.
Now we need only 1 bit to signal if a rational number is negative/signed or not.
My propose was:

Following is a one-byte long second size field, an unsigned integer specifying the length of the following numerator field; let's call this field numerator_size (in bytes)

The numerator_size value has a range from 1 to 8.
That means only 3 Bits are used and we could use the first Bit for the sign option.
If the first Bit is set to 1, than is the rational number negative.
For Example a rational number value: -"180/1"
The denominator has the default value "1" and can be omitted to keep this example simple.
The length of "180" is one byte. -> numerator_size in Bits 0-0-0-0-0-0-0-1
Now we set the sign Bit -> 1-0-0-0-0-0-0-1
It is more or less the same like a VINT with a length of one byte.
The word numerator_size have to changed because there are more information in this Byte now.
Maybe sign_and_numerator_size.
That was some basic thoughts and I hope I'm not wasting your time.

In my original proposal the denominator is a signed integer already. Not sure what you're missing there.

I did not check. I just added a reference to an existing (efficient) EBML-like encoding for signed values, so we don't forget about it.
Looking at your proposal it seems easier to use than this (no need code needed for those not supporting EBML lacing in Matroska).


Add implementation for pipectl tool

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #1199
Does this PR introduce a user-facing change?:

Introducing the CLI tool: pipectl

/cc @khanhtc1202

I just realized that we need a wait-status after syncing an application for our B team.
So changed this PR to be WIP.

/lgtm

/approve


Random "invalid signature" errors when more than one parameter

Some time i get strange random "invalid signature" errors on requests with two parameters like this:
client.get_transactions('BTC', starting_after='', limit=100)
But if limit parameter is omitted all works fine:
client.get_transactions('BTC', starting_after='')
Cant say exactly where is this problem comes from, but seems that ordering of GET parameters is different on client and on coinbase server and that causes "invalid signature" errors, because GET parameters included in message for signing request.

1000D

I also get this error under the exact same circumstance as your example. If I try to use a limit of 100 and pagination to get all of my transactions, the first call works, but the second call that includes the starting_after parameter will fail with an invalid signature. If I remove the limit and just use starting_after everything works fine. This code used to work for me but I had not run it in a while. Not sure when it broke.


toObject  property does not exist on type UserModel

Versions

NodeJS: 10.15.1
Typegoose(NPM): 5.2.1
mongoose: 6.1.2

TypeScript does not see property toObject in instances of models (that are extending typegoose)
I only able to fix it by adding my custom types, but maybe it can be added to @types at library?

typegoose is based on typescript, so it dosnt need to be on the @types collection
could you add your model & how you use it to verify it?

const result: User = await this.authService.register(payload);

result.toObject(); // here's the error property toObject does not exist on type User

And here's the model
export class User extends Typegoose {
@prop()
username: string;
}

And there's is authServices

const data = new userModel(payload);

await data.save();

return data;

sorry, cant verify, using latest master / npm 5.8.1:
import * as mongoose from "mongoose";
import { prop, Typegoose } from "./typegoose/src/typegoose";

class Dummy extends Typegoose {
    @prop({ required: true })
    public main1?: string;
}

const DummyModel = new Dummy().getModelForClass(Dummy);

(async () => {
    mongoose.set("debug", true);
    await mongoose.connect(`mongodb://sylvi:27017/`, {
        useNewUrlParser: true,
        useFindAndModify: true,
        useCreateIndex: true,
        dbName: "verify338",
        user: "hasezoey",
        pass: "FhU2MdPxWzWvESHN",
        authSource: "admin",
        autoIndex: true
    });

    const doc = new DummyModel({ main1: "main1" });
    await doc.save();
    console.log(doc.toObject());

    await mongoose.disconnect();
})();

PS: yes, on type User there will never be an .toObject, you are using the wrong type, use this!

cannot reproduce, sorry, using latest master / npm 5.8.1:
import * as mongoose from "mongoose";
import { prop, Typegoose } from "./typegoose/src/typegoose";

class Dummy extends Typegoose {
    @prop({ required: true })
    public main1?: string;
}

const DummyModel = new Dummy().getModelForClass(Dummy);

(async () => {
    mongoose.set("debug", true);
    await mongoose.connect(`mongodb://mongodb:27017/`, {
        useNewUrlParser: true,
        useFindAndModify: true,
        useCreateIndex: true,
        dbName: "verify338",
        user: "user",
        pass: "passwd",
        authSource: "admin",
        autoIndex: true
    });

    const doc = new DummyModel({ main1: "main1" });
    await doc.save();
    console.log(doc.toObject());

    await mongoose.disconnect();
})();

PS: you are using the wrong type, use this
const result: User = await this.authService.register(payload); -> const result: InstanceType<User> = await this.authService.register(payload);

I consider this as closed & Invalid(as a bug)
@Ben305 you could add the "invalid" label on here, thanks

@hasezoey Could you please help me with the same issue but for the nested property:
import { getModelForClass, modelOptions, prop } from "@typegoose/typegoose";

class Location {
  @prop()
  public latitude: number;
  @prop()
  public longitude: number;
}

class LastKnownLocation {
  @prop({ _id: false })
  public location: Location;
  // ...removed some props for example
}

@modelOptions({ schemaOptions: { timestamps: true } })
export class VehicleLocation {
  @prop({ required: true })
  public vehicleId: string; // string is just as example

  @prop({ required: true })
  public location: LastKnownLocation;
}

const VehicleLocationModel = getModelForClass(VehicleLocation);

(async () => {
  await VehicleLocationModel.create({
    vehicleId: "123",
    location: { location: { latitude: 1, longitude: 2 } },
  });
  const currVehicleLocation = await VehicleLocationModel.findOne({
    vehicleId: "123",
  }).sort({ _id: -1 });

  // Property 'toObject' does not exist on type 'LastKnownLocation'.ts(2339)
  currVehicleLocation?.location.toObject();
})();

@alexeychikk please use the new repo this one is deprecated


Refactor code for official release

[x] Replace local paths with variables with descriptive names
[x] Remove comments in the code

Huge thanks Jon! I know I can always trust you! ❤ Could you please review my changes one more time?

Oh, a couple other questions:
Are all these scripts intended to run on a server or local machine or in a C-PAC image or some of each or user's choice?
Also are there any files that are needed but not included, like maybe https://github.com/XinhuiLi/PipelineHarmonization/blob/893b53b2f7ffc0fc3243ce423252d9a13e696b33/figure/s1/abcd.py#L94-L97 these templates?
(and /usr/local/fsl/data/standard/MNI152_T1_2mm_brain.nii.gz on Lisa is probably the same as /usr/share/data/fsl-mni152-templates/MNI152_T1_2mm_brain.nii.gz in the C-PAC image, right?)

Are all these scripts intended to run on a server or local machine or in a C-PAC image or some of each or user's choice?

Users can run on any machine. We usually do computation (like ICC) on server and download results from server to local machine to plot figures.

Also are there any files that are needed but not included, like maybe

https://github.com/XinhuiLi/PipelineHarmonization/blob/893b53b2f7ffc0fc3243ce423252d9a13e696b33/figure/s1/abcd.py#L94-L97

these templates?

Good catch! I'll add notes for these templates.

(and /usr/local/fsl/data/standard/MNI152_T1_2mm_brain.nii.gz on Lisa is probably the same as /usr/share/data/fsl-mni152-templates/MNI152_T1_2mm_brain.nii.gz in the C-PAC image, right?)

Yes!


x/playground: easily runs into build timeouts by downloading modules

For example, see https://go.dev/play/p/b4S-VWK-yWR, which needs to download<EMAIL_ADDRESS>and its dependencies. It currently times out at build time, presumably while still downloading the module zips:
timeout running go build
go: downloading cuelang.org/go v0.5.0-beta.1
go: downloading github.com/cockroachdb/apd/v2 v2.0.2
go: downloading github.com/mpvl/unique v0.0.0-20150818121801-cbe035fff7de
go: downloading golang.org/x/text v0.3.8
go: downloading golang.org/x/net v0.0.0-20220722155237-a158d28d115b
go: downloading github.com/google/uuid v1.2.0
go: downloading github.com/pkg/errors v0.8.1
go: downloading gopkg.in/yaml.v3 v3.0.1

Go build failed.

It appears that the timeout happens after about ten seconds.
I wrote a small program a while ago to estimate how big all the zips in go list -m all are. The results, starting with the largest in bytes, are as follows:
$ go-mod-size | sort -k2 -n -r
<EMAIL_ADDRESS>8615981
<EMAIL_ADDRESS>3946738
<EMAIL_ADDRESS>2062942
<EMAIL_ADDRESS>1797601
<EMAIL_ADDRESS>1582191
<EMAIL_ADDRESS>320918
<EMAIL_ADDRESS>271386
<EMAIL_ADDRESS>219375
<EMAIL_ADDRESS>210263
<EMAIL_ADDRESS>169515

So a clean go build does download a fair bit - at least 15MiB or so. The total number of modules is 32, per go list -m all | wc -l. I think the playground should be able to download these modules by doing any of:

Talking to a fast GOPROXY. GOMODCACHE=$PWD/tmp time go build reports a total runtime of about 3s on my laptop, with a 500Mb ethernet connection and a 11ms ping to proxy.golang.org. I'm sure that a playground server could do better.

Reusing a GOMODCACHE on a best-effort basis. The same could be said about GOCACHE for build caching for compile timeouts.

Increasing the timeout for multi-file playground inputs with go.mod and go.sum files. For example, given the number of lines in either go.mod or go.sum, we could estimate that this example needs a 20s timeout rather than 10s. Presumably this shouldn't be too harmful, as the playground should cache the result once it runs without a timeout.

Implementing a more sophisticated timeout. For example, the playground could have very generous timeouts for downloading modules, given that network traffic is presumably cheap. The timeout for compiling and linking could similarly be more generous than the timeout for running the program, as there's no chance for running arbitrary code to burn CPU or abusing free compute.

cc @toothrot per https://dev.golang.org/owners

Worth noting that, since go build prints what modules it's downloading, but not what packages it's buildling, it could be that my timeout above is during the compilation or linking of the program. I don't think I can know for sure. If it is indeed not a "module download" timeout, I think that should be made clearer.

We discussed this in the tools call yesterday, and I feel like I may have prematurely determined that this problem is not easily solvable.
If timeouts are during the build, not the download, we should consider increasing the size of the playground app engine instances:
https://cs.opensource.google/go/x/playground/+/master:app.yaml;l=8;drc=9bfb5bdef13596a261db44e1c28a9fc8ddd7bc77
Right now, they appear to be at the minimum number of CPU:
https://cloud.google.com/appengine/docs/flexible/reference/app-yaml?tab=go#resource-settings
It may be worth just 2x'ing or 4x'ing the instance shape to see if it improves anything.
CC @golang/release

Two CPUs and two gigabytes of memory don't sound like enough to build some non-trivial examples with dependencies in a reasonable amount of time :) I would agree with that. Though I would also like to see better caching and longer build timeouts.

For what it's worth, the higher resources don't seem to get us out of the timeouts. Relatively light examples like https://go.dev/play/p/qPDCgnJgO39 still trigger a timeout. Some dependencies like x/text and x/tools still weigh megabytes, but surely that shouldn't be out of the norm :)


waffle.io Badge

Merge this to receive a badge indicating the number of issues in the ready column on your waffle.io board at https://waffle.io/sjkingo/virtualenv-api
This was requested by a real person (user sjkingo) on waffle.io, we're not trying to spam you.

Merged in 5284a8b


Change line length to 120 for black

Added just the pyproject.toml file for the black settings and applied this new setting to the repository

Well well, the idea was just a "quick and easy" change of the formatting settings for black (set line length to 120).
But black uses only pyproject.toml for it's settings and if there is a pyproject.toml file in your package repository, you have to follow the new PEP 440 rules. There you have to define which build-system you would like to use.
And if you define the build-package, you have also to define the version number in a proper way.
This was not the case in our package, cause I used a dirty hack with a github action which changes the version attribute in the setup.cfg file with the help of some bash script magic.
So I looked for some good practises. At the moment there are six ways to define the version number.
The closest one to our workflow is the setuptools-scm workflow. It uses also the git-tag to define the version number.
Therefore I had to do some further changes at:

pyproject.toml
setup.cfg
README.md
.github/python-publish.yml


Documentation and single test execution for kuttl tests

What this PR does / why we need it:

Allows running a single integration/e2e/upgrade test with make TEST=plan-trigger e2e-test
Moved kuttle config files into test folder
Added artifacts directory so the kind-log folders don't clutter the root directory
Updated README in tests to reflect the changes


Fix PyMuPDF version as 1.20.2

Fixes https://github.com/ambuda-org/ambuda/issues/134. Installs cleanly with this fix.

Wonderful! Thank you very much.


Document use of [elasticnet] in [forest]

It requires ((varname)) in t() ... this is odd! But it works....

labeling also fails here, varname needs to be cleaned on passthrough (will strtoname() be sufficient?)


Option --filter pandoc-citeproc  causes pandoc: PATH: getEnv: does not exist (no environment variable)

This target works:
pandoc(
    name = "html",
    src = ":manual_mdx",
    from_format = "markdown",
    output = "manual.html",
    to_format = "html",
)

This target fails:
pandoc(
    name = "html",
    src = ":manual_mdx",
    from_format = "markdown",
    output = "manual.html",
    to_format = "html",
    options = ["--filter", "pandoc-citeproc"],
)

ERROR:[REDACTED]/Projects/jflex/docs/BUILD:11:1: Pandoc docs/manual.html failed (Exit 1)
pandoc: PATH: getEnv: does not exist (no environment variable)
Target //docs:html failed to build

bazel-bin: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42/execroot/main/bazel-out/k8-fastbuild/bin
bazel-genfiles: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42/execroot/main/bazel-out/k8-fastbuild/genfiles
bazel-testlogs: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42/execroot/main/bazel-out/k8-fastbuild/testlogs
character-encoding: file.encoding = ISO-8859-1, defaultCharset = ISO-8859-1
command_log: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42/command.log
committed-heap-size: 1277MB
execution_root: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42/execroot/main
gc-count: 25
gc-time: 859ms
install_base: [REDACTED]/.cache/bazel/_bazel_regisd/install/792a28b07894763eaa2bd870f8776b23
java-home: [REDACTED]/.cache/bazel/_bazel_regisd/install/792a28b07894763eaa2bd870f8776b23/_embedded_binaries/embedded_tools/jdk
java-runtime: OpenJDK Runtime Environment (build <IP_ADDRESS>+1) by Azul Systems, Inc.
java-vm: OpenJDK 64-Bit Server VM (build <IP_ADDRESS>+1, mixed mode) by Azul Systems, Inc.
max-heap-size: 15008MB
output_base: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42
output_path: [REDACTED]/.cache/bazel/_bazel_regisd/d1298cc00c93bfbf087444414b619b42/execroot/main/bazel-out
package_path: %workspace%
release: release 0.17.2
repository_cache: [REDACTED]/.cache/bazel/_bazel_regisd/cache/repos/v1
server_pid: 117524
used-heap-size: 199MB
workspace: [REDACTED]/Projects/jflex

It looks like pandoc-citeproc is part of the same tarball/zip as pandoc, so it should be relatively simple to add support for this. That said, we don't use this feature at Prodrive, meaning we'd rather have that such a feature is contributed.

Sorry for the long delay between opening this issue and merging the changes with master, but this issue is resolved now after I merged #1 with master.

No worries about delay, we are all doing this on our spare time. However, I don't think PR 1 has anything to do with this issue. I tried and I still have:

ERROR: /Users/regis/Projects/jflex/docs/BUILD:9:1: Pandoc docs/manual.html failed (Exit 1) pandoc failed: error executing command
(cd /private/var/tmp/_bazel_regis/297b145b0aac08cfb2c9cf2ea0f2624b/sandbox/darwin-sandbox/7/execroot/main &&
exec env -
external/pandoc_bin_macOS/bin/pandoc --filter pandoc-citeproc --from markdown --to html -o bazel-out/darwin-fastbuild/bin/docs/manual.html bazel-out/darwin-fastbuild/bin/docs/md/concatenated_manual.md)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox

Can you reopen the issue?

No worries about delay, we are all doing this on our spare time. However, I don't think PR 1 has anything to do with this issue. I tried and I still have:

ERROR: /Users/regis/Projects/jflex/docs/BUILD:9:1: Pandoc docs/manual.html failed (Exit 1) pandoc failed: error executing command
(cd /private/var/tmp/_bazel_regis/297b145b0aac08cfb2c9cf2ea0f2624b/sandbox/darwin-sandbox/7/execroot/main &&
exec env -
external/pandoc_bin_macOS/bin/pandoc --filter pandoc-citeproc --from markdown --to html -o bazel-out/darwin-fastbuild/bin/docs/manual.html bazel-out/darwin-fastbuild/bin/docs/md/concatenated_manual.md)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox

Can you reopen the issue?

Ah, I see. I thought this issue was related to #1 because it said regisd mentioned this issue on Oct 15, 2018, my bad!
I believe no one within our organization currently uses bazel-pandoc like this, because we did not run into this issue yet. Feel free to contribute the needed changes.

Yes, you can reassign the issue to me, but I'm not going to look into it anytime soon, tbh.