OidcClient.LoginAsync() Method not found Error

Hello,
I'm trying to implement IdentityModel.OidcClient for our WinForms solution in order to authenticate against a local IdentityServer. I followed the provided WinForm sample (https://github.com/IdentityModel/IdentityModel.OidcClient.Samples/tree/master/WinFormsWebView) and included the WebView .cs files in my project. This is what my code looks like:
var options = new IdentityModel.OidcClient.OidcClientOptions
                {
                    Authority = _authority,
                    ClientId = _clientId,
                    Scope = _scope,
                    RedirectUri = _redirectUri,

                    Browser = new WinFormsEmbeddedBrowser()
                };

IdentityModel.OidcClient.OidcClient _oidcClient = new IdentityModel.OidcClient.OidcClient(options);
var result = await _oidcClient.LoginAsync(IdentityModel.OidcClient.Browser.DisplayMode.Visible);

When I try to go past the last line with LoginAsync(), I get the following error:
Method not found: 'Void IdentityModel.Client.DiscoveryClient..ctor(System.String, System.Net.Http.HttpMessageHandler)'.
I have  enabled in my .csproj file, which generated the following redirects:
<dependentAssembly>
        <assemblyIdentity name="Microsoft.Ink" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
</dependentAssembly>

I tried to recreate the issue in a new console project with just the IdentityModel.OidcClient nuget package, WebView files for the browser, and the same code snippet as above. When I run it, the program just exits as soon as it hits the LoginAsync() part of the code without warnings or any exceptions.
I've been banging my head against the wall on this for a couple of days, so any help would be greatly appreciated! Thanks in advance.
Edit: I forgot to mention that I'm using .NET 4.6.1 and the latest stable nuget package (2.5.1).

I can confirm this error exists on the latest update on IdentityModel to 3.4.1. When using IdentityModel version 3.3.1, the exception is not raised, but when using package IdentityModel 3.4.1, the exception is raised.

Looking into it..

I just pushed a 2.6.0 preview to nuget and tried it with WinForms - works fine for me.

I haven't had a chance to try out the 2.6.0 package but after updating some of the associated projects in my solution to .NET 4.7, it seems to have done the trick. I'll give it a go with the 2.6.0 package when I have some time, but we can probably close this one for now.
Thanks.


Update generated CRD manifest file

A prior commit (0f10f39) updated the allowed codes and their
documentation but not the generated files. This commit simply includes
the modified files generated by the make process.

/assign @yiqigao217

/ok-to-test
/lgtm
/approve
/hold
@adrianludwin  @yiqigao217
feel free to remove hold

/hold cancel


Manage landscape preset configuration

I propose saving/reading the preset configuration into a JSON file, similar to how history.json works. Here is what I think the .jhipster/modules/preset.json should look like:
[
  {
    "name": "angular + spring boot",
    "date": "2024-07-09T15:10:00.521252170Z",
    "modules": [
      "init",
      "application-service-hexagonal-architecture-documentation",
      "maven-java",
      "prettier",
      "angular-core",
      "java-base",
      "maven-wrapper",
      "spring-boot",
      "spring-boot-mvc-empty",
      "logs-spy",
      "spring-boot-tomcat"
    ]
  }
]

When applying the selected preset, the properties defined by the landscape interface or default will be used, which is why they won't be stored in the json.
    "properties": {
      "packageName": "com.mycompany.myapp",
      "projectName": "JHipster Sample Application",
      "baseName": "jhipsterSampleApplication",
      "serverPort": 8080,
      "endOfLine": "lf",
      "indentSize": 2,
      "springConfigurationFormat": "yaml"
    },

@murdos and @pascalgrimaud: feel free to drop any suggestions or feedback. I am going to work on this issue 😀
I added a TO-DO/Roadmap to the following issue description:

https://github.com/jhipster/jhipster-lite/issues/9491

I'm not sure to understand what problem we're trying to solve here.
IMO a preset is something that the landscape should be able to ingest, in order to:

either have a way to manually load a serialized, pre-configured list of modules that will be selected for generation. The intent here is to save time for users that frequently generated similar projects. The serialized preset could be a json file, but it will not belong to a generated project (so I don't understand the .jhipster/modules/preset.json file).
or be the final step once users have chosen main project characteristics from a simple page that will guide them (aka the alternative page to Landscape)

It's a good idea to use a json file to load a pre-selected modules.
But this json file should belong to JHLite project, and not in .jhipster/modules/preset.json

Thank you for the feedback, @murdos and @pascalgrimaud! 😃✌
Short answer: I agree with both of you and will implement loading presets from a JSON file. Here is the JSON file I envision:
[
  {
    "name": "angular + spring boot",
    "modules": [
      "init",
      "application-service-hexagonal-architecture-documentation",
      "maven-java",
      "prettier",
      "angular-core",
      "java-base",
      "maven-wrapper",
      "spring-boot",
      "spring-boot-mvc-empty",
      "logs-spy",
      "spring-boot-tomcat"
    ]
  }
]

I removed the "date": "2024-07-09T15:10:00.521252170Z" attribute because the change commit will have this information.
Long answer:

either have a way to manually load a serialized, pre-configured list of modules that will be selected for generation. The intent here is to save time for users that frequently generated similar projects. The serialized preset could be a json file, but it will not belong to a generated project (so I don't understand the .jhipster/modules/preset.json file).

My intention was to implement something similar to the preset you defined. I had imagined using the alternative landscape interface to allow the creation of new presets via a web application. But how exactly could we add new presets to the generator using the running application? I think it's not possible.
I see another use for presets besides facilitating the generation of repeated projects. I think presets could also function as a tutorial to help users understand how the landscape screen works.


fix: docker build

Fix RM-XYZ

🧰 Changes

fix docker build error on mac

🧬 QA & Testing

Broken on production.
Working in this PR app.

This PR was released!
🚀 Changes included in v7.7.3


Set ~/.electrum/config to Local Server

I'm always careful to launch Electrum with electrum --oneserver --server localhost:50002:s as is recommended to not connect to any other server and share my xpub. However, I'd appreciate instructions for changing my ~/.electrum/config file to automatically use these options, lest I one day accidentally type "electrum" into my terminal and launch it with default settings.

You can use the setconfig option of electrum to do it for you:
electrum setconfig oneserver true
electrum setconfig server localhost:50002:s

The config file will then contain:
"oneserver": true,
"server": "localhost:50002:s",

You can use the setconfig option of electrum to do it for you:
electrum setconfig oneserver true
electrum setconfig server localhost:50002:s

The config file will then contain:
"oneserver": true,
"server": "localhost:50002:s",


Either document how to use php -S with Yii or create console wrapper for it

That's the easiest way to run PHP for a quick exploring the framework. Should be mentioned in installation.

Also could enable clean URLs by default. The problem with these is with Apache only (requires no config for nginx or php -S).

Or URLs config could be there but commented.


Allow images to be loaded into kind using 'kind.local'.

This allows ko to load images into kind nodes when the user is setting the KO_DOCKER_REPO to kind.local.
This replicates some of kind's CLI in that it talks to the container hosting the respective K8s node and loads an image into it by execing into the containerd instance of the respective container and loading the image in via stdin.

@markusthoemmes Can you update the README to call this out?  Generally this seems great.


Partials fail for .js.erb files when building

Partials called from a .js.erb file fail when building, with an error message in the generated file.
They do not fail when running the Middleman server.
This is the entire content of the generated file:
throw Error("NoMethodError: undefined method sub' for nil:NilClass\n (in /Users/jon/workspace/rivertrails/www/source/js.maps.js.erb")`
There is no error in the Middleman build log, even with --verbose.
I did a test where I removed all of my partial calls, and it generated fine. As soon as I try to use any partial, I get this failure. i.e. I created a partial that does nothing - simply includes a one-line comment, and it fails in the same way.
<%= partial :test %>
_test.js.erb
/* This is a test */

I guess this is back middleman-sprockets (3.4.2) and using /source/javascripts/denpendency.coffee.erb results in error  build/javascripts/contact/denpendency.html

@Sinetheta Not sure I follow. Can you open a new ticket with a full bug report

Sure @tdreyno, the bug is that using erb templating on an asset causes a build error. The workaround is adding the file to your ignore list.
ignore 'javascripts/denpendency.coffee.erb'

I'll file a ticket when I get a chance.


ARROW-12410: [C++][Gandiva] Implement regexp_replace function on Gandiva

@projjal needs a rebase

@praveenbingo The rebase was applied in the Pull Request.


fix(Generator): Bug async after generate

Bug async after generate
see #23
close #23

Coverage increased (+0.02%) to 98.986% when pulling 5c26583233d0835bc33f90ff284963c6a6a9bf63 on fix/Generator/Bug-async-after-generate into 95d0f41deed4c10c4ddc00de9e3fa7d9383584f0 on master.


Add Middleclick

Emulate a scroll wheel click with three finger Click or Tap on MacBook trackpad and Magic Mouse with macOS Catalina10.15 support!
Project URL
https://github.com/DaFuqtor/MiddleClick-Catalina
Category
productivity, utilities, extensions
Description

Why it should be included to Awesome macOS open source applications  (optional)
This is what I was missing using (great btw) trachpad in mac
Checklist

[ x] Edit applications.json instead of README.md.
[ x] Only one project/change is in this pull request
[x ] Screenshots(s) added if any
[ x] Has a commit from less than 2 years ago
[ x] Has a clear README in English

          1 Warning

:warning:
Found 2 link issues, a project collaborator will take care of these, thanks :)

Link issues by awesome_bot

Line
Status
Link

7566
Dupe
https://github.com/DaFuqtor/MiddleClick-Catalina

7568
302
https://github.com/DaFuqtor/MiddleClick-Catalina/raw/master/MiddleClick/Images.xcassets/AppIcon.appiconset/mouse128x128.png redirects tohttps://raw.githubusercontent.com/DaFuqtor/MiddleClick-Catalina/master/MiddleClick/Images.xcassets/AppIcon.appiconset/mouse128x128.png

  Generated by :no_entry_sign: Danger

@wojcikm Thank you! 👍


Use kthvalue instead of sort

Fixes https://github.com/neuralmagic/sparseml/issues/827
Important notes:

the proposed solution in this PR doesn't produce any memory overhead while returning the desired threshold value, so pruning experiments with huge models are runnable again and won't crash with cuda OOM
unfortunately, it's much slower than torch.sort (see https://github.com/pytorch/pytorch/issues/75599#issuecomment-1120483478) and this might affect pruners that are pruning with high frequencies
for example, running a GMP 10x per epoch over 30 epochs adds ~3 hours of run-time
torch.topk function might seem like a solution for both problems, memory with torch.sort and speed with torch.kthvalue, but unfortunately the source code of the torch.topk calls torch.sort when the given tensor is huge which gets us back to the memory issue

Given that both solutions have some drawbacks, I think that the torch.kthvalue should still be preferred as it makes experiments with large models runnable without cuda OOM issues. torch.sort is simply too memory-expensive (check examples in the linked pytorch issue: https://github.com/pytorch/pytorch/issues/77049).
To discuss how to get the best of the both worlds @bfineran  @markurtz :

torch.sort should be preferred when memory is not the issue, which happens for example when global_sparsity=False or when the model is simply small, or when the GPU device is large enough to fit the sort call
torch.kthvalue should be preferred in all other cases to make experiments runnable on a GPU (so I think it's should be a default solution)
a potential solution to get the best of both worlds: we could introduce an env variable MEMORY_BOUNDED (or something similar) which we can set when we know that the model is small enough for the device and this can trigger usage of torch.sort instead of the slower torch.kthvalue

thanks for the detailed description @eldarkurtic.  Agreed that we should try and keep sort in when memory is not the issue.  is this error occurring in a wide variety of experiments for you with a reasonable amount of GPU memory?
using an env var sounds reasonable, we could also surround the sort call in a try block and raise an exception pointing to the var, or just set the var then so future runs can use kth element instead

@bfineran unfortunately it occurs quite often. It's impossible to run pruning with global_sparsity=True on the BERT model with "only" 85M params on the 24GB RTX 3090. Also, unfortunately, the peak memory usage grows quickly with the number of params to sort, so pruning larger models (>100M params like BERT-Large/T5) is probably going to be impossible even with the simple magnitude pruner.
A few questions regarding the implementation:

Any preference on the name of the env variable?
Would this be a reasonable flow:

check the env variable and do as it says
if not set (for example a user is not aware of its existence), call sort in a try-block and if it fails print the warning message to set the env variable for future runs

@bfineran any suggestions on where to document the MEMORY_BOUNDED env variable?


Config entry has already been setup

Did you read the instructions?

[X] I have read the README, including the advanced debugging section.

The problem
Disable the integration, and enable it.
What version of this integration (ha-tahoma) has the issue?
2.7.1
What version of Home Assistant Core has the issue?
2021.9.5
Which gateway / hub do you use?
Somfy Tahoma
Device model
N/A
Device type
N/A
Additional information
No response
Relevant log output
2021-09-10 13:30:37 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry my.emaill-aol.com for binary_sensor
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/homeassistant/config_entries.py", line 304, in async_setup
    result = await component.async_setup_entry(hass, self)  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/homeassistant/components/binary_sensor/__init__.py", line 148, in async_setup_entry
    return await component.async_setup_entry(entry)
  File "/usr/local/lib/python3.9/site-packages/homeassistant/helpers/entity_component.py", line 164, in async_setup_entry
    raise ValueError("Config entry has already been setup!")
ValueError: Config entry has already been setup!
2021-09-10 13:30:37 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry<EMAIL_ADDRESS>for sensor
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/homeassistant/config_entries.py", line 304, in async_setup
    result = await component.async_setup_entry(hass, self)  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/homeassistant/components/sensor/__init__.py", line 121, in async_setup_entry
    return await component.async_setup_entry(entry)
  File "/usr/local/lib/python3.9/site-packages/homeassistant/helpers/entity_component.py", line 164, in async_setup_entry
    raise ValueError("Config entry has already been setup!")
ValueError: Config entry has already been setup!

I recall I have seen the same issue when you reload the integration.


Develop

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

What is the current behavior? (You can also link to an open issue here)

What is the new behavior (if this is a feature change)?

Other information:

:tada: This PR is included in version 0.5.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


ActiveStorage decorators

I suggest including support for decorating ActiveStorage attachment. Since ActiveStorage does not derive from ActiveRecord, this is not automatic. I currently do it by putting
Draper.setup_orm(ActiveStorage::Attached)

in a initializer. Thanks!

Thanks for the suggestion!
It would be highly appreciated if you provided some examples and use cases that demonstrate this as a generic feature a regular app could benefit of.

Sorry for the delay. I use it, for example, to generate various html snippets (for instance image tags in active_storage/blob_decorator.rb), to set to_partial_path, to return its url, etc.

@mkamensky, thanks for the reply.
While your solution work technically, I'd like to be sure the implemented pattern is quite generic to be supported out of the box.
Could you please provide close to real life usage examples for decorated objects? One of the thing of a particular interest for me is the rendering context: are they rendered in views or some other places?
Unfortunately, Draper is hard to develop now due to lack of maintainers needed to review the changes. I tried to fix as many things as possible without a major refactoring then. Meanwhile I consider Draper to be bloated with legacy stuff and cumbersome solutions that make it hard to develop while maintaining backward compatibility.
I was considering a refactoring of Draper for the next major release, but then I developed 3 gems to replace it:

Magic Lookup for class inference,
Magic Decorator to implement a generic decorator pattern based on SimpleDelegator from the standard library, and
Magic Presenter for Rails presenters.

That's why I'm asking for your use cases. I'd like to consider supporting them in my libs too.


[Archive] Character Selector, NoteSkin Selector, SideInfo, Complex Accuracy, and Watermark

What is your question?
not a question lol here's the download
Archive.zip

Omg you're the best, thank you now I have the watermark script!

This seems useful! Thanks!


Final steps / thoughts

So we're coming close to the exam :)
Is there anything you think that should be added or modified?
As discussed with @passuf, I think it would make sense to try to finish the exam summary until 16:00 (UTC+2).
@kilianrisse @Robin-des-Bois

Some possibilities:

Jensen's inequality.
Maybe some small section about general EM
Dictonary learning: how to perform step 2
What do you think?

Jensen's inequality can be seen as a tool for the proofs and derivations in this lecture. Since they are not part of the summary I don't think we need Jensen as well. Don't get me wrong, I think knowing it's details is very important for the exam, I just think that one has to understand it anyway to use it properly. Anyway, it is on the exam already (kind of), in section 5, MLE. I think this is enough and paired with general understanding of Jensen one should be able to use it.

EM...maybe, not sure what to put there though...

Yes, I think this is the same as #10

For EM I thought about something like
\subsection*{Latent variable}
We denote the latent variable indicating the component the point is sampled from by Z, which takes on values in $\{1,...,k\}$.

\subsection*{E-step: Posterior probabilities}
$\gamma_j^t(x_i) = P(Z=j|x_i, \theta_t) = \frac{P(x_i|Z=j, \theta_t) P(Z=j|\theta_t)}{Z}$

\subsection*{M-step: maximizing expected log likelihood}
$\mathbb{E}_{\gamma^t}[\log P(\mathcal{D;\theta})] =
\mathbb{E}_{\gamma^t}[\log \Pi_{i=1}^nP(x_i,z_i;\theta)] = $ \\
$\sum_{i=1}^n \mathbb{E}_{\gamma^t}[\log P(x_i,z_i;\theta)] = $ \\
$\sum_{i=1}^n \sum_{j=1}^k \gamma_j^t(x_i) \log (P(x_i|z_i=j;\theta) P(z_i=j;\theta))$ \\
$\theta_{t+1} = \underset{\theta}{\operatorname{argmax}} \mathbb{E}_{\gamma^t}[\log P(\mathcal{D;\theta})]$

I don't really think this is gonna fit on the summary and for me personally this is not of big use; I think the other stuff on the summary is more important.

I think most of these things are mentioned implicitely already


ADR-0008: Simplify workflow states based reading of TTM in TP-156

A fully fleshed out ADR 0008 as discussed previously in #49

Codecov Report

Merging #59 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #59   +/-   ##
=======================================
  Coverage   65.12%   65.12%
=======================================
  Files         101      101
  Lines        4238     4238
  Branches      258      258
=======================================
  Hits         2760     2760
  Misses       1407     1407
  Partials       71       71

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e27151...b2c26d7. Read the comment docs.

Weird that the linter is failing on files you haven't touched in this PR. Maybe try rebasing to master?

It's already on top of master :(

🎉


Pack everything in an instantiable struct instead of using global vars.

Hi!
Love what you've done, the "prepared statements" are a nice idea.
This PR removes most global vars (just kept the Pool, as I don't see a reason to not keep it that way), and allow for several instances in one application (for different output writers for example).
Tell me what you think

Coverage decreased (-0.3%) to 94.821% when pulling c72a799508cb0efa30fa3ad8d61e633e56573aa3 on hectorj:logWriter-struct into d0caa4858a6994c4ad0390a101d3470868800ad6 on ScottMansfield:master.

Coverage increased (+0.2%) to 95.357% when pulling b031ea86da2b780bf79a4d9ef4aebabf1d535191 on hectorj:logWriter-struct into d0caa4858a6994c4ad0390a101d3470868800ad6 on ScottMansfield:master.

Coverage increased (+0.2%) to 95.357% when pulling ac5d0da7d2338fc391c38c146dd5fd099905954b on hectorj:logWriter-struct into d0caa4858a6994c4ad0390a101d3470868800ad6 on ScottMansfield:master.

awesome, looks good. Thanks, Hector!


Test

Initial issue not caused by Svelte - updating to a more current version created conflicts with code - created branch from commit before update - updated what had been done after Svelte change, and merged back into main


Failing to serialise an array of structs containing primitive values

Hi
Keen to give this a try, but have hit a roadblock. I have a simple struct representing a trading tick. HS serialises and deserialises a single tick struct successfully. But when I try to serialise an array of ticks, it barfs with
"The type initializer for 'Hyper.HyperSerializer`1' threw an exception."
If this is a bug, a fix would be much appreciated. If I'm misunderstanding something please accept my apology and point out my mistake!
    public static class Scratchpad2
    {
        public struct SymbolTick
        {
            public long Timestamp;
            public double Bid;
            public double Ask;
            public int SymbolId;

            public SymbolTick(long timestamp, double bid, double ask, int symbolId)
            {
                Timestamp = timestamp;
                Bid = bid;
                Ask = ask;
                SymbolId = symbolId;
            }
        }

        internal static void Run()
        {
            int numRecords = 1000;

            var tick = new SymbolTick { Timestamp =<PHONE_NUMBER>987, Bid = 1.23456, Ask = 1.12345, SymbolId = 12 };

            var singleSpan = HyperSerializer<SymbolTick>.Serialize(tick);
            var singleSpanOut = HyperSerializer<SymbolTick>.Deserialize(singleSpan);

            Console.WriteLine($"Single span: {singleSpanOut.Ask}");

            SymbolTick[] ticks = new SymbolTick[numRecords];
            for (int i = 0; i < numRecords; i++) { ticks[i] = tick; }

            // Crashing here
            var tickArray = HyperSerializer<SymbolTick[]>.Serialize(ticks).ToArray();
            var hsOut = HyperSerializer<SymbolTick[]>.Deserialize(tickArray);
        }
    }

It appears to be a bug.  Will take a look shortly.
From: gcaplan @.>
Sent: Thursday, September 14, 2023 09:39
To: adam-dot-cohen/HyperSerializer @.>
Cc: Subscribed @.***>
Subject: [adam-dot-cohen/HyperSerializer] Failing to serialise an array of structs containing primitive values (Issue #4)
Hi
Keen to give this a try, but have hit a roadblock. I have a simple struct representing a trading tick. HS serialises and deserialises a single tick struct successfully. But when I try to serialise an array of ticks, it barfs with
"The type initializer for 'Hyper.HyperSerializer`1' threw an exception."
If this is a bug, a fix would be much appreciated. If I'm misunderstanding something please accept my apology and point out my mistake!
public static class Scratchpad2

{

    public struct SymbolTick

    {

        public long Timestamp;

        public double Bid;

        public double Ask;

        public int SymbolId;

        public SymbolTick(long timestamp, double bid, double ask, int symbolId)

        {

            Timestamp = timestamp;

            Bid = bid;

            Ask = ask;

            SymbolId = symbolId;

        }

    }

    internal static void Run()

    {

        int numRecords = 1000;

        var tick = new SymbolTick { Timestamp =<PHONE_NUMBER>987, Bid = 1.23456, Ask = 1.12345, SymbolId = 12 };

        var singleSpan = HyperSerializer<SymbolTick>.Serialize(tick);

        var singleSpanOut = HyperSerializer<SymbolTick>.Deserialize(singleSpan);

        Console.WriteLine($"Single span: {singleSpanOut.Ask}");

        SymbolTick[] ticks = new SymbolTick[numRecords];

        for (int i = 0; i < numRecords; i++) { ticks[i] = tick; }

        // Crashing here

        var tickArray = HyperSerializer<SymbolTick[]>.Serialize(ticks).ToArray();

        var hsOut = HyperSerializer<SymbolTick[]>.Deserialize(tickArray);

    }

}

—
Reply to this email directly, view it on GitHubhttps://github.com/adam-dot-cohen/HyperSerializer/issues/4, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYWHZZGCE6YRXO3HGHJFE33X2MCF5ANCNFSM6AAAAAA4YEDIH4.
You are receiving this because you are subscribed to this thread.Message ID<EMAIL_ADDRESS>

Issue has been resolved.  See the following test with your code above for reference...
Also, you can continue using HyperSerializer as you were or the new non-generic convenience type that I added HyperSerializer (only have to specify type param on Deserialize.
I released a NuGet as well.
I'll wait to for you to confirm the fix before closing this out.  LMK.
https://github.com/adam-dot-cohen/HyperSerializer/blob/700d721c645719d72c9dbc6cd341a2f200cc639c/HyperSerializer.Test/SerializerTests.cs#L289

Note, the library only serializes fields on structs.  If you happen to attempt to serialize a class, only properties with getters and setters will be serialized.  Fields will be ignored.
In related news, if you are planning to invest substantial time into algo trading - let me know.

Hi Adam
Sorry for the delay - I've been somewhat overcommitted the past few days...
I've just updated to the new release, but hit another issue. When I import the HyperSerializer namespace, I'm not seeing any of the Serialize or Deserialize methods. All I see is a namespace HyperSerializer.Dynamic. Have you changed any of the permissions? I'm new to .Net so may be doing something daft, but had no issues importing the old version.
Since I posted the issue I've found the package MemoryPack by Yoshifumi Kawai. This is a guy who has been working obsessively on serialisation for years now. MemoryPack is his latest offering and it has approaching 500 commits! I'm getting pretty good performance with it. Given that you're saying that HyperSerialiser is more of a personal experiment, do you think there's a realistic chance that it will significantly outperform Yoshifumi's package? If not, I'll step aside and leave you in peace.
Please advise.
On the wider issue of building trading apps, can I ask what you are up to?
My own project is rather specialised, so it's a bit unlikely there will be much overlap. I'll outline where I'm at and you can judge for yourself.
We are little group of siblings doing very nicely with manual trading. But we're retired and it's not a lifestyle we're enjoying, so I've dusted off my old coding chops (the first programme I wrote was on punchcards) and am hoping move into algo trading as a less stressful option.
My problems are:

I'm interested in active day-trading strategies. This means I need to backtest at tick-level resolution, and none of the consumer-level platforms or OS offerings can do that properly at reasonable speed.

I'm using price-based bars rather than time-based aggregations, and again, backtesting these has proven beyond the scope of the consumer platforms.

We're specialising in FX, and most of what's available is focused on crypto and equities.

My first attempt at a fairly general platform was becoming too complex, so I drew in my horns. I'm focusing on trading FX on the Spotware platform. This offers no less than 3 APIs and decent trading conditions, plus 9 years of good quality free tick data. Narrowing the scope has unblocked the project and it's now going fairly smoothly.
We are emphatically not quants - I trade simple price patterns. So I'm focusing on event-based backtesting with very careful attention to slippage and costs. I'm a few weeks in, and so far I've:

Written a bot to download and serialise the tick history from IC Markets. I have some billions of ticks on my local disk.

Written a loader to read in the ticks and push them into the strategy manager. It can handle portfolio testing with multiple strategies and instruments, each with their own runtime parameters. Once cached, I can run an 8-year tick-resolution backtest on the 5 most liquid FX pairs in 55 secs, with all the ticks sorted by time for accurate market replay. This is good enough for my needs. The app is processing 15,000,000 ticks per second, before any serious optimisation - literally orders of magnitude faster than my tests of other platforms. Quite impressed with C# performance.

Written modules to generate the bars and update the indicators.

Written a pipeline of reusable modules to handle trade management - eg filters, signal generators, trade sizers, risk managers etc. These can be plugged into the strategy in any sane combination.

Developed a chart to display the bars, indicators and trades using the excellent AnyChart JS library. Legitimately free for personal use!

Still to do are plugins to simulate the account and trading engine of the broker venue. We only use a few features for our simple strategies, so this isn't too daunting.
I've found it quite a challenging project - a couple of major refactors before I settled on a workable design. Github is littered with abandoned trading platforms, and I'm beginning to understand why. The key has been to take YAGNI seriously!
I haven't open-sourced it because it's too quirky and specialised and under-documented. Plus I'm new to C# so doubtless much of the code is a bit agricultural. But if anything is of interest let's discuss. Equally, if you have anything that you think would help with this rather niche venture I'm be more than grateful to hear from you.
As Boswell once wrote, I'm sorry this is so long but I don't have the time to make it shorter! Thanks for your tolerance.

Use HyperSerializer.Serialize(array) and HyperSerializer.Deserialize<SymbolTick[]>(bytes) to serialize and deserialize.
Regarding MemoryPack vs HyperSerializer, see the link below for benchmarks.   This test includes types with more strings that one would find in trading data.  As a result, you can expect 2-3x increase in the performance difference between HyperSerializer and MemoryPack for your use case.  I'll respond back on the rest of your post when I have time later this week...
https://github.com/pairbit/IT.Serialization/tree/main/IT.Serialization.Benchmarks

Sorry, the namespacing got changed by accident from Hyper to HyperSerializer.  The issue has been resolved with the namespace moved back to Hyper.  Please update to v1.3 on NuGet.

Everything working now - many thanks for fixing the bug! Much appreciated.
You're right - I am seeing a speedup on deserialize, which is a pleasant surprise given the hype around MemoryPack.
It's not quite as big as you predicted - but it's not far short of double the throughput.
Interestingly the two serialised files are exactly the same size, but it seems you're doing something different deep in the weeds. I guess I should see if I can understand your code - though as a line-of-business sort of guy this system-level stuff is a bit above my pay-grade.
My tick struct contains a DateTime, two doubles and an int. It seems that HS is optimal for this kind of numerical work.
This is quite a big deal for me, because deserialising north of a billion ticks per backtest is the major bottleneck. So I'm a very happy user!
Given the exceptional results I'm seeing you might want to promote this a bit to the community? MemoryPack has 2k stars and you are faster for the types that you cover, so clearly people would find it useful. But it's quite hard to find...

Glad to hear it worked out!
The smaller performance gap you're seeing is due to the fact that you're using stack based structs, which I suspect you're going to find problematic in the not so distant future for a variety of reasons (i.e. consider the size of the datasets you're working with and the number of copies of the data you're going to end up with as you work with the data - subsets, aggregates, display, etc.).   If and when you decide to transition to classes, the performance gap will grow substantially (I've included the benchmark stats below for your type struct vs class).
I have a codebase that may suit your needs - but would need to discuss to figure it out.  I'm a little further along than you with respect to operationalizing with respect to storage, live data pipes, etc.  I won't have time the next couple of months to take it further so I'd be open to exploring joining forces if it's mutually beneficial.
Shoot me an email if you're interested in discussing -<EMAIL_ADDRESS>1 MILLION ROUND TRIPS (SERIALIZE/DESERIALIZE)
------STRUCT------

Method
iterations
Mean
Gen0
Allocated
Alloc Ratio

HyperSerializer
1000000
13.64 ms
4000.0000
56000552 B
1.00

MemoryPack
1000000
20.43 ms
4000.0000
56000600 B
1.00

-----CLASS-------
Method | iterations |      Mean |       Gen0 |   Allocated | Alloc Ratio |
|---------------- |----------- |----------:|-----------:|------------:|------------:|
| HyperSerializer |    1000000 |  19.03 ms |  8000.0000 | 104000552 B |        1.00 |
|      MemoryPack |    1000000 | 101.61 ms |  8000.0000 | 104000600 B |        1.00 |

Hmm - you may be right about the stack issue, but it's been OK to date.
I'm storing and loading month-by month, and am very unlikely to trade more than the 10 most liquid pairs FX pairs, so it's not as though I'm having to screen hundreds of equities.
The ticks are out of scope by the end of the month - I keep some stats but not the ticks themselves. It's only the bars that are stored for the duration.
I did check that I had ample overhead. Even in a worst-case scenario I didn't hit an overflow.
If I do hit a problem it would only take a day or so to switch over, but I'll take your suggestion and test the performance implications of switching now.
I'll drop you an email about potential collaboration.

Sorry about the delay in responding to this, I've spent a lot of time on
speed, both evaluating data provider latency, code infrastructure side
(streaming via Microsoft Trill, blazing fast w/ storage via Microsoft
FASTER, etc, etc, etc.).  I'm also a ML quant so I can provide some
guidance if needed. We'd have to speak to see if collaboration makes sense.
And thanks for the kind words re: HyperSerializer!  Give the repo a start
if it continues to meet your expectations!
Adam


Fix failing builds (adaptation to PSR7 and Travis updates)

A few fixes which make the builds on Travis CI green again!

Problem 1

testCreatesGenericErrorResponseException() in Exception/RequestExceptionTest.php tries to test the creation of a GuzzleHttp\Exception\RequestException:

https://github.com/guzzle/guzzle/blob/de7437d73816cb67ed35d476784d69e010a688b5/tests/Exception/RequestExceptionTest.php#L61-L63

However, due to recent update https://github.com/guzzle/psr7/commit/c5aea30ff82a3565e6d3decd18d74257c033daf6#diff-f89246dce7e387482b2c6c584a5e7e9dR151 this test now errors:

InvalidArgumentException : Status code must be an integer value between 1xx and 5xx.

Research for problem 1

The InvalidArgumentException is correctly thrown, as this range is not valid as defined in PSR7 (https://tools.ietf.org/html/rfc7231#section-6)

RequestException is triggered to be instantiated unextended due to any exception outside of the 4xx and 5xx range:

https://github.com/guzzle/guzzle/blob/de7437d73816cb67ed35d476784d69e010a688b5/src/Exception/RequestException.php#L81-L91
Solution for problem 1

Test RequestException for 300 instead of invalid 600
Add regression test to expect InvalidArgumentException when using a 600 code response

Problem 2
https://github.com/guzzle/guzzle/blob/2297b43ad7c07a6b70a54c19e3fdd27ddd266d0b/tests/MiddlewareTest.php#L22-L34 Throws error:
InvalidArgumentException : Header value must be scalar or null but GuzzleHttp\Cookie\SetCookie provided.

Research for problem 2

Changes in https://github.com/guzzle/psr7/pull/283/commits/91eebe82a902e87fec0742a77d914202e6f66320#diff-5abca4d9d5693da46d10a54795b1192dR189 now require cookies to be scalar.
All other tests use a string for the set-cookie header
There's a perfectly fine __toString() method for casting usage

Solution for problem 2
Cast new setCookie() as a string, as it would happen in the real world.
Problem 3
https://github.com/guzzle/guzzle/blob/2297b43ad7c07a6b70a54c19e3fdd27ddd266d0b/src/RedirectMiddleware.php#L186-L190
Failed with error:
Failed asserting that two strings are identical.
Expected :'http://example.com?a=b'
Actual   :'http://:@example.com?a=b'

Research for problem 3

http://:@example.com?a=b is a valid URL, but empty UserInfo should not be added during normalizing when it's not in an original URL

Class RedirectMiddleWare attemps to clear the UserInfo when redirecting, but it's trying to do this by providing both an empty user and empty password string: https://github.com/guzzle/guzzle/blob/2297b43ad7c07a6b70a54c19e3fdd27ddd266d0b/src/RedirectMiddleware.php#L186-L189

php-fig/http-message describes that the UserInfo is to be removed when only an empty user has string been supplied: https://github.com/php-fig/http-message/blob/master/src/UriInterface.php#L199-L201

Solution to problem 3
Clear the UserInfo during a redirect by only supplying an empty string for the username, as per instruction in php-fig/http-message.
Problem 4
Travis CI builds for php 5.5 were failing (ref) because it couldn't load the PHP version:
0.00s0.02s$ phpenv global 5.5
rbenv: version `5.5' not installed
The command "phpenv global 5.5" failed and exited with 1 during .

Research for problem 4
Travis CI announced in a Apr 15, 2019 blogpost (https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment#what-does-this-mean-for-your-projects) that the default build environment changes. It instructs to use the dist: config parameter to keep using your current build image:

Repositories without an explicit dist: YAML key in their .travis.yml file will be routed to Xenial instead of Trusty.

Solution for problem 4
Configure Travis to use dist: Trusty for PHP 5.5, as was used by default prior to CI updates.

@baspeeters Wow, this is awesome! Thanks a lot.
There is one tiny change I would like to ask: only use trusty for PHP 5.4 and 5.5 as seen here.

@sagikazarmark great tip! I've factored it in :+1:

:(
https://github.com/guzzle/guzzle/pull/2327

:(
#2327

Oh, I'm sorry @gmponos! I had globally scanned through the open PR list looking at those that had passing builds. Didn't mean to bypass your PR contents.

No problem...  I know that there are no bad intentions  👍
Glad you fixed the travis though since I didn't know what was the cause of that.

Sorry @gmponos :(